Amazon Sales Channel stopped communicating with Amazon suddenly

[osclau]

Not sure if anyone else is using the Amazon Sales Channel extension from Adobe, the extension's page on Marketplace is:

https://marketplace.magento.com/magento-module-amazon.html

We are experiencing an issue where our Magento is no longer communicating with Amazon, apparently we are getting 504 response with this error message: Endpoint request timed out.

Because of this, we're not able to receive any new orders from Amazon into our Magento install, and also now cannot update order status, inventory, shipping info to Amazon.

We run a stock Magento 2.4.1, with the extension at v4.4.1 (now upgraded to v4.4.4), the same time out issue is still happening.

This also happens with another stock Magento install (v2.4.4), with extension v4.4.3.

Both are Open Source editions, so unfortunately no support contact from Adobe to ask for help.

Starting around April 25 18:41 GMT, the Amazon Sales Channel extension started writing the "Endpoint request timed out." error in its log file: channel_amazon.log

From the error response header included with the error stack trace in this log file, we're seeing something like this:

"X-Cache":["Error from cloudfront"]

"x-amzn-ErrorType": ["InternalServerErrorException"]

"statusCode": 504

Anyone knows or can confirm what is the "endpoint" that this Amazon Sales Channel extension is talking to?

It looks like the endpoint could be down, or for some other reasons is giving a time out.

And if anyone else is experiencing the same communication issue, would you know how to resolve this error or find out what is wrong?

Thanks very much in advance,

Catherine

[simssmithc3266]

Hello.

We're relieved to hear that somebody else is running into this problem. We are experiencing the exact same issue (and it began at precisely the same time, 04/25/2023 between 6:35pm UTC and 6:40pm UTC.)

We've tried a few things in order to troubleshoot this. We've upgraded the Amazon Sales Channel extension to the latest version (4.4.4) and we've attempted to remove and recreate our Amazon store within Magento. The problem persists and the behavior is exactly the same.

Here's what we believe is happening with the Amazon Sales Channel extension.

1. Magento's cron jobs fire and trigger the extension's various routines (retrieve updates aka logs from Amazon, push changes from Magento to Amazon, retrieve Amazon merchant information, etc.)
2. The extension makes requests to api.magento.com (via the Commerce Services Connector extension) to perform these tasks.
3. Requests made to api.magento.com are routed through AWS's API Gateway. (This is really common for any web services or apps that are hosted in AWS.)
4. The AWS API Gateway manages the requests it receives and makes sure they get routed to where they need to go. It enforces a strict 30 second time limit. Once it receives a request, it must obtain a response from whatever backend service it connects to. If that time limit is exceeded, the AWS API Gateway returns that "Endpoint request timed out" error with a 504 status code.

So to us, it seems that whatever services are actually responsible for processing the requests our stores make to api.magento.com (AWS Lambda functions, EC2 instances, or something else) are experiencing some issue that is preventing them from returning a response within the time allotted by the AWS API Gateway.

Unfortunately, we don't have any visibility into the situation beyond that. We have emailed channels@adobe.com our findings hoping they can provide some guidance.

Below is a sample of Guzzle debugging output from one of the extension's requests made to api.magento.com:

* Found bundle for host api.magento.com: 0x563cc6ff0c40 [serially]
* Re-using existing connection! (#0) with host api.magento.com
* Connected to api.magento.com (18.67.39.104) port 443 (#0)
> POST /channels/amazon/na/v2/merchant/REDACTED/command HTTP/1.1
Host: api.magento.com
magento-api-key: REDACTED
User-Agent: Magento Services Connector (Magento: Community 2.4.2)
X-AMAZON-CLIENT-VERSION: 4.4.1
Content-Type: application/json
Content-Length: 3075

* upload completely sent off: 3075 out of 3075 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 504 Gateway Timeout
< Content-Type: application/json
< Content-Length: 41
< Connection: keep-alive
< Date: Fri, 28 Apr 2023 15:56:32 GMT
< x-amzn-RequestId: 9ce217fa-ee96-4df0-b019-31804a0debf3
< x-amzn-ErrorType: InternalServerErrorException
< x-amz-apigw-id: EGHnBG1cIAMFyIw=
< X-Amzn-Trace-Id: Root=1-644bec93-4e39a9e74277cfa775098367
< X-Cache: Error from cloudfront
< Via: 1.1 04fa8a9e73b27e301fb4b6d36f313186.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: YTO50-P2
< X-Amz-Cf-Id: j-KA6a5DRNUOiTselAkByWXRFfolyKhziodOx24mfquvWrYA-OVX5g==
<
* Connection #0 to host api.magento.com left intact

[miketankprd1be]

Same issue as well! Stopped on the 25th, trying to get a resolution with Amazon and Adobe asap.

[ramanathan1dc8]

We are also facing the same issue, did you get any solution for this issue?

[simssmithc3266]

Unfortunately the issue is still ongoing and we have not received any response from Adobe.

We did originally receive a prompt response from Amazon Seller Support (although we knew there was little chance they'd be able to provide a resolution.)

---

Hello,

Greetings from Amazon,

My name is Rick, from the Selling Partner API Developer Support (SPADS) team.

Regarding your concern about requests timing out from the server, I have reviewed your account, confirmed it is professional, and in complete set up process.

From our end we don't see anything preventing you from using any application. Your applications are correctly authorized, however if you experience errors while using them, we will not be able to troubleshot, since errors occurring on a third party application are outside our scope, as we did not develop the application.

We will be resolving this case since there is no existing issue with your Selling on Amazon Account.

Please refer further inquiries about server timeouts to your developer.

If you wish to receive further Support, please reply to this email with the steps you have taken to resolve the issue that have not worked and a screenshot of the conversation you had with the Developer so that we can troubleshoot further.

The Developer may also contact us directly, just inform them to include the case ID REDACTED when opening the case to have a better reference.

[simssmithc3266]

It appears that as of 3:06pm UTC today, the Amazon Sales Channel integration has resumed working.

[osclau]

Can confirm. Amazon Sales Channel integration between Magento and Amazon appears to resume working as of today (May 4th) 15:06 GMT for us as well.