429 Error with SW API

[aaron-pham]

It appears that SW has added a request limiter or something, I had two check-ins fail today.

2022-09-13T18:10:09.748Z	a52a7626-6b2b-5e56-a22f-2dbca386fb0f	INFO	Checking in at September 13, 2022, 6:10:09 PM UTC
2022-09-13T18:10:10.183Z	8152a56e-485c-52dd-9b59-fef480f49c0c	ERROR	Invoke Error 	
{
    "errorType": "HTTPError",
    "errorMessage": "Response code 429 (Too Many Requests)",
    "code": "ERR_NON_2XX_3XX_RESPONSE",
    "name": "HTTPError",
    "timings": {
        "start": 1663092609742,
        "socket": 1663092609743,
        "lookup": 1663092609743,
        "connect": 1663092609752,
        "secureConnect": 1663092609762,
        "upload": 1663092609762,
        "response": 1663092610135,
        "end": 1663092610135,
        "phases": {
            "wait": 1,
            "dns": 0,
            "tcp": 9,
            "tls": 10,
            "request": 0,
            "firstByte": 373,
            "download": 0,
            "total": 393
        }
    },
    "stack": [
        "HTTPError: Response code 429 (Too Many Requests)",
        "    at Request.<anonymous> (/var/task/node_modules/got/dist/source/as-promise/index.js:117:42)",
        "    at processTicksAndRejections (internal/process/task_queues.js:95:5)"
    ]
}

2022-09-13T18:10:10.016Z	a52a7626-6b2b-5e56-a22f-2dbca386fb0f	ERROR	Invoke Error 	
{
    "errorType": "HTTPError",
    "errorMessage": "Response code 429 (Too Many Requests)",
    "code": "ERR_NON_2XX_3XX_RESPONSE",
    "name": "HTTPError",
    "timings": {
        "start": 1663092609751,
        "socket": 1663092609752,
        "lookup": 1663092609752,
        "connect": 1663092609759,
        "secureConnect": 1663092609767,
        "upload": 1663092609767,
        "response": 1663092609962,
        "end": 1663092609962,
        "phases": {
            "wait": 1,
            "dns": 0,
            "tcp": 7,
            "tls": 8,
            "request": 0,
            "firstByte": 195,
            "download": 0,
            "total": 211
        }
    },
    "stack": [
        "HTTPError: Response code 429 (Too Many Requests)",
        "    at Request.<anonymous> (/var/task/node_modules/got/dist/source/as-promise/index.js:117:42)",
        "    at runMicrotasks (<anonymous>)",
        "    at processTicksAndRejections (internal/process/task_queues.js:95:5)"
    ]
}

I had about 50~ or so request happen before the server responded and I got the error. I think a possible fix may be to not start checking in 5 seconds before and maybe even doing the check-ins in longer intervals. In all my check-ins history, it's never checked-in before the open time.

On average, it's usually about 8 seconds after check-in is suppose to open, but as short as 4 seconds after.

[aaron-pham]

Really busy at the moment, so I think I'm going to try a naïve solution and see if that fixes anything. I'm going to start check-ins at the actual time instead of 5 seconds earlier, and move the milliseconds between request to .5 seconds which I think is reasonable for someone to click.

[Rezer]

I just ran into the same issue, I don't think this is a rate limiting issue. I logged the full headers and body being provided to the checkin page, and when I put the same information into postman I get this response:

{
    "code": 429999999,
    "message": "Error.",
    "messageKey": "ERROR",
    "httpStatusCode": "BAD_REQUEST",
    "requestId": "",
    "infoList": []
   }

However, if I then add a User-Agent string (doesn't seem to matter what), then the checkin succeeds and it returns the usual checkin confirmation page. I'm unfamiliar with the specifics of how Got.get works, but would I be right in assuming that it doesn't add a User-Agent string to the headers unless explicitly told to do so?

[aaron-pham]

Are you adding the user agent string for getting the headers or the actual checkin?

[Rezer]

I was adding the user agent for the actual checkin, but after banging my head against this one all day I'm at a loss as to what the actual problem is. Postman reports the above error without the user agent string, but from what I understand Got actual does insert a default user agent, so it probably wasn't the issue. However, no matter what I try, I'm always getting the same error from the AWS logs. I can often copy the "advancedHeaders" out of the log file along with the checkin json body and post it verbatim in postman and it works fine there, even if I spam the submit button. However, sometimes using the same method from a different run on AWS without even changing any code will also fail in postman. I tried different permutations of the following:

• Including 'User-Agent' explicitly
• Removing 'Content-Type' and relying on Got including it automatically
• Including 'user-agent' and 'content-type' in case Got was adding a duplicate with the different casing (who knows?)
• Adding a 10 second delay between each of the 40 iterations of the checkin json body retrieval (since it's already in the checkin window only one is ever submitted) and then an additional 10 seconds before it posts the actual checkin query, eliminating it being an actual rate limit error
• Retrying the checkin a total of 10 times with the 10 second delay

Nothing but 429 errors all the way down. I'm fairly confident SW is doing something to detect automated requests, but the puppeteer portion always works fine. I don't know enough to try debugging exactly what the Got post request looks like, all I can do is log the inputs before calling it and compare it to what postman sends, and there's nothing obvious that jumps out at me other than Got using all lowercase headers and postman retaining capital letters (postman still works if changed to lowercase, btw).

I did notice one time one of the headers came back named EE30zvQLWf-a0 along with the usual EE30zvQLWf-a, and it wasn't caught since the 0 doesn't match the regex. However this is definitely the exception and not the rule. Out of a couple dozen runs I only saw this once.

[Rezer]

So just to further confuse matters...

I went ahead and pulled the headers and the request body from the mobile checkin site manually by just putting the details in a browser and hardcoded those values in handle-scheduled-checkin.ts. I left in all the regular retrieval of the headers and the json request body, the only difference is I provided the hardcoded values to SwClient.loadJsonPage<CheckIn.CheckinSuccessfulResponse>, and....it worked fine. I pared down the headers and wound up with the following being all that's needed for success:

• x-api-key
• x-channel-id
• x-user-experience-id
• ee30zvqlwf-f
• ee30zvqlwf-b
• ee30zvqlwf-c
• ee30zvqlwf-d
• ee30zvqlwf-z
• ee30zvqlwf-a

I guess the question now is why don't the values retrieved programmatically work when they all seem to be perfectly valid. Maybe something's happening earlier in the process that is somehow poisoning one or more of the values? But then why does postman sometimes work with the same data...

Edit: Behavior seems to be the same with just hardcoded headers while letting it retrieve the request body, it works fine (well, until the headers expire in 30 minutes or so). Also I've noticed ee30zvqlwf-a is consistently significantly longer when pulled out of a desktop browser than what's returned by puppeteer.

[aaron-pham]

@Rezer I suspect they might be doing some smart bot detection and rejecting the request that come from AWS. They've done similar things in flight searches with ips coming from heroku and digitalocean.

Also, there's this repo that appears to have the same issue: byalextran/southwest-checkin#13

[Rezer]

That's the thing though, it's not as simple as just rejecting all requests from AWS. When I copied and pasted valid headers into the AWS app it checks in successfully. It appears as though the ee30zvqlwf headers are being used as the bot detection, and everything appears to behave normally until final checkin. Trying to figure out what's going on in their mangled javascript to generate those headers is way above my level of ability, but I suspect the ee30zvqlwf-a value consists of a bunch of encoded browser metadata that may allow them to determine that the browser that originally created them is either headless or coming from AWS or wherever else they may be rejecting on the final step.

[swtools0]

@aaron-pham @Rezer This is really great detective work, y'all. How can I help diagnose the issue?