[Feature Request] - Option to pass an custom TLS JA3

[vikasg603]

Hey team, It's being almost an year i have been using an HTTP Toolkit and quite recently @pimterry written a blog on Fighting TLS fingerprinting with Node.js

This worked great till now, But now it seems Cloudflare started whitelisting the TLS cipher for different browsers and starts blocking unknown TLS fingerprint.

This can be reproduced from https://flyairseoul.com/CW/ko/main.do? (You have to use south korean proxy to verify it)
Just visit the website and try searching any flight, When using the HTTP toolkit with korean proxy, The website will start showing the cloudflare challenge, But if same tried with chrome directly, It will not show the cloudflare challenge.

This creates an issue while doing the HTTP debugging.

Solution in my mind: As such NodeJS doesn't support changing the TLS JA3 directly and this creates an issue, since the whole HTTPToolkit server and the proxies are implemented in NodeJS.

I really feel, we have to somehow shift the proxy implementation to low level language like GoLang for more better configuration.

What do you guys think?

[pimterry]

It's an interesting idea, and I'm definitely open to improvements here. That said, there's a few options today anyway:

• In many cases, the fingerprinting used here isn't actually based on TLS now (some browsers such as Chrome now intentionally randomize their fingerprint to try to make this more difficult) and it can be worth changing other settings. Forcing HTTP/2 (in the Pro settings) can help with this in many cases, for example.
• It should be possible to build a TLS-fingerprint-configuring pure-TLS proxy (no HTTP parsing or anything, just tunnelling TLS connections and passing data directly) as a standalone project, and then you could send HTTP Toolkit traffic (and any other tool you like) via this proxy.

I really feel, we have to somehow shift the proxy implementation to low level language like GoLang for more better configuration.

This is an interesting option, as golang (via https://github.com/refraction-networking/utls/) does seem to be the only language where TLS is currently sufficiently configurable to do this.

Rewriting the entire proxy in go though is a project that would take years of full time development. Node is actually a really really good language for async IO & higher-level HTTP processing, and rebuilding everything is going to be significantly more difficult in Go.

I think you could combine the two though. Currently HTTP Toolkit's internal TLS handling for upstream connections is tiny and very focused in a single place: here. That makeRequest method uses Node's HTTP/HTTPS/HTTP2 APIs to make the HTTP & TLS (when required) connection upstream (there's various TLS options there).

It's possible to make the TLS connection independently, and to pass it (as a Node.js duplex stream) to that same method call - so the TLS is handled elsewhere, but Node keeps doing HTTP exactly the same way on top. If somebody built a native node add-on module with Go (they're normally written in C++, but Go seems possible too: https://github.com/akshayganeshen/napi-go) that wrapped UTLS and exposed it as a Node.js stream, then we could use that to make the TLS connection in the code above. That would effectively run all HTTP Toolkit's HTTP code as-is on top of Go's TLS, and in theory everything would work immediately. Interesting!

Unfortunately, I don't know Go, and I don't have a lot of time to explore this myself any time soon. If anybody does though, get in touch - I would be happy to support this and help with any guideance you like. The resulting module would be useful for HTTP Toolkit but it'd also be usable by any Node.js project, so all of a sudden all Node.js code would be able to configure the TLS fingerprints freely, which would be pretty cool!

[pimterry]

I've done a bit more digging into how utls actually works. In effect I think it's just directly modifying the client hello, and the underlying crypto/tls connection state (so that the related signed hashes that are included later on in the connection match those changes automatically).

As an approach it's OK, but there are limitations - notably it requires telling the server that the client supports features it doesn't actually support, which can cause problems if the server actually uses that.

It might be possible to do something very similar directly on top of OpenSSL ourselves though. Modifying the bytes is definitely easy - the challenging part is stopping OpenSSL breaking afterwards. That approach would be nicer though, since it would let us do this entirely within Node.js's existing TLS setup (either just via OpenSSL improvements like openssl/openssl#19220 & openssl/openssl#18790, or with a native add-on that reconfigures & tweaks things).

To explore that, I've tried comparing the latest Chrome fingerprint to HTTP Toolkit, ignoring ordering:

• HTTP Toolkit sends a TLS_EMPTY_RENEGOTIATION_INFO_SCSV cipher that Chrome doesn't (OpenSSL always does this - it's a valid and more-compatible alternative to an empty renegotiate extension, but it's noticeably different from all browsers).
• Chrome has some extra extensions we don't:
  • 0x5 status_request - this is an OCSP request. Not sure why it isn't sent, but it's safe to send blindly and ignore the response. This is supported actually I think...
  • 0xff01 renegotiation info extension - when sending this in the empty case, it's equivalent to our extra cipher, so we could include this blindly in that case anyway (and in the non-empty case, I think OpenSSL probably supports this anyway, and that wouldn't be used for fingerprinting anyway since it can vary)
  • 0x10 ALPN - we notably don't send this for HTTP/1 connections, which we use by default, but we could.
  • 0x4469 ALPS HTTP/2 settings - Chrome sends the HTTP/2 settings frame in the TLS handshake to speed up connections. Not clear if OpenSSL supports this. We could probably blindly copy this upstream, since we'll always follow up with our own SETTINGS at the start of the connection anyway, and that will override the TLS settings.
  • 0xfe0d encrypted client hello (ECH). We don't support this, and if we forward it and the server actually replies to it we'll probably have problems (I'm not totally sure but seems v likely). I think few servers support it though, and OpenSSL is planning to add support soon, so this might not be a big issue in practice.
  • 0x12 certificate transparency request - we can just blindly forward this and ignore I think. It's supported in OpenSSL but not Node AFAICT.
  • 0x1b compressed certificates - if we do forward this and the server does send us a compressed cert then the connection would fail. Hard to know without testing, but I'd hope that's not a big problem in practice. OpenSSL 3.2+ supports this but Node uses 3.0.
  • 0x0a0a GREASE - this does nothing, by design, so we could easily forward it and ignore any response
• Chrome sends more ECH curves (29-23-24-25-256-257 vs 29-23-24) - I think OpenSSL can send all of these, so we could do that with mild reconfiguration
• Chrome sends fewer ECH curve formats (0 vs 0-1-2) - I'm not sure we can configure this in OpenSSL easily, but if we rewrite the client hello somehow then this will work fine.

Not something I can immediately test out, but if anybody is comfortable messing around with C & Node.js add-ons, it might be an interesting project to test out. Let me know if so and I can probably come up with some pointers.