-
Notifications
You must be signed in to change notification settings - Fork 257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CONC-654] Stop leaking client identifying information to the server before the TLS handshake #227
base: 3.3
Are you sure you want to change the base?
Conversation
… completion MariaDB Connector/C does not distinguish [application-layer error packets](https://mariadb.com/kb/en/err_packet) that it receives prior to TLS handshake completion from those that it receives immediately after. (A trivially modified server built from dlenski/mariadb-server@demonstration_of_CONC-648_vulnerability can easily be used to demonstrate this.) Pre-TLS error packet received from this trivially modified server. This packet should NOT be trusted to actually originate from the server: $ mariadb --ssl --ssl-verify-server-cert -uUsername -pVerySecretPassword -h CONC-648.vuln.demo.server.com ERROR 1815 (HY000): Internal error: Client will accept this error as genuine even if running with --ssl --ssl-verify-server-cert, and even though this error is sent in plaintext PRIOR TO TLS HANDSHAKE. Post-(TLS handshake) error packet received from a normal MariaDB server upon an attempt to connect with incorrect credentials. This error packet CAN be trusted to actually originate from the server, assuming transitive trust in the TLS protocol implementation and PKI-based certificate validation: $ mariadb --ssl --ssl-verify-server-cert -uUsername -pWrongPassword -h $NORMAL_MARIADB10.6.14_SERVER ERROR 1045 (28000): Access denied for user 'Username'@'A.B.C.D' (using password: YES) This client behavior opens up MariaDB Connector/C clients to an extremely straightforward [downgrade attack](https://en.wikipedia.org/wiki/Downgrade_attack). An on-path or pervasive attacker can inject errors into MariaDB client→server connections that are intended to be protected by TLS, and the client has no clear mechanism to distinguish such errors from errors that actually come from the server. An attacker could easily use this to DOS a client, or even influence its behavior. For example, consider a client application which is configured… 1. To use TLS with server certificate validation (`--ssl --ssl-verify-server-cert`), and 2. To wait for a back-off period and then *retry* connection attempts if the server responds with `ER_CON_COUNT_ERROR` ("Too many connections") from the server, and 3. To give up and shut down if its connection attempts fail with `ER_ACCESS_DENIED_ERROR` ("Access denied for user"), on the assumption that this is due to an incorrect or expired password, and cannot be resolved without human intervention. An attacker could completely disable the retry mechanism of this application by intercepting connection attempts and replying with `ER_ACCESS_DENIED_ERROR` packets. This patch modifies MariaDB Connector/C so that if the client is configured to use TLS, error packets received prior to the completion of the TLS handshake are untrusted, and are changed to a generic `CR_CONNECTION_ERROR`. $ mariadb --ssl --ssl-verify-server-cert -uUsername -pVerySecretPassword -h CONC-648.vuln.demo.server.com ERROR 2002 (HY000): Received error packet before completion of TLS handshake. Suppressing its details so that the client cannot vary its behavior based on this UNTRUSTED input. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
…s requested The 2015 commit mariadb-corporation@23895fbd4#diff-4339ae6506ef1fb201f6f836085257e72c191d2b4498df507d499fc30d891005 was described as "Fixed gnutls support", which is an extremely incomplete description of what it actually does. In that commit, the Connector/C client authentication plugin was modified to abort following the initial server greeting packet if the client has requested a secure transport (`mariadb --ssl`), but the server does not advertise support for SSL/TLS. However, there's a crucial caveat here: If the client has requested secure transport (`mariadb --ssl`) but has not requested verification of the server's TLS certificate (`mariadb --ssl --ssl-verify-server-cert`), then the client will silently accept an unencrypted connection, without even printing a diagnostic message. Thus, any such client is susceptible to a trivial downgrade to an unencrypted connection by an on-path attacker who simply flips the TLS/SSL capability bit in the advertised server capabilities in the greeting packet. The entire design of MariaDB's TLS support is severely flawed in terms of user expectations surrounding secure-by-default connections: the `--ssl` option SHOULD imply `--ssl-verify-server-cert` BY DEFAULT; if a client actually wants a TLS connection that's susceptible to a trivial MITM'ed by any pervasive or on-path attacker, that should be an exceptional case (e.g. `--insecure-ssl-without-server-cert-verification`) rather than the default. Even without resolving the issue of no default verification of server certificates, the issue of silent downgrade to unencrypted connections can and should be resolved. Backwards compatibility: This is an INTENTIONAL backwards-incompatible change to prevent clients from being silently downgraded to unencrypted connections, when they've specified `--ssl` and thus clearly indicated that they do not want unencrypted connections. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
…_connect Two reasons: 1. Reduction of attack surface As soon as the client receives the server's capability flags, it knows whether the server supports TLS/SSL. If the server does not support TLS/SSL, but the client expects and requires it, the client should immediately abort at this point in order to truncate any code paths by which it could inadvertently continue to communicate without TLS/SSL. 2. Separation of concerns Whether or not the server supports TLS/SSL encryption at the transport layer (TLS stands for TRANSPORT-layer security) is a logically separate issue from what APPLICATION-layer authentication modes the client and server support or should use. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
907e73d
to
1b38393
Compare
plugins/auth/my_auth.c
Outdated
{ | ||
my_set_error(mysql, CR_SERVER_LOST, SQLSTATE_UNKNOWN, | ||
ER(CR_SERVER_LOST_EXTENDED), | ||
"sending connection information to server", | ||
"sending dummy client reply packet to server", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's not a dummy packet unless server_allows_dummy_packet
is set, better write something like
server_allows_dummy_packet ? "sending dummy client reply packet to server"
: "sending connection information to server"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even in the status quo situation where this packet contains "too much information", it still doesn't contain complete "connection information."
-
Pre-(TLS handshake) packet is called SSLRequest Packet in the protocol docs: https://mariadb.com/kb/en/connection/#sslrequest-packet
Note that it doesn't contain the CLIENT_CONNECT_ATTRS or CLIENT_PLUGIN_AUTH information
-
Post-(TLS handshake) packet sent by the client is called Handshake Response Packet in the protocol docs: https://mariadb.com/kb/en/connection/#handshake-response-packet
Does contain that information.
Part of the problem here is that the naming of these exchanges is inconsistent both internally to the code and with the documentation, so it's hard to know what the "best" name to call any of it is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤷♂️ okay
First, I only looked at the CONC-654 commit, not at other commits that this PR incorporates. I think this looks pretty good. One conceptual question, though. Should it be a very specific flag "client can send dummy packet" or should it generally mean, like, "secure TLS mode"? Specifically if the server sets SECURE_SSL_MODE capability and the client replies with a 2-byte packet, equal to CLIENT_SSL, then the client will ignore all unencrypted packets from the server (or replace with a generic error, whatever). Also if there're more problems in SSL, they can all be fixed under the same flag |
@vuvova wrote:
Right, that makes sense. Those also exist in separate PRs, fine to consider them separately.
Yeah, I thought about this as well. The problem is that we have no idea what other TLS-related bugs are still remaining; unfortunately, I won't be surprised to find more of them, given the very ad-hoc nature of the TLS setup code. If we call this flag All that we can really guarantee at the time of introducing this flag is that the server has fixed this one, specific TLS bug MDEV-31585 "Server improperly requires client capability bits to be sent IN PLAINTEXT prior to TLS handshake". Perhaps the name could be improved on though. Maybe something like |
Another thought. In all the discussions about protocol weaknesses you were considering a model with the active MitM attacker. But if there's an active MitM attacker, he can turn this flag off, cannot he? And the client will leak again. Should it be a concern? |
@vuvova wrote:
Yes, absolutely! That's a straightforward downgrade attack, akin to POODLE wherein MITM attackers downgrade clients who want to use TLSv1.x to SSLv3 by modifying the handshake packets to tell the client that the server only supports SSLv3. That's why we need to make a client-side change which is backwards-incompatible by default in order to fix this. In order not to be susceptible to this downgrade attack, the client would need to send the send the stripped-down "dummy" pre-(TLS handshake) unconditionally, even if the server doesn't actually advertise support for it (as I wrote in MariaDB/server#2684 (comment)). However, but we can provide a If you agree with this, I'll add those further changes to this PR; I haven't done it so far, since I'm trying not to combine too many changes all at once in here. |
This would bifurcate the client ecosystem into MySQL clients and MariaDB clients. I'm the author of MySqlConnector, a .NET client library that supports MySQL, MariaDB, Percona Server, Amazon Aurora, Google Cloud SQL, Azure Database for MySQL, and anything else that can speak the MySQL protocol. There are probably dozens of people like me who maintain the OSS Python, Rust, Go, JavaScript, Perl, Haskell, etc. etc. drivers. (Some of these are FFI wrappers around the mysql C library, but many are full reimplementations in that language/runtime that avoid a native code dependency.) These OSS authors would have the following choices:
I don't collect analytics on server type, but I would guess it's (at least) 90% MySQL, 10% MariaDB. Unless the server-side change first gets accepted into MySQL Server Community Edition, and then also into the majority of cloud-hosted MySQL servers (some of which are proprietary forks of MySQL Server or reimplementations of the protocol), I don't think any client developer will choose option (3) and break their library for the vast majority of their users. It will probably be option (1) and then maybe option (2) if Oracle picks this up for MySQL Server or enough MariaDB users (of each library) specifically ask for it (or if MariaDB Server makes a breaking change that stops all those clients connecting). Maybe MariaDB will solve this in their server and client implementation, but for now I don't see this achieving widespread adoption across the entire MySQL ecosystem. |
So, let's consider three cases:
In the first case what a client can do to reduce the exposure?
all this is safe and will work with any server implementation. In the second case:
In the third case
This approach doesn't seem to break anything and only leaks client capabilities in case of the old server or active MitM. |
@bgrainger wrote:
I would prefer to describe this as "bifurcate the client ecosystem into clients that leak too much information before the TLS handshake, and clients that don't". There's pretty clear evidence that MySQL/Oracle have long been aware of the fact that the clients were sending "too much information" before the TLS handshake… … and yet the MySQL server implementation still requires the client to send a full 4 bytes of capabilities in the pre-TLS client intro packet. Just like MariaDB, it will not read the capability fields from the post-TLS client intro packet. So MySQL is equally vulnerable to fingerprinting clients based on the capability bits sent in the pre-TLS client intro packet, and equally incapable of fixing this on the client side in a way that's backwards-compatible with existing servers. |
@vuvova wrote:
Unfortunately, this is not quite true; the MySQL server does require the charset in the pre-TLS client packet to precisely match* the charset value in the post-TLS client packet, otherwise the connection aborts: The comment a few lines above claims that they "don't parse" the fields in the post-TLS client packet, but this is not quite correct: There appears to be an escape hatch in the MySQL code, which is that if the pre-TLS client packet is only 4 bytes long (= capability fields only) and has … but if you follow the following lines carefully, you'll see that they've completely botched it:
tl;dr:
|
@vuvova wrote:
If we take an approach that doesn't defend against an active MiTM downgrading the connection, we aren't appreciably improving the security of M/M client-server connections on the Internet. Active MiTM is the threat model that everyone has worried about, because we've known that it is actively being employed by multiple governments for at least a decade:
Active MiTM is the reason that the only remedy for POODLE was to disable SSLv2/3 entirely; for the same reason, the only real fix for this issue is to make the client refuse to downgrade, even if the server says it has to downgrade. |
@dlenski wrote:
See above, it still detects active MitM post-factum, after the TLS was established, and…
… the client can abort the connection at once, so that it couldn't be statefully tracked. Downgrade won't be allowed after MDEV-28634, MitM or not. |
@dlenski wrote:
Ouch. Indeed, sorry. I only checked the oldest MySQL from 2000 (when SSL support was added) and the recent MariaDB. But apparently at some point Okay, then. MariaDB server
MariaDB connectors
This way,
There are trade-offs, particularly in the "new client old server" configuration, but everything will still work. |
Do new clients also need to eliminate the "Preferred" and "Required" SSL Modes (to use MySQL terminology) and only support "VerifyCA"? Otherwise, a malicious proxy/MitM could establish a TLS connection (using a self-signed cert) with the new client and downgrade the MariaDB version that it sends, tricking the client into thinking it's talking to an old MariaDB server and activating the "old server" code path that doesn't terminate the connection. This could allow all data to be logged as it's proxied, even though it's sent over a "secure" connection. |
first, by "new client" I meant "new releases of MariaDB Connector/C, Connector/J, etc", because I only described what we in MariaDB can do to secure the connection and without breaking other clients and servers. Of course, I cannot speak for what MySqlConnector will do. Although we might create a pull request for you at some point. Now, about "modes", I suggested an approach in MDEV-28634, which is,
This looks mostly backward compatible. But doesn't have a "required" mode (which is "I require ssl but I don't care who will decrypt the data, server or someone in the middle") |
@vuvova wrote:
To which I replied…
To which @bgrainger replied…
——— It turns out there is a good way around this (MariaDB/server#2684 (comment)). Summary:
This will largely blunk the MITM attack, while preserving backwards-compatibility-by-default with servers that require the client to keep leaking. What do you think? |
This sounds good to me. It occurs to me that because this will trigger a new code path both in both client and server (client: if "send dummy" bit is set, send the dummy handshake; server: if client sends the dummy handshake, load everything from the post-TLS handshake), that we could take the opportunity to fix anything else that's broken in the handshake. (Obviously there needs to be some overlap in the location of the capabilities bits flags, to handle the case where MitM is changing the packets, and the server doesn't know which handshake to expect, but there's a lot of leeway beyond that.) Nothing's jumping out at me immediately, but it seemed worth bringing up in case anyone else had ideas.
I'd suggest that this whole path be named "SSL Request V2", not "dummy handshake". (This is not going as far as calling it "SECURE_SSL_MODE" as suggested above, but simply denoting that this is indeed a new version of the SSL Handshake packet.)
|
@bgrainger wrote:
I'm not aware of any other desired changes that could fit neatly into the scope of this change, but perhaps @vuvova and other core devs are.
👍, good idea. My naming was focusing too much on the technical details, rather than the overall purpose. I'm on board. |
In the latest updates, I've changed the naming to In MariaDB/server@eb5c26a and cf8638c, I also moved the MySQL is affected by these vulnerabilities as well, and if we want to enable a better TLS handshake for all client/server combinations, this capability bit needs to be sent between all client/server combinations. |
Oracle has already defined a meaning for that bit: https://github.com/mysql/mysql-server/blob/ea7087d885006918ad54458e7aad215b1650312c/include/mysql_com.h#L734-L747 |
Thanks @bgrainger. 😭 … looks like we're all out of the low 32 capability bits which could plausibly be shared between Oracle/MySQL and MariaDB. Well, we can just pop the top commit off here and keep this a MariaDB-only fix, huh? 😬 |
Strictly speaking, there's no reason MySQL couldn't use That could theoretically be changed to:
(Or you could change the MariaDB flag to be |
I'd still suggest a simpler approach. Something like:
This works for all clients and all servers. Doesn't show Additionally we can make the server to prefer capability bits from after the TLS. But the client still has to send them before TLS too, see above, for non-MariaDB servers and to have the same behavior for different servers. |
@vuvova wrote:
Hmmm. This seems like a more complicated approach to me. In particular:
What does "prefer" mean, exactly? What happens if the client sends a Requiring a client to send these twice means allowing them to send them inconsistently. Inevitably, some clients will send them inconsistently, and it will be very difficult to discover the inconsistencies and decide what to do with them. What's currently implemented here and in MariaDB/server#2684 seems more futureproof and less error-prone to me:
|
@bgrainger wrote:
Yep, agreed. Okay, I've popped the top commit from here and MariaDB/server#2684. If/when Oracle/MySQL want to start doing the "v2" SSL handshake, they can start looking at bit 37 🥳 |
@dlenski wrote:
First, it was "additionally", not part of the proposal, more like an idea that needs to be discussed.
In the "simpler approach" none of this happens, because the client replies with the packet exactly the same as before. Like any other old or a third-party client does. To all servers with any capabilities. |
…before the TLS handshake The server implementation here was incorrect as well, unnecessarily reading—and TRUSTING—client identifying information sent before the TLS handshake. That's in MDEV-31585. As a result of the server's mishandling of this information, it's not possible for the client to fix this in a way that's backwards-compatible with old servers. We rely on the server sending a capability bit to indicate that the server-side bug has been fixed: /* This capability is set if: * * - The CLIENT knows how to send a truncated 2-byte SSLRequest * packet, containing no information other than the CLIENT_SSL flag * which is necessary to trigger the TLS handshake, and to send its * complete capability flags and other identifying information after * the TLS handshake. * - The SERVER knows how to receive this truncated 2-byte SSLRequest * packet, and to receive the client's complete capability bits * after the TLS handshake. * */ #define CLIENT_CAN_SSL_V2 (1ULL << 37) All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
The server implementation here was incorrect as well, unnecessarily
reading—and TRUSTING—client identifying information sent before the TLS
handshake. That's in MDEV-31585.
As a result of the server's mishandling of this information, it's not
possible for the client to fix this in a way that's backwards-compatible
with old servers.
We rely on the server sending a capability bit to indicate that the
server-side bug has been fixed:
This PR also incorporates fixes for two previously-identified client-security vulnerabilities: