Add support for gzip decompressing in Ruby gem

[dhughes]

Description

I use the Mailchimp Marketing Ruby gem in two integration projects that sync information from two ecommerce platforms to Mailchimp. I recently noticed a spike in errors with a message of "illegal/malformed utf-8" while parsing JSON data.
Keeping a long story short, what I learned is that was apparently only occurring on the us18 server and only within Ruby.

I was able to use Postman to make the same request to us17 and us18 without any problems. However, when using the Mailchimp marketing gem, I'd run into the problem.

I've narrowed the problem down to the gem not being able to handle gzip data, and us18 only returning gzip data.

Here's an example that should reproduce the problem. This assumes you have the gem installed.

require 'MailchimpMarketing'

def test_request(access_token, server, store_id)
  customer_params = '{"id":"F76FNX39H8YWJ02262A0MH6YD4","opt_in_status":false,"company":""}'

  client = MailchimpMarketing::Client.new.tap do |client|
    client.set_config(access_token: access_token, server: server)
  end

  begin
    client.ecommerce.add_store_customer(store_id, customer_params)
  rescue MailchimpMarketing::ApiError => e
    puts e
  end
end

Please note that I've intentionally provided bad customer_params. I've been seeing the issue specifically with error responses, though maybe it occurs with successes too. I'm using the add_store_customer method, since that's where I was running into the problem.

Now, call this method with a valid access token for a user on a server other than us18. You should see output like this:

access_token = 'some-token'
server = 'us17'
store_id = 'some-store'

test_request(my_access_token, my_server, my_store_id)

This should output something like:

{:status=>400, :response_body=>"{\"type\":\"https://mailchimp.com/developer/marketing/docs/errors/\",\"title\":\"Invalid Resource\",\"status\":400,\"detail\":\"The resource submitted could not be validated. For field-specific details, see the 'errors' array.\",\"instance\":\"70d8f7af-3098-1af3-2016-d03113c729dc\",\"errors\":[{\"field\":\"email_address\",\"message\":\"An email address is required to create a customer.\"}]}"}

Next, call this same method for a user on us18:

access_token = 'some-token'
server = 'us18'
store_id = 'some-store'

test_request(my_access_token, my_server, my_store_id)

This should output something like:

{:status=>400, :response_body=>"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03-\x90\xBDn\xC30\f\x84_\x85\xD0\x92%\xFEk\xDC8\xF5\xD6\xA5@\xD7\xA2[Q\x14\xB2D'Bm\xC9%\xE9\x00A\x90w/\x1Dg\x12\xC0;~w\xD4\xD5\xC8eB\xD3\x9A\x93\xC8\xC4mQ\x8C6\f\xEE\x14\xC6)wi,<\x9EqH\x13\x92\xCE\xE9\x17%\xC4c\xE1\x93\xE3\x02\x89\x12qa\xB6F\x82\f\v\xE0=\x9E\xED\x10<| \xA7\x99\x1C\xAA\xC4bef\xD3\xD6e\xB95\x1EE\xD1j\xFC<!\xD0\xC3\x04<wc\x10A\x0F.\xCD\x83\x87\x98\x04:\x84;\xCA\xEA8\x87\xB7D\xD0\a\x1C|\xC6\x13\xBA\xD0\a\a+\x8A\xB7\xC0\x88 \x8A\xDB\xACu6`\x89\xEC%\xD7\xE8\x105<\xBA\xA5X\xF5\xB2\xF3Me\x9F\xB3\xB2\xDE\xEF3W\xEF\xCA\xAC\xA9\x0EOY\xDDT\x9D\xB5\xFB\xA6\xEAw\xA5n\xAC\b\xD3~]\xCD=N7q\xF9\x8C\x1F\xEB\xBD\xD6e\xB5\x8C\xFA\xD8\xE3\xC2|\x8Dp\x17\xE1!B`\xBD\xE9o\x0E\xA4\x97H\x02G\xA8\xED\xC1\x82\x9BY\xD2\x88\x94\x9B\xDB\xF7\xED\x1F\x9Cm\x9B5m\x01\x00\x00"}

That weird response body is gzip data that the Excon HTTP library doesn't know what to do with.

I tried various approaches to addressing this, but the only one I found that worked was to use the Excon::Middleware::Decompress middleware. This recognizes gzip responses and decodes them.

If you pull my PR, regenerate and install the gem, running both of the examples above will produce the expected output.

With regards to this PR, I looked to see if there was a useful way to add a test for this, but wasn't able to find one. If you require a test for this PR, please let me know what you're looking for and I'll take care of it.

Known Issues

[cla-bot]

Contributor License Agreement Instructions
Thanks for your pull request. Before we can review your work, you’ll need to sign a Contributor License Agreement (CLA).

Please download the appropriate CLA below. Once downloaded, please read, sign, and send back to us at [email protected]. Please note, this account is not monitored so please visit https://mailchimp.com/contact/ if you need support.

Individual CLA: Mailchimp Individual CLA
Corporate CLA: Mailchimp Corporate CLA

Once you’ve emailed us the signed CLA, please reply here (e.g. CLA signed and sent!) and we’ll verify it.

What to do if you already signed the CLA
Individual signers
• If you’ve previously sent us a signed CLA, please reply here letting us know and we’ll verify. If we are unable to verify, It’s possible we don’t have your GitHub username or you’re using a different email address on your Git commit. Check that the CLA you previously submitted was sent to us using the email address associated with your GitHub username and verify that your email is set on your Git commits.
Corporate signers
• Your company has a Point of Contact (POC) who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you’ve previously sent us an updated CLA, please reply here letting us know and we’ll verify.
• The email used to register you as an authorized contributor must be the email used for the Git commit.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

[dhughes]

CLA signed and sent!

[dhughes]

As a note, I accidentally sent the CLA from an incorrect email. I've resent it from my personal email.

[cla-bot]

Contributor License Agreement Instructions
Thanks for your pull request. Before we can review your work, you’ll need to sign a Contributor License Agreement (CLA).

Please download the appropriate CLA below. Once downloaded, please read, sign, and send back to us at [email protected]. Please note, this account is not monitored so please visit https://mailchimp.com/contact/ if you need support.

Individual CLA: Mailchimp Individual CLA
Corporate CLA: Mailchimp Corporate CLA

Once you’ve emailed us the signed CLA, please reply here (e.g. CLA signed and sent!) and we’ll verify it.

What to do if you already signed the CLA
Individual signers
• If you’ve previously sent us a signed CLA, please reply here letting us know and we’ll verify. If we are unable to verify, It’s possible we don’t have your GitHub username or you’re using a different email address on your Git commit. Check that the CLA you previously submitted was sent to us using the email address associated with your GitHub username and verify that your email is set on your Git commits.
Corporate signers
• Your company has a Point of Contact (POC) who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you’ve previously sent us an updated CLA, please reply here letting us know and we’ll verify.
• The email used to register you as an authorized contributor must be the email used for the Git commit.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

[dhughes]

This is the magic line. It sets the middlewares used by the new Excon connection to be the defaults (Excon.defaults[:middlewares]) and Excon::Middleware::Decompress, which knows how to decompress gzip response data.

[mc-keith]

Hey @dhughes thanks for opening this PR!

We believe this issue may have been due to a configuration issue on our end, which has been fixed. Would you mind seeing if you're still seeing the encoding issues outside of this fix?

[dylanbromby]

This is happening on us17 for us, but only when calling add_store_cart.