Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert library to use built-in Net::HTTP #813

Merged
merged 1 commit into from
Jul 29, 2019

Conversation

brandur
Copy link
Contributor

@brandur brandur commented Jul 24, 2019

Moves the library off of Faraday and over onto the standard library's
built-in Net::HTTP module. The upside of the transition is that we
break away from a few dependencies that have caused us a fair bit of
trouble in the past, the downside is that we need more of our own code
to do things (although surprisingly, not that much more).

The biggest new pieces are:

  • ConnectionManager: A per-thread class that manages a connection to
    each Stripe infrastructure URL (like api.stripe.com,
    connect.stripe.com, etc.) so that we can reuse them between
    requests. It's also responsible for setting up and configuring new
    Net::HTTP connections, which is a little more heavyweight
    code-wise compared to other libraries. All of this could have lived in
    StripeClient, but I extracted it because that class has gotten so
    big.

  • MultipartEncoder: A class that does multipart form encoding for file
    uploads. Unfortunately, Ruby doesn't bundle anything like this. I
    built this by referencing the Go implementation because the original
    RFC is not very detailed or well-written. I also made sure that it was
    behaving similarly to our other custom implementations like
    stripe-node, and that it can really upload a file outside the test
    suite.

There's some risk here in that it's easy to miss something across one of
these big transitions. I've tried to test out various error cases
through tests, but also by leaving scripts running as I terminate my
network connection and bring it back. That said, we'd certainly release
on a major version bump because some of the interface (like setting
Stripe.default_client) changes.

r? @ob-stripe Sorry, this one is kinda huge, but can I get your thoughts
on it? It might help just to do an initial pass for just the overall
design/big picture stuff first.

cc @stripe/api-libraries


Note: Targets the branch integration-v5 in #815 instead of master.

@brandur-stripe
Copy link
Contributor

brandur-stripe commented Jul 24, 2019

And minor note that I still want to do another refactor pass on StripeClient in the future. I've tried to clean up places where I could easily, but it's still overall too big/bad/obscure.

(And happy to do that on a major version integration branch in the style that OB does for .NET if we think that's the right move.)

Copy link
Contributor

@ob-stripe ob-stripe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work Brandur!

  • Left a couple of minor comments
  • Looks like you didn't update the Gemfile to remove Faraday and Net::Http::Persistent
  • This should probably be released as a new major version, since it's a breaking change for users using custom Faraday adapters

context = dup
context.account = headers["Stripe-Account"]
context.api_version = headers["Stripe-Version"]
context.idempotency_key = headers["Idempotency-Key"]
context.request_id = headers["Request-Id"]
context.request_id = headers["Request-ID"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT the API always returns either Request-Id or request-id. It doesn't matter since it's case insensitive but I'm curious if there's any specific reason for this change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, yeah I also prefer the header casing style of "strict uppercase after hyphen, strict downcase everywhre else".

I think I'd changed this to all lowercase during an early pass, and then decided to change it back and just put the wrong casing in when I was reverting it.

Fixed now.


params.each do |name, val|
if val.is_a?(::File)
val.rewind
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tempfile instances would need this to. In our test we manually call #rewind before passing the instance to Stripe::File.create but it would be nice if the encoder did this automatically.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh man, I was confused here because I added this rewind explicitly so that this could work with with a tempfile that I was using in the encoder's test suite.

You're totally right in that a Tempfile is not a File:

irb(main):001:0> require 'tempfile'
=> true
irb(main):002:0> t = Tempfile.new
=> #<Tempfile:/var/folders/b7/50czt19n47vf99rd769dq8wc0000gn/T/20190725-551-1kjvpws>
irb(main):003:0> t.is_a?(File)
=> false

But crazy enough, when you use Tempfile in its block form, the pointer you're handed is a file:

irb(main):009:0> Tempfile.create do |f|
irb(main):010:1* f.is_a?(File)
irb(main):011:1> end
=> true

Which is why this code works with its test suite.

So given that, do you have thoughts on:

  1. Whether we should still whitelist Tempfile as well as File to be rewound like you suggested? (I could see it either way TBH.)
  2. Whether we should rewind at all from the encoder? I could just change the test suite to rewind before handing off the tempfile like in the file tests. I was thinking that just doing a rewind should never be harmful, but it's possible there's some case that I hadn't though of (like if a user was purposely trying to upload only part of a file).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But crazy enough, when you use Tempfile in its block form, the pointer you're handed is a file:

Oh Ruby :shakes_fist:

Yeah, I'm starting to think it might be better to not call rewind at all and let the user do it themselves if they want. Will leave final decision up to you.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh Ruby :shakes_fist:

Haha, I know right??

Okay, thanks OB. I thought about it a little more, and in the end have decide to remove the rewind from the encoder and just have it present in the test suite like we were doing in the file resource tests. Doing the automatic rewind might be convenient in some cases, but it's just a little too "magic", and might obstruct the legitimate (albeit probably very rare) use case of passing in a partially read file.

I also added Tempfile to the same case that File was in where we sent a specific filename instead of just the generic word blob. Tempfile filenames will often be meaningless, but they can be given a prefix, and in cases where they have, their filename will be better than blob. So the code now looks like this:

        if val.is_a?(::File) || val.is_a?(::Tempfile)
          write_field(name, val.read, filename: ::File.basename(val.path))
        elsif val.respond_to?(:read)
          write_field(name, val.read, filename: "blob")
        else
          write_field(name, val, filename: nil)
        end

@brandur-stripe
Copy link
Contributor

And just regarding your other notes:

Looks like you didn't update the Gemfile to remove Faraday and Net::Http::Persistent

Ah this is kind of confusing, but the Gemfile using the gemspec annotation to source stripe.gemspec's contents for non-development dependencies, and I removed Faraday and co. from there, so I think we're good.

This should probably be released as a new major version, since it's a breaking change for users using custom Faraday adapters

Yep, totally +1.

Moves the library off of Faraday and over onto the standard library's
built-in `Net::HTTP` module. The upside of the transition is that we
break away from a few dependencies that have caused us a fair bit of
trouble in the past, the downside is that we need more of our own code
to do things (although surprisingly, not that much more).

The biggest new pieces are:

* `ConnectionManager`: A per-thread class that manages a connection to
  each Stripe infrastructure URL (like `api.stripe.com`,
  `connect.stripe.com`, etc.) so that we can reuse them between
  requests. It's also responsible for setting up and configuring new
  `Net::HTTP` connections, which is a little more heavyweight
  code-wise compared to other libraries. All of this could have lived in
  `StripeClient`, but I extracted it because that class has gotten so
  big.

* `MultipartEncoder`: A class that does multipart form encoding for file
  uploads. Unfortunately, Ruby doesn't bundle anything like this. I
  built this by referencing the Go implementation because the original
  RFC is not very detailed or well-written. I also made sure that it was
  behaving similarly to our other custom implementations like
  stripe-node, and that it can really upload a file outside the test
  suite.

There's some risk here in that it's easy to miss something across one of
these big transitions. I've tried to test out various error cases
through tests, but also by leaving scripts running as I terminate my
network connection and bring it back. That said, we'd certainly release
on a major version bump because some of the interface (like setting
`Stripe.default_client`) changes.
@brandur-stripe brandur-stripe changed the base branch from master to integration-v5 July 29, 2019 20:40
@brandur-stripe
Copy link
Contributor

Alright, I've retargeted this PR to the branch integration-v5 and am going to merge to that. I'll open a separate PR to collect other V5 changes.

@brandur-stripe brandur-stripe merged commit 2c80bc8 into integration-v5 Jul 29, 2019
@brandur-stripe brandur-stripe deleted the brandur-net-http branch July 29, 2019 20:47
@brandur brandur mentioned this pull request Jul 29, 2019
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants