Operations sometimes throw AsyncHTTPClient.HTTPClientError

[aviflax]

Hi, I’m a seasoned developer but I’m new to Apple development and new to Swift.

I’m writing a Mac CLI program and I’m trying to upload some files to BackBlaze B2 using its S3-compatible API. I’m using Soto 5.11.0 and Xcode 13.2, and I’m using the ~new Swift concurrency features (async/await).

I’m seeing that my calls to S3.putObject are usually succeeding, but they’re sometimes failing by throwing AsyncHTTPClient.HTTPClientError — specifically, HTTPClientError.getConnectionFromPoolTimeout.

I think I have two questions here:

1. If this is an expected failure mode, shouldn’t it be mentioned in the docs on error handling?
   • That page only discusses AWSClientError, AWSServerError, and AWSResponseError. When I wrote my catch clauses expecting those error types, they didn’t catch these HTTPClientError values because of course they are of a different type. This could just be due to my lack of experience with Swift, but I find this surprising. I’d have thought that the docs would describe all the possible error types that could be thrown. Often, a library will constrain the errors that are thrown by wrapping various types of errors in a standard error type… couldn’t these instances of HTTPClientError be wrapped in instances of AWSClientError and exposed via a property? Or am I missing something?
2. How can I figure out what’s actually the problem here?
   • Right now I’m basically just printing the error to stdout with print (in a string interpolation) and all I get is HTTPClientError.getConnectionFromPoolTimeout — literally just that. I looked at the source for HTTPClientError in HTTPClient.swift from the AsyncHTTPClient library, and this type doesn’t seem to include any properties containing details about what actually went wrong, which is confusing to me.
   • I searched the docs and source for the AsyncHTTPClient library in an attempt to learn more about this error case, but didn’t find much
   • If the answer is "ask over there" that’s fine, I just thought I’d ask here first because Soto is the library that I’m using directly.

Thank you!

[adam-fowler]

Hi @aviflax,

AsyncHTTPClient's connection pooling code has recently gone through a major refactor. This is a new error being returned by the package. From the error name it sounds like you are running many uploads at the same time and the HTTP client is running out of connections. It would probably be best to ask on the AsyncHTTPClient what is the best action to take.

If the cause is because you are pushing too many requests to S3 simultaneously you will have to build a queue to process these requests. I have an implementation of a queue which controls how many tasks can run concurrently here if you want to use that. Warning it is not heavily tested.

[aviflax]

Hi @adam-fowler thank you so much for the rapid and thorough reply!

AsyncHTTPClient's connection pooling code has recently gone through a major refactor. This is a new error being returned by the package. From the error name it sounds like you are running many uploads at the same time and the HTTP client is running out of connections. It would probably be best to ask on the AsyncHTTPClient what is the best action to take.

Makes sense. I’ll do that.

BTW what do you think about my idea that Soto should either wrap this error, or add it to the page Error Handling in the documentation, so that users of the library know that they might want to handle it?

If the cause is because you are pushing too many requests to S3 simultaneously you will have to build a queue to process these requests. I have an implementation of a queue which controls how many tasks can run concurrently here if you want to use that. Warning it is not heavily tested.

That makes sense. I’ll give it a try.

Thanks again!

[aviflax]

BTW what do you think about my idea that Soto should either wrap this error, or add it to the page Error Handling in the documentation, so that users of the library know that they might want to handle it?

I suspect my Swift skills are insufficient at this point for me to contribute the former potential change, but if there’s interest I’d be happy to contribute the latter documentation change.

[adam-fowler]

I would probably add it to the error handling docs, once I get full details of what it means. Did you contact the AsyncHTTPClient guys?