Large volumes of rollbars results in out of file handles

[timlesallen]

Just wanted to bring this one up, if we raise a lot of rollbars (e.g multiple per second) over a sustained period, we exhaust the file handles, and you scenario which triggers an error like this (see last line for where it fails):

I, [2021-08-10T02:17:39.980197 #9]  INFO -- : [Rollbar] Success
I, [2021-08-10T02:17:39.980266 #9]  INFO -- : [Rollbar] Details: https://rollbar.com/instance/uuid?uuid=5335f55d-52e4-492e-896e-2e192f88f74b (only available if report was su
ccessful)
I, [2021-08-10T02:17:39.990623 #9]  INFO -- : [Rollbar] Scheduling item
I, [2021-08-10T02:17:39.994481 #9]  INFO -- : [Rollbar] Sending item
I, [2021-08-10T02:17:39.995434 #9]  INFO -- : [Rollbar] Sending json
I, [2021-08-10T02:17:40.004166 #9]  INFO -- : [Rollbar] Scheduling item
I, [2021-08-10T02:17:40.004233 #9]  INFO -- : [Rollbar] Sending item
I, [2021-08-10T02:17:40.004951 #9]  INFO -- : [Rollbar] Sending json
I, [2021-08-10T02:17:40.084451 #9]  INFO -- : [Rollbar] Success
I, [2021-08-10T02:17:40.084554 #9]  INFO -- : [Rollbar] Details: https://rollbar.com/instance/uuid?uuid=29f6104c-0d6a-437d-aa2a-a33f2a5d51f5 (only available if report was su
ccessful)
I, [2021-08-10T02:17:40.103217 #9]  INFO -- : [Rollbar] Scheduling item
I, [2021-08-10T02:17:40.103308 #9]  INFO -- : [Rollbar] Sending item
I, [2021-08-10T02:17:40.104105 #9]  INFO -- : [Rollbar] Sending json
I, [2021-08-10T02:17:40.111458 #9]  INFO -- : [Rollbar] Success
I, [2021-08-10T02:17:40.111566 #9]  INFO -- : [Rollbar] Details: https://rollbar.com/instance/uuid?uuid=1fada12a-8271-45b0-9594-2ef9afdabd5b (only available if report was successful)
E, [2021-08-10T02:17:39.227624 #7] ERROR -- : [Rollbar] Error processing the item: Errno::EMFILE, Failed to open TCP connection to api.rollbar.com:443 (Too many open files - getaddrinfo).

A couple of points:

• Are there suggestions or best practices for handling this better?
• Would it be possible mitigate this in the rollbar gem in some way? For example, could multiple rollbars be send over the same TCP connection?

[waltjones]

@timlesallen It could be that the OS limit for file descriptors needs to be set higher. For context, what is the current limit when you hit this issue? (Using ulimit -n for example.)

[timlesallen]

I think we are probably using whatever the default is on ECS. But I guess I was more raising this from the perspective of - should rollbar be reusing its connection rather than opening a fresh connection for each rollbar...? Just some feedback. Thanks, we'll increase the file handle limit if need be. :). Feel free to close.

[waltjones]

Yes I agree, reusing the socket would be best practice and better performance. That said, I did some testing today with bursts of traffic and wasn't able to get any significant number of open descriptors. They are freed when the socket closes, and this seems to be working. Results might depend on Net::HTTP version and OS. (I used Ruby 2.7.2 and Ubuntu 18.04 for this testing.)

I'd like to keep this open and consider improving the connection management.