Client can't reconnect

[haledv]

Describe the bug

Hi,

we have some very random and very hard reproducable bug, that "kills" our connection to rabbitmq, and the Application needs to restart. It seems like the rabbitmq-server closes the connection to the client and the reconnect doesn't work. Other Applications continue to run without any errors, means Server is still up and running.

The error messages that appear:

Error while declaring Topic OUR_EXCHANGE_NAME. Exception: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text='Unexpected Exception',

Followed by a "Error occured while executing action (within EnsureConnectedAndExecute) The Operation has timed out.".

And also : Error while declaring Topic OUR_EXCHANGE_NAME. Unkown Exception: (non=OperationInterruptedException): Pipelining of requests forbidden

Followed by a "Error occured while executing action (within EnsureConnectedAndExecute) Pipelining of requests forbidden".

There is also a warning appearing: Channel was Closed and will be recreated. Reason {"Cause":null, "ClassId":60, "initiator":2, "MethodID:"80, "ReplyCode': ED - unkown delivery tag 1878

Any help regarding this would be appreciated. I guess, we maybe use some of libs functionality the wrong way.

Reproduction steps

Can't really reproduce this

Expected behavior

Reconnect works

Additional context

We are using rabbitmq-server-3.11.2 with RabbitMq.client 6.4.0. It seems to happen when a durable queu is quite "full" (100k messages), and both reading and writing occurs, but the reading isn't successfully and the client application needs to reque the Message back.

[michaelklishin]

unkown delivery tag 1878

means your client double acknowledges a delivery.
Usually this happens when multiple consumers share a channel, or a channel is shared between threads or tasks, and one or both consumers acknowledge with multiple set to true.

Then the channel is closed and any attempts to use it close the connection.

[michaelklishin]

Perhaps it's worth clarifying that the above client error will not allow any more channels (or any other activity) on the closed connection. New connections will not be blocked.

However, if clients then reconnect and repeat the error, and enough clients do it repeatedly,
that can create a high connection churn scenario, and with a small number of open file handles that can lead to clients' inability to connect until the old connection exist the TIME_WAIT state (the post is about Linux but TCP connection termination algorithm would be no different on Windows or any other OS).

A traffic capture would provide more information about
what exactly clients send and how RabbitMQ nodes respond.

[haledv]

Thank you for your answer. I inherited the wrapper codebase, so I need to do a bit of a deep dive in it, to understand it better. Any "official" wrapper library you know of?

It happens so rarely, that I didn't have a chance to really see it in action, as the support restarts the consumer clients asap to "fix" the issue.

So to sum it up: one persistant Queu with an client acting as Producer and an other client as Consumer. Reports from the field are that when this happens, both, Producer and Consumer can't connect to the specific Queu. The logs above are from the Consumer side. One thing I am still wondering is, could this lead also to Problems in other clients on the same Queu (or even on the same Exchange)? Reports are, as soon as the consumer client is restarted, the producer client gets back to a "normal" state, before that, it seems that on his side also appear logs of reconneting etc. Sry, we have a bit of limited logging, so I can't really see what happend on the producer side (and my limited knowledge about RabbitMq), but I was expecting that a client consumer, can't interupt a producer client in any way on the same Queu.