Net: Resend previously-failed-to-send messages on reconnect

[solardiz]

Nov 9, 2022

Right now, we first read a message from kmsg, then try to send it. If the sending fails, we'd reconnect and would try reading/sending further messages, but that one is already lost. We could instead resend it upon reconnect.

Further, we find out of a connection failure with a delay - when attempt to send a message fails, it means the previous one could have failed to actually be sent too - but our non-blocking call just buffered it. We could maybe preserve the contents of socket buffer (non-acknowledged) and resend that upon reconnect. This is tricky and it needs research. There's also risk of duplicating messages in this way (not acknowledged does not mean not received), so maybe this should somehow integrate with replay protection.

We could want to split this issue in two (for the above paragraphs).

[solardiz]

Nov 10, 2022

Right now, we first read a message from kmsg, then try to send it. If the sending fails, we'd reconnect and would try reading/sending further messages, but that one is already lost. We could instead resend it upon reconnect.

This is now implemented. As expected, one message is getting lost anyway because of the issue described in the second paragraph (quoted below), but previously multiple messages were getting lost (one per invocation of the read/send loop while the connection couldn't be re-established).

Further, we find out of a connection failure with a delay - when attempt to send a message fails, it means the previous one could have failed to actually be sent too - but our non-blocking call just buffered it. We could maybe preserve the contents of socket buffer (non-acknowledged) and resend that upon reconnect.

It's even trickier than that: the socket buffer contains already encrypted data, and the encryption would need to be different upon reconnect (different ephemeral key(s), different message number within the session). Maybe we could merely treat non-empty buffer as a Boolean, and plan on resending (and re-encrypting first) the last message we had sent to the socket.