Downtimes with volatile packet is discarded since the transport is not currently writable

[severinolorillajr]

Our socket server generally works fine.

"socket.io": "^4.5.3",
"socket.io-client": "^4.5.4",

But we've been having downtimes twice a week (more or less).
We've been bombarded with these messages:

Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.802Z socket.io:client volatile packet is discarded since the transport is not currently writable
Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.802Z socket.io:client volatile packet is discarded since the transport is not currently writable
Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.802Z socket.io:client volatile packet is discarded since the transport is not currently writable
Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.802Z socket.io:client volatile packet is discarded since the transport is not currently writable
Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.804Z socket.io:client volatile packet is discarded since the transport is not currently writable
Jul 9 02:09:36 ip-10-0-31-159 web[255737]: 2024-07-09T02:09:36.805Z socket.io:client volatile packet is discarded since the transport is not currently writable

Any ideas how to approach or work around this?
Can't seem to find anything on Socket IO docs and the web discussing this.

Was this because we are looping volatile emit events?
But it doesn't seem to happen that frequently though.

[darrachequesne]

Hi! What do you mean by "downtimes"? Does your server crash?

Volatile packets can indeed be discarded if the underlying transport is not ready to be used, for example if there is no pending GET request when using HTTP long-polling.

Reference: https://socket.io/docs/v4/emitting-events/#volatile-events

[darrachequesne]

Like if I have volatile events for different destinations, will that matter?

The server checks for the state of each client individually. The volatile packets may be discarded for some, and sent for others.

My problem here though once I reach a state where I'm spammed with these errors.

Please note that these are debug logs, not errors. This might suggest an actual problem though, as rebooting the server should not be needed.

Would there be a way for us to keep track when a transports is about to reach it's limit?

I guess you could check the bufferedAmount value of the WebSocket object, though I'm not sure how it can help in your case.

Reference: https://github.com/websockets/ws/blob/master/doc/ws.md#websocketbufferedamount

[severinolorillajr]

Appreciate the time you spent answering my questions @darrachequesne.
I think this is an issue on code level from our side.

I'll add more once I have more information.
For now, I am closing this discussion.

Cheers!

[severinolorillajr]

@darrachequesne just a quick confirmation though.
You mentioned that websocket transport can be come unwritable when a lot of data gets written.
And in return could result to dropped packets.

Have a few questions though:

1. If I added a "polling" transport would it automatically downgrade from websocket when transport becomes unwriteable?
2. If 1 above automatically does that, will the websocket transport be able to recover, i.e. become writable again?
3. If I am downgraded to polling, is there a mechanism that automatically reverts it back to websocket once available?
4. If i want to have polling as fallback.. i can add the transports in the clients in this order right? websocket, then polling?

[severinolorillajr]

Add'l info here:

I noticed that the dropped packets has been going on for sometime.
Even if we did not really experience the server being unresponsive.
But the volume of this dropped packets, is already concerning.

And at times, it makes the system totally unusable, needing a reboot.

[darrachequesne]

The transport becomes unwritable for a short amount of time (encoding + writing to underlying WebSocket):

socket.io/packages/engine.io-client/lib/transports/websocket.ts

Lines 91 to 120 in b79d80a

	override write(packets) {
	this.writable = false;
	// encodePacket efficient as it uses WS framing
	// no need for encodePayload
	for (let i = 0; i < packets.length; i++) {
	const packet = packets[i];
	const lastPacket = i === packets.length - 1;
	encodePacket(packet, this.supportsBinary, (data) => {
	// Sometimes the websocket has already been closed but the browser didn't
	// have a chance of informing us about it yet, in that case send will
	// throw an error
	try {
	this.doWrite(packet, data);
	} catch (e) {
	debug("websocket closed before onclose event");
	}
	if (lastPacket) {
	// fake drain
	// defer to next tick to allow Socket to clear writeBuffer
	nextTick(() => {
	this.writable = true;
	this.emitReserved("drain");
	}, this.setTimeoutFn);
	}
	});
	}
	}

There is currently no way to downgrade: by default, polling is used first, and upgraded to WebSocket if possible.

There will be a few performance improvements on the server side in the next version, maybe that will help (diff).