go client reports invalid connection after idle for wait_timeout

[djshow832]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

dbt.mustExec("SET @@SESSION.wait_timeout = 2")

// wait for TiDB to close our connection
time.Sleep(3 * time.Second)

tx, err := dbt.db.Begin()
if err != nil {
	dbt.Fatal(err)
}

2. What did you expect to see? (Required)

The driver logs packets.go:122: closing bad idle connection: EOF 3 times but dbt.db.Begin doesn't report any errors because it creates a new connection internally and then retry.

3. What did you see instead (Required)

The driver logs packets.go:37: unexpected EOF and dbt.db.Begin reports invalid connection.

4. What is your version? (Required)

master.

The reason:
TiProxy doesn't close the connection immediately after wait_timeout and still receives requests. After it receives the request, it finds that TiDB has disconnected and then closes the client connection. So the client encounters an ErrInvalidConn in readPacket instead of ErrBadConn in writePacket.
The client will retry by creating a new connection if encounters ErrBadConn but won't retry if encounters ErrInvalidConn, see https://github.com/golang/go/blob/master/src/database/sql/sql.go#L1536-1546
I think there's no way to solve it.

[djshow832]

Maybe a workaround is to add a client idle timeout to tiproxy configuration, like client timeout in haproxy: https://www.papertrail.com/solution/tips/haproxy-logging-how-to-tune-timeouts-for-performance/