panic (close of closed channel) in TestCallReceiverRpc

[zenhack]

Per #346; haven't dug into this at all yet, but I assume this has been there and is just rare enough to have not cropped up before. It passed ~250 times in that run before failing...

https://github.com/capnproto/go-capnproto2/actions/runs/3537277747/jobs/5937100516#step:7:488

--- FAIL: TestCallReceiverAnswerRpc (0.00s)
panic: close of closed channel [recovered]
	panic: close of closed channel

goroutine 39 [running]:
testing.tRunner.func1.2({0x829bd80, 0x8334130})
	/opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1396 +0x2ab
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1399 +0x41f
panic({0x829bd80, 0x8334130})
	/opt/hostedtoolcache/go/1.19.3/x64/src/runtime/panic.go:884 +0x1c3
capnproto.org/go/capnp/v3.Client.Release({0x8c604b0})
	/home/runner/work/go-capnproto2/go-capnproto2/capability.go:584 +0xe6
capnproto.org/go/capnp/v3.(*Message).Reset(0x8c64540, {0x0, 0x0})
	/home/runner/work/go-capnproto2/go-capnproto2/message.go:134 +0xc6
capnproto.org/go/capnp/v3/rpc/transport.(*transport).RecvMessage.func1()
	/home/runner/work/go-capnproto2/go-capnproto2/rpc/transport/transport.go:183 +0x30
capnproto.org/go/capnp/v3/rpc.(*question).PipelineSend.func3()
	/home/runner/work/go-capnproto2/go-capnproto2/rpc/question.go:182 +0x4e
capnproto.org/go/capnp/v3/rpc.TestCallReceiverAnswerRpc(0x8c97770)
	/home/runner/work/go-capnproto2/go-capnproto2/rpc/receiveranswer_test.go:137 +0x583
testing.tRunner(0x8c97770, 0x82e0b2c)
	/opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x113
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x374
exit status 2
FAIL	capnproto.org/go/capnp/v3/rpc	0.115s
Error: Process completed with exit code 1.

[zenhack]

Per discussion on matrix: I am having a really hard time reproducing this, and it seems likely that #366 could have fixed this as well given the nature of the change (though I'm having a hard time reproducing it locally before that commit too). I'm going to close this one for now; we can re-open if we see it again, but otherwise this seems likely to turn into a wild goose chase.

[zenhack]

Re-opening; I just hit this again locally. I wasn't able to reproduce it a second time (ran the tests 700 times), but now we have some evidence that this is still there, and just incredibly rare.

[lthibault]

Is there any chance this might have been resolved by #396 or #406 ? I ask because this issue doesn't feel very actionable, and I would very much like an excuse to close it! 😅

If not, is there a low-effort approach we could take to flushing it out, e.g. running unit tests a few thousand times on a dev machine?

Assuming we can't reproduce it, how do you feel about closing optimistically?

[zenhack]

I'm ok closing it until it comes up again.

[zenhack]

I hit this again, re-opening. The stack trace below starts in a different spot, but still ends in Client.Release, so that gives me some confidence that this isn't terribly specific to any one test, and is a more general bug with the way Clients work.

panic: close of closed channel

goroutine 3094 [running]:
capnproto.org/go/capnp/v3.Client.Release({0x73be20?})
	/home/isd/src/go-wk/go-capnproto2/capability.go:589 +0x167
capnproto.org/go/capnp/v3.(*Message).Reset(0xc00061bcc0, {0x0?, 0x0})
	/home/isd/src/go-wk/go-capnproto2/message.go:141 +0x12c
capnproto.org/go/capnp/v3/rpc/transport.(*transport).RecvMessage.func1()
	/home/isd/src/go-wk/go-capnproto2/rpc/transport/transport.go:172 +0x2f
capnproto.org/go/capnp/v3/rpc.(*Conn).handleReturn.func1.2()
	/home/isd/src/go-wk/go-capnproto2/rpc/rpc.go:1130 +0x142
created by capnproto.org/go/capnp/v3/rpc.(*Conn).handleReturn.func1
	/home/isd/src/go-wk/go-capnproto2/rpc/rpc.go:1122 +0x837
exit status 2
FAIL	capnproto.org/go/capnp/v3/rpc	0.252s