IllegalStateException in IteratingCallback.iterate during async request when client has died

[ekupcik]

Jetty version(s)
12.0.7

Jetty Environment
ee10

Java version/vendor (use: java -version)
21

OS type/version
Windows

Description
While running load tests (100 concurrent requests with 100% CPU load on a 6/12 core CPU) for an REST API using async requests and Jersey I found this on my server side (probably) after I have killed my test client. I tried to reproduce it a few times but without any luck.

2024/04/01 11:17:12.585  WARN  [] [GateKeeper API worker [96]] org.eclipse.jetty.ee10.servlet.ServletChannel /gatekeeper/services/rest/api/7.2/SysLogin
java.lang.IllegalStateException: SendCallback@79107574[CLOSED][i=null,cb=ChannelResponse@45d3bca8{200,POST@863c9c14 https://localhost:8808/gatekeeper/services/rest/api/7.2/SysLogin?client=GATEKEEPER HTTP/1.1}]
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:227)
	at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1419)
	at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1501)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:760)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:421)
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1199)
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257)
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.complete(ServletChannelState.java:783)
	at org.eclipse.jetty.ee10.servlet.AsyncContextState.complete(AsyncContextState.java:61)
	at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:102)
	at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:173)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:404)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:749)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:385)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:375)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:905)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:937)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:893)
	at de.usu.gatekeeper.server.command.rest.ApiResource.sendResponse(ApiResource.java:330)
	at de.usu.gatekeeper.server.command.rest.ApiResource.lambda$0(ApiResource.java:227)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)



Apr. 01, 2024 11:17:12 AM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
SCHWERWIEGEND: Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.unhandle(ServletChannelState.java:417)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:587)
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1199)
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257)
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.complete(ServletChannelState.java:783)
	at org.eclipse.jetty.ee10.servlet.AsyncContextState.complete(AsyncContextState.java:61)
	at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:102)
	at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:173)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:404)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:749)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:385)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:375)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:905)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:937)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:893)
	at de.usu.gatekeeper.server.command.rest.ApiResource.sendResponse(ApiResource.java:330)
	at de.usu.gatekeeper.server.command.rest.ApiResource.lambda$0(ApiResource.java:227)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[lorban]

@sbordet this seems related to #11016, maybe we did not fill all the gaps?

[gregw]

Jersey is calling AsyncContext.complete() on a request whose connection has already been closed. I think it is reasonable for the complete to throw, else how will the application know that the response has not been correctly sent? Maybe ISE is not the right thing to throw and should perhaps throw an IOException, as we know the iterator has been CLOSED?

The second exception is a little bit more of a worry. How did we get to IDLE state with a thread in a call to handle?

[lorban]

This seems to be another manifestation of eclipse-ee4j/jersey#5675