IllegalStateException: Handshake has already been started

[swissiety]

Hi,
thanks for creating this great library which supports developers to create secure software.

Version 0.8.1

Expected Behavior When I execute openssl s_client -connect $IP:$PORT to examine the example code of the non-blocking server works as expected and I can send messages to the example program.

Unexpected Behavior
But when I use the same example - except the SSLEngine setup: I used a different Certificate created via BouncyCastle - in an Android app the following Exception occurs.

Exception

java.lang.IllegalStateException: Handshake has already been started at com.android.org.conscrypt.OpenSSLEngineImpl.beginHandshakeInternal(OpenSSLEngineImpl.java:335)
at com.android.org.conscrypt.OpenSSLEngineImpl.beginHandshake(OpenSSLEngineImpl.java:325)
at tlschannel.impl.TlsChannelImpl.doHandshake(TlsChannelImpl.java:502)
at tlschannel.impl.TlsChannelImpl.handshake(TlsChannelImpl.java:485)
at tlschannel.impl.TlsChannelImpl.read(TlsChannelImpl.java:163)
at tlschannel.ServerTlsChannel.read(ServerTlsChannel.java:266)
at tlschannel.ServerTlsChannel.read(ServerTlsChannel.java:271) 
at tlschannel.ServerTlsChannel.read(ServerTlsChannel.java:276)

As far as I understood, what seems to happen is that there is a read() which throws a NeedsReadException inside writeToChannel(), then the exception bubbles up to doHandshake( false) and therefore skips the rest of the try block and so in the next execution of tlsChannel.read(...) the code tries to begin the handshake again and the SSLEngine complains.

Excerpt from TlsChannelImpl.java:492

private void doHandshake(boolean force) throws IOException, EofException {
     if (!force && negotiated) {
         return;
     }
     try {
            if (invalid || shutdownSent) {
                throw new ClosedChannelException();
            }
            if (force || !negotiated) {
                logger.log(Level.FINEST, "Calling SSLEngine.beginHandshake()");
                engine.beginHandshake();   // <-- this is called the first time and then after the NeedsReadException when we run into the next read(...) call, its executed the second time which leads to the ISE
                writeAndHandshake();      // <-- here a NeedsReadException is thrown so we jump to the finally block

                if (engine.getSession().getProtocol().startsWith("DTLS")) {
                    throw new IllegalArgumentException("DTLS not supported");
                }

                // call client code
                try {
                    initSessionCallback.accept(engine.getSession());
                } catch (Exception e) {
                    logger.log(Level.FINEST, "client code threw exception in session initialization callback", e);
                    throw new TlsChannelCallbackException("session initialization callback failed", e);
                }
                negotiated = true;
        } finally {
            initLock.unlock();
        }
}

[marianobarrios]

Fixed in #201