Conduit (hence Cohttp and mirage.io) hangs at end of client input

[vog]

When a conduit server handles a client connection, it hangs at the end of client input, instead of providing a proper EOF mark for the input channel.

This can be reproduced as follows. First, compile and run the following server:

let main =
  Conduit_lwt_unix.serve ~ctx:Conduit_lwt_unix.default_ctx ~mode:(`TCP (`Port 1234)) (fun _flow ic _oc ->
      let%lwt () = Lwt_io.printf "Client connected\n" in
      let%lwt data = Lwt_io.read ic in
      Lwt_io.printf "Received data: %s\n" data
    )
let () = Lwt_main.run main

Then, send a test string with proper EOF, e.g. using the Netcat tool:

echo 'test' | nc localhost 1234

You'll note that nc hangs, because the server keeps the connection open. The server's output is:

Client connected

which also shows that the server hangs. Once you kill Netcat, e.g. via Ctrl+C, the server finally handles that client input:

Client connected
Received data: test

Workaround: The following workaround appears to used by Cohttp (at least accidentally), and perhaps by other projects as well:

Perform a partial read with a large buffer, and never dare to read again:

      let%lwt data = Lwt_io.read ~count:100000 ic in

If you now run the above Netcat command, it will return immediately, and the server's output at the console shows that request handing indeed completed:

Client connected
Received data: test

Note, however, that this workaround only works if the client sends at least one byte to the server. Otherwise, even that first call will hang.

Cohttp and mirage.io are affected: Cohttp is indeed affected by the limitation of the above workaround. For any Cohttp server, such as mirage.io on port 80, we can observe the following behaviour:

If you send at least one byte to it, Netcat will return immediately, i.e. the server will close the connection properly:

echo 'test' | nc mirage.io 80

However, if you send nothing, the server will not close the connection, hence Netcat will not return:

echo -n | nc mirage.io 80

(you need to use echo -n, because a plain echo would send one byte: the newline character)

[vog]

After digging further, I finally found the root cause.

On network connections, closing the output channel should go hand in hand with a call to:

let callback ... =
    ...
    Lwt_unix.shutdown client_socket Lwt_unix.SHUTDOWN_SEND
    ...

Since this is only possible with access to client_socket (of type Lwt_unix.file_descr), this is only possible when using the following modern server function, which is available since Lwt 4.1.0:

Lwt_io.establish_server_with_client_socket ... callback

This is not possible when using the old server function establish_server_with_client_address.

I propose to fix conduit accordingly.