The courier protocol is a message-based client-server protocol with asynchronous capabilities. It is intended to be high throughput, reliable, and compatible with common firewall implementations.
+---+---+---+---+---+---+---+---+
| Type ID (4B) | Length (4B) |
+---+---+---+---+---+---+---+---+
| Data...
+
Every message starts with a 4 byte message type identifier, and is followed by the length of the message and then the data.
All lengths and numbers are transmitted as 32-bit unsigned integers in Big Endian format.
There is a special length identifier, which is signified by the max 32-bit unsigned value possible (0xFFFFFFFF in hex.) This length has special meaning and signifies that the message is streaming. Only specific message type IDs support streaming and the end of the message is specific to each type that supports it.
Messages identified as "Requests" MUST only be transmitted by a client that has initiated a connection to a server.
Messages identified as "Responses" MUST only be transmitted by a server that has accepted a connection from a client.
Request
Implementations MUST accept a length of up to 32 bytes, and discard any unknown data. Unknown data is reserved for future protocol expansions and MUST be all zero.
This message MUST be the first request sent on the connection. The server MAY respond with a VERS message if it supports protocol negotaton. If the server responds with ???? or an event message the client MUST assume a VERS response that is all zeros.
If a server receives this message not at the start of a connection it MUST return a ???? message, stating the message is unknown.
Data for this message is currently defined as follows:
+---+---+---+---+
| Flags |
+---+---+---+---+
| Major Version |
+---+---+---+---+
| Minor Version |
+---+---+---+---+
| Patch Version |
+---+---+---+---+
| Client |
+---+---+---+---+
| 28-bytes Reserved...
+
Flags = 1-byte of Bit Flags defined as follows:
- Bit 1: EVNT message stream is supported
- Bits 2-8: Reserved
Major Version = Major version of client
Minor Version = Minor version of client
Patch Version = Patch version of client
Client = 4-character ASCII identifier of the client. All "LC" prefix are reserved. See Known Clients.
Response
Implementations MUST accept a length of up to 32 bytes, and discard any unknown data. Unknown data is reserved for future protocol expansions and MUST be all zero.
This message MUST be sent in response to receiving a HELO message at the start of a connection.
The structure of this message is the same as the HELO message.
Request
Mandatory length of 0 and no data.
A PING message SHOULD be sent after a period of inactivity on the connection. Sending PING messages periodically ensures that stateful firewalls do not cause the connection to be broken when the connection is needed again.
Response
Mandatory length of 0 and no data.
A PONG message MUST be sent after a PONG message has been received.
Request
A JSON payload of events, identified by a Nonce, compressed using the ZLIB compression format. The length of the message MUST be the length of the compressed data plus 16 bytes for the Nonce.
The Nonce MUST be 16-bytes and SHOULD be of randomly generated bytes that uniquely identifies this payload of events globally. The generated bytes MUST uniquely identify this payload at a very minumum within the current connection.
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| Nonce (16B) |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| Compressed data...
+
Each event within the compressed data MUST be back-to-back and structured as shown below, with a 4-byte length preceding the JSON object for the event.
+---+---+---+---+---+---+---+---+
| Length (4B) | JSON data ...
+---+---+---+---+---+---+---+---+
| Length (4B) | JSON data ...
+---+---+---+---+---+---+---+---+
| ...
+
Multiple JDAT messages CAN be transmitted consecutively without waiting for an ACKN message response from the server.
If a server fails to decompress a JDAT message, it MUST disconnect the client immediately.
Implementations MAY enforce a limit on the length of the message, and the sending and receiving side SHOULD be configured with the same limits.
Request
A client MUST not send EVNT messages unless previously negotiated via the HELO and VERS handshake, with the VERS response including the bit flag for EVNT message support.
See JDAT. This is the same data format as the JDAT message, with the exception that the length of the message is specified as a stream, using 0xFFFFFFF. The end of the message is demarcated by the end of the compression.
Implementations MAY enforce a limit on the length of the stream, and the sending and receiving side SHOULD be configured with the same limits.
Response
An acknowledgement for a payload of events, containing the Nonce that identifies the payload globally and the number of events processed so far in that payload. The length of the message MUST be 20 bytes.
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| Nonce (16B) |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| Sequence No |
+---+---+---+---+
If a client receives an ACKN message with a Nonce that does not correspond to any payload of events the client submitted to the server on the same connection, it MUST ignore the message.
If a client receives an ACKN message with a Nonce that identifies a payload of events previously transmitted to the server on the same connection, but the sequence number specified is out of range (i.e. higher than the number of events transmitted,) it MUST process the ACKN message as if the sequence number was exactly the number of events in the payload. Similarly, if the sequence number is less than the sequence received in a previous ACKN message for the same payload of events, it MUST be ignored.
If a client receives an ACKN message with a Nonce that identified a payload of events that has not yet completed transmission, it MUST still process this message asynchronously while the payload of events is still being transmitted.
A client MUST be able to process ACKN asynchronously whilst sending data in order to prevent blocking within the connection.
Mandatory length of 0 and no data.
This message should be sent in response to a message that does not exist in the protocol version the server or client connected implements. This is used to allow for backwards compatibility in future versions of the protocol.
- \x00\x00\x00\x00 = (All zeros) Unspecified client
- LCOR = Log Courier
- LCVR = Log Carver