feat!: sesame stack which does not execute in interrupt context (#514)

* Add infra/stream/AtomicByteDeque * Add infra/event/AtomicTriggerScheduler * hal/interfaces/SerialCommunication: Add BufferedSerialCommunication and BufferedSerialCommunicationOnUnbuffered * services/util: Change MessageCommunicationCobs and MessageCommunicationWindowed to get them off interrupts * infra/stream/AtomicByteDeque: Add missing include * services/echo_console/Main: Use BufferedSerialCommunication * Resolve SonarQube warnings * services/util/MessageCommunication: Remove MessageCommunicationReceiveOnInterruptObserver * infra/util/BoundedDeque: Make insert/erase at begin() efficient * services/util/MessageCommunicationCobs: Refactor * services/util: Add TracingMessageCommunicationWindowed * infra/util/BoundedDeque: Make behaviour more logical when inserting on an empty deque * Apply clang-format * Add SESAME documentation * Sesame.adoc: Small improvement on Message packet * services/util/MessageCommunicationCobs: Send starting delimiter only on first packet * services/util/MessageCommunicationWindowed: Only send release window packets when window size is available * Update services/util/test/TestMessageCommunicationWindowed.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * services/util/MessageCommunicationWindowed: initResponse and releaseWindow now consume window size * services/util/MessageCommunicationWindowed: Consume actual COBS size * services/util/MessageCommunication: Refactor sizes * services/util: Rename MessageCommunication to Sesame * services/util: Rename MessageCommunication to Sesame * services/util/SesameWindowed: Compute max message size such that two messages fit in the cobs layer * services/util/SesameWindowed: Compute max message size such that two messages fit in the cobs layer * services/util: Rename MessageCommunication to Sesame * infra/util/BoundedString: Add non-const StdStringAsByteRange * services/util/SesameWindowed: Correctly administrate released window sizes * services/util/SesameWindowed: Don't advertise larger MaxSendMessageSize than the peer can handle * Revert "infra/util/BoundedString: Add non-const StdStringAsByteRange" This reverts commit 79a7b1a. * services/util/test/TestSesameWindowed: Try to unbreak SonarQube * infra/event/AtomicTriggerScheduler: First copy action before resetting scheduled * services/util/TracingSesameWindowed: Adding tracing for sending packets * infra/util/BoundedDeque: correctly modify start * infra/util/BoundedDeque: correctly modify start * services/util/SesameCobs: Handle new message during SendMessageStreamAvailable * services/uti/SesameWindowed: Fix moment of releasing sending * services/echo_console/Console: Concatenate incoming data * services/util/SesameWindowed: Trace SettingOperational * services/util/SesameWindowed: Avoid underflow * services/util/SesameCobs: Don't recursively call DataReceived * services/echo_console/Main: Fix usage of substr * services/echo_console/Main: Fix usage of substr * documents/.../Sesame.adoc: Fix endianness * services/util/SesameWindowed: Refactor * services/util/EchoInstantiation: Use BufferedSerialCommunication iin EchoForwardedToSerial * infra/util/BoundedVector: Fix cases where move_up results in uninitialized memory * Move TracingSesameWindowed from services/util to services/tracer to break a dependency cycle * services/util/SesameSecured: Don't mark destructor as override * services/echo_console/Console: Fix for Concatenate incoming data * services/util/Sesame: Add controlled stopping * protobuf/echo/TracingEcho: Generalize TracingEchoOnStreamsDescendant from TracingEchoOnConnection, add TracingEchoOnSesame * protobuf/echo/TracingEcho: Correctly set serializer * protobuf/echo/TracingEcho: Mark overridden functions as override * services/tracer/Tracer: Extract TracerToStream from Tracer, add TracerToDelegate and TracerColoured * protobuf/echo/TracingEcho: Fix direction of < and > * Use TracerToStream instead of Tracer in tests and applications * protobuf/echo/TracingEcho: Handle tracing of messages split over multiple packets * Sesame.adoc: Add sequence diagram * services/tracer/Tracer: Add resetColour * protobuf/echo/TracingEcho: Fix tracing of forwarded messages * protobuf/echo/TracingEcho: Apply clang-format * protobuf/echo: When a message is split over multiple packets, and when an iniitalize is received in between those packets, the rest of the message is not sent * services/util/Sesame: Add Reset() * services/util/EchoOnSesame: Add Reset() * services/util/SesameWindowed: Fix tracing otherAvailableWindow in Initialized messages * services/util/EchoOnSesame: Reset initialized * Update hal/interfaces/SerialCommunication.hpp * services/network_instantiations/NetworkAdapter: Add ConnectionFactoryWithNameResolver * protobuf/echo/TracingEcho: fix check on contents type * services/network/EchoOnConnection: Only forward AckReceived if still attached * services/network/WebSocketServerConnectionObserver: On AckReceived, when erasing data from the receive buffer also reset the receive stream * services/network_instantiations/NetworkAdapter: Add ExecuteUntil and NetworkActivity * services/network/WebSocketClientConnectionObserver: Remove unused parameter * services/network/TracingEchoOnConnection: Fix merge error * protobuf/protoc_echo_plugin/ProtoCEchoPlugin: Generate field sizes * protobuf/protoc_echo_plugin/ProtoCEchoPlugin: Make field sizes constexpr * fix echo console * Make requests more robust * Add CancelRequestSend to EchoMock * Reinstate MessageCommunication for backwards compatibility * Fix services/util/EchoInstantiation * Process Sonar findings * feat: add echo instantiation functions (#682) * services/util/EchoInstantiation: Add TracingEchoOnUart * Add infra/timer/Waiting * Add OpenEcho * Exclude uart instantiation from embedded builds * services/network_intantiations/EchoInstantiation: Add OpenTracingEcho * protobuf/echo/Echo.cpp: Refactor so that services not registered still get logged if their tracer is registered * services/network/test/TestHttpClient: Remove ambiguity * services/echo_console/Main: Remove ambiguity * protobuf/echo/Echo: Refactor readers * services/util/test/TestEchoOnSesame: Update test * protobuf/echo/test_doubles/EchoSingleLoopback: Keep data alive until it is read * protobuf/echo/TracingEcho: fix skipping messages that are too long * protobuf/echo/TracingEcho: fix skipping messages that are too long * protobuf/echo/TracingEcho: Fix tracing of messages for unknown services * Process review comments * Remove commented out code * Replace UartWindows/UartUnix by UartGeneric * Replace UartWindows/UartUnix by UartGeneric --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Arun Magi <[email protected]>
philips-software · Jul 25, 2024 · 4dc5736 · 4dc5736
1 parent 9cd8ba1
commit 4dc5736
Show file tree

Hide file tree

Showing 112 changed files with 4,880 additions and 725 deletions.
diff --git a/documents/modules/ROOT/examples/SesameSecurity.proto b/documents/modules/ROOT/examples/SesameSecurity.proto
@@ -0,0 +1 @@
+../../../../services/util/SesameSecurity.proto
diff --git a/documents/modules/ROOT/nav.adoc b/documents/modules/ROOT/nav.adoc
@@ -2,5 +2,6 @@
 * xref:MemoryRange.adoc[MemoryRange]
 * xref:Containers.adoc[Containers]
 * xref:Echo.adoc[Echo]
+* xref:Sesame.adoc[Sesame]
 * xref:NetworkConnections.adoc[NetworkConnections]
 * xref:CodingStandard.adoc[Coding Standard]
diff --git a/documents/modules/ROOT/pages/Sesame.adoc b/documents/modules/ROOT/pages/Sesame.adoc
@@ -0,0 +1,216 @@
+= SESAME (SErial Secure Adaptable Message Exchange)
+:source-highlighter: highlight.js
+:highlightjs-languages: protobuf
+
+== Introduction
+
+Microcontrollers often communicate with each other using a UART connection.
+Before such a serial connection can be used by the xref:Echo.adoc[ECHO] protocol,
+a mechanism is needed to initialize communication and prevent overflowing buffers
+on the peer. Furthermore, a means to secure the communication is required.
+For these purposes the SESAME protocol stack is developed.
+
+SESAME consists of several layers, first using link:https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing[COBS] 
+to divide the stream of bytes into delimited packets,
+second a window protocol to prevent overflowing buffers,
+and third (optional) a layer that provides security.
+
+[plantuml]
+----
+@startuml
+node node [
+ECHO
+====
+Secured
+....
+Windowed
+....
+COBS
+====
+Serial Communication
+]
+@enduml
+----
+
+== SESAME layer COBS
+
+The first layer divides the stream of bytes of the serial communication
+into a number of packets. As specified by the link:https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing[COBS]
+protocol, packets are separated by 0 bytes, and any 0 bytes in the payload
+are exchanged for non-0 bytes on the serial communication layer. Please see
+the link:https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing[COBS] specification
+for details.
+
+== SESAME layer Windowed
+
+Using the Windowed layer, both sides of the communication notify and update their peer of
+available window size. A peer can only send packets up to the available window. Four
+packet types are defined:
+
+1. Init
+2. InitResponse
+3. ReleaseWindow
+4. Message
+
+=== Init
+
+Sending an Init packet initializes the protocol. An Init packet has one 16-bit unsigned parameter `size`
+which specifies the available window size, encoded in little endian order. An Init packet must be responded to with an InitResponse
+packet. Before receiving an InitResponse, a peer must assume that no window is available, so
+further packets may not be sent.
+
+.Init packet
+[bytefield]
+----
+(def boxes-per-row 3)
+(draw-column-headers)
+(draw-box 1)
+(draw-box "size" {:span 2})
+----
+
+An Init packet consumes no window, and may always be sent. This results in an edge case that an
+Init packet sent at an inconvenient moment may overflow the peer's buffer. In that case, that
+overflow must result in the peer sending an Init packet of its own. Since a sender
+has an empty buffer before sending an Init packet, that will not result in a repeated exchange of
+Init packets.
+
+=== InitResponse
+
+When an Init packet has been received, a host will clear its buffer, and advertise available buffer
+space using the InitResponse packet. Similar to the Init packet, the InitResponse packet
+has one 16-bit unsigned parameter `size` encoded in little endian which advertises available buffer size.
+
+.InitResponse packet
+[bytefield]
+----
+(def boxes-per-row 3)
+(draw-column-headers)
+(draw-box 2)
+(draw-box "size" {:span 2})
+----
+
+An InitResponse packet consumes 5 window bytes: 3 for the packet contents, 2 for the 
+COBS layer (1 for the COBS overhead byte, one for the terminating 0).
+
+=== ReleaseWindow
+
+When a host has processed incoming packets, and therefore frees up buffer space,
+it advertises the freed up space using the ReleaseWindow packet. The ReleaseWindow packet
+has one 16-bit unsigned parameter `size` encoded in little endian which specifies the amount of bytes in the buffer
+freed up in addition to already known free space.
+
+.ReleaseWindow packet
+[bytefield]
+----
+(def boxes-per-row 3)
+(draw-column-headers)
+(draw-box 3)
+(draw-box "size" {:span 2})
+----
+
+In order to avoid continuous exchanges of ReleaseWindow packets, hosts should only release
+window sizes bigger than 5. When determining the buffer size needed to exchange packets, the
+size of the biggest packet should be increased by 5 to accommodate for a ReleaseWindow packet.
+
+=== Message
+
+Data sent by higher protocol layers are sent by Message packets. Each Message packet consumes a window
+amount equal to the size of that packet plus the COBS overhead of that specific packet, plus
+its terminating 0.
+
+.Message packet
+[bytefield]
+----
+(def boxes-per-row 8)
+(draw-column-headers)
+(draw-box 4)
+(draw-gap "message")
+(draw-bottom)
+----
+
+== SESAME layer Secured
+
+This protocol layer provides confidentiality and integrity by encrypting packets and appending
+a MAC (Message Authentication Code). Since SESAME is geared towards embedded systems, and is
+not intended to be as flexible as a protocol like TLS, for simplicity some very specific choices
+are made regarding cryptographic schemes.
+
+Encryption and authentication is done with AES-128 in
+link:https://en.wikipedia.org/wiki/Galois/Counter_Mode[GCM] mode. AES-128-GCM requires a 128 bit
+key and an 128 bit IV. Both directions of communication require their own unique key.
+
+Each message is encrypted with AES-128-GCM with the key and IV as parameters. This results in
+a cyphertext of size equal to the size of the message, plus a 16 byte MAC. Since in GCM IVs may
+not be reused, the IV is incremented in a specific way to avoid using the same IV twice: After
+each message, treat the IV as a 128-bit unsigned number, and add 2^64^ to it, discarding any overflow.
+This way, the protocol can securely handle 2^64^ messages each of size at most 2^64^ before a
+re-negotiation of keys is necessary.
+
+Key establishment is not a responsiblity of the Secured layer; but care must be taken that
+the same key/IV pair should not be reused for different sessions, since encrypting two messages
+using the same key/IV pair will result in the XOR of the bit pattern of those two plaintext
+messages to be equal to the XOR of the bit pattern of the two cyphertexts of those messages.
+
+There is one exception to that rule: If the first message sent consists of at least 128 bit of
+random data, then security is not compromised. The only information leaked is the XOR of two
+times 128 bits of random data, from which an attacker still learns nothing. One possible scheme
+of key establishment is therefore to hardcode the four Key and IV values for sending and receiving 
+on both hosts, and to choose random new Key/IV values right after initialization.
+
+After each initialization of the lower protocol layers (after receiving the Init packet in the
+Windowed layer), keys are reset to their default value, and key negotiation must start over
+before other messages are sent.
+
+=== ECHO on SESAME
+
+ECHO is used over SESAME by serializing ECHO messages, dividing those messages into a series
+of chunks suitable to send over SESAME (so the maximum buffer size advertised by the peer must
+be taken into account), and sending those chunks. On the receiving end, those chunks are reassembled,
+and parsed as ECHO messages. Since ECHO is able to handle a continuous stream of data, multiple
+ECHO messages being concatenated do not pose any difficulties.
+
+=== Symmetric key establishment over ECHO
+
+Specifying the exact key negotiation scheme is outside of the scope of this specification,
+since there are many different situations which require different negotiation schemes. However,
+if a key negotiation scheme is used that results in a host communicating to its peer that
+a specific Key/IV pair is to be used, then the SymmetricKeyEstablishment ECHO service can be used.
+This service has one method ActivateNewKeyMaterial with parameters Key and IV. After receiving this
+message, a sender of that message will use those Key/IV for subsequent messages towards the receiver.
+A full key negotiation will therefore result in ActivateNewKeyMaterial invocations in both directions.
+
+== Appendix: Typical sequence
+
+This sequence diagram shows a typical sequence of the SESAME protocol being initialized and a message
+being sent.
+
+.Typical sequence
+[plantuml]
+----
+@startuml
+COBS <- Windowed : Init\n01 10 00
+note right
+  16 bytes of window are advertised, which is 0x10 0x00
+end note
+[<- COBS : 00 03 01 10 01 00
+note right
+  The init message is wrapped right and left with 00
+  A 03 indicates that the third byte must be replaced with 00
+  The replaced byte contained 01 pointing to the end
+end note
+[-> COBS : 00 03 02 10 01 00
+COBS -> Windowed : InitResponse\n02 10 00
+Windowed <- Application : Message\n12 34 56 78
+COBS <- Windowed : Message\n03 12 34 56 78
+[<- COBS : 06 03 12 34 56 78 00
+[-> COBS : 03 03 07 01 00
+COBS -> Windowed : ReleaseWindow\n03 07 00
+@enduml
+----
+
+== Appendix: SesameSecurity.proto
+
+[source,protobuf]
+----
+include::SesameSecurity.proto[]
+----
diff --git a/documents/modules/ROOT/pages/index.adoc b/documents/modules/ROOT/pages/index.adoc
@@ -3,7 +3,7 @@
 
 == Introduction
 
-amp-embedded-infra-lib is a set of C++ libraries and headers that provide heap-less, https://en.wikipedia.org/wiki/Standard_Template_Library[STL] like, infrastructure for embedded software development. It includes, amongst others; a hardware abstraction layer (HAL), a xref:Echo.adoc[remote procedure call] (RPC) implementation, a xref:NetworkConnections.adoc[networking layer], a secure upgrade mechanism and several other re-usable utility classes.
+amp-embedded-infra-lib is a set of C++ libraries and headers that provide heap-less, https://en.wikipedia.org/wiki/Standard_Template_Library[STL] like, infrastructure for embedded software development. It includes, amongst others; a hardware abstraction layer (HAL), a xref:Echo.adoc[remote procedure call] (RPC) implementation for TCP/IP and xref:Sesame.adoc[Serial communication], a xref:NetworkConnections.adoc[networking layer], a secure upgrade mechanism and several other re-usable utility classes.
 
 amp-embedded-infra-lib is available as https://philips-software.github.io/[Philips Open Source] under a https://choosealicense.com/licenses/mit/[MIT] license.
 
@@ -32,7 +32,6 @@ Rating]]
         component echo
         component echo_console
         component protoc_echo_plugin
-        component services
         component more_protobuf [
             ...
         ]

diff --git a/examples/serial_net/Main.cpp b/examples/serial_net/Main.cpp
@@ -41,7 +41,7 @@ int main(int argc, const char* argv[], const char* env[])
     static main_::NetworkAdapter network;
     static hal::TimerServiceGeneric timerService;
     static infra::IoOutputStream ioOutputStream;
-    static services::Tracer tracer(ioOutputStream);
+    static services::TracerToStream tracer(ioOutputStream);
 
     static infra::BoundedVector<SerialBridge>::WithMaxSize<8> serialBridges;
 

diff --git a/hal/generic/CMakeLists.txt b/hal/generic/CMakeLists.txt
@@ -7,6 +7,10 @@ target_link_libraries(hal.generic PUBLIC
     $<$<OR:$<BOOL:${EMIL_BUILD_UNIX}>,$<BOOL:${EMIL_BUILD_DARWIN}>>:pthread>
 )
 
+target_compile_definitions(hal.generic PUBLIC
+    EMIL_HAL_GENERIC
+)
+
 target_sources(hal.generic PRIVATE
     FileSystemGeneric.cpp
     FileSystemGeneric.hpp

diff --git a/hal/interfaces/CMakeLists.txt b/hal/interfaces/CMakeLists.txt
@@ -39,6 +39,7 @@ target_sources(hal.interfaces PRIVATE
     QuadSpi.hpp
     RandomDataGenerator.hpp
     Reset.hpp
+    SerialCommunication.cpp
     SerialCommunication.hpp
     Spi.cpp
     Spi.hpp

diff --git a/hal/interfaces/SerialCommunication.cpp b/hal/interfaces/SerialCommunication.cpp
@@ -0,0 +1,40 @@
+#include "hal/interfaces/SerialCommunication.hpp"
+
+namespace hal
+{
+    BufferedSerialCommunicationOnUnbuffered::BufferedSerialCommunicationOnUnbuffered(infra::AtomicByteQueue& buffer, SerialCommunication& delegate)
+        : buffer(buffer)
+        , delegate(delegate)
+    {
+        delegate.ReceiveData([this](infra::ConstByteRange data)
+            {
+                this->buffer.Push(data);
+                scheduler.Schedule([this]()
+                    {
+                        if (HasObserver())
+                            GetObserver().DataReceived();
+                    });
+            });
+    }
+
+    BufferedSerialCommunicationOnUnbuffered::~BufferedSerialCommunicationOnUnbuffered()
+    {
+        delegate.ReceiveData([](infra::ConstByteRange) {});
+    }
+
+    void BufferedSerialCommunicationOnUnbuffered::SendData(infra::ConstByteRange data, infra::Function<void()> actionOnCompletion)
+    {
+        delegate.SendData(data, actionOnCompletion);
+    }
+
+    infra::StreamReaderWithRewinding& BufferedSerialCommunicationOnUnbuffered::Reader()
+    {
+        return reader;
+    }
+
+    void BufferedSerialCommunicationOnUnbuffered::AckReceived()
+    {
+        buffer.Pop(reader.ConstructSaveMarker());
+        reader.Rewind(0);
+    }
+}
diff --git a/hal/interfaces/SerialCommunication.hpp b/hal/interfaces/SerialCommunication.hpp
@@ -1,8 +1,11 @@
 #ifndef HAL_SERIAL_COMMUNICATION_HPP
 #define HAL_SERIAL_COMMUNICATION_HPP
 
+#include "infra/event/AtomicTriggerScheduler.hpp"
+#include "infra/stream/AtomicByteQueue.hpp"
 #include "infra/util/ByteRange.hpp"
 #include "infra/util/Function.hpp"
+#include "infra/util/Observer.hpp"
 
 namespace hal
 {
@@ -16,6 +19,50 @@ namespace hal
         virtual void SendData(infra::ConstByteRange data, infra::Function<void()> actionOnCompletion) = 0;
         virtual void ReceiveData(infra::Function<void(infra::ConstByteRange data)> dataReceived) = 0;
     };
+
+    class BufferedSerialCommunication;
+
+    class BufferedSerialCommunicationObserver
+        : public infra::SingleObserver<BufferedSerialCommunicationObserver, BufferedSerialCommunication>
+    {
+    public:
+        using infra::SingleObserver<BufferedSerialCommunicationObserver, BufferedSerialCommunication>::SingleObserver;
+
+        virtual void DataReceived() = 0;
+    };
+
+    class BufferedSerialCommunication
+        : public infra::Subject<BufferedSerialCommunicationObserver>
+    {
+    public:
+        virtual void SendData(infra::ConstByteRange data, infra::Function<void()> actionOnCompletion) = 0;
+
+        virtual infra::StreamReaderWithRewinding& Reader() = 0;
+        virtual void AckReceived() = 0;
+    };
+
+    class BufferedSerialCommunicationOnUnbuffered
+        : public BufferedSerialCommunication
+    {
+    public:
+        template<std::size_t Size>
+        using WithStorage = infra::WithStorage<BufferedSerialCommunicationOnUnbuffered, infra::AtomicByteQueue::WithStorage<Size + 1>>;
+
+        BufferedSerialCommunicationOnUnbuffered(infra::AtomicByteQueue& buffer, SerialCommunication& delegate);
+        ~BufferedSerialCommunicationOnUnbuffered();
+
+        // Implementation of BufferedSerialCommunication
+        void SendData(infra::ConstByteRange data, infra::Function<void()> actionOnCompletion) override;
+        infra::StreamReaderWithRewinding& Reader() override;
+        void AckReceived() override;
+
+    private:
+        infra::AtomicByteQueue& buffer;
+        SerialCommunication& delegate;
+
+        infra::AtomicByteQueueReader reader{ buffer };
+        infra::AtomicTriggerScheduler scheduler;
+    };
 }
 
 #endif
diff --git a/hal/interfaces/test/CMakeLists.txt b/hal/interfaces/test/CMakeLists.txt
@@ -17,4 +17,5 @@ target_sources(hal.interfaces_test PRIVATE
     TestI2cRegisterAccess.cpp
     TestMacAddress.cpp
     TestQuadSpi.cpp
+    TestSerialCommunication.cpp
 )