lib: user provided threads and packets - v5 #11769
Conversation
Add library source and runmode modules. Reorganized library example to create a worker thread and replay a pcap file using the library mode. No API layer is added at this stage. Edits by Jason Ish: - fix guard - add copyright/license headers
To keep the simple example simple, move the lib based capture method example to its own example.
- the generated binaries for lib examples - LSP files - man pages
Worker threads not created by Suricata, but instead a library user should not be joined, as Suricata does not have access to their thread handle, and it may in-fact be an unjoinable thread, such as the main process. When the thread ID is 0, assume the thread is "externally" managed, but still mark is as dead to satisfy Suricata's view of the thread.
Use the more conventional "--" command line handling to separate the arguments. The first set will be passed to Suricata, and the args after "--" will be handled by the example. Currently this is a single PCAP filename, but will be extended to a list of PCAP filenames.
Also use a proper return type (ThreadVars *).
Update ThreadVars creation in lib mode to have the worker_id provided by the user.
In the library capture example, show how the packet counter can be updated.
The variable run is not needed, we can just run an infinite loop and break when needed.
Refactor TmThreadsSlotPktAcqLoop for user provided thread by breaking out the init and finish code into their own functions. For user provided threads, Suricata should not "drive" the thread, but the setup and finish code is the same. The finish function is exported so it can be called by the user application when its receive loop or equivalent is done.
And add SC prefix.
Also remove function to set the library mode. This is easy enough to do with SCRunmodeSet, and we don't want to add a specific setter for each and every runmode.
Add a release packet callback where the action can be checked for drop.
| PKT_SET_SRC(p, PKT_SRC_WIRE); | ||
| p->ts = SCTIME_FROM_TIMEVAL(&pkthdr.ts); | ||
| p->datalink = datalink; | ||
| p->livedev = device; | ||
| p->ReleasePacket = ReleasePacket; |
There was a problem hiding this comment.
These all need functions to avoid internal access to packet.
| usleep(100); | ||
|
|
||
| SuricataPostInit(); | ||
|
|
There was a problem hiding this comment.
Need something like SuricataMainLoop here.
| int TmModuleLibHandlePacket(ThreadVars *tv, LiveDevice *device, const uint8_t *data, int datalink, | ||
| struct timeval ts, uint32_t len, uint32_t tenant_id, uint32_t flags) |
There was a problem hiding this comment.
This can now go. I think most users of the lib who want to drive their own packets through won't be satisfied with this abstraction where they need a better handle on the Packet pre and post, and perform IPS as can be seen with the updated example in this PR.
| if (data == NULL) { | ||
| TmThreadsSetFlag(tv, THV_CAPTURE_INJECT_PKT); | ||
| TmThreadsCaptureHandleTimeout(tv, NULL); | ||
| SCReturnInt(TM_ECODE_OK); | ||
| } |
There was a problem hiding this comment.
@victorjulien Any guidelines as when a custom capture method should push something like this through?
| /* If we are processing a PCAP and it is the first packet we need to set the timestamp. */ | ||
| SCTime_t timestamp = SCTIME_FROM_TIMEVAL(&ts); | ||
| if (!time_set && !TimeModeIsLive()) { | ||
| TmThreadsInitThreadsTimestamp(timestamp); | ||
| time_set = true; | ||
| } |
There was a problem hiding this comment.
Left this out of the example app, I think if a library user is doing offline mode, they could "prime" this early by peaking into their packet source, or locking on some conditional until set. Anyways, when the threads and packets are in the users control there are a few ways to do. Just need documentation and it should still be in the example.
| static void *TmThreadsSlotPktAcqLoop(void *td) | ||
| static bool TmThreadsSlotPktAcqLoopInit(ThreadVars *tv) |
There was a problem hiding this comment.
@victorjulien Would like your eyes on how I broke this function up as a user driving there thread needs this to return, but then do the finish stuff later on.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #11769 +/- ##
==========================================
- Coverage 82.62% 82.49% -0.14%
==========================================
Files 919 921 +2
Lines 248979 249105 +126
==========================================
- Hits 205722 205499 -223
- Misses 43257 43606 +349
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
ERROR: ERROR: QA failed on SURI_TLPR1_alerts_cmp. Pipeline 22592 |
Update of #11711 to show how packet dropping might be handled.