WAF Ruleset Builder & Release v1.8.0 (#138)

CHANGELOG.md

@@ -1,5 +1,15 @@
 # libddwaf release
 
+### v1.8.0 ([unstable](https://github.com/DataDog/libddwaf/blob/master/README.md#versioning-semantics))
+### API \& Breaking Changes
+- Add `ddwaf_update` for all-in-one ruleset updates ([#138](https://github.com/DataDog/libddwaf/pull/138))
+- Remove `ddwaf_required_rule_data_ids` ([#138](https://github.com/DataDog/libddwaf/pull/138))
+- Remove `ddwaf_update_rule_data` ([#138](https://github.com/DataDog/libddwaf/pull/138))
+- Remove `ddwaf_toggle_rules` ([#138](https://github.com/DataDog/libddwaf/pull/138))
+
+### Changes
+- Add WAF Builder ([#138](https://github.com/DataDog/libddwaf/pull/138))
+
 ### v1.7.0  ([unstable](https://github.com/DataDog/libddwaf/blob/master/README.md#versioning-semantics)) - 2023/02/06
 #### Changes
 - Handle lifetime extension ([#135](https://github.com/DataDog/libddwaf/pull/135))

CMakeLists.txt

@@ -62,9 +62,9 @@ include(GNUInstallDirs)
 add_subdirectory(third_party)
 
 set(LIBDDWAF_SOURCE
+    ${libddwaf_SOURCE_DIR}/src/ruleset_builder.cpp
     ${libddwaf_SOURCE_DIR}/src/clock.cpp
     ${libddwaf_SOURCE_DIR}/src/parameter.cpp
-    ${libddwaf_SOURCE_DIR}/src/waf.cpp
     ${libddwaf_SOURCE_DIR}/src/interface.cpp
     ${libddwaf_SOURCE_DIR}/src/context.cpp
     ${libddwaf_SOURCE_DIR}/src/event.cpp

UPGRADING.md

@@ -1,5 +1,70 @@
 # Upgrading libddwaf
 
+### Upgrading from `1.7.x` to `1.8.0`
+
+Version `1.8.0` introduces the WAF builder, a new module with the ability to generate a new WAF instance from an existing one. This new module works transparently through the `ddwaf_update` function, which allows the user to update one, some or all of the following:
+- The complete ruleset through the `rules` key.
+- The `on_match` or `enabled` field of specific rules through the `rules_override` key.
+- Exclusion filters through the `exclusions` key.
+- Rule data through the `rules_data` key.
+
+The WAF builder has a number of objectives:
+- Provide a mechanism to generate and update the WAF as needed.
+- Remove all existing mutexes.
+- Remove all side-effects on running contexts.
+- __Potentially__ provide efficiency gains: 
+  - Avoiding the need to parse a whole ruleset on every update.
+  - Reusing internal structures, objects and containers whenever possible.
+
+With the introduction of `ddwaf_update`, the following functions have been deprecated and removed:
+- `ddwaf_toggle_rules`
+- `ddwaf_update_rule_data`
+- `ddwaf_required_rule_data_ids`
+
+The first two functions have been removed due to the added complexity of supporting multiple interfaces with a similar outcome but different inputs. On the other hand, the last function was simply removed in favour of letting the WAF handle unexpected rule data IDs more gracefully, however this function can be reintroduced later if deemed necessary.
+
+Typically, the new interface will be used as follows on all instances:
+
+```c
+    ddwaf_handle old_handle = ddwaf_init(&ruleset, &config, &info);
+    ddwaf_object_free(&ruleset);
+
+    ddwaf_handle new_handle = ddwaf_update(old_handle, &update, &new_info);
+    ddwaf_object_free(&update);
+    if (new_handle != NULL) {
+        ddwaf_destroy(old_handle);
+    }
+```
+
+The `ddwaf_update` function returns a new `ddwaf_handle` which will be a valid pointer if the update succeeded, or `NULL` if there was nothing to update or there was an error. Creating contexts while calling `ddwaf_update` is, in theory, perfectly legal as well as destroying a handle while associated contexts are still in use, for example:
+
+```c
+    ddwaf_handle old_handle = ddwaf_init(&ruleset, &config, &info);
+    ddwaf_object_free(&rule);
+
+    ddwaf_context context = ddwaf_context_init(old_handle);
+
+    ddwaf_handle new_handle = ddwaf_update(old_handle, &new_ruleset, &new_info);
+    ddwaf_object_free(&new_rule);
+    if (new_handle != NULL) {
+        // Destroying the handle should not invalidate the context
+        ddwaf_destroy(old_handle);
+    }
+    
+    // Both the context and the handle are destroyed here
+    ddwaf_context_destroy(context);
+```
+Note that the `ddwaf_update` function also has an optional input parameter for the `ruleset_info` structure, this will only provide useful diagnostics when the update provided contains new rules (within the `rules` key), also note that the `ruleset_info` should either be a fresh new structure or the previously used after calling `ddwaf_ruleset_info_free`.
+
+Finally, you can call `ddwaf_init` with all previously mentioned keys, or a combination of them, however the `rules` key is mandatory. This does not apply to `ddwaf_update.
+
+#### Notes on thread-safety
+
+The thread-safety of any operations on the handle depends on whether they act on the ruleset or the builder itself, generally:
+- Calling `ddwaf_update` concurrently, regardless of the handle, is never thread-safe.
+- Calling `ddwaf_context_init` concurrently on the same handle is thread-safe.
+- Calling `ddwaf_context_init` and `ddwaf_update` concurrently on the same handle is also thread-safe.
+
 ### Upgrading from `1.6.x` to `1.7.0`
 
 There are no API changes in 1.7.0, however `ddwaf_handle` is now reference-counted and shared between the user and each `ddwaf_context`. This means that it is now possible to call `ddwaf_destroy` on a `ddwaf_handle` without invalidating any `ddwaf_context` in use instantiated from said `ddwaf_handle`. For example, the following snippet is now perfectly legal and will work as expected (note that any checks have been omitted for brevity):

include/ddwaf.h

@@ -8,12 +8,11 @@
 #define DDWAF_H
 
 #ifdef __cplusplus
-#include <memory>
 namespace ddwaf{
 class waf;
 class context;
 } // namespace ddwaf
-using ddwaf_handle = std::shared_ptr<ddwaf::waf> *;
+using ddwaf_handle = ddwaf::waf *;
 using ddwaf_context = ddwaf::context *;
 
 extern "C"
@@ -207,46 +206,40 @@ typedef void (*ddwaf_log_cb)(
  *
  * Initialize a ddwaf instance
  *
- * @param rule ddwaf::object containing the patterns to be used by the WAF. (nonnull)
+ * @param rule ddwaf::object map containing rules, exclusions, rules_override and rules_data. (nonnull)
  * @param config Optional configuration of the WAF. (nullable)
  * @param info Optional ruleset parsing diagnostics. (nullable)
  *
- * @return Handle to the WAF instance.
+ * @return Handle to the WAF instance or NULL on error.
  *
  * @note If config is NULL, default values will be used, including the default
  *       free function (ddwaf_object_free).
  **/
-ddwaf_handle ddwaf_init(const ddwaf_object *rule,
+ddwaf_handle ddwaf_init(const ddwaf_object *ruleset,
     const ddwaf_config* config, ddwaf_ruleset_info *info);
 
 /**
- * ddwaf_destroy
- *
- * Destroy a WAF instance.
+ * ddwaf_update
  *
- * @param Handle to the WAF instance.
- */
-void ddwaf_destroy(ddwaf_handle handle);
-
-/**
- * ddwaf_update_rule_data
+ * Update a ddwaf instance
  *
- * Update existing rules with new rule data.
+ * @param rule ddwaf::object map containing rules, exclusions, rules_override and rules_data. (nonnull)
+ * @param info Optional ruleset parsing diagnostics. (nullable)
  *
- * @param handle to the WAF instance.
- * @param data A ddwaf_object with the format [{id, type, [data]}].
- */
-DDWAF_RET_CODE ddwaf_update_rule_data(ddwaf_handle handle, ddwaf_object *data);
+ * @return Handle to the new WAF instance or NULL if there were no new updates
+ *         or there was an error processing the ruleset.
+ **/
+ddwaf_handle ddwaf_update(ddwaf_handle handle, const ddwaf_object *ruleset,
+    ddwaf_ruleset_info *info);
 
 /**
- * ddwaf_toggle_rules
+ * ddwaf_destroy
  *
- * Enable or disable rules (true -> rule enabled, false -> rule disabled).
+ * Destroy a WAF instance.
  *
- * @param handle to the WAF instance.
- * @param data A ddwaf_object with the format {rule_id : boolean}.
+ * @param Handle to the WAF instance.
  */
-DDWAF_RET_CODE ddwaf_toggle_rules(ddwaf_handle handle, ddwaf_object *rule_map);
+void ddwaf_destroy(ddwaf_handle handle);
 
 /**
  * ddwaf_ruleset_info_free
@@ -268,18 +261,6 @@ void ddwaf_ruleset_info_free(ddwaf_ruleset_info *info);
  * @return NULL if empty, otherwise a pointer to an array with size elements.
  **/
 const char* const* ddwaf_required_addresses(const ddwaf_handle handle, uint32_t *size);
-/**
- * ddwaf_required_rule_data_ids
- *
- * Get a list of required rule data IDs (if any). The memory is owned by the
- * WAF and should not be freed.
- *
- * @param Handle to the WAF instance.
- * @param size Output parameter in which the size will be returned. The value of
- *             size will be 0 if the return value is NULL.
- * @return NULL if empty, otherwise a pointer to an array with size elements.
- **/
-const char* const* ddwaf_required_rule_data_ids(const ddwaf_handle handle, uint32_t *size);
 
 /**
  * ddwaf_context_init

libddwaf.def

@@ -1,12 +1,10 @@
 LIBRARY ddwaf
 EXPORTS
   ddwaf_init
+  ddwaf_update
   ddwaf_destroy
-  ddwaf_update_rule_data
-  ddwaf_toggle_rules
   ddwaf_ruleset_info_free
   ddwaf_required_addresses
-  ddwaf_required_rule_data_ids
   ddwaf_context_init
   ddwaf_run
   ddwaf_context_destroy

src/collection.cpp

@@ -12,9 +12,10 @@ namespace ddwaf {
 
 namespace {
 std::optional<event> match_rule(const rule::ptr &rule, const object_store &store,
-    const ddwaf::manifest &manifest, std::unordered_map<rule::ptr, rule::cache_type> &cache,
+    std::unordered_map<rule::ptr, rule::cache_type> &cache,
     const std::unordered_set<rule::ptr> &rules_to_exclude,
     const std::unordered_map<rule::ptr, collection::object_set> &objects_to_exclude,
+    const std::unordered_map<std::string, rule_processor::base::ptr> &dynamic_processors,
     ddwaf::timer &deadline)
 {
     const auto &id = rule->id;
@@ -47,9 +48,9 @@ std::optional<event> match_rule(const rule::ptr &rule, const object_store &store
         auto exclude_it = objects_to_exclude.find(rule);
         if (exclude_it != objects_to_exclude.end()) {
             const auto &objects_excluded = exclude_it->second;
-            event = rule->match(store, manifest, rule_cache, objects_excluded, deadline);
+            event = rule->match(store, rule_cache, objects_excluded, dynamic_processors, deadline);
         } else {
-            event = rule->match(store, manifest, rule_cache, {}, deadline);
+            event = rule->match(store, rule_cache, {}, dynamic_processors, deadline);
         }
 
         return event;
@@ -64,18 +65,18 @@ std::optional<event> match_rule(const rule::ptr &rule, const object_store &store
 
 void collection::match(std::vector<event> &events,
     std::unordered_set<std::string_view> & /*seen_actions*/, const object_store &store,
-    const ddwaf::manifest &manifest, collection_cache &cache,
-    const std::unordered_set<rule::ptr> &rules_to_exclude,
+    collection_cache &cache, const std::unordered_set<rule::ptr> &rules_to_exclude,
     const std::unordered_map<rule::ptr, object_set> &objects_to_exclude,
+    const std::unordered_map<std::string, rule_processor::base::ptr> &dynamic_processors,
     ddwaf::timer &deadline) const
 {
     if (cache.result) {
         return;
     }
 
     for (const auto &rule : rules_) {
-        auto event = match_rule(rule, store, manifest, cache.rule_cache, rules_to_exclude,
-            objects_to_exclude, deadline);
+        auto event = match_rule(rule, store, cache.rule_cache, rules_to_exclude, objects_to_exclude,
+            dynamic_processors, deadline);
         if (event.has_value()) {
             cache.result = true;
             events.emplace_back(std::move(*event));
@@ -87,9 +88,9 @@ void collection::match(std::vector<event> &events,
 
 void priority_collection::match(std::vector<event> &events,
     std::unordered_set<std::string_view> &seen_actions, const object_store &store,
-    const ddwaf::manifest &manifest, collection_cache &cache,
-    const std::unordered_set<rule::ptr> &rules_to_exclude,
+    collection_cache &cache, const std::unordered_set<rule::ptr> &rules_to_exclude,
     const std::unordered_map<rule::ptr, object_set> &objects_to_exclude,
+    const std::unordered_map<std::string, rule_processor::base::ptr> &dynamic_processors,
     ddwaf::timer &deadline) const
 {
     auto &remaining_actions = cache.remaining_actions;
@@ -103,15 +104,15 @@ void priority_collection::match(std::vector<event> &events,
 
     // If there are no remaining actions, we treat this collection as a regular one
     if (remaining_actions.empty()) {
-        collection::match(events, seen_actions, store, manifest, cache, rules_to_exclude,
-            objects_to_exclude, deadline);
+        collection::match(events, seen_actions, store, cache, rules_to_exclude, objects_to_exclude,
+            dynamic_processors, deadline);
         return;
     }
 
     // If there are still remaining actions, we treat this collection as a priority tone
     for (const auto &rule : rules_) {
-        auto event = match_rule(rule, store, manifest, cache.rule_cache, rules_to_exclude,
-            objects_to_exclude, deadline);
+        auto event = match_rule(rule, store, cache.rule_cache, rules_to_exclude, objects_to_exclude,
+            dynamic_processors, deadline);
         if (event.has_value()) {
             // If there has been a match, we set the result to true to ensure
             // that the equivalent regular collection doesn't attempt to match

src/collection.hpp

@@ -51,9 +51,10 @@ class collection {
 
     virtual void match(std::vector<event> &events /* output */,
         std::unordered_set<std::string_view> &seen_actions /* input & output */,
-        const object_store &store, const ddwaf::manifest &manifest, collection_cache &cache,
+        const object_store &store, collection_cache &cache,
         const std::unordered_set<rule::ptr> &rules_to_exclude,
         const std::unordered_map<rule::ptr, object_set> &objects_to_exclude,
+        const std::unordered_map<std::string, rule_processor::base::ptr> &dynamic_processors,
         ddwaf::timer &deadline) const;
 
     [[nodiscard]] virtual collection_cache get_cache() const { return {}; }
@@ -79,9 +80,10 @@ class priority_collection : public collection {
 
     void match(std::vector<event> &events /* output */,
         std::unordered_set<std::string_view> &seen_actions /* input & output */,
-        const object_store &store, const ddwaf::manifest &manifest, collection_cache &cache,
+        const object_store &store, collection_cache &cache,
         const std::unordered_set<rule::ptr> &rules_to_exclude,
         const std::unordered_map<rule::ptr, object_set> &objects_to_exclude,
+        const std::unordered_map<std::string, rule_processor::base::ptr> &dynamic_processors,
         ddwaf::timer &deadline) const override;
 
     [[nodiscard]] collection_cache get_cache() const override { return {false, {}, actions_}; }