implement release_all_elements method on StreamTracer

[lucila]

What and why? 🤔

We want to track when filters are the cause of stream exhaustion. This can happen when a filter releases all elements, the remaining elements is empty and if we don't set retry and overfetch properly, it can cause stream exhaustion.

On this PR we implement release_all_elements on StreamTracer. We call this method on the filter step of StreamFilter so that we can track when a filter releases all the elements on the filter step.

Testing Steps ✍️

Provide adequate testing steps (including examples if necessary).
Don't assume someone knows how to test your changes; be thorough.

• Bullet point checklists
• Are useful for some tests

You're it! 👑

@Automattic/stream-builders

[coveralls]

coverage: 79.439% (+0.04%) from 79.4%
when pulling e3022ab on lucila/tick-all-elements-released
into 6cca8f6 on main.

[nicola-barbieri]

can we ALSO report the % of released elements ?
we can interpret it more clearly: e.g. this filter is filtering out 80% of the posts

[lucila]

if we report the % of released elements, do we need the release_all tick?

this would be the diff for the % of released elements implementation:

diff --git a/lib/Tumblr/StreamBuilder/StreamFilters/StreamFilter.php b/lib/Tumblr/StreamBuilder/StreamFilters/StreamFilter.php
index 621cb17..70a772e 100644
--- a/lib/Tumblr/StreamBuilder/StreamFilters/StreamFilter.php
+++ b/lib/Tumblr/StreamBuilder/StreamFilters/StreamFilter.php
@@ -118,7 +118,8 @@ abstract class StreamFilter extends Templatable
             $tracer->end_filter(
                 $this,
                 $filtered->get_released_count(),
-                [$t0, microtime(true) - $t0]
+                [$t0, microtime(true) - $t0],
+                count($elements)
             );
             foreach ($filtered->get_released() as $element) {
                 $tracer->release_element($this, $element);
diff --git a/lib/Tumblr/StreamBuilder/StreamTracers/StreamTracer.php b/lib/Tumblr/StreamBuilder/StreamTracers/StreamTracer.php
index 8963fc2..8736ad3 100644
--- a/lib/Tumblr/StreamBuilder/StreamTracers/StreamTracer.php
+++ b/lib/Tumblr/StreamBuilder/StreamTracers/StreamTracer.php
@@ -387,21 +387,28 @@ abstract class StreamTracer
      * @param StreamFilter $filter The stream filter is applied.
      * @param int $released_count The number of items being filtered.
      * @param array $timing Zero indexed tuple of the start time and duration of the operation (in that order)
+     * @param int|null $total_count The number of elements that went through the filter step, out of which $release_count were filtered.
      * @return void
      */
     final public function end_filter(
         StreamFilter $filter,
         int $released_count,
-        array $timing
+        array $timing,
+        ?int $total_count = 0
     ): void {
+        $meta = [static::META_COUNT => $released_count];
+
+        if ($total_count > 0) {
+            $ratio = $released_count / $total_count;
+            $meta['released_ratio'] = $ratio;
+        }
+
         $this->trace_event(
             static::CATEGORY_FILTER,
             $filter,
             static::EVENT_END,
             $timing,
-            [
-                static::META_COUNT => $released_count,
-            ]
+            $meta
         );
     }

[nicola-barbieri]

if we report the % of released elements, do we need the release_all tick?

I would keep the release_all tick because for our use-case it's more easy to monitor that, but % release is more informative.

[cmantas]

LGTM. I agree with logging both the % and the boolean of exhaustion, but I don't think doing one of the 2 is a blocker.