Add a ticket level alert for prep volume disk pressure

Our prep filesystems can have large and rapid fluctuations
in utilization. They are also often close to full without this
being a problem. This makes alerts challenging. However, it
seems reasonable to receive a ticket level alert for 98% full
on any prep filesystem, regardless of fstype. This alert
as written does rely on prep filesystems having the word
prep in their mount point, but that is almost always the case.

templates/profile/prometheus/rules.yml.erb

@@ -141,6 +141,24 @@ groups:
     for: 5m
     labels:
       severity: page
+  - alert: PrepDiskPressure
+    annotations:
+      summary: 'Prep filesystem {{$labels.hostname}}:{{$labels.mountpoint}} is more than 98% full.'
+    expr: >
+      (
+        (
+          avg_over_time(
+            node_filesystem_size_bytes{mountpoint=~"prep"}[1m]
+          ) - avg_over_time(
+            node_filesystem_avail_bytes[1m]
+          )
+        ) / avg_over_time(
+          node_filesystem_size_bytes[1m]
+        )
+      ) > 0.98
+    for: 30m
+    labels:
+      severity: ticket
   - alert: HTDataDenIsFull
     annotations:
       summary: 'Filesystem {{$labels.hostname}}:{{$labels.mountpoint}} is full.'