Create rate limiting monitor for Search

[jilladams]

[2024-10-04] Blocked on confirming our rate limit with Search.gov and completing cutover to the new API endpoint

Description

Today when we moved to the new search API endpoint (#18736) , we hit a rate limit, but we didn't get a forewarning. We learned it when the VA.gov - Search Browser Test alarmed, showing that Search calls in the browser were failing.

We don't have an alarm that explicitly goes off related to Search rate limiting and we probably should.

We do this for VA Forms: https://vagov.ddog-gov.com/monitors/160407, based on the rate limit for LH API user key. The monitor counts calls to the API every minute, warning at 400, alarming at 450.

We already know we have weird rate limiting problems with the search_click_tracking endpoint (#14819), and that doesn't appear to affect traffic to the actual search endpoint / get it rate limited. Maybe our rate limits are per endpoint? Or if they're per API key, we are using different keys per endpoint?

Either way, there's some kind of current firewall between v0/search and search_click_tracking already, so once Search.gov verifies our rate limits for v0/search to the new gsa endpoint, we could set up a monitor similar to the Forms one.

ACs

• A Datadog alarm is configured to warn when traffic to v0/search approaches our Search.gov rate limit
• Monitor warns / alarms to #public-websites-dd-search
• Search monitoring docs are updated to describe the new alarm: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/on-site-search/engineering/monitoring.md