Add prometheus metrics to measure hook execution time

[mcastorina]

Description:

Add prometheus metrics to measure hook execution time.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[ahrav]

How do you feel about putting this in a metrics.go file similar to how we currently do it for other parts of the code. (ie github, gitlab, etc)?

[ahrav]

question: have you tested this in a dev/local env?

Since this needs to be sent over the network even with compression there is a chance this exceeds our current gRPC payload limit. I only mention this because this one of the main reasons I had to change our metrics for detectors to only include 3-4 bucket values due to the enormous size of the payload.

[ahrav]

I added compression and also increased the payload limit on the API server, but i'm not sure if that would be enough for these metrics. You might be able to also increase it some more if needed here

[mcastorina]

Ah, good callout. I wasn't aware that the number of buckets affected the size of the payload. These buckets were copied from another metric in thog (but on the API side, so no GRPC messaging). I'll reduce them.

[ahrav]

Funnily enough I did the exact same thing and copied the exact same metrics when I added them for our detectors 🤣 .

[rosecodym]

Ooh, I didn't know that either.

[mcastorina]

How do you feel about putting this in a metrics.go file similar to how we currently do it for other parts of the code. (ie github, gitlab, etc)?

I considered it, but felt like I was adding too many files to the sources package. Should SourceManager et al. be in it's own package?

[ahrav]

How do you feel about putting this in a metrics.go file similar to how we currently do it for other parts of the code. (ie github, gitlab, etc)?

I considered it, but felt like I was adding too many files to the sources package. Should SourceManager et al. be in it's own package?

I personally don't mind either approach tbh. Gonna leave that up to you, or anyone else that feels strongly. You can also do that in a separate PR if that's easier.

[rosecodym]

I weakly vote for moving it into its own metrics file - I've found it nice to be able to see how metrics work by looking at just one file - but I really don't think it matters much either way.

[rosecodym]

Ooh, I didn't know that either.