chore: changing health status thresholds

[samhith-kakarla]

Explain what this PR does.

[samhith-kakarla]

Pipline is reaching warning/critical state when we stop scaling:

[samhith-kakarla]

unit tests were erroring because of change in thresholds so i changed the cutoff values in the tests to be dependent on DefaultBufferUsageLimit and match the new thresholds. Also added TestHealthThresholds to show that thresholds are actually changing as expected when BufferUsageLimit changes

[codecov]

Codecov Report

Attention: Patch coverage is 68.75000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 58.24%. Comparing base (9cf7624) to head (3b8a8a8).
Report is 4 commits behind head on main.

Files	Patch %	Lines
pkg/daemon/server/service/health_status.go	68.75%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1905      +/-   ##
==========================================
- Coverage   59.82%   58.24%   -1.59%     
==========================================
  Files         286      291       +5     
  Lines       23689    24390     +701     
==========================================
+ Hits        14172    14205      +33     
- Misses       8588     9243     +655     
- Partials      929      942      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[whynowy]

The threshold for different vertex should be different - based on the vertex limits configuration, is this right? @kohlisid

[kohlisid]

Yes now that we have it relative, it can vary vertex to vertex.
The code will need changes to compare each timeline using the limits for the corresponding vertex limits.

[KeranYang]

Hey @samhith-kakarla , Yashash and I just made changes to our e2e tests and committed to the main. If you continue working on your branch, you might see some un-expected e2e testing failures. To fix them, please rebase. Thanks!