fix: creating a retryable kong client for tests and retry logic in konnect authentication for CI rate-limit errors

[Prashansa-K]

• chore: creating a retryable kong client for tests to account for rate-limit errors
• fix: added retry and backoff logic in Konnect authentication layer to account for rate-limit errors during authentication.

Fixes #1364

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Project coverage is 22.42%. Comparing base (e44d6c4) to head (2125626).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/common_konnect.go	0.00%	13 Missing ⚠️
tests/integration/test_utils.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1381   +/-   ##
=======================================
  Coverage   22.42%   22.42%           
=======================================
  Files          54       54           
  Lines        4517     4530   +13     
=======================================
+ Hits         1013     1016    +3     
- Misses       3404     3415   +11     
+ Partials      100       99    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GGabriele]

Don't we already have retries supported in our client? https://github.com/Kong/go-database-reconciler/blob/25ccc04e6c42e1b447070391e3938913deaed9fb/pkg/utils/types.go#L182

[Prashansa-K]

Don't we already have retries supported in our client? https://github.com/Kong/go-database-reconciler/blob/25ccc04e6c42e1b447070391e3938913deaed9fb/pkg/utils/types.go#L182

Yeah the client does have it, but the authentication layer still returns 429. The Konnect's login_service uses a normal http request here and is not retried. Does the client retry it by default?
I added my own retry layer but it still fails with 429 with 5 retries. Trying with an increased backoff period.

[Prashansa-K]

Works fine with an increased backoff period.
I tried the pipeline run thrice and it succeeds each time.