Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

//test/extensions/filters/network/rbac:integration_test flakily times out #35653

Closed
RyanTheOptimist opened this issue Aug 9, 2024 · 10 comments · Fixed by #35899
Closed

//test/extensions/filters/network/rbac:integration_test flakily times out #35653

RyanTheOptimist opened this issue Aug 9, 2024 · 10 comments · Fixed by #35899
Assignees
@yanavlasov
Labels
area/test flakes

Comments

@RyanTheOptimist
Copy link
Contributor

https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_build/results?buildId=177453&tracking_data=ew0KICAic291cmNlIjogIlNsYWNrUGlwZWxpbmVzQXBwIiwNCiAgInNvdXJjZV9ldmVudF9uYW1lIjogIm1zLnZzcy1waXBlbGluZXMucnVuLXN0YXRlLWNoYW5nZWQtZXZlbnQiDQp9
https://dev.azure.com/cncf/envoy/_build/results?buildId=177474&view=logs&j=4930ecaf-18f4-5b3c-dea3-309729c3b3ae&t=573d8780-d7b9-52e3-b4e0-a89886b0b9ff&l=2823

Yan, can you take a look? Looks like it might be related to #35531
@RyanTheOptimist
Copy link
Contributor Author

@antoniovleonti it looks like this might be your PR.

@antoniovleonti
Copy link
Contributor

I find this strange because I only made a change to the http filter while it's the network filter test that's timing out.

@antoniovleonti
Copy link
Contributor

Yeah, this should be caused by #33875.

@antoniovleonti
Copy link
Contributor

@yangminzhu

@antoniovleonti
Copy link
Contributor

The timeout flake still exists at head. I don't know if we want to forward fix or revert but I'm going to make a PR to un-revert #35531.

@antoniovleonti
Copy link
Contributor

I reproduced with bazel test //test/extensions/filters/network/rbac:integration_test --runs_per_test=10000 but it looks like a smaller number would work too.

@antoniovleonti antoniovleonti mentioned this issue Aug 13, 2024
Merged
@antoniovleonti
Copy link
Contributor

I'm wondering if this can be fixed by just reducing the durations used in that test. I'm testing this now.

@antoniovleonti
Copy link
Contributor

No luck. PTAL @yangminzhu

@antoniovleonti antoniovleonti mentioned this issue Aug 13, 2024
Closed
yanavlasov pushed a commit that referenced this issue Aug 15, 2024
@antoniovleonti
Reapply "implement per-route override for rbac stat prefixes (#35531)… (
40dddc9 
#35693)

…" (#35655)

This reverts commit 7ae5ca5.

The flakiness in #35653 is not
caused by this PR. If you look at [the
failure](https://btx.cloud.google.com/invocations/08f19e64-5934-4ed9-9f4c-e0d14e113360/targets/%2F%2Ftest%2Fextensions%2Ffilters%2Fnetwork%2Frbac:integration_test;config=a48d3689347bfb6aaa2bb9b4c87edaab1bb2d2c4710bec9b6690537c48f15140/log)
(you can also find this by going to the issue and clicking the link),
the failing test is
`RoleBasedAccessControlNetworkFilterIntegrationTest.DelayDenied`. That
test was added in #33875 and is
still flaky at head.

Signed-off-by: antoniovleonti <[email protected]>
@adisuissa
Copy link
Contributor

The RBAC test seems to be highly flaky, and we've experienced a few failures every day.
If #33875 is the cause, should it be reverted?

@phlax
Copy link
Member

phlax commented Aug 29, 2024

If #33875 is the cause, should it be reverted?

@adisuissa @yangminzhu

i was just tracking this again as its an ongoing ~frequent flake and also came to the conclusion that this PR is the culprit (apologies @RyanTheOptimist i think i suggested the other previously)

im going to raise a revert PR and we can figure out from there

@phlax phlax mentioned this issue Aug 29, 2024
Merged
@wbpcode wbpcode mentioned this issue Aug 31, 2024
Closed
@phlax phlax closed this as completed in b7e13f1 Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yanavlasov yanavlasov

Labels
area/test flakes
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "rbac: add delay_deny implementation in RBAC network filter (#… phlax/envoy
5 participants
@phlax @adisuissa @yanavlasov @RyanTheOptimist @antoniovleonti