Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] OSB hangs when runs carried out with large number of clients and target-throughput enabled #633

Open
gkamat opened this issue Sep 9, 2024 · 0 comments
Assignees
Labels
bug Something isn't working Medium Priority

Comments

@gkamat
Copy link
Collaborator

gkamat commented Sep 9, 2024

Describe the bug

For tests where the target throughput is > 4000 ops/s and search_clients > 4000, OSB stalls and does not report results. This only happens if target throughput is set. Based on the OSB logs, some workers never meet at the joinpoint and this is preventing OSB from reporting results.

On the target cluster, the run appears to have completed in the expected time and the target throughput is achieved.

The cause is not clear but it seems to be related to the target-throughput pathway and also in how Thespianpy workers are missing (the suspicion is that they exit prematurely and cause OSB to stall.)

This may be related to the issue reported in #318.

@gkamat gkamat added bug Something isn't working untriaged labels Sep 9, 2024
@gkamat gkamat self-assigned this Sep 9, 2024
@gkamat gkamat removed the untriaged label Sep 9, 2024
@gkamat gkamat changed the title [BUG] OSB hangs when runs carried out with [BUG] OSB hangs when runs carried out with large number of clients Sep 9, 2024
@IanHoang IanHoang changed the title [BUG] OSB hangs when runs carried out with large number of clients [BUG] OSB hangs when runs carried out with large number of clients and target-throughput enabled Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Medium Priority
Projects
Status: Now(This Quarter)
Status: This Quarter
Status: In Progress
Development

No branches or pull requests

2 participants