[BBPBGLIB-1102] Add load balance based on dry run estimate #111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
e514246
to
b4cf01f
Compare
This comment has been minimized.
This comment has been minimized.
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
6e8d575
to
4abfd4e
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
matz-e
reviewed
Feb 5, 2024
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
1669b5b
to
82319af
Compare
This comment has been minimized.
This comment has been minimized.
matz-e
reviewed
Feb 6, 2024
st4rl3ss
changed the title
matz-e
reviewed
Feb 6, 2024
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
matz-e
reviewed
Feb 7, 2024
This comment has been minimized.
This comment has been minimized.
matz-e
reviewed
Feb 7, 2024
matz-e
reviewed
Feb 7, 2024
matz-e
reviewed
Feb 7, 2024
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
3a33bc4
to
824051d
Compare
matz-e
reviewed
Feb 8, 2024
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
a1e32a4
to
4c667f6
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…e into account synapses in load balancing
…d load balancing stats
st4rl3ss
force-pushed
the
bellotta/dryrun_load_balance
branch
from
ba005e9
to
e5aa8d4
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
matz-e
reviewed
Feb 13, 2024
ferdonline
previously approved these changes
Feb 13, 2024
ferdonline
approved these changes
Feb 13, 2024
|
Logfiles from GitLab pipeline #193103 (:white_check_mark:) have been uploaded here! Status and direct links: |
WeinaJi
pushed a commit
that referenced
this pull request
Oct 14, 2024
…ate (#111) ## Context This MR includes new mechanisms to generate a detailed load balancing using the memory estimates generated by the dry run. It distributes all the gids of the circuit to a number of ranks specified using the `--num-target-ranks` CLI flag taking care of distributing so that the allocation of each rank is balanced. The system is based on a greedy algorithm, assigning cells in blocks of 10 to the emptiest rank available at that moment. It also includes automatic pickling and gzipping of the allocation dictionary to `allocation.pkl.gz` file. Loading of the generated load balance will be integrated in a future MR. ## Scope The MR briefly expands the capabilities of the dry run workflow with automatic load balancing and save to pickle file for future use. All the structure involved stay the same and no significant change is made to workflow. ## Testing Most of the new functionalities should be automatically tested using the `test_dry_run_workflow` integration tests. In any case a specific test that loads the pickled `allocation.pkl.gz` file and checks against expected values was integrated. ## Review * [x] PR description is complete * [x] Coding style (imports, function length, New functions, classes or files) are good * [x] Unit/Scientific test added * [x] Updated Readme, in-code, developer documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
This MR includes new mechanisms to generate a detailed load balancing using the memory estimates generated by the dry run. It distributes all the gids of the circuit to a number of ranks specified using the
--num-target-ranksCLI flag taking care of distributing so that the allocation of each rank is balanced. The system is based on a greedy algorithm, assigning cells in blocks of 10 to the emptiest rank available at that moment.It also includes automatic pickling and gzipping of the allocation dictionary to
allocation.pkl.gzfile. Loading of the generated load balance will be integrated in a future MR.Scope
The MR briefly expands the capabilities of the dry run workflow with automatic load balancing and save to pickle file for future use. All the structure involved stay the same and no significant change is made to workflow.
Testing
Most of the new functionalities should be automatically tested using the
test_dry_run_workflowintegration tests. In any case a specific test that loads the pickledallocation.binfile and checks against expected values was integrated.Review