optimize locking in katalogus.py, reuse available data #3752
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
We see long running locks when many organizations are present due to fetching the plugin information from the katalogus.
This blocks the boefje and normalizer threads.
We also do tree requests for every organization to the same katalogus endpoint, once for plugins, once for boefjes and once for normalizers.
This Pr minimizes the lock duration by building the new dict locally before copying. it also reuses this info for the boefjes and normalizer loops.
Issue link
You have to create an issue to link to this PR. If this really is not possible, write a very detailed description here and add this PR to the project board directly.
Please add the link to the issue after "Closes".
Closes ...
Demo
Please add some proof in the form of screenshots or screen recordings to show (off) new functionality, if there are interesting new features for end-users.
QA notes
Please add some information for QA on how to test the newly created code.
Code Checklist
.env
changes files if required and changed the.env-dist
accordingly.Checklist for code reviewers:
Copy-paste the checklist from the docs/source/templates folder into your comment.
Checklist for QA:
Copy-paste the checklist from the docs/source/templates folder into your comment.