loaddata: Allow increasing the size of submitted files #7016
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This would make it easier to get rough benchmarks of encryption steps.
zenmonkeykstop
requested changes
Oct 19, 2023
| @@ -202,7 +202,7 @@ def submit_file(source: Source, journalist_who_saw: Optional[Journalist]) -> Non | |||
| source.interaction_count, | |||
| source.journalist_filename, | |||
| "memo.txt", | |||
| io.BytesIO(b"This is an example of a plain text file upload."), | |||
| io.BytesIO(b"A" * size), | |||
There was a problem hiding this comment.
We compress then encrypt, so this will compress down to pretty much nothing before further processing, making it less useful for timing encryption or file transfers. Consider adding some randomness?
There was a problem hiding this comment.
Hmm, good point. Also there's a lot of overhead from SecureTemporaryFile. Maybe just a simple benchmarking script that just benchmarks exactly what we want (GPG vs redwood) would be better. Let me try to come up with something...
|
I don't want to merge this anymore. I created a separate benchmark script, which we could or could not merge, but IMO probably not. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Status
Ready for review
Description of Changes
This would make it easier to get rough benchmarks of encryption steps.
This is pretty quick/rough, not sure if there's a better way we want to go about this. Note that this would need to be manually applied to a 2.6.0 install to get numbers for gpg.
Testing
How should the reviewer test this PR?
time ./loaddata.py --file-size 500000000, etc.Deployment
Any special considerations for deployment? No
Checklist
make lint) and tests (make test) pass in the development container