Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interleave benchmark iterations, not just processes #139

Open
fitzgen opened this issue Jun 3, 2021 · 0 comments · May be fixed by #143
Open

Interleave benchmark iterations, not just processes #139

fitzgen opened this issue Jun 3, 2021 · 0 comments · May be fixed by #143
Milestone
Measurement Bias ...

Comments

@fitzgen
Copy link
Member

fitzgen commented Jun 3, 2021

From #138:

Interleave benchmark runs appropriately. Right now, it looks like the top-level runner does a batch of runs with one engine, then a batch of runs with another. If the runs for different engines/configurations were interleaved at the innermost loop, then system effects that vary over time would at least impact all configurations roughly equally.
@fitzgen fitzgen added this to the Measurement Bias Mitigated milestone Jun 3, 2021
@fitzgen fitzgen mentioned this issue Jun 3, 2021
Open
fitzgen added a commit to fitzgen/sightglass that referenced this issue Jun 4, 2021
@fitzgen
Always spawn subprocesses for recording benchmarks
0ee07d9 
Even when we are only using one process for all samples. This will make
everything easier when we parent<-->child communication protocol changes to
support interleaving iterations from different processes as part of bytecodealliance#139.
fitzgen added a commit to fitzgen/sightglass that referenced this issue Jun 4, 2021
@fitzgen
Interleave benchmark iterations from different processes
489641e 
This commit makes it so that, rather than spawning a process for a Wasm
benchmark and engine pair and running all iterations for that process
immediately, we now spawn a bunch of processes and then run one iteration from a
random one of them at a time. This interleaves benchmark iterations, not just
processes.

This means that we need to have communication back and forth between the parent
process and the child processes. Before, we had fork/join style subprocesses
where we just spawned a child and waited for it to finish before starting the
next subprocess and there was no communication between the parent and child
before the child was done executing. In order to interleave benchmark
iterations, we need to spawn many processes, have them wait on the parent, and
have the parent tell one subprocess at a time to run one iteration and then wait
again. The way that this is done is with a very simple `stdin` and
`stdout`-based protocol. All children are waiting to read a newline on their
`stdin` before they run an iteration. The parent chooses a child, writes a
newline and then waits for the child to finish by reading a newline from the
child's `stdout`. When the child finishes its iteration, it writes a newline to
its `stdout`. When all iterations are complete, then the child writes its
results as JSON to `stdout`.

Fixes bytecodealliance#139
@fitzgen fitzgen linked a pull request Jun 4, 2021 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Measurement Bias Mitigated
Development

Successfully merging a pull request may close this issue.

Interleave benchmark iterations from different processes fitzgen/sightglass
1 participant
@fitzgen