diff --git a/README.md b/README.md index 8aa2af5..c851a6c 100644 --- a/README.md +++ b/README.md @@ -120,6 +120,11 @@ or `echo aeiou | ./build/count_vowels.hermit.com` +## Benchmarks + +- Hermit-cli : `./benchmarks/bench-cli.sh` benchmark hermit cli, for more details check [docs](benchmarks/README.md). +- Cli binaries produced by Hermit-cli : `./benchmarks/bench-artifacts.sh` benchmark produced binaries, for more details check [docs](benchmarks/README.md). + ## Community Hermit shares the [Extism Discord](https://discord.gg/cx3usBCWnc). Join `#hermit` to discuss working with or building Hermit. diff --git a/benchmarks/.gitignore b/benchmarks/.gitignore new file mode 100644 index 0000000..bd44e87 --- /dev/null +++ b/benchmarks/.gitignore @@ -0,0 +1,4 @@ +*.json +*.com +benchmarks/bench-artifacts/* +benchmarks/bench-cli/* \ No newline at end of file diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..0799798 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,76 @@ +## Benchmarks + +The purpose of this tool is to benchmark the performance of the `hermit` CLI.This will let us track performance changes from one release to another. + +Example of current benchmarks with a VM(4xCPU : Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, RAM 6GB) : + +- Debug mode +```log +Benching with cat. +Benchmark 1: hermit.com with cat + Time (mean ± σ): 1737.8 ms ± 103.2 ms [User: 1725.2 ms, System: 12.1 ms] + Range (min … max): 1595.7 ms … 1915.7 ms 10 runs + +Benching with count_vowels. +Benchmark 1: hermit.com with count_vowels + Time (mean ± σ): 50272.0 ms ± 2673.0 ms [User: 50248.9 ms, System: 19.0 ms] + Range (min … max): 47441.7 ms … 54378.3 ms 10 runs + +Benching with cowsay. +Benchmark 1: hermit.com with cowsay + Time (mean ± σ): 30065.3 ms ± 1294.7 ms [User: 30047.6 ms, System: 15.0 ms] + Range (min … max): 28296.4 ms … 32019.9 ms 10 runs +``` + +- Release mode +```log +Benching with cat. +Benchmark 1: hermit.com with cat + Time (mean ± σ): 46.0 ms ± 4.6 ms [User: 44.2 ms, System: 1.8 ms] + Range (min … max): 40.0 ms … 60.3 ms 67 runs + +Benching with count_vowels. +Benchmark 1: hermit.com with count_vowels + Time (mean ± σ): 795.7 ms ± 33.1 ms [User: 789.6 ms, System: 6.0 ms] + Range (min … max): 734.3 ms … 841.0 ms 10 runs + +Benching with cowsay. +Benchmark 1: hermit.com with cowsay + Time (mean ± σ): 472.0 ms ± 18.6 ms [User: 471.0 ms, System: 1.0 ms] + Range (min … max): 447.8 ms … 508.2 ms 10 runs +``` + +## Requirements + +- Install [Hyperfine](https://github.com/sharkdp/hyperfine) : `cargo install --locked hyperfine` + + More info can be found in [installation docs](https://github.com/sharkdp/hyperfine#installation). + +- Build Hermit, preferably in release mode. + +## Usage + +### Hermit-cli + +Run `./benchmarks/bench-cli.sh`, this will run benchmarks using hermit-cli with available samples. + +Default samples used during the bench : +- [cat](/src/cat/) +- [count_vowels](/src/count_vowels/) +- [cowsay](/src/cowsay/) + +You can provide your own samples to benchmark against it : +- Create a folder under `benchmarks/bench-cli/custom/` with the '$name' of your sample (ex.: benchmarks/custom/my_cli) +- Create a HermitFile under `benchmarks/bench-cli/custom/$name/` (ex.: benchmarks/custom/my_cli/HermitFile) +- Run `./benchmarks/bench-cli.sh --only-custom` + +### Cli binaires produced by Hermit-cli + +Run `./benchmarks/bench-artifacts.sh`, this will run benchmarks using cli binaries produced by hermit-cli. + +Default samples used during the bench : +- [cat](/src/cat/) +- [count_vowels](/src/count_vowels/) +- [cowsay](/src/cowsay/) + +You can benchmark your own samples, follow instructions provided by: `./benchmarks/bench-artifacts.sh --only-custom` \ No newline at end of file diff --git a/benchmarks/bench-artifacts.sh b/benchmarks/bench-artifacts.sh new file mode 100755 index 0000000..2701346 --- /dev/null +++ b/benchmarks/bench-artifacts.sh @@ -0,0 +1,56 @@ +#!/bin/bash +#set -x + +only_custom=false +remove_artifacts_after_bench=false +script_folder_name=$(basename "$(dirname "$(readlink -f "$0")")")/bench-artifacts +mkdir -p $script_folder_name +#uncomment line below if you wan't to remove +#stored bench stats from previous runs +#rm -rf $script_folder_name/*.json + +# Iterate through the command-line arguments +for arg in "$@"; do + if [ "$arg" == "--only-custom" ]; then + only_custom=true + break + fi +done + +run_hyperfine(){ + #https://github.com/sharkdp/hyperfine/issues/94#issuecomment-432708897 + #samples are small, silence the warning msg. + export_file="benchmarks/bench-artifacts/benchmark_$1_$(date +%s%3N).json" + hyperfine \ + --export-json="$export_file" \ + -N \ + --min-runs 10 \ + --warmup=3 \ + --time-unit=millisecond \ + --command-name="bench of $1" "$2" 2> /dev/null +} + +# Check if --only-custom was passed +if [ "$only_custom" == false ]; then + # Benchmark build/cat.hermit.com, build/count_vowels.hermit.com and build/cowsay.hermit.com examples. + run_hyperfine "Cat" "build/cat.hermit.com src/cat/cat.c" + run_hyperfine "Count_vowels" "echo eeeUIaoopaskjdfhiiioozzmmmwze | build/count_vowels.hermit.com" + run_hyperfine "Cowsay" "build/cowsay.hermit.com Hermooooooooot" +fi + +if [ "$only_custom" == true ]; then +cat </dev/null)) + for example_path in "${custom_examples_paths[@]}"; do + example_name=$(basename "$example_path") + printf "Benching with ${example_name}.\n" + + export_file="${example_path}/benchmark_${example_name}_$(date +%s%3N).json" + hyperfine \ + --export-json="$export_file" \ + --warmup=3 \ + --time-unit=millisecond \ + --command-name="hermit.com with ${example_name}" "build/hermit.com -f ${example_path}/Hermitfile -o ${example_path}/${example_name}.com" \ + --cleanup "(${remove_artifacts_after_bench} && rm -rf ${example_path}/${example_name}.com) || true" + done +fi \ No newline at end of file