feat: add benchmarks

README.md

@@ -120,6 +120,11 @@ or
 
 `echo aeiou | ./build/count_vowels.hermit.com`
 
+## Benchmarks
+
+- Hermit-cli : `./benchmarks/bench-cli.sh` benchmark hermit cli, for more details check [docs](benchmarks/README.md).
+- Cli binaries produced by Hermit-cli : `./benchmarks/bench-artifacts.sh` benchmark produced binaries, for more details check [docs](benchmarks/README.md).
+
 ## Community
 
 Hermit shares the [Extism Discord](https://discord.gg/cx3usBCWnc). Join `#hermit` to discuss working with or building Hermit.

benchmarks/.gitignore

@@ -0,0 +1,4 @@
+*.json
+*.com
+benchmarks/bench-artifacts/*
+benchmarks/bench-cli/*

benchmarks/README.md

@@ -0,0 +1,76 @@
+## Benchmarks
+
+The purpose of this tool is to benchmark the performance of the `hermit` CLI.This will let us track performance changes from one release to another.
+
+Example of current benchmarks with a VM(4xCPU : Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, RAM 6GB) :
+
+- Debug mode
+```log
+Benching with cat.
+Benchmark 1: hermit.com with cat
+  Time (mean ± σ):     1737.8 ms ± 103.2 ms    [User: 1725.2 ms, System: 12.1 ms]
+  Range (min … max):   1595.7 ms … 1915.7 ms    10 runs
+
+Benching with count_vowels.
+Benchmark 1: hermit.com with count_vowels
+  Time (mean ± σ):     50272.0 ms ± 2673.0 ms    [User: 50248.9 ms, System: 19.0 ms]
+  Range (min … max):   47441.7 ms … 54378.3 ms    10 runs
+
+Benching with cowsay.
+Benchmark 1: hermit.com with cowsay
+  Time (mean ± σ):     30065.3 ms ± 1294.7 ms    [User: 30047.6 ms, System: 15.0 ms]
+  Range (min … max):   28296.4 ms … 32019.9 ms    10 runs
+```
+
+- Release mode
+```log
+Benching with cat.
+Benchmark 1: hermit.com with cat
+  Time (mean ± σ):      46.0 ms ±   4.6 ms   [User: 44.2 ms, System: 1.8 ms]
+  Range (min … max):    40.0 ms …  60.3 ms    67 runs
+
+Benching with count_vowels.
+Benchmark 1: hermit.com with count_vowels
+  Time (mean ± σ):     795.7 ms ±  33.1 ms    [User: 789.6 ms, System: 6.0 ms]
+  Range (min … max):   734.3 ms … 841.0 ms    10 runs
+
+Benching with cowsay.
+Benchmark 1: hermit.com with cowsay
+  Time (mean ± σ):     472.0 ms ±  18.6 ms    [User: 471.0 ms, System: 1.0 ms]
+  Range (min … max):   447.8 ms … 508.2 ms    10 runs
+```
+
+## Requirements
+
+- Install [Hyperfine](https://github.com/sharkdp/hyperfine) : `cargo install --locked hyperfine` 
+
+  More info can be found in [installation docs](https://github.com/sharkdp/hyperfine#installation).
+
+- Build Hermit, preferably in release mode.
+
+## Usage
+
+### Hermit-cli
+
+Run `./benchmarks/bench-cli.sh`, this will run benchmarks using hermit-cli with available samples.
+
+Default samples used during the bench :
+- [cat](/src/cat/)
+- [count_vowels](/src/count_vowels/)
+- [cowsay](/src/cowsay/)
+
+You can provide your own samples to benchmark against it :
+- Create a folder under `benchmarks/bench-cli/custom/` with the '$name' of your sample (ex.: benchmarks/custom/my_cli)
+- Create a HermitFile under `benchmarks/bench-cli/custom/$name/` (ex.: benchmarks/custom/my_cli/HermitFile)
+- Run `./benchmarks/bench-cli.sh --only-custom`
+
+### Cli binaires produced by Hermit-cli
+
+Run `./benchmarks/bench-artifacts.sh`, this will run benchmarks using cli binaries produced by hermit-cli.
+
+Default samples used during the bench :
+- [cat](/src/cat/)
+- [count_vowels](/src/count_vowels/)
+- [cowsay](/src/cowsay/)
+
+You can benchmark your own samples, follow instructions provided by: `./benchmarks/bench-artifacts.sh --only-custom`

benchmarks/bench-artifacts.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+#set -x	
+
+only_custom=false
+remove_artifacts_after_bench=false
+script_folder_name=$(basename "$(dirname "$(readlink -f "$0")")")/bench-artifacts
+mkdir -p $script_folder_name
+#uncomment line below if you wan't to remove
+#stored bench stats from previous runs
+#rm -rf $script_folder_name/*.json
+
+# Iterate through the command-line arguments
+for arg in "$@"; do
+    if [ "$arg" == "--only-custom" ]; then
+        only_custom=true
+        break
+    fi
+done
+
+run_hyperfine(){
+    #https://github.com/sharkdp/hyperfine/issues/94#issuecomment-432708897
+    #samples are small, silence the warning msg.
+    export_file="benchmarks/bench-artifacts/benchmark_$1_$(date +%s%3N).json"
+    hyperfine \
+        --export-json="$export_file" \
+        -N \
+        --min-runs 10 \
+        --warmup=3 \
+        --time-unit=millisecond \
+        --command-name="bench of $1" "$2" 2> /dev/null
+}
+
+# Check if --only-custom was passed
+if [ "$only_custom" == false ]; then
+    # Benchmark build/cat.hermit.com, build/count_vowels.hermit.com and  build/cowsay.hermit.com examples.
+    run_hyperfine "Cat" "build/cat.hermit.com src/cat/cat.c"
+    run_hyperfine "Count_vowels" "echo eeeUIaoopaskjdfhiiioozzmmmwze | build/count_vowels.hermit.com"
+    run_hyperfine "Cowsay" "build/cowsay.hermit.com Hermooooooooot"
+fi
+
+if [ "$only_custom" == true ]; then
+cat <<EOF
+# to benchmark using custom artifact
+# place your cli binary in "./benchmarks/bench-artifacts/custom/"
+# set values of "example_name" and "cmd"
+# run commands below
+export example_name="my_cli"
+export cmd="path/to/my_cli.com params of my_cli.com"
+export stats_file="benchmarks/bench-artifacts/benchmark_\$example_name_\$(date +%s%3N).json"
+hyperfine \\
+    --export-json="\$stats_file" \\
+    --warmup=3 \\
+    --time-unit=millisecond \\
+    --command-name="bench of \$example_name" "\$cmd" 
+EOF
+fi

benchmarks/bench-cli.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+#set -x	
+
+only_custom=false
+remove_artifacts_after_bench=false
+script_folder_name=$(basename "$(dirname "$(readlink -f "$0")")")/bench-cli
+mkdir -p $script_folder_name
+rm -rf $script_folder_name/*.com
+#uncomment line below if you wan't to remove
+#stored bench stats from previous runs
+#rm -rf $script_folder_name/*.json
+
+# Iterate through the command-line arguments
+for arg in "$@"; do
+    if [ "$arg" == "--only-custom" ]; then
+        only_custom=true
+        break
+    fi
+done
+
+# Check if --only-custom was passed
+if [ "$only_custom" == false ]; then
+    # Benchmark build/hermit.com using `cat`, `count_vowels` and `cowsay` examples.
+    examples_paths=("src/cat" "src/count_vowels" "src/cowsay")
+    for example_path in "${examples_paths[@]}"; do
+        example_name=$(basename "$example_path")
+        printf "Benching with ${example_name} ${example_path}.\n"
+
+        export_file="${script_folder_name}/benchmark_${example_name}_$(date +%s%3N).json"
+        hyperfine \
+            --export-json="$export_file" \
+            --warmup=3 \
+            --time-unit=millisecond \
+            --command-name="hermit.com with ${example_name}" "build/hermit.com -f ${example_path}/Hermitfile -o ${script_folder_name}/${example_name}.com" \
+            --cleanup "(${remove_artifacts_after_bench} && rm -rf ${script_folder_name}/${example_name}.com) || true"
+    done
+
+fi
+
+# Benchmark build/hermit.com using custom examples.
+if [ "$only_custom" == true ]; then
+    custom_examples_paths=($(ls -d $script_folder_name/custom/* 2>/dev/null))
+    for example_path in "${custom_examples_paths[@]}"; do
+        example_name=$(basename "$example_path")
+        printf "Benching with ${example_name}.\n"
+
+        export_file="${example_path}/benchmark_${example_name}_$(date +%s%3N).json"
+        hyperfine \
+            --export-json="$export_file" \
+            --warmup=3 \
+            --time-unit=millisecond \
+            --command-name="hermit.com with ${example_name}" "build/hermit.com -f ${example_path}/Hermitfile -o ${example_path}/${example_name}.com" \
+            --cleanup "(${remove_artifacts_after_bench} && rm -rf ${example_path}/${example_name}.com) || true"
+    done
+fi