Benchmarks

[seriyps]

It's quite experimental, but it's only enabled in test profile.

Sample output:

$ make bench
 ./rebar3 as test bench
 ===> Verifying dependencies...
 ===> Compiling gen_smtp
 test/gen_smtp_util_test.erl:3: Warning: export_all flag enabled - all functions will be exported

===> Calling deprecated rebar_erlc_compiler compiler module. This module has been replaced by rebar_compiler and rebar_compiler_erl, but will remain available.
test/gen_smtp_util_test.erl:3: Warning: export_all flag enabled - all functions will be exported

===> Testing bench_session:bench_session()
===> Stats for wall_time
Min:               6890.36μs (-   31.22μs /  0.5%)
25 percentile:     6959.27μs (-  426.40μs /  5.8%)
Median:            7631.72μs (-   38.27μs /  0.5%)
75 percentile:     7774.30μs (-  192.35μs /  2.4%)
Max:               9195.65μs (- 5291.78μs / 36.5%)
Outliers:         Lo: 0; Hi: 1; Sum: 1
Outlier variance:   0.584795 (severe)
> Bootstrapped
Mean:              7474.21μs (-  587.10μs /  7.3%)
Std deviation:      454.61μs
> Relative
Difference at 95.0 confidence
  -587.10μs ±   286.67μs
      -7.28%  ±  3.56%
       (Student's t, pooled s = 1.03420e+6

===> Outlier variance is too high! Benchmark result might be non-representative. Try to repeat the benchmarks in more calm environment (no heavy background OS tasks) or run benchmark for a longer time
===> Testing bench_mimemail:bench_parse_plain()
===> Stats for wall_time
Min:                187.34μs (- 5039.00ns /  2.6%)
25 percentile:      189.39μs (- 6912.00ns /  3.5%)
Median:             191.15μs (- 7122.00ns /  3.6%)
75 percentile:      193.16μs (-   11.07μs /  5.4%)
Max:                212.63μs (-  108.49μs / 33.8%)
Outliers:         Lo: 0; Hi: 5; Sum: 5
Outlier variance:   0.132375 (moderate)
> Bootstrapped
Mean:               191.90μs (-   14.80μs /  7.2%)
Std deviation:     3846.00ns
> Relative
Difference at 95.0 confidence
   -14.80μs ±  4218.00ns
       -7.16%  ±  2.04%
        (Student's t, pooled s = 1.52176e+4

===> Testing bench_mimemail:bench_parse_quoted_printable()
===> Stats for wall_time
Min:               1777.63μs (-   18.63μs /  1.0%)
25 percentile:     1810.64μs (-   26.84μs /  1.5%)
Median:            1833.41μs (-   21.28μs /  1.1%)
75 percentile:     1849.58μs (-   19.64μs /  1.1%)
Max:               2158.69μs (- 6326.00ns /  0.3%)
Outliers:         Lo: 0; Hi: 4; Sum: 4
Outlier variance:   0.209611 (moderate)
> Bootstrapped
Mean:              1838.44μs (-   25.43μs /  1.4%)
Std deviation:       49.60μs
> Relative
Difference at 95.0 confidence
   -25.43μs ±    14.09μs
       -1.36%  ±  0.76%
        (Student's t, pooled s = 5.08313e+4

===> Testing bench_mimemail:bench_parse_base64()
===> Stats for wall_time
Min:                631.92μs (+ 2029.00ns /  0.3%)
25 percentile:      637.23μs (+ 2118.00ns /  0.3%)
Median:             641.01μs (+ 1783.00ns /  0.3%)
75 percentile:      652.04μs (- 5631.00ns /  0.9%)
Max:                690.12μs (-  172.48μs / 20.0%)
Outliers:         Lo: 0; Hi: 4; Sum: 4
Outlier variance:   0.113353 (moderate)
> Bootstrapped
Mean:               645.79μs (-   13.50μs /  2.0%)
Std deviation:       11.88μs
> Relative
Difference at 95.0 confidence
   -13.50μs ±  9323.00ns
       -2.05%  ±  1.41%
        (Student's t, pooled s = 3.36346e+4

===> Testing bench_mimemail:bench_parse_gamut()
===> Stats for wall_time
Min:               1860.02μs (- 2933.00ns /  0.2%)
25 percentile:     1875.35μs (-   14.79μs /  0.8%)
Median:            1885.45μs (-   16.18μs /  0.9%)
75 percentile:     1905.02μs (-   18.45μs /  1.0%)
Max:               2349.37μs (-  221.40μs /  8.6%)
Outliers:         Lo: 0; Hi: 11; Sum: 11
Outlier variance:   0.385208 (moderate)
> Bootstrapped
Mean:              1910.38μs (-   41.77μs /  2.1%)
Std deviation:       78.54μs
> Relative
Difference at 95.0 confidence
   -41.77μs ±    30.84μs
       -2.14%  ±  1.58%
        (Student's t, pooled s = 1.11274e+5

===> Testing bench_mimemail:bench_encode_plain()
===> Stats for wall_time
Min:                273.65μs (-  682.00ns /  0.2%)
25 percentile:      279.11μs (+ 2706.00ns /  1.0%)
Median:             280.72μs (+ 2072.00ns /  0.7%)
75 percentile:      283.64μs (+  620.00ns /  0.2%)
Max:                294.76μs (-   89.69μs / 23.3%)
Outliers:         Lo: 0; Hi: 6; Sum: 6
Outlier variance: 7.56186e-2 (slight)
> Bootstrapped
Mean:               281.73μs (+  454.00ns /  0.2%)
Std deviation:     4256.00ns
> Relative
No difference found at 95.0% confidence level

===> Testing bench_mimemail:bench_encode_bin_attach()
===> Stats for wall_time
Min:                533.97μs (- 3565.00ns /  0.7%)
25 percentile:      536.30μs (- 9442.00ns /  1.7%)
Median:             538.96μs (- 8829.00ns /  1.6%)
75 percentile:      544.11μs (- 8650.00ns /  1.6%)
Max:                589.18μs (-   17.52μs /  2.9%)
Outliers:         Lo: 0; Hi: 4; Sum: 4
Outlier variance: 7.53255e-2 (slight)
> Bootstrapped
Mean:               541.46μs (- 9036.00ns /  1.6%)
Std deviation:     7919.00ns
> Relative
Difference at 95.0 confidence
 -9036.00ns ±  2450.00ns
     -1.64%  ±  0.45%
      (Student's t, pooled s = 8840.11

===> Testing bench_mimemail:bench_encode_dkim_rs()
===> Stats for wall_time
Min:               1782.56μs (-   16.19μs /  0.9%)
25 percentile:     1795.41μs (-   25.85μs /  1.4%)
Median:            1801.29μs (-   65.89μs /  3.5%)
75 percentile:     1823.76μs (-  168.81μs /  8.5%)
Max:               2052.16μs (-  392.79μs / 16.1%)
Outliers:         Lo: 0; Hi: 9; Sum: 9
Outlier variance:   0.151655 (moderate)
> Bootstrapped
Mean:              1816.47μs (-  110.47μs /  5.7%)
Std deviation:       39.77μs
> Relative
Difference at 95.0 confidence
  -110.47μs ±    29.92μs
      -5.73%  ±  1.55%
       (Student's t, pooled s = 1.07944e+5

===> Testing bench_mimemail:bench_encode_dkim_ss()
===> Stats for wall_time
Min:               1698.18μs (-   15.14μs /  0.9%)
25 percentile:     1708.22μs (-   15.98μs /  0.9%)
Median:            1715.14μs (-   15.07μs /  0.9%)
75 percentile:     1727.40μs (-   19.66μs /  1.1%)
Max:               2010.47μs (-  498.85μs / 19.9%)
Outliers:         Lo: 0; Hi: 10; Sum: 10
Outlier variance:   0.199742 (moderate)
> Bootstrapped
Mean:              1729.38μs (-   40.73μs /  2.3%)
Std deviation:       44.82μs
> Relative
Difference at 95.0 confidence
   -40.73μs ±    26.66μs
       -2.30%  ±  1.51%
        (Student's t, pooled s = 9.61875e+4

[mworrell]

Needs a bit of rebasing 😨

[seriyps]

True. I think I will try to rework an API of rebar3 bench first. It's still in alpha and I was working on this PR partially to test how easy would it be to write benchmarks for real-world projects and I dound a couple of things that I want to improve.