[Discussion] Global settings/defaults

[asinghvi17]

I noticed that there was some instability when benchmarking functions with a runtime of ~3ms, which was reduced significantly when I set seconds=1.

Would it be possible to set "global defaults" when using Chairmarks, so that I could set the default value of seconds to be 1s instead of 0.1s?

[LilithHafner]

This seems like a good idea. Global variables are a bit of an issue for composability and general usage (e.g. someone using @b inside an optimization loop in order to dispatch to a different algorithm depending on whether the target function is fast to evaluate) but I think we can document that the default seconds value should not be relied on in that sort of usage.

Current:

Defaults to 0.1 seconds unless samples is specified in which case it defaults to 1 second.

Proposed:

Defaults to Charimarks.DEFAULTS.seconds (which is 0.1 by default) unless samples is specified, in which case it defaults to 10 times as long (1 second, by default). Users are free to modify Charimarks.DEFAULTS.seconds for their own interactive usage and its default value may change in the future.

And if you've asked to spend a little longer on benchmarks to get more accurate results and an optimization function author uses @b without thinking about (or specifying) seconds, then it's quite reasonable (and even preferred) for that usage of @b to take longer and produce more accurate results for you.

Here are some plausible names:

Charimarks.DEFAULTS.seconds = 1
Charimarks.DEFAULT_PARAMETERS.seconds = 1
Charimarks.DEFAULT_SECONDS[] = 1

I prefer the foremost name because it is formulaic and I think it's unambiguous. What other defaults could we possibly have? The second name is what BenchmarkTools.jl uses, but making the same design choices as BenchmarkTools is not a priority for this package.

[asinghvi17]

Nice! I agree the first one looks best.

[JeffreySarnoff]

I agree.

[LilithHafner]

Okay, another thought: what if these defaults are module/Main specific? For example, if BaseBenchmarks uses Chairmarks, I don't really want BaseBenchmarks to have to worry about whether users are going to define new defaults.

Because @b and @be are macros, they know where they are being invoked from. Maybe the defaults only apply if they are being invoked from Main.

Then we have some more options:

Chairmarks.DEFAULTS.seconds = 1 # Effects only Main
Chairmarks.DEFAULTS[Mod].seconds = 1 # Effects only Mod
Chairmarks.@set_default seconds = 1 samples = 10_000 # Effects only the module that this macro was called in

[asinghvi17]

That would be difficult to set with Documenter, because it creates a new module for every @example block.

I do get the appeal of module-specific settings, though.

How about this: if a module specific setting is not detected, then it falls back to the settings for Main?

[JeffreySarnoff]

The default parameters for BenchmarkTools are named
BenchmarkTools.DEFAULT_PARAMETERS.seconds = 1.0
BenchmarkTools.DEFAULT_PARAMETERS.samples = 10000
BenchmarkTools.DEFAULT_PARAMETERS.time_tolerance = 0.15
BenchmarkTools.DEFAULT_PARAMETERS.memory_tolerance = 0.01

so how about using
julia> Chairmarks.DEFAULTS.seconds = 1
to set the number of seconds Chairmarks uses in Main
and anywhere else it is not shadowed (e.g. within a module's benchmarking)
and also allow module local use of
Chairmarks.DEFAULTS.seconds = 0.5?

[LilithHafner]

For the Documenter usecase, a scoped value set with

Charimarks.with_defaults(seconds = 1) do
    makedocs(...)
end

would work.

It would be possible to have a whole hierarchy

Function arguments > Scoped values > Module defaults > Globals > Chairmark's defaults

Which is altogether too many ways to pass an argument to a function. We already have (and ought to keep) function arguments and Charimark's defaults. Module defaults are the least invasive thing to put in users' startup.jl files and scoped values are the least invasive thing to support reconfiguring defaults for Documentation deployment. Scoped values are also ideal for BaseBenchmarks usage. Plain globals are the best thing for RegressionTests.jl, though. I want to set the default runtime to .01 due to the outer loop that that package employs, globals are safe because RegressionTests is working with self-contained Julia processes that is spawns and owns, and globals are better than scoped values because they don't require special invocation by the user.

Given that plain globals are needed by at least one use case and that globals can imitate scoped values and module-specific globals while the reverse is not the case, I suppose it makes sense to implement globals now and only implement the others if global conflicts become an issue.