Move cli_testing_examples under crates/cli/tests

[jschear]

Fixes #6557. When these fixtures are under crates/cli, the file paths in expect failures are property relativized to the CWD.

This touches the benchmark stuff a bit, not sure how to run those on CI.

[Anton-4]

Thanks for working on this @jschear :)

Would it be possible to get rid of cli_testing_examples and place its subfolders directly in crates/cli/tests?
crates/cli/tests/cli_testing_examples reads kind of redundant 😄

[jschear]

Heh, great point. Updated.

[Anton-4]

Looks like the paths in this file still need to be updated

[jschear]

Good catch, updated.

[Anton-4]

Thanks @jschear, looks good :)

[jschear]

Looks like something failed with the benchmarks check; I'll have some time to look into it later today.

[jschear]

Hmm, hitting a different issue when trying to run the benchmarks locally (maybe I need to run them in a noninteractive shell somehow?):

nix-shell-env ❯ nix develop -c ./ci/benchmarks/bench-runner/target/release/bench-runner --check-executables-changed

Doing a test run to verify benchmarks are working correctly and generate executables.

>>bench "branch": "Gnuplot not found, using plotters backend"
s>>bench "branch": "Benchmarking bench-group_wall-time/Benchmarking \"nQueens\""
>>bench "branch": "Benchmarking bench-group_wall-time/Benchmarking \"nQueens\": Warming up for 1.0000 s"
>>bench "branch": "Benchmarking bench-group_wall-time/Benchmarking \"nQueens\": Collecting 10 samples in estimated 1.3578 s (30 iterations)"
>>bench "branch": "Benchmarking bench-group_wall-time/Benchmarking \"nQueens\": Analyzing"
>>bench "branch": "bench-group_wall-time/Benchmarking \"nQueens\""
>>bench "branch": "                        time:   [44.852 ms 44.946 ms 45.053 ms]"
>>bench "branch": "Found 1 outliers among 10 measurements (10.00%)"
>>bench "branch": "  1 (10.00%) high mild"
thread 'main' panicked at 'expected output to end with "396354 & 396354\n" but instead got Out {
    cmd_str: "/Users/jschear/dev/roc/bench-folder-branch/crates/cli/tests/benchmarks/cFold",
    stdout: "Please enter an integer\n",
    stderr: "",
    status: ExitStatus(
        unix_wait_status(
            11,
        ),
    ),
}', crates/cli_utils/src/bench_utils.rs:60:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
>>bench "branch": ""
thread 'main' panicked at 'Error: time-bench execution failed with exit code exit status: 101.
See output above for error info.
Command was:
        "./bench-folder-branch/target/release/deps/time_bench" "--bench" "--noplot"', src/main.rs:164:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[jschear]

Oh I see, the CI job checks out main, invokes the script to prepare a benchmark folder, then checks out this branch and invokes the script to create another folder. Those folders are going to have a different structure (on main there will still be cli_testing_examples in the path, on this branch there won't be), and then the updated bench-runner binary doesn't handle both path formats. @Anton-4 if that makes sense, any tips on how to handle this? Merge through the failure; or update the bench-runner to handle the both the old and new layout temporarily?

[Anton-4]

I'll test this by checking out the current branch twice for the comparison, if that's good we can change one back to main and force the merge after.

[Anton-4]

Benchmarks passed :)

[Anton-4]

Can you force merge this @rtfeldman? The benchmarks failure is expected.