support for automatic bisection quite lacking

[matthiaskrgr]

I'm looking into adding automatic bisection to icemaker reports and it turns out to be quite difficult.

I was hoping I could do something like cargo-bisect-rustc --preserve --test-dir=/home/matthias/vcs/github/rust_bisect/foo --start 2021-01-01 --access github --regress ice -- rustc /tmp/icemaker/113793.rs -Zmir-opt-level=3 apparently this requires cargo rustc to be a subcommand :/

When bisecting manually I used a --script where I would just put in a simple ! rustc args | grep "panic" command to force a bad exit code on crash, but writing, chmodding and executing a bash script which you then run as arg to cargo-bisect seems kinda silly.
I need to be able to use (prlimit run ..) rustc/rustdoc/clippy-driver <file> <args>) to pipe through the binary, rustflags and filepath that I get from icemaker, creating an extra "cargo" project and putting the file in there, and setting its RUSTFLAGS and then running "cargo build" also does not seem like the right way.

edit: I found cargo x which seems to be a simple passthrough, so that cargo x cmd simply runs cmd, maybe cargo bisect ... x rustc .. can make it work, will try that next

Plans to just use cargo-bisect as a dependency in icemaker were quickly abolished when I saw not even the Config struct was marked public 😆 so that would probably require a bunch of changes here and there.

It would be great to have something like a --cmd "rustc --foo --bar crash.rs" --interesting-exitcodes="101,42,134" --interesting-output="panicked" --interesting-output="stack overflow" --interesting-output="internal compiler error" to make it a bit more versatile

[ehuss]

This sounds like there are multiple issues here, can we maybe separate it out into separate things?

For running without cargo, running without cargo explains what you need to do. Roughly, your first command would be rewritten as cargo bisect-rustc … --script=rustc -- /tmp/icemaker/113793.rs -Zmir-opt-level=3.

You mention adding various --interesting flags, but don't quite explain what they mean. Can you detail what you are after there?

The --regress flag already has various options for different regression checks. I personally want one to handle grepping for output, and I'm not sure if that is what you mean by --interesting-output?

What is the purpose of --interesting-exitcodes? I don't recall rustc really having different exit codes.

[matthiaskrgr]

--script=rustc ...

I do not know why this never crossed my mind :D
Thanks, I'll try that.

--interesting flags

The problem I ran into is for example:
rustc can have a "classic" internal compiler error, caused by a panic!(), bug!(), assertion failure etc
rustc can stack overflow
rustc can get sigabrt/sigsegv'd (might be caused by a miscompilation)
LLVM can run into an assertion failure
LLVM can run into a LLVM_ERROR
a process gets killed by the OOM killer

from my experience, these can have different exit codes, so just having a hardcoded exit_code == 130 is not enough in all cases unfortunately.
https://github.com/matthiaskrgr/icemaker/blob/dcf6aea556b4ed0f082bd89950dcfdda9fb87538/src/main.rs#L1116

You may also find yourself wanting to bisect with some custom condition like "exit code is 130 AND stacktrace contains "argument to drop_in_place is not a raw ptr" to make sure you catch "one ice turning into another" or something like that, but its a bit of a niche case.
Or maybe later we also want to bisect clippy FPs, miri UB, rustc diagnostic suggestions that do not apply, etc..

I knew the grepping example, but this again needs a custom script to passed to cargo-bisect which I would like to avoid :)