Likely incorrect bisection result produced

[camelid]

As seen in rust-lang/rust#80101 (comment), cargo-bisect-rustc seems to have produced an incorrect result. Additionally, it seemed to be installing a commit then skipped to a different one and then another without printing any information. And it checked a commit twice (cc #85).

Here's the full log from my usage of cargo-bisect-rustc in case it helps in diagnosing the problem:

(pnkfelix note: Of course, everywhere you see ==> No is a spot where the author chose "mark baseline", and ==> Yes means the choice was "mark regressed")

❯ cargo bisect-rustc --preserve --prompt --start=2020-11-17 --end=2020-11-20 -- doc --document-private-items
installing nightly-2020-11-17
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 8.72s


nightly-2020-11-17 finished with exit code Some(0).
please select an action to take:
RESULT: nightly-2020-11-17, ===> No

installing nightly-2020-11-20
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 8.04s


nightly-2020-11-20 finished with exit code Some(0).
please select an action to take:
RESULT: nightly-2020-11-20, ===> Yes

installing nightly-2020-11-18
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 10.66s


nightly-2020-11-18 finished with exit code Some(0).
please select an action to take:
RESULT: nightly-2020-11-18, ===> No

installing nightly-2020-11-19
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 9.87s


nightly-2020-11-19 finished with exit code Some(0).
please select an action to take:
RESULT: nightly-2020-11-19, ===> Yes

searched toolchains nightly-2020-11-17 through nightly-2020-11-20


********************************************************************************
Regression in nightly-2020-11-19
********************************************************************************

fetching https://static.rust-lang.org/dist/2020-11-18/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-11-18: 40 B / 40 B [================] 100.00 % 43.49 KB/s converted 2020-11-18 to c919f490bbcd2b29b74016101f7ec71aaa24bdbb
fetching https://static.rust-lang.org/dist/2020-11-19/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2020-11-19: 40 B / 40 B [=================] 100.00 % 1.14 MB/s converted 2020-11-19 to 8256379832b5ecb7f71e8c5e2018446482223c12
looking for regression commit between 2020-11-18 and 2020-11-19
opening existing repository at "/Users/user/rust"
refreshing repository
fetching (via local git) commits from c919f490bbcd2b29b74016101f7ec71aaa24bdbb to 8256379832b5ecb7f71e8c5e2018446482223c12
opening existing repository at "/Users/user/rust"
refreshing repository
looking up first commit
looking up second commit
checking that commits are by bors and thus have ci artifacts...
finding bors merge commits
found 6 bors merge commits in the specified range
  commit[0] 2020-11-17UTC: Auto merge of #79138 - m-ou-se:rollup-owel5ld, r=m-ou-se
  commit[1] 2020-11-18UTC: Auto merge of #79132 - pietroalbini:beta-next, r=Mark-Simulacrum
  commit[2] 2020-11-18UTC: Auto merge of #77820 - jyn514:from-inner, r=petrochenkov
  commit[3] 2020-11-18UTC: Auto merge of #79159 - pietroalbini:woops, r=pietroalbini
  commit[4] 2020-11-18UTC: Auto merge of #79167 - m-ou-se:rollup-4g15apk, r=m-ou-se
  commit[5] 2020-11-18UTC: Auto merge of #78995 - Nadrieril:clean-empty-match, r=varkor
validated commits found, specifying toolchains

installing c919f490bbcd2b29b74016101f7ec71aaa24bdbb
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 10.60s


c919f490bbcd2b29b74016101f7ec71aaa24bdbb finished with exit code Some(0).
please select an action to take:
RESULT: c919f490bbcd2b29b74016101f7ec71aaa24bdbb, ===> No

installing 8256379832b5ecb7f71e8c5e2018446482223c12
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 13.33s


8256379832b5ecb7f71e8c5e2018446482223c12 finished with exit code Some(0).
please select an action to take:
RESULT: 8256379832b5ecb7f71e8c5e2018446482223c12, ===> Yes

installing c4f836ad1aceb83507810d9499f56988fd24578d
installing 87776d7d5322422e5239e153e793b687f7f9c379
installing 7d747db0d5dd8f08f2efb073e2e77a34553465a7
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 8.63s


7d747db0d5dd8f08f2efb073e2e77a34553465a7 finished with exit code Some(0).
please select an action to take:
RESULT: 7d747db0d5dd8f08f2efb073e2e77a34553465a7, ===> Yes

installing c919f490bbcd2b29b74016101f7ec71aaa24bdbb
testing...
 Documenting foo v0.1.0 (/Users/user/rust-issues/rust-issue-80101)
    Finished dev [unoptimized + debuginfo] target(s) in 8.58s


c919f490bbcd2b29b74016101f7ec71aaa24bdbb finished with exit code Some(0).
please select an action to take:
RESULT: c919f490bbcd2b29b74016101f7ec71aaa24bdbb, ===> No

searched toolchains c919f490bbcd2b29b74016101f7ec71aaa24bdbb through 8256379832b5ecb7f71e8c5e2018446482223c12


********************************************************************************
Regression in 7d747db0d5dd8f08f2efb073e2e77a34553465a7
********************************************************************************

==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2020-11-17 to nightly-2020-11-20
regressed nightly: nightly-2020-11-19
searched commits: from https://github.com/rust-lang/rust/commit/c919f490bbcd2b29b74016101f7ec71aaa24bdbb to https://github.com/rust-lang/rust/commit/8256379832b5ecb7f71e8c5e2018446482223c12
regressed commit: https://github.com/rust-lang/rust/commit/7d747db0d5dd8f08f2efb073e2e77a34553465a7

<details>
<summary>bisected with <a href='https://github.com/rust-lang/cargo-bisect-rustc'>cargo-bisect-rustc</a> v0.6.0</summary>


Host triple: x86_64-apple-darwin
Reproduce with:
```bash
cargo bisect-rustc --preserve --prompt --start=2020-11-17 --end=2020-11-20 -- doc --document-private-items 
```
</details>

Note that the issue seems to be when bisecting CI artifacts; I think it's right about which nightly the issue is in. Also note that I was manually testing with --prompt because I was bisecting a visual rustdoc regression. However, I bisected three times and got the same result, so I think it's unlikely that it's human error.

[camelid]

Seems like it's potentially an off-by-one error since the commit that actually regressed was one later than the one cargo-bisect-rustc thought – the correct one was rust-lang/rust@c4f836a.

[dtolnay]

I agree something is off-by-one. This came up as well in #134 (comment).

[ehuss]

I'm going to close, as I suspect this is likely caused by #113. I published version 0.6.1, feel free to reopen if it continues with the newer version.