github: compliance: use shallow fetches

[fabiobaltieri]

Change the checkout config for the compliance run to use shallow fetching, then add a couple of extra fetch to fetch just the necessary commits for the rebase and the other steps to work correctly.

This speeds up the fetching time significantly.

DNM while it has the debug prints in

[fabiobaltieri]

@marc-hb how about this? (well I realize now it was mentioned in actions/checkout#552 (comment))

1. let the checkout do a shallow fetch of the PR head, takes like 11 seconds rather than 50s-1m
2. run another fetch to deepen the data by the commit count, that ends up fetching the merge base
3. fetch the current base branch HEAD
4. now we have enough commits to do the rebase and nothing more

See the log:

### initial state

3db0337a (grafted, HEAD) github: compliance: use shallow fetches

### deepen

From https://github.com/zephyrproject-rtos/zephyr
 * branch              3db0337af302e4367cffeedd2681b147c7e1550b -> FETCH_HEAD
3db0337a (HEAD) github: compliance: use shallow fetches
72f416f3 (grafted) scripts: Add workflow for "common" directories in find_tests()

### fetch origin/main

From https://github.com/zephyrproject-rtos/zephyr
 * branch                main       -> FETCH_HEAD
 * [new branch]          main       -> origin/main
* 3db0337af (HEAD) github: compliance: use shallow fetches
| * e7340282a (origin/main) west.yml: update hal_intel
| * b2eade6bc west: update open-amp repo
| * ad6d8f281 boards/arm/serpente: fix system clock rate
| * 2235a8253 scripts: tests: twister: runner test expansion
| * fffe0b9fa twister: pytest: Parametrize scope of the dut fixture
| * e57e7f28a Revert "Revert "modules: tinycrypt: Options only when module is available""
| * a6c3230b7 unittest: Generate symbol for existent modules
| * 49f0c9f22 samples: modules: canopennode: suggest using manifest.project-filter
| * 0aed42f2e mgmt: ec_host_cmd: improve handling buffer sizes
| * d0961756a drivers: watchdog: Add xmc4xxx support
| * 460c2167e Revert "drivers: intel: ssp: Correct FIFO depth value for CAVS25 platforms"
| * 3e4c50b0e Revert "drivers: intel: ssp: Revise receive FIFO draining"
| * 7b2cb95cd tests/x86: Reverting back to old logic
| * f5ce4ddc7 arch/x86: Remove useless legacy ACPI code
| * c294b7d27 lib/acpi: Fix the behavior to fit with how it used to be
|/  
* 72f416f38 (grafted) scripts: Add workflow for "common" directories in find_tests()

### rebase

Rebasing (1/1)                                                                             
Successfully rebased and updated detached HEAD.
1998a20fd (HEAD) github: compliance: use shallow fetches
e7340282a (origin/main) west.yml: update hal_intel
b2eade6bc west: update open-amp repo
ad6d8f281 boards/arm/serpente: fix system clock rate
2235a8253 scripts: tests: twister: runner test expansion
fffe0b9fa twister: pytest: Parametrize scope of the dut fixture
e57e7f28a Revert "Revert "modules: tinycrypt: Options only when module is available""
a6c3230b7 unittest: Generate symbol for existent modules
49f0c9f22 samples: modules: canopennode: suggest using manifest.project-filter
0aed42f2e mgmt: ec_host_cmd: improve handling buffer sizes
d0961756a drivers: watchdog: Add xmc4xxx support
460c2167e Revert "drivers: intel: ssp: Correct FIFO depth value for CAVS25 platforms"
3e4c50b0e Revert "drivers: intel: ssp: Revise receive FIFO draining"
7b2cb95cd tests/x86: Reverting back to old logic
f5ce4ddc7 arch/x86: Remove useless legacy ACPI code
c294b7d27 lib/acpi: Fix the behavior to fit with how it used to be
72f416f38 (grafted) scripts: Add workflow for "common" directories in find_tests()

[marc-hb]

run another fetch to deepen the data by the commit count, that ends up fetching the merge base

Yes this is good with linear git histories. In my experience it's inefficient and sometimes even SLOWER when you have a significant number of git merges. Reminder: depth != number of commits! It can be slower in some cases because you have 2 git fetches instead of one. But for Zephyr it should work well (no time to look at the timings of your dry-run right now sorry)

Just make sure you use a big timeout command somewhere or some other cap for whenever someone pushes something massive / unusual.

I think requesting the list of commits from Github would be more generic and maybe even enough for a Github Action that always works in the general case - but overkill for Zephyr's linear history.

[fabiobaltieri]

Yes this is good with linear git histories. In my experience it's inefficient and sometimes even SLOWER when you have a significant number of git merges. Reminder: depth != number of commits! It can be slower in some cases because you have 2 git fetches instead of one. But for Zephyr it should work well (no time to look at the timings of your dry-run right now sorry)

Ah there is the caveat! Yeah I noticed that fetch being unreasonably slow for some reason I don't quite understand if you don't explicitly specify the remote sha. This seems to work well for most cases, but then as you pointed out if someone has a branch with merges this would probably fail quite bad.

Alright, was a good experiment but I think I'll leave it at that for now, the Zephyr case at the moment is not that bad, probably not (yet) worth the extra steps.

Many thanks for the pointers.

[marc-hb]

It can be slower in some cases because you have 2 git fetches instead of one

I just remembered something else: I think I noticed that the LOCAL house-keeping performed by --deepen could also be pretty slow sometimes, I mean slow even after the actual download.

This area is unfortunately and deceptively full of bad surprises and edge cases.