Nightly release CI action is broken

[TommyMurphyTM1234]

See here:

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.10.23
• https://github.com/riscv-collab/riscv-gnu-toolchain/actions/runs/11470734480

Nightly Release
Error when evaluating 'strategy' for job 'upload-assets'. .github/workflows/nightly-release.yaml (Line: 199, Col: 15): Matrix must define at least one vector

Only source bundles generated, no binary toolchains.

[TommyMurphyTM1234]

This seems to be the culprit but I don't really understand it yet...

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Lines 198 to 199 in 935b263

  	strategy:
  	matrix: ${{ fromJson( needs.create-release.outputs.asset_matrix ) }}

[TommyMurphyTM1234]

The last successful nightly release was 3rd September 2024:

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases
  • https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.09.03

so I presume that one of the commits since that date caused the problem?

• https://github.com/riscv-collab/riscv-gnu-toolchain/commits/master/.github/workflows/nightly-release.yaml

I hope it's not one of mine! :-)

[cmuellner]

I created a PR that should address this issue: #1592

[TommyMurphyTM1234]

I created a PR that should address this issue: #1592

Thanks @cmuellner. 👍

[TommyMurphyTM1234]

Any idea why the nightly build still doesn't seem to be working or, at least, hasn't completed and uploaded a complete set of built artifacts yet?

[cmuellner]

I was waiting for a review of my PRs that address issues in the CI/CD scripts (#1582 and #1592).
I just merged them without receiving a review.

[TommyMurphyTM1234]

Still something wrong I guess? Only sources in the latest release again.

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.10.26
• https://github.com/riscv-collab/riscv-gnu-toolchain/actions/runs/11527279579

Edit: oh, out of disk space? Even though it's supposed to clean up after itself as far as I can see?

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Line 62 in 7aa6d5e

  - name: Remove unneeded frameworks to recover disk space

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Line 92 in 7aa6d5e

  - name: recover space

Does it maybe need to do more to clean up?
Do older release artifacts need to be deleted?
Are the changes to enable additional Linux musl and uClibc builds exceeding the available resources?

[jordancarlin]

It looks like it is the "create release" job that is running out of space. It downloads all of the artifacts from previous steps, which take up 25 GB but the runner only has 21 GB available. Each job is run on a separate runner, so the space needs to be cleaned up in this job too.

[cmuellner]

The CI seems to be regularly (I've observed this multiple times since we added the musl builds into the CI/CD) broken because of git/musl issues:

error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: expected 'packfile'
fatal: clone of 'https://git.musl-libc.org/git/musl' into submodule path '/home/runner/work/riscv-gnu-toolchain/riscv-gnu-toolchain/musl' failed
Failed to clone 'musl' a second time, aborting

I'm not sure what the best way to move forward here is.

[TommyMurphyTM1234]

https://git.musl-libc.org/git/musl

That's the wrong URL as far as I can see:

• https://musl.libc.org/
  • https://git.musl-libc.org/cgit/musl

Edit: ah - sorry - ignore that...

• https://git.musl-libc.org/cgit/musl

[mickflemm]

Maybe we are hitting an issue with HTTP (e.g. http.postBuffer is not enough to hold the pack file), does doing a shallow clone solve this ? Yocto at some point used this mirror: https://github.com/kraj/musl/tree/master it seems to be up to date.

Maybe @richfelker can help.

[richfelker]

Can you provide a minimal test case to reproduce the failure to git-clone? I just cloned successfully.

[richfelker]

FWIW if you're re-cloning on every CI job, the polite thing to do is make it a shallow clone. The more polite thing to do would be to cache git clones. But I don't think this is related to the problem.

[TommyMurphyTM1234]

FWIW if you're re-cloning on every CI job, the polite thing to do is make it a shallow clone.

FWIW that's what this recent PR was intended to deal with but it's closed pending further investigations:

• Use Shallow Submodule Init to Reduce Clone Times #1603

The more polite thing to do would be to cache git clones. But I don't think this is related to the problem.

Do you know what this would involve for this repo's actions?

[richfelker]

Do you know what this would involve for this repo's actions?

No, I don't. I've really avoided getting into CI workflows myself because I deem them a gigantically irresponsible abuse of resources. So I'm not sure what tooling there is to fix this (avoid downloading the same thing thousands of times), but it's something I very much hope someone is working on.

[cmuellner]

I've really avoided getting into CI workflows myself because I deem them a gigantically irresponsible abuse of resources.

I'm in the same camp. However, there is a significant demand for pre-built releases of this repo.
The automatic builds, which trigger new releases if new changes are merged, broke in August. Since then, people have regularly reached out as they want them back.

A possible solution is to have a mirror repo on Github, which regularly pulls the changes from the official repo. This reduces the load on upstream git servers.

[TommyMurphyTM1234]

A possible solution is to have a mirror repo on Github, which regularly pulls the changes from the official repo. This reduces the load on upstream git servers.

Another possibility might be to wget source tarballs for those components that have clearly defined releases?

[TommyMurphyTM1234]

In case this helps at all (may belong elsewhere?):

• binutils
  • https://ftp.gnu.org/gnu/binutils/
  • https://ftp.gnu.org/gnu/binutils/binutils-2.43.1.tar.lz
• dejagnu
  • https://ftp.gnu.org/gnu/dejagnu/
  • https://ftp.gnu.org/gnu/dejagnu/dejagnu-1.6.tar.gz
• gcc
  • https://ftp.gnu.org/gnu/gcc/gcc-14.2.0/
  • https://ftp.gnu.org/gnu/gcc/gcc-14.2.0/gcc-14.2.0.tar.xz
• gdb
  • https://ftp.gnu.org/gnu/gdb/
  • https://ftp.gnu.org/gnu/gdb/gdb-15.2.tar.xz
• glibc
  • https://ftp.gnu.org/gnu/glibc/
  • https://ftp.gnu.org/gnu/glibc/glibc-2.40.tar.xz
• llvm
  • https://releases.llvm.org/
  • https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.8
• musl
  • https://musl.libc.org/releases
  • https://musl.libc.org/releases/musl-1.2.5.tar.gz
• newlib
  • https://sourceware.org/ftp/newlib/index.html
  • ftp://sourceware.org/pub/newlib/newlib-4.4.0.20231231.tar.gz
• pk
  • https://github.com/riscv-software-src/riscv-pk/tags
  • https://github.com/riscv-software-src/riscv-pk/archive/refs/tags/v1.0.0.tar.gz
• qemu
  • https://download.qemu.org/
  • https://download.qemu.org/qemu-9.1.0.tar.xz
• spike
  • https://github.com/riscv-software-src/riscv-isa-sim/releases
  • https://github.com/riscv-software-src/riscv-isa-sim/archive/refs/tags/dummy-tag-for-ci-storage.tar.gz
• uclibc-ng
  • https://downloads.uclibc-ng.org/releases/
  • https://downloads.uclibc-ng.org/releases/1.0.50/uClibc-ng-1.0.50.tar.xz

Notes:

1. For each component the master tarball link is listed along with a link to the version currently used by riscv-gnu-toolchain (except where noted otherwise below)
2. Where different tarball formats are available the smallest/most compressed option was selected
3. Not sure what specific llvm tarball is relevant
4. Ditto for dejagnu
5. Spike/pk repos don't seem to create regular release/snapshot tarballs so this may be an option for these components yet?

[mickflemm]

On one hand if we run CI very often it's indeed a waste of resources, on the other hand it's useful for regression testing (and we can't solve that with a mirror repo btw, we can't check for example pull requests that way, and it's super useful), and it's an even worse waste of resources to have the users of this repo (or their CIs) building this repo again and again. That being said there are a few ways to optimize the flow, here are a few suggestions:

1. Mark all submodules for shallow cloning, this is better than wget IMHO since it'll work for all repos (even for those that don't create release tarballs), and will also be easier to update them. It'll make the build process faster for our users too.

2. I don't know if we can preserve the build environment across builds, that would help a lot since at this point we clone everything on every job. One approach would be to use https://github.com/actions/cache for sharing the cloned repos (or maybe have one job that would clone everything and cache its output for the others) but I haven't tested it (I've seen others using artifact upload/download to share data across jobs).

3. Instead of nigthly releases we can do weekly releases, it's a small change on nightly-release.yaml, it doesn't make much sense to have a release every day, even monthly releases would be fine.

4. For improving the size of the generated toolchains, we could deduplicate files using their hashes and switch them to hardlinks (I've tested it and it works fine), tar will preserve those so when user unpacks it there will also be a benefit, not only for the tarball's size. Then we could switch form gz to a more efficient compression, I use xz and to improve its efficiency further I first create the tarball and then compress it with xz -e -T0 (this is better than tar cvJf since it has the opportunity to create a better dictionary).

[TommyMurphyTM1234]

Then we could switch form gz to a more efficient compression, I use xz

Tarball repositories that I've seen (e.g. see above) suggest that LZ compression may be even better than XZ (at least from a compression perspective, not sure if it's slower?).

[cmuellner]

1. Mark all submodules for shallow cloning, this is better than wget IMHO since it'll work for all repos (even for those that don't create release tarballs), and will also be easier to update them. It'll make the build process faster for our users too.

PR exists (#1605).

2. I don't know if we can preserve the build environment across builds, that would help a lot since at this point we clone everything on every job. One approach would be to use https://github.com/actions/cache for sharing the cloned repos (or maybe have one job that would clone everything and cache its output for the others) but I haven't tested it (I've seen others using artifact upload/download to share data across jobs).

I also thought of this, but I have zero experience with it. It is hard to get up and running if it cannot be tested locally.

3. Instead of nigthly releases we can do weekly releases, it's a small change on nightly-release.yaml, it doesn't make much sense to have a release every day, even monthly releases would be fine.

We trigger the build every night, but it will not download/build anything if there were no changes in the last 24 hours.

4. For improving the size of the generated toolchains, we could deduplicate files using their hashes and switch them to hardlinks (I've tested it and it works fine), tar will preserve those so when user unpacks it there will also be a benefit, not only for the tarball's size. Then we could switch form gz to a more efficient compression, I use xz and to improve its efficiency further I first create the tarball and then compress it with xz -e -T0 (this is better than tar cvJf since it has the opportunity to create a better dictionary).

I will look into this. I usually use --threads=0 -6e for toolchain releases, as this gave the best results when I tested it a few years ago.

Thanks!

[TShapinsky]

2. I don't know if we can preserve the build environment across builds, that would help a lot since at this point we clone everything on every job. One approach would be to use https://github.com/actions/cache for sharing the cloned repos (or maybe have one job that would clone everything and cache its output for the others) but I haven't tested it (I've seen others using artifact upload/download to share data across jobs).

I'm working on a branch on my fork on this topic. Not happy with it quite yet. TShapinsky#2

[TShapinsky]

Another way toolchain size can be reduced is if stripped versions of the programs are used. A good portion of the dependencies already support a variant of make install-strip. I did some testing and it reduced the final toolchain output size by more than half.