Please rethink the "minimal image" idea for runner images

[Nuru]

Currently the Runner Set runner image ships with no more packages than are necessary. This makes it extremely awkward to develop workflows (especially shared and Marketplace workflows) that run on both GitHub hosted runners and self-hosted runners, as you have to have the workflows install commands which are preinstalled on the GitHub hosted runners.

Obviously you want to slim the image down from the 20 GB of the GitHub hosted image to something that can be pulled all the time, and I'm not arguing for full feature parity, but please reconsider installing some extremely commonly used tools, such as

• git
• curl
• wget

It just seems insane on the face of it that we have to install git in an Official GitHub action runner image. Being able to retrieve files via curl and wget are so basic to bootstrapping anything else that the only argument I can see against including them is that they duplicate each other. Nevertheless, both are so popular (and small) that it seems needlessly dogmatic not to include them both.

I estimate that including just these 3 extra tools will cover 80% of the issues. I would prefer you also include jq, which, along with the other 3, would cover nearly all of my workflows, but I agree is not so basic.

The argument that we can build our own image neglects the fact that any marketplace or shared workflow that wants to be able to run on a self-hosted Runner Set must assume that only the tools on the official GitHub images are present. It also neglects the significant effort it takes to build, publish, and keep up-to-date a custom image.

I think you should seriously consider installing all the tools installed on the legacy open source self-hosted runner image. That way workflow writers will only have to deal with 2 sets of preinstalled software rather than 3.

[naikrovek]

A better, future-proof, solution would be for GHA users to avoid relying on any software being installed whatsoever, except for what is supported by GHA itself. Create a container which contains everything you need, then specify that your job runs in your container.

That way you have everything you need, with the exact versions you've built in, and you can count on those tools being present no matter where your job runs.

I've been recommending that GHA users at my employer do this for a couple of years, now.

[Nuru]

@naikrovek wrote:

A better, future-proof, solution would be for GHA users to avoid relying on any software being installed whatsoever, except for what is supported by GHA itself. Create a container which contains everything you need, then specify that your job runs in your container.

How would this be "future-proof"? It would break every time an action (whether newly added to your workflow or, via maintenance, needs a new tool or a new version of a tool maintained) needs new support.

What do you mean by "supported by GHA itself?" The GitHub Hosted Action Runners are VMs that have over 18 GB of software and libraries installed or cached. The Runner Set Docker image is a little over 1 GB, and obviously missing a ton of support relative to the GitHub Hosted Runners.

I think there is a lot of value in having a smaller image that can be more quickly downloaded, but I also think a small image could cover more than 80% of the use cases, the way the Summerwind Runner Docker Image did at under 500 MB.

That way you have everything you need, with the exact versions you've built in, and you can count on those tools being present no matter where your job runs.

We went down that road briefly. The problem is that there is a large collection of publicly available actions that have been written with the expectation they will be running on GitHub Hosted Action Runners and naïvely expect those tools to be installed, to the extent that they do not document (and the maintainers often appear to be unaware) of what tools they are relying on, so we cannot just read the documentation and install the appropriate software. For example the setup-python action on Ubuntu requires that the build-essential package be installed, but that is not in any documentation I have seen.

The availability of powerful, free, well-documented, and well-maintained actions is an important part of the overall value of the GitHub Action feature. The fact that the officially supported and maintained runner image for self-hosting cannot run many (if not most) of them is a significant hinderance.

Having to build and maintain your own runner image requires the kind of needless duplication of effort that the entire open source movement was created to alleviate.

[Nuru]

The ability to run your action in a container is not something that will be taken away by changing what software is preinstalled on a runner.

The ability to run other people's actions, the whole premise of GitHub Marketplace and the Open Source movement in general, has been taken away by limiting what software is preinstalled on the runner.

Asking everyone to create their own container adds back the effort intended to be removed by sharing open source code, and leaves Marketplace actions with the unpleasant tradeoff of having to check to see if their prerequisites have been installed and installing them dynamically if they or not, or, as most have today, simply failing to run on self-hosted runners and requiring the GitHub hosted runner environment.

@naikrovek There is nothing about providing non-minimal images for people who want to use them that prevents people from building and using their own images the way you envision and recommend. On the other hand, the lack of images with a reasonable baseline of capability is a significant roadblock to adoption of GitHubs self-hosted runners versus Summerwind's.

[naikrovek]

The ability to run your action in a container is not something that will be taken away by changing what software is preinstalled on a runner.

The ability to run other people's actions, the whole premise of GitHub Marketplace and the Open Source movement in general, has been taken away by limiting what software is preinstalled on the runner.

I don't think you understand what happens when you run your job in a container. You can still install whatever actions you want as part of the job. You can do anything you want, but when you bring your own container, you bring a container which already has all the software you want installed, because you created the container.

[Nuru]

I don't think you understand what happens when you run your job in a container. You can still install whatever actions you want as part of the job. You can do anything you want, but when you bring your own container, you bring a container which already has all the software you want installed, because you created the container.

I fully understand what happens when I run a job in my own container. I don't think you understand how much extra, unnecessary hassle it is to determine what prerequisites are needed to use all the open source Actions I want to use, and then configure, build, maintain, and publish my own container to use them.

GitHub could have required all Marketplace actions to be published as containers, but they didn't, and for good reason. GitHub did publish a list of pre-installed software on GitHub hosted runners, and developers of Marketplace and other shared, open source actions built Node.js actions based on the expectation that those tools would be available. As I said before, I don't expect full feature parity, but Summerwind managed to figure out a subset that covered most use cases and made adoption relatively easy.

[naikrovek]

The runner software itself includes a node.js runtime, which is what most actions use.

I don't know why you're so angry about my suggestion to run actions in containers. Look to see what runtimes the actions you want to use require, then install them. There are only a few, at most.

This "problem" you're describing is a non-issue; I've never encountered it and I support over 10k GHA users on multiple GitHub Enterprise Server instances.

Just put what you need in the container. It's very easy.

I am done replying to you, by the way. You are a waste of time.

[Nuru]

I'm not angry about your suggestion, but strongly disagree that what I'm describing is a non-issue that you have never encountered, since it is an issue that you have encountered so frequently (the runner not having the required prerequisites for the action) that you have built a whole infrastructure to work around it:

[Instead of running actions on runners,] look to see what runtimes the actions you want to use require, then install them [on a container you build, maintain, and publish, and then write your workflows to run your actions in your custom container (which of course means that to make your workflow a public, reusable workflow, you also have to make your container publicly available)]. There are only a few, at most.

Your comment "there are only a few, at most" exactly aligns with my recommendation and request. Since there are only a few, at most, pre-install them on the runners (or at least a "fat" version of a GitHub maintained runner), and then people who do not have the time, energy, and resources to support over 10k GHA users on multiple GitHub Enterprise Server instances don't have to suffer for it.

[OmpahDev]

At the very least, make a few different runner images in addition to the minimal one. Something similar to the Summerwind image as an additionally supported runner image would be really nice. The effort involved in having to maintain custom docker images to do anything is a really big annoyance. A lot of organizations that use GitHub Actions are not set up to do this, they just rely on pulling a ready-to-go image.

In my own organization, this issue is literally the one thing that's keeping us on the old self-hosted runner image setup and preventing us from migrating to ARC. We don't want to be maintaining our own docker images.

[zickzackv]

I can understand the request of installing at least some basic software. What puzzles me the most is that marketplace actions are relying on software being installed on a runner, but on the minimal images is neither curl, wget nor git installed. so most actions for setting up other packages like node can not run bc theses packages are missing.

We needed to create our own images:

FROM ghcr.io/actions/actions-runner:2.314.1

RUN sudo apt-get update && sudo apt-get install -y \
    jq unzip git curl

But now need to track the releases of the actions-runner docker images.

[naikrovek]

But now need to track the releases of the actions-runner docker images.

FROM ghcr.io/actions/actions-runner:latest doesn't do that?

[zickzackv]

Yeah, true fair point. But its a bit lazy, since we actually needed to check whats has changed and we are not using latest tags when using the image. (its company policy)

[so-jelly]

the more tools added, the more the github team has to keep track of.
you should be using immutable digests instead of tags and pinning your dependencies

# 2.314.1
ARG RUNNER_DIGEST=sha256:da21d3177504ba153bd0665c6bc5155a0b572ee19f3b12413ec043a6d94d9fde
USER root

RUN \
  apt-get update -y &&\
  apt-get install --no-install-recommends -y \
  curl=7.81.0-1ubuntu1.16 \
  jq=1.6-2.1ubuntu3 \
  git=1:2.45.2-0ppa1~ubuntu22.04.1 \
  gpg=2.2.27-3ubuntu2.1 \
  unzip=6.0-26ubuntu3.2 \
  wget=1.21.2-2ubuntu1 &&\
  rm -rf /var/lib/apt/lists/*

USER runner

check if there are updates

docker pull ghcr.io/actions/actions-runner
[[ $(docker inspect ghcr.io/actions/actions-runner | jq -r '.[].Id') == $(grep <Dockerfile 'ARG RUNNER_DIGEST=' | cut -d= -f2) ]]

[caiocsgomes]

Hey team! Any ideas on this? We are migrating to the new Scaleset architecture and this is being an issue. While migrating we have workflows failing due to the tools missing. Sorry for the ping btw.

@Link- @nikola-jokic

[naikrovek]

Create your own image, or install things as part of your workflow.

I know both options are a pain compared to using a container image that someone else maintains, but those are your options unless someone promises to maintain an image for this.

[nsheaps]

You could try https://github.com/actions-runner-controller/runner-images or https://github.com/quipper/actions-runner

Been looking for this myself, was caught off guard when disabling a required job container and finding a ton of tools missing

[caiocsgomes]

hey! We took the custom image based on the standard image road @nsheaps

[avdhoot]

The absence of essential software (which are available in the Summerwind runner image) is increasing the workload during the migration. Unfortunately, things aren't backward compatible.

[so-jelly]

for some context, github adding tools to their runner: actions/runner#3056