[Github] Build PGO optimized toolchain in container

[boomanaiden154]

This patch adjusts the Docker container intended for CI use to contain a PGO+ThinLTO+BOLT optimized clang. The toolchain is built within a Github action and takes ~3.5 hours. No caching is utilized. The current PGO optimization is fairly minimal, only running clang over hello world. This can be adjusted as needed.

[llvmbot]

@llvm/pr-subscribers-github-workflow

Author: Aiden Grossman (boomanaiden154)

Changes

This patch adjusts the Docker container intended for CI use to contain a PGO+ThinLTO+BOLT optimized clang. The toolchain is built within a Github action and takes ~3.5 hours. No caching is utilized. The current PGO optimization is fairly minimal, only running clang over hello world. This can be adjusted as needed.

---

Full diff: https://github.com/llvm/llvm-project/pull/80096.diff

1 Files Affected:

• (modified) .github/workflows/containers/github-action-ci/Dockerfile (+22-16)

diff --git a/.github/workflows/containers/github-action-ci/Dockerfile b/.github/workflows/containers/github-action-ci/Dockerfile
index d91a7ad3a9d06..6c8a2045bbdc2 100644
--- a/.github/workflows/containers/github-action-ci/Dockerfile
+++ b/.github/workflows/containers/github-action-ci/Dockerfile
@@ -2,29 +2,35 @@ FROM docker.io/library/ubuntu:22.04 as base
 ENV LLVM_SYSROOT=/opt/llvm/
 
 FROM base as toolchain
-ENV LLVM_MAJOR=17
-ENV LLVM_VERSION=${LLVM_MAJOR}.0.6
-ENV LLVM_DIRNAME=clang+llvm-${LLVM_VERSION}-x86_64-linux-gnu-ubuntu-22.04
-ENV LLVM_FILENAME=${LLVM_DIRNAME}.tar.xz
+ENV LLVM_VERSION=17.0.6
 
 RUN apt-get update && \
     apt-get install -y \
-    curl \
-    xz-utils
+    wget \
+    gcc \
+    g++ \
+    cmake \
+    ninja-build \
+    python3
+
+RUN wget https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-$LLVM_VERSION.tar.gz && tar -xf llvmorg-$LLVM_VERSION.tar.gz
 
-RUN mkdir -p $LLVM_SYSROOT/bin/ $LLVM_SYSROOT/lib/
+WORKDIR /llvm-project-llvmorg-$LLVM_VERSION
 
-RUN curl -O -L https://github.com/llvm/llvm-project/releases/download/llvmorg-$LLVM_VERSION/$LLVM_FILENAME
+RUN mkdir build
 
-RUN tar -C $LLVM_SYSROOT --strip-components=1 -xJf $LLVM_FILENAME \
-    $LLVM_DIRNAME/bin/clang \
-    $LLVM_DIRNAME/bin/clang++ \
-    $LLVM_DIRNAME/bin/clang-cl \
-    $LLVM_DIRNAME/bin/clang-$LLVM_MAJOR \
-    $LLVM_DIRNAME/bin/lld \
-    $LLVM_DIRNAME/bin/ld.lld \
-    $LLVM_DIRNAME/lib/clang/
+RUN cmake -B ./build -G Ninja ./llvm \
+  -C ./clang/cmake/caches/BOLT-PGO.cmake \
+  -DBOOTSTRAP_LLVM_ENABLE_LLD=ON \
+  -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LLD=ON \
+  -DPGO_INSTRUMENT_LTO=Thin \
+  -DLLVM_ENABLE_RUNTIMES="compiler-rt" \
+  -DCMAKE_INSTALL_PREFIX="$LLVM_SYSROOT" \
+  -DLLVM_ENABLE_PROJECTS="bolt;clang;lld;clang-tools-extra" \
+  -DLLVM_DISTRIBUTION_COMPONENTS="lld;compiler-rt;clang-format" \
+  -DCLANG_DEFAULT_LINKER="lld"
 
+RUN ninja -C ./build stage2-install-distribution && ninja -C ./build install-distribution && rm -rf ./build
 
 FROM base
 

[boomanaiden154]

The container ends up being 808MB fully unpacked. It's only a couple hundred MB in download size (not sure what the full one is currently). I think that should probably be fine. I've seen pretty good performance improvements with just this configuration (~20% when I last tested it), but it's not the theoretical max. I think we can land this, start moving jobs over to the container, and then once some other patches land, we can do better perf-training and see what the results are like.

[tstellar]

#78879 should make it possible to optimize the build further.

[boomanaiden154]

#78879 should make it possible to optimize the build further.

Yep. My plan is to adjust this script once that lands and do some proper compile time experiments.