[SPARK-49861][INFRA] Add Python 3.13 to Infra docker image

### What changes were proposed in this pull request? This PR aims to add `Python 3.13` to Infra docker image. Note that SPARK-49862 `Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13` will track the Python package readiness. ### Why are the changes needed? `Python 3.13` release is scheduled on next Monday, 2024-10-07. - https://peps.python.org/pep-0719/ This is a part of `Python 3.13 Support` preparation for Apache Spark 4.0.0 on 2025 February. ### Does this PR introduce _any_ user-facing change? No, this is an infra only change. ### How was this patch tested? Pass the CIs and verify manually like the following. ``` $ docker run -it --rm ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-11152626644 python3.13 --version Python 3.13.0rc3 ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #48330 from dongjoon-hyun/SPARK-49861. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
apache · Oct 3, 2024 · 68fd17d · 68fd17d
1 parent 38f067d
commit 68fd17d
Showing 1 changed file with 11 additions and 1 deletion.
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image"
 # Overwrite this label to avoid exposing the underlying Ubuntu OS version label
 LABEL org.opencontainers.image.version=""
 
-ENV FULL_REFRESH_DATE 20240903
+ENV FULL_REFRESH_DATE 20241002
 
 ENV DEBIAN_FRONTEND noninteractive
 ENV DEBCONF_NONINTERACTIVE_SEEN true
@@ -142,6 +142,16 @@ RUN python3.12 -m pip install $BASIC_PIP_PKGS $CONNECT_PIP_PKGS lxml && \
     python3.12 -m pip install torcheval && \
     python3.12 -m pip cache purge
 
+# Install Python 3.13 at the last stage to avoid breaking the existing Python installations
+RUN apt-get update && apt-get install -y \
+    python3.13 \
+    && rm -rf /var/lib/apt/lists/*
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13
+# TODO(SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13
+RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs this
+RUN python3.13 -m pip install lxml && \
+    python3.13 -m pip cache purge
+
 # Remove unused installation packages to free up disk space
 RUN apt-get remove --purge -y 'gfortran-11' 'humanity-icon-theme' 'nodejs-doc' || true
 RUN apt-get autoremove --purge -y