Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade from python3.8 to python3.10 #5311

Closed
wants to merge 18 commits into from
Closed

Upgrade from python3.8 to python3.10 #5311

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d228f34
Update from python3.8 to python3.10
May 30, 2023
9724cf2
Update docker image
May 30, 2023
a57d335
fmt
May 30, 2023
3505591
VMs
May 30, 2023
94c283a
Fix docs
May 30, 2023
2352527
Merge branch 'main' into python_3_10
jumaffre May 31, 2023
46dc9d2
Fix?
May 31, 2023
99faa76
Merge branch 'python_3_10' of github.com:microsoft/CCF into python_3_10
May 31, 2023
a468ed3
GitHub pages
May 31, 2023
9d0b034
.
May 31, 2023
8668e5a
Try this?
May 31, 2023
f4c1374
.
May 31, 2023
9f054d0
Merge branch 'main' into python_3_10
jumaffre Jun 1, 2023
cf3596a
Merge branch 'main' into python_3_10
jumaffre Jun 2, 2023
6f861e1
Merge branch 'main' into python_3_10
eddyashton Jun 2, 2023
5ea9c28
Merge branch 'main' into python_3_10
jumaffre Jun 2, 2023
b26219e
Changelog
Jun 2, 2023
ac40551
.
Jun 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .azure-pipelines-gh-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ trigger:

jobs:
- job: build_and_publish_docs
# Note: Still on ubuntu-20.04 with python3.8 because of
# https://github.com/python-babel/babel/issues/990
container: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15
pool:
vmImage: ubuntu-20.04
displayName: Build and Publish Docs

steps:
- checkout: self
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-quictls.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ parameters:

jobs:
- job: build_quictls
container: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-clang15
container: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15
pool: 1es-dv4-focal

strategy:
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ jobs:
clean: true
fetchDepth: 1

- script: python3.8 scripts/extract-release-notes.py --target-git-version
- script: python3.10 scripts/extract-release-notes.py --target-git-version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we ought to switch to calling python3 everywhere, and communicate the version requirement through https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#python-requires

Wherever we can, using a shebang line or setuptools entry points would be nicer too, to avoid having to specify which python, now that 2.x is thoroughly obsolete.

displayName: "Check presence of release notes entry"
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/configure.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:
Codeql.SkipTaskAutoInjection: true
skipComponentGovernanceDetection: true
pool:
vmImage: "ubuntu-20.04"
vmImage: "ubuntu-22.04"
steps:
- checkout: self
clean: true
Expand Down
22 changes: 11 additions & 11 deletions .azure-pipelines-templates/deploy_aci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
Codeql.SkipTaskAutoInjection: true
skipComponentGovernanceDetection: true
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
steps:
- checkout: none

Expand All @@ -26,7 +26,7 @@ jobs:
skipComponentGovernanceDetection: true
sshKey: $[ dependencies.generate_ssh_key.outputs['generate_ssh_key.sshKey'] ]
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
steps:
- script: |
env
Expand Down Expand Up @@ -65,10 +65,10 @@ jobs:

- script: |
set -ex
python3.8 -m venv ./scripts/azure_deployment/.env
python3.10 -m venv ./scripts/azure_deployment/.env
source ./scripts/azure_deployment/.env/bin/activate
pip install -r ./scripts/azure_deployment/requirements.txt
python3.8 scripts/azure_deployment/arm_template.py deploy aci \
python3.10 scripts/azure_deployment/arm_template.py deploy aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group ccf-aci \
--region northeurope \
Expand All @@ -91,7 +91,7 @@ jobs:
- job: deploy_secondary_aci
displayName: "Deploy Secondary ACI"
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
dependsOn:
- generate_ssh_key
variables:
Expand Down Expand Up @@ -132,10 +132,10 @@ jobs:

- script: |
set -ex
python3.8 -m venv ./scripts/azure_deployment/.env
python3.10 -m venv ./scripts/azure_deployment/.env
source ./scripts/azure_deployment/.env/bin/activate
pip install -r ./scripts/azure_deployment/requirements.txt
python3.8 scripts/azure_deployment/arm_template.py deploy aci \
python3.10 scripts/azure_deployment/arm_template.py deploy aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group ccf-aci \
--region northeurope \
Expand Down Expand Up @@ -185,7 +185,7 @@ jobs:
- job: cleanup_aci
displayName: "Cleanup ACI"
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also updating some of the dependencies to 22.04, which feels like a bigger change than Python 3.10. Can we describe that in the PR title, and changelog if necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have only been done on CI jobs that used Python (3.8 but now 3.10) but do not make use of any of our containers. In other words, it's not a user-facing change, but simply for us to use 22.04 for a few CI agents. I'll reflect that in the PR description.

dependsOn:
- generate_ssh_key
- deploy_primary_aci
Expand Down Expand Up @@ -219,10 +219,10 @@ jobs:

- script: |
set -ex
python3.8 -m venv ./scripts/azure_deployment/.env
python3.10 -m venv ./scripts/azure_deployment/.env
source ./scripts/azure_deployment/.env/bin/activate
pip install -r ./scripts/azure_deployment/requirements.txt
python3.8 scripts/azure_deployment/arm_template.py remove aci \
python3.10 scripts/azure_deployment/arm_template.py remove aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group ccf-aci \
--aci-type dynamic-agent \
Expand All @@ -234,7 +234,7 @@ jobs:
- script: |
source ./scripts/azure_deployment/.env/bin/activate
if [[ ${{ parameters.secondaries.count }} != 0 ]]; then
python3.8 scripts/azure_deployment/arm_template.py remove aci \
python3.10 scripts/azure_deployment/arm_template.py remove aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group ccf-aci \
--aci-type dynamic-agent \
Expand Down
14 changes: 7 additions & 7 deletions .azure-pipelines-templates/deploy_attestation_container.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:
Codeql.SkipTaskAutoInjection: true
skipComponentGovernanceDetection: true
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
steps:
- template: azure_cli.yml
parameters:
Expand All @@ -22,10 +22,10 @@ jobs:

- script: |
set -ex
python3.8 -m venv ./scripts/azure_deployment/.env
python3.10 -m venv ./scripts/azure_deployment/.env
source ./scripts/azure_deployment/.env/bin/activate
pip install -r ./scripts/azure_deployment/requirements.txt
python3.8 scripts/azure_deployment/arm_template.py deploy aci \
python3.10 scripts/azure_deployment/arm_template.py deploy aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group attestationContainer \
--aci-type dynamic-agent \
Expand All @@ -40,7 +40,7 @@ jobs:
- job: cleanup_aci
displayName: "Cleanup Attestation Container"
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
dependsOn:
- deploy_attestation_container
- ${{ parameters.used_by }}
Expand All @@ -54,10 +54,10 @@ jobs:

- script: |
set -ex
python3.8 -m venv ./scripts/azure_deployment/.env
python3.10 -m venv ./scripts/azure_deployment/.env
source ./scripts/azure_deployment/.env/bin/activate
pip install -r ./scripts/azure_deployment/requirements.txt
python3.8 scripts/azure_deployment/arm_template.py remove aci \
python3.10 scripts/azure_deployment/arm_template.py remove aci \
--subscription-id $(CCF_AZURE_SUBSCRIPTION_ID) \
--resource-group attestationContainer \
--aci-type dynamic-agent \
Expand All @@ -68,7 +68,7 @@ jobs:
- job: cleanup_container_image
displayName: "Cleanup Attestation Container image"
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
dependsOn:
- deploy_attestation_container
- ${{ parameters.used_by }}
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/install_others.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
steps:
- script: |
set -ex
python3.8 -m venv env
python3.10 -m venv env
source ./env/bin/activate
pip install -r requirements.txt
pip install wheel
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ parameters:
Hosted:
container: virtual
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
Virtual:
container: virtual
pool: 1es-dv4-focal
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:

- script: |
set -ex
python3.8 ./scripts/extract-release-notes.py --target-git-version | tee $(Build.BinariesDirectory)/rel-notes.md
python3.10 ./scripts/extract-release-notes.py --target-git-version | tee $(Build.BinariesDirectory)/rel-notes.md
displayName: Extract release notes

- script: |
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/test_attestation_container.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
dependsOn: ${{ parameters.depends_on }}
condition: ${{ parameters.condition }}
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
timeoutInMinutes: 120
variables:
Codeql.SkipTaskAutoInjection: true
Expand Down
2 changes: 1 addition & 1 deletion .azure-pipelines-templates/test_on_remote.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
dependsOn: ${{ parameters.depends_on }}
condition: ${{ parameters.condition }}
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
timeoutInMinutes: 120
variables:
runOn: ${{ parameters.run_on }}
Expand Down
6 changes: 3 additions & 3 deletions .azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@ schedules:
resources:
containers:
- container: virtual
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_PTRACE -v /lib/modules:/lib/modules:ro

- container: snp
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-snp-clang15
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-snp-clang15
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_PTRACE -v /lib/modules:/lib/modules:ro

- container: sgx
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-sgx
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-sgx
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --device /dev/sgx_enclave:/dev/sgx_enclave --device /dev/sgx_provision:/dev/sgx_provision -v /dev/sgx:/dev/sgx -v /lib/modules:/lib/modules:ro

variables:
Expand Down
6 changes: 3 additions & 3 deletions .daily.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ schedules:
resources:
containers:
- container: virtual
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_PTRACE

- container: snp
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-snp-clang15
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-snp-clang15
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_PTRACE -v /lib/modules:/lib/modules:ro

- container: sgx
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-sgx
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-sgx
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --device /dev/sgx_enclave:/dev/sgx_enclave --device /dev/sgx_provision:/dev/sgx_provision -v /dev/sgx:/dev/sgx

jobs:
Expand Down
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "CCF Development Environment",
"image": "ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15",
"image": "ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15",
"runArgs": [],
"extensions": [
"eamodio.gitlens",
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/build-ci-container.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ jobs:
run: docker login -u $ACR_TOKEN_NAME -p ${{ secrets.ACR_CI_PUSH_TOKEN_PASSWORD }} $ACR_REGISTRY

- name: Pull CI container
run: docker pull $ACR_REGISTRY/ccf/ci:27-04-2023-snp-clang15
run: docker pull $ACR_REGISTRY/ccf/ci:30-05-2023-snp-clang15

- name: Build CCF CI SNP container
run: docker build -f docker/ccf_ci_built . --build-arg="base=ccfmsrc.azurecr.io/ccf/ci:27-04-2023-snp-clang15" --build-arg="platform=snp" -t $ACR_REGISTRY/ccf/ci:pr-`git rev-parse HEAD`
run: docker build -f docker/ccf_ci_built . --build-arg="base=ccfmsrc.azurecr.io/ccf/ci:30-05-2023-snp-clang15" --build-arg="platform=snp" -t $ACR_REGISTRY/ccf/ci:pr-`git rev-parse HEAD`

- name: Push CI container
run: docker push $ACR_REGISTRY/ccf/ci:pr-`git rev-parse HEAD`
2 changes: 1 addition & 1 deletion .github/workflows/ci-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
jobs:
checks:
runs-on: ubuntu-latest
container: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15
container: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15

steps:
- run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
jobs:
analyze:
name: Analyze
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

strategy:
fail-fast: false
Expand Down
2 changes: 1 addition & 1 deletion .multi-thread.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pr:
resources:
containers:
- container: virtual
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-virtual-clang15
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-virtual-clang15
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_PTRACE -v /lib/modules:/lib/modules:ro

jobs:
Expand Down
2 changes: 1 addition & 1 deletion .stress.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ schedules:
resources:
containers:
- container: sgx
image: ccfmsrc.azurecr.io/ccf/ci:27-04-2023-sgx
image: ccfmsrc.azurecr.io/ccf/ci:30-05-2023-sgx
options: --publish-all --cap-add NET_ADMIN --cap-add NET_RAW --device /dev/sgx_enclave:/dev/sgx_enclave --device /dev/sgx_provision:/dev/sgx_provision -v /dev/sgx:/dev/sgx

jobs:
Expand Down
2 changes: 1 addition & 1 deletion docker/ccf_ci_built
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

# Latest image as of this change
ARG platform=sgx
ARG base=ccfmsrc.azurecr.io/ccf/ci:27-04-2023-snp-clang-15
ARG base=ccfmsrc.azurecr.io/ccf/ci:30-05-2023-snp-clang-15
FROM ${base}

# SSH. Note that this could (should) be done in the base ccf_ci image instead
Expand Down
7 changes: 7 additions & 0 deletions getting_started/setup_vm/roles/ccf_build/tasks/install.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
- name: Add Python repository
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused by this, doesn't 22.04 ship with 3.10? If so, why do we need the ppa?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR doesn't update the base container image to 22.04, and only updates Python to 3.10. Updating our container images to 22.04 will be a bigger change (in a separate PR) which cannot be completed just yet because of the lack of Open Enclave packages on 22.04.

apt_repository:
repo: "ppa:deadsnakes/ppa"
state: present
update_cache: yes
become: yes

- name: Include vars for Clang
include_vars: "clang{{ clang_version }}.yml"

Expand Down
4 changes: 2 additions & 2 deletions getting_started/setup_vm/roles/ccf_build/vars/clang11.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ debs:
- libuv1-dev
- libc++-{{ clang_ver }}-dev
- libc++abi-{{ clang_ver }}-dev
- python3.8-dev
- python3.8-venv
- python3.10-dev
- python3.10-venv
- llvm-{{ clang_ver }}
- clang-{{ clang_ver }}
- clang-format-11
Expand Down
4 changes: 2 additions & 2 deletions getting_started/setup_vm/roles/ccf_build/vars/clang15.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ debs:
- libuv1-dev
- libc++-{{ clang_ver }}-dev
- libc++abi-{{ clang_ver }}-dev
- python3.8-dev
- python3.8-venv
- python3.10-dev
- python3.10-venv
- llvm-{{ clang_ver }}
- clang-{{ clang_ver }}
- clang-format-11 # On purpose, to avoid formatting conflicts
Expand Down
2 changes: 1 addition & 1 deletion livehtml.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ fi
echo "Setting up Python environment..."
if [ ! -f "env/bin/activate" ]
then
python3.8 -m venv env
python3.10 -m venv env
fi

source env/bin/activate
Expand Down
2 changes: 1 addition & 1 deletion scripts/azure_deployment/arm_aci.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ def parse_aci_args(parser: ArgumentParser) -> Namespace:
"--aci-image",
help="The name of the image to deploy in the ACI",
type=str,
default="ccfmsrc.azurecr.io/ccf/ci:27-04-2023-snp",
default="ccfmsrc.azurecr.io/ccf/ci:30-05-2023-snp",
)
parser.add_argument(
"--aci-type",
Expand Down
2 changes: 1 addition & 1 deletion scripts/check-cmake-format.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ fi

if [ ! -f "scripts/env/bin/activate" ]
then
python3.8 -m venv scripts/env
python3.10 -m venv scripts/env
fi

source scripts/env/bin/activate
Expand Down
Loading