Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images #12027

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

DaveLak
Copy link
Contributor

@DaveLak DaveLak commented Jun 4, 2024

Note

I was looking for somewhere to get feedback from maintainers about this approach to the Python 3.10 upgrade before attempting it, but the discussion surrounding a Python upgrade has been rather fragmented across many issues, PRs, and comment chains.

For that reason, I felt it would be easier to propose with a working example and dedicated PR.

Fixes:

Supersedes:

Changes

The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images.

Base Image Changes

Image Before Changes After Changes
base-builder Compiled Python 3.8 from source using official release servers at https://www.python.org/ftp/python/. Compiles Python 3.10.14 (the latest 3.10 release) from source using official release servers at https://www.python.org/ftp/python/.
base-runner Installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image. Uses a multi-stage build to copy the Python 3.10.14 interpreter compiled by the base-builder image, ensuring version sync and saving build time by re-using a pre-built version.

Known Impact on Projects

3.9 Workarounds That Can Be Removed

Project Fix Link
dask DaveLak@417bbf5
docutils DaveLak@e4c21ff
dovecot DaveLak@7ab3ab6
nbclassic DaveLak@5509b4e
pandas DaveLak@0642a7a
pybind11 DaveLak@a5bbdb3
pyodbc DaveLak@afa2b5e
qpid-proton DaveLak@f5bf756

Anticipated Build Failures

Preexisting Failures

Fix is Prepared
Project Fix Link
airflow DaveLak@60a0368
ipython DaveLak@21ac68e
networkx DaveLak@fc2f8c5
numpy DaveLak@9383c87
tensorflow-addons DaveLak@eed2bea
django (coverage build) DaveLak@c724d61
proto-plus-python DaveLak@37d973e
dnspython The upgraded pip version in the base-builder fixes the currently failing build.
Fix Requires Upstream Changes
Project Issue
pyvex Currently failing on python 3.9 because archinfo dependency requires >=3.10. Fails after the 3.10 upgrade because the upstream build script needs python3.9 replaced with python3.
Requires More Investigation
Project Issue
matplotlib Upgrading Python & Pyinstaller does resolve the build issues, but an error in the fuzz harness is exposed and must be resolved for check_build to pass. The exception: TypeError: Parser.non_math() takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py", line 43, in TestOneInput.
scipy Upgrading Python & Pyinstaller does resolve the build issues, but an error in the build step causes the build to fail. The error seems related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25". When export LDFLAGS="-fuse-ld=lld" is set, the error becomes: "ld.lld: error: undefined symbol: __asan_report_store4".
pandas (Introspector only) This workaround in build.sh is the issue.
pycrypto Failing with error: "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats". Seems like the issue described here. Pycrypto is deprecated and this is unlikely to be fixed upstream.

Possible Future Improvements

Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how base-clang works, and use that as the multi-stage builder in images that need it.

Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work.


Motivation

The changes introduced here upgrade Python from 3.8 to 3.10.14 inside
the base-builder and base-runner images.

 ### base-builder changes:

Prior to these changes, base-builder compiled Python 3.8 from source
using sources downloaded from the official release servers at
https://www.python.org/ftp/python/. This updates the compiled version
to 3.10.14 (the latest 3.10 release) instead.

 ### base-runner changes:

Prior to these changes, base-runner installed Python 3.8 from the
default apt repository provided by the Ubuntu 20.04 image it's based
on. These apt repositories do not have a version of Python 3.10
available by default. This updates the base-runner to instead use a
multi-stage build to copy the same Python interpreter compiled by the
base-builder image into the runner image, which ensures both Python
versions remain in-sync while saving build time by re-using a pre-built
version.

 ## Motivation

- Code coverage does not work on Python projects that use Python 3.10+
  syntax, and will not work until this or similar changes are landed
  (see google#11419)
- Upgrading the base-image to use Ubuntu 22.04 (which provides more
  recent Python versions via apt) has been stated as being unlikely to
  happen any time soon (see google#3290)
- Many OSS-Fuzz integrated Python projects no longer support Python 3.8
  and have resorted to implementing ad-hoc workarounds to upgrade to
  newer Python versions, including installing Python from the Dead
  Snakes PPA.
  - This leads to fragmentation and hard to debug issues. Maintenance
    is easier when everyone is using the same version without issue.
- With [Python 3.8 reaching end of life soon (in 2024-10)][python-
  versions-EOL], it is likely that more Python projects will begin
  dropping support for 3.8, further increasing the number of broken
  builds and ad-hoc workarounds.
- Previous attempts at upgrading Python have stalled.

 ## Known & Expected Issues

Several project Dockerfiles and build scripts contain hard coded
references to python3.8 file system paths, and many more have implanted
ad-hoc workarounds to upgrade to newer Python versions than 3.8
(typically 3.9.) Additional changes are required to each of these
projects to ensure they successfully build after this upgrade to Python
3.10.

 ### Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is
not expected to introduce any new issues, it was not tested on these
changes and may require additional work.

 ## Possible Areas of Improvement

Using the base-builder image in a multi-stage build to copy the pre-
compiled Python into base-runner is effective, but feels like a
workaround that may be introducing tech debt. A cleaner approach would
be to extract the Python compilation into a discrete base image similar
to how `base-clang` works, and use that as the multi-stage builder in
images that need it.

---

Fixes:
- google#11419

Supersedes:
- google#9532
- google#11420

[python-versions-EOL]: https://devguide.python.org/versions/
@jonathanmetzman
Copy link
Contributor

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

1 similar comment
@jonathanmetzman
Copy link
Contributor

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

@DaveLak
Copy link
Contributor Author

DaveLak commented Jun 20, 2024

Thanks for the runs. I'll check the timeouts in about 24 hours from now.

`MarkupSafe` is a transitive dependency through `code_coverage`'s
Jinja2 requirement. The previously pinned version, `MarkupSafe==0.23`,
is incompatible with Python 3.10 raising the following error:

```
ImportError: cannot import name 'Mapping' from 'collections'
```

Upgrading MarkupSafe to a compatible version requires `code_coverage`'s
Jinja2 requirement to be bumped from Jinja2==2.10 to 2.10.3

The `sed` change introduced here is not ideal, but is required until the
upstream requirement is bumped. At that point, the `sed` should become a
no-op.
@DaveLak
Copy link
Contributor Author

DaveLak commented Jun 22, 2024

@jonathanmetzman I think e1a6e9f should fix the broken coverage builds.

@jonathanmetzman
Copy link
Contributor

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

@DonggeLiu
Copy link
Contributor

Step #1: INFO:root:----------------------------Build result----------------------------
Step #1: INFO:root:Trial build end time: 2024-07-01 23:47:03.076958
Step #1: INFO:root:Failed project, Statuses, Logs
Step #1: INFO:root:adal, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-59e9438d-1ff0-4ffa-8224-8111645a98af.txt
Step #1: INFO:root:antlr3-java, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0433cbe9-0538-439b-a4d4-dbe4f848e62f.txt
Step #1: INFO:root:antlr4-java, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-acc02e30-e5ac-4799-a338-6b39bbbb0344.txt
Step #1: INFO:root:apache-commons-bcel, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5bd6cbdc-bf98-4e8f-9b7c-696c93692f68.txt
Step #1: INFO:root:apache-commons-configuration, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-385d8181-ff5c-4bd1-8737-cc2c18942098.txt
Step #1: INFO:root:apache-commons-io, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-62d9a1a7-b2dd-4630-84c2-ce547d42b242.txt
Step #1: INFO:root:args4j, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6a2d6890-f82b-44e7-9974-c7c022d22235.txt
Step #1: INFO:root:augeas, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-68014ad0-7f36-4c36-b6a7-7727142ecb5d.txt
Step #1: INFO:root:black, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cd576622-e346-4e48-ac1a-5d584253a818.txt
Step #1: INFO:root:capstone, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f4bc5b49-1878-4646-8427-1188c9e404c8.txt
Step #1: INFO:root:capstone, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ece1df41-1086-4071-9782-5f01c06446a9.txt
Step #1: INFO:root:charset_normalizer, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-80b23fcb-6554-4247-bbd8-4daf9f1b75d5.txt
Step #1: INFO:root:connexion, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e8a74cea-bb59-47a1-844b-820bd384fb00.txt
Step #1: INFO:root:cryptography, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0d10efa6-1e38-49f9-b617-91b11f6da301.txt
Step #1: INFO:root:filesystem_spec, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5f73e2aa-1b04-4de6-9fe9-442d936b6d97.txt
Step #1: INFO:root:flask-restx, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4a9d8672-dc50-4dce-a231-cfcf1b82ba03.txt
Step #1: INFO:root:fwupd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-65397082-aaf0-434d-9475-5a75186e9e74.txt
Step #1: INFO:root:fwupd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1bd67925-c598-4ef4-ace1-b2f29ceaa747.txt
Step #1: INFO:root:g-api-pubsub, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f8832604-6ab1-403f-bb90-2f96a6e7e505.txt
Step #1: INFO:root:g-api-py-oauthlib, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-c704ea64-418d-4a89-b173-1cbd521ce444.txt
Step #1: INFO:root:g-api-python-bigquery-storage, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7eea83fa-516f-404a-8968-ebb6f2c3f70a.txt
Step #1: INFO:root:g-api-python-cloud-core, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3cab03b6-ae60-4c00-8527-902ed1ee6d90.txt
Step #1: INFO:root:g-api-python-firestore, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e1fb8d0c-eee8-41c7-aec4-17c3e92151fa.txt
Step #1: INFO:root:g-api-python-tasks, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-a396dbc7-5019-4cf4-9cec-3442a9d5bb32.txt
Step #1: INFO:root:g-api-resource-manager, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-c9bf8e2b-8746-4900-b9d0-c09422b979e2.txt
Step #1: INFO:root:g-api-secret-manager, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-aafc7ad3-65ec-4d6c-b736-05e795bc939c.txt
Step #1: INFO:root:g-cloud-logging-py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5e55f45d-7a25-40ed-9f2b-40834a82406c.txt
Step #1: INFO:root:g-http-java-client, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-355832cb-00bc-40fd-a994-31f7268f08bd.txt
Step #1: INFO:root:g-py-bigquery, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-69f63c0d-d1b1-4b1c-a21d-4f7fdd77e679.txt
Step #1: INFO:root:gc-iam, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5aca40d1-a633-4ae1-baa1-1081cefdc067.txt
Step #1: INFO:root:gcloud-error-py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-523d22bc-41e0-424a-9ed8-56715e56998e.txt
Step #1: INFO:root:gcp-python-cloud-storage, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2dc76ef5-0ab5-49f6-90b1-f29bc1ddc4bf.txt
Step #1: INFO:root:h5py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-722b2644-4760-42c3-9850-d6d8c2791fec.txt
Step #1: INFO:root:hadoop, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-dea4441f-e119-4172-8a82-e58a5b0c8174.txt
Step #1: INFO:root:httplib2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d0b67e02-e149-4ca6-9cca-d3070f687446.txt
Step #1: INFO:root:httpretty, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-73678cae-3e84-4112-89eb-b35376c28327.txt
Step #1: INFO:root:ijson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ac4359f7-49a4-4b36-be70-84a418eb6116.txt
Step #1: INFO:root:ipykernel, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bfa3c47a-9c0c-497d-b518-e3303ab29a82.txt
Step #1: INFO:root:isodate, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-be3ab31f-f67b-4c3b-af35-2b45cb73d873.txt
Step #1: INFO:root:jedi, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0d6209c9-1078-41ff-bfae-d2ab2964c791.txt
Step #1: INFO:root:jupyter_server, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-490864fa-ccd8-49f8-bc01-cb51a7b2bb7c.txt
Step #1: INFO:root:kie-soup, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-863ff132-84b4-4db3-bfdb-ba18c3629c5b.txt
Step #1: INFO:root:knot-dns, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d7b332d7-011e-4bde-89c4-8b5f83f5f052.txt
Step #1: INFO:root:libidn, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f2cb8817-7a0b-46ab-9241-fc35d4c9ad21.txt
Step #1: INFO:root:libidn2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3cf3ca45-dd6d-4d69-b649-ba3429c637ba.txt
Step #1: INFO:root:libpsl, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3fd2f1aa-74e8-4ac0-86c4-26713a0163f8.txt
Step #1: INFO:root:libtasn1, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e84497e0-96d7-41da-a6ab-9fd27257e39e.txt
Step #1: INFO:root:libxls, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b615294d-15c4-47de-9873-f3a16f3bcf54.txt
Step #1: INFO:root:libxls, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0cb093ce-d7f1-4f50-be29-8d6d01bbb100.txt
Step #1: INFO:root:libxml2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2ee82ccc-0618-410a-af96-5d6138762889.txt
Step #1: INFO:root:looker-sdk, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-8f4b474f-514e-4c85-9f28-522aacc6469f.txt
Step #1: INFO:oauth2client.transport:Refreshing due to a 401 (attempt 1/2)
Step #1: INFO:root:msal, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-8acd07c4-6ffe-4d90-a6c3-676590175d08.txt
Step #1: INFO:root:nbclassic, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1eb60faa-2968-4ee5-9834-73cc3e93cc7e.txt
Step #1: INFO:root:nbclassic, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d0ab947b-d5f0-445b-93b0-3d793dc02cca.txt
Step #1: INFO:root:nbformat, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-21a0e098-a263-4820-80fa-0438f9da2470.txt
Step #1: INFO:root:ntlm2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7434edaf-572f-4f89-9446-d0e670996ade.txt
Step #1: INFO:root:orjson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f76408e1-fbc1-46b7-a36a-1ca528bc55d3.txt
Step #1: INFO:root:orjson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5bd98fe6-5b20-4c73-97a1-56e0bb9902aa.txt
Step #1: INFO:root:paramiko, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-dc135c35-a9e7-4b35-8d5d-b672feafa2fe.txt
Step #1: INFO:root:parso, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5ce6388d-5599-40d9-ab0a-d8cb7611b732.txt
Step #1: INFO:root:pasta, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7ecf9b62-0452-4eb4-a950-0faaa57a1af1.txt
Step #1: INFO:root:pffft, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2d82a68e-2374-4dbb-886a-d8c3253bd34d.txt
Step #1: INFO:root:pffft, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cb3bbe5f-437e-41f6-927d-cc1bec8ccc10.txt
Step #1: INFO:root:pillow, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e19625fb-a984-47b1-be32-f92b4c76fc7f.txt
Step #1: INFO:root:pillow, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-80ededd3-8c64-45c5-9913-6447e6f734e6.txt
Step #1: INFO:root:proto-plus-python, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6868d820-88c1-4fe8-a7e4-054e86554da7.txt
Step #1: INFO:root:pyasn1-modules, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-df5d0051-08d2-4f08-b02c-314da42fea63.txt
Step #1: INFO:root:pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e5f9f0f2-d1d8-4a2c-bf41-4c89232e3cec.txt
Step #1: INFO:root:pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-380ecce4-400f-42ff-8d58-5ee5e8cb90ca.txt
Step #1: INFO:root:pycrypto, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f655175f-cd8e-4f76-abb5-d4092e72c715.txt
Step #1: INFO:root:pyparsing, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-829f2448-5777-44a8-9f20-70dd21c7e7d5.txt
Step #1: INFO:root:python-pypdf, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-263bb117-8446-4b64-8a18-ce0c17e87db1.txt
Step #1: INFO:root:pyxdg, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ee6476d9-3fe2-487f-8d78-529ea375a416.txt
Step #1: INFO:root:pyzmq, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d8550a00-608c-46fa-9210-cb6c33ed4d86.txt
Step #1: INFO:root:sacremoses, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-611d1c7e-f36f-4bf7-9c62-59950b039685.txt
Step #1: INFO:root:simplejson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4eb149c7-8afc-48a1-a757-2916f02aa7db.txt
Step #1: INFO:root:six, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1d47f866-910f-4b22-b220-6f354e445360.txt
Step #1: INFO:root:soupsieve, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bf945104-19ff-4319-bf98-e553bdde0f66.txt
Step #1: INFO:root:tablesaw, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4ef78f17-4fe6-446e-90a2-66c40033c153.txt
Step #1: INFO:root:tinyusb, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1a6f010e-cf64-4d63-a1e3-7ca9a97c168f.txt
Step #1: INFO:root:toolbelt, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2642efe6-d83b-4817-ae9a-f241358e7141.txt
Step #1: INFO:root:wget2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-a1c02746-8f70-472d-85b3-31f56bb520b4.txt
Step #1: INFO:root:xlrd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2a2bad59-7dd1-44f7-8ab9-f65b9487a2b6.txt
Step #1: INFO:root:xmltodict, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-085add0f-851d-495a-8389-60b9bf7fc449.txt
Step #1: INFO:root:yamlbeans, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-857d16ce-6d90-4e3f-bd95-13e644a8855c.txt
Step #1: INFO:root:yarl, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-efd06c32-7445-4fed-a470-82b9f33afae2.txt
Step #1: INFO:root:django, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b92e8311-f2dc-412f-83fa-601d54c74c47.txt
Step #1: INFO:root:python-markdown, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1b32acfa-f5be-4220-880c-10a1b67c5074.txt

@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 10, 2024

Thanks for grabbing that list @DonggeLiu! I've started looking over them and will report back.

The few I've seen so far appear to be caused by ad-hock Python version updated in build scripts which is reassuring.


FYI for anyone interested:

Here's the list formatted as table:

Failures Table
Project Status Log URL
adal FAILURE log
antlr3-java FAILURE log
antlr4-java FAILURE log
apache-commons-bcel FAILURE log
apache-commons-configuration FAILURE log
apache-commons-io FAILURE log
args4j FAILURE log
augeas FAILURE log
black FAILURE log
capstone FAILURE log
capstone FAILURE log
charset_normalizer FAILURE log
connexion FAILURE log
cryptography FAILURE log
filesystem_spec FAILURE log
flask-restx FAILURE log
fwupd FAILURE log
fwupd FAILURE log
g-api-pubsub FAILURE log
g-api-py-oauthlib FAILURE log
g-api-python-bigquery-storage FAILURE log
g-api-python-cloud-core FAILURE log
g-api-python-firestore FAILURE log
g-api-python-tasks FAILURE log
g-api-resource-manager FAILURE log
g-api-secret-manager FAILURE log
g-cloud-logging-py FAILURE log
g-http-java-client FAILURE log
g-py-bigquery FAILURE log
gc-iam FAILURE log
gcloud-error-py FAILURE log
gcp-python-cloud-storage FAILURE log
h5py FAILURE log
hadoop FAILURE log
httplib2 FAILURE log
httpretty FAILURE log
ijson FAILURE log
ipykernel FAILURE log
isodate FAILURE log
jedi FAILURE log
jupyter_server FAILURE log
kie-soup FAILURE log
knot-dns FAILURE log
libidn FAILURE log
libidn2 FAILURE log
libpsl FAILURE log
libtasn1 FAILURE log
libxls FAILURE log
libxls FAILURE log
libxml2 FAILURE log
looker-sdk FAILURE log
msal FAILURE log
nbclassic FAILURE log
nbclassic FAILURE log
nbformat FAILURE log
ntlm2 FAILURE log
orjson FAILURE log
orjson FAILURE log
paramiko FAILURE log
parso FAILURE log
pasta FAILURE log
pffft FAILURE log
pffft FAILURE log
pillow FAILURE log
pillow FAILURE log
proto-plus-python FAILURE log
pyasn1-modules FAILURE log
pybind11 FAILURE log
pybind11 FAILURE log
pycrypto FAILURE log
pyparsing FAILURE log
python-pypdf FAILURE log
pyxdg FAILURE log
pyzmq FAILURE log
sacremoses FAILURE log
simplejson FAILURE log
six FAILURE log
soupsieve FAILURE log
tablesaw FAILURE log
tinyusb FAILURE log
toolbelt FAILURE log
wget2 FAILURE log
xlrd FAILURE log
xmltodict FAILURE log
yamlbeans FAILURE log
yarl FAILURE log
django FAILURE log
python-markdown FAILURE log

Updated the hook_pre_exec_eval function in command_injection.py to
accept additional arguments (*args, **kwargs). This resolves a TypeError
encountered in Python 3.10 where the function was called with more
arguments than expected.

The change ensures compatibility with Python 3.10 by aligning the
function signature with the arguments passed by the add_hook mechanism.

Also replaces the deprecated `importlib.find_loader` methoc call with
the recommended ` importlib.util.find_spec` alternative.

These changes were tested by running the "proof-of-exploit" examples,
the pyscan tests in this project, and by running `check_build` on
several projects (such as `black`) that enable Pyscan.
Atheris: Among many useful patches, the Python 3.10 compatability fixes
in v2.2.2 are of particular note.
See https://github.com/google/atheris/releases/tag/2.2.2

Pyinstaller: Dependency collection was improved significantly between
Pyintstaller v5 and v6, in both the core library, and the more recent
`pyinstaller-hooks-contrib` package it ships with.

Pyinstaller versions 3.9.0 & 3.10.0 are particularly noteworthy.
3.9.0 includes updates for scipy, numpy 2.0.0, and Django to fix
compatibility issues.
3.10.0 implements support for setuptools >= 71.0.0 and its new approach
to vendoring its dependencies.
See: https://setuptools.pypa.io/en/latest/history.html

Setuptools: Many projects expect a more recent version of setuptools
than was previously installed, including the pyscanner sanatizer from
this repo:
`infra/base-images/base-builder/sanitizers/pysecsan/`
@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 13, 2024

@jonathanmetzman @DonggeLiu Could you please start a trial build with the latest changes?

Thanks in advance!

The following information provides context for the changes and is intended for reference.


Analysis of Recent Failures

After reviewing the recent run failures, here's what I found:

  • PySecSan Integration: Several projects that failed have pyscan sanitizer enabled (ENABLE_PYSECSAN="1"). The resolution should be achieved with commit
    3e3fa52

  • Dependency Resolution: Many failures were due to issues with dependency resolution caused by outdated Pyinstaller and Setuptools versions, which are resolved by commit e6fc52c. Initially, I wasn’t going to attempt a version bump of those deps, but seeing as how those very old dependency versions are a common source of broken builds in both Python 3.8 & 3.10, it's apparent that the bump is necessary.

  • Pillow will continue to fail but a patch is already prepared via [Pillow] Removed python symlink #12326

JVM/Java project falures

The failures in JVM/Java projects appear to be network errors unrelated to this PR. It seems that repo.maven.apache.org might be dropping requests due to excessive download attempts from the OSS-Fuzz infrastructure. The following error is common in the JVM project logs:

[ERROR] Failed to execute goal on project antlr3-maven-plugin: 
        Could not resolve dependencies for project 
        org.antlr:antlr3-maven-plugin:maven-plugin:3.5.4-SNAPSHOT: 
        Could not transfer artifact 
        org.apache.maven:maven-artifact:jar:3.8.4 from/to central 
        (https://repo.maven.apache.org/maven2): transfer failed for 
        https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.8.4/maven-artifact-3.8.4.jar: 
        Connection reset -> [Help 1]

Ensures the CI actions use the same Python version as OSS-Fuzz images.
Versions with multiple digits after the forst "." in the version number
must be quoted strings, otherwise the GH action runner does not read the
whole version number and actions fail with an error similar to:
> Error: The version '3.1' with architecture 'x64' was not found
Also upgraddes the Cloud SDK version to the latest availiable to attempt
to avoid a python 3.10 compat issue:" module 'collections' has no
attribute 'MutableMapping'" tracked here:
https://issuetracker.google.com/issues/202172882

This also resolves an error in the GH actions prompting for upgrade:
> The v0 series of google-github-actions/setup-gcloud is no longer
> maintained. It will not receive updates, improvements,
> or security patches.
The `>=` was unintentionally changed to `==` in commit:
e6fc52c

This reverts that change.
for consistentcy with pip commands in other files
The issue these attempted to solve appear to be related to GH Action caching and not the python version, meanwhile upgrading python in these actions introduces additional issues that would need to be addressed.

- Revert "Bump Python Version from 3.8 to 3.10 in GitHub Actions" from commit 8b056dc.

- Revert "Specify Python Version as Strings" from commit c4957f5.

- Revert "Bump  google-github-actions/setup-gcloud from v0 to v2" from commit 26a5c01.
@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 21, 2024

Edit: Disregard the comment below. Rerunning it on a merge was successful.

The CI check that is failing appears to be due to a transient network request failure and unrelated to the changes in this PR. From the logs:

fatal: unable to access 'https://github.com/llvm/llvm-project.git/': Failed to connect to github.com port 443: Connection timed out

@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 22, 2024

@jonathanmetzman, could you trigger another trial build whenever you have a chance? Thanks!

@DonggeLiu
Copy link
Contributor

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

DonggeLiu pushed a commit that referenced this pull request Aug 27, 2024
Pillow's oss-fuzz is currently failing, because Pillow main has [dropped
support for Python
3.8](python-pillow/Pillow#8183).

#12027 attempts to resolve this by upgrading the base builder to Python
3.10, but the [latest status
update](#12027 (comment))
over there says that Pillow [fails in that branch
with](https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e19625fb-a984-47b1-be32-f92b4c76fc7f.txt)
> Step 17: Step 6/11 : RUN ln -s /usr/local/bin/python3
/usr/local/bin/python && ln -s /bin/true /usr/local/bin/yum_install &&
ln -s /bin/true /usr/local/bin/yum && cd $SRC/Pillow && git submodule
update --init wheels/multibuild && bash $SRC/build_depends.sh
Step 17:  ---> Running in 932abd1dc89a
Step 17: �[91mln: failed to create symbolic link
'/usr/local/bin/python': File exists

So this pull request removes the python symlink in order to try and
help.

cc @hugovk

---------

Co-authored-by: Andrew Murray <[email protected]>
@radarhere
Copy link
Contributor

Pillow will continue to fail but a patch is already prepared via #12326

That PR has now been merged into master.

@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 28, 2024

Thanks, @DonggeLiu.

I've looked into the latest trial build failures from the build at 9db721e (GH Run: https://github.com/google/oss-fuzz/runs/29280838754) and outlined my findings below.

Latest trial build failures

Fix Identified

I've identified fixes for the failed projects in this section and committed them to a new branch on my fork for now.

Project: configparser

  • Log URL: log
  • Reason for Failure: Does not work with `setuptools>=70'.
  • Fix: DaveLak@c44388e

Project: fwupd

  • Log URL: log
  • Reason for Failure: Missing Python package dependency.
  • Fix: DaveLak@d04205e

Project: gitpython

  • Log URL: log
  • Reason for Failure: The fuzzing harness that failed is flaky.
  • Fix: I maintain that integration and will fix it. This is not a blocker for the PR.

Project: nbclassic

  • Log URL: log
  • Reason for Failure: Hardcoded references to Python 3.8 installation paths in build script.
  • Fix: DaveLak@b338e1f

Project: pffft

  • Log URL: log
  • Reason for Failure: pffft/generate_seed_corpus.py helper script dependency numpy was installed via apt-get but should use pip instead.
  • Fix: DaveLak@708a18e

Project: proto-plus-python

Project: pybind11

  • Log URL: log
  • Reason for Failure: Hardcoded references to Python 3.8 installation paths in build script.
  • Fix: DaveLak@2633c02

Project: pyzmq

  • Log URL: log
  • Reason for Failure: Missing libzmq3-dev apt-get dependency.
  • Fix: DaveLak@41b04bf

Project: django

  • Log URL: log
  • Reason for Failure: An ad-hoc workaround using the Dead Snakes PPA to install Python 3.10
  • Fix: DaveLak@185ac6b

Failure Related but Require Further Investigation

Project: six

  • Log URL: log
  • Reason for Failure: TBD
  • Fix: TBD

Project: ijson

  • Log URL: log
  • Reason for Failure: TBD
  • Fix: N/A

ijson appears to be the only coverage failure in the latest trial build run.

Project: pycrypto

As noted in the PR description above, Pycrypto is deprecated and this is unlikely to be fixed upstream.
The latest failure is different, but even if it is fixes, it will continue failing with the error: "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats". It seems like the issue described here.

Likely Unrelated?

The project below failed with the following message:

docker: Error response from daemon: image with reference gcr.io/oss-fuzz-base/base-runner-testing-upgrade-python-to-3-10-14 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.

I was not able to reproduce this locally.

Project: lua

Project: libxml2

Project: tinyusb

Project: tarantool

Unrelated Failures

Project: bloaty

@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 28, 2024

I'm not sure what these failed "Project tests / build" CI checks are about, but it seems to happen intermittently on this PR and the logs make me think it has something to do with a stale Docker image cache?

@DaveLak
Copy link
Contributor Author

DaveLak commented Aug 28, 2024

@DonggeLiu or @jonathanmetzman, could either of you comment on what next steps are necessary to get this landed?

Given the latest trial build is showing the changes proposed in this PR will only break ~9 project builds (7 of which I've prepared a patch for) and the significantly larger number projects that would benefit from the Python version bump, I'm eager to bring this over the finish line.

Is there anything more I can do for the time being?

@DaveLak
Copy link
Contributor Author

DaveLak commented Sep 25, 2024

@DonggeLiu @jonathanmetzman Just bumping this up in case it got missed. Could you share any thoughts on the feasibility of this Python upgrade approach and whether it’s worth keeping the branch updated?

@cujomalainey
Copy link
Contributor

Fyi any project that depends on zephyr is likely broken until this lands see zephyrproject-rtos/zephyr@9d1b361

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants