[Bug]: PythonDocs fail to build after moving to Python3.9

[Abacn]

What happened?

https://github.com/apache/beam/actions/workflows/beam_PreCommit_PythonDocs.yml is actually failing:

Traceback (most recent call last):
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/registry.py", line 483, in load_extension
    metadata = mod.setup(app)
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/alabaster/__init__.py", line 31, in setup
    app.require_sphinx("3.4")
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/application.py", line 462, in require_sphinx
    raise VersionRequirementError(version)
sphinx.errors.VersionRequirementError: 3.4

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/cmd/build.py", line 300, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/application.py", line 224, in __init__
    self.setup_extension(extension)
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/application.py", line 449, in setup_extension
    self.registry.load_extension(self, extname)
  File "/runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-docs/docs/lib/python3.9/site-packages/sphinx/registry.py", line 486, in load_extension
    raise VersionRequirementError(
sphinx.errors.VersionRequirementError: The alabaster extension used by this project needs at least Sphinx v3.4; it therefore cannot be built with this version.

Sphinx version error:
The alabaster extension used by this project needs at least Sphinx v3.4; it therefore cannot be built with this version.
9.56user 4.21system 0:10.35elapsed 133%CPU (0avgtext+0avgdata 221000maxresident)k
Running Sphinx v1.8.5
88inputs+12808outputs (3major+92663minor)pagefaults 0swaps

Sphinx version error:
The alabaster extension used by this project needs at least Sphinx v3.4; it therefore cannot be built with this version.
Browse to file:///runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/docs/_build/index.html
docs: commands_post[0]> bash /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/scripts/run_tox_cleanup.sh

However, looks like our build target does not check return code, and the task as well as GHA workflow always pass

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

This prevents generating site PR for Beam 2.60.0RC1. For now I am downgrading workflow python version to 3.8, but still need to be fixed by the next release cc: @jrmccluskey

[Abacn]

after moving back to Python3.8, Build failing https://github.com/apache/beam/actions/runs/11266727364/job/31330701433 due to some commit originally should fail pydoc test slipped in

[Abacn]

I am trying to downgrade the PreCommit PythonDocs to run with Python 3.8, find error:

https://github.com/apache/beam/actions/runs/11267199804/job/31331924456

[jrmccluskey]

Should hopefully just be a sphinx version upgrade, let me see how intensive that is

[jrmccluskey]

For reference, we pin sphinx<=2.0, the latest sphinx release is 8.0.2. The last release supporting Python 3.9 is 7.4.7, so the goal is to catch us up to speed there

[jrmccluskey]

Okay https://github.com/apache/beam/actions/runs/11276977276/job/31362056928 should pass with modernized dependencies. The biggest issue is that the original pydoc script will fail if there are any warnings generated, not just errors, and I don't know if it's particularly worth it to try and plug all of those holes while this is a release blocker. Assuming I don't find other issues in subsequent runs, I can open a PR to get these changes into master

[Abacn]

Okay https://github.com/apache/beam/actions/runs/11276977276/job/31362056928 should pass with modernized dependencies. The biggest issue is that the original pydoc script will fail if there are any warnings generated, not just errors, and I don't know if it's particularly worth it to try and plug all of those holes while this is a release blocker. Assuming I don't find other issues in subsequent runs, I can open a PR to get these changes into master

Thanks, that sounds good