[Task]: Include python-snappy in Python 3.11 containers once python-snappy supports Python 3.11

[AnandInguva]

What needs to happen?

What needs to happen?

Currently, the installation of python-snappy from sdist fails. the wheels for python 3.11 is not yet available. Beam doesn't directly depend on python-snappy but we would eventually want to include it in the python 3.11 container.

Tracking issue to make sure Beam includes python-snappy in the Python 3.11 containers.

python-snappy issue: intake/python-snappy#124

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[smeet07]

.take-issue

[riteshghorse]

Are we on track for these for 2.48? @AnandInguva
I'll be cutting the branch today

[AnandInguva]

3.11 wheels are not available for this yet. We can change the milestone on this.

[Abacn]

no new python-snappy release yet, move onto 2.50.0

[lostluck]

2.50 release manager here.
This issue is currently tagged for the 2.50.0 release, which cuts in a week on August 9th.

Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15

[riteshghorse]

No new releases from python-snappy to support it. Moving to 2.51.0

[damccorm]

No new release, bumping it again.

[damccorm]

Bumping this again since there's no new release. From intake/python-snappy#124 it looks like this may not actually ever happen. @AnandInguva any reason we need this in the container or can we just close this?

[tvalentyn]

Actually snappy is installable on Python 3.11 as long as one installs snappy linux headers:

sudo apt install libsnappy-dev

Dockerfile already includes this dep, and snappy is installable in Beam Python 3.11 docker containers. However requirement generation fails to process this manual dependency, since on most beam committers machines (and maybe on GHA workers?) libsnappy-dev is not installed.

[tvalentyn]

snappy is optional ; i think it might be used by some fastavro users if they use snappy codec, and it optimizies TFRecordIO, which falls back on less efficient implementation if snappy is not provided. Given that snappy is no longer maintained, there might be more efficient alternatives (zstd?)

[tvalentyn]

However requirement generation fails to process this manual dependency, since on most beam committers machines (and maybe on GHA workers?) libsnappy-dev is not installed.

One workaround for issues like this (and also to enable macbook users), would be to generate requirements using Beam's docker container, as defined by Dockerfile, instead of a virtual environment on a committer's local machine. Although that would require having access to docker or cloudbuild.

[jrmccluskey]

@tvalentyn do you anticipate finishing this task before the branch cut?

[gliptak]

python-snappy released support for Python 3.11

intake/python-snappy#124