Add standalone installation using zipapp

[BeyondEvil]

I have a use case where I need to install hatch in isolation (using direnv), and currently I solve that using pipx (installed as a zipapp). But I figured I could add the possibility of installing Hatch directly using a zipapp. That way I could ditch the dependency on pipx.

Hopefully others will find it useful as well.

Stolen like a thief in the night from: pypa/pipx#895

In draft because I want to test the flow first.

Successful run here.

[BeyondEvil]

ping @ofek

[manfred-kaiser]

There is one problem with zipapps. They are built with a specific python version.

The idea is very good, but the zipapp should be created by the user for each system and for each used python version.

Perhaps the "hatch" zipapp will work for different python version and systems, but it's not recommended.

[BeyondEvil]

Perhaps the "hatch" zipapp will work for different python version and systems, but it's not recommended.

Seems to work for pipx: https://github.com/pypa/pipx/blob/main/.github/workflows/tests.yml#L118

I've also successfully used pipx for internal company tools. Never had an issue with python version.

Unless of course the code itself uses features not available in older python versions.

[manfred-kaiser]

One problem of zipapps is, theay are built against a specific version.

While a lot of python applications only consist of pure python files, there are some libraries, which have compile dependencies to specific versions.

For example, if you create a shiv containing "cffi", a python version dependent package will be installed. For most python versions (cpython and pypy) there is a specific package available on pypi.org: https://pypi.org/project/cffi/#files

I have created a zipapp with shiv and python 3.10 to pack "hatch":

shiv hatch -o hatch.pyz

When opening the zipapp with an archive manager, you can find a _cffi_backend.cpython-310-x86_64-linux-gnu.so in the site-packages directory.

When built with python 3.11 following cffi backend is used: _cffi_backend.cpython-311-x86_64-linux-gnu.so

This is an indicator, that the zipapp is built with a specific python version and if the app requires the cffi backend there is a chance that the application will crash.

You can also notice it during the creation of the zip app.

When building with python 3.10 following file is installed:

Collecting cffi>=1.12 (from cryptography>=2.0->SecretStorage>=3.2->keyring>=23.5.0->hatch)
  Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

When built with python 3.11 follofing file is installed:

Collecting cffi>=1.12 (from cryptography>=2.0->SecretStorage>=3.2->keyring>=23.5.0->hatch)
  Using cached cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

As you can see, the cffi package is a dependency from cryptography, which is also a package which is built for a specific python version.

Your test will succeed, because it does not try to load the cryptography and the cffi package:

   - name: Test zipapp
      run: python ./hatch.pyz --version

@BeyondEvil python zipapps are cool and I'm also using shiv, because this avoids headaches during package management on production systems. But I had some python applications which depends on specific python versions, because of libraries like cryptography or cffi.

[BeyondEvil]

The obvious solution is to create zip-apps for all the supported python versions. Easy enough to matrix-ify my code.

[manfred-kaiser]

I like the idea of having a standalone version of hatch, because this avoids messing up the build system. Using shiv completely isolates hatch from the dev and build environments.

When providing a software as zipapp with shiv, you have to consider a few things.

Hatch itself has a lot of dependencies which receive regular updates. See the attached requirements.txt for a full list of direct and indirect packages: requirements.txt

You could recreate the zipapp every day. This allows to have an up to date zipapp. but with the drawback of reproducibility.

If you regulary update the zipapp, you should keep reproducable builds in mind. If you recreate the zipapp with updated dependencies, the behavior of the whole application can change and if dependencies are incompatible this can break the application.

To avoid such problems, you need a maintainer, who is responsible for updating and testing the zipapp. The minimum is to have a "requirements.txt" with the latest tested dependencies and those are used to build the zipapp.

One problem is to update the requirements.txt, because each version should be reproducable. This has security reasons because if you want to protect your software from supply chain attacks, you should know your dependencies. If you recreate the zipapp every day you should also set a tag, a version or something else, but this will mess up your git tags and versions in a short time.

If you have to check your software against known security risks, you must know which versions of your software are used. This includes the software itself, the build system and all used dependencies.

One can argue, that this version information are included in the metadata of the installed packages of the zipapp, but this is not the way how good dependency management works. The recommended way is to have a dependency file for each built and the best is to include it in the version control system connected to a release tag or a specific version.

matrix-ify the github action

If you want to provide prebuilt hatch packages as zipapp,matrix-ify your github action seems the only possible way.

I have also the problem of different python versions for my production systems and I have to setup my build system for each python version.

You should add more tests to your github action and deal with the mentioned problems for dependency management of your zipapp.

@BeyondEvil please do not take these comments as negative criticism. i am interested in such an implementation myself and would like to find a good solution.

[ofek]

Hatch v1.8.0 revamped the installation process: https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/#installation-made-easy

I appreciate the effort here but as was mentioned a few transitive dependencies like cryptography are not pure Python so indeed this would have required a matrix which I think wouldn't have been worth the effort.

Sorry for the wait and thank you again!