PythonBundle should perform 1 single pip check instead of each python package repeating it

[Micket]

Currently a bundle build has repeated

  >> running shell command:
        /apps/Test2/software/Python/3.12.3-GCCcore-13.3.0/bin/python -m pip check
        [started at: 2024-08-22 16:50:45]
        [working dir: /apps/Test2/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0]
        [output and state saved to /dev/shm/eb-ht0fx9fv/run-shell-cmd-output/python-ytqm4_tu]

due to each and every extension in the bundle doing this. But as far as i understand, this is checking the same thing every time. The bundle should do this instead, and only once (at the start)

[boegel]

Yeah, this has been annoying me for a while too, but I never got around to looking into it...

pip check is indeed a general check for all installed Python packages, it doesn't check for just a single extension, so just running it once is enough

[Flamefire]

I don't agree here that it is doing the same: Python packages have dependencies and as we specifically tell pip to not look at any dependency it is on us to make sure the order of the packages we install is correct, i.e. dependencies are installed before the dependent package. pip check would detect violations and might be able to report early when there is an issue instead of at the end.

At least that was the intention. However it looks like for PythonBundle the sanity check for each extension is run only at the end and one after an other, so the benefit of running it for each package vanishes.

Same applies to the unversioned_packages check done afterwards. Only issue I see there is the unversioned_packages EC param: I guess for extensions/pythonbundle we'd rather need to aggregate those values over all PythonPackage extensions.

Or better: Don't allow sanity_pip_check or unversioned_packages for extensions and force it in the top-level EC as it doesn't make sense to be set differently for multiple extensions: Both checks are checking everything.

Similar issue with module existence checks: easybuilders/easybuild-framework#4618

[Micket]

Sanity checks has always only ever run at the end of everything (including bundles), after even generating the module file and will also run after a --module-only rebuild, or a --skip --rebuild, (so redesigning bundles to run sanity checks in during the installation steps would not be easy)
pip does allow circular dependencies as well, and we do have those already, just a quick check i found Sphinx (pipdeptree --warn fail (don't understand why they call it fail, it's allowed as far as i understand))

* sphinxcontrib-serializinghtml => Sphinx => sphinxcontrib-serializinghtml
* sphinxcontrib-htmlhelp => Sphinx => sphinxcontrib-htmlhelp
* sphinxcontrib-applehelp => Sphinx => sphinxcontrib-applehelp
* sphinxcontrib-qthelp => Sphinx => sphinxcontrib-qthelp
* sphinxcontrib-devhelp => Sphinx => sphinxcontrib-devhelp
* Sphinx => sphinxcontrib-applehelp => Sphinx

so checking these one at a time would presumably fail

Yes the idea would be to disable these checks for pyhonpackage extensions and repeat it once and for all in the pythonbundle.

[Flamefire]

I'm working on it

[Flamefire]

I think I'm done: #3428

Tested with a more or less random collection of PythonPackage and PythonBundle ECs and counting the number of pip check log lines shows that it is now only called once per EC