use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x #20879

bedroge · 2024-06-21T12:21:44Z

I ran into this when installing ReFrame on a system with Cython 3:

==> [+pygelf] python3 -m pip install --no-cache-dir -q -r /tmp/eb-myh20jsw/tmp.V9e2PYU6tR --target=external/ --upgrade
DEPRECATION: Loading egg at /cvmfs/riscv.eessi.io/versions/20240402/software/linux/riscv64/generic/software/ReFrame/4.3.3/lib/python3.11/site-packages/pip-21.3.1-py3.11.egg is deprecated. pi
p 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
  error: subprocess-exited-with-error
  
   Getting requirements to build wheel did not run successfully.
   exit code: 1
  > [54 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb/easybuild/build/ReFrame/4.3.3/system-system/reframe/reframe-4.3.3/external/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 288, in <module>
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 976, in run_command
          super().run_command(command)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 204, in get_source_files
        File "/tmp/eb-myh20jsw/pip-build-env-afkyp1z7/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

 Getting requirements to build wheel did not run successfully.
 exit code: 1

Apparently PyYAML 6.0 has a known issue regarding Cython 3, and is not compatible with this version:
yaml/pyyaml#736
They worked around it by pinning the Cython build constraint to < 3.0:
yaml/pyyaml#702
And they released 6.0.1 with (only) this workaround: https://github.com/yaml/pyyaml/blob/6.0.1/announcement.msg

This PR replaces 6.0 in the requirements.txt by 6.0.1 for recent ReFrame version. Since nothing else is changed in PyYAML, this should be a safe change.

ReFrame 4.4.x and newer already use 6.0.1, so this fix can be removed for those versions.

bedroge · 2024-06-21T12:30:16Z

@boegelbot please test @ generoso

boegelbot · 2024-06-21T12:35:08Z

@bedroge: Request for testing this PR well received on login1

PR test command 'EB_PR=20879 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_20879 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 13796

Test results coming soon (I hope)...

- notification for comment with ID 2182662114 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-06-21T12:38:44Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/117eee9aac35a9f844a8afba0acc2f5c for a full test report.

bedroge · 2024-06-21T12:42:14Z

Test report by @bedroge
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
interactive1 - Linux Rocky Linux 8.9, x86_64, AMD EPYC-Milan Processor (zen2), Python 3.6.8
See https://gist.github.com/bedroge/3f8321df1b5f0d9edea0b85258e3329b for a full test report.

bedroge · 2024-06-21T12:46:05Z

@boegelbot please test @ jsc-zen3

boegelbot · 2024-06-21T12:50:08Z

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=20879 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_20879 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 4442

Test results coming soon (I hope)...

- notification for comment with ID 2182687643 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-06-21T12:52:20Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/9c6a4f21f69bc8e97596787bb0bc7047 for a full test report.

boegel

lgtm

boegel · 2024-07-03T17:00:18Z

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3158.skitty.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/boegel/5a90b66337fffe8d6253fa184122ce54 for a full test report.

boegel · 2024-07-03T17:06:15Z

Going in, thanks @bedroge!

bedroge added 2 commits June 21, 2024 14:15

let ReFrame install PyYAML 6.0.1 instead of 6.0 to fix Cython issue

3d5643f

add comment about sed command

830385d

bedroge added bug fix EESSI Related to EESSI project labels Jun 21, 2024

bedroge mentioned this pull request Jun 21, 2024

Notes on kickstarting the RISC-V software layer EESSI/software-layer#552

Open

switch order of lines to make CI happy about comment

859aac5

boegel added this to the release after 4.9.2 milestone Jul 3, 2024

boegel approved these changes Jul 3, 2024

View reviewed changes

boegel merged commit 83a74ba into easybuilders:develop Jul 3, 2024
9 checks passed

bedroge deleted the fix_reframe433_dep_pyyaml branch July 4, 2024 09:20

boegel changed the title ~~use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions~~ use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x Sep 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x #20879

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x #20879

bedroge commented Jun 21, 2024

bedroge commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegelbot commented Jun 21, 2024

bedroge commented Jun 21, 2024

bedroge commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegel left a comment

boegel commented Jul 3, 2024

boegel commented Jul 3, 2024

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x #20879

use PyYAML 6.0.1 instead of 6.0 for recent ReFrame versions to fix problem with Cython 3.x #20879

Conversation

bedroge commented Jun 21, 2024

bedroge commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegelbot commented Jun 21, 2024

bedroge commented Jun 21, 2024

bedroge commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegelbot commented Jun 21, 2024

boegel left a comment

Choose a reason for hiding this comment

boegel commented Jul 3, 2024

boegel commented Jul 3, 2024