Build failure with pyyaml 6.0 and PEP517 #8287

hadialqattan · 2023-08-09T08:24:07Z

Poetry version: 1.5.1
Python version: 3.11
OS version and name: Linux
pyproject.toml: https://github.com/hadialqattan/pycln/blob/749cb325aa936a303c2f5ade671f4320e3387ae8/pyproject.toml

I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I'm experiencing a build failure when trying to install pyyaml 6.0 using Poetry on my CD system (Github Actions).
The log shows that pyyaml 6.0 does not support PEP 517 builds; however, I've confirmed that it does by running pip wheel --use-pep517 "pyyaml (==6.0)", which worked with no errors.

Logs full logs:

Creating virtualenv pycln-cDQ_M3T8-py3.11 in /github/home/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.

Package operations: 31 installs, 0 updates, 0 removals

  • Installing mypy-extensions (1.0.0)
  • Installing pyparsing (3.0.7)
  • Installing typing-extensions (4.6.2)
  • Installing attrs (22.2.0)
  • Installing iniconfig (1.1.1)
  • Installing pluggy (1.0.0)
  • Installing packaging (21.3)
  • Installing py (1.11.0)
  • Installing pyyaml (6.0)
  • Installing sortedcontainers (2.4.0)
  • Installing tomli (1.2.3)
  • Installing typing-inspect (0.9.0)

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  running egg_info
  writing lib/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/local/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 288, in <module>
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<string>", line 204, in get_source_files
    File "/tmp/tmpoagz8bvr/.venv/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  

  at /usr/local/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      [143](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:144)│ 
      [144](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:145)│                 error = ChefBuildError("\n\n".join(message_parts))
      [145](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:146)│ 
      [146](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:147)│             if error is not None:
    → [147](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:148)│                 raise error from None
      [148](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:149)│ 
      [149](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:150)│             return path
      [150](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:151)│ 
      [151](https://github.com/hadialqattan/pycln/actions/runs/5765910598/job/15632822881#step:4:152)│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyyaml (6.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyyaml (==6.0)"'.

The text was updated successfully, but these errors were encountered:

Secrus · 2023-08-09T08:34:27Z

Check if you have 6.0.1 in the lock file. 6.0 was broken by Cython 3.0 release, they patched it in 6.0.1

MajorDallas · 2023-08-09T22:02:04Z

I also ran into this issue, but I'm currently locked to PyYAML 5.4.1. There's a workaround described in yaml/pyyaml#736 and it does work....

I'm hoping it's possible (I missed it in the docs?) to get poetry to express an equivalent of the PIP_CONSTRAINT env var in pyproject.toml. Maybe eg.:

PyYAML = {version = '5.4.1', build_constraints = ['Cython < 3.0']}
# or some key in a section other than [tool.poetry.dependencies]?

That way I'm not left just hoping that my teammates will notice that the README now says to use this obscure env var with the add and install commands.

Secrus · 2023-08-09T22:12:14Z

I also ran into this issue, but I'm currently locked to PyYAML 5.4.1. There's a workaround described in yaml/pyyaml#736 and it does work....

I'm hoping it's possible (I missed it in the docs?) to get poetry to express an equivalent of the PIP_CONSTRAINT env var in pyproject.toml. Maybe eg.:
PyYAML = {version = '5.4.1', build_constraints = ['Cython < 3.0']}
# or some key in a section other than [tool.poetry.dependencies]?
That way I'm not left just hoping that my teammates will notice that the README now says to use this obscure env var with the add and install commands.

No, we don't handle build dependencies overriding, as this is considered a job for the project to express those. Also, the solution you linked won't work in modern versions of Poetry, since we don't use pip in modern installer.

If you need a workaround for older pyyaml versions, it would be to either grab a wheel from PyPI directly, or build wheel locally in a constrained environment and reference it as file dependency.

MajorDallas · 2023-08-09T22:24:36Z

we don't handle build dependencies overriding, as this is considered a job for the project to express those.

At risk of displaying an embarrassing lack of knowledge, isn't poetry (or pipenv or whatever) the way for a project to express its dependencies? If I can't capture this comprehensively in poetry, I must use something to fill in the gaps like a script, Make, etc. I'm not opposed to that, really, but I feel I've at least partially misunderstood either the aim of poetry or the distinction between a dependency and a "build dependency override."

hadialqattan · 2023-08-10T12:45:38Z

Check if you have 6.0.1 in the lock file. 6.0 was broken by Cython 3.0 release, they patched it in 6.0.1

Yeah upgrading to v6.0.1 works. Thanks a lot.

We experienced a build failure on GitHub Actions exactly the same as python-poetry/poetry#8287 See also: yaml/pyyaml#601

… python 3.12 See python-poetry/poetry#8287

AuroraTea · 2023-11-15T01:40:26Z

Amazingly enough, it was working fine for me before. When I switched computers and git cloned and then poetry installed, I had this problem.

I solved the problem by deleting the lock file.

python-poetry/poetry#8287

jankatins · 2024-01-29T18:07:36Z

This worked arround the problem in a python 3.12 poetry install (my other dependencies wanted <6:-(, so this ended up as 5.3.1):

pyyaml = { version = "!=6.0.0,!=5.4.0,!=5.4.1" }

github-actions · 2024-02-29T01:24:07Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

hadialqattan added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Aug 9, 2023

hadialqattan mentioned this issue Aug 10, 2023

Update: PyYaml 6.0 ~> 6.0.1 hadialqattan/pycln#216

Merged

hadialqattan closed this as completed in hadialqattan/pycln#216 Aug 10, 2023

tueda added a commit to tueda/py-check-updates that referenced this issue Aug 15, 2023

fix(ci): update PyYAML to 6.0.1 to fix Cython build

b4ff277

We experienced a build failure on GitHub Actions exactly the same as python-poetry/poetry#8287 See also: yaml/pyyaml#601

mmwinther mentioned this issue Aug 18, 2023

Action to update dependencies regularly statisticsnorway/dapla-statbank-authenticator#19

Merged

This was referenced Aug 16, 2023

Fix Staging Deployment hackforla/CivicTechJobs#430

Closed

Fixing staging deployment attempt 2 hackforla/CivicTechJobs#433

Merged

qkaiser added a commit to onekey-sec/unblob that referenced this issue Oct 13, 2023

chore(deps): upgrade pyyaml from 6.0 to 6.0.1 to fix build failure on…

b38f3b9

… python 3.12 See python-poetry/poetry#8287

aokblast mentioned this issue Oct 29, 2023

feat: update lock file because pyyaml 6.0 cannot be built in PEP517 ntnu-cp/noj-cli#11

Open

artis3n mentioned this issue Nov 1, 2023

chore: update dependencies artis3n/ansible-role-tailscale#387

Merged

iainelder mentioned this issue Nov 30, 2023

replace circleci docker image Sceptre/sceptre#1394

Closed

cozybear-dev mentioned this issue Dec 13, 2023

Issue with pyyaml in combination with poetry lock file, causing build failure cmehay/docker-tor-hidden-service#96

Open

liranbg mentioned this issue Dec 17, 2023

[Installation]: Getting requirements to build wheel did not run successfully. mlrun/mlrun#4204

Open

1 task

withtwoemms mentioned this issue Jan 8, 2024

Pyyaml fails to install scanapi/scanapi#675

Closed

jacobian mentioned this issue Jan 17, 2024

Figure out testing jacobian/personal-app-template#1

Open

otosky added a commit to otosky/home-ops that referenced this issue Jan 28, 2024

bump pyyaml to 6.0.1

2066a2d

python-poetry/poetry#8287

pcperera mentioned this issue Jan 29, 2024

Upgrade sdk, python and pyyaml in cognite-replicator cognitedata/cognite-replicator#222

Merged

shdobxr mentioned this issue Feb 26, 2024

Ran into some requirements issues with PyYaml==6.0 aws-samples/access-analyzer-automated-policy-analysis-blog#1

Open

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failure with pyyaml 6.0 and PEP517 #8287

Build failure with pyyaml 6.0 and PEP517 #8287

hadialqattan commented Aug 9, 2023 •

edited

Loading

Secrus commented Aug 9, 2023

MajorDallas commented Aug 9, 2023 •

edited by Secrus

Loading

Secrus commented Aug 9, 2023

MajorDallas commented Aug 9, 2023

hadialqattan commented Aug 10, 2023

AuroraTea commented Nov 15, 2023

jankatins commented Jan 29, 2024

github-actions bot commented Feb 29, 2024

Build failure with pyyaml 6.0 and PEP517 #8287

Build failure with pyyaml 6.0 and PEP517 #8287

Comments

hadialqattan commented Aug 9, 2023 • edited Loading

Issue

Secrus commented Aug 9, 2023

MajorDallas commented Aug 9, 2023 • edited by Secrus Loading

Secrus commented Aug 9, 2023

MajorDallas commented Aug 9, 2023

hadialqattan commented Aug 10, 2023

AuroraTea commented Nov 15, 2023

jankatins commented Jan 29, 2024

github-actions bot commented Feb 29, 2024

hadialqattan commented Aug 9, 2023 •

edited

Loading

MajorDallas commented Aug 9, 2023 •

edited by Secrus

Loading