platform_system not correctly respected

[amin-nejad]

pip-compile does not respect platform_system correctly when it is specified multiple times for the same package. This is useful in cases where you need to have different versions of the same package on different OSs.

Environment Versions

1. Mac M1
2. Python version: 3.9.12
3. pip version: 22.3.1
4. pip-tools version: 6.12.1

Steps to replicate

Run pip-compile on the following requirements.in file:

grpcio-tools~=1.48.0 ; platform_system=="Darwin" 
grpcio-tools>=1.48.0 ; platform_system!="Darwin" 

Expected result

grpcio-tools should occur twice in the requirements.txt in a similar way to how it was specified

Actual result

Instead you get something like this where the dependency we specified only occurs once with the first platform_system.

grpcio==1.51.1
    # via grpcio-tools
grpcio-tools==1.48.2 ; platform_system == "Darwin"
    # via -r requirements/test.in
protobuf==3.20.3
    # via grpcio-tools

[webknjaz]

Not “the first”, just “current env”. See https://pip-tools.readthedocs.io/en/latest/#cross-environment-usage-of-requirements-in-requirements-txt-and-pip-compile and #826 (comment).

[amin-nejad]

Thanks, that's very helpful. I didn't realise it was because I was running on Mac and also that the feature I thought existed does not exist (yet). Will close this in favour of #826 then

[gwerbin]

Not “the first”, just “current env”. See https://pip-tools.readthedocs.io/en/latest/#cross-environment-usage-of-requirements-in-requirements-txt-and-pip-compile and #826 (comment).

Is there any feature planned to support cross-platform builds?

For example, my team is lucky enough to have M1 Macs to develop on, but for unhappy reasons beyond our control we are stuck deploying to x86-64. We've been OK for now just running pip-compile on developers' ARM64 machines. Fortunately it's very rare to use environment markers like platform_machine, but you never know, especially with complicated "scientific" libraries.

As far as I understand, the pedantically-correct alternative would be to run pip-compile in an x86-64 VM. Using Docker Desktop, this is much slower than running natively (~5 minutes for my current project!).

It would be great if I could run something like pip-compile --platform x86-64 to circumvent this problem.

[webknjaz]

@gwerbin I don't think it's possible to implement with the current state of the ecosystem. pip-tools doesn't do anything special around building individual dists — it just uses pip and whatever underlying tooling each dist calls. It's up to you to provide an environment that is actually capable of building those things.
Although, the question of cross-compliling has been brought up this year on the PyCon's Python Packaging Summit. Still, there's a lot of work to make this happen so don't hold your breath.