Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix encoding mismatch between python child process and uv #7757

Merged
merged 1 commit into from
Sep 28, 2024

Conversation

kahojyun
Copy link
Contributor

Summary

This PR fixes #7733. According to CPython documentation on sys.stdout, when stdout/stderr is non-character device like pipe, the encoding will be set to system locale on windows. However, on the Rust side stdout_reader and stderr_reader expect them to be encoded in UTF-8 and will fail when child process write non-ASCII character to stdout/stderr, e.g., build directory name containing non-ASCII character.

Both CPython3 and PyPy support environment variable PYTHONIOENCODING. When it is set to utf-8, python will use UTF-8 encoding for stdin/stdout/stderr. Since stdin is not used by the spawned python process and we expect stdout/stderr to use UTF-8, this fix should work as expected.

Test Plan

I only tested it on my computer with CPython 3.12 and 3.7. With the fix applied I confirmed that the case I described is fixed.

I'm using Windows 11 with system locale set to code page 936.

@charliermarsh charliermarsh added bug Something isn't working windows Specific to the Windows platform labels Sep 28, 2024
@charliermarsh
Copy link
Member

\cc @BurntSushi

Copy link
Member

@BurntSushi BurntSushi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds right to me. Nice catch.

@charliermarsh charliermarsh enabled auto-merge (squash) September 28, 2024 12:38
@charliermarsh
Copy link
Member

Great find, thanks!

@charliermarsh charliermarsh merged commit 1cae78d into astral-sh:main Sep 28, 2024
60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working windows Specific to the Windows platform
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UTF-8 Encoding Errors Prevent Package Installation
3 participants