Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(python): Add security warning in LazyFrame.deserialize() docstring #15282

Merged
merged 2 commits into from
Apr 13, 2024

Conversation

douglas-raillard-arm
Copy link
Contributor

Add indication that LazyFrame.deserialize() might execute arbitrary code coming from the deserialized data.

Fixes #14623

Copy link
Member

@stinodego stinodego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the text looks good. One note about formatting.

I think this also applies to Expr.deserialize, right?

Comment on lines 613 to 616
.. warning::
This function uses :mod:`pickle` under some circumstances, and as
such inherits the security implications. Deserializing can execute
arbitrary code so it should only be attempted on trusted data.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add this as a Warnings section rather than an admonition? The section should be below Parameters / above See Also. See https://numpydoc.readthedocs.io/en/latest/format.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should elaborate here. If we put a big fat security warning we should explain under which conditions. Polars only uses pickle if the logical plan was serialized with python UDFs. In most queries Polars won't use pickle.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ritchie46 is that relevant for deserialize() though ? From the security perspective, what matters is whether you trust the data. If so, it does not matter whether it's using pickle. However, if you don't trust the data, it's very unlikely you would trust a promise that this JSON does not contain the UDF. As such, if the warning applies to your case, you'd have to assume it contains a malicious bit of pickle. So unless there is a way for the user to prevent loading pickle at the loss of some feature (e.g. a deserialize(..., forbid_pickle=True) parameter), the conditions of production don't really matter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It gives you more information on when we call pickle, I think that's important. You can evenly verify that the JSON doesn't have any UDFs as it is human readable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for what it's worth, I can see us adding a forbid pickle.

Copy link

codecov bot commented Mar 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.32%. Comparing base (8f616b8) to head (88b1a58).
Report is 75 commits behind head on main.

❗ Current head 88b1a58 differs from pull request most recent head 299f73c. Consider uploading reports for the commit 299f73c to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #15282      +/-   ##
==========================================
- Coverage   81.32%   81.32%   -0.01%     
==========================================
  Files        1359     1359              
  Lines      176072   176072              
  Branches     2526     2526              
==========================================
- Hits       143195   143185      -10     
- Misses      32393    32403      +10     
  Partials      484      484              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…r.deserialize() docstring

Add indication that LazyFrame.deserialize() and Expr.deserialize() might
execute arbitrary code coming from the deserialized data.

Fixes pola-rs#14623
@douglas-raillard-arm
Copy link
Contributor Author

Updated the PR with:

  • Use of warnings section instead of Sphinx warning directive
  • Added information on the circumstances leading to having pickle blobs in the serialize() JSON

@ritchie46 ritchie46 merged commit ded6302 into pola-rs:main Apr 13, 2024
11 of 12 checks passed
@ritchie46
Copy link
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LazyFrame.deserialize() should document the security implications
3 participants