Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX-#7381: Fix Series binary operators ignoring fill_value #7394

Merged
merged 9 commits into from
Sep 17, 2024

Conversation

noloerino
Copy link
Collaborator

@noloerino noloerino commented Sep 12, 2024

What do these changes do?

Many binary operators in the frontend Series class were not passing their fillna parameters to the query compiler (not sure why this wasn't caught in linting). For arithmetic functions, the fix is simply to propagate fillna, but for logical operators (eq/ge etc.), the equivalent DataFrame method does not take a fillna parameter, so we need to add a separate query compiler method to handle this case.

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves BUG: Series binary operators don't respect fill_value #7381
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

@noloerino noloerino marked this pull request as ready for review September 12, 2024 20:55
@noloerino noloerino changed the title FIX-#7381: Fix Series binary operators ignoring fillna FIX-#7381: Fix Series binary operators ignoring fill_value Sep 12, 2024
@noloerino noloerino marked this pull request as draft September 12, 2024 22:17
@noloerino noloerino marked this pull request as ready for review September 12, 2024 23:19


@pytest.mark.parametrize("op", ["eq", "ge", "gt", "le", "lt", "ne"])
def test_logical_binary_with_list(op):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this relate to the problem with fill_value?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first pass at implementing fill_value created errors for these functions when the second operand was not a Series, so I added this test to make sure it didn't break.

level=level,
fill_value=fill_value,
axis=axis,
squeeze_other=isinstance(other, (pandas.Series, Series)),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to do squeeze in case of a series?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using the Binary.register/BinaryDefault.register operators in the query compiler to implement these functions, which treats the two arguments as pandas.DataFrames when applying the passed function to partitions. I'm not sure if there's a more intelligent way of forcing the arguments to be treated as pandas.Series instead.

modin/pandas/series.py Outdated Show resolved Hide resolved
modin/pandas/series.py Outdated Show resolved Hide resolved
modin/pandas/series.py Outdated Show resolved Hide resolved
modin/pandas/series.py Outdated Show resolved Hide resolved
modin/pandas/series.py Outdated Show resolved Hide resolved
modin/pandas/series.py Outdated Show resolved Hide resolved
@anmyachev anmyachev merged commit 6cf3ca2 into modin-project:main Sep 17, 2024
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: Series binary operators don't respect fill_value
2 participants