FEAT-#7308: Interoperability between query compilers #7376

arunjose696 · 2024-08-26T14:45:30Z

What do these changes do?

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Interoperability between DataFrames using different query compilers #7308
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

Signed-off-by: arunjose696 <[email protected]>

Signed-off-by: Igoshev, Iaroslav <[email protected]>

…rialized_dtypes to query compiler layer as in the code in multiple places the methods of private _modin_frame were used

Signed-off-by: Igoshev, Iaroslav <[email protected]>

Co-authored-by: Iaroslav Igoshev <[email protected]> Signed-off-by: arunjose696 <[email protected]>

Co-authored-by: Iaroslav Igoshev <[email protected]>

Signed-off-by: arunjose696 <[email protected]>

…fferent query compilers

Co-authored-by: Iaroslav Igoshev <[email protected]>

modin/core/storage_formats/pandas/native_query_compiler.py

modin/core/storage_formats/pandas/query_compiler_validator.py

modin/tests/pandas/native_df_mode/test_default.py

+            try:
+                operation()
+            # `except` for non callable attributes
+            except TypeError:


modin/tests/pandas/native_df_mode/test_default.py

+        modin_series_without_index, _ = create_test_series(
+            np.arange(col_len), data_frame_mode=data_frame_mode_pair[1]
+        )
+        modin_df @ modin_series_without_index


modin/tests/pandas/native_df_mode/test_indexing.py

modin/tests/pandas/utils.py

Signed-off-by: arunjose696 <[email protected]>

modin/core/storage_formats/pandas/query_compiler_validator.py

modin/tests/pandas/utils.py

anmyachev · 2024-08-27T16:09:34Z

modin/pandas/dataframe.py

-        assert (
-            isinstance(new_query_compiler, type(self._query_compiler))
-            or type(new_query_compiler) in self._query_compiler.__class__.__bases__


new_query_compiler can also be an instance of a different query compiler(Nativequerycompiler or PandasQueryCompiler),

Eg during operations like insert in the _create_or_update_from_compiler the constructor gets called directly eg ,

Thus it would be normal for the cases where constructor may return a new query compiler for a case where user changes the query compiler mode between creating data_frame and insert operation

modin/core/storage_formats/pandas/query_compiler_validator.py

modin/core/storage_formats/pandas/native_query_compiler.py

anmyachev · 2024-08-28T08:40:55Z

modin/tests/pandas/utils.py

+    if data_frame_mode:
+        actual_data_frame_mode = NativeDataframeMode().get()
+        NativeDataframeMode().put(data_frame_mode)


@arunjose696 I would recommend doing this at a higher level, using fixture modify_config.

It would not be possible to use this in the fixture , as the purpose of this is to set two different NativeDataframeMode for 2 different dataframes in the pytest, We dont want to be setting the NativeDataframeMode in the fixture as these dataframes wont be created in the fixture.

In that case let's create create_test_series_in_defined_mode function in modin/tests/pandas/native_df_mode/utils.py. Something like:

def create_test_series_in_defined_mode(vals, sort=False, backend=None, df_mode=None, **kwargs): with context(NativeDataframeMode=df_mode): return create_test_series(vals, sort=False, backend=None, **kwargs)

And similar for the dataframes.

The main motivation is not to introduce functionality where it is rarely used. Also, given that there is a separate folder with tests, it seems better to try to localize the necessary changes.

modin/core/storage_formats/pandas/query_compiler_validator.py

Co-authored-by: Iaroslav Igoshev <[email protected]> Co-authored-by: Anatoly Myachev <[email protected]>

modin/tests/pandas/native_df_mode/utils.py

+    modin_df2, pandas_df2 = create_test_dfs(data, backend=backend, df_mode=df_mode2)
+    md_kwargs, pd_kwargs = {}, {}
+
+    def execute_callable(fn, inplace=False, md_kwargs={}, pd_kwargs={}):


Signed-off-by: arunjose696 <[email protected]>

modin/core/storage_formats/pandas/native_query_compiler.py

 from modin.core.storage_formats.base.query_compiler import BaseQueryCompiler
+from modin.core.storage_formats.pandas.query_compiler_caster import QueryCompilerCaster


modin/core/storage_formats/pandas/query_compiler_caster.py

+
+from pandas.core.indexes.frozen import FrozenList
+
+from modin.core.storage_formats.base.query_compiler import BaseQueryCompiler


Signed-off-by: arunjose696 <[email protected]>

YarShev

@arunjose696, please take a look at the failing tests.

modin/tests/pandas/native_df_mode/utils.py

modin/tests/pandas/utils.py

Signed-off-by: arunjose696 <[email protected]>

anmyachev · 2024-09-02T11:58:46Z

modin/core/storage_formats/pandas/query_compiler_caster.py

+        return staticmethod(apply_argument_cast(obj.__func__))
+
+    @functools.wraps(obj)
+    def cast_args(*args: Tuple, **kwargs: Dict) -> Any:


@arunjose696 doesn't this function break type hints?

I am not clear how this would break type hints, as this function is similar to run_and_log function used in modin logging which would also be wrapping the qc layer.

I am not clear how this would break type hints, as this function is similar to run_and_log function used in modin logging which would also be wrapping the qc layer

It should be visible in IDE. Do you see problems with it or not?

I can see it in IDE

Thanks for checking. | Any seems redundant, someone will need to sort this out in the future.

arunjose696 and others added 28 commits August 26, 2024 16:37

FEAT-modin-project#4605: Add small query compiler

ba48d29

fixing tests

40e1cd4

removing additional parameter from try_cast_to_pandas

a574172

Signed-off-by: arunjose696 <[email protected]>

test_iter passing

6f9795b

fixing isin unique and clip

07d8d3a

Signed-off-by: arunjose696 <[email protected]>

Enable test_default.py and test_join_sort.py

ac75854

Signed-off-by: Igoshev, Iaroslav <[email protected]>

fixed test_map_metadata by adding set_frame_dtypes_cache and has_mate…

52b3136

…rialized_dtypes to query compiler layer as in the code in multiple places the methods of private _modin_frame were used

Fix test_dot

4aed823

Signed-off-by: Igoshev, Iaroslav <[email protected]>

test_udf passing

7149ba3

All tests except one passing in modin/tests/pandas/dataframe

b0e0a82

All tests in modin/tests/pandas/dataframe/ passing

507d58a

PR comments

88a6207

renaming to PlainPandasQueryCompiler to NativeDataframeMode

b01f7b9

renaming to PlainPandasQueryCompiler to NativeDataframeMode

1a61931

PR comments + changes

a105677

Apply suggestions from code review

8e29fbe

Co-authored-by: Iaroslav Igoshev <[email protected]> Signed-off-by: arunjose696 <[email protected]>

fix conflict

1620f82

Apply suggestions from code review

4392165

Co-authored-by: Iaroslav Igoshev <[email protected]>

PR comments

7a1a30f

Signed-off-by: arunjose696 <[email protected]>

FEAT-modin-project#7308: Interoperability between DataFrames using di…

4e24de2

…fferent query compilers

Started tests for interop

2308295

modified test_binary and test_default

d240ac8

Started adding testindexing

a4c1697

fixing envvars repeatation

89dabf9

adding tests

b212c3b

PR comments and tests

90ba1a4

Update modin/core/storage_formats/pandas/query_compiler_validator.py

e9060a7

Co-authored-by: Iaroslav Igoshev <[email protected]>

PR comments and tests

60f9512

arunjose696 requested review from devin-petersohn and mvashishtha as code owners August 26, 2024 14:45

arunjose696 force-pushed the arun-sqc-interop branch from fbe9e15 to 6a21aaf Compare August 26, 2024 14:48

github-advanced-security bot found potential problems Aug 26, 2024

View reviewed changes

arunjose696 force-pushed the arun-sqc-interop branch from 6a21aaf to cd70589 Compare August 26, 2024 14:54

arunjose696 changed the title ~~FEAT-#7308: Fix inserting datelike values into a DataFrame~~ FEAT-#7308: Interoperability between query compilers Aug 26, 2024

arunjose696 force-pushed the arun-sqc-interop branch from cd70589 to 9af55f2 Compare August 26, 2024 15:29

Doc updates

80ce01c

Signed-off-by: arunjose696 <[email protected]>

arunjose696 force-pushed the arun-sqc-interop branch from 9af55f2 to 80ce01c Compare August 26, 2024 17:11

YarShev reviewed Aug 27, 2024

View reviewed changes

anmyachev requested changes Aug 28, 2024

View reviewed changes

arunjose696 and others added 2 commits August 28, 2024 10:55

PR comments

6a068dc

Apply suggestions from code review

cac89d2

Co-authored-by: Iaroslav Igoshev <[email protected]> Co-authored-by: Anatoly Myachev <[email protected]>

github-advanced-security bot found potential problems Aug 28, 2024

View reviewed changes

arunjose696 force-pushed the arun-sqc-interop branch 2 times, most recently from 94e8e48 to af0a7ba Compare August 28, 2024 10:10

PR changes

8e8ec46

Signed-off-by: arunjose696 <[email protected]>

arunjose696 force-pushed the arun-sqc-interop branch from af0a7ba to 8e8ec46 Compare August 28, 2024 10:15

github-advanced-security bot found potential problems Aug 28, 2024

View reviewed changes

arunjose696 force-pushed the arun-sqc-interop branch 3 times, most recently from 04b7f60 to 03d137c Compare August 29, 2024 07:15

creating function for create_test_df_in_defined_mode

c1b0942

Signed-off-by: arunjose696 <[email protected]>

arunjose696 force-pushed the arun-sqc-interop branch from 03d137c to c1b0942 Compare August 29, 2024 08:25

YarShev reviewed Aug 29, 2024

View reviewed changes

creating function for create_test_df_in_defined_mode

f488872

Signed-off-by: arunjose696 <[email protected]>

arunjose696 force-pushed the arun-sqc-interop branch 2 times, most recently from 136bf1e to f488872 Compare September 2, 2024 08:03

YarShev approved these changes Sep 2, 2024

View reviewed changes

anmyachev reviewed Sep 2, 2024

View reviewed changes

anmyachev approved these changes Sep 2, 2024

View reviewed changes

YarShev merged commit cf5d638 into modin-project:main Sep 2, 2024
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#7308: Interoperability between query compilers #7376

FEAT-#7308: Interoperability between query compilers #7376

arunjose696 commented Aug 26, 2024

anmyachev Aug 27, 2024

arunjose696 Aug 28, 2024

anmyachev Aug 28, 2024

arunjose696 Aug 28, 2024

anmyachev Aug 28, 2024

YarShev left a comment

anmyachev Sep 2, 2024

arunjose696 Sep 2, 2024 •

edited

Loading

anmyachev Sep 2, 2024

arunjose696 Sep 2, 2024

anmyachev Sep 3, 2024

		from modin.core.storage_formats.base.query_compiler import BaseQueryCompiler
		from modin.core.storage_formats.pandas.query_compiler_caster import QueryCompilerCaster


		from pandas.core.indexes.frozen import FrozenList

		from modin.core.storage_formats.base.query_compiler import BaseQueryCompiler

FEAT-#7308: Interoperability between query compilers #7376

FEAT-#7308: Interoperability between query compilers #7376

Conversation

arunjose696 commented Aug 26, 2024

What do these changes do?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YarShev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arunjose696 Sep 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arunjose696 Sep 2, 2024 •

edited

Loading