-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: add support for Spark Connect #63
feature: add support for Spark Connect #63
Commits on Aug 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 2e86208 - Browse repository at this point
Copy the full SHA 2e86208View commit details
Commits on Sep 10, 2024
-
Merge remote-tracking branch 'origin/main' into 33-feature-ensure-tha…
…t-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for 0aa6a53 - Browse repository at this point
Copy the full SHA 0aa6a53View commit details
Commits on Sep 19, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a9fe361 - Browse repository at this point
Copy the full SHA a9fe361View commit details
Commits on Sep 30, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f2ab79f - Browse repository at this point
Copy the full SHA f2ab79fView commit details
Commits on Oct 8, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f3e6014 - Browse repository at this point
Copy the full SHA f3e6014View commit details
Commits on Oct 9, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f24ed3e - Browse repository at this point
Copy the full SHA f24ed3eView commit details -
refactor: replace spark_minor_version with SPARK_MINOR_VERSION consta…
…nt for consistency
Configuration menu - View commit details
-
Copy full SHA for ecbdc9a - Browse repository at this point
Copy the full SHA ecbdc9aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 18ff011 - Browse repository at this point
Copy the full SHA 18ff011View commit details -
Merge remote-tracking branch 'origin/33-feature-ensure-that-we-can-su…
…pport-dbr-143lts' into 33-feature-ensure-that-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for 8a84038 - Browse repository at this point
Copy the full SHA 8a84038View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a291b2 - Browse repository at this point
Copy the full SHA 6a291b2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3bfe84a - Browse repository at this point
Copy the full SHA 3bfe84aView commit details -
Configuration menu - View commit details
-
Copy full SHA for a3a5ad6 - Browse repository at this point
Copy the full SHA a3a5ad6View commit details
Commits on Oct 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 31bb7d7 - Browse repository at this point
Copy the full SHA 31bb7d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c837cd4 - Browse repository at this point
Copy the full SHA c837cd4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3b09e5e - Browse repository at this point
Copy the full SHA 3b09e5eView commit details -
Configuration menu - View commit details
-
Copy full SHA for ac9eee5 - Browse repository at this point
Copy the full SHA ac9eee5View commit details
Commits on Oct 15, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b360d8e - Browse repository at this point
Copy the full SHA b360d8eView commit details -
Configuration menu - View commit details
-
Copy full SHA for cfab89f - Browse repository at this point
Copy the full SHA cfab89fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 891c7f5 - Browse repository at this point
Copy the full SHA 891c7f5View commit details
Commits on Oct 16, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 349770c - Browse repository at this point
Copy the full SHA 349770cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1b47c75 - Browse repository at this point
Copy the full SHA 1b47c75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c93701 - Browse repository at this point
Copy the full SHA 4c93701View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1a77512 - Browse repository at this point
Copy the full SHA 1a77512View commit details
Commits on Oct 21, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 224c0cc - Browse repository at this point
Copy the full SHA 224c0ccView commit details -
Configuration menu - View commit details
-
Copy full SHA for 42b7d86 - Browse repository at this point
Copy the full SHA 42b7d86View commit details -
fix: improve session handling and type annotations in connect_utils a…
…nd delta readers
Configuration menu - View commit details
-
Copy full SHA for 0c3b4f1 - Browse repository at this point
Copy the full SHA 0c3b4f1View commit details
Commits on Oct 22, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 586d76a - Browse repository at this point
Copy the full SHA 586d76aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 46a18ca - Browse repository at this point
Copy the full SHA 46a18caView commit details -
Configuration menu - View commit details
-
Copy full SHA for 16290e3 - Browse repository at this point
Copy the full SHA 16290e3View commit details -
Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…
…rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for fd415ad - Browse repository at this point
Copy the full SHA fd415adView commit details -
Configuration menu - View commit details
-
Copy full SHA for 77aa482 - Browse repository at this point
Copy the full SHA 77aa482View commit details -
Configuration menu - View commit details
-
Copy full SHA for a8219f8 - Browse repository at this point
Copy the full SHA a8219f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for e32e9a7 - Browse repository at this point
Copy the full SHA e32e9a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9fda4df - Browse repository at this point
Copy the full SHA 9fda4dfView commit details -
Configuration menu - View commit details
-
Copy full SHA for f251e87 - Browse repository at this point
Copy the full SHA f251e87View commit details -
Configuration menu - View commit details
-
Copy full SHA for a7319f6 - Browse repository at this point
Copy the full SHA a7319f6View commit details -
Merge remote-tracking branch 'origin/main' into 33-feature-ensure-tha…
…t-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for 53bb8ec - Browse repository at this point
Copy the full SHA 53bb8ecView commit details -
Configuration menu - View commit details
-
Copy full SHA for b570064 - Browse repository at this point
Copy the full SHA b570064View commit details -
Configuration menu - View commit details
-
Copy full SHA for 916e1a8 - Browse repository at this point
Copy the full SHA 916e1a8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 635a525 - Browse repository at this point
Copy the full SHA 635a525View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5856960 - Browse repository at this point
Copy the full SHA 5856960View commit details -
Configuration menu - View commit details
-
Copy full SHA for d377bb1 - Browse repository at this point
Copy the full SHA d377bb1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 95a9d70 - Browse repository at this point
Copy the full SHA 95a9d70View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9baca2c - Browse repository at this point
Copy the full SHA 9baca2cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e63806 - Browse repository at this point
Copy the full SHA 3e63806View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b28ba4 - Browse repository at this point
Copy the full SHA 7b28ba4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29e784f - Browse repository at this point
Copy the full SHA 29e784fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e9b0aca - Browse repository at this point
Copy the full SHA e9b0acaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 988fb03 - Browse repository at this point
Copy the full SHA 988fb03View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0fd123 - Browse repository at this point
Copy the full SHA b0fd123View commit details -
Configuration menu - View commit details
-
Copy full SHA for b426f66 - Browse repository at this point
Copy the full SHA b426f66View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14c6519 - Browse repository at this point
Copy the full SHA 14c6519View commit details -
Configuration menu - View commit details
-
Copy full SHA for a1ce806 - Browse repository at this point
Copy the full SHA a1ce806View commit details -
Configuration menu - View commit details
-
Copy full SHA for 754a21e - Browse repository at this point
Copy the full SHA 754a21eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7fddd06 - Browse repository at this point
Copy the full SHA 7fddd06View commit details -
Configuration menu - View commit details
-
Copy full SHA for ccaed64 - Browse repository at this point
Copy the full SHA ccaed64View commit details
Commits on Oct 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 56e4f6c - Browse repository at this point
Copy the full SHA 56e4f6cView commit details
Commits on Oct 24, 2024
-
33 part 2 making snowflake work with connect (#84)
Snowflake integrations is extensively reworked to be able to support Spark Connect. Highlights: - Original spark module has been split across respective spark and python modules (original modules retain the API through imports) - new package: src/koheesio/integrations/snowflake - new module: src/koheesio/integrations/snowflake/test_utils - new module: src/koheesio/integrations/spark/snowflake Detailed changes ---------------- File: src/koheesio/integrations/snowflake.py - Now a package (code moved to __init__.py) - Fixed links in module docs File: src/koheesio/integrations/spark/snowflake.py - TagSnowflakeQuery, and map_spark_type added to __all__ - breaking API changes for CreateOrReplaceTableFromDataFrame, AddColumn, SynchronizeDeltaToSnowflakeTask, 'account' field now is mandatory as they now use python snowflake connector (removing calls to Spark's JVM) - Spark specific SnowflakeSparkStep introduced - RunQuery is deprecated in favor of Python implementation File: src/koheesio/integrations/snowflake/test_utils.py - defines a reusable pytest fixture named mock_query that mocks the query execution for SnowflakeRunQueryPython, allowing tests to simulate query execution without connecting to Snowflake. File: tests/spark/integrations/snowflake/test_spark_snowflake.py - Added test to catch the deprecation warning in TestRunQuery. - Added test to catch RuntimeError when connect is being used, this test is skipped with a regular SparkSession - Fixed TestAddColumn tests since it is a DDL statement. - Updated all relevant tests to use dummy_spark fixture File: tests/spark/integrations/snowflake/test_sync_task.py - Minor refactoring of test body along with switching and updating some of the mocks / fixtures
Configuration menu - View commit details
-
Copy full SHA for a1bc658 - Browse repository at this point
Copy the full SHA a1bc658View commit details -
Configuration menu - View commit details
-
Copy full SHA for a9fbd1c - Browse repository at this point
Copy the full SHA a9fbd1cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5b9c716 - Browse repository at this point
Copy the full SHA 5b9c716View commit details
Commits on Oct 28, 2024
-
Configuration menu - View commit details
-
Copy full SHA for dcdf3d9 - Browse repository at this point
Copy the full SHA dcdf3d9View commit details -
Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…
…rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for 18f8873 - Browse repository at this point
Copy the full SHA 18f8873View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1f85306 - Browse repository at this point
Copy the full SHA 1f85306View commit details
Commits on Oct 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 4f88889 - Browse repository at this point
Copy the full SHA 4f88889View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c742b4 - Browse repository at this point
Copy the full SHA 1c742b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for b63dc7c - Browse repository at this point
Copy the full SHA b63dc7cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4751375 - Browse repository at this point
Copy the full SHA 4751375View commit details -
Configuration menu - View commit details
-
Copy full SHA for 67c1e68 - Browse repository at this point
Copy the full SHA 67c1e68View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3da04ae - Browse repository at this point
Copy the full SHA 3da04aeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a3733e - Browse repository at this point
Copy the full SHA 5a3733eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6af1f56 - Browse repository at this point
Copy the full SHA 6af1f56View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2e13df2 - Browse repository at this point
Copy the full SHA 2e13df2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 767974c - Browse repository at this point
Copy the full SHA 767974cView commit details -
Merge remote-tracking branch 'origin/33-feature-ensure-that-we-can-su…
…pport-dbr-143lts' into 33-feature-ensure-that-we-can-support-dbr-143lts # Conflicts: # src/koheesio/integrations/snowflake/__init__.py # src/koheesio/models/__init__.py # src/koheesio/steps/http.py # src/koheesio/utils.py
Configuration menu - View commit details
-
Copy full SHA for c724154 - Browse repository at this point
Copy the full SHA c724154View commit details -
Configuration menu - View commit details
-
Copy full SHA for d027370 - Browse repository at this point
Copy the full SHA d027370View commit details -
Configuration menu - View commit details
-
Copy full SHA for dba6204 - Browse repository at this point
Copy the full SHA dba6204View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc29864 - Browse repository at this point
Copy the full SHA cc29864View commit details -
Configuration menu - View commit details
-
Copy full SHA for 122f3ef - Browse repository at this point
Copy the full SHA 122f3efView commit details -
Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…
…rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
Configuration menu - View commit details
-
Copy full SHA for 117c2ba - Browse repository at this point
Copy the full SHA 117c2baView commit details -
Configuration menu - View commit details
-
Copy full SHA for ed3b1bf - Browse repository at this point
Copy the full SHA ed3b1bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for c9190ba - Browse repository at this point
Copy the full SHA c9190baView commit details -
fix: update GitHub Actions workflow to fallback to 'main' branch if b…
…ase ref is not set
Configuration menu - View commit details
-
Copy full SHA for e324ce2 - Browse repository at this point
Copy the full SHA e324ce2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7d6bbfe - Browse repository at this point
Copy the full SHA 7d6bbfeView commit details -
Configuration menu - View commit details
-
Copy full SHA for aad8c3a - Browse repository at this point
Copy the full SHA aad8c3aView commit details