Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add support for Spark Connect #63

Merged

Commits on Aug 9, 2024

  1. Configuration menu
    Copy the full SHA
    2e86208 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Merge remote-tracking branch 'origin/main' into 33-feature-ensure-tha…

    …t-we-can-support-dbr-143lts
    mikita-sakalouski committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0aa6a53 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    a9fe361 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Configuration menu
    Copy the full SHA
    f2ab79f View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Configuration menu
    Copy the full SHA
    f3e6014 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Configuration menu
    Copy the full SHA
    f24ed3e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ecbdc9a View commit details
    Browse the repository at this point in the history
  3. few more fixes

    dannymeijer committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    18ff011 View commit details
    Browse the repository at this point in the history
  4. Merge remote-tracking branch 'origin/33-feature-ensure-that-we-can-su…

    …pport-dbr-143lts' into 33-feature-ensure-that-we-can-support-dbr-143lts
    dannymeijer committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    8a84038 View commit details
    Browse the repository at this point in the history
  5. few more fixes

    dannymeijer committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    6a291b2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3bfe84a View commit details
    Browse the repository at this point in the history
  7. few more fixes

    dannymeijer committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    a3a5ad6 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Configuration menu
    Copy the full SHA
    31bb7d7 View commit details
    Browse the repository at this point in the history
  2. fix typo

    dannymeijer committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    c837cd4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3b09e5e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ac9eee5 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. last 24

    dannymeijer committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    b360d8e View commit details
    Browse the repository at this point in the history
  2. fix formatting

    dannymeijer committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    cfab89f View commit details
    Browse the repository at this point in the history
  3. one more test

    dannymeijer committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    891c7f5 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. 17 more remaining

    dannymeijer committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    349770c View commit details
    Browse the repository at this point in the history
  2. Last 21

    dannymeijer committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    1b47c75 View commit details
    Browse the repository at this point in the history
  3. Last 21

    dannymeijer committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    4c93701 View commit details
    Browse the repository at this point in the history
  4. Last 20

    dannymeijer committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    1a77512 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. EOD

    dannymeijer committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    224c0cc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    42b7d86 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0c3b4f1 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. fix: improve tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    586d76a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    46a18ca View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    16290e3 View commit details
    Browse the repository at this point in the history
  4. Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…

    …rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    fd415ad View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    77aa482 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    a8219f8 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e32e9a7 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    9fda4df View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f251e87 View commit details
    Browse the repository at this point in the history
  10. fix: improve tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a7319f6 View commit details
    Browse the repository at this point in the history
  11. Merge remote-tracking branch 'origin/main' into 33-feature-ensure-tha…

    …t-we-can-support-dbr-143lts
    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    53bb8ec View commit details
    Browse the repository at this point in the history
  12. fix: spark imports

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    b570064 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    916e1a8 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    635a525 View commit details
    Browse the repository at this point in the history
  15. fix: conftest

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    5856960 View commit details
    Browse the repository at this point in the history
  16. fix: tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    d377bb1 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    95a9d70 View commit details
    Browse the repository at this point in the history
  18. fix: remote port

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    9baca2c View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    3e63806 View commit details
    Browse the repository at this point in the history
  20. fix: tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    7b28ba4 View commit details
    Browse the repository at this point in the history
  21. fix: tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    29e784f View commit details
    Browse the repository at this point in the history
  22. fix: fail fast

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    e9b0aca View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    988fb03 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    b0fd123 View commit details
    Browse the repository at this point in the history
  25. fix: delta packages

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    b426f66 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    14c6519 View commit details
    Browse the repository at this point in the history
  27. fix: tests

    mikita-sakalouski committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a1ce806 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    754a21e View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    7fddd06 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    ccaed64 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. fix: test

    mikita-sakalouski committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    56e4f6c View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. 33 part 2 making snowflake work with connect (#84)

    Snowflake integrations is extensively reworked to be able to support
    Spark Connect.
    
    Highlights:
    - Original spark module has been split across respective spark and
    python modules (original modules retain the API through imports)
    - new package: src/koheesio/integrations/snowflake
    - new module: src/koheesio/integrations/snowflake/test_utils
    - new module: src/koheesio/integrations/spark/snowflake
    
    Detailed changes
    ----------------
    File: src/koheesio/integrations/snowflake.py
    - Now a package (code moved to __init__.py)
    - Fixed links in module docs
    
    File: src/koheesio/integrations/spark/snowflake.py
    - TagSnowflakeQuery, and map_spark_type added to __all__
    - breaking API changes for CreateOrReplaceTableFromDataFrame, AddColumn,
    SynchronizeDeltaToSnowflakeTask, 'account' field now is mandatory as
    they now use python snowflake connector (removing calls to Spark's JVM)
    - Spark specific SnowflakeSparkStep introduced
    - RunQuery is deprecated in favor of Python implementation
    
    File: src/koheesio/integrations/snowflake/test_utils.py
    - defines a reusable pytest fixture named mock_query that mocks the
    query execution for SnowflakeRunQueryPython, allowing tests to simulate
    query execution without connecting to Snowflake.
    
    File: tests/spark/integrations/snowflake/test_spark_snowflake.py
    - Added test to catch the deprecation warning in TestRunQuery.
    - Added test to catch RuntimeError when connect is being used, this test
    is skipped with a regular SparkSession
    - Fixed TestAddColumn tests since it is a DDL statement.
    - Updated all relevant tests to use dummy_spark fixture
    
    File: tests/spark/integrations/snowflake/test_sync_task.py
    - Minor refactoring of test body along with switching and updating some
    of the mocks / fixtures
    dannymeijer authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a1bc658 View commit details
    Browse the repository at this point in the history
  2. ran make fmt

    dannymeijer committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a9fbd1c View commit details
    Browse the repository at this point in the history
  3. small fix

    dannymeijer committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5b9c716 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Configuration menu
    Copy the full SHA
    dcdf3d9 View commit details
    Browse the repository at this point in the history
  2. Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…

    …rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
    mikita-sakalouski committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    18f8873 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1f85306 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. mypy down to 318

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    4f88889 View commit details
    Browse the repository at this point in the history
  2. progress

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    1c742b4 View commit details
    Browse the repository at this point in the history
  3. as good as I can make it

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    b63dc7c View commit details
    Browse the repository at this point in the history
  4. version bump

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    4751375 View commit details
    Browse the repository at this point in the history
  5. typo

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    67c1e68 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3da04ae View commit details
    Browse the repository at this point in the history
  7. fix: tests

    mikita-sakalouski committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    5a3733e View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    6af1f56 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2e13df2 View commit details
    Browse the repository at this point in the history
  10. some more improvements

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    767974c View commit details
    Browse the repository at this point in the history
  11. Merge remote-tracking branch 'origin/33-feature-ensure-that-we-can-su…

    …pport-dbr-143lts' into 33-feature-ensure-that-we-can-support-dbr-143lts
    
    # Conflicts:
    #	src/koheesio/integrations/snowflake/__init__.py
    #	src/koheesio/models/__init__.py
    #	src/koheesio/steps/http.py
    #	src/koheesio/utils.py
    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    c724154 View commit details
    Browse the repository at this point in the history
  12. some more improvements

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    d027370 View commit details
    Browse the repository at this point in the history
  13. small bugfix

    dannymeijer committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    dba6204 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    cc29864 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    122f3ef View commit details
    Browse the repository at this point in the history
  16. Merge branch '33-feature-ensure-that-we-can-support-dbr-143lts' of pe…

    …rsonal.github.com:Nike-Inc/koheesio into 33-feature-ensure-that-we-can-support-dbr-143lts
    mikita-sakalouski committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    117c2ba View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    ed3b1bf View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    c9190ba View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    e324ce2 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    7d6bbfe View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    aad8c3a View commit details
    Browse the repository at this point in the history