Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Support creation and reading of StructuredDataset with local or remote uri #2914

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[BUG] Support creation and reading of StructuredDataset with local or remote uri #2914

wants to merge 8 commits into from
+120 −0

Commits on Nov 8, 2024

  1. Manually fill sd literal and metadata 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    8d3ea89 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2024

  1. Use StructuredDatasetTransformerEngine to set sd literal 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 10, 2024
    Configuration menu
    Copy the full SHA
    204d549 View commit details
    Browse the repository at this point in the history

  2. Add unit test for reading sd from uri 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 10, 2024
    Configuration menu
    Copy the full SHA
    56d9fd3 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Put tasks into wf to mimic real-world use cases 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e2344e9 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. add env 

    Signed-off-by: Future-Outlier <[email protected]>
    @Future-Outlier
    Future-Outlier committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2a689b8 View commit details
    Browse the repository at this point in the history

  2. env again 

    Signed-off-by: Future-Outlier <[email protected]>
    @Future-Outlier
    Future-Outlier committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a30df5e View commit details
    Browse the repository at this point in the history

  3. Modify python ff reading logic to a flyte task 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    72382ca View commit details
    Browse the repository at this point in the history

  4. Use task param instead of global path const 

    Signed-off-by: JiaWei Jiang <[email protected]>
    @JiangJiaWei1103
    JiangJiaWei1103 committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    14df985 View commit details
    Browse the repository at this point in the history