Create new transform to ingest markdown (.md) files and convert to parquet format #364
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rquet format ## Why are these changes needed? Convert .md files to parquet files so that they can be processed by data prep pipeline This is the preferred input for InstructLab ## Related issue number (if any). 178
daw3rd
requested changes
Jul 2, 2024
There was a problem hiding this comment.
Thank you for the contribution. This ingest2parquet tool is destined to be deprecated in favor of the new code2parquet transform which allows for more scalable conversions (i.e. think TB of data). I might suggest using code2parquet as a template to create a new transform, name it markdown2parquet and put it in a new directory called transforms/language/markdown2parquet. Let me know if you have questions
|
Also, in the future can you sign your commits? |
|
Oh and the code2parquet transform is in transforms/code/code2parquet |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Convert .md files to parquet files so that they can be processed by data prep pipeline
This is the preferred input for InstructLab
Related issue number (if any).
178