Develop data transformation pipeline for optimal RAG and fine-tuning performance

[branhoff]

Description

In this paper RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
, a rough structure is outlined for transforming data into a usable state for RAG and fine-tuning enhancements of an LLM.

This ticket seeks to develop an initial process for transforming data that we scrape (car diagnostic manuals for instance) into a Q&A format.

Acceptance Criteria

1. Data should be in jsonl formats
2. Data should be structured as Q&A's
3. Q&A's should be reviewed and filtered by LLM's according to the criteria laid out in the paper
4. The implementation should be generic enough that this process should be easily repeatable.