Wrote simple streaming dataloader

[lennart-finke]

Description

Added a streaming dataloader to avoid storing large datasets.

Related Issue

Progress towards #8

[danbraunai-apollo]

Does this still work with distributed data parallel? I'm a bit concerned that this setup is more complex than needed. In general I think we'll need to support:

1. The dataset and tokenizer will be hosted on huggingface.
2. The pre-tokenized dataset will be hosted on huggingface (so we don't have to tokenize it on the fly everytime we train).

I think we can just get away with using huggingface's load_dataset with streaming=True. An example is here, which supports loading tokenized or untokenized datasets. Then we would just need to set it up to work for DDP. Not sure of the easiest way, there's probably standard setups here, maybe using a distributed sampler.

I've added the above to #8 (sorry I should have done this earlier).

[lennart-finke]

When writing it I was thinking about distributed data parallel, and it could work, but I have not been able to test that.

Currently the code assumes that the the tokenizer comes from tiktoken and that the streamed dataset is not tokenized.

Am open to closing the PR if there is an easier solution; that decision I would leave to you.

[lennart-finke]

After looking into this a bit more, there is also a builtin in Huggingface for this, the function split_dataset_by_node . If that makes sense, next I'll adapt the file you mentioned to use this.