New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Huge datasets #233

Open

atl333 opened this issue Dec 30, 2021 · 0 comments

atl333 commented Dec 30, 2021

I have 100k+ message datasets from discord id like to use, is there a specific format i could use for it?

for example im using a script and could export them as a two column csv with name and message

or a txt with name: message

or just
1 message
2 message
1 message
3 message

etc

would it work? or would it think it is all a huge conversation and not learn properly?

i could maybe make it so if it has been more than 1-2 hours between messages the conversation ended and i format it as a new convo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment