Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiline prompts #1279

Merged
merged 14 commits into from
May 10, 2024
Merged

Allow multiline prompts #1279

merged 14 commits into from
May 10, 2024

Conversation

rasbt
Copy link
Collaborator

@rasbt rasbt commented Apr 12, 2024

This workaround let's users submit multiline prompts. Either via copy & paste or by typing on the command line and using Enter once to start a new line. If the newline is blank and a user hits Enter again, it submits the prompt.

If a user only wants to submit a single line prompt, pressing Enter twice might be extra effort, but I think it's worth it given that users can now use multiline prompts without "bugs".

We can also explore other options based on external libraries some time later but I think it's kind of useful to fix this so users can use CodeGemma etc.

Fixes #1275

litgpt/chat/base.py Outdated Show resolved Hide resolved
@rasbt
Copy link
Collaborator Author

rasbt commented May 3, 2024

I was spending way too much time on this, but yeah, there is no nice way to do that. For some reason it's not easy to reliably log CTRL+Enter info. You could use the keyboard library but that might be too intrusive because it monitors key board inputs and I think may not work on systems with limited user rights.

Maybe the best solution is to just have a special keyword to stop parsing.

The way I updated it is that if you enter :submit: on a new line, it will submit the prompt. From using this a bit, it is cumbersome to type a special word instead of just pressing enter or CTRL+Enter. n

Not sure how to move forward with this. What do you think here @carmocca and @awaelchli , should we just leave it as is, use :submit: (or something similar), or perhaps something else?

@awaelchli
Copy link
Contributor

I'm good with whatever the solution if it does not impact the single-line user experience.

However, keep in mind that if you add special syntax for the multi-line input, it won't solve your original issue #1275 where you copy-pasted multi-line into the terminal.

@rasbt
Copy link
Collaborator Author

rasbt commented May 7, 2024

Ok, thinking about this more, I think there is no one-size-fits-all approach.

So, I changed it back so that by default nothing changes.

But you can now additionally pass the --multiline_prompts arg:

litgpt chat \
   --checkpoint_dir checkpoints/microsoft/phi-2 \
   --multiline_prompts True

to work with multiline prompts. This will then require users to enter !submit to submit. (I found that :submit: is just too annoying to type on US keyboards.)

This should also address #1275 based on trying it.

The code should be ready for review @carmocca and @awaelchli .

Copy link
Contributor

@carmocca carmocca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tests are failing

litgpt/chat/base.py Outdated Show resolved Hide resolved
litgpt/chat/base.py Outdated Show resolved Hide resolved
@rasbt
Copy link
Collaborator Author

rasbt commented May 8, 2024

All the feedback should be addressed now. Thanks @carmocca @awaelchli for keeping the bar high!

The default usage is the same as before, except that you can now exit via !exit and !quit in addition to 'Enter' on a blank line:

⚡ multiline ~/litgpt litgpt chat --checkpoint_dir checkpoints/microsoft/phi-2
Now chatting with phi-2.
To exit, press 'Enter' on an empty prompt.

Seed set to 1234
>> Prompt: what do llamas eat/
>> Reply:  Llamas eat grass, shrubs, and leaves.

Time for inference: 0.74 sec total, 17.47 tokens/sec, 13 tokens

>> Prompt: !exit
⚡ multiline ~/litgpt 

And with --multiline true, you exit via !exit and !submit as well:

>> Prompt: (Type '!submit' on a new line to end input).
!exit
⚡ multiline ~/litgpt litgpt chat --checkpoint_dir checkpoints/microsoft/phi-2 --multiline true
Now chatting with phi-2.
To exit, enter '!quit' or '!exit' on an empty prompt and press 'Enter'.

Seed set to 1234
>> Prompt: (Type '!submit' on a new line to end input).
Hi
I have a question:
What do Llamas eat?
!submit
>> Reply:  llamas eat grass and hay.

Time for inference: 0.75 sec total, 10.64 tokens/sec, 8 tokens

>> Prompt: (Type '!submit' on a new line to end input).
!exit

litgpt/chat/base.py Outdated Show resolved Hide resolved
@rasbt rasbt enabled auto-merge (squash) May 10, 2024 12:36
@rasbt rasbt merged commit d317902 into main May 10, 2024
9 checks passed
@rasbt rasbt deleted the multiline branch May 10, 2024 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LitGPT chat terminates weirdly
4 participants