-
Notifications
You must be signed in to change notification settings - Fork 435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add NeedleInAHaystack Test Support #714
Conversation
Please move the visualize and preprocess scripts into tools. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As inline comments
Co-authored-by: Songyang Zhang <[email protected]>
@tonysy I have made the changes as per your request and moved the scripts to the tools directory. Please review. Thank you! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
tools/gen_needleinahaystack.py
Outdated
# This creates a list of different context lengths (in tokens), | ||
# ranging from 1000 to 8000, incremented by 1000. | ||
# Each length represents a different test scenario for the dataset. | ||
context_lengths = list(range(1000, 9000, 1000)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needle, and retrieval_question should be set as args
* Add NeedleInAHaystack Test * Apply pre-commit formatting * Update configs/eval_hf_internlm_chat_20b_cdme.py Co-authored-by: Songyang Zhang <[email protected]> * add needle in haystack test * update needle in haystack test --------- Co-authored-by: Songyang Zhang <[email protected]>
* Add NeedleInAHaystack Test * Apply pre-commit formatting * Update configs/eval_hf_internlm_chat_20b_cdme.py Co-authored-by: Songyang Zhang <[email protected]> * add needle in haystack test * update needle in haystack test --------- Co-authored-by: Songyang Zhang <[email protected]>
No description provided.