Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NeedleInAHaystack Test Support #714

Merged
merged 7 commits into from
Dec 23, 2023
Merged

Add NeedleInAHaystack Test Support #714

merged 7 commits into from
Dec 23, 2023

Conversation

Mor-Li
Copy link
Collaborator

@Mor-Li Mor-Li commented Dec 19, 2023

No description provided.

@tonysy
Copy link
Collaborator

tonysy commented Dec 19, 2023

Please move the visualize and preprocess scripts into tools.

Copy link
Collaborator

@tonysy tonysy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As inline comments

configs/eval_hf_internlm_chat_20b_cdme.py Outdated Show resolved Hide resolved
@Mor-Li
Copy link
Collaborator Author

Mor-Li commented Dec 20, 2023

@tonysy I have made the changes as per your request and moved the scripts to the tools directory. Please review. Thank you!

Copy link
Collaborator

@tonysy tonysy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# This creates a list of different context lengths (in tokens),
# ranging from 1000 to 8000, incremented by 1000.
# Each length represents a different test scenario for the dataset.
context_lengths = list(range(1000, 9000, 1000))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needle, and retrieval_question should be set as args

@tonysy tonysy merged commit 0e24f42 into open-compass:main Dec 23, 2023
7 checks passed
Leymore pushed a commit that referenced this pull request Jan 8, 2024
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <[email protected]>

* add needle in haystack test

* update needle in haystack test

---------

Co-authored-by: Songyang Zhang <[email protected]>
liuyaox pushed a commit to liuyaox/opencompass that referenced this pull request Jun 26, 2024
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <[email protected]>

* add needle in haystack test

* update needle in haystack test

---------

Co-authored-by: Songyang Zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants