Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Add dingo test #1529

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

[Feature] Add dingo test #1529

wants to merge 13 commits into from

Conversation

shijinpjlab
Copy link

@shijinpjlab shijinpjlab commented Sep 14, 2024 •
edited
Loading

Motivation

Add the test ability for data produced by llm.

Modification

Add eval_dingo.py in configs and ding.py in datasets.

Use cases (Optional)

model config:

image

result

image

dingo_benc.zip

eval_dingo_20240914_135811.zip

dingo_result.zip

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

e06084
e06084 reviewed Sep 14, 2024
View reviewed changes
configs/eval_dingo.py
@@ -0,0 +1,41 @@
from mmengine.config import read_base
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充一下新增PR功能说明，和测试记录吧

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

title也修改成[feature] Add xxx 这样的格式吧

e06084
e06084 reviewed Sep 14, 2024
View reviewed changes
opencompass/datasets/dingo.py Outdated
def score(self, origin_prompt: List, predictions: List) -> dict:
current_time = time.strftime('%Y%m%d_%H%M%S', time.localtime())
file_data = [{'prompt':pmt, 'prediction':prd} for pmt, prd in zip(origin_prompt, predictions)]
file_name = 'dingo_file_' + current_time + '.jsonl'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个要看看有没有更好的实现方式

@shijinpjlab shijinpjlab changed the title add dingo test [feature] Add dingo test Sep 14, 2024
e06084
e06084 reviewed Sep 14, 2024
View reviewed changes
opencompass/datasets/dingo.py
input_data = {
"eval_models": ["llm_base"],
"input_path": file_name,
"output_path": "./outputs/dingo/",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个看看有没有全局的环境变量

@tonysy
Copy link
Collaborator

tonysy commented Sep 14, 2024

Also please fix the lint issue with pre-commit hook

@tonysy tonysy changed the title [feature] Add dingo test [Feature] Add dingo test Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonysy tonysy tonysy left review comments

@e06084 e06084 e06084 left review comments

@MaiziXiao MaiziXiao MaiziXiao left review comments

@bittersweet1999 bittersweet1999 Awaiting requested review from bittersweet1999

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@shijinpjlab @tonysy @e06084 @MaiziXiao @shijinPJ