Add KoCommonGEN v2 benchmark #2208
Conversation
metterian
requested review from
haileyschoelkopf and
lintangsutawika
as code owners
|
|
| # "choices": [f"{doc[str(i+1)]}" for i in range(4)], | ||
| # "choices": [f'{str(i+1)}. ' + doc['{i}'.format(i=i + 1)] for i in range(4)], # The list of choices. | ||
| # "choices": [str(i+1) for i in range(4)], # The list of choices. |
There was a problem hiding this comment.
Just want to check if this is an alternative option (which is why it's commented but left in)?
There was a problem hiding this comment.
+1, if this comment is safe to delete let's do so!
| @@ -0,0 +1,19 @@ | |||
| task: ko_commongen_v2 | |||
There was a problem hiding this comment.
| task: ko_commongen_v2 |
There was a problem hiding this comment.
let's remove the task: field since this is a template/stub config
| # "choices": [f"{doc[str(i+1)]}" for i in range(4)], | ||
| # "choices": [f'{str(i+1)}. ' + doc['{i}'.format(i=i + 1)] for i in range(4)], # The list of choices. | ||
| # "choices": [str(i+1) for i in range(4)], # The list of choices. |
There was a problem hiding this comment.
+1, if this comment is safe to delete let's do so!
| out_doc = { | ||
| "query": query, | ||
| "choices": [f"{i+1}. {doc[str(i+1)]}" for i in range(4)], | ||
| # "choices": [f"{doc[str(i+1)]}" for i in range(4)], |
|
Hi @metterian , just following up to see if you'd be able to make these final few changes so we can merge this task! If not we'll try to get to them ourselves. Note also that we'd ideally have an entry in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/README.md describing the task as well, so users know about your task! |
Description:
This PR adds support for the KoCommonGEN v2 benchmark, a new dataset for evaluating Korean commonsense reasoning in large language models.
Changes:
KoCommonGEN v2 Details:
This benchmark provides a valuable resource for evaluating Korean language models on commonsense reasoning tasks. Adding it to our evaluation suite will help broaden our coverage of multilingual NLP capabilities.
Please review and let me know if any changes or additional information is needed.