Skip to content

Commit

Permalink
updated asdiv readme to include informaiton about new asdiv_cot_llama…
Browse files Browse the repository at this point in the history
… task
  • Loading branch information
Cameron7195 committed Aug 22, 2024
1 parent f712591 commit ff2141b
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions lm_eval/tasks/asdiv/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ Homepage: https://github.com/chaochun/nlu-asdiv-dataset
#### Tasks

* `asdiv`
* `asdiv-cot-llama`: ASDIV with prompt formatting modified to conform to the evaluation settings described by Meta here: https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals/viewer/Meta-Llama-3.1-8B-Instruct-evals__gsm8k__details?row=0
- Note that the CoT prompt from (https://arxiv.org/pdf/2201.11903) is used exactly as in GSM8k-CoT
- This file is setup to run identically to the task `gsm8k-cot-llama` but for asdiv.
- Use this task with --fewshot_as_multiturn and --apply_chat_template to run correctly with Llama Instruct models.


### Checklist

Expand Down

0 comments on commit ff2141b

Please sign in to comment.