Contamination analysis for MMLU, Hellaswag, and ARC_c

[liyucheng09]

This PR is to extend contamination analysis to MMLU, Hellaswag, and ARC_c.

Now OpenCompass should be able to conduct contamination analysis on CEval, MMLU, Hellaswag, and ARC_c.

In Contamination Detector, actually Winogrande and CommonsenseQA are also supported, but since they have very little comtaminated samples, I don't think it's necessary to add them in OpenCompass.

See the example results below:

dataset     version    mode    baichuan2-7b-base-hf    -                              -                                        qwen-7b-hf        -                              -                                        llama_30b_autogptq    -                              -
----------  ---------  ------  ----------------------  -----------------------------  ---------------------------------------  ----------------  -----------------------------  ---------------------------------------  --------------------  -----------------------------  ---------------------------------------
-           -          -       accuracy - clean        accuracy - input contaminated  accuracy - input-and-label contaminated  accuracy - clean  accuracy - input contaminated  accuracy - input-and-label contaminated  accuracy - clean      accuracy - input contaminated  accuracy - input-and-label contaminated
mmlu        -          ppl     56.76                   44.69                          54.93                                    58.74             48.67                          58.28                                    57.46                 45.72                          57.16
hellaswag   47bff9     ppl     66.87                   57.14                          70.97                                    86.42             89.29                          90.88                                    76.71                 57.14                          82.37

Note that, for ARC_c the contamination detection was done on the test set, but the default test split of ARC_c in OpenCompass is dev. So, it may require to rerun ARC_c on test to finish the data contamination analysis.

[liyucheng09]

And btw I am submitting a paper about data contamination to NAACL where the deadline is 15 Dec.

I have finished experiments on all Llama models (see table below), but the paper would be stronger to have analysis of more LLMs.

For example, we could add Multilingual LLMs (Qwen, Yi, baichuan) on CEval or more recent model (Mistral etc.).

If you could provide some sort of data, it would really helpful. I would love to add you as co-authors.

Let me know what do you think. @tonysy @Leymore

[liyucheng09]

Found a bug in contamination annotation from the Contamination Detector side. Fixed.

but if you have run contamination analysis for MMLU, please clean the cache file data/mmlu/test/MMLU_test_contamination_annotations.json and try again.

[liyucheng09]

@Leymore Hi, how was your holiday? DO we have any updates on this pr?

[Leymore]

LGTM