[Task Submission] Frequency based mathematics (frequency_based_mathematics)

[dieuwkehupkes]

Frequency based mathematics (example task)

This task quantifies generalisation by comparing accuracies with pretraining term frequencies.

Authors

• Dieuwke Hupkes [email protected]

Implementation

This task reimplements the evaluation function.

Usage

Copied from the docs, this is an example usage:

# Load the task
task = load_task("frequency_based_mathematics")
ds = task.get_prepared_datasets(
    PreparationStrategy.PROMPT_BASED_TESTING,
    shot_list=[0])[0]

# Load your pretraining frequencies and model predictions
pretraining_freqs = ... 
preds = ... 

for pred_type, preds in preds.items():
    for freq_type, pretraining_freq in pretraining_freqs.items():
        scores = task.evaluate_predictions(
            predictions=preds,
            gold=ds,
            term_freqs=pretraining_freq
        )
 
print(f'Scores: {scores}')

Checklist:

• I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• I have submitted or will submit an accompanying paper to the GenBench workshop.