[Task Submission] Quantifier Understanding (quantifier_understanding)

[lerow]

Quantifier Understanding

The task evaluates generalization in the understanding of quantifiers. It aims to measure
how well can language models capture the semantics of logical quantifiers in natural language.

Authors

• Leroy Wang [email protected]
• Shane Steinert-Threlkeld [email protected]

Implementation

The task re-implements the evaluation function to compute accuracy scores.

Usage

Given predictions and gold labels, evaluate_predictions() outputs the accuracy score.

Checklist

• I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• I have submitted or will submit an accompanying paper to the GenBench workshop.