[Task Submission] Shifted Pauq (shifted_pauq)

[ai-spiderweb]

Shifted Pauq

Custom splits(iid and target query length) for English and Russian versions of text-to-SQL tasks for measuring compositional generalization, based on original Spider text-to-SQL dataset.

Authors

• Oleg Somov [email protected]
• Ekaterina Dmitrieva [email protected]
• Elena Tutubalina [email protected]

Implementation

Right now we do not make custom implementations in task files.

Usage

We only modified config.jsonnet files, therefore our evaluation should be run in the default way.

Checklist:

• I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• I have submitted or will submit an accompanying paper to the GenBench workshop.

[vernadankers]

Hello!

We are getting quite close to the deadline (September 1, 11:59PM anywhere on earth), so please make any final changes to your PR if required,
and don't forget to submit your accompanying paper to Openreview via https://openreview.net/group?id=GenBench.org/2023/Workshop by September 1.

Good luck finalising your PR and paper, feel free to tag us if you have questions.
Cheers, Verna
On behalf of the GenBench team

[ai-spiderweb]

Hello!

I've updated the PR with splits used in the Camera Ready paper.