[Task Submission] Natural Language Codesearch Classification (nl_codesearch_clf )

[drndr]

[Natural Language Codesearch Classification]

The task consists of 8 subtasks and measures cross-lingual and domain generalization, and robustness to covariate shift
Includes a binary classification evaluation:
Given a natural language query, determine if a given code snippet is matches the natural language description or not.

Authors

• Andor Diera [email protected]
• Abdelhalim Dahou [email protected]
• Lukas Galke [email protected]
• Fabian Karl [email protected]
• Florian Sihler [email protected]
• Ansgar Scherp [email protected]

Implementation

For the binary classification setup the config files were used with task_type "multiple_choice"
The task.py script includes a custom get_dataset_raw method, where the negativ samples are created.

Usage

For binary classification the default "multiple_choice" usage.

Checklist:

• I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• I have submitted or will submit an accompanying paper to the GenBench workshop.

[drndr]

test_task failed due to non existent id
our submission includes two tasks nl_codesearch_clf and nl_codesearch_mrr

[vernadankers]

Hi, that sounds like something that can be fixed on our side! Could you open an issue for that, please? :-)

[drndr]

Thanks! i was also considering submitting two different branches, but if it can be fixed on your side even better :)

[kazemnejad]

Hi, the problem was that the automated CI was trying to read the task_id (which can be the parent task, but, your initial title contained nl_codesearch. nl_codesearch didn't exist as a task_id. I changed the title to contain nl_codesearch_clf and it passed all the test cases.

You are submitting two parent tasks nl_codesearch_clf and nl_codesearch_mrr. Currently, we only support one per task-submission. Please, open another PR for the second task.

[drndr]

Thanks! I removed the mrr part from this clf PR, and created a seperate one for mrr

[vernadankers]

Hello!

We are getting quite close to the deadline (September 1, 11:59PM anywhere on earth), so if your PR needs any final changes, please make them now,
and don't forget to submit your accompanying paper to Openreview via https://openreview.net/group?id=GenBench.org/2023/Workshop by September 1.

Good luck finalising your PR and paper, feel free to tag us if you have questions.
Cheers, Verna
On behalf of the GenBench team

[drndr]

Hi,

Sorry there were some style issues introduced in the last commit at submission and forgot the rerun the make commands.
Fixed these issues in the latest commit.

[kazemnejad]

@drndr We're in the process of merging the tasks into the repo.

Could you please include a single file usage_example.py of each task where you use each task. It seems your tasks supports both finetuning/in-context-learning. It'd be nice to have examples of both usages. (Preferably a pretrained huggingface model).

Please also include requirements-usage-example.txt for the python dependencies needed to be installed for running the example.

[kazemnejad]

This looks great. Just a minor comment. Could you replace the task loading using the official method? for here and the rest of this file?

from genbench import load_task
task = load_task("nl_codesearch_clf:codesearchnet_adv")

[kazemnejad]

Manually merged in
https://github.com/GenBench/genbench_cbt/pull/41