1.5.2

1.5.2 (2024-04-04)

Fix

• fix: Minor fixes to metadata (#315)

• Update MindSmallReranking.py

• fix: Updated wrong metadata (e0eddf9)

Unknown

• Adding French team contribution points (#302)

• Update points.md

• Update docs/mmteb/points.md

• Update points.md

• Update points.md (23c9fdd)

1.5.1

1.5.1 (2024-04-03)

Fix

• fix: Added tests for checking datasets (#307)

• fix: Fixed hf_hub_name for WikiCitiesClustering

• Added points for this PR and a 3 other minor dataset fixes

• feat: Added tests which validated that datasets are available

• fix: Updated hf references and revisions to multiple datasets

• Added points for submission

• fix: Added suggestions from the review

• Apply suggestions from code review

Co-authored-by: Imene Kerboua <[email protected]>

• fix: sped up async test for whether datasets exist

• fix: Updated revisions

• fix: reuploaded scandeval datasets

• fix: Applied formatter

---

Co-authored-by: Imene Kerboua <[email protected]> (8d804f4)

1.5.0

1.5.0 (2024-04-02)

Feature

• feat: Allow extending the load_dataset parameters in custom tasks inheriting AbsTask (#299)

• Allow extending the load_dataset parameters

• format

• Fix test

• remove duplicated logic from AbsTask, now handled in the metadata

• add tests

• remove comments, moved to PR

• format

• extend metadata dict from super class

• Remove additional load_data

• test: adding very high level test

• Remove hf_hub_name and add test

• Fix revision in output file

---

Co-authored-by: gbmarc1 <[email protected]> (953780d)

1.4.1

1.4.1 (2024-04-01)

Fix

• fix: hf_hub_name for WikiCitiesClustering (#305)

• fix: Fixed hf_hub_name for WikiCitiesClustering

• Added points for this PR and a 3 other minor dataset fixes (b447235)

1.4.0

1.4.0 (2024-04-01)

Feature

• feat: Added windows support by replacing pytrec-eval with pytrec-eval-terrier (#292)

• ci: Added windows to test suite

• feat: Changed to pytrec-eval-terrier to add support for windows installs (fc0e105)

1.3.4

1.3.4 (2024-04-01)

Fix

• fix: Update MindSmallReranking.py to have the correct hf reference (#303) (102e24e)

1.3.3

1.3.3 (2024-03-31)

Documentation

• docs: Added information related to the automatic release (#290)

• docs: added information related to the automatic release

• docs: removed test-parallel from docs

• docs: minor additions to contributing guidelines

• ci: removed changelog

As it already present in the git releases

• Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <[email protected]>

---

Co-authored-by: Niklas Muennighoff <[email protected]> (6821d23)

Fix

• fix: fixed bug introduced in TatoebaBitextMining causing it to use a different dataset (#297) (d0549a3)

• fix: Fixed mispecified rev. id for datasets (#298)

• fix: fixed wrong rev. id for ToxicConversationsClassification

• fix: fixed wrong rev. id with RedditClusteringP2P (e1ae0d3)

1.3.2

1.3.2 (2024-03-29)

Documentation

• docs: Update links in README.md (#296) (76056b5)

Fix

• fix: Added tasks from SEB (#287)

• Added tasks from SEB

• docs: fix link

• fix: ran linting

• fix typing for 3.8

• fixed annotation for v3.8 (39cff49)

1.3.1

1.3.1 (2024-03-26)

Fix

• fix: updated version in transition to semantic release ci (238ab82)

v0.10.0

v0.10.0 (2024-03-26)

Ci

• ci: renamed test job and workflow (#282)

ci: Added tests (6675bb8)

Documentation

• docs: typos in readme (#268) (aa9234c)

• docs: add dataset schemas (#255)

• docs: update AbsTaskClassification.py document schema for classification task

• update AbsTaskBitextMining.py

• update BornholmskBitextMining.py

• update AbsTaskClustering.py and BlurbsClusteringP2P.py

• update 8 files

• update 9 files

• update AbsTaskReranking.py

• update BlurbsClusteringP2P.py

• update CMTEBPairClassification.py

• update GerDaLIRRetrieval.py

• update 7 files

• update AbsTaskBitextMining.py

• update AbsTaskClassification.py (c3ce1ac)

• docs: Add development installation instructions (#246)

• docs: Add development installation instructions

• removed unused requirements file

I don't believe this is nec. with the setup.py specifying the same dependencies

• docs: Updated make file with new dependencies

• ci: Update ci to use make commands

This ensure that the user runs exactly what the CI expects

• ci: Avoid specifying tests folder as it causes issuew ith tests

• ci: removed unec. args for test ci

• Added dev install (0048878)

Feature

• feat: update revision id of wikicitiesclustering task (fb90c02)

Fix

• fix: dead link in readme (ecbb776)

• fix: Added sizes to the metadata (#276)

• restructing the readme

• added mmteb

• removed unec. method

• Added docstring to metadata

• Updated outdated examples

• formatting documents

• fix: Updated form to be parsed correctly

• fix: Added sizes to the metadata

this allow for automatic metadata generations

• Updated based on feedback

• Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <[email protected]>

• updated based on feedback

• Added suggestion from review

• added correction based on review

• reformatted empty fields to None

---

Co-authored-by: Niklas Muennighoff <[email protected]> (cd4a012)

• fix: remove debugging print statement (d292d93)

• fix: pass parallel_retrieval kwarg to use DenseRetrievalParallelExactSearch (19b8f66)

• fix: msmarco-v2 uses dev.tsv, not dev1.tsv (6908d21)

• fix: add missing task-langs attribute (#152) (bc22909)

Refactor

• refactor: add metadata basemodel (#260)

• refactor: rename description to metadata dict

• refactor: add TaskMetadata and first example

• update 9 files

• update TaskMetadata.py

• update TaskMetadata.py

• update TaskMetadata.py

• update LICENSE, TaskMetadata.py and requirements.dev.txt

• update 151 files

• update 150 files

• update 43 files and delete 1 file

• update 106 files

• update 45 files

• update 6 files

• update 14 files

• Added model results to repo and updated CLI to create consistent folder structure. (#254)

• Added model results to repo and updated CLI to create consistent folder structure.

• ci: updated ci to use make install

• Added missing pytest dependencies

• Update README.md

Co-authored-by: Niklas Muennighoff <[email protected]>

---

Co-authored-by: Niklas Muennighoff <[email protected]>

• Restructing the readme (#262)

• restructing the readme

• removed double specification of versions and moved all setup to pyproject.toml

• correctly use flat-layout for the package

• build(deps): update TaskMetadata.py and pyproject.toml

• update 221 files

• build(deps): update pyproject.toml

• build(deps): update pyproject.toml

• build(deps): update pyproject.toml

---

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Niklas Muennighoff <[email protected]> (dd5d617)

Unknown

• Ci-fix (#289)

• added release pipeline

• v1.3.0

• ci: moved release to the correct folder (7f56c1a)

• v1.3.0

• added release pipeline

• v1.3.0 (5e4d10e)

• tests: speed up tests (#283)

update Makefile and test_all_abstasks.py (2155bf6)

• update TaskMetadata.py (#281) (acfd7d4)

• Merge branch 'main' of https://github.com/embeddings-benchmark/mteb (c9d1a03)

• Enable ruff ci (#279)

• restructing the readme

• added mmteb

• removed unec. method

• Added docstring to metadata

• Updated outdated examples

• formatting documents

• fix: Updated form to be parsed correctly

• fix: Added sizes to the metadata

this allow for automatic metadata generations

• Updated based on feedback

• Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <[email protected]>

• updated based on feedback

• Added suggestion from review

• added correction based on review

• reformatted empty fields to None

• CI: Enable linter

---

Co-authored-by: Niklas Muennighoff <[email protected]> (a16eb07)

• Added MMTEB (#275)

• restructing the readme

• added mmteb

• removed unec. method

• Added docstring to metadata

• Updated outdated examples

• formatting documents

• fix: Updated form to be parsed correctly

• Updated based on feedback

• Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <[email protected]>

• updated based on feedback

• Added suggestion from review

• added correction based on review

---

Co-authored-by: Niklas Muennighoff <[email protected]> (c0dc49a)

• dev: add ruff as suggested extension (#274) (b08913f)

• dev: add isort (#271)

• dev: add isort

• dev: add isort (845099d)

• dev: run tests on pull request towards any branch (13f759a)

• Merge branch 'main' of https://github.com/embeddings-benchmark/mteb (b42abe4)

• replaced linter with ruff (#265)

• restructing the readme

• removed double specification of versions and moved all setup to pyproject.toml

• correctly use flat-layout for the package

• replaced linter with ruff

• rerun tests

• ci: Added in newer workflow

some of them are disables as they require other issues to be solved

• Update Makefile

Co-authored-by: Niklas Muennighoff <[email protected]>

---

Co-authored-by: Niklas Muennighoff <[email protected]> (023e881)

• Restructing the readme (#262)

• restructing the readme

• removed double specification of versions and moved all setup to pyproject.toml

• correctly use flat-layout for the package (769157b)

• restructing the readme (364be7f)

• Added model results to repo and updated CLI to create consistent folder structure. (#254)

• Added model results to repo and updated CLI to create consistent folder structure.

• ci: updated ci to use make install

• Added missing pytest dependencies

• Update README.md

Co-authored-by: Niklas Muennighoff <[email protected]>

---

Co-authored-by: Niklas Muennighoff <[email protected]> (8a758bc)

• dev: add workspace defaults in VSCode (#253)

• dev: add black as default formatter in vscode

• Update .vscode/settings.json

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (30e5b9e)

• Add Danish Discourse dataset (#247)

• misc.

• update dd...