1.18.5

1.18.5 (2024-10-31)

Fix

• fix: Speed up leaderboard by caching and skipping validation (#1365)

• Made loading and filtering faster by removing unnecessary validation

• Made select_tasks faster by removing validation

• Added caching to leaderboard

• Ran linting

• Added missing future import (f1bc375)

1.18.4

1.18.4 (2024-10-30)

Fix

• fix: make sure test is the default split for FEVER (#1361)

The other splits can still be run as long as they are specified. (d9626ab)

1.18.3

1.18.3 (2024-10-30)

Fix

• fix: Update KorSarcasm to avoid trust-remote code (#1364) (756ba7e)

Unknown

• Leaderboard updates: Model meta + task and benchmark info (#1345)

• Added benchmark description and citation to leaderboard

• Added model information to main table

• Fixed citation box

• Added table tab with task information

• Added button for benchmark link if specified

• Formatted model column in per_task table properly

• Implemented model filtering based on metadata

• Fixed maximum minimum model sizes

• Ran linting

• Replaced mean rank with borda rank in main table (298b0bd)

1.18.2

1.18.2 (2024-10-30)

Fix

• fix: upload BrazilianToxicTweetsClassification to hf (#1352)

upload to hf (9c7a1c2)

1.18.1

1.18.1 (2024-10-30)

Fix

• fix: Add jina, uae, stella models (#1319)

• add models

• fix

• fix

• fix prompt

• Update mteb/models/jina_models.py

Co-authored-by: Wang Bo <[email protected]>

• Update mteb/models/jina_models.py

Co-authored-by: Wang Bo <[email protected]>

• try reeval stella

• change to e5

• change to e5

• add metadata

• update languages

• Update mteb/models/jina_models.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• remove docstring

• remove trust remote

• update model meta

• Set minimal version

---

Co-authored-by: Wang Bo <[email protected]>
Co-authored-by: Kenneth Enevoldsen <[email protected]> (0b846ff)

• fix: remove accidentally commited file (16a333e)

1.18.0

1.18.0 (2024-10-28)

Feature

• feat: update English benchmarks and mark MMTEB benchmarks as beta (#1341)

• feat: update English benchmarks and mark MMTEB benchmarks as beta

• Added summEvalv2

• Update docs with new MTEB_EN_MAIN rename (61371dd)

1.17.0

1.17.0 (2024-10-26)

Feature

• feat: Update metadata for all models (#1316)

• Added model meta

• format

• fixed metadata

• Metadata update for voyage models

• Update mteb/models/cohere_models.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/models/cohere_models.py

Co-authored-by: Roman Solomatin <[email protected]>

• Added corrections from review

• fix spelling error

---

Co-authored-by: Roman Solomatin <[email protected]> (f8fed9b)

Unknown

• WIP: Leaderboard UI improvements (#1320)

• Fixed typos in task_results

• Fixed typos in task_results

• Added Tailwind, reorganized layout and fixed scrolling

• Ran linting

• Removed faux benchmark

• Updated layout

• Changed table number format

• Table highlights highest values by making them bold

• Added rank to table, removed organization from model_name

• Added mean rank to table

• Ran linting (5af36c5)

• Cache the embeddings when requested (#1307)

• add caching

• update test to use close

• change from json to pkl

• fix for window

• cleanup on Windows again

• infer dimension

• move cachewrapper

• add wrapper

• fix

• updates

• fix tests

• fix lint

• lint

• add test (650e8b8)

• Update tasks table (4a04042)

• Add multilingual mFollowIR dataset (#1308)

• add mFollowIR

• paper name

• edit warning->info

• convert to parquet

• lint (b580b95)

1.16.5

1.16.5 (2024-10-25)

Fix

• fix: Add implementations of common reranker models (#1309)

• init

• revert

• revert

• add metadata

• lint

• add reqs

• change to float16

• benchmark lint fix (f5f90d3)

1.16.4

1.16.4 (2024-10-25)

Fix

• fix: Re-upload dataset to hub to avoid using script upload (#1322)

• fix dataset upload

• add linting (f00a262)

Unknown

• Update tasks table (e5b6c12)

1.16.3

1.16.3 (2024-10-24)

Fix

• fix: remove duplicate multilingual (2f14519)