1.19.4

1.19.4 (2024-11-11)

Fix

• fix: Add missing benchmarks in benchmarks.py (#1431)

Fixes #1423 (a240ea0)

• fix: Add Korean AutoRAGRetrieval (#1388)

• feat: add AutoRAG Korean embedding retrieval benchmark

• fix: run --- 🧹 Running linters ---
  ruff format . # running ruff formatting
  716 files left unchanged
  ruff check . --fix # running ruff linting
  All checks passed!

• fix: add metadata for AutoRAGRetrieval

• change link for markers_bm

• add AutoRAGRetrieval to init.py and update metadata

• add precise metadata

• update metadata: description and license

• delete descriptive_stats in AutoRAGRetrieval.py and run calculate_matadata_metrics.py (f79d9ba)

• fix: make samples_per_label a task attribute (#1419)

make samples_per_label a task attr (7f1a1d3)

Unknown

• Update tasks table (d069aba)

1.19.3

1.19.3 (2024-11-11)

Documentation

• docs: Fix a typo in README (#1430)

Fix typo in readme (9681eb3)

• docs: Update recommendation for pushing results (#1401)

fix: Update recommendation for pushing results (fccf034)

Fix

• fix: add logging for RetrievalEvaluator NaN values for similarity scores (#1398)

Fixes #1389 (cc7a106)

1.19.2

1.19.2 (2024-11-07)

Fix

• fix: Added the necessary trust_remote_code (#1406) (fd8b283)

1.19.1

1.19.1 (2024-11-07)

Fix

• fix: Add the_ugly_duckling.txt for speedtask to Python wheel (#1402)

Add the_ugly_duckling.txt for speedtask to Python wheel (b1a0ec6)

1.19.0

1.19.0 (2024-11-06)

Feature

• feat: Standardize descriptive stats (#1375)

• init

• mock tests

• remove debug prints

• remove descriptive stats and move to n_samples

• fix typo

• fix create task table

• fix citations

• remove n_samples

• metadata per task

• add test

• reformat task table

• add n_samples property

• fix tests

• rename total_symbols to number_of_characters

• Update mteb/abstasks/AbsTaskRetrieval.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• rename in tests

• lint

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (2854fa2)

Unknown

• Update tasks table (537b974)

1.18.9

1.18.9 (2024-11-06)

Fix

• fix: Disable rich output with verbosity=0 on evaluation.run (#1395)

• feat: verbose=0 now supress rich console output

• fix: removed unnecesary list comprenhension (1bb1ca3)

1.18.8

1.18.8 (2024-11-04)

Fix

• fix: Update logging verbosity levels in MTEB (#1384)

• Fix verbosity handling in MTEB.py for consistent logging

• updates

• update docstrings

• linting code (35daf58)

1.18.7

1.18.7 (2024-11-04)

Fix

• fix: Leaderboard UI improvements (#1370)

• Added elementary search bar with RegEx functionality

• Improved layout

• Table improvements, adjusted column width and added links

• Update benchmark filter description

Co-authored-by: Isaac Chung <[email protected]>

• Added instructions on multiple search queries

Co-authored-by: Isaac Chung <[email protected]>

---

Co-authored-by: Isaac Chung <[email protected]> (92fe9cb)

1.18.6

1.18.6 (2024-10-31)

Fix

• fix: Integrate prompts to task metadata (#1300)

• init

• add DatasetDict

• add classification

• add clustering

• add pair classification

• add retrieval

• add all prompts

• start integrating prompts

• refactor instruct models

• lint

• fix test

• fix

• fix no prompt in prompt dict

• add more logging

• add more logging

• Apply suggestions from code review

Co-authored-by: Isaac Chung <[email protected]>

• fix code review

• fix use_instructions

• add log if instruction template not set

• fix metadata

• lint

• fix brazilian

• remove MetadataDatasetDict

• rollback test metadata

---

Co-authored-by: Isaac Chung <[email protected]> (029d378)

1.18.5

1.18.5 (2024-10-31)

Fix

• fix: Speed up leaderboard by caching and skipping validation (#1365)

• Made loading and filtering faster by removing unnecessary validation

• Made select_tasks faster by removing validation

• Added caching to leaderboard

• Ran linting

• Added missing future import (f1bc375)