Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat] Update mine_hard_negatives to using a full corpus and multiple positives #2848

Merged
merged 25 commits into from
Sep 11, 2024
Merged

[feat] Update mine_hard_negatives to using a full corpus and multiple positives #2848

merged 25 commits into from
Sep 11, 2024
+159 −60

Commits on Jul 9, 2024

  1. updated mine_hard_negatives method to include a seperate corpus for m… 

    …ining hard negatives.
    Christian Geishauser authored and Christian Geishauser committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8e5927c View commit details
    Browse the repository at this point in the history

  2. Run 'make check'

    @tomaarsen
    tomaarsen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    bcac963 View commit details
    Browse the repository at this point in the history

  3. Update "corpus" to just a list of strings

    @tomaarsen
    tomaarsen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    e63938f View commit details
    Browse the repository at this point in the history

  4. Prevent duplicate embeddings if no separate corpus

    @tomaarsen
    tomaarsen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    0736190 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Deduplicate corpus 

    Add a positive to corpus indices mapping, useful to get non-deduplicated positives and to filter away positives taken from the corpus
    @tomaarsen
    tomaarsen committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    46ae230 View commit details
    Browse the repository at this point in the history

  2. Skip rescoring positive pairs via pos_to_corpus_indices instead

    @tomaarsen
    tomaarsen committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    aba1fb7 View commit details
    Browse the repository at this point in the history

  3. Add a mine_hard_negatives_from_corpus util

    @ArthurCamara
    ArthurCamara committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    ecbfc3c View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Merge pull request #1 from ChrisGeishauser/mine_hard_negatives_with_c… 

    …orpus

Mine hard negatives with corpus
    @ArthurCamara
    ArthurCamara committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    1d5c985 View commit details
    Browse the repository at this point in the history

  2. Speedup pos_to_corpus_indices for large corpora

    @ArthurCamara
    ArthurCamara committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    344b6b4 View commit details
    Browse the repository at this point in the history

  3. Fix range_max by number of max_positives in dataset

    @ArthurCamara
    ArthurCamara committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    2e86570 View commit details
    Browse the repository at this point in the history

  4. encode in chunks, ensure at least one positive per query always

    @ArthurCamara
    ArthurCamara committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    330ea40 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Hard_negative_mining with corpus and multiple positives is possible

    @ArthurCamara
    ArthurCamara committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    d75282f View commit details
    Browse the repository at this point in the history

  2. docstring

    @ArthurCamara
    ArthurCamara committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    0962fe3 View commit details
    Browse the repository at this point in the history

  3. Fix for random sampling

    @ArthurCamara
    ArthurCamara committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    9f00605 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. fix for return_triplets=False

    @ArthurCamara
    ArthurCamara committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    adcd06a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into hard_negatives_from_corpus

    @ArthurCamara
    ArthurCamara committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    b5ce939 View commit details
    Browse the repository at this point in the history

  3. Typo on list

    @ArthurCamara
    ArthurCamara committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    defbfe5 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Fix bug with multiple positives. More efficient creation of some tens… 

    …ors.
    @ArthurCamara
    ArthurCamara committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    33c74c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. Fix offset of positives scoring with multiple chunks

    @ArthurCamara
    ArthurCamara committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    83bd387 View commit details
    Browse the repository at this point in the history

  2. fix pytorch copy warning

    @ArthurCamara
    ArthurCamara committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    c15c7a6 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Only embed each text once; no need for chunking if convert_to_numpy=True

    @tomaarsen
    tomaarsen committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    93b4d69 View commit details
    Browse the repository at this point in the history

  2. Undo unintended changes

    @tomaarsen
    tomaarsen committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    691917e View commit details
    Browse the repository at this point in the history

  3. Fix mismatch in anchor/positive and negatives if multiple positives p… 

    …er query
    @tomaarsen
    tomaarsen committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    5320a23 View commit details
    Browse the repository at this point in the history

  4. Don't repeat positive_scores as it inflates the positive score counts

    @tomaarsen
    tomaarsen committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    762c0fe View commit details
    Browse the repository at this point in the history

  5. Remove the "Count" for Difference as it's rather confusing

    @tomaarsen
    tomaarsen committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    0d4d73e View commit details
    Browse the repository at this point in the history