Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multimodal prototyping #2243

Merged
merged 69 commits into from
Sep 13, 2024
Merged

Multimodal prototyping #2243

merged 69 commits into from
Sep 13, 2024

Commits on Jul 2, 2024

  1. add WIP hf vlm class

    @haileyschoelkopf
    haileyschoelkopf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    5b62a52 View commit details
    Browse the repository at this point in the history

  2. add doc_to_image

    @lintangsutawika
    lintangsutawika committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    34a079e View commit details
    Browse the repository at this point in the history

  3. add mmmu tasks

    @lintangsutawika
    lintangsutawika committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    8bce8cf View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge branch 'hailey-multimodal-prototyping' into multimodal-prototyping

    @haileyschoelkopf
    haileyschoelkopf committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    6cc6e9c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into multimodal-prototyping

    @haileyschoelkopf
    haileyschoelkopf committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    e4db76c View commit details
    Browse the repository at this point in the history

  3. fix merge conflicts

    @haileyschoelkopf
    haileyschoelkopf committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    1c94a54 View commit details
    Browse the repository at this point in the history

  4. add lintang's changes to hf_vlms.py

    @haileyschoelkopf
    haileyschoelkopf committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    9692aa0 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. fix doc_to_image

    @lintangsutawika
    lintangsutawika committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    90ba03a View commit details
    Browse the repository at this point in the history

  2. added yaml_path for config-loading

    @lintangsutawika
    lintangsutawika committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    aa6c50e View commit details
    Browse the repository at this point in the history

  3. revert

    @lintangsutawika
    lintangsutawika committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    7c76574 View commit details
    Browse the repository at this point in the history

  4. add line to process str type v

    @lintangsutawika
    lintangsutawika committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    1b9deaa View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. update

    @lintangsutawika
    lintangsutawika committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    8db0a47 View commit details
    Browse the repository at this point in the history

  2. modeling cleanup

    @haileyschoelkopf
    haileyschoelkopf committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    9b9ca7b View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. merge with lintang-multimodal-prototyping

    @haileyschoelkopf
    haileyschoelkopf committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    df7fee6 View commit details
    Browse the repository at this point in the history

  2. add aggregation for mmmu

    @haileyschoelkopf
    haileyschoelkopf committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    8d92a68 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. rewrite MMMU processing code based on only MMMU authors' repo (doc_to… 

    …_image still WIP)
    @haileyschoelkopf
    haileyschoelkopf committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    f410d35 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. implemented doc_to_image

    @lintangsutawika
    lintangsutawika committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    ebf54d8 View commit details
    Browse the repository at this point in the history

  2. update doc_to_image to accept list of features

    @lintangsutawika
    lintangsutawika committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    941b502 View commit details
    Browse the repository at this point in the history

  3. update functions

    @lintangsutawika
    lintangsutawika committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    8e4c1d6 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2024

  1. readd image processed

    @lintangsutawika
    lintangsutawika committed Aug 10, 2024
    Configuration menu
    Copy the full SHA
    63bcbc5 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2024

  1. update args process

    @lintangsutawika
    lintangsutawika committed Aug 11, 2024
    Configuration menu
    Copy the full SHA
    15dda35 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. bugfix for repeated images fed to model

    @haileyschoelkopf
    haileyschoelkopf committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d811a3a View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. push WIP loglikelihood code

    @haileyschoelkopf
    haileyschoelkopf committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    2242ed3 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. commit most recent code (generative ; qwen2-vl testing)

    @haileyschoelkopf
    haileyschoelkopf committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    be14ac1 View commit details
    Browse the repository at this point in the history

  2. preliminary image_token_id handling

    @haileyschoelkopf
    haileyschoelkopf committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    7516b88 View commit details
    Browse the repository at this point in the history

  3. small mmmu update: some qs have >4 mcqa options

    @haileyschoelkopf
    haileyschoelkopf committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    5a65d10 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. push updated modeling code

    @haileyschoelkopf
    haileyschoelkopf committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    54d317d View commit details
    Browse the repository at this point in the history

  2. merge with most recent main

    @haileyschoelkopf
    haileyschoelkopf committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    9789f83 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. use processor.apply_chat_template

    @baberabb
    baberabb committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    40a48c2 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. add mathvista draft

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8148fe4 View commit details
    Browse the repository at this point in the history

  2. nit

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    38b6fe3 View commit details
    Browse the repository at this point in the history

  3. nit

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d348106 View commit details
    Browse the repository at this point in the history

  4. ensure no footguns in text<>multimodal LM<>task incompatibility

    @haileyschoelkopf
    haileyschoelkopf committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    295a825 View commit details
    Browse the repository at this point in the history

  5. add notification to readme regarding launch of prototype!

    @haileyschoelkopf
    haileyschoelkopf committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    2c9fd79 View commit details
    Browse the repository at this point in the history

  6. fix compatibility check

    @haileyschoelkopf
    haileyschoelkopf committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    80e1711 View commit details
    Browse the repository at this point in the history

  7. reorganize mmmu configs

    @haileyschoelkopf
    haileyschoelkopf committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d84b9fc View commit details
    Browse the repository at this point in the history

  8. chat_template=None

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    207778f View commit details
    Browse the repository at this point in the history

  9. add interleave chat_template

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    45d0f3c View commit details
    Browse the repository at this point in the history

  10. add condition

    @baberabb
    baberabb committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5d55e6c View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. add max_images; interleave=true

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    af8382b View commit details
    Browse the repository at this point in the history

  2. nit

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    f1185a2 View commit details
    Browse the repository at this point in the history

  3. testmini_mcq

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c67d810 View commit details
    Browse the repository at this point in the history

  4. nit

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5848698 View commit details
    Browse the repository at this point in the history

  5. pass image string; convert img

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    294dc01 View commit details
    Browse the repository at this point in the history

  6. add vllm

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    b45d295 View commit details
    Browse the repository at this point in the history

  7. add init

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    db554e0 View commit details
    Browse the repository at this point in the history

  8. vlm add multi attr

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    90b4601 View commit details
    Browse the repository at this point in the history

  9. fixup

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    94731a7 View commit details
    Browse the repository at this point in the history

  10. pass max images to vllm model init

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d5cbc48 View commit details
    Browse the repository at this point in the history

  11. nit

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    3b4ce83 View commit details
    Browse the repository at this point in the history

  12. encoding to device

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    357cf64 View commit details
    Browse the repository at this point in the history

  13. fix HFMultimodalLM.chat_template ?

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    f9cf90e View commit details
    Browse the repository at this point in the history

  14. Merge branch 'multimodal-prototyping' of https://github.com/EleutherA… 

    …I/lm-evaluation-harness into multimodal-prototyping
    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    f04be6c View commit details
    Browse the repository at this point in the history

  15. add mmmu readme

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    8349e12 View commit details
    Browse the repository at this point in the history

  16. Merge branch 'main' into multimodal-prototyping

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    805a115 View commit details
    Browse the repository at this point in the history

  17. remove erroneous prints

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    05f0dd6 View commit details
    Browse the repository at this point in the history

  18. Merge branch 'multimodal-prototyping' of https://github.com/EleutherA… 

    …I/lm-evaluation-harness into multimodal-prototyping
    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    4623768 View commit details
    Browse the repository at this point in the history

  19. use HFMultimodalLM.chat_template ; restore tasks/__init__.py

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    9ddb2ec View commit details
    Browse the repository at this point in the history

  20. add docstring for replace_placeholders in utils

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d1aadbc View commit details
    Browse the repository at this point in the history

  21. fix replace_placeholders; set image_string=None

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    19d6874 View commit details
    Browse the repository at this point in the history

  22. fix typo

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5665bc6 View commit details
    Browse the repository at this point in the history

  23. cleanup + fix merge conflicts

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    31e5ab0 View commit details
    Browse the repository at this point in the history

  24. Merge branch 'multimodal-prototyping' of https://github.com/EleutherA… 

    …I/lm-evaluation-harness into multimodal-prototyping
    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    2d0c1b0 View commit details
    Browse the repository at this point in the history

  25. update MMMU readme

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c0b585d View commit details
    Browse the repository at this point in the history

  26. del mathvista

    @baberabb
    baberabb committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5c0bd54 View commit details
    Browse the repository at this point in the history

  27. add some sample scores

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    a3bb2f1 View commit details
    Browse the repository at this point in the history

  28. Update README.md

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5f76efd View commit details
    Browse the repository at this point in the history

  29. add log msg for image_string value

    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    b3e87ae View commit details
    Browse the repository at this point in the history

  30. Merge branch 'multimodal-prototyping' of https://github.com/EleutherA… 

    …I/lm-evaluation-harness into multimodal-prototyping
    @haileyschoelkopf
    haileyschoelkopf committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d85c3b6 View commit details
    Browse the repository at this point in the history