Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] Verbosity rework #958

Merged
merged 14 commits into from
Nov 17, 2023
Merged

[Refactor] Verbosity rework #958

merged 14 commits into from
Nov 17, 2023

Conversation

lintangsutawika
Copy link
Contributor

@lintangsutawika lintangsutawika commented Nov 2, 2023

Make the args.verbosity work as initially intended across the whole repo.

The command

lm-eval --model hf --model_args "pretrained=EleutherAI/pythia-2.8b" --task mmlu --limit 10 --verbosity WARNING

now suppresses DEBUG level logs in the lm_eval/tasks/__init__.py

I'll inspect and see if other logs in lm_eval should be set to DEBUG instead of INFO.

cc: @haileyschoelkopf

@lintangsutawika lintangsutawika added this to the v0.3.0 milestone Nov 3, 2023
@lintangsutawika
Copy link
Contributor Author

@haileyschoelkopf

  1. Added initialize_task to handle all task initialization so that it doesn't do it when lm_eval.tasks or other modules are imported.
  2. Imports are moved back to the top
  3. eval_logger is part of utils.py

Copy link
Collaborator

@haileyschoelkopf haileyschoelkopf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good, just some minor comments about style, and about making sure that evaluate() functions work outside of running main.py.

I think we'll also have to make sure the scripts/ directories call initialize_tasks() as well.

lm_eval/__main__.py Outdated Show resolved Hide resolved
lm_eval/evaluator.py Outdated Show resolved Hide resolved
lm_eval/__init__.py Outdated Show resolved Hide resolved
@haileyschoelkopf haileyschoelkopf merged commit afda655 into big-refactor Nov 17, 2023
8 checks passed
@haileyschoelkopf haileyschoelkopf deleted the verbosity-rework branch November 17, 2023 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants