Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature][refactor] Optimum-Benchmark API #118

Merged
merged 29 commits into from
Feb 6, 2024
Merged

Conversation

IlyasMoutawwakil
Copy link
Member

@IlyasMoutawwakil IlyasMoutawwakil commented Feb 5, 2024

In order to support an API that's very simple to use and doesn't require passing by hydra CLI and subprocess.
This refactoring removes most of hydra magic, especially resolvers that made the usage of optimum-benchmark as an API impossible.

Example:

from pprint import pprint
from optimum_benchmark.logging_utils import setup_logging
from optimum_benchmark.experiment import launch, ExperimentConfig
from optimum_benchmark.backends.pytorch.config import PyTorchConfig
from optimum_benchmark.launchers.process.config import ProcessConfig
from optimum_benchmark.benchmarks.inference.config import InferenceConfig

if __name__ == "__main__":
    setup_logging(level="INFO")
    backend_config = PyTorchConfig(model="gpt2", no_weights=True, device="cuda")
    launcher_config = ProcessConfig(device_isolation=True)
    benchmark_config = InferenceConfig(memory=True)
    experiment_config = ExperimentConfig(
        experiment_name="python-api-launch-experiment",
        benchmark=benchmark_config,
        launcher=launcher_config,
        backend=backend_config,
    )
    benchmark_report = launch(experiment_config)
    pprint(benchmark_report)
[2024-02-06 12:58:42,933][backend][WARNING] - Multiple GPUs detected but CUDA_VISIBLE_DEVICES is not 
[...]
[2024-02-06 12:59:14,971][isolation][INFO] -    + Closing device(s) isolation process...
{'decode.latency(s)': 0.7354095666084248,
 'decode.throughput(tokens/s)': 269.2377268263452,
 'forward.latency(s)': 0.0073284958915750915,
 'forward.max_memory_allocated(MB)': 530,
 'forward.max_memory_reserved(MB)': 574,
 'forward.max_memory_used(MB)': 2025,
 'forward.peak_memory(MB)': 2025,
 'forward.throughput(samples/s)': 272.9072963388325,
 'generate.latency(s)': 0.7427380625,
 'generate.max_memory_allocated(MB)': 555,
 'generate.max_memory_reserved(MB)': 618,
 'generate.max_memory_used(MB)': 2069,
 'generate.peak_memory(MB)': 2069,
 'generate.throughput(tokens/s)': 269.2739339718436}

model, device, task and library are deprecated in experiment, in favor of backend.model and backend.device, backend.task and backend.library.
No breaking changes, only more flexibility and fine grained testing 😊.

@IlyasMoutawwakil IlyasMoutawwakil changed the title [refactor] Optimum-Benchmark API [feature][refactor] Optimum-Benchmark API Feb 5, 2024
@IlyasMoutawwakil IlyasMoutawwakil merged commit 6be304d into main Feb 6, 2024
21 checks passed
@IlyasMoutawwakil IlyasMoutawwakil deleted the api-integration branch March 29, 2024 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant