Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable storage #497

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
28f7e7f
Add Data Preparator cookiecutter template
aristizabal95 Mar 3, 2023
6f9e19e
Rename cookiecutter folder
aristizabal95 Mar 3, 2023
df6e6a2
Temporarily remove possibly offending files
aristizabal95 Mar 3, 2023
a7db0cf
Remove cookicutter conditionals
aristizabal95 Mar 3, 2023
a7a6d15
Inclube back missing pieces of template
aristizabal95 Mar 3, 2023
e2f7108
remove cookiecutter typo
aristizabal95 Mar 3, 2023
581b5bb
Use project_name attribute
aristizabal95 Mar 3, 2023
fd77804
Change cookiecutter fields order
aristizabal95 Mar 3, 2023
6eebd59
Create empty directories on hook
aristizabal95 Mar 3, 2023
5ef86a2
Fix empty folders paths
aristizabal95 Mar 3, 2023
d04baf8
Create evaluator mlcube cookiecutter template
aristizabal95 Mar 6, 2023
02cec01
Fix JSON Syntax Error
aristizabal95 Mar 6, 2023
b3d7a1d
Update template default values
aristizabal95 Mar 6, 2023
7338236
Remove reference to undefined template variable
aristizabal95 Mar 6, 2023
d1cec5e
Implement model mlcube cookiecutter template
aristizabal95 Mar 6, 2023
7338755
Update cookiecutter variable default values
aristizabal95 Mar 6, 2023
3ae9226
Create medperf CLI command for creating MLCubes
aristizabal95 Mar 6, 2023
e07cde2
Provide additional options for mlcube create
aristizabal95 Mar 6, 2023
68e136a
Start working on tests
aristizabal95 Mar 7, 2023
b8e03ac
Add tests for cube create
aristizabal95 Mar 7, 2023
7896b25
Ignore invalid syntax on cookiecutter conditionals
aristizabal95 Mar 7, 2023
4f78981
Ignore more flake8 errors
aristizabal95 Mar 7, 2023
f5dab5e
Remove unused import
aristizabal95 Mar 7, 2023
a03d7f6
Empty commit for cloudbuild
aristizabal95 Mar 8, 2023
6bb60d0
Fix inconsistency with labels paths
aristizabal95 Mar 8, 2023
43b6cab
Update mlcube.yaml so it can be commented on docs
aristizabal95 Mar 8, 2023
55b5d22
Don't render noqa comments on template
aristizabal95 Mar 8, 2023
135c598
Remove flake8 specific ignores
aristizabal95 Mar 8, 2023
e9e2c32
Exclude templates from lint checks
aristizabal95 Mar 8, 2023
e95dab8
Remove specific flake8 ignores
aristizabal95 Mar 8, 2023
d059b7a
Fix labels_paht being passed in he wrong situation
aristizabal95 Mar 10, 2023
fcdaa7b
Add requirements to cookiecutters
aristizabal95 Mar 13, 2023
37f3f3c
Set separate labels as true by default
aristizabal95 Mar 14, 2023
b45fdb9
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Mar 22, 2023
fbf02b4
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Mar 31, 2023
7a33c23
Remove duplicate templates
aristizabal95 Mar 31, 2023
e9a1190
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 5, 2023
f2ff354
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 12, 2023
a682021
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 19, 2023
50592ee
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 21, 2023
31c6bbf
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 21, 2023
69ae2ed
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Apr 26, 2023
8a47ad9
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 May 8, 2023
5f32e50
Abstract field-error dict formatting
aristizabal95 May 16, 2023
3951af8
Reformat errors dictionary for printing
aristizabal95 May 16, 2023
09b8d69
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 May 16, 2023
e7a8ae4
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 May 23, 2023
091b69c
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Jun 23, 2023
c69bf4c
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Jul 25, 2023
d0c2b77
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Jul 26, 2023
c70bb9b
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Aug 8, 2023
90b5cb3
Merge branch 'main' of https://github.com/mlcommons/medperf
aristizabal95 Oct 6, 2023
1ffa543
implement configurable storage path
aristizabal95 Oct 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion cli/medperf/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@
keyring_access_token_service_name = "medperf_access_token"
keyring_refresh_token_service_name = "medperf_refresh_token"

storage = str(Path.home().resolve() / ".medperf")
base_storage = str(Path.home().resolve() / ".medperf")
storage = base_storage
logs_storage = "logs"
tmp_storage = "tmp"
data_storage = "data"
Expand Down Expand Up @@ -93,6 +94,7 @@
mlcube_inspect_timeout = None

configurable_parameters = [
"storage",
"server",
"certificate",
"auth_class",
Expand Down
11 changes: 10 additions & 1 deletion cli/medperf/decorators.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import os
import sys
import typer
import logging
Expand Down Expand Up @@ -67,7 +68,13 @@ def configurable(func: Callable) -> Callable:
# by changing configurable_parameters
@merge_args(func)
def wrapper(
ctx: typer.Context,
*args,
storage: str = typer.Option(
config.storage,
"--storage",
help="Location to store medperf-related artifacts",
),
server: str = typer.Option(
config.server, "--server", help="URL of a hosted MedPerf API instance"
),
Expand Down Expand Up @@ -153,6 +160,8 @@ def wrapper(
),
**kwargs,
):
return func(*args, **kwargs)
# Transform the context as necessary
ctx.params["storage"] = os.path.expanduser(storage)
return func(ctx, *args, **kwargs)

return wrapper
2 changes: 1 addition & 1 deletion cli/medperf/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def storage_path(subpath: str):

def base_storage_path(subpath: str):
"""Helper function that converts a path to base storage-related path"""
return os.path.join(config.storage, subpath)
return os.path.join(config.base_storage, subpath)


def get_file_hash(path: str) -> str:
Expand Down
Loading