-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Small fixes & improvements in tutorials #517
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
b74bac0
Data tutorial: removed predefined dataset hash as it varies depending…
VukW 693b471
Benchmark tutorial: added src for mlcube uids
VukW 5d4979c
Benchmark tutorial: minor fix for dataset archivation
VukW e028fef
Tutorials: include links to medperf client installation
VukW 880e5c9
Tutorials: changed asset links to main repo
VukW cf06c12
Benchmark tutorial: added tutorial data description
VukW d04708c
Added tutorial content description for data & model tutorials
VukW 4ad0c66
Bugfix: incorrect JSON for dataprep template
VukW 51f5edd
Small fix
VukW c1da318
Added doc for data preparator mlcube
VukW 874b2a9
Data Prep MLCube: improved language
VukW 072f8f2
Added doc for Metric Evaluator MLCube
VukW 4ceead1
Fixed some TODOs
VukW 732a71d
typos
VukW d418c30
Update docs/getting_started/data_owner_demo.md
VukW d224b8b
Update docs/getting_started/data_owner_demo.md
VukW 47558df
Update docs/getting_started/benchmark_owner_demo.md
VukW 122c252
Update docs/getting_started/benchmark_owner_demo.md
VukW 8a20364
Update docs/getting_started/setup.md
VukW 928cd52
Hid empty pages:
VukW dd05813
Hid mlcubes docs as they require massive rework
VukW 3862af6
Update docs/getting_started/shared/tutorials_content_overview/data.md
VukW 031199d
Update docs/getting_started/shared/tutorials_content_overview/benchma…
VukW 767f29a
Update docs/getting_started/shared/tutorials_content_overview/benchma…
VukW 75ec2ce
Update docs/getting_started/shared/tutorials_content_overview/benchma…
VukW 8a530f7
Update docs/getting_started/shared/tutorials_content_overview/benchma…
VukW 97a7965
Update docs/getting_started/shared/tutorials_content_overview/benchma…
VukW File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
# In Progress | ||
TODO: the page is hidden now. If implemented, find all usages and uncomment them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
# In Progress | ||
TODO: the page is hidden now. If implemented, find all usages and uncomment them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
# In Progress | ||
TODO: the page is hidden now. If implemented, find all usages and uncomment them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
# In Progress | ||
TODO: the page is hidden now. If implemented, find all usages and uncomment them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
# In Progress | ||
TODO: the page is hidden now. If implemented, find all usages and uncomment them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
24 changes: 24 additions & 0 deletions
24
docs/getting_started/shared/tutorials_content_overview/benchmark.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
In this tutorial we will create a benchmark that classifies chest X-Ray images. | ||
|
||
### Demo Data | ||
|
||
The `medperf_tutorial/demo_data/` folder contains the demo dataset content. | ||
|
||
- `images/` folder includes sample images. | ||
- `labels/labels.csv` provides a basic ground truth markup, indicating the class each image belongs to. | ||
|
||
The demo dataset is a sample dataset used for the development of your benchmark and used by Model Owners for the development of their models. More details are available in the [section below](#2-develop-a-demo-dataset) | ||
|
||
### Data Preparator MLCube | ||
|
||
The `medperf_tutorial/data_preparator/` contains a [DataPreparator MLCube](../../../mlcubes/mlcube_data.md) that you must implement. This MLCube: | ||
- Transforms raw data into a format convenient for model consumption, such as converting DICOM images into numpy tensors, cropping patches, normalizing columns, etc. It's up to you to define the format that is handy for future models. | ||
- Ensures its output is in a standardized format, allowing Model Owners/Developers to rely on its consistency. | ||
|
||
### Model MLCube | ||
|
||
The `medperf_tutorial/model_custom_cnn/` is an example of a [Model MLCube](../../../mlcubes/mlcube_models.md). You need to implement a reference model which will be used by data owners to test the compatibility of their data with your pipeline. Also, Model Developers joining your benchmark will follow the input/output specifications of this model when building their own models. | ||
|
||
### Metrics MLCube | ||
|
||
The `medperf_tutorial/metrics/` houses a [Metrics MLCube](../../../mlcubes/mlcube_metrics.md) that processes ground truth data, model predictions, and computes performance metrics - such as classification accuracy, loss, etc. After a Dataset Owner runs the benchmark pipeline on their data, these final metric values will be shared with you as the Benchmark Owner. |
11 changes: 11 additions & 0 deletions
11
docs/getting_started/shared/tutorials_content_overview/data.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
### Tutorial's Dataset Example | ||
|
||
The `medperf_tutorial/sample_raw_data/` folder contains your data for the specified Benchmark. In this tutorial, where the benchmark involves classifying chest X-Ray images, your data comprises: | ||
|
||
- `images/` folder contains your images | ||
- `labels/labels.csv`, which provides the ground truth markup, specifying the class of each image. | ||
|
||
The format of this data is dictated by the Benchmark Owner, as it must be compatible with the benchmark's Data Preparation MLCube. In a real-world scenario, the expected data format would differ from this toy example. Refer to the Benchmark Owner to get a format specifications and details for your practical case. | ||
|
||
As previously mentioned, your data itself never leaves your machine. During the dataset submission, only basic metadata is transferred, for which you will be prompted to confirm. | ||
|
5 changes: 5 additions & 0 deletions
5
docs/getting_started/shared/tutorials_content_overview/model.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
### Model MLCube | ||
|
||
The `medperf_tutorial/model_mobilenetv2/` is a toy [Model MLCube](../../../mlcubes/mlcube_models.md). Once you submit your model to the benchmark, all participating Data Owners would be able to run the model within the benchmark pipeline. Therefore, your MLCube must support the specific input/output formats defined by the Benchmark Owners. | ||
|
||
For the purposes of this tutorial, you will work with a pre-prepared toy benchmark. In a real-world scenario, you should refer to your Benchmark Owner to get a format specifications and details for your practical case. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed as they are duplicated a few lines higher