mlcommons · hasan7n · Dec 18, 2023 · Nov 22, 2023 · Nov 22, 2023 · Nov 22, 2023
@@ -3,7 +3,10 @@ demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo
 model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz
 assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/
 tutorial_id: benchmark
+hide:
+  - toc
 ---
+![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"}
 
 # Hands-on Tutorial for Bechmark Committee
 
@@ -27,10 +30,12 @@ In this guide, you will learn how a user can use MedPerf to create a benchmark.
 
 It's assumed that you have already set up the general testing environment as explained in the [setup guide](setup.md).
 
+
 {% include "getting_started/shared/before_we_start.md" %}
 
 ## 1. Implement a Valid Workflow
 
+![Benchmark Committee implements MLCubes for workflow](../tutorial_images/bc-1-bc-implements-mlcubes.png){class="tutorial-sticky-image-content"}
 The implementation of a valid workflow is accomplished by implementing three [MLCubes](../mlcubes/mlcubes.md):
 
 1. **Data Preparator MLCube:** This MLCube will transform raw data into a dataset ready for the AI model execution. All data owners willing to participate in this benchmark will have their data prepared using this MLCube. A guide on how to implement data preparation MLCubes can be found [here](../mlcubes/mlcube_data.md).
@@ -43,6 +48,7 @@ For this tutorial, you are provided with following three already implemented mlc
 
 ## 2. Develop a Demo Dataset
 
+![Benchmark Committee develops a demo dataset](../tutorial_images/bc-2-bc-develops-dataset.png){class="tutorial-sticky-image-content"}
 A demo dataset is a small reference dataset. It contains a few data records and their labels, which will be used to test the benchmark's workflow in two scenarios:
 
 1. It is used for testing the benchmark's default workflow. The MedPerf client automatically runs a compatibility test of the benchmark's three mlcubes prior to its submission. The test is run using the benchmark's demo dataset as input.
@@ -76,8 +82,66 @@ medperf test run \
 
 Assuming the test passes successfully, you are ready to submit the MLCubes to the MedPerf server.
 
-## 4. Submitting the MLCubes
+## 4. Host the Demo Dataset
+
+The demo dataset should be packaged in a specific way as a compressed tarball file. The folder stucture in the workspace currently looks like the following:
+
+```text
+.
+└── medperf_tutorial
+    ├── demo_data
+    │   ├── images
+    │   └── labels
+    │
+    ...
+```
+
+The goal is to package the folder `demo_data`. You must first create a file called `paths.yaml`. This file will provide instructions on how to locate the data records path and the labels path. The `paths.yaml` file should specify both the data records path and the labels path.
+
+In your workspace directory (`medperf_tutorial`), create a file `paths.yaml` and fill it with the following:
+
+```text
+data_path: demo_data/images
+labels_path: demo_data/labels
+```
+
+!!! note
+    The paths are determined by the Data Preparator MLCube's expected input path.
+
+After that, the workspace should look like the following:
+
+```text
+.
+└── medperf_tutorial
+    ├── demo_data
+    │   ├── images
+    │   └── labels
+    ├── paths.yaml
+    │
+    ...
+```
+
+Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command in your workspace directory:
+
+```bash
+tar -czf demo_data.tar.gz demo_data paths.yaml
+```
+
+And that's it! Now you have to host the tarball file (`demo_data.tar.gz`) on the internet.
+
+For the tutorial to run smoothly, the file is already hosted at the following URL:
+
+```text
+{{ demo_url }}
+```
+
+If you wish to host it by yourself, you can find the list of supported options and details about hosting files in [this page](../concepts/hosting_files.md).
+
+Finally, now after having the MLCubes submitted and the demo dataset hosted, you can submit the benchmark to the MedPerf server.
+
+## 5. Submitting the MLCubes
 
+![Benchmark Committee submits MLCubes](../tutorial_images/bc-3-bc-submits-mlcubes.png){class="tutorial-sticky-image-content"}
 ### How does MedPerf Recognize an MLCube?
 
 {% include "getting_started/shared/mlcube_submission_overview.md" %}
@@ -98,6 +162,7 @@ For each MLCube, this script will create a new folder named `assets` in the MLCu
 
 #### Data Preparator MLCube
 
+![Benchmark Committee submits the Data Preparator MLCube](../tutorial_images/bc-4-bc-submits-data-preparator.png){class="tutorial-sticky-image-content"}
 For the Data Preparator MLCube, the submission should include:
 
 - The URL to the hosted mlcube manifest file, which is:
@@ -124,6 +189,7 @@ medperf mlcube submit \
 
 #### Reference Model MLCube
 
+![Benchmark Committee submits the reference Model MLCube](../tutorial_images/bc-5-bc-submits-ref-model.png){class="tutorial-sticky-image-content"}
 For the Reference Model MLCube, the submission should include:
 
 - The URL to the hosted mlcube manifest file:
@@ -156,6 +222,7 @@ medperf mlcube submit \
 
 #### Metrics MLCube
 
+![Benchmark Committee submits the Evaluation Metrics MLCube](../tutorial_images/bc-6-bc-submits-evalmetrics.png){class="tutorial-sticky-image-content"}
 For the Metrics MLCube, the submission should include:
 
 - The URL to the hosted mlcube manifest file:
@@ -187,65 +254,9 @@ medperf mlcube ls --mine
 
 Next, you will learn how to host the demo dataset.
 
-## 5. Host the Demo Dataset
-
-The demo dataset should be packaged in a specific way as a compressed tarball file. The folder stucture in the workspace currently looks like the following:
-
-```text
-.
-└── medperf_tutorial
-    ├── demo_data
-    │   ├── images
-    │   └── labels
-    │
-    ...
-```
-
-The goal is to package the folder `demo_data`. You must first create a file called `paths.yaml`. This file will provide instructions on how to locate the data records path and the labels path. The `paths.yaml` file should specify both the data records path and the labels path.
-
-In your workspace directory (`medperf_tutorial`), create a file `paths.yaml` and fill it with the following:
-
-```text
-data_path: demo_data/images
-labels_path: demo_data/labels
-```
-
-!!! note
-    The paths are determined by the Data Preparator MLCube's expected input path.
-
-After that, the workspace should look like the following:
-
-```text
-.
-└── medperf_tutorial
-    ├── demo_data
-    │   ├── images
-    │   └── labels
-    ├── paths.yaml
-    │
-    ...
-```
-
-Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command in your workspace directory:
-
-```bash
-tar -czf demo_data.tar.gz demo_data paths.yaml
-```
-
-And that's it! Now you have to host the tarball file (`demo_data.tar.gz`) on the internet.
-
-For the tutorial to run smoothly, the file is already hosted at the following URL:
-
-```text
-{{ demo_url }}
-```
-
-If you wish to host it by yourself, you can find the list of supported options and details about hosting files in [this page](../concepts/hosting_files.md).
-
-Finally, now after having the MLCubes submitted and the demo dataset hosted, you can submit the benchmark to the MedPerf server.
-
 ## 6. Submit your Benchmark
 
+![Benchmark Committee submits the Benchmark Metadata](../tutorial_images/bc-7-bc-submits-benchmark.png){class="tutorial-sticky-image-content"}
 You need to keep at hand the following information:
 
 - The Demo Dataset URL. Here, the URL will be:
@@ -282,6 +293,7 @@ That's it! You can check your benchmark's server UID by running:
 medperf benchmark ls --mine
 ```
 
+![The end of the tutorial](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"}
 {% include "getting_started/shared/cleanup.md" %}
 
 ## See Also

@@ -1,7 +1,10 @@
 ---
 prepared_hash: 9d56e799a9e63a6c3ced056ebd67eb6381483381
 tutorial_id: data
+hide:
+  - toc
 ---
+![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"}
 # Hands-on Tutorial for Data Owners
 
 ## Overview
@@ -20,6 +23,7 @@ It is assumed that you have the general testing environment [set up](setup.md).
 
 ## 1. Prepare your Data
 
+![Dataset Owner prepares the dataset](../tutorial_images/do-1-do-prepares-datset.png){class="tutorial-sticky-image-content"}
 To prepare your data, you need to collect the following information:
 
 - A name you wish to have for your dataset.
@@ -56,6 +60,7 @@ After that, your dataset will be successfully prepared! This command will also c
 
 ## 2. Submit your Data Information
 
+![Dataset Owner registers the dataset metadata](../tutorial_images/do-2-do-registers-dataset.png){class="tutorial-sticky-image-content"}
 To submit your data information, you need to know the generated UID of your prepared dataset. Normally, you will see it in the output of the previous command. You can always check local datasets information by running:
 
 ```bash
@@ -79,6 +84,7 @@ After successfully submitting your dataset, you can proceed to request participa
 
 ## 3. Request Participation
 
+![Dataset Owner requests to participate in the benchmark](../tutorial_images/do-3-do-requests-participation.png){class="tutorial-sticky-image-content"}
 For submitting the results of executing the benchmark models on your data in the future, you must associate your data with the benchmark.
 
 Once you have submitted your dataset to the MedPerf server, it will be assigned a server UID, which you can find by running `medperf dataset ls --mine`. Your dataset's server UID is also `1`.
@@ -93,6 +99,7 @@ This command will first run the benchmark's reference model on your dataset to e
 
 #### How to proceed after requesting association
 
+![Benchmark Committee accepts / rejects datasets](../tutorial_images/do-4-bc-accepts-rejects-datasets.png){class="tutorial-sticky-image-content"}
 When participating with a real benchmark, you must wait for the Benchmark Committee to approve the association request. You can check the status of your association requests by running `medperf association ls`. The association is identified by the server UIDs of your dataset and the benchmark with which you are requesting association.
 
 _For the sake of continuing the tutorial only_, run the following to simulate the benchmark committee approving your association (make sure you are in the MedPerf's root directory):
@@ -105,6 +112,7 @@ You can verify if your association request has been approved by running `medperf
 
 ## 4. Execute the Benchmark
 
+![Dataset Owner runs Benchmark models](../tutorial_images/do-5-do-runs-models.png){class="tutorial-sticky-image-content"}
 MedPerf provides a command that runs all the models of a benchmark effortlessly. You only need to provide two parameters:
 
 - The benchmark ID you want to run, which is `1`.
@@ -146,6 +154,7 @@ For now, your results are only local. Next, you will learn how to submit the res
 
 ## 5. Submit a Result
 
+![Dataset Owner submits evaluation results](../tutorial_images/do-6-do-submits-eval-results.png){class="tutorial-sticky-image-content"}
 After executing the benchmark, you will submit a result to the MedPerf server. To do so, you have to find the target result generated UID.
 
 As an example, you will be submitting the result of UID `b1m4d1`. To do this, run the following command:
@@ -156,6 +165,7 @@ medperf result submit --result b1m4d1
 
 The information that is going to be submitted will be printed to the screen and you will be prompted to confirm that you want to submit.
 
+![The end](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"}
 {% include "getting_started/shared/cleanup.md" %}
 
 ## See Also

@@ -3,8 +3,10 @@ demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo
 model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz
 assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/
 tutorial_id: model
+hide:
+  - toc
 ---
-
+![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"}
 # Hands-on Tutorial for Model Owners
 
 {% set model_mlcube = assets_url+"model_mobilenetv2/mlcube/mlcube.yaml" %}
@@ -26,6 +28,7 @@ It's assumed that you have already set up the general testing environment as exp
 
 ## 1. Test your MLCube Compatibility
 
+![Model Owner implements & tests MLCube](../tutorial_images/mo-1-mo-implements-cube.png){class="tutorial-sticky-image-content"}
 Before submitting your MLCube, it is highly recommended that you test your MLCube compatibility with the benchmarks of interest to avoid later edits and multiple submissions. Your MLCube should be compatible with the benchmark workflow in two main ways:
 
 1. It should expect a specific data input structure
@@ -50,6 +53,7 @@ Assuming the test passes successfuly, you are ready to submit the MLCube to the
 
 ## 2. Submit the MLCube
 
+![Model Owner submits Model MLCube](../tutorial_images/mo-2-mo-submits-model.png){class="tutorial-sticky-image-content"}
 ### How does MedPerf Recognize an MLCube?
 
 {% include "getting_started/shared/mlcube_submission_overview.md" %}
@@ -104,6 +108,7 @@ medperf mlcube ls --mine
 
 ## 3. Request Participation
 
+![Model Owner requests to participate in the benchmark](../tutorial_images/mo-3-mo-requests-participation.png){class="tutorial-sticky-image-content"}
 Benchmark workflows are run by Data Owners, who will get notified when a new model is added to a benchmark. You must request the association for your model to be part of the benchmark.
 
 To initiate an association request, you need to collect the following information:
@@ -120,7 +125,8 @@ medperf mlcube associate --benchmark 1 --model_uid 4
 This command will first run the benchmark's workflow on your model to ensure your model is compatible with the benchmark workflow. Then, the association request information is printed on the screen, which includes an executive summary of the test mentioned. You will be prompted to confirm sending this information and initiating this association request.
 
 #### What Happens After Requesting the Association?
-
+![Benchmark Committee accepts / rejects models](../tutorial_images/mo-4-bc-accepts-rejects-models.png){class="tutorial-sticky-image-content"}
 When participating with a real benchmark, you must wait for the Benchmark Committee to approve the association request. You can check the status of your association requests by running `medperf association ls`. The association is identified by the server UIDs of your MLCube and the benchmark with which you are requesting association.
 
+![The end](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"}
 {% include "getting_started/shared/cleanup.md" %}
@@ -1,5 +1,13 @@
 ## Before You Start
 
+#### First steps
+
+To start experimenting with MedPerf through this tutorial, you need to start by following these quick steps:
+
+  1. **[Create an Account](../signup)**
+  2. **[Install Medperf](../installation)**
+  3. **[Set up Medperf](../setup)**
+
 #### Prepare the Local MedPerf Server
 
 For the purpose of the tutorial, you have to initialize a local MedPerf server with a fresh database and then create the necessary entities that you will be interacting with. To do so, run the following: (make sure you are in MedPerf's root folder)

@@ -1,15 +1,38 @@
 # Overview
 
-To ensure users have the best experience in learning the fundamentals of MedPerf and utilizing the MedPerf client, the following set of tutorials are provided:
-
-- [How a Benchmark Committee can create and submit a benchmark.](benchmark_owner_demo.md)
+The tutorials simulate a benchmarking example for the task of detecting thoracic diseases from chest X-ray scans. You can find the description of the used data [here](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Throughout the tutorials, you will be interacting with a temporary local MedPerf server as described in the [setup page](setup.md). This allows you to freely experiment with the MedPerf client and rerun the tutorials as many times as you want, providing you with an immersive learning experience. Please note that these tutorials also serve as a general guidance to be followed when using the MedPerf client in a real scenario.
 
-- [How a Model Owner can submit a model.](model_owner_demo.md)
+Before proceeding to the tutorials, make sure you have the general tutorial environment [set up](setup.md).
 
-- [How a Data Owner can prepare their data and execute a benchmark.](data_owner_demo.md)
+To ensure users have the best experience in learning the fundamentals of MedPerf and utilizing the MedPerf client, the following set of tutorials are provided:
 
-The tutorials simulate a benchmarking example for the task of detecting thoracic diseases from chest X-ray scans. You can find the description of the used data [here](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Throughout the tutorials, you will be interacting with a temporary local MedPerf server as described in the [setup page](setup.md). This allows you to freely experiment with the MedPerf client and rerun the tutorials as many times as you want, providing you with an immersive learning experience. Please note that these tutorials also serve as a general guidance to be followed when using the MedPerf client in a real scenario.
 
-Before proceeding to the tutorials, make sure you have the general tutorial environment [set up](setup.md).
+<div class="tutorial_card_container">
+        <a class="card" href="../benchmark_owner_demo/">
+            <div class="card_image image1">
+            </div>
+            <div class="card_text">
+                <h2 class='card_title'>Benchmark Committee</h2>
+                <p>Click here to see the documentation specifically for benchmark owners.</p>
+            </div>
+        </a>
+        <a class="card" href="../model_owner_demo/">
+            <div class="card_image image2">
+            </div>
+            <div class="card_text">
+                <h2 class='card_title'>Model Owner</h2>
+                <p>Click here to see the documentation specifically for model owners.</p>
+            </div>
+        </a>
+        <a class="card" href="../data_owner_demo/">
+            <div class="card_image image3">
+            </div>
+            <div class="card_text">
+                <h2 class='card_title'>Data Owner</h2>
+                <p>Click here to see the documentation specifically for data owners.</p>
+            </div>
+        </a>
+    </div>
 
 For a detailed reference on the commands used throughout the tutorials, you can always refer to the [command line interface documentation](../cli_reference.md).
+