Small fixes & improvements in tutorials

cli/medperf/cli.py

@@ -23,8 +23,6 @@
 app.add_typer(result.app, name="result", help="Manage results")
 app.add_typer(dataset.app, name="dataset", help="Manage datasets")
 app.add_typer(benchmark.app, name="benchmark", help="Manage benchmarks")
-app.add_typer(mlcube.app, name="mlcube", help="Manage mlcubes")
-app.add_typer(result.app, name="result", help="Manage results")
 app.add_typer(association.app, name="association", help="Manage associations")
 app.add_typer(profile.app, name="profile", help="Manage profiles")
 app.add_typer(compatibility_test.app, name="test", help="Manage compatibility tests")

cli/medperf/templates/data_preparator_mlcube/cookiecutter.json

@@ -4,5 +4,5 @@
     "description": "Data Preparator MLCube Template. Provided by MLCommons",
     "author_name": "John Smith",
     "accelerator_count": "0",
-    "docker_image_name": "docker/image:latest",
+    "docker_image_name": "docker/image:latest"
 }

docs/cli_reference.md

@@ -1 +1,2 @@
 # In Progress
+TODO: the page is hidden now. If implemented, find all usages and uncomment them.

docs/concepts/associations.md

@@ -1 +1,2 @@
 # In Progress
+TODO: the page is hidden now. If implemented, find all usages and uncomment them.

docs/concepts/auth.md

@@ -23,21 +23,21 @@ After entering your email address, you will be provided with a verification URL
 
 - **Step2** Open the verification URL and confirm the code:
 
-Open the printed URL in your browser. You will be presented with a code and you will be asked to confirm if that code is the same one printed in your terminal.
+Open the printed URL in your browser. You will be presented with a code, and you will be asked to confirm if that code is the same one printed in your terminal.
 
 ![Code Confirmation](../assets/auth/code_confirmation.png)
 
 - **Step3** After confirmation, you will be asked to enter your email address. Enter your email address and press "Continue". You will see the following screen:
 
 ![Login code](../assets/auth/login_code.png)
 
-- **Step4** Check your inbox. You should recieve an email similar to the following:
+- **Step4** Check your inbox. You should receive an email similar to the following:
 
 ![Login email](../assets/auth/login_email.png)
 
-Enter the recieved code in the previous screen.
+Enter the received code in the previous screen.
 
-- **Step5** If there is no problem with your account, the login will be successful and you will see a screen similar to the following:
+- **Step5** If there is no problem with your account, the login will be successful, and you will see a screen similar to the following:
 
 ![Login success](../assets/auth/login_success.png)
 
@@ -51,7 +51,8 @@ medperf auth logout
 
 ## Checking the authentication status
 
-Note that when you login, the MedPerf client will remember you as long as you are using the same `profile`. If you switch to another profile by running `medperf profile activate <other-profile>`, you may have to login again. If you switch back again to a profile where you previously logged in, your login state will be restored. Read more about profiles [here](profiles.md).
+Note that when you log in, the MedPerf client will remember you as long as you are using the same `profile`. If you switch to another profile by running `medperf profile activate <other-profile>`, you may have to log in again. If you switch back again to a profile where you previously logged in, your login state will be restored.
+<!-- TODO: uncomment once profiles.md are filled. Read more about profiles [here](profiles.md). -->
 
 You can always check the current login status by the running the following command:
 

docs/concepts/priorities.md

@@ -1 +1,2 @@
 # In Progress
+TODO: the page is hidden now. If implemented, find all usages and uncomment them.

docs/concepts/profiles.md

@@ -1 +1,2 @@
 # In Progress
+TODO: the page is hidden now. If implemented, find all usages and uncomment them.

docs/concepts/single_run.md

@@ -1 +1,2 @@
 # In Progress
+TODO: the page is hidden now. If implemented, find all usages and uncomment them.

docs/getting_started/benchmark_owner_demo.md

@@ -1,7 +1,7 @@
 ---
 demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz
 model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz
-assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/
+assets_url: https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/
 tutorial_id: benchmark
 hide:
   - toc
@@ -28,7 +28,7 @@ In this guide, you will learn how a user can use MedPerf to create a benchmark.
 5. Host the demo dataset.
 6. Submit the benchmark to the MedPerf server.
 
-It's assumed that you have already set up the general testing environment as explained in the [setup guide](setup.md).
+It's assumed that you have already set up the general testing environment as explained in the [installation](installation.md) and [setup guide](setup.md).
 
 
 {% include "getting_started/shared/before_we_start.md" %}
@@ -121,10 +121,12 @@ After that, the workspace should look like the following:
     ...
 ```
 
-Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command in your workspace directory:
+Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command:
 
 ```bash
+cd medperf_tutorial
 tar -czf demo_data.tar.gz demo_data paths.yaml
+cd ..
 ```
 
 And that's it! Now you have to host the tarball file (`demo_data.tar.gz`) on the internet.
@@ -265,10 +267,16 @@ You need to keep at hand the following information:
 {{ demo_url }}
 ```
 
-- The server UIDs of the three MLCubes:
-    - Data preparator UID: `1`
-    - Reference model UID: `2`
-    - Evaluator UID: `3`
+- The server UIDs of the three MLCubes can be found by running:
+
+```bash
+ medperf mlcube ls
+```
+
+- For this tutorial, the UIDs are as follows: 
+  - Data preparator UID: `1`
+  - Reference model UID: `2`
+  - Evaluator UID: `3`
 
 You can create and submit your benchmark using the following command:
 
@@ -296,7 +304,10 @@ medperf benchmark ls --mine
 ![The end of the tutorial](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"}
 {% include "getting_started/shared/cleanup.md" %}
 
+<!--
+TODO: uncomment once pages are filled
 ## See Also
 
 - [Benchmark Associations.](../concepts/associations.md)
 - [Models Priorities](../concepts/priorities.md)
+-->

docs/getting_started/data_owner_demo.md

@@ -9,6 +9,12 @@ hide:
 
 ## Overview
 
+As a data owner, you plan to run a benchmark on your own dataset. Using MedPerf, you will prepare your (raw) dataset and submit information about it to the MedPerf server. You may have to consult the benchmark committee to make sure that your raw dataset aligns with the benchmark's expected input format.
+
+!!!Note
+    A key concept of MedPerf is the stringent confidentiality of your data. It remains exclusively on your machine. Only minimal information about your dataset, such as the hash of its contents, is submitted. Once your Dataset is submitted and associated with a benchmark, you can run all benchmark models on your data within your own infrastructure and see the results / predictions.
+
+
 This guide provides you with the necessary steps to use MedPerf as a Data Owner. The key tasks can be summarized as follows:
 
 1. Prepare your data.
@@ -70,12 +76,10 @@ medperf dataset ls --local
 !!! note
     You will be submitting general information about the data, not the data itself. The data never leaves your machine.
 
-The unique identifier for your generated data is `{{ page.meta.prepared_hash }}`.
-
 Run the following command to submit your dataset information to the MedPerf server:
 
 ```bash
-medperf dataset submit --data_uid {{ page.meta.prepared_hash }}
+medperf dataset submit --data_uid YOUR_DATASET_ID_HERE
 ```
 
 Once you run this command, the information to be submitted will be displayed on the screen and you will be asked to confirm your submission.
@@ -168,6 +172,9 @@ The information that is going to be submitted will be printed to the screen and
 ![The end](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"}
 {% include "getting_started/shared/cleanup.md" %}
 
+<!--
+TODO: uncomment once single_run is filled.
 ## See Also
 
 - [Running a Single Model.](../concepts/single_run.md)
+-->

docs/getting_started/model_owner_demo.md

@@ -1,7 +1,7 @@
 ---
 demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz
 model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz
-assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/
+assets_url: https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/
 tutorial_id: model
 hide:
   - toc

docs/getting_started/setup.md

@@ -2,6 +2,10 @@
 
 This setup is only for running the tutorials. If you are using MedPerf with a real benchmark and real experiments, skip to [this section](#choose-the-container-runner) to optionally change your container runner. Then, follow the tutorials as a general guidance for your real experiments.
 
+## Install the MedPerf Client
+
+If this is your first time using MedPerf, install the MedPerf client library as described [here](installation.md).  
+
 ## Run a Local MedPerf Server
 
 For this tutorial, you should spawn a local MedPerf server for the MedPerf client to communicate with. Note that this server will be hosted on your `localhost` and not on the internet.
@@ -27,7 +31,8 @@ After that, you will be configuring the MedPerf client to communicate with the l
 
 ## Configure the MedPerf Client
 
-The MedPerf client can be configured by creating or modifying ["`profiles`"](../concepts/profiles.md). A profile is a set of configuration parameters used by the client during runtime. By default, the profile named `default` will be active.
+<!-- TODO: set links to ["`profiles`"](../concepts/profiles.md) once profiles are filled -->
+The MedPerf client can be configured by creating or modifying "`profiles`". A profile is a set of configuration parameters used by the client during runtime. By default, the profile named `default` will be active.
 
 The `default` profile is preconfigured so that the client communicates with the main MedPerf server ([api.medperf.org](https://api.medperf.org){target="\_blank"}). For the purposes of the tutorial, you will be using the `local` profile as it is preconfigured so that the client communicates with the local MedPerf server.
 

docs/getting_started/shared/before_we_start.md

@@ -27,7 +27,14 @@ A script is provided to download all the necessary files so that you follow the
 sh tutorials_scripts/setup_{{page.meta.tutorial_id}}_tutorial.sh
 ```
 
-This will create a workspace folder `medperf_tutorial` where all necessary files are downloaded.
+This will create a workspace folder `medperf_tutorial` where all necessary files are downloaded. The folder contains the following content:
+
+<details markdown>
+<summary>Toy content description</summary>
+{% include "getting_started/shared/tutorials_content_overview/"+page.meta.tutorial_id+".md" %}
+</details>
+
+In real life all the listed artifacts and files have to be created on your own. However, for tutorial's sake you may use this toy data.   
 
 #### Login to the Local MedPerf Server
 

docs/getting_started/shared/tutorials_content_overview/benchmark.md

@@ -0,0 +1,24 @@
+In this tutorial we will create a benchmark that classifies chest X-Ray images.
+
+### Demo Data
+
+The `medperf_tutorial/demo_data/` folder contains the demo dataset content.
+
+  - `images/` folder includes sample images.
+  - `labels/labels.csv` provides a basic ground truth markup, indicating the class each image belongs to.
+
+The demo dataset is a sample dataset used for the development of your benchmark and used by Model Owners for the development of their models. More details are available in the [section below](#2-develop-a-demo-dataset)
+
+### Data Preparator MLCube
+
+The `medperf_tutorial/data_preparator/` contains a [DataPreparator MLCube](../../../mlcubes/mlcube_data.md) that you must implement. This MLCube:
+  - Transforms raw data into a format convenient for model consumption, such as converting DICOM images into numpy tensors, cropping patches, normalizing columns, etc. It's up to you to define the format that is handy for future models.
+  - Ensures its output is in a standardized format, allowing Model Owners/Developers to rely on its consistency.
+
+### Model MLCube
+
+The `medperf_tutorial/model_custom_cnn/` is an example of a [Model MLCube](../../../mlcubes/mlcube_models.md). You need to implement a reference model which will be used by data owners to test the compatibility of their data with your pipeline. Also, Model Developers joining your benchmark will follow the input/output specifications of this model when building their own models.
+
+### Metrics MLCube
+
+The `medperf_tutorial/metrics/` houses a [Metrics MLCube](../../../mlcubes/mlcube_metrics.md) that processes ground truth data, model predictions, and computes performance metrics - such as classification accuracy, loss, etc. After a Dataset Owner runs the benchmark pipeline on their data, these final metric values will be shared with you as the Benchmark Owner.

docs/getting_started/shared/tutorials_content_overview/data.md

@@ -0,0 +1,11 @@
+### Tutorial's Dataset Example
+
+The `medperf_tutorial/sample_raw_data/` folder contains your data for the specified Benchmark. In this tutorial, where the benchmark involves classifying chest X-Ray images, your data comprises:
+
+- `images/` folder contains your images
+- `labels/labels.csv`, which provides the ground truth markup, specifying the class of each image.
+
+The format of this data is dictated by the Benchmark Owner, as it must be compatible with the benchmark's Data Preparation MLCube. In a real-world scenario, the expected data format would differ from this toy example. Refer to the Benchmark Owner to get a format specifications and details for your practical case.
+
+As previously mentioned, your data itself never leaves your machine. During the dataset submission, only basic metadata is transferred, for which you will be prompted to confirm.
+

docs/getting_started/shared/tutorials_content_overview/model.md

@@ -0,0 +1,5 @@
+### Model MLCube
+
+The `medperf_tutorial/model_mobilenetv2/` is a toy [Model MLCube](../../../mlcubes/mlcube_models.md). Once you submit your model to the benchmark, all participating Data Owners would be able to run the model within the benchmark pipeline. Therefore, your MLCube must support the specific input/output formats defined by the Benchmark Owners.
+
+For the purposes of this tutorial, you will work with a pre-prepared toy benchmark. In a real-world scenario,  you should refer to your Benchmark Owner to get a format specifications and details for your practical case.

docs/getting_started/tutorials_overview.md

@@ -34,5 +34,6 @@ To ensure users have the best experience in learning the fundamentals of MedPerf
         </a>
     </div>
 
-For a detailed reference on the commands used throughout the tutorials, you can always refer to the [command line interface documentation](../cli_reference.md).
+<!--TODO: uncomment once cli_reference is filled.
+For a detailed reference on the commands used throughout the tutorials, you can always refer to the [command line interface documentation](../cli_reference.md).-->
 

docs/medperf_components.md

@@ -14,7 +14,7 @@ The MedPerf client contains all the necessary tools to interact with the server,
 
 The client communicates to the server through the API to, for example, authenticate a user, retrieve benchmarks/MLcubes and send results.
 
-The client is currently available to the user through a command-line interface (CLI). See the [CLI reference](cli_reference.md).
+The client is currently available to the user through a command-line interface (CLI). <!--TODO: uncomment once cli_reference is filled. See the [CLI reference](cli_reference.md).-->
 
 ## Auth Provider