diff --git a/docs/getting_started/benchmark_owner_demo.md b/docs/getting_started/benchmark_owner_demo.md index 6fcbbecd5..ed849e17a 100644 --- a/docs/getting_started/benchmark_owner_demo.md +++ b/docs/getting_started/benchmark_owner_demo.md @@ -3,7 +3,10 @@ demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/ tutorial_id: benchmark +hide: + - toc --- +![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"} # Hands-on Tutorial for Bechmark Committee @@ -27,10 +30,12 @@ In this guide, you will learn how a user can use MedPerf to create a benchmark. It's assumed that you have already set up the general testing environment as explained in the [setup guide](setup.md). + {% include "getting_started/shared/before_we_start.md" %} ## 1. Implement a Valid Workflow +![Benchmark Committee implements MLCubes for workflow](../tutorial_images/bc-1-bc-implements-mlcubes.png){class="tutorial-sticky-image-content"} The implementation of a valid workflow is accomplished by implementing three [MLCubes](../mlcubes/mlcubes.md): 1. **Data Preparator MLCube:** This MLCube will transform raw data into a dataset ready for the AI model execution. All data owners willing to participate in this benchmark will have their data prepared using this MLCube. A guide on how to implement data preparation MLCubes can be found [here](../mlcubes/mlcube_data.md). @@ -43,6 +48,7 @@ For this tutorial, you are provided with following three already implemented mlc ## 2. Develop a Demo Dataset +![Benchmark Committee develops a demo dataset](../tutorial_images/bc-2-bc-develops-dataset.png){class="tutorial-sticky-image-content"} A demo dataset is a small reference dataset. It contains a few data records and their labels, which will be used to test the benchmark's workflow in two scenarios: 1. It is used for testing the benchmark's default workflow. The MedPerf client automatically runs a compatibility test of the benchmark's three mlcubes prior to its submission. The test is run using the benchmark's demo dataset as input. @@ -76,8 +82,66 @@ medperf test run \ Assuming the test passes successfully, you are ready to submit the MLCubes to the MedPerf server. -## 4. Submitting the MLCubes +## 4. Host the Demo Dataset + +The demo dataset should be packaged in a specific way as a compressed tarball file. The folder stucture in the workspace currently looks like the following: + +```text +. +└── medperf_tutorial + ├── demo_data + │ ├── images + │ └── labels + │ + ... +``` + +The goal is to package the folder `demo_data`. You must first create a file called `paths.yaml`. This file will provide instructions on how to locate the data records path and the labels path. The `paths.yaml` file should specify both the data records path and the labels path. + +In your workspace directory (`medperf_tutorial`), create a file `paths.yaml` and fill it with the following: + +```text +data_path: demo_data/images +labels_path: demo_data/labels +``` + +!!! note + The paths are determined by the Data Preparator MLCube's expected input path. + +After that, the workspace should look like the following: + +```text +. +└── medperf_tutorial + ├── demo_data + │ ├── images + │ └── labels + ├── paths.yaml + │ + ... +``` + +Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command in your workspace directory: + +```bash +tar -czf demo_data.tar.gz demo_data paths.yaml +``` + +And that's it! Now you have to host the tarball file (`demo_data.tar.gz`) on the internet. + +For the tutorial to run smoothly, the file is already hosted at the following URL: + +```text +{{ demo_url }} +``` + +If you wish to host it by yourself, you can find the list of supported options and details about hosting files in [this page](../concepts/hosting_files.md). + +Finally, now after having the MLCubes submitted and the demo dataset hosted, you can submit the benchmark to the MedPerf server. + +## 5. Submitting the MLCubes +![Benchmark Committee submits MLCubes](../tutorial_images/bc-3-bc-submits-mlcubes.png){class="tutorial-sticky-image-content"} ### How does MedPerf Recognize an MLCube? {% include "getting_started/shared/mlcube_submission_overview.md" %} @@ -98,6 +162,7 @@ For each MLCube, this script will create a new folder named `assets` in the MLCu #### Data Preparator MLCube +![Benchmark Committee submits the Data Preparator MLCube](../tutorial_images/bc-4-bc-submits-data-preparator.png){class="tutorial-sticky-image-content"} For the Data Preparator MLCube, the submission should include: - The URL to the hosted mlcube manifest file, which is: @@ -124,6 +189,7 @@ medperf mlcube submit \ #### Reference Model MLCube +![Benchmark Committee submits the reference Model MLCube](../tutorial_images/bc-5-bc-submits-ref-model.png){class="tutorial-sticky-image-content"} For the Reference Model MLCube, the submission should include: - The URL to the hosted mlcube manifest file: @@ -156,6 +222,7 @@ medperf mlcube submit \ #### Metrics MLCube +![Benchmark Committee submits the Evaluation Metrics MLCube](../tutorial_images/bc-6-bc-submits-evalmetrics.png){class="tutorial-sticky-image-content"} For the Metrics MLCube, the submission should include: - The URL to the hosted mlcube manifest file: @@ -187,65 +254,9 @@ medperf mlcube ls --mine Next, you will learn how to host the demo dataset. -## 5. Host the Demo Dataset - -The demo dataset should be packaged in a specific way as a compressed tarball file. The folder stucture in the workspace currently looks like the following: - -```text -. -└── medperf_tutorial - ├── demo_data - │ ├── images - │ └── labels - │ - ... -``` - -The goal is to package the folder `demo_data`. You must first create a file called `paths.yaml`. This file will provide instructions on how to locate the data records path and the labels path. The `paths.yaml` file should specify both the data records path and the labels path. - -In your workspace directory (`medperf_tutorial`), create a file `paths.yaml` and fill it with the following: - -```text -data_path: demo_data/images -labels_path: demo_data/labels -``` - -!!! note - The paths are determined by the Data Preparator MLCube's expected input path. - -After that, the workspace should look like the following: - -```text -. -└── medperf_tutorial - ├── demo_data - │ ├── images - │ └── labels - ├── paths.yaml - │ - ... -``` - -Finally, compress the required assets (`demo_data` and `paths.yaml`) into a tarball file by running the following command in your workspace directory: - -```bash -tar -czf demo_data.tar.gz demo_data paths.yaml -``` - -And that's it! Now you have to host the tarball file (`demo_data.tar.gz`) on the internet. - -For the tutorial to run smoothly, the file is already hosted at the following URL: - -```text -{{ demo_url }} -``` - -If you wish to host it by yourself, you can find the list of supported options and details about hosting files in [this page](../concepts/hosting_files.md). - -Finally, now after having the MLCubes submitted and the demo dataset hosted, you can submit the benchmark to the MedPerf server. - ## 6. Submit your Benchmark +![Benchmark Committee submits the Benchmark Metadata](../tutorial_images/bc-7-bc-submits-benchmark.png){class="tutorial-sticky-image-content"} You need to keep at hand the following information: - The Demo Dataset URL. Here, the URL will be: @@ -282,6 +293,7 @@ That's it! You can check your benchmark's server UID by running: medperf benchmark ls --mine ``` +![The end of the tutorial](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"} {% include "getting_started/shared/cleanup.md" %} ## See Also diff --git a/docs/getting_started/data_owner_demo.md b/docs/getting_started/data_owner_demo.md index 2121cda3b..7b30f2f8d 100644 --- a/docs/getting_started/data_owner_demo.md +++ b/docs/getting_started/data_owner_demo.md @@ -1,7 +1,10 @@ --- prepared_hash: 9d56e799a9e63a6c3ced056ebd67eb6381483381 tutorial_id: data +hide: + - toc --- +![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"} # Hands-on Tutorial for Data Owners ## Overview @@ -20,6 +23,7 @@ It is assumed that you have the general testing environment [set up](setup.md). ## 1. Prepare your Data +![Dataset Owner prepares the dataset](../tutorial_images/do-1-do-prepares-datset.png){class="tutorial-sticky-image-content"} To prepare your data, you need to collect the following information: - A name you wish to have for your dataset. @@ -56,6 +60,7 @@ After that, your dataset will be successfully prepared! This command will also c ## 2. Submit your Data Information +![Dataset Owner registers the dataset metadata](../tutorial_images/do-2-do-registers-dataset.png){class="tutorial-sticky-image-content"} To submit your data information, you need to know the generated UID of your prepared dataset. Normally, you will see it in the output of the previous command. You can always check local datasets information by running: ```bash @@ -79,6 +84,7 @@ After successfully submitting your dataset, you can proceed to request participa ## 3. Request Participation +![Dataset Owner requests to participate in the benchmark](../tutorial_images/do-3-do-requests-participation.png){class="tutorial-sticky-image-content"} For submitting the results of executing the benchmark models on your data in the future, you must associate your data with the benchmark. Once you have submitted your dataset to the MedPerf server, it will be assigned a server UID, which you can find by running `medperf dataset ls --mine`. Your dataset's server UID is also `1`. @@ -93,6 +99,7 @@ This command will first run the benchmark's reference model on your dataset to e #### How to proceed after requesting association +![Benchmark Committee accepts / rejects datasets](../tutorial_images/do-4-bc-accepts-rejects-datasets.png){class="tutorial-sticky-image-content"} When participating with a real benchmark, you must wait for the Benchmark Committee to approve the association request. You can check the status of your association requests by running `medperf association ls`. The association is identified by the server UIDs of your dataset and the benchmark with which you are requesting association. _For the sake of continuing the tutorial only_, run the following to simulate the benchmark committee approving your association (make sure you are in the MedPerf's root directory): @@ -105,6 +112,7 @@ You can verify if your association request has been approved by running `medperf ## 4. Execute the Benchmark +![Dataset Owner runs Benchmark models](../tutorial_images/do-5-do-runs-models.png){class="tutorial-sticky-image-content"} MedPerf provides a command that runs all the models of a benchmark effortlessly. You only need to provide two parameters: - The benchmark ID you want to run, which is `1`. @@ -146,6 +154,7 @@ For now, your results are only local. Next, you will learn how to submit the res ## 5. Submit a Result +![Dataset Owner submits evaluation results](../tutorial_images/do-6-do-submits-eval-results.png){class="tutorial-sticky-image-content"} After executing the benchmark, you will submit a result to the MedPerf server. To do so, you have to find the target result generated UID. As an example, you will be submitting the result of UID `b1m4d1`. To do this, run the following command: @@ -156,6 +165,7 @@ medperf result submit --result b1m4d1 The information that is going to be submitted will be printed to the screen and you will be prompted to confirm that you want to submit. +![The end](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"} {% include "getting_started/shared/cleanup.md" %} ## See Also diff --git a/docs/getting_started/model_owner_demo.md b/docs/getting_started/model_owner_demo.md index e7401d7ec..c4dd3e6e2 100644 --- a/docs/getting_started/model_owner_demo.md +++ b/docs/getting_started/model_owner_demo.md @@ -3,8 +3,10 @@ demo_url: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo model_add: https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz assets_url: https://raw.githubusercontent.com/hasan7n/medperf/99b0d84bc107415d9fc6f69c4ea3fcdfbf22315d/examples/chestxray_tutorial/ tutorial_id: model +hide: + - toc --- - +![Overview](../tutorial_images/overview.png){class="tutorial-sticky-image-content"} # Hands-on Tutorial for Model Owners {% set model_mlcube = assets_url+"model_mobilenetv2/mlcube/mlcube.yaml" %} @@ -26,6 +28,7 @@ It's assumed that you have already set up the general testing environment as exp ## 1. Test your MLCube Compatibility +![Model Owner implements & tests MLCube](../tutorial_images/mo-1-mo-implements-cube.png){class="tutorial-sticky-image-content"} Before submitting your MLCube, it is highly recommended that you test your MLCube compatibility with the benchmarks of interest to avoid later edits and multiple submissions. Your MLCube should be compatible with the benchmark workflow in two main ways: 1. It should expect a specific data input structure @@ -50,6 +53,7 @@ Assuming the test passes successfuly, you are ready to submit the MLCube to the ## 2. Submit the MLCube +![Model Owner submits Model MLCube](../tutorial_images/mo-2-mo-submits-model.png){class="tutorial-sticky-image-content"} ### How does MedPerf Recognize an MLCube? {% include "getting_started/shared/mlcube_submission_overview.md" %} @@ -104,6 +108,7 @@ medperf mlcube ls --mine ## 3. Request Participation +![Model Owner requests to participate in the benchmark](../tutorial_images/mo-3-mo-requests-participation.png){class="tutorial-sticky-image-content"} Benchmark workflows are run by Data Owners, who will get notified when a new model is added to a benchmark. You must request the association for your model to be part of the benchmark. To initiate an association request, you need to collect the following information: @@ -120,7 +125,8 @@ medperf mlcube associate --benchmark 1 --model_uid 4 This command will first run the benchmark's workflow on your model to ensure your model is compatible with the benchmark workflow. Then, the association request information is printed on the screen, which includes an executive summary of the test mentioned. You will be prompted to confirm sending this information and initiating this association request. #### What Happens After Requesting the Association? - +![Benchmark Committee accepts / rejects models](../tutorial_images/mo-4-bc-accepts-rejects-models.png){class="tutorial-sticky-image-content"} When participating with a real benchmark, you must wait for the Benchmark Committee to approve the association request. You can check the status of your association requests by running `medperf association ls`. The association is identified by the server UIDs of your MLCube and the benchmark with which you are requesting association. +![The end](../tutorial_images/the-end.png){class="tutorial-sticky-image-content"} {% include "getting_started/shared/cleanup.md" %} diff --git a/docs/getting_started/shared/before_we_start.md b/docs/getting_started/shared/before_we_start.md index fe006fc26..873b27494 100644 --- a/docs/getting_started/shared/before_we_start.md +++ b/docs/getting_started/shared/before_we_start.md @@ -1,5 +1,13 @@ ## Before You Start +#### First steps + +To start experimenting with MedPerf through this tutorial, you need to start by following these quick steps: + + 1. **[Create an Account](../signup)** + 2. **[Install Medperf](../installation)** + 3. **[Set up Medperf](../setup)** + #### Prepare the Local MedPerf Server For the purpose of the tutorial, you have to initialize a local MedPerf server with a fresh database and then create the necessary entities that you will be interacting with. To do so, run the following: (make sure you are in MedPerf's root folder) diff --git a/docs/getting_started/tutorials_overview.md b/docs/getting_started/tutorials_overview.md index c91d98fd1..df997797f 100644 --- a/docs/getting_started/tutorials_overview.md +++ b/docs/getting_started/tutorials_overview.md @@ -1,15 +1,38 @@ # Overview -To ensure users have the best experience in learning the fundamentals of MedPerf and utilizing the MedPerf client, the following set of tutorials are provided: - -- [How a Benchmark Committee can create and submit a benchmark.](benchmark_owner_demo.md) +The tutorials simulate a benchmarking example for the task of detecting thoracic diseases from chest X-ray scans. You can find the description of the used data [here](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Throughout the tutorials, you will be interacting with a temporary local MedPerf server as described in the [setup page](setup.md). This allows you to freely experiment with the MedPerf client and rerun the tutorials as many times as you want, providing you with an immersive learning experience. Please note that these tutorials also serve as a general guidance to be followed when using the MedPerf client in a real scenario. -- [How a Model Owner can submit a model.](model_owner_demo.md) +Before proceeding to the tutorials, make sure you have the general tutorial environment [set up](setup.md). -- [How a Data Owner can prepare their data and execute a benchmark.](data_owner_demo.md) +To ensure users have the best experience in learning the fundamentals of MedPerf and utilizing the MedPerf client, the following set of tutorials are provided: -The tutorials simulate a benchmarking example for the task of detecting thoracic diseases from chest X-ray scans. You can find the description of the used data [here](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Throughout the tutorials, you will be interacting with a temporary local MedPerf server as described in the [setup page](setup.md). This allows you to freely experiment with the MedPerf client and rerun the tutorials as many times as you want, providing you with an immersive learning experience. Please note that these tutorials also serve as a general guidance to be followed when using the MedPerf client in a real scenario. -Before proceeding to the tutorials, make sure you have the general tutorial environment [set up](setup.md). +
+ +
+
+
+

Benchmark Committee

+

Click here to see the documentation specifically for benchmark owners.

+
+ + +
+
+
+

Model Owner

+

Click here to see the documentation specifically for model owners.

+
+ + +
+
+
+

Data Owner

+

Click here to see the documentation specifically for data owners.

+
+ +
For a detailed reference on the commands used throughout the tutorials, you can always refer to the [command line interface documentation](../cli_reference.md). + diff --git a/docs/images/homepage/logo.png b/docs/images/homepage/logo.png new file mode 100644 index 000000000..478481561 Binary files /dev/null and b/docs/images/homepage/logo.png differ diff --git a/docs/images/homepage/logo.svg b/docs/images/homepage/logo.svg new file mode 100644 index 000000000..6de426663 --- /dev/null +++ b/docs/images/homepage/logo.svg @@ -0,0 +1,209 @@ + + + + + + + + + + + + diff --git a/docs/index.md b/docs/index.md index 75b01a37a..607285c8c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,52 +1,4 @@ -# Introduction - -Medical Artificial Intelligence (AI) has the potential to revolutionize healthcare by advancing evidence-based medicine, personalizing patient care, and reducing costs. Unlocking this potential requires reliable methods for evaluating the efficacy of medical machine learning (ML) models on large-scale heterogeneous data while maintaining patient privacy. - -## What is MedPerf? - -MedPerf is an open-source framework for benchmarking ML models to deliver clinical efficacy while prioritizing patient privacy and mitigating legal and regulatory risks. It enables federated evaluation in which models are securely distributed to different facilities for evaluation. The goal of federated evaluation is to make it simple and reliable to share models with many data providers, evaluate those models against their data in controlled settings, then aggregate and analyze the findings. - -The MedPerf approach empowers healthcare organizations to assess and verify the performance of ML models in an efficient and human-supervised process without sharing any patient data across facilities during the process. - -## Why MedPerf? - -MedPerf reduces the risks and costs associated with data sharing, maximizing medical and patient outcomes. The platform leads to an effective, broader, and cost-effective adoption of medical ML and improves patient outcomes. - -Anyone who joins our platform can get several benefits, regardless of the role they will assume. - -**Benefits if you are a [Data Provider](roles.md#data-providers):** - -* Evaluate how well machine learning models perform on your population’s data. -* Connect to Model Owners to help them improve medical ML in a specific domain. -* Help define impactful medical ML benchmarks. - -**Benefits if you are a [Model Owner](roles.md#model-owners):** - -* Measure model performance on private datasets that you would never have access to. -* Connect to specific Data Providers that can help you increase the performance of your model. - -**Benefits if you own a benchmark ([Benchmark Committee](roles.md#benchmark-committee)):** - -* Hold a leading role in the MedPerf ecosystem by defining specifications of a benchmark for a particular medical ML task. -* Get help to create a strong community around a specific area. -* Rule point on creating the guidelines to generate impactful ML for a specific area. -* Help improve best practices in your area of interest. -* Ensure the choice of the metrics as well as the proper reference implementations. - -**Benefits to the Broad Community:** - -* Provide consistent and rigorous approaches for evaluating the accuracy of ML models for real-world use in a standardized manner. -* Enable model usability measurement across institutions while maintaining data privacy and model reliability. -* Connect with a community of expert groups to employ scientific evaluation methodologies and technical approaches to operate benchmarks that not only have well-defined clinical aspects, such as clinical impact, clinical workflow integration and patient outcome, but also support robust technical aspects, including proper metrics, data preprocessing and reference implementation. - -## What is a benchmark in the MedPerf perspective? - -A benchmark is a collection of assets used by the platform to test the performance of ML models for a specific clinical problem. The primary components of a benchmark are: - -1. **Specifications**: precise definition of the clinical setting (e.g., problem or task and specific patient population) on which trained ML models are to be evaluated. It also includes the labeling (annotation) methodology as well as the choice of evaluation metrics. -2. **Dataset Preparation**: a process that prepares datasets for use in evaluation, and can also test the prepared datasets for quality and compatibility. This is implemented as an MLCube (see [Data Preparator MLCube](mlcubes/mlcubes.md#data-preparator-mlcube)). -3. **Registered Datasets**: a list of registered datasets prepared according to the benchmark criteria and approved for evaluation use by their owners (e.g. patient data from multiple facilities representing (as a whole) a diverse patient population). -4. **Evaluation**: a consistent implementation of the testing pipelines and evaluation metrics. -5. **Reference Implementation**: a detailed example of a benchmark submission consisting of example model code, the evaluation component, and de-identified or synthetic publicly available sample data. -6. **Registered Models**: a list of registered models to run in this benchmark. -7. **Documentation**: documents for understanding and using the benchmark. +--- +title: Title +template: home.html +--- diff --git a/docs/overrides/home.html b/docs/overrides/home.html new file mode 100644 index 000000000..85460378b --- /dev/null +++ b/docs/overrides/home.html @@ -0,0 +1,367 @@ + + +{% extends "main.html" %} +{% block tabs %} +{{ super() }} + + + +
+
+
+
+ +
+

Explore our documentation hub to understand everything you need to get started with MedPerf, + including definitions, setup, tutorials, advanced concepts, and more.

+ +
+
+
+
+
+ + +
+

Overview

+
+ + +
+
+
+

What is MedPerf?

+

Click here to see understand what is MedPerf.

+
+ + + +
+
+
+

Roles

+

Click here to see the documentation on User Roles and Responsibilities.

+
+ + + +
+
+
+

Workflow

+

Click here to see the documentation on MedPerf Benchmarking Workflow.

+
+ + +
+
+ +
+

Tutorials

+
+ + +
+
+
+

Benchmark Owner

+

Click here to see the documentation specifically for benchmark owners.

+
+ + + +
+
+
+

Model Owner

+

Click here to see the documentation specifically for model owners.

+
+ + + +
+
+
+

Data Owner

+

Click here to see the documentation specifically for data owners.

+
+ + +
+
+ + + +{% endblock %} +{% block content %}{% endblock %} +{% block footer %}{% endblock %} \ No newline at end of file diff --git a/docs/overrides/partials/header.html b/docs/overrides/partials/header.html new file mode 100644 index 000000000..61b1db1ec --- /dev/null +++ b/docs/overrides/partials/header.html @@ -0,0 +1,157 @@ + + + +{% set class = "md-header" %} +{% if "navigation.tabs.sticky" in features %} + {% set class = class ~ " md-header--lifted" %} +{% endif %} + + +
+ + + + {% if "navigation.tabs.sticky" in features %} + {% if "navigation.tabs" in features %} + {% include "partials/tabs.html" %} + {% endif %} + {% endif %} +
\ No newline at end of file diff --git a/docs/scripts/script.js b/docs/scripts/script.js new file mode 100644 index 000000000..e0db65f74 --- /dev/null +++ b/docs/scripts/script.js @@ -0,0 +1,76 @@ + +function createSideImageContainer() { + const containerElement = document.createElement('div'); + containerElement.classList.add('side-container'); + + const imageElement = document.createElement('img'); + imageElement.src = ''; + imageElement.alt = ''; + + imageElement.setAttribute('class', 'tutorial-sticky-image'); + imageElement.setAttribute('id', 'tutorial-sticky-image'); + + containerElement.appendChild(imageElement); + + return containerElement; +} + +function imagesAppending(imageElement) { + let imageSrc = ''; + let smthWasSet = false; + + const contentImages = document.getElementsByClassName("tutorial-sticky-image-content") + for (let contentImageElement of contentImages) { + + const rect = contentImageElement.getBoundingClientRect(); + if (300 > rect.top) { + imageSrc = contentImageElement.src; + smthWasSet = true; + console.log(rect.top, "changing image to", imageSrc) + if (imageElement) { + imageElement.src = imageSrc; + imageElement.style.display="block"; + } + } + } + if (!smthWasSet) { + console.log("no image was chosen. Hid the image"); + imageElement.style.display="none"; + } +} + +// ADDS IMAGING SCROLL TO TUTORIALS +window.addEventListener('scroll', function() { + let containerElement = document.querySelector('.side-container'); + + if (containerElement) { + const imageElement = containerElement.querySelector('img'); + imagesAppending(imageElement); + } +}) + +// ADDS HOME BUTTON TO HEADER +document.addEventListener('DOMContentLoaded', function() { + const headerTitle = document.querySelector(".md-header__ellipsis"); + + if (headerTitle && window.location.pathname !== "/") { + const newElement = document.createElement("a"); + newElement.textContent = "Home"; + newElement.classList.add('header_home_btn'); + newElement.href = "/"; + + headerTitle.appendChild(newElement); + console.log(document.querySelector(".header_home_btn")) + } + + const contentImages = document.getElementsByClassName("tutorial-sticky-image-content"); + if (contentImages.length) { + console.log('content images found: ', contentImages) + // we add an element only if there are content images on this page to be shown + const tutorialStickyImageElement = createSideImageContainer(); + const contentElement = document.getElementsByClassName('md-content')[0]; + contentElement.parentNode.insertBefore(tutorialStickyImageElement, contentElement.nextSibling); + imagesAppending(tutorialStickyImageElement.querySelector('img')); + } + +}); \ No newline at end of file diff --git a/docs/styles/styles.css b/docs/styles/styles.css new file mode 100644 index 000000000..d754b62a1 --- /dev/null +++ b/docs/styles/styles.css @@ -0,0 +1,183 @@ +:root, [data-md-color-scheme=default] { + --md-card-bg-color: rgb(255, 255, 255); + --md-card-box-shadow: 0 10px 20px #282a3017; + --md-card-border-color: rgba(0, 0, 0, 0.147); + --md-white-color: rgb(255, 255, 255); +} + +:root, [data-md-color-scheme=slate] { + --md-card-bg-color: rgba(90, 90, 90, 0.512); + --md-card-box-shadow: 0 10px 20px rgba(210, 210, 210, 0.315); + --md-card-border-color: rgba(83, 83, 83, 0.581); + --md-white-color: rgb(255, 255, 255); +} + +.side-container { + min-width: 10rem; + max-width: 40rem; + align-self: flex-start; + flex-shrink: 0; + width: 35%; + position: sticky; + top: 100px; + background-color: transparent; + z-index: 0 !important; +} + +@media screen and (max-width: 70em) { + .side-container { + display: none; + } +} + +.tutorial-sticky-image { + width: 100%; + height: 100%; + z-index: 0 !important; +} + +.tutorial-sticky-image-content { + width: 0; + height: 0; +} + +.md-main__inner.md-grid { + max-width: 75rem; +} + + +.md-header__inner { + max-width: 70rem !important; +} + +.md-nav__title { + text-transform: uppercase; + margin-left: 0 !important; +} + +.md-nav__link { + margin-left: 15px; +} + +#__nav_6_label, #__nav_5_label { + margin-left: 0 !important; +} + +.md-nav__item--section {/* md-nav__item--nested */ + margin-top: 40px; +} + +.tutorial_card h2 { + margin-top: 0; + margin-bottom: 10px; + font-weight: 500; +} + +.tutorial_card p { + margin: 0; +} + +.md-header__ellipsis { + justify-content: space-around; +} + +.header_home_btn { + background-color: transparent; + top: 10px; + margin: 0; + margin-left: 215px; + font-size: 16px; +} + +.md-header__topic { + font-weight: 700; +} + +.tutorial_card_container { + justify-content: center; + align-items: center; + display: grid; + grid-template-columns: repeat(auto-fill, minmax(200px, 1fr)); + gap: 20px; +} + +.tutorial_card_container .card { + background-color: var(--md-card-bg-color); + border: 1px solid var(--md-card-border-color); + border-radius: 6px; + display: flex; + flex-direction: column; + padding: 12px; + transition: all 0.3s ease, border 0.1s ease; + flex: 1; + align-self: stretch; + color: var(--md-typeset-color); +} + +.tutorial_card_container .card:hover { + border: 1px solid transparent; + box-shadow: var(--md-card-box-shadow); + text-decoration: none; + color: inherit !important; + transform: translateY(-2px); +} + +.tutorial_card_container .card .card_image { + background-position: center; + background-size: cover; + border-radius: 6px; + height: 80px; + padding: 10px; +} + +.tutorial_card_container .card .image1 { + background-image: url('https://raw.githubusercontent.com/gabrielraeder/website/main/docs/static/images/cards/card1_img.png'); +} + +.tutorial_card_container .card .image2 { + background-image: url('https://raw.githubusercontent.com/gabrielraeder/website/main/docs/static/images/cards/card2_img.png'); +} + +.tutorial_card_container .card .image3 { + background-image: url('https://raw.githubusercontent.com/gabrielraeder/website/main/docs/static/images/cards/card3_img.png'); +} + +.tutorial_card_container .card_text { + padding-top: 12px; + margin-left: 1px; +} + +.tutorial_card_container .card_title { + margin: 5px 0; + font-weight: bold; + font-size: 16.5px; +} + +.tutorial_card_container .card_text p { + margin: 8px 0 0 0; + min-height: 40px; + height: fit-content; +} + +.tutorial_card_container .card a { + border-radius: 15px; + font-weight: 700; + margin-top: 8px; + padding: 3px 0; + text-decoration: none; + transition: color 0.3s ease; + width: fit-content; +} + +.tutorial_card_container .card:hover a { + text-decoration: underline !important; + color: inherit !important; +} + +.tutorial_card_container .card p { + font-size: 14px; + line-height: 1.8em; + -webkit-font-smoothing: antialiased; + margin: 0 0 10px; + display: block; +} \ No newline at end of file diff --git a/docs/tutorial_images/bc-1-bc-implements-mlcubes.png b/docs/tutorial_images/bc-1-bc-implements-mlcubes.png new file mode 100644 index 000000000..026e2b511 Binary files /dev/null and b/docs/tutorial_images/bc-1-bc-implements-mlcubes.png differ diff --git a/docs/tutorial_images/bc-2-bc-develops-dataset.png b/docs/tutorial_images/bc-2-bc-develops-dataset.png new file mode 100644 index 000000000..d4b514c5a Binary files /dev/null and b/docs/tutorial_images/bc-2-bc-develops-dataset.png differ diff --git a/docs/tutorial_images/bc-3-bc-submits-mlcubes.png b/docs/tutorial_images/bc-3-bc-submits-mlcubes.png new file mode 100644 index 000000000..50ad3485a Binary files /dev/null and b/docs/tutorial_images/bc-3-bc-submits-mlcubes.png differ diff --git a/docs/tutorial_images/bc-4-bc-submits-data-preparator.png b/docs/tutorial_images/bc-4-bc-submits-data-preparator.png new file mode 100644 index 000000000..d22f3be97 Binary files /dev/null and b/docs/tutorial_images/bc-4-bc-submits-data-preparator.png differ diff --git a/docs/tutorial_images/bc-5-bc-submits-ref-model.png b/docs/tutorial_images/bc-5-bc-submits-ref-model.png new file mode 100644 index 000000000..9890e82e3 Binary files /dev/null and b/docs/tutorial_images/bc-5-bc-submits-ref-model.png differ diff --git a/docs/tutorial_images/bc-6-bc-submits-evalmetrics.png b/docs/tutorial_images/bc-6-bc-submits-evalmetrics.png new file mode 100644 index 000000000..ae1f88b21 Binary files /dev/null and b/docs/tutorial_images/bc-6-bc-submits-evalmetrics.png differ diff --git a/docs/tutorial_images/bc-7-bc-submits-benchmark.png b/docs/tutorial_images/bc-7-bc-submits-benchmark.png new file mode 100644 index 000000000..6823d9396 Binary files /dev/null and b/docs/tutorial_images/bc-7-bc-submits-benchmark.png differ diff --git a/docs/tutorial_images/do-1-do-prepares-datset.png b/docs/tutorial_images/do-1-do-prepares-datset.png new file mode 100644 index 000000000..bc20081fd Binary files /dev/null and b/docs/tutorial_images/do-1-do-prepares-datset.png differ diff --git a/docs/tutorial_images/do-2-do-registers-dataset.png b/docs/tutorial_images/do-2-do-registers-dataset.png new file mode 100644 index 000000000..a6871186f Binary files /dev/null and b/docs/tutorial_images/do-2-do-registers-dataset.png differ diff --git a/docs/tutorial_images/do-3-do-requests-participation.png b/docs/tutorial_images/do-3-do-requests-participation.png new file mode 100644 index 000000000..108bfec64 Binary files /dev/null and b/docs/tutorial_images/do-3-do-requests-participation.png differ diff --git a/docs/tutorial_images/do-4-bc-accepts-rejects-datasets.png b/docs/tutorial_images/do-4-bc-accepts-rejects-datasets.png new file mode 100644 index 000000000..ae49b9f86 Binary files /dev/null and b/docs/tutorial_images/do-4-bc-accepts-rejects-datasets.png differ diff --git a/docs/tutorial_images/do-5-do-runs-models.png b/docs/tutorial_images/do-5-do-runs-models.png new file mode 100644 index 000000000..ecf87bda8 Binary files /dev/null and b/docs/tutorial_images/do-5-do-runs-models.png differ diff --git a/docs/tutorial_images/do-6-do-submits-eval-results.png b/docs/tutorial_images/do-6-do-submits-eval-results.png new file mode 100644 index 000000000..7ed77e1ec Binary files /dev/null and b/docs/tutorial_images/do-6-do-submits-eval-results.png differ diff --git a/docs/tutorial_images/do-7-bc-reads-results.png b/docs/tutorial_images/do-7-bc-reads-results.png new file mode 100644 index 000000000..d0b876ba6 Binary files /dev/null and b/docs/tutorial_images/do-7-bc-reads-results.png differ diff --git a/docs/tutorial_images/mo-1-mo-implements-cube.png b/docs/tutorial_images/mo-1-mo-implements-cube.png new file mode 100644 index 000000000..436be7251 Binary files /dev/null and b/docs/tutorial_images/mo-1-mo-implements-cube.png differ diff --git a/docs/tutorial_images/mo-2-mo-submits-model.png b/docs/tutorial_images/mo-2-mo-submits-model.png new file mode 100644 index 000000000..d7de7a294 Binary files /dev/null and b/docs/tutorial_images/mo-2-mo-submits-model.png differ diff --git a/docs/tutorial_images/mo-3-mo-requests-participation.png b/docs/tutorial_images/mo-3-mo-requests-participation.png new file mode 100644 index 000000000..6ff264000 Binary files /dev/null and b/docs/tutorial_images/mo-3-mo-requests-participation.png differ diff --git a/docs/tutorial_images/mo-4-bc-accepts-rejects-models.png b/docs/tutorial_images/mo-4-bc-accepts-rejects-models.png new file mode 100644 index 000000000..00825179e Binary files /dev/null and b/docs/tutorial_images/mo-4-bc-accepts-rejects-models.png differ diff --git a/docs/tutorial_images/overview.png b/docs/tutorial_images/overview.png new file mode 100644 index 000000000..f466144c2 Binary files /dev/null and b/docs/tutorial_images/overview.png differ diff --git a/docs/tutorial_images/the-end.png b/docs/tutorial_images/the-end.png new file mode 100644 index 000000000..d91c64a50 Binary files /dev/null and b/docs/tutorial_images/the-end.png differ diff --git a/docs/what_is_medperf.md b/docs/what_is_medperf.md new file mode 100644 index 000000000..75b01a37a --- /dev/null +++ b/docs/what_is_medperf.md @@ -0,0 +1,52 @@ +# Introduction + +Medical Artificial Intelligence (AI) has the potential to revolutionize healthcare by advancing evidence-based medicine, personalizing patient care, and reducing costs. Unlocking this potential requires reliable methods for evaluating the efficacy of medical machine learning (ML) models on large-scale heterogeneous data while maintaining patient privacy. + +## What is MedPerf? + +MedPerf is an open-source framework for benchmarking ML models to deliver clinical efficacy while prioritizing patient privacy and mitigating legal and regulatory risks. It enables federated evaluation in which models are securely distributed to different facilities for evaluation. The goal of federated evaluation is to make it simple and reliable to share models with many data providers, evaluate those models against their data in controlled settings, then aggregate and analyze the findings. + +The MedPerf approach empowers healthcare organizations to assess and verify the performance of ML models in an efficient and human-supervised process without sharing any patient data across facilities during the process. + +## Why MedPerf? + +MedPerf reduces the risks and costs associated with data sharing, maximizing medical and patient outcomes. The platform leads to an effective, broader, and cost-effective adoption of medical ML and improves patient outcomes. + +Anyone who joins our platform can get several benefits, regardless of the role they will assume. + +**Benefits if you are a [Data Provider](roles.md#data-providers):** + +* Evaluate how well machine learning models perform on your population’s data. +* Connect to Model Owners to help them improve medical ML in a specific domain. +* Help define impactful medical ML benchmarks. + +**Benefits if you are a [Model Owner](roles.md#model-owners):** + +* Measure model performance on private datasets that you would never have access to. +* Connect to specific Data Providers that can help you increase the performance of your model. + +**Benefits if you own a benchmark ([Benchmark Committee](roles.md#benchmark-committee)):** + +* Hold a leading role in the MedPerf ecosystem by defining specifications of a benchmark for a particular medical ML task. +* Get help to create a strong community around a specific area. +* Rule point on creating the guidelines to generate impactful ML for a specific area. +* Help improve best practices in your area of interest. +* Ensure the choice of the metrics as well as the proper reference implementations. + +**Benefits to the Broad Community:** + +* Provide consistent and rigorous approaches for evaluating the accuracy of ML models for real-world use in a standardized manner. +* Enable model usability measurement across institutions while maintaining data privacy and model reliability. +* Connect with a community of expert groups to employ scientific evaluation methodologies and technical approaches to operate benchmarks that not only have well-defined clinical aspects, such as clinical impact, clinical workflow integration and patient outcome, but also support robust technical aspects, including proper metrics, data preprocessing and reference implementation. + +## What is a benchmark in the MedPerf perspective? + +A benchmark is a collection of assets used by the platform to test the performance of ML models for a specific clinical problem. The primary components of a benchmark are: + +1. **Specifications**: precise definition of the clinical setting (e.g., problem or task and specific patient population) on which trained ML models are to be evaluated. It also includes the labeling (annotation) methodology as well as the choice of evaluation metrics. +2. **Dataset Preparation**: a process that prepares datasets for use in evaluation, and can also test the prepared datasets for quality and compatibility. This is implemented as an MLCube (see [Data Preparator MLCube](mlcubes/mlcubes.md#data-preparator-mlcube)). +3. **Registered Datasets**: a list of registered datasets prepared according to the benchmark criteria and approved for evaluation use by their owners (e.g. patient data from multiple facilities representing (as a whole) a diverse patient population). +4. **Evaluation**: a consistent implementation of the testing pipelines and evaluation metrics. +5. **Reference Implementation**: a detailed example of a benchmark submission consisting of example model code, the evaluation component, and de-identified or synthetic publicly available sample data. +6. **Registered Models**: a list of registered models to run in this benchmark. +7. **Documentation**: documents for understanding and using the benchmark. diff --git a/mkdocs.yml b/mkdocs.yml index 42777b96d..1f766e15b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,28 +3,29 @@ site_url: https://medperf.com/ repo_url: https://github.com/mlcommons/medperf repo_name: mlcommons/medperf nav: -- Introduction: index.md +- What's Medperf?: what_is_medperf.md - Benchmark Roles: roles.md - Benchmark Workflow: workflow.md - MedPerf Components: medperf_components.md -- Getting Started: - - Overview: getting_started/overview.md - - Create an Account: getting_started/signup.md - - Installation: getting_started/installation.md - - Tutorials - User Guide: - - Introduction: getting_started/tutorials_overview.md +- GETTING STARTED: + - Roles Capabilities: getting_started/overview.md + - First Steps: + - Create an Account: getting_started/signup.md + - Installation: getting_started/installation.md - Setup: getting_started/setup.md - - Hands-on Tutorial for Bechmark Committee: getting_started/benchmark_owner_demo.md - - Hands-on Tutorial for Model Owners: getting_started/model_owner_demo.md - - Hands-on Tutorial for Data Owners: getting_started/data_owner_demo.md + - Tutorials: + - Introduction: getting_started/tutorials_overview.md + - Bechmark Committee: getting_started/benchmark_owner_demo.md + - Model Owners: getting_started/model_owner_demo.md + - Data Owners: getting_started/data_owner_demo.md + +- ADVANCED CONCEPTS: - MedPerf MLCubes: - Introduction: mlcubes/mlcubes.md - Creating Data Preparator MLCubes from Scratch: mlcubes/mlcube_data.md - Creating Model MLCubes from Scratch: mlcubes/mlcube_models.md - Creating Metrics MLCubes from Scratch: mlcubes/mlcube_metrics.md - Creating Model MLCubes from GaNDLF: mlcubes/gandlf_mlcube.md - -- Concepts: - Authentication: concepts/auth.md - Client Configuration: concepts/profiles.md - MLCube Components: concepts/mlcube_files.md @@ -34,13 +35,15 @@ nav: - Running Specific Models: concepts/single_run.md - CLI Reference: cli_reference.md # - Code Reference: reference/ -theme: +theme: + custom_dir: docs/overrides features: - search.suggest - search.highlight - search.share - content.code.annotate - content.code.copy + - navigation.sections icon: admonition: note: octicons/tag-16 @@ -122,4 +125,10 @@ markdown_extensions: - pymdownx.superfences - toc: permalink: True - - attr_list \ No newline at end of file + - attr_list + +extra_css: + - styles/styles.css + +extra_javascript: + - scripts/script.js \ No newline at end of file