Adding basic multi-account mlops usecase (#18)

Signed-off-by: Anton Kukushkin <[email protected]> Co-authored-by: fotinosk <[email protected]> Co-authored-by: Viktor Malesevic <[email protected]> Co-authored-by: kukushking <[email protected]> Co-authored-by: Leon Luttenberger <[email protected]>
awslabs · Mar 7, 2024 · 64fe4ed · 64fe4ed
1 parent 57ad3e7
commit 64fe4ed
Show file tree

Hide file tree

Showing 82 changed files with 4,088 additions and 5 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### **Added**
 
+- added `sagemaker-templates` module with `multi_account_basic` project template
 - bump cdk & ecr deployment version to fix deprecated custom resource runtimes issue in `mlflow-image`
 - added `sagemaker-jumpstart-fm-endpoint` module
 - added RDS persistence layer to MLFlow modules

diff --git a/README.md b/README.md
@@ -18,11 +18,12 @@ All modules in this repository adhere to the module structure defined in the the
 
 ### SageMaker Modules
 
-| Type                                                                                     | Description                                                                                                                                                                    |
-|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md)              | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group                                           |
-| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)                  | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config |
-| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md)     | Creates SageMaker Notebook Instances                                                                                                                                           |
+| Type                                                                                  | Description                                                                                                                                                                    |
+|---------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md)               | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config |
+| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md)           | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group                                           |
+| [SageMaker Project Templates Module](modules/sagemaker/sagemaker-templates/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog                                      |
+| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md)  | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace                                                                     |
 
 ### Mlflow Modules
 

diff --git a/manifests/deployment.yaml b/manifests/deployment.yaml
@@ -8,6 +8,8 @@ groups:
     path: manifests/storage-modules.yaml
   - name: sagemaker-studio
     path: manifests/sagemaker-studio-modules.yaml
+  - name: sagemaker-templates
+    path: manifests/sagemaker-templates-modules.yaml
   - name: images
     path: manifests/images-modules.yaml
   - name: mlflow

diff --git a/manifests/sagemaker-templates-modules.yaml b/manifests/sagemaker-templates-modules.yaml
@@ -0,0 +1,10 @@
+name: templates
+path: modules/sagemaker/sagemaker-templates
+targetAccount: primary
+parameters:
+  - name: portfolio-access-role-arn
+    valueFrom:
+      moduleMetadata:
+        group: sagemaker-studio
+        name: studio
+        key: LeadDataScientistRoleArn
diff --git a/modules/sagemaker/sagemaker-templates/README.md b/modules/sagemaker/sagemaker-templates/README.md
@@ -0,0 +1,54 @@
+# SageMaker Project Templates
+
+This module creates organizational SageMaker Project Templates. 
+
+The templates are registered in Service Catalog and available via SageMaker Studio Classic.
+
+### Architecture
+
+![SageMaker Templates Module Architecture](docs/_static/sagemaker-templates-module-architecture.png "SageMaker Templates Module Architecture")
+
+### Project Templates
+
+The module contains ogranizational SageMaker Project Templates vended as Service Catalog Products. Using the templates is available through SageMaker Studio Classic and AWS Service Catalog.
+
+#### Basic Multi-Account Template
+
+This project template contains an example of basic multi-account template from [AWS Enterprise MLOps Framework](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/README.md#sagemaker-project-stack).
+
+TODO: add detailed description and architecture diagram.
+
+## Inputs and outputs:
+### Required inputs:
+  - `portfolio-access-role-arn` - the ARN of the IAM Role used to access the Service Catalog Portfolio or SageMaker projects
+
+### Optional Inputs:
+  - `portfolio-name` - name of the Service Catalog Portfolio
+  - `portfolio-owner` - owner of the Service Catalog Portfolio
+
+### Sample manifest declaration
+
+```yaml
+name: templates
+path: modules/sagemaker/sagemaker-templates
+targetAccount: primary
+parameters:
+  - name: portfolio-access-role-arn
+    valueFrom:
+      moduleMetadata:
+        group: sagemaker-studio
+        name: studio
+        key: LeadDataScientistRoleArn
+```
+
+### Outputs (module metadata):
+  - `ServiceCatalogPortfolioName` - the name of the Service Catalog Portfolio
+  - `ServiceCatalogPortfolioOwner` - the owner of the Service Catalog Portfolio
+
+### Example Output:
+```yaml
+{
+  "ServiceCatalogPortfolioName": "MLOps SageMaker Project Templates",
+  "ServiceCatalogPortfolioOwner": "administrator"
+}
+```
diff --git a/modules/sagemaker/sagemaker-templates/app.py b/modules/sagemaker/sagemaker-templates/app.py
@@ -0,0 +1,56 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+
+import aws_cdk
+
+from stack import ServiceCatalogStack
+
+project_name = os.getenv("SEEDFARMER_PROJECT_NAME", "")
+deployment_name = os.getenv("SEEDFARMER_DEPLOYMENT_NAME", "")
+module_name = os.getenv("SEEDFARMER_MODULE_NAME", "")
+app_prefix = f"{project_name}-{deployment_name}-{module_name}"
+
+DEFAULT_PORTFOLIO_NAME = "MLOps SageMaker Project Templates"
+DEFAULT_PORTFOLIO_OWNER = "administrator"
+
+
+def _param(name: str) -> str:
+    return f"SEEDFARMER_PARAMETER_{name}"
+
+
+environment = aws_cdk.Environment(
+    account=os.environ["CDK_DEFAULT_ACCOUNT"],
+    region=os.environ["CDK_DEFAULT_REGION"],
+)
+
+portfolio_name = os.getenv(_param("PORTFOLIO_NAME"), DEFAULT_PORTFOLIO_NAME)
+portfolio_owner = os.getenv(_param("PORTFOLIO_OWNER"), DEFAULT_PORTFOLIO_OWNER)
+portfolio_access_role_arn = os.getenv(_param("PORTFOLIO_ACCESS_ROLE_ARN"))
+
+if not portfolio_access_role_arn:
+    raise ValueError("Missing input parameter portfolio-access-role-arn")
+
+app = aws_cdk.App()
+stack = ServiceCatalogStack(
+    app,
+    app_prefix,
+    portfolio_name=portfolio_name,
+    portfolio_owner=portfolio_owner,
+    portfolio_access_role_arn=portfolio_access_role_arn,
+)
+
+
+aws_cdk.CfnOutput(
+    scope=stack,
+    id="metadata",
+    value=stack.to_json_string(
+        {
+            "ServiceCatalogPortfolioName": stack.portfolio_name,
+            "ServiceCatalogPortfolioOwner": stack.portfolio_owner,
+        }
+    ),
+)
+
+app.synth()
diff --git a/modules/sagemaker/sagemaker-templates/coverage.ini b/modules/sagemaker/sagemaker-templates/coverage.ini
@@ -0,0 +1,3 @@
+[run]
+omit =
+    tests/*
diff --git a/modules/sagemaker/sagemaker-templates/deployspec.yaml b/modules/sagemaker/sagemaker-templates/deployspec.yaml
@@ -0,0 +1,23 @@
+publishGenericEnvVariables: true
+deploy:
+  phases:
+    install:
+      commands:
+      - env
+      # Install whatever additional build libraries
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    build:
+      commands:
+      - cdk deploy --require-approval never --progress events --app "python app.py" --outputs-file ./cdk-exports.json
+destroy:
+  phases:
+    install:
+      commands:
+      # Install whatever additional build libraries
+      - npm install -g [email protected]
+      - pip install -r requirements.txt
+    build:
+      commands:
+      # execute the CDK
+      - cdk destroy --force --app "python app.py"
diff --git a/modules/sagemaker/sagemaker-templates/images/zip-image/Dockerfile b/modules/sagemaker/sagemaker-templates/images/zip-image/Dockerfile
@@ -0,0 +1,3 @@
+FROM public.ecr.aws/docker/library/alpine:latest
+
+RUN apk --no-cache add zip make
diff --git a/modules/sagemaker/sagemaker-templates/pyproject.toml b/modules/sagemaker/sagemaker-templates/pyproject.toml
@@ -0,0 +1,38 @@
+[tool.black]
+line-length = 120
+target-version = ["py36", "py37", "py38"]
+exclude = '''
+/(
+    \.eggs
+  | \.git
+  | \.hg
+  | \.mypy_cache
+  | \.tox
+  | \.venv
+  | \.env
+  | _build
+  | buck-out
+  | build
+  | dist
+  | codeseeder.out
+)/
+'''
+
+[tool.isort]
+multi_line_output = 3
+include_trailing_comma = true
+force_grid_wrap = 0
+use_parentheses = true
+ensure_newline_before_comments = true
+line_length = 120
+py_version = 36
+skip_gitignore = false
+
+[tool.pytest.ini_options]
+addopts = "-v --cov=. --cov-report term --cov-config=coverage.ini --cov-fail-under=80"
+pythonpath = [
+  "."
+]
+testpaths = [
+  "tests"
+]
diff --git a/modules/sagemaker/sagemaker-templates/requirements.in b/modules/sagemaker/sagemaker-templates/requirements.in
@@ -0,0 +1,3 @@
+aws-cdk-lib==2.130.0
+cdk-nag==2.28.27
+boto3==1.34.35
diff --git a/modules/sagemaker/sagemaker-templates/requirements.txt b/modules/sagemaker/sagemaker-templates/requirements.txt
@@ -0,0 +1,84 @@
+#
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
+#
+#    pip-compile --output-file=requirements.txt requirements.in
+#
+attrs==23.2.0
+    # via
+    #   cattrs
+    #   jsii
+aws-cdk-asset-awscli-v1==2.2.202
+    # via aws-cdk-lib
+aws-cdk-asset-kubectl-v20==2.1.2
+    # via aws-cdk-lib
+aws-cdk-asset-node-proxy-agent-v6==2.0.1
+    # via aws-cdk-lib
+aws-cdk-lib==2.130.0
+    # via
+    #   -r requirements.in
+    #   cdk-nag
+boto3==1.34.35
+    # via -r requirements.in
+botocore==1.34.51
+    # via
+    #   boto3
+    #   s3transfer
+cattrs==23.2.3
+    # via jsii
+cdk-nag==2.28.27
+    # via -r requirements.in
+constructs==10.3.0
+    # via
+    #   aws-cdk-lib
+    #   cdk-nag
+exceptiongroup==1.2.0
+    # via cattrs
+importlib-resources==6.1.2
+    # via jsii
+jmespath==1.0.1
+    # via
+    #   boto3
+    #   botocore
+jsii==1.94.0
+    # via
+    #   aws-cdk-asset-awscli-v1
+    #   aws-cdk-asset-kubectl-v20
+    #   aws-cdk-asset-node-proxy-agent-v6
+    #   aws-cdk-lib
+    #   cdk-nag
+    #   constructs
+publication==0.0.3
+    # via
+    #   aws-cdk-asset-awscli-v1
+    #   aws-cdk-asset-kubectl-v20
+    #   aws-cdk-asset-node-proxy-agent-v6
+    #   aws-cdk-lib
+    #   cdk-nag
+    #   constructs
+    #   jsii
+python-dateutil==2.8.2
+    # via
+    #   botocore
+    #   jsii
+s3transfer==0.10.0
+    # via boto3
+six==1.16.0
+    # via python-dateutil
+typeguard==2.13.3
+    # via
+    #   aws-cdk-asset-awscli-v1
+    #   aws-cdk-asset-kubectl-v20
+    #   aws-cdk-asset-node-proxy-agent-v6
+    #   aws-cdk-lib
+    #   cdk-nag
+    #   constructs
+    #   jsii
+typing-extensions==4.10.0
+    # via
+    #   cattrs
+    #   jsii
+urllib3==1.26.18
+    # via botocore
+zipp==3.17.0
+    # via importlib-resources
diff --git a/modules/sagemaker/sagemaker-templates/setup.cfg b/modules/sagemaker/sagemaker-templates/setup.cfg
@@ -0,0 +1,28 @@
+[metadata]
+license_files =
+  LICENSE
+  NOTICE
+  VERSION
+
+[flake8]
+max-line-length = 120
+extend-ignore = E203, W503
+exclude =
+    .git,
+    __pycache__,
+    docs/source/conf.py,
+    old,
+    build,
+    dist,
+    .venv,
+    codeseeder.out,
+    bundle
+
+[mypy]
+python_version = 3.7
+strict = True
+ignore_missing_imports = True
+allow_untyped_decorators = True
+exclude =
+  codeseeder.out/|example/|tests/
+warn_unused_ignores = False