chore: pin module versions; cleanup manifests (#226)

* pin module versions

* fix one-click-launch.yaml

* changelog

* remove "uber" manifest

* update example manifests

* add bedrock fine-tuning manifests

* update readme & changelog

CHANGELOG.md

@@ -10,19 +10,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### **Added**
 
 - added documentation for Ray on EKS manifests
-- Added network isolation and inter container encryption for xgboost template
-- Added partition support for modules:
+- added network isolation and inter container encryption for xgboost template
+- added partition support for modules:
   - `fmops/sagemaker-jumpstart-fm-endpoint`
   - `sagemaker/sagemaker-endpoint`
   - `sagemaker/sagemaker-notebook`
   - `sagemaker/sagemaker-studio`
+- added Bedrock fine-tuning manifest
 
 ### **Changed**
-- Added accelerate as extra for transformers in finetune llm template
-- Limited bucket name length in templates to avoid pipeline failures when using long project names
-- Increased timeout on finetune_llm_evaluation project from 1 hour (default) to 4 hours
-
+- added accelerate as extra for transformers in finetune llm template
+- limited bucket name length in templates to avoid pipeline failures when using long project names
+- increased timeout on finetune_llm_evaluation project from 1 hour (default) to 4 hours
 - pin `ray-operator`, `ray-cluster`, and `ray-image` modules versions
+- pin module versions for all manifests
 
 ## v1.4.0
 

README.md

@@ -23,13 +23,15 @@ See deployment steps in the [Deployment Guide](DEPLOYMENT.md).
 
 End-to-end example use-cases built using modules in this repository.
 
-| Type                                                                                        | Description                                                                                                                                                                                                  |
-|---------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [MLOps with Amazon SageMaker](manifests/mlops-sagemaker/)                                   | Set up environment for MLOps with Amazon SageMaker. Deploy secure Amazon SageMaker Studio Domain, and provisions SageMaker Project Templates using Service Catalog, including model training and deployment. |
-| [Mlflow experiments tracking with Amazon SageMaker](manifests/mlflow-experiments-tracking/) | An example using Mlflow experiments tracking with Amazon SageMaker. Deploy self-hosted Mlflow instance on AWS Fargate, and Amazon SageMaker Studio Domain environment.                                       |
-| [MWAA ML Training](manifests/mwaa-ml-training/)                                             | An example orchestrating ML training jobs with Managed Workflows for Apache Airflow (MWAA). Deploys MWAA and an example ML training DAG.                                                                     |
-| [Q&A RAG](manifests/fmops-qna-rag/)                                                         | Deploy AppSync GraphQL endpoint for Q&A chatbot with RAG based on OpenSearch, and data ingestion infrastructure.                                                                                             |
-| [Ray on EKS](manifests/ray-on-eks/)                                                         | Run Ray on AWS EKS. Deploys an AWS EKS cluster, KubeRay Ray Operator, and a Ray Cluster with autoscaling enabled.                                                                                            |
+| Type                                                                                                      | Description                                                                                                                                                                                                  |
+|-----------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [MLOps with Amazon SageMaker](manifests/mlops-sagemaker/)                                                 | Set up environment for MLOps with Amazon SageMaker. Deploy secure Amazon SageMaker Studio Domain, and provisions SageMaker Project Templates using Service Catalog, including model training and deployment. |
+| [Mlflow experiments tracking with Amazon SageMaker](manifests/mlflow-experiments-tracking/)               | An example using Mlflow experiments tracking with Amazon SageMaker. Deploy self-hosted Mlflow instance on AWS Fargate, and Amazon SageMaker Studio Domain environment.                                       |
+| [Managed Workflows with Apache Airflow (MWAA) for Machine Learning Training](manifests/mwaa-ml-training/) | An example orchestrating ML training jobs with Managed Workflows for Apache Airflow (MWAA). Deploys MWAA and an example ML training DAG.                                                                     |
+| [Q&A on PDF documents with RAG](manifests/fmops-qna-rag/)                                                 | Deploy AppSync GraphQL endpoint for Q&A chatbot with RAG based on OpenSearch, and data ingestion infrastructure.                                                                                             |
+| [Ray on Amazon Elastic Kubernetes Service (EKS)](manifests/ray-on-eks/)                                   | Run Ray on AWS EKS. Deploys an AWS EKS cluster, KubeRay Ray Operator, and a Ray Cluster with autoscaling enabled.                                                                                            |
+| [Bedrock Fine-Tuning with Step Functions](manifests/bedrock-finetuning-sfn/)                              | Continuously Fine-tune a Foundation Model with Bedrock Fine-Tuning jobs and AWS Step Functions.                                                                                                              |
+
 
 ## Modules
 

examples/manifests/deployment.yaml

@@ -16,6 +16,8 @@ groups:
     path: examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml
   - name: personas
     path: examples/manifests/personas-modules.yaml
+  - name: fmops
+    path: examples/manifests/fmops-modules.yaml
 targetAccountMappings:
   - alias: primary
     accountId:

examples/manifests/event-bus-modules.yaml

@@ -1,5 +1,5 @@
 name: event-bus
-path: modules/examples/event-bus
+path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/event-bus?ref=release/1.4.0&depth=1
 targetAccount: tooling
 parameters:
   - name: event_bus_name

manifests/fmops-modules.yaml → examples/manifests/fmops-modules.yaml

@@ -1,5 +1,5 @@
 name: jumpstart-hf-asrwhisper-endpoint
-path: modules/fmops/sagemaker-jumpstart-fm-endpoint
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-jumpstart-fm-endpoint?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: jump-start-model-name
@@ -17,4 +17,4 @@ parameters:
       moduleMetadata:
         group: networking
         name: networking
-        key: PrivateSubnetIds
+        key: PrivateSubnetIds

examples/manifests/networking-modules.yaml

@@ -1,5 +1,5 @@
 name: networking
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: primary
 parameters:
   - name: internet-accessible

examples/manifests/personas-modules.yaml

@@ -1,5 +1,5 @@
 name: personas
-path: modules/examples/personas
+path: git::https://github.com/awslabs/aiops-modules.git//modules/examples/personas?ref=release/1.4.0&depth=1
 parameters:
   - name: bucket-name
     value: my-bucket

examples/manifests/sagemaker-endpoints-modules.yaml

@@ -1,7 +1,7 @@
 # This is an example manifest group.
 # Replace the parameters with the parameters for your model below prior the deployment.
 name: endpoint
-path: modules/sagemaker/sagemaker-endpoint
+path: git::https://github.com/awslabs/aiops-modules.git//sagemaker/fmops/sagemaker-endpoint?ref=release/1.4.0&depth=1
 parameters:
   - name: sagemaker_project_id
     value: project-1

examples/manifests/sagemaker-hugging-face.yml

@@ -1,5 +1,5 @@
 name: hugging-face-mistral-endpoint
-path: modules/fmops/sagemaker-hugging-face-endpoint
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/sagemaker-hugging-face-endpoint?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: hugging-face-model-id
@@ -21,4 +21,4 @@ parameters:
       moduleMetadata:
         group: networking
         name: networking
-        key: PrivateSubnetIds
+        key: PrivateSubnetIds

examples/manifests/sagemaker-model-monitoring-modules.yaml

@@ -1,7 +1,7 @@
 # This is an example manifest group.
 # Replace the parameters with the parameters for your model below prior the deployment.
 name: monitor
-path: modules/sagemaker/sagemaker-model-monitoring
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/1.4.0&depth=1
 parameters:
   - name: sagemaker_project_id
     value: project-1
@@ -38,4 +38,4 @@ parameters:
       moduleMetadata:
         group: networking
         name: networking
-        key: PrivateSubnetIds
+        key: PrivateSubnetIds

examples/manifests/sagemaker-model-package-group-modules.yaml

@@ -1,5 +1,5 @@
 name: source-model-package-group
-path: modules/sagemaker/sagemaker-model-package-group
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-group?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: model_package_group_name

examples/manifests/sagemaker-model-package-promote-pipeline-modules.yaml

@@ -1,5 +1,5 @@
-name: rappi-b2
-path: modules/sagemaker/sagemaker-model-package-promote-pipeline
+name: model-pipeline
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-package-promote-pipeline?ref=release/1.4.0&depth=1
 targetAccount: tooling
 parameters:
   - name: source_model_package_group_arn

examples/manifests/sagemaker-notebook-modules.yaml

@@ -1,6 +1,5 @@
-
 name: notebook
-path: modules/sagemaker/sagemaker-notebook
+path: git::https://github.com/awslabs/aiops-modules.git//modules/modules/sagemaker/sagemaker-notebook?ref=release/1.4.0&depth=1
 parameters:
   - name: notebook_name
     value: dummy
@@ -17,4 +16,4 @@ parameters:
       moduleMetadata:
         group: networking
         name: networking
-        key: VpcId
+        key: VpcId

manifests/bedrock-finetuning-modules.yaml → manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml

@@ -1,5 +1,5 @@
 name: bedrock-finetuning
-path: modules/fmops/bedrock-finetuning
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/bedrock-finetuning?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: bedrock-base-model-ID
@@ -17,4 +17,4 @@ parameters:
         name: networking
         key: PrivateSubnetIds
   - name: removal-policy
-    value: DESTROY
+    value: DESTROY

manifests/bedrock-finetuning-sfn/deployment.yaml

@@ -0,0 +1,17 @@
+name: mlops
+toolchainRegion: us-east-1
+forceDependencyRedeploy: true
+groups:
+  - name: networking
+    path: manifests/bedrock-finetuning-sfn/networking-modules.yaml
+  - name: bedrock-finetuning
+    path: manifests/bedrock-finetuning-sfn/bedrock-finetuning-modules.yaml
+targetAccountMappings:
+  - alias: primary
+    accountId:
+      valueFrom:
+        envVariable: PRIMARY_ACCOUNT
+    default: true
+    regionMappings:
+      - region: us-east-1
+        default: true

manifests/networking-modules.yaml → manifests/bedrock-finetuning-sfn/networking-modules.yaml

@@ -1,5 +1,5 @@
 name: networking
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.7.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: primary
 parameters:
   - name: internet-accessible

manifests/fmops-qna-rag/networking-modules.yaml

@@ -1,5 +1,5 @@
 name: networking
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 parameters:
   - name: internet-accessible
     value: True

manifests/fmops-qna-rag/qna-rag-modules.yaml

@@ -1,5 +1,5 @@
 name: qna-rag
-path: modules/fmops/qna-rag
+path: git::https://github.com/awslabs/aiops-modules.git//modules/fmops/qna-rag?ref=release/1.4.0&depth=1
 parameters:
   - name: cognito-pool-id
     #Replace below value with valid congnito pool id

manifests/mlflow-experiments-tracking/images-modules.yaml

@@ -1,5 +1,5 @@
 name: mlflow-image
-path: modules/mlflow/mlflow-image
+path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-image?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: ecr-repository-name

manifests/mlflow-experiments-tracking/mlflow-modules.yaml

@@ -1,5 +1,5 @@
 name: mlflow-fargate
-path: modules/mlflow/mlflow-fargate
+path: git::https://github.com/awslabs/aiops-modules.git//modules/mlflow/mlflow-fargate?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: vpc-id

manifests/mlflow-experiments-tracking/networking-modules.yaml

@@ -1,5 +1,5 @@
 name: networking
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.7.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: primary
 parameters:
   - name: internet-accessible

manifests/mlflow-experiments-tracking/sagemaker-studio-modules.yaml

@@ -1,5 +1,5 @@
 name: studio
-path: modules/sagemaker/sagemaker-studio
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-studio?ref=release/1.4.0&depth=1
 targetAccount: primary
 parameters:
   - name: vpc_id

manifests/mlflow-experiments-tracking/storage-modules.yaml

@@ -21,7 +21,7 @@ parameters:
     value: DESTROY
 ---
 name: mlflow-mysql
-path: git::https://github.com/awslabs/idf-modules.git//modules/database/rds?ref=release/1.4.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/database/rds?ref=release/1.11.0&depth=1
 targetAccount: primary
 parameters:
   - name: vpc-id

manifests/mlops-sagemaker-multiacc/kernels-modules.yaml

@@ -1,5 +1,5 @@
 name: sagemaker-custom-kernel
-path: modules/sagemaker/sagemaker-custom-kernel/
+path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-custom-kernel?ref=release/1.4.0&depth=1
 targetAccount: dev
 parameters:
   - name: ecr-repo-name

manifests/mlops-sagemaker-multiacc/networking-modules.yaml

@@ -1,19 +1,19 @@
 name: networking-dev
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: dev
 parameters:
   - name: internet-accessible
     value: True
 ---
 name: networking-pre-prod
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: pre-prod
 parameters:
   - name: internet-accessible
     value: True
 ---
 name: networking-prod
-path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.3.0&depth=1
+path: git::https://github.com/awslabs/idf-modules.git//modules/network/basic-cdk?ref=release/1.11.0&depth=1
 targetAccount: prod
 parameters:
   - name: internet-accessible