Build and deploy Dagster AMI #924
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Created docker-compose and .env templates/files for running dagster in containers. Created a DAGSTER_VERSION default. Created testing infrastructure for running dagster+docker-compose along side a traditional dagster deployment without being disruptive. Created a dagster ami based on the docker-baseline. Created an ugly non-minimal vault policy for dagster docker-compose servers. Adding a missing environment variable to the dagster .env template file. Added nginx with basic auth infront of dagster. fix: Resolve basic auth and env issues with Dagster - The basic auth config for Dagster was not using a properly formatted htpasswd file - Add explicit reference to the env file for dagit/dagster-daemon in the docker-compose file Paramtizer the dagster bucket value that will be shared with dagster.yaml. Squashed a few bugs. Updated the version of dagster. Removed the legacy salt-stack code from the dagster stacks. Adding the .env file to the edx-pipeline. Tweaking the dagster stack to use fewer security groups and be a bit neater. Fixed a templating problem with the xpro pipeline. Fixed another templating problem with the xpro pipeline. Fixed a few copy and paste errors with the mitxonline template. Created production configuration for dagster and fixed a few changed stack keys in the consul data.
…ased on the environment in which theny are running.
…d added a few missing vault permissions for production.
blarghmatey
commented
Jul 11, 2022
src/bilder/images/dagster/files/pipelines/edx-pipeline/micromasters.yaml.tmpl
Outdated
Show resolved
Hide resolved
src/bilder/images/dagster/files/pipelines/edx-pipeline/mitx_bigquery.yaml.tmpl
Outdated
Show resolved
Hide resolved
src/bilder/images/dagster/files/pipelines/edx-pipeline/mitxonline_edx.yaml.tmpl
Outdated
Show resolved
Hide resolved
...der/images/dagster/files/pipelines/edx-pipeline/open-discussions-enrollment-update.yaml.tmpl
Outdated
Show resolved
Hide resolved
src/bilder/images/dagster/files/pipelines/edx-pipeline/open-discussions.yaml.tmpl
Outdated
Show resolved
Hide resolved
blarghmatey
commented
Jul 11, 2022
…secret-data/data-qa to secret-data/data in vault.
blarghmatey
commented
Jul 12, 2022
src/bilder/images/dagster/files/pipelines/edx-pipeline/mitxonline_edx.yaml.tmpl
Outdated
Show resolved
Hide resolved
src/ol_infrastructure/applications/dagster/dagster_server_policy.hcl
Outdated
Show resolved
Hide resolved
|
👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updated the Dagster deployment to pre-bake an AMI and deploy using docker containers
Description
This adds code for building an AMI to run the Dagster data orchestrator using the containers being created as part of mitodl/ol-data-platform#240 and updates the Pulumi code to use that new image for deployment
Motivation and Context
This is related to mitodl/ol-data-platform#233
How Has This Been Tested?
The built image has been deployed and validated that the UI is accessible and pipelines render properly.
Types of changes
Checklist:
Closed #900