SpeedRun 1A: S3 Data Lake

[aaronsteers]

As a training tool, as a test for ease-of-use, and as proof of value, we're creating a "speed run" video that demonstrates how to get up and running quickly with the Infrastructure Catalog and a basic DataOps pipeline. This will uncover usability issues and bugs which we'll need to resolve before we can promote the platform broadly.

---

Stop Point:

• all infrastructure is deployed, including the data lake, the VPC, and the public/private subnets

Start Point:

• one-time setup:

  • Installed software:

    • choco install vscode python3 docker awscli github-desktop
    • choco install git.install --params "/GitOnlyOnPath /SChannel /NoAutoCrlf /WindowsTerminal"

  • Access to LinuxAcademy, will be used to create a new 4-hour limited AWS account

• environment setup (each time):
  • Hourglass app open with timer title: SpeedRun Goal: 12 Minutes
  • Open browser tabs:
    • the speedrun checklist (this page)
    • linux academy playbround
    • slalom-ggp/dataops-project-template

---

Speed Target: 10 minutes

---

Other Details:

• Must stay on each screen for at least 3 seconds
• Scale browser windows and VS Code to: ~125%

Blockers:

• None!

---

Steps:

Create Repo and AWS Account (0:00-2:00, approx. 2m):

• Create new repo from the Slalom DataOps Template, clone repo locally and open in VS Code (60s)
• Get AWS credentials from LInux Academy (30s)
• Use the linux-academy link to log in to AWS in the web browser (30s)

Configure Creds (2:00-3:00, approx. 1m):

• In the .secrets folder, rename credentials.template to .secrets/credentials, copy-paste credentials into file (30s)
• In the .secrets folder, rename aws-secrets-manager-secrets.yml.template to aws-secrets-manager-secrets.yml (no addl. secrets needed in this exercise) (30s)

Configure Project (3:00-3:30, approx 0.5m):

• Rename infra-config-template.yml to infra-config.yml - update email address and project shortname (30s)

Configure and Deploy Terraform (3:30-7:30, approx. 4m):

• Open the infra folder, review and update each file (90s)
• Run terraform init and terraform apply, type 'yes' (30s)
• Wait for terraform apply to complete (2m)
  • Switch to the git tab, review code changes while apply is running

Confirm resource creation (7:30-9:30, approx. 2m):

• Copy-paste and run the provided AWS User Switch command so aws-cli can locate our AWS credentials (30s)
• Upload infra-config.yml to the data bucket: aws s3 cp infra-config.yml s3://... (30s)
• List the bucket contents with aws s3 ls s3://... (30s)
• In the web browser, browse to the bucket and confirm the file has landed. (30s)
• Stop the time once the transfer is successfully confirmed. (DONE!)

[aaronsteers]

Completed in 14 minutes. Slower primarily due to setup, cloning, and logging into LA. Was already 2 minutes behind by the minute 3 benchmark.