COVID Redux

Modernizing the ETL of my largest project

Goals

• Transition all csv/xlsx output to database tables
• Transition manual diagnostic checking & etl to dbt+dagster
• Only clean and write updates for newest data

Tools

• Local

  • Python
  • Docker
  • Postgres
  • Dagster

• Cloud (soon)

  • AWS
  • PowerBI

Next steps

• Vitals data loaded and cleaned
• Vaccination data loaded and cleaned
• Dbt models reorganized
• Tests added to all scripts
• Weekly run set up
• Migrate local parquet storage to aws s3
• Migrate local postgres to aws rds
• Set up config to run on ECS cluster