Initialize Django with admin, superuser, and existing config data

[thekaveman]

This depends on #1847

The following set of tasks/criteria should be performed and checked after the secrets refactor. Once complete, the app will officially be "live" with the Admin interface.

Acceptance Criteria

• bin/init.sh runs: python manage.py migrate, python manage.py compilemessages, and python manage.py collectstatic --no-input
• bin/init.sh no longer tries to delete/recreate the database file or a superuser
• A new script like bin/reset_db.sh contains logic for creating the database and superuser
• Remove the data migration, use fixtures for local/CI runs
• Remove terraform config for DJANGO_ADMIN, DJANGO_DB_RESET, DJANGO_SUPERUSER_* -- keep DJANGO_DB_DIR
• The ADMIN variable and associated if statements in settings.py go away -- we assume Django admin is enabled. Also update the test in https://github.com/cal-itp/benefits/blob/dev/tests/pytest/core/test_settings.py

Before merge #1881

• Configure DJANGO_DB_DIR for test to ensure database is saved in the file share
• Configure DJANGO_DB_DIR for prod to ensure database is saved in the file share
• Export existing data from dev database locally
• Run the bin/reset_db.sh for dev locally with SUPERUSER values from shared LastPass entry

After merge #1881

• Upload new dev database to Azure, reset site, test
• Clean up unused secrets in dev, see list below
• Export existing data from test database locally
• Run the bin/reset_db.sh for test locally with SUPERUSER values from shared LastPass entry
• Deploy dev to test Deploy to test #1909

After merge #1909

• Upload new test database to Azure, reset site, test
• Clean up unused secrets in test, see list below
• Export existing data from prod database locally
• Run the bin/reset_db.sh for prod locally with SUPERUSER values from shared LastPass entry
• Deploy test to prod

After deploy test to prod

• Upload new prod database to Azure, reset site, test
• Clean up unused secrets in prod, see list below

Export data

See the Django docs on the dumpdata management command.

For exporting dev, test, and prod data, this has to be done on the existing corresponding branch (before merging #1881 up) because that PR changes the structure of the existing databases. This causes a mismatch when the below command is run and the database schema is checked against the (since updated in #1881) model schema.

Temporarily rename the environment-specific database file, e.g. dev_django.db, to django.db, then follow the environment specific steps below.

dev:

Since we've already merged the basic Admin changes to dev (i.e. #1855, #1874) the below command excludes some data types from export that will be recreated automatically when the dev database is reinitialized with changes for #1881.

python manage.py dumpdata \
  --natural-foreign \
  --natural-primary \
  --exclude auth \
  --exclude contenttypes > dev_fixtures.json

test and prod:

Since these two environments do not have any Admin changes merged to them yet, the process will be slightly different. First the export command does not need to exclude any data types (since they don't exist anyway):

python manage.py dumpdata \
  --natural-foreign \
  --natural-primary > [env]_fixtures.json

And the resulting fixtures need to be hand-edited to account for the move to secret-backed fields in #1874:

PemData

1. Delete the value of all core.pemdata.fields.text entries (i.e. the certs/keys)
2. Rename this field to text_secret_name
3. Add the value of the corresponding Key Vault secret name as the value of these fields

AuthProvider

1. Delete the value of all core.authprovider.client_id entries (i.e. the actual client IDs)
2. Rename this field to client_id_secret_name
3. Add the value of the corresponding Key Vault secret name as the value of these fields

EligibilityVerifier

1. Delete the value of all core.eligibilityverifier.api_auth_key entries (i.e. the actual API auth keys)
2. Rename this field to api_auth_key_secret_name
3. Add the value of the corresponding Key Vault secret name as the value of these fields

(The last step only applies to the 2 Eligibility API verifiers: Courtesy Cards and Mobility Pass).

Reinitialize database with fixtures

Once the fixtures are generated for each environment, the following steps should be performed to reinitialize a local database from the environment configuration:

1. Update the .env with SUPERUSER values from the corresponding environment (from LastPass)
2. Update the .env to point to the corresponding environment fixtures file
3. Run bin/reset_db.sh to generate a new django.db from the environment fixtures
4. Keep note / temporarily rename django.db to reflect which environment it corresponds with
5. Before uploading to the corresponding environment in Azure, rename to django.db

Unused secrets

These secrets can be removed from Key Vault in all environments: dev, test, prod -- they are no longer used after the Admin is live in all environments:

sorted in alpha order to match Key Vault

auth-provider-authority
courtesy-card-verifier-active
courtesy-card-verifier-api-auth-header
courtesy-card-verifier-api-url
courtesy-card-verifier-jwe-cek-enc
courtesy-card-verifier-jwe-encryption-alg
courtesy-card-verifier-jws-signing-alg
courtesy-card-verifier-name
django-admin
django-db-reset
django-superuser-email
django-superuser-password
django-superuser-username
mobility-pass-verifier-active
mobility-pass-verifier-api-auth-header
mobility-pass-verifier-api-url
mobility-pass-verifier-jwe-cek-enc
mobility-pass-verifier-jwe-encryption-alg
mobility-pass-verifier-jws-signing-alg
mobility-pass-verifier-name
mst-agency-active
mst-agency-jws-signing-alg
mst-agency-long-name
mst-agency-short-name
mst-courtesy-card-group-id
mst-payment-processor-api-access-token-endpoint
mst-payment-processor-api-access-token-request-key
mst-payment-processor-api-access-token-request-val
mst-payment-processor-api-base-url
mst-payment-processor-card-tokenize-env
mst-payment-processor-card-tokenize-func
mst-payment-processor-card-tokenize-url
mst-payment-processor-name
mst-senior-group-id
mst-senior-verifier-active
mst-senior-verifier-name
mst-server-public-key-url
mst-veteran-group-id
mst-veteran-verifier-active
mst-veteran-verifier-name
sacrt-agency-active
sacrt-agency-jws-signing-alg
sacrt-agency-long-name
sacrt-agency-merchant-id
sacrt-agency-short-name
sacrt-payment-processor-api-access-token-endpoint
sacrt-payment-processor-api-access-token-request-key
sacrt-payment-processor-api-access-token-request-val
sacrt-payment-processor-api-base-url
sacrt-payment-processor-card-tokenize-env
sacrt-payment-processor-card-tokenize-func
sacrt-payment-processor-card-tokenize-url
sacrt-payment-processor-name
sacrt-senior-group-id
sacrt-senior-verifier-active
sacrt-senior-verifier-name
sbmtd-agency-active
sbmtd-agency-jws-signing-alg
sbmtd-agency-long-name
sbmtd-agency-merchant-id
sbmtd-agency-short-name
sbmtd-mobility-pass-group-id
sbmtd-payment-processor-api-access-token-endpoint
sbmtd-payment-processor-api-access-token-request-key
sbmtd-payment-processor-api-access-token-request-val
sbmtd-payment-processor-api-base-url
sbmtd-payment-processor-card-tokenize-env
sbmtd-payment-processor-card-tokenize-func
sbmtd-payment-processor-card-tokenize-url
sbmtd-payment-processor-name
sbmtd-senior-group-id
sbmtd-senior-verifier-active
sbmtd-senior-verifier-name
sbmtd-server-public-key-url
senior-auth-provider-claim
senior-auth-provider-client-name
senior-auth-provider-scheme
senior-auth-provider-scope
veteran-auth-provider-claim
veteran-auth-provider-client-name
veteran-auth-provider-scheme
veteran-auth-provider-scope

Additional context

Discussion around #1851 led us to decide that it was simpler to just move forward with enabling the Admin for good, vs. trying to make the code work for both the current situation (resetting the db, maybe/maybe not enabling Admin) and the new situation (Admin is always enabled, db is never reset, etc.)

[machikoyasuda]

This PR #1855 now completes the last to do item for this issue. It's necessary for the test to run properly.

[angela-tran]

@thekaveman Do you think this task should remove the data migration?

[thekaveman]

@thekaveman Do you think this task should remove the data migration?

Yes I think that makes sense! We might want to think about some kind of local setup e.g. fixtures that can be imported to get up and running locally. But for the production code, I think we won't need the data migrations.

[thekaveman]

Blocked until #1881 is merged.