Add NPI activation triggers to Gleam simulations

[gavento]

Description

We want to be able to plot simulations with NPIs triggered by the current situation.

NPI triggers are:

• Number of deaths in the last N days (e.g. N=14)
• Number of infections in the last N days (e.g. N=14)
  with different limits for activation and deactivation.

The NPIs do not have to be concrete, only the R reduction (resp. beta reduction) matters.

Design

(User interface has some details fleshed out)

• Have 4 levels of NPIs (0-3), where 0 is initial and has the refault R.
• Let forecasters generate estimates for trigger limits for all levels, and the R reduction factor, for each country.
• Forecasters input those into spreadsheet (google sheet), for any number of countries (or areas), downloaded as CSV
• Repeat cca 4 times (more is fine), starting with no triggered NPIs in the input spec file:
  • Run the model in Gleam with the latest batch file, import results into the batch file
  • Create a new batch file with 0-1 triggered NPI added for each country
    • For each country, see if some new trigger would be activated after the last one was
    • If so, add a new setting for beta (dep. on R reduction) into the XML def starting at that date
    • A new table triggered_NPIs in the batch HDF5 file is used to store which NPI levels are activated and when (hard to distinguish them from the exceptions in the XMLs)
    • Existing trigger activations get never modified, even if they would shift for some reason (randomness, migration from other country)
    • The sequence data is copied to the new batch file but truncated (relaced by NaNs) after when the new intervention would be added - so that we ca nuse it directly for web export and only have valid data there.

User interface and process

Template for NPI triggers doc (to be downloaded as csv)

The steps are marked [old] if they are exactly the same as without triggers, [new] if new or modified.

• [old] Create the HDF5 file A0.hdf5 (example name) with scenarios with GenerateGleamBatch as usual (from base def XML, infections, ...)
• [old] Export the definitions in A0.hd5 to GLEAM with ExportSimulationDefinitions
• [old] Run GLEAM for the exported simulations
• [old] Import simulation results with ExtractSimulationsResults into A0.hdf5
• [new] Download the triggers spreadsheet as CSV, (example name triggers.csv)
• [new] Create batch file A1.hdf5 from A0.hdf5 and triggers.csv with UpdateBatchTriggers (new task)
  • This reports when the new triggers are added and for how many countries
  • After this, A1 trajectories are same as in A0 but truncated (with NaNs or -1) after the newly added trigger (as they are invalid).

Repeat few times (suggestion: 3) for N=1,2,3,...

• [old] Export the definitions in A(N).hd5 to GLEAM with ExportSimulationDefinitions
• [old] Run GLEAM for the exported simulations
• [old] Import simulation results with ExtractSimulationsResults into A(N).hdf5
• [new] Create batch file A(N+1).hdf5 from A(N).hdf5 and triggers.csv with UpdateBatchTriggers (new task)
  • This reports when the new triggers are added and for how many countries
  • After this, A(N+1) trajectories are same as in A(N) but truncated (with NaNs or -1) after the newly added trigger (as they are invalid).

Then:

• [old] Use the last created file A(N+1).hdf5 in WebExport

Problems and caveats

• Each subsequent trigger added for one country may influence development in other countries that already added trigger far in future. This should not be a big shift, I would ignore it.
• Having the deactivation limit too close to activation may trigger fast on-off cycle, requiring a lot of gleam repeats.
• The triggers activate for the actual number of infections, not observed. The forecasters need to take this into account.

[gavento]

Notes: Commands to execute the loop:

# Add one iteration of triggers
poetry run ./run_luigi UpdateNPITriggers --input-batch data-dir/outputs/gleam-models.hdf5 --output-batch data-dir/outputs/gleam-models-1.hdf5 --NPITriggers-triggers ~/download/Example\ NPI\ triggers\ -\ Sheet1.csv

# Force Luigi to re-export the (new) batch defs
rm ~/GLEAMviz-data/sims/ExportSimulationDefinitions.success
poetry run ./run_luigi ExportSimulationDefinitions --GenerateGleamBatch-generated-batch-filename data-dir/outputs/gleam-models-1.hdf5

### <run gleam>

# Import gleam results
./run_luigi ExtractSimulationsResults --models-file data-dir/outputs/gleam-models-1-ran.hdf5 --GenerateGleamBatch-generated-batch-filename data-dir/outputs/gleam-models-1.hdf5

# Repeat from the start for a while ...

poetry run ./run_luigi WebExport --overwrite --WebExport-export-name test-t1 --ExtractSimulationsResults-models-file data-dir/outputs/gleam-models-2.hdf5
poetry run ./run_luigi WebUpload --WebUpload-channel test-t1 --WebUpload-exported-data data-dir/outputs/web-exports/test-t1