Incorporating Balochistan workflow to master #59

mathijshenquet · 2020-05-12T20:58:55Z

Resolves epidemics/covid#452

Allow overwriting region names in config.yml
Add Balochistan data to sites/balochistan/
Moves data to generated the main site to sites/main/
Made it possible to specify in config.yaml and estimates.csv when the model was updated/when the estimates were made.

NB: The sites/ directory would be how I would organize the different sites going forward.

Blocked by #49

Add region hierarchy inference, make Level an ordered enum (only in lib)

fix enum json encoding

lgtm-com · 2020-05-14T16:23:54Z

This pull request introduces 2 alerts when merging f426f16 into fa06024 - view on LGTM.com

new alerts:

2 for Unused import

hnykda · 2020-05-21T17:26:27Z

I believe this is superseded by switching to luigi. The way how this could be changed is by providing a different data-dir (or just it outputs directory) https://epidemicforecasting.slack.com/archives/CV93A58G7/p1590040992047800?thread_ts=1590019434.047600 . You can then have a config per site (or just provide the parameter on CLI)

Should I document it?

specific configs settings docs

hnykda · 2020-05-22T19:48:03Z

I tried to document how to have config per locations: 02883a7 .

Other than that, I think that the PR looks generally good. Does anyone have resources to rebase it to current master with luigi? I think the steps would be to:

move the data to https://github.com/epidemics/epimodel/tree/master/data-dir/inputs, directory balochistan (sites is also fine, but ideally consistent...)
move the parameters from config.yaml to a luigi-compatible balochistan.cfg as documented in the commit above (e.g. gleam_resample and so on)
keep the actual business logic changes as is in this PR

mathijshenquet · 2020-05-25T10:21:55Z

I will do that

hnykda · 2020-05-25T15:46:47Z

Cool. Any ETAs?

* fix test that results in a dtype of int64 * change types_to_json to be more architecture independent * make luigi and other shell scripts check out with LF

* parse spreadsheet based on colab code. * fix df-parsing issues * load gsheet with single url parameter * xml definition: minor refactor + add definitions for everything, properly scaled * xml definition: more getters/setters * add logic to import Gleam definition from a DataFrame * make separate class to generate definition from df * add scenario cartesian product * add support for definition param multipliers * rename scenario generation file * rows with no class are applied to all scenarios * store Regions in ScenarioGenerator df to avoid passing down rds * refactor/rename scenario classes * get first unittest.TestCase working * get pytest fixtures working with unittest * improve test compatibility with numpy * fix broken gleam definition tests * test region-based lookup * test foretold integration using mocks * separate out colab functions * add test for SimulationSet * fix SimulationSet bug * start testing DefinitionGenerator * DefinitionGenerator: test all global parameters * DefinitionGenerator: test compartment variables * DefinitionGenerator: fix issues with exception aggregation & add validation * DefinitionGenerator: finish testing exceptions * DefinitionGenerator: test multipliers * scenario integration testing + support name & id params * test that scenario xml output is correct * fix test error * ScenarioSet holds DefinitionGenerators, which manage filenames * integrate scenario with batch * make tests pass again * add config as input to SimulationSet * fix failing tests + better xml formatting * black formatting * start work on adding estimates to gleam definitions * parse and add estimates in scenario * update GleamDefinition interface for seeds * update most tests to handle estimates * add/update tests for estimates * estimates parsing & setting fully tested & functional * fix minor formatting bug in xml * move multi-seed setting logic to GleamDefinition * fix branch lgtm issues * use test default.xml + add initialCompartments * convert GleamDefinition tests to unittest + test xml output * fix lgtm alerts * client version in xml is 7.0 * test scenario Batch integration * scenario takes name from config, not params * add context manager for default xml path * move generate_simulations code to scenario & use new classes * fix test isolation problem with tmp_path * update dependencies & fix CI test failure * change GenerateGleamBatch task to use new logic * fix lint * move sims generation to Batch & change sims storage params * update config for new scenarios * enable GleamDefinition parse from string * update WebExport for new config & simulations format * fix lint * don't pickle RegionsDataset * bugfix + remove dill from dependencies * rename GenerateSimulationDefinitions => ExportSimulationDefinitions since it doesn't actually generate them * define Gleam parameters file in luigi config * clean up traces of old luigi tasks * gleam stores initial compartments in tenths * Use population to fill in susceptible initial compartment And XML stopped alphebetizing attrs for some reason * move part of ParseInput _get_region method to RegionDataset * web-export bugfixes * fix ExportSimulationDefinitions success file bug * update/refactor example-run.sh * fix gleam results path expansion issue + test config * add docs for manual inputs in readme

* the single_result parameters is actually interpreted as pointing to the result directory and not the actual result file * do not append the web-export directory name to the gcs path * change the GleamvizResults to be easier to configure

add a r estimation script and integrate with luigi

lgtm-com · 2020-05-28T13:54:53Z

This pull request introduces 8 alerts when merging 03e1b4e into 671562b - view on LGTM.com

new alerts:

3 for Unused import
2 for Wrong name for an argument in a call
2 for Testing equality to None
1 for Module is imported with 'import' and 'import from'

mathijshenquet · 2020-05-28T14:12:52Z

With this commit it has become impossible to reproduce the website builds using the hdf5 files in fixtures. The reason is the reworking of how the trace/group names work and how it expects the traces to be named in hdf5 and how it puts them in the exported json file.

The breakage for the normal site, renamed main here, is limited and I will make a issue about it - epidemics/covid#459. For balochistan it might be more severe and I have no idea how to approach it at this point.

hnykda · 2020-05-28T19:26:43Z

And do we still need it?

* add a way to have a overrides config to contain configurable secrets * rename overrides to secrets

# Conflicts: # README.md # epimodel/tasks.py

lgtm-com · 2020-06-02T09:31:20Z

This pull request introduces 8 alerts when merging 43c4db1 into 6131ea8 - view on LGTM.com

new alerts:

3 for Unused import
2 for Wrong name for an argument in a call
2 for Testing equality to None
1 for Module is imported with 'import' and 'import from'

gavento and others added 30 commits April 7, 2020 16:39

Update github testing (hopefully making it faster)

7b8d0b4

Fix github action script

ae25bd7

Update github workflow

585bfab

Add more merging code, fir region writing

12732b8

Add better level handling and infer region hierarchy

8e28a7d

Update to Level enum, add parent tests

8c0fdbd

masks model

cc90484

Merge pull request #22 from epidemics/region-structure

2c450de

Add region hierarchy inference, make Level an ordered enum (only in lib)

upadte gitignore to ignore data folder

fa63ba1

modify gitignore

0f3c055

minimum example for refactored code

2356777

black formatting

01612d7

Refactor/reshape the model library

96e6d8b

WIP on model library (model not finished yet)

6c85615

Blacked!

fe573fe

Fix ./do with Level, lint get-poetry.py, archive pycountry import

da6642d

fix enum json encoding

8767162

Update model library, add V2 and V2g models (make converge again)

ca8961e

Update v2_lib NB

ccc6180

Update notebook, add util function

1eb8d2d

Backed!

75e72d8

Remove duplicate checks

4362f53

Merge pull request #27 from epidemics/bug/fix-enum-encoding

e986ae0

fix enum json encoding

Minor model fixes

38ec28a

Update JH output file name

e99e0ff

Update model to match v2 NB

1c99114

Fix model param

d3db1d8

Add feature splitting into loader

09806b2

pipe summary

ceb64e3

readme update

9890b58

mathijshenquet requested review from hnykda and wolverdude May 14, 2020 15:46

luigi integration PR #60

39f4845

hnykda force-pushed the master branch from 899f854 to 39f4845 Compare May 19, 2020 06:59

mathijshenquet mentioned this pull request May 22, 2020

use created instead of generated since generated does not exist epidemics/covid#453

Closed

hnykda and others added 2 commits May 22, 2020 21:29

specific configs

0831578

specific configs settings docs #61

02883a7

specific configs settings docs

witzatom and others added 7 commits May 26, 2020 13:45

Minor compatibility issues (#63)

a8631d4

* fix test that results in a dtype of int64 * change types_to_json to be more architecture independent * make luigi and other shell scripts check out with LF

Pipeline fixes (#64)

f25c67b

* the single_result parameters is actually interpreted as pointing to the result directory and not the actual result file * do not append the web-export directory name to the gcs path * change the GleamvizResults to be easier to configure

R estimation (#65)

671562b

add a r estimation script and integrate with luigi

fix broken requirements in merge

f33f1f2

Merge branch 'master' into balochistan

5c23e75

Fix merge

03e1b4e

witzatom added 4 commits June 2, 2020 09:12

Luigi secrets (#69)

c79d8d7

* add a way to have a overrides config to contain configurable secrets * rename overrides to secrets

CO-448 add a overwrite flag to the WebExport task (#68)

58f7d67

formatting

6131ea8

Merge remote-tracking branch 'upstream/master' into balochistan

43c4db1

# Conflicts: # README.md # epimodel/tasks.py

MrinankSharma force-pushed the master branch from a6e4ba4 to 0b36157 Compare June 13, 2020 13:06

MrinankSharma closed this Sep 7, 2020

MrinankSharma deleted the balochistan branch September 7, 2020 14:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporating Balochistan workflow to master #59

Incorporating Balochistan workflow to master #59

mathijshenquet commented May 12, 2020 •

edited

Loading

lgtm-com bot commented May 14, 2020

hnykda commented May 21, 2020 •

edited

Loading

hnykda commented May 22, 2020

mathijshenquet commented May 25, 2020

hnykda commented May 25, 2020

lgtm-com bot commented May 28, 2020

mathijshenquet commented May 28, 2020 •

edited by witzatom

Loading

hnykda commented May 28, 2020

lgtm-com bot commented Jun 2, 2020

Incorporating Balochistan workflow to master #59

Incorporating Balochistan workflow to master #59

Conversation

mathijshenquet commented May 12, 2020 • edited Loading

lgtm-com bot commented May 14, 2020

hnykda commented May 21, 2020 • edited Loading

hnykda commented May 22, 2020

mathijshenquet commented May 25, 2020

hnykda commented May 25, 2020

lgtm-com bot commented May 28, 2020

mathijshenquet commented May 28, 2020 • edited by witzatom Loading

hnykda commented May 28, 2020

lgtm-com bot commented Jun 2, 2020

mathijshenquet commented May 12, 2020 •

edited

Loading

hnykda commented May 21, 2020 •

edited

Loading

mathijshenquet commented May 28, 2020 •

edited by witzatom

Loading