WIP: Memory allocation #59
Conversation
|
Hi @wytbella, now #55 has been merged there has been a fairly substantial change to |
|
We've got what we needed from this - thanks very much @wytbella. We'll keep the branch alive and can merge in these changes in the future on an ad-hoc basis to rerun the memory profiling. |
|
Hi @claireh93 , sorry about the delay. I meant to include the latest development on Before I spend more time to solve the python issue, just want to ask how important it is to profile the latest changes? As mentioned in the meeting, I think this PR provides more of a tooling/reference rather than a concrete conclusion (sorry about that!). It includes examples to use |
|
Hi @wytbella, yes I think you are right - it's more important that we have the tools on this branch so that we can run them to explore the code as it changes, so I wouldn't worry about running it if there are python problems! As Richard said, it would be great to chat about how these things work at some point in the future, but there is no rush :) |
* dev: (150 commits)
Revert rand MedianGenerator Binomial removal.
changes from Richard's PR review
Apply suggestions from code review
Add codecov reports
Remove old unused EpiList constructor.
Fix testing and examples to new EpiList API.
Switch to new API for EpiList so that human_abun and virus_abun store all information about human and virus states in a dataframe so disease_classes is no longer needed. Introduce DiseaseState enum to define Susceptible, Infectious, etc. Other small fixes: - first Susceptible age group is now Susceptible1 for consistency - virus needs initialisation with a vector like humans when there are age categories
rand call didnt have rng
Update SEI3HRD_example to new movement kernel format - number grid cells instead of classes
things missed from merge
lower case kwarg
needed type anntations for ::Type{R} where R<:Random.AbstractRNG & VecRNGType
Remove redundant parameter setting
Fix commit problem
Fix data frames so they are easier to read.
work in progress
Tidying and clarification
Fixing documentation and tidying
starting work to merge in from dev
Use dot product instead of allocating
...
# Conflicts:
# Project.toml
# examples/Epidemiology/SEI3HRD_example.jl
# examples/Epidemiology/Scotland_run.jl
# src/Epidemiology/EpiGenerate.jl
# src/Simulation.jl
|
@claireh93 - I've updated this to the latest version of dev. There's still a lot of work to do here (happily?) when someone has time. |
Codecov Report
@@ Coverage Diff @@
## dev #59 +/- ##
==========================================
+ Coverage 61.97% 62.20% +0.22%
==========================================
Files 46 46
Lines 2735 2741 +6
==========================================
+ Hits 1695 1705 +10
+ Misses 1040 1036 -4
|
To look into issue: #51
I tracked the time/memory use in
update!, here are some initial results onScotland_run.jlfor single thread (see the table below)────────────────────────────────────────────────────────────────────────────── Time Allocations ────────────────────── ─────────────────────── Tot / % measured: 597s / 53.7% 38.0GiB / 68.4% Section ncalls time %tot avg alloc %tot avg ────────────────────────────────────────────────────────────────────────────── virusupdate! 90 227s 70.7% 2.52s 25.4GiB 97.5% 289MiB firstloop 2.70k 172s 53.7% 63.7ms 22.7GiB 87.4% 8.62MiB birth draw 146M 71.2s 22.2% 487ns 10.9GiB 41.9% 80.0B virusmove 2.69M 35.0s 10.9% 13.0μs 0.00B 0.00% 0.00B death draw 146M 25.0s 7.80% 171ns 3.06GiB 11.8% 22.4B rest virus update 90 668ms 0.21% 7.42ms 112MiB 0.42% 1.24MiB classupdate! 90 90.2s 28.2% 1.00s 586MiB 2.20% 6.51MiB invalidatecaches! 90 3.07s 0.96% 34.1ms 0.00B 0.00% 0.00B habitatupdate! 90 544ms 0.17% 6.05ms 67.4MiB 0.25% 767KiB applycontrols! 90 39.7μs 0.00% 441ns 0.00B 0.00% 0.00B ──────────────────────────────────────────────────────────────────────────────@claireh93 @richardreeve , the tool (TimerOutputs.jl) I used doesn't seem to support multi-threading well, so I might have to use other tools for further investigation. Just want to ask, does the result (the table above) on a single thread give you any insights? Any suggestions on what's next?
Edit: some results from 1 or 2 threads (TimerOutputs could work if I don't track in details
Update to new dev:
──────────────────────────────────────────────────────────────────────────────── Time Allocations ────────────────────── ─────────────────────── Tot / % measured: 252s / 36.5% 54.1GiB / 44.3% Section ncalls time %tot avg alloc %tot avg ──────────────────────────────────────────────────────────────────────────────── virusupdate! 60 83.8s 90.9% 1.40s 19.5GiB 81.3% 333MiB secondloop 31.5M 69.1s 74.9% 2.19μs 15.4GiB 64.2% 524B loop 31.5M 18.3s 19.8% 581ns 4.59GiB 19.1% 156B Poisson 4.04M 3.10s 3.36% 767ns 185MiB 0.75% 48.0B Poisson rand 4.04M 1.38s 1.49% 341ns 61.7MiB 0.25% 16.0B Poisson crea... 4.04M 855ms 0.93% 212ns 123MiB 0.50% 32.0B binomial2 31.5M 8.58s 9.31% 272ns 1.88GiB 7.83% 64.0B binomial 31.5M 6.67s 7.23% 212ns 0.94GiB 3.92% 32.0B alloc 31.5M 3.24s 3.52% 103ns 2.82GiB 11.8% 96.0B traitfn 31.5M 2.59s 2.81% 82.4ns 0.00B 0.00% 0.00B .assign 31.5M 2.17s 2.35% 68.9ns 0.00B 0.00% 0.00B firstloop 180 7.62s 8.26% 42.3ms 1.94GiB 8.08% 11.0MiB birth draw 14.4M 5.09s 5.52% 352ns 1.08GiB 4.50% 80.1B virusmove 1.13M 171ms 0.19% 152ns 0.00B 0.00% 0.00B rest virus update 60 436ms 0.47% 7.27ms 140MiB 0.57% 2.33MiB classupdate! 60 7.50s 8.13% 125ms 4.41GiB 18.4% 75.3MiB habitatupdate! 60 735ms 0.80% 12.3ms 62.0MiB 0.25% 1.03MiB invalidatecaches! 60 122ms 0.13% 2.04ms 0.00B 0.00% 0.00B applycontrols! 60 18.1μs 0.00% 302ns 0.00B 0.00% 0.00B ────────────────────────────────────────────────────────────────────────────────