MultiscaleRun is a Python package to run brain cells simulation at different scales. It orchestrates the coupling between several brain simulators like Neuron and STEPS but also solvers like AstroVascPy for the cerebral blood flow. The package also embeds a Julia solver to simulate the astrocytes activity.
The Python package includes a program called multiscale-run
that lets you run
and analyze multiscale simulations from start to finish.
This Python package requires Python 3.8 or higher.
Apart from supercomputers where MultiscaleRun may be provided as a module, the general way to install MultiscaleRun is via the pip
utility:
pip install multiscale-run
The multiscale-run
executable provides several commands to initialize, configure and execute simulations
multiscale-run init /path/to/my-sim
This command creates the following files in /path/to/my-sim
providing both the circuit, the configuration files, and the runtime dependencies:
circuit_config.json
: description of the circuit to simulatenode_sets.json
: subsets of cells acting as targets for different reports or stimulations. See also https://github.com/AllenInstitute/sonata/blob/master/docs/SONATA_DEVELOPER_GUIDE.md#node-sets-filesimulation_config.json
: ties together the definition of the simulation on the circuit, see section Simulation Configuration below to have an understanding of the dedicated "multiscale_run" section.simulation.sbatch
: An example of SLURM script to launch the simulation on BB5postproc.ipynb
: An example of Jupyter notebook making use of the simulation results to build a report
The generated setup is ready to compute, but feel free to browse and tweak the JSON configuration files at will,
especially the "multiscale_run" section of file simulation_config.json
📒 See
multiscale-run init --help
for more information
This command performs a series of check to identify common mistakes in the configuration. It is recommended to perform this operation before starting a simulation.
multiscale-run check [/path/to/my/sim]
multiscale-run compute [/path/to/my-sim]
📒 To use multiple ranks, use
srun -n X multiscale-run compute
where X is the number of ranks. Notice that steps requires this to be a power of 2.
📒 See
multiscale-run compute --help
for more information
❗ You may need to load the
intel-oneapi-mkl
module on BB5 if not already loaded otherwise you will probably experience the following error when running the compute phase:libmkl_intel_thread.so.1: undefined symbol: omp_get_num_procs
Three more folders will be created during the simulation:
cache
: it keeps some cached communication matrices, useful for subsequent runsmesh
: mesh files for steps. If the folder is missing a standard mesh will be generated Just In TimeRESULTS
: the results of the simulation. Various data are recorded here in HDF5 files. The variables are per-neuron based. MultiscaleRun mimics the structure of Neurodamus result files so that they can be post-processed with the same method
:For efficiency reasons, MultiscaleRun result files are filled with 0s at the beginning of the simulation. Thus, if the simulation dies early, these files will be full of 0s for the time steps after the simulation died
sbatch simulation.sbatch
This operation clones the Neurodamus mod files library for local editing.
multiscale-run edit-mod-files [/path/to/my-sim]
The command build_neurodamus.sh mod
may fail with the following error:
=> LINKING shared library ./libnrnmech.so
/usr/bin/ld: /path/to/install_oneapi-2023.2.0-skylake/neuron-9.0.a15-lrspl6/lib/libnrniv.so: unable to initialize decompress status for section .debug_info
/usr/bin/ld: /path/to/install_oneapi-2023.2.0-skylake/neuron-9.0.a15-lrspl6/lib/libnrniv.so: unable to initialize decompress status for section .debug_info
/path/to/install_oneapi-2023.2.0-skylake/neuron-9.0.a15-lrspl6/lib/libnrniv.so: file not recognized: File format not recognized
icpx: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [/path/to/install_oneapi-2023.2.0-skylake/neuron-9.0.a15-lrspl6/bin/nrnmech_makefile:133: mech_lib_shared] Error 1
This happens because Neuron was built with Intel oneAPI compiler but the compiler is not available in the environment. Loading the proper module on BB5 may fix the issue module load unstable intel-oneapi-compilers
- Setup a development environment with the
multiscale-run virtualenv
operation - Clone the bluepy-configfile repository
- Go to the repository and execute:
pip install .
- In the BlueConfig (BC) of interest comment out the
RunMode
field, theCircuit GLIA
section, theProjection NeuroGlia
section, and the theProjection GlioVascular
section. - Go back to the MultiscaleRun repo and run
blueconfig convert-to-sonata /gpfs/bbp.cscs.ch/project/proj62/Circuits/O1/20190912_spines/sonata/circuit_config.json ngv.json user.target BlueConfig
. The first argument, i.e. circuit_config, should point to an existing SONATA circuit config file, which can be found by searching the folder defined inCircuitPath
field of the BlueConfig. The second argument is the name of the SONATA simulation config file to be created from the BlueConfig. The third argument is the path to an existing SONATA nodesets file, and the fourth argument is the BlueConfig that we want to convert. - Now the
ngv.json
can be used from the Jupyter notebook for the visualizations.
- Load ARM MAP in the environment:
module load arm-forge
- Run the simulation:
map --profile --output OUT.map srun [...] multiscale-run compute [/path/to/my-sim]
- Open
OUT.map
with the Arm Forge Client (locally)
For more on how to use ARM MAP on BB5, please check this page.
- Remove upload gitlab logs as the CI logs are used only internally.
- New vasculature-segment-based reports are available now. Postproc and docs have been updated as well to address these new functionalities.
- Open-source release on GitHub https://github.com/BlueBrain/MultiscaleRun
- Publication of the documentation on https://multiscalerun.rtfd.io
- Move some of the GitLab CI/CD tasks to GitHub workflows.
- Add the command
multiscale_run stats
to retrieve some simple stats from the simulation. virtualenv
was not setting correctlyHOC_LIBRARY_PATH
when called from a python environment itself.
edit-mod-files
was not working anymore because neurodamus changed the position of themod
folder. Updated.edit-mod-files
was giving wrong suggestions since we should usebuild_neurodamus.sh mod --only-neuron
now.
- The configuration file has changed in the
connections
section:- you can specify where the connection takes place and if you want to write the connection results (
vals
) in the source simulator too. - matrices are now part of the
op
operation and can be used throughout the code. - A
dest_simulator
key is now required for every connection asconnect_to
keys do not dictate which simulator is the destination simulator anymore. - Previous
simulation_config.json
files must be adapted since backward compatibility is not possible. However, changes should be small and self-explanatory if a new template is compared to an old version (v 0.7,config_format
: 2) of the configuration file.
- you can specify where the connection takes place and if you want to write the connection results (
- New thorough verification of the MultiscaleRun configuration in file
simulation_config.json
during thecheck
andcompute
operations.
- Improve memory usage of class
multiscale_run.MsrConfig
- Add
update_currents.sh
in root for temporary storage. We still need to figure out where to put it or if we want to keep it at all. Sofia Farina and Alexis Arnaudon know more about this file.
- The configuration file
msr_config.json
does not exist anymore. All the MultiscaleRun configurations have been merged intosimulation_config.json
in a new root section:multiscale_run
. This is not yet SONATA. The configuration is in the same file but Neurodamus is not aware of it. [BBPP40-455]. - The configuration has changed significantly from 0.6. The documentation has now a dedicated page to explain how to use it.
- All the
magic numbers
in metabolism have been moved to the configuration or eliminated. For this reason, simulation behavior changed. Currently, neurons develop an ATP deficiency that breaks the simulations after a certain amount of time. This requires studies that will be incorporated in a future release.
- All the
magic numbers
that were used as glue among the various simulators have been moved to the configuration file (simulation_config.json
) or removed. - Value checking before a metabolism time step can now be changed from the config file. For example you can now say that if a neuron has an ATP that is lower than 0.25 it should be kicked out of the simulation and the simulation needs to continue without it. More information on how to do it is available in the configuration documentation.
- The metabolism model and metabolism object are not connected with fixed numbers anymore. Thus, if you want to change the model just a change in the configuration file is required.
- Connections among simulators can now be changed programmatically from the configuration. Just after a simulator (called
destination simulator
) does anadvance
the connection manager loops over its connections, picks thesource simulator
and performs the connections. Currently there are 3 types of connections:-
sum
sums the values from thesource
and thedestination
in the destination simulator. -
set
sets the values fromsource
intodestination
. -
merge
merges the two discording values$V_{\text{source}}$ and$V_{\text{destination}}$ using the formula:$V_{\text{source}} + V_{\text{destination}} - V_{n-1} = V_n$ .$V_n$ is then set in bothsource
anddestination
. This is the only connection type that also changes thesource
. Notice that swappingsource
anddestination
is not exactly equivalent because the sync is performed after adestination
advance
.
-
- The reporting structure has been changed significantly (mostly simplified and improved). However, these changes are mostly under the hood except for the changes in the configuration file. Consider reading the appropriate section in the documentation to know more MultiscaleRun reports.
- Added documentation for the configuration file.
- Added docstrings in the code.
bf
->bloodflow
keyword change for simpler future development [BBPP40-440].- Added type suggestions for function signatures in the code.
- Formatted docstrings for
sphinx
documentation. - Updated
postproc
to handle the new configuration. - Reworked the Python
import
statements and the initialization of the MPI library to fix possible hanging issues whenMPI_Finalize
was called. - Add static analysis of the code in the continuous-integration process.
- Fix a bug in the pytest CI that made it pass even if some tests were failing
- Fix hanging simulations when an error is thrown [BBPP40-427] when MPI is used
- virtualenv command:
- Improvement: the command now installs spack pkg if necessary. There is no need to run spack commands manually anymore. [!104]
- Fix issue when MultiscaleRun was loaded from BB5 module
py-multiscale-run
. [BBPP40-430]
- edit-mod-file command: load intel compilers module if required (!104)
- improved the documentation of the Python API
- Fix: init with julia create was not pointing to the correct location
- Improved README
msr_config.json
is a template now. Prepared for the bbp workflowbase_path
is now specified in the main config file. This is breaking change. Add to yourmsr_config.json
file a field:"base_path": "."
ndam
is now forced to useRoundRobin
- New commands:
edit-mod-files
: Clone the Neurodamus mod files library for local editingvirtualenv
: Create a Python virtual environment to ease development of MultiscaleRun Python package
- Fix possible program hang due to sensitive MPI initialization
- Improved README
- Added a section for
spack environments
- Increased standard mesh refinement for ratV10 to not have ranks without tets (omega_h cannot handle them)
- Align to bbp workflow
- new
post-processing
command creates an HTML report based on simulation results. Usage:multiscale-run post-processing [sim-path]
. Usemultiscale-run post-processing -h
for more information. init
command:- new option
--no-check
to skip the tests of Julia environment which lasts several minutes - new keys in
msr_config.json
:config_format
: an integer, the version of this file structuremsr_version
: a string indicating the version of MultiscaleRun that created this config
- new option
compute
command: now creates a Neurodamus success file at the end of the simulation- Compatibility notes for simulations created with previous versions of MultiscaleRun
- rename directory
julia_environment
to.julia_environment
- it is not possible to override the JSON configuration keys with environment variables.
- rename directory
- GitLab CI on BB5 now relies on spack
- Rework reporting [BBPP40-402, BBPP40-407, BBPP40-410, BBPP40-411, BBPP40-415]
First release of the code shipped as a Python package
Polina Shichkova, Alessandro Cattabiani, Christos Kotsalos, and Tristan Carel
The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.
Copyright (c) 2023-2024 Blue Brain Project/EPFL