-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #351 from SainsburyWellcomeCentre/datajoint_pipeline
Datajoint pipeline PR into `main` branch
- Loading branch information
Showing
85 changed files
with
20,467 additions
and
3,780 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,84 +1,84 @@ | ||
| # aeon_mecha | ||
|  | ||
| [](https://codecov.io/gh/SainsburyWellcomeCentre/aeon_mecha) | ||
|
|
||
| Project Aeon's main repository for manipulating acquired data. Includes modules for loading raw data, performing quality control on raw data, processing raw data, and ingesting processed data into a DataJoint MySQL database. | ||
|
|
||
| ## Set-up Instructions | ||
|
|
||
| The various set-up tools mentioned below do some combination of python version, environment, package, and package dependency management. For basic information on the differences between these tools, see this [blog post](https://dev.to/bowmanjd/python-tools-for-managing-virtual-environments-3bko#hatch). | ||
|
|
||
| ### Remote set-up on SWC's HPC | ||
|
|
||
| #### Prereqs | ||
|
|
||
| 1. Ssh into the HPC and clone this repository to your home directory. | ||
| ``` | ||
| ssh <your_SWC_username>@ssh.swc.ucl.ac.uk | ||
| mkdir ~/ProjectAeon | ||
| cd ~/ProjectAeon | ||
| git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha | ||
| cd aeon_mecha | ||
| ``` | ||
|
|
||
| #### Set-up | ||
|
|
||
| Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below. | ||
|
|
||
| [Option 1](./docs/env_setup/remote/miniconda_conda_remote_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager) | ||
|
|
||
| - *Note*: [mamba](https://mamba.readthedocs.io/en/latest/), a faster alternative to conda, is now installed as a module on the HPC, so the above instructions can be followed using 'mamba' instead of 'conda' if you prefer. | ||
|
|
||
| [Option 2](./docs/env_setup/remote/pip_venv_remote_setup.md): **pip** (python package manager) and **venv** (python environment manager) | ||
|
|
||
| ### Local set-up | ||
|
|
||
| #### Prereqs | ||
|
|
||
| All commands below should be run in a bash shell (Windows users can use the 'mingw64' terminal that is included when installing git). | ||
|
|
||
| 1. Clone this repository: create a 'ProjectAeon' directory in your home directory, clone this repository there, and `cd` into the cloned directory: | ||
| ``` | ||
| mkdir ~/ProjectAeon | ||
| cd ~/ProjectAeon | ||
| git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha | ||
| cd aeon_mecha | ||
| ``` | ||
|
|
||
| #### Set-up | ||
|
|
||
| Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below. | ||
|
|
||
| [Option 1](./docs/env_setup/local/miniconda_conda_local_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager) | ||
|
|
||
| - *Note*: **mambaforge** and **mamba** can be used as faster, drop-in replacements for 'miniconda' and 'conda', respectively. You can set up the Aeon environment using them, following roughly the same instructions as above. See [here](https://biapol.github.io/blog/mara_lampert/getting_started_with_mambaforge_and_python/readme.html) for more info. | ||
|
|
||
| [Option 2](./docs/env_setup/local/pip_venv_local_setup.md): **pip** (python package manager) and **venv** (python environment manager) | ||
|
|
||
| ## Repository Contents | ||
|
|
||
| - `.github/workflows/` : GitHub actions workflows for building the environment and running tests | ||
| - `aeon/` : Source code for the Aeon Python package | ||
| - `aeon/dj_pipeline`: Source code for the Aeon DataJoint MySQL database pipeline | ||
| - `aeon/io`: Source code for loading raw data | ||
| - `aeon/processing`: Source code for processing raw data | ||
| - `aeon/qc`: Source code for quality control of raw data | ||
| - `aeon/schema`: Examples of 'experiment schemas': variables that can be used to load raw data from particular experiments | ||
| - `docker/` : Dockerfiles for building Docker images for the Aeon DataJoint MySQL database pipeline. | ||
| - `docs/` : Documentation for the Aeon project | ||
| - `docs/devs/` : Documentation for developers | ||
| - `docs/env_setup/` : Documentation for setting up the Aeon Python environment | ||
| - `docs/examples/` : Aeon usecase examples | ||
| - `docs/using_hpc_jupyterhub.md` : Instructions for using Jupyter notebooks to access Aeon data via SWC's HPC | ||
| - `docs/using_online_dashboard.md` : Instructions for connecting to Aeon's online dashboard | ||
| - `env_config/` : Configuration files for the Aeon Python environment | ||
| - `tests/` : Unit and integration tests | ||
| - `tests/data` : Data used by tests | ||
|
|
||
| ## Citation Policy | ||
|
|
||
| If you use this software, please cite it as below: | ||
|
|
||
| Sainsbury Wellcome Centre Foraging Behaviour Working Group. (2023). Aeon: An open-source platform to study the neural basis of ethological behaviours over naturalistic timescales, https://doi.org/10.5281/zenodo.8411157 | ||
|
|
||
| [](https://zenodo.org/doi/10.5281/zenodo.8411157) | ||
| # aeon_mecha | ||
|  | ||
| [](https://codecov.io/gh/SainsburyWellcomeCentre/aeon_mecha) | ||
|
|
||
| Project Aeon's main repository for manipulating acquired data. Includes modules for loading raw data, performing quality control on raw data, processing raw data, and ingesting processed data into a DataJoint MySQL database. | ||
|
|
||
| ## Set-up Instructions | ||
|
|
||
| The various set-up tools mentioned below do some combination of python version, environment, package, and package dependency management. For basic information on the differences between these tools, see this [blog post](https://dev.to/bowmanjd/python-tools-for-managing-virtual-environments-3bko#hatch). | ||
|
|
||
| ### Remote set-up on SWC's HPC | ||
|
|
||
| #### Prereqs | ||
|
|
||
| 1. Ssh into the HPC and clone this repository to your home directory. | ||
| ``` | ||
| ssh <your_SWC_username>@ssh.swc.ucl.ac.uk | ||
| mkdir ~/ProjectAeon | ||
| cd ~/ProjectAeon | ||
| git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha | ||
| cd aeon_mecha | ||
| ``` | ||
|
|
||
| #### Set-up | ||
|
|
||
| Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below. | ||
|
|
||
| [Option 1](./docs/env_setup/remote/miniconda_conda_remote_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager) | ||
|
|
||
| - *Note*: [mamba](https://mamba.readthedocs.io/en/latest/), a faster alternative to conda, is now installed as a module on the HPC, so the above instructions can be followed using 'mamba' instead of 'conda' if you prefer. | ||
|
|
||
| [Option 2](./docs/env_setup/remote/pip_venv_remote_setup.md): **pip** (python package manager) and **venv** (python environment manager) | ||
|
|
||
| ### Local set-up | ||
|
|
||
| #### Prereqs | ||
|
|
||
| All commands below should be run in a bash shell (Windows users can use the 'mingw64' terminal that is included when installing git). | ||
|
|
||
| 1. Clone this repository: create a 'ProjectAeon' directory in your home directory, clone this repository there, and `cd` into the cloned directory: | ||
| ``` | ||
| mkdir ~/ProjectAeon | ||
| cd ~/ProjectAeon | ||
| git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha | ||
| cd aeon_mecha | ||
| ``` | ||
|
|
||
| #### Set-up | ||
|
|
||
| Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below. | ||
|
|
||
| [Option 1](./docs/env_setup/local/miniconda_conda_local_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager) | ||
|
|
||
| - *Note*: **mambaforge** and **mamba** can be used as faster, drop-in replacements for 'miniconda' and 'conda', respectively. You can set up the Aeon environment using them, following roughly the same instructions as above. See [here](https://biapol.github.io/blog/mara_lampert/getting_started_with_mambaforge_and_python/readme.html) for more info. | ||
|
|
||
| [Option 2](./docs/env_setup/local/pip_venv_local_setup.md): **pip** (python package manager) and **venv** (python environment manager) | ||
|
|
||
| ## Repository Contents | ||
|
|
||
| - `.github/workflows/` : GitHub actions workflows for building the environment and running tests | ||
| - `aeon/` : Source code for the Aeon Python package | ||
| - `aeon/dj_pipeline`: Source code for the Aeon DataJoint MySQL database pipeline | ||
| - `aeon/io`: Source code for loading raw data | ||
| - `aeon/processing`: Source code for processing raw data | ||
| - `aeon/qc`: Source code for quality control of raw data | ||
| - `aeon/schema`: Examples of 'experiment schemas': variables that can be used to load raw data from particular experiments | ||
| - `docker/` : Dockerfiles for building Docker images for the Aeon DataJoint MySQL database pipeline. | ||
| - `docs/` : Documentation for the Aeon project | ||
| - `docs/devs/` : Documentation for developers | ||
| - `docs/env_setup/` : Documentation for setting up the Aeon Python environment | ||
| - `docs/examples/` : Aeon usecase examples | ||
| - `docs/using_hpc_jupyterhub.md` : Instructions for using Jupyter notebooks to access Aeon data via SWC's HPC | ||
| - `docs/using_online_dashboard.md` : Instructions for connecting to Aeon's online dashboard | ||
| - `env_config/` : Configuration files for the Aeon Python environment | ||
| - `tests/` : Unit and integration tests | ||
| - `tests/data` : Data used by tests | ||
|
|
||
| ## Citation Policy | ||
|
|
||
| If you use this software, please cite it as below: | ||
|
|
||
| Sainsbury Wellcome Centre Foraging Behaviour Working Group. (2023). Aeon: An open-source platform to study the neural basis of ethological behaviours over naturalistic timescales, https://doi.org/10.5281/zenodo.8411157 | ||
|
|
||
| [](https://zenodo.org/doi/10.5281/zenodo.8411157) |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| import os | ||
| import pathlib | ||
| from colorsys import hls_to_rgb, rgb_to_hls | ||
| from contextlib import contextmanager | ||
| from pathlib import Path | ||
|
|
||
| import matplotlib.pyplot as plt | ||
| import numpy as np | ||
| import pandas as pd | ||
| import plotly | ||
| import plotly.express as px | ||
| import plotly.graph_objs as go | ||
| import seaborn as sns | ||
| from numpy.lib.stride_tricks import as_strided | ||
|
|
||
| """Standardize subject colors, patch colors, and markers.""" | ||
|
|
||
| subject_colors = plotly.colors.qualitative.Plotly | ||
| subject_colors_dict = { | ||
| "BAA-1104045": subject_colors[0], | ||
| "BAA-1104047": subject_colors[1], | ||
| "BAA-1104048": subject_colors[2], | ||
| "BAA-1104049": subject_colors[3], | ||
| } | ||
| patch_colors = plotly.colors.qualitative.Dark2 | ||
| patch_markers = [ | ||
| "circle", | ||
| "bowtie", | ||
| "square", | ||
| "hourglass", | ||
| "diamond", | ||
| "cross", | ||
| "x", | ||
| "triangle", | ||
| "star", | ||
| ] | ||
| patch_markers_symbols = ["●", "⧓", "■", "⧗", "♦", "✖", "×", "▲", "★"] | ||
| patch_markers_dict = { | ||
| marker: symbol for marker, symbol in zip(patch_markers, patch_markers_symbols) | ||
| } | ||
| patch_markers_linestyles = ["solid", "dash", "dot", "dashdot", "longdashdot"] | ||
|
|
||
|
|
||
| def gen_hex_grad(hex_col, vals, min_l=0.3): | ||
| """Generates an array of hex color values based on a gradient defined by unit-normalized values.""" | ||
| # Convert hex to rgb to hls | ||
| h, l, s = rgb_to_hls( | ||
| *[int(hex_col.lstrip("#")[i: i + 2], 16) / 255 for i in (0, 2, 4)] | ||
| ) | ||
| grad = np.empty(shape=(len(vals),), dtype="<U10") # init grad | ||
| for i, val in enumerate(vals): | ||
| cur_l = (l * val) + ( | ||
| min_l * (1 - val) | ||
| ) # get cur lightness relative to `hex_col` | ||
| cur_l = max(min(cur_l, l), min_l) # set min, max bounds | ||
| cur_rgb_col = hls_to_rgb(h, cur_l, s) # convert to rgb | ||
| cur_hex_col = "#%02x%02x%02x" % tuple( | ||
| int(c * 255) for c in cur_rgb_col | ||
| ) # convert to hex | ||
| grad[i] = cur_hex_col | ||
|
|
||
| return grad |
Oops, something went wrong.