Updated docs

docs/env_setup/local/miniconda_conda_local_setup.md

@@ -1,6 +1,6 @@
 # Local set-up using miniconda and conda
 
-1) Install [miniconda](https://docs.conda.io/en/latest/miniconda.html)
+1. Install [miniconda](https://docs.conda.io/en/latest/miniconda.html)
 	- If prompted with an "Install for", select "Just Me" (instead of "All Users")
 	- Ensure installation is in your home directory:
 		- On Windows: `C:\Users\<your_username>\anaconda3`
@@ -10,15 +10,21 @@
 	- Ensure you do *not* register anaconda as the default version of Python.
 	- _Note_: These installation settings can always be changed posthoc.
 
-2) Create conda environment and install the code dependencies from the `env.yml` file:
+2. Create conda environment and install the code dependencies from the `env.yml` file:
 ```
 conda update conda
 conda init
-conda env create --file env.yml
+conda env create --file env_config/env.yml
 ```
 
-3) Using the virtual environment:
+3. Optionally install development dependencies:
+```
+conda activate aeon
+conda env update --file env_config/env_dev.yml
+```
+
+4. Using the virtual environment:
 
-`conda activate aeon`: activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
+	- `conda activate aeon`: activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
 
-`conda deactivate aeon`: deactivates the virtual environment.
+	- `conda deactivate aeon`: deactivates the virtual environment.

docs/env_setup/local/pip_venv_local_setup.md

@@ -1,23 +1,29 @@
 # Local set-up using pip and venv
 
-1) Ensure that you have python 3.9 and pip installed on your computer:
+1. Ensure that python >=3.9 and pip are installed (if not, install python >=3.9)
 ```
-python -V         # should return >=3.9
+python -V          # should return >=3.9
 python -m pip -V
 ```
 
-2) Install virtualenv: `python -m pip install virtualenv`
+2. Install virtualenv: `python -m pip install virtualenv`
 
-3) Create virtual environment: `python -m venv aeon`
+3. Create virtual environment: `python -m venv aeon`
 
-4) Activate the virtual environment and install the code dependencies:
+4. Activate the virtual environment and install the code dependencies:
 ```
 source aeon/bin/activate
-python -m pip install -r requirements.txt
+python -m pip install -e .
 ```
 
-5) Using the virtual environment:
+5. Optionally install the development dependencies:
+```
+source aeon/bin/activate
+python -m pip install -e .[dev]
+```
+
+6. Using the virtual environment:
 
-`source aeon/bin/activate` activates the virtual environment.
+`source aeon/bin/activate` activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
 
 `deactivate` deactivates the virtual environment.

docs/env_setup/remote/developing_on_hpc.md

@@ -2,12 +2,6 @@ In the examples below, replace `<your_username>` with your SWC HPC username.
 
 # Developing while on the HPC
 
-0) After you've finished creating the virtual `aeon` environment, activate the environment. Then, add this repository to your python path within the activated environment by opening a terminal and running `pip install -e <path_to_aeon_mecha>`.
+1. After you've finished creating the virtual `aeon` environment, activate the environment. Then, add this repository to your python path within the activated environment by opening a terminal and running `pip install -e <path_to_aeon_mecha>`.
 
-1) For using an IDE (e.g. PyCharm, VSCode, Jupyter, etc.) from your local machine, you will need to set up local port forwarding from a specified port on the HPC: 
-
-* First, open a terminal and set up SSH local port forwarding to HPC-GW1:  `ssh -L 9999:hpc-gw1:22 <your_username>@ssh.swc.ucl.ac.uk`
-
-* Then, in a new terminal, SSH with trusted x11 forwarding into the HPC via the forwarded port on localhost: `ssh -Y <your_username>@localhost -p 9999`
-
-* Lastly, set up the remote interpreter in your IDE to use the forwarded port (here 9999) on your localhost, and point to the python installed in the `aeon` environment. These instructions can typically be found in your IDE's online documentation. [Here are instructions for PyCharm Professional](https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html), and [here for VSCode](https://code.visualstudio.com/docs/remote/ssh).
+2. For using an IDE (e.g. PyCharm, VSCode, Jupyter, etc.), you will need to set up local port forwarding from a specified port on the HPC. These instructions can typically be found in your IDE's online documentation. [Here are instructions for PyCharm Professional](https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html), and [here for VSCode](https://code.visualstudio.com/docs/remote/ssh).

docs/env_setup/remote/miniconda_conda_remote_setup.md

@@ -1,20 +1,27 @@
 # Remote set-up using miniconda and conda
 
-1) Add miniconda to your system path and create the `aeon` python env from the `env.yml` file:
+1. Add miniconda to your system path and create the `aeon` python env from the `env.yml` file:
 
 ```
 module load miniconda
-conda env create -f env.yml
+conda env create -f env_config/env.yml
 ```
 
-2) Using the virtual environment:
+2. Optionally install development dependencies:
 
-`conda activate aeon`: activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
+```
+conda activate aeon
+conda env update -f env_config/env_dev.yml
+```
+
+3. Using the virtual environment:
 
-`conda deactivate aeon`: deactivates the virtual environment.
+    - `conda activate aeon`: activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
 
-3) (Optional) Add commands to `.profile` to add miniconda as an environment module and Bonsai and its dependencies to your system path on startup:
+    - `conda deactivate aeon`: deactivates the virtual environment.
 
-Copy the commands in the `.profile` file in this folder to your HPC home directory `.profile` file.
+4. (Optional) Add commands to the `.profile` file to add miniconda as an environment module and Bonsai and its dependencies to your system path on startup (this will be initialized each time you SSH into the HPC).
+
+    - Copy the commands in the `.profile` file in this folder to your HPC home directory `.profile` file.
 
-4) For instructions on developing within the `aeon` environment, see [`developing_on_hpc.md`](./developing_on_hpc.md)
+5. For instructions on developing within the `aeon` environment, see [`developing_on_hpc.md`](./developing_on_hpc.md)

docs/env_setup/remote/pip_venv_remote_setup.md

@@ -1,25 +1,31 @@
 # Remote set-up using pip and venv
 
-1) Ensure that you have python 3.9 and pip installed on your computer:
+1. Ensure that python >=3.9 and pip are installed (if not, install python >=3.9)
 ```
-python -V         # should return >=3.9
+python -V          # should return >=3.9
 python -m pip -V
 ```
 
-2) Install virtualenv: `python -m pip install virtualenv`
+2. Install virtualenv: `python -m pip install virtualenv`
 
-3) Create virtual environment: `python -m venv aeon`
+3. Create virtual environment: `python -m venv aeon`
 
-4) Activate the virtual environment and install the code dependencies:
+4. Activate the virtual environment and install the code dependencies:
 ```
 source aeon/bin/activate
-python -m pip install -r requirements.txt
+python -m pip install -e .
 ```
 
-5) Using the virtual environment:
+5. Optionally install the development dependencies:
+```
+source aeon/bin/activate
+python -m pip install -e .[dev]
+```
+
+6. Using the virtual environment:
 
-`source aeon/bin/activate` activates the virtual environment.
+`source aeon/bin/activate` activates the virtual environment; any commands now run within this terminal will take place within the virtual environment.
 
 `deactivate` deactivates the virtual environment.
 
-6) For instructions on developing within the `aeon` environment, see [`developing_on_hpc.md`](./developing_on_hpc.md)
+7. For instructions on developing within the `aeon` environment, see [`developing_on_hpc.md`](./developing_on_hpc.md)

readme.md

@@ -2,7 +2,7 @@
 ![aeon_mecha_env_build_and_tests](https://github.com/SainsburyWellcomeCentre/aeon_mecha/actions/workflows/build_env_run_tests.yml/badge.svg?branch=main)
 [![aeon_mecha_tests_code_coverage](https://codecov.io/gh/SainsburyWellcomeCentre/aeon_mecha/branch/main/graph/badge.svg?token=973EC1CG03)](https://codecov.io/gh/SainsburyWellcomeCentre/aeon_mecha)
 
-Project Aeon's main repository for manipulating acquired data. Includes preprocessing, querying, plotting, and analysis modules.
+Project Aeon's main repository for manipulating acquired data. Includes modules for loading raw data, performing quality control on raw data, processing raw data, and ingesting processed data into a DataJoint MySQL database.
 
 ## Set-up Instructions
 
@@ -12,11 +12,10 @@ The various set-up tools mentioned below do some combination of python version,
 
 #### Prereqs
 
-1. Ssh into the HPC GW1 node and clone this repo to your home directory.
+1. Ssh into the HPC and clone this repository to your home directory.
 
 ```
 ssh <your_SWC_username>@ssh.swc.ucl.ac.uk
-ssh hpc-gw1
 mkdir ~/ProjectAeon
 cd ~/ProjectAeon
 git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha
@@ -26,19 +25,17 @@ git clone https://github.com/SainsburyWellcomeCentre/aeon_mecha
 
 Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below.
 
-[Option 1](./docs/env_setup/remote/miniconda_conda_remote_setup.md): miniconda (python distribution) and conda (python version manager, environment manager, package manager, and package dependency manager)
+[Option 1](./docs/env_setup/remote/miniconda_conda_remote_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager)
 
-[Option 2](./docs/env_setup/remote/pyenv_poetry_remote_setup.md): pyenv (python version manager) and poetry (python environment manager, package manager, and package dependency manager)
-
-[Option 3](./docs/env_setup/remote/pip_venv_remote_setup.md): pip (python package manager) and venv (python environment manager)
+[Option 2](./docs/env_setup/remote/pip_venv_remote_setup.md): **pip** (python package manager) and **venv** (python environment manager)
 
 ### Local set-up
 
 #### Prereqs
 
-1. Install [git](https://git-scm.com/downloads). If you are not familiar with git, just confirm the default settings during installation.
+All commands below should be run in a bash shell (Windows users can use the 'mingw64' terminal that is included when installing git).
 
-2. Clone this repository: create a 'ProjectAeon' directory in your home directory, clone this repository there, and `cd` into the cloned directory:
+1. Clone this repository: create a 'ProjectAeon' directory in your home directory, clone this repository there, and `cd` into the cloned directory:
 ```
 mkdir ~/ProjectAeon
 cd ~/ProjectAeon
@@ -48,16 +45,25 @@ cd aeon_mecha
 
 #### Set-up
 
-Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below. All commands below should be run in a bash terminal (Windows users can use the 'mingw64' terminal that comes installed with git).
-
-[Option 1](./docs/env_setup/local/miniconda_conda_local_setup.md): miniconda (python distribution) and conda (python version manager, environment manager, package manager, and package dependency manager)
+Ensure you stay in the `~/ProjectAeon/aeon_mecha` directory for the rest of the set-up instructions, regardless of which set-up procedure you follow below.
 
-[Option 2](./docs/env_setup/local/pyenv_poetry_local_setup.md): pyenv (python version manager) and poetry (python environment manager, package manager, and package dependency manager)
+[Option 1](./docs/env_setup/local/miniconda_conda_local_setup.md): **miniconda** (python distribution) and **conda** (python version manager, environment manager, package manager, and package dependency manager)
 
-[Option 3](./docs/env_setup/local/pip_venv_local_setup.md): pip (python package manager) and venv (python environment manager)
+[Option 2](./docs/env_setup/local/pip_venv_local_setup.md): **pip** (python package manager) and **venv** (python environment manager)
 
 ## Repository Contents
 
-## Todos
-
-- add to [repository contents](#repository-contents)
+- `.github/workflows/` : GitHub actions workflows for building the environment and running tests 
+- `aeon/` : Source code for the Aeon Python package 
+    - `aeon/dj_pipeline`: Source code for the Aeon DataJoint MySQL database pipeline
+    - `aeon/io`: Source code for loading raw data
+    - `aeon/processing`: Source code for processing raw data
+    - `aeon/qc`: Source code for quality control of raw data
+    - `aeon/schema`: Examples of 'experiment schemas': variables that can be used to load raw data from particular experiments
+- `docker/` : Dockerfiles for building Docker images for the Aeon DataJoint MySQL database pipeline.
+- `docs/` : Documentation for the Aeon project
+    - `docs/devs/` : Documentation for developers
+    - `docs/env_setup/` : Documentation for setting up the Aeon Python environment
+    - `docs/examples/` : Aeon usecase examples
+- `env_config/` : Configuration files that get used when setting up the Aeon Python environment
+- `tests/` : Unit and integration tests