See details. Notebook work, config overhaul

gitignore: add exclude example config mkdocs: new notebook names notebooks: complete revamp for minirec data and more links to docs init: add new load_config, isort imports common_lab: - adjust to accept names in Last, First format (nwb-compliant) - continue to use First Last name structure in database - yes? common_nwb: use new load_config, change `assert` to `raise` insert_sessions: permit paths, use file name, use raw dir storage_dirs: remove redundant funcs for base_dir settings: implement new base_dir system - allows base/raw/etc to be independent - defaults to dj.config, then env vars, then sets default rel paths
CBroz1 · Jul 19, 2023 · 7d43f80 · 7d43f80
1 parent a6c80cd
commit 7d43f80
Show file tree

Hide file tree

Showing 17 changed files with 3,145 additions and 917 deletions.
diff --git a/.gitignore b/.gitignore
@@ -159,7 +159,6 @@ dmypy.json
 temp_nwb/*s
 
 *.mp4
-dj_local_conf.json
 *.png
 *.npy
 *.nc
@@ -168,6 +167,8 @@ dj_local_conf.json
 *.json
 *.gz
 *.pdf
+dj_local_conf.json
+!dj_local_conf_example.json
 
 !/.vscode/extensions.json
 !/.vscode/settings.json
@@ -183,4 +184,4 @@ docs/src/notebooks/
 temp*
 
 # Version
-_version.py
+_version.py
diff --git a/dj_local_conf_example.json b/dj_local_conf_example.json
@@ -0,0 +1,36 @@
+{
+  "database.host": "localhost",
+  "database.password": "tutorial",
+  "database.user": "root",
+  "database.port": 3306,
+  "database.reconnect": true,
+  "connection.init_function": null,
+  "connection.charset": "",
+  "loglevel": "INFO",
+  "safemode": true,
+  "fetch_format": "array",
+  "display.limit": 12,
+  "display.width": 14,
+  "display.show_tuple_count": true,
+  "database.use_tls": null,
+  "enable_python_native_blobs": true,
+  "filepath_checksum_size_limit": null,
+  "stores": {
+    "raw": {
+      "protocol": "file",
+      "location": "/your/base/path/raw",
+      "stage": "/your/base/path/raw"
+    },
+    "analysis": {
+      "protocol": "file",
+      "location": "/your/base/path/analysis",
+      "stage": "/your/base/path/analysis"
+    }
+  },
+  "custom": {
+    "database.prefix": "username_",
+    "spyglass_dirs": {
+      "base": "/your/base/path"
+    }
+  }
+}
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -55,8 +55,9 @@ nav:
       - Insert Data: misc/insert_data.md
       - Merge Tables: misc/merge_tables.md
   - Tutorials:
-      - Introduction: notebooks/00_intro.ipynb
-      - Spike Sorting: notebooks/01_spikesorting.ipynb
+      - Database Connection: notebooks/00_Dabase_Connection.ipynb
+      - Insert Data: notebooks/01_Insert_Data.ipynb
+      - Spike Sorting: notebooks/02_Spike_Sorting.ipynb
       - Curation: notebooks/02_curation.ipynb
       - LFP: notebooks/03_lfp.ipynb
       - Position:
@@ -70,7 +71,6 @@ nav:
       - 1D Clusterless Decoding: notebooks/10_1D_Clusterless_Decoding.ipynb
       - 2D Clusterless Decoding: notebooks/11_2D_Clusterless_Decoding.ipynb
       - Ripple Detection: notebooks/12_Ripple_Detection.ipynb
-      - Docker: notebooks/docker_mysql_tutorial.ipynb
       - Spyglass Kachery Setup: notebooks/Spyglass_kachery_setup.ipynb
   - API Reference: api/ # defer to gen-files + literate-nav
   - How to Contribute: contribute.md

diff --git a/notebooks/00_Database_Connection.ipynb b/notebooks/00_Database_Connection.ipynb
@@ -0,0 +1,279 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "deaf3687",
+   "metadata": {},
+   "source": [
+    "# Database Connection\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cbb74150",
+   "metadata": {},
+   "source": [
+    "Welcome to [Spyglass](https://lorenfranklab.github.io/spyglass/0.4/),\n",
+    "a [DataJoint](https://github.com/datajoint/datajoint-python/)\n",
+    "pipeline maintained by the [Frank Lab](https://franklab.ucsf.edu/) at UCSF.\n",
+    "\n",
+    "Spyglass will help you take an [NWB](https://www.nwb.org/) file from raw data to\n",
+    "analysis-ready preprocessed formats using DataJoint to (a) connect to a\n",
+    "[relational database](https://www.youtube.com/watch?v=q-PMUSC5P5o) (here,\n",
+    "MySQL), and (b) automate processing steps. You have a few options for databases.\n",
+    "\n",
+    "1. Connect to an existing database.\n",
+    "2. Use GitHub Codespaces (coming soon...)\n",
+    "3. Run your own database with [Docker](#running-your-own-database)\n",
+    "\n",
+    "Once your database is set up, be sure to [configure](#configure) the connection.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a63761cc-437f-4e4a-a777-664b321b9b94",
+   "metadata": {},
+   "source": [
+    "## Running your own database\n",
+    "\n",
+    "- First, [install Docker](https://docs.docker.com/engine/install/).\n",
+    "- Add yourself to the\n",
+    "  [`docker` group](https://docs.docker.com/engine/install/linux-postinstall/) so\n",
+    "  that you don't have to be sudo to run docker.\n",
+    "- Download the docker image for datajoint/mysql\n",
+    "\n",
+    "  ```bash\n",
+    "  docker pull datajoint/mysql\n",
+    "  ```\n",
+    "\n",
+    "- When run, this is referred to as a 'Docker container'\n",
+    "- Next start the container with a couple additional pieces of info...\n",
+    "\n",
+    "  - Root password. We use `tutorial`.\n",
+    "  - Database name. Here, we use `spyglass-db`.\n",
+    "  - Port mapping. Here, we map 3306 across the local machine and container.\n",
+    "\n",
+    "  ```bash\n",
+    "  docker run --name spyglass-db -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n",
+    "  ```\n",
+    "\n",
+    "- For data to persist after terminating the container,\n",
+    "  [attach a volume](https://docs.docker.com/storage/volumes/) when running:\n",
+    "\n",
+    "  ```bash\n",
+    "  docker volume create dj-vol\n",
+    "  docker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n",
+    "  ```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11487cb5",
+   "metadata": {},
+   "source": [
+    "## Configure\n",
+    "\n",
+    "The `dj_local_conf_example.json` contains all the relevant items for setting up\n",
+    "a database connection. Simply rename to `dj_local_conf.json` and modify the\n",
+    "contents accordingly. The defaults reflect a local Docker connection.\n",
+    "\n",
+    "For Spyglass, you'll want to set your base path under `custom`:\n",
+    "\n",
+    "```json\n",
+    "{\n",
+    "  \"custom\": {\n",
+    "    \"database.prefix\": \"username_\",\n",
+    "    \"spyglass_dirs\": {\n",
+    "      \"base\": \"/your/base/path\"\n",
+    "    }\n",
+    "  }\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "Alternatively, you can leave the `spyglass_dirs` section blank and use an\n",
+    "environment variable, `SPYGLASS_BASE_DIR`. This is where Spyglass will look for\n",
+    "various data types using subdirectories, including ...\n",
+    "\n",
+    "- raw: raw data\n",
+    "- analysis: analyzed data\n",
+    "- recording: spike sorting recordings\n",
+    "- spike_sorting_storage\n",
+    "- waveforms: spike sorting waveforms\n",
+    "- temp: temporary files\n",
+    "- storage: [kachery](https://github.com/kacheryhub/kachery-client) storage\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "912ac84b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from spyglass.settings import load_config\n",
+    "\n",
+    "config = load_config()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b5c15d54",
+   "metadata": {},
+   "source": [
+    "## Connect\n",
+    "\n",
+    "Now, you should be able to connect to the database you set up.\n",
+    "\n",
+    "Let's demonstrate with an example table:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "afb63913-4e6b-4049-ae1d-55ab1ac8d42c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    \n",
+       "    <style type=\"text/css\">\n",
+       "        .Table{\n",
+       "            border-collapse:collapse;\n",
+       "        }\n",
+       "        .Table th{\n",
+       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
+       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
+       "        }\n",
+       "        .Table td{\n",
+       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
+       "        }\n",
+       "        .Table tr:nth-child(odd){\n",
+       "            background: #ffffff;\n",
+       "            color: #000000;\n",
+       "        }\n",
+       "        .Table tr:nth-child(even){\n",
+       "            background: #f3f1ff;\n",
+       "            color: #000000;\n",
+       "        }\n",
+       "        /* Tooltip container */\n",
+       "        .djtooltip {\n",
+       "        }\n",
+       "        /* Tooltip text */\n",
+       "        .djtooltip .djtooltiptext {\n",
+       "            visibility: hidden;\n",
+       "            width: 120px;\n",
+       "            background-color: black;\n",
+       "            color: #fff;\n",
+       "            text-align: center;\n",
+       "            padding: 5px 0;\n",
+       "            border-radius: 6px;\n",
+       "            /* Position the tooltip text - see examples below! */\n",
+       "            position: absolute;\n",
+       "            z-index: 1;\n",
+       "        }\n",
+       "        #primary {\n",
+       "            font-weight: bold;\n",
+       "            color: black;\n",
+       "        }\n",
+       "        #nonprimary {\n",
+       "            font-weight: normal;\n",
+       "            color: white;\n",
+       "        }\n",
+       "\n",
+       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
+       "        .djtooltip:hover .djtooltiptext {\n",
+       "            visibility: visible;\n",
+       "        }\n",
+       "    </style>\n",
+       "    \n",
+       "    <b></b>\n",
+       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
+       "        <table border=\"1\" class=\"Table\">\n",
+       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">nwb_file_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">sort_group_id</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">sort_interval_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">filter_parameter_set_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">sorter_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"primary\">spikesorter_parameter_set_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"nonprimary\">sorting_id</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"nonprimary\">analysis_file_name</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"nonprimary\">time_of_sort</p>\n",
+       "                            <span class=\"djtooltiptext\">in Unix time, to the nearest second</span>\n",
+       "                        </div></th><th><div class=\"djtooltip\">\n",
+       "                            <p id=\"nonprimary\">units_object_id</p>\n",
+       "                            <span class=\"djtooltiptext\"></span>\n",
+       "                        </div> </th> </tr> </thead>\n",
+       "            <tbody> <tr>  </tr> </tbody>\n",
+       "        </table>\n",
+       "        \n",
+       "        <p>Total: 0</p></div>\n",
+       "        "
+      ],
+      "text/plain": [
+       "*nwb_file_name *sort_group_id *sort_interval *filter_parame *sorter_name   *spikesorter_p sorting_id     analysis_file_ time_of_sort   units_object_i\n",
+       "+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\n",
+       "\n",
+       " (Total: 0)"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from spyglass.common import SpikeSortingBackUp\n",
+    "\n",
+    "SpikeSortingBackUp()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c6850095",
+   "metadata": {},
+   "source": [
+    "Next, we'll try [inserting data](./01_Insert_Data.ipynb)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}