forked from LorenFrankLab/spyglass
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
See details. Notebook work, config overhaul
gitignore: add exclude example config mkdocs: new notebook names notebooks: complete revamp for minirec data and more links to docs init: add new load_config, isort imports common_lab: - adjust to accept names in Last, First format (nwb-compliant) - continue to use First Last name structure in database - yes? common_nwb: use new load_config, change `assert` to `raise` insert_sessions: permit paths, use file name, use raw dir storage_dirs: remove redundant funcs for base_dir settings: implement new base_dir system - allows base/raw/etc to be independent - defaults to dj.config, then env vars, then sets default rel paths
- Loading branch information
Showing
17 changed files
with
3,145 additions
and
917 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
{ | ||
"database.host": "localhost", | ||
"database.password": "tutorial", | ||
"database.user": "root", | ||
"database.port": 3306, | ||
"database.reconnect": true, | ||
"connection.init_function": null, | ||
"connection.charset": "", | ||
"loglevel": "INFO", | ||
"safemode": true, | ||
"fetch_format": "array", | ||
"display.limit": 12, | ||
"display.width": 14, | ||
"display.show_tuple_count": true, | ||
"database.use_tls": null, | ||
"enable_python_native_blobs": true, | ||
"filepath_checksum_size_limit": null, | ||
"stores": { | ||
"raw": { | ||
"protocol": "file", | ||
"location": "/your/base/path/raw", | ||
"stage": "/your/base/path/raw" | ||
}, | ||
"analysis": { | ||
"protocol": "file", | ||
"location": "/your/base/path/analysis", | ||
"stage": "/your/base/path/analysis" | ||
} | ||
}, | ||
"custom": { | ||
"database.prefix": "username_", | ||
"spyglass_dirs": { | ||
"base": "/your/base/path" | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,279 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"id": "deaf3687", | ||
"metadata": {}, | ||
"source": [ | ||
"# Database Connection\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "cbb74150", | ||
"metadata": {}, | ||
"source": [ | ||
"Welcome to [Spyglass](https://lorenfranklab.github.io/spyglass/0.4/),\n", | ||
"a [DataJoint](https://github.com/datajoint/datajoint-python/)\n", | ||
"pipeline maintained by the [Frank Lab](https://franklab.ucsf.edu/) at UCSF.\n", | ||
"\n", | ||
"Spyglass will help you take an [NWB](https://www.nwb.org/) file from raw data to\n", | ||
"analysis-ready preprocessed formats using DataJoint to (a) connect to a\n", | ||
"[relational database](https://www.youtube.com/watch?v=q-PMUSC5P5o) (here,\n", | ||
"MySQL), and (b) automate processing steps. You have a few options for databases.\n", | ||
"\n", | ||
"1. Connect to an existing database.\n", | ||
"2. Use GitHub Codespaces (coming soon...)\n", | ||
"3. Run your own database with [Docker](#running-your-own-database)\n", | ||
"\n", | ||
"Once your database is set up, be sure to [configure](#configure) the connection.\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "a63761cc-437f-4e4a-a777-664b321b9b94", | ||
"metadata": {}, | ||
"source": [ | ||
"## Running your own database\n", | ||
"\n", | ||
"- First, [install Docker](https://docs.docker.com/engine/install/).\n", | ||
"- Add yourself to the\n", | ||
" [`docker` group](https://docs.docker.com/engine/install/linux-postinstall/) so\n", | ||
" that you don't have to be sudo to run docker.\n", | ||
"- Download the docker image for datajoint/mysql\n", | ||
"\n", | ||
" ```bash\n", | ||
" docker pull datajoint/mysql\n", | ||
" ```\n", | ||
"\n", | ||
"- When run, this is referred to as a 'Docker container'\n", | ||
"- Next start the container with a couple additional pieces of info...\n", | ||
"\n", | ||
" - Root password. We use `tutorial`.\n", | ||
" - Database name. Here, we use `spyglass-db`.\n", | ||
" - Port mapping. Here, we map 3306 across the local machine and container.\n", | ||
"\n", | ||
" ```bash\n", | ||
" docker run --name spyglass-db -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n", | ||
" ```\n", | ||
"\n", | ||
"- For data to persist after terminating the container,\n", | ||
" [attach a volume](https://docs.docker.com/storage/volumes/) when running:\n", | ||
"\n", | ||
" ```bash\n", | ||
" docker volume create dj-vol\n", | ||
" docker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n", | ||
" ```\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "11487cb5", | ||
"metadata": {}, | ||
"source": [ | ||
"## Configure\n", | ||
"\n", | ||
"The `dj_local_conf_example.json` contains all the relevant items for setting up\n", | ||
"a database connection. Simply rename to `dj_local_conf.json` and modify the\n", | ||
"contents accordingly. The defaults reflect a local Docker connection.\n", | ||
"\n", | ||
"For Spyglass, you'll want to set your base path under `custom`:\n", | ||
"\n", | ||
"```json\n", | ||
"{\n", | ||
" \"custom\": {\n", | ||
" \"database.prefix\": \"username_\",\n", | ||
" \"spyglass_dirs\": {\n", | ||
" \"base\": \"/your/base/path\"\n", | ||
" }\n", | ||
" }\n", | ||
"}\n", | ||
"```\n", | ||
"\n", | ||
"Alternatively, you can leave the `spyglass_dirs` section blank and use an\n", | ||
"environment variable, `SPYGLASS_BASE_DIR`. This is where Spyglass will look for\n", | ||
"various data types using subdirectories, including ...\n", | ||
"\n", | ||
"- raw: raw data\n", | ||
"- analysis: analyzed data\n", | ||
"- recording: spike sorting recordings\n", | ||
"- spike_sorting_storage\n", | ||
"- waveforms: spike sorting waveforms\n", | ||
"- temp: temporary files\n", | ||
"- storage: [kachery](https://github.com/kacheryhub/kachery-client) storage\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 2, | ||
"id": "912ac84b", | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from spyglass.settings import load_config\n", | ||
"\n", | ||
"config = load_config()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "b5c15d54", | ||
"metadata": {}, | ||
"source": [ | ||
"## Connect\n", | ||
"\n", | ||
"Now, you should be able to connect to the database you set up.\n", | ||
"\n", | ||
"Let's demonstrate with an example table:\n" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 3, | ||
"id": "afb63913-4e6b-4049-ae1d-55ab1ac8d42c", | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"data": { | ||
"text/html": [ | ||
"\n", | ||
" \n", | ||
" <style type=\"text/css\">\n", | ||
" .Table{\n", | ||
" border-collapse:collapse;\n", | ||
" }\n", | ||
" .Table th{\n", | ||
" background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n", | ||
" font-weight: normal; font-family: monospace; font-size: 100%;\n", | ||
" }\n", | ||
" .Table td{\n", | ||
" padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n", | ||
" }\n", | ||
" .Table tr:nth-child(odd){\n", | ||
" background: #ffffff;\n", | ||
" color: #000000;\n", | ||
" }\n", | ||
" .Table tr:nth-child(even){\n", | ||
" background: #f3f1ff;\n", | ||
" color: #000000;\n", | ||
" }\n", | ||
" /* Tooltip container */\n", | ||
" .djtooltip {\n", | ||
" }\n", | ||
" /* Tooltip text */\n", | ||
" .djtooltip .djtooltiptext {\n", | ||
" visibility: hidden;\n", | ||
" width: 120px;\n", | ||
" background-color: black;\n", | ||
" color: #fff;\n", | ||
" text-align: center;\n", | ||
" padding: 5px 0;\n", | ||
" border-radius: 6px;\n", | ||
" /* Position the tooltip text - see examples below! */\n", | ||
" position: absolute;\n", | ||
" z-index: 1;\n", | ||
" }\n", | ||
" #primary {\n", | ||
" font-weight: bold;\n", | ||
" color: black;\n", | ||
" }\n", | ||
" #nonprimary {\n", | ||
" font-weight: normal;\n", | ||
" color: white;\n", | ||
" }\n", | ||
"\n", | ||
" /* Show the tooltip text when you mouse over the tooltip container */\n", | ||
" .djtooltip:hover .djtooltiptext {\n", | ||
" visibility: visible;\n", | ||
" }\n", | ||
" </style>\n", | ||
" \n", | ||
" <b></b>\n", | ||
" <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n", | ||
" <table border=\"1\" class=\"Table\">\n", | ||
" <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">nwb_file_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">sort_group_id</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">sort_interval_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">filter_parameter_set_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">sorter_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"primary\">spikesorter_parameter_set_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"nonprimary\">sorting_id</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"nonprimary\">analysis_file_name</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"nonprimary\">time_of_sort</p>\n", | ||
" <span class=\"djtooltiptext\">in Unix time, to the nearest second</span>\n", | ||
" </div></th><th><div class=\"djtooltip\">\n", | ||
" <p id=\"nonprimary\">units_object_id</p>\n", | ||
" <span class=\"djtooltiptext\"></span>\n", | ||
" </div> </th> </tr> </thead>\n", | ||
" <tbody> <tr> </tr> </tbody>\n", | ||
" </table>\n", | ||
" \n", | ||
" <p>Total: 0</p></div>\n", | ||
" " | ||
], | ||
"text/plain": [ | ||
"*nwb_file_name *sort_group_id *sort_interval *filter_parame *sorter_name *spikesorter_p sorting_id analysis_file_ time_of_sort units_object_i\n", | ||
"+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\n", | ||
"\n", | ||
" (Total: 0)" | ||
] | ||
}, | ||
"execution_count": 3, | ||
"metadata": {}, | ||
"output_type": "execute_result" | ||
} | ||
], | ||
"source": [ | ||
"from spyglass.common import SpikeSortingBackUp\n", | ||
"\n", | ||
"SpikeSortingBackUp()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "c6850095", | ||
"metadata": {}, | ||
"source": [ | ||
"Next, we'll try [inserting data](./01_Insert_Data.ipynb)\n" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3 (ipykernel)", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.9.16" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |
Oops, something went wrong.