From 34f53ee080d9d41ad909692905193e771cd45c20 Mon Sep 17 00:00:00 2001
From: CBroz1 <CBrozdowski@yahoo.com>
Date: Thu, 21 Mar 2024 13:43:32 -0600
Subject: [PATCH] Deployed 3de6cdcc to 0.5 with MkDocs 1.5.3 and mike 2.0.0

---
 0.5/notebooks/03_Merge_Tables/index.html |   8 ++++----
 0.5/notebooks/30_LFP/index.html          |   8 ++++----
 0.5/notebooks/index.html                 |  10 +++++++---
 0.5/search/search_index.json             |   2 +-
 0.5/sitemap.xml.gz                       | Bin 1303 -> 1303 bytes
 5 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/0.5/notebooks/03_Merge_Tables/index.html b/0.5/notebooks/03_Merge_Tables/index.html
index de3efa725..76eb2133c 100644
--- a/0.5/notebooks/03_Merge_Tables/index.html
+++ b/0.5/notebooks/03_Merge_Tables/index.html
@@ -5445,9 +5445,9 @@ <h2 id="example-data">Example data<a class="anchor-link" href="#example-data">¶
 <div class="jp-OutputArea jp-Cell-outputArea">
 <div class="jp-OutputArea-child">
 <div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
-<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="8f211fcc-fed0-48ef-bcc1-b3ba53568c98" tabindex="0">
+<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="f4bcd321-7b50-4179-bee6-df73ac1af0f6" tabindex="0">
 <script type="text/javascript">
-var element = document.getElementById('8f211fcc-fed0-48ef-bcc1-b3ba53568c98');
+var element = document.getElementById('f4bcd321-7b50-4179-bee6-df73ac1af0f6');
 (function(root) {
   function now() {
     return new Date();
@@ -6942,9 +6942,9 @@ <h2 id="example-data">Example data<a class="anchor-link" href="#example-data">¶
 </div>
 <div class="jp-OutputArea-child">
 <div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
-<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="8211d74f-37f3-4fc3-9b93-625a2cd6d4b6" tabindex="0">
+<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="3ed2e5e4-1b4e-41ba-955b-82824af3a98b" tabindex="0">
 <script type="text/javascript">
-var element = document.getElementById('8211d74f-37f3-4fc3-9b93-625a2cd6d4b6');
+var element = document.getElementById('3ed2e5e4-1b4e-41ba-955b-82824af3a98b');
 
 if ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {
   window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}
diff --git a/0.5/notebooks/30_LFP/index.html b/0.5/notebooks/30_LFP/index.html
index 605af9175..c9d44dc97 100644
--- a/0.5/notebooks/30_LFP/index.html
+++ b/0.5/notebooks/30_LFP/index.html
@@ -5336,9 +5336,9 @@ <h2 id="create-filters">Create Filters<a class="anchor-link" href="#create-filte
 <div class="jp-OutputArea jp-Cell-outputArea">
 <div class="jp-OutputArea-child">
 <div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
-<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="5fed768a-c3a9-4f27-878c-8c4641cd20a5" tabindex="0">
+<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="ec01e763-0bad-4687-9ee7-a53d3539814a" tabindex="0">
 <script type="text/javascript">
-var element = document.getElementById('5fed768a-c3a9-4f27-878c-8c4641cd20a5');
+var element = document.getElementById('ec01e763-0bad-4687-9ee7-a53d3539814a');
 (function(root) {
   function now() {
     return new Date();
@@ -5619,9 +5619,9 @@ <h2 id="create-filters">Create Filters<a class="anchor-link" href="#create-filte
 </div>
 <div class="jp-OutputArea-child">
 <div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
-<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="1ea88e2d-b21a-49f2-b7db-a3690af610ec" tabindex="0">
+<div class="jp-RenderedJavaScript jp-OutputArea-output" data-mime-type="application/javascript" id="d94694b4-690b-4f79-a86b-bb43b20e59e6" tabindex="0">
 <script type="text/javascript">
-var element = document.getElementById('1ea88e2d-b21a-49f2-b7db-a3690af610ec');
+var element = document.getElementById('d94694b4-690b-4f79-a86b-bb43b20e59e6');
 
 if ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {
   window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}
diff --git a/0.5/notebooks/index.html b/0.5/notebooks/index.html
index a1cd8632c..c312de58e 100644
--- a/0.5/notebooks/index.html
+++ b/0.5/notebooks/index.html
@@ -4636,13 +4636,13 @@ <h2 id="4-decoding-pipeline">4. Decoding Pipeline<a class="headerlink" href="#4-
 Decoding can be from sorted or from unsorted data using spike waveform features
 (so-called clusterless decoding).</p>
 <p>The first notebook
-(<a href="./41_Extracting_Clusterless_Waveform_Features.ipynb">Extracting Clusterless Waveform Features</a>)
+(<a href="40_Extracting_Clusterless_Waveform_Features/">Extracting Clusterless Waveform Features</a>)
 in this series shows how to retrieve the spike waveform features used for
 clusterless decoding.</p>
-<p>The second notebook (<a href="./42_Decoding_Clusterless.ipynb">Clusterless Decoding</a>)
+<p>The second notebook (<a href="41_Decoding_Clusterless/">Clusterless Decoding</a>)
 shows a detailed example of how to decode the position of the animal from the
 spike waveform features. The third notebook
-(<a href="./43_Decoding_SortedSpikes.ipynb">Decoding</a>) shows how to decode the position
+(<a href="42_Decoding_SortedSpikes/">Decoding</a>) shows how to decode the position
 of the animal from the sorted spikes.</p>
 <h2 id="developer-note">Developer note<a class="headerlink" href="#developer-note" title="Permanent link">&para;</a></h2>
 <p>The <code>py_scripts</code> directory contains the same notebook data in <code>.py</code> form to
@@ -4654,6 +4654,10 @@ <h2 id="developer-note">Developer note<a class="headerlink" href="#developer-not
 </span><span id="__span-0-4"><a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a>black<span class="w"> </span>.
 </span></code></pre></div>
 <p>Unfortunately, jupytext-generated py script are not black-compliant by default.</p>
+<p>You can ensure black compliace with the <code>pre-commit</code> hook by running</p>
+<div class="language-bash highlight"><pre><span></span><code><span id="__span-1-1"><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>pip<span class="w"> </span>install<span class="w"> </span>pre-commit
+</span></code></pre></div>
+<p>This will run black whenever you commit changes to the repository.</p>
 
 
 
diff --git a/0.5/search/search_index.json b/0.5/search/search_index.json
index f187ea6fc..9c7c54114 100644
--- a/0.5/search/search_index.json
+++ b/0.5/search/search_index.json
@@ -1 +1 @@
-{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Spyglass", "text": "<p>Spyglass is an open-source software framework designed to offer reliable and reproducible analysis of neuroscience data and sharing of the results with collaborators and the broader community.</p> <p>Features of Spyglass include:</p> <ul> <li>Standardized data storage - Spyglass uses the open-source     Neurodata Without Borders: Neurophysiology (NWB:N)     format to ingest and store processed data. NWB:N is a standard set by the     BRAIN Initiative for neurophysiological data     (R\u00fcbel et al., 2022).</li> <li>Reproducible analysis - Spyglass uses DataJoint     to ensure that all analysis is reproducible. DataJoint is a data management     system that automatically tracks dependencies between data and analysis     code. This ensures that all analysis is reproducible and that the results     are automatically updated when the data or analysis code changes.</li> <li>Common analysis tools - Spyglass provides easy usage of the open-source     packages SpikeInterface,     Ghostipy, and     DeepLabCut for common analysis     tasks. These packages are well-documented and have active developer     communities.</li> <li>Interactive data visualization - Spyglass uses     figurl to create interactive     data visualizations that can be shared with collaborators and the broader     community. These visualizations are hosted on the web and can be viewed in     any modern web browser. The interactivity allows users to explore the data     and analysis results in detail.</li> <li>Sharing results - Spyglass enables sharing of data and analysis results     via Kachery, a     decentralized content addressable data sharing platform. Kachery Cloud     allows users to access the database and pull data and analysis results     directly to their local machine.</li> <li>Pipeline versioning - Processing and analysis of data in neuroscience is     often dynamic, requiring new features. Spyglass uses Merge tables to     ensure that analysis pipelines can be versioned. This allows users to easily     use and compare results from different versions of the analysis pipeline     while retaining the ability to access previously generated results.</li> <li>Cautious Delete - Spyglass uses a <code>cautious delete</code> feature to ensure that     data is not accidentally deleted by other users. When a user deletes data,     Spyglass will first check to see if the data belongs to another team of     users. This enables teams of users to work collaboratively on the same     database without worrying about accidentally deleting each other's data.</li> </ul>"}, {"location": "#getting-started", "title": "Getting Started", "text": "<p>This site hosts both installation instructions and tutorials to help you get started with Spyglass. We recommend running the notebooks yourself. They can be downloaded from GitHub here.</p>"}, {"location": "#diving-deeper", "title": "Diving Deeper", "text": "<p>The API Reference provides a detailed description of all the tables and class functions in Spyglass via python docstrings. Potential contributors should also read the Developer Guide. Those interested in in hosting a Spyglass instance for their own data should read the database management guide.</p> <p>We have a series of additional docs under the misc folder that may be helpful. Our changelog highlights the changes that have been made to Spyglass over time and the copyright page contains license information.</p>"}, {"location": "#citing-spyglass", "title": "Citing Spyglass", "text": "<p>Lee, K.H.*, Denovellis, E.L.*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Tritt, A., R\u00fcbel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. 10.1101/2024.01.25.577295.</p> <p>* Equal contribution</p> <p>See paper related code here.</p>"}, {"location": "CHANGELOG/", "title": "Change Log", "text": ""}, {"location": "CHANGELOG/#052-unreleased", "title": "[0.5.2] (Unreleased)", "text": "<ul> <li>Refactor <code>TableChain</code> to include <code>_searched</code> attribute. #867</li> <li>Fix errors in config import #882</li> </ul>"}, {"location": "CHANGELOG/#051-march-7-2024", "title": "[0.5.1] (March 7, 2024)", "text": ""}, {"location": "CHANGELOG/#infrastructure", "title": "Infrastructure", "text": "<ul> <li>Add user roles to <code>database_settings.py</code>. #832</li> <li>Fix redundancy in <code>waveforms_dir</code> #857</li> <li>Revise <code>dj_chains</code> to permit undirected paths for paths with multiple Merge     Tables. #846</li> </ul>"}, {"location": "CHANGELOG/#pipelines", "title": "Pipelines", "text": "<ul> <li>Position:<ul> <li>Fixes to <code>environment-dlc.yml</code> restricting tensortflow #834</li> <li>Video restriction for multicamera epochs #834</li> <li>Fixes to <code>_convert_mp4</code> #834</li> <li>Replace deprecated calls to <code>yaml.safe_load()</code> #834</li> </ul> </li> <li>Spikesorting:<ul> <li>Increase<code>spikeinterface</code> version to &gt;=0.99.1, &lt;0.100 #852</li> <li>Bug fix in single artifact interval edge case #859</li> <li>Bug fix in FigURL #871</li> </ul> </li> <li>LFP<ul> <li>In LFPArtifactDetection, only apply referencing if explicitly selected #863</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#050-february-9-2024", "title": "0.5.0 (February 9, 2024)", "text": ""}, {"location": "CHANGELOG/#infrastructure_1", "title": "Infrastructure", "text": "<ul> <li>Docs:<ul> <li>Additional documentation. #690</li> <li>Add overview of Spyglass to docs. #779</li> <li>Update docs to reflect new notebooks. #776</li> </ul> </li> <li>Mixin:<ul> <li>Add Mixin class to centralize <code>fetch_nwb</code> functionality. #692, #734</li> <li>Refactor restriction use in <code>delete_downstream_merge</code> #703</li> <li>Add <code>cautious_delete</code> to Mixin class<ul> <li>Initial implementation. #711, #762</li> <li>More robust caching of join to downstream tables. #806</li> <li>Overwrite datajoint <code>delete</code> method to use <code>cautious_delete</code>. #806</li> <li>Reverse join order for session summary. #821</li> <li>Add temporary logging of use to <code>common_usage</code>. #811, #821</li> </ul> </li> </ul> </li> <li>Merge Tables:<ul> <li>UUIDs: Revise Merge table uuid generation to include source. #824</li> <li>UUIDs: Remove mutual exclusivity logic due to new UUID generation. #824</li> <li>Add method for <code>merge_populate</code>. #824</li> </ul> </li> <li>Linting:<ul> <li>Clean up following pre-commit checks. #688</li> <li>Update linting for Black 24. #808</li> </ul> </li> <li>Misc:<ul> <li>Add <code>deprecation_factory</code> to facilitate table migration. #717</li> <li>Add Spyglass logger. #730</li> <li>Increase pytest coverage for <code>common</code>, <code>lfp</code>, and <code>utils</code>. #743</li> <li>Steamline dependency management. #822</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#pipelines_1", "title": "Pipelines", "text": "<ul> <li>Common:<ul> <li><code>IntervalList</code>: Add secondary key <code>pipeline</code> #742</li> <li>Add <code>common_usage</code> table. #811, #821, #824</li> <li>Add catch errors during <code>populate_all_common</code>. #824</li> </ul> </li> <li>Spike sorting:<ul> <li>Add SpikeSorting V1 pipeline. #651</li> <li>Move modules into spikesorting.v0 #807</li> </ul> </li> <li>LFP:<ul> <li>Minor fixes to LFPBandV1 populator and <code>make</code>. #706, #795</li> <li>LFPV1: Fix error for multiple lfp settings on same data #775</li> </ul> </li> <li>Linearization:<ul> <li>Minor fixes to LinearizedPositionV1 pipeline #695</li> <li>Rename <code>position_linearization</code> -&gt; <code>linearization</code>. #717</li> <li>Migrate tables: <code>common_position</code> -&gt; <code>linearization.v0</code>. #717</li> </ul> </li> <li>Position:<ul> <li>Refactor input validation in DLC pipeline. #688</li> <li>DLC path handling from config, and normalize naming convention. #722</li> <li>Fix in place column bug #752</li> </ul> </li> <li>Decoding:<ul> <li>Add <code>decoding</code> pipeline V1. #731, #769, #819</li> <li>Add a table to store the decoding results #731</li> <li>Use the new <code>non_local_detector</code> package for decoding #731</li> <li>Allow multiple spike waveform features for clusterless decoding #731</li> <li>Reorder notebooks #731</li> <li>Add fetch class functionality to <code>Merge</code> table. #783, #786</li> <li>Add ability to filter sorted units in decoding #807</li> <li>Rename SortedSpikesGroup.SortGroup to SortedSpikesGroup.Units #807</li> <li>Change methods with load_... to fetch_... for consistency #807</li> <li>Use merge table methods to access part methods #807</li> </ul> </li> <li>MUA<ul> <li>Add MUA pipeline V1. #731, #819</li> </ul> </li> <li>Ripple<ul> <li>Add figurl to Ripple pipeline #819</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#043-november-7-2023", "title": "0.4.3 (November 7, 2023)", "text": "<ul> <li>Migrate <code>config</code> helper scripts to Spyglass codebase. #662</li> <li>Revise contribution guidelines. #655</li> <li>Minor bug fixes. #656, #657, #659, #651, #671</li> <li>Add setup instruction specificity.</li> <li>Reduce primary key varchar allocation aross may tables. #664</li> </ul>"}, {"location": "CHANGELOG/#042-october-10-2023", "title": "0.4.2 (October 10, 2023)", "text": ""}, {"location": "CHANGELOG/#infrastructure-support", "title": "Infrastructure / Support", "text": "<ul> <li>Bumped Python version to 3.9. #583</li> <li>Updated user management helper scripts for MySQL 8. #650</li> <li>Centralized config/path handling to permit setting via datajoint config. #593</li> <li>Fixed Merge Table deletes: error specificity and transaction context. #617</li> </ul>"}, {"location": "CHANGELOG/#pipelines_2", "title": "Pipelines", "text": "<ul> <li>Common:<ul> <li>Added support multiple cameras per epoch. #557</li> <li>Removed <code>common_backup</code> schema. #631</li> <li>Added support for multiple position objects per NWB in <code>common_behav</code> via     PositionSource.SpatialSeries and RawPosition.PosObject #628, #616. Note:     Existing functions have been made compatible, but column labels for     <code>RawPosition.fetch1_dataframe</code> may change.</li> </ul> </li> <li>Spike sorting:<ul> <li>Added pipeline populator. #637, #646, #647</li> <li>Fixed curation functionality for <code>nn_isolation</code>. #597, #598</li> </ul> </li> <li>Position: Added position interval/epoch mapping via PositionIntervalMap. #620,     #621, #627</li> <li>LFP: Refactored pipeline. #594, #588, #605, #606, #607, #608, #615, #629</li> </ul>"}, {"location": "CHANGELOG/#041-june-30-2023", "title": "0.4.1 (June 30, 2023)", "text": "<ul> <li>Add mkdocs automated deployment. #527, #537, #549, #551</li> <li>Add class for Merge Tables. #556, #564, #565</li> </ul>"}, {"location": "CHANGELOG/#040-may-22-2023", "title": "0.4.0 (May 22, 2023)", "text": "<ul> <li>Updated call to <code>spikeinterface.preprocessing.whiten</code> to use dtype np.float16.     #446,</li> <li>Updated default spike sorting metric parameters. #447</li> <li>Updated whitening to be compatible with recent changes in spikeinterface when     using mountainsort. #449</li> <li>Moved LFP pipeline to <code>src/spyglass/lfp/v1</code> and addressed related usability     issues. #468, #478, #482, #484, #504</li> <li>Removed whiten parameter for clusterless thresholder. #454</li> <li>Added plot to plot all DIO events in a session. #457</li> <li>Added file sharing functionality through kachery_cloud. #458, #460</li> <li>Pinned numpy version to <code>numpy&lt;1.24</code></li> <li>Added scripts to add guests and collaborators as users. #463</li> <li>Cleaned up installation instructions in repo README. #467</li> <li>Added checks in decoding visualization to ensure time dimensions are the     correct length.</li> <li>Fixed artifact removed valid times. #472</li> <li>Added codespell workflow for spell checking and fixed typos. #471</li> <li>Updated LFP code to save LFP as <code>pynwb.ecephys.LFP</code> type. #475</li> <li>Added artifact detection to LFP pipeline. #473</li> <li>Replaced calls to <code>spikeinterface.sorters.get_default_params</code> with     <code>spikeinterface.sorters.get_default_sorter_params</code>. #486</li> <li>Updated position pipeline and added functionality to handle pose estimation     through DeepLabCut. #367, #505</li> <li>Updated <code>environment_position.yml</code>. #502</li> <li>Renamed <code>FirFilter</code> class to <code>FirFilterParameters</code>. #512</li> </ul>"}, {"location": "CHANGELOG/#034-march-30-2023", "title": "0.3.4 (March 30, 2023)", "text": "<ul> <li>Fixed error in spike sorting pipeline referencing the \"probe_type\" column     which is no longer accessible from the <code>Electrode</code> table. #437</li> <li>Fixed error when inserting an NWB file that does not have a probe     manufacturer. #433, #436</li> <li>Fixed error when adding a new <code>DataAcquisitionDevice</code> and a new <code>ProbeType</code>.     #436</li> <li>Fixed inconsistency between capitalized/uncapitalized versions of \"Intan\" for     DataAcquisitionAmplifier and DataAcquisitionDevice.adc_circuit. #430, #438</li> </ul>"}, {"location": "CHANGELOG/#033-march-29-2023", "title": "0.3.3 (March 29, 2023)", "text": "<ul> <li>Fixed errors from referencing the changed primary key for <code>Probe</code>. #429</li> </ul>"}, {"location": "CHANGELOG/#032-march-28-2023", "title": "0.3.2 (March 28, 2023)", "text": "<ul> <li>Fixed import of <code>common_nwbfile</code>. #424</li> </ul>"}, {"location": "CHANGELOG/#031-march-24-2023", "title": "0.3.1 (March 24, 2023)", "text": "<ul> <li>Fixed import error due to <code>sortingview.Workspace</code>. #421</li> </ul>"}, {"location": "CHANGELOG/#030-march-24-2023", "title": "0.3.0 (March 24, 2023)", "text": "<ul> <li>Refactor common for non Frank Lab data, allow file-based mods #420</li> <li>Allow creation and linkage of device metadata from YAML #400</li> <li>Move helper functions to utils directory #386</li> </ul>"}, {"location": "LICENSE/", "title": "Copyright", "text": "<p>Copyright (c) 2020-present Loren Frank</p> <p>Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:</p> <p>The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.</p> <p>THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BEsq LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.</p>"}, {"location": "contribute/", "title": "Developer notes", "text": "<p>Notes on how the repo / database is organized, intended for a new developer.</p>"}, {"location": "contribute/#development-workflow", "title": "Development workflow", "text": "<p>New contributors should follow the Fork-and-Branch workflow. See GitHub instructions here.</p> <p>Regular contributors may choose to follow the Feature Branch Workflow for features that will involve multiple contributors.</p>"}, {"location": "contribute/#code-organization", "title": "Code organization", "text": "<ul> <li>Tables are grouped into schemas by topic (e.g., <code>common_metrics</code>)</li> <li>Schemas<ul> <li>Are defined in a <code>py</code> pile.</li> <li>Correspond to MySQL 'databases'.</li> <li>Are organized into modules (e.g., <code>common</code>) by folders.</li> </ul> </li> <li>The common module<ul> <li>In principle, contains schema that are shared across all projects.</li> <li>In practice, contains shared tables (e.g., Session) and the first draft of     schemas that have since been split into their own     modality-specific\\     modules (e.g., <code>lfp</code>)</li> <li>Should not be added to without discussion.</li> </ul> </li> <li>A pipeline<ul> <li>Refers to a set of tables used for processing data of a particular modality     (e.g., LFP, spike sorting, position tracking).</li> <li>May span multiple schema.</li> </ul> </li> <li>For analysis that will be only useful to you, create your own schema.</li> </ul>"}, {"location": "contribute/#types-of-tables", "title": "Types of tables", "text": "<p>Spyglass uses DataJoint's default table tiers.</p> <p>By convention, an individual pipeline has one or more the following table types:</p> <ul> <li>Common/Multi-pipeline table</li> <li>NWB ingestion table</li> <li>Parameters table</li> <li>Selection table</li> <li>Data table</li> <li>Merge Table (see also doc</li> </ul>"}, {"location": "contribute/#commonmulti-pipeline", "title": "Common/Multi-pipeline", "text": "<p>Tables shared across multiple pipelines for shared data types.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Manual</code></li> <li>Examples: <code>IntervalList</code> (time interval for any analysis), <code>AnalysisNwbfile</code>     (analysis NWB files)</li> </ul> <p>Note: Because these are stand-alone tables not part of the dependency structure, developers should include enough information to link entries back to the pipeline where the data is used.</p>"}, {"location": "contribute/#nwb-ingestion", "title": "NWB ingestion", "text": "<p>Automatically populated when an NWB file is ingested (i.e., <code>dj.Imported</code>) to keep track of object hashes (i.e., <code>object_id</code>) in the NWB file. All such tables should be included in the <code>make</code> method of <code>Session</code>.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Imported</code></li> <li>Primary key: foreign key from <code>Session</code></li> <li>Non-primary key: <code>object_id</code>, the unique hash of an object in the NWB file.</li> <li>Examples: <code>Raw</code>, <code>Institution</code>, etc.</li> <li>Required methods:<ul> <li><code>make</code>: must read information from an NWB file and insert it to the table.</li> <li><code>fetch_nwb</code>: retrieve the data specified by the object ID.</li> </ul> </li> </ul>"}, {"location": "contribute/#parameters", "title": "Parameters", "text": "<p>Stores the set of values that may be used in an analysis.</p> <ul> <li>Naming convention: end with <code>Parameters</code> or <code>Params</code></li> <li>Data tier: <code>dj.Manual</code>, or <code>dj.Lookup</code></li> <li>Primary key: <code>{pipeline}_params_name</code>, <code>varchar</code></li> <li>Non-primary key: <code>{pipeline}_params</code>, <code>blob</code> - dict of parameters</li> <li>Examples: <code>RippleParameters</code>, <code>DLCModelParams</code></li> <li>Possible method: if <code>dj.Manual</code>, include <code>insert_default</code></li> </ul> <p>Notes: Some early instances of Parameter tables (a) used non-primary keys for each individual parameter, and (b) use the Manual rather than Lookup tier, requiring a class method to insert defaults.</p>"}, {"location": "contribute/#selection", "title": "Selection", "text": "<p>A staging area to pair sessions with parameter sets, allowing us to be selective in the analyses we run. It may not make sense to pair every paramset with every session.</p> <ul> <li>Naming convention: end with <code>Selection</code></li> <li>Data tier: <code>dj.Manual</code></li> <li>Primary key(s): Foreign key references to<ul> <li>one or more NWB or data tables</li> <li>optionally, one or more parameter tables</li> </ul> </li> <li>Non-primary key: None</li> <li>Examples: <code>MetricSelection</code>, <code>LFPSelection</code></li> </ul> <p>It is possible for a Selection table to collect information from more than one Parameter table. For example, the Selection table for spike sorting holds information about both the interval (<code>SortInterval</code>) and the group of electrodes (<code>SortGroup</code>) to be sorted.</p>"}, {"location": "contribute/#data", "title": "Data", "text": "<p>The output of processing steps associated with a selection table. Has a <code>make</code> method that carries out the computation specified in the Selection table when <code>populate</code> is called.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Computed</code></li> <li>Primary key: Foreign key reference to a Selection table.</li> <li>Non-primary key: <code>analysis_file_name</code> inherited from <code>AnalysisNwbfile</code> table     (i.e., name of the analysis NWB file that will hold the output of the     computation).</li> <li>Required methods:<ul> <li><code>make</code>: carries out the computation and insert a new entry; must also create     an analysis NWB file and insert it to the <code>AnalysisNwbfile</code> table. Note     that this method is never called directly; it is called via <code>populate</code>.     Multiple entries can be run in parallel when called with     <code>reserve_jobs=True</code>.</li> <li><code>delete</code>: extension of the <code>delete</code> method that checks user privilege before     deleting entries as a way to prevent accidental deletion of computations     that take a long time (see below).</li> </ul> </li> <li>Example: <code>QualityMetrics</code>, <code>LFPV1</code></li> </ul>"}, {"location": "contribute/#merge", "title": "Merge", "text": "<p>Following a convention outlined in the dedicated doc, merges the output of different pipelines dedicated to the same modality as part tables (e.g., common LFP, LFP v1, imported LFP) to permit unified downstream processing.</p> <ul> <li>Naming convention: <code>{Pipeline}Output</code></li> <li>Data tier: custom <code>_Merge</code> class</li> <li>Primary key: <code>merge_id</code>, <code>uuid</code></li> <li>Non-primary key: <code>source</code>, <code>varchar</code> table name associated with that entry</li> <li>Required methods: None - see custom class methods with <code>merge_</code> prefix</li> <li>Example: <code>LFPOutput</code>, <code>PositionOutput</code></li> </ul>"}, {"location": "contribute/#integration-with-nwb", "title": "Integration with NWB", "text": ""}, {"location": "contribute/#nwb-files", "title": "NWB files", "text": "<p>NWB files contain everything about the experiment and form the starting point of all analyses.</p> <ul> <li>Naming: <code>{animal name}YYYYMMDD.nwb</code></li> <li>Storage:<ul> <li>On disk, directory identified by <code>settings.py</code> as <code>raw_dir</code> (e.g.,     <code>/stelmo/nwb/raw</code>)</li> <li>In database, in the <code>Nwbfile</code> table</li> </ul> </li> <li>Copies:<ul> <li>made with an underscore <code>{animal name}YYYYMMDD_.nwb</code></li> <li>stored in the same <code>raw_dir</code></li> <li>contain pointers to objects in original file</li> <li>permit adding new parts to the NWB file without risk of corrupting the     original data</li> </ul> </li> </ul>"}, {"location": "contribute/#analysis-files", "title": "Analysis files", "text": "<p>Hold the results of intermediate steps in the analysis.</p> <ul> <li>Naming: <code>{animal name}YYYYMMDD_{10-character random string}.nwb</code></li> <li>Storage:<ul> <li>On disk, directory identified by <code>settings.py</code> as <code>analysis_dir</code> (e.g.,     <code>/stelmo/nwb/analysis</code>). Items are further sorted into folders matching     original NWB file name</li> <li>In database, in the <code>AnalysisNwbfile</code> table.</li> </ul> </li> <li>Examples: filtered recordings, spike times of putative units after sorting, or     waveform snippets.</li> </ul> <p>Note: Because NWB files and analysis files exist both on disk and listed in tables, these can become out of sync. You can 'equalize' the database table lists and the set of files on disk by running <code>cleanup</code> method, which deletes any files not listed in the table from disk.</p>"}, {"location": "contribute/#reading-and-writing-recordings", "title": "Reading and writing recordings", "text": "<p>Recordings start out as an NWB file, which is opened as a <code>NwbRecordingExtractor</code>, a class in <code>spikeinterface</code>. When using <code>sortingview</code> for visualizing the results of spike sorting, this recording is saved again in HDF5 format. This duplication should be resolved in the future.</p>"}, {"location": "contribute/#naming-convention", "title": "Naming convention", "text": "<p>The following objects should be uniquely named.</p> <ul> <li>Recordings: Underscore-separated concatenations of uniquely defining     features,     <code>NWBFileName_IntervalName_ElectrodeGroupName_PreprocessingParamsName</code>.</li> <li>SpikeSorting: Adds <code>SpikeSorter_SorterParamName</code> to the name of the     recording.</li> <li>Waveforms: Adds <code>_WaveformParamName</code> to the name of the sorting.</li> <li>Quality metrics: Adds <code>_MetricParamName</code> to the name of the waveform.</li> <li>Analysis NWB files:     <code>NWBFileName_IntervalName_ElectrodeGroupName_PreprocessingParamsName.nwb</code></li> <li>Each recording and sorting is given truncated UUID strings as part of     concatenations.</li> </ul> <p>Following broader Python conventions, methods a method that will not be explicitly called by the user should start with <code>_</code></p>"}, {"location": "contribute/#time", "title": "Time", "text": "<p>The <code>IntervalList</code> table stores all time intervals in the following format: <code>[start_time, stop_time]</code>, which represents a contiguous time of valid data. These are used to exclude any invalid timepoints, such as missing data from a faulty connection.</p> <ul> <li>Intervals can be nested for a set of disjoint intervals.</li> <li>Some recordings have explicit     PTP timestamps     associated with each sample. Some older recordings are missing PTP times,     and times must be inferred from the TTL pulses from the camera.</li> </ul>"}, {"location": "contribute/#misc", "title": "Misc", "text": "<ul> <li>During development, we suggest using a Docker container. See     example.</li> <li>DataJoint is unable to set delete permissions on a per-table basis. If a user     is able to delete entries in a given table, she can delete entries in any     table in the schema. The <code>SpikeSorting</code> table extends the built-in <code>delete</code>     method to check if the username matches a list of allowed users when     <code>delete</code> is called. Issues #226 and #586 track the progress of generalizing     this feature.</li> <li><code>numpy</code> style docstrings will be interpreted by API docs. To check for     compliance, monitor the std out when building docs (see <code>docs/README.md</code>)</li> <li><code>fetch_nwb</code> is currently reperated across many tables. For progress on a fix,     follow issue #530</li> </ul>"}, {"location": "contribute/#making-a-release", "title": "Making a release", "text": "<p>Spyglass follows Semantic Versioning with versioning of the form <code>X.Y.Z</code> (e.g., <code>0.4.2</code>).</p> <ol> <li>In <code>CITATION.cff</code>, update the <code>version</code> key.</li> <li>Make a pull request with changes.</li> <li>After the pull request is merged, pull this merge commit and tag it with     <code>git tag {version}</code></li> <li>Publish the new release tag. Run <code>git push origin {version}</code>. This will     rebuild docs and push updates to PyPI.</li> <li>Make a new     release on GitHub.</li> </ol>"}, {"location": "installation/", "title": "Installation", "text": "<p>Note: Developers, or those who wish to add features or otherwise work on the codebase should follow the same steps below, but install Spyglass as editable with the <code>-e</code> flag: <code>pip install -e /path/to/spyglass</code></p>"}, {"location": "installation/#basic-installation", "title": "Basic Installation", "text": "<p>For basic installation steps, see the Setup notebook 'local installation' section, including python, mamba (for managing a virtual environment), VSCode, Jupyter, and git. This notebook also covers database access.</p>"}, {"location": "installation/#additional-packages", "title": "Additional Packages", "text": "<p>Some pipelines require installation of additional packages.</p> <p>The spike sorting pipeline relies on <code>spikeinterface</code> and optionally <code>mountainsort4</code>.</p> <pre><code>pip install spikeinterface[full,widgets]\npip install mountainsort4\n</code></pre> <p>WARNING: If you are on an M1 Mac, you need to install <code>pyfftw</code> via <code>conda</code> BEFORE installing <code>ghostipy</code>:</p> <pre><code>conda install -c conda-forge pyfftw\n</code></pre> <p>The LFP pipeline uses <code>ghostipy</code>:</p> <pre><code>pip install ghostipy\n</code></pre>"}, {"location": "installation/#database-access", "title": "Database access", "text": "<p>For basic installation steps, see the Setup notebook 'database connection' section. For additional details, see the DataJoint documentation.</p>"}, {"location": "installation/#config", "title": "Config", "text": ""}, {"location": "installation/#via-file-recommended", "title": "Via File (Recommended)", "text": "<p>A <code>dj_local_conf.json</code> file in your current directory when launching python can hold all the specifics needed to connect to a database. This can include different directories for different pipelines. If only the Spyglass <code>base</code> is specified, other subfolder names are assumed from defaults. See <code>dj_local_conf_example.json</code> for the full set of options. This example can be copied and saved as <code>dj_local_conf.json</code> to set the configuration for a given folder. Alternatively, it can be saved as <code>.datajoint_config.json</code> in a user's home directory to be accessed globally. See DataJoint docs for more details.</p> <p>Note that raw and analysis folder locations should be specified under both <code>stores</code> and <code>custom</code> sections of the config file. The <code>stores</code> section is used by DataJoint to store the location of files referenced in database, while the <code>custom</code> section is used by Spyglass. Spyglass will check that these sections match on startup.</p>"}, {"location": "installation/#via-environment-variables", "title": "Via Environment Variables", "text": "<p>Older versions of Spyglass relied exclusively on environment for config. If <code>spyglass_dirs</code> is not found in the config file, Spyglass will look for environment variables. These can be set either once in a terminal session, or permanently in a unix settings file (e.g., <code>.bashrc</code> or <code>.bash_profile</code>) in your home directory.</p> <pre><code>export SPYGLASS_BASE_DIR=\"/stelmo/nwb\"\nexport SPYGLASS_RECORDING_DIR=\"$SPYGLASS_BASE_DIR/recording\"\nexport SPYGLASS_SORTING_DIR=\"$SPYGLASS_BASE_DIR/sorting\"\nexport SPYGLASS_VIDEO_DIR=\"$SPYGLASS_BASE_DIR/video\"\nexport SPYGLASS_WAVEFORMS_DIR=\"$SPYGLASS_BASE_DIR/waveforms\"\nexport SPYGLASS_TEMP_DIR=\"$SPYGLASS_BASE_DIR/tmp\"\nexport DJ_SUPPORT_FILEPATH_MANAGEMENT=\"TRUE\"\n</code></pre> <p>To load variables from a <code>.bashrc</code> file, run <code>source ~/.bashrc</code> in a terminal.</p>"}, {"location": "installation/#temporary-directory", "title": "Temporary directory", "text": "<p>A temporary directory will speed up spike sorting. If unspecified by either method above, it will be assumed as a <code>tmp</code> subfolder relative to the base path. Be sure it has enough free space (ideally at least 500GB).</p>"}, {"location": "installation/#subfolders", "title": "Subfolders", "text": "<p>If subfolders do not exist, they will be created automatically. If unspecified by either method above, they will be assumed as <code>recording</code>, <code>sorting</code>, <code>video</code>, etc. subfolders relative to the base path.</p>"}, {"location": "installation/#file-manager", "title": "File manager", "text": "<p><code>kachery-cloud</code> is a file manager for Frank Lab collaborators who do not have access to the lab's production database.</p> <p>To customize <code>kachery</code> file paths, see <code>dj_local_conf_example.json</code> or set the following variables in your unix settings file (e.g., <code>.bashrc</code>). If unspecified, the defaults below are assumed.</p> <pre><code>export KACHERY_CLOUD_DIR=\"$SPYGLASS_BASE_DIR/.kachery-cloud\"\nexport KACHERY_TEMP_DIR=\"$SPYGLASS_BASE_DIR/tmp\"\n</code></pre> <p>Be sure to load these with <code>source ~/.bashrc</code> to persist changes.</p>"}, {"location": "installation/#test-connection", "title": "Test connection", "text": "<p>Finally, open up a python console (e.g., run <code>ipython</code> from terminal) and import <code>spyglass</code> to check that the installation has worked.</p> <pre><code>from spyglass.common import Nwbfile\n\nNwbfile()\n</code></pre>"}, {"location": "api/", "title": "API Docs", "text": "<p>The files in this directory are automatically generated from the docstrings in the source code. They include descriptions of each of the DataJoint tables and other classes/methods within Spyglass.</p> <p>These docs are updated any time a new release is made or a tag is pushed to the repository.</p>"}, {"location": "api/settings/", "title": "settings.py", "text": ""}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig", "title": "<code>SpyglassConfig</code>", "text": "<p>Gets Spyglass dirs from dj.config or environment variables.</p> <p>Uses SpyglassConfig.relative_dirs to (a) gather user settings from dj.config or os environment variables or defaults relative to base, in that order (b) set environment variables, and (c) make dirs that don't exist. NOTE: when passed a base_dir, it will ignore env vars to facilitate testing.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>class SpyglassConfig:\n    \"\"\"Gets Spyglass dirs from dj.config or environment variables.\n\n    Uses SpyglassConfig.relative_dirs to (a) gather user\n    settings from dj.config or os environment variables or defaults relative to\n    base, in that order (b) set environment variables, and (c) make dirs that\n    don't exist. NOTE: when passed a base_dir, it will ignore env vars to\n    facilitate testing.\n    \"\"\"\n\n    def __init__(self, base_dir: str = None, **kwargs):\n        \"\"\"\n        Initializes a new instance of the class.\n\n        Parameters\n        ----------\n        base_dir (str)\n            The base directory.\n\n        Attributes\n        ----------\n        supplied_base_dir (str)\n            The base directory passed to the class.\n        config_defaults (dict)\n            Default settings for the config.\n        relative_dirs (dict)\n            Relative dirs for each prefix (spyglass, kachery, dlc). Relative\n            to respective base_dir. Created on init.\n        dj_defaults (dict)\n            Default settings for datajoint.\n        env_defaults (dict)\n            Default settings for environment variables.\n        _config (dict)\n            Cached config settings.\n        _debug_mode (bool)\n            True if debug_mode is set. Supports skipping known bugs in test env.\n        _test_mode (bool)\n            True if test_mode is set. Required for pytests to run without\n            prompts.\n        \"\"\"\n        self.supplied_base_dir = base_dir\n        self._config = dict()\n        self.config_defaults = dict(prepopulate=True)\n        self._debug_mode = kwargs.get(\"debug_mode\", False)\n        self._test_mode = kwargs.get(\"test_mode\", False)\n        self._dlc_base = None\n        self.load_failed = False\n\n        self.relative_dirs = {\n            # {PREFIX}_{KEY}_DIR, default dir relative to base_dir\n            \"spyglass\": {\n                \"raw\": \"raw\",\n                \"analysis\": \"analysis\",\n                \"recording\": \"recording\",\n                \"sorting\": \"spikesorting\",\n                \"waveforms\": \"waveforms\",\n                \"temp\": \"tmp\",\n                \"video\": \"video\",\n            },\n            \"kachery\": {\n                \"cloud\": \".kachery-cloud\",\n                \"storage\": \"kachery_storage\",\n                \"temp\": \"tmp\",\n            },\n            \"dlc\": {\n                \"project\": \"projects\",\n                \"video\": \"video\",\n                \"output\": \"output\",\n            },\n        }\n        self.dj_defaults = {\n            \"database.host\": kwargs.get(\"database_host\", \"lmf-db.cin.ucsf.edu\"),\n            \"database.user\": kwargs.get(\"database_user\"),\n            \"database.port\": kwargs.get(\"database_port\", 3306),\n            \"database.use_tls\": kwargs.get(\"database_use_tls\", True),\n            \"filepath_checksum_size_limit\": 1 * 1024**3,\n            \"enable_python_native_blobs\": True,\n        }\n        self.env_defaults = {\n            \"FIGURL_CHANNEL\": \"franklab2\",\n            \"DJ_SUPPORT_FILEPATH_MANAGEMENT\": \"TRUE\",\n            \"KACHERY_CLOUD_EPHEMERAL\": \"TRUE\",\n            \"HD5_USE_FILE_LOCKING\": \"FALSE\",\n        }\n\n    def load_config(\n        self,\n        base_dir=None,\n        force_reload=False,\n        on_startup: bool = False,\n        **kwargs,\n    ):\n        \"\"\"\n        Loads the configuration settings for the object.\n\n        Order of precedence, where X is base, raw, analysis, etc.:\n            1. SpyglassConfig(base_dir=\"string\") for base dir only\n            2. dj.config['custom']['{spyglass/kachery}_dirs']['X']\n            3. os.environ['{SPYGLASS/KACHERY}_{X}_DIR']\n            4. resolved_base_dir/X for non-base dirs\n\n        Parameters\n        ----------\n        base_dir: str\n            Optional. Default None. The base directory. If not provided, will\n            use the env variable or existing config.\n        force_reload: bool\n            Optional. Default False. Default skip load if already completed.\n\n        Raises\n        ------\n        ValueError\n            If base_dir is not set in either dj.config or os.environ.\n\n        Returns\n        -------\n        dict\n            list of relative_dirs and other settings (e.g., prepopulate).\n        \"\"\"\n        if not force_reload and self._config:\n            return self._config\n\n        dj_custom = dj.config.get(\"custom\", {})\n        dj_spyglass = dj_custom.get(\"spyglass_dirs\", {})\n        dj_kachery = dj_custom.get(\"kachery_dirs\", {})\n        dj_dlc = dj_custom.get(\"dlc_dirs\", {})\n\n        self._debug_mode = dj_custom.get(\"debug_mode\", False)\n        self._test_mode = kwargs.get(\"test_mode\") or dj_custom.get(\n            \"test_mode\", False\n        )\n\n        resolved_base = (\n            base_dir\n            or self.supplied_base_dir\n            or dj_spyglass.get(\"base\")\n            or os.environ.get(\"SPYGLASS_BASE_DIR\")\n        )\n\n        if resolved_base and not Path(resolved_base).exists():\n            resolved_base = Path(resolved_base).expanduser()\n        if not resolved_base or not Path(resolved_base).exists():\n            if not on_startup:  # Only warn if not on startup\n                logger.error(\n                    f\"Could not find SPYGLASS_BASE_DIR: {resolved_base}\"\n                    + \"\\n\\tCheck dj.config['custom']['spyglass_dirs']['base']\"\n                    + \"\\n\\tand os.environ['SPYGLASS_BASE_DIR']\"\n                )\n            self.load_failed = True\n            return\n\n        self._dlc_base = (\n            dj_dlc.get(\"base\")\n            or os.environ.get(\"DLC_BASE_DIR\")\n            or os.environ.get(\"DLC_PROJECT_PATH\", \"\").split(\"projects\")[0]\n            or str(Path(resolved_base) / \"deeplabcut\")\n        )\n        Path(self._dlc_base).mkdir(exist_ok=True)\n\n        config_dirs = {\"SPYGLASS_BASE_DIR\": str(resolved_base)}\n        for prefix, dirs in self.relative_dirs.items():\n            this_base = self._dlc_base if prefix == \"dlc\" else resolved_base\n            for dir, dir_str in dirs.items():\n                dir_env_fmt = self.dir_to_var(dir=dir, dir_type=prefix)\n\n                env_loc = (  # Ignore env vars if base was passed to func\n                    os.environ.get(dir_env_fmt)\n                    if not self.supplied_base_dir\n                    else None\n                )\n\n                source_config = (\n                    dj_dlc\n                    if prefix == \"dlc\"\n                    else dj_kachery if prefix == \"kachery\" else dj_spyglass\n                )\n                dir_location = (\n                    source_config.get(dir)\n                    or env_loc\n                    or str(Path(this_base) / dir_str)\n                ).replace('\"', \"\")\n\n                config_dirs.update({dir_env_fmt: str(dir_location)})\n\n        kachery_zone_dict = {\n            \"KACHERY_ZONE\": (\n                os.environ.get(\"KACHERY_ZONE\")\n                or dj.config.get(\"custom\", {}).get(\"kachery_zone\")\n                or \"franklab.default\"\n            )\n        }\n\n        loaded_env = self._load_env_vars()\n        self._set_env_with_dict(\n            {**config_dirs, **kachery_zone_dict, **loaded_env}\n        )\n        self._mkdirs_from_dict_vals(config_dirs)\n\n        self._config = dict(\n            debug_mode=self._debug_mode,\n            test_mode=self._test_mode,\n            **self.config_defaults,\n            **config_dirs,\n            **kachery_zone_dict,\n            **loaded_env,\n        )\n\n        self._set_dj_config_stores()\n\n        return self._config\n\n    def _load_env_vars(self):\n        loaded_dict = {}\n        for var, val in self.env_defaults.items():\n            loaded_dict[var] = os.getenv(var, val)\n        return loaded_dict\n\n    def _set_env_with_dict(self, env_dict):\n        # NOTE: Kept for backwards compatibility. Should be removed in future\n        # for custom paths. Keep self.env_defaults.\n        for var, val in env_dict.items():\n            os.environ[var] = str(val)\n\n    def _mkdirs_from_dict_vals(self, dir_dict):\n        if self._debug_mode:\n            return\n        for dir_str in dir_dict.values():\n            Path(dir_str).mkdir(exist_ok=True)\n\n    def _set_dj_config_stores(self, check_match=True, set_stores=True):\n        \"\"\"\n        Checks dj.config['stores'] match resolved dirs. Ensures stores set.\n\n        Parameters\n        ----------\n        dir_dict: dict\n            Dictionary of resolved dirs.\n        check_match: bool\n            Optional. Default True. Check that dj.config['stores'] match\n            resolved dirs.\n        set_stores: bool\n            Optional. Default True. Set dj.config['stores'] to resolved dirs.\n        \"\"\"\n\n        mismatch_analysis = False\n        mismatch_raw = False\n\n        if check_match:\n            dj_stores = dj.config.get(\"stores\", {})\n            store_r = dj_stores.get(\"raw\", {}).get(\"location\")\n            store_a = dj_stores.get(\"analysis\", {}).get(\"location\")\n            mismatch_raw = store_r and Path(store_r) != Path(self.raw_dir)\n            mismatch_analysis = store_a and Path(store_a) != Path(\n                self.analysis_dir\n            )\n\n        if set_stores:\n            if mismatch_raw or mismatch_analysis:\n                logger.warning(\n                    \"Setting config DJ stores to resolve mismatch.\\n\\t\"\n                    + f\"raw     : {self.raw_dir}\\n\\t\"\n                    + f\"analysis: {self.analysis_dir}\"\n                )\n            dj.config.update(self._dj_stores)\n            return\n\n        if mismatch_raw or mismatch_analysis:\n            raise ValueError(\n                \"dj.config['stores'] does not match resolved dirs.\"\n                + f\"\\n\\tdj.config['stores']: {dj_stores}\"\n                + f\"\\n\\tResolved dirs: {self._dj_stores}\"\n            )\n\n        return\n\n    def dir_to_var(self, dir: str, dir_type: str = \"spyglass\"):\n        \"\"\"Converts a dir string to an env variable name.\"\"\"\n        return f\"{dir_type.upper()}_{dir.upper()}_DIR\"\n\n    def _generate_dj_config(\n        self,\n        base_dir: str = None,\n        database_user: str = None,\n        database_password: str = None,\n        database_host: str = \"lmf-db.cin.ucsf.edu\",\n        database_port: int = 3306,\n        database_use_tls: bool = True,\n        **kwargs,\n    ):\n        \"\"\"Generate a datajoint configuration file.\n\n        Parameters\n        ----------\n        database_user : str, optional\n            The database user. If not provided, resulting config will not\n            specify.\n        database_password : str, optional\n            The database password. If not provided, resulting config will not\n            specify.\n        database_host : str, optional\n            Default lmf-db.cin.ucsf.edu. MySQL host name.\n        dapabase_port : int, optional\n            Default 3306. Port number for MySQL server.\n        database_use_tls : bool, optional\n            Default True. Use TLS encryption.\n        **kwargs: dict, optional\n            Any other valid datajoint configuration parameters.\n            Note: python will raise error for params with `.` in name.\n        \"\"\"\n\n        if database_user:\n            kwargs.update({\"database.user\": database_user})\n        if database_password:\n            kwargs.update({\"database.password\": database_password})\n\n        kwargs.update(\n            {\n                \"database.host\": database_host,\n                \"database.port\": database_port,\n                \"database.use_tls\": database_use_tls,\n            }\n        )\n\n        # `|` merges dictionaries\n        return self.dj_defaults | self._dj_stores | self._dj_custom | kwargs\n\n    def save_dj_config(\n        self,\n        save_method: str = \"global\",\n        output_filename: str = None,\n        base_dir=None,\n        set_password=True,\n        **kwargs,\n    ):\n        \"\"\"Set the dj.config parameters, set password, and save config to file.\n\n        Parameters\n        ----------\n        save_method : {'local', 'global', 'custom'}, optional\n            The method to use to save the config. If either 'local' or 'global',\n            datajoint builtins will be used to save.\n        output_filename : str or Path, optional\n            Default to datajoint global config. If save_method = 'custom', name\n            of file to generate. Must end in either be either yaml or json.\n        base_dir : str, optional\n            The base directory. If not provided, will default to the env var\n        set_password : bool, optional\n            Default True. Set the database password.\n        kwargs: dict, optional\n            Any other valid datajoint configuration parameters, including\n            database_user, database_password, database_host, database_port, etc.\n            Note: python will raise error for params with `.` in name, so use\n            underscores instead.\n        \"\"\"\n        if base_dir:\n            self.load_config(\n                base_dir=base_dir, force_reload=True, on_startup=False\n            )\n\n        if output_filename:\n            save_method = \"custom\"\n            path = Path(output_filename).expanduser()  # Expand ~\n            filepath = path if path.is_absolute() else path.absolute()\n            filepath.parent.mkdir(exist_ok=True, parents=True)\n            filepath = (\n                filepath.with_suffix(\".json\")  # ensure suffix, default json\n                if filepath.suffix not in [\".json\", \".yaml\"]\n                else filepath\n            )\n        elif save_method == \"local\":\n            filepath = Path(\".\") / dj.settings.LOCALCONFIG\n        elif save_method == \"global\":\n            filepath = Path(\"~\").expanduser() / dj.settings.GLOBALCONFIG\n        else:\n            raise ValueError(\n                \"For save_dj_config, either (a) save_method must be 'local' \"\n                + \" or 'global' or (b) must provide custom output_filename.\"\n            )\n\n        dj.config.update(self._generate_dj_config(**kwargs))\n\n        if set_password:\n            try:\n                dj.set_password()\n            except OperationalError as e:\n                warnings.warn(f\"Database connection issues. Wrong pass?\\n\\t{e}\")\n\n        user_warn = (\n            f\"Replace existing file? {filepath.resolve()}\\n\\t\"\n            + \"\\n\\t\".join([f\"{k}: {v}\" for k, v in config.items()])\n            + \"\\n\"\n        )\n\n        if (\n            not self.test_mode\n            and filepath.exists()\n            and dj.utils.user_choice(user_warn)[0] != \"y\"\n        ):\n            return\n\n        if save_method == \"global\":\n            dj.config.save_global(verbose=True)\n            return\n\n        if save_method == \"local\":\n            dj.config.save_local(verbose=True)\n            return\n\n        with open(filepath, \"w\") as outfile:\n            if filepath.suffix == \".yaml\":\n                yaml.dump(dj.config._conf, outfile, default_flow_style=False)\n            else:\n                json.dump(dj.config._conf, outfile, indent=2)\n            logger.info(f\"Saved config to {filepath}\")\n\n    @property\n    def _dj_stores(self) -&gt; dict:\n        self.load_config()\n        return {\n            \"stores\": {\n                \"raw\": {\n                    \"protocol\": \"file\",\n                    \"location\": self.raw_dir,\n                    \"stage\": self.raw_dir,\n                },\n                \"analysis\": {\n                    \"protocol\": \"file\",\n                    \"location\": self.analysis_dir,\n                    \"stage\": self.analysis_dir,\n                },\n            }\n        }\n\n    @property\n    def _dj_custom(self) -&gt; dict:\n        self.load_config()\n        return {\n            \"custom\": {\n                \"debug_mode\": str(self.debug_mode).lower(),\n                \"test_mode\": str(self._test_mode).lower(),\n                \"spyglass_dirs\": {\n                    \"base\": self.base_dir,\n                    \"raw\": self.raw_dir,\n                    \"analysis\": self.analysis_dir,\n                    \"recording\": self.recording_dir,\n                    \"sorting\": self.sorting_dir,\n                    \"waveforms\": self.waveforms_dir,\n                    \"temp\": self.temp_dir,\n                    \"video\": self.video_dir,\n                },\n                \"kachery_dirs\": {\n                    \"cloud\": self.config.get(\n                        self.dir_to_var(\"cloud\", \"kachery\")\n                    ),\n                    \"storage\": self.config.get(\n                        self.dir_to_var(\"storage\", \"kachery\")\n                    ),\n                    \"temp\": self.config.get(self.dir_to_var(\"temp\", \"kachery\")),\n                },\n                \"dlc_dirs\": {\n                    \"base\": self._dlc_base,\n                    \"project\": self.dlc_project_dir,\n                    \"video\": self.dlc_video_dir,\n                    \"output\": self.dlc_output_dir,\n                },\n                \"kachery_zone\": \"franklab.default\",\n            }\n        }\n\n    @property\n    def config(self) -&gt; dict:\n        self.load_config()\n        return self._config\n\n    @property\n    def base_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"base\"))\n\n    @property\n    def raw_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"raw\"))\n\n    @property\n    def analysis_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"analysis\"))\n\n    @property\n    def recording_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"recording\"))\n\n    @property\n    def sorting_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"sorting\"))\n\n    @property\n    def waveforms_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"waveforms\"))\n\n    @property\n    def temp_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"temp\"))\n\n    @property\n    def video_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"video\"))\n\n    @property\n    def debug_mode(self) -&gt; bool:\n        \"\"\"Returns True if debug_mode is set.\n\n        Supports skipping inserts for Dockerized development.\n        \"\"\"\n        return self._debug_mode\n\n    @property\n    def test_mode(self) -&gt; bool:\n        \"\"\"Returns True if test_mode is set.\n\n        Required for pytests to run without prompts.\"\"\"\n        return self._test_mode\n\n    @property\n    def dlc_project_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"project\", \"dlc\"))\n\n    @property\n    def dlc_video_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"video\", \"dlc\"))\n\n    @property\n    def dlc_output_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"output\", \"dlc\"))\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.__init__", "title": "<code>__init__(base_dir=None, **kwargs)</code>", "text": "<p>Initializes a new instance of the class.</p> <p>Parameters:</p> Name Type Description Default <code>base_dir</code> <code>str</code> <p>The base directory.</p> <code>None</code> <p>Attributes:</p> Name Type Description <code>supplied_base_dir (str)</code> <p>The base directory passed to the class.</p> <code>config_defaults (dict)</code> <p>Default settings for the config.</p> <code>relative_dirs (dict)</code> <p>Relative dirs for each prefix (spyglass, kachery, dlc). Relative to respective base_dir. Created on init.</p> <code>dj_defaults (dict)</code> <p>Default settings for datajoint.</p> <code>env_defaults (dict)</code> <p>Default settings for environment variables.</p> <code>_config (dict)</code> <p>Cached config settings.</p> <code>_debug_mode (bool)</code> <p>True if debug_mode is set. Supports skipping known bugs in test env.</p> <code>_test_mode (bool)</code> <p>True if test_mode is set. Required for pytests to run without prompts.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def __init__(self, base_dir: str = None, **kwargs):\n    \"\"\"\n    Initializes a new instance of the class.\n\n    Parameters\n    ----------\n    base_dir (str)\n        The base directory.\n\n    Attributes\n    ----------\n    supplied_base_dir (str)\n        The base directory passed to the class.\n    config_defaults (dict)\n        Default settings for the config.\n    relative_dirs (dict)\n        Relative dirs for each prefix (spyglass, kachery, dlc). Relative\n        to respective base_dir. Created on init.\n    dj_defaults (dict)\n        Default settings for datajoint.\n    env_defaults (dict)\n        Default settings for environment variables.\n    _config (dict)\n        Cached config settings.\n    _debug_mode (bool)\n        True if debug_mode is set. Supports skipping known bugs in test env.\n    _test_mode (bool)\n        True if test_mode is set. Required for pytests to run without\n        prompts.\n    \"\"\"\n    self.supplied_base_dir = base_dir\n    self._config = dict()\n    self.config_defaults = dict(prepopulate=True)\n    self._debug_mode = kwargs.get(\"debug_mode\", False)\n    self._test_mode = kwargs.get(\"test_mode\", False)\n    self._dlc_base = None\n    self.load_failed = False\n\n    self.relative_dirs = {\n        # {PREFIX}_{KEY}_DIR, default dir relative to base_dir\n        \"spyglass\": {\n            \"raw\": \"raw\",\n            \"analysis\": \"analysis\",\n            \"recording\": \"recording\",\n            \"sorting\": \"spikesorting\",\n            \"waveforms\": \"waveforms\",\n            \"temp\": \"tmp\",\n            \"video\": \"video\",\n        },\n        \"kachery\": {\n            \"cloud\": \".kachery-cloud\",\n            \"storage\": \"kachery_storage\",\n            \"temp\": \"tmp\",\n        },\n        \"dlc\": {\n            \"project\": \"projects\",\n            \"video\": \"video\",\n            \"output\": \"output\",\n        },\n    }\n    self.dj_defaults = {\n        \"database.host\": kwargs.get(\"database_host\", \"lmf-db.cin.ucsf.edu\"),\n        \"database.user\": kwargs.get(\"database_user\"),\n        \"database.port\": kwargs.get(\"database_port\", 3306),\n        \"database.use_tls\": kwargs.get(\"database_use_tls\", True),\n        \"filepath_checksum_size_limit\": 1 * 1024**3,\n        \"enable_python_native_blobs\": True,\n    }\n    self.env_defaults = {\n        \"FIGURL_CHANNEL\": \"franklab2\",\n        \"DJ_SUPPORT_FILEPATH_MANAGEMENT\": \"TRUE\",\n        \"KACHERY_CLOUD_EPHEMERAL\": \"TRUE\",\n        \"HD5_USE_FILE_LOCKING\": \"FALSE\",\n    }\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.load_config", "title": "<code>load_config(base_dir=None, force_reload=False, on_startup=False, **kwargs)</code>", "text": "<p>Loads the configuration settings for the object.</p> <p>Order of precedence, where X is base, raw, analysis, etc.:     1. SpyglassConfig(base_dir=\"string\") for base dir only     2. dj.config['custom']['{spyglass/kachery}dirs']['X']     3. os.environ['{SPYGLASS/KACHERY}_DIR']     4. resolved_base_dir/X for non-base dirs</p> <p>Parameters:</p> Name Type Description Default <code>base_dir</code> <p>Optional. Default None. The base directory. If not provided, will use the env variable or existing config.</p> <code>None</code> <code>force_reload</code> <p>Optional. Default False. Default skip load if already completed.</p> <code>False</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>If base_dir is not set in either dj.config or os.environ.</p> <p>Returns:</p> Type Description <code>dict</code> <p>list of relative_dirs and other settings (e.g., prepopulate).</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def load_config(\n    self,\n    base_dir=None,\n    force_reload=False,\n    on_startup: bool = False,\n    **kwargs,\n):\n    \"\"\"\n    Loads the configuration settings for the object.\n\n    Order of precedence, where X is base, raw, analysis, etc.:\n        1. SpyglassConfig(base_dir=\"string\") for base dir only\n        2. dj.config['custom']['{spyglass/kachery}_dirs']['X']\n        3. os.environ['{SPYGLASS/KACHERY}_{X}_DIR']\n        4. resolved_base_dir/X for non-base dirs\n\n    Parameters\n    ----------\n    base_dir: str\n        Optional. Default None. The base directory. If not provided, will\n        use the env variable or existing config.\n    force_reload: bool\n        Optional. Default False. Default skip load if already completed.\n\n    Raises\n    ------\n    ValueError\n        If base_dir is not set in either dj.config or os.environ.\n\n    Returns\n    -------\n    dict\n        list of relative_dirs and other settings (e.g., prepopulate).\n    \"\"\"\n    if not force_reload and self._config:\n        return self._config\n\n    dj_custom = dj.config.get(\"custom\", {})\n    dj_spyglass = dj_custom.get(\"spyglass_dirs\", {})\n    dj_kachery = dj_custom.get(\"kachery_dirs\", {})\n    dj_dlc = dj_custom.get(\"dlc_dirs\", {})\n\n    self._debug_mode = dj_custom.get(\"debug_mode\", False)\n    self._test_mode = kwargs.get(\"test_mode\") or dj_custom.get(\n        \"test_mode\", False\n    )\n\n    resolved_base = (\n        base_dir\n        or self.supplied_base_dir\n        or dj_spyglass.get(\"base\")\n        or os.environ.get(\"SPYGLASS_BASE_DIR\")\n    )\n\n    if resolved_base and not Path(resolved_base).exists():\n        resolved_base = Path(resolved_base).expanduser()\n    if not resolved_base or not Path(resolved_base).exists():\n        if not on_startup:  # Only warn if not on startup\n            logger.error(\n                f\"Could not find SPYGLASS_BASE_DIR: {resolved_base}\"\n                + \"\\n\\tCheck dj.config['custom']['spyglass_dirs']['base']\"\n                + \"\\n\\tand os.environ['SPYGLASS_BASE_DIR']\"\n            )\n        self.load_failed = True\n        return\n\n    self._dlc_base = (\n        dj_dlc.get(\"base\")\n        or os.environ.get(\"DLC_BASE_DIR\")\n        or os.environ.get(\"DLC_PROJECT_PATH\", \"\").split(\"projects\")[0]\n        or str(Path(resolved_base) / \"deeplabcut\")\n    )\n    Path(self._dlc_base).mkdir(exist_ok=True)\n\n    config_dirs = {\"SPYGLASS_BASE_DIR\": str(resolved_base)}\n    for prefix, dirs in self.relative_dirs.items():\n        this_base = self._dlc_base if prefix == \"dlc\" else resolved_base\n        for dir, dir_str in dirs.items():\n            dir_env_fmt = self.dir_to_var(dir=dir, dir_type=prefix)\n\n            env_loc = (  # Ignore env vars if base was passed to func\n                os.environ.get(dir_env_fmt)\n                if not self.supplied_base_dir\n                else None\n            )\n\n            source_config = (\n                dj_dlc\n                if prefix == \"dlc\"\n                else dj_kachery if prefix == \"kachery\" else dj_spyglass\n            )\n            dir_location = (\n                source_config.get(dir)\n                or env_loc\n                or str(Path(this_base) / dir_str)\n            ).replace('\"', \"\")\n\n            config_dirs.update({dir_env_fmt: str(dir_location)})\n\n    kachery_zone_dict = {\n        \"KACHERY_ZONE\": (\n            os.environ.get(\"KACHERY_ZONE\")\n            or dj.config.get(\"custom\", {}).get(\"kachery_zone\")\n            or \"franklab.default\"\n        )\n    }\n\n    loaded_env = self._load_env_vars()\n    self._set_env_with_dict(\n        {**config_dirs, **kachery_zone_dict, **loaded_env}\n    )\n    self._mkdirs_from_dict_vals(config_dirs)\n\n    self._config = dict(\n        debug_mode=self._debug_mode,\n        test_mode=self._test_mode,\n        **self.config_defaults,\n        **config_dirs,\n        **kachery_zone_dict,\n        **loaded_env,\n    )\n\n    self._set_dj_config_stores()\n\n    return self._config\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.dir_to_var", "title": "<code>dir_to_var(dir, dir_type='spyglass')</code>", "text": "<p>Converts a dir string to an env variable name.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def dir_to_var(self, dir: str, dir_type: str = \"spyglass\"):\n    \"\"\"Converts a dir string to an env variable name.\"\"\"\n    return f\"{dir_type.upper()}_{dir.upper()}_DIR\"\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.save_dj_config", "title": "<code>save_dj_config(save_method='global', output_filename=None, base_dir=None, set_password=True, **kwargs)</code>", "text": "<p>Set the dj.config parameters, set password, and save config to file.</p> <p>Parameters:</p> Name Type Description Default <code>save_method</code> <code>(local, 'global', custom)</code> <p>The method to use to save the config. If either 'local' or 'global', datajoint builtins will be used to save.</p> <code>'local'</code> <code>output_filename</code> <code>str or Path</code> <p>Default to datajoint global config. If save_method = 'custom', name of file to generate. Must end in either be either yaml or json.</p> <code>None</code> <code>base_dir</code> <code>str</code> <p>The base directory. If not provided, will default to the env var</p> <code>None</code> <code>set_password</code> <code>bool</code> <p>Default True. Set the database password.</p> <code>True</code> <code>kwargs</code> <p>Any other valid datajoint configuration parameters, including database_user, database_password, database_host, database_port, etc. Note: python will raise error for params with <code>.</code> in name, so use underscores instead.</p> <code>{}</code> Source code in <code>src/spyglass/settings.py</code> <pre><code>def save_dj_config(\n    self,\n    save_method: str = \"global\",\n    output_filename: str = None,\n    base_dir=None,\n    set_password=True,\n    **kwargs,\n):\n    \"\"\"Set the dj.config parameters, set password, and save config to file.\n\n    Parameters\n    ----------\n    save_method : {'local', 'global', 'custom'}, optional\n        The method to use to save the config. If either 'local' or 'global',\n        datajoint builtins will be used to save.\n    output_filename : str or Path, optional\n        Default to datajoint global config. If save_method = 'custom', name\n        of file to generate. Must end in either be either yaml or json.\n    base_dir : str, optional\n        The base directory. If not provided, will default to the env var\n    set_password : bool, optional\n        Default True. Set the database password.\n    kwargs: dict, optional\n        Any other valid datajoint configuration parameters, including\n        database_user, database_password, database_host, database_port, etc.\n        Note: python will raise error for params with `.` in name, so use\n        underscores instead.\n    \"\"\"\n    if base_dir:\n        self.load_config(\n            base_dir=base_dir, force_reload=True, on_startup=False\n        )\n\n    if output_filename:\n        save_method = \"custom\"\n        path = Path(output_filename).expanduser()  # Expand ~\n        filepath = path if path.is_absolute() else path.absolute()\n        filepath.parent.mkdir(exist_ok=True, parents=True)\n        filepath = (\n            filepath.with_suffix(\".json\")  # ensure suffix, default json\n            if filepath.suffix not in [\".json\", \".yaml\"]\n            else filepath\n        )\n    elif save_method == \"local\":\n        filepath = Path(\".\") / dj.settings.LOCALCONFIG\n    elif save_method == \"global\":\n        filepath = Path(\"~\").expanduser() / dj.settings.GLOBALCONFIG\n    else:\n        raise ValueError(\n            \"For save_dj_config, either (a) save_method must be 'local' \"\n            + \" or 'global' or (b) must provide custom output_filename.\"\n        )\n\n    dj.config.update(self._generate_dj_config(**kwargs))\n\n    if set_password:\n        try:\n            dj.set_password()\n        except OperationalError as e:\n            warnings.warn(f\"Database connection issues. Wrong pass?\\n\\t{e}\")\n\n    user_warn = (\n        f\"Replace existing file? {filepath.resolve()}\\n\\t\"\n        + \"\\n\\t\".join([f\"{k}: {v}\" for k, v in config.items()])\n        + \"\\n\"\n    )\n\n    if (\n        not self.test_mode\n        and filepath.exists()\n        and dj.utils.user_choice(user_warn)[0] != \"y\"\n    ):\n        return\n\n    if save_method == \"global\":\n        dj.config.save_global(verbose=True)\n        return\n\n    if save_method == \"local\":\n        dj.config.save_local(verbose=True)\n        return\n\n    with open(filepath, \"w\") as outfile:\n        if filepath.suffix == \".yaml\":\n            yaml.dump(dj.config._conf, outfile, default_flow_style=False)\n        else:\n            json.dump(dj.config._conf, outfile, indent=2)\n        logger.info(f\"Saved config to {filepath}\")\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.debug_mode", "title": "<code>debug_mode: bool</code>  <code>property</code>", "text": "<p>Returns True if debug_mode is set.</p> <p>Supports skipping inserts for Dockerized development.</p>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.test_mode", "title": "<code>test_mode: bool</code>  <code>property</code>", "text": "<p>Returns True if test_mode is set.</p> <p>Required for pytests to run without prompts.</p>"}, {"location": "api/cli/cli/", "title": "cli.py", "text": ""}, {"location": "api/common/common_behav/", "title": "common_behav.py", "text": ""}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource", "title": "<code>PositionSource</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass PositionSource(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Session\n    -&gt; IntervalList\n    ---\n    source: varchar(200)             # source of data (e.g., trodes, dlc)\n    import_file_name: varchar(2000)  # path to import file if importing\n    \"\"\"\n\n    class SpatialSeries(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        id = 0 : int unsigned            # index of spatial series\n        ---\n        name=null: varchar(32)       # name of spatial series\n        \"\"\"\n\n    def populate(self, keys=None):\n        \"\"\"Insert position source data from NWB file.\n\n        WARNING: populate method on Manual table is not protected by transaction\n                protections like other DataJoint tables.\n        \"\"\"\n        if not isinstance(keys, list):\n            keys = [keys]\n        if isinstance(keys[0], (dj.Table, dj.expression.QueryExpression)):\n            keys = [k for tbl in keys for k in tbl.fetch(\"KEY\", as_dict=True)]\n        for key in keys:\n            nwb_file_name = key.get(\"nwb_file_name\")\n            if not nwb_file_name:\n                raise ValueError(\n                    \"PositionSource.populate is an alias for a non-computed table \"\n                    + \"and must be passed a key with nwb_file_name\"\n                )\n            self.insert_from_nwbfile(nwb_file_name, skip_duplicates=True)\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwb_file_name, skip_duplicates=False) -&gt; None:\n        \"\"\"Add intervals to ItervalList and PositionSource.\n\n        Given an NWB file name, get the spatial series and interval lists from\n        the file, add the interval lists to the IntervalList table, and\n        populate the RawPosition table if possible.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        \"\"\"\n        nwbf = get_nwb_file(nwb_file_name)\n        all_pos = get_all_spatial_series(nwbf, verbose=True)\n        sess_key = Nwbfile.get_file_key(nwb_file_name)\n        src_key = dict(**sess_key, source=\"trodes\", import_file_name=\"\")\n\n        if all_pos is None:\n            return\n\n        sources = []\n        intervals = []\n        spat_series = []\n\n        for epoch, epoch_list in all_pos.items():\n            ind_key = dict(interval_list_name=cls.get_pos_interval_name(epoch))\n\n            sources.append(dict(**src_key, **ind_key))\n            intervals.append(\n                dict(\n                    **sess_key,\n                    **ind_key,\n                    valid_times=epoch_list[0][\"valid_times\"],\n                    pipeline=\"position\",\n                )\n            )\n\n            for index, pdict in enumerate(epoch_list):\n                spat_series.append(\n                    dict(\n                        **sess_key,\n                        **ind_key,\n                        id=index,\n                        name=pdict.get(\"name\"),\n                    )\n                )\n\n        with cls.connection.transaction:\n            IntervalList.insert(intervals, skip_duplicates=skip_duplicates)\n            cls.insert(sources, skip_duplicates=skip_duplicates)\n            cls.SpatialSeries.insert(\n                spat_series, skip_duplicates=skip_duplicates\n            )\n\n        # make map from epoch intervals to position intervals\n        populate_position_interval_map_session(nwb_file_name)\n\n    @staticmethod\n    def get_pos_interval_name(epoch_num: int) -&gt; str:\n        \"\"\"Return string of the interval name from the epoch number.\n\n        Parameters\n        ----------\n        epoch_num : int\n            Input epoch number\n\n        Returns\n        -------\n        str\n            Position interval name (e.g., pos 2 valid times)\n        \"\"\"\n        try:\n            int(epoch_num)\n        except ValueError:\n            raise ValueError(\n                f\"Epoch number must must be an integer. Received: {epoch_num}\"\n            )\n        return f\"pos {epoch_num} valid times\"\n\n    @staticmethod\n    def _is_valid_name(name) -&gt; bool:\n        return name.startswith(\"pos \") and name.endswith(\" valid times\")\n\n    @staticmethod\n    def get_epoch_num(name: str) -&gt; int:\n        \"\"\"Return the epoch number from the interval name.\n\n        Parameters\n        ----------\n        name : str\n            Name of position interval (e.g., pos epoch 1 index 2 valid times)\n\n        Returns\n        -------\n        int\n            epoch number\n        \"\"\"\n        if not PositionSource._is_valid_name(name):\n            raise ValueError(f\"Invalid interval name: {name}\")\n        return int(name.replace(\"pos \", \"\").replace(\" valid times\", \"\"))\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.populate", "title": "<code>populate(keys=None)</code>", "text": "<p>Insert position source data from NWB file.</p> <p>WARNING: populate method on Manual table is not protected by transaction         protections like other DataJoint tables.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def populate(self, keys=None):\n    \"\"\"Insert position source data from NWB file.\n\n    WARNING: populate method on Manual table is not protected by transaction\n            protections like other DataJoint tables.\n    \"\"\"\n    if not isinstance(keys, list):\n        keys = [keys]\n    if isinstance(keys[0], (dj.Table, dj.expression.QueryExpression)):\n        keys = [k for tbl in keys for k in tbl.fetch(\"KEY\", as_dict=True)]\n    for key in keys:\n        nwb_file_name = key.get(\"nwb_file_name\")\n        if not nwb_file_name:\n            raise ValueError(\n                \"PositionSource.populate is an alias for a non-computed table \"\n                + \"and must be passed a key with nwb_file_name\"\n            )\n        self.insert_from_nwbfile(nwb_file_name, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwb_file_name, skip_duplicates=False)</code>  <code>classmethod</code>", "text": "<p>Add intervals to ItervalList and PositionSource.</p> <p>Given an NWB file name, get the spatial series and interval lists from the file, add the interval lists to the IntervalList table, and populate the RawPosition table if possible.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwb_file_name, skip_duplicates=False) -&gt; None:\n    \"\"\"Add intervals to ItervalList and PositionSource.\n\n    Given an NWB file name, get the spatial series and interval lists from\n    the file, add the interval lists to the IntervalList table, and\n    populate the RawPosition table if possible.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    \"\"\"\n    nwbf = get_nwb_file(nwb_file_name)\n    all_pos = get_all_spatial_series(nwbf, verbose=True)\n    sess_key = Nwbfile.get_file_key(nwb_file_name)\n    src_key = dict(**sess_key, source=\"trodes\", import_file_name=\"\")\n\n    if all_pos is None:\n        return\n\n    sources = []\n    intervals = []\n    spat_series = []\n\n    for epoch, epoch_list in all_pos.items():\n        ind_key = dict(interval_list_name=cls.get_pos_interval_name(epoch))\n\n        sources.append(dict(**src_key, **ind_key))\n        intervals.append(\n            dict(\n                **sess_key,\n                **ind_key,\n                valid_times=epoch_list[0][\"valid_times\"],\n                pipeline=\"position\",\n            )\n        )\n\n        for index, pdict in enumerate(epoch_list):\n            spat_series.append(\n                dict(\n                    **sess_key,\n                    **ind_key,\n                    id=index,\n                    name=pdict.get(\"name\"),\n                )\n            )\n\n    with cls.connection.transaction:\n        IntervalList.insert(intervals, skip_duplicates=skip_duplicates)\n        cls.insert(sources, skip_duplicates=skip_duplicates)\n        cls.SpatialSeries.insert(\n            spat_series, skip_duplicates=skip_duplicates\n        )\n\n    # make map from epoch intervals to position intervals\n    populate_position_interval_map_session(nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.get_pos_interval_name", "title": "<code>get_pos_interval_name(epoch_num)</code>  <code>staticmethod</code>", "text": "<p>Return string of the interval name from the epoch number.</p> <p>Parameters:</p> Name Type Description Default <code>epoch_num</code> <code>int</code> <p>Input epoch number</p> required <p>Returns:</p> Type Description <code>str</code> <p>Position interval name (e.g., pos 2 valid times)</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@staticmethod\ndef get_pos_interval_name(epoch_num: int) -&gt; str:\n    \"\"\"Return string of the interval name from the epoch number.\n\n    Parameters\n    ----------\n    epoch_num : int\n        Input epoch number\n\n    Returns\n    -------\n    str\n        Position interval name (e.g., pos 2 valid times)\n    \"\"\"\n    try:\n        int(epoch_num)\n    except ValueError:\n        raise ValueError(\n            f\"Epoch number must must be an integer. Received: {epoch_num}\"\n        )\n    return f\"pos {epoch_num} valid times\"\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.get_epoch_num", "title": "<code>get_epoch_num(name)</code>  <code>staticmethod</code>", "text": "<p>Return the epoch number from the interval name.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>Name of position interval (e.g., pos epoch 1 index 2 valid times)</p> required <p>Returns:</p> Type Description <code>int</code> <p>epoch number</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@staticmethod\ndef get_epoch_num(name: str) -&gt; int:\n    \"\"\"Return the epoch number from the interval name.\n\n    Parameters\n    ----------\n    name : str\n        Name of position interval (e.g., pos epoch 1 index 2 valid times)\n\n    Returns\n    -------\n    int\n        epoch number\n    \"\"\"\n    if not PositionSource._is_valid_name(name):\n        raise ValueError(f\"Invalid interval name: {name}\")\n    return int(name.replace(\"pos \", \"\").replace(\" valid times\", \"\"))\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition", "title": "<code>RawPosition</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition--notes", "title": "Notes", "text": "<p>The position timestamps come from: .pos_cameraHWSync.dat. If PTP is not used, the position timestamps are inferred by finding the closest timestamps from the neural recording via the trodes time.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass RawPosition(SpyglassMixin, dj.Imported):\n    \"\"\"\n\n    Notes\n    -----\n    The position timestamps come from: .pos_cameraHWSync.dat.\n    If PTP is not used, the position timestamps are inferred by finding the\n    closest timestamps from the neural recording via the trodes time.\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionSource\n    \"\"\"\n\n    class PosObject(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        -&gt; PositionSource.SpatialSeries.proj('id')\n        ---\n        raw_position_object_id: varchar(40) # id of spatial series in NWB file\n        \"\"\"\n\n        _nwb_table = Nwbfile\n\n        def fetch1_dataframe(self):\n            id_rp = [(n[\"id\"], n[\"raw_position\"]) for n in self.fetch_nwb()]\n\n            if len(set(rp.interval for _, rp in id_rp)) &gt; 1:\n                logger.warn(\"Loading DataFrame with multiple intervals.\")\n\n            df_list = [\n                pd.DataFrame(\n                    data=rp.data,\n                    index=pd.Index(rp.timestamps, name=\"time\"),\n                    columns=self._get_column_names(rp, pos_id),\n                )\n                for pos_id, rp in id_rp\n            ]\n\n            return reduce(lambda x, y: pd.merge(x, y, on=\"time\"), df_list)\n\n        @staticmethod\n        def _get_column_names(rp, pos_id):\n            INDEX_ADJUST = 1  # adjust 0-index to 1-index (e.g., xloc0 -&gt; xloc1)\n            n_pos_dims = rp.data.shape[1]\n            column_names = [\n                (\n                    col  # use existing columns if already numbered\n                    if \"1\" in rp.description or \"2\" in rp.description\n                    # else number them by id\n                    else col + str(pos_id + INDEX_ADJUST)\n                )\n                for col in rp.description.split(\", \")\n            ]\n            if len(column_names) != n_pos_dims:\n                # if the string split didn't work, use default names\n                column_names = [\"x\", \"y\", \"z\"][:n_pos_dims]\n            return column_names\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        interval_list_name = key[\"interval_list_name\"]\n\n        nwbf = get_nwb_file(nwb_file_name)\n        indices = (PositionSource.SpatialSeries &amp; key).fetch(\"id\")\n\n        # incl_times = False -&gt; don't do extra processing for valid_times\n        spat_objs = get_all_spatial_series(nwbf, incl_times=False)[\n            PositionSource.get_epoch_num(interval_list_name)\n        ]\n\n        self.insert1(key)\n        self.PosObject.insert(\n            [\n                dict(\n                    nwb_file_name=nwb_file_name,\n                    interval_list_name=interval_list_name,\n                    id=index,\n                    raw_position_object_id=obj[\"raw_position_object_id\"],\n                )\n                for index, obj in enumerate(spat_objs)\n                if index in indices\n            ]\n        )\n\n    def fetch_nwb(self, *attrs, **kwargs) -&gt; list:\n        \"\"\"\n        Returns a condatenated list of nwb objects from RawPosition.PosObject\n        \"\"\"\n        return (\n            self.PosObject()\n            .restrict(self.restriction)  # Avoids fetch_nwb on whole table\n            .fetch_nwb(*attrs, **kwargs)\n        )\n\n    def fetch1_dataframe(self):\n        \"\"\"Returns a dataframe with all RawPosition.PosObject items.\n\n        Uses interval_list_name as column index.\n        \"\"\"\n        ret = {}\n\n        pos_obj_set = self.PosObject &amp; self.restriction\n        unique_intervals = set(pos_obj_set.fetch(\"interval_list_name\"))\n\n        for interval in unique_intervals:\n            ret[interval] = (\n                pos_obj_set &amp; {\"interval_list_name\": interval}\n            ).fetch1_dataframe()\n\n        if len(unique_intervals) == 1:\n            return next(iter(ret.values()))\n\n        return pd.concat(ret, axis=1)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>Returns a condatenated list of nwb objects from RawPosition.PosObject</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs) -&gt; list:\n    \"\"\"\n    Returns a condatenated list of nwb objects from RawPosition.PosObject\n    \"\"\"\n    return (\n        self.PosObject()\n        .restrict(self.restriction)  # Avoids fetch_nwb on whole table\n        .fetch_nwb(*attrs, **kwargs)\n    )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Returns a dataframe with all RawPosition.PosObject items.</p> <p>Uses interval_list_name as column index.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Returns a dataframe with all RawPosition.PosObject items.\n\n    Uses interval_list_name as column index.\n    \"\"\"\n    ret = {}\n\n    pos_obj_set = self.PosObject &amp; self.restriction\n    unique_intervals = set(pos_obj_set.fetch(\"interval_list_name\"))\n\n    for interval in unique_intervals:\n        ret[interval] = (\n            pos_obj_set &amp; {\"interval_list_name\": interval}\n        ).fetch1_dataframe()\n\n    if len(unique_intervals) == 1:\n        return next(iter(ret.values()))\n\n    return pd.concat(ret, axis=1)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.StateScriptFile", "title": "<code>StateScriptFile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass StateScriptFile(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; TaskEpoch\n    ---\n    file_object_id: varchar(40)  # the object id of the file object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        \"\"\"Add a new row to the StateScriptFile table.\"\"\"\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n        associated_files = nwbf.processing.get(\n            \"associated_files\"\n        ) or nwbf.processing.get(\"associated files\")\n        if associated_files is None:\n            logger.info(\n                \"Unable to import StateScriptFile: no processing module named \"\n                + '\"associated_files\" found in {nwb_file_name}.'\n            )\n            return  # See #849\n\n        for associated_file_obj in associated_files.data_interfaces.values():\n            if not isinstance(\n                associated_file_obj, ndx_franklab_novela.AssociatedFiles\n            ):\n                logger.info(\n                    f\"Data interface {associated_file_obj.name} within \"\n                    + '\"associated_files\" processing module is not '\n                    + \"of expected type ndx_franklab_novela.AssociatedFiles\\n\"\n                )\n                return\n\n            # parse the task_epochs string\n            # TODO: update associated_file_obj.task_epochs to be an array of\n            # 1-based ints, not a comma-separated string of ints\n\n            epoch_list = associated_file_obj.task_epochs.split(\",\")\n            # only insert if this is the statescript file\n            logger.info(associated_file_obj.description)\n            if (\n                \"statescript\".upper() in associated_file_obj.description.upper()\n                or \"state_script\".upper()\n                in associated_file_obj.description.upper()\n                or \"state script\".upper()\n                in associated_file_obj.description.upper()\n            ):\n                # find the file associated with this epoch\n                if str(key[\"epoch\"]) in epoch_list:\n                    key[\"file_object_id\"] = associated_file_obj.object_id\n                    self.insert1(key)\n            else:\n                logger.info(\"not a statescript file\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.StateScriptFile.make", "title": "<code>make(key)</code>", "text": "<p>Add a new row to the StateScriptFile table.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def make(self, key):\n    \"\"\"Add a new row to the StateScriptFile table.\"\"\"\n    nwb_file_name = key[\"nwb_file_name\"]\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    nwbf = get_nwb_file(nwb_file_abspath)\n\n    associated_files = nwbf.processing.get(\n        \"associated_files\"\n    ) or nwbf.processing.get(\"associated files\")\n    if associated_files is None:\n        logger.info(\n            \"Unable to import StateScriptFile: no processing module named \"\n            + '\"associated_files\" found in {nwb_file_name}.'\n        )\n        return  # See #849\n\n    for associated_file_obj in associated_files.data_interfaces.values():\n        if not isinstance(\n            associated_file_obj, ndx_franklab_novela.AssociatedFiles\n        ):\n            logger.info(\n                f\"Data interface {associated_file_obj.name} within \"\n                + '\"associated_files\" processing module is not '\n                + \"of expected type ndx_franklab_novela.AssociatedFiles\\n\"\n            )\n            return\n\n        # parse the task_epochs string\n        # TODO: update associated_file_obj.task_epochs to be an array of\n        # 1-based ints, not a comma-separated string of ints\n\n        epoch_list = associated_file_obj.task_epochs.split(\",\")\n        # only insert if this is the statescript file\n        logger.info(associated_file_obj.description)\n        if (\n            \"statescript\".upper() in associated_file_obj.description.upper()\n            or \"state_script\".upper()\n            in associated_file_obj.description.upper()\n            or \"state script\".upper()\n            in associated_file_obj.description.upper()\n        ):\n            # find the file associated with this epoch\n            if str(key[\"epoch\"]) in epoch_list:\n                key[\"file_object_id\"] = associated_file_obj.object_id\n                self.insert1(key)\n        else:\n            logger.info(\"not a statescript file\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile", "title": "<code>VideoFile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile--notes", "title": "Notes", "text": "<p>The video timestamps come from: videoTimeStamps.cameraHWSync if PTP is used. If PTP is not used, the video timestamps come from videoTimeStamps.cameraHWFrameCount .</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass VideoFile(SpyglassMixin, dj.Imported):\n    \"\"\"\n\n    Notes\n    -----\n    The video timestamps come from: videoTimeStamps.cameraHWSync if PTP is\n    used. If PTP is not used, the video timestamps come from\n    videoTimeStamps.cameraHWFrameCount .\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; TaskEpoch\n    video_file_num = 0: int\n    ---\n    camera_name: varchar(80)\n    video_file_object_id: varchar(40)  # the object id of the file object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        self._no_transaction_make(key)\n\n    def _no_transaction_make(self, key, verbose=True):\n        if not self.connection.in_transaction:\n            self.populate(key)\n            return\n\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        videos = get_data_interface(\n            nwbf, \"video\", pynwb.behavior.BehavioralEvents\n        )\n\n        if videos is None:\n            logger.warn(f\"No video data interface found in {nwb_file_name}\\n\")\n            return\n        else:\n            videos = videos.time_series\n\n        # get the interval for the current TaskEpoch\n        interval_list_name = (TaskEpoch() &amp; key).fetch1(\"interval_list_name\")\n        valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        cam_device_str = r\"camera_device (\\d+)\"\n        is_found = False\n        for ind, video in enumerate(videos.values()):\n            if isinstance(video, pynwb.image.ImageSeries):\n                video = [video]\n            for video_obj in video:\n                # check to see if the times for this video_object are largely\n                # overlapping with the task epoch times\n\n                if len(\n                    interval_list_contains(valid_times, video_obj.timestamps)\n                    &gt; 0.9 * len(video_obj.timestamps)\n                ):\n                    nwb_cam_device = video_obj.device.name\n                    # returns whatever was captured in the first group (within the parentheses) of the regular expression -- in this case, 0\n                    key[\"video_file_num\"] = int(\n                        re.match(cam_device_str, nwb_cam_device)[1]\n                    )\n                    camera_name = video_obj.device.camera_name\n                    if CameraDevice &amp; {\"camera_name\": camera_name}:\n                        key[\"camera_name\"] = video_obj.device.camera_name\n                    else:\n                        raise KeyError(\n                            f\"No camera with camera_name: {camera_name} found \"\n                            + \"in CameraDevice table.\"\n                        )\n                    key[\"video_file_object_id\"] = video_obj.object_id\n                    self.insert1(key)\n                    is_found = True\n\n        if not is_found and verbose:\n            logger.info(\n                f\"No video found corresponding to file {nwb_file_name}, \"\n                + f\"epoch {interval_list_name}\"\n            )\n\n    @classmethod\n    def update_entries(cls, restrict={}):\n        existing_entries = (cls &amp; restrict).fetch(\"KEY\")\n        for row in existing_entries:\n            if (cls &amp; row).fetch1(\"camera_name\"):\n                continue\n            video_nwb = (cls &amp; row).fetch_nwb()[0]\n            if len(video_nwb) != 1:\n                raise ValueError(\n                    f\"Expecting 1 video file per entry. {len(video_nwb)} found\"\n                )\n            row[\"camera_name\"] = video_nwb[0][\"video_file\"].device.camera_name\n            cls.update1(row=row)\n\n    @classmethod\n    def get_abs_path(cls, key: Dict):\n        \"\"\"Return the absolute path for a stored video file given a key.\n\n        Key must include the nwb_file_name and epoch number. The\n        SPYGLASS_VIDEO_DIR environment variable must be set.\n\n        Parameters\n        ----------\n        key : dict\n            dictionary with nwb_file_name and epoch as keys\n\n        Returns\n        -------\n        nwb_video_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        video_path_obj = pathlib.Path(video_dir)\n        video_info = (cls &amp; key).fetch1()\n        nwb_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n        nwbf = get_nwb_file(nwb_path)\n        nwb_video = nwbf.objects[video_info[\"video_file_object_id\"]]\n        video_filename = nwb_video.name\n        # see if the file exists and is stored in the base analysis dir\n        nwb_video_file_abspath = pathlib.Path(video_path_obj / video_filename)\n        if nwb_video_file_abspath.exists():\n            return nwb_video_file_abspath.as_posix()\n        else:\n            raise FileNotFoundError(\n                f\"video file with filename: {video_filename} \"\n                f\"does not exist in {video_path_obj}/\"\n            )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile.get_abs_path", "title": "<code>get_abs_path(key)</code>  <code>classmethod</code>", "text": "<p>Return the absolute path for a stored video file given a key.</p> <p>Key must include the nwb_file_name and epoch number. The SPYGLASS_VIDEO_DIR environment variable must be set.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary with nwb_file_name and epoch as keys</p> required <p>Returns:</p> Name Type Description <code>nwb_video_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@classmethod\ndef get_abs_path(cls, key: Dict):\n    \"\"\"Return the absolute path for a stored video file given a key.\n\n    Key must include the nwb_file_name and epoch number. The\n    SPYGLASS_VIDEO_DIR environment variable must be set.\n\n    Parameters\n    ----------\n    key : dict\n        dictionary with nwb_file_name and epoch as keys\n\n    Returns\n    -------\n    nwb_video_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    video_path_obj = pathlib.Path(video_dir)\n    video_info = (cls &amp; key).fetch1()\n    nwb_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n    nwbf = get_nwb_file(nwb_path)\n    nwb_video = nwbf.objects[video_info[\"video_file_object_id\"]]\n    video_filename = nwb_video.name\n    # see if the file exists and is stored in the base analysis dir\n    nwb_video_file_abspath = pathlib.Path(video_path_obj / video_filename)\n    if nwb_video_file_abspath.exists():\n        return nwb_video_file_abspath.as_posix()\n    else:\n        raise FileNotFoundError(\n            f\"video file with filename: {video_filename} \"\n            f\"does not exist in {video_path_obj}/\"\n        )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.convert_epoch_interval_name_to_position_interval_name", "title": "<code>convert_epoch_interval_name_to_position_interval_name(key, populate_missing=True)</code>", "text": "<p>Converts IntervalList key to the corresponding position interval name.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Lookup key</p> required <code>populate_missing</code> <code>bool</code> <p>Whether to populate PositionIntervalMap for the key if missing. Should be False if this function is used inside of another populate call. Defaults to True</p> <code>True</code> <p>Returns:</p> Name Type Description <code>position_interval_name</code> <code>str</code> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def convert_epoch_interval_name_to_position_interval_name(\n    key: dict, populate_missing: bool = True\n) -&gt; str:\n    \"\"\"Converts IntervalList key to the corresponding position interval name.\n\n    Parameters\n    ----------\n    key : dict\n        Lookup key\n    populate_missing: bool\n        Whether to populate PositionIntervalMap for the key if missing. Should\n        be False if this function is used inside of another populate call.\n        Defaults to True\n\n    Returns\n    -------\n    position_interval_name : str\n    \"\"\"\n    # get the interval list name if given epoch but not interval list name\n    if \"interval_list_name\" not in key and \"epoch\" in key:\n        key[\"interval_list_name\"] = get_interval_list_name_from_epoch(\n            key[\"nwb_file_name\"], key[\"epoch\"]\n        )\n\n    pos_query = PositionIntervalMap &amp; key\n\n    if len(pos_query) == 0:\n        if populate_missing:\n            PositionIntervalMap()._no_transaction_make(key)\n        else:\n            raise KeyError(\n                f\"{key} must be populated in the PositionIntervalMap table \"\n                + \"prior to your current populate call\"\n            )\n\n    if len(pos_query) == 0:\n        logger.info(f\"No position intervals found for {key}\")\n        return []\n\n    if len(pos_query) == 1:\n        return pos_query.fetch1(\"position_interval_name\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.get_interval_list_name_from_epoch", "title": "<code>get_interval_list_name_from_epoch(nwb_file_name, epoch)</code>", "text": "<p>Returns the interval list name for the given epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required <code>epoch</code> <code>int</code> <p>The epoch number.</p> required <p>Returns:</p> Name Type Description <code>interval_list_name</code> <code>str</code> <p>The interval list name.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def get_interval_list_name_from_epoch(nwb_file_name: str, epoch: int) -&gt; str:\n    \"\"\"Returns the interval list name for the given epoch.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    epoch : int\n        The epoch number.\n\n    Returns\n    -------\n    interval_list_name : str\n        The interval list name.\n    \"\"\"\n    interval_names = (\n        TaskEpoch &amp; {\"nwb_file_name\": nwb_file_name, \"epoch\": epoch}\n    ).fetch(\"interval_list_name\")\n\n    if len(interval_names) != 1:\n        logger.info(\n            f\"Found {len(interval_names)} interval list names found for \"\n            + f\"{nwb_file_name} epoch {epoch}\"\n        )\n        return None\n\n    return interval_names[0]\n</code></pre>"}, {"location": "api/common/common_device/", "title": "common_device.py", "text": ""}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice", "title": "<code>DataAcquisitionDevice</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass DataAcquisitionDevice(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    data_acquisition_device_name: varchar(80)\n    ---\n    -&gt; DataAcquisitionDeviceSystem\n    -&gt; DataAcquisitionDeviceAmplifier\n    adc_circuit = NULL: varchar(2000)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, config):\n        \"\"\"Insert data acquisition devices from an NWB file.\n\n        Note that this does not link the DataAcquisitionDevices with a Session.\n        For that, see DataAcquisitionDeviceList.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n        \"\"\"\n        _, ndx_devices, _ = cls.get_all_device_names(nwbf, config)\n\n        for device_name in ndx_devices:\n            new_device_dict = dict()\n\n            # read device properties into new_device_dict from PyNWB extension\n            # device object\n            nwb_device_obj = ndx_devices[device_name]\n\n            name = nwb_device_obj.name\n            adc_circuit = nwb_device_obj.adc_circuit\n\n            # transform system value. check if value is in DB. if not, prompt\n            # user to add an entry or cancel.\n            system = cls._add_system(nwb_device_obj.system)\n\n            # transform amplifier value. check if value is in DB. if not, prompt\n            # user to add an entry or cancel.\n            amplifier = cls._add_amplifier(nwb_device_obj.amplifier)\n\n            # standardize how Intan is represented in the database\n            if adc_circuit.title() == \"Intan\":\n                adc_circuit = \"Intan\"\n\n            new_device_dict[\"data_acquisition_device_name\"] = name\n            new_device_dict[\"data_acquisition_device_system\"] = system\n            new_device_dict[\"data_acquisition_device_amplifier\"] = amplifier\n            new_device_dict[\"adc_circuit\"] = adc_circuit\n\n            cls._add_device(new_device_dict)\n\n        if ndx_devices:\n            logger.info(\n                \"Inserted or referenced data acquisition device(s): \"\n                + f\"{ndx_devices.keys()}\"\n            )\n        else:\n            logger.warn(\"No conforming data acquisition device metadata found.\")\n\n    @classmethod\n    def get_all_device_names(cls, nwbf, config) -&gt; tuple:\n        \"\"\"\n        Return device names in the NWB file, after appending and overwriting by\n        the config file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : tuple\n            List of data acquisition object names found in the NWB file.\n        \"\"\"\n        # make a dict mapping device name to PyNWB device object for all devices\n        # in the NWB file that are of type ndx_franklab_novela.DataAcqDevice and\n        # thus have the required metadata\n        ndx_devices = {\n            device_obj.name: device_obj\n            for device_obj in nwbf.devices.values()\n            if isinstance(device_obj, ndx_franklab_novela.DataAcqDevice)\n        }\n\n        # make a list of device names that are associated with this NWB file\n        if \"DataAcquisitionDevice\" in config:\n            config_devices = [\n                device_dict[\"data_acquisition_device_name\"]\n                for device_dict in config[\"DataAcquisitionDevice\"]\n            ]\n        else:\n            config_devices = list()\n\n        all_device_names = set(ndx_devices.keys()).union(set(config_devices))\n\n        return all_device_names, ndx_devices, config_devices\n\n    @classmethod\n    def _add_device(cls, new_device_dict):\n        \"\"\"Ensure match between NWB file info &amp; database entry.\n\n        If no DataAcquisitionDevice with the given name exists in the database,\n        check whether the user wants to add a new entry instead of referencing\n        an existing entry. If so, return. If not, raise an exception.\n\n        Parameters\n        ----------\n        new_device_dict : dict\n            Dict of new device properties\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device to the database when prompted or\n            if the device properties from the NWB file do not match the\n            properties of the corresponding database entry.\n        \"\"\"\n        name = new_device_dict[\"data_acquisition_device_name\"]\n        all_values = DataAcquisitionDevice.fetch(\n            \"data_acquisition_device_name\"\n        ).tolist()\n        if prompt_insert(name=name, all_values=all_values):\n            cls.insert1(new_device_dict, skip_duplicates=True)\n            return\n\n        # Check if values provided match the values stored in the database\n        db_dict = (\n            DataAcquisitionDevice &amp; {\"data_acquisition_device_name\": name}\n        ).fetch1()\n        if db_dict != new_device_dict:\n            raise PopulateException(\n                \"Data acquisition device properties of PyNWB Device object \"\n                + f\"with name '{name}': {new_device_dict} do not match \"\n                f\"properties of the corresponding database entry: {db_dict}.\"\n            )\n\n    @classmethod\n    def _add_system(cls, system):\n        \"\"\"Check the system value. If not in the db, prompt user to add it.\n\n        This method also renames the system value \"MCU\" to \"SpikeGadgets\".\n\n        Parameters\n        ----------\n        system : str\n            The system value to check.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device system value to the database\n            when prompted.\n\n        Returns\n        -------\n        system : str\n            The system value that was added to the database.\n        \"\"\"\n        if system == \"MCU\":\n            system = \"SpikeGadgets\"\n\n        all_values = DataAcquisitionDeviceSystem.fetch(\n            \"data_acquisition_device_system\"\n        ).tolist()\n        if prompt_insert(\n            name=system, all_values=all_values, table_type=\"system\"\n        ):\n            key = {\"data_acquisition_device_system\": system}\n            DataAcquisitionDeviceSystem.insert1(key, skip_duplicates=True)\n        return system\n\n    @classmethod\n    def _add_amplifier(cls, amplifier):\n        \"\"\"Check amplifier value. If not in db, prompt user to add.\n\n        Parameters\n        ----------\n        amplifier : str\n            The amplifier value to check.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device amplifier value to the database\n            when prompted.\n\n        Returns\n        -------\n        amplifier : str\n            The amplifier value that was added to the database.\n        \"\"\"\n        # standardize how Intan is represented in the database\n        if amplifier.title() == \"Intan\":\n            amplifier = \"Intan\"\n\n        all_values = DataAcquisitionDeviceAmplifier.fetch(\n            \"data_acquisition_device_amplifier\"\n        ).tolist()\n        if prompt_insert(\n            name=amplifier, all_values=all_values, table_type=\"amplifier\"\n        ):\n            key = {\"data_acquisition_device_amplifier\": amplifier}\n            DataAcquisitionDeviceAmplifier.insert1(key, skip_duplicates=True)\n        return amplifier\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Insert data acquisition devices from an NWB file.</p> <p>Note that this does not link the DataAcquisitionDevices with a Session. For that, see DataAcquisitionDeviceList.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, config):\n    \"\"\"Insert data acquisition devices from an NWB file.\n\n    Note that this does not link the DataAcquisitionDevices with a Session.\n    For that, see DataAcquisitionDeviceList.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n    \"\"\"\n    _, ndx_devices, _ = cls.get_all_device_names(nwbf, config)\n\n    for device_name in ndx_devices:\n        new_device_dict = dict()\n\n        # read device properties into new_device_dict from PyNWB extension\n        # device object\n        nwb_device_obj = ndx_devices[device_name]\n\n        name = nwb_device_obj.name\n        adc_circuit = nwb_device_obj.adc_circuit\n\n        # transform system value. check if value is in DB. if not, prompt\n        # user to add an entry or cancel.\n        system = cls._add_system(nwb_device_obj.system)\n\n        # transform amplifier value. check if value is in DB. if not, prompt\n        # user to add an entry or cancel.\n        amplifier = cls._add_amplifier(nwb_device_obj.amplifier)\n\n        # standardize how Intan is represented in the database\n        if adc_circuit.title() == \"Intan\":\n            adc_circuit = \"Intan\"\n\n        new_device_dict[\"data_acquisition_device_name\"] = name\n        new_device_dict[\"data_acquisition_device_system\"] = system\n        new_device_dict[\"data_acquisition_device_amplifier\"] = amplifier\n        new_device_dict[\"adc_circuit\"] = adc_circuit\n\n        cls._add_device(new_device_dict)\n\n    if ndx_devices:\n        logger.info(\n            \"Inserted or referenced data acquisition device(s): \"\n            + f\"{ndx_devices.keys()}\"\n        )\n    else:\n        logger.warn(\"No conforming data acquisition device metadata found.\")\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice.get_all_device_names", "title": "<code>get_all_device_names(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Return device names in the NWB file, after appending and overwriting by the config file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>tuple</code> <p>List of data acquisition object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef get_all_device_names(cls, nwbf, config) -&gt; tuple:\n    \"\"\"\n    Return device names in the NWB file, after appending and overwriting by\n    the config file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : tuple\n        List of data acquisition object names found in the NWB file.\n    \"\"\"\n    # make a dict mapping device name to PyNWB device object for all devices\n    # in the NWB file that are of type ndx_franklab_novela.DataAcqDevice and\n    # thus have the required metadata\n    ndx_devices = {\n        device_obj.name: device_obj\n        for device_obj in nwbf.devices.values()\n        if isinstance(device_obj, ndx_franklab_novela.DataAcqDevice)\n    }\n\n    # make a list of device names that are associated with this NWB file\n    if \"DataAcquisitionDevice\" in config:\n        config_devices = [\n            device_dict[\"data_acquisition_device_name\"]\n            for device_dict in config[\"DataAcquisitionDevice\"]\n        ]\n    else:\n        config_devices = list()\n\n    all_device_names = set(ndx_devices.keys()).union(set(config_devices))\n\n    return all_device_names, ndx_devices, config_devices\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.CameraDevice", "title": "<code>CameraDevice</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass CameraDevice(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    camera_name: varchar(80)\n    ---\n    meters_per_pixel = 0: float  # height / width of pixel in meters\n    manufacturer = \"\": varchar(2000)\n    model = \"\": varchar(2000)\n    lens = \"\": varchar(2000)\n    camera_id = -1: int\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert camera devices from an NWB file\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n\n        Returns\n        -------\n        device_name_list : list\n            List of camera device object names found in the NWB file.\n        \"\"\"\n        device_name_list = list()\n        for device in nwbf.devices.values():\n            if isinstance(device, ndx_franklab_novela.CameraDevice):\n                device_dict = dict()\n                # TODO ideally the ID is not encoded in the name formatted in a\n                # particular way device.name must have the form \"[any string\n                # without a space, usually camera] [int]\"\n                device_dict = {\n                    \"camera_id\": int(device.name.split()[1]),\n                    \"camera_name\": device.camera_name,\n                    \"manufacturer\": device.manufacturer,\n                    \"model\": device.model,\n                    \"lens\": device.lens,\n                    \"meters_per_pixel\": device.meters_per_pixel,\n                }\n                cls.insert1(device_dict, skip_duplicates=True)\n                device_name_list.append(device_dict[\"camera_name\"])\n        if device_name_list:\n            logger.info(f\"Inserted camera devices {device_name_list}\")\n        else:\n            logger.warn(\"No conforming camera device metadata found.\")\n        return device_name_list\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.CameraDevice.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert camera devices from an NWB file</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of camera device object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert camera devices from an NWB file\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n\n    Returns\n    -------\n    device_name_list : list\n        List of camera device object names found in the NWB file.\n    \"\"\"\n    device_name_list = list()\n    for device in nwbf.devices.values():\n        if isinstance(device, ndx_franklab_novela.CameraDevice):\n            device_dict = dict()\n            # TODO ideally the ID is not encoded in the name formatted in a\n            # particular way device.name must have the form \"[any string\n            # without a space, usually camera] [int]\"\n            device_dict = {\n                \"camera_id\": int(device.name.split()[1]),\n                \"camera_name\": device.camera_name,\n                \"manufacturer\": device.manufacturer,\n                \"model\": device.model,\n                \"lens\": device.lens,\n                \"meters_per_pixel\": device.meters_per_pixel,\n            }\n            cls.insert1(device_dict, skip_duplicates=True)\n            device_name_list.append(device_dict[\"camera_name\"])\n    if device_name_list:\n        logger.info(f\"Inserted camera devices {device_name_list}\")\n    else:\n        logger.warn(\"No conforming camera device metadata found.\")\n    return device_name_list\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe", "title": "<code>Probe</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass Probe(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # A configuration of a ProbeType. For most probe types, there is only one,\n    # which should always be used. For Neuropixels, the channel map (which\n    # electrodes used, where they are, and in what order) can differ between\n    # users and sessions. Each config should have a different ProbeType.\n    probe_id: varchar(80)     # a unique ID for this probe &amp; dynamic config\n    ---\n    -&gt; ProbeType              # Type of probe, selected from a controlled list\n    -&gt; [nullable] DataAcquisitionDevice  # the data acquisition device used\n    contact_side_numbering: enum(\"True\", \"False\")  # Facing you when numbering\n    \"\"\"\n\n    class Shank(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; Probe\n        probe_shank: int              # unique shank number within probe.\n        \"\"\"\n\n    class Electrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Electrode configuration, with ID, contact size, X/Y/Z coordinates\n        -&gt; Probe.Shank\n        probe_electrode: int          # electrode ID, output from acquisition\n                                      # system. Unique within a Probe\n        ---\n        contact_size = NULL: float    # (um) contact size\n        rel_x = NULL: float           # (um) x coordinate of electrode\n        rel_y = NULL: float           # (um) y coordinate of electrode\n        rel_z = NULL: float           # (um) z coordinate of electrode\n        \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, config):\n        \"\"\"Insert probe devices from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : list\n            List of probe device types found in the NWB file.\n        \"\"\"\n        all_probes_types, ndx_probes, _ = cls.get_all_probe_names(nwbf, config)\n\n        for probe_type in all_probes_types:\n            new_probe_type_dict = dict()\n            new_probe_dict = dict()\n            shank_dict = dict()\n            elect_dict = dict()\n            num_shanks = 0\n\n            if probe_type in ndx_probes:\n                # read probe properties into new_probe_dict from PyNWB extension\n                # probe object\n                nwb_probe_obj = ndx_probes[probe_type]\n                cls.__read_ndx_probe_data(\n                    nwb_probe_obj,\n                    new_probe_type_dict,\n                    new_probe_dict,\n                    shank_dict,\n                    elect_dict,\n                )\n\n            # check that number of shanks is consistent\n            num_shanks = new_probe_type_dict[\"num_shanks\"]\n            assert num_shanks == 0 or num_shanks == len(\n                shank_dict\n            ), \"`num_shanks` is not equal to the number of shanks.\"\n\n            # if probe id already exists, do not overwrite anything or create\n            # new Shanks and Electrodes\n            # TODO: test whether the Shanks and Electrodes in the NWB file match\n            # the ones in the database\n            query = Probe &amp; {\"probe_id\": new_probe_dict[\"probe_id\"]}\n            if len(query) &gt; 0:\n                logger.info(\n                    f\"Probe ID '{new_probe_dict['probe_id']}' already exists in\"\n                    \" the database. Spyglass will use that and not create a new\"\n                    \" Probe, Shanks, or Electrodes.\"\n                )\n                continue\n\n            cls.insert1(new_probe_dict, skip_duplicates=True)\n\n            for shank in shank_dict.values():\n                cls.Shank.insert1(shank, skip_duplicates=True)\n            for electrode in elect_dict.values():\n                cls.Electrode.insert1(electrode, skip_duplicates=True)\n\n        if all_probes_types:\n            logger.info(f\"Inserted probes {all_probes_types}\")\n        else:\n            logger.warn(\"No conforming probe metadata found.\")\n\n        return all_probes_types\n\n    @classmethod\n    def get_all_probe_names(cls, nwbf, config):\n        \"\"\"Get a list of all device names in the NWB.\n\n        Includes all devices, after appending/overwriting by the config file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : list\n            List of data acquisition object names found in the NWB file.\n        \"\"\"\n\n        # make a dict mapping probe type to PyNWB object for all devices in the\n        # NWB file that are of type ndx_franklab_novela.Probe and thus have the\n        # required metadata\n        ndx_probes = {\n            device_obj.probe_type: device_obj\n            for device_obj in nwbf.devices.values()\n            if isinstance(device_obj, ndx_franklab_novela.Probe)\n        }\n\n        # make a dict mapping probe type to dict of device metadata from the\n        # config YAML if exists\n        config_probes = (\n            [probe_dict[\"probe_type\"] for probe_dict in config[\"Probe\"]]\n            if \"Probe\" in config\n            else list()\n        )\n\n        # get all the probe types from the NWB file plus the config YAML\n        all_probes_types = set(ndx_probes.keys()).union(set(config_probes))\n\n        return all_probes_types, ndx_probes, config_probes\n\n    @classmethod\n    def __read_ndx_probe_data(\n        cls,\n        nwb_probe_obj: ndx_franklab_novela.Probe,\n        new_probe_type_dict: dict,\n        new_probe_dict: dict,\n        shank_dict: dict,\n        elect_dict: dict,\n    ):\n        # construct dictionary of values to add to ProbeType\n        new_probe_type_dict.update(\n            {\n                \"manufacturer\": getattr(nwb_probe_obj, \"manufacturer\") or \"\",\n                \"probe_type\": nwb_probe_obj.probe_type,\n                \"probe_description\": nwb_probe_obj.probe_description,\n                \"num_shanks\": len(nwb_probe_obj.shanks),\n            }\n        )\n\n        cls._add_probe_type(new_probe_type_dict)\n\n        new_probe_dict.update(\n            {\n                \"probe_id\": nwb_probe_obj.probe_type,\n                \"probe_type\": nwb_probe_obj.probe_type,\n                \"contact_side_numbering\": (\n                    \"True\" if nwb_probe_obj.contact_side_numbering else \"False\"\n                ),\n            }\n        )\n        # go through the shanks and add each one to the Shank table\n        for shank in nwb_probe_obj.shanks.values():\n            shank_dict[shank.name] = {\n                \"probe_id\": new_probe_dict[\"probe_type\"],\n                \"probe_shank\": int(shank.name),\n            }\n\n            # go through the electrodes and add each one to the Electrode table\n            for electrode in shank.shanks_electrodes.values():\n                # the next line will need to be fixed if we have different sized\n                # contacts on a shank\n                elect_dict[electrode.name] = {\n                    \"probe_id\": new_probe_dict[\"probe_type\"],\n                    \"probe_shank\": shank_dict[shank.name][\"probe_shank\"],\n                    \"contact_size\": nwb_probe_obj.contact_size,\n                    \"probe_electrode\": int(electrode.name),\n                    \"rel_x\": electrode.rel_x,\n                    \"rel_y\": electrode.rel_y,\n                    \"rel_z\": electrode.rel_z,\n                }\n\n    @classmethod\n    def _add_probe_type(cls, new_probe_type_dict):\n        \"\"\"Check the probe type value against the values in the database.\n\n        Parameters\n        ----------\n        new_probe_type_dict : dict\n            Dictionary of probe type properties. See ProbeType for keys.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a probe type to the database when\n            prompted.\n\n        Returns\n        -------\n        probe_type : str\n            The probe type value that was added to the database.\n        \"\"\"\n        probe_type = new_probe_type_dict[\"probe_type\"]\n        all_values = ProbeType.fetch(\"probe_type\").tolist()\n        if prompt_insert(probe_type, all_values, table=\"probe type\"):\n            ProbeType.insert1(new_probe_type_dict, skip_duplicates=True)\n            return\n\n        # else / entry exists: check whether the values provided match the\n        # values stored in the database\n        db_dict = (ProbeType &amp; {\"probe_type\": probe_type}).fetch1()\n        if db_dict != new_probe_type_dict:\n            raise PopulateException(\n                \"\\nProbe type properties of PyNWB Probe object with name \"\n                f\"'{probe_type}': {new_probe_type_dict} do not match properties\"\n                f\" of the corresponding database entry: {db_dict}.\"\n            )\n        return probe_type\n\n    @classmethod\n    def create_from_nwbfile(\n        cls,\n        nwb_file_name: str,\n        nwb_device_name: str,\n        probe_id: str,\n        probe_type: str,\n        contact_side_numbering: bool,\n    ):\n        \"\"\"Create master/part Probe entry from the NWB file.\n\n        This method will parse the electrodes in the electrodes table, electrode\n        groups (as shanks), and devices (as probes) in the NWB file, but only\n        ones that are associated with the device that matches the given\n        `nwb_device_name`.\n\n        Note that this code assumes the relatively standard convention where the\n        NWB device corresponds to a Probe, the NWB electrode group corresponds\n        to a Shank, and the NWB electrode corresponds to an Electrode.\n\n        Example usage: ``` sgc.Probe.create_from_nwbfile(\n            nwbfile=nwb_file_name, nwb_device_name=\"Device\",\n            probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",\n            probe_type=\"Neuropixels 1.0\", contact_side_numbering=True\n        )\n        ```\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        nwb_device_name : str\n            The name of the PyNWB Device object that represents the probe to\n            read in the NWB file.\n        probe_id : str\n            A unique ID for the probe and its configuration, to be used as the\n            primary key for the new Probe entry.\n        probe_type : str\n            The existing ProbeType entry that represents the type of probe being\n            created. It must exist.\n        contact_side_numbering : bool\n            Whether the electrode contacts are facing you when numbering them.\n            Stored in the new Probe entry.\n        \"\"\"\n\n        from .common_nwbfile import Nwbfile\n\n        nwb_file_path = Nwbfile.get_abs_path(nwb_file_name)\n        nwbfile = get_nwb_file(nwb_file_path)\n\n        query = ProbeType &amp; {\"probe_type\": probe_type}\n        if len(query) == 0:\n            logger.warn(\n                f\"No ProbeType found with probe_type '{probe_type}'. Aborting.\"\n            )\n            return\n\n        new_probe_dict = {\n            \"probe_id\": probe_id,\n            \"probe_type\": probe_type,\n            \"contact_side_numbering\": (\n                \"True\" if contact_side_numbering else \"False\"\n            ),\n        }\n        shank_dict = {}\n        elect_dict = {}\n\n        # iterate through the electrodes table in the NWB file\n        # and use the group column (ElectrodeGroup) to create shanks\n        # and use the device attribute of each ElectrodeGroup to create a probe\n        created_shanks = {}  # map device name to shank_index (int)\n        device_found = False\n        for elec_index in range(len(nwbfile.electrodes)):\n            electrode_group = nwbfile.electrodes[elec_index, \"group\"]\n            eg_device_name = electrode_group.device.name\n\n            # only look at electrodes where the associated device is the one\n            # specified\n            if eg_device_name == nwb_device_name:\n                device_found = True\n\n                # if a Shank has not yet been created from the electrode group,\n                # then create it\n                if electrode_group.name not in created_shanks:\n                    shank_index = len(created_shanks)\n                    created_shanks[electrode_group.name] = shank_index\n\n                    # build the dictionary of Probe.Shank data\n                    shank_dict[shank_index] = {\n                        \"probe_id\": new_probe_dict[\"probe_id\"],\n                        \"probe_shank\": shank_index,\n                    }\n\n                # get the probe shank index associated with this Electrode\n                probe_shank = created_shanks[electrode_group.name]\n\n                # build the dictionary of Probe.Electrode data\n                elect_dict[elec_index] = {\n                    \"probe_id\": new_probe_dict[\"probe_id\"],\n                    \"probe_shank\": probe_shank,\n                    \"probe_electrode\": elec_index,\n                }\n                if \"rel_x\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_x\"] = nwbfile.electrodes[\n                        elec_index, \"rel_x\"\n                    ]\n                if \"rel_y\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_y\"] = nwbfile.electrodes[\n                        elec_index, \"rel_y\"\n                    ]\n                if \"rel_z\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_z\"] = nwbfile.electrodes[\n                        elec_index, \"rel_z\"\n                    ]\n\n        if not device_found:\n            logger.warn(\n                \"No electrodes in the NWB file were associated with a device \"\n                + f\"named '{nwb_device_name}'.\"\n            )\n            return\n\n        # insert the Probe, then the Shank parts, and then the Electrode parts\n        cls.insert1(new_probe_dict, skip_duplicates=True)\n\n        for shank in shank_dict.values():\n            cls.Shank.insert1(shank, skip_duplicates=True)\n        for electrode in elect_dict.values():\n            cls.Electrode.insert1(electrode, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Insert probe devices from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of probe device types found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, config):\n    \"\"\"Insert probe devices from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : list\n        List of probe device types found in the NWB file.\n    \"\"\"\n    all_probes_types, ndx_probes, _ = cls.get_all_probe_names(nwbf, config)\n\n    for probe_type in all_probes_types:\n        new_probe_type_dict = dict()\n        new_probe_dict = dict()\n        shank_dict = dict()\n        elect_dict = dict()\n        num_shanks = 0\n\n        if probe_type in ndx_probes:\n            # read probe properties into new_probe_dict from PyNWB extension\n            # probe object\n            nwb_probe_obj = ndx_probes[probe_type]\n            cls.__read_ndx_probe_data(\n                nwb_probe_obj,\n                new_probe_type_dict,\n                new_probe_dict,\n                shank_dict,\n                elect_dict,\n            )\n\n        # check that number of shanks is consistent\n        num_shanks = new_probe_type_dict[\"num_shanks\"]\n        assert num_shanks == 0 or num_shanks == len(\n            shank_dict\n        ), \"`num_shanks` is not equal to the number of shanks.\"\n\n        # if probe id already exists, do not overwrite anything or create\n        # new Shanks and Electrodes\n        # TODO: test whether the Shanks and Electrodes in the NWB file match\n        # the ones in the database\n        query = Probe &amp; {\"probe_id\": new_probe_dict[\"probe_id\"]}\n        if len(query) &gt; 0:\n            logger.info(\n                f\"Probe ID '{new_probe_dict['probe_id']}' already exists in\"\n                \" the database. Spyglass will use that and not create a new\"\n                \" Probe, Shanks, or Electrodes.\"\n            )\n            continue\n\n        cls.insert1(new_probe_dict, skip_duplicates=True)\n\n        for shank in shank_dict.values():\n            cls.Shank.insert1(shank, skip_duplicates=True)\n        for electrode in elect_dict.values():\n            cls.Electrode.insert1(electrode, skip_duplicates=True)\n\n    if all_probes_types:\n        logger.info(f\"Inserted probes {all_probes_types}\")\n    else:\n        logger.warn(\"No conforming probe metadata found.\")\n\n    return all_probes_types\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.get_all_probe_names", "title": "<code>get_all_probe_names(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Get a list of all device names in the NWB.</p> <p>Includes all devices, after appending/overwriting by the config file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of data acquisition object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef get_all_probe_names(cls, nwbf, config):\n    \"\"\"Get a list of all device names in the NWB.\n\n    Includes all devices, after appending/overwriting by the config file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : list\n        List of data acquisition object names found in the NWB file.\n    \"\"\"\n\n    # make a dict mapping probe type to PyNWB object for all devices in the\n    # NWB file that are of type ndx_franklab_novela.Probe and thus have the\n    # required metadata\n    ndx_probes = {\n        device_obj.probe_type: device_obj\n        for device_obj in nwbf.devices.values()\n        if isinstance(device_obj, ndx_franklab_novela.Probe)\n    }\n\n    # make a dict mapping probe type to dict of device metadata from the\n    # config YAML if exists\n    config_probes = (\n        [probe_dict[\"probe_type\"] for probe_dict in config[\"Probe\"]]\n        if \"Probe\" in config\n        else list()\n    )\n\n    # get all the probe types from the NWB file plus the config YAML\n    all_probes_types = set(ndx_probes.keys()).union(set(config_probes))\n\n    return all_probes_types, ndx_probes, config_probes\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.create_from_nwbfile", "title": "<code>create_from_nwbfile(nwb_file_name, nwb_device_name, probe_id, probe_type, contact_side_numbering)</code>  <code>classmethod</code>", "text": "<p>Create master/part Probe entry from the NWB file.</p> <p>This method will parse the electrodes in the electrodes table, electrode groups (as shanks), and devices (as probes) in the NWB file, but only ones that are associated with the device that matches the given <code>nwb_device_name</code>.</p> <p>Note that this code assumes the relatively standard convention where the NWB device corresponds to a Probe, the NWB electrode group corresponds to a Shank, and the NWB electrode corresponds to an Electrode.</p> <p>Example usage: <code>sgc.Probe.create_from_nwbfile(     nwbfile=nwb_file_name, nwb_device_name=\"Device\",     probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",     probe_type=\"Neuropixels 1.0\", contact_side_numbering=True )</code></p>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.create_from_nwbfile--parameters", "title": "Parameters", "text": "<p>nwb_file_name : str     The name of the NWB file. nwb_device_name : str     The name of the PyNWB Device object that represents the probe to     read in the NWB file. probe_id : str     A unique ID for the probe and its configuration, to be used as the     primary key for the new Probe entry. probe_type : str     The existing ProbeType entry that represents the type of probe being     created. It must exist. contact_side_numbering : bool     Whether the electrode contacts are facing you when numbering them.     Stored in the new Probe entry.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef create_from_nwbfile(\n    cls,\n    nwb_file_name: str,\n    nwb_device_name: str,\n    probe_id: str,\n    probe_type: str,\n    contact_side_numbering: bool,\n):\n    \"\"\"Create master/part Probe entry from the NWB file.\n\n    This method will parse the electrodes in the electrodes table, electrode\n    groups (as shanks), and devices (as probes) in the NWB file, but only\n    ones that are associated with the device that matches the given\n    `nwb_device_name`.\n\n    Note that this code assumes the relatively standard convention where the\n    NWB device corresponds to a Probe, the NWB electrode group corresponds\n    to a Shank, and the NWB electrode corresponds to an Electrode.\n\n    Example usage: ``` sgc.Probe.create_from_nwbfile(\n        nwbfile=nwb_file_name, nwb_device_name=\"Device\",\n        probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",\n        probe_type=\"Neuropixels 1.0\", contact_side_numbering=True\n    )\n    ```\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    nwb_device_name : str\n        The name of the PyNWB Device object that represents the probe to\n        read in the NWB file.\n    probe_id : str\n        A unique ID for the probe and its configuration, to be used as the\n        primary key for the new Probe entry.\n    probe_type : str\n        The existing ProbeType entry that represents the type of probe being\n        created. It must exist.\n    contact_side_numbering : bool\n        Whether the electrode contacts are facing you when numbering them.\n        Stored in the new Probe entry.\n    \"\"\"\n\n    from .common_nwbfile import Nwbfile\n\n    nwb_file_path = Nwbfile.get_abs_path(nwb_file_name)\n    nwbfile = get_nwb_file(nwb_file_path)\n\n    query = ProbeType &amp; {\"probe_type\": probe_type}\n    if len(query) == 0:\n        logger.warn(\n            f\"No ProbeType found with probe_type '{probe_type}'. Aborting.\"\n        )\n        return\n\n    new_probe_dict = {\n        \"probe_id\": probe_id,\n        \"probe_type\": probe_type,\n        \"contact_side_numbering\": (\n            \"True\" if contact_side_numbering else \"False\"\n        ),\n    }\n    shank_dict = {}\n    elect_dict = {}\n\n    # iterate through the electrodes table in the NWB file\n    # and use the group column (ElectrodeGroup) to create shanks\n    # and use the device attribute of each ElectrodeGroup to create a probe\n    created_shanks = {}  # map device name to shank_index (int)\n    device_found = False\n    for elec_index in range(len(nwbfile.electrodes)):\n        electrode_group = nwbfile.electrodes[elec_index, \"group\"]\n        eg_device_name = electrode_group.device.name\n\n        # only look at electrodes where the associated device is the one\n        # specified\n        if eg_device_name == nwb_device_name:\n            device_found = True\n\n            # if a Shank has not yet been created from the electrode group,\n            # then create it\n            if electrode_group.name not in created_shanks:\n                shank_index = len(created_shanks)\n                created_shanks[electrode_group.name] = shank_index\n\n                # build the dictionary of Probe.Shank data\n                shank_dict[shank_index] = {\n                    \"probe_id\": new_probe_dict[\"probe_id\"],\n                    \"probe_shank\": shank_index,\n                }\n\n            # get the probe shank index associated with this Electrode\n            probe_shank = created_shanks[electrode_group.name]\n\n            # build the dictionary of Probe.Electrode data\n            elect_dict[elec_index] = {\n                \"probe_id\": new_probe_dict[\"probe_id\"],\n                \"probe_shank\": probe_shank,\n                \"probe_electrode\": elec_index,\n            }\n            if \"rel_x\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_x\"] = nwbfile.electrodes[\n                    elec_index, \"rel_x\"\n                ]\n            if \"rel_y\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_y\"] = nwbfile.electrodes[\n                    elec_index, \"rel_y\"\n                ]\n            if \"rel_z\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_z\"] = nwbfile.electrodes[\n                    elec_index, \"rel_z\"\n                ]\n\n    if not device_found:\n        logger.warn(\n            \"No electrodes in the NWB file were associated with a device \"\n            + f\"named '{nwb_device_name}'.\"\n        )\n        return\n\n    # insert the Probe, then the Shank parts, and then the Electrode parts\n    cls.insert1(new_probe_dict, skip_duplicates=True)\n\n    for shank in shank_dict.values():\n        cls.Shank.insert1(shank, skip_duplicates=True)\n    for electrode in elect_dict.values():\n        cls.Electrode.insert1(electrode, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.prompt_insert", "title": "<code>prompt_insert(name, all_values, table='Data Acquisition Device', table_type=None)</code>", "text": "<p>Prompt user to add an item to the database. Return True if yes.</p> <p>Assume insert during test mode.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name of the item to add.</p> required <code>all_values</code> <code>list</code> <p>List of all values in the database.</p> required <code>table</code> <code>str</code> <p>The name of the table to add to, by default Data Acquisition Device</p> <code>'Data Acquisition Device'</code> <code>table_type</code> <code>str</code> <p>The type of item to add, by default None. Data Acquisition Device X</p> <code>None</code> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>def prompt_insert(\n    name: str,\n    all_values: list,\n    table: str = \"Data Acquisition Device\",\n    table_type: str = None,\n) -&gt; bool:\n    \"\"\"Prompt user to add an item to the database. Return True if yes.\n\n    Assume insert during test mode.\n\n    Parameters\n    ----------\n    name : str\n        The name of the item to add.\n    all_values : list\n        List of all values in the database.\n    table : str, optional\n        The name of the table to add to, by default Data Acquisition Device\n    table_type : str, optional\n        The type of item to add, by default None. Data Acquisition Device X\n    \"\"\"\n    if name in all_values:\n        return False\n\n    if test_mode:\n        return True\n\n    if table_type:\n        table_type += \" \"\n\n    logger.info(\n        f\"{table}{table_type} '{name}' was not found in the\"\n        f\"database. The current values are: {all_values}.\\n\"\n        \"Please ensure that the device you want to add does not already\"\n        \"exist in the database under a different name or spelling. If you\"\n        \"want to use an existing device in the database, please change the\"\n        \"corresponding Device object in the NWB file.\\nEntering 'N' will \"\n        \"raise an exception.\"\n    )\n    msg = f\"Do you want to add {table}{table_type} '{name}' to the database?\"\n    if dj.utils.user_choice(msg).lower() in [\"y\", \"yes\"]:\n        return True\n\n    raise PopulateException(\n        f\"User chose not to add {table}{table_type} '{name}' to the database.\"\n    )\n</code></pre>"}, {"location": "api/common/common_dio/", "title": "common_dio.py", "text": ""}, {"location": "api/common/common_dio/#src.spyglass.common.common_dio.DIOEvents", "title": "<code>DIOEvents</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_dio.py</code> <pre><code>@schema\nclass DIOEvents(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; Session\n    dio_event_name: varchar(80)   # the name assigned to this DIO event\n    ---\n    dio_object_id: varchar(40)    # the object id of the data in the NWB file\n    -&gt; IntervalList               # the list of intervals for this object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n        behav_events = get_data_interface(\n            nwbf, \"behavioral_events\", pynwb.behavior.BehavioralEvents\n        )\n        if behav_events is None:\n            logger.warn(\n                \"No conforming behavioral events data interface found in \"\n                + f\"{nwb_file_name}\\n\"\n            )\n            return  # See #849\n\n        # Times for these events correspond to the valid times for the raw data\n        key[\"interval_list_name\"] = (\n            Raw() &amp; {\"nwb_file_name\": nwb_file_name}\n        ).fetch1(\"interval_list_name\")\n        for event_series in behav_events.time_series.values():\n            key[\"dio_event_name\"] = event_series.name\n            key[\"dio_object_id\"] = event_series.object_id\n            self.insert1(key, skip_duplicates=True)\n\n    def plot_all_dio_events(self, return_fig=False):\n        \"\"\"Plot all DIO events in the session.\n\n        Examples\n        --------\n        &gt; restr1 = {'nwb_file_name': 'arthur20220314_.nwb'}\n        &gt; restr2 = {'nwb_file_name': 'arthur20220316_.nwb'}\n        &gt; (DIOEvents &amp; restr1).plot_all_dio_events()\n        &gt; (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()\n\n        \"\"\"\n        behavioral_events = self.fetch_nwb()\n        nwb_file_names = np.unique(\n            [event[\"nwb_file_name\"] for event in behavioral_events]\n        )\n        epoch_valid_times = (\n            pd.DataFrame(\n                IntervalList()\n                &amp; [\n                    {\"nwb_file_name\": nwb_file_name}\n                    for nwb_file_name in nwb_file_names\n                ]\n            )\n            .set_index(\"interval_list_name\")\n            .filter(regex=r\"^[0-9]\", axis=0)\n            .valid_times\n        )\n\n        n_events = len(behavioral_events)\n\n        _, axes = plt.subplots(\n            n_events,\n            1,\n            figsize=(15, n_events * 0.3),\n            dpi=100,\n            sharex=True,\n            constrained_layout=True,\n        )\n\n        for ind, (ax, event) in enumerate(zip(axes.flat, behavioral_events)):\n            for epoch_name, epoch in epoch_valid_times.items():\n                start_time, stop_time = epoch.squeeze()\n                ax.axvspan(start_time, stop_time, alpha=0.5)\n                if ind == 0:\n                    ax.text(\n                        start_time + (stop_time - start_time) / 2,\n                        1.001,\n                        epoch_name,\n                        ha=\"center\",\n                        va=\"bottom\",\n                    )\n            ax.step(\n                np.asarray(event[\"dio\"].timestamps),\n                np.asarray(event[\"dio\"].data),\n                where=\"post\",\n                color=\"black\",\n            )\n            ax.set_ylabel(\n                event[\"dio_event_name\"], rotation=0, ha=\"right\", va=\"center\"\n            )\n            ax.set_yticks([])\n        ax.set_xlabel(\"Time\")\n\n        if len(nwb_file_names) == 1:\n            plt.suptitle(f\"DIO events in {nwb_file_names[0]}\")\n        else:\n            plt.suptitle(f\"DIO events in {', '.join(nwb_file_names)}\")\n\n        if return_fig:\n            return plt.gcf()\n</code></pre>"}, {"location": "api/common/common_dio/#src.spyglass.common.common_dio.DIOEvents.plot_all_dio_events", "title": "<code>plot_all_dio_events(return_fig=False)</code>", "text": "<p>Plot all DIO events in the session.</p> <p>Examples:</p> <p>restr1 = {'nwb_file_name': 'arthur20220314_.nwb'} restr2 = {'nwb_file_name': 'arthur20220316_.nwb'} (DIOEvents &amp; restr1).plot_all_dio_events() (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()</p> Source code in <code>src/spyglass/common/common_dio.py</code> <pre><code>def plot_all_dio_events(self, return_fig=False):\n    \"\"\"Plot all DIO events in the session.\n\n    Examples\n    --------\n    &gt; restr1 = {'nwb_file_name': 'arthur20220314_.nwb'}\n    &gt; restr2 = {'nwb_file_name': 'arthur20220316_.nwb'}\n    &gt; (DIOEvents &amp; restr1).plot_all_dio_events()\n    &gt; (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()\n\n    \"\"\"\n    behavioral_events = self.fetch_nwb()\n    nwb_file_names = np.unique(\n        [event[\"nwb_file_name\"] for event in behavioral_events]\n    )\n    epoch_valid_times = (\n        pd.DataFrame(\n            IntervalList()\n            &amp; [\n                {\"nwb_file_name\": nwb_file_name}\n                for nwb_file_name in nwb_file_names\n            ]\n        )\n        .set_index(\"interval_list_name\")\n        .filter(regex=r\"^[0-9]\", axis=0)\n        .valid_times\n    )\n\n    n_events = len(behavioral_events)\n\n    _, axes = plt.subplots(\n        n_events,\n        1,\n        figsize=(15, n_events * 0.3),\n        dpi=100,\n        sharex=True,\n        constrained_layout=True,\n    )\n\n    for ind, (ax, event) in enumerate(zip(axes.flat, behavioral_events)):\n        for epoch_name, epoch in epoch_valid_times.items():\n            start_time, stop_time = epoch.squeeze()\n            ax.axvspan(start_time, stop_time, alpha=0.5)\n            if ind == 0:\n                ax.text(\n                    start_time + (stop_time - start_time) / 2,\n                    1.001,\n                    epoch_name,\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n        ax.step(\n            np.asarray(event[\"dio\"].timestamps),\n            np.asarray(event[\"dio\"].data),\n            where=\"post\",\n            color=\"black\",\n        )\n        ax.set_ylabel(\n            event[\"dio_event_name\"], rotation=0, ha=\"right\", va=\"center\"\n        )\n        ax.set_yticks([])\n    ax.set_xlabel(\"Time\")\n\n    if len(nwb_file_names) == 1:\n        plt.suptitle(f\"DIO events in {nwb_file_names[0]}\")\n    else:\n        plt.suptitle(f\"DIO events in {', '.join(nwb_file_names)}\")\n\n    if return_fig:\n        return plt.gcf()\n</code></pre>"}, {"location": "api/common/common_ephys/", "title": "common_ephys.py", "text": ""}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.Electrode", "title": "<code>Electrode</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass Electrode(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; ElectrodeGroup\n    electrode_id: int                      # the unique number for this electrode\n    ---\n    -&gt; [nullable] Probe.Electrode\n    -&gt; BrainRegion\n    name = \"\": varchar(200)                 # unique label for each contact\n    original_reference_electrode = -1: int  # the configured reference electrode for this electrode\n    x = NULL: float                         # the x coordinate of the electrode position in the brain\n    y = NULL: float                         # the y coordinate of the electrode position in the brain\n    z = NULL: float                         # the z coordinate of the electrode position in the brain\n    filtering: varchar(2000)                # description of the signal filtering\n    impedance = NULL: float                 # electrode impedance\n    bad_channel = \"False\": enum(\"True\", \"False\")  # if electrode is \"good\" or \"bad\" as observed during recording\n    x_warped = NULL: float                  # x coordinate of electrode position warped to common template brain\n    y_warped = NULL: float                  # y coordinate of electrode position warped to common template brain\n    z_warped = NULL: float                  # z coordinate of electrode position warped to common template brain\n    contacts: varchar(200)                  # label of electrode contacts used for a bipolar signal - current workaround\n    \"\"\"\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        config = get_config(nwb_file_abspath)\n\n        if \"Electrode\" in config:\n            electrode_config_dicts = {\n                electrode_dict[\"electrode_id\"]: electrode_dict\n                for electrode_dict in config[\"Electrode\"]\n            }\n        else:\n            electrode_config_dicts = dict()\n\n        electrodes = nwbf.electrodes.to_dataframe()\n        for elect_id, elect_data in electrodes.iterrows():\n            key[\"electrode_id\"] = elect_id\n            key[\"name\"] = str(elect_id)\n            key[\"electrode_group_name\"] = elect_data.group_name\n            key[\"region_id\"] = BrainRegion.fetch_add(\n                region_name=elect_data.group.location\n            )\n            key[\"x\"] = elect_data.x\n            key[\"y\"] = elect_data.y\n            key[\"z\"] = elect_data.z\n            key[\"x_warped\"] = 0\n            key[\"y_warped\"] = 0\n            key[\"z_warped\"] = 0\n            key[\"contacts\"] = \"\"\n            key[\"filtering\"] = elect_data.filtering\n            key[\"impedance\"] = elect_data.get(\"imp\")\n\n            # rough check of whether the electrodes table was created by\n            # rec_to_nwb and has the appropriate custom columns used by\n            # rec_to_nwb\n\n            # TODO this could be better resolved by making an extension for the\n            # electrodes table\n\n            if (\n                isinstance(elect_data.group.device, ndx_franklab_novela.Probe)\n                and \"probe_shank\" in elect_data\n                and \"probe_electrode\" in elect_data\n                and \"bad_channel\" in elect_data\n                and \"ref_elect_id\" in elect_data\n            ):\n                key[\"probe_id\"] = elect_data.group.device.probe_type\n                key[\"probe_shank\"] = elect_data.probe_shank\n                key[\"probe_electrode\"] = elect_data.probe_electrode\n                key[\"bad_channel\"] = (\n                    \"True\" if elect_data.bad_channel else \"False\"\n                )\n                key[\"original_reference_electrode\"] = elect_data.ref_elect_id\n\n            # override with information from the config YAML based on primary\n            # key (electrode id)\n\n            if elect_id in electrode_config_dicts:\n                # check whether the Probe.Electrode being referenced exists\n                query = Probe.Electrode &amp; electrode_config_dicts[elect_id]\n                if len(query) == 0:\n                    warnings.warn(\n                        \"No Probe.Electrode exists that matches the data: \"\n                        + f\"{electrode_config_dicts[elect_id]}. \"\n                        \"The config YAML for Electrode with electrode_id \"\n                        + f\"{elect_id} will be ignored.\"\n                    )\n                else:\n                    key.update(electrode_config_dicts[elect_id])\n\n            self.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def create_from_config(cls, nwb_file_name: str):\n        \"\"\"Create or update Electrode entries from what is specified in the config YAML file.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        \"\"\"\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        config = get_config(nwb_file_abspath)\n        if \"Electrode\" not in config:\n            return  # See #849\n\n        # map electrode id to dictof electrode information from config YAML\n        electrode_dicts = {\n            electrode_dict[\"electrode_id\"]: electrode_dict\n            for electrode_dict in config[\"Electrode\"]\n        }\n\n        electrodes = nwbf.electrodes.to_dataframe()\n        for nwbfile_elect_id, elect_data in electrodes.iterrows():\n            if nwbfile_elect_id in electrode_dicts:\n                # use the information in the electrodes table to start and then\n                # add (or overwrite) values from the config YAML\n\n                key = dict()\n                key[\"nwb_file_name\"] = nwb_file_name\n                key[\"name\"] = str(nwbfile_elect_id)\n                key[\"electrode_group_name\"] = elect_data.group_name\n                key[\"region_id\"] = BrainRegion.fetch_add(\n                    region_name=elect_data.group.location\n                )\n                key[\"x\"] = elect_data.x\n                key[\"y\"] = elect_data.y\n                key[\"z\"] = elect_data.z\n                key[\"x_warped\"] = 0\n                key[\"y_warped\"] = 0\n                key[\"z_warped\"] = 0\n                key[\"contacts\"] = \"\"\n                key[\"filtering\"] = elect_data.filtering\n                key[\"impedance\"] = elect_data.get(\"imp\")\n                key.update(electrode_dicts[nwbfile_elect_id])\n                query = Electrode &amp; {\"electrode_id\": nwbfile_elect_id}\n                if len(query):\n                    cls.update1(key)\n                    logger.info(\n                        f\"Updated Electrode with ID {nwbfile_elect_id}.\"\n                    )\n                else:\n                    cls.insert1(\n                        key, skip_duplicates=True, allow_direct_insert=True\n                    )\n                    logger.info(\n                        f\"Inserted Electrode with ID {nwbfile_elect_id}.\"\n                    )\n            else:\n                warnings.warn(\n                    f\"Electrode ID {nwbfile_elect_id} exists in the NWB file \"\n                    + \"but has no corresponding config YAML entry.\"\n                )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.Electrode.create_from_config", "title": "<code>create_from_config(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Create or update Electrode entries from what is specified in the config YAML file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@classmethod\ndef create_from_config(cls, nwb_file_name: str):\n    \"\"\"Create or update Electrode entries from what is specified in the config YAML file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    \"\"\"\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    nwbf = get_nwb_file(nwb_file_abspath)\n    config = get_config(nwb_file_abspath)\n    if \"Electrode\" not in config:\n        return  # See #849\n\n    # map electrode id to dictof electrode information from config YAML\n    electrode_dicts = {\n        electrode_dict[\"electrode_id\"]: electrode_dict\n        for electrode_dict in config[\"Electrode\"]\n    }\n\n    electrodes = nwbf.electrodes.to_dataframe()\n    for nwbfile_elect_id, elect_data in electrodes.iterrows():\n        if nwbfile_elect_id in electrode_dicts:\n            # use the information in the electrodes table to start and then\n            # add (or overwrite) values from the config YAML\n\n            key = dict()\n            key[\"nwb_file_name\"] = nwb_file_name\n            key[\"name\"] = str(nwbfile_elect_id)\n            key[\"electrode_group_name\"] = elect_data.group_name\n            key[\"region_id\"] = BrainRegion.fetch_add(\n                region_name=elect_data.group.location\n            )\n            key[\"x\"] = elect_data.x\n            key[\"y\"] = elect_data.y\n            key[\"z\"] = elect_data.z\n            key[\"x_warped\"] = 0\n            key[\"y_warped\"] = 0\n            key[\"z_warped\"] = 0\n            key[\"contacts\"] = \"\"\n            key[\"filtering\"] = elect_data.filtering\n            key[\"impedance\"] = elect_data.get(\"imp\")\n            key.update(electrode_dicts[nwbfile_elect_id])\n            query = Electrode &amp; {\"electrode_id\": nwbfile_elect_id}\n            if len(query):\n                cls.update1(key)\n                logger.info(\n                    f\"Updated Electrode with ID {nwbfile_elect_id}.\"\n                )\n            else:\n                cls.insert1(\n                    key, skip_duplicates=True, allow_direct_insert=True\n                )\n                logger.info(\n                    f\"Inserted Electrode with ID {nwbfile_elect_id}.\"\n                )\n        else:\n            warnings.warn(\n                f\"Electrode ID {nwbfile_elect_id} exists in the NWB file \"\n                + \"but has no corresponding config YAML entry.\"\n            )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPSelection", "title": "<code>LFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass LFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; Session\n     \"\"\"\n\n    class LFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPSelection\n        -&gt; Electrode\n        \"\"\"\n\n    def set_lfp_electrodes(self, nwb_file_name, electrode_list):\n        \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the nwb file for the desired session\n        electrode_list : list\n            list of electrodes to be used for LFP\n\n        \"\"\"\n        # remove the session and then recreate the session and Electrode list\n        (LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).delete(\n            safemode=not test_mode\n        )\n        # check to see if the user allowed the deletion\n        if (\n            len((LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).fetch())\n            == 0\n        ):\n            LFPSelection().insert1({\"nwb_file_name\": nwb_file_name})\n\n            # TODO: do this in a better way\n            all_electrodes = (\n                Electrode() &amp; {\"nwb_file_name\": nwb_file_name}\n            ).fetch(as_dict=True)\n            primary_key = Electrode.primary_key\n            for e in all_electrodes:\n                # create a dictionary so we can insert new elects\n                if e[\"electrode_id\"] in electrode_list:\n                    lfpelectdict = {\n                        k: v for k, v in e.items() if k in primary_key\n                    }\n                    LFPSelection().LFPElectrode.insert1(\n                        lfpelectdict, replace=True\n                    )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(nwb_file_name, electrode_list)</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the nwb file for the desired session</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes to be used for LFP</p> required Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>def set_lfp_electrodes(self, nwb_file_name, electrode_list):\n    \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the nwb file for the desired session\n    electrode_list : list\n        list of electrodes to be used for LFP\n\n    \"\"\"\n    # remove the session and then recreate the session and Electrode list\n    (LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).delete(\n        safemode=not test_mode\n    )\n    # check to see if the user allowed the deletion\n    if (\n        len((LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).fetch())\n        == 0\n    ):\n        LFPSelection().insert1({\"nwb_file_name\": nwb_file_name})\n\n        # TODO: do this in a better way\n        all_electrodes = (\n            Electrode() &amp; {\"nwb_file_name\": nwb_file_name}\n        ).fetch(as_dict=True)\n        primary_key = Electrode.primary_key\n        for e in all_electrodes:\n            # create a dictionary so we can insert new elects\n            if e[\"electrode_id\"] in electrode_list:\n                lfpelectdict = {\n                    k: v for k, v in e.items() if k in primary_key\n                }\n                LFPSelection().LFPElectrode.insert1(\n                    lfpelectdict, replace=True\n                )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPBandSelection", "title": "<code>LFPBandSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass LFPBandSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; LFP\n    -&gt; FirFilterParameters                   # the filter to use for the data\n    -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n    lfp_band_sampling_rate: int    # the sampling rate for this band\n    ---\n    min_interval_len = 1: float  # the minimum length of a valid interval to filter\n    \"\"\"\n\n    class LFPBandElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPBandSelection\n        -&gt; LFPSelection.LFPElectrode  # the LFP electrode to be filtered\n        reference_elect_id = -1: int  # the reference electrode to use; -1 for no reference\n        ---\n        \"\"\"\n\n    def set_lfp_band_electrodes(\n        self,\n        nwb_file_name,\n        electrode_list,\n        filter_name,\n        interval_list_name,\n        reference_electrode_list,\n        lfp_band_sampling_rate,\n    ):\n        \"\"\"\n        Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and\n        reference electrode.\n        Also removes any entries that have the same filter, interval list and reference electrode but are not\n        in the electrode_list.\n        :param nwb_file_name: string - the name of the nwb file for the desired session\n        :param electrode_list: list of LFP electrodes to be filtered\n        :param filter_name: the name of the filter (from the FirFilterParameters schema)\n        :param interval_name: the name of the interval list (from the IntervalList schema)\n        :param reference_electrode_list: A single electrode id corresponding to the reference to use for all\n        electrodes or a list with one element per entry in the electrode_list\n        :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an\n        integer divisor of the LFP sampling rate\n        :return: none\n        \"\"\"\n        # Error checks on parameters\n        # electrode_list\n        query = LFPSelection().LFPElectrode() &amp; {\"nwb_file_name\": nwb_file_name}\n        available_electrodes = query.fetch(\"electrode_id\")\n        if not np.all(np.isin(electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in electrode_list must be valid electrode_ids in the LFPSelection table\"\n            )\n        # sampling rate\n        lfp_sampling_rate = (LFP() &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\n            \"lfp_sampling_rate\"\n        )\n        decimation = lfp_sampling_rate // lfp_band_sampling_rate\n        if lfp_sampling_rate // decimation != lfp_band_sampling_rate:\n            raise ValueError(\n                f\"lfp_band_sampling rate {lfp_band_sampling_rate} is not an integer divisor of lfp \"\n                f\"samping rate {lfp_sampling_rate}\"\n            )\n        # filter\n        query = FirFilterParameters() &amp; {\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": lfp_sampling_rate,\n        }\n        if not query:\n            raise ValueError(\n                f\"filter {filter_name}, sampling rate {lfp_sampling_rate} is not in the FirFilterParameters table\"\n            )\n        # interval_list\n        query = IntervalList() &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_name\": interval_list_name,\n        }\n        if not query:\n            raise ValueError(\n                f\"interval list {interval_list_name} is not in the IntervalList table; the list must be \"\n                \"added before this function is called\"\n            )\n        # reference_electrode_list\n        if len(reference_electrode_list) != 1 and len(\n            reference_electrode_list\n        ) != len(electrode_list):\n            raise ValueError(\n                \"reference_electrode_list must contain either 1 or len(electrode_list) elements\"\n            )\n        # add a -1 element to the list to allow for the no reference option\n        available_electrodes = np.append(available_electrodes, [-1])\n        if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in reference_electrode_list must be valid electrode_ids in the LFPSelection \"\n                \"table\"\n            )\n\n        # make a list of all the references\n        ref_list = np.zeros((len(electrode_list),))\n        ref_list[:] = reference_electrode_list\n\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"filter_name\"] = filter_name\n        key[\"filter_sampling_rate\"] = lfp_sampling_rate\n        key[\"target_interval_list_name\"] = interval_list_name\n        key[\"lfp_band_sampling_rate\"] = lfp_sampling_rate // decimation\n        # insert an entry into the main LFPBandSelectionTable\n        self.insert1(key, skip_duplicates=True)\n\n        # get all of the current entries and delete any that are not in the list\n        elect_id, ref_id = (self.LFPBandElectrode() &amp; key).fetch(\n            \"electrode_id\", \"reference_elect_id\"\n        )\n        for e, r in zip(elect_id, ref_id):\n            if not len(np.where((electrode_list == e) &amp; (ref_list == r))[0]):\n                key[\"electrode_id\"] = e\n                key[\"reference_elect_id\"] = r\n                (self.LFPBandElectrode() &amp; key).delete()\n\n        # iterate through all of the new elements and add them\n        for e, r in zip(electrode_list, ref_list):\n            key[\"electrode_id\"] = e\n            query = Electrode &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_id\": e,\n            }\n            key[\"electrode_group_name\"] = query.fetch1(\"electrode_group_name\")\n            key[\"reference_elect_id\"] = r\n            self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPBandSelection.set_lfp_band_electrodes", "title": "<code>set_lfp_band_electrodes(nwb_file_name, electrode_list, filter_name, interval_list_name, reference_electrode_list, lfp_band_sampling_rate)</code>", "text": "<p>Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and reference electrode. Also removes any entries that have the same filter, interval list and reference electrode but are not in the electrode_list. :param nwb_file_name: string - the name of the nwb file for the desired session :param electrode_list: list of LFP electrodes to be filtered :param filter_name: the name of the filter (from the FirFilterParameters schema) :param interval_name: the name of the interval list (from the IntervalList schema) :param reference_electrode_list: A single electrode id corresponding to the reference to use for all electrodes or a list with one element per entry in the electrode_list :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an integer divisor of the LFP sampling rate :return: none</p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>def set_lfp_band_electrodes(\n    self,\n    nwb_file_name,\n    electrode_list,\n    filter_name,\n    interval_list_name,\n    reference_electrode_list,\n    lfp_band_sampling_rate,\n):\n    \"\"\"\n    Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and\n    reference electrode.\n    Also removes any entries that have the same filter, interval list and reference electrode but are not\n    in the electrode_list.\n    :param nwb_file_name: string - the name of the nwb file for the desired session\n    :param electrode_list: list of LFP electrodes to be filtered\n    :param filter_name: the name of the filter (from the FirFilterParameters schema)\n    :param interval_name: the name of the interval list (from the IntervalList schema)\n    :param reference_electrode_list: A single electrode id corresponding to the reference to use for all\n    electrodes or a list with one element per entry in the electrode_list\n    :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an\n    integer divisor of the LFP sampling rate\n    :return: none\n    \"\"\"\n    # Error checks on parameters\n    # electrode_list\n    query = LFPSelection().LFPElectrode() &amp; {\"nwb_file_name\": nwb_file_name}\n    available_electrodes = query.fetch(\"electrode_id\")\n    if not np.all(np.isin(electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in electrode_list must be valid electrode_ids in the LFPSelection table\"\n        )\n    # sampling rate\n    lfp_sampling_rate = (LFP() &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\n        \"lfp_sampling_rate\"\n    )\n    decimation = lfp_sampling_rate // lfp_band_sampling_rate\n    if lfp_sampling_rate // decimation != lfp_band_sampling_rate:\n        raise ValueError(\n            f\"lfp_band_sampling rate {lfp_band_sampling_rate} is not an integer divisor of lfp \"\n            f\"samping rate {lfp_sampling_rate}\"\n        )\n    # filter\n    query = FirFilterParameters() &amp; {\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": lfp_sampling_rate,\n    }\n    if not query:\n        raise ValueError(\n            f\"filter {filter_name}, sampling rate {lfp_sampling_rate} is not in the FirFilterParameters table\"\n        )\n    # interval_list\n    query = IntervalList() &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_name\": interval_list_name,\n    }\n    if not query:\n        raise ValueError(\n            f\"interval list {interval_list_name} is not in the IntervalList table; the list must be \"\n            \"added before this function is called\"\n        )\n    # reference_electrode_list\n    if len(reference_electrode_list) != 1 and len(\n        reference_electrode_list\n    ) != len(electrode_list):\n        raise ValueError(\n            \"reference_electrode_list must contain either 1 or len(electrode_list) elements\"\n        )\n    # add a -1 element to the list to allow for the no reference option\n    available_electrodes = np.append(available_electrodes, [-1])\n    if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in reference_electrode_list must be valid electrode_ids in the LFPSelection \"\n            \"table\"\n        )\n\n    # make a list of all the references\n    ref_list = np.zeros((len(electrode_list),))\n    ref_list[:] = reference_electrode_list\n\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"filter_name\"] = filter_name\n    key[\"filter_sampling_rate\"] = lfp_sampling_rate\n    key[\"target_interval_list_name\"] = interval_list_name\n    key[\"lfp_band_sampling_rate\"] = lfp_sampling_rate // decimation\n    # insert an entry into the main LFPBandSelectionTable\n    self.insert1(key, skip_duplicates=True)\n\n    # get all of the current entries and delete any that are not in the list\n    elect_id, ref_id = (self.LFPBandElectrode() &amp; key).fetch(\n        \"electrode_id\", \"reference_elect_id\"\n    )\n    for e, r in zip(elect_id, ref_id):\n        if not len(np.where((electrode_list == e) &amp; (ref_list == r))[0]):\n            key[\"electrode_id\"] = e\n            key[\"reference_elect_id\"] = r\n            (self.LFPBandElectrode() &amp; key).delete()\n\n    # iterate through all of the new elements and add them\n    for e, r in zip(electrode_list, ref_list):\n        key[\"electrode_id\"] = e\n        query = Electrode &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_id\": e,\n        }\n        key[\"electrode_group_name\"] = query.fetch1(\"electrode_group_name\")\n        key[\"reference_elect_id\"] = r\n        self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_filter/", "title": "common_filter.py", "text": ""}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters", "title": "<code>FirFilterParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>@schema\nclass FirFilterParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    filter_name: varchar(80)           # descriptive name of this filter\n    filter_sampling_rate: int          # sampling rate for this filter\n    ---\n    filter_type: enum(\"lowpass\", \"highpass\", \"bandpass\")\n    filter_low_stop = 0: float     # lowest freq for stop band for low filt\n    filter_low_pass = 0: float     # lowest freq for pass band of low filt\n    filter_high_pass = 0: float    # highest freq for pass band for high filt\n    filter_high_stop = 0: float    # highest freq for stop band of high filt\n    filter_comments: varchar(2000) # comments about the filter\n    filter_band_edges: blob        # numpy array of filter bands\n                                   # redundant with individual parameters\n    filter_coeff: longblob         # numpy array of filter coefficients\n    \"\"\"\n\n    def add_filter(\n        self,\n        filter_name: str,\n        fs: float,\n        filter_type: str,\n        band_edges: list,\n        comments: str = \"\",\n    ) -&gt; None:\n        \"\"\"Add filter to the Filter table.\n\n        Parameters\n        ----------\n        filter_name: str\n            The name of the filter.\n        fs: float\n            The filter sampling rate.\n        filter_type: str\n            The type of the filter ('lowpass', 'highpass', or 'bandpass').\n        band_edges: List[float]\n            The band edges for the filter.\n        comments: str, optional)\n            Additional comments for the filter. Default \"\".\n\n        Returns\n        -------\n        None\n            Returns None if there is an error in the filter type or band\n            frequencies.\n\n        Raises\n        ------\n        Exception:\n            Raises an exception if an unexpected filter type is encountered.\n        \"\"\"\n        VALID_FILTERS = {\"lowpass\": 2, \"highpass\": 2, \"bandpass\": 4}\n        FILTER_ERR = \"Error in Filter.add_filter: \"\n        FILTER_N_ERR = FILTER_ERR + \"filter {} requires {} band_frequencies.\"\n\n        # add an FIR bandpass filter of the specified type.\n        # band_edges should be as follows:\n        #   low pass : [high_pass high_stop]\n        #   high pass: [low stop low pass]\n        #   band pass: [low_stop low_pass high_pass high_stop].\n        if filter_type not in VALID_FILTERS:\n            logger.error(\n                FILTER_ERR\n                + f\"{filter_type} not valid type: {VALID_FILTERS.keys()}\"\n            )\n            return None\n\n        if not len(band_edges) == VALID_FILTERS[filter_type]:\n            logger.error(\n                FILTER_N_ERR.format(filter_name, VALID_FILTERS[filter_type])\n            )\n            return None\n\n        gsp = _import_ghostipy()\n        TRANS_SPLINE = 2  # transition spline will be quadratic\n\n        if filter_type != \"bandpass\":\n            transition_width = band_edges[1] - band_edges[0]\n\n        else:\n            # transition width is mean of left and right transition regions\n            transition_width = (\n                (band_edges[1] - band_edges[0])\n                + (band_edges[3] - band_edges[2])\n            ) / 2.0\n\n        numtaps = gsp.estimate_taps(fs, transition_width)\n        filterdict = {\n            \"filter_type\": filter_type,\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": fs,\n            \"filter_comments\": comments,\n            \"filter_low_stop\": 0,\n            \"filter_low_pass\": 0,\n            \"filter_high_pass\": 0,\n            \"filter_high_stop\": 0,\n            \"filter_band_edges\": np.asarray(band_edges),\n        }\n\n        # set the desired frequency response\n        if filter_type == \"lowpass\":\n            desired = [1, 0]\n            pass_stop_dict = {\n                \"filter_high_pass\": band_edges[0],\n                \"filter_high_stop\": band_edges[1],\n            }\n        elif filter_type == \"highpass\":\n            desired = [0, 1]\n            pass_stop_dict = {\n                \"filter_low_stop\": band_edges[0],\n                \"filter_low_pass\": band_edges[1],\n            }\n        else:\n            desired = [0, 1, 1, 0]\n            pass_stop_dict = {\n                \"filter_low_stop\": band_edges[0],\n                \"filter_low_pass\": band_edges[1],\n                \"filter_high_pass\": band_edges[2],\n                \"filter_high_stop\": band_edges[3],\n            }\n\n        # create 1d array for coefficients\n        filterdict.update(\n            {\n                **pass_stop_dict,\n                \"filter_coeff\": np.array(\n                    gsp.firdesign(\n                        numtaps, band_edges, desired, fs=fs, p=TRANS_SPLINE\n                    ),\n                    ndmin=1,\n                ),\n            }\n        )\n\n        self.insert1(filterdict, skip_duplicates=True)\n\n    def _filter_restrict(self, filter_name, fs):\n        return (\n            self &amp; {\"filter_name\": filter_name} &amp; {\"filter_sampling_rate\": fs}\n        ).fetch1()\n\n    def plot_magnitude(self, filter_name, fs, return_fig=False):\n        filter_dict = self._filter_restrict(filter_name, fs)\n        plt.figure()\n        w, h = signal.freqz(filter_dict[\"filter_coeff\"], worN=65536)\n        magnitude = 20 * np.log10(np.abs(h))\n        plt.plot(w / np.pi * fs / 2, magnitude)\n        plt.xlabel(\"Frequency (Hz)\")\n        plt.ylabel(\"Magnitude\")\n        plt.title(\"Frequency Response\")\n        plt.xlim(0, np.max(filter_dict[\"filter_band_edges\"] * 2))\n        plt.ylim(np.min(magnitude), -1 * np.min(magnitude) * 0.1)\n        plt.grid(True)\n        if return_fig:\n            return plt.gcf()\n\n    def plot_fir_filter(self, filter_name, fs, return_fig=False):\n        filter_dict = self._filter_restrict(filter_name, fs)\n        plt.figure()\n        plt.clf()\n        plt.plot(filter_dict[\"filter_coeff\"], \"k\")\n        plt.xlabel(\"Coefficient\")\n        plt.ylabel(\"Magnitude\")\n        plt.title(\"Filter Taps\")\n        plt.grid(True)\n        if return_fig:\n            return plt.gcf()\n\n    def filter_delay(self, filter_name, fs):\n        return self.calc_filter_delay(\n            self._filter_restrict(filter_name, fs)[\"filter_coeff\"]\n        )\n\n    def _time_bound_check(self, start, stop, all, nsamples):\n        timestamp_warn = \"Interval time warning: \"\n        if start &lt; all[0]:\n            warnings.warn(\n                timestamp_warn\n                + \"start time smaller than first timestamp, \"\n                + f\"substituting first: {start} &lt; {all[0]}\"\n            )\n            start = all[0]\n\n        if stop &gt; all[-1]:\n            logger.warning(\n                timestamp_warn\n                + \"stop time larger than last timestamp, \"\n                + f\"substituting last: {stop} &lt; {all[-1]}\"\n            )\n            stop = all[-1]\n\n        frm, to = np.searchsorted(all, (start, stop))\n        to = min(to, nsamples)\n        return frm, to\n\n    def filter_data_nwb(\n        self,\n        analysis_file_abs_path: str,\n        eseries: pynwb.ecephys.ElectricalSeries,\n        filter_coeff: np.ndarray,\n        valid_times: np.ndarray,\n        electrode_ids: list,\n        decimation: int,\n        description: str = \"filtered data\",\n        data_type: Union[None, str] = None,\n    ):\n        \"\"\"\n        Filter data from an NWB electrical series using the ghostipy package,\n        and save the result as a new electrical series in the analysis NWB file.\n\n        Parameters\n        ----------\n        analysis_file_abs_path : str\n            Full path to the analysis NWB file.\n        eseries : pynwb.ecephys.ElectricalSeries\n            Electrical series with data to be filtered.\n        filter_coeff : np.ndarray\n            Array with filter coefficients for FIR filter.\n        valid_times : np.ndarray\n            Array with start and stop times of intervals to be filtered.\n        electrode_ids : list\n            List of electrode IDs to filter.\n        decimation : int\n            Decimation factor.\n        description : str\n            Description of the filtered data.\n        data_type : Union[None, str]\n            Type of data (e.g., \"LFP\").\n\n        Returns\n        -------\n        tuple\n            The NWB object ID of the filtered data and a list containing the\n            first and last timestamps.\n        \"\"\"\n        # Note: type -&gt; data_type to avoid conflict with builtin type\n        # All existing refs to this func use positional args, so no need to\n        # adjust elsewhere, but low probability of issues with custom scripts\n\n        MEM_USE_LIMIT = 0.9  # % of RAM use permitted\n\n        gsp = _import_ghostipy()\n\n        data_on_disk = eseries.data\n        timestamps_on_disk = eseries.timestamps\n\n        n_samples = len(timestamps_on_disk)\n        time_axis = 0 if data_on_disk.shape[0] == n_samples else 1\n        electrode_axis = 1 - time_axis\n\n        n_electrodes = data_on_disk.shape[electrode_axis]\n        input_dim_restrictions = [None] * len(data_on_disk.shape)\n\n        # Get input dimension restrictions\n        input_dim_restrictions[electrode_axis] = np.s_[\n            get_electrode_indices(eseries, electrode_ids)\n        ]\n\n        indices = []\n        output_shape_list = [0] * len(data_on_disk.shape)\n        output_shape_list[electrode_axis] = len(electrode_ids)\n        data_dtype = data_on_disk[0][0].dtype\n\n        filter_delay = self.calc_filter_delay(filter_coeff)\n\n        output_offsets = [0]\n\n        for a_start, a_stop in valid_times:\n            frm, to = self._time_bound_check(\n                a_start, a_stop, timestamps_on_disk, n_samples\n            )\n\n            indices.append((frm, to))\n\n            shape, _ = gsp.filter_data_fir(\n                data_on_disk,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[frm, to - 1],\n                output_index_bounds=[filter_delay, filter_delay + to - frm],\n                describe_dims=True,\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n            )\n            output_offsets.append(output_offsets[-1] + shape[time_axis])\n            output_shape_list[time_axis] += shape[time_axis]\n\n        # Create dynamic table region and electrode series, write/close file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n\n            # get the indices of the electrodes in the electrode table\n            elect_ind = get_electrode_indices(nwbf, electrode_ids)\n\n            electrode_table_region = nwbf.create_electrode_table_region(\n                elect_ind, \"filtered electrode table\"\n            )\n            es = pynwb.ecephys.ElectricalSeries(\n                name=\"filtered data\",\n                data=np.empty(tuple(output_shape_list), dtype=data_dtype),\n                electrodes=electrode_table_region,\n                timestamps=np.empty(output_shape_list[time_axis]),\n                description=description,\n            )\n            if data_type == \"LFP\":\n                ecephys_module = nwbf.create_processing_module(\n                    name=\"ecephys\", description=description\n                )\n                ecephys_module.add(pynwb.ecephys.LFP(electrical_series=es))\n            else:\n                nwbf.add_scratch(es)\n\n            io.write(nwbf)\n\n        # Reload NWB file to get h5py objects for data/timestamps\n        # NOTE: CBroz - why io context within io context? Unindenting\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            es = nwbf.objects[es.object_id]\n            filtered_data = es.data\n            new_timestamps = es.timestamps\n            indices = np.array(indices, ndmin=2)\n            # Filter and write the output dataset\n            ts_offset = 0\n\n            logger.info(\"Filtering data\")\n            for ii, (start, stop) in enumerate(indices):\n                # Calc size of timestamps + data, check if &lt; 90% of RAM\n                interval_samples = stop - start\n                req_mem = interval_samples * (\n                    timestamps_on_disk[0].itemsize\n                    + n_electrodes * data_on_disk[0][0].itemsize\n                )\n                if req_mem &lt; MEM_USE_LIMIT * psutil.virtual_memory().available:\n                    logger.info(f\"Interval {ii}: loading data into memory\")\n                    timestamps = np.asarray(\n                        timestamps_on_disk[start:stop],\n                        dtype=timestamps_on_disk[0].dtype,\n                    )\n                    if time_axis == 0:\n                        data = np.asarray(\n                            data_on_disk[start:stop, :], dtype=data_dtype\n                        )\n                    else:\n                        data = np.asarray(\n                            data_on_disk[:, start:stop], dtype=data_dtype\n                        )\n                    extracted_ts = timestamps[0::decimation]\n                    new_timestamps[\n                        ts_offset : ts_offset + len(extracted_ts)\n                    ] = extracted_ts\n                    ts_offset += len(extracted_ts)\n                    input_index_bounds = [0, interval_samples - 1]\n\n                else:\n                    logger.info(f\"Interval {ii}: leaving data on disk\")\n                    data = data_on_disk\n                    timestamps = timestamps_on_disk\n                    extracted_ts = timestamps[start:stop:decimation]\n                    new_timestamps[\n                        ts_offset : ts_offset + len(extracted_ts)\n                    ] = extracted_ts\n                    ts_offset += len(extracted_ts)\n                    input_index_bounds = [start, stop]\n\n                # filter the data\n                gsp.filter_data_fir(\n                    data,\n                    filter_coeff,\n                    axis=time_axis,\n                    input_index_bounds=input_index_bounds,\n                    output_index_bounds=[\n                        filter_delay,\n                        filter_delay + stop - start,\n                    ],\n                    ds=decimation,\n                    input_dim_restrictions=input_dim_restrictions,\n                    outarray=filtered_data,\n                    output_offset=output_offsets[ii],\n                )\n\n            start_end = [new_timestamps[0], new_timestamps[-1]]\n\n            io.write(nwbf)\n\n        return es.object_id, start_end\n\n    def filter_data(\n        self,\n        timestamps,\n        data,\n        filter_coeff,\n        valid_times,\n        electrodes,\n        decimation,\n    ):\n        \"\"\"\n        Parameters\n        ----------\n        timestamps: numpy array\n            List of timestamps for data\n        data:\n            original data array\n        filter_coeff: numpy array\n            Filter coefficients for FIR filter\n        valid_times: 2D numpy array\n            Start and stop times of intervals to be filtered\n        electrodes: list\n            Electrodes to filter\n        decimation:\n            decimation factor\n\n        Return\n        ------\n        filtered_data, timestamps\n        \"\"\"\n\n        gsp = _import_ghostipy()\n\n        n_dim = len(data.shape)\n        n_samples = len(timestamps)\n        time_axis = 0 if data.shape[0] == n_samples else 1\n        electrode_axis = 1 - time_axis\n        input_dim_restrictions = [None] * n_dim\n        input_dim_restrictions[electrode_axis] = np.s_[electrodes]\n\n        indices = []\n        output_shape_list = [0] * n_dim\n        output_shape_list[electrode_axis] = len(electrodes)\n        output_offsets = [0]\n\n        filter_delay = self.calc_filter_delay(filter_coeff)\n        for a_start, a_stop in valid_times:\n            frm, to = self._time_bound_check(\n                a_start, a_stop, timestamps, n_samples\n            )\n            if np.isclose(frm, to, rtol=0, atol=1e-8):\n                continue\n            indices.append((frm, to))\n\n            shape, _ = gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[frm, to],\n                output_index_bounds=[filter_delay, filter_delay + to - frm],\n                describe_dims=True,\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n            )\n            output_offsets.append(output_offsets[-1] + shape[time_axis])\n            output_shape_list[time_axis] += shape[time_axis]\n\n        # create the dataset and the timestamps array\n        filtered_data = np.empty(tuple(output_shape_list), dtype=data.dtype)\n\n        new_timestamps = np.empty(\n            (output_shape_list[time_axis],), timestamps.dtype\n        )\n\n        indices = np.array(indices, ndmin=2)\n\n        # Filter  the output dataset\n        ts_offset = 0\n\n        for ii, (start, stop) in enumerate(indices):\n            extracted_ts = timestamps[start:stop:decimation]\n\n            new_timestamps[ts_offset : ts_offset + len(extracted_ts)] = (\n                extracted_ts\n            )\n            ts_offset += len(extracted_ts)\n\n            # finally ready to filter data!\n            gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[start, stop],\n                output_index_bounds=[filter_delay, filter_delay + stop - start],\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n                outarray=filtered_data,\n                output_offset=output_offsets[ii],\n            )\n\n        return filtered_data, new_timestamps\n\n    def calc_filter_delay(self, filter_coeff):\n        \"\"\"\n        Parameters\n        ----------\n        filter_coeff: numpy array\n\n        Return\n        ------\n        filter delay: int\n        \"\"\"\n        return (len(filter_coeff) - 1) // 2\n\n    def create_standard_filters(self):\n        \"\"\"Add standard filters to the Filter table\n\n        Includes 0-400 Hz low pass for continuous raw data -&gt; LFP\n        \"\"\"\n        self.add_filter(\n            \"LFP 0-400 Hz\",\n            20000,\n            \"lowpass\",\n            [400, 425],\n            \"standard LFP filter for 20 KHz data\",\n        )\n        self.add_filter(\n            \"LFP 0-400 Hz\",\n            30000,\n            \"lowpass\",\n            [400, 425],\n            \"standard LFP filter for 30 KHz data\",\n        )\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.add_filter", "title": "<code>add_filter(filter_name, fs, filter_type, band_edges, comments='')</code>", "text": "<p>Add filter to the Filter table.</p> <p>Parameters:</p> Name Type Description Default <code>filter_name</code> <code>str</code> <p>The name of the filter.</p> required <code>fs</code> <code>float</code> <p>The filter sampling rate.</p> required <code>filter_type</code> <code>str</code> <p>The type of the filter ('lowpass', 'highpass', or 'bandpass').</p> required <code>band_edges</code> <code>list</code> <p>The band edges for the filter.</p> required <code>comments</code> <code>str</code> <p>Additional comments for the filter. Default \"\".</p> <code>''</code> <p>Returns:</p> Type Description <code>None</code> <p>Returns None if there is an error in the filter type or band frequencies.</p> <p>Raises:</p> Type Description <code>Exception:</code> <p>Raises an exception if an unexpected filter type is encountered.</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def add_filter(\n    self,\n    filter_name: str,\n    fs: float,\n    filter_type: str,\n    band_edges: list,\n    comments: str = \"\",\n) -&gt; None:\n    \"\"\"Add filter to the Filter table.\n\n    Parameters\n    ----------\n    filter_name: str\n        The name of the filter.\n    fs: float\n        The filter sampling rate.\n    filter_type: str\n        The type of the filter ('lowpass', 'highpass', or 'bandpass').\n    band_edges: List[float]\n        The band edges for the filter.\n    comments: str, optional)\n        Additional comments for the filter. Default \"\".\n\n    Returns\n    -------\n    None\n        Returns None if there is an error in the filter type or band\n        frequencies.\n\n    Raises\n    ------\n    Exception:\n        Raises an exception if an unexpected filter type is encountered.\n    \"\"\"\n    VALID_FILTERS = {\"lowpass\": 2, \"highpass\": 2, \"bandpass\": 4}\n    FILTER_ERR = \"Error in Filter.add_filter: \"\n    FILTER_N_ERR = FILTER_ERR + \"filter {} requires {} band_frequencies.\"\n\n    # add an FIR bandpass filter of the specified type.\n    # band_edges should be as follows:\n    #   low pass : [high_pass high_stop]\n    #   high pass: [low stop low pass]\n    #   band pass: [low_stop low_pass high_pass high_stop].\n    if filter_type not in VALID_FILTERS:\n        logger.error(\n            FILTER_ERR\n            + f\"{filter_type} not valid type: {VALID_FILTERS.keys()}\"\n        )\n        return None\n\n    if not len(band_edges) == VALID_FILTERS[filter_type]:\n        logger.error(\n            FILTER_N_ERR.format(filter_name, VALID_FILTERS[filter_type])\n        )\n        return None\n\n    gsp = _import_ghostipy()\n    TRANS_SPLINE = 2  # transition spline will be quadratic\n\n    if filter_type != \"bandpass\":\n        transition_width = band_edges[1] - band_edges[0]\n\n    else:\n        # transition width is mean of left and right transition regions\n        transition_width = (\n            (band_edges[1] - band_edges[0])\n            + (band_edges[3] - band_edges[2])\n        ) / 2.0\n\n    numtaps = gsp.estimate_taps(fs, transition_width)\n    filterdict = {\n        \"filter_type\": filter_type,\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": fs,\n        \"filter_comments\": comments,\n        \"filter_low_stop\": 0,\n        \"filter_low_pass\": 0,\n        \"filter_high_pass\": 0,\n        \"filter_high_stop\": 0,\n        \"filter_band_edges\": np.asarray(band_edges),\n    }\n\n    # set the desired frequency response\n    if filter_type == \"lowpass\":\n        desired = [1, 0]\n        pass_stop_dict = {\n            \"filter_high_pass\": band_edges[0],\n            \"filter_high_stop\": band_edges[1],\n        }\n    elif filter_type == \"highpass\":\n        desired = [0, 1]\n        pass_stop_dict = {\n            \"filter_low_stop\": band_edges[0],\n            \"filter_low_pass\": band_edges[1],\n        }\n    else:\n        desired = [0, 1, 1, 0]\n        pass_stop_dict = {\n            \"filter_low_stop\": band_edges[0],\n            \"filter_low_pass\": band_edges[1],\n            \"filter_high_pass\": band_edges[2],\n            \"filter_high_stop\": band_edges[3],\n        }\n\n    # create 1d array for coefficients\n    filterdict.update(\n        {\n            **pass_stop_dict,\n            \"filter_coeff\": np.array(\n                gsp.firdesign(\n                    numtaps, band_edges, desired, fs=fs, p=TRANS_SPLINE\n                ),\n                ndmin=1,\n            ),\n        }\n    )\n\n    self.insert1(filterdict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data_nwb", "title": "<code>filter_data_nwb(analysis_file_abs_path, eseries, filter_coeff, valid_times, electrode_ids, decimation, description='filtered data', data_type=None)</code>", "text": "<p>Filter data from an NWB electrical series using the ghostipy package, and save the result as a new electrical series in the analysis NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_abs_path</code> <code>str</code> <p>Full path to the analysis NWB file.</p> required <code>eseries</code> <code>ElectricalSeries</code> <p>Electrical series with data to be filtered.</p> required <code>filter_coeff</code> <code>ndarray</code> <p>Array with filter coefficients for FIR filter.</p> required <code>valid_times</code> <code>ndarray</code> <p>Array with start and stop times of intervals to be filtered.</p> required <code>electrode_ids</code> <code>list</code> <p>List of electrode IDs to filter.</p> required <code>decimation</code> <code>int</code> <p>Decimation factor.</p> required <code>description</code> <code>str</code> <p>Description of the filtered data.</p> <code>'filtered data'</code> <code>data_type</code> <code>Union[None, str]</code> <p>Type of data (e.g., \"LFP\").</p> <code>None</code> <p>Returns:</p> Type Description <code>tuple</code> <p>The NWB object ID of the filtered data and a list containing the first and last timestamps.</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def filter_data_nwb(\n    self,\n    analysis_file_abs_path: str,\n    eseries: pynwb.ecephys.ElectricalSeries,\n    filter_coeff: np.ndarray,\n    valid_times: np.ndarray,\n    electrode_ids: list,\n    decimation: int,\n    description: str = \"filtered data\",\n    data_type: Union[None, str] = None,\n):\n    \"\"\"\n    Filter data from an NWB electrical series using the ghostipy package,\n    and save the result as a new electrical series in the analysis NWB file.\n\n    Parameters\n    ----------\n    analysis_file_abs_path : str\n        Full path to the analysis NWB file.\n    eseries : pynwb.ecephys.ElectricalSeries\n        Electrical series with data to be filtered.\n    filter_coeff : np.ndarray\n        Array with filter coefficients for FIR filter.\n    valid_times : np.ndarray\n        Array with start and stop times of intervals to be filtered.\n    electrode_ids : list\n        List of electrode IDs to filter.\n    decimation : int\n        Decimation factor.\n    description : str\n        Description of the filtered data.\n    data_type : Union[None, str]\n        Type of data (e.g., \"LFP\").\n\n    Returns\n    -------\n    tuple\n        The NWB object ID of the filtered data and a list containing the\n        first and last timestamps.\n    \"\"\"\n    # Note: type -&gt; data_type to avoid conflict with builtin type\n    # All existing refs to this func use positional args, so no need to\n    # adjust elsewhere, but low probability of issues with custom scripts\n\n    MEM_USE_LIMIT = 0.9  # % of RAM use permitted\n\n    gsp = _import_ghostipy()\n\n    data_on_disk = eseries.data\n    timestamps_on_disk = eseries.timestamps\n\n    n_samples = len(timestamps_on_disk)\n    time_axis = 0 if data_on_disk.shape[0] == n_samples else 1\n    electrode_axis = 1 - time_axis\n\n    n_electrodes = data_on_disk.shape[electrode_axis]\n    input_dim_restrictions = [None] * len(data_on_disk.shape)\n\n    # Get input dimension restrictions\n    input_dim_restrictions[electrode_axis] = np.s_[\n        get_electrode_indices(eseries, electrode_ids)\n    ]\n\n    indices = []\n    output_shape_list = [0] * len(data_on_disk.shape)\n    output_shape_list[electrode_axis] = len(electrode_ids)\n    data_dtype = data_on_disk[0][0].dtype\n\n    filter_delay = self.calc_filter_delay(filter_coeff)\n\n    output_offsets = [0]\n\n    for a_start, a_stop in valid_times:\n        frm, to = self._time_bound_check(\n            a_start, a_stop, timestamps_on_disk, n_samples\n        )\n\n        indices.append((frm, to))\n\n        shape, _ = gsp.filter_data_fir(\n            data_on_disk,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[frm, to - 1],\n            output_index_bounds=[filter_delay, filter_delay + to - frm],\n            describe_dims=True,\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n        )\n        output_offsets.append(output_offsets[-1] + shape[time_axis])\n        output_shape_list[time_axis] += shape[time_axis]\n\n    # Create dynamic table region and electrode series, write/close file\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n\n        # get the indices of the electrodes in the electrode table\n        elect_ind = get_electrode_indices(nwbf, electrode_ids)\n\n        electrode_table_region = nwbf.create_electrode_table_region(\n            elect_ind, \"filtered electrode table\"\n        )\n        es = pynwb.ecephys.ElectricalSeries(\n            name=\"filtered data\",\n            data=np.empty(tuple(output_shape_list), dtype=data_dtype),\n            electrodes=electrode_table_region,\n            timestamps=np.empty(output_shape_list[time_axis]),\n            description=description,\n        )\n        if data_type == \"LFP\":\n            ecephys_module = nwbf.create_processing_module(\n                name=\"ecephys\", description=description\n            )\n            ecephys_module.add(pynwb.ecephys.LFP(electrical_series=es))\n        else:\n            nwbf.add_scratch(es)\n\n        io.write(nwbf)\n\n    # Reload NWB file to get h5py objects for data/timestamps\n    # NOTE: CBroz - why io context within io context? Unindenting\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        es = nwbf.objects[es.object_id]\n        filtered_data = es.data\n        new_timestamps = es.timestamps\n        indices = np.array(indices, ndmin=2)\n        # Filter and write the output dataset\n        ts_offset = 0\n\n        logger.info(\"Filtering data\")\n        for ii, (start, stop) in enumerate(indices):\n            # Calc size of timestamps + data, check if &lt; 90% of RAM\n            interval_samples = stop - start\n            req_mem = interval_samples * (\n                timestamps_on_disk[0].itemsize\n                + n_electrodes * data_on_disk[0][0].itemsize\n            )\n            if req_mem &lt; MEM_USE_LIMIT * psutil.virtual_memory().available:\n                logger.info(f\"Interval {ii}: loading data into memory\")\n                timestamps = np.asarray(\n                    timestamps_on_disk[start:stop],\n                    dtype=timestamps_on_disk[0].dtype,\n                )\n                if time_axis == 0:\n                    data = np.asarray(\n                        data_on_disk[start:stop, :], dtype=data_dtype\n                    )\n                else:\n                    data = np.asarray(\n                        data_on_disk[:, start:stop], dtype=data_dtype\n                    )\n                extracted_ts = timestamps[0::decimation]\n                new_timestamps[\n                    ts_offset : ts_offset + len(extracted_ts)\n                ] = extracted_ts\n                ts_offset += len(extracted_ts)\n                input_index_bounds = [0, interval_samples - 1]\n\n            else:\n                logger.info(f\"Interval {ii}: leaving data on disk\")\n                data = data_on_disk\n                timestamps = timestamps_on_disk\n                extracted_ts = timestamps[start:stop:decimation]\n                new_timestamps[\n                    ts_offset : ts_offset + len(extracted_ts)\n                ] = extracted_ts\n                ts_offset += len(extracted_ts)\n                input_index_bounds = [start, stop]\n\n            # filter the data\n            gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=input_index_bounds,\n                output_index_bounds=[\n                    filter_delay,\n                    filter_delay + stop - start,\n                ],\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n                outarray=filtered_data,\n                output_offset=output_offsets[ii],\n            )\n\n        start_end = [new_timestamps[0], new_timestamps[-1]]\n\n        io.write(nwbf)\n\n    return es.object_id, start_end\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data", "title": "<code>filter_data(timestamps, data, filter_coeff, valid_times, electrodes, decimation)</code>", "text": "<p>Parameters:</p> Name Type Description Default <code>timestamps</code> <p>List of timestamps for data</p> required <code>data</code> <p>original data array</p> required <code>filter_coeff</code> <p>Filter coefficients for FIR filter</p> required <code>valid_times</code> <p>Start and stop times of intervals to be filtered</p> required <code>electrodes</code> <p>Electrodes to filter</p> required <code>decimation</code> <p>decimation factor</p> required"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data--return", "title": "Return", "text": "<p>filtered_data, timestamps</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def filter_data(\n    self,\n    timestamps,\n    data,\n    filter_coeff,\n    valid_times,\n    electrodes,\n    decimation,\n):\n    \"\"\"\n    Parameters\n    ----------\n    timestamps: numpy array\n        List of timestamps for data\n    data:\n        original data array\n    filter_coeff: numpy array\n        Filter coefficients for FIR filter\n    valid_times: 2D numpy array\n        Start and stop times of intervals to be filtered\n    electrodes: list\n        Electrodes to filter\n    decimation:\n        decimation factor\n\n    Return\n    ------\n    filtered_data, timestamps\n    \"\"\"\n\n    gsp = _import_ghostipy()\n\n    n_dim = len(data.shape)\n    n_samples = len(timestamps)\n    time_axis = 0 if data.shape[0] == n_samples else 1\n    electrode_axis = 1 - time_axis\n    input_dim_restrictions = [None] * n_dim\n    input_dim_restrictions[electrode_axis] = np.s_[electrodes]\n\n    indices = []\n    output_shape_list = [0] * n_dim\n    output_shape_list[electrode_axis] = len(electrodes)\n    output_offsets = [0]\n\n    filter_delay = self.calc_filter_delay(filter_coeff)\n    for a_start, a_stop in valid_times:\n        frm, to = self._time_bound_check(\n            a_start, a_stop, timestamps, n_samples\n        )\n        if np.isclose(frm, to, rtol=0, atol=1e-8):\n            continue\n        indices.append((frm, to))\n\n        shape, _ = gsp.filter_data_fir(\n            data,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[frm, to],\n            output_index_bounds=[filter_delay, filter_delay + to - frm],\n            describe_dims=True,\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n        )\n        output_offsets.append(output_offsets[-1] + shape[time_axis])\n        output_shape_list[time_axis] += shape[time_axis]\n\n    # create the dataset and the timestamps array\n    filtered_data = np.empty(tuple(output_shape_list), dtype=data.dtype)\n\n    new_timestamps = np.empty(\n        (output_shape_list[time_axis],), timestamps.dtype\n    )\n\n    indices = np.array(indices, ndmin=2)\n\n    # Filter  the output dataset\n    ts_offset = 0\n\n    for ii, (start, stop) in enumerate(indices):\n        extracted_ts = timestamps[start:stop:decimation]\n\n        new_timestamps[ts_offset : ts_offset + len(extracted_ts)] = (\n            extracted_ts\n        )\n        ts_offset += len(extracted_ts)\n\n        # finally ready to filter data!\n        gsp.filter_data_fir(\n            data,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[start, stop],\n            output_index_bounds=[filter_delay, filter_delay + stop - start],\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n            outarray=filtered_data,\n            output_offset=output_offsets[ii],\n        )\n\n    return filtered_data, new_timestamps\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.calc_filter_delay", "title": "<code>calc_filter_delay(filter_coeff)</code>", "text": "<p>Parameters:</p> Name Type Description Default <code>filter_coeff</code> required"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.calc_filter_delay--return", "title": "Return", "text": "<p>filter delay: int</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def calc_filter_delay(self, filter_coeff):\n    \"\"\"\n    Parameters\n    ----------\n    filter_coeff: numpy array\n\n    Return\n    ------\n    filter delay: int\n    \"\"\"\n    return (len(filter_coeff) - 1) // 2\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.create_standard_filters", "title": "<code>create_standard_filters()</code>", "text": "<p>Add standard filters to the Filter table</p> <p>Includes 0-400 Hz low pass for continuous raw data -&gt; LFP</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def create_standard_filters(self):\n    \"\"\"Add standard filters to the Filter table\n\n    Includes 0-400 Hz low pass for continuous raw data -&gt; LFP\n    \"\"\"\n    self.add_filter(\n        \"LFP 0-400 Hz\",\n        20000,\n        \"lowpass\",\n        [400, 425],\n        \"standard LFP filter for 20 KHz data\",\n    )\n    self.add_filter(\n        \"LFP 0-400 Hz\",\n        30000,\n        \"lowpass\",\n        [400, 425],\n        \"standard LFP filter for 30 KHz data\",\n    )\n</code></pre>"}, {"location": "api/common/common_interval/", "title": "common_interval.py", "text": ""}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.IntervalList", "title": "<code>IntervalList</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>@schema\nclass IntervalList(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Time intervals used for analysis\n    -&gt; Session\n    interval_list_name: varchar(170)  # descriptive name of this interval list\n    ---\n    valid_times: longblob  # numpy array with start/end times for each interval\n    pipeline = \"\": varchar(64)  # type of interval list (e.g. 'position', 'spikesorting_recording_v1')\n    \"\"\"\n\n    # See #630, #664. Excessive key length.\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, *, nwb_file_name):\n        \"\"\"Add each entry in the NWB file epochs table to the IntervalList.\n\n        The interval list name for each epoch is set to the first tag for the\n        epoch. If the epoch has no tags, then 'interval_x' will be used as the\n        interval list name, where x is the index (0-indexed) of the epoch in the\n        epochs table. The start time and stop time of the epoch are stored in\n        the valid_times field as a numpy array of [start time, stop time] for\n        each epoch.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        nwb_file_name : str\n            The file name of the NWB file, used as a primary key to the Session\n            table.\n        \"\"\"\n        if nwbf.epochs is None:\n            logger.info(\"No epochs found in NWB file.\")\n            return\n\n        epochs = nwbf.epochs.to_dataframe()\n\n        for _, epoch_data in epochs.iterrows():\n            epoch_dict = {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": (\n                    epoch_data.tags[0]\n                    if epoch_data.tags\n                    else f\"interval_{epoch_data[0]}\"\n                ),\n                \"valid_times\": np.asarray(\n                    [[epoch_data.start_time, epoch_data.stop_time]]\n                ),\n            }\n\n            cls.insert1(epoch_dict, skip_duplicates=True)\n\n    def plot_intervals(self, figsize=(20, 5), return_fig=False):\n        interval_list = pd.DataFrame(self)\n        fig, ax = plt.subplots(figsize=figsize)\n        interval_count = 0\n        for row in interval_list.itertuples(index=False):\n            for interval in row.valid_times:\n                ax.plot(interval, [interval_count, interval_count])\n                ax.scatter(\n                    interval,\n                    [interval_count, interval_count],\n                    alpha=0.8,\n                    zorder=2,\n                )\n            interval_count += 1\n        ax.set_yticks(np.arange(interval_list.shape[0]))\n        ax.set_yticklabels(interval_list.interval_list_name)\n        ax.set_xlabel(\"Time [s]\")\n        ax.grid(True)\n        if return_fig:\n            return fig\n\n    def plot_epoch_pos_raw_intervals(self, figsize=(20, 5), return_fig=False):\n        interval_list = pd.DataFrame(self)\n        fig, ax = plt.subplots(figsize=(30, 3))\n\n        raw_data_valid_times = interval_list.loc[\n            interval_list.interval_list_name == \"raw data valid times\"\n        ].valid_times\n        interval_y = 1\n\n        for interval in np.asarray(raw_data_valid_times)[0]:\n            ax.plot(interval, [interval_y, interval_y])\n            ax.scatter(interval, [interval_y, interval_y], alpha=0.8, zorder=2)\n\n        epoch_valid_times = (\n            interval_list.set_index(\"interval_list_name\")\n            .filter(regex=r\"^[0-9]\", axis=0)\n            .valid_times\n        )\n        interval_y = 2\n        for epoch, valid_times in zip(\n            epoch_valid_times.index, epoch_valid_times\n        ):\n            for interval in valid_times:\n                ax.plot(interval, [interval_y, interval_y])\n                ax.scatter(\n                    interval, [interval_y, interval_y], alpha=0.8, zorder=2\n                )\n                ax.text(\n                    interval[0] + np.diff(interval)[0] / 2,\n                    interval_y,\n                    epoch,\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n\n        pos_valid_times = (\n            interval_list.set_index(\"interval_list_name\")\n            .filter(regex=r\"^pos \\d+ valid times$\", axis=0)\n            .valid_times\n        ).sort_index(key=lambda index: [int(name.split()[1]) for name in index])\n        interval_y = 0\n        for epoch, valid_times in zip(pos_valid_times.index, pos_valid_times):\n            for interval in valid_times:\n                ax.plot(interval, [interval_y, interval_y])\n                ax.scatter(\n                    interval, [interval_y, interval_y], alpha=0.8, zorder=2\n                )\n                ax.text(\n                    interval[0] + np.diff(interval)[0] / 2,\n                    interval_y,\n                    epoch.strip(\" valid times\"),\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n\n        ax.set_ylim((-0.25, 2.25))\n        ax.set_yticks(np.arange(3))\n        ax.set_yticklabels([\"pos valid times\", \"raw data valid times\", \"epoch\"])\n        ax.set_xlabel(\"Time [s]\")\n        ax.grid(True)\n        if return_fig:\n            return fig\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.IntervalList.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, *, nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Add each entry in the NWB file epochs table to the IntervalList.</p> <p>The interval list name for each epoch is set to the first tag for the epoch. If the epoch has no tags, then 'interval_x' will be used as the interval list name, where x is the index (0-indexed) of the epoch in the epochs table. The start time and stop time of the epoch are stored in the valid_times field as a numpy array of [start time, stop time] for each epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>nwb_file_name</code> <code>str</code> <p>The file name of the NWB file, used as a primary key to the Session table.</p> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, *, nwb_file_name):\n    \"\"\"Add each entry in the NWB file epochs table to the IntervalList.\n\n    The interval list name for each epoch is set to the first tag for the\n    epoch. If the epoch has no tags, then 'interval_x' will be used as the\n    interval list name, where x is the index (0-indexed) of the epoch in the\n    epochs table. The start time and stop time of the epoch are stored in\n    the valid_times field as a numpy array of [start time, stop time] for\n    each epoch.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    nwb_file_name : str\n        The file name of the NWB file, used as a primary key to the Session\n        table.\n    \"\"\"\n    if nwbf.epochs is None:\n        logger.info(\"No epochs found in NWB file.\")\n        return\n\n    epochs = nwbf.epochs.to_dataframe()\n\n    for _, epoch_data in epochs.iterrows():\n        epoch_dict = {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": (\n                epoch_data.tags[0]\n                if epoch_data.tags\n                else f\"interval_{epoch_data[0]}\"\n            ),\n            \"valid_times\": np.asarray(\n                [[epoch_data.start_time, epoch_data.stop_time]]\n            ),\n        }\n\n        cls.insert1(epoch_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.intervals_by_length", "title": "<code>intervals_by_length(interval_list, min_length=0.0, max_length=10000000000.0)</code>", "text": "<p>Select intervals of certain lengths from an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>min_length</code> <code>float</code> <p>Minimum interval length in seconds. Defaults to 0.0.</p> <code>0.0</code> <code>max_length</code> <code>float</code> <p>Maximum interval length in seconds. Defaults to 1e10.</p> <code>10000000000.0</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def intervals_by_length(interval_list, min_length=0.0, max_length=1e10):\n    \"\"\"Select intervals of certain lengths from an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    min_length : float, optional\n        Minimum interval length in seconds. Defaults to 0.0.\n    max_length : float, optional\n        Maximum interval length in seconds. Defaults to 1e10.\n    \"\"\"\n    lengths = np.ravel(np.diff(interval_list))\n    return interval_list[\n        np.logical_and(lengths &gt; min_length, lengths &lt; max_length)\n    ]\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_contains_ind", "title": "<code>interval_list_contains_ind(interval_list, timestamps)</code>", "text": "<p>Find indices of list of timestamps contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_contains_ind(interval_list, timestamps):\n    \"\"\"Find indices of list of timestamps contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    ind = []\n    for interval in interval_list:\n        ind += np.ravel(\n            np.argwhere(\n                np.logical_and(\n                    timestamps &gt;= interval[0], timestamps &lt;= interval[1]\n                )\n            )\n        ).tolist()\n    return np.asarray(ind)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_contains", "title": "<code>interval_list_contains(interval_list, timestamps)</code>", "text": "<p>Find timestamps that are contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_contains(interval_list, timestamps):\n    \"\"\"Find timestamps that are contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    ind = []\n    for interval in interval_list:\n        ind += np.ravel(\n            np.argwhere(\n                np.logical_and(\n                    timestamps &gt;= interval[0], timestamps &lt;= interval[1]\n                )\n            )\n        ).tolist()\n    return timestamps[ind]\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_excludes_ind", "title": "<code>interval_list_excludes_ind(interval_list, timestamps)</code>", "text": "<p>Find indices of timestamps that are not contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_excludes_ind(interval_list, timestamps):\n    \"\"\"Find indices of timestamps that are not contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n\n    contained_inds = interval_list_contains_ind(interval_list, timestamps)\n    return np.setdiff1d(np.arange(len(timestamps)), contained_inds)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_excludes", "title": "<code>interval_list_excludes(interval_list, timestamps)</code>", "text": "<p>Find timestamps that are not contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_excludes(interval_list, timestamps):\n    \"\"\"Find timestamps that are not contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    contained_times = interval_list_contains(interval_list, timestamps)\n    return np.setdiff1d(timestamps, contained_times)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_intersect", "title": "<code>interval_list_intersect(interval_list1, interval_list2, min_length=0)</code>", "text": "<p>Finds the intersections between two interval lists</p> <p>Parameters:</p> Name Type Description Default <code>interval_list1</code> <code>np.array, (N,2) where N = number of intervals</code> required <code>interval_list2</code> <code>np.array, (N,2) where N = number of intervals</code> required <code>min_length</code> <code>float, optional.</code> <p>Minimum length of intervals to include, default 0</p> <code>0</code> <p>Each interval is (start time, stop time)</p> <p>Returns:</p> Name Type Description <code>interval_list</code> <code>(array, (N, 2))</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_intersect(interval_list1, interval_list2, min_length=0):\n    \"\"\"Finds the intersections between two interval lists\n\n    Parameters\n    ----------\n    interval_list1 : np.array, (N,2) where N = number of intervals\n    interval_list2 : np.array, (N,2) where N = number of intervals\n    min_length : float, optional.\n        Minimum length of intervals to include, default 0\n\n    Each interval is (start time, stop time)\n\n    Returns\n    -------\n    interval_list: np.array, (N,2)\n    \"\"\"\n\n    # Consolidate interval lists to disjoint int'ls by sorting &amp; applying union\n    interval_list1 = consolidate_intervals(interval_list1)\n    interval_list2 = consolidate_intervals(interval_list2)\n\n    # then do pairwise comparison and collect intersections\n    intersecting_intervals = [\n        _intersection(interval2, interval1)\n        for interval2 in interval_list2\n        for interval1 in interval_list1\n        if _intersection(interval2, interval1) is not None\n    ]\n\n    # if no intersection, then return an empty list\n    if not intersecting_intervals:\n        return []\n\n    intersecting_intervals = np.asarray(intersecting_intervals)\n    intersecting_intervals = intersecting_intervals[\n        np.argsort(intersecting_intervals[:, 0])\n    ]\n\n    return intervals_by_length(intersecting_intervals, min_length=min_length)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.union_adjacent_index", "title": "<code>union_adjacent_index(interval1, interval2)</code>", "text": "<p>Union index-adjacent intervals. If not adjacent, just concatenate.</p> <p>e.g. [a,b] and [b+1, c] is converted to [a,c]</p> <p>Parameters:</p> Name Type Description Default <code>interval1</code> <code>array</code> required <code>interval2</code> <code>array</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def union_adjacent_index(interval1, interval2):\n    \"\"\"Union index-adjacent intervals. If not adjacent, just concatenate.\n\n    e.g. [a,b] and [b+1, c] is converted to [a,c]\n\n    Parameters\n    ----------\n    interval1 : np.array\n    interval2 : np.array\n    \"\"\"\n    interval1 = np.atleast_2d(interval1)\n    interval2 = np.atleast_2d(interval2)\n\n    if (\n        interval1[-1][1] + 1 == interval2[0][0]\n        or interval2[0][1] + 1 == interval1[-1][0]\n    ):\n        x = np.array(\n            [\n                [\n                    np.min([interval1[-1][0], interval2[0][0]]),\n                    np.max([interval1[-1][1], interval2[0][1]]),\n                ]\n            ]\n        )\n        return np.concatenate((interval1[:-1], x), axis=0)\n    else:\n        return np.concatenate((interval1, interval2), axis=0)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_union", "title": "<code>interval_list_union(interval_list1, interval_list2, min_length=0.0, max_length=10000000000.0)</code>", "text": "<p>Finds the union (all times in one or both) for two interval lists</p> <p>Parameters:</p> Name Type Description Default <code>interval_list1</code> <code>ndarray</code> <p>The first interval list [start, stop]</p> required <code>interval_list2</code> <code>ndarray</code> <p>The second interval list [start, stop]</p> required <code>min_length</code> <code>float</code> <p>Minimum length of interval for inclusion in output, default 0.0</p> <code>0.0</code> <code>max_length</code> <code>float</code> <p>Maximum length of interval for inclusion in output, default 1e10</p> <code>10000000000.0</code> <p>Returns:</p> Type Description <code>ndarray</code> <p>Array of intervals [start, stop]</p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_union(\n    interval_list1: np.ndarray,\n    interval_list2: np.ndarray,\n    min_length: float = 0.0,\n    max_length: float = 1e10,\n) -&gt; np.ndarray:\n    \"\"\"Finds the union (all times in one or both) for two interval lists\n\n    Parameters\n    ----------\n    interval_list1 : np.ndarray\n        The first interval list [start, stop]\n    interval_list2 : np.ndarray\n        The second interval list [start, stop]\n    min_length : float, optional\n        Minimum length of interval for inclusion in output, default 0.0\n    max_length : float, optional\n        Maximum length of interval for inclusion in output, default 1e10\n\n    Returns\n    -------\n    np.ndarray\n        Array of intervals [start, stop]\n    \"\"\"\n\n    il1, il1_start_end = _parallel_union(interval_list1)\n    il2, il2_start_end = _parallel_union(interval_list2)\n\n    # Concatenate the two lists so we can resort the intervals and apply the\n    # same sorting to the start-end arrays\n    combined_intervals = np.concatenate((il1, il2))\n    ss = np.concatenate((il1_start_end, il2_start_end))\n    sort_ind = np.argsort(combined_intervals)\n    combined_intervals = combined_intervals[sort_ind]\n\n    # a cumulative sum of 1 indicates the beginning of a joint interval; a\n    # cumulative sum of 0 indicates the end\n    union_starts = np.ravel(np.array(np.where(np.cumsum(ss[sort_ind]) == 1)))\n    union_stops = np.ravel(np.array(np.where(np.cumsum(ss[sort_ind]) == 0)))\n    union = [\n        [combined_intervals[start], combined_intervals[stop]]\n        for start, stop in zip(union_starts, union_stops)\n    ]\n\n    return np.asarray(union)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_censor", "title": "<code>interval_list_censor(interval_list, timestamps)</code>", "text": "<p>Returns new interval list that starts/ends at first/last timestamp</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>numpy array of intervals [start, stop]</code> <p>interval list from IntervalList valid times</p> required <code>timestamps</code> <code>numpy array or list</code> required <p>Returns:</p> Type Description <code>interval_list (numpy array of intervals [start, stop])</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_censor(interval_list, timestamps):\n    \"\"\"Returns new interval list that starts/ends at first/last timestamp\n\n    Parameters\n    ----------\n    interval_list : numpy array of intervals [start, stop]\n        interval list from IntervalList valid times\n    timestamps : numpy array or list\n\n    Returns\n    -------\n    interval_list (numpy array of intervals [start, stop])\n    \"\"\"\n    # check that all timestamps are in the interval list\n    if len(interval_list_contains_ind(interval_list, timestamps)) != len(\n        timestamps\n    ):\n        raise ValueError(\"Interval_list must contain all timestamps\")\n\n    timestamps_interval = np.asarray([[timestamps[0], timestamps[-1]]])\n    return interval_list_intersect(interval_list, timestamps_interval)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_from_inds", "title": "<code>interval_from_inds(list_frames)</code>", "text": "<p>Converts a list of indices to a list of intervals.</p> <p>e.g. [2,3,4,6,7,8,9,10] -&gt; [[2,4],[6,10]]</p> <p>Parameters:</p> Name Type Description Default <code>list_frames</code> <code>array_like of int</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_from_inds(list_frames):\n    \"\"\"Converts a list of indices to a list of intervals.\n\n    e.g. [2,3,4,6,7,8,9,10] -&gt; [[2,4],[6,10]]\n\n    Parameters\n    ----------\n    list_frames : array_like of int\n    \"\"\"\n    list_frames = np.unique(list_frames)\n    interval_list = []\n    for key, group in itertools.groupby(\n        enumerate(list_frames), lambda t: t[1] - t[0]\n    ):\n        group = list(group)\n        interval_list.append([group[0][1], group[-1][1]])\n    return np.asarray(interval_list)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_set_difference_inds", "title": "<code>interval_set_difference_inds(intervals1, intervals2)</code>", "text": "<p>e.g. intervals1 = [(0, 5), (8, 10)] intervals2 = [(1, 2), (3, 4), (6, 9)]</p> <p>result = [(0, 1), (4, 5), (9, 10)]</p> <p>Parameters:</p> Name Type Description Default <code>intervals1</code> <code>_type_</code> <p>description</p> required <code>intervals2</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_set_difference_inds(intervals1, intervals2):\n    \"\"\"\n    e.g.\n    intervals1 = [(0, 5), (8, 10)]\n    intervals2 = [(1, 2), (3, 4), (6, 9)]\n\n    result = [(0, 1), (4, 5), (9, 10)]\n\n    Parameters\n    ----------\n    intervals1 : _type_\n        _description_\n    intervals2 : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    result = []\n    i = j = 0\n    while i &lt; len(intervals1) and j &lt; len(intervals2):\n        if intervals1[i][1] &lt;= intervals2[j][0]:\n            result.append(intervals1[i])\n            i += 1\n        elif intervals2[j][1] &lt;= intervals1[i][0]:\n            j += 1\n        else:\n            if intervals1[i][0] &lt; intervals2[j][0]:\n                result.append((intervals1[i][0], intervals2[j][0]))\n            if intervals1[i][1] &gt; intervals2[j][1]:\n                intervals1[i] = (intervals2[j][1], intervals1[i][1])\n                j += 1\n            else:\n                i += 1\n    result += intervals1[i:]\n    return result\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_complement", "title": "<code>interval_list_complement(intervals1, intervals2, min_length=0.0)</code>", "text": "<p>Finds intervals in intervals1 that are not in intervals2</p> <p>Parameters:</p> Name Type Description Default <code>min_length</code> <code>float</code> <p>Minimum interval length in seconds. Defaults to 0.0.</p> <code>0.0</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_complement(intervals1, intervals2, min_length=0.0):\n    \"\"\"\n    Finds intervals in intervals1 that are not in intervals2\n\n    Parameters\n    ----------\n    min_length : float, optional\n        Minimum interval length in seconds. Defaults to 0.0.\n    \"\"\"\n\n    result = []\n\n    for start1, end1 in intervals1:\n        subtracted = [(start1, end1)]\n\n        for start2, end2 in intervals2:\n            new_subtracted = []\n\n            for s, e in subtracted:\n                if start2 &lt;= s and e &lt;= end2:\n                    continue\n\n                if e &lt;= start2 or end2 &lt;= s:\n                    new_subtracted.append((s, e))\n                    continue\n\n                if start2 &gt; s:\n                    new_subtracted.append((s, start2))\n\n                if end2 &lt; e:\n                    new_subtracted.append((end2, e))\n\n            subtracted = new_subtracted\n\n        result.extend(subtracted)\n\n    return intervals_by_length(\n        np.asarray(result), min_length=min_length, max_length=1e100\n    )\n</code></pre>"}, {"location": "api/common/common_lab/", "title": "common_lab.py", "text": "<p>Schema for institution, lab team/name/members. Session-independent.</p>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember", "title": "<code>LabMember</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass LabMember(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    lab_member_name: varchar(80)\n    ---\n    first_name: varchar(200)\n    last_name: varchar(200)\n    \"\"\"\n\n    # NOTE that names must be unique here. If there are two neuroscientists\n    # named Jack Black that have data in this database, this will create an\n    # incorrect linkage. NWB does not yet provide unique IDs for names.\n\n    # NOTE: PR requires table alter.\n    class LabMemberInfo(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Information about lab member in the context of Frank lab network\n        -&gt; LabMember\n        ---\n        google_user_name         : varchar(200) # For permission to curate\n        datajoint_user_name = \"\" : varchar(200) # For permission to delete\n        admin = 0                : bool         # Ignore permission checks\n        unique index (datajoint_user_name)\n        unique index (google_user_name)\n        \"\"\"\n\n        # NOTE: index present for new instances. pending surgery, not enforced\n        # for existing instances.\n\n    _admin = []\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert lab member information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf: pynwb.NWBFile\n            The NWB file with experimenter information.\n        \"\"\"\n        if isinstance(nwbf, str):\n            nwb_file_abspath = Nwbfile.get_abs_path(nwbf, new_file=True)\n            nwbf = get_nwb_file(nwb_file_abspath)\n\n        if nwbf.experimenter is None:\n            logger.info(\"No experimenter metadata found.\\n\")\n            return\n\n        for experimenter in nwbf.experimenter:\n            cls.insert_from_name(experimenter)\n\n            # each person is by default the member of their own LabTeam\n            # (same as their name)\n\n            full_name, _, _ = decompose_name(experimenter)\n            LabTeam.create_new_team(\n                team_name=full_name, team_members=[full_name]\n            )\n\n    @classmethod\n    def insert_from_name(cls, full_name):\n        \"\"\"Insert a lab member by name.\n\n        Parameters\n        ----------\n        full_name : str\n            The name to be added.\n        \"\"\"\n        _, first, last = decompose_name(full_name)\n        cls.insert1(\n            dict(\n                lab_member_name=f\"{first} {last}\",\n                first_name=first,\n                last_name=last,\n            ),\n            skip_duplicates=True,\n        )\n\n    def _load_admin(cls):\n        \"\"\"Load admin list.\"\"\"\n        cls._admin = list(\n            (cls.LabMemberInfo &amp; {\"admin\": True}).fetch(\"datajoint_user_name\")\n        )\n\n    @property\n    def admin(cls) -&gt; list:\n        \"\"\"Return the list of admin users.\n\n        Note: This is cached. If adding a new admin, run _load_admin\n        \"\"\"\n        if not cls._admin:\n            cls._load_admin()\n        return cls._admin\n\n    def get_djuser_name(cls, dj_user) -&gt; str:\n        \"\"\"Return the lab member name associated with a datajoint user name.\n\n        Parameters\n        ----------\n        dj_user: str\n            The datajoint user name.\n\n        Returns\n        -------\n        str\n            The lab member name.\n        \"\"\"\n        query = (cls.LabMemberInfo &amp; {\"datajoint_user_name\": dj_user}).fetch(\n            \"lab_member_name\"\n        )\n\n        if len(query) != 1:\n            raise ValueError(\n                f\"Could not find name for datajoint user {dj_user}\"\n                + f\" in common.LabMember.LabMemberInfo: {query}\"\n            )\n\n        return query[0]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert lab member information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <p>The NWB file with experimenter information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert lab member information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf: pynwb.NWBFile\n        The NWB file with experimenter information.\n    \"\"\"\n    if isinstance(nwbf, str):\n        nwb_file_abspath = Nwbfile.get_abs_path(nwbf, new_file=True)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n    if nwbf.experimenter is None:\n        logger.info(\"No experimenter metadata found.\\n\")\n        return\n\n    for experimenter in nwbf.experimenter:\n        cls.insert_from_name(experimenter)\n\n        # each person is by default the member of their own LabTeam\n        # (same as their name)\n\n        full_name, _, _ = decompose_name(experimenter)\n        LabTeam.create_new_team(\n            team_name=full_name, team_members=[full_name]\n        )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.insert_from_name", "title": "<code>insert_from_name(full_name)</code>  <code>classmethod</code>", "text": "<p>Insert a lab member by name.</p> <p>Parameters:</p> Name Type Description Default <code>full_name</code> <code>str</code> <p>The name to be added.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_name(cls, full_name):\n    \"\"\"Insert a lab member by name.\n\n    Parameters\n    ----------\n    full_name : str\n        The name to be added.\n    \"\"\"\n    _, first, last = decompose_name(full_name)\n    cls.insert1(\n        dict(\n            lab_member_name=f\"{first} {last}\",\n            first_name=first,\n            last_name=last,\n        ),\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.admin", "title": "<code>admin: list</code>  <code>property</code>", "text": "<p>Return the list of admin users.</p> <p>Note: This is cached. If adding a new admin, run _load_admin</p>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.get_djuser_name", "title": "<code>get_djuser_name(dj_user)</code>", "text": "<p>Return the lab member name associated with a datajoint user name.</p> <p>Parameters:</p> Name Type Description Default <code>dj_user</code> <p>The datajoint user name.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The lab member name.</p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def get_djuser_name(cls, dj_user) -&gt; str:\n    \"\"\"Return the lab member name associated with a datajoint user name.\n\n    Parameters\n    ----------\n    dj_user: str\n        The datajoint user name.\n\n    Returns\n    -------\n    str\n        The lab member name.\n    \"\"\"\n    query = (cls.LabMemberInfo &amp; {\"datajoint_user_name\": dj_user}).fetch(\n        \"lab_member_name\"\n    )\n\n    if len(query) != 1:\n        raise ValueError(\n            f\"Could not find name for datajoint user {dj_user}\"\n            + f\" in common.LabMember.LabMemberInfo: {query}\"\n        )\n\n    return query[0]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam", "title": "<code>LabTeam</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass LabTeam(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    team_name: varchar(80)\n    ---\n    team_description = \"\": varchar(2000)\n    \"\"\"\n\n    class LabTeamMember(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LabTeam\n        -&gt; LabMember\n        \"\"\"\n\n    _shared_teams = {}\n\n    def get_team_members(cls, member, reload=False):\n        \"\"\"Return the set of lab members that share a team with member.\n\n        Parameters\n        ----------\n        member : str\n            The lab member to check.\n        reload : bool\n            If True, reload the shared teams cache.\n        \"\"\"\n        if reload or member not in cls._shared_teams:\n            teams = (cls.LabTeamMember &amp; {\"lab_member_name\": member}).fetch(\n                \"team_name\"\n            )\n            cls._shared_teams[member] = set(\n                (\n                    LabTeam.LabTeamMember\n                    &amp; [{\"team_name\": team} for team in teams]\n                ).fetch(\"lab_member_name\")\n            )\n        return cls._shared_teams[member]\n\n    @classmethod\n    def create_new_team(\n        cls, team_name: str, team_members: list, team_description: str = \"\"\n    ):\n        \"\"\"Create a new team with a list of team members.\n\n        If the lab member does not exist in the database, they will be added.\n\n        Parameters\n        ----------\n        team_name : str\n            The name of the team.\n        team_members : str\n            The full names of the lab members that are part of the team.\n        team_description: str\n            The description of the team.\n        \"\"\"\n        labteam_dict = {\n            \"team_name\": team_name,\n            \"team_description\": team_description,\n        }\n\n        member_list = []\n        for team_member in team_members:\n            LabMember.insert_from_name(team_member)\n            query = (\n                LabMember.LabMemberInfo() &amp; {\"lab_member_name\": team_member}\n            ).fetch(\"google_user_name\")\n            if not query:\n                logger.info(\n                    f\"Please add the Google user ID for {team_member} in \"\n                    + \"LabMember.LabMemberInfo to help manage permissions.\"\n                )\n            labteammember_dict = {\n                \"team_name\": team_name,\n                \"lab_member_name\": team_member,\n            }\n            member_list.append(labteammember_dict)\n            # clear cache for this member\n            _ = cls._shared_teams.pop(team_member, None)\n\n        cls.insert1(labteam_dict, skip_duplicates=True)\n        cls.LabTeamMember.insert(member_list, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam.get_team_members", "title": "<code>get_team_members(member, reload=False)</code>", "text": "<p>Return the set of lab members that share a team with member.</p> <p>Parameters:</p> Name Type Description Default <code>member</code> <code>str</code> <p>The lab member to check.</p> required <code>reload</code> <code>bool</code> <p>If True, reload the shared teams cache.</p> <code>False</code> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def get_team_members(cls, member, reload=False):\n    \"\"\"Return the set of lab members that share a team with member.\n\n    Parameters\n    ----------\n    member : str\n        The lab member to check.\n    reload : bool\n        If True, reload the shared teams cache.\n    \"\"\"\n    if reload or member not in cls._shared_teams:\n        teams = (cls.LabTeamMember &amp; {\"lab_member_name\": member}).fetch(\n            \"team_name\"\n        )\n        cls._shared_teams[member] = set(\n            (\n                LabTeam.LabTeamMember\n                &amp; [{\"team_name\": team} for team in teams]\n            ).fetch(\"lab_member_name\")\n        )\n    return cls._shared_teams[member]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam.create_new_team", "title": "<code>create_new_team(team_name, team_members, team_description='')</code>  <code>classmethod</code>", "text": "<p>Create a new team with a list of team members.</p> <p>If the lab member does not exist in the database, they will be added.</p> <p>Parameters:</p> Name Type Description Default <code>team_name</code> <code>str</code> <p>The name of the team.</p> required <code>team_members</code> <code>str</code> <p>The full names of the lab members that are part of the team.</p> required <code>team_description</code> <code>str</code> <p>The description of the team.</p> <code>''</code> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef create_new_team(\n    cls, team_name: str, team_members: list, team_description: str = \"\"\n):\n    \"\"\"Create a new team with a list of team members.\n\n    If the lab member does not exist in the database, they will be added.\n\n    Parameters\n    ----------\n    team_name : str\n        The name of the team.\n    team_members : str\n        The full names of the lab members that are part of the team.\n    team_description: str\n        The description of the team.\n    \"\"\"\n    labteam_dict = {\n        \"team_name\": team_name,\n        \"team_description\": team_description,\n    }\n\n    member_list = []\n    for team_member in team_members:\n        LabMember.insert_from_name(team_member)\n        query = (\n            LabMember.LabMemberInfo() &amp; {\"lab_member_name\": team_member}\n        ).fetch(\"google_user_name\")\n        if not query:\n            logger.info(\n                f\"Please add the Google user ID for {team_member} in \"\n                + \"LabMember.LabMemberInfo to help manage permissions.\"\n            )\n        labteammember_dict = {\n            \"team_name\": team_name,\n            \"lab_member_name\": team_member,\n        }\n        member_list.append(labteammember_dict)\n        # clear cache for this member\n        _ = cls._shared_teams.pop(team_member, None)\n\n    cls.insert1(labteam_dict, skip_duplicates=True)\n    cls.LabTeamMember.insert(member_list, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Institution", "title": "<code>Institution</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass Institution(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    institution_name: varchar(80)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert institution information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The NWB file with institution information.\n        \"\"\"\n        if nwbf.institution is None:\n            logger.info(\"No institution metadata found.\\n\")\n            return\n\n        cls.insert1(\n            dict(institution_name=nwbf.institution), skip_duplicates=True\n        )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Institution.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert institution information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The NWB file with institution information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert institution information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The NWB file with institution information.\n    \"\"\"\n    if nwbf.institution is None:\n        logger.info(\"No institution metadata found.\\n\")\n        return\n\n    cls.insert1(\n        dict(institution_name=nwbf.institution), skip_duplicates=True\n    )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Lab", "title": "<code>Lab</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass Lab(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    lab_name: varchar(80)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert lab name information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The NWB file with lab name information.\n        \"\"\"\n        if nwbf.lab is None:\n            logger.info(\"No lab metadata found.\\n\")\n            return\n        cls.insert1(dict(lab_name=nwbf.lab), skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Lab.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert lab name information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The NWB file with lab name information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert lab name information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The NWB file with lab name information.\n    \"\"\"\n    if nwbf.lab is None:\n        logger.info(\"No lab metadata found.\\n\")\n        return\n    cls.insert1(dict(lab_name=nwbf.lab), skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.decompose_name", "title": "<code>decompose_name(full_name)</code>", "text": "<p>Return the full, first and last name parts of a full name.</p> <p>Names capitalized. Decomposes either comma- or space-separated. If name includes spaces, must use exactly one comma.</p> <p>Parameters:</p> Name Type Description Default <code>full_name</code> <code>str</code> <p>The full name to decompose.</p> required <p>Returns:</p> Type Description <code>tuple</code> <p>A tuple of the full, first, last name parts.</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>When full_name is in an unsupported format</p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def decompose_name(full_name: str) -&gt; tuple:\n    \"\"\"Return the full, first and last name parts of a full name.\n\n    Names capitalized. Decomposes either comma- or space-separated.\n    If name includes spaces, must use exactly one comma.\n\n    Parameters\n    ----------\n    full_name: str\n        The full name to decompose.\n\n    Returns\n    -------\n    tuple\n        A tuple of the full, first, last name parts.\n\n    Raises\n    ------\n    ValueError\n        When full_name is in an unsupported format\n    \"\"\"\n\n    full_trimmed = \"\" if not full_name else full_name.strip()\n\n    delim = (\n        \", \"\n        if full_trimmed.count(\", \") == 1\n        else \" \" if full_trimmed.count(\" \") == 1 else None\n    )\n\n    parts = full_trimmed.split(delim)\n\n    if delim is None or len(parts) != 2:  # catch unsupported format\n        raise ValueError(\n            f\"Name has unsupported format for {full_name}. \\n\\t\"\n            + \"Must use exactly one comma+space (i.e., ', ') or space.\\n\\t\"\n            + \"And provide exactly two name parts.\\n\\t\"\n            + \"For example, 'First Last' or 'Last1 Last2, First1 First2' \"\n            + \"if name(s) includes spaces.\"\n        )\n\n    first, last = parts if delim == \" \" else parts[::-1]\n    full = f\"{first} {last}\"\n\n    return full, first, last\n</code></pre>"}, {"location": "api/common/common_nwbfile/", "title": "common_nwbfile.py", "text": ""}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile", "title": "<code>Nwbfile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@schema\nclass Nwbfile(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Table for holding the NWB files.\n    nwb_file_name: varchar(64)   # name of the NWB file\n    ---\n    nwb_file_abs_path: filepath@raw\n    INDEX (nwb_file_abs_path)\n    \"\"\"\n    # NOTE the INDEX above is implicit from filepath@... above but needs to be explicit\n    # so that alter() can work\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    @classmethod\n    def insert_from_relative_file_name(cls, nwb_file_name):\n        \"\"\"Insert a new session from an existing NWB file.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The relative path to the NWB file.\n        \"\"\"\n        nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n        assert os.path.exists(\n            nwb_file_abs_path\n        ), f\"File does not exist: {nwb_file_abs_path}\"\n\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"nwb_file_abs_path\"] = nwb_file_abs_path\n        cls.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def _get_file_name(cls, nwb_file_name: str) -&gt; str:\n        \"\"\"Get valid nwb file name given substring.\"\"\"\n        query = cls &amp; f'nwb_file_name LIKE \"%{nwb_file_name}%\"'\n\n        if len(query) == 1:\n            return query.fetch1(\"nwb_file_name\")\n\n        raise ValueError(\n            f\"Found {len(query)} matches for {nwb_file_name} in Nwbfile table:\"\n            + f\" \\n{query}\"\n        )\n\n    @classmethod\n    def get_file_key(cls, nwb_file_name: str) -&gt; dict:\n        \"\"\"Return primary key using nwb_file_name substring.\"\"\"\n        return {\"nwb_file_name\": cls._get_file_name(nwb_file_name)}\n\n    @classmethod\n    def get_abs_path(cls, nwb_file_name, new_file=False) -&gt; str:\n        \"\"\"Return absolute path for a stored raw NWB file given file name.\n\n        The SPYGLASS_BASE_DIR must be set, either as an environment or part of\n        dj.config['custom']. See spyglass.settings.load_config\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file that has been inserted into the Nwbfile()\n            table. May be file substring. May include % wildcard(s).\n        new_file : bool, optional\n            Adding a new file to Nwbfile table. Defaults to False.\n\n        Returns\n        -------\n        nwb_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        if new_file:\n            return raw_dir + \"/\" + nwb_file_name\n\n        return raw_dir + \"/\" + cls._get_file_name(nwb_file_name)\n\n    @staticmethod\n    def add_to_lock(nwb_file_name):\n        \"\"\"Add the specified NWB file to the file with the list of NWB files to be locked.\n\n        The NWB_LOCK_FILE environment variable must be set.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file that has been inserted into the Nwbfile() schema.\n        \"\"\"\n        key = {\"nwb_file_name\": nwb_file_name}\n        # check to make sure the file exists\n        assert (\n            len((Nwbfile() &amp; key).fetch()) &gt; 0\n        ), f\"Error adding {nwb_file_name} to lock file, not in Nwbfile() schema\"\n\n        lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"a+\")\n        lock_file.write(f\"{nwb_file_name}\\n\")\n        lock_file.close()\n\n    @staticmethod\n    def cleanup(delete_files=False):\n        \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n        This does not delete the files themselves unless delete_files=True is specified\n        Run this after deleting the Nwbfile() entries themselves.\n        \"\"\"\n        schema.external[\"raw\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.insert_from_relative_file_name", "title": "<code>insert_from_relative_file_name(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Insert a new session from an existing NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The relative path to the NWB file.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef insert_from_relative_file_name(cls, nwb_file_name):\n    \"\"\"Insert a new session from an existing NWB file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The relative path to the NWB file.\n    \"\"\"\n    nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n    assert os.path.exists(\n        nwb_file_abs_path\n    ), f\"File does not exist: {nwb_file_abs_path}\"\n\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"nwb_file_abs_path\"] = nwb_file_abs_path\n    cls.insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.get_file_key", "title": "<code>get_file_key(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Return primary key using nwb_file_name substring.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_file_key(cls, nwb_file_name: str) -&gt; dict:\n    \"\"\"Return primary key using nwb_file_name substring.\"\"\"\n    return {\"nwb_file_name\": cls._get_file_name(nwb_file_name)}\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.get_abs_path", "title": "<code>get_abs_path(nwb_file_name, new_file=False)</code>  <code>classmethod</code>", "text": "<p>Return absolute path for a stored raw NWB file given file name.</p> <p>The SPYGLASS_BASE_DIR must be set, either as an environment or part of dj.config['custom']. See spyglass.settings.load_config</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file that has been inserted into the Nwbfile() table. May be file substring. May include % wildcard(s).</p> required <code>new_file</code> <code>bool</code> <p>Adding a new file to Nwbfile table. Defaults to False.</p> <code>False</code> <p>Returns:</p> Name Type Description <code>nwb_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_abs_path(cls, nwb_file_name, new_file=False) -&gt; str:\n    \"\"\"Return absolute path for a stored raw NWB file given file name.\n\n    The SPYGLASS_BASE_DIR must be set, either as an environment or part of\n    dj.config['custom']. See spyglass.settings.load_config\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file that has been inserted into the Nwbfile()\n        table. May be file substring. May include % wildcard(s).\n    new_file : bool, optional\n        Adding a new file to Nwbfile table. Defaults to False.\n\n    Returns\n    -------\n    nwb_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    if new_file:\n        return raw_dir + \"/\" + nwb_file_name\n\n    return raw_dir + \"/\" + cls._get_file_name(nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.add_to_lock", "title": "<code>add_to_lock(nwb_file_name)</code>  <code>staticmethod</code>", "text": "<p>Add the specified NWB file to the file with the list of NWB files to be locked.</p> <p>The NWB_LOCK_FILE environment variable must be set.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file that has been inserted into the Nwbfile() schema.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef add_to_lock(nwb_file_name):\n    \"\"\"Add the specified NWB file to the file with the list of NWB files to be locked.\n\n    The NWB_LOCK_FILE environment variable must be set.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file that has been inserted into the Nwbfile() schema.\n    \"\"\"\n    key = {\"nwb_file_name\": nwb_file_name}\n    # check to make sure the file exists\n    assert (\n        len((Nwbfile() &amp; key).fetch()) &gt; 0\n    ), f\"Error adding {nwb_file_name} to lock file, not in Nwbfile() schema\"\n\n    lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"a+\")\n    lock_file.write(f\"{nwb_file_name}\\n\")\n    lock_file.close()\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.cleanup", "title": "<code>cleanup(delete_files=False)</code>  <code>staticmethod</code>", "text": "<p>Remove the filepath entries for NWB files that are not in use.</p> <p>This does not delete the files themselves unless delete_files=True is specified Run this after deleting the Nwbfile() entries themselves.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef cleanup(delete_files=False):\n    \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n    This does not delete the files themselves unless delete_files=True is specified\n    Run this after deleting the Nwbfile() entries themselves.\n    \"\"\"\n    schema.external[\"raw\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile", "title": "<code>AnalysisNwbfile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@schema\nclass AnalysisNwbfile(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Table for holding the NWB files that contain results of analysis, such as spike sorting.\n    analysis_file_name: varchar(64)               # name of the file\n    ---\n    -&gt; Nwbfile                                     # name of the parent NWB file. Used for naming and metadata copy\n    analysis_file_abs_path: filepath@analysis      # the full path to the file\n    analysis_file_description = \"\": varchar(2000)  # an optional description of this analysis\n    analysis_parameters = NULL: blob               # additional relevant parameters. Currently used only for analyses\n                                                   # that span multiple NWB files\n    INDEX (analysis_file_abs_path)\n    \"\"\"\n    # NOTE the INDEX above is implicit from filepath@... above but needs to be explicit\n    # so that alter() can work\n\n    # See #630, #664. Excessive key length.\n\n    def create(self, nwb_file_name):\n        \"\"\"Open the NWB file, create a copy, write the copy to disk and return the name of the new file.\n\n        Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file to be copied.\n\n        Returns\n        -------\n        analysis_file_name : str\n            The name of the new NWB file.\n        \"\"\"\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        with pynwb.NWBHDF5IO(\n            path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # pop off the unnecessary elements to save space\n            nwb_fields = nwbf.fields\n            for field in nwb_fields:\n                if field not in NWB_KEEP_FIELDS:\n                    nwb_object = getattr(nwbf, field)\n                    if isinstance(nwb_object, pynwb.core.LabelledDict):\n                        for module in list(nwb_object.keys()):\n                            nwb_object.pop(module)\n\n            analysis_file_name = self.__get_new_file_name(nwb_file_name)\n            # write the new file\n            logger.info(f\"Writing new NWB file {analysis_file_name}\")\n            analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n                analysis_file_name\n            )\n            # export the new NWB file\n            with pynwb.NWBHDF5IO(\n                path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n            ) as export_io:\n                export_io.export(io, nwbf)\n\n        # change the permissions to only allow owner to write\n        permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n        os.chmod(analysis_file_abs_path, permissions)\n\n        return analysis_file_name\n\n    @classmethod\n    def __get_new_file_name(cls, nwb_file_name):\n        # each file ends with a random string of 10 digits, so we generate that\n        # string and redo if by some miracle it's already there\n        file_in_table = True\n        while file_in_table:\n            analysis_file_name = (\n                os.path.splitext(nwb_file_name)[0]\n                + \"\".join(\n                    random.choices(string.ascii_uppercase + string.digits, k=10)\n                )\n                + \".nwb\"\n            )\n            file_in_table = AnalysisNwbfile &amp; {\n                \"analysis_file_name\": analysis_file_name\n            }\n\n        return analysis_file_name\n\n    @classmethod\n    def __get_analysis_file_dir(cls, analysis_file_name: str):\n        # strip off everything after and including the final underscore and return the result\n        return analysis_file_name[0 : analysis_file_name.rfind(\"_\")]\n\n    @classmethod\n    def copy(cls, nwb_file_name):\n        \"\"\"Make a copy of an analysis NWB file.\n\n        Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the analysis NWB file to be copied.\n\n        Returns\n        -------\n        analysis_file_name : str\n            The name of the new NWB file.\n        \"\"\"\n        nwb_file_abspath = AnalysisNwbfile.get_abs_path(nwb_file_name)\n        with pynwb.NWBHDF5IO(\n            path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # get the current number of analysis files related to this nwb file\n            query = AnalysisNwbfile &amp; {\"analysis_file_name\": nwb_file_name}\n            original_nwb_file_name = query.fetch(\"nwb_file_name\")[0]\n            analysis_file_name = cls.__get_new_file_name(original_nwb_file_name)\n            # write the new file\n            logger.info(f\"Writing new NWB file {analysis_file_name}...\")\n            analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n                analysis_file_name\n            )\n            # export the new NWB file\n            with pynwb.NWBHDF5IO(\n                path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n            ) as export_io:\n                export_io.export(io, nwbf)\n\n        return analysis_file_name\n\n    def add(self, nwb_file_name, analysis_file_name):\n        \"\"\"Add the specified file to AnalysisNWBfile table.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the parent NWB file.\n        analysis_file_name : str\n            The name of the analysis NWB file that was created.\n        \"\"\"\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"analysis_file_name\"] = analysis_file_name\n        key[\"analysis_file_description\"] = \"\"\n        key[\"analysis_file_abs_path\"] = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        self.insert1(key)\n\n    @staticmethod\n    def get_abs_path(analysis_nwb_file_name):\n        \"\"\"Return the absolute path for a stored analysis NWB file given just the file name.\n\n        The spyglass config from settings.py must be set.\n\n        Parameters\n        ----------\n        analysis_nwb_file_name : str\n            The name of the NWB file that has been inserted into the AnalysisNwbfile() schema\n\n        Returns\n        -------\n        analysis_nwb_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        # see if the file exists and is stored in the base analysis dir\n        test_path = f\"{analysis_dir}/{analysis_nwb_file_name}\"\n\n        if os.path.exists(test_path):\n            return test_path\n        else:\n            # use the new path\n            analysis_file_base_path = Path(\n                analysis_dir\n            ) / AnalysisNwbfile.__get_analysis_file_dir(analysis_nwb_file_name)\n            if not analysis_file_base_path.exists():\n                os.mkdir(str(analysis_file_base_path))\n            return str(analysis_file_base_path / analysis_nwb_file_name)\n\n    def add_nwb_object(\n        self, analysis_file_name, nwb_object, table_name=\"pandas_table\"\n    ):\n        # TODO: change to add_object with checks for object type and a name parameter, which should be specified if\n        # it is not an NWB container\n        \"\"\"Add an NWB object to the analysis file in the scratch area and returns the NWB object ID\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        nwb_object : pynwb.core.NWBDataInterface\n            The NWB object created by PyNWB.\n        table_name : str (optional, defaults to 'pandas_table')\n            The name of the DynamicTable made from a dataframe.\n\n        Returns\n        -------\n        nwb_object_id : str\n            The NWB object ID of the added object.\n        \"\"\"\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            if isinstance(nwb_object, pd.DataFrame):\n                dt_object = DynamicTable.from_dataframe(\n                    name=table_name, df=nwb_object\n                )\n                nwbf.add_scratch(dt_object)\n                io.write(nwbf)\n                return dt_object.object_id\n            else:\n                nwbf.add_scratch(nwb_object)\n                io.write(nwbf)\n                return nwb_object.object_id\n\n    def add_units(\n        self,\n        analysis_file_name,\n        units,\n        units_valid_times,\n        units_sort_interval,\n        metrics=None,\n        units_waveforms=None,\n        labels=None,\n    ):\n        \"\"\"Add units to analysis NWB file\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        units : dict\n            keys are unit ids, values are spike times\n        units_valid_times : dict\n            Dictionary of units and valid times with unit ids as keys.\n        units_sort_interval : dict\n            Dictionary of units and sort_interval with unit ids as keys.\n        units_waveforms : dict, optional\n            Dictionary of unit waveforms with unit ids as keys.\n        metrics : dict, optional\n            Cluster metrics.\n        labels : dict, optional\n            Curation labels for clusters\n\n        Returns\n        -------\n        units_object_id, waveforms_object_id : str, str\n            The NWB object id of the Units object and the object id of the waveforms object ('' if None)\n        \"\"\"\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            sort_intervals = list()\n            if len(units.keys()):\n                # Add spike times and valid time range for the sort\n                for id in units.keys():\n                    nwbf.add_unit(\n                        spike_times=units[id],\n                        id=id,\n                        # waveform_mean = units_templates[id],\n                        obs_intervals=units_valid_times[id],\n                    )\n                    sort_intervals.append(units_sort_interval[id])\n                # Add a column for the sort interval (subset of valid time)\n                nwbf.add_unit_column(\n                    name=\"sort_interval\",\n                    description=\"the interval used for spike sorting\",\n                    data=sort_intervals,\n                )\n                # If metrics were specified, add one column per metric\n                if metrics is not None:\n                    for metric in metrics:\n                        if metrics[metric]:\n                            unit_ids = np.array(list(metrics[metric].keys()))\n                            metric_values = np.array(\n                                list(metrics[metric].values())\n                            )\n                            # sort by unit_ids and apply that sorting to values to ensure that things go in the right order\n                            metric_values = metric_values[np.argsort(unit_ids)]\n                            logger.info(\n                                f\"Adding metric {metric} : {metric_values}\"\n                            )\n                            nwbf.add_unit_column(\n                                name=metric,\n                                description=f\"{metric} metric\",\n                                data=metric_values,\n                            )\n                if labels is not None:\n                    unit_ids = np.array(list(units.keys()))\n                    for unit in unit_ids:\n                        if unit not in labels:\n                            labels[unit] = \"\"\n                    label_values = np.array(list(labels.values()))\n                    label_values = label_values[np.argsort(unit_ids)].tolist()\n                    nwbf.add_unit_column(\n                        name=\"label\",\n                        description=\"label given during curation\",\n                        data=label_values,\n                    )\n                # If the waveforms were specified, add them as a dataframe to scratch\n                waveforms_object_id = \"\"\n                if units_waveforms is not None:\n                    waveforms_df = pd.DataFrame.from_dict(\n                        units_waveforms, orient=\"index\"\n                    )\n                    waveforms_df.columns = [\"waveforms\"]\n                    nwbf.add_scratch(\n                        waveforms_df,\n                        name=\"units_waveforms\",\n                        notes=\"spike waveforms for each unit\",\n                    )\n                    waveforms_object_id = nwbf.scratch[\n                        \"units_waveforms\"\n                    ].object_id\n\n                io.write(nwbf)\n                return nwbf.units.object_id, waveforms_object_id\n            else:\n                return \"\"\n\n    def add_units_waveforms(\n        self,\n        analysis_file_name,\n        waveform_extractor: si.WaveformExtractor,\n        metrics=None,\n        labels=None,\n    ):\n        \"\"\"Add units to analysis NWB file along with the waveforms\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        waveform_extractor : si.WaveformExtractor object\n        metrics : dict, optional\n            Cluster metrics.\n        labels : dict, optional\n            Curation labels for clusters\n\n        Returns\n        -------\n        units_object_id : str\n            The NWB object id of the Units object\n        \"\"\"\n\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            for id in waveform_extractor.sorting.get_unit_ids():\n                # (spikes, samples, channels)\n                waveforms = waveform_extractor.get_waveforms(unit_id=id)\n                # (channels, spikes, samples)\n                waveforms = np.moveaxis(waveforms, source=2, destination=0)\n                nwbf.add_unit(\n                    spike_times=waveform_extractor.sorting.get_unit_spike_train(\n                        unit_id=id\n                    ),\n                    id=id,\n                    electrodes=waveform_extractor.recording.get_channel_ids(),\n                    waveforms=waveforms,\n                )\n\n            # The following is a rough sketch of AnalysisNwbfile().add_waveforms\n            # analysis_file_name = AnalysisNwbfile().create(key['nwb_file_name'])\n            # or\n            # nwbfile = pynwb.NWBFile(...)\n            # (channels, spikes, samples)\n            # wfs = [\n            #         [     # elec 1\n            #             [1, 2, 3],  # spike 1, [sample 1, sample 2, sample 3]\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ], [  # elec 2\n            #             [1, 2, 3],  # spike 1\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ], [  # elec 3\n            #             [1, 2, 3],  # spike 1\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ]\n            # ]\n            # elecs = ... # DynamicTableRegion referring to three electrodes (rows) of the electrodes table\n            # nwbfile.add_unit(spike_times=[1, 2, 3], electrodes=elecs, waveforms=wfs)\n\n            # If metrics were specified, add one column per metric\n            if metrics is not None:\n                for metric_name, metric_dict in metrics.items():\n                    logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                    metric_data = metric_dict.values().to_list()\n                    nwbf.add_unit_column(\n                        name=metric_name,\n                        description=metric_name,\n                        data=metric_data,\n                    )\n            if labels is not None:\n                nwbf.add_unit_column(\n                    name=\"label\",\n                    description=\"label given during curation\",\n                    data=labels,\n                )\n\n            io.write(nwbf)\n            return nwbf.units.object_id\n\n    def add_units_metrics(self, analysis_file_name, metrics):\n        \"\"\"Add units to analysis NWB file along with the waveforms\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        metrics : dict, optional\n            Cluster metrics.\n\n        Returns\n        -------\n        units_object_id : str\n            The NWB object id of the Units object\n        \"\"\"\n        metric_names = list(metrics.keys())\n        unit_ids = list(metrics[metric_names[0]].keys())\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            for id in unit_ids:\n                nwbf.add_unit(id=id)\n\n            for metric_name, metric_dict in metrics.items():\n                logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                metric_data = list(metric_dict.values())\n                nwbf.add_unit_column(\n                    name=metric_name, description=metric_name, data=metric_data\n                )\n\n            io.write(nwbf)\n            return nwbf.units.object_id\n\n    @classmethod\n    def get_electrode_indices(cls, analysis_file_name, electrode_ids):\n        \"\"\"Given an analysis NWB file name, returns the indices of the specified electrode_ids.\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        electrode_ids : numpy array or list\n            Array or list of electrode IDs.\n\n        Returns\n        -------\n        electrode_indices : numpy array\n            Array of indices in the electrodes table for the given electrode IDs.\n        \"\"\"\n        nwbf = get_nwb_file(cls.get_abs_path(analysis_file_name))\n        return get_electrode_indices(nwbf.electrodes, electrode_ids)\n\n    @staticmethod\n    def cleanup(delete_files=False):\n        \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n        Does not delete the files themselves unless delete_files=True is specified.\n        Run this after deleting the Nwbfile() entries themselves.\n\n        Parameters\n        ----------\n        delete_files : bool, optional\n            Whether the original files should be deleted (default False).\n        \"\"\"\n        schema.external[\"analysis\"].delete(delete_external_files=delete_files)\n\n    @staticmethod\n    def nightly_cleanup():\n        child_tables = get_child_tables(AnalysisNwbfile)\n        (AnalysisNwbfile - child_tables).delete_quick()\n\n        # a separate external files clean up required - this is to be done\n        # during times when no other transactions are in progress.\n        AnalysisNwbfile.cleanup(True)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.create", "title": "<code>create(nwb_file_name)</code>", "text": "<p>Open the NWB file, create a copy, write the copy to disk and return the name of the new file.</p> <p>Note that this does NOT add the file to the schema; that needs to be done after data are written to it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file to be copied.</p> required <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <p>The name of the new NWB file.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def create(self, nwb_file_name):\n    \"\"\"Open the NWB file, create a copy, write the copy to disk and return the name of the new file.\n\n    Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file to be copied.\n\n    Returns\n    -------\n    analysis_file_name : str\n        The name of the new NWB file.\n    \"\"\"\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        # pop off the unnecessary elements to save space\n        nwb_fields = nwbf.fields\n        for field in nwb_fields:\n            if field not in NWB_KEEP_FIELDS:\n                nwb_object = getattr(nwbf, field)\n                if isinstance(nwb_object, pynwb.core.LabelledDict):\n                    for module in list(nwb_object.keys()):\n                        nwb_object.pop(module)\n\n        analysis_file_name = self.__get_new_file_name(nwb_file_name)\n        # write the new file\n        logger.info(f\"Writing new NWB file {analysis_file_name}\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n        ) as export_io:\n            export_io.export(io, nwbf)\n\n    # change the permissions to only allow owner to write\n    permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n    os.chmod(analysis_file_abs_path, permissions)\n\n    return analysis_file_name\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.copy", "title": "<code>copy(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Make a copy of an analysis NWB file.</p> <p>Note that this does NOT add the file to the schema; that needs to be done after data are written to it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the analysis NWB file to be copied.</p> required <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <p>The name of the new NWB file.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef copy(cls, nwb_file_name):\n    \"\"\"Make a copy of an analysis NWB file.\n\n    Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the analysis NWB file to be copied.\n\n    Returns\n    -------\n    analysis_file_name : str\n        The name of the new NWB file.\n    \"\"\"\n    nwb_file_abspath = AnalysisNwbfile.get_abs_path(nwb_file_name)\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        # get the current number of analysis files related to this nwb file\n        query = AnalysisNwbfile &amp; {\"analysis_file_name\": nwb_file_name}\n        original_nwb_file_name = query.fetch(\"nwb_file_name\")[0]\n        analysis_file_name = cls.__get_new_file_name(original_nwb_file_name)\n        # write the new file\n        logger.info(f\"Writing new NWB file {analysis_file_name}...\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n        ) as export_io:\n            export_io.export(io, nwbf)\n\n    return analysis_file_name\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add", "title": "<code>add(nwb_file_name, analysis_file_name)</code>", "text": "<p>Add the specified file to AnalysisNWBfile table.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the parent NWB file.</p> required <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file that was created.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add(self, nwb_file_name, analysis_file_name):\n    \"\"\"Add the specified file to AnalysisNWBfile table.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the parent NWB file.\n    analysis_file_name : str\n        The name of the analysis NWB file that was created.\n    \"\"\"\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"analysis_file_name\"] = analysis_file_name\n    key[\"analysis_file_description\"] = \"\"\n    key[\"analysis_file_abs_path\"] = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    self.insert1(key)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.get_abs_path", "title": "<code>get_abs_path(analysis_nwb_file_name)</code>  <code>staticmethod</code>", "text": "<p>Return the absolute path for a stored analysis NWB file given just the file name.</p> <p>The spyglass config from settings.py must be set.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_nwb_file_name</code> <code>str</code> <p>The name of the NWB file that has been inserted into the AnalysisNwbfile() schema</p> required <p>Returns:</p> Name Type Description <code>analysis_nwb_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef get_abs_path(analysis_nwb_file_name):\n    \"\"\"Return the absolute path for a stored analysis NWB file given just the file name.\n\n    The spyglass config from settings.py must be set.\n\n    Parameters\n    ----------\n    analysis_nwb_file_name : str\n        The name of the NWB file that has been inserted into the AnalysisNwbfile() schema\n\n    Returns\n    -------\n    analysis_nwb_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    # see if the file exists and is stored in the base analysis dir\n    test_path = f\"{analysis_dir}/{analysis_nwb_file_name}\"\n\n    if os.path.exists(test_path):\n        return test_path\n    else:\n        # use the new path\n        analysis_file_base_path = Path(\n            analysis_dir\n        ) / AnalysisNwbfile.__get_analysis_file_dir(analysis_nwb_file_name)\n        if not analysis_file_base_path.exists():\n            os.mkdir(str(analysis_file_base_path))\n        return str(analysis_file_base_path / analysis_nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_nwb_object", "title": "<code>add_nwb_object(analysis_file_name, nwb_object, table_name='pandas_table')</code>", "text": "<p>Add an NWB object to the analysis file in the scratch area and returns the NWB object ID</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>nwb_object</code> <code>NWBDataInterface</code> <p>The NWB object created by PyNWB.</p> required <code>table_name</code> <code>str (optional, defaults to 'pandas_table')</code> <p>The name of the DynamicTable made from a dataframe.</p> <code>'pandas_table'</code> <p>Returns:</p> Name Type Description <code>nwb_object_id</code> <code>str</code> <p>The NWB object ID of the added object.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_nwb_object(\n    self, analysis_file_name, nwb_object, table_name=\"pandas_table\"\n):\n    # TODO: change to add_object with checks for object type and a name parameter, which should be specified if\n    # it is not an NWB container\n    \"\"\"Add an NWB object to the analysis file in the scratch area and returns the NWB object ID\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    nwb_object : pynwb.core.NWBDataInterface\n        The NWB object created by PyNWB.\n    table_name : str (optional, defaults to 'pandas_table')\n        The name of the DynamicTable made from a dataframe.\n\n    Returns\n    -------\n    nwb_object_id : str\n        The NWB object ID of the added object.\n    \"\"\"\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        if isinstance(nwb_object, pd.DataFrame):\n            dt_object = DynamicTable.from_dataframe(\n                name=table_name, df=nwb_object\n            )\n            nwbf.add_scratch(dt_object)\n            io.write(nwbf)\n            return dt_object.object_id\n        else:\n            nwbf.add_scratch(nwb_object)\n            io.write(nwbf)\n            return nwb_object.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units", "title": "<code>add_units(analysis_file_name, units, units_valid_times, units_sort_interval, metrics=None, units_waveforms=None, labels=None)</code>", "text": "<p>Add units to analysis NWB file</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>units</code> <code>dict</code> <p>keys are unit ids, values are spike times</p> required <code>units_valid_times</code> <code>dict</code> <p>Dictionary of units and valid times with unit ids as keys.</p> required <code>units_sort_interval</code> <code>dict</code> <p>Dictionary of units and sort_interval with unit ids as keys.</p> required <code>units_waveforms</code> <code>dict</code> <p>Dictionary of unit waveforms with unit ids as keys.</p> <code>None</code> <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> <code>None</code> <code>labels</code> <code>dict</code> <p>Curation labels for clusters</p> <code>None</code> <p>Returns:</p> Type Description <code>units_object_id, waveforms_object_id : str, str</code> <p>The NWB object id of the Units object and the object id of the waveforms object ('' if None)</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units(\n    self,\n    analysis_file_name,\n    units,\n    units_valid_times,\n    units_sort_interval,\n    metrics=None,\n    units_waveforms=None,\n    labels=None,\n):\n    \"\"\"Add units to analysis NWB file\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    units : dict\n        keys are unit ids, values are spike times\n    units_valid_times : dict\n        Dictionary of units and valid times with unit ids as keys.\n    units_sort_interval : dict\n        Dictionary of units and sort_interval with unit ids as keys.\n    units_waveforms : dict, optional\n        Dictionary of unit waveforms with unit ids as keys.\n    metrics : dict, optional\n        Cluster metrics.\n    labels : dict, optional\n        Curation labels for clusters\n\n    Returns\n    -------\n    units_object_id, waveforms_object_id : str, str\n        The NWB object id of the Units object and the object id of the waveforms object ('' if None)\n    \"\"\"\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        sort_intervals = list()\n        if len(units.keys()):\n            # Add spike times and valid time range for the sort\n            for id in units.keys():\n                nwbf.add_unit(\n                    spike_times=units[id],\n                    id=id,\n                    # waveform_mean = units_templates[id],\n                    obs_intervals=units_valid_times[id],\n                )\n                sort_intervals.append(units_sort_interval[id])\n            # Add a column for the sort interval (subset of valid time)\n            nwbf.add_unit_column(\n                name=\"sort_interval\",\n                description=\"the interval used for spike sorting\",\n                data=sort_intervals,\n            )\n            # If metrics were specified, add one column per metric\n            if metrics is not None:\n                for metric in metrics:\n                    if metrics[metric]:\n                        unit_ids = np.array(list(metrics[metric].keys()))\n                        metric_values = np.array(\n                            list(metrics[metric].values())\n                        )\n                        # sort by unit_ids and apply that sorting to values to ensure that things go in the right order\n                        metric_values = metric_values[np.argsort(unit_ids)]\n                        logger.info(\n                            f\"Adding metric {metric} : {metric_values}\"\n                        )\n                        nwbf.add_unit_column(\n                            name=metric,\n                            description=f\"{metric} metric\",\n                            data=metric_values,\n                        )\n            if labels is not None:\n                unit_ids = np.array(list(units.keys()))\n                for unit in unit_ids:\n                    if unit not in labels:\n                        labels[unit] = \"\"\n                label_values = np.array(list(labels.values()))\n                label_values = label_values[np.argsort(unit_ids)].tolist()\n                nwbf.add_unit_column(\n                    name=\"label\",\n                    description=\"label given during curation\",\n                    data=label_values,\n                )\n            # If the waveforms were specified, add them as a dataframe to scratch\n            waveforms_object_id = \"\"\n            if units_waveforms is not None:\n                waveforms_df = pd.DataFrame.from_dict(\n                    units_waveforms, orient=\"index\"\n                )\n                waveforms_df.columns = [\"waveforms\"]\n                nwbf.add_scratch(\n                    waveforms_df,\n                    name=\"units_waveforms\",\n                    notes=\"spike waveforms for each unit\",\n                )\n                waveforms_object_id = nwbf.scratch[\n                    \"units_waveforms\"\n                ].object_id\n\n            io.write(nwbf)\n            return nwbf.units.object_id, waveforms_object_id\n        else:\n            return \"\"\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units_waveforms", "title": "<code>add_units_waveforms(analysis_file_name, waveform_extractor, metrics=None, labels=None)</code>", "text": "<p>Add units to analysis NWB file along with the waveforms</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>waveform_extractor</code> <code>si.WaveformExtractor object</code> required <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> <code>None</code> <code>labels</code> <code>dict</code> <p>Curation labels for clusters</p> <code>None</code> <p>Returns:</p> Name Type Description <code>units_object_id</code> <code>str</code> <p>The NWB object id of the Units object</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units_waveforms(\n    self,\n    analysis_file_name,\n    waveform_extractor: si.WaveformExtractor,\n    metrics=None,\n    labels=None,\n):\n    \"\"\"Add units to analysis NWB file along with the waveforms\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    waveform_extractor : si.WaveformExtractor object\n    metrics : dict, optional\n        Cluster metrics.\n    labels : dict, optional\n        Curation labels for clusters\n\n    Returns\n    -------\n    units_object_id : str\n        The NWB object id of the Units object\n    \"\"\"\n\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        for id in waveform_extractor.sorting.get_unit_ids():\n            # (spikes, samples, channels)\n            waveforms = waveform_extractor.get_waveforms(unit_id=id)\n            # (channels, spikes, samples)\n            waveforms = np.moveaxis(waveforms, source=2, destination=0)\n            nwbf.add_unit(\n                spike_times=waveform_extractor.sorting.get_unit_spike_train(\n                    unit_id=id\n                ),\n                id=id,\n                electrodes=waveform_extractor.recording.get_channel_ids(),\n                waveforms=waveforms,\n            )\n\n        # The following is a rough sketch of AnalysisNwbfile().add_waveforms\n        # analysis_file_name = AnalysisNwbfile().create(key['nwb_file_name'])\n        # or\n        # nwbfile = pynwb.NWBFile(...)\n        # (channels, spikes, samples)\n        # wfs = [\n        #         [     # elec 1\n        #             [1, 2, 3],  # spike 1, [sample 1, sample 2, sample 3]\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ], [  # elec 2\n        #             [1, 2, 3],  # spike 1\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ], [  # elec 3\n        #             [1, 2, 3],  # spike 1\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ]\n        # ]\n        # elecs = ... # DynamicTableRegion referring to three electrodes (rows) of the electrodes table\n        # nwbfile.add_unit(spike_times=[1, 2, 3], electrodes=elecs, waveforms=wfs)\n\n        # If metrics were specified, add one column per metric\n        if metrics is not None:\n            for metric_name, metric_dict in metrics.items():\n                logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                metric_data = metric_dict.values().to_list()\n                nwbf.add_unit_column(\n                    name=metric_name,\n                    description=metric_name,\n                    data=metric_data,\n                )\n        if labels is not None:\n            nwbf.add_unit_column(\n                name=\"label\",\n                description=\"label given during curation\",\n                data=labels,\n            )\n\n        io.write(nwbf)\n        return nwbf.units.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units_metrics", "title": "<code>add_units_metrics(analysis_file_name, metrics)</code>", "text": "<p>Add units to analysis NWB file along with the waveforms</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> required <p>Returns:</p> Name Type Description <code>units_object_id</code> <code>str</code> <p>The NWB object id of the Units object</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units_metrics(self, analysis_file_name, metrics):\n    \"\"\"Add units to analysis NWB file along with the waveforms\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    metrics : dict, optional\n        Cluster metrics.\n\n    Returns\n    -------\n    units_object_id : str\n        The NWB object id of the Units object\n    \"\"\"\n    metric_names = list(metrics.keys())\n    unit_ids = list(metrics[metric_names[0]].keys())\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        for id in unit_ids:\n            nwbf.add_unit(id=id)\n\n        for metric_name, metric_dict in metrics.items():\n            logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n            metric_data = list(metric_dict.values())\n            nwbf.add_unit_column(\n                name=metric_name, description=metric_name, data=metric_data\n            )\n\n        io.write(nwbf)\n        return nwbf.units.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.get_electrode_indices", "title": "<code>get_electrode_indices(analysis_file_name, electrode_ids)</code>  <code>classmethod</code>", "text": "<p>Given an analysis NWB file name, returns the indices of the specified electrode_ids.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>electrode_ids</code> <code>numpy array or list</code> <p>Array or list of electrode IDs.</p> required <p>Returns:</p> Name Type Description <code>electrode_indices</code> <code>numpy array</code> <p>Array of indices in the electrodes table for the given electrode IDs.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_electrode_indices(cls, analysis_file_name, electrode_ids):\n    \"\"\"Given an analysis NWB file name, returns the indices of the specified electrode_ids.\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    electrode_ids : numpy array or list\n        Array or list of electrode IDs.\n\n    Returns\n    -------\n    electrode_indices : numpy array\n        Array of indices in the electrodes table for the given electrode IDs.\n    \"\"\"\n    nwbf = get_nwb_file(cls.get_abs_path(analysis_file_name))\n    return get_electrode_indices(nwbf.electrodes, electrode_ids)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.cleanup", "title": "<code>cleanup(delete_files=False)</code>  <code>staticmethod</code>", "text": "<p>Remove the filepath entries for NWB files that are not in use.</p> <p>Does not delete the files themselves unless delete_files=True is specified. Run this after deleting the Nwbfile() entries themselves.</p> <p>Parameters:</p> Name Type Description Default <code>delete_files</code> <code>bool</code> <p>Whether the original files should be deleted (default False).</p> <code>False</code> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef cleanup(delete_files=False):\n    \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n    Does not delete the files themselves unless delete_files=True is specified.\n    Run this after deleting the Nwbfile() entries themselves.\n\n    Parameters\n    ----------\n    delete_files : bool, optional\n        Whether the original files should be deleted (default False).\n    \"\"\"\n    schema.external[\"analysis\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_position/", "title": "common_position.py", "text": ""}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionInfoParameters", "title": "<code>PositionInfoParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Parameters for extracting the smoothed position, orientation and velocity.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass PositionInfoParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"\n    Parameters for extracting the smoothed position, orientation and velocity.\n    \"\"\"\n\n    definition = \"\"\"\n    position_info_param_name : varchar(32) # name for this set of parameters\n    ---\n    max_separation = 9.0  : float   # max distance (in cm) between head LEDs\n    max_speed = 300.0     : float   # max speed (in cm / s) of animal\n    position_smoothing_duration = 0.125 : float # size of moving window (s)\n    speed_smoothing_std_dev = 0.100 : float # smoothing standard deviation (s)\n    head_orient_smoothing_std_dev = 0.001 : float # smoothing std deviation (s)\n    led1_is_front = 1 : int # 1 if 1st LED is front LED, else 1st LED is back\n    is_upsampled = 0 : int # upsample the position to higher sampling rate\n    upsampling_sampling_rate = NULL : float # The rate to be upsampled to\n    upsampling_interpolation_method = linear : varchar(80) # see\n        # pandas.DataFrame.interpolation for list of methods\n    \"\"\"\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfoSelection", "title": "<code>IntervalPositionInfoSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Combines the parameters for position extraction and a time interval to extract the smoothed position on.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass IntervalPositionInfoSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Combines the parameters for position extraction and a time interval to\n    extract the smoothed position on.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionInfoParameters\n    -&gt; IntervalList\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfo", "title": "<code>IntervalPositionInfo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Computes the smoothed head position, orientation and velocity for a given interval.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass IntervalPositionInfo(SpyglassMixin, dj.Computed):\n    \"\"\"Computes the smoothed head position, orientation and velocity for a given\n    interval.\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalPositionInfoSelection\n    ---\n    -&gt; AnalysisNwbfile\n    head_position_object_id : varchar(40)\n    head_orientation_object_id : varchar(40)\n    head_velocity_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing position for: {key}\")\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        raw_position = RawPosition.PosObject &amp; key\n        spatial_series = raw_position.fetch_nwb()[0][\"raw_position\"]\n        spatial_df = raw_position.fetch1_dataframe()\n\n        position_info_parameters = (PositionInfoParameters() &amp; key).fetch1()\n\n        position_info = self.calculate_position_info(\n            spatial_df=spatial_df,\n            meters_to_pixels=spatial_series.conversion,\n            **position_info_parameters,\n        )\n\n        key.update(\n            dict(\n                analysis_file_name=analysis_file_name,\n                **self.generate_pos_components(\n                    spatial_series=spatial_series,\n                    position_info=position_info,\n                    analysis_fname=analysis_file_name,\n                ),\n            )\n        )\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        self.insert1(key)\n\n    @staticmethod\n    def generate_pos_components(\n        spatial_series,\n        position_info,\n        analysis_fname,\n        prefix=\"head_\",\n        add_frame_ind=False,\n        video_frame_ind=None,\n    ):\n        \"\"\"Generate position, orientation and velocity components.\"\"\"\n        METERS_PER_CM = 0.01\n\n        position = pynwb.behavior.Position()\n        orientation = pynwb.behavior.CompassDirection()\n        velocity = pynwb.behavior.BehavioralTimeSeries()\n\n        # NOTE: CBroz1 removed a try/except ValueError that surrounded all\n        #       .create_X_series methods. dpeg22 could not recall purpose\n\n        time_comments = dict(\n            comments=spatial_series.comments,\n            timestamps=position_info[\"time\"],\n        )\n        time_comments_ref = dict(\n            **time_comments,\n            reference_frame=spatial_series.reference_frame,\n        )\n\n        # create nwb objects for insertion into analysis nwb file\n        position.create_spatial_series(\n            name=f\"{prefix}position\",\n            conversion=METERS_PER_CM,\n            data=position_info[\"position\"],\n            description=f\"{prefix}x_position, {prefix}y_position\",\n            **time_comments_ref,\n        )\n\n        orientation.create_spatial_series(\n            name=f\"{prefix}orientation\",\n            conversion=1.0,\n            data=position_info[\"orientation\"],\n            description=f\"{prefix}orientation\",\n            **time_comments_ref,\n        )\n\n        velocity.create_timeseries(\n            name=f\"{prefix}velocity\",\n            conversion=METERS_PER_CM,\n            unit=\"m/s\",\n            data=np.concatenate(\n                (\n                    position_info[\"velocity\"],\n                    position_info[\"speed\"][:, np.newaxis],\n                ),\n                axis=1,\n            ),\n            description=f\"{prefix}x_velocity, {prefix}y_velocity, \"\n            + f\"{prefix}speed\",\n            **time_comments,\n        )\n\n        if add_frame_ind:\n            if video_frame_ind is not None:\n                velocity.create_timeseries(\n                    name=\"video_frame_ind\",\n                    unit=\"index\",\n                    data=video_frame_ind.to_numpy(),\n                    description=\"video_frame_ind\",\n                    **time_comments,\n                )\n            else:\n                logger.info(\n                    \"No video frame index found. Assuming all camera frames \"\n                    + \"are present.\"\n                )\n                velocity.create_timeseries(\n                    name=\"video_frame_ind\",\n                    unit=\"index\",\n                    data=np.arange(len(position_info[\"time\"])),\n                    description=\"video_frame_ind\",\n                    **time_comments,\n                )\n\n        # Insert into analysis nwb file\n        nwba = AnalysisNwbfile()\n\n        return {\n            f\"{prefix}position_object_id\": nwba.add_nwb_object(\n                analysis_fname, position\n            ),\n            f\"{prefix}orientation_object_id\": nwba.add_nwb_object(\n                analysis_fname, orientation\n            ),\n            f\"{prefix}velocity_object_id\": nwba.add_nwb_object(\n                analysis_fname, velocity\n            ),\n        }\n\n    @staticmethod\n    def _fix_kwargs(\n        orient_smoothing_std_dev=None,\n        speed_smoothing_std_dev=None,\n        max_LED_separation=None,\n        max_plausible_speed=None,\n        **kwargs,\n    ):\n        \"\"\"Handles discrepancies between common and v1 param names.\"\"\"\n        if not orient_smoothing_std_dev:\n            orient_smoothing_std_dev = kwargs.get(\n                \"head_orient_smoothing_std_dev\"\n            )\n        if not speed_smoothing_std_dev:\n            speed_smoothing_std_dev = kwargs.get(\"head_speed_smoothing_std_dev\")\n        if not max_LED_separation:\n            max_LED_separation = kwargs.get(\"max_separation\")\n        if not max_plausible_speed:\n            max_plausible_speed = kwargs.get(\"max_speed\")\n        if not all(\n            [speed_smoothing_std_dev, max_LED_separation, max_plausible_speed]\n        ):\n            raise ValueError(\n                \"Missing at least one required parameter:\\n\\t\"\n                + f\"speed_smoothing_std_dev: {speed_smoothing_std_dev}\\n\\t\"\n                + f\"max_LED_separation: {max_LED_separation}\\n\\t\"\n                + f\"max_plausible_speed: {max_plausible_speed}\"\n            )\n        return (\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n        )\n\n    @staticmethod\n    def _upsample(\n        front_LED,\n        back_LED,\n        time,\n        sampling_rate,\n        upsampling_sampling_rate,\n        upsampling_interpolation_method,\n        **kwargs,\n    ):\n        position_df = pd.DataFrame(\n            {\n                \"time\": time,\n                \"back_LED_x\": back_LED[:, 0],\n                \"back_LED_y\": back_LED[:, 1],\n                \"front_LED_x\": front_LED[:, 0],\n                \"front_LED_y\": front_LED[:, 1],\n            }\n        ).set_index(\"time\")\n\n        upsampling_start_time = time[0]\n        upsampling_end_time = time[-1]\n\n        n_samples = (\n            int(\n                np.ceil(\n                    (upsampling_end_time - upsampling_start_time)\n                    * upsampling_sampling_rate\n                )\n            )\n            + 1\n        )\n        new_time = np.linspace(\n            upsampling_start_time, upsampling_end_time, n_samples\n        )\n        new_index = pd.Index(\n            np.unique(np.concatenate((position_df.index, new_time))),\n            name=\"time\",\n        )\n        position_df = (\n            position_df.reindex(index=new_index)\n            .interpolate(method=upsampling_interpolation_method)\n            .reindex(index=new_time)\n        )\n\n        time = np.asarray(position_df.index)\n        back_LED = np.asarray(position_df.loc[:, [\"back_LED_x\", \"back_LED_y\"]])\n        front_LED = np.asarray(\n            position_df.loc[:, [\"front_LED_x\", \"front_LED_y\"]]\n        )\n\n        sampling_rate = upsampling_sampling_rate\n\n        return front_LED, back_LED, time, sampling_rate\n\n    def calculate_position_info(\n        self,\n        spatial_df: pd.DataFrame,\n        meters_to_pixels: float,\n        position_smoothing_duration,\n        led1_is_front,\n        is_upsampled,\n        upsampling_sampling_rate,\n        upsampling_interpolation_method,\n        orient_smoothing_std_dev=None,\n        speed_smoothing_std_dev=None,\n        max_LED_separation=None,\n        max_plausible_speed=None,\n        **kwargs,\n    ):\n        CM_TO_METERS = 100\n\n        (\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n        ) = self._fix_kwargs(\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n            **kwargs,\n        )\n\n        spatial_df = _fix_col_names(spatial_df)\n        # Get spatial series properties\n        time = np.asarray(spatial_df.index)  # seconds\n        position = np.asarray(spatial_df.iloc[:, :4])  # meters\n\n        # remove NaN times\n        is_nan_time = np.isnan(time)\n        position = position[~is_nan_time]\n        time = time[~is_nan_time]\n\n        dt = np.median(np.diff(time))\n        sampling_rate = 1 / dt\n\n        if position.shape[1] &lt; 4:\n            front_LED = position.astype(float)\n            back_LED = position.astype(float)\n        else:\n            # If there are 4 columns, then there are 2 LEDs\n            if led1_is_front:\n                front_LED = position[:, [0, 1]].astype(float)\n                back_LED = position[:, [2, 3]].astype(float)\n            else:\n                back_LED = position[:, [0, 1]].astype(float)\n                front_LED = position[:, [2, 3]].astype(float)\n\n        # Convert to cm\n        back_LED *= meters_to_pixels * CM_TO_METERS\n        front_LED *= meters_to_pixels * CM_TO_METERS\n\n        # Set points to NaN where the front and back LEDs are too separated\n        dist_between_LEDs = get_distance(back_LED, front_LED)\n        is_too_separated = dist_between_LEDs &gt;= max_LED_separation\n        if np.all(is_too_separated):\n            raise ValueError(\n                \"All points are too far apart. If this is single LED data,\"\n                + \"please check that using a parameter set with large max_LED_seperation.\"\n                + f\"Current max_LED_separation: {max_LED_separation}\"\n            )\n\n        back_LED[is_too_separated] = np.nan\n        front_LED[is_too_separated] = np.nan\n\n        # Calculate speed\n        front_LED_speed = get_speed(\n            front_LED,\n            time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )\n        back_LED_speed = get_speed(\n            back_LED,\n            time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )\n\n        # Set to points to NaN where the speed is too fast\n        is_too_fast = (front_LED_speed &gt; max_plausible_speed) | (\n            back_LED_speed &gt; max_plausible_speed\n        )\n        back_LED[is_too_fast] = np.nan\n        front_LED[is_too_fast] = np.nan\n\n        # Interpolate the NaN points\n        back_LED = interpolate_nan(back_LED)\n        front_LED = interpolate_nan(front_LED)\n\n        # Smooth\n        moving_average_window = int(position_smoothing_duration * sampling_rate)\n        back_LED = bottleneck.move_mean(\n            back_LED, window=moving_average_window, axis=0, min_count=1\n        )\n        front_LED = bottleneck.move_mean(\n            front_LED, window=moving_average_window, axis=0, min_count=1\n        )\n\n        if is_upsampled:\n            front_LED, back_LED, time, sampling_rate = self._upsample(\n                front_LED,\n                back_LED,\n                time,\n                sampling_rate,\n                upsampling_sampling_rate,\n                upsampling_interpolation_method,\n            )\n\n        # Calculate position, orientation, velocity, speed\n        position = get_centroid(back_LED, front_LED)  # cm\n\n        orientation = get_angle(back_LED, front_LED)  # radians\n        is_nan = np.isnan(orientation)\n\n        # Unwrap orientation before smoothing\n        orientation[~is_nan] = np.unwrap(orientation[~is_nan])\n        orientation[~is_nan] = gaussian_smooth(\n            orientation[~is_nan],\n            orient_smoothing_std_dev,\n            sampling_rate,\n            axis=0,\n            truncate=8,\n        )\n        # convert back to between -pi and pi\n        orientation[~is_nan] = np.angle(np.exp(1j * orientation[~is_nan]))\n\n        velocity = get_velocity(\n            position,\n            time=time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )  # cm/s\n        speed = np.sqrt(np.sum(velocity**2, axis=1))  # cm/s\n\n        return {\n            \"time\": time,\n            \"position\": position,\n            \"orientation\": orientation,\n            \"velocity\": velocity,\n            \"speed\": speed,\n        }\n\n    def fetch1_dataframe(self):\n        return self._data_to_df(self.fetch_nwb()[0])\n\n    @staticmethod\n    def _data_to_df(data, prefix=\"head_\", add_frame_ind=False):\n        pos, ori, vel = [\n            prefix + c for c in [\"position\", \"orientation\", \"velocity\"]\n        ]\n\n        COLUMNS = [\n            f\"{pos}_x\",\n            f\"{pos}_y\",\n            ori,\n            f\"{vel}_x\",\n            f\"{vel}_y\",\n            f\"{prefix}speed\",\n        ]\n\n        df = pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(data[pos].get_spatial_series().data),\n                    np.asarray(data[ori].get_spatial_series().data)[\n                        :, np.newaxis\n                    ],\n                    np.asarray(data[vel].time_series[vel].data),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=pd.Index(\n                np.asarray(data[pos].get_spatial_series().timestamps),\n                name=\"time\",\n            ),\n        )\n\n        if add_frame_ind:\n            df.insert(\n                0,\n                \"video_frame_ind\",\n                np.asarray(\n                    data[vel].time_series[\"video_frame_ind\"].data,\n                    dtype=int,\n                ),\n            )\n\n        return df\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfo.generate_pos_components", "title": "<code>generate_pos_components(spatial_series, position_info, analysis_fname, prefix='head_', add_frame_ind=False, video_frame_ind=None)</code>  <code>staticmethod</code>", "text": "<p>Generate position, orientation and velocity components.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@staticmethod\ndef generate_pos_components(\n    spatial_series,\n    position_info,\n    analysis_fname,\n    prefix=\"head_\",\n    add_frame_ind=False,\n    video_frame_ind=None,\n):\n    \"\"\"Generate position, orientation and velocity components.\"\"\"\n    METERS_PER_CM = 0.01\n\n    position = pynwb.behavior.Position()\n    orientation = pynwb.behavior.CompassDirection()\n    velocity = pynwb.behavior.BehavioralTimeSeries()\n\n    # NOTE: CBroz1 removed a try/except ValueError that surrounded all\n    #       .create_X_series methods. dpeg22 could not recall purpose\n\n    time_comments = dict(\n        comments=spatial_series.comments,\n        timestamps=position_info[\"time\"],\n    )\n    time_comments_ref = dict(\n        **time_comments,\n        reference_frame=spatial_series.reference_frame,\n    )\n\n    # create nwb objects for insertion into analysis nwb file\n    position.create_spatial_series(\n        name=f\"{prefix}position\",\n        conversion=METERS_PER_CM,\n        data=position_info[\"position\"],\n        description=f\"{prefix}x_position, {prefix}y_position\",\n        **time_comments_ref,\n    )\n\n    orientation.create_spatial_series(\n        name=f\"{prefix}orientation\",\n        conversion=1.0,\n        data=position_info[\"orientation\"],\n        description=f\"{prefix}orientation\",\n        **time_comments_ref,\n    )\n\n    velocity.create_timeseries(\n        name=f\"{prefix}velocity\",\n        conversion=METERS_PER_CM,\n        unit=\"m/s\",\n        data=np.concatenate(\n            (\n                position_info[\"velocity\"],\n                position_info[\"speed\"][:, np.newaxis],\n            ),\n            axis=1,\n        ),\n        description=f\"{prefix}x_velocity, {prefix}y_velocity, \"\n        + f\"{prefix}speed\",\n        **time_comments,\n    )\n\n    if add_frame_ind:\n        if video_frame_ind is not None:\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                data=video_frame_ind.to_numpy(),\n                description=\"video_frame_ind\",\n                **time_comments,\n            )\n        else:\n            logger.info(\n                \"No video frame index found. Assuming all camera frames \"\n                + \"are present.\"\n            )\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                data=np.arange(len(position_info[\"time\"])),\n                description=\"video_frame_ind\",\n                **time_comments,\n            )\n\n    # Insert into analysis nwb file\n    nwba = AnalysisNwbfile()\n\n    return {\n        f\"{prefix}position_object_id\": nwba.add_nwb_object(\n            analysis_fname, position\n        ),\n        f\"{prefix}orientation_object_id\": nwba.add_nwb_object(\n            analysis_fname, orientation\n        ),\n        f\"{prefix}velocity_object_id\": nwba.add_nwb_object(\n            analysis_fname, velocity\n        ),\n    }\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionVideo", "title": "<code>PositionVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass PositionVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalPositionInfo\n    \"\"\"\n\n    def make(self, key):\n        M_TO_CM = 100\n\n        logger.info(\"Loading position data...\")\n        raw_position_df = (\n            RawPosition()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch1_dataframe()\n        position_info_df = (\n            IntervalPositionInfo()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n                \"position_info_param_name\": key[\"position_info_param_name\"],\n            }\n        ).fetch1_dataframe()\n\n        logger.info(\"Loading video data...\")\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n        video_info = (\n            VideoFile()\n            &amp; {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        ).fetch1()\n        io = pynwb.NWBHDF5IO(raw_dir + \"/\" + video_info[\"nwb_file_name\"], \"r\")\n        nwb_file = io.read()\n        nwb_video = nwb_file.objects[video_info[\"video_file_object_id\"]]\n        video_filename = nwb_video.external_file[0]\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        output_video_filename = (\n            f\"{nwb_base_filename}_{epoch:02d}_\"\n            f'{key[\"position_info_param_name\"]}.mp4'\n        )\n\n        # ensure standardized column names\n        raw_position_df = _fix_col_names(raw_position_df)\n        # if IntervalPositionInfo supersampled position, downsample to video\n        if position_info_df.shape[0] &gt; raw_position_df.shape[0]:\n            ind = np.digitize(\n                raw_position_df.index, position_info_df.index, right=True\n            )\n            position_info_df = position_info_df.iloc[ind]\n\n        centroids = {\n            \"red\": np.asarray(raw_position_df[[\"xloc\", \"yloc\"]]),\n            \"green\": np.asarray(raw_position_df[[\"xloc2\", \"yloc2\"]]),\n        }\n        head_position_mean = np.asarray(\n            position_info_df[[\"head_position_x\", \"head_position_y\"]]\n        )\n        head_orientation_mean = np.asarray(\n            position_info_df[[\"head_orientation\"]]\n        )\n        video_time = np.asarray(nwb_video.timestamps)\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = nwb_video.device.meters_per_pixel * M_TO_CM\n\n        logger.info(\"Making video...\")\n        self.make_video(\n            f\"{video_dir}/{video_filename}\",\n            centroids,\n            head_position_mean,\n            head_orientation_mean,\n            video_time,\n            position_time,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n        )\n\n    @staticmethod\n    def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n        \"\"\"Converts from cm to pixels and flips the y-axis.\n        Parameters\n        ----------\n        data : ndarray, shape (n_time, 2)\n        frame_size : array_like, shape (2,)\n        cm_to_pixels : float\n\n        Returns\n        -------\n        converted_data : ndarray, shape (n_time, 2)\n        \"\"\"\n        return data / cm_to_pixels\n\n    @staticmethod\n    def fill_nan(variable, video_time, variable_time):\n        video_ind = np.digitize(variable_time, video_time[1:])\n\n        n_video_time = len(video_time)\n        try:\n            n_variable_dims = variable.shape[1]\n            filled_variable = np.full((n_video_time, n_variable_dims), np.nan)\n        except IndexError:\n            filled_variable = np.full((n_video_time,), np.nan)\n        filled_variable[video_ind] = variable\n\n        return filled_variable\n\n    def make_video(\n        self,\n        video_filename,\n        centroids,\n        head_position_mean,\n        head_orientation_mean,\n        video_time,\n        position_time,\n        output_video_filename=\"output.mp4\",\n        cm_to_pixels=1.0,\n        disable_progressbar=False,\n        arrow_radius=15,\n        circle_radius=8,\n    ):\n        import cv2  # noqa: F401\n\n        RGB_PINK = (234, 82, 111)\n        RGB_YELLOW = (253, 231, 76)\n        RGB_WHITE = (255, 255, 255)\n\n        video = cv2.VideoCapture(video_filename)\n        fourcc = cv2.VideoWriter_fourcc(*\"mp4v\")\n        frame_size = (int(video.get(3)), int(video.get(4)))\n        frame_rate = video.get(5)\n        n_frames = int(head_orientation_mean.shape[0])\n\n        out = cv2.VideoWriter(\n            output_video_filename, fourcc, frame_rate, frame_size, True\n        )\n\n        centroids = {\n            color: self.fill_nan(data, video_time, position_time)\n            for color, data in centroids.items()\n        }\n        head_position_mean = self.fill_nan(\n            head_position_mean, video_time, position_time\n        )\n        head_orientation_mean = self.fill_nan(\n            head_orientation_mean, video_time, position_time\n        )\n\n        for time_ind in tqdm(\n            range(n_frames - 1), desc=\"frames\", disable=disable_progressbar\n        ):\n            is_grabbed, frame = video.read()\n            if is_grabbed:\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n\n                red_centroid = centroids[\"red\"][time_ind]\n                green_centroid = centroids[\"green\"][time_ind]\n\n                head_position = head_position_mean[time_ind]\n                head_position = self.convert_to_pixels(\n                    head_position, frame_size, cm_to_pixels\n                )\n                head_orientation = head_orientation_mean[time_ind]\n\n                if np.all(~np.isnan(red_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(red_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_YELLOW,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(green_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(green_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_PINK,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(head_position)) &amp; np.all(\n                    ~np.isnan(head_orientation)\n                ):\n                    arrow_tip = (\n                        int(\n                            head_position[0]\n                            + arrow_radius * np.cos(head_orientation)\n                        ),\n                        int(\n                            head_position[1]\n                            + arrow_radius * np.sin(head_orientation)\n                        ),\n                    )\n                    cv2.arrowedLine(\n                        img=frame,\n                        pt1=tuple(head_position.astype(int)),\n                        pt2=arrow_tip,\n                        color=RGB_WHITE,\n                        thickness=4,\n                        line_type=8,\n                        shift=cv2.CV_8U,\n                        tipLength=0.25,\n                    )\n\n                if np.all(~np.isnan(head_position)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(head_position.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_WHITE,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)\n                out.write(frame)\n            else:\n                break\n\n        video.release()\n        out.release()\n        cv2.destroyAllWindows()\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionVideo.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>  <code>staticmethod</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@staticmethod\ndef convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/common/common_region/", "title": "common_region.py", "text": ""}, {"location": "api/common/common_region/#src.spyglass.common.common_region.BrainRegion", "title": "<code>BrainRegion</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/common/common_region.py</code> <pre><code>@schema\nclass BrainRegion(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    region_id: smallint auto_increment\n    ---\n    region_name: varchar(200)             # the name of the brain region\n    subregion_name=NULL: varchar(200)     # subregion name\n    subsubregion_name=NULL: varchar(200)  # subregion within subregion\n    \"\"\"\n\n    # TODO consider making (region_name, subregion_name, subsubregion_name) a\n    # primary key subregion_name='' and subsubregion_name='' will be necessary\n    # but that seems OK\n\n    @classmethod\n    def fetch_add(\n        cls, region_name, subregion_name=None, subsubregion_name=None\n    ):\n        \"\"\"Return the region ID for names. If no match, add to the BrainRegion.\n\n        The combination of (region_name, subregion_name, subsubregion_name) is\n        effectively unique, then.\n\n        Parameters\n        ----------\n        region_name : str\n            The name of the brain region.\n        subregion_name : str, optional\n            The name of the subregion within the brain region.\n        subsubregion_name : str, optional\n            The name of the subregion within the subregion.\n\n        Returns\n        -------\n        region_id : int\n            The index of the region in the BrainRegion table.\n        \"\"\"\n        key = dict(\n            region_name=region_name,\n            subregion_name=subregion_name,\n            subsubregion_name=subsubregion_name,\n        )\n        query = BrainRegion &amp; key\n        if not query:\n            cls.insert1(key)\n            query = BrainRegion &amp; key\n        return query.fetch1(\"region_id\")\n</code></pre>"}, {"location": "api/common/common_region/#src.spyglass.common.common_region.BrainRegion.fetch_add", "title": "<code>fetch_add(region_name, subregion_name=None, subsubregion_name=None)</code>  <code>classmethod</code>", "text": "<p>Return the region ID for names. If no match, add to the BrainRegion.</p> <p>The combination of (region_name, subregion_name, subsubregion_name) is effectively unique, then.</p> <p>Parameters:</p> Name Type Description Default <code>region_name</code> <code>str</code> <p>The name of the brain region.</p> required <code>subregion_name</code> <code>str</code> <p>The name of the subregion within the brain region.</p> <code>None</code> <code>subsubregion_name</code> <code>str</code> <p>The name of the subregion within the subregion.</p> <code>None</code> <p>Returns:</p> Name Type Description <code>region_id</code> <code>int</code> <p>The index of the region in the BrainRegion table.</p> Source code in <code>src/spyglass/common/common_region.py</code> <pre><code>@classmethod\ndef fetch_add(\n    cls, region_name, subregion_name=None, subsubregion_name=None\n):\n    \"\"\"Return the region ID for names. If no match, add to the BrainRegion.\n\n    The combination of (region_name, subregion_name, subsubregion_name) is\n    effectively unique, then.\n\n    Parameters\n    ----------\n    region_name : str\n        The name of the brain region.\n    subregion_name : str, optional\n        The name of the subregion within the brain region.\n    subsubregion_name : str, optional\n        The name of the subregion within the subregion.\n\n    Returns\n    -------\n    region_id : int\n        The index of the region in the BrainRegion table.\n    \"\"\"\n    key = dict(\n        region_name=region_name,\n        subregion_name=subregion_name,\n        subsubregion_name=subsubregion_name,\n    )\n    query = BrainRegion &amp; key\n    if not query:\n        cls.insert1(key)\n        query = BrainRegion &amp; key\n    return query.fetch1(\"region_id\")\n</code></pre>"}, {"location": "api/common/common_ripple/", "title": "common_ripple.py", "text": ""}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleLFPSelection", "title": "<code>RippleLFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleLFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; LFPBand\n     group_name = 'CA1' : varchar(80)\n     \"\"\"\n\n    class RippleLFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; RippleLFPSelection\n        -&gt; LFPBandSelection.LFPBandElectrode\n        \"\"\"\n\n    def insert1(self, key, **kwargs):\n        filter_name = (LFPBand &amp; key).fetch1(\"filter_name\")\n        if \"ripple\" not in filter_name.lower():\n            logger.warning(\"Please use a ripple filter\")\n        super().insert1(key, **kwargs)\n\n    @staticmethod\n    def set_lfp_electrodes(\n        key,\n        electrode_list=None,\n        group_name=\"CA1\",\n        **kwargs,\n    ):\n        \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n        Parameters\n        ----------\n        key : dict\n            dictionary corresponding to the LFPBand entry to use for ripple detection\n        electrode_list : list\n            list of electrodes from LFPBandSelection.LFPBandElectrode\n            to be used as the ripple LFP during detection\n        group_name : str, optional\n            description of the electrode group, by default \"CA1\"\n        \"\"\"\n\n        RippleLFPSelection().insert1(\n            {**key, \"group_name\": group_name},\n            skip_duplicates=True,\n            **kwargs,\n        )\n        if not electrode_list:\n            electrode_list = (\n                (LFPBandSelection.LFPBandElectrode() &amp; key)\n                .fetch(\"electrode_id\")\n                .tolist()\n            )\n        electrode_list.sort()\n        electrode_keys = (\n            pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n            .set_index(\"electrode_id\")\n            .loc[electrode_list]\n            .reset_index()\n            .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n        )\n        electrode_keys[\"group_name\"] = group_name\n        electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n        RippleLFPSelection().RippleLFPElectrode.insert(\n            electrode_keys.to_dict(orient=\"records\"),\n            replace=True,\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleLFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(key, electrode_list=None, group_name='CA1', **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary corresponding to the LFPBand entry to use for ripple detection</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes from LFPBandSelection.LFPBandElectrode to be used as the ripple LFP during detection</p> <code>None</code> <code>group_name</code> <code>str</code> <p>description of the electrode group, by default \"CA1\"</p> <code>'CA1'</code> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@staticmethod\ndef set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name=\"CA1\",\n    **kwargs,\n):\n    \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n    Parameters\n    ----------\n    key : dict\n        dictionary corresponding to the LFPBand entry to use for ripple detection\n    electrode_list : list\n        list of electrodes from LFPBandSelection.LFPBandElectrode\n        to be used as the ripple LFP during detection\n    group_name : str, optional\n        description of the electrode group, by default \"CA1\"\n    \"\"\"\n\n    RippleLFPSelection().insert1(\n        {**key, \"group_name\": group_name},\n        skip_duplicates=True,\n        **kwargs,\n    )\n    if not electrode_list:\n        electrode_list = (\n            (LFPBandSelection.LFPBandElectrode() &amp; key)\n            .fetch(\"electrode_id\")\n            .tolist()\n        )\n    electrode_list.sort()\n    electrode_keys = (\n        pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n        .set_index(\"electrode_id\")\n        .loc[electrode_list]\n        .reset_index()\n        .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n    )\n    electrode_keys[\"group_name\"] = group_name\n    electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n    RippleLFPSelection().RippleLFPElectrode.insert(\n        electrode_keys.to_dict(orient=\"records\"),\n        replace=True,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleParameters", "title": "<code>RippleParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleParameters(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    ripple_param_name : varchar(80) # a name for this set of parameters\n    ----\n    ripple_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\"\"\"\n        default_dict = {\n            \"speed_name\": \"head_speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\"\"\"\n    default_dict = {\n        \"speed_name\": \"head_speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleTimes", "title": "<code>RippleTimes</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleTimes(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; RippleParameters\n    -&gt; RippleLFPSelection\n    -&gt; IntervalPositionInfo\n    ---\n    -&gt; AnalysisNwbfile\n    ripple_times_object_id : varchar(40)\n     \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing ripple times for: {key}\")\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        ripple_detection_algorithm = ripple_params[\"ripple_detection_algorithm\"]\n        ripple_detection_params = ripple_params[\"ripple_detection_params\"]\n\n        (\n            speed,\n            interval_ripple_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n\n        ripple_times = RIPPLE_DETECTION_ALGORITHMS[ripple_detection_algorithm](\n            time=np.asarray(interval_ripple_lfps.index),\n            filtered_lfps=np.asarray(interval_ripple_lfps),\n            speed=np.asarray(speed),\n            sampling_frequency=sampling_frequency,\n            **ripple_detection_params,\n        )\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(\n            key[\"nwb_file_name\"]\n        )\n        key[\"ripple_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=ripple_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"ripple_times\"] for data in self.fetch_nwb()]\n\n    @staticmethod\n    def get_ripple_lfps_and_position_info(key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        interval_list_name = key[\"target_interval_list_name\"]\n        position_info_param_name = key[\"position_info_param_name\"]\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        speed_name = ripple_params[\"speed_name\"]\n\n        electrode_keys = (RippleLFPSelection.RippleLFPElectrode() &amp; key).fetch(\n            \"electrode_id\"\n        )\n\n        # warn/validate that there is only one wire per electrode\n        lfp_key = key.copy()\n        del lfp_key[\"interval_list_name\"]\n        ripple_lfp_nwb = (LFPBand &amp; lfp_key).fetch_nwb()[0]\n        ripple_lfp_electrodes = ripple_lfp_nwb[\"filtered_data\"].electrodes.data[\n            :\n        ]\n        elec_mask = np.full_like(ripple_lfp_electrodes, 0, dtype=bool)\n        elec_mask[\n            [\n                ind\n                for ind, elec in enumerate(ripple_lfp_electrodes)\n                if elec in electrode_keys\n            ]\n        ] = True\n        ripple_lfp = pd.DataFrame(\n            ripple_lfp_nwb[\"filtered_data\"].data,\n            index=pd.Index(\n                ripple_lfp_nwb[\"filtered_data\"].timestamps, name=\"time\"\n            ),\n        )\n        sampling_frequency = ripple_lfp_nwb[\"lfp_band_sampling_rate\"]\n\n        ripple_lfp = ripple_lfp.loc[:, elec_mask]\n\n        position_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        position_info = (\n            IntervalPositionInfo()\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n                \"position_info_param_name\": position_info_param_name,\n            }\n        ).fetch1_dataframe()\n\n        position_info = pd.concat(\n            [\n                position_info.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=1,\n        )\n        interval_ripple_lfps = pd.concat(\n            [\n                ripple_lfp.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=1,\n        )\n\n        position_info = interpolate_to_new_time(\n            position_info, interval_ripple_lfps.index\n        )\n\n        return (\n            position_info[speed_name],\n            interval_ripple_lfps,\n            sampling_frequency,\n        )\n\n    @staticmethod\n    def get_Kay_ripple_consensus_trace(\n        ripple_filtered_lfps, sampling_frequency, smoothing_sigma=0.004\n    ):\n        ripple_consensus_trace = np.full_like(ripple_filtered_lfps, np.nan)\n        not_null = np.all(pd.notnull(ripple_filtered_lfps), axis=1)\n\n        ripple_consensus_trace[not_null] = get_envelope(\n            np.asarray(ripple_filtered_lfps)[not_null]\n        )\n        ripple_consensus_trace = np.sum(ripple_consensus_trace**2, axis=1)\n        ripple_consensus_trace[not_null] = gaussian_smooth(\n            ripple_consensus_trace[not_null],\n            smoothing_sigma,\n            sampling_frequency,\n        )\n        return pd.DataFrame(\n            np.sqrt(ripple_consensus_trace), index=ripple_filtered_lfps.index\n        )\n\n    @staticmethod\n    def plot_ripple_consensus_trace(\n        ripple_consensus_trace,\n        ripple_times,\n        ripple_label=1,\n        offset=0.100,\n        relative=True,\n        ax=None,\n    ):\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n\n        start_offset = ripple_start if relative else 0\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, 1))\n        ax.plot(\n            ripple_consensus_trace.loc[time_slice].index - start_offset,\n            ripple_consensus_trace.loc[time_slice],\n        )\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_xlabel(\"Time [s]\")\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n\n    @staticmethod\n    def plot_ripple(\n        lfps, ripple_times, ripple_label=1, offset=0.100, relative=True, ax=None\n    ):\n        lfp_labels = lfps.columns\n        n_lfps = len(lfp_labels)\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, n_lfps * 0.20))\n\n        start_offset = ripple_start if relative else 0\n\n        for lfp_ind, lfp_label in enumerate(lfp_labels):\n            lfp = lfps.loc[time_slice, lfp_label]\n            ax.plot(\n                lfp.index - start_offset,\n                lfp_ind + (lfp - lfp.mean()) / (lfp.max() - lfp.min()),\n                color=\"black\",\n            )\n\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_ylim((-1, n_lfps))\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n        ax.set_ylabel(\"LFPs\")\n        ax.set_xlabel(\"Time [s]\")\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleTimes.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/common/common_sensors/", "title": "common_sensors.py", "text": "<p>Schema for headstage or other environmental sensors.</p>"}, {"location": "api/common/common_session/", "title": "common_session.py", "text": ""}, {"location": "api/common/common_subject/", "title": "common_subject.py", "text": ""}, {"location": "api/common/common_subject/#src.spyglass.common.common_subject.Subject", "title": "<code>Subject</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_subject.py</code> <pre><code>@schema\nclass Subject(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    subject_id: varchar(80)\n    ---\n    age = NULL: varchar(200)\n    description = NULL: varchar(2000)\n    genotype = NULL: varchar(2000)\n    sex = \"U\": enum(\"M\", \"F\", \"U\")\n    species = NULL: varchar(200)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Get the subject info from the NWBFile, insert into the Subject.\"\"\"\n        sub = nwbf.subject\n        if sub is None:\n            logger.warn(\"No subject metadata found.\\n\")\n            return\n        subject_dict = dict()\n        subject_dict[\"subject_id\"] = sub.subject_id\n        subject_dict[\"age\"] = sub.age\n        subject_dict[\"description\"] = sub.description\n        subject_dict[\"genotype\"] = sub.genotype\n        if sub.sex in (\"Male\", \"male\", \"M\", \"m\"):\n            sex = \"M\"\n        elif sub.sex in (\"Female\", \"female\", \"F\", \"f\"):\n            sex = \"F\"\n        else:\n            sex = \"U\"\n        subject_dict[\"sex\"] = sex\n        subject_dict[\"species\"] = sub.species\n        cls.insert1(subject_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_subject/#src.spyglass.common.common_subject.Subject.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Get the subject info from the NWBFile, insert into the Subject.</p> Source code in <code>src/spyglass/common/common_subject.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Get the subject info from the NWBFile, insert into the Subject.\"\"\"\n    sub = nwbf.subject\n    if sub is None:\n        logger.warn(\"No subject metadata found.\\n\")\n        return\n    subject_dict = dict()\n    subject_dict[\"subject_id\"] = sub.subject_id\n    subject_dict[\"age\"] = sub.age\n    subject_dict[\"description\"] = sub.description\n    subject_dict[\"genotype\"] = sub.genotype\n    if sub.sex in (\"Male\", \"male\", \"M\", \"m\"):\n        sex = \"M\"\n    elif sub.sex in (\"Female\", \"female\", \"F\", \"f\"):\n        sex = \"F\"\n    else:\n        sex = \"U\"\n    subject_dict[\"sex\"] = sex\n    subject_dict[\"species\"] = sub.species\n    cls.insert1(subject_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_task/", "title": "common_task.py", "text": ""}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task", "title": "<code>Task</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@schema\nclass Task(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     task_name: varchar(80)\n     ---\n     task_description = NULL: varchar(2000)    # description of this task\n     task_type = NULL: varchar(2000)           # type of task\n     task_subtype = NULL: varchar(2000)        # subtype of task\n     \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert tasks from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        \"\"\"\n        tasks_mod = nwbf.processing.get(\"tasks\")\n        if tasks_mod is None:\n            logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n            return\n        for task in tasks_mod.data_interfaces.values():\n            if cls.check_task_table(task):\n                cls.insert_from_task_table(task)\n\n    @classmethod\n    def insert_from_task_table(cls, task_table):\n        \"\"\"Insert tasks from a pynwb DynamicTable containing task metadata.\n\n        Duplicate tasks will not be added.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n        \"\"\"\n        taskdf = task_table.to_dataframe()\n        for task_entry in taskdf.iterrows():\n            task_dict = dict()\n            task_dict[\"task_name\"] = task_entry[1].task_name\n            task_dict[\"task_description\"] = task_entry[1].task_description\n            cls.insert1(task_dict, skip_duplicates=True)\n\n    @classmethod\n    def check_task_table(cls, task_table):\n        \"\"\"Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.\n\n\n        The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and\n        'task_description'.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n\n        Returns\n        -------\n        bool\n            Whether the DynamicTable conforms to the expected format for loading data into the Task table.\n        \"\"\"\n        return (\n            isinstance(task_table, pynwb.core.DynamicTable)\n            and hasattr(task_table, \"task_name\")\n            and hasattr(task_table, \"task_description\")\n        )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert tasks from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert tasks from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    \"\"\"\n    tasks_mod = nwbf.processing.get(\"tasks\")\n    if tasks_mod is None:\n        logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n        return\n    for task in tasks_mod.data_interfaces.values():\n        if cls.check_task_table(task):\n            cls.insert_from_task_table(task)\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.insert_from_task_table", "title": "<code>insert_from_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Insert tasks from a pynwb DynamicTable containing task metadata.</p> <p>Duplicate tasks will not be added.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef insert_from_task_table(cls, task_table):\n    \"\"\"Insert tasks from a pynwb DynamicTable containing task metadata.\n\n    Duplicate tasks will not be added.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n    \"\"\"\n    taskdf = task_table.to_dataframe()\n    for task_entry in taskdf.iterrows():\n        task_dict = dict()\n        task_dict[\"task_name\"] = task_entry[1].task_name\n        task_dict[\"task_description\"] = task_entry[1].task_description\n        cls.insert1(task_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.check_task_table", "title": "<code>check_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.</p> <p>The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and 'task_description'.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether the DynamicTable conforms to the expected format for loading data into the Task table.</p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef check_task_table(cls, task_table):\n    \"\"\"Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.\n\n\n    The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and\n    'task_description'.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n\n    Returns\n    -------\n    bool\n        Whether the DynamicTable conforms to the expected format for loading data into the Task table.\n    \"\"\"\n    return (\n        isinstance(task_table, pynwb.core.DynamicTable)\n        and hasattr(task_table, \"task_name\")\n        and hasattr(task_table, \"task_description\")\n    )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.TaskEpoch", "title": "<code>TaskEpoch</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@schema\nclass TaskEpoch(SpyglassMixin, dj.Imported):\n    # Tasks, session and time intervals\n    definition = \"\"\"\n     -&gt; Session\n     epoch: int  # the session epoch for this task and apparatus(1 based)\n     ---\n     -&gt; Task\n     -&gt; [nullable] CameraDevice\n     -&gt; IntervalList\n     task_environment = NULL: varchar(200)  # the environment the animal was in\n     camera_names : blob # list of keys corresponding to entry in CameraDevice\n     \"\"\"\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile().get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        camera_names = dict()\n\n        # the tasks refer to the camera_id which is unique for the NWB file but\n        # not for CameraDevice schema, so we need to look up the right camera\n        # map camera ID (in camera name) to camera_name\n\n        for device in nwbf.devices.values():\n            if isinstance(device, ndx_franklab_novela.CameraDevice):\n                # get the camera ID\n                camera_id = int(str.split(device.name)[1])\n                camera_names[camera_id] = device.camera_name\n\n        # find the task modules and for each one, add the task to the Task\n        # schema if it isn't there and then add an entry for each epoch\n\n        tasks_mod = nwbf.processing.get(\"tasks\")\n        if tasks_mod is None:\n            logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n            return\n\n        for task in tasks_mod.data_interfaces.values():\n            if self.check_task_table(task):\n                # check if the task is in the Task table and if not, add it\n                Task.insert_from_task_table(task)\n                key[\"task_name\"] = task.task_name[0]\n\n                # get the CameraDevice used for this task (primary key is\n                # camera name so we need to map from ID to name)\n\n                camera_ids = task.camera_id[0]\n                valid_camera_ids = [\n                    camera_id\n                    for camera_id in camera_ids\n                    if camera_id in camera_names.keys()\n                ]\n                if valid_camera_ids:\n                    key[\"camera_names\"] = [\n                        {\"camera_name\": camera_names[camera_id]}\n                        for camera_id in valid_camera_ids\n                    ]\n                else:\n                    logger.warn(\n                        f\"No camera device found with ID {camera_ids} in NWB \"\n                        + f\"file {nwbf}\\n\"\n                    )\n                # Add task environment\n                if hasattr(task, \"task_environment\"):\n                    key[\"task_environment\"] = task.task_environment[0]\n\n                # get the interval list for this task, which corresponds to the\n                # matching epoch for the raw data. Users should define more\n                # restrictive intervals as required for analyses\n\n                session_intervals = (\n                    IntervalList() &amp; {\"nwb_file_name\": nwb_file_name}\n                ).fetch(\"interval_list_name\")\n                for epoch in task.task_epochs[0]:\n                    # TODO in beans file, task_epochs[0] is 1x2 dset of ints,\n                    # so epoch would be an int\n\n                    key[\"epoch\"] = epoch\n                    target_interval = str(epoch).zfill(2)\n                    for interval in session_intervals:\n                        if (\n                            target_interval in interval\n                        ):  # TODO this is not true for the beans file\n                            break\n                    # TODO case when interval is not found is not handled\n                    key[\"interval_list_name\"] = interval\n                    self.insert1(key)\n\n    @classmethod\n    def update_entries(cls, restrict={}):\n        existing_entries = (cls &amp; restrict).fetch(\"KEY\")\n        for row in existing_entries:\n            if (cls &amp; row).fetch1(\"camera_names\"):\n                continue\n            row[\"camera_names\"] = [\n                {\"camera_name\": (cls &amp; row).fetch1(\"camera_name\")}\n            ]\n            cls.update1(row=row)\n\n    @classmethod\n    def check_task_table(cls, task_table) -&gt; bool:\n        \"\"\"Check whether the pynwb DynamicTable containing task metadata\n        conforms to the expected format.\n\n        The table should be an instance of pynwb.core.DynamicTable and contain\n        the columns 'task_name', 'task_description', 'camera_id', 'and\n        'task_epochs'.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n\n        Returns\n        -------\n        bool\n            Whether the DynamicTable conforms to the expected format for\n            loading data into the TaskEpoch table.\n        \"\"\"\n\n        # TODO this could be more strict and check data types, but really it\n        # should be schematized\n        return (\n            Task.check_task_table(task_table)\n            and hasattr(task_table, \"camera_id\")\n            and hasattr(task_table, \"task_epochs\")\n        )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.TaskEpoch.check_task_table", "title": "<code>check_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.</p> <p>The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name', 'task_description', 'camera_id', 'and 'task_epochs'.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether the DynamicTable conforms to the expected format for loading data into the TaskEpoch table.</p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef check_task_table(cls, task_table) -&gt; bool:\n    \"\"\"Check whether the pynwb DynamicTable containing task metadata\n    conforms to the expected format.\n\n    The table should be an instance of pynwb.core.DynamicTable and contain\n    the columns 'task_name', 'task_description', 'camera_id', 'and\n    'task_epochs'.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n\n    Returns\n    -------\n    bool\n        Whether the DynamicTable conforms to the expected format for\n        loading data into the TaskEpoch table.\n    \"\"\"\n\n    # TODO this could be more strict and check data types, but really it\n    # should be schematized\n    return (\n        Task.check_task_table(task_table)\n        and hasattr(task_table, \"camera_id\")\n        and hasattr(task_table, \"task_epochs\")\n    )\n</code></pre>"}, {"location": "api/common/common_usage/", "title": "common_usage.py", "text": "<p>A schema to store the usage of advanced Spyglass features.</p> <p>Records show usage of features such as cautious delete and fault-permitting insert, which will be used to determine which features are used, how often, and by whom. This will help plan future development of Spyglass.</p>"}, {"location": "api/common/errors/", "title": "errors.py", "text": ""}, {"location": "api/common/populate_all_common/", "title": "populate_all_common.py", "text": ""}, {"location": "api/common/populate_all_common/#src.spyglass.common.populate_all_common.populate_all_common", "title": "<code>populate_all_common(nwb_file_name)</code>", "text": "<p>Insert all common tables for a given NWB file.</p> Source code in <code>src/spyglass/common/populate_all_common.py</code> <pre><code>def populate_all_common(nwb_file_name):\n    \"\"\"Insert all common tables for a given NWB file.\"\"\"\n    from spyglass.spikesorting.imported import ImportedSpikeSorting\n\n    key = [(Nwbfile &amp; f\"nwb_file_name LIKE '{nwb_file_name}'\").proj()]\n    tables = [\n        Session,\n        # NwbfileKachery, # Not used by default\n        ElectrodeGroup,\n        Electrode,\n        Raw,\n        SampleCount,\n        DIOEvents,\n        # SensorData, # Not used by default. Generates large files\n        RawPosition,\n        TaskEpoch,\n        StateScriptFile,\n        VideoFile,\n        PositionSource,\n        RawPosition,\n        ImportedSpikeSorting,\n    ]\n    error_constants = dict(\n        dj_user=dj.config[\"database.user\"],\n        connection_id=dj.conn().connection_id,\n        nwb_file_name=nwb_file_name,\n    )\n\n    for table in tables:\n        logger.info(f\"Populating {table.__name__}...\")\n        try:\n            table.populate(key)\n        except Exception as e:\n            InsertError.insert1(\n                dict(\n                    **error_constants,\n                    table=table.__name__,\n                    error_type=type(e).__name__,\n                    error_message=str(e),\n                    error_raw=str(e),\n                )\n            )\n    query = InsertError &amp; error_constants\n    if query:\n        err_tables = query.fetch(\"table\")\n        logger.error(\n            f\"Errors occurred during population for {nwb_file_name}:\\n\\t\"\n            + f\"Failed tables {err_tables}\\n\\t\"\n            + \"See common_usage.InsertError for more details\"\n        )\n        return query.fetch(\"KEY\")\n</code></pre>"}, {"location": "api/common/signal_processing/", "title": "signal_processing.py", "text": ""}, {"location": "api/common/signal_processing/#src.spyglass.common.signal_processing.hilbert_decomp", "title": "<code>hilbert_decomp(lfp_band_object, sampling_rate=1)</code>", "text": "<p>Generates analytical decomposition of signals in the lfp_band_object</p> <p>NOTE: This function is not currently used in the pipeline.</p> <p>Parameters:</p> Name Type Description Default <code>lfp_band_object</code> <code>ElectricalSeries</code> <p>bandpass filtered LFP</p> required <code>sampling_rate</code> <code>int</code> <p>bandpass filtered LFP sampling rate (defaults to 1; only used for instantaneous frequency)</p> <code>1</code> <p>Returns:</p> Name Type Description <code>envelope</code> <code>ElectricalSeries</code> <p>envelope of the signal</p> Source code in <code>src/spyglass/common/signal_processing.py</code> <pre><code>def hilbert_decomp(lfp_band_object, sampling_rate=1):\n    \"\"\"Generates analytical decomposition of signals in the lfp_band_object\n\n    NOTE: This function is not currently used in the pipeline.\n\n    Parameters\n    ----------\n    lfp_band_object : pynwb.ecephys.ElectricalSeries\n        bandpass filtered LFP\n    sampling_rate : int, optional\n        bandpass filtered LFP sampling rate\n        (defaults to 1; only used for instantaneous frequency)\n\n    Returns\n    -------\n    envelope : pynwb.ecephys.ElectricalSeries\n        envelope of the signal\n    \"\"\"\n    analytical_signal = signal.hilbert(lfp_band_object.data, axis=0)\n\n    eseries_name = \"envelope\"\n    envelope = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=np.abs(analytical_signal),\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n\n    eseries_name = \"phase\"\n    instantaneous_phase = np.unwrap(np.angle(analytical_signal))\n    phase = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=instantaneous_phase,\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n\n    eseries_name = \"frequency\"\n    instantaneous_frequency = (\n        np.diff(instantaneous_phase) / (2.0 * np.pi) * sampling_rate\n    )\n    frequency = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=instantaneous_frequency,\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n    return envelope, phase, frequency\n</code></pre>"}, {"location": "api/common/prepopulate/prepopulate/", "title": "prepopulate.py", "text": ""}, {"location": "api/common/prepopulate/prepopulate/#src.spyglass.common.prepopulate.prepopulate.prepopulate_default", "title": "<code>prepopulate_default()</code>", "text": "<p>Populate the database with default values in SPYGLASS_BASE_DIR/entries.yaml</p> Source code in <code>src/spyglass/common/prepopulate/prepopulate.py</code> <pre><code>def prepopulate_default():\n    \"\"\"\n    Populate the database with default values in SPYGLASS_BASE_DIR/entries.yaml\n    \"\"\"\n    yaml_path = pathlib.Path(base_dir) / \"entries.yaml\"\n    if os.path.exists(yaml_path):\n        populate_from_yaml(yaml_path)\n</code></pre>"}, {"location": "api/common/prepopulate/prepopulate/#src.spyglass.common.prepopulate.prepopulate.populate_from_yaml", "title": "<code>populate_from_yaml(yaml_path)</code>", "text": "<p>Populate the database from specially formatted YAML files.</p> Source code in <code>src/spyglass/common/prepopulate/prepopulate.py</code> <pre><code>def populate_from_yaml(yaml_path: str):\n    \"\"\"Populate the database from specially formatted YAML files.\"\"\"\n    if not os.path.exists(yaml_path):\n        raise ValueError(f\"There is no file found with the path: {yaml_path}\")\n    with open(yaml_path, \"r\") as stream:\n        d = yaml.safe_load(stream)\n\n    for table_name, table_entries in d.items():\n        table_cls = _get_table_cls(table_name)\n        for entry_dict in table_entries:\n            # test whether an entity with the primary key(s) already exists\n            if not issubclass(table_cls, (dj.Manual, dj.Lookup, dj.Part)):\n                raise ValueError(\n                    f\"Prepopulate YAML ('{yaml_path}') contains table \"\n                    + f\"{table_name}' that cannot be prepopulated. Only Manual \"\n                    + \"and Lookup tables can be prepopulated.\"\n                )\n            if hasattr(table_cls, \"fetch_add\"):\n                # if the table has defined a fetch_add method, use that instead\n                # of insert1. this is useful for tables where the primary key\n                # is an ID that auto-increments. first check whether an entry\n                # exists with the same information.\n\n                query = table_cls &amp; entry_dict\n                if not query:\n                    logger.info(\n                        f\"Populate: Populating table {table_cls.__name__} with \"\n                        + f\"data {entry_dict} using fetch_add.\"\n                    )\n                    table_cls.fetch_add(**entry_dict)\n                continue\n\n            primary_key_values = {\n                k: v\n                for k, v in entry_dict.items()\n                if k in table_cls.primary_key\n            }\n            if not primary_key_values:\n                logger.warn(\n                    f\"Populate: No primary key provided in data {entry_dict} \"\n                    + f\"for table {table_cls.__name__}\"\n                )\n                continue\n            if primary_key_values not in table_cls.fetch(\n                *table_cls.primary_key, as_dict=True\n            ):\n                logger.info(\n                    f\"Populate: Populating table {table_cls.__name__} with data\"\n                    + f\" {entry_dict} using insert1.\"\n                )\n                table_cls.insert1(entry_dict)\n            else:\n                logging.warn(\n                    f\"Populate: Entry in {table_cls.__name__} with primary keys\"\n                    + f\" {primary_key_values} already exists.\"\n                )\n</code></pre>"}, {"location": "api/data_import/insert_sessions/", "title": "insert_sessions.py", "text": ""}, {"location": "api/data_import/insert_sessions/#src.spyglass.data_import.insert_sessions.insert_sessions", "title": "<code>insert_sessions(nwb_file_names)</code>", "text": "<p>Populate the dj database with new sessions.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_names</code> <code>str or List of str</code> <p>File names in raw directory ($SPYGLASS_RAW_DIR) pointing to existing .nwb files. Each file represents a session. Also accepts strings with glob wildcards (e.g., *) so long as the wildcard specifies exactly one file.</p> required Source code in <code>src/spyglass/data_import/insert_sessions.py</code> <pre><code>def insert_sessions(nwb_file_names: Union[str, List[str]]):\n    \"\"\"\n    Populate the dj database with new sessions.\n\n    Parameters\n    ----------\n    nwb_file_names : str or List of str\n        File names in raw directory ($SPYGLASS_RAW_DIR) pointing to\n        existing .nwb files. Each file represents a session. Also accepts\n        strings with glob wildcards (e.g., *) so long as the wildcard specifies\n        exactly one file.\n    \"\"\"\n\n    if not isinstance(nwb_file_names, list):\n        nwb_file_names = [nwb_file_names]\n\n    for nwb_file_name in nwb_file_names:\n        if \"/\" in nwb_file_name:\n            nwb_file_name = nwb_file_name.split(\"/\")[-1]\n\n        nwb_file_abs_path = Path(\n            Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n        )\n\n        if not nwb_file_abs_path.exists():\n            possible_matches = sorted(Path(raw_dir).glob(f\"*{nwb_file_name}*\"))\n\n            if len(possible_matches) == 1:\n                nwb_file_abs_path = possible_matches[0]\n                nwb_file_name = nwb_file_abs_path.name\n\n            else:\n                raise FileNotFoundError(\n                    f\"File not found: {nwb_file_abs_path}\\n\\t\"\n                    + f\"{len(possible_matches)} possible matches:\"\n                    + f\"{possible_matches}\"\n                )\n\n        # file name for the copied raw data\n        out_nwb_file_name = get_nwb_copy_filename(nwb_file_abs_path.name)\n\n        # Check whether the file already exists in the Nwbfile table\n        if len(Nwbfile() &amp; {\"nwb_file_name\": out_nwb_file_name}):\n            warnings.warn(\n                f\"Cannot insert data from {nwb_file_name}: {out_nwb_file_name}\"\n                + \" is already in Nwbfile table.\"\n            )\n            continue\n\n        # Make a copy of the NWB file that ends with '_'.\n        # This has everything except the raw data but has a link to\n        # the raw data in the original file\n        copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)\n        Nwbfile().insert_from_relative_file_name(out_nwb_file_name)\n        populate_all_common(out_nwb_file_name)\n</code></pre>"}, {"location": "api/data_import/insert_sessions/#src.spyglass.data_import.insert_sessions.copy_nwb_link_raw_ephys", "title": "<code>copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)</code>", "text": "<p>Copies an NWB file with a link to raw ephys data.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file to be copied.</p> required <code>out_nwb_file_name</code> <code>str</code> <p>The name of the new NWB file with the link to raw ephys data.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The absolute path of the new NWB file.</p> Source code in <code>src/spyglass/data_import/insert_sessions.py</code> <pre><code>def copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name):\n    \"\"\"Copies an NWB file with a link to raw ephys data.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file to be copied.\n    out_nwb_file_name : str\n        The name of the new NWB file with the link to raw ephys data.\n\n    Returns\n    -------\n    str\n        The absolute path of the new NWB file.\n    \"\"\"\n    logger.info(\n        f\"Creating a copy of NWB file {nwb_file_name} \"\n        + f\"with link to raw ephys data: {out_nwb_file_name}\"\n    )\n\n    nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n\n    if not os.path.exists(nwb_file_abs_path):\n        raise FileNotFoundError(f\"Could not find raw file: {nwb_file_abs_path}\")\n\n    out_nwb_file_abs_path = Nwbfile.get_abs_path(\n        out_nwb_file_name, new_file=True\n    )\n\n    if os.path.exists(out_nwb_file_abs_path):\n        if debug_mode:\n            return out_nwb_file_abs_path\n        logger.warning(\n            f\"Output file {out_nwb_file_abs_path} exists and will be \"\n            + \"overwritten.\"\n        )\n\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abs_path, mode=\"r\", load_namespaces=True\n    ) as input_io:\n        nwbf = input_io.read()\n\n        # pop off acquisition electricalseries\n        eseries_list = get_raw_eseries(nwbf)\n        for eseries in eseries_list:\n            nwbf.acquisition.pop(eseries.name)\n\n        # pop off analog processing module\n        analog_processing = nwbf.processing.get(\"analog\")\n        if analog_processing:\n            nwbf.processing.pop(\"analog\")\n\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=out_nwb_file_abs_path, mode=\"w\", manager=input_io.manager\n        ) as export_io:\n            export_io.export(input_io, nwbf)\n\n    # add link from new file back to raw ephys data in raw data file using fresh build manager and container cache\n    # where the acquisition electricalseries objects have not been removed\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abs_path, mode=\"r\", load_namespaces=True\n    ) as input_io:\n        nwbf_raw = input_io.read()\n        eseries_list = get_raw_eseries(nwbf_raw)\n        analog_processing = nwbf_raw.processing.get(\"analog\")\n\n        with pynwb.NWBHDF5IO(\n            path=out_nwb_file_abs_path, mode=\"a\", manager=input_io.manager\n        ) as export_io:\n            nwbf_export = export_io.read()\n\n            # add link to raw ephys ElectricalSeries in raw data file\n            for eseries in eseries_list:\n                nwbf_export.add_acquisition(eseries)\n\n            # add link to processing module in raw data file\n            if analog_processing:\n                nwbf_export.add_processing_module(analog_processing)\n\n            nwbf_export.set_modified()\n            export_io.write(nwbf_export)\n\n    # change the permissions to only allow owner to write\n    permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n    os.chmod(out_nwb_file_abs_path, permissions)\n\n    return out_nwb_file_abs_path\n</code></pre>"}, {"location": "api/decoding/decoding_merge/", "title": "decoding_merge.py", "text": ""}, {"location": "api/decoding/decoding_merge/#src.spyglass.decoding.decoding_merge.DecodingOutput", "title": "<code>DecodingOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/decoding/decoding_merge.py</code> <pre><code>@schema\nclass DecodingOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class ClusterlessDecodingV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ClusterlessDecodingV1\n        \"\"\"\n\n    class SortedSpikesDecodingV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; SortedSpikesDecodingV1\n        \"\"\"\n\n    def cleanup(self, dry_run=False):\n        \"\"\"Remove any decoding outputs that are not in the merge table\"\"\"\n        if dry_run:\n            logger.info(\"Dry run, not removing any files\")\n        else:\n            logger.info(\"Cleaning up decoding outputs\")\n        table_results_paths = list(\n            chain(\n                *[\n                    part_parent_table.fetch(\"results_path\").tolist()\n                    for part_parent_table in self.merge_get_parent(\n                        multi_source=True\n                    )\n                ]\n            )\n        )\n        for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.nc\"):\n            if str(path) not in table_results_paths:\n                logger.info(f\"Removing {path}\")\n                if not dry_run:\n                    try:\n                        path.unlink(missing_ok=True)  # Ignore FileNotFoundError\n                    except PermissionError:\n                        logger.warning(f\"Unable to remove {path}, skipping\")\n\n        table_model_paths = list(\n            chain(\n                *[\n                    part_parent_table.fetch(\"classifier_path\").tolist()\n                    for part_parent_table in self.merge_get_parent(\n                        multi_source=True\n                    )\n                ]\n            )\n        )\n        for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.pkl\"):\n            if str(path) not in table_model_paths:\n                logger.info(f\"Removing {path}\")\n                if not dry_run:\n                    try:\n                        path.unlink()\n                    except (PermissionError, FileNotFoundError):\n                        logger.warning(f\"Unable to remove {path}, skipping\")\n\n    @classmethod\n    def fetch_results(cls, key):\n        return cls().merge_get_parent_class(key).fetch_results()\n\n    @classmethod\n    def fetch_model(cls, key):\n        return cls().merge_get_parent_class(key).fetch_model()\n\n    @classmethod\n    def fetch_environments(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_environments(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_position_info(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_position_info(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_linear_position_info(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_linear_position_info(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_spike_data(cls, key, filter_by_interval=True):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_linear_position_info(\n                decoding_selection_key, filter_by_interval=filter_by_interval\n            )\n        )\n\n    @classmethod\n    def create_decoding_view(cls, key, head_direction_name=\"head_orientation\"):\n        results = cls.fetch_results(key)\n        posterior = (\n            results.squeeze()\n            .acausal_posterior.unstack(\"state_bins\")\n            .drop_sel(state=[\"Local\", \"No-Spike\"], errors=\"ignore\")\n            .sum(\"state\")\n        )\n        posterior /= posterior.sum(\"position\")\n        env = cls.fetch_environments(key)[0]\n\n        if \"x_position\" in results.coords:\n            position_info, position_variable_names = cls.fetch_position_info(\n                key\n            )\n            # Not 1D\n            bin_size = (\n                np.nanmedian(np.diff(np.unique(results.x_position.values))),\n                np.nanmedian(np.diff(np.unique(results.y_position.values))),\n            )\n            return create_2D_decode_view(\n                position_time=position_info.index,\n                position=position_info[position_variable_names],\n                interior_place_bin_centers=env.place_bin_centers_[\n                    env.is_track_interior_.ravel(order=\"C\")\n                ],\n                place_bin_size=bin_size,\n                posterior=posterior,\n                head_dir=position_info[head_direction_name],\n            )\n        else:\n            (\n                position_info,\n                position_variable_names,\n            ) = cls.fetch_linear_position_info(key)\n            return create_1D_decode_view(\n                posterior=posterior,\n                linear_position=position_info[\"linear_position\"],\n            )\n</code></pre>"}, {"location": "api/decoding/decoding_merge/#src.spyglass.decoding.decoding_merge.DecodingOutput.cleanup", "title": "<code>cleanup(dry_run=False)</code>", "text": "<p>Remove any decoding outputs that are not in the merge table</p> Source code in <code>src/spyglass/decoding/decoding_merge.py</code> <pre><code>def cleanup(self, dry_run=False):\n    \"\"\"Remove any decoding outputs that are not in the merge table\"\"\"\n    if dry_run:\n        logger.info(\"Dry run, not removing any files\")\n    else:\n        logger.info(\"Cleaning up decoding outputs\")\n    table_results_paths = list(\n        chain(\n            *[\n                part_parent_table.fetch(\"results_path\").tolist()\n                for part_parent_table in self.merge_get_parent(\n                    multi_source=True\n                )\n            ]\n        )\n    )\n    for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.nc\"):\n        if str(path) not in table_results_paths:\n            logger.info(f\"Removing {path}\")\n            if not dry_run:\n                try:\n                    path.unlink(missing_ok=True)  # Ignore FileNotFoundError\n                except PermissionError:\n                    logger.warning(f\"Unable to remove {path}, skipping\")\n\n    table_model_paths = list(\n        chain(\n            *[\n                part_parent_table.fetch(\"classifier_path\").tolist()\n                for part_parent_table in self.merge_get_parent(\n                    multi_source=True\n                )\n            ]\n        )\n    )\n    for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.pkl\"):\n        if str(path) not in table_model_paths:\n            logger.info(f\"Removing {path}\")\n            if not dry_run:\n                try:\n                    path.unlink()\n                except (PermissionError, FileNotFoundError):\n                    logger.warning(f\"Unable to remove {path}, skipping\")\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/", "title": "clusterless.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from unclustered spikes and spike waveform features. See [1] for details.</p>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters", "title": "<code>MarkParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Defines the type of waveform feature computed for a given spike time.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass MarkParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Defines the type of waveform feature computed for a given spike time.\"\"\"\n\n    definition = \"\"\"\n    mark_param_name : varchar(32) # a name for this set of parameters\n    ---\n    # the type of mark. Currently only 'amplitude' is supported\n    mark_type = 'amplitude':  varchar(40)\n    mark_param_dict: BLOB # dict of parameters for the mark extraction function\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\n\n        Examples\n        --------\n        {'peak_sign': 'neg', 'threshold' : 100}\n        corresponds to negative going waveforms of at least 100 uV size\n        \"\"\"\n        default_dict = {}\n        self.insert1(\n            {\"mark_param_name\": \"default\", \"mark_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n\n    @staticmethod\n    def supported_mark_type(mark_type):\n        \"\"\"Checks whether the requested mark type is supported.\n\n        Currently only 'amplitude\" is supported.\n\n        Parameters\n        ----------\n        mark_type : str\n\n        \"\"\"\n        supported_types = [\"amplitude\"]\n        return mark_type in supported_types\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> <p>Examples:</p> <p>{'peak_sign': 'neg', 'threshold' : 100} corresponds to negative going waveforms of at least 100 uV size</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\n\n    Examples\n    --------\n    {'peak_sign': 'neg', 'threshold' : 100}\n    corresponds to negative going waveforms of at least 100 uV size\n    \"\"\"\n    default_dict = {}\n    self.insert1(\n        {\"mark_param_name\": \"default\", \"mark_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters.supported_mark_type", "title": "<code>supported_mark_type(mark_type)</code>  <code>staticmethod</code>", "text": "<p>Checks whether the requested mark type is supported.</p> <p>Currently only 'amplitude\" is supported.</p> <p>Parameters:</p> Name Type Description Default <code>mark_type</code> <code>str</code> required Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef supported_mark_type(mark_type):\n    \"\"\"Checks whether the requested mark type is supported.\n\n    Currently only 'amplitude\" is supported.\n\n    Parameters\n    ----------\n    mark_type : str\n\n    \"\"\"\n    supported_types = [\"amplitude\"]\n    return mark_type in supported_types\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarks", "title": "<code>UnitMarks</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Compute spike waveform features for each spike time.</p> <p>For each spike time, compute a spike waveform feature associated with that spike. Used for clusterless decoding.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarks(SpyglassMixin, dj.Computed):\n    \"\"\"Compute spike waveform features for each spike time.\n\n    For each spike time, compute a spike waveform feature associated with that\n    spike. Used for clusterless decoding.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarkParameters\n    ---\n    -&gt; AnalysisNwbfile\n    marks_object_id: varchar(40) # the NWB object that stores the marks\n    \"\"\"\n\n    def make(self, key):\n        # get the list of mark parameters\n        mark_param = (MarkParameters &amp; key).fetch1()\n\n        # check that the mark type is supported\n        if not MarkParameters().supported_mark_type(mark_param[\"mark_type\"]):\n            Warning(\n                f'Mark type {mark_param[\"mark_type\"]} not supported; skipping'\n            )\n            return\n\n        # retrieve the units from the NWB file\n        nwb_units = (CuratedSpikeSorting() &amp; key).fetch_nwb()[0][\"units\"]\n\n        recording = Curation.get_recording(key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n        sorting = Curation.get_curated_sorting(key)\n        waveform_extractor_name = (\n            f'{key[\"nwb_file_name\"]}_{str(uuid.uuid4())[0:8]}_'\n            f'{key[\"curation_id\"]}_clusterless_waveforms'\n        )\n        waveform_extractor_path = str(\n            Path(waveforms_dir) / Path(waveform_extractor_name)\n        )\n        if os.path.exists(waveform_extractor_path):\n            shutil.rmtree(waveform_extractor_path)\n\n        WAVEFORM_PARAMS = {\n            \"ms_before\": 0.5,\n            \"ms_after\": 0.5,\n            \"max_spikes_per_unit\": None,\n            \"n_jobs\": 5,\n            \"total_memory\": \"5G\",\n        }\n        waveform_extractor = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveform_extractor_path,\n            **WAVEFORM_PARAMS,\n        )\n\n        if mark_param[\"mark_type\"] == \"amplitude\":\n            sorter = (CuratedSpikeSorting() &amp; key).fetch1(\"sorter\")\n            if sorter == \"clusterless_thresholder\":\n                estimate_peak_time = False\n            else:\n                estimate_peak_time = True\n\n            peak_sign = mark_param[\"mark_param_dict\"].get(\"peak_sign\")\n\n            marks = np.concatenate(\n                [\n                    UnitMarks._get_peak_amplitude(\n                        waveform=waveform_extractor.get_waveforms(unit_id),\n                        peak_sign=peak_sign,\n                        estimate_peak_time=estimate_peak_time,\n                    )\n                    for unit_id in nwb_units.index\n                ],\n                axis=0,\n            )\n\n            timestamps = np.concatenate(np.asarray(nwb_units[\"spike_times\"]))\n            sorted_timestamp_ind = np.argsort(timestamps)\n            marks = marks[sorted_timestamp_ind]\n            timestamps = timestamps[sorted_timestamp_ind]\n\n        if \"threshold\" in mark_param[\"mark_param_dict\"]:\n            timestamps, marks = UnitMarks._threshold(\n                timestamps, marks, mark_param[\"mark_param_dict\"]\n            )\n\n        # create a new AnalysisNwbfile and a timeseries for the marks and save\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        nwb_object = pynwb.TimeSeries(\n            name=\"marks\",\n            data=marks,\n            unit=\"uV\",\n            timestamps=timestamps,\n            description=\"spike features for clusterless decoding\",\n        )\n        key[\"marks_object_id\"] = AnalysisNwbfile().add_nwb_object(\n            key[\"analysis_file_name\"], nwb_object\n        )\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], key[\"analysis_file_name\"])\n        self.insert1(key)\n\n    def fetch1_dataframe(self) -&gt; pd.DataFrame:\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n        return [self._convert_to_dataframe(data) for data in self.fetch_nwb()]\n\n    @staticmethod\n    def _convert_to_dataframe(nwb_data) -&gt; pd.DataFrame:\n        \"\"\"Converts the marks from an NWB object to a pandas dataframe\"\"\"\n        n_marks = nwb_data[\"marks\"].data.shape[1]\n        columns = [f\"amplitude_{ind:04d}\" for ind in range(n_marks)]\n        return pd.DataFrame(\n            nwb_data[\"marks\"].data,\n            index=pd.Index(nwb_data[\"marks\"].timestamps, name=\"time\"),\n            columns=columns,\n        )\n\n    @staticmethod\n    def _get_peak_amplitude(\n        waveform: np.array,\n        peak_sign: str = \"neg\",\n        estimate_peak_time: bool = False,\n    ) -&gt; np.array:\n        \"\"\"Returns the amplitudes of all channels at the time of the peak.\n\n        Amplitude across channels.\n\n        Parameters\n        ----------\n        waveform : np.array\n            array-like, shape (n_spikes, n_time, n_channels)\n        peak_sign : str, optional\n            One of 'pos', 'neg', 'both'. Direction of the peak in the waveform\n        estimate_peak_time : bool, optional\n            Find the peak times for each spike because some spikesorters do not\n            align the spike time (at index n_time // 2) to the peak\n\n        Returns\n        -------\n        peak_amplitudes : np.array\n            array-like, shape (n_spikes, n_channels)\n\n        \"\"\"\n        if estimate_peak_time:\n            if peak_sign == \"neg\":\n                peak_inds = np.argmin(np.min(waveform, axis=2), axis=1)\n            elif peak_sign == \"pos\":\n                peak_inds = np.argmax(np.max(waveform, axis=2), axis=1)\n            elif peak_sign == \"both\":\n                peak_inds = np.argmax(np.max(np.abs(waveform), axis=2), axis=1)\n\n            # Get mode of peaks to find the peak time\n            values, counts = np.unique(peak_inds, return_counts=True)\n            spike_peak_ind = values[counts.argmax()]\n        else:\n            spike_peak_ind = waveform.shape[1] // 2\n\n        return waveform[:, spike_peak_ind]\n\n    @staticmethod\n    def _threshold(\n        timestamps: np.array, marks: np.array, mark_param_dict: dict\n    ):\n        \"\"\"Filter the marks by an amplitude threshold\n\n        Parameters\n        ----------\n        timestamps : np.array\n            array-like, shape (n_time,)\n        marks : np.array\n            array-like, shape (n_time, n_channels)\n        mark_param_dict : dict\n\n        Returns\n        -------\n        filtered_timestamps : np.array\n            array-like, shape (n_filtered_time,)\n        filtered_marks : np.array\n            array-like, shape (n_filtered_time, n_channels)\n\n        \"\"\"\n        if mark_param_dict[\"peak_sign\"] == \"neg\":\n            include = np.min(marks, axis=1) &lt;= -1 * mark_param_dict[\"threshold\"]\n        elif mark_param_dict[\"peak_sign\"] == \"pos\":\n            include = np.max(marks, axis=1) &gt;= mark_param_dict[\"threshold\"]\n        elif mark_param_dict[\"peak_sign\"] == \"both\":\n            include = (\n                np.max(np.abs(marks), axis=1) &gt;= mark_param_dict[\"threshold\"]\n            )\n        return timestamps[include], marks[include]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarks.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1_dataframe(self) -&gt; pd.DataFrame:\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicatorSelection", "title": "<code>UnitMarksIndicatorSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Pairing of a UnitMarksIndicator with a time interval and sampling rate</p> <p>Bins the spike times and associated spike waveform features for a given time interval into regular time bins determined by the sampling rate.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarksIndicatorSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Pairing of a UnitMarksIndicator with a time interval and sampling rate\n\n    Bins the spike times and associated spike waveform features for a given\n    time interval into regular time bins determined by the sampling rate.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarks\n    -&gt; IntervalList\n    sampling_rate=500 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator", "title": "<code>UnitMarksIndicator</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Bins spike times and waveforms into regular time bins.</p> <p>Bins the spike times and associated spike waveform features into regular time bins according to the sampling rate. Features that fall into the same time bin are averaged.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarksIndicator(SpyglassMixin, dj.Computed):\n    \"\"\"Bins spike times and waveforms into regular time bins.\n\n    Bins the spike times and associated spike waveform features into regular\n    time bins according to the sampling rate. Features that fall into the same\n    time bin are averaged.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarks\n    -&gt; UnitMarksIndicatorSelection\n    ---\n    -&gt; AnalysisNwbfile\n    marks_indicator_object_id: varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        # TODO: intersection of sort interval and interval list\n        interval_times = (IntervalList &amp; key).fetch1(\"valid_times\")\n\n        sampling_rate = (UnitMarksIndicatorSelection &amp; key).fetch(\n            \"sampling_rate\"\n        )\n\n        marks_df = (UnitMarks &amp; key).fetch1_dataframe()\n\n        time = self.get_time_bins_from_interval(interval_times, sampling_rate)\n\n        # Bin marks into time bins. No spike bins will have NaN\n        marks_df = marks_df.loc[time.min() : time.max()]\n        time_index = np.digitize(marks_df.index, time[1:-1])\n        marks_indicator_df = (\n            marks_df.groupby(time[time_index])\n            .mean()\n            .reindex(index=pd.Index(time, name=\"time\"))\n        )\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(\n            key[\"nwb_file_name\"]\n        )\n\n        key[\"marks_indicator_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=marks_indicator_df.reset_index(),\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    @staticmethod\n    def get_time_bins_from_interval(\n        interval_times: np.array, sampling_rate: int\n    ) -&gt; np.array:\n        \"\"\"Picks the superset of the interval\"\"\"\n        start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n        n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n        return np.linspace(start_time, end_time, n_samples)\n\n    @staticmethod\n    def plot_all_marks(\n        marks_indicators: xr.DataArray,\n        plot_size: int = 5,\n        marker_size: int = 10,\n        plot_limit: int = None,\n    ):\n        \"\"\"Plot all marks for all electrodes.\n\n        Plots 2D slices of each of the spike features against each other\n        for all electrodes.\n\n        Parameters\n        ----------\n        marks_indicators : xr.DataArray, shape (n_time, n_electrodes, n_features)\n            Spike times and associated spike waveform features binned into\n        plot_size : int, optional\n            Default 5. Matplotlib figure size for each mark.\n        marker_size : int, optional\n            Default 10. Marker size\n        plot_limit : int, optional\n            Default None. Limits to first N electrodes.\n        \"\"\"\n        if not plot_limit:\n            plot_limit = len(marks_indicators.electrodes)\n\n        for electrode_ind in marks_indicators.electrodes[:plot_limit]:\n            marks = (\n                marks_indicators.sel(electrodes=electrode_ind)\n                .dropna(\"time\", how=\"all\")\n                .dropna(\"marks\")\n            )\n            n_features = len(marks.marks)\n            fig, axes = plt.subplots(\n                n_features,\n                n_features,\n                constrained_layout=True,\n                sharex=True,\n                sharey=True,\n                figsize=(plot_size * n_features, plot_size * n_features),\n            )\n            for ax_ind1, feature1 in enumerate(marks.marks):\n                for ax_ind2, feature2 in enumerate(marks.marks):\n                    try:\n                        axes[ax_ind1, ax_ind2].scatter(\n                            marks.sel(marks=feature1),\n                            marks.sel(marks=feature2),\n                            s=marker_size,\n                        )\n                    except TypeError:\n                        axes.scatter(\n                            marks.sel(marks=feature1),\n                            marks.sel(marks=feature2),\n                            s=marker_size,\n                        )\n\n    def fetch1_dataframe(self) -&gt; pd.DataFrame:\n        \"\"\"Convenience function for returning the first dataframe\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n        \"\"\"Fetches the marks indicators as a list of pandas dataframes\"\"\"\n        return [\n            data[\"marks_indicator\"].set_index(\"time\")\n            for data in self.fetch_nwb()\n        ]\n\n    def fetch_xarray(self):\n        \"\"\"Fetches the marks indicators as an xarray DataArray\"\"\"\n        # sort_group_electrodes = (\n        #     SortGroup.SortGroupElectrode() &amp;\n        #     pd.DataFrame(self).to_dict('records'))\n        # brain_region = (sort_group_electrodes * Electrode *\n        #                 BrainRegion).fetch('region_name')\n\n        marks_indicators = (\n            xr.concat(\n                [\n                    df.to_xarray().to_array(\"marks\")\n                    for df in self.fetch_dataframe()\n                ],\n                dim=\"electrodes\",\n            )\n            .transpose(\"time\", \"marks\", \"electrodes\")\n            .assign_coords({\"electrodes\": self.fetch(\"sort_group_id\")})\n            .sortby([\"electrodes\", \"marks\"])\n        )\n\n        # hacky way to keep the marks in order\n        def reformat_name(name):\n            mark_type, number = name.split(\"_\")\n            return f\"{mark_type}_{int(number):04d}\"\n\n        new_mark_names = [\n            reformat_name(name) for name in marks_indicators.marks.values\n        ]\n\n        return marks_indicators.assign_coords({\"marks\": new_mark_names}).sortby(\n            [\"electrodes\", \"marks\"]\n        )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.get_time_bins_from_interval", "title": "<code>get_time_bins_from_interval(interval_times, sampling_rate)</code>  <code>staticmethod</code>", "text": "<p>Picks the superset of the interval</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef get_time_bins_from_interval(\n    interval_times: np.array, sampling_rate: int\n) -&gt; np.array:\n    \"\"\"Picks the superset of the interval\"\"\"\n    start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n    return np.linspace(start_time, end_time, n_samples)\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.plot_all_marks", "title": "<code>plot_all_marks(marks_indicators, plot_size=5, marker_size=10, plot_limit=None)</code>  <code>staticmethod</code>", "text": "<p>Plot all marks for all electrodes.</p> <p>Plots 2D slices of each of the spike features against each other for all electrodes.</p> <p>Parameters:</p> Name Type Description Default <code>marks_indicators</code> <code>(DataArray, shape(n_time, n_electrodes, n_features))</code> <p>Spike times and associated spike waveform features binned into</p> required <code>plot_size</code> <code>int</code> <p>Default 5. Matplotlib figure size for each mark.</p> <code>5</code> <code>marker_size</code> <code>int</code> <p>Default 10. Marker size</p> <code>10</code> <code>plot_limit</code> <code>int</code> <p>Default None. Limits to first N electrodes.</p> <code>None</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef plot_all_marks(\n    marks_indicators: xr.DataArray,\n    plot_size: int = 5,\n    marker_size: int = 10,\n    plot_limit: int = None,\n):\n    \"\"\"Plot all marks for all electrodes.\n\n    Plots 2D slices of each of the spike features against each other\n    for all electrodes.\n\n    Parameters\n    ----------\n    marks_indicators : xr.DataArray, shape (n_time, n_electrodes, n_features)\n        Spike times and associated spike waveform features binned into\n    plot_size : int, optional\n        Default 5. Matplotlib figure size for each mark.\n    marker_size : int, optional\n        Default 10. Marker size\n    plot_limit : int, optional\n        Default None. Limits to first N electrodes.\n    \"\"\"\n    if not plot_limit:\n        plot_limit = len(marks_indicators.electrodes)\n\n    for electrode_ind in marks_indicators.electrodes[:plot_limit]:\n        marks = (\n            marks_indicators.sel(electrodes=electrode_ind)\n            .dropna(\"time\", how=\"all\")\n            .dropna(\"marks\")\n        )\n        n_features = len(marks.marks)\n        fig, axes = plt.subplots(\n            n_features,\n            n_features,\n            constrained_layout=True,\n            sharex=True,\n            sharey=True,\n            figsize=(plot_size * n_features, plot_size * n_features),\n        )\n        for ax_ind1, feature1 in enumerate(marks.marks):\n            for ax_ind2, feature2 in enumerate(marks.marks):\n                try:\n                    axes[ax_ind1, ax_ind2].scatter(\n                        marks.sel(marks=feature1),\n                        marks.sel(marks=feature2),\n                        s=marker_size,\n                    )\n                except TypeError:\n                    axes.scatter(\n                        marks.sel(marks=feature1),\n                        marks.sel(marks=feature2),\n                        s=marker_size,\n                    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the first dataframe</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1_dataframe(self) -&gt; pd.DataFrame:\n    \"\"\"Convenience function for returning the first dataframe\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch_dataframe", "title": "<code>fetch_dataframe()</code>", "text": "<p>Fetches the marks indicators as a list of pandas dataframes</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n    \"\"\"Fetches the marks indicators as a list of pandas dataframes\"\"\"\n    return [\n        data[\"marks_indicator\"].set_index(\"time\")\n        for data in self.fetch_nwb()\n    ]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch_xarray", "title": "<code>fetch_xarray()</code>", "text": "<p>Fetches the marks indicators as an xarray DataArray</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch_xarray(self):\n    \"\"\"Fetches the marks indicators as an xarray DataArray\"\"\"\n    # sort_group_electrodes = (\n    #     SortGroup.SortGroupElectrode() &amp;\n    #     pd.DataFrame(self).to_dict('records'))\n    # brain_region = (sort_group_electrodes * Electrode *\n    #                 BrainRegion).fetch('region_name')\n\n    marks_indicators = (\n        xr.concat(\n            [\n                df.to_xarray().to_array(\"marks\")\n                for df in self.fetch_dataframe()\n            ],\n            dim=\"electrodes\",\n        )\n        .transpose(\"time\", \"marks\", \"electrodes\")\n        .assign_coords({\"electrodes\": self.fetch(\"sort_group_id\")})\n        .sortby([\"electrodes\", \"marks\"])\n    )\n\n    # hacky way to keep the marks in order\n    def reformat_name(name):\n        mark_type, number = name.split(\"_\")\n        return f\"{mark_type}_{int(number):04d}\"\n\n    new_mark_names = [\n        reformat_name(name) for name in marks_indicators.marks.values\n    ]\n\n    return marks_indicators.assign_coords({\"marks\": new_mark_names}).sortby(\n        [\"electrodes\", \"marks\"]\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.make_default_decoding_parameters_cpu", "title": "<code>make_default_decoding_parameters_cpu()</code>", "text": "<p>Default parameters for decoding on CPU</p> <p>Returns:</p> Name Type Description <code>classifier_parameters</code> <code>dict</code> <code>fit_parameters</code> <code>dict</code> <code>predict_parameters</code> <code>dict</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def make_default_decoding_parameters_cpu() -&gt; tuple[dict, dict, dict]:\n    \"\"\"Default parameters for decoding on CPU\n\n    Returns\n    -------\n    classifier_parameters : dict\n    fit_parameters : dict\n    predict_parameters : dict\n    \"\"\"\n\n    classifier_parameters = dict(\n        environments=[_DEFAULT_ENVIRONMENT],\n        observation_models=None,\n        continuous_transition_types=_DEFAULT_CONTINUOUS_TRANSITIONS,\n        discrete_transition_type=DiagonalDiscrete(0.98),\n        initial_conditions_type=UniformInitialConditions(),\n        infer_track_interior=True,\n        clusterless_algorithm=\"multiunit_likelihood_integer\",\n        clusterless_algorithm_params=_DEFAULT_CLUSTERLESS_MODEL_KWARGS,\n    )\n\n    predict_parameters = {\n        \"is_compute_acausal\": True,\n        \"use_gpu\": False,\n        \"state_names\": [\"Continuous\", \"Fragmented\"],\n    }\n    fit_parameters = dict()\n\n    return classifier_parameters, fit_parameters, predict_parameters\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.make_default_decoding_parameters_gpu", "title": "<code>make_default_decoding_parameters_gpu()</code>", "text": "<p>Default parameters for decoding on GPU</p> <p>Returns:</p> Name Type Description <code>classifier_parameters</code> <code>dict</code> <code>fit_parameters</code> <code>dict</code> <code>predict_parameters</code> <code>dict</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def make_default_decoding_parameters_gpu() -&gt; tuple[dict, dict, dict]:\n    \"\"\"Default parameters for decoding on GPU\n\n    Returns\n    -------\n    classifier_parameters : dict\n    fit_parameters : dict\n    predict_parameters : dict\n    \"\"\"\n\n    classifier_parameters = dict(\n        environments=[_DEFAULT_ENVIRONMENT],\n        observation_models=None,\n        continuous_transition_types=_DEFAULT_CONTINUOUS_TRANSITIONS,\n        discrete_transition_type=DiagonalDiscrete(0.98),\n        initial_conditions_type=UniformInitialConditions(),\n        infer_track_interior=True,\n        clusterless_algorithm=\"multiunit_likelihood_integer_gpu\",\n        clusterless_algorithm_params={\n            \"mark_std\": 24.0,\n            \"position_std\": 6.0,\n        },\n    )\n\n    predict_parameters = {\n        \"is_compute_acausal\": True,\n        \"use_gpu\": True,\n        \"state_names\": [\"Continuous\", \"Fragmented\"],\n    }\n\n    fit_parameters = dict()\n\n    return classifier_parameters, fit_parameters, predict_parameters\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters", "title": "<code>ClusterlessClassifierParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Decodes animal's mental position.</p> <p>Decodes the animal's mental position and some category of interest from unclustered spikes and spike waveform features</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass ClusterlessClassifierParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Decodes animal's mental position.\n\n    Decodes the animal's mental position and some category of interest\n    from unclustered spikes and spike waveform features\n    \"\"\"\n\n    definition = \"\"\"\n    classifier_param_name : varchar(80) # a name for this set of parameters\n    ---\n    classifier_params :   BLOB    # initialization parameters\n    fit_params :          BLOB    # fit parameters\n    predict_params :      BLOB    # prediction parameters\n    \"\"\"\n\n    def insert_default(self) -&gt; None:\n        \"\"\"Insert the default parameter set\"\"\"\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_cpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_cpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_gpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_gpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n    def insert1(self, key, **kwargs) -&gt; None:\n        \"\"\"Custom insert1 to convert classes to dicts\"\"\"\n        super().insert1(convert_classes_to_dict(key), **kwargs)\n\n    def fetch1(self, *args, **kwargs) -&gt; dict:\n        \"\"\"Custom fetch1 to convert dicts to classes\"\"\"\n        return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert_default(self) -&gt; None:\n    \"\"\"Insert the default parameter set\"\"\"\n    (\n        classifier_parameters,\n        fit_parameters,\n        predict_parameters,\n    ) = make_default_decoding_parameters_cpu()\n    self.insert1(\n        {\n            \"classifier_param_name\": \"default_decoding_cpu\",\n            \"classifier_params\": classifier_parameters,\n            \"fit_params\": fit_parameters,\n            \"predict_params\": predict_parameters,\n        },\n        skip_duplicates=True,\n    )\n\n    (\n        classifier_parameters,\n        fit_parameters,\n        predict_parameters,\n    ) = make_default_decoding_parameters_gpu()\n    self.insert1(\n        {\n            \"classifier_param_name\": \"default_decoding_gpu\",\n            \"classifier_params\": classifier_parameters,\n            \"fit_params\": fit_parameters,\n            \"predict_params\": predict_parameters,\n        },\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.insert1", "title": "<code>insert1(key, **kwargs)</code>", "text": "<p>Custom insert1 to convert classes to dicts</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert1(self, key, **kwargs) -&gt; None:\n    \"\"\"Custom insert1 to convert classes to dicts\"\"\"\n    super().insert1(convert_classes_to_dict(key), **kwargs)\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.fetch1", "title": "<code>fetch1(*args, **kwargs)</code>", "text": "<p>Custom fetch1 to convert dicts to classes</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1(self, *args, **kwargs) -&gt; dict:\n    \"\"\"Custom fetch1 to convert dicts to classes\"\"\"\n    return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.get_decoding_data_for_epoch", "title": "<code>get_decoding_data_for_epoch(nwb_file_name, interval_list_name, position_info_param_name='default_decoding', additional_mark_keys={})</code>", "text": "<p>Collects necessary data for decoding.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>interval_list_name</code> <code>str</code> required <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> <code>additional_mark_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_columns))</code> <code>marks</code> <code>(DataArray, shape(n_time, n_marks, n_electrodes))</code> <code>valid_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def get_decoding_data_for_epoch(\n    nwb_file_name: str,\n    interval_list_name: str,\n    position_info_param_name: str = \"default_decoding\",\n    additional_mark_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, xr.DataArray, list[slice]]:\n    \"\"\"Collects necessary data for decoding.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    interval_list_name : str\n    position_info_param_name : str, optional\n    additional_mark_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_columns)\n    marks : xr.DataArray, shape (n_time, n_marks, n_electrodes)\n    valid_slices : list[slice]\n    \"\"\"\n\n    valid_ephys_position_times_by_epoch = (\n        get_valid_ephys_position_times_by_epoch(nwb_file_name)\n    )\n    valid_ephys_position_times = valid_ephys_position_times_by_epoch[\n        interval_list_name\n    ]\n    valid_slices = convert_valid_times_to_slice(valid_ephys_position_times)\n    position_interval_name = (\n        convert_epoch_interval_name_to_position_interval_name(\n            {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        )\n    )\n\n    position_info = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": position_interval_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch1_dataframe()\n\n    position_info = pd.concat(\n        [position_info.loc[times] for times in valid_slices]\n    )\n\n    marks = (\n        (\n            UnitMarksIndicator()\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": position_interval_name,\n                **additional_mark_keys,\n            }\n        )\n    ).fetch_xarray()\n\n    marks = xr.concat(\n        [marks.sel(time=times) for times in valid_slices], dim=\"time\"\n    )\n\n    return position_info, marks, valid_slices\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.get_data_for_multiple_epochs", "title": "<code>get_data_for_multiple_epochs(nwb_file_name, epoch_names, position_info_param_name='default_decoding', additional_mark_keys={})</code>", "text": "<p>Collects necessary data for decoding multiple environments</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>epoch_names</code> <code>list[str]</code> required <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> <code>additional_mark_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_columns))</code> <code>marks</code> <code>(DataArray, shape(n_time, n_marks, n_electrodes))</code> <code>valid_slices</code> <code>dict[str, list[slice]]</code> <code>environment_labels</code> <code>(ndarray, shape(n_time))</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def get_data_for_multiple_epochs(\n    nwb_file_name: str,\n    epoch_names: list[str],\n    position_info_param_name=\"default_decoding\",\n    additional_mark_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, xr.DataArray, dict[str, list[slice]], np.ndarray]:\n    \"\"\"Collects necessary data for decoding multiple environments\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    epoch_names : list[str]\n    position_info_param_name : str, optional\n    additional_mark_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_columns)\n    marks : xr.DataArray, shape (n_time, n_marks, n_electrodes)\n    valid_slices : dict[str, list[slice]]\n    environment_labels : np.ndarray, shape (n_time,)\n    \"\"\"\n    data = []\n    environment_labels = []\n\n    for epoch in epoch_names:\n        data.append(\n            get_decoding_data_for_epoch(\n                nwb_file_name,\n                epoch,\n                position_info_param_name=position_info_param_name,\n                additional_mark_keys=additional_mark_keys,\n            )\n        )\n        n_time = data[-1][0].shape[0]\n        environment_labels.append([epoch] * n_time)\n\n    environment_labels = np.concatenate(environment_labels, axis=0)\n    position_info, marks, valid_slices = list(zip(*data))\n    position_info = pd.concat(position_info, axis=0)\n    marks = xr.concat(marks, dim=\"time\")\n    valid_slices = {\n        epoch: valid_slice\n        for epoch, valid_slice in zip(epoch_names, valid_slices)\n    }\n\n    assert position_info.shape[0] == marks.shape[0]\n\n    return position_info, marks, valid_slices, environment_labels\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.populate_mark_indicators", "title": "<code>populate_mark_indicators(spikesorting_selection_keys, mark_param_name='default', position_info_param_name='default_decoding')</code>", "text": "<p>Populate mark indicators</p> <p>Populates for all units in a given spike sorting selection.</p> <p>This function is a way to do several pipeline steps at once. It will: 1. Populate the SpikeSortingSelection table 2. Populate the SpikeSorting table 3. Populate the Curation table 4. Populate the CuratedSpikeSortingSelection table 5. Populate UnitMarks 6. Compute UnitMarksIndicator for each position epoch</p> <p>Parameters:</p> Name Type Description Default <code>spikesorting_selection_keys</code> <code>dict</code> required <code>mark_param_name</code> <code>str</code> <code>'default'</code> <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def populate_mark_indicators(\n    spikesorting_selection_keys: dict,\n    mark_param_name: str = \"default\",\n    position_info_param_name: str = \"default_decoding\",\n):\n    \"\"\"Populate mark indicators\n\n    Populates for all units in a given spike sorting selection.\n\n    This function is a way to do several pipeline steps at once. It will:\n    1. Populate the SpikeSortingSelection table\n    2. Populate the SpikeSorting table\n    3. Populate the Curation table\n    4. Populate the CuratedSpikeSortingSelection table\n    5. Populate UnitMarks\n    6. Compute UnitMarksIndicator for each position epoch\n\n    Parameters\n    ----------\n    spikesorting_selection_keys : dict\n    mark_param_name : str, optional\n    position_info_param_name : str, optional\n    \"\"\"\n    spikesorting_selection_keys = deepcopy(spikesorting_selection_keys)\n    # Populate spike sorting\n    SpikeSortingSelection().insert(\n        spikesorting_selection_keys,\n        skip_duplicates=True,\n    )\n    SpikeSorting.populate(spikesorting_selection_keys)\n\n    # Skip any curation\n    curation_keys = [\n        Curation.insert_curation(key) for key in spikesorting_selection_keys\n    ]\n\n    CuratedSpikeSortingSelection().insert(curation_keys, skip_duplicates=True)\n    CuratedSpikeSorting.populate(CuratedSpikeSortingSelection() &amp; curation_keys)\n\n    # Populate marks\n    mark_parameters_keys = pd.DataFrame(CuratedSpikeSorting &amp; curation_keys)\n    mark_parameters_keys[\"mark_param_name\"] = mark_param_name\n    mark_parameters_keys = mark_parameters_keys.loc[\n        :, UnitMarkParameters.primary_key\n    ].to_dict(\"records\")\n    UnitMarkParameters().insert(mark_parameters_keys, skip_duplicates=True)\n    UnitMarks.populate(UnitMarkParameters &amp; mark_parameters_keys)\n\n    # Compute mark indicators for each position epoch\n    nwb_file_name = spikesorting_selection_keys[0][\"nwb_file_name\"]\n    position_interval_names = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch(\"interval_list_name\")\n\n    for interval_name in tqdm(position_interval_names):\n        position_interval = IntervalList &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": interval_name,\n        }\n\n        marks_selection = (UnitMarks &amp; mark_parameters_keys) * position_interval\n        marks_selection = (\n            pd.DataFrame(marks_selection)\n            .loc[:, marks_selection.primary_key]\n            .to_dict(\"records\")\n        )\n        UnitMarksIndicatorSelection.insert(\n            marks_selection, skip_duplicates=True\n        )\n        UnitMarksIndicator.populate(marks_selection)\n</code></pre>"}, {"location": "api/decoding/v0/core/", "title": "core.py", "text": ""}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_valid_ephys_position_times_from_interval", "title": "<code>get_valid_ephys_position_times_from_interval(interval_list_name, nwb_file_name)</code>", "text": "<p>Finds the intersection of the valid times for the interval list, the valid times for the ephys data, and the valid times for the position data.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list_name</code> <code>str</code> required <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>valid_ephys_position_times</code> <code>(ndarray, shape(n_valid_times, 2))</code> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_valid_ephys_position_times_from_interval(\n    interval_list_name: str, nwb_file_name: str\n) -&gt; np.ndarray:\n    \"\"\"Finds the intersection of the valid times for the interval list, the valid times for the ephys data,\n    and the valid times for the position data.\n\n    Parameters\n    ----------\n    interval_list_name : str\n    nwb_file_name : str\n\n    Returns\n    -------\n    valid_ephys_position_times : np.ndarray, shape (n_valid_times, 2)\n\n    \"\"\"\n    interval_valid_times = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": interval_list_name,\n        }\n    ).fetch1(\"valid_times\")\n\n    position_interval_names = (\n        RawPosition\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n        }\n    ).fetch(\"interval_list_name\")\n    position_interval_names = position_interval_names[\n        np.argsort(\n            [\n                int(name.strip(\"pos valid time\"))\n                for name in position_interval_names\n            ]\n        )\n    ]\n    valid_pos_times = [\n        (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": pos_interval_name,\n            }\n        ).fetch1(\"valid_times\")\n        for pos_interval_name in position_interval_names\n    ]\n\n    valid_ephys_times = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": \"raw data valid times\",\n        }\n    ).fetch1(\"valid_times\")\n\n    return interval_list_intersect(\n        interval_list_intersect(interval_valid_times, valid_ephys_times),\n        np.concatenate(valid_pos_times),\n    )\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_epoch_interval_names", "title": "<code>get_epoch_interval_names(nwb_file_name)</code>", "text": "<p>Find the interval names that are epochs.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>epoch_names</code> <code>list[str]</code> <p>List of interval names that are epochs.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_epoch_interval_names(nwb_file_name: str) -&gt; list[str]:\n    \"\"\"Find the interval names that are epochs.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n\n    Returns\n    -------\n    epoch_names : list[str]\n        List of interval names that are epochs.\n    \"\"\"\n    interval_list = pd.DataFrame(\n        IntervalList() &amp; {\"nwb_file_name\": nwb_file_name}\n    )\n\n    interval_list = interval_list.loc[\n        interval_list.interval_list_name.str.contains(\n            r\"^(?:\\d+)_(?:\\w+)$\", regex=True, na=False\n        )\n    ]\n\n    return interval_list.interval_list_name.tolist()\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_valid_ephys_position_times_by_epoch", "title": "<code>get_valid_ephys_position_times_by_epoch(nwb_file_name)</code>", "text": "<p>Get the valid ephys position times for each epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>valid_ephys_position_times_by_epoch</code> <code>dict[str, ndarray]</code> <p>Dictionary of epoch names and valid ephys position times.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_valid_ephys_position_times_by_epoch(\n    nwb_file_name: str,\n) -&gt; dict[str, np.ndarray]:\n    \"\"\"Get the valid ephys position times for each epoch.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n\n    Returns\n    -------\n    valid_ephys_position_times_by_epoch : dict[str, np.ndarray]\n        Dictionary of epoch names and valid ephys position times.\n\n    \"\"\"\n    return {\n        epoch: get_valid_ephys_position_times_from_interval(\n            epoch, nwb_file_name\n        )\n        for epoch in get_epoch_interval_names(nwb_file_name)\n    }\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.convert_valid_times_to_slice", "title": "<code>convert_valid_times_to_slice(valid_times)</code>", "text": "<p>Converts the valid times to a list of slices so that arrays can be indexed easily.</p> <p>Parameters:</p> Name Type Description Default <code>valid_times</code> <code>(ndarray, shape(n_valid_times, 2))</code> <p>Start and end times for each valid time.</p> required <p>Returns:</p> Name Type Description <code>valid_time_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def convert_valid_times_to_slice(valid_times: np.ndarray) -&gt; list[slice]:\n    \"\"\"Converts the valid times to a list of slices so that arrays can be indexed easily.\n\n    Parameters\n    ----------\n    valid_times : np.ndarray, shape (n_valid_times, 2)\n        Start and end times for each valid time.\n\n    Returns\n    -------\n    valid_time_slices : list[slice]\n\n    \"\"\"\n    return [slice(times[0], times[1]) for times in valid_times]\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.create_model_for_multiple_epochs", "title": "<code>create_model_for_multiple_epochs(epoch_names, env_kwargs)</code>", "text": "<p>Creates the observation model, environment, and continuous transition types for multiple epochs for decoding</p> <p>Parameters:</p> Name Type Description Default <code>epoch_names</code> <code>(list[str], length(n_epochs))</code> required <code>env_kwargs</code> <code>dict</code> <p>Environment keyword arguments.</p> required <p>Returns:</p> Name Type Description <code>observation_models</code> <code>tuple[list[ObservationModel]</code> <p>Observation model for each epoch.</p> <code>environments</code> <code>list[Environment]</code> <p>Environment for each epoch.</p> <code>continuous_transition_types</code> <code>list[list[object]]]</code> <p>Continuous transition types for each epoch.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def create_model_for_multiple_epochs(\n    epoch_names: list[str], env_kwargs: dict\n) -&gt; tuple[list[ObservationModel], list[Environment], list[list[object]]]:\n    \"\"\"Creates the observation model, environment, and continuous transition types for multiple epochs for decoding\n\n    Parameters\n    ----------\n    epoch_names : list[str], length (n_epochs)\n    env_kwargs : dict\n        Environment keyword arguments.\n\n    Returns\n    -------\n    observation_models: tuple[list[ObservationModel]\n        Observation model for each epoch.\n    environments : list[Environment]\n        Environment for each epoch.\n    continuous_transition_types : list[list[object]]]\n        Continuous transition types for each epoch.\n\n    \"\"\"\n    observation_models = []\n    environments = []\n    continuous_transition_types = []\n\n    for epoch in epoch_names:\n        observation_models.append(ObservationModel(epoch))\n        environments.append(Environment(epoch, **env_kwargs))\n\n    for epoch1 in epoch_names:\n        continuous_transition_types.append([])\n        for epoch2 in epoch_names:\n            if epoch1 == epoch2:\n                continuous_transition_types[-1].append(\n                    RandomWalk(epoch1, use_diffusion=False)\n                )\n            else:\n                continuous_transition_types[-1].append(Uniform(epoch1, epoch2))\n\n    return observation_models, environments, continuous_transition_types\n</code></pre>"}, {"location": "api/decoding/v0/dj_decoder_conversion/", "title": "dj_decoder_conversion.py", "text": "<p>Converts decoder classes into dictionaries and dictionaries into classes so that datajoint can store them in tables.</p>"}, {"location": "api/decoding/v0/dj_decoder_conversion/#src.spyglass.decoding.v0.dj_decoder_conversion.restore_classes", "title": "<code>restore_classes(params)</code>", "text": "<p>Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes</p> Source code in <code>src/spyglass/decoding/v0/dj_decoder_conversion.py</code> <pre><code>def restore_classes(params: dict) -&gt; dict:\n    \"\"\"Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes\"\"\"\n    continuous_state_transition_types = {\n        \"RandomWalk\": RandomWalk,\n        \"RandomWalkDirection1\": RandomWalkDirection1,\n        \"RandomWalkDirection2\": RandomWalkDirection2,\n        \"Uniform\": Uniform,\n        \"Identity\": Identity,\n    }\n\n    discrete_state_transition_types = {\n        \"DiagonalDiscrete\": DiagonalDiscrete,\n        \"UniformDiscrete\": UniformDiscrete,\n        \"RandomDiscrete\": RandomDiscrete,\n        \"UserDefinedDiscrete\": UserDefinedDiscrete,\n    }\n\n    initial_conditions_types = {\n        \"UniformInitialConditions\": UniformInitialConditions,\n        \"UniformOneEnvironmentInitialConditions\": UniformOneEnvironmentInitialConditions,\n    }\n\n    params[\"classifier_params\"][\"continuous_transition_types\"] = [\n        [\n            _convert_dict_to_class(st, continuous_state_transition_types)\n            for st in sts\n        ]\n        for sts in params[\"classifier_params\"][\"continuous_transition_types\"]\n    ]\n    params[\"classifier_params\"][\"environments\"] = [\n        _convert_env_dict(env_params)\n        for env_params in params[\"classifier_params\"][\"environments\"]\n    ]\n    params[\"classifier_params\"][\"discrete_transition_type\"] = (\n        _convert_dict_to_class(\n            params[\"classifier_params\"][\"discrete_transition_type\"],\n            discrete_state_transition_types,\n        )\n    )\n    params[\"classifier_params\"][\"initial_conditions_type\"] = (\n        _convert_dict_to_class(\n            params[\"classifier_params\"][\"initial_conditions_type\"],\n            initial_conditions_types,\n        )\n    )\n\n    if params[\"classifier_params\"].get(\"observation_models\"):\n        params[\"classifier_params\"][\"observation_models\"] = [\n            ObservationModel(obs)\n            for obs in params[\"classifier_params\"][\"observation_models\"]\n        ]\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v0/dj_decoder_conversion/#src.spyglass.decoding.v0.dj_decoder_conversion.convert_classes_to_dict", "title": "<code>convert_classes_to_dict(key)</code>", "text": "<p>Converts the classifier parameters into a dictionary so that datajoint can store it.</p> Source code in <code>src/spyglass/decoding/v0/dj_decoder_conversion.py</code> <pre><code>def convert_classes_to_dict(key: dict) -&gt; dict:\n    \"\"\"Converts the classifier parameters into a dictionary so that datajoint can store it.\"\"\"\n    try:\n        key[\"classifier_params\"][\"environments\"] = [\n            _convert_environment_to_dict(env)\n            for env in key[\"classifier_params\"][\"environments\"]\n        ]\n    except TypeError:\n        key[\"classifier_params\"][\"environments\"] = [\n            _convert_environment_to_dict(\n                key[\"classifier_params\"][\"environments\"]\n            )\n        ]\n    key[\"classifier_params\"][\"continuous_transition_types\"] = (\n        _convert_transitions_to_dict(\n            key[\"classifier_params\"][\"continuous_transition_types\"]\n        )\n    )\n    key[\"classifier_params\"][\"discrete_transition_type\"] = _to_dict(\n        key[\"classifier_params\"][\"discrete_transition_type\"]\n    )\n    key[\"classifier_params\"][\"initial_conditions_type\"] = _to_dict(\n        key[\"classifier_params\"][\"initial_conditions_type\"]\n    )\n\n    if key[\"classifier_params\"][\"observation_models\"] is not None:\n        key[\"classifier_params\"][\"observation_models\"] = [\n            vars(obs) for obs in key[\"classifier_params\"][\"observation_models\"]\n        ]\n\n    try:\n        key[\"classifier_params\"][\"clusterless_algorithm_params\"] = (\n            _convert_algorithm_params(\n                key[\"classifier_params\"][\"clusterless_algorithm_params\"]\n            )\n        )\n    except KeyError:\n        pass\n\n    return key\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/", "title": "sorted_spikes.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from clustered spikes times. See [1] for details.</p>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicatorSelection", "title": "<code>SortedSpikesIndicatorSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Bins spike times into regular intervals given by the sampling rate. Start and stop time of the interval are defined by the interval list.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesIndicatorSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Bins spike times into regular intervals given by the sampling rate.\n    Start and stop time of the interval are defined by the interval list.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; CuratedSpikeSorting\n    -&gt; IntervalList\n    sampling_rate=500 : float\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicator", "title": "<code>SortedSpikesIndicator</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Bins spike times into regular intervals given by the sampling rate. Useful for GLMs and for decoding.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesIndicator(SpyglassMixin, dj.Computed):\n    \"\"\"Bins spike times into regular intervals given by the sampling rate.\n    Useful for GLMs and for decoding.\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; SortedSpikesIndicatorSelection\n    ---\n    -&gt; AnalysisNwbfile\n    spike_indicator_object_id: varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        pprint.pprint(key)\n        # TODO: intersection of sort interval and interval list\n        interval_times = (IntervalList &amp; key).fetch1(\"valid_times\")\n\n        sampling_rate = (SortedSpikesIndicatorSelection &amp; key).fetch(\n            \"sampling_rate\"\n        )\n\n        time = self.get_time_bins_from_interval(interval_times, sampling_rate)\n\n        spikes_nwb = (CuratedSpikeSorting &amp; key).fetch_nwb()\n        # restrict to cases with units\n        spikes_nwb = [entry for entry in spikes_nwb if \"units\" in entry]\n        spike_times_list = [\n            np.asarray(n_trode[\"units\"][\"spike_times\"])\n            for n_trode in spikes_nwb\n        ]\n        if len(spike_times_list) &gt; 0:  # if units\n            spikes = np.concatenate(spike_times_list)\n\n            # Bin spikes into time bins\n            spike_indicator = []\n            for spike_times in spikes:\n                spike_times = spike_times[\n                    (spike_times &gt; time[0]) &amp; (spike_times &lt;= time[-1])\n                ]\n                spike_indicator.append(\n                    np.bincount(\n                        np.digitize(spike_times, time[1:-1]),\n                        minlength=time.shape[0],\n                    )\n                )\n\n            column_names = np.concatenate(\n                [\n                    [\n                        f'{n_trode[\"sort_group_id\"]:04d}_{unit_number:04d}'\n                        for unit_number in n_trode[\"units\"].index\n                    ]\n                    for n_trode in spikes_nwb\n                ]\n            )\n            spike_indicator = pd.DataFrame(\n                np.stack(spike_indicator, axis=1),\n                index=pd.Index(time, name=\"time\"),\n                columns=column_names,\n            )\n\n            # Insert into analysis nwb file\n            nwb_analysis_file = AnalysisNwbfile()\n            key[\"analysis_file_name\"] = nwb_analysis_file.create(\n                key[\"nwb_file_name\"]\n            )\n\n            key[\"spike_indicator_object_id\"] = nwb_analysis_file.add_nwb_object(\n                analysis_file_name=key[\"analysis_file_name\"],\n                nwb_object=spike_indicator.reset_index(),\n            )\n\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n\n            self.insert1(key)\n\n    @staticmethod\n    def get_time_bins_from_interval(interval_times, sampling_rate):\n        \"\"\"Gets the superset of the interval.\"\"\"\n        start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n        n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n        return np.linspace(start_time, end_time, n_samples)\n\n    def fetch1_dataframe(self):\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return pd.concat(\n            [\n                data[\"spike_indicator\"].set_index(\"time\")\n                for data in self.fetch_nwb()\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicator.get_time_bins_from_interval", "title": "<code>get_time_bins_from_interval(interval_times, sampling_rate)</code>  <code>staticmethod</code>", "text": "<p>Gets the superset of the interval.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@staticmethod\ndef get_time_bins_from_interval(interval_times, sampling_rate):\n    \"\"\"Gets the superset of the interval.\"\"\"\n    start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n    return np.linspace(start_time, end_time, n_samples)\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesClassifierParameters", "title": "<code>SortedSpikesClassifierParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Stores parameters for decoding with sorted spikes</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesClassifierParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Stores parameters for decoding with sorted spikes\"\"\"\n\n    definition = \"\"\"\n    classifier_param_name : varchar(80) # a name for this set of parameters\n    ---\n    classifier_params :   BLOB    # initialization parameters\n    fit_params :          BLOB    # fit parameters\n    predict_params :      BLOB    # prediction parameters\n    \"\"\"\n\n    def insert_default(self):\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_cpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_cpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_gpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_gpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n    def insert1(self, key, **kwargs):\n        super().insert1(convert_classes_to_dict(key), **kwargs)\n\n    def fetch1(self, *args, **kwargs):\n        return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_spike_indicator", "title": "<code>get_spike_indicator(key, time_range, sampling_rate=500.0)</code>", "text": "<p>For a given key, returns a dataframe with the spike indicator for each unit</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> required <code>time_range</code> <code>tuple[float, float]</code> <p>Start and end time of the spike indicator</p> required <code>sampling_rate</code> <code>float</code> <code>500.0</code> <p>Returns:</p> Name Type Description <code>spike_indicator</code> <code>(DataFrame, shape(n_time, n_units))</code> <p>A dataframe with the spike indicator for each unit</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_spike_indicator(\n    key: dict, time_range: tuple[float, float], sampling_rate: float = 500.0\n) -&gt; pd.DataFrame:\n    \"\"\"For a given key, returns a dataframe with the spike indicator for each unit\n\n    Parameters\n    ----------\n    key : dict\n    time_range : tuple[float, float]\n        Start and end time of the spike indicator\n    sampling_rate : float, optional\n\n    Returns\n    -------\n    spike_indicator : pd.DataFrame, shape (n_time, n_units)\n        A dataframe with the spike indicator for each unit\n    \"\"\"\n    start_time, end_time = time_range\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n    time = np.linspace(start_time, end_time, n_samples)\n\n    spike_indicator = dict()\n    spikes_nwb_table = CuratedSpikeSorting() &amp; key\n\n    for n_trode in spikes_nwb_table.fetch_nwb():\n        try:\n            for unit_id, unit_spike_times in n_trode[\"units\"][\n                \"spike_times\"\n            ].items():\n                unit_spike_times = unit_spike_times[\n                    (unit_spike_times &gt; time[0])\n                    &amp; (unit_spike_times &lt;= time[-1])\n                ]\n                unit_name = f'{n_trode[\"sort_group_id\"]:04d}_{unit_id:04d}'\n                spike_indicator[unit_name] = np.bincount(\n                    np.digitize(unit_spike_times, time[1:-1]),\n                    minlength=time.shape[0],\n                )\n        except KeyError:\n            pass\n\n    return pd.DataFrame(\n        spike_indicator,\n        index=pd.Index(time, name=\"time\"),\n    )\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_decoding_data_for_epoch", "title": "<code>get_decoding_data_for_epoch(nwb_file_name, interval_list_name, position_info_param_name='default', additional_spike_keys={})</code>", "text": "<p>Collects the data needed for decoding</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>interval_list_name</code> <code>str</code> required <code>position_info_param_name</code> <code>str</code> <code>'default'</code> <code>additional_spike_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_position_features))</code> <code>spikes</code> <code>(DataFrame, shape(n_time, n_units))</code> <code>valid_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_decoding_data_for_epoch(\n    nwb_file_name: str,\n    interval_list_name: str,\n    position_info_param_name: str = \"default\",\n    additional_spike_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, pd.DataFrame, list[slice]]:\n    \"\"\"Collects the data needed for decoding\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    interval_list_name : str\n    position_info_param_name : str, optional\n    additional_spike_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_position_features)\n    spikes : pd.DataFrame, shape (n_time, n_units)\n    valid_slices : list[slice]\n\n    \"\"\"\n    # valid slices\n    valid_ephys_position_times_by_epoch = (\n        get_valid_ephys_position_times_by_epoch(nwb_file_name)\n    )\n    valid_ephys_position_times = valid_ephys_position_times_by_epoch[\n        interval_list_name\n    ]\n    valid_slices = convert_valid_times_to_slice(valid_ephys_position_times)\n\n    # position interval\n    position_interval_name = (\n        convert_epoch_interval_name_to_position_interval_name(\n            {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        )\n    )\n\n    # spikes\n    valid_times = np.asarray(\n        [(times.start, times.stop) for times in valid_slices]\n    )\n\n    curated_spikes_key = {\n        \"nwb_file_name\": nwb_file_name,\n        **additional_spike_keys,\n    }\n    spikes = get_spike_indicator(\n        curated_spikes_key,\n        (valid_times.min(), valid_times.max()),\n        sampling_rate=500,\n    )\n    spikes = pd.concat([spikes.loc[times] for times in valid_slices])\n\n    # position\n    position_info = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": position_interval_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch1_dataframe()\n    new_time = spikes.index.to_numpy()\n    new_index = pd.Index(\n        np.unique(np.concatenate((position_info.index, new_time))),\n        name=\"time\",\n    )\n    position_info = (\n        position_info.reindex(index=new_index)\n        .interpolate(method=\"linear\")\n        .reindex(index=new_time)\n    )\n\n    return position_info, spikes, valid_slices\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_data_for_multiple_epochs", "title": "<code>get_data_for_multiple_epochs(nwb_file_name, epoch_names, position_info_param_name='decoding', additional_spike_keys={})</code>", "text": "<p>Collects the data needed for decoding for multiple epochs</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>epoch_names</code> <code>list</code> required <code>position_info_param_name</code> <code>str</code> <code>'decoding'</code> <code>additional_spike_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_position_features))</code> <code>spikes</code> <code>(DataFrame, shape(n_time, n_units))</code> <code>valid_slices</code> <code>list[slice]</code> <code>environment_labels</code> <code>(ndarray, shape(n_time))</code> <p>The environment label for each time point</p> <code>sort_group_ids</code> <code>(ndarray, shape(n_units))</code> <p>The sort group of each unit</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_data_for_multiple_epochs(\n    nwb_file_name: str,\n    epoch_names: list,\n    position_info_param_name: str = \"decoding\",\n    additional_spike_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, pd.DataFrame, list[slice], np.ndarray, np.ndarray]:\n    \"\"\"Collects the data needed for decoding for multiple epochs\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    epoch_names : list\n    position_info_param_name : str, optional\n    additional_spike_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_position_features)\n    spikes : pd.DataFrame, shape (n_time, n_units)\n    valid_slices : list[slice]\n    environment_labels : np.ndarray, shape (n_time,)\n        The environment label for each time point\n    sort_group_ids : np.ndarray, shape (n_units,)\n        The sort group of each unit\n    \"\"\"\n    data = []\n    environment_labels = []\n\n    for epoch in epoch_names:\n        logger.info(epoch)\n        data.append(\n            get_decoding_data_for_epoch(\n                nwb_file_name,\n                epoch,\n                position_info_param_name=position_info_param_name,\n                additional_spike_keys=additional_spike_keys,\n            )\n        )\n        n_time = data[-1][0].shape[0]\n        environment_labels.append([epoch] * n_time)\n\n    environment_labels = np.concatenate(environment_labels, axis=0)\n    position_info, spikes, valid_slices = list(zip(*data))\n    position_info = pd.concat(position_info, axis=0)\n    spikes = pd.concat(spikes, axis=0)\n    valid_slices = {\n        epoch: valid_slice\n        for epoch, valid_slice in zip(epoch_names, valid_slices)\n    }\n\n    assert position_info.shape[0] == spikes.shape[0]\n\n    sort_group_ids = np.asarray(\n        [int(col.split(\"_\")[0]) for col in spikes.columns]\n    )\n\n    return (\n        position_info,\n        spikes,\n        valid_slices,\n        environment_labels,\n        sort_group_ids,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/visualization/", "title": "visualization.py", "text": ""}, {"location": "api/decoding/v0/visualization_1D_view/", "title": "visualization_1D_view.py", "text": ""}, {"location": "api/decoding/v0/visualization_1D_view/#src.spyglass.decoding.v0.visualization_1D_view.create_1D_decode_view", "title": "<code>create_1D_decode_view(posterior, linear_position=None, ref_time_sec=None)</code>", "text": "<p>Creates a view of an interactive heatmap of position vs. time.</p> <p>Parameters:</p> Name Type Description Default <code>posterior</code> <code>(DataArray, shape(n_time, n_position_bins))</code> required <code>linear_position</code> <code>(ndarray, shape(n_time))</code> <code>None</code> <code>ref_time_sec</code> <code>float64</code> <p>Reference time for the purpose of offsetting the start time</p> <code>None</code> <p>Returns:</p> Name Type Description <code>view</code> <code>DecodedLinearPositionData</code> Source code in <code>src/spyglass/decoding/v0/visualization_1D_view.py</code> <pre><code>def create_1D_decode_view(\n    posterior: xr.DataArray,\n    linear_position: np.ndarray = None,\n    ref_time_sec: Union[np.float64, None] = None,\n) -&gt; vvf.DecodedLinearPositionData:\n    \"\"\"Creates a view of an interactive heatmap of position vs. time.\n\n    Parameters\n    ----------\n    posterior : xr.DataArray, shape (n_time, n_position_bins)\n    linear_position : np.ndarray, shape (n_time, ), optional\n    ref_time_sec : np.float64, optional\n        Reference time for the purpose of offsetting the start time\n\n    Returns\n    -------\n    view : vvf.DecodedLinearPositionData\n\n    \"\"\"\n    if linear_position is not None:\n        linear_position = np.asarray(linear_position).squeeze()\n\n    trimmed_posterior = discretize_and_trim(posterior)\n    observations_per_time = get_observations_per_time(\n        trimmed_posterior, posterior\n    )\n    sampling_freq = get_sampling_freq(posterior.time)\n    start_time_sec = posterior.time.values[0]\n    if ref_time_sec is not None:\n        start_time_sec = start_time_sec - ref_time_sec\n\n    trimmed_bin_center_index = get_trimmed_bin_center_index(\n        posterior.position.values, trimmed_posterior.position.values\n    )\n\n    return vvf.DecodedLinearPositionData(\n        values=trimmed_posterior.values,\n        positions=trimmed_bin_center_index,\n        frame_bounds=observations_per_time,\n        positions_key=posterior.position.values.astype(np.float32),\n        observed_positions=linear_position,\n        start_time_sec=start_time_sec,\n        sampling_frequency=sampling_freq,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/visualization_2D_view/", "title": "visualization_2D_view.py", "text": ""}, {"location": "api/decoding/v0/visualization_2D_view/#src.spyglass.decoding.v0.visualization_2D_view.create_2D_decode_view", "title": "<code>create_2D_decode_view(position_time, position, interior_place_bin_centers, place_bin_size, posterior, head_dir=None)</code>", "text": "<p>Creates a 2D decoding movie view</p> <p>Parameters:</p> Name Type Description Default <code>position_time</code> <code>(ndarray, shape(n_time))</code> required <code>position</code> <code>(ndarray, shape(n_time, 2))</code> required <code>interior_place_bin_centers</code> <code>ndarray</code> required <code>place_bin_size</code> <code>(ndarray, shape(2, 1))</code> required <code>posterior</code> <code>(DataArray, shape(n_time, n_position_bins))</code> required <code>head_dir</code> <code>ndarray</code> <code>None</code> <p>Returns:</p> Name Type Description <code>view</code> <code>TrackPositionAnimationV1</code> Source code in <code>src/spyglass/decoding/v0/visualization_2D_view.py</code> <pre><code>def create_2D_decode_view(\n    position_time: np.ndarray,\n    position: np.ndarray,\n    interior_place_bin_centers: np.ndarray,\n    place_bin_size: np.ndarray,\n    posterior: xr.DataArray,\n    head_dir: np.ndarray = None,\n) -&gt; vvf.TrackPositionAnimationV1:\n    \"\"\"Creates a 2D decoding movie view\n\n    Parameters\n    ----------\n    position_time : np.ndarray, shape (n_time,)\n    position : np.ndarray, shape (n_time, 2)\n    interior_place_bin_centers: np.ndarray, shape (n_track_bins, 2)\n    place_bin_size : np.ndarray, shape (2, 1)\n    posterior : xr.DataArray, shape (n_time, n_position_bins)\n    head_dir : np.ndarray, optional\n\n    Returns\n    -------\n    view : vvf.TrackPositionAnimationV1\n\n    \"\"\"\n    assert (\n        position_time.shape[0] == position.shape[0]\n    ), \"position_time and position must have the same length\"\n    assert (\n        posterior.shape[0] == position.shape[0]\n    ), \"posterior and position must have the same length\"\n\n    position_time = np.squeeze(np.asarray(position_time)).copy()\n    position = np.asarray(position)\n    if head_dir is not None:\n        head_dir = np.squeeze(np.asarray(head_dir))\n\n    track_bin_width = place_bin_size[0]\n    track_bin_height = place_bin_size[1]\n    # NOTE: We expect caller to have converted from fortran ordering already\n    # i.e. somewhere upstream, centers = env.place_bin_centers_[env.is_track_interior_.ravel(order=\"F\")]\n    upper_left_points = get_ul_corners(\n        track_bin_width, track_bin_height, interior_place_bin_centers\n    )\n\n    data = create_static_track_animation(\n        ul_corners=upper_left_points,\n        track_rect_height=track_bin_height,\n        track_rect_width=track_bin_width,\n        timestamps=position_time,\n        positions=position.T,\n        head_dir=head_dir,\n        compute_real_time_rate=True,\n    )\n    data[\"decodedData\"] = process_decoded_data(posterior)\n\n    return create_track_animation_object(static_track_animation=data)\n</code></pre>"}, {"location": "api/decoding/v1/clusterless/", "title": "clusterless.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from unclustered spikes and spike waveform features. See [1] for details.</p>"}, {"location": "api/decoding/v1/clusterless/#src.spyglass.decoding.v1.clusterless--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v1/core/", "title": "core.py", "text": ""}, {"location": "api/decoding/v1/core/#src.spyglass.decoding.v1.core.DecodingParameters", "title": "<code>DecodingParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Parameters for decoding the animal's mental position and some category of interest</p> Source code in <code>src/spyglass/decoding/v1/core.py</code> <pre><code>@schema\nclass DecodingParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Parameters for decoding the animal's mental position and some category of interest\"\"\"\n\n    definition = \"\"\"\n    decoding_param_name : varchar(80)  # a name for this set of parameters\n    ---\n    decoding_params : BLOB             # initialization parameters for model\n    decoding_kwargs = NULL : BLOB      # additional keyword arguments\n    \"\"\"\n\n    contents = [\n        {\n            \"decoding_param_name\": f\"contfrag_clusterless_{non_local_detector_version}\",\n            \"decoding_params\": ContFragClusterlessClassifier(),\n        },\n        {\n            \"decoding_param_name\": f\"nonlocal_clusterless_{non_local_detector_version}\",\n            \"decoding_params\": NonLocalClusterlessDetector(),\n        },\n        {\n            \"decoding_param_name\": f\"contfrag_sorted_{non_local_detector_version}\",\n            \"decoding_params\": ContFragSortedSpikesClassifier(),\n        },\n        {\n            \"decoding_param_name\": f\"nonlocal_sorted_{non_local_detector_version}\",\n            \"decoding_params\": NonLocalSortedSpikesDetector(),\n        },\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        cls.insert(cls.contents, skip_duplicates=True)\n\n    def insert(self, rows, *args, **kwargs):\n        for row in rows:\n            row[\"decoding_params\"] = convert_classes_to_dict(\n                vars(row[\"decoding_params\"])\n            )\n        super().insert(rows, *args, **kwargs)\n\n    def fetch(self, *args, **kwargs):\n        rows = super().fetch(*args, **kwargs)\n        if len(rows) &gt; 0 and len(rows[0]) &gt; 1:\n            content = []\n            for (\n                decoding_param_name,\n                decoding_params,\n                decoding_kwargs,\n            ) in rows:\n                content.append(\n                    (\n                        decoding_param_name,\n                        restore_classes(decoding_params),\n                        decoding_kwargs,\n                    )\n                )\n        else:\n            content = rows\n        return content\n\n    def fetch1(self, *args, **kwargs):\n        row = super().fetch1(*args, **kwargs)\n        row[\"decoding_params\"] = restore_classes(row[\"decoding_params\"])\n        return row\n</code></pre>"}, {"location": "api/decoding/v1/dj_decoder_conversion/", "title": "dj_decoder_conversion.py", "text": "<p>Converts decoder classes into dictionaries and dictionaries into classes so that datajoint can store them in tables.</p>"}, {"location": "api/decoding/v1/dj_decoder_conversion/#src.spyglass.decoding.v1.dj_decoder_conversion.restore_classes", "title": "<code>restore_classes(params)</code>", "text": "<p>Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes</p> <p>Parameters:</p> Name Type Description Default <code>params</code> <code>dict</code> <p>The parameters to convert</p> required <p>Returns:</p> Name Type Description <code>converted_params</code> <code>dict</code> <p>The converted parameters</p> Source code in <code>src/spyglass/decoding/v1/dj_decoder_conversion.py</code> <pre><code>def restore_classes(params: dict) -&gt; dict:\n    \"\"\"Converts a dictionary of parameters into a dictionary of classes\n    since datajoint cannot handle classes\n\n    Parameters\n    ----------\n    params : dict\n        The parameters to convert\n\n    Returns\n    -------\n    converted_params : dict\n        The converted parameters\n    \"\"\"\n\n    params = copy.deepcopy(params)\n\n    continuous_state_transition_types = _map_class_name_to_class(cst)\n    discrete_state_transition_types = _map_class_name_to_class(dst)\n    continuous_initial_conditions_types = _map_class_name_to_class(ic)\n\n    params[\"environments\"] = [\n        _convert_env_dict(env_params) for env_params in params[\"environments\"]\n    ]\n\n    params[\"continuous_transition_types\"] = [\n        [\n            _convert_dict_to_class(st, continuous_state_transition_types)\n            for st in sts\n        ]\n        for sts in params[\"continuous_transition_types\"]\n    ]\n    params[\"discrete_transition_type\"] = _convert_dict_to_class(\n        params[\"discrete_transition_type\"],\n        discrete_state_transition_types,\n    )\n    params[\"continuous_initial_conditions_types\"] = [\n        _convert_dict_to_class(cont_ic, continuous_initial_conditions_types)\n        for cont_ic in params[\"continuous_initial_conditions_types\"]\n    ]\n\n    if params[\"observation_models\"] is not None:\n        params[\"observation_models\"] = [\n            ObservationModel(**obs) for obs in params[\"observation_models\"]\n        ]\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v1/dj_decoder_conversion/#src.spyglass.decoding.v1.dj_decoder_conversion.convert_classes_to_dict", "title": "<code>convert_classes_to_dict(params)</code>", "text": "<p>Converts the classifier parameters into a dictionary so that datajoint can store it.</p> Source code in <code>src/spyglass/decoding/v1/dj_decoder_conversion.py</code> <pre><code>def convert_classes_to_dict(params: dict) -&gt; dict:\n    \"\"\"Converts the classifier parameters into a dictionary so that datajoint can store it.\"\"\"\n    params = copy.deepcopy(params)\n    try:\n        params[\"environments\"] = [\n            _convert_environment_to_dict(env) for env in params[\"environments\"]\n        ]\n    except TypeError:\n        params[\"environments\"] = [\n            _convert_environment_to_dict(params[\"environments\"])\n        ]\n    params[\"continuous_transition_types\"] = _convert_transitions_to_dict(\n        params[\"continuous_transition_types\"]\n    )\n    params[\"discrete_transition_type\"] = _to_dict(\n        params[\"discrete_transition_type\"]\n    )\n    params[\"continuous_initial_conditions_types\"] = [\n        _to_dict(cont_ic)\n        for cont_ic in params[\"continuous_initial_conditions_types\"]\n    ]\n\n    if params[\"observation_models\"] is not None:\n        params[\"observation_models\"] = [\n            vars(obs) for obs in params[\"observation_models\"]\n        ]\n\n    try:\n        params[\"clusterless_algorithm_params\"] = _convert_algorithm_params(\n            params[\"clusterless_algorithm_params\"]\n        )\n    except KeyError:\n        pass\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v1/sorted_spikes/", "title": "sorted_spikes.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from clustered spikes times. See [1] for details.</p>"}, {"location": "api/decoding/v1/sorted_spikes/#src.spyglass.decoding.v1.sorted_spikes--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v1/temp/", "title": "temp.py", "text": ""}, {"location": "api/decoding/v1/waveform_features/", "title": "waveform_features.py", "text": ""}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams", "title": "<code>WaveformFeaturesParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Defines the types of spike waveform features computed for a given spike time.</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@schema\nclass WaveformFeaturesParams(SpyglassMixin, dj.Lookup):\n    \"\"\"Defines the types of spike waveform features computed for a given spike\n    time.\"\"\"\n\n    definition = \"\"\"\n    features_param_name : varchar(80) # a name for this set of parameters\n    ---\n    params : longblob # the parameters for the waveform features\n    \"\"\"\n    _default_waveform_feature_params = {\n        \"amplitude\": {\n            \"peak_sign\": \"neg\",\n            \"estimate_peak_time\": False,\n        }\n    }\n    _default_waveform_extraction_params = {\n        \"ms_before\": 0.5,\n        \"ms_after\": 0.5,\n        \"max_spikes_per_unit\": None,\n        \"n_jobs\": 5,\n        \"chunk_duration\": \"1000s\",\n    }\n    contents = [\n        [\n            \"amplitude\",\n            {\n                \"waveform_features_params\": _default_waveform_feature_params,\n                \"waveform_extraction_params\": _default_waveform_extraction_params,\n            },\n        ],\n        [\n            \"amplitude, spike_location\",\n            {\n                \"waveform_features_params\": {\n                    \"amplitude\": _default_waveform_feature_params[\"amplitude\"],\n                    \"spike_location\": {},\n                },\n                \"waveform_extraction_params\": _default_waveform_extraction_params,\n            },\n        ],\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        cls.insert(cls.contents, skip_duplicates=True)\n\n    @staticmethod\n    def check_supported_waveform_features(waveform_features: list[str]) -&gt; bool:\n        \"\"\"Checks whether the requested waveform features types are supported\n\n        Parameters\n        ----------\n        waveform_features : list\n        \"\"\"\n        supported_features = set(WAVEFORM_FEATURE_FUNCTIONS)\n        return set(waveform_features).issubset(supported_features)\n\n    @property\n    def supported_waveform_features(self) -&gt; list[str]:\n        \"\"\"Returns the list of supported waveform features\"\"\"\n        return list(WAVEFORM_FEATURE_FUNCTIONS)\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams.check_supported_waveform_features", "title": "<code>check_supported_waveform_features(waveform_features)</code>  <code>staticmethod</code>", "text": "<p>Checks whether the requested waveform features types are supported</p> <p>Parameters:</p> Name Type Description Default <code>waveform_features</code> <code>list</code> required Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@staticmethod\ndef check_supported_waveform_features(waveform_features: list[str]) -&gt; bool:\n    \"\"\"Checks whether the requested waveform features types are supported\n\n    Parameters\n    ----------\n    waveform_features : list\n    \"\"\"\n    supported_features = set(WAVEFORM_FEATURE_FUNCTIONS)\n    return set(waveform_features).issubset(supported_features)\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams.supported_waveform_features", "title": "<code>supported_waveform_features: list[str]</code>  <code>property</code>", "text": "<p>Returns the list of supported waveform features</p>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.UnitWaveformFeatures", "title": "<code>UnitWaveformFeatures</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>For each spike time, compute a spike waveform feature associated with that spike. Used for clusterless decoding.</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@schema\nclass UnitWaveformFeatures(SpyglassMixin, dj.Computed):\n    \"\"\"For each spike time, compute a spike waveform feature associated with that\n    spike. Used for clusterless decoding.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitWaveformFeaturesSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # the NWB object that stores the waveforms\n    \"\"\"\n\n    def make(self, key):\n        # get the list of feature parameters\n        params = (WaveformFeaturesParams &amp; key).fetch1(\"params\")\n\n        # check that the feature type is supported\n        if not WaveformFeaturesParams.check_supported_waveform_features(\n            params[\"waveform_features_params\"]\n        ):\n            raise NotImplementedError(\n                f\"Features {set(params['waveform_features_params'])} are not supported\"\n            )\n\n        merge_key = {\"merge_id\": key[\"spikesorting_merge_id\"]}\n        waveform_extractor = self._fetch_waveform(\n            merge_key, params[\"waveform_extraction_params\"]\n        )\n\n        source_key = SpikeSortingOutput().merge_get_parent(merge_key).fetch1()\n        # v0 pipeline\n        if \"sorter\" in source_key and \"nwb_file_name\" in source_key:\n            sorter = source_key[\"sorter\"]\n            nwb_file_name = source_key[\"nwb_file_name\"]\n            analysis_nwb_key = \"units\"\n        # v1 pipeline\n        else:\n            sorting_id = (SpikeSortingOutput.CurationV1 &amp; merge_key).fetch1(\n                \"sorting_id\"\n            )\n            sorter, nwb_file_name = (\n                SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}\n            ).fetch1(\"sorter\", \"nwb_file_name\")\n            analysis_nwb_key = \"object_id\"\n\n        waveform_features = {}\n\n        for feature, feature_params in params[\n            \"waveform_features_params\"\n        ].items():\n            waveform_features[feature] = self._compute_waveform_features(\n                waveform_extractor,\n                feature,\n                feature_params,\n                sorter,\n            )\n\n        spike_times = SpikeSortingOutput().fetch_nwb(merge_key)[0][\n            analysis_nwb_key\n        ][\"spike_times\"]\n\n        (\n            key[\"analysis_file_name\"],\n            key[\"object_id\"],\n        ) = _write_waveform_features_to_nwb(\n            nwb_file_name,\n            waveform_extractor,\n            spike_times,\n            waveform_features,\n        )\n\n        AnalysisNwbfile().add(\n            nwb_file_name,\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @staticmethod\n    def _fetch_waveform(\n        merge_key: dict, waveform_extraction_params: dict\n    ) -&gt; si.WaveformExtractor:\n        # get the recording from the parent table\n        recording = SpikeSortingOutput().get_recording(merge_key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n        # get the sorting from the parent table\n        sorting = SpikeSortingOutput().get_sorting(merge_key)\n\n        waveforms_temp_dir = temp_dir + \"/\" + str(merge_key[\"merge_id\"])\n        os.makedirs(waveforms_temp_dir, exist_ok=True)\n\n        return si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveforms_temp_dir,\n            overwrite=True,\n            **waveform_extraction_params,\n        )\n\n    @staticmethod\n    def _compute_waveform_features(\n        waveform_extractor: si.WaveformExtractor,\n        feature: str,\n        feature_params: dict,\n        sorter: str,\n    ) -&gt; dict:\n        feature_func = WAVEFORM_FEATURE_FUNCTIONS[feature]\n        if sorter == \"clusterless_thresholder\" and feature == \"amplitude\":\n            feature_params[\"estimate_peak_time\"] = False\n\n        return {\n            unit_id: feature_func(waveform_extractor, unit_id, **feature_params)\n            for unit_id in waveform_extractor.sorting.get_unit_ids()\n        }\n\n    def fetch_data(self) -&gt; tuple[list[np.ndarray], list[np.ndarray]]:\n        \"\"\"Fetches the spike times and features for each unit.\n\n        Returns\n        -------\n        spike_times : list of np.ndarray\n            List of spike times for each unit\n        features : list of np.ndarray\n            List of features for each unit\n\n        \"\"\"\n        return tuple(\n            zip(\n                *list(\n                    chain(\n                        *[self._convert_data(data) for data in self.fetch_nwb()]\n                    )\n                )\n            )\n        )\n\n    @staticmethod\n    def _convert_data(nwb_data) -&gt; list[tuple[np.ndarray, np.ndarray]]:\n        feature_df = nwb_data[\"object_id\"]\n\n        feature_columns = [\n            column for column in feature_df.columns if column != \"spike_times\"\n        ]\n\n        return [\n            (\n                unit.spike_times,\n                np.concatenate(unit[feature_columns].to_numpy(), axis=1),\n            )\n            for _, unit in feature_df.iterrows()\n        ]\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.UnitWaveformFeatures.fetch_data", "title": "<code>fetch_data()</code>", "text": "<p>Fetches the spike times and features for each unit.</p> <p>Returns:</p> Name Type Description <code>spike_times</code> <code>list of np.ndarray</code> <p>List of spike times for each unit</p> <code>features</code> <code>list of np.ndarray</code> <p>List of features for each unit</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>def fetch_data(self) -&gt; tuple[list[np.ndarray], list[np.ndarray]]:\n    \"\"\"Fetches the spike times and features for each unit.\n\n    Returns\n    -------\n    spike_times : list of np.ndarray\n        List of spike times for each unit\n    features : list of np.ndarray\n        List of features for each unit\n\n    \"\"\"\n    return tuple(\n        zip(\n            *list(\n                chain(\n                    *[self._convert_data(data) for data in self.fetch_nwb()]\n                )\n            )\n        )\n    )\n</code></pre>"}, {"location": "api/lfp/lfp_electrode/", "title": "lfp_electrode.py", "text": ""}, {"location": "api/lfp/lfp_electrode/#src.spyglass.lfp.lfp_electrode.LFPElectrodeGroup", "title": "<code>LFPElectrodeGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lfp/lfp_electrode.py</code> <pre><code>@schema\nclass LFPElectrodeGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; Session                             # the session to which this LFP belongs\n     lfp_electrode_group_name: varchar(200) # the name of this group of electrodes\n     \"\"\"\n\n    class LFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPElectrodeGroup # the group of electrodes to be filtered\n        -&gt; Electrode        # the electrode to be filtered\n        \"\"\"\n\n    @staticmethod\n    def create_lfp_electrode_group(\n        nwb_file_name: str, group_name: str, electrode_list: list[int]\n    ):\n        \"\"\"Adds an LFPElectrodeGroup and the individual electrodes\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the nwb file (e.g. the session)\n        group_name : str\n            The name of this group (&lt; 200 char)\n        electrode_list : list\n            A list of the electrode ids to include in this group.\n        \"\"\"\n        # remove the session and then recreate the session and Electrode list\n        # check to see if the user allowed the deletion\n        key = {\n            \"nwb_file_name\": nwb_file_name,\n            \"lfp_electrode_group_name\": group_name,\n        }\n        LFPElectrodeGroup().insert1(key, skip_duplicates=True)\n\n        # TODO: do this in a better way\n        all_electrodes = (Electrode() &amp; {\"nwb_file_name\": nwb_file_name}).fetch(\n            as_dict=True\n        )\n        primary_key = Electrode.primary_key\n        if isinstance(electrode_list, ndarray):\n            # convert to list if it is an numpy array\n            electrode_list = list(electrode_list.astype(int).reshape(-1))\n        for e in all_electrodes:\n            # create a dictionary so we can insert the electrodes\n            if e[\"electrode_id\"] in electrode_list:\n                lfpelectdict = {k: v for k, v in e.items() if k in primary_key}\n                lfpelectdict[\"lfp_electrode_group_name\"] = group_name\n                LFPElectrodeGroup().LFPElectrode.insert1(\n                    lfpelectdict, skip_duplicates=True\n                )\n</code></pre>"}, {"location": "api/lfp/lfp_electrode/#src.spyglass.lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group", "title": "<code>create_lfp_electrode_group(nwb_file_name, group_name, electrode_list)</code>  <code>staticmethod</code>", "text": "<p>Adds an LFPElectrodeGroup and the individual electrodes</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the nwb file (e.g. the session)</p> required <code>group_name</code> <code>str</code> <p>The name of this group (&lt; 200 char)</p> required <code>electrode_list</code> <code>list</code> <p>A list of the electrode ids to include in this group.</p> required Source code in <code>src/spyglass/lfp/lfp_electrode.py</code> <pre><code>@staticmethod\ndef create_lfp_electrode_group(\n    nwb_file_name: str, group_name: str, electrode_list: list[int]\n):\n    \"\"\"Adds an LFPElectrodeGroup and the individual electrodes\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the nwb file (e.g. the session)\n    group_name : str\n        The name of this group (&lt; 200 char)\n    electrode_list : list\n        A list of the electrode ids to include in this group.\n    \"\"\"\n    # remove the session and then recreate the session and Electrode list\n    # check to see if the user allowed the deletion\n    key = {\n        \"nwb_file_name\": nwb_file_name,\n        \"lfp_electrode_group_name\": group_name,\n    }\n    LFPElectrodeGroup().insert1(key, skip_duplicates=True)\n\n    # TODO: do this in a better way\n    all_electrodes = (Electrode() &amp; {\"nwb_file_name\": nwb_file_name}).fetch(\n        as_dict=True\n    )\n    primary_key = Electrode.primary_key\n    if isinstance(electrode_list, ndarray):\n        # convert to list if it is an numpy array\n        electrode_list = list(electrode_list.astype(int).reshape(-1))\n    for e in all_electrodes:\n        # create a dictionary so we can insert the electrodes\n        if e[\"electrode_id\"] in electrode_list:\n            lfpelectdict = {k: v for k, v in e.items() if k in primary_key}\n            lfpelectdict[\"lfp_electrode_group_name\"] = group_name\n            LFPElectrodeGroup().LFPElectrode.insert1(\n                lfpelectdict, skip_duplicates=True\n            )\n</code></pre>"}, {"location": "api/lfp/lfp_imported/", "title": "lfp_imported.py", "text": ""}, {"location": "api/lfp/lfp_merge/", "title": "lfp_merge.py", "text": ""}, {"location": "api/lfp/lfp_merge/#src.spyglass.lfp.lfp_merge.LFPOutput", "title": "<code>LFPOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/lfp/lfp_merge.py</code> <pre><code>@schema\nclass LFPOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class LFPV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; LFPV1\n        \"\"\"\n\n    class ImportedLFP(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ImportedLFP\n        \"\"\"\n\n    class CommonLFP(SpyglassMixin, dj.Part):  # noqa: F811\n        \"\"\"Table to pass-through legacy LFP\"\"\"\n\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CommonLFP\n        \"\"\"\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        # Note: `proj` below facilitates operator syntax eg Table &amp; restrict\n        nwb_lfp = self.fetch_nwb(self.proj())[0]\n        return pd.DataFrame(\n            nwb_lfp[\"lfp\"].data,\n            index=pd.Index(nwb_lfp[\"lfp\"].timestamps, name=\"time\"),\n        )\n</code></pre>"}, {"location": "api/lfp/lfp_merge/#src.spyglass.lfp.lfp_merge.LFPOutput.CommonLFP", "title": "<code>CommonLFP</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through legacy LFP</p> Source code in <code>src/spyglass/lfp/lfp_merge.py</code> <pre><code>class CommonLFP(SpyglassMixin, dj.Part):  # noqa: F811\n    \"\"\"Table to pass-through legacy LFP\"\"\"\n\n    definition = \"\"\"\n    -&gt; master\n    ---\n    -&gt; CommonLFP\n    \"\"\"\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/", "title": "lfp_band.py", "text": ""}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandSelection", "title": "<code>LFPBandSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>The user's selection of LFP data to be filtered in a given frequency band.</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>@schema\nclass LFPBandSelection(SpyglassMixin, dj.Manual):\n    \"\"\"The user's selection of LFP data to be filtered in a given frequency band.\"\"\"\n\n    definition = \"\"\"\n    -&gt; LFPOutput.proj(lfp_merge_id='merge_id')                            # the LFP data to be filtered\n    -&gt; FirFilterParameters                                                # the filter to use for the data\n    -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n    lfp_band_sampling_rate: int                                           # the sampling rate for this band\n    ---\n    min_interval_len = 1.0: float  # the minimum length of a valid interval to filter\n    \"\"\"\n\n    class LFPBandElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPBandSelection # the LFP band selection\n        -&gt; LFPElectrodeGroup.LFPElectrode  # the LFP electrode to be filtered\n        reference_elect_id = -1: int  # the reference electrode to use; -1 for no reference\n        ---\n        \"\"\"\n\n    def set_lfp_band_electrodes(\n        self,\n        nwb_file_name: str,\n        lfp_merge_id: int,\n        electrode_list: list[int],\n        filter_name: str,\n        interval_list_name: str,\n        reference_electrode_list: list[int],\n        lfp_band_sampling_rate: int,\n    ):\n        \"\"\"Sets the electrodes to be filtered for a given LFP\n\n        Parameters\n        ----------\n        nwb_file_name: str\n            The name of the NWB file containing the LFP data\n        lfp_merge_id: int\n            The uuid of the LFP data to be filtered\n        electrode_list: list\n            A list of the electrodes to be filtered\n        filter_name: str\n            The name of the filter to be used\n        interval_list_name: str\n            The name of the interval list to be used\n        reference_electrode_list: list\n            A list of the reference electrodes to be used\n        lfp_band_sampling_rate: int\n        \"\"\"\n        # Error checks on parameters\n        # electrode_list\n\n        lfp_key = {\"merge_id\": lfp_merge_id}\n        lfp_part_table = LFPOutput.merge_get_part(lfp_key)\n\n        query = LFPElectrodeGroup().LFPElectrode() &amp; lfp_key\n        available_electrodes = query.fetch(\"electrode_id\")\n        if not np.all(np.isin(electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in electrode_list must be valid electrode_ids in\"\n                + \" the LFPElectodeGroup table: \"\n                + f\"{electrode_list} not in {available_electrodes}\"\n            )\n        # sampling rate\n        lfp_sampling_rate = LFPOutput.merge_get_parent(lfp_key).fetch1(\n            \"lfp_sampling_rate\"\n        )\n        decimation = lfp_sampling_rate // lfp_band_sampling_rate\n        # filter\n        filter_query = FirFilterParameters() &amp; {\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": lfp_sampling_rate,\n        }\n        if not filter_query:\n            raise ValueError(\n                f\"Filter {filter_name}, sampling rate {lfp_sampling_rate} is \"\n                + \"not in the FirFilterParameters table\"\n            )\n        # interval_list\n        interval_query = IntervalList() &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_name\": interval_list_name,\n        }\n        if not interval_query:\n            raise ValueError(\n                f\"interval list {interval_list_name} is not in the IntervalList\"\n                \" table; the list must be added before this function is called\"\n            )\n        # reference_electrode_list\n        if len(reference_electrode_list) != 1 and len(\n            reference_electrode_list\n        ) != len(electrode_list):\n            raise ValueError(\n                \"reference_electrode_list must contain either 1 or \"\n                + \"len(electrode_list) elements\"\n            )\n        # add a -1 element to the list to allow for the no reference option\n        available_electrodes = np.append(available_electrodes, [-1])\n        if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in reference_electrode_list must be valid \"\n                \"electrode_ids in the LFPSelection table\"\n            )\n\n        # make a list of all the references\n        ref_list = np.zeros((len(electrode_list),))\n        ref_list[:] = reference_electrode_list\n\n        key = dict(\n            nwb_file_name=nwb_file_name,\n            lfp_merge_id=lfp_merge_id,\n            filter_name=filter_name,\n            filter_sampling_rate=lfp_sampling_rate,\n            target_interval_list_name=interval_list_name,\n            lfp_band_sampling_rate=lfp_sampling_rate // decimation,\n        )\n        # insert an entry into the main LFPBandSelectionTable\n        self.insert1(key, skip_duplicates=True)\n\n        key[\"lfp_electrode_group_name\"] = lfp_part_table.fetch1(\n            \"lfp_electrode_group_name\"\n        )\n        # iterate through all of the new elements and add them\n        for e, r in zip(electrode_list, ref_list):\n            elect_key = (\n                LFPElectrodeGroup.LFPElectrode\n                &amp; {\n                    \"nwb_file_name\": nwb_file_name,\n                    \"lfp_electrode_group_name\": key[\"lfp_electrode_group_name\"],\n                    \"electrode_id\": e,\n                }\n            ).fetch1(\"KEY\")\n            for item in elect_key:\n                key[item] = elect_key[item]\n            query = Electrode &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_id\": e,\n            }\n            key[\"reference_elect_id\"] = r\n            self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandSelection.set_lfp_band_electrodes", "title": "<code>set_lfp_band_electrodes(nwb_file_name, lfp_merge_id, electrode_list, filter_name, interval_list_name, reference_electrode_list, lfp_band_sampling_rate)</code>", "text": "<p>Sets the electrodes to be filtered for a given LFP</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file containing the LFP data</p> required <code>lfp_merge_id</code> <code>int</code> <p>The uuid of the LFP data to be filtered</p> required <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to be filtered</p> required <code>filter_name</code> <code>str</code> <p>The name of the filter to be used</p> required <code>interval_list_name</code> <code>str</code> <p>The name of the interval list to be used</p> required <code>reference_electrode_list</code> <code>list[int]</code> <p>A list of the reference electrodes to be used</p> required <code>lfp_band_sampling_rate</code> <code>int</code> required Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def set_lfp_band_electrodes(\n    self,\n    nwb_file_name: str,\n    lfp_merge_id: int,\n    electrode_list: list[int],\n    filter_name: str,\n    interval_list_name: str,\n    reference_electrode_list: list[int],\n    lfp_band_sampling_rate: int,\n):\n    \"\"\"Sets the electrodes to be filtered for a given LFP\n\n    Parameters\n    ----------\n    nwb_file_name: str\n        The name of the NWB file containing the LFP data\n    lfp_merge_id: int\n        The uuid of the LFP data to be filtered\n    electrode_list: list\n        A list of the electrodes to be filtered\n    filter_name: str\n        The name of the filter to be used\n    interval_list_name: str\n        The name of the interval list to be used\n    reference_electrode_list: list\n        A list of the reference electrodes to be used\n    lfp_band_sampling_rate: int\n    \"\"\"\n    # Error checks on parameters\n    # electrode_list\n\n    lfp_key = {\"merge_id\": lfp_merge_id}\n    lfp_part_table = LFPOutput.merge_get_part(lfp_key)\n\n    query = LFPElectrodeGroup().LFPElectrode() &amp; lfp_key\n    available_electrodes = query.fetch(\"electrode_id\")\n    if not np.all(np.isin(electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in electrode_list must be valid electrode_ids in\"\n            + \" the LFPElectodeGroup table: \"\n            + f\"{electrode_list} not in {available_electrodes}\"\n        )\n    # sampling rate\n    lfp_sampling_rate = LFPOutput.merge_get_parent(lfp_key).fetch1(\n        \"lfp_sampling_rate\"\n    )\n    decimation = lfp_sampling_rate // lfp_band_sampling_rate\n    # filter\n    filter_query = FirFilterParameters() &amp; {\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": lfp_sampling_rate,\n    }\n    if not filter_query:\n        raise ValueError(\n            f\"Filter {filter_name}, sampling rate {lfp_sampling_rate} is \"\n            + \"not in the FirFilterParameters table\"\n        )\n    # interval_list\n    interval_query = IntervalList() &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_name\": interval_list_name,\n    }\n    if not interval_query:\n        raise ValueError(\n            f\"interval list {interval_list_name} is not in the IntervalList\"\n            \" table; the list must be added before this function is called\"\n        )\n    # reference_electrode_list\n    if len(reference_electrode_list) != 1 and len(\n        reference_electrode_list\n    ) != len(electrode_list):\n        raise ValueError(\n            \"reference_electrode_list must contain either 1 or \"\n            + \"len(electrode_list) elements\"\n        )\n    # add a -1 element to the list to allow for the no reference option\n    available_electrodes = np.append(available_electrodes, [-1])\n    if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in reference_electrode_list must be valid \"\n            \"electrode_ids in the LFPSelection table\"\n        )\n\n    # make a list of all the references\n    ref_list = np.zeros((len(electrode_list),))\n    ref_list[:] = reference_electrode_list\n\n    key = dict(\n        nwb_file_name=nwb_file_name,\n        lfp_merge_id=lfp_merge_id,\n        filter_name=filter_name,\n        filter_sampling_rate=lfp_sampling_rate,\n        target_interval_list_name=interval_list_name,\n        lfp_band_sampling_rate=lfp_sampling_rate // decimation,\n    )\n    # insert an entry into the main LFPBandSelectionTable\n    self.insert1(key, skip_duplicates=True)\n\n    key[\"lfp_electrode_group_name\"] = lfp_part_table.fetch1(\n        \"lfp_electrode_group_name\"\n    )\n    # iterate through all of the new elements and add them\n    for e, r in zip(electrode_list, ref_list):\n        elect_key = (\n            LFPElectrodeGroup.LFPElectrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"lfp_electrode_group_name\": key[\"lfp_electrode_group_name\"],\n                \"electrode_id\": e,\n            }\n        ).fetch1(\"KEY\")\n        for item in elect_key:\n            key[item] = elect_key[item]\n        query = Electrode &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_id\": e,\n        }\n        key[\"reference_elect_id\"] = r\n        self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1", "title": "<code>LFPBandV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>@schema\nclass LFPBandV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; LFPBandSelection              # the LFP band selection\n    ---\n    -&gt; AnalysisNwbfile               # the name of the nwb file with the lfp data\n    -&gt; IntervalList                  # the final interval list of valid times for the data\n    lfp_band_object_id: varchar(40)  # the NWB object ID for loading this object from the file\n    \"\"\"\n\n    def make(self, key):\n        # get the NWB object with the lfp data; FIX: change to fetch with additional infrastructure\n        lfp_key = {\"merge_id\": key[\"lfp_merge_id\"]}\n        lfp_object = (LFPOutput &amp; lfp_key).fetch_nwb()[0][\"lfp\"]\n\n        # get the electrodes to be filtered and their references\n        lfp_band_elect_id, lfp_band_ref_id = (\n            LFPBandSelection().LFPBandElectrode() &amp; key\n        ).fetch(\"electrode_id\", \"reference_elect_id\")\n\n        # sort the electrodes to make sure they are in ascending order\n        lfp_band_elect_id = np.asarray(lfp_band_elect_id)\n        lfp_band_ref_id = np.asarray(lfp_band_ref_id)\n        lfp_sort_order = np.argsort(lfp_band_elect_id)\n        lfp_band_elect_id = lfp_band_elect_id[lfp_sort_order]\n        lfp_band_ref_id = lfp_band_ref_id[lfp_sort_order]\n\n        lfp_sampling_rate, lfp_interval_list = LFPOutput.merge_get_parent(\n            lfp_key\n        ).fetch1(\"lfp_sampling_rate\", \"interval_list_name\")\n        interval_list_name, lfp_band_sampling_rate = (\n            LFPBandSelection() &amp; key\n        ).fetch1(\"target_interval_list_name\", \"lfp_band_sampling_rate\")\n        valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        # the valid_times for this interval may be slightly beyond the valid\n        # times for the lfp itself, so we have to intersect the two lists\n        lfp_valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": lfp_interval_list,\n            }\n        ).fetch1(\"valid_times\")\n        min_length = (LFPBandSelection &amp; key).fetch1(\"min_interval_len\")\n        lfp_band_valid_times = interval_list_intersect(\n            valid_times, lfp_valid_times, min_length=min_length\n        )\n\n        filter_name, filter_sampling_rate, lfp_band_sampling_rate = (\n            LFPBandSelection() &amp; key\n        ).fetch1(\n            \"filter_name\", \"filter_sampling_rate\", \"lfp_band_sampling_rate\"\n        )\n\n        decimation = int(lfp_sampling_rate) // lfp_band_sampling_rate\n\n        # load in the timestamps\n        timestamps = np.asarray(lfp_object.timestamps)\n        # get the indices of the first timestamp and the last timestamp that\n        # are within the valid times\n        included_indices = interval_list_contains_ind(\n            lfp_band_valid_times, timestamps\n        )\n        # pad the indices by 1 on each side to avoid message in filter_data\n        if included_indices[0] &gt; 0:\n            included_indices[0] -= 1\n        if included_indices[-1] != len(timestamps) - 1:\n            included_indices[-1] += 1\n\n        timestamps = timestamps[included_indices[0] : included_indices[-1]]\n\n        # load all the data to speed filtering\n        lfp_data = np.asarray(\n            lfp_object.data[included_indices[0] : included_indices[-1], :],\n            dtype=type(lfp_object.data[0][0]),\n        )\n\n        # get the indices of the electrodes to be filtered and the references\n        lfp_band_elect_index = get_electrode_indices(\n            lfp_object, lfp_band_elect_id\n        )\n        lfp_band_ref_index = get_electrode_indices(lfp_object, lfp_band_ref_id)\n\n        # subtract off the references for the selected channels\n        lfp_data_original = lfp_data.copy()\n        for index, elect_index in enumerate(lfp_band_elect_index):\n            if lfp_band_ref_id[index] != -1:\n                lfp_data[:, elect_index] = (\n                    lfp_data_original[:, elect_index]\n                    - lfp_data_original[:, lfp_band_ref_index[index]]\n                )\n\n        # get the LFP filter that matches the raw data\n        filter = (\n            FirFilterParameters()\n            &amp; {\"filter_name\": filter_name}\n            &amp; {\"filter_sampling_rate\": filter_sampling_rate}\n        ).fetch(as_dict=True)\n\n        filter_coeff = filter[0][\"filter_coeff\"]\n        if len(filter_coeff) == 0:\n            logger.error(\n                \"LFPBand: no filter found with data \"\n                + f\"sampling rate of {lfp_band_sampling_rate}\"\n            )\n            return None\n\n        # create the analysis nwb file to store the results.\n        lfp_band_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n        lfp_band_file_abspath = AnalysisNwbfile().get_abs_path(\n            lfp_band_file_name\n        )\n        # filter the data and write to an the nwb file\n        filtered_data, new_timestamps = FirFilterParameters().filter_data(\n            timestamps,\n            lfp_data,\n            filter_coeff,\n            lfp_band_valid_times,\n            lfp_band_elect_index,\n            decimation,\n        )\n\n        # now that the LFP is filtered, we create an electrical series for it and add it to the file\n        with pynwb.NWBHDF5IO(\n            path=lfp_band_file_abspath, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # get the indices of the electrodes in the electrode table of the file to get the right values\n            elect_index = get_electrode_indices(nwbf, lfp_band_elect_id)\n            electrode_table_region = nwbf.create_electrode_table_region(\n                elect_index, \"filtered electrode table\"\n            )\n            eseries_name = \"filtered data\"\n            # TODO: use datatype of data\n            es = pynwb.ecephys.ElectricalSeries(\n                name=eseries_name,\n                data=filtered_data,\n                electrodes=electrode_table_region,\n                timestamps=new_timestamps,\n            )\n            lfp = pynwb.ecephys.LFP(electrical_series=es)\n            ecephys_module = nwbf.create_processing_module(\n                name=\"ecephys\",\n                description=f\"LFP data processed with {filter_name}\",\n            )\n            ecephys_module.add(lfp)\n            io.write(nwbf)\n            lfp_band_object_id = es.object_id\n        #\n        # add the file to the AnalysisNwbfile table\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], lfp_band_file_name)\n        key[\"analysis_file_name\"] = lfp_band_file_name\n        key[\"lfp_band_object_id\"] = lfp_band_object_id\n\n        # finally, we need to censor the valid times to account for the downsampling if this is the first time we've\n        # downsampled these data\n        key[\"interval_list_name\"] = (\n            interval_list_name\n            + \" lfp band \"\n            + str(lfp_band_sampling_rate)\n            + \"Hz\"\n        )\n        tmp_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch(\"valid_times\")\n        if len(tmp_valid_times) == 0:\n            lfp_band_valid_times = interval_list_censor(\n                lfp_band_valid_times, new_timestamps\n            )\n            # add an interval list for the LFP valid times\n            IntervalList.insert1(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"interval_list_name\": key[\"interval_list_name\"],\n                    \"valid_times\": lfp_band_valid_times,\n                    \"pipeline\": \"lfp band\",\n                }\n            )\n        else:\n            lfp_band_valid_times = interval_list_censor(\n                lfp_band_valid_times, new_timestamps\n            )\n            # check that the valid times are the same\n            assert np.isclose(\n                tmp_valid_times[0], lfp_band_valid_times\n            ).all(), (\n                \"previously saved lfp band times do not match current times\"\n            )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        \"\"\"Fetches the filtered data as a dataframe\"\"\"\n        filtered_nwb = self.fetch_nwb()[0]\n        return pd.DataFrame(\n            filtered_nwb[\"lfp_band\"].data,\n            index=pd.Index(filtered_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n        )\n\n    def compute_analytic_signal(self, electrode_list: list[int], **kwargs):\n        \"\"\"Computes the hilbert transform of a given LFPBand signal\n\n        Uses scipy.signal.hilbert to compute the hilbert transform of the signal\n\n        Parameters\n        ----------\n        electrode_list: list[int]\n            A list of the electrodes to compute the hilbert transform of\n\n        Returns\n        -------\n        analytic_signal_df: pd.DataFrame\n            DataFrame containing hilbert transform of signal\n\n        Raises\n        ------\n        ValueError\n            If items in electrode_list are invalid for the dataset\n        \"\"\"\n\n        filtered_band = self.fetch_nwb()[0][\"lfp_band\"]\n        electrode_index = np.isin(\n            filtered_band.electrodes.data[:], electrode_list\n        )\n        if len(electrode_list) != np.sum(electrode_index):\n            raise ValueError(\n                \"Some of the electrodes specified in electrode_list are missing\"\n                + \" in the current LFPBand table.\"\n            )\n        analytic_signal_df = pd.DataFrame(\n            hilbert(filtered_band.data[:, electrode_index], axis=0),\n            index=pd.Index(filtered_band.timestamps, name=\"time\"),\n            columns=[f\"electrode {e}\" for e in electrode_list],\n        )\n        return analytic_signal_df\n\n    def compute_signal_phase(\n        self, electrode_list: list[int] = None, **kwargs\n    ) -&gt; pd.DataFrame:\n        \"\"\"Computes the phase of a given LFPBand signals using the hilbert transform\n\n        Parameters\n        ----------\n        electrode_list : list[int], optional\n            A list of the electrodes to compute the phase of, by default None\n\n        Returns\n        -------\n        signal_phase_df : pd.DataFrame\n            DataFrame containing the phase of the signals\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = []\n\n        analytic_signal_df = self.compute_analytic_signal(\n            electrode_list, **kwargs\n        )\n\n        return pd.DataFrame(\n            np.angle(analytic_signal_df) + np.pi,\n            columns=analytic_signal_df.columns,\n            index=analytic_signal_df.index,\n        )\n\n    def compute_signal_power(\n        self, electrode_list: list[int] = None, **kwargs\n    ) -&gt; pd.DataFrame:\n        \"\"\"Computes the power of a given LFPBand signals using the hilbert transform\n\n        Parameters\n        ----------\n        electrode_list : list[int], optional\n            A list of the electrodes to compute the power of, by default None\n\n        Returns\n        -------\n        signal_power_df : pd.DataFrame\n            DataFrame containing the power of the signals\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = []\n\n        analytic_signal_df = self.compute_analytic_signal(\n            electrode_list, **kwargs\n        )\n\n        return pd.DataFrame(\n            np.abs(analytic_signal_df) ** 2,\n            columns=analytic_signal_df.columns,\n            index=analytic_signal_df.index,\n        )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.fetch1_dataframe", "title": "<code>fetch1_dataframe(*attrs, **kwargs)</code>", "text": "<p>Fetches the filtered data as a dataframe</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def fetch1_dataframe(self, *attrs, **kwargs):\n    \"\"\"Fetches the filtered data as a dataframe\"\"\"\n    filtered_nwb = self.fetch_nwb()[0]\n    return pd.DataFrame(\n        filtered_nwb[\"lfp_band\"].data,\n        index=pd.Index(filtered_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n    )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_analytic_signal", "title": "<code>compute_analytic_signal(electrode_list, **kwargs)</code>", "text": "<p>Computes the hilbert transform of a given LFPBand signal</p> <p>Uses scipy.signal.hilbert to compute the hilbert transform of the signal</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the hilbert transform of</p> required <p>Returns:</p> Name Type Description <code>analytic_signal_df</code> <code>DataFrame</code> <p>DataFrame containing hilbert transform of signal</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>If items in electrode_list are invalid for the dataset</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_analytic_signal(self, electrode_list: list[int], **kwargs):\n    \"\"\"Computes the hilbert transform of a given LFPBand signal\n\n    Uses scipy.signal.hilbert to compute the hilbert transform of the signal\n\n    Parameters\n    ----------\n    electrode_list: list[int]\n        A list of the electrodes to compute the hilbert transform of\n\n    Returns\n    -------\n    analytic_signal_df: pd.DataFrame\n        DataFrame containing hilbert transform of signal\n\n    Raises\n    ------\n    ValueError\n        If items in electrode_list are invalid for the dataset\n    \"\"\"\n\n    filtered_band = self.fetch_nwb()[0][\"lfp_band\"]\n    electrode_index = np.isin(\n        filtered_band.electrodes.data[:], electrode_list\n    )\n    if len(electrode_list) != np.sum(electrode_index):\n        raise ValueError(\n            \"Some of the electrodes specified in electrode_list are missing\"\n            + \" in the current LFPBand table.\"\n        )\n    analytic_signal_df = pd.DataFrame(\n        hilbert(filtered_band.data[:, electrode_index], axis=0),\n        index=pd.Index(filtered_band.timestamps, name=\"time\"),\n        columns=[f\"electrode {e}\" for e in electrode_list],\n    )\n    return analytic_signal_df\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_signal_phase", "title": "<code>compute_signal_phase(electrode_list=None, **kwargs)</code>", "text": "<p>Computes the phase of a given LFPBand signals using the hilbert transform</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the phase of, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>signal_phase_df</code> <code>DataFrame</code> <p>DataFrame containing the phase of the signals</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_signal_phase(\n    self, electrode_list: list[int] = None, **kwargs\n) -&gt; pd.DataFrame:\n    \"\"\"Computes the phase of a given LFPBand signals using the hilbert transform\n\n    Parameters\n    ----------\n    electrode_list : list[int], optional\n        A list of the electrodes to compute the phase of, by default None\n\n    Returns\n    -------\n    signal_phase_df : pd.DataFrame\n        DataFrame containing the phase of the signals\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = []\n\n    analytic_signal_df = self.compute_analytic_signal(\n        electrode_list, **kwargs\n    )\n\n    return pd.DataFrame(\n        np.angle(analytic_signal_df) + np.pi,\n        columns=analytic_signal_df.columns,\n        index=analytic_signal_df.index,\n    )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_signal_power", "title": "<code>compute_signal_power(electrode_list=None, **kwargs)</code>", "text": "<p>Computes the power of a given LFPBand signals using the hilbert transform</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the power of, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>signal_power_df</code> <code>DataFrame</code> <p>DataFrame containing the power of the signals</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_signal_power(\n    self, electrode_list: list[int] = None, **kwargs\n) -&gt; pd.DataFrame:\n    \"\"\"Computes the power of a given LFPBand signals using the hilbert transform\n\n    Parameters\n    ----------\n    electrode_list : list[int], optional\n        A list of the electrodes to compute the power of, by default None\n\n    Returns\n    -------\n    signal_power_df : pd.DataFrame\n        DataFrame containing the power of the signals\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = []\n\n    analytic_signal_df = self.compute_analytic_signal(\n        electrode_list, **kwargs\n    )\n\n    return pd.DataFrame(\n        np.abs(analytic_signal_df) ** 2,\n        columns=analytic_signal_df.columns,\n        index=analytic_signal_df.index,\n    )\n</code></pre>"}, {"location": "api/lfp/v1/lfp/", "title": "lfp.py", "text": ""}, {"location": "api/lfp/v1/lfp/#src.spyglass.lfp.v1.lfp.LFPSelection", "title": "<code>LFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>The user's selection of LFP data to be filtered</p> <p>This table is used to select the LFP data to be filtered.  The user can select the LFP data by specifying the electrode group and the interval list to be used. The interval list is used to select the times from the raw data that will be filtered.  The user can also specify the filter to be used.</p> <p>The LFP data is filtered and downsampled to the user-defined sampling rate, specified as lfp_sampling_rate.  The filtered data is stored in the AnalysisNwbfile table. The valid times for the filtered data are stored in the IntervalList table.</p> Source code in <code>src/spyglass/lfp/v1/lfp.py</code> <pre><code>@schema\nclass LFPSelection(SpyglassMixin, dj.Manual):\n    \"\"\"The user's selection of LFP data to be filtered\n\n    This table is used to select the LFP data to be filtered.  The user can select\n    the LFP data by specifying the electrode group and the interval list to be used.\n    The interval list is used to select the times from the raw data that will be\n    filtered.  The user can also specify the filter to be used.\n\n    The LFP data is filtered and downsampled to the user-defined sampling rate, specified\n    as lfp_sampling_rate.  The filtered data is stored in the AnalysisNwbfile table.\n    The valid times for the filtered data are stored in the IntervalList table.\n    \"\"\"\n\n    definition = \"\"\"\n     -&gt; LFPElectrodeGroup                                                  # the group of electrodes to be filtered\n     -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n     -&gt; FirFilterParameters                                                # the filter to be used\n     ---\n     target_sampling_rate = 1000 : float                                     # the desired output sampling rate, in HZ\n     \"\"\"\n</code></pre>"}, {"location": "api/lfp/v1/lfp/#src.spyglass.lfp.v1.lfp.LFPV1", "title": "<code>LFPV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>The filtered LFP data</p> Source code in <code>src/spyglass/lfp/v1/lfp.py</code> <pre><code>@schema\nclass LFPV1(SpyglassMixin, dj.Computed):\n    \"\"\"The filtered LFP data\"\"\"\n\n    definition = \"\"\"\n    -&gt; LFPSelection             # the user's selection of LFP data to be filtered\n    ---\n    -&gt; AnalysisNwbfile          # the name of the nwb file with the lfp data\n    -&gt; IntervalList             # the final interval list of valid times for the data\n    lfp_object_id: varchar(40)  # the NWB object ID for loading this object from the file\n    lfp_sampling_rate: float    # the sampling rate, in HZ\n    \"\"\"\n\n    def make(self, key):\n        # get the NWB object with the data\n        nwbf_key = {\"nwb_file_name\": key[\"nwb_file_name\"]}\n        rawdata = (Raw &amp; nwbf_key).fetch_nwb()[0][\"raw\"]\n\n        # CBroz: assumes Raw sampling rate matches FirFilterParameters set?\n        #        if we just pull rate from Raw, why include in Param table?\n        sampling_rate, raw_interval_list_name = (Raw &amp; nwbf_key).fetch1(\n            \"sampling_rate\", \"interval_list_name\"\n        )\n        sampling_rate = int(np.round(sampling_rate))\n        target_sampling_rate = (LFPSelection &amp; key).fetch1(\n            \"target_sampling_rate\"\n        )\n\n        # to get the list of valid times, we need to combine those from the\n        # user with those from the raw data\n        orig_key = copy.deepcopy(key)\n        orig_key[\"interval_list_name\"] = key[\"target_interval_list_name\"]\n        user_valid_times = (IntervalList() &amp; orig_key).fetch1(\"valid_times\")\n        # we remove the extra entry so we can insert this into the LFPOutput table.\n        del orig_key[\"interval_list_name\"]\n\n        raw_valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": raw_interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        valid_times = interval_list_intersect(\n            user_valid_times,\n            raw_valid_times,\n            min_length=MIN_LFP_INTERVAL_DURATION,\n        )\n        logger.info(\n            f\"LFP: found {len(valid_times)} intervals &gt; \"\n            + f\"{MIN_LFP_INTERVAL_DURATION} sec long.\"\n        )\n        # target user-specified sampling rate\n        decimation = int(sampling_rate // target_sampling_rate)\n\n        # get the LFP filter that matches the raw data\n        filter = (\n            FirFilterParameters()\n            &amp; {\n                \"filter_name\": key[\"filter_name\"],\n                \"filter_sampling_rate\": sampling_rate,\n            }  # not key['filter_sampling_rate']?\n        ).fetch(as_dict=True)[0]\n\n        # there should only be one filter that matches, so we take the first of\n        # the dictionaries\n\n        key[\"filter_name\"] = filter[\"filter_name\"]\n        key[\"filter_sampling_rate\"] = filter[\"filter_sampling_rate\"]\n\n        filter_coeff = filter[\"filter_coeff\"]\n        if len(filter_coeff) == 0:\n            logger.error(\n                \"LFP: no filter found with data sampling rate of \"\n                + f\"{sampling_rate}\"\n            )\n            return None  # See #849\n\n        # get the list of selected LFP Channels from LFPElectrode\n        electrode_keys = (LFPElectrodeGroup.LFPElectrode &amp; key).fetch(\"KEY\")\n        electrode_id_list = list(k[\"electrode_id\"] for k in electrode_keys)\n        electrode_id_list.sort()\n\n        lfp_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        lfp_file_abspath = AnalysisNwbfile().get_abs_path(lfp_file_name)\n        (\n            lfp_object_id,\n            timestamp_interval,\n        ) = FirFilterParameters().filter_data_nwb(\n            lfp_file_abspath,\n            rawdata,\n            filter_coeff,\n            valid_times,\n            electrode_id_list,\n            decimation,\n        )\n\n        # now that the LFP is filtered and in the file, add the file to the\n        # AnalysisNwbfile table\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], lfp_file_name)\n\n        key[\"analysis_file_name\"] = lfp_file_name\n        key[\"lfp_object_id\"] = lfp_object_id\n        key[\"lfp_sampling_rate\"] = sampling_rate // decimation\n\n        # need to censor the valid times to account for the downsampling\n        lfp_valid_times = interval_list_censor(valid_times, timestamp_interval)\n\n        # add an interval list for the LFP valid times, or check that it matches the existing one\n        key[\"interval_list_name\"] = \"_\".join(\n            (\n                \"lfp\",\n                key[\"lfp_electrode_group_name\"],\n                key[\"target_interval_list_name\"],\n                \"valid times\",\n            )\n        )\n\n        tmp_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch(\"valid_times\")\n        if len(tmp_valid_times) == 0:\n            IntervalList.insert1(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"interval_list_name\": key[\"interval_list_name\"],\n                    \"valid_times\": lfp_valid_times,\n                    \"pipeline\": \"lfp_v1\",\n                },\n                replace=True,\n            )\n        elif not np.allclose(tmp_valid_times[0], lfp_valid_times):\n            raise ValueError(\n                \"previously saved lfp times do not match current times\"\n            )\n        self.insert1(key)\n\n        # finally, we insert this into the LFP output table.\n        from spyglass.lfp.lfp_merge import LFPOutput\n\n        orig_key[\"analysis_file_name\"] = lfp_file_name\n        orig_key[\"lfp_object_id\"] = lfp_object_id\n        LFPOutput.insert1(orig_key)\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        nwb_lfp = self.fetch_nwb()[0]\n        return pd.DataFrame(\n            nwb_lfp[\"lfp\"].data,\n            index=pd.Index(nwb_lfp[\"lfp\"].timestamps, name=\"time\"),\n        )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact/", "title": "lfp_artifact.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact/#src.spyglass.lfp.v1.lfp_artifact.LFPArtifactDetectionParameters", "title": "<code>LFPArtifactDetectionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact.py</code> <pre><code>@schema\nclass LFPArtifactDetectionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for detecting LFP artifact times within a LFP group.\n    artifact_params_name: varchar(64)\n    ---\n    artifact_params: blob  # dictionary of parameters\n    \"\"\"\n\n    # See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Insert the default artifact parameters.\"\"\"\n        diff_params = [\n            \"default_difference\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": 0.1,\n                    \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": 0.05,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                    \"local_window_ms\": 40.0,  # in milliseconds\n                },\n            },\n        ]\n\n        diff_ref_params = [\n            \"default_difference_ref\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": 0.1,\n                    \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": 0.05,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                    \"local_window_ms\": 40.0,  # in milliseconds\n                },\n                \"referencing\": {\n                    \"ref_on\": 1,\n                    \"reference_list\": [0, 0, 0, 0, 0],\n                    \"electrode_list\": [0, 0],\n                },\n            },\n        ]\n\n        no_params = [\n            \"none\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": None,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": None,\n                    \"amplitude_thresh_2nd\": None,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": None,\n                    \"removal_window_ms\": None,  # in milliseconds\n                    \"local_window_ms\": None,  # in milliseconds\n                },\n            },\n        ]\n\n        mad_params = [\n            \"default_mad\",\n            {\n                \"artifact_detection_algorithm\": \"mad\",\n                \"artifact_detection_algorithm_params\": {\n                    # akin to z-score std dev if the distribution is normal\n                    \"mad_thresh\": 6.0,\n                    \"proportion_above_thresh\": 0.1,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                },\n            },\n        ]\n\n        self.insert(\n            [diff_params, diff_ref_params, no_params, mad_params],\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact/#src.spyglass.lfp.v1.lfp_artifact.LFPArtifactDetectionParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default artifact parameters.</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default artifact parameters.\"\"\"\n    diff_params = [\n        \"default_difference\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": 0.1,\n                \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": 0.05,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n                \"local_window_ms\": 40.0,  # in milliseconds\n            },\n        },\n    ]\n\n    diff_ref_params = [\n        \"default_difference_ref\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": 0.1,\n                \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": 0.05,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n                \"local_window_ms\": 40.0,  # in milliseconds\n            },\n            \"referencing\": {\n                \"ref_on\": 1,\n                \"reference_list\": [0, 0, 0, 0, 0],\n                \"electrode_list\": [0, 0],\n            },\n        },\n    ]\n\n    no_params = [\n        \"none\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": None,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": None,\n                \"amplitude_thresh_2nd\": None,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": None,\n                \"removal_window_ms\": None,  # in milliseconds\n                \"local_window_ms\": None,  # in milliseconds\n            },\n        },\n    ]\n\n    mad_params = [\n        \"default_mad\",\n        {\n            \"artifact_detection_algorithm\": \"mad\",\n            \"artifact_detection_algorithm_params\": {\n                # akin to z-score std dev if the distribution is normal\n                \"mad_thresh\": 6.0,\n                \"proportion_above_thresh\": 0.1,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n            },\n        },\n    ]\n\n    self.insert(\n        [diff_params, diff_ref_params, no_params, mad_params],\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact_MAD_detection/", "title": "lfp_artifact_MAD_detection.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact_MAD_detection/#src.spyglass.lfp.v1.lfp_artifact_MAD_detection.mad_artifact_detector", "title": "<code>mad_artifact_detector(recording, mad_thresh=6.0, proportion_above_thresh=0.1, removal_window_ms=10.0, sampling_frequency=1000.0, *args, **kwargs)</code>", "text": "<p>Detect LFP artifacts using the median absolute deviation method.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>RecordingExtractor</code> <p>The recording extractor object</p> required <code>mad_thresh</code> <code>float</code> <p>Threshold on the median absolute deviation scaled LFPs, defaults to 6.0</p> <code>6.0</code> <code>proportion_above_thresh</code> <code>float</code> <p>Proportion of electrodes that need to be above the threshold, defaults to 1.0</p> <code>0.1</code> <code>removal_window_ms</code> <code>float</code> <p>Width of the window in milliseconds to mask out per artifact (window/2 removed on each side of threshold crossing), defaults to 1 ms</p> <code>10.0</code> <code>sampling_frequency</code> <code>float</code> <p>Sampling frequency of the recording extractor, defaults to 1000.0</p> <code>1000.0</code> <p>Returns:</p> Name Type Description <code>artifact_removed_valid_times</code> <code>ndarray</code> <p>Intervals of valid times where artifacts were not detected, unit: seconds</p> <code>artifact_intervals</code> <code>ndarray</code> <p>Intervals in which artifacts are detected (including removal windows), unit: seconds</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact_MAD_detection.py</code> <pre><code>def mad_artifact_detector(\n    recording: None,\n    mad_thresh: float = 6.0,\n    proportion_above_thresh: float = 0.1,\n    removal_window_ms: float = 10.0,\n    sampling_frequency: float = 1000.0,\n    *args,\n    **kwargs,\n) -&gt; tuple[np.ndarray, np.ndarray]:\n    \"\"\"Detect LFP artifacts using the median absolute deviation method.\n\n    Parameters\n    ----------\n    recording : RecordingExtractor\n        The recording extractor object\n    mad_thresh : float, optional\n        Threshold on the median absolute deviation scaled LFPs, defaults to 6.0\n    proportion_above_thresh : float, optional\n        Proportion of electrodes that need to be above the threshold, defaults to 1.0\n    removal_window_ms : float, optional\n        Width of the window in milliseconds to mask out per artifact\n        (window/2 removed on each side of threshold crossing), defaults to 1 ms\n    sampling_frequency : float, optional\n        Sampling frequency of the recording extractor, defaults to 1000.0\n\n    Returns\n    -------\n    artifact_removed_valid_times : np.ndarray\n        Intervals of valid times where artifacts were not detected, unit: seconds\n    artifact_intervals : np.ndarray\n        Intervals in which artifacts are detected (including removal windows), unit: seconds\n\n    \"\"\"\n\n    timestamps = np.asarray(recording.timestamps)\n    lfps = np.asarray(recording.data)\n\n    mad = median_abs_deviation(lfps, axis=0, nan_policy=\"omit\", scale=\"normal\")\n    is_artifact = _is_above_proportion_thresh(\n        _mad_scale_lfps(lfps, mad), mad_thresh, proportion_above_thresh\n    )\n\n    MILLISECONDS_PER_SECOND = 1000.0\n    half_removal_window_s = (removal_window_ms / MILLISECONDS_PER_SECOND) * 0.5\n    half_removal_window_idx = int(half_removal_window_s * sampling_frequency)\n    is_artifact = _extend_array_by_window(is_artifact, half_removal_window_idx)\n\n    artifact_intervals_s = _get_time_intervals_from_bool_array(\n        is_artifact, timestamps\n    )\n\n    valid_times = _get_time_intervals_from_bool_array(~is_artifact, timestamps)\n\n    return valid_times, artifact_intervals_s\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact_difference_detection/", "title": "lfp_artifact_difference_detection.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact_difference_detection/#src.spyglass.lfp.v1.lfp_artifact_difference_detection.difference_artifact_detector", "title": "<code>difference_artifact_detector(recording, timestamps, amplitude_thresh_1st=None, amplitude_thresh_2nd=None, proportion_above_thresh_1st=1.0, proportion_above_thresh_2nd=1.0, removal_window_ms=1.0, local_window_ms=1.0, sampling_frequency=1000.0, referencing=False)</code>", "text": "<p>Detects times during which artifacts do and do not occur.</p> <p>Artifacts are defined as periods where the absolute value of the change in LFP exceeds amplitude change thresholds on the proportion of channels specified, with the period extended by the removal_window_ms/2 on each side. amplitude change threshold values of None are ignored.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>lfp eseries zscore_thresh : float</code> <p>Stdev threshold for exclusion, should be &gt;=0, defaults to None</p> required <code>amplitude_thresh_1st</code> <code>float</code> <p>Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to None</p> <code>None</code> <code>amplitude_thresh_2nd</code> <code>float</code> <p>Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to None</p> <code>None</code> <code>proportion_above_thresh_1st</code> <code>float, optional, should be&gt;0 and &lt;=1</code> <p>Proportion of electrodes that need to have threshold crossings, defaults to 1</p> <code>1.0</code> <code>proportion_above_thresh_2nd</code> <code>float, optional, should be&gt;0 and &lt;=1</code> <p>Proportion of electrodes that need to have threshold crossings, defaults to 1</p> <code>1.0</code> <code>removal_window_ms</code> <code>float</code> <p>Width of the window in milliseconds to mask out per artifact (window/2 removed on each side of threshold crossing), defaults to 1 ms</p> <code>1.0</code> <code>referencing</code> <code>bool</code> <p>Whether or not the data passed to this function is referenced, defaults to False</p> <code>False</code> <p>Returns:</p> Name Type Description <code>artifact_removed_valid_times</code> <code>ndarray</code> <p>Intervals of valid times where artifacts were not detected, unit: seconds</p> <code>artifact_intervals</code> <code>ndarray</code> <p>Intervals in which artifacts are detected (including removal windows), unit: seconds</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact_difference_detection.py</code> <pre><code>def difference_artifact_detector(\n    recording: None,\n    timestamps: None,\n    amplitude_thresh_1st: Union[float, None] = None,\n    amplitude_thresh_2nd: Union[float, None] = None,\n    proportion_above_thresh_1st: float = 1.0,\n    proportion_above_thresh_2nd: float = 1.0,\n    removal_window_ms: float = 1.0,\n    local_window_ms: float = 1.0,\n    sampling_frequency: float = 1000.0,\n    referencing: bool = False,\n):\n    \"\"\"Detects times during which artifacts do and do not occur.\n\n    Artifacts are defined as periods where the absolute value of the change in\n    LFP exceeds amplitude change thresholds on the proportion of channels\n    specified, with the period extended by the removal_window_ms/2 on each side.\n    amplitude change threshold values of None are ignored.\n\n    Parameters\n    ----------\n    recording : lfp eseries zscore_thresh : float, optional\n        Stdev threshold for exclusion, should be &gt;=0, defaults to None\n    amplitude_thresh_1st : float, optional\n        Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to\n        None\n    amplitude_thresh_2nd : float, optional\n        Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to\n        None\n    proportion_above_thresh_1st : float, optional, should be&gt;0 and &lt;=1\n        Proportion of electrodes that need to have threshold crossings, defaults\n        to 1\n    proportion_above_thresh_2nd : float, optional, should be&gt;0 and &lt;=1\n        Proportion of electrodes that need to have threshold crossings, defaults\n        to 1\n    removal_window_ms : float, optional\n        Width of the window in milliseconds to mask out per artifact (window/2\n        removed on each side of threshold crossing), defaults to 1 ms\n    referencing : bool, optional\n        Whether or not the data passed to this function is referenced, defaults to False\n\n    Returns\n    -------\n    artifact_removed_valid_times : np.ndarray\n        Intervals of valid times where artifacts were not detected, unit:\n        seconds\n    artifact_intervals : np.ndarray\n        Intervals in which artifacts are detected (including removal windows),\n        unit: seconds\n    \"\"\"\n\n    # NOTE: 7-17-23 updated to remove recording.data, since it will converted to\n    # numpy array before referencing check for referencing flag\n\n    if referencing:\n        logger.info(\"referencing activated. may be set to -1\")\n\n    # valid_timestamps = recording.timestamps\n    valid_timestamps = timestamps\n\n    local_window = int(local_window_ms / 2)\n\n    # if both thresholds are None, we skip artifact detection\n    if amplitude_thresh_1st is None:\n        recording_interval = np.asarray(\n            [valid_timestamps[0], valid_timestamps[-1]]\n        )\n        artifact_times_empty = np.asarray([])\n        logger.info(\"Amplitude threshold is None, skipping artifact detection\")\n        return recording_interval, artifact_times_empty\n\n    # verify threshold parameters\n    (\n        amplitude_thresh_1st,\n        amplitude_thresh_2nd,\n        proportion_above_thresh_1st,\n        proportion_above_thresh_2nd,\n    ) = _check_artifact_thresholds(\n        amplitude_thresh_1st,\n        amplitude_thresh_2nd,\n        proportion_above_thresh_1st,\n        proportion_above_thresh_2nd,\n    )\n\n    # want to detect frames without parallel processing\n    # compute the number of electrodes that have to be above threshold\n    nelect_above_1st = np.ceil(proportion_above_thresh_1st * recording.shape[1])\n    nelect_above_2nd = np.ceil(proportion_above_thresh_2nd * recording.shape[1])\n    logger.info(\"num tets 1\", nelect_above_1st, \"num tets 2\", nelect_above_2nd)\n    logger.info(\"data shape\", recording.shape)\n\n    # find the artifact occurrences using one or both thresholds, across\n    # channels\n\n    if amplitude_thresh_1st is not None:\n        # first find times with large amp change: sum diff over several timebins\n\n        diff_array = np.diff(recording, axis=0)\n        window = np.ones((3, 1)) if referencing else np.ones((15, 1))\n\n        # sum differences over bins using convolution for speed\n        width = int((window.size - 1) / 2)\n        diff_array = np.pad(\n            diff_array,\n            pad_width=((width, width), (0, 0)),\n            mode=\"constant\",\n        )\n        diff_array_5 = scipy.signal.convolve(diff_array, window, mode=\"valid\")\n\n        artifact_times_all_5 = np.sum(\n            (np.abs(diff_array_5) &gt; amplitude_thresh_1st), axis=1\n        )\n\n        above_thresh_1st = np.where(artifact_times_all_5 &gt;= nelect_above_1st)[0]\n\n        # second, find artifacts with large baseline change\n        logger.info(\"thresh\", amplitude_thresh_2nd, \"window\", local_window)\n\n        big_artifacts = np.zeros(\n            (recording.shape[1], above_thresh_1st.shape[0])\n        )\n        for art_count in np.arange(above_thresh_1st.shape[0]):\n            if above_thresh_1st[art_count] &lt;= local_window:\n                local_min = local_max = above_thresh_1st[art_count]\n            else:\n                local_max = np.max(\n                    recording[\n                        above_thresh_1st[art_count]\n                        - local_window : above_thresh_1st[art_count]\n                        + local_window,\n                        :,\n                    ],\n                    axis=0,\n                )\n                local_min = np.min(\n                    recording[\n                        above_thresh_1st[art_count]\n                        - local_window : above_thresh_1st[art_count]\n                        + local_window,\n                        :,\n                    ],\n                    axis=0,\n                )\n            big_artifacts[:, art_count] = (\n                np.abs(local_max - local_min) &gt; amplitude_thresh_2nd\n            )\n\n        # sum columns in big artficat, then compare to nelect_above_2nd\n        above_thresh = above_thresh_1st[\n            np.sum(big_artifacts, axis=0) &gt;= nelect_above_2nd\n        ]\n\n    artifact_frames = above_thresh.copy()\n    logger.info(\"detected \", artifact_frames.shape[0], \" artifacts\")\n\n    # Convert to s to remove from either side of each detected artifact\n    half_removal_window_s = removal_window_ms / 1000 * 0.5\n\n    if len(artifact_frames) == 0:\n        recording_interval = np.asarray(\n            [[valid_timestamps[0], valid_timestamps[-1]]]\n        )\n        artifact_times_empty = np.asarray([])\n        logger.info(\"No artifacts detected.\")\n        return recording_interval, artifact_times_empty\n\n    artifact_intervals = interval_from_inds(artifact_frames)\n\n    artifact_intervals_s = np.zeros(\n        (len(artifact_intervals), 2), dtype=np.float64\n    )\n\n    for interval_idx, interval in enumerate(artifact_intervals):\n        artifact_intervals_s[interval_idx] = [\n            valid_timestamps[interval[0]] - half_removal_window_s,\n            valid_timestamps[interval[1]] + half_removal_window_s,\n        ]\n    artifact_intervals_s = reduce(_union_concat, artifact_intervals_s)\n\n    valid_intervals = get_valid_intervals(\n        valid_timestamps, sampling_frequency, 1.5, 0.000001\n    )\n\n    # these are artifact times - need to subtract these from valid timestamps\n    artifact_valid_times = interval_list_intersect(\n        valid_intervals, artifact_intervals_s\n    )\n\n    # note: this is a slow step\n    list_triggers = []\n    for interval in artifact_valid_times:\n        list_triggers.append(\n            np.arange(\n                np.searchsorted(valid_timestamps, interval[0]),\n                np.searchsorted(valid_timestamps, interval[1]),\n            )\n        )\n\n    new_array = np.array(np.concatenate(list_triggers))\n\n    new_timestamps = np.delete(valid_timestamps, new_array)\n\n    artifact_removed_valid_times = get_valid_intervals(\n        new_timestamps, sampling_frequency, 1.5, 0.000001\n    )\n\n    return artifact_removed_valid_times, artifact_intervals_s\n</code></pre>"}, {"location": "api/linearization/merge/", "title": "merge.py", "text": ""}, {"location": "api/linearization/v0/main/", "title": "main.py", "text": ""}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.LinearizationParameters", "title": "<code>LinearizationParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Choose whether to use an HMM to linearize position.</p> <p>This can help when the euclidean distances between separate arms are too close and the previous position has some information about which arm the animal is on.</p> <p>route_euclidean_distance_scaling: How much to prefer route distances between successive time points that are closer to the euclidean distance. Smaller numbers mean the route distance is more likely to be close to the euclidean distance.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass LinearizationParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Choose whether to use an HMM to linearize position.\n\n    This can help when the euclidean distances between separate arms are too\n    close and the previous position has some information about which arm the\n    animal is on.\n\n    route_euclidean_distance_scaling: How much to prefer route distances between\n    successive time points that are closer to the euclidean distance. Smaller\n    numbers mean the route distance is more likely to be close to the euclidean\n    distance.\n    \"\"\"\n\n    definition = \"\"\"\n    linearization_param_name : varchar(80)   # name for this set of parameters\n    ---\n    use_hmm = 0 : int   # use HMM to determine linearization\n    route_euclidean_distance_scaling = 1.0 : float # Preference for euclidean.\n    sensor_std_dev = 5.0 : float   # Uncertainty of position sensor (in cm).\n    # Biases the transition matrix to prefer the current track segment.\n    diagonal_bias = 0.5 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph", "title": "<code>TrackGraph</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Graph representation of track representing the spatial environment.</p> <p>Used for linearizing position.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass TrackGraph(SpyglassMixin, dj.Manual):\n    \"\"\"Graph representation of track representing the spatial environment.\n\n    Used for linearizing position.\n    \"\"\"\n\n    definition = \"\"\"\n    track_graph_name : varchar(80)\n    ----\n    environment : varchar(80)  # Type of Environment\n    node_positions : blob      # 2D position of nodes, (n_nodes, 2)\n    edges: blob                # shape (n_edges, 2)\n    linear_edge_order : blob   # order of edges in linear space, (n_edges, 2)\n    linear_edge_spacing : blob # space btwn edges in linear space, (n_edges,)\n    \"\"\"\n\n    def get_networkx_track_graph(self, track_graph_parameters=None):\n        if track_graph_parameters is None:\n            track_graph_parameters = self.fetch1()\n        return make_track_graph(\n            node_positions=track_graph_parameters[\"node_positions\"],\n            edges=track_graph_parameters[\"edges\"],\n        )\n\n    def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n        \"\"\"Plot the track graph in 2D position space.\"\"\"\n        track_graph = self.get_networkx_track_graph()\n        plot_track_graph(\n            track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n        )\n\n    def plot_track_graph_as_1D(\n        self,\n        ax=None,\n        axis=\"x\",\n        other_axis_start=0.0,\n        draw_edge_labels=False,\n        node_size=300,\n        node_color=\"#1f77b4\",\n    ):\n        \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n        track_graph_parameters = self.fetch1()\n        track_graph = self.get_networkx_track_graph(\n            track_graph_parameters=track_graph_parameters\n        )\n        plot_graph_as_1D(\n            track_graph,\n            edge_order=track_graph_parameters[\"linear_edge_order\"],\n            edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n            ax=ax,\n            axis=axis,\n            other_axis_start=other_axis_start,\n            draw_edge_labels=draw_edge_labels,\n            node_size=node_size,\n            node_color=node_color,\n        )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph.plot_track_graph", "title": "<code>plot_track_graph(ax=None, draw_edge_labels=False, **kwds)</code>", "text": "<p>Plot the track graph in 2D position space.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n    \"\"\"Plot the track graph in 2D position space.\"\"\"\n    track_graph = self.get_networkx_track_graph()\n    plot_track_graph(\n        track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n    )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph.plot_track_graph_as_1D", "title": "<code>plot_track_graph_as_1D(ax=None, axis='x', other_axis_start=0.0, draw_edge_labels=False, node_size=300, node_color='#1f77b4')</code>", "text": "<p>Plot the track graph in 1D to see how the linearization is set up.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>def plot_track_graph_as_1D(\n    self,\n    ax=None,\n    axis=\"x\",\n    other_axis_start=0.0,\n    draw_edge_labels=False,\n    node_size=300,\n    node_color=\"#1f77b4\",\n):\n    \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n    track_graph_parameters = self.fetch1()\n    track_graph = self.get_networkx_track_graph(\n        track_graph_parameters=track_graph_parameters\n    )\n    plot_graph_as_1D(\n        track_graph,\n        edge_order=track_graph_parameters[\"linear_edge_order\"],\n        edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n        ax=ax,\n        axis=axis,\n        other_axis_start=other_axis_start,\n        draw_edge_labels=draw_edge_labels,\n        node_size=node_size,\n        node_color=node_color,\n    )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.IntervalLinearizedPosition", "title": "<code>IntervalLinearizedPosition</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Linearized position for a given interval</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass IntervalLinearizedPosition(SpyglassMixin, dj.Computed):\n    \"\"\"Linearized position for a given interval\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalLinearizationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    linearized_position_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing linear position for: {key}\")\n\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n\n        position_nwb = (\n            IntervalPositionInfo\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n                \"position_info_param_name\": key[\"position_info_param_name\"],\n            }\n        ).fetch_nwb()[0]\n\n        position = np.asarray(\n            position_nwb[\"head_position\"].get_spatial_series().data\n        )\n        time = np.asarray(\n            position_nwb[\"head_position\"].get_spatial_series().timestamps\n        )\n\n        linearization_parameters = (\n            LinearizationParameters()\n            &amp; {\"linearization_param_name\": key[\"linearization_param_name\"]}\n        ).fetch1()\n        track_graph_info = (\n            TrackGraph() &amp; {\"track_graph_name\": key[\"track_graph_name\"]}\n        ).fetch1()\n\n        track_graph = make_track_graph(\n            node_positions=track_graph_info[\"node_positions\"],\n            edges=track_graph_info[\"edges\"],\n        )\n\n        linear_position_df = get_linearized_position(\n            position=position,\n            track_graph=track_graph,\n            edge_spacing=track_graph_info[\"linear_edge_spacing\"],\n            edge_order=track_graph_info[\"linear_edge_order\"],\n            use_HMM=linearization_parameters[\"use_hmm\"],\n            route_euclidean_distance_scaling=linearization_parameters[\n                \"route_euclidean_distance_scaling\"\n            ],\n            sensor_std_dev=linearization_parameters[\"sensor_std_dev\"],\n            diagonal_bias=linearization_parameters[\"diagonal_bias\"],\n        )\n\n        linear_position_df[\"time\"] = time\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n\n        key[\"linearized_position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=linear_position_df,\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        return self.fetch_nwb()[0][\"linearized_position\"].set_index(\"time\")\n</code></pre>"}, {"location": "api/linearization/v1/main/", "title": "main.py", "text": ""}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.LinearizationParameters", "title": "<code>LinearizationParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Choose whether to use an HMM to linearize position. This can help when the eucledian distances between separate arms are too close and the previous position has some information about which arm the animal is on.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass LinearizationParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Choose whether to use an HMM to linearize position. This can help when\n    the eucledian distances between separate arms are too close and the previous\n    position has some information about which arm the animal is on.\"\"\"\n\n    definition = \"\"\"\n    linearization_param_name : varchar(80)   # name for this set of parameters\n    ---\n    use_hmm = 0 : int   # use HMM to determine linearization\n    # How much to prefer route distances between successive time points that are closer to the euclidean distance. Smaller numbers mean the route distance is more likely to be close to the euclidean distance.\n    route_euclidean_distance_scaling = 1.0 : float\n    sensor_std_dev = 5.0 : float   # Uncertainty of position sensor (in cm).\n    # Biases the transition matrix to prefer the current track segment.\n    diagonal_bias = 0.5 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph", "title": "<code>TrackGraph</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Graph representation of track representing the spatial environment. Used for linearizing position.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass TrackGraph(SpyglassMixin, dj.Manual):\n    \"\"\"Graph representation of track representing the spatial environment.\n    Used for linearizing position.\"\"\"\n\n    definition = \"\"\"\n    track_graph_name : varchar(80)\n    ----\n    environment : varchar(80)    # Type of Environment\n    node_positions : blob  # 2D position of track_graph nodes, shape (n_nodes, 2)\n    edges: blob                  # shape (n_edges, 2)\n    linear_edge_order : blob  # order of track graph edges in the linear space, shape (n_edges, 2)\n    linear_edge_spacing : blob  # amount of space between edges in the linear space, shape (n_edges,)\n    \"\"\"\n\n    def get_networkx_track_graph(self, track_graph_parameters=None):\n        if track_graph_parameters is None:\n            track_graph_parameters = self.fetch1()\n        return make_track_graph(\n            node_positions=track_graph_parameters[\"node_positions\"],\n            edges=track_graph_parameters[\"edges\"],\n        )\n\n    def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n        \"\"\"Plot the track graph in 2D position space.\"\"\"\n        track_graph = self.get_networkx_track_graph()\n        plot_track_graph(\n            track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n        )\n\n    def plot_track_graph_as_1D(\n        self,\n        ax=None,\n        axis=\"x\",\n        other_axis_start=0.0,\n        draw_edge_labels=False,\n        node_size=300,\n        node_color=\"#1f77b4\",\n    ):\n        \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n        track_graph_parameters = self.fetch1()\n        track_graph = self.get_networkx_track_graph(\n            track_graph_parameters=track_graph_parameters\n        )\n        plot_graph_as_1D(\n            track_graph,\n            edge_order=track_graph_parameters[\"linear_edge_order\"],\n            edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n            ax=ax,\n            axis=axis,\n            other_axis_start=other_axis_start,\n            draw_edge_labels=draw_edge_labels,\n            node_size=node_size,\n            node_color=node_color,\n        )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph.plot_track_graph", "title": "<code>plot_track_graph(ax=None, draw_edge_labels=False, **kwds)</code>", "text": "<p>Plot the track graph in 2D position space.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n    \"\"\"Plot the track graph in 2D position space.\"\"\"\n    track_graph = self.get_networkx_track_graph()\n    plot_track_graph(\n        track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n    )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph.plot_track_graph_as_1D", "title": "<code>plot_track_graph_as_1D(ax=None, axis='x', other_axis_start=0.0, draw_edge_labels=False, node_size=300, node_color='#1f77b4')</code>", "text": "<p>Plot the track graph in 1D to see how the linearization is set up.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>def plot_track_graph_as_1D(\n    self,\n    ax=None,\n    axis=\"x\",\n    other_axis_start=0.0,\n    draw_edge_labels=False,\n    node_size=300,\n    node_color=\"#1f77b4\",\n):\n    \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n    track_graph_parameters = self.fetch1()\n    track_graph = self.get_networkx_track_graph(\n        track_graph_parameters=track_graph_parameters\n    )\n    plot_graph_as_1D(\n        track_graph,\n        edge_order=track_graph_parameters[\"linear_edge_order\"],\n        edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n        ax=ax,\n        axis=axis,\n        other_axis_start=other_axis_start,\n        draw_edge_labels=draw_edge_labels,\n        node_size=node_size,\n        node_color=node_color,\n    )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.LinearizedPositionV1", "title": "<code>LinearizedPositionV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Linearized position for a given interval</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass LinearizedPositionV1(SpyglassMixin, dj.Computed):\n    \"\"\"Linearized position for a given interval\"\"\"\n\n    definition = \"\"\"\n    -&gt; LinearizationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    linearized_position_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n        logger.info(f\"Computing linear position for: {key}\")\n\n        position_nwb = PositionOutput().fetch_nwb(\n            {\"merge_id\": key[\"pos_merge_id\"]}\n        )[0]\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            position_nwb[\"nwb_file_name\"]\n        )\n        position = np.asarray(\n            position_nwb[\"position\"].get_spatial_series().data\n        )\n        time = np.asarray(\n            position_nwb[\"position\"].get_spatial_series().timestamps\n        )\n\n        linearization_parameters = (\n            LinearizationParameters()\n            &amp; {\"linearization_param_name\": key[\"linearization_param_name\"]}\n        ).fetch1()\n        track_graph_info = (\n            TrackGraph() &amp; {\"track_graph_name\": key[\"track_graph_name\"]}\n        ).fetch1()\n\n        track_graph = make_track_graph(\n            node_positions=track_graph_info[\"node_positions\"],\n            edges=track_graph_info[\"edges\"],\n        )\n\n        linear_position_df = get_linearized_position(\n            position=position,\n            track_graph=track_graph,\n            edge_spacing=track_graph_info[\"linear_edge_spacing\"],\n            edge_order=track_graph_info[\"linear_edge_order\"],\n            use_HMM=linearization_parameters[\"use_hmm\"],\n            route_euclidean_distance_scaling=linearization_parameters[\n                \"route_euclidean_distance_scaling\"\n            ],\n            sensor_std_dev=linearization_parameters[\"sensor_std_dev\"],\n            diagonal_bias=linearization_parameters[\"diagonal_bias\"],\n        )\n\n        linear_position_df[\"time\"] = time\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n\n        key[\"linearized_position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=linear_position_df,\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=position_nwb[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n        from spyglass.linearization.merge import LinearizedPositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n\n        LinearizedPositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    def fetch1_dataframe(self):\n        return self.fetch_nwb()[0][\"linearized_position\"].set_index(\"time\")\n</code></pre>"}, {"location": "api/lock/file_lock/", "title": "file_lock.py", "text": ""}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.NwbfileLock", "title": "<code>NwbfileLock</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>@schema\nclass NwbfileLock(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Nwbfile\n    \"\"\"\n\n    def populate_from_lock_file(self):\n        \"\"\"\n        Reads from the NWB_LOCK_FILE (defined by an environment variable),\n        adds the entries to this schema, and then removes the file\n        \"\"\"\n        if os.path.exists(os.getenv(\"NWB_LOCK_FILE\")):\n            lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"r\")\n            for line in lock_file:\n                logger.info(line)\n                key = {\"nwb_file_name\": line.strip()}\n                self.insert1(key, skip_duplicates=\"True\")\n            lock_file.close()\n            os.remove(os.getenv(\"NWB_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.NwbfileLock.populate_from_lock_file", "title": "<code>populate_from_lock_file()</code>", "text": "<p>Reads from the NWB_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and then removes the file</p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>def populate_from_lock_file(self):\n    \"\"\"\n    Reads from the NWB_LOCK_FILE (defined by an environment variable),\n    adds the entries to this schema, and then removes the file\n    \"\"\"\n    if os.path.exists(os.getenv(\"NWB_LOCK_FILE\")):\n        lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"r\")\n        for line in lock_file:\n            logger.info(line)\n            key = {\"nwb_file_name\": line.strip()}\n            self.insert1(key, skip_duplicates=\"True\")\n        lock_file.close()\n        os.remove(os.getenv(\"NWB_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.AnalysisNwbfileLock", "title": "<code>AnalysisNwbfileLock</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>@schema\nclass AnalysisNwbfileLock(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; AnalysisNwbfile\n    \"\"\"\n\n    def populate_from_lock_file(self):\n        \"\"\"Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and\n        then removes the file\n        \"\"\"\n        if os.path.exists(os.getenv(\"ANALYSIS_LOCK_FILE\")):\n            lock_file = open(os.getenv(\"ANALYSIS_LOCK_FILE\"), \"r\")\n            for line in lock_file:\n                key = {\"analysis_file_name\": line.strip()}\n                self.insert1(key, skip_duplicates=\"True\")\n            os.remove(os.getenv(\"ANALYSIS_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.AnalysisNwbfileLock.populate_from_lock_file", "title": "<code>populate_from_lock_file()</code>", "text": "<p>Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and then removes the file</p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>def populate_from_lock_file(self):\n    \"\"\"Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and\n    then removes the file\n    \"\"\"\n    if os.path.exists(os.getenv(\"ANALYSIS_LOCK_FILE\")):\n        lock_file = open(os.getenv(\"ANALYSIS_LOCK_FILE\"), \"r\")\n        for line in lock_file:\n            key = {\"analysis_file_name\": line.strip()}\n            self.insert1(key, skip_duplicates=\"True\")\n        os.remove(os.getenv(\"ANALYSIS_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/mua/v1/mua/", "title": "mua.py", "text": ""}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsParameters", "title": "<code>MuaEventsParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Params to extract times of high mulitunit activity during immobility.</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@schema\nclass MuaEventsParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Params to extract times of high mulitunit activity during immobility.\"\"\"\n\n    definition = \"\"\"\n    mua_param_name : varchar(80) # a name for this set of parameters\n    ----\n    mua_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n    contents = [\n        {\n            \"mua_param_name\": \"default\",\n            \"mua_param_dict\": {\n                \"minimum_duration\": 0.015,  # seconds\n                \"zscore_threshold\": 2.0,\n                \"close_event_threshold\": 0.0,  # seconds\n                \"speed_threshold\": 4.0,  # cm/s\n            },\n        },\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        \"\"\"Insert the default parameter set\"\"\"\n        cls.insert(cls.contents, skip_duplicates=True)\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsParameters.insert_default", "title": "<code>insert_default()</code>  <code>classmethod</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@classmethod\ndef insert_default(cls):\n    \"\"\"Insert the default parameter set\"\"\"\n    cls.insert(cls.contents, skip_duplicates=True)\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsV1", "title": "<code>MuaEventsV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@schema\nclass MuaEventsV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; MuaEventsParameters\n    -&gt; SortedSpikesGroup\n    -&gt; PositionOutput.proj(pos_merge_id='merge_id')\n    -&gt; IntervalList.proj(detection_interval='interval_list_name')\n    ---\n    -&gt; AnalysisNwbfile\n    mua_times_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        speed = self.get_speed(key)\n        time = speed.index.to_numpy()\n        speed = speed.to_numpy()\n\n        spike_indicator = SortedSpikesGroup.get_spike_indicator(key, time)\n        spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        mua_params = (MuaEventsParameters &amp; key).fetch1(\"mua_param_dict\")\n\n        valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"detection_interval\"],\n            }\n        ).fetch1(\"valid_times\")\n        mask = np.zeros_like(time, dtype=bool)\n        for start, end in valid_times:\n            mask = mask | ((time &gt;= start) &amp; (time &lt;= end))\n\n        time = time[mask]\n        speed = speed[mask]\n        spike_indicator = spike_indicator[mask]\n\n        mua_times = multiunit_HSE_detector(\n            time, spike_indicator, speed, sampling_frequency, **mua_params\n        )\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        nwb_file_name = (SortedSpikesGroup &amp; key).fetch1(\"nwb_file_name\")\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(nwb_file_name)\n        key[\"mua_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=mua_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=nwb_file_name,\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"mua_times\"] for data in self.fetch_nwb()]\n\n    @classmethod\n    def get_firing_rate(cls, key, time):\n        return SortedSpikesGroup.get_firing_rate(key, time, multiunit=True)\n\n    @staticmethod\n    def get_speed(key):\n        position_info = (\n            PositionOutput &amp; {\"merge_id\": key[\"pos_merge_id\"]}\n        ).fetch1_dataframe()\n        speed_name = (\n            \"speed\" if \"speed\" in position_info.columns else \"head_speed\"\n        )\n        return position_info[speed_name]\n\n    def create_figurl(\n        self,\n        zscore_mua=True,\n        mua_times_color=\"red\",\n        speed_color=\"black\",\n        mua_color=\"black\",\n        view_height=800,\n    ):\n        key = self.fetch1(\"KEY\")\n        speed = self.get_speed(key)\n        time = speed.index.to_numpy()\n        multiunit_firing_rate = self.get_firing_rate(key, time)\n        if zscore_mua:\n            multiunit_firing_rate = zscore(multiunit_firing_rate)\n\n        mua_times = self.fetch1_dataframe()\n\n        multiunit_firing_rate_view = vv.TimeseriesGraph()\n        multiunit_firing_rate_view.add_interval_series(\n            name=\"MUA Events\",\n            t_start=mua_times.start_time.to_numpy(),\n            t_end=mua_times.end_time.to_numpy(),\n            color=mua_times_color,\n        )\n        name = \"Z-Scored Multiunit Rate\" if zscore_mua else \"Multiunit Rate\"\n        multiunit_firing_rate_view.add_line_series(\n            name=name,\n            t=np.asarray(time),\n            y=np.asarray(multiunit_firing_rate, dtype=np.float32),\n            color=mua_color,\n            width=1,\n        )\n        if zscore_mua:\n            mua_params = (MuaEventsParameters &amp; key).fetch1(\"mua_param_dict\")\n            zscore_threshold = mua_params.get(\"zscore_threshold\")\n            multiunit_firing_rate_view.add_line_series(\n                name=\"Z-Score Threshold\",\n                t=np.asarray(time).squeeze(),\n                y=np.ones_like(\n                    multiunit_firing_rate, dtype=np.float32\n                ).squeeze()\n                * zscore_threshold,\n                color=mua_times_color,\n                width=1,\n            )\n        speed_view = vv.TimeseriesGraph().add_line_series(\n            name=\"Speed [cm/s]\",\n            t=np.asarray(time),\n            y=np.asarray(speed, dtype=np.float32),\n            color=speed_color,\n            width=1,\n        )\n        speed_view.add_interval_series(\n            name=\"MUA Events\",\n            t_start=mua_times.start_time.to_numpy(),\n            t_end=mua_times.end_time.to_numpy(),\n            color=mua_times_color,\n        )\n        vertical_panel_content = [\n            vv.LayoutItem(\n                multiunit_firing_rate_view, stretch=2, title=\"Multiunit\"\n            ),\n            vv.LayoutItem(speed_view, stretch=2, title=\"Speed\"),\n        ]\n\n        view = vv.Box(\n            direction=\"horizontal\",\n            show_titles=True,\n            height=view_height,\n            items=[\n                vv.LayoutItem(\n                    vv.Box(\n                        direction=\"vertical\",\n                        show_titles=True,\n                        items=vertical_panel_content,\n                    )\n                ),\n            ],\n        )\n\n        return view.url(label=\"Multiunit Detection\")\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsV1.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/position/position_merge/", "title": "position_merge.py", "text": ""}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput", "title": "<code>PositionOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> <p>Table to identify source of Position Information from upstream options (e.g. DLC, Trodes, etc...) To add another upstream option, a new Part table should be added in the same syntax as DLCPos and TrodesPos.</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>@schema\nclass PositionOutput(_Merge, SpyglassMixin):\n    \"\"\"\n    Table to identify source of Position Information from upstream options\n    (e.g. DLC, Trodes, etc...) To add another upstream option, a new Part table\n    should be added in the same syntax as DLCPos and TrodesPos.\n    \"\"\"\n\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class DLCPosV1(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream DLC Pose Estimation information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; DLCPosV1\n        \"\"\"\n\n    class TrodesPosV1(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream Trodes Position Tracking information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; TrodesPosV1\n        \"\"\"\n\n    class CommonPos(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream Trodes Position Tracking information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; CommonPos\n        \"\"\"\n\n    def fetch1_dataframe(self):\n        # proj replaces operator restriction to enable\n        # (TableName &amp; restriction).fetch1_dataframe()\n        key = self.merge_restrict(self.proj()).proj()\n        query = (\n            source_class_dict[\n                to_camel_case(self.merge_get_parent(self.proj()).table_name)\n            ]\n            &amp; key\n        )\n        return query.fetch1_dataframe()\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.DLCPosV1", "title": "<code>DLCPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream DLC Pose Estimation information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class DLCPosV1(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream DLC Pose Estimation information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; DLCPosV1\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.TrodesPosV1", "title": "<code>TrodesPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream Trodes Position Tracking information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class TrodesPosV1(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream Trodes Position Tracking information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; TrodesPosV1\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.CommonPos", "title": "<code>CommonPos</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream Trodes Position Tracking information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class CommonPos(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream Trodes Position Tracking information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; CommonPos\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionVideo", "title": "<code>PositionVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>@schema\nclass PositionVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionVideoSelection\n    ---\n    \"\"\"\n\n    def make(self, key):\n        raise NotImplementedError(\"work in progress -DPG\")\n\n        plot = key.get(\"plot\")\n        if plot not in [\"DLC\", \"Trodes\", \"Common\", \"All\"]:\n            raise ValueError(f\"Plot {key['plot']} not supported\")\n            # CBroz: I was told only tests should `assert`, code should `raise`\n\n        M_TO_CM = 100\n        output_dir = (PositionVideoSelection &amp; key).fetch1(\"output_dir\")\n\n        logger.info(\"Loading position data...\")\n        # raw_position_df = (\n        #     RawPosition()\n        #     &amp; {\n        #         \"nwb_file_name\": key[\"nwb_file_name\"],\n        #         \"interval_list_name\": key[\"interval_list_name\"],\n        #     }\n        # ).fetch1_dataframe()\n\n        query = {\n            \"nwb_file_name\": key[\"nwb_file_name\"],\n            \"interval_list_name\": key[\"interval_list_name\"],\n        }\n        merge_entries = {\n            \"DLC\": PositionOutput.DLCPosV1 &amp; query,\n            \"Trodes\": PositionOutput.TrodesPosV1 &amp; query,\n            \"Common\": PositionOutput.CommonPos &amp; query,\n        }\n\n        position_mean_dict = {}\n        if plot == \"All\":\n            # Check which entries exist in PositionOutput\n            merge_dict = {}\n            for source, entries in merge_entries.items():\n                if entries:\n                    merge_dict[source] = entries.fetch1_dataframe().drop(\n                        columns=[\"velocity_x\", \"velocity_y\", \"speed\"]\n                    )\n\n            pos_df = ft.reduce(\n                lambda left, right,: pd.merge(\n                    left[1],\n                    right[1],\n                    left_index=True,\n                    right_index=True,\n                    suffixes=[f\"_{left[0]}\", f\"_{right[0]}\"],\n                ),\n                merge_dict.items(),\n            )\n            position_mean_dict = {\n                source: {\n                    \"position\": np.asarray(\n                        pos_df[[f\"position_x_{source}\", f\"position_y_{source}\"]]\n                    ),\n                    \"orientation\": np.asarray(\n                        pos_df[[f\"orientation_{source}\"]]\n                    ),\n                }\n                for source in merge_dict.keys()\n            }\n        else:\n            if plot == \"DLC\":\n                # CBroz - why is this extra step needed for DLC?\n                pos_df_key = merge_entries[plot].fetch1(as_dict=True)\n                pos_df = (PositionOutput &amp; pos_df_key).fetch1_dataframe()\n            elif plot in [\"Trodes\", \"Common\"]:\n                pos_df = merge_entries[plot].fetch1_dataframe()\n\n            position_mean_dict[plot][\"position\"] = np.asarray(\n                pos_df[[\"position_x\", \"position_y\"]]\n            )\n            position_mean_dict[plot][\"orientation\"] = np.asarray(\n                pos_df[[\"orientation\"]]\n            )\n\n        logger.info(\"Loading video data...\")\n        epoch = int(\"\".join(filter(str.isdigit, key[\"interval_list_name\"]))) + 1\n\n        (\n            video_path,\n            video_filename,\n            meters_per_pixel,\n            video_time,\n        ) = get_video_path(\n            {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        )\n        video_dir = os.path.dirname(video_path) + \"/\"\n        video_frame_col_name = [\n            col for col in pos_df.columns if \"video_frame_ind\" in col\n        ][0]\n        video_frame_inds = pos_df[video_frame_col_name].astype(int).to_numpy()\n        if plot in [\"DLC\", \"All\"]:\n            video_path = (\n                DLCPoseEstimationSelection\n                &amp; (PositionOutput.DLCPosV1 &amp; key).fetch1(\"KEY\")\n            ).fetch1(\"video_path\")\n        else:\n            video_path = check_videofile(\n                video_dir, key[\"output_dir\"], video_filename\n            )[0]\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        output_video_filename = Path(\n            f\"{Path(output_dir).as_posix()}/{nwb_base_filename}{epoch:02d}_\"\n            f\"{key['plot']}_pos_overlay.mp4\"\n        ).as_posix()\n\n        # centroids = {'red': np.asarray(raw_position_df[['xloc', 'yloc']]),\n        #              'green':  np.asarray(raw_position_df[['xloc2', 'yloc2']])}\n\n        logger.info(\"Making video...\")\n\n        make_video(\n            video_path,\n            video_frame_inds,\n            position_mean_dict,\n            video_time,\n            np.asarray(pos_df.index),\n            processor=\"opencv\",\n            output_video_filename=output_video_filename,\n            cm_to_pixels=meters_per_pixel * M_TO_CM,\n            disable_progressbar=False,\n        )\n        self.insert1(key)\n</code></pre>"}, {"location": "api/position/v1/dlc_decorators/", "title": "dlc_decorators.py", "text": ""}, {"location": "api/position/v1/dlc_reader/", "title": "dlc_reader.py", "text": ""}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.read_yaml", "title": "<code>read_yaml(fullpath, filename='*')</code>", "text": "<p>Return contents of yml in fullpath. If available, defer to DJ-saved version</p> <p>Parameters:</p> Name Type Description Default <code>fullpath</code> required <code>filename</code> <code>'*'</code> <p>Returns filepath and contents as dict</p> Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def read_yaml(fullpath, filename=\"*\"):\n    \"\"\"Return contents of yml in fullpath. If available, defer to DJ-saved version\n\n    Parameters\n    ----------\n    fullpath: String or pathlib path. Directory with yaml files\n    filename: String. Filename, no extension. Permits wildcards.\n\n    Returns filepath and contents as dict\n    \"\"\"\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    # Take the DJ-saved if there. If not, return list of available\n    yml_paths = list(Path(fullpath).glob(\"dj_dlc_config.yaml\")) or sorted(\n        list(Path(fullpath).glob(f\"{filename}.y*ml\"))\n    )\n\n    assert (  # If more than 1 and not DJ-saved,\n        len(yml_paths) == 1\n    ), f\"Found more yaml files than expected: {len(yml_paths)}\\n{fullpath}\"\n\n    return yml_paths[0], read_config(yml_paths[0])\n</code></pre>"}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.save_yaml", "title": "<code>save_yaml(output_dir, config_dict, filename='dj_dlc_config', mkdir=True)</code>", "text": "<p>Save config_dict to output_path as filename.yaml. By default, preserves original.</p> <p>Parameters:</p> Name Type Description Default <code>output_dir</code> required <code>config_dict</code> required <code>filename</code> <pre><code>  Set to 'config' to overwrite original file.\n  If extension is included, removed and replaced with \"yaml\".\n</code></pre> <code>'dj_dlc_config'</code> <code>mkdir</code> <code>True</code> <p>Returns:</p> Type Description <code>str</code> <p>path of saved file as string - due to DLC func preference for strings</p> Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def save_yaml(output_dir, config_dict, filename=\"dj_dlc_config\", mkdir=True):\n    \"\"\"Save config_dict to output_path as filename.yaml. By default, preserves original.\n\n    Parameters\n    ----------\n    output_dir: where to save yaml file\n    config_dict: dict of config params or element-deeplabcut model.Model dict\n    filename: Optional, default 'dj_dlc_config' or preserve original 'config'\n              Set to 'config' to overwrite original file.\n              If extension is included, removed and replaced with \"yaml\".\n    mkdir (bool): Optional, True. Make new directory if output_dir not exist\n\n    Returns\n    -------\n    str\n        path of saved file as string - due to DLC func preference for strings\n    \"\"\"\n    from deeplabcut.utils.auxiliaryfunctions import write_config\n\n    if \"config_template\" in config_dict:  # if passed full model.Model dict\n        config_dict = config_dict[\"config_template\"]\n    if mkdir:\n        Path(output_dir).mkdir(exist_ok=True)\n    if \".\" in filename:  # if user provided extension, remove\n        filename = filename.split(\".\")[0]\n\n    output_filepath = Path(output_dir) / f\"{filename}.yaml\"\n    write_config(output_filepath, config_dict)\n    return str(output_filepath)\n</code></pre>"}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.do_pose_estimation", "title": "<code>do_pose_estimation(video_filepaths, dlc_model, project_path, output_dir, videotype='', gputouse=None, save_as_csv=False, batchsize=None, cropping=None, TFGPUinference=True, dynamic=(False, 0.5, 10), robust_nframes=False, allow_growth=False, use_shelve=False)</code>", "text": "<p>Launch DLC's analyze_videos within element-deeplabcut</p> <p>Other optional parameters may be set other than those described below. See deeplabcut.analyze_videos parameters for descriptions/defaults.</p> <p>Parameters:</p> Name Type Description Default <code>video_filepaths</code> required <code>dlc_model</code> required <code>project_path</code> required <code>output_dir</code> required Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def do_pose_estimation(\n    video_filepaths,\n    dlc_model,\n    project_path,\n    output_dir,\n    videotype=\"\",\n    gputouse=None,\n    save_as_csv=False,\n    batchsize=None,\n    cropping=None,\n    TFGPUinference=True,\n    dynamic=(False, 0.5, 10),\n    robust_nframes=False,\n    allow_growth=False,\n    use_shelve=False,\n):\n    \"\"\"Launch DLC's analyze_videos within element-deeplabcut\n\n    Other optional parameters may be set other than those described below. See\n    deeplabcut.analyze_videos parameters for descriptions/defaults.\n\n    Parameters\n    ----------\n    video_filepaths: list of videos to analyze\n    dlc_model: element-deeplabcut dlc.Model dict\n    project_path: path to project config.yml\n    output_dir: where to save output\n    \"\"\"\n    from deeplabcut.pose_estimation_tensorflow import analyze_videos\n\n    # ---- Build and save DLC configuration (yaml) file ----\n    dlc_config = dlc_model[\"config_template\"]\n    dlc_project_path = Path(project_path)\n    dlc_config[\"project_path\"] = dlc_project_path.as_posix()\n\n    # ---- Write config files ----\n    # To output dir: Important for loading/parsing output in datajoint\n    _ = save_yaml(output_dir, dlc_config)\n    # To project dir: Required by DLC to run the analyze_videos\n    if dlc_project_path != output_dir:\n        config_filepath = save_yaml(dlc_project_path, dlc_config)\n    # ---- Trigger DLC prediction job ----\n    analyze_videos(\n        config=config_filepath,\n        videos=video_filepaths,\n        shuffle=dlc_model[\"shuffle\"],\n        trainingsetindex=dlc_model[\"trainingsetindex\"],\n        destfolder=output_dir,\n        modelprefix=dlc_model[\"model_prefix\"],\n        videotype=videotype,\n        gputouse=gputouse,\n        save_as_csv=save_as_csv,\n        batchsize=batchsize,\n        cropping=cropping,\n        TFGPUinference=TFGPUinference,\n        dynamic=dynamic,\n        robust_nframes=robust_nframes,\n        allow_growth=allow_growth,\n        use_shelve=use_shelve,\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/", "title": "dlc_utils.py", "text": ""}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_option", "title": "<code>validate_option(option=None, options=None, name='option', types=None, val_range=None, permit_none=False)</code>", "text": "<p>Validate that option is in a list options or a list of types.</p> <p>Parameters:</p> Name Type Description Default <code>option</code> <code>str</code> <p>If none, runs no checks.</p> <code>None</code> <code>options</code> <code>lis</code> <p>If provided, option must be in options.</p> <code>None</code> <code>name</code> <code>st</code> <p>If provided, name of option to use in error message.</p> <code>'option'</code> <code>types</code> <code>tuple</code> <p>If provided, option must be an instance of one of the types in types.</p> <code>None</code> <code>val_range</code> <code>tuple</code> <p>If provided, option must be in range (min, max)</p> <code>None</code> <code>permit_none</code> <code>bool</code> <p>If True, permit option to be None. Default False.</p> <code>False</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>If option is not in options.</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_option(\n    option=None,\n    options: list = None,\n    name=\"option\",\n    types: tuple = None,\n    val_range: tuple = None,\n    permit_none=False,\n):\n    \"\"\"Validate that option is in a list options or a list of types.\n\n    Parameters\n    ----------\n    option : str, optional\n        If none, runs no checks.\n    options : lis, optional\n        If provided, option must be in options.\n    name : st, optional\n        If provided, name of option to use in error message.\n    types : tuple, optional\n        If provided, option must be an instance of one of the types in types.\n    val_range : tuple, optional\n        If provided, option must be in range (min, max)\n    permit_none : bool, optional\n        If True, permit option to be None. Default False.\n\n    Raises\n    ------\n    ValueError\n        If option is not in options.\n    \"\"\"\n    if option is None and not permit_none:\n        raise ValueError(f\"{name} cannot be None\")\n\n    if options and option not in options:\n        raise KeyError(\n            f\"Unknown {name}: {option} \" f\"Available options: {options}\"\n        )\n\n    if types and not isinstance(option, tuple(types)):\n        raise TypeError(f\"{name} is {type(option)}. Available types {types}\")\n\n    if val_range and not (val_range[0] &lt;= option &lt;= val_range[1]):\n        raise ValueError(f\"{name} must be in range {val_range}\")\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_list", "title": "<code>validate_list(required_items, option_list=None, name='List', condition='', permit_none=False)</code>", "text": "<p>Validate that option_list contains all items in required_items.</p> <p>Parameters:</p> Name Type Description Default <code>required_items</code> <code>list</code> required <code>option_list</code> <code>list</code> <p>If provided, option_list must contain all items in required_items.</p> <code>None</code> <code>name</code> <code>str</code> <p>If provided, name of option_list to use in error message.</p> <code>'List'</code> <code>condition</code> <code>str</code> <p>If provided, condition in error message as 'when using X'.</p> <code>''</code> <code>permit_none</code> <code>bool</code> <p>If True, permit option_list to be None. Default False.</p> <code>False</code> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_list(\n    required_items: list,\n    option_list: list = None,\n    name=\"List\",\n    condition=\"\",\n    permit_none=False,\n):\n    \"\"\"Validate that option_list contains all items in required_items.\n\n    Parameters\n    ---------\n    required_items : list\n    option_list : list, optional\n        If provided, option_list must contain all items in required_items.\n    name : str, optional\n        If provided, name of option_list to use in error message.\n    condition : str, optional\n        If provided, condition in error message as 'when using X'.\n    permit_none : bool, optional\n        If True, permit option_list to be None. Default False.\n    \"\"\"\n    if option_list is None:\n        if permit_none:\n            return\n        else:\n            raise ValueError(f\"{name} cannot be None\")\n    if condition:\n        condition = f\" when using {condition}\"\n    if any(x not in required_items for x in option_list):\n        raise KeyError(\n            f\"{name} must contain all items in {required_items}{condition}.\"\n        )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_smooth_params", "title": "<code>validate_smooth_params(params)</code>", "text": "<p>If params['smooth'], validate method is in list and duration type</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_smooth_params(params):\n    \"\"\"If params['smooth'], validate method is in list and duration type\"\"\"\n    if not params.get(\"smooth\"):\n        return\n    smoothing_params = params.get(\"smoothing_params\")\n    validate_option(smoother=smoothing_params, name=\"smoothing_params\")\n    validate_option(\n        option=smoothing_params.get(\"smooth_method\"),\n        name=\"smooth_method\",\n        options=_key_to_smooth_func_dict,\n    )\n    validate_option(\n        option=smoothing_params.get(\"smoothing_duration\"),\n        name=\"smoothing_duration\",\n        types=(int, float),\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.OutputLogger", "title": "<code>OutputLogger</code>", "text": "<p>A class to wrap a logging.Logger object in order to provide context manager capabilities.</p> <p>This class uses contextlib.redirect_stdout to temporarily redirect sys.stdout and thus print statements to the log file instead of, or as well as the console.</p> <p>Attributes:</p> Name Type Description <code>logger</code> <code>Logger</code> <p>logger object</p> <code>name</code> <code>str</code> <p>name of logger</p> <code>level</code> <code>int</code> <p>level of logging that the logger is set to handle</p> <p>Methods:</p> Name Description <code>setup_logger</code> <p>initialize or get logger object with name_logfile that writes to path_logfile</p> <p>Examples:</p> <pre><code>&gt;&gt;&gt; with OutputLogger(name, path, print_console=True) as logger:\n...    print(\"this will print to logfile\")\n...    logger.logger.info(\"this will log to the logfile\")\n... print(\"this will print to the console\")\n... logger.logger.info(\"this will log to the logfile\")\n</code></pre> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>class OutputLogger:  # TODO: migrate to spyglass.utils.logger\n    \"\"\"\n    A class to wrap a logging.Logger object in order to provide context manager capabilities.\n\n    This class uses contextlib.redirect_stdout to temporarily redirect sys.stdout and thus\n    print statements to the log file instead of, or as well as the console.\n\n    Attributes\n    ----------\n    logger : logging.Logger\n        logger object\n    name : str\n        name of logger\n    level : int\n        level of logging that the logger is set to handle\n\n    Methods\n    -------\n    setup_logger(name_logfile, path_logfile, print_console=False)\n        initialize or get logger object with name_logfile\n        that writes to path_logfile\n\n    Examples\n    --------\n    &gt;&gt;&gt; with OutputLogger(name, path, print_console=True) as logger:\n    ...    print(\"this will print to logfile\")\n    ...    logger.logger.info(\"this will log to the logfile\")\n    ... print(\"this will print to the console\")\n    ... logger.logger.info(\"this will log to the logfile\")\n\n    \"\"\"\n\n    def __init__(self, name, path, level=\"INFO\", **kwargs):\n        self.logger = self.setup_logger(name, path, **kwargs)\n        self.name = self.logger.name\n        self.level = getattr(logging, level)\n\n    def setup_logger(\n        self, name_logfile, path_logfile, print_console=False\n    ) -&gt; logging.Logger:\n        \"\"\"\n        Sets up a logger for that outputs to a file, and optionally, the console\n\n        Parameters\n        ----------\n        name_logfile : str\n            name of the logfile to use\n        path_logfile : str\n            path to the file that should be used as the file handler\n        print_console : bool, default-False\n            if True, prints to console as well as log file.\n\n        Returns\n        -------\n        logger : logging.Logger\n            the logger object with specified handlers\n        \"\"\"\n\n        logger = logging.getLogger(name_logfile)\n        # check to see if handlers already exist for this logger\n        if logger.handlers:\n            for handler in logger.handlers:\n                # if it's a file handler\n                # type is used instead of isinstance,\n                # which doesn't work properly with logging.StreamHandler\n                if type(handler) == logging.FileHandler:\n                    # if paths don't match, change file handler path\n                    if not os.path.samefile(handler.baseFilename, path_logfile):\n                        handler.close()\n                        logger.removeHandler(handler)\n                        file_handler = self._get_file_handler(path_logfile)\n                        logger.addHandler(file_handler)\n                # if a stream handler exists and\n                # if print_console is False remove streamHandler\n                if type(handler) == logging.StreamHandler:\n                    if not print_console:\n                        handler.close()\n                        logger.removeHandler(handler)\n            if print_console and not any(\n                type(handler) == logging.StreamHandler\n                for handler in logger.handlers\n            ):\n                logger.addHandler(self._get_stream_handler())\n\n        else:\n            file_handler = self._get_file_handler(path_logfile)\n            logger.addHandler(file_handler)\n            if print_console:\n                logger.addHandler(self._get_stream_handler())\n        logger.setLevel(logging.INFO)\n        return logger\n\n    def _get_file_handler(self, path):\n        output_dir = pathlib.Path(os.path.dirname(path))\n        if not os.path.exists(output_dir):\n            output_dir.mkdir(parents=True, exist_ok=True)\n        file_handler = logging.FileHandler(path, mode=\"a\")\n        file_handler.setFormatter(self._get_formatter())\n        return file_handler\n\n    def _get_stream_handler(self):\n        stream_handler = logging.StreamHandler()\n        stream_handler.setFormatter(self._get_formatter())\n        return stream_handler\n\n    def _get_formatter(self):\n        return logging.Formatter(\n            \"[%(asctime)s] in %(pathname)s, line %(lineno)d: %(message)s\",\n            datefmt=\"%d-%b-%y %H:%M:%S\",\n        )\n\n    def write(self, msg):\n        if msg and not msg.isspace():\n            self.logger.log(self.level, msg)\n\n    def flush(self):\n        pass\n\n    def __enter__(self):\n        self._redirector = redirect_stdout(self)\n        self._redirector.__enter__()\n        return self\n\n    def __exit__(self, exc_type, exc_value, traceback):\n        # let contextlib do any exception handling here\n        self._redirector.__exit__(exc_type, exc_value, traceback)\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.OutputLogger.setup_logger", "title": "<code>setup_logger(name_logfile, path_logfile, print_console=False)</code>", "text": "<p>Sets up a logger for that outputs to a file, and optionally, the console</p> <p>Parameters:</p> Name Type Description Default <code>name_logfile</code> <code>str</code> <p>name of the logfile to use</p> required <code>path_logfile</code> <code>str</code> <p>path to the file that should be used as the file handler</p> required <code>print_console</code> <code>(bool, default - False)</code> <p>if True, prints to console as well as log file.</p> <code>False</code> <p>Returns:</p> Name Type Description <code>logger</code> <code>Logger</code> <p>the logger object with specified handlers</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def setup_logger(\n    self, name_logfile, path_logfile, print_console=False\n) -&gt; logging.Logger:\n    \"\"\"\n    Sets up a logger for that outputs to a file, and optionally, the console\n\n    Parameters\n    ----------\n    name_logfile : str\n        name of the logfile to use\n    path_logfile : str\n        path to the file that should be used as the file handler\n    print_console : bool, default-False\n        if True, prints to console as well as log file.\n\n    Returns\n    -------\n    logger : logging.Logger\n        the logger object with specified handlers\n    \"\"\"\n\n    logger = logging.getLogger(name_logfile)\n    # check to see if handlers already exist for this logger\n    if logger.handlers:\n        for handler in logger.handlers:\n            # if it's a file handler\n            # type is used instead of isinstance,\n            # which doesn't work properly with logging.StreamHandler\n            if type(handler) == logging.FileHandler:\n                # if paths don't match, change file handler path\n                if not os.path.samefile(handler.baseFilename, path_logfile):\n                    handler.close()\n                    logger.removeHandler(handler)\n                    file_handler = self._get_file_handler(path_logfile)\n                    logger.addHandler(file_handler)\n            # if a stream handler exists and\n            # if print_console is False remove streamHandler\n            if type(handler) == logging.StreamHandler:\n                if not print_console:\n                    handler.close()\n                    logger.removeHandler(handler)\n        if print_console and not any(\n            type(handler) == logging.StreamHandler\n            for handler in logger.handlers\n        ):\n            logger.addHandler(self._get_stream_handler())\n\n    else:\n        file_handler = self._get_file_handler(path_logfile)\n        logger.addHandler(file_handler)\n        if print_console:\n            logger.addHandler(self._get_stream_handler())\n    logger.setLevel(logging.INFO)\n    return logger\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_dlc_processed_data_dir", "title": "<code>get_dlc_processed_data_dir()</code>", "text": "<p>Returns session_dir relative to custom 'dlc_output_dir' root</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_dlc_processed_data_dir() -&gt; str:\n    \"\"\"Returns session_dir relative to custom 'dlc_output_dir' root\"\"\"\n    if \"custom\" in dj.config:\n        if \"dlc_output_dir\" in dj.config[\"custom\"]:\n            dlc_output_dir = dj.config.get(\"custom\", {}).get(\"dlc_output_dir\")\n    if dlc_output_dir:\n        return pathlib.Path(dlc_output_dir)\n    else:\n        return pathlib.Path(\"/nimbus/deeplabcut/output/\")\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.find_full_path", "title": "<code>find_full_path(root_directories, relative_path)</code>", "text": "<p>from Datajoint Elements - unused Given a relative path, search and return the full-path  from provided potential root directories (in the given order)     :param root_directories: potential root directories     :param relative_path: the relative path to find the valid root directory     :return: full-path (pathlib.Path object)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def find_full_path(root_directories, relative_path):\n    \"\"\"\n    from Datajoint Elements - unused\n    Given a relative path, search and return the full-path\n     from provided potential root directories (in the given order)\n        :param root_directories: potential root directories\n        :param relative_path: the relative path to find the valid root directory\n        :return: full-path (pathlib.Path object)\n    \"\"\"\n    relative_path = _to_Path(relative_path)\n\n    if relative_path.exists():\n        return relative_path\n\n    # Turn to list if only a single root directory is provided\n    if isinstance(root_directories, (str, pathlib.Path)):\n        root_directories = [_to_Path(root_directories)]\n\n    for root_dir in root_directories:\n        if (_to_Path(root_dir) / relative_path).exists():\n            return _to_Path(root_dir) / relative_path\n\n    raise FileNotFoundError(\n        f\"No valid full-path found (from {root_directories})\"\n        f\" for {relative_path}\"\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.find_root_directory", "title": "<code>find_root_directory(root_directories, full_path)</code>", "text": "<p>From datajoint elements - unused Given multiple potential root directories and a full-path, search and return one directory that is the parent of the given path     :param root_directories: potential root directories     :param full_path: the full path to search the root directory     :return: root_directory (pathlib.Path object)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def find_root_directory(root_directories, full_path):\n    \"\"\"\n    From datajoint elements - unused\n    Given multiple potential root directories and a full-path,\n    search and return one directory that is the parent of the given path\n        :param root_directories: potential root directories\n        :param full_path: the full path to search the root directory\n        :return: root_directory (pathlib.Path object)\n    \"\"\"\n    full_path = _to_Path(full_path)\n\n    if not full_path.exists():\n        raise FileNotFoundError(f\"{full_path} does not exist!\")\n\n    # Turn to list if only a single root directory is provided\n    if isinstance(root_directories, (str, pathlib.Path)):\n        root_directories = [_to_Path(root_directories)]\n\n    try:\n        return next(\n            _to_Path(root_dir)\n            for root_dir in root_directories\n            if _to_Path(root_dir) in set(full_path.parents)\n        )\n\n    except StopIteration as exc:\n        raise FileNotFoundError(\n            f\"No valid root directory found (from {root_directories})\"\n            f\" for {full_path}\"\n        ) from exc\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.infer_output_dir", "title": "<code>infer_output_dir(key, makedir=True)</code>", "text": "<p>Return the expected pose_estimation_output_dir.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> required Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def infer_output_dir(key, makedir=True):\n    \"\"\"Return the expected pose_estimation_output_dir.\n\n    Parameters\n    ----------\n    key: DataJoint key specifying a pairing of VideoFile and Model.\n    \"\"\"\n    # TODO: add check to make sure interval_list_name refers to a single epoch\n    # Or make key include epoch in and of itself instead of interval_list_name\n    nwb_file_name = key[\"nwb_file_name\"].split(\"_.\")[0]\n    output_dir = pathlib.Path(dlc_output_dir) / pathlib.Path(\n        f\"{nwb_file_name}/{nwb_file_name}_{key['epoch']:02}\"\n        f\"_model_\" + key[\"dlc_model_name\"].replace(\" \", \"-\")\n    )\n    if makedir is True:\n        if not os.path.exists(output_dir):\n            output_dir.mkdir(parents=True, exist_ok=True)\n    return output_dir\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_video_path", "title": "<code>get_video_path(key)</code>", "text": "<p>Given nwb_file_name and interval_list_name returns specified video file filename and path</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Dictionary containing nwb_file_name and interval_list_name as keys</p> required <p>Returns:</p> Name Type Description <code>video_filepath</code> <code>str</code> <p>path to the video file, including video filename</p> <code>video_filename</code> <code>str</code> <p>filename of the video</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_video_path(key):\n    \"\"\"\n    Given nwb_file_name and interval_list_name returns specified\n    video file filename and path\n\n    Parameters\n    ----------\n    key : dict\n        Dictionary containing nwb_file_name and interval_list_name as keys\n\n    Returns\n    -------\n    video_filepath : str\n        path to the video file, including video filename\n    video_filename : str\n        filename of the video\n    \"\"\"\n    import pynwb\n\n    vf_key = {k: val for k, val in key.items() if k in VideoFile.heading.names}\n    if not VideoFile &amp; vf_key:\n        VideoFile()._no_transaction_make(vf_key, verbose=False)\n    video_query = VideoFile &amp; vf_key\n\n    if len(video_query) != 1:\n        print(f\"Found {len(video_query)} videos for {vf_key}\")\n        return None, None, None, None\n\n    video_info = video_query.fetch1()\n    nwb_path = f\"{raw_dir}/{video_info['nwb_file_name']}\"\n\n    with pynwb.NWBHDF5IO(path=nwb_path, mode=\"r\") as in_out:\n        nwb_file = in_out.read()\n        nwb_video = nwb_file.objects[video_info[\"video_file_object_id\"]]\n        video_filepath = VideoFile.get_abs_path(vf_key)\n        video_dir = os.path.dirname(video_filepath) + \"/\"\n        video_filename = video_filepath.split(video_dir)[-1]\n        meters_per_pixel = nwb_video.device.meters_per_pixel\n        timestamps = np.asarray(nwb_video.timestamps)\n\n    return video_dir, video_filename, meters_per_pixel, timestamps\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.check_videofile", "title": "<code>check_videofile(video_path, output_path=dlc_video_dir, video_filename=None, video_filetype='h264')</code>", "text": "<p>Checks the file extension of a video file to make sure it is .mp4 for DeepLabCut processes. Converts to MP4 if not already.</p> <p>Parameters:</p> Name Type Description Default <code>video_path</code> <code>str or PosixPath object</code> <p>path to directory of the existing video file without filename</p> required <code>output_path</code> <code>str or PosixPath object</code> <p>path to directory where converted video will be saved</p> <code>dlc_video_dir</code> <code>video_filename</code> <code>(str, Optional)</code> <p>filename of the video to convert, if not provided, video_filetype must be and all video files of video_filetype in the directory will be converted</p> <code>None</code> <code>video_filetype</code> <code>str or List, Default 'h264', Optional</code> <p>If video_filename is not provided, all videos of this filetype will be converted to .mp4</p> <code>'h264'</code> <p>Returns:</p> Name Type Description <code>output_files</code> <code>List of PosixPath objects</code> <p>paths to converted video file(s)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def check_videofile(\n    video_path: Union[str, pathlib.PosixPath],\n    output_path: Union[str, pathlib.PosixPath] = dlc_video_dir,\n    video_filename: str = None,\n    video_filetype: str = \"h264\",\n):\n    \"\"\"\n    Checks the file extension of a video file to make sure it is .mp4 for\n    DeepLabCut processes. Converts to MP4 if not already.\n\n    Parameters\n    ----------\n    video_path : str or PosixPath object\n        path to directory of the existing video file without filename\n    output_path : str or PosixPath object\n        path to directory where converted video will be saved\n    video_filename : str, Optional\n        filename of the video to convert, if not provided, video_filetype must be\n        and all video files of video_filetype in the directory will be converted\n    video_filetype : str or List, Default 'h264', Optional\n        If video_filename is not provided,\n        all videos of this filetype will be converted to .mp4\n\n    Returns\n    -------\n    output_files : List of PosixPath objects\n        paths to converted video file(s)\n    \"\"\"\n\n    if not video_filename:\n        video_files = pathlib.Path(video_path).glob(f\"*.{video_filetype}\")\n    else:\n        video_files = [pathlib.Path(f\"{video_path}/{video_filename}\")]\n    output_files = []\n    for video_filepath in video_files:\n        if video_filepath.exists():\n            if video_filepath.suffix == \".mp4\":\n                output_files.append(video_filepath)\n                continue\n        video_file = (\n            video_filepath.as_posix()\n            .rsplit(video_filepath.parent.as_posix(), maxsplit=1)[-1]\n            .split(\"/\")[-1]\n        )\n        output_files.append(\n            _convert_mp4(video_file, video_path, output_path, videotype=\"mp4\")\n        )\n    return output_files\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_gpu_memory", "title": "<code>get_gpu_memory()</code>", "text": "<p>Queries the gpu cluster and returns the memory use for each core. This is used to evaluate which GPU cores are available to run jobs on (i.e. pose estimation, DLC model training)</p> <p>Returns:</p> Name Type Description <code>memory_use_values</code> <code>dict</code> <p>dictionary with core number as key and memory in use as value.</p> <p>Raises:</p> Type Description <code>RuntimeError</code> <p>if subproccess command errors.</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_gpu_memory():\n    \"\"\"Queries the gpu cluster and returns the memory use for each core.\n    This is used to evaluate which GPU cores are available to run jobs on\n    (i.e. pose estimation, DLC model training)\n\n    Returns\n    -------\n    memory_use_values : dict\n        dictionary with core number as key and memory in use as value.\n\n    Raises\n    ------\n    RuntimeError\n        if subproccess command errors.\n    \"\"\"\n\n    def output_to_list(x):\n        return x.decode(\"ascii\").split(\"\\n\")[:-1]\n\n    query_cmd = \"nvidia-smi --query-gpu=memory.used --format=csv\"\n    try:\n        memory_use_info = output_to_list(\n            subprocess.check_output(query_cmd.split(), stderr=subprocess.STDOUT)\n        )[1:]\n    except subprocess.CalledProcessError as err:\n        raise RuntimeError(\n            f\"command {err.cmd} return with error (code {err.returncode}): \"\n            + f\"{err.output}\"\n        ) from err\n    memory_use_values = {\n        i: int(x.split()[0]) for i, x in enumerate(memory_use_info)\n    }\n    return memory_use_values\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_span_start_stop", "title": "<code>get_span_start_stop(indices)</code>", "text": "<p>summary</p> <p>Parameters:</p> Name Type Description Default <code>indices</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_span_start_stop(indices):\n    \"\"\"_summary_\n\n    Parameters\n    ----------\n    indices : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    span_inds = []\n    # Get start and stop index of spans of consecutive indices\n    for k, g in groupby(enumerate(indices), lambda x: x[1] - x[0]):\n        group = list(map(itemgetter(1), g))\n        span_inds.append((group[0], group[-1]))\n    return span_inds\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/", "title": "position_dlc_centroid.py", "text": ""}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams", "title": "<code>DLCCentroidParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for calculating the centroid</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroidParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for calculating the centroid\n    \"\"\"\n\n    # TODO: whether to keep all params in a params dict\n    # or break out into individual secondary keys\n    definition = \"\"\"\n    dlc_centroid_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        \"\"\"\n        Inserts default centroid parameters. Assumes 2 LEDs tracked\n        \"\"\"\n        params = {\n            \"centroid_method\": \"two_pt_centroid\",\n            \"points\": {\n                \"point1\": \"greenLED\",\n                \"point2\": \"redLED_C\",\n            },\n            \"interpolate\": True,\n            \"interp_params\": {\"max_cm_to_interp\": 15},\n            \"smooth\": True,\n            \"smoothing_params\": {\n                \"smoothing_duration\": 0.05,\n                \"smooth_method\": \"moving_avg\",\n            },\n            \"max_LED_separation\": 12,\n            \"speed_smoothing_std_dev\": 0.100,\n        }\n        cls.insert1(\n            {\"dlc_centroid_params_name\": \"default\", \"params\": params}, **kwargs\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_centroid_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (cls &amp; {\"dlc_centroid_params_name\": \"default\"}).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n\n    def insert1(self, key, **kwargs):\n        \"\"\"\n        Check provided parameter dictionary to make sure\n        it contains all necessary items\n        \"\"\"\n        params = key[\"params\"]\n        centroid_method = params.get(\"centroid_method\")\n        validate_option(  # Ensure centroid method is valid\n            option=centroid_method,\n            options=_key_to_points.keys(),\n            name=\"centroid_method\",\n        )\n        validate_list(  # Ensure points are valid for centroid method\n            required_items=set(params[\"points\"].keys()),\n            option_list=params[\"points\"],\n            name=\"points\",\n            condition=centroid_method,\n        )\n\n        validate_option(\n            option=params.get(\"max_LED_separation\"),\n            name=\"max_LED_separation\",\n            types=(int, float),\n            permit_none=True,\n        )\n\n        validate_smooth_params(params)\n\n        super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams.insert_default", "title": "<code>insert_default(**kwargs)</code>  <code>classmethod</code>", "text": "<p>Inserts default centroid parameters. Assumes 2 LEDs tracked</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@classmethod\ndef insert_default(cls, **kwargs):\n    \"\"\"\n    Inserts default centroid parameters. Assumes 2 LEDs tracked\n    \"\"\"\n    params = {\n        \"centroid_method\": \"two_pt_centroid\",\n        \"points\": {\n            \"point1\": \"greenLED\",\n            \"point2\": \"redLED_C\",\n        },\n        \"interpolate\": True,\n        \"interp_params\": {\"max_cm_to_interp\": 15},\n        \"smooth\": True,\n        \"smoothing_params\": {\n            \"smoothing_duration\": 0.05,\n            \"smooth_method\": \"moving_avg\",\n        },\n        \"max_LED_separation\": 12,\n        \"speed_smoothing_std_dev\": 0.100,\n    }\n    cls.insert1(\n        {\"dlc_centroid_params_name\": \"default\", \"params\": params}, **kwargs\n    )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams.insert1", "title": "<code>insert1(key, **kwargs)</code>", "text": "<p>Check provided parameter dictionary to make sure it contains all necessary items</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def insert1(self, key, **kwargs):\n    \"\"\"\n    Check provided parameter dictionary to make sure\n    it contains all necessary items\n    \"\"\"\n    params = key[\"params\"]\n    centroid_method = params.get(\"centroid_method\")\n    validate_option(  # Ensure centroid method is valid\n        option=centroid_method,\n        options=_key_to_points.keys(),\n        name=\"centroid_method\",\n    )\n    validate_list(  # Ensure points are valid for centroid method\n        required_items=set(params[\"points\"].keys()),\n        option_list=params[\"points\"],\n        name=\"points\",\n        condition=centroid_method,\n    )\n\n    validate_option(\n        option=params.get(\"max_LED_separation\"),\n        name=\"max_LED_separation\",\n        types=(int, float),\n        permit_none=True,\n    )\n\n    validate_smooth_params(params)\n\n    super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidSelection", "title": "<code>DLCCentroidSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to pair a cohort of bodypart entries with the parameters for calculating their centroid</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroidSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to pair a cohort of bodypart entries with\n    the parameters for calculating their centroid\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpCohort\n    -&gt; DLCCentroidParams\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroid", "title": "<code>DLCCentroid</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to calculate the centroid of a group of bodyparts</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroid(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to calculate the centroid of a group of bodyparts\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCCentroidSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_position_object_id : varchar(80)\n    dlc_velocity_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        idx = pd.IndexSlice\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            logger.logger.info(\"Centroid Calculation\")\n\n            # Get labels to smooth from Parameters table\n            cohort_entries = DLCSmoothInterpCohort.BodyPart &amp; key\n            params = (DLCCentroidParams() &amp; key).fetch1(\"params\")\n            centroid_method = params.pop(\"centroid_method\")\n            bodyparts_avail = cohort_entries.fetch(\"bodypart\")\n            speed_smoothing_std_dev = params.pop(\"speed_smoothing_std_dev\")\n\n            if not centroid_method:\n                raise ValueError(\"Please specify a centroid method to use.\")\n            validate_option(option=centroid_method, options=_key_to_func_dict)\n\n            points = params.get(\"points\")\n            required_points = _key_to_points.get(centroid_method)\n            validate_list(\n                required_items=required_points,\n                option_list=points,\n                name=\"params points\",\n                condition=centroid_method,\n            )\n            for point in required_points:\n                bodypart = points[point]\n                if bodypart not in bodyparts_avail:\n                    raise ValueError(\n                        \"Bodypart in points not in model.\"\n                        f\"\\tBodypart {bodypart}\"\n                        f\"\\tIn Model {bodyparts_avail}\"\n                    )\n            bodyparts_to_use = [points[point] for point in required_points]\n\n            pos_df = pd.concat(\n                {\n                    bodypart: (\n                        DLCSmoothInterpCohort.BodyPart\n                        &amp; {**key, **{\"bodypart\": bodypart}}\n                    ).fetch1_dataframe()\n                    for bodypart in bodyparts_to_use\n                },\n                axis=1,\n            )\n            dt = np.median(np.diff(pos_df.index.to_numpy()))\n            sampling_rate = 1 / dt\n            logger.logger.info(\n                \"Calculating centroid with %s\", str(centroid_method)\n            )\n            centroid_func = _key_to_func_dict.get(centroid_method)\n            centroid = centroid_func(pos_df, **params)\n            centroid_df = pd.DataFrame(\n                centroid,\n                columns=[\"x\", \"y\"],\n                index=pos_df.index.to_numpy(),\n            )\n            if params[\"interpolate\"]:\n                if np.any(np.isnan(centroid)):\n                    logger.logger.info(\"interpolating over NaNs\")\n                    nan_inds = (\n                        pd.isnull(centroid_df.loc[:, idx[(\"x\", \"y\")]])\n                        .any(axis=1)\n                        .to_numpy()\n                        .nonzero()[0]\n                    )\n                    nan_spans = get_span_start_stop(nan_inds)\n                    interp_df = interp_pos(\n                        centroid_df.copy(), nan_spans, **params[\"interp_params\"]\n                    )\n                else:\n                    logger.logger.info(\"no NaNs to interpolate over\")\n                    interp_df = centroid_df.copy()\n            else:\n                interp_df = centroid_df.copy()\n            if params[\"smooth\"]:\n                smoothing_duration = params[\"smoothing_params\"].get(\n                    \"smoothing_duration\"\n                )\n                if not smoothing_duration:\n                    raise KeyError(\n                        \"smoothing_duration needs to be passed within smoothing_params\"\n                    )\n                dt = np.median(np.diff(pos_df.index.to_numpy()))\n                sampling_rate = 1 / dt\n                logger.logger.info(\"smoothing position\")\n                smooth_func = _key_to_smooth_func_dict[\n                    params[\"smoothing_params\"][\"smooth_method\"]\n                ]\n                logger.logger.info(\n                    \"Smoothing using method: %s\",\n                    str(params[\"smoothing_params\"][\"smooth_method\"]),\n                )\n                final_df = smooth_func(\n                    interp_df,\n                    sampling_rate=sampling_rate,\n                    **params[\"smoothing_params\"],\n                )\n            else:\n                final_df = interp_df.copy()\n            logger.logger.info(\"getting velocity\")\n            velocity = get_velocity(\n                final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                time=pos_df.index.to_numpy(),\n                sigma=speed_smoothing_std_dev,\n                sampling_frequency=sampling_rate,\n            )  # cm/s\n            speed = np.sqrt(np.sum(velocity**2, axis=1))  # cm/s\n            # Create dataframe\n            velocity_df = pd.DataFrame(\n                np.concatenate((velocity, speed[:, np.newaxis]), axis=1),\n                columns=[\"velocity_x\", \"velocity_y\", \"speed\"],\n                index=pos_df.index.to_numpy(),\n            )\n            total_nan = np.sum(\n                final_df.loc[:, idx[(\"x\", \"y\")]].isna().any(axis=1)\n            )\n            pretrack_nan = np.sum(\n                final_df.iloc[:1000].loc[:, idx[(\"x\", \"y\")]].isna().any(axis=1)\n            )\n            logger.logger.info(\"total NaNs in centroid dataset: %d\", total_nan)\n            logger.logger.info(\n                \"NaNs in centroid dataset before ind 1000: %d\", pretrack_nan\n            )\n            position = pynwb.behavior.Position()\n            velocity = pynwb.behavior.BehavioralTimeSeries()\n            spatial_series = (RawPosition() &amp; key).fetch_nwb()[0][\n                \"raw_position\"\n            ]\n            METERS_PER_CM = 0.01\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=final_df.index.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=spatial_series.reference_frame,\n                comments=spatial_series.comments,\n                description=\"x_position, y_position\",\n            )\n            velocity.create_timeseries(\n                name=\"velocity\",\n                timestamps=velocity_df.index.to_numpy(),\n                conversion=METERS_PER_CM,\n                unit=\"m/s\",\n                data=velocity_df.loc[\n                    :, idx[(\"velocity_x\", \"velocity_y\", \"speed\")]\n                ].to_numpy(),\n                comments=spatial_series.comments,\n                description=\"x_velocity, y_velocity, speed\",\n            )\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                timestamps=final_df.index.to_numpy(),\n                data=pos_df[\n                    pos_df.columns.levels[0][0]\n                ].video_frame_ind.to_numpy(),\n                description=\"video_frame_ind\",\n                comments=\"no comments\",\n            )\n            # Add to Analysis NWB file\n            analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n            nwb_analysis_file = AnalysisNwbfile()\n            key.update(\n                {\n                    \"analysis_file_name\": analysis_file_name,\n                    \"dlc_position_object_id\": nwb_analysis_file.add_nwb_object(\n                        analysis_file_name, position\n                    ),\n                    \"dlc_velocity_object_id\": nwb_analysis_file.add_nwb_object(\n                        analysis_file_name, velocity\n                    ),\n                }\n            )\n\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.insert1(key)\n            logger.logger.info(\"inserted entry into DLCCentroid\")\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_position\"].get_spatial_series().timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"position_x\",\n            \"position_y\",\n            \"velocity_x\",\n            \"velocity_y\",\n            \"speed\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"dlc_velocity\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"dlc_position\"].get_spatial_series().data\n                    ),\n                    np.asarray(\n                        nwb_data[\"dlc_velocity\"].time_series[\"velocity\"].data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.four_led_centroid", "title": "<code>four_led_centroid(pos_df, **params)</code>", "text": "<p>Determines the centroid of 4 LEDS on an implant LED ring. Assumed to be the Green LED, and 3 red LEDs called: redLED_C, redLED_L, redLED_R By default, uses (greenled + redLED_C) / 2 to calculate centroid If Green LED is NaN, but red center LED is not,     then the red center LED is called the centroid If green and red center LEDs are NaN, but red left and red right LEDs are not,     then the centroid is (redLED_L + redLED_R) / 2 If red center LED is NaN, but the other 3 LEDS are not,     then the centroid is (greenled + (redLED_L + redLED_R) / 2) / 2 If red center and left LEDs are NaN, but green and red right LEDs are not,     then the centroid is (greenled + redLED_R) / 2 If red center and right LEDs are NaN, but green and red left LEDs are not,     then the centroid is (greenled + redLED_L) / 2 If all red LEDs are NaN, but green LED is not,     then the green LED is called the centroid If all LEDs are NaN, then the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for each LED of interest, index is timestamps. Column names specified by params</p> required <code>**params</code> <code>dict</code> <p>contains 'greenLED' and 'redLED_C', 'redLED_R', 'redLED_L' keys, whose values specify the column names in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def four_led_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines the centroid of 4 LEDS on an implant LED ring.\n    Assumed to be the Green LED, and 3 red LEDs called: redLED_C, redLED_L, redLED_R\n    By default, uses (greenled + redLED_C) / 2 to calculate centroid\n    If Green LED is NaN, but red center LED is not,\n        then the red center LED is called the centroid\n    If green and red center LEDs are NaN, but red left and red right LEDs are not,\n        then the centroid is (redLED_L + redLED_R) / 2\n    If red center LED is NaN, but the other 3 LEDS are not,\n        then the centroid is (greenled + (redLED_L + redLED_R) / 2) / 2\n    If red center and left LEDs are NaN, but green and red right LEDs are not,\n        then the centroid is (greenled + redLED_R) / 2\n    If red center and right LEDs are NaN, but green and red left LEDs are not,\n        then the centroid is (greenled + redLED_L) / 2\n    If all red LEDs are NaN, but green LED is not,\n        then the green LED is called the centroid\n    If all LEDs are NaN, then the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for each LED of interest,\n        index is timestamps. Column names specified by params\n    **params : dict\n        contains 'greenLED' and 'redLED_C', 'redLED_R', 'redLED_L' keys,\n        whose values specify the column names in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n    centroid = np.zeros(shape=(len(pos_df), 2))\n    idx = pd.IndexSlice\n    # TODO: this feels messy, clean-up\n    green_led = params[\"points\"].pop(\"greenLED\", None)\n    red_led_C = params[\"points\"].pop(\"redLED_C\", None)\n    red_led_L = params[\"points\"].pop(\"redLED_L\", None)\n    red_led_R = params[\"points\"].pop(\"redLED_R\", None)\n    green_nans = pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].isna().any(axis=1)\n    red_C_nans = pos_df.loc[:, idx[red_led_C, (\"x\", \"y\")]].isna().any(axis=1)\n    red_L_nans = pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].isna().any(axis=1)\n    red_R_nans = pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].isna().any(axis=1)\n    # TODO: implement checks to make sure not rewriting previously set index in centroid\n    # If all given LEDs are not NaN\n    dist_between_green_red = get_distance(\n        pos_df.loc[:, idx[red_led_C, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    g_c_is_too_separated = (\n        dist_between_green_red &gt;= params[\"max_LED_separation\"]\n    )\n    all_good_mask = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            ~red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~g_c_is_too_separated,\n        ),\n    )\n    centroid[all_good_mask] = [\n        *zip(\n            (\n                pos_df.loc[idx[all_good_mask], idx[red_led_C, \"x\"]]\n                + pos_df.loc[idx[all_good_mask], idx[green_led, \"x\"]]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[all_good_mask], idx[red_led_C, \"y\"]]\n                + pos_df.loc[idx[all_good_mask], idx[green_led, \"y\"]]\n            )\n            / 2,\n        )\n    ]\n    # If green LED and red center LED are both not NaN\n    green_red_C = np.logical_and(\n        ~green_nans, ~red_C_nans, ~g_c_is_too_separated\n    )\n    if np.sum(green_red_C) &gt; 0:\n        centroid[green_red_C] = [\n            *zip(\n                (\n                    pos_df.loc[idx[green_red_C], idx[red_led_C, \"x\"]]\n                    + pos_df.loc[idx[green_red_C], idx[green_led, \"x\"]]\n                )\n                / 2,\n                (\n                    pos_df.loc[idx[green_red_C], idx[red_led_C, \"y\"]]\n                    + pos_df.loc[idx[green_red_C], idx[green_led, \"y\"]]\n                )\n                / 2,\n            )\n        ]\n    # If all given LEDs are NaN\n    all_bad_mask = reduce(\n        np.logical_and, (green_nans, red_C_nans, red_L_nans, red_R_nans)\n    )\n    centroid[all_bad_mask, :] = np.nan\n    # If green LED is NaN, but red center LED is not\n    no_green_red_C = np.logical_and(green_nans, ~red_C_nans)\n    if np.sum(no_green_red_C) &gt; 0:\n        centroid[no_green_red_C] = [\n            *zip(\n                pos_df.loc[idx[no_green_red_C], idx[red_led_C, \"x\"]],\n                pos_df.loc[idx[no_green_red_C], idx[red_led_C, \"y\"]],\n            )\n        ]\n    # If green and red center LEDs are NaN, but red left and red right LEDs are not\n    dist_between_left_right = get_distance(\n        pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].to_numpy(),\n    )\n    l_r_is_too_separated = (\n        dist_between_left_right &gt;= params[\"max_LED_separation\"]\n    )\n    no_green_no_red_C_red_L_red_R = reduce(\n        np.logical_and,\n        (\n            green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~l_r_is_too_separated,\n        ),\n    )\n    if np.sum(no_green_no_red_C_red_L_red_R) &gt; 0:\n        centroid[no_green_no_red_C_red_L_red_R] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_L, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_R, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_L, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_R, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center LED is NaN, but green, red left, and right LEDs are not\n    dist_between_left_green = get_distance(\n        pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    dist_between_right_green = get_distance(\n        pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    l_g_is_too_separated = (\n        dist_between_left_green &gt;= params[\"max_LED_separation\"]\n    )\n    r_g_is_too_separated = (\n        dist_between_right_green &gt;= params[\"max_LED_separation\"]\n    )\n    green_red_L_red_R_no_red_C = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~l_r_is_too_separated,\n            ~l_g_is_too_separated,\n            ~r_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_L_red_R_no_red_C) &gt; 0:\n        midpoint = (\n            (\n                pos_df.loc[idx[green_red_L_red_R_no_red_C], idx[red_led_L, \"x\"]]\n                + pos_df.loc[\n                    idx[green_red_L_red_R_no_red_C], idx[red_led_R, \"x\"]\n                ]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[green_red_L_red_R_no_red_C], idx[red_led_L, \"y\"]]\n                + pos_df.loc[\n                    idx[green_red_L_red_R_no_red_C], idx[red_led_R, \"y\"]\n                ]\n            )\n            / 2,\n        )\n        centroid[green_red_L_red_R_no_red_C] = [\n            *zip(\n                (\n                    midpoint[0]\n                    + pos_df.loc[\n                        idx[green_red_L_red_R_no_red_C], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    midpoint[1]\n                    + pos_df.loc[\n                        idx[green_red_L_red_R_no_red_C], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center and left LED is NaN, but green and red right LED are not\n    green_red_R_no_red_C_no_red_L = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            red_L_nans,\n            ~red_R_nans,\n            ~r_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_R_no_red_C_no_red_L) &gt; 0:\n        centroid[green_red_R_no_red_C_no_red_L] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[red_led_R, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[red_led_R, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center and right LED is NaN, but green and red left LED are not\n    green_red_L_no_red_C_no_red_R = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            red_R_nans,\n            ~l_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_L_no_red_C_no_red_R) &gt; 0:\n        centroid[green_red_L_no_red_C_no_red_R] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[red_led_L, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[red_led_L, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If all LEDS are NaN except red left LED\n    red_L_no_green_no_red_C_no_red_R = reduce(\n        np.logical_and, (green_nans, red_C_nans, ~red_L_nans, red_R_nans)\n    )\n    if np.sum(red_L_no_green_no_red_C_no_red_R) &gt; 0:\n        centroid[red_L_no_green_no_red_C_no_red_R] = [\n            *zip(\n                pos_df.loc[\n                    idx[red_L_no_green_no_red_C_no_red_R], idx[red_led_L, \"x\"]\n                ],\n                pos_df.loc[\n                    idx[red_L_no_green_no_red_C_no_red_R], idx[red_led_L, \"y\"]\n                ],\n            )\n        ]\n    # If all LEDS are NaN except red right LED\n    red_R_no_green_no_red_C_no_red_L = reduce(\n        np.logical_and, (green_nans, red_C_nans, red_L_nans, ~red_R_nans)\n    )\n    if np.sum(red_R_no_green_no_red_C_no_red_L) &gt; 0:\n        centroid[red_R_no_green_no_red_C_no_red_L] = [\n            *zip(\n                pos_df.loc[\n                    idx[red_R_no_green_no_red_C_no_red_L], idx[red_led_R, \"x\"]\n                ],\n                pos_df.loc[\n                    idx[red_R_no_green_no_red_C_no_red_L], idx[red_led_R, \"y\"]\n                ],\n            )\n        ]\n    # If all red LEDs are NaN, but green LED is not\n    green_no_red = reduce(\n        np.logical_and, (~green_nans, red_C_nans, red_L_nans, red_R_nans)\n    )\n    if np.sum(green_no_red) &gt; 0:\n        centroid[green_no_red] = [\n            *zip(\n                pos_df.loc[idx[green_no_red], idx[green_led, \"x\"]],\n                pos_df.loc[idx[green_no_red], idx[green_led, \"y\"]],\n            )\n        ]\n    too_separated_inds = reduce(\n        np.logical_or,\n        (\n            g_c_is_too_separated,\n            l_r_is_too_separated,\n            l_g_is_too_separated,\n            r_g_is_too_separated,\n        ),\n    )\n    if np.sum(too_separated_inds) &gt; 0:\n        centroid[too_separated_inds, :] = np.nan\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.two_pt_centroid", "title": "<code>two_pt_centroid(pos_df, **params)</code>", "text": "<p>Determines the centroid of 2 points using (point1 + point2) / 2 For a given timestamp, if one point is NaN, then the other point is assigned as the centroid. If both are NaN, the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for each point of interest, index is timestamps. Column names specified by params</p> required <code>**params</code> <code>dict</code> <p>contains 'point1' and 'point2' keys, whose values specify the column names in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def two_pt_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"\n    Determines the centroid of 2 points using (point1 + point2) / 2\n    For a given timestamp, if one point is NaN,\n    then the other point is assigned as the centroid.\n    If both are NaN, the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for each point of interest,\n        index is timestamps. Column names specified by params\n    **params : dict\n        contains 'point1' and 'point2' keys,\n        whose values specify the column names in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n\n    idx = pd.IndexSlice\n    centroid = np.zeros(shape=(len(pos_df), 2))\n    PT1 = params[\"points\"].pop(\"point1\", None)\n    PT2 = params[\"points\"].pop(\"point2\", None)\n    pt1_nans = pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].isna().any(axis=1)\n    pt2_nans = pos_df.loc[:, idx[PT2, (\"x\", \"y\")]].isna().any(axis=1)\n    dist_between_points = get_distance(\n        pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[PT2, (\"x\", \"y\")]].to_numpy(),\n    )\n    is_too_separated = dist_between_points &gt;= params[\"max_LED_separation\"]\n    all_good_mask = np.logical_and(~pt1_nans, ~pt2_nans, ~is_too_separated)\n    centroid[all_good_mask] = [\n        *zip(\n            (\n                pos_df.loc[idx[all_good_mask], idx[PT1, \"x\"]]\n                + pos_df.loc[idx[all_good_mask], idx[PT2, \"x\"]]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[all_good_mask], idx[PT1, \"y\"]]\n                + pos_df.loc[idx[all_good_mask], idx[PT2, \"y\"]]\n            )\n            / 2,\n        )\n    ]\n    # If only point1 is good\n    pt1_mask = np.logical_and(~pt1_nans, pt2_nans)\n    if np.sum(pt1_mask) &gt; 0:\n        centroid[pt1_mask] = [\n            *zip(\n                pos_df.loc[idx[pt1_mask], idx[PT1, \"x\"]],\n                pos_df.loc[idx[pt1_mask], idx[PT1, \"y\"]],\n            )\n        ]\n    # If only point2 is good\n    pt2_mask = np.logical_and(pt1_nans, ~pt2_nans)\n    if np.sum(pt2_mask) &gt; 0:\n        centroid[pt2_mask] = [\n            *zip(\n                pos_df.loc[idx[pt2_mask], idx[PT2, \"x\"]],\n                pos_df.loc[idx[pt2_mask], idx[PT2, \"y\"]],\n            )\n        ]\n    # If neither point is not NaN\n    all_bad_mask = np.logical_and(pt1_nans, pt2_nans)\n    centroid[all_bad_mask, :] = np.nan\n    # If LEDs are too far apart\n    centroid[is_too_separated, :] = np.nan\n\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.one_pt_centroid", "title": "<code>one_pt_centroid(pos_df, **params)</code>", "text": "<p>Passes through the provided point as the centroid For a given timestamp, if the point is NaN, then the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for the point of interest, index is timestamps. Column name specified by params</p> required <code>**params</code> <code>dict</code> <p>contains a 'point1' key, whose value specifies the column name in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def one_pt_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"\n    Passes through the provided point as the centroid\n    For a given timestamp, if the point is NaN,\n    then the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for the point of interest,\n        index is timestamps. Column name specified by params\n    **params : dict\n        contains a 'point1' key,\n        whose value specifies the column name in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n    idx = pd.IndexSlice\n    PT1 = params[\"points\"].pop(\"point1\", None)\n    centroid = pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].to_numpy()\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_cohort/", "title": "position_dlc_cohort.py", "text": ""}, {"location": "api/position/v1/position_dlc_cohort/#src.spyglass.position.v1.position_dlc_cohort.DLCSmoothInterpCohortSelection", "title": "<code>DLCSmoothInterpCohortSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to specify which combination of bodyparts from DLCSmoothInterp get combined into a cohort</p> Source code in <code>src/spyglass/position/v1/position_dlc_cohort.py</code> <pre><code>@schema\nclass DLCSmoothInterpCohortSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to specify which combination of bodyparts from DLCSmoothInterp\n    get combined into a cohort\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_si_cohort_selection_name : varchar(120)\n    -&gt; DLCPoseEstimation\n    ---\n    bodyparts_params_dict   : blob      # Dict with bodypart as key and desired dlc_si_params_name as value\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_cohort/#src.spyglass.position.v1.position_dlc_cohort.DLCSmoothInterpCohort", "title": "<code>DLCSmoothInterpCohort</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to combine multiple bodyparts from DLCSmoothInterp to enable centroid/orientation calculations</p> Source code in <code>src/spyglass/position/v1/position_dlc_cohort.py</code> <pre><code>@schema\nclass DLCSmoothInterpCohort(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to combine multiple bodyparts from DLCSmoothInterp\n    to enable centroid/orientation calculations\n    \"\"\"\n\n    # Need to ensure that nwb_file_name/epoch/interval list name endure as primary keys\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpCohortSelection\n    ---\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCSmoothInterpCohort\n        -&gt; DLCSmoothInterp\n        ---\n        -&gt; AnalysisNwbfile\n        dlc_smooth_interp_position_object_id : varchar(80)\n        dlc_smooth_interp_info_object_id : varchar(80)\n        \"\"\"\n\n        def fetch1_dataframe(self):\n            nwb_data = self.fetch_nwb()[0]\n            index = pd.Index(\n                np.asarray(\n                    nwb_data[\"dlc_smooth_interp_position\"]\n                    .get_spatial_series()\n                    .timestamps\n                ),\n                name=\"time\",\n            )\n            COLUMNS = [\n                \"video_frame_ind\",\n                \"x\",\n                \"y\",\n            ]\n            return pd.DataFrame(\n                np.concatenate(\n                    (\n                        np.asarray(\n                            nwb_data[\"dlc_smooth_interp_info\"]\n                            .time_series[\"video_frame_ind\"]\n                            .data,\n                            dtype=int,\n                        )[:, np.newaxis],\n                        np.asarray(\n                            nwb_data[\"dlc_smooth_interp_position\"]\n                            .get_spatial_series()\n                            .data\n                        ),\n                    ),\n                    axis=1,\n                ),\n                columns=COLUMNS,\n                index=index,\n            )\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            logger.logger.info(\"Bodypart Cohort\")\n            # from Jen Guidera\n            self.insert1(key)\n            cohort_selection = (DLCSmoothInterpCohortSelection &amp; key).fetch1()\n            table_entries = []\n            bodyparts_params_dict = cohort_selection.pop(\n                \"bodyparts_params_dict\"\n            )\n            temp_key = cohort_selection.copy()\n            for bodypart, params in bodyparts_params_dict.items():\n                temp_key[\"bodypart\"] = bodypart\n                temp_key[\"dlc_si_params_name\"] = params\n                table_entries.append((DLCSmoothInterp &amp; temp_key).fetch())\n            assert len(table_entries) == len(\n                bodyparts_params_dict\n            ), \"more entries found in DLCSmoothInterp than specified in bodyparts_params_dict\"\n            table_column_names = list(table_entries[0].dtype.fields.keys())\n            for table_entry in table_entries:\n                entry_key = {\n                    **{\n                        k: v for k, v in zip(table_column_names, table_entry[0])\n                    },\n                    **key,\n                }\n                DLCSmoothInterpCohort.BodyPart.insert1(\n                    entry_key, skip_duplicates=True\n                )\n        logger.logger.info(\"Inserted entry into DLCSmoothInterpCohort\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/", "title": "position_dlc_model.py", "text": ""}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelInput", "title": "<code>DLCModelInput</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to hold model path if model is being input from local disk instead of Spyglass</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelInput(SpyglassMixin, dj.Manual):\n    \"\"\"Table to hold model path if model is being input\n    from local disk instead of Spyglass\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_model_name : varchar(64)  # Different than dlc_model_name in DLCModelSource... not great\n    -&gt; DLCProject\n    ---\n    project_path         : varchar(255) # Path to project directory\n    \"\"\"\n\n    def insert1(self, key, **kwargs):\n        # expects key from DLCProject with config_path\n        project_path = Path(key[\"config_path\"]).parent\n        assert project_path.exists(), \"project path does not exist\"\n        key[\"dlc_model_name\"] = f'{project_path.name.split(\"model\")[0]}model'\n        key[\"project_path\"] = project_path.as_posix()\n        del key[\"config_path\"]\n        super().insert1(key, **kwargs)\n        DLCModelSource.insert_entry(\n            dlc_model_name=key[\"dlc_model_name\"],\n            project_name=key[\"project_name\"],\n            source=\"FromImport\",\n            key=key,\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelSource", "title": "<code>DLCModelSource</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to determine whether model originates from upstream DLCModelTraining table, or from local directory</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelSource(SpyglassMixin, dj.Manual):\n    \"\"\"Table to determine whether model originates from\n    upstream DLCModelTraining table, or from local directory\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCProject\n    dlc_model_name : varchar(64)    # User-friendly model name\n    ---\n    source         : enum ('FromUpstream', 'FromImport')\n    \"\"\"\n\n    class FromImport(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCModelSource\n        -&gt; DLCModelInput\n        ---\n        project_path : varchar(255)\n        \"\"\"\n\n    class FromUpstream(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCModelSource\n        -&gt; DLCModelTraining\n        ---\n        project_path : varchar(255)\n        \"\"\"\n\n    @classmethod\n    @accepts(None, None, (\"FromUpstream\", \"FromImport\"), None)\n    def insert_entry(\n        cls,\n        dlc_model_name: str,\n        project_name: str,\n        source: str = \"FromUpstream\",\n        key: dict = None,\n        **kwargs,\n    ):\n        cls.insert1(\n            {\n                \"dlc_model_name\": dlc_model_name,\n                \"project_name\": project_name,\n                \"source\": source,\n            },\n            **kwargs,\n        )\n        part_table = getattr(cls, source)\n        table_query = dj.FreeTable(\n            dj.conn(), full_table_name=part_table.parents()[-1]\n        ) &amp; {\"project_name\": project_name}\n        project_path = table_query.fetch1(\"project_path\")\n        part_table.insert1(\n            {\n                \"dlc_model_name\": dlc_model_name,\n                \"project_name\": project_name,\n                \"project_path\": project_path,\n                **key,\n            },\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelEvaluation", "title": "<code>DLCModelEvaluation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelEvaluation(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCModel\n    ---\n    train_iterations   : int   # Training iterations\n    train_error=null   : float # Train error (px)\n    test_error=null    : float # Test error (px)\n    p_cutoff=null      : float # p-cutoff used\n    train_error_p=null : float # Train error with p-cutoff\n    test_error_p=null  : float # Test error with p-cutoff\n    \"\"\"\n\n    def make(self, key):\n        \"\"\".populate() method will launch evaluation for each unique entry in Model.\"\"\"\n        import csv\n\n        from deeplabcut import evaluate_network\n        from deeplabcut.utils.auxiliaryfunctions import get_evaluation_folder\n\n        dlc_config, project_path, model_prefix, shuffle, trainingsetindex = (\n            DLCModel &amp; key\n        ).fetch1(\n            \"config_template\",\n            \"project_path\",\n            \"model_prefix\",\n            \"shuffle\",\n            \"trainingsetindex\",\n        )\n\n        yml_path, _ = dlc_reader.read_yaml(project_path)\n\n        evaluate_network(\n            yml_path,\n            Shuffles=[shuffle],  # this needs to be a list\n            trainingsetindex=trainingsetindex,\n            comparisonbodyparts=\"all\",\n        )\n\n        eval_folder = get_evaluation_folder(\n            trainFraction=dlc_config[\"TrainingFraction\"][trainingsetindex],\n            shuffle=shuffle,\n            cfg=dlc_config,\n            modelprefix=model_prefix,\n        )\n        eval_path = project_path / eval_folder\n        assert (\n            eval_path.exists()\n        ), f\"Couldn't find evaluation folder:\\n{eval_path}\"\n\n        eval_csvs = list(eval_path.glob(\"*csv\"))\n        max_modified_time = 0\n        for eval_csv in eval_csvs:\n            modified_time = os.path.getmtime(eval_csv)\n            if modified_time &gt; max_modified_time:\n                eval_csv_latest = eval_csv\n        with open(eval_csv_latest, newline=\"\") as f:\n            results = list(csv.DictReader(f, delimiter=\",\"))[0]\n        # in testing, test_error_p returned empty string\n        self.insert1(\n            dict(\n                key,\n                train_iterations=results[\"Training iterations:\"],\n                train_error=results[\" Train error(px)\"],\n                test_error=results[\" Test error(px)\"],\n                p_cutoff=results[\"p-cutoff used\"],\n                train_error_p=results[\"Train error with p-cutoff\"],\n                test_error_p=results[\"Test error with p-cutoff\"],\n            )\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelEvaluation.make", "title": "<code>make(key)</code>", "text": "<p>.populate() method will launch evaluation for each unique entry in Model.</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>def make(self, key):\n    \"\"\".populate() method will launch evaluation for each unique entry in Model.\"\"\"\n    import csv\n\n    from deeplabcut import evaluate_network\n    from deeplabcut.utils.auxiliaryfunctions import get_evaluation_folder\n\n    dlc_config, project_path, model_prefix, shuffle, trainingsetindex = (\n        DLCModel &amp; key\n    ).fetch1(\n        \"config_template\",\n        \"project_path\",\n        \"model_prefix\",\n        \"shuffle\",\n        \"trainingsetindex\",\n    )\n\n    yml_path, _ = dlc_reader.read_yaml(project_path)\n\n    evaluate_network(\n        yml_path,\n        Shuffles=[shuffle],  # this needs to be a list\n        trainingsetindex=trainingsetindex,\n        comparisonbodyparts=\"all\",\n    )\n\n    eval_folder = get_evaluation_folder(\n        trainFraction=dlc_config[\"TrainingFraction\"][trainingsetindex],\n        shuffle=shuffle,\n        cfg=dlc_config,\n        modelprefix=model_prefix,\n    )\n    eval_path = project_path / eval_folder\n    assert (\n        eval_path.exists()\n    ), f\"Couldn't find evaluation folder:\\n{eval_path}\"\n\n    eval_csvs = list(eval_path.glob(\"*csv\"))\n    max_modified_time = 0\n    for eval_csv in eval_csvs:\n        modified_time = os.path.getmtime(eval_csv)\n        if modified_time &gt; max_modified_time:\n            eval_csv_latest = eval_csv\n    with open(eval_csv_latest, newline=\"\") as f:\n        results = list(csv.DictReader(f, delimiter=\",\"))[0]\n    # in testing, test_error_p returned empty string\n    self.insert1(\n        dict(\n            key,\n            train_iterations=results[\"Training iterations:\"],\n            train_error=results[\" Train error(px)\"],\n            test_error=results[\" Test error(px)\"],\n            p_cutoff=results[\"p-cutoff used\"],\n            train_error_p=results[\"Train error with p-cutoff\"],\n            test_error_p=results[\"Test error with p-cutoff\"],\n        )\n    )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.str_to_bool", "title": "<code>str_to_bool(value)</code>", "text": "<p>Return whether the provided string represents true. Otherwise false.</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>def str_to_bool(value) -&gt; bool:\n    \"\"\"Return whether the provided string represents true. Otherwise false.\"\"\"\n    # Due to distutils equivalent depreciation in 3.10\n    # Adopted from github.com/PostHog/posthog/blob/master/posthog/utils.py\n    if not value:\n        return False\n    return str(value).lower() in (\"y\", \"yes\", \"t\", \"true\", \"on\", \"1\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/", "title": "position_dlc_orient.py", "text": ""}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.DLCOrientationParams", "title": "<code>DLCOrientationParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for determining and smoothing the orientation of a set of BodyParts</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>@schema\nclass DLCOrientationParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for determining and smoothing the orientation of a set of BodyParts\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_orientation_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @classmethod\n    def insert_params(cls, params_name: str, params: dict, **kwargs):\n        cls.insert1(\n            {\"dlc_orientation_params_name\": params_name, \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        params = {\n            \"orient_method\": \"red_green_orientation\",\n            \"bodypart1\": \"greenLED\",\n            \"bodypart2\": \"redLED_C\",\n            \"orientation_smoothing_std_dev\": 0.001,\n        }\n        cls.insert1(\n            {\"dlc_orientation_params_name\": \"default\", \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_orientation_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (\n                cls &amp; {\"dlc_orientation_params_name\": \"default\"}\n            ).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.DLCOrientation", "title": "<code>DLCOrientation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Determines and smooths orientation of a set of bodyparts given a specified method</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>@schema\nclass DLCOrientation(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Determines and smooths orientation of a set of bodyparts given a specified method\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCOrientationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_orientation_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        # Get labels to smooth from Parameters table\n        cohort_entries = DLCSmoothInterpCohort.BodyPart &amp; key\n        pos_df = pd.concat(\n            {\n                bodypart: (\n                    DLCSmoothInterpCohort.BodyPart\n                    &amp; {**key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in cohort_entries.fetch(\"bodypart\")\n            },\n            axis=1,\n        )\n        params = (DLCOrientationParams() &amp; key).fetch1(\"params\")\n        orientation_smoothing_std_dev = params.pop(\n            \"orientation_smoothing_std_dev\", None\n        )\n        dt = np.median(np.diff(pos_df.index.to_numpy()))\n        sampling_rate = 1 / dt\n        orient_func = _key_to_func_dict[params[\"orient_method\"]]\n        orientation = orient_func(pos_df, **params)\n        if not params[\"orient_method\"] == \"none\":\n            # Smooth orientation\n            is_nan = np.isnan(orientation)\n            unwrap_orientation = orientation.copy()\n            # Only unwrap non nan values, while keeping nans in dataset for interpolation\n            unwrap_orientation[~is_nan] = np.unwrap(orientation[~is_nan])\n            unwrap_df = pd.DataFrame(\n                unwrap_orientation, columns=[\"orientation\"], index=pos_df.index\n            )\n            nan_spans = get_span_start_stop(np.where(is_nan)[0])\n            orient_df = interp_orientation(\n                unwrap_df,\n                nan_spans,\n            )\n            orientation = gaussian_smooth(\n                orient_df[\"orientation\"].to_numpy(),\n                orientation_smoothing_std_dev,\n                sampling_rate,\n                axis=0,\n                truncate=8,\n            )\n            # convert back to between -pi and pi\n            orientation = np.angle(np.exp(1j * orientation))\n        final_df = pd.DataFrame(\n            orientation, columns=[\"orientation\"], index=pos_df.index\n        )\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        spatial_series = (RawPosition() &amp; key).fetch_nwb()[0][\"raw_position\"]\n        orientation = pynwb.behavior.CompassDirection()\n        orientation.create_spatial_series(\n            name=\"orientation\",\n            timestamps=final_df.index.to_numpy(),\n            conversion=1.0,\n            data=final_df[\"orientation\"].to_numpy(),\n            reference_frame=spatial_series.reference_frame,\n            comments=spatial_series.comments,\n            description=\"orientation\",\n        )\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"dlc_orientation_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], orientation\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_orientation\"].get_spatial_series().timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"orientation\",\n        ]\n        return pd.DataFrame(\n            np.asarray(nwb_data[\"dlc_orientation\"].get_spatial_series().data)[\n                :, np.newaxis\n            ],\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.two_pt_head_orientation", "title": "<code>two_pt_head_orientation(pos_df, **params)</code>", "text": "<p>Determines orientation based on vector between two points</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>def two_pt_head_orientation(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines orientation based on vector between two points\"\"\"\n    BP1 = params.pop(\"bodypart1\", None)\n    BP2 = params.pop(\"bodypart2\", None)\n    orientation = np.arctan2(\n        (pos_df[BP1][\"y\"] - pos_df[BP2][\"y\"]),\n        (pos_df[BP1][\"x\"] - pos_df[BP2][\"x\"]),\n    )\n    return orientation\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.red_led_bisector_orientation", "title": "<code>red_led_bisector_orientation(pos_df, **params)</code>", "text": "<p>Determines orientation based on 2 equally-spaced identifiers that are assumed to be perpendicular to the orientation direction. A third object is needed to determine forward/backward</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>def red_led_bisector_orientation(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines orientation based on 2 equally-spaced identifiers\n    that are assumed to be perpendicular to the orientation direction.\n    A third object is needed to determine forward/backward\n    \"\"\"\n    LED1 = params.pop(\"led1\", None)\n    LED2 = params.pop(\"led2\", None)\n    LED3 = params.pop(\"led3\", None)\n    orientation = []\n    for index, row in pos_df.iterrows():\n        x_vec = row[LED1][\"x\"] - row[LED2][\"x\"]\n        y_vec = row[LED1][\"y\"] - row[LED2][\"y\"]\n        if y_vec == 0:\n            if (row[LED3][\"y\"] &gt; row[LED1][\"y\"]) &amp; (\n                row[LED3][\"y\"] &gt; row[LED2][\"y\"]\n            ):\n                orientation.append(np.pi / 2)\n            elif (row[LED3][\"y\"] &lt; row[LED1][\"y\"]) &amp; (\n                row[LED3][\"y\"] &lt; row[LED2][\"y\"]\n            ):\n                orientation.append(-(np.pi / 2))\n            else:\n                raise Exception(\"Cannot determine head direction from bisector\")\n        else:\n            length = np.sqrt(y_vec * y_vec + x_vec * x_vec)\n            norm = np.array([-y_vec / length, x_vec / length])\n            orientation.append(np.arctan2(norm[1], norm[0]))\n        if index + 1 == len(pos_df):\n            break\n    return np.array(orientation)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/", "title": "position_dlc_pose_estimation.py", "text": ""}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection", "title": "<code>DLCPoseEstimationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@schema\nclass DLCPoseEstimationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; VideoFile                           # Session -&gt; Recording + File part table\n    -&gt; DLCModel                                    # Must specify a DLC project_path\n    ---\n    task_mode='load' : enum('load', 'trigger')  # load results or trigger computation\n    video_path : varchar(120)                   # path to video file\n    pose_estimation_output_dir='': varchar(255) # output dir relative to the root dir\n    pose_estimation_params=null  : longblob     # analyze_videos params, if not default\n    \"\"\"\n\n    @classmethod\n    def get_video_crop(cls, video_path):\n        \"\"\"\n        Queries the user to determine the cropping parameters for a given video\n\n        Parameters\n        ----------\n        video_path : str\n            path to the video file\n\n        Returns\n        -------\n        crop_ints : list\n            list of 4 integers [x min, x max, y min, y max]\n        \"\"\"\n        import cv2\n\n        cap = cv2.VideoCapture(video_path)\n        _, frame = cap.read()\n        fig, ax = plt.subplots(figsize=(20, 10))\n        ax.imshow(frame)\n        xlims = ax.get_xlim()\n        ylims = ax.get_ylim()\n        ax.set_xticks(np.arange(xlims[0], xlims[-1], 50))\n        ax.set_yticks(np.arange(ylims[0], ylims[-1], -50))\n        ax.grid(visible=True, color=\"white\", lw=0.5, alpha=0.5)\n        display(fig)\n        crop_input = input(\n            \"Please enter the crop parameters for your video in format xmin, xmax, ymin, ymax, or 'none'\\n\"\n        )\n        plt.close()\n        if crop_input.lower() == \"none\":\n            return None\n        crop_ints = [int(val) for val in crop_input.split(\",\")]\n        assert all(isinstance(val, int) for val in crop_ints)\n        return crop_ints\n\n    @classmethod\n    def insert_estimation_task(\n        cls,\n        key,\n        task_mode=\"trigger\",  # load or trigger\n        params: dict = None,\n        check_crop=None,\n        skip_duplicates=True,\n    ):\n        \"\"\"\n        Insert PoseEstimationTask in inferred output dir.\n        From Datajoint Elements\n\n        Parameters\n        ----------\n        key: dict\n            DataJoint key specifying a pairing of VideoRecording and Model.\n        task_mode: bool, optional\n            Default 'trigger' computation. Or 'load' existing results.\n        params (dict): Optional. Parameters passed to DLC's analyze_videos:\n            videotype, gputouse, save_as_csv, batchsize, cropping,\n            TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve\n        \"\"\"\n        from .dlc_utils import check_videofile, get_video_path\n\n        video_path, video_filename, _, _ = get_video_path(key)\n        output_dir = infer_output_dir(key)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n        ) as logger:\n            logger.logger.info(\"Pose Estimation Selection\")\n            video_dir = os.path.dirname(video_path) + \"/\"\n            logger.logger.info(\"video_dir: %s\", video_dir)\n            video_path = check_videofile(\n                video_path=video_dir, video_filename=video_filename\n            )[0]\n            if check_crop is not None:\n                params[\"cropping\"] = cls.get_video_crop(\n                    video_path=video_path.as_posix()\n                )\n            cls.insert1(\n                {\n                    **key,\n                    \"task_mode\": task_mode,\n                    \"pose_estimation_params\": params,\n                    \"video_path\": video_path,\n                    \"pose_estimation_output_dir\": output_dir,\n                },\n                skip_duplicates=skip_duplicates,\n            )\n        logger.logger.info(\"inserted entry into Pose Estimation Selection\")\n        return {**key, \"task_mode\": task_mode}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection.get_video_crop", "title": "<code>get_video_crop(video_path)</code>  <code>classmethod</code>", "text": "<p>Queries the user to determine the cropping parameters for a given video</p> <p>Parameters:</p> Name Type Description Default <code>video_path</code> <code>str</code> <p>path to the video file</p> required <p>Returns:</p> Name Type Description <code>crop_ints</code> <code>list</code> <p>list of 4 integers [x min, x max, y min, y max]</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@classmethod\ndef get_video_crop(cls, video_path):\n    \"\"\"\n    Queries the user to determine the cropping parameters for a given video\n\n    Parameters\n    ----------\n    video_path : str\n        path to the video file\n\n    Returns\n    -------\n    crop_ints : list\n        list of 4 integers [x min, x max, y min, y max]\n    \"\"\"\n    import cv2\n\n    cap = cv2.VideoCapture(video_path)\n    _, frame = cap.read()\n    fig, ax = plt.subplots(figsize=(20, 10))\n    ax.imshow(frame)\n    xlims = ax.get_xlim()\n    ylims = ax.get_ylim()\n    ax.set_xticks(np.arange(xlims[0], xlims[-1], 50))\n    ax.set_yticks(np.arange(ylims[0], ylims[-1], -50))\n    ax.grid(visible=True, color=\"white\", lw=0.5, alpha=0.5)\n    display(fig)\n    crop_input = input(\n        \"Please enter the crop parameters for your video in format xmin, xmax, ymin, ymax, or 'none'\\n\"\n    )\n    plt.close()\n    if crop_input.lower() == \"none\":\n        return None\n    crop_ints = [int(val) for val in crop_input.split(\",\")]\n    assert all(isinstance(val, int) for val in crop_ints)\n    return crop_ints\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection.insert_estimation_task", "title": "<code>insert_estimation_task(key, task_mode='trigger', params=None, check_crop=None, skip_duplicates=True)</code>  <code>classmethod</code>", "text": "<p>Insert PoseEstimationTask in inferred output dir. From Datajoint Elements</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <p>DataJoint key specifying a pairing of VideoRecording and Model.</p> required <code>task_mode</code> <p>Default 'trigger' computation. Or 'load' existing results.</p> <code>'trigger'</code> <code>params</code> <code>dict</code> <p>videotype, gputouse, save_as_csv, batchsize, cropping, TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@classmethod\ndef insert_estimation_task(\n    cls,\n    key,\n    task_mode=\"trigger\",  # load or trigger\n    params: dict = None,\n    check_crop=None,\n    skip_duplicates=True,\n):\n    \"\"\"\n    Insert PoseEstimationTask in inferred output dir.\n    From Datajoint Elements\n\n    Parameters\n    ----------\n    key: dict\n        DataJoint key specifying a pairing of VideoRecording and Model.\n    task_mode: bool, optional\n        Default 'trigger' computation. Or 'load' existing results.\n    params (dict): Optional. Parameters passed to DLC's analyze_videos:\n        videotype, gputouse, save_as_csv, batchsize, cropping,\n        TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve\n    \"\"\"\n    from .dlc_utils import check_videofile, get_video_path\n\n    video_path, video_filename, _, _ = get_video_path(key)\n    output_dir = infer_output_dir(key)\n    with OutputLogger(\n        name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n        path=f\"{output_dir.as_posix()}/log.log\",\n    ) as logger:\n        logger.logger.info(\"Pose Estimation Selection\")\n        video_dir = os.path.dirname(video_path) + \"/\"\n        logger.logger.info(\"video_dir: %s\", video_dir)\n        video_path = check_videofile(\n            video_path=video_dir, video_filename=video_filename\n        )[0]\n        if check_crop is not None:\n            params[\"cropping\"] = cls.get_video_crop(\n                video_path=video_path.as_posix()\n            )\n        cls.insert1(\n            {\n                **key,\n                \"task_mode\": task_mode,\n                \"pose_estimation_params\": params,\n                \"video_path\": video_path,\n                \"pose_estimation_output_dir\": output_dir,\n            },\n            skip_duplicates=skip_duplicates,\n        )\n    logger.logger.info(\"inserted entry into Pose Estimation Selection\")\n    return {**key, \"task_mode\": task_mode}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimation", "title": "<code>DLCPoseEstimation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@schema\nclass DLCPoseEstimation(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCPoseEstimationSelection\n    ---\n    pose_estimation_time: datetime  # time of generation of this set of DLC results\n    meters_per_pixel : double       # conversion of meters per pixel for analyzed video\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        definition = \"\"\" # uses DeepLabCut h5 output for body part position\n        -&gt; DLCPoseEstimation\n        -&gt; DLCModel.BodyPart\n        ---\n        -&gt; AnalysisNwbfile\n        dlc_pose_estimation_position_object_id : varchar(80)\n        dlc_pose_estimation_likelihood_object_id : varchar(80)\n        \"\"\"\n\n        _nwb_table = AnalysisNwbfile\n\n        def fetch1_dataframe(self):\n            nwb_data = self.fetch_nwb()[0]\n            index = pd.Index(\n                np.asarray(\n                    nwb_data[\"dlc_pose_estimation_position\"]\n                    .get_spatial_series()\n                    .timestamps\n                ),\n                name=\"time\",\n            )\n            COLUMNS = [\n                \"video_frame_ind\",\n                \"x\",\n                \"y\",\n                \"likelihood\",\n            ]\n            return pd.DataFrame(\n                np.concatenate(\n                    (\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_likelihood\"]\n                            .time_series[\"video_frame_ind\"]\n                            .data,\n                            dtype=int,\n                        )[:, np.newaxis],\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_position\"]\n                            .get_spatial_series()\n                            .data\n                        ),\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_likelihood\"]\n                            .time_series[\"likelihood\"]\n                            .data\n                        )[:, np.newaxis],\n                    ),\n                    axis=1,\n                ),\n                columns=COLUMNS,\n                index=index,\n            )\n\n    def make(self, key):\n        \"\"\".populate() method will launch training for each PoseEstimationTask\"\"\"\n        from . import dlc_reader\n        from .dlc_utils import get_video_path\n\n        METERS_PER_CM = 0.01\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n        ) as logger:\n            logger.logger.info(\"----------------------\")\n            logger.logger.info(\"Pose Estimation\")\n            # ID model and directories\n            dlc_model = (DLCModel &amp; key).fetch1()\n            bodyparts = (DLCModel.BodyPart &amp; key).fetch(\"bodypart\")\n            task_mode, analyze_video_params, video_path, output_dir = (\n                DLCPoseEstimationSelection &amp; key\n            ).fetch1(\n                \"task_mode\",\n                \"pose_estimation_params\",\n                \"video_path\",\n                \"pose_estimation_output_dir\",\n            )\n            analyze_video_params = analyze_video_params or {}\n\n            project_path = dlc_model[\"project_path\"]\n\n            # Trigger PoseEstimation\n            if task_mode == \"trigger\":\n                dlc_reader.do_pose_estimation(\n                    video_path,\n                    dlc_model,\n                    project_path,\n                    output_dir,\n                    **analyze_video_params,\n                )\n            dlc_result = dlc_reader.PoseEstimation(output_dir)\n            creation_time = datetime.fromtimestamp(\n                dlc_result.creation_time\n            ).strftime(\"%Y-%m-%d %H:%M:%S\")\n\n            logger.logger.info(\"getting raw position\")\n            interval_list_name = (\n                convert_epoch_interval_name_to_position_interval_name(\n                    {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"epoch\": key[\"epoch\"],\n                    },\n                    populate_missing=False,\n                )\n            )\n            spatial_series = (\n                RawPosition()\n                &amp; {**key, \"interval_list_name\": interval_list_name}\n            ).fetch_nwb()[0][\"raw_position\"]\n            _, _, _, video_time = get_video_path(key)\n            pos_time = spatial_series.timestamps\n            # TODO: should get timestamps from VideoFile, but need the video_frame_ind from RawPosition,\n            # which also has timestamps\n            key[\"meters_per_pixel\"] = spatial_series.conversion\n\n            # Insert entry into DLCPoseEstimation\n            logger.logger.info(\n                \"Inserting %s, epoch %02d into DLCPoseEsimation\",\n                key[\"nwb_file_name\"],\n                key[\"epoch\"],\n            )\n            self.insert1({**key, \"pose_estimation_time\": creation_time})\n            meters_per_pixel = key[\"meters_per_pixel\"]\n            del key[\"meters_per_pixel\"]\n            body_parts = dlc_result.df.columns.levels[0]\n            body_parts_df = {}\n            # Insert dlc pose estimation into analysis NWB file for\n            # each body part.\n            for body_part in bodyparts:\n                if body_part in body_parts:\n                    body_parts_df[body_part] = pd.DataFrame.from_dict(\n                        {\n                            c: dlc_result.df.get(body_part).get(c).values\n                            for c in dlc_result.df.get(body_part).columns\n                        }\n                    )\n            idx = pd.IndexSlice\n            for body_part, part_df in body_parts_df.items():\n                logger.logger.info(\"converting to cm\")\n                part_df = convert_to_cm(part_df, meters_per_pixel)\n                logger.logger.info(\"adding timestamps to DataFrame\")\n                part_df = add_timestamps(\n                    part_df, pos_time=pos_time, video_time=video_time\n                )\n                key[\"bodypart\"] = body_part\n                key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                    key[\"nwb_file_name\"]\n                )\n                position = pynwb.behavior.Position()\n                likelihood = pynwb.behavior.BehavioralTimeSeries()\n                position.create_spatial_series(\n                    name=\"position\",\n                    timestamps=part_df.time.to_numpy(),\n                    conversion=METERS_PER_CM,\n                    data=part_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                    reference_frame=spatial_series.reference_frame,\n                    comments=spatial_series.comments,\n                    description=\"x_position, y_position\",\n                )\n                likelihood.create_timeseries(\n                    name=\"likelihood\",\n                    timestamps=part_df.time.to_numpy(),\n                    data=part_df.loc[:, idx[\"likelihood\"]].to_numpy(),\n                    unit=\"likelihood\",\n                    comments=\"no comments\",\n                    description=\"likelihood\",\n                )\n                likelihood.create_timeseries(\n                    name=\"video_frame_ind\",\n                    timestamps=part_df.time.to_numpy(),\n                    data=part_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                    unit=\"index\",\n                    comments=\"no comments\",\n                    description=\"video_frame_ind\",\n                )\n                nwb_analysis_file = AnalysisNwbfile()\n                key[\"dlc_pose_estimation_position_object_id\"] = (\n                    nwb_analysis_file.add_nwb_object(\n                        analysis_file_name=key[\"analysis_file_name\"],\n                        nwb_object=position,\n                    )\n                )\n                key[\"dlc_pose_estimation_likelihood_object_id\"] = (\n                    nwb_analysis_file.add_nwb_object(\n                        analysis_file_name=key[\"analysis_file_name\"],\n                        nwb_object=likelihood,\n                    )\n                )\n                nwb_analysis_file.add(\n                    nwb_file_name=key[\"nwb_file_name\"],\n                    analysis_file_name=key[\"analysis_file_name\"],\n                )\n                self.BodyPart.insert1(key)\n\n    def fetch_dataframe(self, *attrs, **kwargs):\n        entries = (self.BodyPart &amp; self).fetch(\"KEY\")\n        nwb_data_dict = {\n            entry[\"bodypart\"]: (self.BodyPart() &amp; entry).fetch_nwb()[0]\n            for entry in entries\n        }\n        index = pd.Index(\n            np.asarray(\n                nwb_data_dict[entries[0][\"bodypart\"]][\n                    \"dlc_pose_estimation_position\"\n                ]\n                .get_spatial_series()\n                .timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"x\",\n            \"y\",\n            \"likelihood\",\n        ]\n        return pd.concat(\n            {\n                entry[\"bodypart\"]: pd.DataFrame(\n                    np.concatenate(\n                        (\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_likelihood\"\n                                ]\n                                .time_series[\"video_frame_ind\"]\n                                .data,\n                                dtype=int,\n                            )[:, np.newaxis],\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_position\"\n                                ]\n                                .get_spatial_series()\n                                .data\n                            ),\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_likelihood\"\n                                ]\n                                .time_series[\"likelihood\"]\n                                .data\n                            )[:, np.newaxis],\n                        ),\n                        axis=1,\n                    ),\n                    columns=COLUMNS,\n                    index=index,\n                )\n                for entry in entries\n            },\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimation.make", "title": "<code>make(key)</code>", "text": "<p>.populate() method will launch training for each PoseEstimationTask</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>def make(self, key):\n    \"\"\".populate() method will launch training for each PoseEstimationTask\"\"\"\n    from . import dlc_reader\n    from .dlc_utils import get_video_path\n\n    METERS_PER_CM = 0.01\n\n    output_dir = infer_output_dir(key=key, makedir=False)\n    with OutputLogger(\n        name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n        path=f\"{output_dir.as_posix()}/log.log\",\n    ) as logger:\n        logger.logger.info(\"----------------------\")\n        logger.logger.info(\"Pose Estimation\")\n        # ID model and directories\n        dlc_model = (DLCModel &amp; key).fetch1()\n        bodyparts = (DLCModel.BodyPart &amp; key).fetch(\"bodypart\")\n        task_mode, analyze_video_params, video_path, output_dir = (\n            DLCPoseEstimationSelection &amp; key\n        ).fetch1(\n            \"task_mode\",\n            \"pose_estimation_params\",\n            \"video_path\",\n            \"pose_estimation_output_dir\",\n        )\n        analyze_video_params = analyze_video_params or {}\n\n        project_path = dlc_model[\"project_path\"]\n\n        # Trigger PoseEstimation\n        if task_mode == \"trigger\":\n            dlc_reader.do_pose_estimation(\n                video_path,\n                dlc_model,\n                project_path,\n                output_dir,\n                **analyze_video_params,\n            )\n        dlc_result = dlc_reader.PoseEstimation(output_dir)\n        creation_time = datetime.fromtimestamp(\n            dlc_result.creation_time\n        ).strftime(\"%Y-%m-%d %H:%M:%S\")\n\n        logger.logger.info(\"getting raw position\")\n        interval_list_name = (\n            convert_epoch_interval_name_to_position_interval_name(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"epoch\": key[\"epoch\"],\n                },\n                populate_missing=False,\n            )\n        )\n        spatial_series = (\n            RawPosition()\n            &amp; {**key, \"interval_list_name\": interval_list_name}\n        ).fetch_nwb()[0][\"raw_position\"]\n        _, _, _, video_time = get_video_path(key)\n        pos_time = spatial_series.timestamps\n        # TODO: should get timestamps from VideoFile, but need the video_frame_ind from RawPosition,\n        # which also has timestamps\n        key[\"meters_per_pixel\"] = spatial_series.conversion\n\n        # Insert entry into DLCPoseEstimation\n        logger.logger.info(\n            \"Inserting %s, epoch %02d into DLCPoseEsimation\",\n            key[\"nwb_file_name\"],\n            key[\"epoch\"],\n        )\n        self.insert1({**key, \"pose_estimation_time\": creation_time})\n        meters_per_pixel = key[\"meters_per_pixel\"]\n        del key[\"meters_per_pixel\"]\n        body_parts = dlc_result.df.columns.levels[0]\n        body_parts_df = {}\n        # Insert dlc pose estimation into analysis NWB file for\n        # each body part.\n        for body_part in bodyparts:\n            if body_part in body_parts:\n                body_parts_df[body_part] = pd.DataFrame.from_dict(\n                    {\n                        c: dlc_result.df.get(body_part).get(c).values\n                        for c in dlc_result.df.get(body_part).columns\n                    }\n                )\n        idx = pd.IndexSlice\n        for body_part, part_df in body_parts_df.items():\n            logger.logger.info(\"converting to cm\")\n            part_df = convert_to_cm(part_df, meters_per_pixel)\n            logger.logger.info(\"adding timestamps to DataFrame\")\n            part_df = add_timestamps(\n                part_df, pos_time=pos_time, video_time=video_time\n            )\n            key[\"bodypart\"] = body_part\n            key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                key[\"nwb_file_name\"]\n            )\n            position = pynwb.behavior.Position()\n            likelihood = pynwb.behavior.BehavioralTimeSeries()\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=part_df.time.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=part_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=spatial_series.reference_frame,\n                comments=spatial_series.comments,\n                description=\"x_position, y_position\",\n            )\n            likelihood.create_timeseries(\n                name=\"likelihood\",\n                timestamps=part_df.time.to_numpy(),\n                data=part_df.loc[:, idx[\"likelihood\"]].to_numpy(),\n                unit=\"likelihood\",\n                comments=\"no comments\",\n                description=\"likelihood\",\n            )\n            likelihood.create_timeseries(\n                name=\"video_frame_ind\",\n                timestamps=part_df.time.to_numpy(),\n                data=part_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                unit=\"index\",\n                comments=\"no comments\",\n                description=\"video_frame_ind\",\n            )\n            nwb_analysis_file = AnalysisNwbfile()\n            key[\"dlc_pose_estimation_position_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=position,\n                )\n            )\n            key[\"dlc_pose_estimation_likelihood_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=likelihood,\n                )\n            )\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.BodyPart.insert1(key)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.add_timestamps", "title": "<code>add_timestamps(df, pos_time, video_time)</code>", "text": "<p>Takes timestamps from raw_pos_df and adds to df, which is returned with timestamps and their matching video frame index</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>pose estimation dataframe to add timestamps</p> required <code>pos_time</code> <code>ndarray</code> <p>numpy array containing timestamps from the raw position object</p> required <code>video_time</code> <code>ndarray</code> <p>numpy array containing timestamps from the video file</p> required <p>Returns:</p> Type Description <code>DataFrame</code> <p>original df with timestamps and video_frame_ind as new columns</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>def add_timestamps(\n    df: pd.DataFrame, pos_time: np.ndarray, video_time: np.ndarray\n) -&gt; pd.DataFrame:\n    \"\"\"\n    Takes timestamps from raw_pos_df and adds to df,\n    which is returned with timestamps and their matching video frame index\n\n    Parameters\n    ----------\n    df : pd.DataFrame\n        pose estimation dataframe to add timestamps\n    pos_time : np.ndarray\n        numpy array containing timestamps from the raw position object\n    video_time: np.ndarray\n        numpy array containing timestamps from the video file\n\n    Returns\n    -------\n    pd.DataFrame\n        original df with timestamps and video_frame_ind as new columns\n    \"\"\"\n    first_video_frame = np.searchsorted(video_time, pos_time[0])\n    video_frame_ind = np.arange(first_video_frame, len(video_time))\n    time_df = pd.DataFrame(\n        index=video_frame_ind,\n        data=video_time[first_video_frame:],\n        columns=[\"time\"],\n    )\n    df = df.join(time_df)\n    # Drop indices where time is NaN\n    df = df.dropna(subset=[\"time\"])\n    # Add video_frame_ind as column\n    df = df.rename_axis(\"video_frame_ind\").reset_index()\n    return df\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/", "title": "position_dlc_position.py", "text": ""}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.DLCSmoothInterpParams", "title": "<code>DLCSmoothInterpParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for extracting the smoothed head position.</p> <p>Attributes:</p> Name Type Description <code>interpolate</code> <code>bool, default True</code> <p>whether to interpolate over NaN spans</p> <code>smooth</code> <code>bool, default True</code> <p>whether to smooth the dataset</p> <code>smoothing_params</code> <code>dict</code> <p>smoothing_duration : float, default 0.05     number of frames to smooth over: sampling_rate*smoothing_duration = num_frames</p> <code>interp_params</code> <code>dict</code> <p>max_cm_to_interp : int, default 20     maximum distance between high likelihood points on either side of a NaN span     to interpolate over</p> <code>likelihood_thresh</code> <code>float, default 0.95</code> <p>likelihood below which to NaN and interpolate over</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>@schema\nclass DLCSmoothInterpParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for extracting the smoothed head position.\n\n    Attributes\n    ----------\n    interpolate : bool, default True\n        whether to interpolate over NaN spans\n    smooth : bool, default True\n        whether to smooth the dataset\n    smoothing_params : dict\n        smoothing_duration : float, default 0.05\n            number of frames to smooth over: sampling_rate*smoothing_duration = num_frames\n    interp_params : dict\n        max_cm_to_interp : int, default 20\n            maximum distance between high likelihood points on either side of a NaN span\n            to interpolate over\n    likelihood_thresh : float, default 0.95\n        likelihood below which to NaN and interpolate over\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_si_params_name : varchar(80) # name for this set of parameters\n    ---\n    params: longblob # dictionary of parameters\n    \"\"\"\n\n    @classmethod\n    def insert_params(cls, params_name: str, params: dict, **kwargs):\n        cls.insert1(\n            {\"dlc_si_params_name\": params_name, \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        default_params = {\n            \"smooth\": True,\n            \"smoothing_params\": {\n                \"smoothing_duration\": 0.05,\n                \"smooth_method\": \"moving_avg\",\n            },\n            \"interpolate\": True,\n            \"likelihood_thresh\": 0.95,\n            \"interp_params\": {\"max_cm_to_interp\": 15},\n            \"max_cm_between_pts\": 20,\n            # This is for use when finding \"good spans\" and is how many indices\n            # to bridge in between good spans see inds_to_span in get_good_spans\n            \"num_inds_to_span\": 20,\n        }\n        cls.insert1(\n            {\"dlc_si_params_name\": \"default\", \"params\": default_params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_nan_params(cls, **kwargs):\n        nan_params = {\n            \"smooth\": False,\n            \"interpolate\": False,\n            \"likelihood_thresh\": 0.95,\n            \"max_cm_between_pts\": 20,\n            \"num_inds_to_span\": 20,\n        }\n        cls.insert1(\n            {\"dlc_si_params_name\": \"just_nan\", \"params\": nan_params}, **kwargs\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_si_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (cls &amp; {\"dlc_si_params_name\": \"default\"}).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n\n    @classmethod\n    def get_nan_params(cls):\n        query = cls &amp; {\"dlc_si_params_name\": \"just_nan\"}\n        if not len(query) &gt; 0:\n            cls().insert_nan_params(skip_duplicates=True)\n            nan_params = (cls &amp; {\"dlc_si_params_name\": \"just_nan\"}).fetch1()\n        else:\n            nan_params = query.fetch1()\n        return nan_params\n\n    @staticmethod\n    def get_available_methods():\n        return _key_to_smooth_func_dict.keys()\n\n    def insert1(self, key, **kwargs):\n        params = key.get(\"params\")\n        if not isinstance(params, dict):\n            raise KeyError(\"'params' must be a dict in key\")\n\n        validate_option(\n            option=params.get(\"max_cm_between_pts\"), name=\"max_cm_between_pts\"\n        )\n        validate_smooth_params(params)\n\n        validate_option(\n            params.get(\"likelihood_thresh\"),\n            name=\"likelihood_thresh\",\n            types=(float),\n            val_range=(0, 1),\n        )\n\n        super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.DLCSmoothInterp", "title": "<code>DLCSmoothInterp</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Interpolates across low likelihood periods and smooths the position Can take a few minutes.</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>@schema\nclass DLCSmoothInterp(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Interpolates across low likelihood periods and smooths the position\n    Can take a few minutes.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_smooth_interp_position_object_id : varchar(80)\n    dlc_smooth_interp_info_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        METERS_PER_CM = 0.01\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            idx = pd.IndexSlice\n            # Get labels to smooth from Parameters table\n            params = (DLCSmoothInterpParams() &amp; key).fetch1(\"params\")\n            # Get DLC output dataframe\n            logger.logger.info(\"fetching Pose Estimation Dataframe\")\n            dlc_df = (DLCPoseEstimation.BodyPart() &amp; key).fetch1_dataframe()\n            dt = np.median(np.diff(dlc_df.index.to_numpy()))\n            sampling_rate = 1 / dt\n            logger.logger.info(\"Identifying indices to NaN\")\n            df_w_nans, bad_inds = nan_inds(\n                dlc_df.copy(),\n                params[\"max_cm_between_pts\"],\n                likelihood_thresh=params.pop(\"likelihood_thresh\"),\n                inds_to_span=params[\"num_inds_to_span\"],\n            )\n\n            nan_spans = get_span_start_stop(np.where(bad_inds)[0])\n            if params[\"interpolate\"]:\n                logger.logger.info(\"interpolating across low likelihood times\")\n                interp_df = interp_pos(\n                    df_w_nans.copy(), nan_spans, **params[\"interp_params\"]\n                )\n            else:\n                interp_df = df_w_nans.copy()\n                logger.logger.info(\"skipping interpolation\")\n            if params[\"smooth\"]:\n                if \"smoothing_duration\" in params[\"smoothing_params\"]:\n                    smoothing_duration = params[\"smoothing_params\"].pop(\n                        \"smoothing_duration\"\n                    )\n                dt = np.median(np.diff(dlc_df.index.to_numpy()))\n                sampling_rate = 1 / dt\n                logger.logger.info(\"smoothing position\")\n                smooth_func = _key_to_smooth_func_dict[\n                    params[\"smoothing_params\"][\"smooth_method\"]\n                ]\n                logger.logger.info(\n                    \"Smoothing using method: %s\",\n                    str(params[\"smoothing_params\"][\"smooth_method\"]),\n                )\n                smooth_df = smooth_func(\n                    interp_df,\n                    smoothing_duration=smoothing_duration,\n                    sampling_rate=sampling_rate,\n                    **params[\"smoothing_params\"],\n                )\n            else:\n                smooth_df = interp_df.copy()\n                logger.logger.info(\"skipping smoothing\")\n            final_df = smooth_df.drop([\"likelihood\"], axis=1)\n            final_df = final_df.rename_axis(\"time\").reset_index()\n            position_nwb_data = (\n                (DLCPoseEstimation.BodyPart() &amp; key)\n                .fetch_nwb()[0][\"dlc_pose_estimation_position\"]\n                .get_spatial_series()\n            )\n            key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                key[\"nwb_file_name\"]\n            )\n            # Add dataframe to AnalysisNwbfile\n            nwb_analysis_file = AnalysisNwbfile()\n            position = pynwb.behavior.Position()\n            video_frame_ind = pynwb.behavior.BehavioralTimeSeries()\n            logger.logger.info(\"Creating NWB objects\")\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=final_df.time.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=position_nwb_data.reference_frame,\n                comments=position_nwb_data.comments,\n                description=\"x_position, y_position\",\n            )\n            video_frame_ind.create_timeseries(\n                name=\"video_frame_ind\",\n                timestamps=final_df.time.to_numpy(),\n                data=final_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                unit=\"index\",\n                comments=\"no comments\",\n                description=\"video_frame_ind\",\n            )\n            key[\"dlc_smooth_interp_position_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=position,\n                )\n            )\n            key[\"dlc_smooth_interp_info_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=video_frame_ind,\n                )\n            )\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.insert1(key)\n            logger.logger.info(\"inserted entry into DLCSmoothInterp\")\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_smooth_interp_position\"]\n                .get_spatial_series()\n                .timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"x\",\n            \"y\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"dlc_smooth_interp_info\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"dlc_smooth_interp_position\"]\n                        .get_spatial_series()\n                        .data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.get_good_spans", "title": "<code>get_good_spans(bad_inds_mask, inds_to_span=50)</code>", "text": "<p>This function takes in a boolean mask of good and bad indices and determines spans of consecutive good indices. It combines two neighboring spans with a separation of less than inds_to_span and treats them as a single good span.</p> <p>Parameters:</p> Name Type Description Default <code>bad_inds_mask</code> <code>boolean mask</code> <p>A boolean mask where True is a bad index and False is a good index.</p> required <code>inds_to_span</code> <code>int</code> <p>This indicates how many indices between two good spans should be bridged to form a single good span. For instance if span A is (1500, 2350) and span B is (2370, 3700), then span A and span B would be combined into span A (1500, 3700) since one would want to identify potential jumps in the space in between the original A and B.</p> <code>50</code> <p>Returns:</p> Name Type Description <code>good_spans</code> <code>list</code> <p>List of spans of good indices, unmodified.</p> <code>modified_spans</code> <code>list</code> <p>spans that are amended to bridge up to inds_to_span consecutive bad indices</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>def get_good_spans(bad_inds_mask, inds_to_span: int = 50):\n    \"\"\"\n    This function takes in a boolean mask of good and bad indices and\n    determines spans of consecutive good indices. It combines two neighboring\n    spans with a separation of less than inds_to_span and treats them as a\n    single good span.\n\n    Parameters\n    ----------\n    bad_inds_mask : boolean mask\n        A boolean mask where True is a bad index and False is a good index.\n    inds_to_span : int, default 50\n        This indicates how many indices between two good spans should\n        be bridged to form a single good span.\n        For instance if span A is (1500, 2350) and span B is (2370, 3700),\n        then span A and span B would be combined into span A (1500, 3700)\n        since one would want to identify potential jumps in the space in between\n        the original A and B.\n\n    Returns\n    -------\n    good_spans : list\n        List of spans of good indices, unmodified.\n    modified_spans : list\n        spans that are amended to bridge up to inds_to_span consecutive bad indices\n    \"\"\"\n    good_spans = get_span_start_stop(\n        np.arange(len(bad_inds_mask))[~bad_inds_mask]\n    )\n    if len(good_spans) &gt; 1:\n        modified_spans = []\n        for (start1, stop1), (start2, stop2) in zip(\n            good_spans[:-1], good_spans[1:]\n        ):\n            check_existing = [\n                entry\n                for entry in modified_spans\n                if start1\n                in range(entry[0] - inds_to_span, entry[1] + inds_to_span)\n            ]\n            if len(check_existing) &gt; 0:\n                modify_ind = modified_spans.index(check_existing[0])\n                if (start2 - stop1) &lt;= inds_to_span:\n                    modified_spans[modify_ind] = (check_existing[0][0], stop2)\n                else:\n                    modified_spans[modify_ind] = (check_existing[0][0], stop1)\n                    modified_spans.append((start2, stop2))\n                continue\n            if (start2 - stop1) &lt;= inds_to_span:\n                modified_spans.append((start1, stop2))\n            else:\n                modified_spans.append((start1, stop1))\n                modified_spans.append((start2, stop2))\n        return good_spans, modified_spans\n    else:\n        return None, good_spans\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/", "title": "position_dlc_project.py", "text": ""}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.BodyPart", "title": "<code>BodyPart</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Holds bodyparts for use in DeepLabCut models</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@schema\nclass BodyPart(SpyglassMixin, dj.Manual):\n    \"\"\"Holds bodyparts for use in DeepLabCut models\"\"\"\n\n    definition = \"\"\"\n    bodypart                : varchar(32)\n    ---\n    bodypart_description='' : varchar(80)\n    \"\"\"\n\n    @classmethod\n    def add_from_config(cls, bodyparts: List, descriptions: List = None):\n        \"\"\"Given a list of bodyparts from the config and\n        an optional list of descriptions, inserts into BodyPart table.\n\n        Parameters\n        ----------\n        bodyparts : List\n            list of bodyparts from config\n        descriptions : List, default None\n            optional list of descriptions for bodyparts.\n            If None, description is set to bodypart name\n        \"\"\"\n        if descriptions is not None:\n            bodyparts_dict = [\n                {\"bodypart\": bp, \"bodypart_description\": desc}\n                for (bp, desc) in zip(bodyparts, descriptions)\n            ]\n        else:\n            bodyparts_dict = [\n                {\"bodypart\": bp, \"bodypart_description\": bp} for bp in bodyparts\n            ]\n        cls.insert(bodyparts_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.BodyPart.add_from_config", "title": "<code>add_from_config(bodyparts, descriptions=None)</code>  <code>classmethod</code>", "text": "<p>Given a list of bodyparts from the config and an optional list of descriptions, inserts into BodyPart table.</p> <p>Parameters:</p> Name Type Description Default <code>bodyparts</code> <code>List</code> <p>list of bodyparts from config</p> required <code>descriptions</code> <code>List</code> <p>optional list of descriptions for bodyparts. If None, description is set to bodypart name</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef add_from_config(cls, bodyparts: List, descriptions: List = None):\n    \"\"\"Given a list of bodyparts from the config and\n    an optional list of descriptions, inserts into BodyPart table.\n\n    Parameters\n    ----------\n    bodyparts : List\n        list of bodyparts from config\n    descriptions : List, default None\n        optional list of descriptions for bodyparts.\n        If None, description is set to bodypart name\n    \"\"\"\n    if descriptions is not None:\n        bodyparts_dict = [\n            {\"bodypart\": bp, \"bodypart_description\": desc}\n            for (bp, desc) in zip(bodyparts, descriptions)\n        ]\n    else:\n        bodyparts_dict = [\n            {\"bodypart\": bp, \"bodypart_description\": bp} for bp in bodyparts\n        ]\n    cls.insert(bodyparts_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject", "title": "<code>DLCProject</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to facilitate creation of a new DeepLabCut model. With ability to edit config, extract frames, label frames</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@schema\nclass DLCProject(SpyglassMixin, dj.Manual):\n    \"\"\"Table to facilitate creation of a new DeepLabCut model.\n    With ability to edit config, extract frames, label frames\n    \"\"\"\n\n    # Add more parameters as secondary keys...\n    # TODO: collapse params into blob dict\n    definition = \"\"\"\n    project_name     : varchar(100) # name of DLC project\n    ---\n    -&gt; LabTeam\n    bodyparts        : blob         # list of bodyparts to label\n    frames_per_video : int          # number of frames to extract from each video\n    config_path      : varchar(120) # path to config.yaml for model\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        \"\"\"Part table to hold bodyparts used in each project.\"\"\"\n\n        definition = \"\"\"\n        -&gt; DLCProject\n        -&gt; BodyPart\n        \"\"\"\n\n    class File(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Paths of training files (e.g., labeled pngs, CSV or video)\n        -&gt; DLCProject\n        file_name: varchar(200) # Concise name to describe file\n        file_ext : enum(\"mp4\", \"csv\", \"h5\") # extension of file\n        ---\n        file_path: varchar(255)\n        \"\"\"\n\n    def insert1(self, key, **kwargs):\n        assert isinstance(\n            key[\"project_name\"], str\n        ), \"project_name must be a string\"\n        assert isinstance(\n            key[\"frames_per_video\"], int\n        ), \"frames_per_video must be of type `int`\"\n        super().insert1(key, **kwargs)\n\n    @classmethod\n    def insert_existing_project(\n        cls,\n        project_name: str,\n        lab_team: str,\n        config_path: str,\n        bodyparts: List = None,\n        frames_per_video: int = None,\n        add_to_files: bool = True,\n        **kwargs,\n    ):\n        \"\"\"\n        insert an existing project into DLCProject table.\n        Parameters\n        ----------\n        project_name : str\n            user-friendly name of project\n        lab_team : str\n            name of lab team. Should match an entry in LabTeam table\n        config_path : str\n            path to project directory\n        bodyparts : list\n            optional list of bodyparts to label that\n            are not already in existing config\n        \"\"\"\n\n        # Read config\n        project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n        if project_name in project_names_in_use:\n            print(f\"project name: {project_name} is already in use.\")\n            return_key = {}\n            return_key[\"project_name\"], return_key[\"config_path\"] = (\n                cls &amp; {\"project_name\": project_name}\n            ).fetch1(\"project_name\", \"config_path\")\n            return return_key\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        cfg = read_config(config_path)\n        if bodyparts:\n            bodyparts_to_add = [\n                bodypart\n                for bodypart in bodyparts\n                if bodypart not in cfg[\"bodyparts\"]\n            ]\n            all_bodyparts = bodyparts_to_add + cfg[\"bodyparts\"]\n        else:\n            all_bodyparts = cfg[\"bodyparts\"]\n        BodyPart.add_from_config(cfg[\"bodyparts\"])\n        for bodypart in all_bodyparts:\n            if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n                raise ValueError(\n                    f\"bodypart: {bodypart} not found in BodyPart table\"\n                )\n        # check bodyparts are in config, if not add\n        if len(bodyparts_to_add) &gt; 0:\n            add_to_config(config_path, bodyparts=bodyparts_to_add)\n        # Get frames per video from config. If passed as arg, check match\n        if frames_per_video:\n            if frames_per_video != cfg[\"numframes2pick\"]:\n                add_to_config(\n                    config_path, **{\"numframes2pick\": frames_per_video}\n                )\n        config_path = Path(config_path)\n        project_path = config_path.parent\n        dlc_project_path = dlc_project_dir\n        if dlc_project_path not in project_path.as_posix():\n            project_dirname = project_path.name\n            dest_folder = Path(f\"{dlc_project_path}/{project_dirname}/\")\n            if dest_folder.exists():\n                new_proj_dir = dest_folder.as_posix()\n            else:\n                new_proj_dir = shutil.copytree(\n                    src=project_path,\n                    dst=f\"{dlc_project_path}/{project_dirname}/\",\n                )\n            new_config_path = Path(f\"{new_proj_dir}/config.yaml\")\n            assert (\n                new_config_path.exists()\n            ), \"config.yaml does not exist in new project directory\"\n            config_path = new_config_path\n            add_to_config(config_path, **{\"project_path\": new_proj_dir})\n        # TODO still need to copy videos over to video dir\n        key = {\n            \"project_name\": project_name,\n            \"team_name\": lab_team,\n            \"bodyparts\": bodyparts,\n            \"config_path\": config_path.as_posix(),\n            \"frames_per_video\": frames_per_video,\n        }\n        cls.insert1(key, **kwargs)\n        cls.BodyPart.insert(\n            [\n                {\"project_name\": project_name, \"bodypart\": bp}\n                for bp in all_bodyparts\n            ],\n            **kwargs,\n        )\n        if add_to_files:\n            del key[\"bodyparts\"]\n            del key[\"team_name\"]\n            del key[\"config_path\"]\n            del key[\"frames_per_video\"]\n            # Check for training files to add\n            cls.add_training_files(key, **kwargs)\n        return {\n            \"project_name\": project_name,\n            \"config_path\": config_path.as_posix(),\n        }\n\n    @classmethod\n    def insert_new_project(\n        cls,\n        project_name: str,\n        bodyparts: List,\n        lab_team: str,\n        frames_per_video: int,\n        video_list: List,\n        groupname: str = None,\n        project_directory: str = dlc_project_dir,\n        output_path: str = dlc_video_dir,\n        **kwargs,\n    ):\n        \"\"\"Insert a new project into DLCProject table.\n\n        Parameters\n        ----------\n        project_name : str\n            user-friendly name of project\n        groupname : str, optional\n            Name for project group. If None, defaults to username\n        bodyparts : list\n            list of bodyparts to label. Should match bodyparts in BodyPart table\n        lab_team : str\n            name of lab team. Should match an entry in LabTeam table\n        project_directory : str\n            directory where to create project.\n            (Default is '/cumulus/deeplabcut/')\n        frames_per_video : int\n            number of frames to extract from each video\n        video_list : list\n            list of (a) dicts of to query VideoFile table for or (b) absolute\n            paths to videos to train on. If dict, use format:\n            [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]\n        output_path : str\n            target path to output converted videos\n            (Default is '/nimbus/deeplabcut/videos/')\n        \"\"\"\n        project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n        if project_name in project_names_in_use:\n            print(f\"project name: {project_name} is already in use.\")\n            return_key = {}\n            return_key[\"project_name\"], return_key[\"config_path\"] = (\n                cls &amp; {\"project_name\": project_name}\n            ).fetch1(\"project_name\", \"config_path\")\n            return return_key\n\n        add_to_files = kwargs.pop(\"add_to_files\", True)\n        if not bool(LabTeam() &amp; {\"team_name\": lab_team}):\n            raise ValueError(f\"team_name: {lab_team} does not exist in LabTeam\")\n        skeleton_node = None\n        # If dict, assume of form {'nwb_file_name': nwb_file_name, 'epoch': epoch}\n        # and pass to get_video_path to reference VideoFile table for path\n\n        if all(isinstance(n, Dict) for n in video_list):\n            videos_to_convert = [\n                get_video_path(video_key) for video_key in video_list\n            ]\n            videos = [\n                check_videofile(\n                    video_path=video[0],\n                    output_path=output_path,\n                    video_filename=video[1],\n                )[0].as_posix()\n                for video in videos_to_convert\n                if video[0] is not None\n            ]\n            if len(videos) &lt; 1:\n                raise ValueError(\n                    f\"no .mp4 videos found in {videos_to_convert[0][0]}\"\n                    + f\" for key: {video_list[0]}\"\n                )\n\n        # If not dict, assume list of video file paths that may or may not need to be converted\n        else:\n            videos = []\n            if not all([Path(video).exists() for video in video_list]):\n                raise OSError(\"at least one file in video_list does not exist\")\n            for video in video_list:\n                video_path = Path(video).parent\n                video_filename = video.rsplit(\n                    video_path.as_posix(), maxsplit=1\n                )[-1].split(\"/\")[-1]\n                videos.extend(\n                    [\n                        check_videofile(\n                            video_path=video_path,\n                            output_path=output_path,\n                            video_filename=video_filename,\n                        )[0].as_posix()\n                    ]\n                )\n            if len(videos) &lt; 1:\n                raise ValueError(f\"no .mp4 videos found in{video_path}\")\n        from deeplabcut import create_new_project\n\n        config_path = create_new_project(\n            project_name,\n            lab_team,\n            videos,\n            working_directory=project_directory,\n            copy_videos=True,\n            multianimal=False,\n        )\n        for bodypart in bodyparts:\n            if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n                raise ValueError(\n                    f\"bodypart: {bodypart} not found in BodyPart table\"\n                )\n        kwargs_copy = copy.deepcopy(kwargs)\n        kwargs_copy.update({\"numframes2pick\": frames_per_video, \"dotsize\": 3})\n        add_to_config(\n            config_path, bodyparts, skeleton_node=skeleton_node, **kwargs_copy\n        )\n        key = {\n            \"project_name\": project_name,\n            \"team_name\": lab_team,\n            \"bodyparts\": bodyparts,\n            \"config_path\": config_path,\n            \"frames_per_video\": frames_per_video,\n        }\n        cls.insert1(key, **kwargs)\n        cls.BodyPart.insert(\n            [\n                {\"project_name\": project_name, \"bodypart\": bp}\n                for bp in bodyparts\n            ],\n            **kwargs,\n        )\n        if add_to_files:\n            del key[\"bodyparts\"]\n            del key[\"team_name\"]\n            del key[\"config_path\"]\n            del key[\"frames_per_video\"]\n            # Add videos to training files\n            cls.add_training_files(key, **kwargs)\n        if isinstance(config_path, PosixPath):\n            config_path = config_path.as_posix()\n        return {\"project_name\": project_name, \"config_path\": config_path}\n\n    @classmethod\n    def add_video_files(\n        cls,\n        video_list,\n        config_path=None,\n        key=None,\n        output_path: str = os.getenv(\"DLC_VIDEO_PATH\"),\n        add_new=False,\n        add_to_files=True,\n        **kwargs,\n    ):\n        has_config_or_key = bool(config_path) or bool(key)\n\n        if add_new and not has_config_or_key:\n            raise ValueError(\"If add_new, must provide key or config_path\")\n        config_path = config_path or (cls &amp; key).fetch1(\"config_path\")\n\n        if (\n            add_to_files\n            and not key\n            and len(cls &amp; {\"config_path\": config_path}) != 1\n        ):\n            raise ValueError(\"Cannot set add_to_files=True without passing key\")\n\n        if all(isinstance(n, Dict) for n in video_list):\n            videos_to_convert = [\n                get_video_path(video_key) for video_key in video_list\n            ]\n            videos = [\n                check_videofile(\n                    video_path=video[0],\n                    output_path=output_path,\n                    video_filename=video[1],\n                )[0].as_posix()\n                for video in videos_to_convert\n            ]\n        # If not dict, assume list of video file paths\n        # that may or may not need to be converted\n        else:\n            videos = []\n            if not all([Path(video).exists() for video in video_list]):\n                raise OSError(\"at least one file in video_list does not exist\")\n            for video in video_list:\n                video_path = Path(video).parent\n                video_filename = video.rsplit(\n                    video_path.as_posix(), maxsplit=1\n                )[-1].split(\"/\")[-1]\n                videos.append(\n                    check_videofile(\n                        video_path=video_path,\n                        output_path=output_path,\n                        video_filename=video_filename,\n                    )[0].as_posix()\n                )\n            if len(videos) &lt; 1:\n                raise ValueError(f\"no .mp4 videos found in{video_path}\")\n        if add_new:\n            from deeplabcut import add_new_videos\n\n            add_new_videos(config=config_path, videos=videos, copy_videos=True)\n        if add_to_files:\n            # Add videos to training files\n            cls.add_training_files(key, **kwargs)\n        return videos\n\n    @classmethod\n    def add_training_files(cls, key, **kwargs):\n        \"\"\"Add training videos and labeled frames .h5\n        and .csv to DLCProject.File\"\"\"\n        config_path = (cls &amp; {\"project_name\": key[\"project_name\"]}).fetch1(\n            \"config_path\"\n        )\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        if \"config_path\" in key:\n            del key[\"config_path\"]\n        cfg = read_config(config_path)\n        video_names = list(cfg[\"video_sets\"].keys())\n        training_files = []\n        for video in video_names:\n            video_name = os.path.splitext(\n                video.split(os.path.dirname(video) + \"/\")[-1]\n            )[0]\n            training_files.extend(\n                glob.glob(\n                    f\"{cfg['project_path']}/\"\n                    f\"labeled-data/{video_name}/*Collected*\"\n                )\n            )\n        for video in video_names:\n            key[\"file_name\"] = f'{os.path.splitext(video.split(\"/\")[-1])[0]}'\n            key[\"file_ext\"] = os.path.splitext(video.split(\"/\")[-1])[-1].split(\n                \".\"\n            )[-1]\n            key[\"file_path\"] = video\n            cls.File.insert1(key, **kwargs)\n        if len(training_files) &gt; 0:\n            for file in training_files:\n                video_name = os.path.dirname(file).split(\"/\")[-1]\n                file_type = os.path.splitext(\n                    file.split(os.path.dirname(file) + \"/\")[-1]\n                )[-1].split(\".\")[-1]\n                key[\"file_name\"] = f\"{video_name}_labeled_data\"\n                key[\"file_ext\"] = file_type\n                key[\"file_path\"] = file\n                cls.File.insert1(key, **kwargs)\n        else:\n            Warning(\"No training files to add\")\n\n    @classmethod\n    def run_extract_frames(cls, key, **kwargs):\n        \"\"\"Convenience function to launch DLC GUI for extracting frames.\n        Must be run on local machine to access GUI,\n        cannot be run through ssh tunnel\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import extract_frames\n\n        extract_frames(config_path, **kwargs)\n\n    @classmethod\n    def run_label_frames(cls, key):\n        \"\"\"Convenience function to launch DLC GUI for labeling frames.\n        Must be run on local machine to access GUI,\n        cannot be run through ssh tunnel\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import label_frames\n\n        label_frames(config_path)\n\n    @classmethod\n    def check_labels(cls, key, **kwargs):\n        \"\"\"Convenience function to check labels on\n        previously extracted and labeled frames\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import check_labels\n\n        check_labels(config_path, **kwargs)\n\n    @classmethod\n    def import_labeled_frames(\n        cls,\n        key: Dict,\n        import_project_path: Union[str, PosixPath],\n        video_filenames: Union[str, List],\n        **kwargs,\n    ):\n        \"\"\"Function to import pre-labeled frames from an existing project\n        into a new project\n\n        Parameters\n        ----------\n        key : Dict\n            key to specify entry in DLCProject table to add labeled frames to\n        import_project_path : str\n            absolute path to project directory containing\n            labeled frames to import\n        video_filenames : str or List\n            filename or list of filenames of video(s)\n            from which to import frames.\n            without file extension\n        \"\"\"\n        project_entry = (cls &amp; key).fetch1()\n        team_name = project_entry[\"team_name\"]\n        current_project_path = Path(project_entry[\"config_path\"]).parent\n        current_labeled_data_path = Path(\n            f\"{current_project_path.as_posix()}/labeled-data\"\n        )\n        if isinstance(import_project_path, PosixPath):\n            assert import_project_path.exists(), (\n                \"import_project_path: \"\n                f\"{import_project_path.as_posix()} does not exist\"\n            )\n            import_labeled_data_path = Path(\n                f\"{import_project_path.as_posix()}/labeled-data\"\n            )\n        else:\n            assert Path(\n                import_project_path\n            ).exists(), (\n                f\"import_project_path: {import_project_path} does not exist\"\n            )\n            import_labeled_data_path = Path(\n                f\"{import_project_path}/labeled-data\"\n            )\n        assert (\n            import_labeled_data_path.exists()\n        ), \"import_project has no directory 'labeled-data'\"\n        if not isinstance(video_filenames, List):\n            video_filenames = [video_filenames]\n        for video_file in video_filenames:\n            h5_file = glob.glob(\n                f\"{import_labeled_data_path.as_posix()}/{video_file}/*.h5\"\n            )[0]\n            dlc_df = pd.read_hdf(h5_file)\n            dlc_df.columns = dlc_df.columns.set_levels([team_name], level=0)\n            dlc_df.to_hdf(\n                Path(\n                    f\"{current_labeled_data_path.as_posix()}/\"\n                    f\"{video_file}/CollectedData_{team_name}.h5\"\n                ).as_posix(),\n                \"df_with_missing\",\n            )\n        cls.add_training_files(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.BodyPart", "title": "<code>BodyPart</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Part table to hold bodyparts used in each project.</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>class BodyPart(SpyglassMixin, dj.Part):\n    \"\"\"Part table to hold bodyparts used in each project.\"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCProject\n    -&gt; BodyPart\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.insert_existing_project", "title": "<code>insert_existing_project(project_name, lab_team, config_path, bodyparts=None, frames_per_video=None, add_to_files=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>insert an existing project into DLCProject table.</p> <p>Parameters:</p> Name Type Description Default <code>project_name</code> <code>str</code> <p>user-friendly name of project</p> required <code>lab_team</code> <code>str</code> <p>name of lab team. Should match an entry in LabTeam table</p> required <code>config_path</code> <code>str</code> <p>path to project directory</p> required <code>bodyparts</code> <code>list</code> <p>optional list of bodyparts to label that are not already in existing config</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef insert_existing_project(\n    cls,\n    project_name: str,\n    lab_team: str,\n    config_path: str,\n    bodyparts: List = None,\n    frames_per_video: int = None,\n    add_to_files: bool = True,\n    **kwargs,\n):\n    \"\"\"\n    insert an existing project into DLCProject table.\n    Parameters\n    ----------\n    project_name : str\n        user-friendly name of project\n    lab_team : str\n        name of lab team. Should match an entry in LabTeam table\n    config_path : str\n        path to project directory\n    bodyparts : list\n        optional list of bodyparts to label that\n        are not already in existing config\n    \"\"\"\n\n    # Read config\n    project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n    if project_name in project_names_in_use:\n        print(f\"project name: {project_name} is already in use.\")\n        return_key = {}\n        return_key[\"project_name\"], return_key[\"config_path\"] = (\n            cls &amp; {\"project_name\": project_name}\n        ).fetch1(\"project_name\", \"config_path\")\n        return return_key\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    cfg = read_config(config_path)\n    if bodyparts:\n        bodyparts_to_add = [\n            bodypart\n            for bodypart in bodyparts\n            if bodypart not in cfg[\"bodyparts\"]\n        ]\n        all_bodyparts = bodyparts_to_add + cfg[\"bodyparts\"]\n    else:\n        all_bodyparts = cfg[\"bodyparts\"]\n    BodyPart.add_from_config(cfg[\"bodyparts\"])\n    for bodypart in all_bodyparts:\n        if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n            raise ValueError(\n                f\"bodypart: {bodypart} not found in BodyPart table\"\n            )\n    # check bodyparts are in config, if not add\n    if len(bodyparts_to_add) &gt; 0:\n        add_to_config(config_path, bodyparts=bodyparts_to_add)\n    # Get frames per video from config. If passed as arg, check match\n    if frames_per_video:\n        if frames_per_video != cfg[\"numframes2pick\"]:\n            add_to_config(\n                config_path, **{\"numframes2pick\": frames_per_video}\n            )\n    config_path = Path(config_path)\n    project_path = config_path.parent\n    dlc_project_path = dlc_project_dir\n    if dlc_project_path not in project_path.as_posix():\n        project_dirname = project_path.name\n        dest_folder = Path(f\"{dlc_project_path}/{project_dirname}/\")\n        if dest_folder.exists():\n            new_proj_dir = dest_folder.as_posix()\n        else:\n            new_proj_dir = shutil.copytree(\n                src=project_path,\n                dst=f\"{dlc_project_path}/{project_dirname}/\",\n            )\n        new_config_path = Path(f\"{new_proj_dir}/config.yaml\")\n        assert (\n            new_config_path.exists()\n        ), \"config.yaml does not exist in new project directory\"\n        config_path = new_config_path\n        add_to_config(config_path, **{\"project_path\": new_proj_dir})\n    # TODO still need to copy videos over to video dir\n    key = {\n        \"project_name\": project_name,\n        \"team_name\": lab_team,\n        \"bodyparts\": bodyparts,\n        \"config_path\": config_path.as_posix(),\n        \"frames_per_video\": frames_per_video,\n    }\n    cls.insert1(key, **kwargs)\n    cls.BodyPart.insert(\n        [\n            {\"project_name\": project_name, \"bodypart\": bp}\n            for bp in all_bodyparts\n        ],\n        **kwargs,\n    )\n    if add_to_files:\n        del key[\"bodyparts\"]\n        del key[\"team_name\"]\n        del key[\"config_path\"]\n        del key[\"frames_per_video\"]\n        # Check for training files to add\n        cls.add_training_files(key, **kwargs)\n    return {\n        \"project_name\": project_name,\n        \"config_path\": config_path.as_posix(),\n    }\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.insert_new_project", "title": "<code>insert_new_project(project_name, bodyparts, lab_team, frames_per_video, video_list, groupname=None, project_directory=dlc_project_dir, output_path=dlc_video_dir, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert a new project into DLCProject table.</p> <p>Parameters:</p> Name Type Description Default <code>project_name</code> <code>str</code> <p>user-friendly name of project</p> required <code>groupname</code> <code>str</code> <p>Name for project group. If None, defaults to username</p> <code>None</code> <code>bodyparts</code> <code>list</code> <p>list of bodyparts to label. Should match bodyparts in BodyPart table</p> required <code>lab_team</code> <code>str</code> <p>name of lab team. Should match an entry in LabTeam table</p> required <code>project_directory</code> <code>str</code> <p>directory where to create project. (Default is '/cumulus/deeplabcut/')</p> <code>dlc_project_dir</code> <code>frames_per_video</code> <code>int</code> <p>number of frames to extract from each video</p> required <code>video_list</code> <code>list</code> <p>list of (a) dicts of to query VideoFile table for or (b) absolute paths to videos to train on. If dict, use format: [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]</p> required <code>output_path</code> <code>str</code> <p>target path to output converted videos (Default is '/nimbus/deeplabcut/videos/')</p> <code>dlc_video_dir</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef insert_new_project(\n    cls,\n    project_name: str,\n    bodyparts: List,\n    lab_team: str,\n    frames_per_video: int,\n    video_list: List,\n    groupname: str = None,\n    project_directory: str = dlc_project_dir,\n    output_path: str = dlc_video_dir,\n    **kwargs,\n):\n    \"\"\"Insert a new project into DLCProject table.\n\n    Parameters\n    ----------\n    project_name : str\n        user-friendly name of project\n    groupname : str, optional\n        Name for project group. If None, defaults to username\n    bodyparts : list\n        list of bodyparts to label. Should match bodyparts in BodyPart table\n    lab_team : str\n        name of lab team. Should match an entry in LabTeam table\n    project_directory : str\n        directory where to create project.\n        (Default is '/cumulus/deeplabcut/')\n    frames_per_video : int\n        number of frames to extract from each video\n    video_list : list\n        list of (a) dicts of to query VideoFile table for or (b) absolute\n        paths to videos to train on. If dict, use format:\n        [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]\n    output_path : str\n        target path to output converted videos\n        (Default is '/nimbus/deeplabcut/videos/')\n    \"\"\"\n    project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n    if project_name in project_names_in_use:\n        print(f\"project name: {project_name} is already in use.\")\n        return_key = {}\n        return_key[\"project_name\"], return_key[\"config_path\"] = (\n            cls &amp; {\"project_name\": project_name}\n        ).fetch1(\"project_name\", \"config_path\")\n        return return_key\n\n    add_to_files = kwargs.pop(\"add_to_files\", True)\n    if not bool(LabTeam() &amp; {\"team_name\": lab_team}):\n        raise ValueError(f\"team_name: {lab_team} does not exist in LabTeam\")\n    skeleton_node = None\n    # If dict, assume of form {'nwb_file_name': nwb_file_name, 'epoch': epoch}\n    # and pass to get_video_path to reference VideoFile table for path\n\n    if all(isinstance(n, Dict) for n in video_list):\n        videos_to_convert = [\n            get_video_path(video_key) for video_key in video_list\n        ]\n        videos = [\n            check_videofile(\n                video_path=video[0],\n                output_path=output_path,\n                video_filename=video[1],\n            )[0].as_posix()\n            for video in videos_to_convert\n            if video[0] is not None\n        ]\n        if len(videos) &lt; 1:\n            raise ValueError(\n                f\"no .mp4 videos found in {videos_to_convert[0][0]}\"\n                + f\" for key: {video_list[0]}\"\n            )\n\n    # If not dict, assume list of video file paths that may or may not need to be converted\n    else:\n        videos = []\n        if not all([Path(video).exists() for video in video_list]):\n            raise OSError(\"at least one file in video_list does not exist\")\n        for video in video_list:\n            video_path = Path(video).parent\n            video_filename = video.rsplit(\n                video_path.as_posix(), maxsplit=1\n            )[-1].split(\"/\")[-1]\n            videos.extend(\n                [\n                    check_videofile(\n                        video_path=video_path,\n                        output_path=output_path,\n                        video_filename=video_filename,\n                    )[0].as_posix()\n                ]\n            )\n        if len(videos) &lt; 1:\n            raise ValueError(f\"no .mp4 videos found in{video_path}\")\n    from deeplabcut import create_new_project\n\n    config_path = create_new_project(\n        project_name,\n        lab_team,\n        videos,\n        working_directory=project_directory,\n        copy_videos=True,\n        multianimal=False,\n    )\n    for bodypart in bodyparts:\n        if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n            raise ValueError(\n                f\"bodypart: {bodypart} not found in BodyPart table\"\n            )\n    kwargs_copy = copy.deepcopy(kwargs)\n    kwargs_copy.update({\"numframes2pick\": frames_per_video, \"dotsize\": 3})\n    add_to_config(\n        config_path, bodyparts, skeleton_node=skeleton_node, **kwargs_copy\n    )\n    key = {\n        \"project_name\": project_name,\n        \"team_name\": lab_team,\n        \"bodyparts\": bodyparts,\n        \"config_path\": config_path,\n        \"frames_per_video\": frames_per_video,\n    }\n    cls.insert1(key, **kwargs)\n    cls.BodyPart.insert(\n        [\n            {\"project_name\": project_name, \"bodypart\": bp}\n            for bp in bodyparts\n        ],\n        **kwargs,\n    )\n    if add_to_files:\n        del key[\"bodyparts\"]\n        del key[\"team_name\"]\n        del key[\"config_path\"]\n        del key[\"frames_per_video\"]\n        # Add videos to training files\n        cls.add_training_files(key, **kwargs)\n    if isinstance(config_path, PosixPath):\n        config_path = config_path.as_posix()\n    return {\"project_name\": project_name, \"config_path\": config_path}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.add_training_files", "title": "<code>add_training_files(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Add training videos and labeled frames .h5 and .csv to DLCProject.File</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef add_training_files(cls, key, **kwargs):\n    \"\"\"Add training videos and labeled frames .h5\n    and .csv to DLCProject.File\"\"\"\n    config_path = (cls &amp; {\"project_name\": key[\"project_name\"]}).fetch1(\n        \"config_path\"\n    )\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    if \"config_path\" in key:\n        del key[\"config_path\"]\n    cfg = read_config(config_path)\n    video_names = list(cfg[\"video_sets\"].keys())\n    training_files = []\n    for video in video_names:\n        video_name = os.path.splitext(\n            video.split(os.path.dirname(video) + \"/\")[-1]\n        )[0]\n        training_files.extend(\n            glob.glob(\n                f\"{cfg['project_path']}/\"\n                f\"labeled-data/{video_name}/*Collected*\"\n            )\n        )\n    for video in video_names:\n        key[\"file_name\"] = f'{os.path.splitext(video.split(\"/\")[-1])[0]}'\n        key[\"file_ext\"] = os.path.splitext(video.split(\"/\")[-1])[-1].split(\n            \".\"\n        )[-1]\n        key[\"file_path\"] = video\n        cls.File.insert1(key, **kwargs)\n    if len(training_files) &gt; 0:\n        for file in training_files:\n            video_name = os.path.dirname(file).split(\"/\")[-1]\n            file_type = os.path.splitext(\n                file.split(os.path.dirname(file) + \"/\")[-1]\n            )[-1].split(\".\")[-1]\n            key[\"file_name\"] = f\"{video_name}_labeled_data\"\n            key[\"file_ext\"] = file_type\n            key[\"file_path\"] = file\n            cls.File.insert1(key, **kwargs)\n    else:\n        Warning(\"No training files to add\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.run_extract_frames", "title": "<code>run_extract_frames(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Convenience function to launch DLC GUI for extracting frames. Must be run on local machine to access GUI, cannot be run through ssh tunnel</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef run_extract_frames(cls, key, **kwargs):\n    \"\"\"Convenience function to launch DLC GUI for extracting frames.\n    Must be run on local machine to access GUI,\n    cannot be run through ssh tunnel\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import extract_frames\n\n    extract_frames(config_path, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.run_label_frames", "title": "<code>run_label_frames(key)</code>  <code>classmethod</code>", "text": "<p>Convenience function to launch DLC GUI for labeling frames. Must be run on local machine to access GUI, cannot be run through ssh tunnel</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef run_label_frames(cls, key):\n    \"\"\"Convenience function to launch DLC GUI for labeling frames.\n    Must be run on local machine to access GUI,\n    cannot be run through ssh tunnel\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import label_frames\n\n    label_frames(config_path)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.check_labels", "title": "<code>check_labels(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Convenience function to check labels on previously extracted and labeled frames</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef check_labels(cls, key, **kwargs):\n    \"\"\"Convenience function to check labels on\n    previously extracted and labeled frames\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import check_labels\n\n    check_labels(config_path, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.import_labeled_frames", "title": "<code>import_labeled_frames(key, import_project_path, video_filenames, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Function to import pre-labeled frames from an existing project into a new project</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>Dict</code> <p>key to specify entry in DLCProject table to add labeled frames to</p> required <code>import_project_path</code> <code>str</code> <p>absolute path to project directory containing labeled frames to import</p> required <code>video_filenames</code> <code>str or List</code> <p>filename or list of filenames of video(s) from which to import frames. without file extension</p> required Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef import_labeled_frames(\n    cls,\n    key: Dict,\n    import_project_path: Union[str, PosixPath],\n    video_filenames: Union[str, List],\n    **kwargs,\n):\n    \"\"\"Function to import pre-labeled frames from an existing project\n    into a new project\n\n    Parameters\n    ----------\n    key : Dict\n        key to specify entry in DLCProject table to add labeled frames to\n    import_project_path : str\n        absolute path to project directory containing\n        labeled frames to import\n    video_filenames : str or List\n        filename or list of filenames of video(s)\n        from which to import frames.\n        without file extension\n    \"\"\"\n    project_entry = (cls &amp; key).fetch1()\n    team_name = project_entry[\"team_name\"]\n    current_project_path = Path(project_entry[\"config_path\"]).parent\n    current_labeled_data_path = Path(\n        f\"{current_project_path.as_posix()}/labeled-data\"\n    )\n    if isinstance(import_project_path, PosixPath):\n        assert import_project_path.exists(), (\n            \"import_project_path: \"\n            f\"{import_project_path.as_posix()} does not exist\"\n        )\n        import_labeled_data_path = Path(\n            f\"{import_project_path.as_posix()}/labeled-data\"\n        )\n    else:\n        assert Path(\n            import_project_path\n        ).exists(), (\n            f\"import_project_path: {import_project_path} does not exist\"\n        )\n        import_labeled_data_path = Path(\n            f\"{import_project_path}/labeled-data\"\n        )\n    assert (\n        import_labeled_data_path.exists()\n    ), \"import_project has no directory 'labeled-data'\"\n    if not isinstance(video_filenames, List):\n        video_filenames = [video_filenames]\n    for video_file in video_filenames:\n        h5_file = glob.glob(\n            f\"{import_labeled_data_path.as_posix()}/{video_file}/*.h5\"\n        )[0]\n        dlc_df = pd.read_hdf(h5_file)\n        dlc_df.columns = dlc_df.columns.set_levels([team_name], level=0)\n        dlc_df.to_hdf(\n            Path(\n                f\"{current_labeled_data_path.as_posix()}/\"\n                f\"{video_file}/CollectedData_{team_name}.h5\"\n            ).as_posix(),\n            \"df_with_missing\",\n        )\n    cls.add_training_files(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.add_to_config", "title": "<code>add_to_config(config, bodyparts=None, skeleton_node=None, **kwargs)</code>", "text": "<p>Add necessary items to the config.yaml for the model</p> <p>Parameters:</p> Name Type Description Default <code>config</code> <code>str</code> <p>Path to config.yaml</p> required <code>bodyparts</code> <code>list</code> <p>list of bodyparts to add to model</p> <code>None</code> <code>skeleton_node</code> <code>str</code> <p>(default is None) node to link LEDs in skeleton</p> <code>None</code> <code>kwargs</code> <code>dict</code> <p>Other parameters of config to modify in key:value pairs</p> <code>{}</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>def add_to_config(\n    config, bodyparts: List = None, skeleton_node: str = None, **kwargs\n):\n    \"\"\"\n    Add necessary items to the config.yaml for the model\n    Parameters\n    ----------\n    config : str\n        Path to config.yaml\n    bodyparts : list\n        list of bodyparts to add to model\n    skeleton_node : str\n        (default is None) node to link LEDs in skeleton\n    kwargs : dict\n        Other parameters of config to modify in key:value pairs\n    \"\"\"\n\n    yaml = ruamel.yaml.YAML()\n    with open(config) as fp:\n        data = yaml.load(fp)\n    if bodyparts:\n        data[\"bodyparts\"] = bodyparts\n        led_parts = [element for element in bodyparts if \"LED\" in element]\n        if skeleton_node is not None:\n            bodypart_skeleton = [\n                list(link)\n                for link in combinations(led_parts, 2)\n                if skeleton_node in link\n            ]\n        else:\n            bodypart_skeleton = list(combinations(led_parts, 2))\n        other_parts = list(set(bodyparts) - set(led_parts))\n        for ind, part in enumerate(other_parts):\n            other_parts[ind] = [part, part]\n        bodypart_skeleton.append(other_parts)\n        data[\"skeleton\"] = bodypart_skeleton\n    for kwarg, val in kwargs.items():\n        if not isinstance(kwarg, str):\n            kwarg = str(kwarg)\n        data[kwarg] = val\n    with open(config, \"w\") as fw:\n        yaml.dump(data, fw)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/", "title": "position_dlc_selection.py", "text": ""}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosSelection", "title": "<code>DLCPosSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Specify collection of upstream DLCCentroid and DLCOrientation entries to combine into a set of position information</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Specify collection of upstream DLCCentroid and DLCOrientation entries\n    to combine into a set of position information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCCentroid.proj(dlc_si_cohort_centroid='dlc_si_cohort_selection_name', centroid_analysis_file_name='analysis_file_name')\n    -&gt; DLCOrientation.proj(dlc_si_cohort_orientation='dlc_si_cohort_selection_name', orientation_analysis_file_name='analysis_file_name')\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosV1", "title": "<code>DLCPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Combines upstream DLCCentroid and DLCOrientation entries into a single entry with a single Analysis NWB file</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosV1(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Combines upstream DLCCentroid and DLCOrientation\n    entries into a single entry with a single Analysis NWB file\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCPosSelection\n    ---\n    -&gt; AnalysisNwbfile\n    position_object_id      : varchar(80)\n    orientation_object_id   : varchar(80)\n    velocity_object_id      : varchar(80)\n    pose_eval_result        : longblob\n    \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n        key[\"pose_eval_result\"] = self.evaluate_pose_estimation(key)\n\n        pos_nwb = (DLCCentroid &amp; key).fetch_nwb()[0]\n        ori_nwb = (DLCOrientation &amp; key).fetch_nwb()[0]\n\n        pos_obj = pos_nwb[\"dlc_position\"].spatial_series[\"position\"]\n        vel_obj = pos_nwb[\"dlc_velocity\"].time_series[\"velocity\"]\n        vid_frame_obj = pos_nwb[\"dlc_velocity\"].time_series[\"video_frame_ind\"]\n        ori_obj = ori_nwb[\"dlc_orientation\"].spatial_series[\"orientation\"]\n\n        position = pynwb.behavior.Position()\n        orientation = pynwb.behavior.CompassDirection()\n        velocity = pynwb.behavior.BehavioralTimeSeries()\n\n        position.create_spatial_series(\n            name=pos_obj.name,\n            timestamps=np.asarray(pos_obj.timestamps),\n            conversion=pos_obj.conversion,\n            data=np.asarray(pos_obj.data),\n            reference_frame=pos_obj.reference_frame,\n            comments=pos_obj.comments,\n            description=pos_obj.description,\n        )\n\n        orientation.create_spatial_series(\n            name=ori_obj.name,\n            timestamps=np.asarray(ori_obj.timestamps),\n            conversion=ori_obj.conversion,\n            data=np.asarray(ori_obj.data),\n            reference_frame=ori_obj.reference_frame,\n            comments=ori_obj.comments,\n            description=ori_obj.description,\n        )\n\n        velocity.create_timeseries(\n            name=vel_obj.name,\n            timestamps=np.asarray(vel_obj.timestamps),\n            conversion=vel_obj.conversion,\n            unit=vel_obj.unit,\n            data=np.asarray(vel_obj.data),\n            comments=vel_obj.comments,\n            description=vel_obj.description,\n        )\n\n        velocity.create_timeseries(\n            name=vid_frame_obj.name,\n            unit=vid_frame_obj.unit,\n            timestamps=np.asarray(vid_frame_obj.timestamps),\n            data=np.asarray(vid_frame_obj.data),\n            description=vid_frame_obj.description,\n            comments=vid_frame_obj.comments,\n        )\n\n        # Add to Analysis NWB file\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"orientation_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], orientation\n        )\n        key[\"position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], position\n        )\n        key[\"velocity_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], velocity\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n        from ..position_merge import PositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n        # TODO: The next line belongs in a merge table function\n        PositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(nwb_data[\"position\"].get_spatial_series().timestamps),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"position_x\",\n            \"position_y\",\n            \"orientation\",\n            \"velocity_x\",\n            \"velocity_y\",\n            \"speed\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"velocity\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(nwb_data[\"position\"].get_spatial_series().data),\n                    np.asarray(\n                        nwb_data[\"orientation\"].get_spatial_series().data\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"velocity\"].time_series[\"velocity\"].data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n\n    @classmethod\n    def evaluate_pose_estimation(cls, key):\n        likelihood_thresh = []\n        valid_fields = (\n            DLCSmoothInterpCohort.BodyPart().fetch().dtype.fields.keys()\n        )\n        centroid_key = {k: val for k, val in key.items() if k in valid_fields}\n        centroid_key[\"dlc_si_cohort_selection_name\"] = key[\n            \"dlc_si_cohort_centroid\"\n        ]\n        orientation_key = centroid_key.copy()\n        orientation_key[\"dlc_si_cohort_selection_name\"] = key[\n            \"dlc_si_cohort_orientation\"\n        ]\n        centroid_bodyparts, centroid_si_params = (\n            DLCSmoothInterpCohort.BodyPart &amp; centroid_key\n        ).fetch(\"bodypart\", \"dlc_si_params_name\")\n        orientation_bodyparts, orientation_si_params = (\n            DLCSmoothInterpCohort.BodyPart &amp; orientation_key\n        ).fetch(\"bodypart\", \"dlc_si_params_name\")\n        for param in np.unique(\n            np.concatenate((centroid_si_params, orientation_si_params))\n        ):\n            likelihood_thresh.append(\n                (\n                    DLCSmoothInterpParams() &amp; {\"dlc_si_params_name\": param}\n                ).fetch1(\"params\")[\"likelihood_thresh\"]\n            )\n\n        if len(np.unique(likelihood_thresh)) &gt; 1:\n            raise ValueError(\"more than one likelihood threshold used\")\n        like_thresh = likelihood_thresh[0]\n        bodyparts = np.unique([*centroid_bodyparts, *orientation_bodyparts])\n        fields = list(DLCPoseEstimation.BodyPart.fetch().dtype.fields.keys())\n        pose_estimation_key = {k: v for k, v in key.items() if k in fields}\n        pose_estimation_df = pd.concat(\n            {\n                bodypart: (\n                    DLCPoseEstimation.BodyPart()\n                    &amp; {**pose_estimation_key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in bodyparts.tolist()\n            },\n            axis=1,\n        )\n        df_filter = {\n            bodypart: pose_estimation_df[bodypart][\"likelihood\"] &lt; like_thresh\n            for bodypart in bodyparts\n            if bodypart in pose_estimation_df.columns\n        }\n        sub_thresh_percent_dict = {\n            bodypart: (\n                len(\n                    np.where(\n                        ~np.isnan(\n                            pose_estimation_df[bodypart][\"likelihood\"].where(\n                                df_filter[bodypart]\n                            )\n                        )\n                    )[0]\n                )\n                / len(pose_estimation_df)\n            )\n            * 100\n            for bodypart in bodyparts\n        }\n        return sub_thresh_percent_dict\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosVideo", "title": "<code>DLCPosVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCPosVideoSelection\n    ---\n    \"\"\"\n\n    def make(self, key):\n        from tqdm import tqdm as tqdm\n\n        params = (DLCPosVideoParams &amp; key).fetch1(\"params\")\n        if \"video_params\" not in params:\n            params[\"video_params\"] = {}\n        M_TO_CM = 100\n        interval_list_name = (\n            convert_epoch_interval_name_to_position_interval_name(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"epoch\": key[\"epoch\"],\n                },\n                populate_missing=False,\n            )\n        )\n        key[\"interval_list_name\"] = interval_list_name\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n        pose_estimation_key = {\n            \"nwb_file_name\": key[\"nwb_file_name\"],\n            \"epoch\": epoch,\n            \"dlc_model_name\": key[\"dlc_model_name\"],\n            \"dlc_model_params_name\": key[\"dlc_model_params_name\"],\n        }\n        pose_estimation_params, video_filename, output_dir = (\n            DLCPoseEstimationSelection() &amp; pose_estimation_key\n        ).fetch1(\n            \"pose_estimation_params\", \"video_path\", \"pose_estimation_output_dir\"\n        )\n        print(f\"video filename: {video_filename}\")\n        meters_per_pixel = (DLCPoseEstimation() &amp; pose_estimation_key).fetch1(\n            \"meters_per_pixel\"\n        )\n        crop = None\n        if \"cropping\" in pose_estimation_params:\n            crop = pose_estimation_params[\"cropping\"]\n        print(\"Loading position data...\")\n        position_info_df = (\n            DLCPosV1()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"epoch\": epoch,\n                \"dlc_si_cohort_centroid\": key[\"dlc_si_cohort_centroid\"],\n                \"dlc_centroid_params_name\": key[\"dlc_centroid_params_name\"],\n                \"dlc_si_cohort_orientation\": key[\"dlc_si_cohort_orientation\"],\n                \"dlc_orientation_params_name\": key[\n                    \"dlc_orientation_params_name\"\n                ],\n            }\n        ).fetch1_dataframe()\n        pose_estimation_df = pd.concat(\n            {\n                bodypart: (\n                    DLCPoseEstimation.BodyPart()\n                    &amp; {**pose_estimation_key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in (\n                    DLCSmoothInterpCohort.BodyPart &amp; pose_estimation_key\n                )\n                .fetch(\"bodypart\")\n                .tolist()\n            },\n            axis=1,\n        )\n        assert len(pose_estimation_df) == len(position_info_df), (\n            f\"length of pose_estimation_df: {len(pose_estimation_df)} \"\n            f\"does not match the length of position_info_df: {len(position_info_df)}.\"\n        )\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        if Path(output_dir).exists():\n            output_video_filename = (\n                f\"{Path(output_dir).as_posix()}/\"\n                f\"{nwb_base_filename}_{epoch:02d}_\"\n                f'{key[\"dlc_si_cohort_centroid\"]}_'\n                f'{key[\"dlc_centroid_params_name\"]}'\n                f'{key[\"dlc_orientation_params_name\"]}.mp4'\n            )\n        else:\n            output_video_filename = (\n                f\"{nwb_base_filename}_{epoch:02d}_\"\n                f'{key[\"dlc_si_cohort_centroid\"]}_'\n                f'{key[\"dlc_centroid_params_name\"]}'\n                f'{key[\"dlc_orientation_params_name\"]}.mp4'\n            )\n        idx = pd.IndexSlice\n        video_frame_inds = (\n            position_info_df[\"video_frame_ind\"].astype(int).to_numpy()\n        )\n        centroids = {\n            bodypart: pose_estimation_df.loc[\n                :, idx[bodypart, (\"x\", \"y\")]\n            ].to_numpy()\n            for bodypart in pose_estimation_df.columns.levels[0]\n        }\n        if params.get(\"incl_likelihood\", None):\n            likelihoods = {\n                bodypart: pose_estimation_df.loc[\n                    :, idx[bodypart, (\"likelihood\")]\n                ].to_numpy()\n                for bodypart in pose_estimation_df.columns.levels[0]\n            }\n        else:\n            likelihoods = None\n        position_mean = {\n            \"DLC\": np.asarray(position_info_df[[\"position_x\", \"position_y\"]])\n        }\n        orientation_mean = {\n            \"DLC\": np.asarray(position_info_df[[\"orientation\"]])\n        }\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = meters_per_pixel * M_TO_CM\n        percent_frames = params.get(\"percent_frames\", None)\n        frames = params.get(\"frames\", None)\n        if frames is not None:\n            frames_arr = np.arange(frames[0], frames[1])\n        else:\n            frames_arr = frames\n\n        print(\"Making video...\")\n        make_video(\n            video_filename=video_filename,\n            video_frame_inds=video_frame_inds,\n            position_mean=position_mean,\n            orientation_mean=orientation_mean,\n            centroids=centroids,\n            likelihoods=likelihoods,\n            position_time=position_time,\n            video_time=None,\n            processor=params.get(\"processor\", \"matplotlib\"),\n            frames=frames_arr,\n            percent_frames=percent_frames,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n            crop=crop,\n            **params[\"video_params\"],\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/", "title": "position_dlc_training.py", "text": ""}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTrainingParams", "title": "<code>DLCModelTrainingParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@schema\nclass DLCModelTrainingParams(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    # Parameters to specify a DLC model training instance\n    # For DLC \u2264 v2.0, include scorer_lecacy = True in params\n    dlc_training_params_name      : varchar(50) # descriptive name of parameter set\n    ---\n    params                        : longblob    # dictionary of all applicable parameters\n    \"\"\"\n\n    required_parameters = (\n        \"shuffle\",\n        \"trainingsetindex\",\n        \"net_type\",\n        \"gputouse\",\n    )\n    skipped_parameters = (\"project_path\", \"video_sets\")\n\n    @classmethod\n    def insert_new_params(cls, paramset_name: str, params: dict, **kwargs):\n        \"\"\"\n        Insert a new set of training parameters into dlc.TrainingParamSet.\n\n        Parameters\n        ----------\n        paramset_name : str\n            Description of parameter set to be inserted\n        params : dict\n            Dictionary including all settings to specify model training.\n            Must include shuffle &amp; trainingsetindex b/c not in config.yaml.\n            project_path and video_sets will be overwritten by config.yaml.\n            Note that trainingsetindex is 0-indexed\n        \"\"\"\n\n        for required_param in cls.required_parameters:\n            assert required_param in params, (\n                \"Missing required parameter: \" + required_param\n            )\n        for skipped_param in cls.skipped_parameters:\n            if skipped_param in params:\n                params.pop(skipped_param)\n\n        param_dict = {\n            \"dlc_training_params_name\": paramset_name,\n            \"params\": params,\n        }\n        param_query = cls &amp; {\n            \"dlc_training_params_name\": param_dict[\"dlc_training_params_name\"]\n        }\n        # If the specified param-set already exists\n        # Not sure we need this part, as much just a check if the name is the same\n        if param_query:\n            existing_paramset_name = param_query.fetch1(\n                \"dlc_training_params_name\"\n            )\n            if (\n                existing_paramset_name == paramset_name\n            ):  # If existing name same:\n                return print(\n                    f\"New param set not added\\n\"\n                    f\"A param set with name: {paramset_name} already exists\"\n                )\n        else:\n            cls.insert1(\n                param_dict, **kwargs\n            )  # if duplicate, will raise duplicate error\n            # if this will raise duplicate error, why is above check needed? @datajoint\n\n    @classmethod\n    def get_accepted_params(cls):\n        from deeplabcut import create_training_dataset, train_network\n\n        return list(\n            set(\n                [\n                    *list(inspect.signature(train_network).parameters),\n                    *list(\n                        inspect.signature(create_training_dataset).parameters\n                    ),\n                ]\n            )\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTrainingParams.insert_new_params", "title": "<code>insert_new_params(paramset_name, params, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert a new set of training parameters into dlc.TrainingParamSet.</p> <p>Parameters:</p> Name Type Description Default <code>paramset_name</code> <code>str</code> <p>Description of parameter set to be inserted</p> required <code>params</code> <code>dict</code> <p>Dictionary including all settings to specify model training. Must include shuffle &amp; trainingsetindex b/c not in config.yaml. project_path and video_sets will be overwritten by config.yaml. Note that trainingsetindex is 0-indexed</p> required Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@classmethod\ndef insert_new_params(cls, paramset_name: str, params: dict, **kwargs):\n    \"\"\"\n    Insert a new set of training parameters into dlc.TrainingParamSet.\n\n    Parameters\n    ----------\n    paramset_name : str\n        Description of parameter set to be inserted\n    params : dict\n        Dictionary including all settings to specify model training.\n        Must include shuffle &amp; trainingsetindex b/c not in config.yaml.\n        project_path and video_sets will be overwritten by config.yaml.\n        Note that trainingsetindex is 0-indexed\n    \"\"\"\n\n    for required_param in cls.required_parameters:\n        assert required_param in params, (\n            \"Missing required parameter: \" + required_param\n        )\n    for skipped_param in cls.skipped_parameters:\n        if skipped_param in params:\n            params.pop(skipped_param)\n\n    param_dict = {\n        \"dlc_training_params_name\": paramset_name,\n        \"params\": params,\n    }\n    param_query = cls &amp; {\n        \"dlc_training_params_name\": param_dict[\"dlc_training_params_name\"]\n    }\n    # If the specified param-set already exists\n    # Not sure we need this part, as much just a check if the name is the same\n    if param_query:\n        existing_paramset_name = param_query.fetch1(\n            \"dlc_training_params_name\"\n        )\n        if (\n            existing_paramset_name == paramset_name\n        ):  # If existing name same:\n            return print(\n                f\"New param set not added\\n\"\n                f\"A param set with name: {paramset_name} already exists\"\n            )\n    else:\n        cls.insert1(\n            param_dict, **kwargs\n        )  # if duplicate, will raise duplicate error\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTraining", "title": "<code>DLCModelTraining</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@schema\nclass DLCModelTraining(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCModelTrainingSelection\n    ---\n    project_path         : varchar(255) # Path to project directory\n    latest_snapshot: int unsigned # latest exact snapshot index (i.e., never -1)\n    config_template: longblob     # stored full config file\n    \"\"\"\n\n    # To continue from previous training snapshot, devs suggest editing pose_cfg.yml\n    # https://github.com/DeepLabCut/DeepLabCut/issues/70\n\n    def make(self, key):\n        \"\"\"Launch training for each entry in DLCModelTrainingSelection via `.populate()`.\"\"\"\n        model_prefix = (DLCModelTrainingSelection &amp; key).fetch1(\"model_prefix\")\n        from deeplabcut import create_training_dataset, train_network\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        from . import dlc_reader\n\n        try:\n            from deeplabcut.utils.auxiliaryfunctions import get_model_folder\n        except ImportError:\n            from deeplabcut.utils.auxiliaryfunctions import (\n                GetModelFolder as get_model_folder,\n            )\n        config_path, project_name = (DLCProject() &amp; key).fetch1(\n            \"config_path\", \"project_name\"\n        )\n        with OutputLogger(\n            name=\"DLC_project_{project_name}_training\",\n            path=f\"{os.path.dirname(config_path)}/log.log\",\n            print_console=True,\n        ) as logger:\n            dlc_config = read_config(config_path)\n            project_path = dlc_config[\"project_path\"]\n            key[\"project_path\"] = project_path\n            # ---- Build and save DLC configuration (yaml) file ----\n            _, dlc_config = dlc_reader.read_yaml(project_path)\n            if not dlc_config:\n                dlc_config = read_config(config_path)\n            dlc_config.update((DLCModelTrainingParams &amp; key).fetch1(\"params\"))\n            dlc_config.update(\n                {\n                    \"project_path\": Path(project_path).as_posix(),\n                    \"modelprefix\": model_prefix,\n                    \"train_fraction\": dlc_config[\"TrainingFraction\"][\n                        int(dlc_config[\"trainingsetindex\"])\n                    ],\n                    \"training_filelist_datajoint\": [  # don't overwrite origin video_sets\n                        Path(fp).as_posix()\n                        for fp in (DLCProject.File &amp; key).fetch(\"file_path\")\n                    ],\n                }\n            )\n            # Write dlc config file to base project folder\n            # TODO: need to make sure this will work\n            dlc_cfg_filepath = dlc_reader.save_yaml(project_path, dlc_config)\n            # ---- create training dataset ----\n            training_dataset_input_args = list(\n                inspect.signature(create_training_dataset).parameters\n            )\n            training_dataset_kwargs = {\n                k: v\n                for k, v in dlc_config.items()\n                if k in training_dataset_input_args\n            }\n            logger.logger.info(\"creating training dataset\")\n            create_training_dataset(dlc_cfg_filepath, **training_dataset_kwargs)\n            # ---- Trigger DLC model training job ----\n            train_network_input_args = list(\n                inspect.signature(train_network).parameters\n            )\n            train_network_kwargs = {\n                k: v\n                for k, v in dlc_config.items()\n                if k in train_network_input_args\n            }\n            for k in [\"shuffle\", \"trainingsetindex\", \"maxiters\"]:\n                if k in train_network_kwargs:\n                    train_network_kwargs[k] = int(train_network_kwargs[k])\n            try:\n                train_network(dlc_cfg_filepath, **train_network_kwargs)\n            except (\n                KeyboardInterrupt\n            ):  # Instructions indicate to train until interrupt\n                logger.logger.info(\n                    \"DLC training stopped via Keyboard Interrupt\"\n                )\n\n            snapshots = list(\n                (\n                    project_path\n                    / get_model_folder(\n                        trainFraction=dlc_config[\"train_fraction\"],\n                        shuffle=dlc_config[\"shuffle\"],\n                        cfg=dlc_config,\n                        modelprefix=dlc_config[\"modelprefix\"],\n                    )\n                    / \"train\"\n                ).glob(\"*index*\")\n            )\n            max_modified_time = 0\n            # DLC goes by snapshot magnitude when judging 'latest' for evaluation\n            # Here, we mean most recently generated\n            for snapshot in snapshots:\n                modified_time = os.path.getmtime(snapshot)\n                if modified_time &gt; max_modified_time:\n                    latest_snapshot = int(snapshot.stem[9:])\n                    max_modified_time = modified_time\n\n            self.insert1(\n                {\n                    **key,\n                    \"latest_snapshot\": latest_snapshot,\n                    \"config_template\": dlc_config,\n                }\n            )\n            from .position_dlc_model import DLCModelSource\n\n            dlc_model_name = f\"{key['project_name']}_{key['dlc_training_params_name']}_{key['training_id']:02d}\"\n            DLCModelSource.insert_entry(\n                dlc_model_name=dlc_model_name,\n                project_name=key[\"project_name\"],\n                source=\"FromUpstream\",\n                key=key,\n                skip_duplicates=True,\n            )\n        print(\n            f\"Inserted {dlc_model_name} from {key['project_name']} into DLCModelSource\"\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTraining.make", "title": "<code>make(key)</code>", "text": "<p>Launch training for each entry in DLCModelTrainingSelection via <code>.populate()</code>.</p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>def make(self, key):\n    \"\"\"Launch training for each entry in DLCModelTrainingSelection via `.populate()`.\"\"\"\n    model_prefix = (DLCModelTrainingSelection &amp; key).fetch1(\"model_prefix\")\n    from deeplabcut import create_training_dataset, train_network\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    from . import dlc_reader\n\n    try:\n        from deeplabcut.utils.auxiliaryfunctions import get_model_folder\n    except ImportError:\n        from deeplabcut.utils.auxiliaryfunctions import (\n            GetModelFolder as get_model_folder,\n        )\n    config_path, project_name = (DLCProject() &amp; key).fetch1(\n        \"config_path\", \"project_name\"\n    )\n    with OutputLogger(\n        name=\"DLC_project_{project_name}_training\",\n        path=f\"{os.path.dirname(config_path)}/log.log\",\n        print_console=True,\n    ) as logger:\n        dlc_config = read_config(config_path)\n        project_path = dlc_config[\"project_path\"]\n        key[\"project_path\"] = project_path\n        # ---- Build and save DLC configuration (yaml) file ----\n        _, dlc_config = dlc_reader.read_yaml(project_path)\n        if not dlc_config:\n            dlc_config = read_config(config_path)\n        dlc_config.update((DLCModelTrainingParams &amp; key).fetch1(\"params\"))\n        dlc_config.update(\n            {\n                \"project_path\": Path(project_path).as_posix(),\n                \"modelprefix\": model_prefix,\n                \"train_fraction\": dlc_config[\"TrainingFraction\"][\n                    int(dlc_config[\"trainingsetindex\"])\n                ],\n                \"training_filelist_datajoint\": [  # don't overwrite origin video_sets\n                    Path(fp).as_posix()\n                    for fp in (DLCProject.File &amp; key).fetch(\"file_path\")\n                ],\n            }\n        )\n        # Write dlc config file to base project folder\n        # TODO: need to make sure this will work\n        dlc_cfg_filepath = dlc_reader.save_yaml(project_path, dlc_config)\n        # ---- create training dataset ----\n        training_dataset_input_args = list(\n            inspect.signature(create_training_dataset).parameters\n        )\n        training_dataset_kwargs = {\n            k: v\n            for k, v in dlc_config.items()\n            if k in training_dataset_input_args\n        }\n        logger.logger.info(\"creating training dataset\")\n        create_training_dataset(dlc_cfg_filepath, **training_dataset_kwargs)\n        # ---- Trigger DLC model training job ----\n        train_network_input_args = list(\n            inspect.signature(train_network).parameters\n        )\n        train_network_kwargs = {\n            k: v\n            for k, v in dlc_config.items()\n            if k in train_network_input_args\n        }\n        for k in [\"shuffle\", \"trainingsetindex\", \"maxiters\"]:\n            if k in train_network_kwargs:\n                train_network_kwargs[k] = int(train_network_kwargs[k])\n        try:\n            train_network(dlc_cfg_filepath, **train_network_kwargs)\n        except (\n            KeyboardInterrupt\n        ):  # Instructions indicate to train until interrupt\n            logger.logger.info(\n                \"DLC training stopped via Keyboard Interrupt\"\n            )\n\n        snapshots = list(\n            (\n                project_path\n                / get_model_folder(\n                    trainFraction=dlc_config[\"train_fraction\"],\n                    shuffle=dlc_config[\"shuffle\"],\n                    cfg=dlc_config,\n                    modelprefix=dlc_config[\"modelprefix\"],\n                )\n                / \"train\"\n            ).glob(\"*index*\")\n        )\n        max_modified_time = 0\n        # DLC goes by snapshot magnitude when judging 'latest' for evaluation\n        # Here, we mean most recently generated\n        for snapshot in snapshots:\n            modified_time = os.path.getmtime(snapshot)\n            if modified_time &gt; max_modified_time:\n                latest_snapshot = int(snapshot.stem[9:])\n                max_modified_time = modified_time\n\n        self.insert1(\n            {\n                **key,\n                \"latest_snapshot\": latest_snapshot,\n                \"config_template\": dlc_config,\n            }\n        )\n        from .position_dlc_model import DLCModelSource\n\n        dlc_model_name = f\"{key['project_name']}_{key['dlc_training_params_name']}_{key['training_id']:02d}\"\n        DLCModelSource.insert_entry(\n            dlc_model_name=dlc_model_name,\n            project_name=key[\"project_name\"],\n            source=\"FromUpstream\",\n            key=key,\n            skip_duplicates=True,\n        )\n    print(\n        f\"Inserted {dlc_model_name} from {key['project_name']} into DLCModelSource\"\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/", "title": "position_trodes_position.py", "text": ""}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosParams", "title": "<code>TrodesPosParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for calculating the position (centroid, velocity, orientation)</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for calculating the position (centroid, velocity, orientation)\n    \"\"\"\n\n    definition = \"\"\"\n    trodes_pos_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @property\n    def default_pk(self):\n        return {\"trodes_pos_params_name\": \"default\"}\n\n    @property\n    def default_params(self):\n        return {\n            \"max_LED_separation\": 9.0,\n            \"max_plausible_speed\": 300.0,\n            \"position_smoothing_duration\": 0.125,\n            \"speed_smoothing_std_dev\": 0.100,\n            \"orient_smoothing_std_dev\": 0.001,\n            \"led1_is_front\": 1,\n            \"is_upsampled\": 0,\n            \"upsampling_sampling_rate\": None,\n            \"upsampling_interpolation_method\": \"linear\",\n        }\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        \"\"\"\n        Insert default parameter set for position determination\n        \"\"\"\n        cls.insert1(\n            {**cls().default_pk, \"params\": cls().default_params},\n            skip_duplicates=True,\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; cls().default_pk\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            return (cls &amp; cls().default_pk).fetch1()\n\n        return query.fetch1()\n\n    @classmethod\n    def get_accepted_params(cls):\n        return [k for k in cls().default_params.keys()]\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosParams.insert_default", "title": "<code>insert_default(**kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert default parameter set for position determination</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@classmethod\ndef insert_default(cls, **kwargs):\n    \"\"\"\n    Insert default parameter set for position determination\n    \"\"\"\n    cls.insert1(\n        {**cls().default_pk, \"params\": cls().default_params},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosSelection", "title": "<code>TrodesPosSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to pair an interval with position data and position determination parameters</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to pair an interval with position data\n    and position determination parameters\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; RawPosition\n    -&gt; TrodesPosParams\n    \"\"\"\n\n    @classmethod\n    def insert_with_default(\n        cls,\n        key: dict,\n        skip_duplicates: bool = False,\n        edit_defaults: dict = {},\n        edit_name: str = None,\n    ) -&gt; None:\n        \"\"\"Insert key with default parameters.\n\n        To change defaults, supply a dict as edit_defaults with a name for\n        the new paramset as edit_name.\n\n        Parameters\n        ----------\n        key: Union[dict, str]\n            Restriction uniquely identifying entr(y/ies) in RawPosition.\n        skip_duplicates: bool, optional\n            Skip duplicate entries.\n        edit_defaults: dict, optional\n            Dictionary of overrides to default parameters.\n        edit_name: str, optional\n            If edit_defauts is passed, the name of the new entry\n\n        Raises\n        ------\n        ValueError\n            Key does not identify any entries in RawPosition.\n        \"\"\"\n        query = RawPosition &amp; key\n        if not query:\n            raise ValueError(f\"Found no entries found for {key}\")\n\n        param_pk, param_name = list(TrodesPosParams().default_pk.items())[0]\n\n        if bool(edit_defaults) ^ bool(edit_name):  # XOR: only one of them\n            raise ValueError(\"Must specify both edit_defauts and edit_name\")\n\n        elif edit_defaults and edit_name:\n            TrodesPosParams.insert1(\n                {\n                    param_pk: edit_name,\n                    \"params\": {\n                        **TrodesPosParams().default_params,\n                        **edit_defaults,\n                    },\n                },\n                skip_duplicates=skip_duplicates,\n            )\n\n        cls.insert(\n            [\n                {**k, param_pk: edit_name or param_name}\n                for k in query.fetch(\"KEY\", as_dict=True)\n            ],\n            skip_duplicates=skip_duplicates,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosSelection.insert_with_default", "title": "<code>insert_with_default(key, skip_duplicates=False, edit_defaults={}, edit_name=None)</code>  <code>classmethod</code>", "text": "<p>Insert key with default parameters.</p> <p>To change defaults, supply a dict as edit_defaults with a name for the new paramset as edit_name.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Restriction uniquely identifying entr(y/ies) in RawPosition.</p> required <code>skip_duplicates</code> <code>bool</code> <p>Skip duplicate entries.</p> <code>False</code> <code>edit_defaults</code> <code>dict</code> <p>Dictionary of overrides to default parameters.</p> <code>{}</code> <code>edit_name</code> <code>str</code> <p>If edit_defauts is passed, the name of the new entry</p> <code>None</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>Key does not identify any entries in RawPosition.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@classmethod\ndef insert_with_default(\n    cls,\n    key: dict,\n    skip_duplicates: bool = False,\n    edit_defaults: dict = {},\n    edit_name: str = None,\n) -&gt; None:\n    \"\"\"Insert key with default parameters.\n\n    To change defaults, supply a dict as edit_defaults with a name for\n    the new paramset as edit_name.\n\n    Parameters\n    ----------\n    key: Union[dict, str]\n        Restriction uniquely identifying entr(y/ies) in RawPosition.\n    skip_duplicates: bool, optional\n        Skip duplicate entries.\n    edit_defaults: dict, optional\n        Dictionary of overrides to default parameters.\n    edit_name: str, optional\n        If edit_defauts is passed, the name of the new entry\n\n    Raises\n    ------\n    ValueError\n        Key does not identify any entries in RawPosition.\n    \"\"\"\n    query = RawPosition &amp; key\n    if not query:\n        raise ValueError(f\"Found no entries found for {key}\")\n\n    param_pk, param_name = list(TrodesPosParams().default_pk.items())[0]\n\n    if bool(edit_defaults) ^ bool(edit_name):  # XOR: only one of them\n        raise ValueError(\"Must specify both edit_defauts and edit_name\")\n\n    elif edit_defaults and edit_name:\n        TrodesPosParams.insert1(\n            {\n                param_pk: edit_name,\n                \"params\": {\n                    **TrodesPosParams().default_params,\n                    **edit_defaults,\n                },\n            },\n            skip_duplicates=skip_duplicates,\n        )\n\n    cls.insert(\n        [\n            {**k, param_pk: edit_name or param_name}\n            for k in query.fetch(\"KEY\", as_dict=True)\n        ],\n        skip_duplicates=skip_duplicates,\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosV1", "title": "<code>TrodesPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to calculate the position based on Trodes tracking</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosV1(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to calculate the position based on Trodes tracking\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; TrodesPosSelection\n    ---\n    -&gt; AnalysisNwbfile\n    position_object_id : varchar(80)\n    orientation_object_id : varchar(80)\n    velocity_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing position for: {key}\")\n        orig_key = copy.deepcopy(key)\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        raw_position = RawPosition.PosObject &amp; key\n        spatial_series = raw_position.fetch_nwb()[0][\"raw_position\"]\n        spatial_df = raw_position.fetch1_dataframe()\n\n        position_info_parameters = (TrodesPosParams() &amp; key).fetch1(\"params\")\n        position_info = self.calculate_position_info(\n            spatial_df=spatial_df,\n            meters_to_pixels=spatial_series.conversion,\n            **position_info_parameters,\n        )\n\n        key.update(\n            dict(\n                analysis_file_name=analysis_file_name,\n                **self.generate_pos_components(\n                    spatial_series=spatial_series,\n                    position_info=position_info,\n                    analysis_fname=analysis_file_name,\n                    prefix=\"\",\n                    add_frame_ind=True,\n                    video_frame_ind=getattr(\n                        spatial_df, \"video_frame_ind\", None\n                    ),\n                ),\n            )\n        )\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        self.insert1(key)\n\n        from ..position_merge import PositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n\n        # TODO: The next line belongs in a merge table function\n        PositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    @staticmethod\n    def generate_pos_components(*args, **kwargs):\n        return IntervalPositionInfo().generate_pos_components(*args, **kwargs)\n\n    @staticmethod\n    def calculate_position_info(*args, **kwargs):\n        \"\"\"Calculate position info from 2D spatial series.\"\"\"\n        return IntervalPositionInfo().calculate_position_info(*args, **kwargs)\n\n    def fetch1_dataframe(self, add_frame_ind=True):\n        pos_params = self.fetch1(\"trodes_pos_params_name\")\n        if (\n            add_frame_ind\n            and (\n                TrodesPosParams &amp; {\"trodes_pos_params_name\": pos_params}\n            ).fetch1(\"params\")[\"is_upsampled\"]\n        ):\n            logger.warning(\n                \"Upsampled position data, frame indices are invalid. \"\n                + \"Setting add_frame_ind=False\"\n            )\n            add_frame_ind = False\n        return IntervalPositionInfo._data_to_df(\n            self.fetch_nwb()[0], prefix=\"\", add_frame_ind=add_frame_ind\n        )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosV1.calculate_position_info", "title": "<code>calculate_position_info(*args, **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Calculate position info from 2D spatial series.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@staticmethod\ndef calculate_position_info(*args, **kwargs):\n    \"\"\"Calculate position info from 2D spatial series.\"\"\"\n    return IntervalPositionInfo().calculate_position_info(*args, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosVideo", "title": "<code>TrodesPosVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; TrodesPosV1\n    ---\n    has_video : bool\n    \"\"\"\n\n    def make(self, key):\n        M_TO_CM = 100\n\n        logger.info(\"Loading position data...\")\n        raw_position_df = (\n            RawPosition.PosObject\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch1_dataframe()\n        position_info_df = (TrodesPosV1() &amp; key).fetch1_dataframe()\n\n        logger.info(\"Loading video data...\")\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n\n        (\n            video_path,\n            video_filename,\n            meters_per_pixel,\n            video_time,\n        ) = get_video_path(\n            {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        )\n\n        if not video_path:\n            self.insert1(dict(**key, has_video=False))\n            return\n\n        video_dir = os.path.dirname(video_path) + \"/\"\n        video_path = check_videofile(\n            video_path=video_dir, video_filename=video_filename\n        )[0].as_posix()\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        current_dir = Path(os.getcwd())\n        output_video_filename = (\n            f\"{current_dir.as_posix()}/{nwb_base_filename}_\"\n            f\"{epoch:02d}_{key['trodes_pos_params_name']}.mp4\"\n        )\n        centroids = {\n            \"red\": np.asarray(raw_position_df[[\"xloc\", \"yloc\"]]),\n            \"green\": np.asarray(raw_position_df[[\"xloc2\", \"yloc2\"]]),\n        }\n        position_mean = np.asarray(\n            position_info_df[[\"position_x\", \"position_y\"]]\n        )\n        orientation_mean = np.asarray(position_info_df[[\"orientation\"]])\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = meters_per_pixel * M_TO_CM\n\n        logger.info(\"Making video...\")\n        self.make_video(\n            video_path,\n            centroids,\n            position_mean,\n            orientation_mean,\n            video_time,\n            position_time,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n        )\n        self.insert1(dict(**key, has_video=True))\n\n    @staticmethod\n    def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n        \"\"\"Converts from cm to pixels and flips the y-axis.\n        Parameters\n        ----------\n        data : ndarray, shape (n_time, 2)\n        frame_size : array_like, shape (2,)\n        cm_to_pixels : float\n\n        Returns\n        -------\n        converted_data : ndarray, shape (n_time, 2)\n        \"\"\"\n        return data / cm_to_pixels\n\n    @staticmethod\n    def fill_nan(variable, video_time, variable_time):\n        video_ind = np.digitize(variable_time, video_time[1:])\n\n        n_video_time = len(video_time)\n        try:\n            n_variable_dims = variable.shape[1]\n            filled_variable = np.full((n_video_time, n_variable_dims), np.nan)\n        except IndexError:\n            filled_variable = np.full((n_video_time,), np.nan)\n        filled_variable[video_ind] = variable\n\n        return filled_variable\n\n    def make_video(\n        self,\n        video_filename,\n        centroids,\n        position_mean,\n        orientation_mean,\n        video_time,\n        position_time,\n        output_video_filename=\"output.mp4\",\n        cm_to_pixels=1.0,\n        disable_progressbar=False,\n        arrow_radius=15,\n        circle_radius=8,\n    ):\n        import cv2\n\n        RGB_PINK = (234, 82, 111)\n        RGB_YELLOW = (253, 231, 76)\n        RGB_WHITE = (255, 255, 255)\n\n        video = cv2.VideoCapture(video_filename)\n        fourcc = cv2.VideoWriter_fourcc(*\"mp4v\")\n        frame_size = (int(video.get(3)), int(video.get(4)))\n        frame_rate = video.get(5)\n        n_frames = int(orientation_mean.shape[0])\n        logger.info(f\"video filepath: {output_video_filename}\")\n        out = cv2.VideoWriter(\n            output_video_filename, fourcc, frame_rate, frame_size, True\n        )\n\n        centroids = {\n            color: self.fill_nan(data, video_time, position_time)\n            for color, data in centroids.items()\n        }\n        position_mean = self.fill_nan(position_mean, video_time, position_time)\n        orientation_mean = self.fill_nan(\n            orientation_mean, video_time, position_time\n        )\n\n        for time_ind in tqdm(\n            range(n_frames - 1), desc=\"frames\", disable=disable_progressbar\n        ):\n            is_grabbed, frame = video.read()\n            if is_grabbed:\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n\n                red_centroid = centroids[\"red\"][time_ind]\n                green_centroid = centroids[\"green\"][time_ind]\n\n                position = position_mean[time_ind]\n                position = self.convert_to_pixels(\n                    position, frame_size, cm_to_pixels\n                )\n                orientation = orientation_mean[time_ind]\n\n                if np.all(~np.isnan(red_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(red_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_YELLOW,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(green_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(green_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_PINK,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(position)) &amp; np.all(~np.isnan(orientation)):\n                    arrow_tip = (\n                        int(position[0] + arrow_radius * np.cos(orientation)),\n                        int(position[1] + arrow_radius * np.sin(orientation)),\n                    )\n                    cv2.arrowedLine(\n                        img=frame,\n                        pt1=tuple(position.astype(int)),\n                        pt2=arrow_tip,\n                        color=RGB_WHITE,\n                        thickness=4,\n                        line_type=8,\n                        shift=cv2.CV_8U,\n                        tipLength=0.25,\n                    )\n\n                if np.all(~np.isnan(position)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(position.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_WHITE,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)\n                out.write(frame)\n            else:\n                break\n\n        video.release()\n        out.release()\n        cv2.destroyAllWindows()\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosVideo.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>  <code>staticmethod</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@staticmethod\ndef convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/ripple/v1/ripple/", "title": "ripple.py", "text": ""}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleLFPSelection", "title": "<code>RippleLFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleLFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; LFPBandV1\n     group_name = 'CA1' : varchar(80)\n     \"\"\"\n\n    class RippleLFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; RippleLFPSelection\n        -&gt; LFPBandSelection.LFPBandElectrode\n        \"\"\"\n\n    @staticmethod\n    def validate_key(key):\n        filter_name = (LFPBandV1 &amp; key).fetch1(\"filter_name\")\n        if \"ripple\" not in filter_name.lower():\n            raise ValueError(\"Please use a ripple filter\")\n\n    @staticmethod\n    def set_lfp_electrodes(\n        key,\n        electrode_list=None,\n        group_name=\"CA1\",\n        **kwargs,\n    ):\n        \"\"\"Removes all electrodes for the specified nwb file and then\n        adds back the electrodes in the list\n\n        Parameters\n        ----------\n        key : dict\n            dictionary corresponding to the LFPBand entry to use for\n            ripple detection\n        electrode_list : list\n            list of electrodes from LFPBandSelection.LFPBandElectrode\n            to be used as the ripple LFP during detection\n        group_name : str, optional\n            description of the electrode group, by default \"CA1\"\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = (\n                (LFPBandSelection.LFPBandElectrode &amp; key)\n                .fetch(\"electrode_id\")\n                .tolist()\n            )\n        electrode_list.sort()\n        try:\n            electrode_keys = (\n                pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n                .set_index(\"electrode_id\")\n                .loc[np.asarray(electrode_list)]\n                .reset_index()\n                .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n            )\n        except KeyError as err:\n            logger.debug(err)\n            raise KeyError(\n                \"Attempting to use electrode_ids that aren't in the associated\"\n                \" LFPBand filtered dataset.\"\n            ) from err\n        electrode_keys[\"group_name\"] = group_name\n        electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n        RippleLFPSelection.validate_key(key)\n        RippleLFPSelection().insert1(\n            {**key, \"group_name\": group_name},\n            skip_duplicates=True,\n            **kwargs,\n        )\n        RippleLFPSelection().RippleLFPElectrode.insert(\n            electrode_keys.to_dict(orient=\"records\"),\n            replace=True,\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleLFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(key, electrode_list=None, group_name='CA1', **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary corresponding to the LFPBand entry to use for ripple detection</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes from LFPBandSelection.LFPBandElectrode to be used as the ripple LFP during detection</p> <code>None</code> <code>group_name</code> <code>str</code> <p>description of the electrode group, by default \"CA1\"</p> <code>'CA1'</code> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@staticmethod\ndef set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name=\"CA1\",\n    **kwargs,\n):\n    \"\"\"Removes all electrodes for the specified nwb file and then\n    adds back the electrodes in the list\n\n    Parameters\n    ----------\n    key : dict\n        dictionary corresponding to the LFPBand entry to use for\n        ripple detection\n    electrode_list : list\n        list of electrodes from LFPBandSelection.LFPBandElectrode\n        to be used as the ripple LFP during detection\n    group_name : str, optional\n        description of the electrode group, by default \"CA1\"\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = (\n            (LFPBandSelection.LFPBandElectrode &amp; key)\n            .fetch(\"electrode_id\")\n            .tolist()\n        )\n    electrode_list.sort()\n    try:\n        electrode_keys = (\n            pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n            .set_index(\"electrode_id\")\n            .loc[np.asarray(electrode_list)]\n            .reset_index()\n            .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n        )\n    except KeyError as err:\n        logger.debug(err)\n        raise KeyError(\n            \"Attempting to use electrode_ids that aren't in the associated\"\n            \" LFPBand filtered dataset.\"\n        ) from err\n    electrode_keys[\"group_name\"] = group_name\n    electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n    RippleLFPSelection.validate_key(key)\n    RippleLFPSelection().insert1(\n        {**key, \"group_name\": group_name},\n        skip_duplicates=True,\n        **kwargs,\n    )\n    RippleLFPSelection().RippleLFPElectrode.insert(\n        electrode_keys.to_dict(orient=\"records\"),\n        replace=True,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleParameters", "title": "<code>RippleParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleParameters(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    ripple_param_name : varchar(80) # a name for this set of parameters\n    ----\n    ripple_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\"\"\"\n        default_dict = {\n            \"speed_name\": \"head_speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n        default_dict_trodes = {\n            \"speed_name\": \"speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\n                \"ripple_param_name\": \"default_trodes\",\n                \"ripple_param_dict\": default_dict_trodes,\n            },\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\"\"\"\n    default_dict = {\n        \"speed_name\": \"head_speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n    default_dict_trodes = {\n        \"speed_name\": \"speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\n            \"ripple_param_name\": \"default_trodes\",\n            \"ripple_param_dict\": default_dict_trodes,\n        },\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleTimesV1", "title": "<code>RippleTimesV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleTimesV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; RippleLFPSelection\n    -&gt; RippleParameters\n    -&gt; PositionOutput.proj(pos_merge_id='merge_id')\n    ---\n    -&gt; AnalysisNwbfile\n    ripple_times_object_id : varchar(40)\n     \"\"\"\n\n    def make(self, key):\n        nwb_file_name = (LFPBandV1 &amp; key).fetch1(\"nwb_file_name\")\n\n        logger.info(f\"Computing ripple times for: {key}\")\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        ripple_detection_algorithm = ripple_params[\"ripple_detection_algorithm\"]\n        ripple_detection_params = ripple_params[\"ripple_detection_params\"]\n\n        (\n            speed,\n            interval_ripple_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n        ripple_times = RIPPLE_DETECTION_ALGORITHMS[ripple_detection_algorithm](\n            time=np.asarray(interval_ripple_lfps.index),\n            filtered_lfps=np.asarray(interval_ripple_lfps),\n            speed=np.asarray(speed),\n            sampling_frequency=sampling_frequency,\n            **ripple_detection_params,\n        )\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(nwb_file_name)\n        key[\"ripple_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=ripple_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=nwb_file_name,\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"ripple_times\"] for data in self.fetch_nwb()]\n\n    @staticmethod\n    def get_ripple_lfps_and_position_info(key):\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        speed_name = ripple_params[\"speed_name\"]\n\n        electrode_keys = (RippleLFPSelection.RippleLFPElectrode() &amp; key).fetch(\n            \"electrode_id\"\n        )\n\n        # warn/validate that there is only one wire per electrode\n        ripple_lfp_nwb = (LFPBandV1 &amp; key).fetch_nwb()[0]\n        ripple_lfp_electrodes = ripple_lfp_nwb[\"lfp_band\"].electrodes.data[:]\n        elec_mask = np.zeros_like(ripple_lfp_electrodes, dtype=bool)\n        valid_elecs = [\n            elec for elec in electrode_keys if elec in ripple_lfp_electrodes\n        ]\n        lfp_indexed_elec_ids = get_electrode_indices(\n            ripple_lfp_nwb[\"lfp_band\"], valid_elecs\n        )\n        elec_mask[lfp_indexed_elec_ids] = True\n        ripple_lfp = pd.DataFrame(\n            ripple_lfp_nwb[\"lfp_band\"].data,\n            index=pd.Index(ripple_lfp_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n        ).loc[:, elec_mask]\n        sampling_frequency = ripple_lfp_nwb[\"lfp_band_sampling_rate\"]\n\n        position_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"target_interval_list_name\"],\n            }\n        ).fetch1(\"valid_times\")\n        position_info = (\n            PositionOutput() &amp; {\"merge_id\": key[\"pos_merge_id\"]}\n        ).fetch1_dataframe()\n\n        # restrict valid times to position time\n        valid_times_interval = np.array(\n            [position_info.index[0], position_info.index[-1]]\n        )\n        position_valid_times = interval_list_intersect(\n            position_valid_times, valid_times_interval\n        )\n\n        position_info = pd.concat(\n            [\n                position_info.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=0,\n        )\n        interval_ripple_lfps = pd.concat(\n            [\n                ripple_lfp.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=0,\n        )\n\n        position_info = interpolate_to_new_time(\n            position_info, interval_ripple_lfps.index\n        )\n\n        return (\n            position_info[speed_name],\n            interval_ripple_lfps,\n            sampling_frequency,\n        )\n\n    @staticmethod\n    def get_Kay_ripple_consensus_trace(\n        ripple_filtered_lfps, sampling_frequency, smoothing_sigma=0.004\n    ):\n        ripple_consensus_trace = np.full_like(ripple_filtered_lfps, np.nan)\n        not_null = np.all(pd.notnull(ripple_filtered_lfps), axis=1)\n\n        ripple_consensus_trace[not_null] = get_envelope(\n            np.asarray(ripple_filtered_lfps)[not_null]\n        )\n        ripple_consensus_trace = np.sum(ripple_consensus_trace**2, axis=1)\n        ripple_consensus_trace[not_null] = gaussian_smooth(\n            ripple_consensus_trace[not_null],\n            smoothing_sigma,\n            sampling_frequency,\n        )\n        return pd.DataFrame(\n            np.sqrt(ripple_consensus_trace), index=ripple_filtered_lfps.index\n        )\n\n    @staticmethod\n    def plot_ripple_consensus_trace(\n        ripple_consensus_trace,\n        ripple_times,\n        ripple_label=1,\n        offset=0.100,\n        relative=True,\n        ax=None,\n    ):\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n\n        start_offset = ripple_start if relative else 0\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, 1))\n        ax.plot(\n            ripple_consensus_trace.loc[time_slice].index - start_offset,\n            ripple_consensus_trace.loc[time_slice],\n        )\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_xlabel(\"Time [s]\")\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n\n    @staticmethod\n    def plot_ripple(\n        lfps, ripple_times, ripple_label=1, offset=0.100, relative=True, ax=None\n    ):\n        lfp_labels = lfps.columns\n        n_lfps = len(lfp_labels)\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, n_lfps * 0.20))\n\n        start_offset = ripple_start if relative else 0\n\n        for lfp_ind, lfp_label in enumerate(lfp_labels):\n            lfp = lfps.loc[time_slice, lfp_label]\n            ax.plot(\n                lfp.index - start_offset,\n                lfp_ind + (lfp - lfp.mean()) / (lfp.max() - lfp.min()),\n                color=\"black\",\n            )\n\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_ylim((-1, n_lfps))\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n        ax.set_ylabel(\"LFPs\")\n        ax.set_xlabel(\"Time [s]\")\n\n    def create_figurl(\n        self,\n        zscore_ripple=False,\n        ripple_times_color=\"red\",\n        consensus_color=\"black\",\n        speed_color=\"black\",\n        view_height=800,\n        use_ripple_filtered_lfps=False,\n        lfp_offset=1,\n        lfp_channel_ind=None,\n    ):\n        ripple_times = self.fetch1_dataframe()\n\n        def _add_ripple_times(\n            view,\n            ripple_times=ripple_times,\n            ripple_times_color=ripple_times_color,\n        ):\n            return view.add_interval_series(\n                name=\"Ripple Events\",\n                t_start=ripple_times.start_time.to_numpy(),\n                t_end=ripple_times.end_time.to_numpy(),\n                color=ripple_times_color,\n            )\n\n        key = self.fetch1(\"KEY\")\n        (\n            speed,\n            ripple_filtered_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n        ripple_consensus_trace = self.get_Kay_ripple_consensus_trace(\n            ripple_filtered_lfps, sampling_frequency\n        )\n\n        if zscore_ripple:\n            ripple_consensus_trace = zscore(ripple_consensus_trace)\n\n        consensus_view = _add_ripple_times(vv.TimeseriesGraph())\n        consensus_name = (\n            \"Z-Scored Consensus Trace\" if zscore_ripple else \"Consensus Trace\"\n        )\n        consensus_view.add_line_series(\n            name=consensus_name,\n            t=np.asarray(ripple_consensus_trace.index).squeeze(),\n            y=np.asarray(ripple_consensus_trace, dtype=np.float32).squeeze(),\n            color=consensus_color,\n            width=1,\n        )\n        if zscore_ripple:\n            ripple_params = (\n                RippleParameters\n                &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n            ).fetch1(\"ripple_param_dict\")\n\n            zscore_threshold = ripple_params[\"ripple_detection_params\"].get(\n                \"zscore_threshold\"\n            )\n            if zscore_threshold is not None:\n                consensus_view.add_line_series(\n                    name=\"Z-Score Threshold\",\n                    t=np.asarray(ripple_consensus_trace.index).squeeze(),\n                    y=np.ones_like(\n                        ripple_consensus_trace, dtype=np.float32\n                    ).squeeze()\n                    * zscore_threshold,\n                    color=ripple_times_color,\n                    width=1,\n                )\n\n        if use_ripple_filtered_lfps:\n            interval_ripple_lfps = ripple_filtered_lfps\n        else:\n            lfp_merge_id = (LFPBandSelection &amp; key).fetch1(\"lfp_merge_id\")\n            lfp_df = (LFPOutput &amp; {\"merge_id\": lfp_merge_id}).fetch1_dataframe()\n            interval_ripple_lfps = lfp_df.loc[speed.index[0] : speed.index[-1]]\n        if lfp_channel_ind is not None:\n            if lfp_channel_ind.max() &gt;= interval_ripple_lfps.shape[1]:\n                raise ValueError(\n                    \"lfp_channel_ind is out of range for the number of LFPs\"\n                )\n            interval_ripple_lfps = interval_ripple_lfps.iloc[:, lfp_channel_ind]\n\n        lfp_view = _add_ripple_times(vv.TimeseriesGraph())\n        max_lfp_value = interval_ripple_lfps.to_numpy().max()\n        lfp_offset *= max_lfp_value\n\n        for i, lfp in enumerate(interval_ripple_lfps.to_numpy().T):\n            lfp_view.add_line_series(\n                name=f\"LFP {i}\",\n                t=np.asarray(interval_ripple_lfps.index).squeeze(),\n                y=np.asarray(lfp + lfp_offset * i, dtype=np.int16).squeeze(),\n                color=\"black\",\n                width=1,\n            )\n        speed_view = _add_ripple_times(vv.TimeseriesGraph())\n        speed_view.add_line_series(\n            name=\"Speed [cm/s]\",\n            t=np.asarray(speed.index).squeeze(),\n            y=np.asarray(speed, dtype=np.float32).squeeze(),\n            color=speed_color,\n            width=1,\n        )\n        vertical_panel_content = [\n            vv.LayoutItem(consensus_view, stretch=2, title=\"Consensus\"),\n            vv.LayoutItem(lfp_view, stretch=8, title=\"LFPs\"),\n            vv.LayoutItem(speed_view, stretch=2, title=\"Speed\"),\n        ]\n\n        view = vv.Box(\n            direction=\"horizontal\",\n            show_titles=True,\n            height=view_height,\n            items=[\n                vv.LayoutItem(\n                    vv.Box(\n                        direction=\"vertical\",\n                        show_titles=True,\n                        items=vertical_panel_content,\n                    )\n                ),\n            ],\n        )\n\n        return view.url(label=\"Ripple Detection\")\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleTimesV1.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/", "title": "sharing_kachery.py", "text": ""}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.kachery_download_file", "title": "<code>kachery_download_file(uri, dest, kachery_zone_name)</code>", "text": "<p>set the kachery resource url and attempt to down load the uri into the destination path</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>def kachery_download_file(uri: str, dest: str, kachery_zone_name: str) -&gt; str:\n    \"\"\"set the kachery resource url and attempt to down load the uri into the destination path\"\"\"\n    KacheryZone.set_resource_url({\"kachery_zone_name\": kachery_zone_name})\n    return kcl.load_file(uri, dest=dest)\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone", "title": "<code>KacheryZone</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@schema\nclass KacheryZone(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    kachery_zone_name: varchar(200) # the name of the kachery zone. Note that this is the same as the name of the kachery resource.\n    ---\n    description: varchar(200) # description of this zone\n    kachery_cloud_dir: varchar(200) # kachery cloud directory on local machine where files are linked\n    kachery_proxy: varchar(200) # kachery sharing proxy\n    -&gt; Lab\n    \"\"\"\n\n    @staticmethod\n    def set_zone(key: dict):\n        \"\"\"Set the kachery zone based on the key to KacheryZone\n\n        Parameters\n        ----------\n        key : dict\n            key defining a single KacheryZone\n\n        \"\"\"\n        try:\n            kachery_zone_name, kachery_cloud_dir = (KacheryZone &amp; key).fetch1(\n                \"kachery_zone_name\", \"kachery_cloud_dir\"\n            )\n        except DataJointError:\n            raise Exception(\n                f\"{key} does not correspond to a single entry in KacheryZone.\"\n            )\n            return None\n        # set the new zone and cloud directory\n        os.environ[kachery_zone_envar] = kachery_zone_name\n        os.environ[kachery_cloud_dir_envar] = kachery_cloud_dir\n\n    @staticmethod\n    def reset_zone():\n        \"\"\"Resets the kachery zone environment variable to the default values.\"\"\"\n        if default_kachery_zone is not None:\n            os.environ[kachery_zone_envar] = default_kachery_zone\n        if default_kachery_cloud_dir is not None:\n            os.environ[kachery_cloud_dir_envar] = default_kachery_cloud_dir\n\n    @staticmethod\n    def set_resource_url(key: dict):\n        \"\"\"Sets the KACHERY_RESOURCE_URL based on the key corresponding to a\n        single Kachery Zone\n\n        Parameters\n        ----------\n        key : dict\n            key to retrieve a single kachery zone\n        \"\"\"\n        try:\n            kachery_zone_name, kachery_proxy = (KacheryZone &amp; key).fetch1(\n                \"kachery_zone_name\", \"kachery_proxy\"\n            )\n        except DataJointError:\n            raise Exception(\n                f\"{key} does not correspond to a single entry in KacheryZone.\"\n            )\n        # set the new zone and cloud directory\n        os.environ[kachery_zone_envar] = kachery_zone_name\n        os.environ[kachery_resource_url_envar] = (\n            kachery_proxy + \"/r/\" + kachery_zone_name\n        )\n\n    @staticmethod\n    def reset_resource_url():\n        KacheryZone.reset_zone()\n        if default_kachery_resource_url is not None:\n            os.environ[kachery_resource_url_envar] = (\n                default_kachery_resource_url\n            )\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.set_zone", "title": "<code>set_zone(key)</code>  <code>staticmethod</code>", "text": "<p>Set the kachery zone based on the key to KacheryZone</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key defining a single KacheryZone</p> required Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef set_zone(key: dict):\n    \"\"\"Set the kachery zone based on the key to KacheryZone\n\n    Parameters\n    ----------\n    key : dict\n        key defining a single KacheryZone\n\n    \"\"\"\n    try:\n        kachery_zone_name, kachery_cloud_dir = (KacheryZone &amp; key).fetch1(\n            \"kachery_zone_name\", \"kachery_cloud_dir\"\n        )\n    except DataJointError:\n        raise Exception(\n            f\"{key} does not correspond to a single entry in KacheryZone.\"\n        )\n        return None\n    # set the new zone and cloud directory\n    os.environ[kachery_zone_envar] = kachery_zone_name\n    os.environ[kachery_cloud_dir_envar] = kachery_cloud_dir\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.reset_zone", "title": "<code>reset_zone()</code>  <code>staticmethod</code>", "text": "<p>Resets the kachery zone environment variable to the default values.</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef reset_zone():\n    \"\"\"Resets the kachery zone environment variable to the default values.\"\"\"\n    if default_kachery_zone is not None:\n        os.environ[kachery_zone_envar] = default_kachery_zone\n    if default_kachery_cloud_dir is not None:\n        os.environ[kachery_cloud_dir_envar] = default_kachery_cloud_dir\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.set_resource_url", "title": "<code>set_resource_url(key)</code>  <code>staticmethod</code>", "text": "<p>Sets the KACHERY_RESOURCE_URL based on the key corresponding to a single Kachery Zone</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key to retrieve a single kachery zone</p> required Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef set_resource_url(key: dict):\n    \"\"\"Sets the KACHERY_RESOURCE_URL based on the key corresponding to a\n    single Kachery Zone\n\n    Parameters\n    ----------\n    key : dict\n        key to retrieve a single kachery zone\n    \"\"\"\n    try:\n        kachery_zone_name, kachery_proxy = (KacheryZone &amp; key).fetch1(\n            \"kachery_zone_name\", \"kachery_proxy\"\n        )\n    except DataJointError:\n        raise Exception(\n            f\"{key} does not correspond to a single entry in KacheryZone.\"\n        )\n    # set the new zone and cloud directory\n    os.environ[kachery_zone_envar] = kachery_zone_name\n    os.environ[kachery_resource_url_envar] = (\n        kachery_proxy + \"/r/\" + kachery_zone_name\n    )\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.AnalysisNwbfileKachery", "title": "<code>AnalysisNwbfileKachery</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@schema\nclass AnalysisNwbfileKachery(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; AnalysisNwbfileKacherySelection\n    ---\n    analysis_file_uri='': varchar(200)  # the uri of the file\n    \"\"\"\n\n    class LinkedFile(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; AnalysisNwbfileKachery\n        linked_file_rel_path: varchar(200) # the path for the linked file relative to the SPYGLASS_BASE_DIR environment variable\n        ---\n        linked_file_uri='': varchar(200) # the uri for the linked file\n        \"\"\"\n\n    def make(self, key):\n        # note that we're assuming that the user has initialized a kachery-cloud\n        # client with kachery-cloud-init. Uncomment the line below once we are\n        # sharing linked files as well.\n\n        # linked_key = copy.deepcopy(key)\n\n        logger.info(f'Linking {key[\"analysis_file_name\"]} in kachery-cloud...')\n        # set the kachery zone\n\n        KacheryZone.set_zone(key)\n\n        key[\"analysis_file_uri\"] = kcl.link_file(\n            AnalysisNwbfile().get_abs_path(key[\"analysis_file_name\"])\n        )\n        logger.info(\n            os.environ[kachery_zone_envar], os.environ[kachery_cloud_dir_envar]\n        )\n        logger.info(AnalysisNwbfile().get_abs_path(key[\"analysis_file_name\"]))\n        logger.info(kcl.load_file(key[\"analysis_file_uri\"]))\n        self.insert1(key)\n\n        # we also need to insert any linked files\n        # TODO: change this to automatically detect all linked files\n        # self.LinkedFile.insert1(key)\n\n        # reset the Kachery zone and cloud_dir to the defaults\n        KacheryZone.reset_zone()\n\n    @staticmethod\n    def download_file(analysis_file_name: str) -&gt; bool:\n        \"\"\"Download the specified analysis file and associated linked files\n        from kachery-cloud if possible\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis file\n\n        Returns\n        ----------\n        is_success : bool\n            True if the file was successfully downloaded, False otherwise\n        \"\"\"\n        fetched_list = (\n            AnalysisNwbfileKachery &amp; {\"analysis_file_name\": analysis_file_name}\n        ).fetch(\"analysis_file_uri\", \"kachery_zone_name\")\n        downloaded = False\n        for uri, kachery_zone_name in zip(fetched_list[0], fetched_list[1]):\n            if len(uri) == 0:\n                return False\n            logger.info(\"uri:\", uri)\n            if kachery_download_file(\n                uri=uri,\n                dest=AnalysisNwbfile.get_abs_path(analysis_file_name),\n                kachery_zone_name=kachery_zone_name,\n            ):\n                downloaded = True\n                # now download the linked file(s)\n                linked_files = (\n                    AnalysisNwbfileKachery.LinkedFile\n                    &amp; {\"analysis_file_name\": analysis_file_name}\n                ).fetch(as_dict=True)\n                for file in linked_files:\n                    uri = file[\"linked_file_uri\"]\n                    logger.info(f\"attempting to download linked file uri {uri}\")\n                    linked_file_path = (\n                        os.environ[\"SPYGLASS_BASE_DIR\"]\n                        + file[\"linked_file_rel_path\"]\n                    )\n                    if not kachery_download_file(\n                        uri=uri,\n                        dest=linked_file_path,\n                        kachery_zone_name=kachery_zone_name,\n                    ):\n                        raise Exception(\n                            f\"Linked file {linked_file_path} cannot be downloaded\"\n                        )\n        if not downloaded:\n            raise Exception(f\"{analysis_file_name} cannot be downloaded\")\n\n        return True\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.AnalysisNwbfileKachery.download_file", "title": "<code>download_file(analysis_file_name)</code>  <code>staticmethod</code>", "text": "<p>Download the specified analysis file and associated linked files from kachery-cloud if possible</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis file</p> required <p>Returns:</p> Name Type Description <code>is_success</code> <code>bool</code> <p>True if the file was successfully downloaded, False otherwise</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef download_file(analysis_file_name: str) -&gt; bool:\n    \"\"\"Download the specified analysis file and associated linked files\n    from kachery-cloud if possible\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis file\n\n    Returns\n    ----------\n    is_success : bool\n        True if the file was successfully downloaded, False otherwise\n    \"\"\"\n    fetched_list = (\n        AnalysisNwbfileKachery &amp; {\"analysis_file_name\": analysis_file_name}\n    ).fetch(\"analysis_file_uri\", \"kachery_zone_name\")\n    downloaded = False\n    for uri, kachery_zone_name in zip(fetched_list[0], fetched_list[1]):\n        if len(uri) == 0:\n            return False\n        logger.info(\"uri:\", uri)\n        if kachery_download_file(\n            uri=uri,\n            dest=AnalysisNwbfile.get_abs_path(analysis_file_name),\n            kachery_zone_name=kachery_zone_name,\n        ):\n            downloaded = True\n            # now download the linked file(s)\n            linked_files = (\n                AnalysisNwbfileKachery.LinkedFile\n                &amp; {\"analysis_file_name\": analysis_file_name}\n            ).fetch(as_dict=True)\n            for file in linked_files:\n                uri = file[\"linked_file_uri\"]\n                logger.info(f\"attempting to download linked file uri {uri}\")\n                linked_file_path = (\n                    os.environ[\"SPYGLASS_BASE_DIR\"]\n                    + file[\"linked_file_rel_path\"]\n                )\n                if not kachery_download_file(\n                    uri=uri,\n                    dest=linked_file_path,\n                    kachery_zone_name=kachery_zone_name,\n                ):\n                    raise Exception(\n                        f\"Linked file {linked_file_path} cannot be downloaded\"\n                    )\n    if not downloaded:\n        raise Exception(f\"{analysis_file_name} cannot be downloaded\")\n\n    return True\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.share_data_to_kachery", "title": "<code>share_data_to_kachery(restriction={}, table_list=[], zone_name=None)</code>", "text": "<p>Share data to kachery</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>dict</code> <p>restriction to select what data should be shared from table, by default {}</p> <code>{}</code> <code>table_list</code> <code>list</code> <p>List of tables to share data from, by default []</p> <code>[]</code> <code>zone_name</code> <code>str</code> <p>What kachery zone to share the data to, by default zone in spyglass.settings.config, which looks for <code>KACHERY_ZONE</code> environmental variable, but defaults to 'franklab.default'</p> <code>None</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>Does not allow sharing of all data in table</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>def share_data_to_kachery(\n    restriction={},\n    table_list=[],\n    zone_name=None,\n):\n    \"\"\"Share data to kachery\n\n    Parameters\n    ----------\n    restriction : dict, optional\n        restriction to select what data should be shared from table, by default {}\n    table_list : list, optional\n        List of tables to share data from, by default []\n    zone_name : str, optional\n        What kachery zone to share the data to, by default zone in spyglass.settings.config,\n        which looks for `KACHERY_ZONE` environmental variable, but defaults to\n        'franklab.default'\n\n    Raises\n    ------\n    ValueError\n        Does not allow sharing of all data in table\n    \"\"\"\n    if not zone_name:\n        zone_name = config[\"KACHERY_ZONE\"]\n    kachery_selection_key = {\"kachery_zone_name\": zone_name}\n    if not restriction:\n        raise ValueError(\"Must provide a restriction to the table\")\n    selection_inserts = []\n    for table in table_list:\n        analysis_file_list = (table &amp; restriction).fetch(\"analysis_file_name\")\n        for file in analysis_file_list:  # Add all analysis to shared list\n            kachery_selection_key[\"analysis_file_name\"] = file\n            selection_inserts.append(kachery_selection_key)\n    AnalysisNwbfileKacherySelection.insert(\n        selection_inserts, skip_duplicates=True\n    )\n    AnalysisNwbfileKachery.populate()\n</code></pre>"}, {"location": "api/spikesorting/imported/", "title": "imported.py", "text": ""}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting", "title": "<code>ImportedSpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>@schema\nclass ImportedSpikeSorting(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; Session\n    ---\n    object_id: varchar(40)\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    class Annotations(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; ImportedSpikeSorting\n        id: int # unit id, corresponds to dataframe index of unit in NWB file\n        ---\n        label = Null: longblob # list of string labels for the unit\n        annotations: longblob # dict of other annotations (e.g. metrics)\n        \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n\n        nwb_file_abs_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n\n        with pynwb.NWBHDF5IO(\n            nwb_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            if not nwbfile.units:\n                logger.warn(\"No units found in NWB file\")\n                return\n\n        from spyglass.spikesorting.spikesorting_merge import (  # noqa: F401\n            SpikeSortingOutput,\n        )\n\n        key[\"object_id\"] = nwbfile.units.object_id\n\n        self.insert1(key, skip_duplicates=True)\n\n        part_name = SpikeSortingOutput._part_name(self.table_name)\n        SpikeSortingOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    @classmethod\n    def get_recording(cls, key):\n        raise NotImplementedError(\n            \"Imported spike sorting does not have a `get_recording` method\"\n        )\n\n    @classmethod\n    def get_sorting(cls, key):\n        raise NotImplementedError(\n            \"Imported spike sorting does not have a `get_sorting` method\"\n        )\n\n    def add_annotation(\n        self, key, id, label=[], annotations={}, merge_annotations=False\n    ):\n        \"\"\"Manually add annotations to the spike sorting output\n\n        Parameters\n        ----------\n        key : dict\n            restriction key for ImportedSpikeSorting\n        id : int\n            unit id\n        label : List[str], optional\n            list of str labels for the unit, by default None\n        annotations : dict, optional\n            dictionary of other annotation values for unit, by default None\n        merge_annotations : bool, optional\n            whether to merge with existing annotations, by default False\n        \"\"\"\n        if isinstance(label, str):\n            label = [label]\n        query = self &amp; key\n        if not len(query) == 1:\n            raise ValueError(\n                f\"ImportedSpikeSorting key must be unique. Found: {query}\"\n            )\n        unit_key = {**key, \"id\": id}\n        annotation_query = ImportedSpikeSorting.Annotations &amp; unit_key\n        if annotation_query and not merge_annotations:\n            raise ValueError(\n                f\"Unit already has annotations: {annotation_query}\"\n            )\n        elif annotation_query:\n            existing_annotations = annotation_query.fetch1()\n            existing_annotations[\"label\"] += label\n            existing_annotations[\"annotations\"].update(annotations)\n            self.Annotations.update1(existing_annotations)\n        else:\n            self.Annotations.insert1(\n                dict(unit_key, label=label, annotations=annotations),\n                skip_duplicates=True,\n            )\n\n    def make_df_from_annotations(self):\n        \"\"\"Convert the annotations part table into a dataframe that can be\n        concatenated to the spikes dataframe in the nwb file.\"\"\"\n        df = []\n        for id, label, annotations in zip(\n            *self.Annotations.fetch(\"id\", \"label\", \"annotations\")\n        ):\n            df.append(\n                dict(\n                    id=id,\n                    label=label,\n                    **annotations,\n                )\n            )\n        df = pd.DataFrame(df)\n        df.set_index(\"id\", inplace=True)\n        return df\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        \"\"\"class method to fetch the nwb and add annotations to the spike dfs returned\"\"\"\n        # get the original nwbs\n        nwbs = super().fetch_nwb(*attrs, **kwargs)\n        # for each nwb, get the annotations and add them to the spikes dataframe\n        for i, key in enumerate(self.fetch(\"KEY\")):\n            if not ImportedSpikeSorting.Annotations &amp; key:\n                continue\n            # make the annotation_df\n            annotation_df = (self &amp; key).make_df_from_annotations()\n            # concatenate the annotations to the spikes dataframe in the returned nwb\n            nwbs[i][\"object_id\"] = pd.concat(\n                [nwbs[i][\"object_id\"], annotation_df], axis=\"columns\"\n            )\n        return nwbs\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.add_annotation", "title": "<code>add_annotation(key, id, label=[], annotations={}, merge_annotations=False)</code>", "text": "<p>Manually add annotations to the spike sorting output</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>restriction key for ImportedSpikeSorting</p> required <code>id</code> <code>int</code> <p>unit id</p> required <code>label</code> <code>List[str]</code> <p>list of str labels for the unit, by default None</p> <code>[]</code> <code>annotations</code> <code>dict</code> <p>dictionary of other annotation values for unit, by default None</p> <code>{}</code> <code>merge_annotations</code> <code>bool</code> <p>whether to merge with existing annotations, by default False</p> <code>False</code> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def add_annotation(\n    self, key, id, label=[], annotations={}, merge_annotations=False\n):\n    \"\"\"Manually add annotations to the spike sorting output\n\n    Parameters\n    ----------\n    key : dict\n        restriction key for ImportedSpikeSorting\n    id : int\n        unit id\n    label : List[str], optional\n        list of str labels for the unit, by default None\n    annotations : dict, optional\n        dictionary of other annotation values for unit, by default None\n    merge_annotations : bool, optional\n        whether to merge with existing annotations, by default False\n    \"\"\"\n    if isinstance(label, str):\n        label = [label]\n    query = self &amp; key\n    if not len(query) == 1:\n        raise ValueError(\n            f\"ImportedSpikeSorting key must be unique. Found: {query}\"\n        )\n    unit_key = {**key, \"id\": id}\n    annotation_query = ImportedSpikeSorting.Annotations &amp; unit_key\n    if annotation_query and not merge_annotations:\n        raise ValueError(\n            f\"Unit already has annotations: {annotation_query}\"\n        )\n    elif annotation_query:\n        existing_annotations = annotation_query.fetch1()\n        existing_annotations[\"label\"] += label\n        existing_annotations[\"annotations\"].update(annotations)\n        self.Annotations.update1(existing_annotations)\n    else:\n        self.Annotations.insert1(\n            dict(unit_key, label=label, annotations=annotations),\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.make_df_from_annotations", "title": "<code>make_df_from_annotations()</code>", "text": "<p>Convert the annotations part table into a dataframe that can be concatenated to the spikes dataframe in the nwb file.</p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def make_df_from_annotations(self):\n    \"\"\"Convert the annotations part table into a dataframe that can be\n    concatenated to the spikes dataframe in the nwb file.\"\"\"\n    df = []\n    for id, label, annotations in zip(\n        *self.Annotations.fetch(\"id\", \"label\", \"annotations\")\n    ):\n        df.append(\n            dict(\n                id=id,\n                label=label,\n                **annotations,\n            )\n        )\n    df = pd.DataFrame(df)\n    df.set_index(\"id\", inplace=True)\n    return df\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>class method to fetch the nwb and add annotations to the spike dfs returned</p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs):\n    \"\"\"class method to fetch the nwb and add annotations to the spike dfs returned\"\"\"\n    # get the original nwbs\n    nwbs = super().fetch_nwb(*attrs, **kwargs)\n    # for each nwb, get the annotations and add them to the spikes dataframe\n    for i, key in enumerate(self.fetch(\"KEY\")):\n        if not ImportedSpikeSorting.Annotations &amp; key:\n            continue\n        # make the annotation_df\n        annotation_df = (self &amp; key).make_df_from_annotations()\n        # concatenate the annotations to the spikes dataframe in the returned nwb\n        nwbs[i][\"object_id\"] = pd.concat(\n            [nwbs[i][\"object_id\"], annotation_df], axis=\"columns\"\n        )\n    return nwbs\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/", "title": "spikesorting_merge.py", "text": ""}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput", "title": "<code>SpikeSortingOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@schema\nclass SpikeSortingOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    # Output of spike sorting pipelines.\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class CurationV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CurationV1\n        \"\"\"\n\n    class ImportedSpikeSorting(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ImportedSpikeSorting\n        \"\"\"\n\n    class CuratedSpikeSorting(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CuratedSpikeSorting\n        \"\"\"\n\n    @classmethod\n    def get_recording(cls, key):\n        \"\"\"get the recording associated with a spike sorting output\"\"\"\n        source_table = source_class_dict[\n            to_camel_case(cls.merge_get_parent(key).table_name)\n        ]\n        query = source_table &amp; cls.merge_get_part(key)\n        return query.get_recording(query.fetch(\"KEY\"))\n\n    @classmethod\n    def get_sorting(cls, key):\n        \"\"\"get the sorting associated with a spike sorting output\"\"\"\n        source_table = source_class_dict[\n            to_camel_case(cls.merge_get_parent(key).table_name)\n        ]\n        query = source_table &amp; cls.merge_get_part(key)\n        return query.get_sorting(query.fetch(\"KEY\"))\n\n    @classmethod\n    def get_spike_times(cls, key):\n        spike_times = []\n        for nwb_file in cls.fetch_nwb(key):\n            # V1 uses 'object_id', V0 uses 'units'\n            file_loc = \"object_id\" if \"object_id\" in nwb_file else \"units\"\n            spike_times.extend(nwb_file[file_loc][\"spike_times\"].to_list())\n        return spike_times\n\n    @classmethod\n    def get_spike_indicator(cls, key, time):\n        time = np.asarray(time)\n        min_time, max_time = time[[0, -1]]\n        spike_times = cls.fetch_spike_data(key)\n        spike_indicator = np.zeros((len(time), len(spike_times)))\n\n        for ind, times in enumerate(spike_times):\n            times = times[np.logical_and(times &gt;= min_time, times &lt;= max_time)]\n            spike_indicator[:, ind] = np.bincount(\n                np.digitize(times, time[1:-1]),\n                minlength=time.shape[0],\n            )\n\n        return spike_indicator\n\n    @classmethod\n    def get_firing_rate(cls, key, time, multiunit=False):\n        spike_indicator = cls.get_spike_indicator(key, time)\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        if multiunit:\n            spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n        return np.stack(\n            [\n                get_multiunit_population_firing_rate(\n                    indicator[:, np.newaxis], sampling_frequency\n                )\n                for indicator in spike_indicator.T\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>get the recording associated with a spike sorting output</p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@classmethod\ndef get_recording(cls, key):\n    \"\"\"get the recording associated with a spike sorting output\"\"\"\n    source_table = source_class_dict[\n        to_camel_case(cls.merge_get_parent(key).table_name)\n    ]\n    query = source_table &amp; cls.merge_get_part(key)\n    return query.get_recording(query.fetch(\"KEY\"))\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>get the sorting associated with a spike sorting output</p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key):\n    \"\"\"get the sorting associated with a spike sorting output\"\"\"\n    source_table = source_class_dict[\n        to_camel_case(cls.merge_get_parent(key).table_name)\n    ]\n    query = source_table &amp; cls.merge_get_part(key)\n    return query.get_sorting(query.fetch(\"KEY\"))\n</code></pre>"}, {"location": "api/spikesorting/analysis/v1/group/", "title": "group.py", "text": ""}, {"location": "api/spikesorting/analysis/v1/group/#src.spyglass.spikesorting.analysis.v1.group.SortedSpikesGroup", "title": "<code>SortedSpikesGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/analysis/v1/group.py</code> <pre><code>@schema\nclass SortedSpikesGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Session\n    -&gt; UnitSelectionParams\n    sorted_spikes_group_name: varchar(80)\n    \"\"\"\n\n    class Units(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        -&gt; SpikeSortingOutput.proj(spikesorting_merge_id='merge_id')\n        \"\"\"\n\n    def create_group(\n        self,\n        group_name: str,\n        nwb_file_name: str,\n        unit_filter_params_name: str = \"all_units\",\n        keys: list[dict] = [],\n    ):\n        group_key = {\n            \"sorted_spikes_group_name\": group_name,\n            \"nwb_file_name\": nwb_file_name,\n            \"unit_filter_params_name\": unit_filter_params_name,\n        }\n        parts_insert = [{**key, **group_key} for key in keys]\n\n        self.insert1(\n            group_key,\n            skip_duplicates=True,\n        )\n        self.Units.insert(parts_insert, skip_duplicates=True)\n\n    @staticmethod\n    def filter_units(\n        labels: list[list[str]],\n        include_labels: list[str],\n        exclude_labels: list[str],\n    ) -&gt; np.ndarray:\n        \"\"\"\n        Filter units based on labels\n        \"\"\"\n        include_labels = np.unique(include_labels)\n        exclude_labels = np.unique(exclude_labels)\n\n        if include_labels.size == 0 and exclude_labels.size == 0:\n            # if no labels are provided, include all units\n            return np.ones(len(labels), dtype=bool)\n\n        include_mask = np.zeros(len(labels), dtype=bool)\n        for ind, unit_labels in enumerate(labels):\n            if isinstance(unit_labels, str):\n                unit_labels = [unit_labels]\n            if (\n                include_labels.size &gt; 0\n                and np.all(~np.isin(unit_labels, include_labels))\n            ) or np.any(np.isin(unit_labels, exclude_labels)):\n                # if the unit does not have any of the include labels\n                # or has any of the exclude labels, skip\n                continue\n            include_mask[ind] = True\n        return include_mask\n\n    @staticmethod\n    def fetch_spike_data(key, time_slice=None):\n        # get merge_ids for SpikeSortingOutput\n        merge_ids = (\n            (\n                SortedSpikesGroup.Units\n                &amp; {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"sorted_spikes_group_name\": key[\"sorted_spikes_group_name\"],\n                }\n            )\n        ).fetch(\"spikesorting_merge_id\")\n\n        # get the filtering parameters\n        include_labels, exclude_labels = (UnitSelectionParams &amp; key).fetch1(\n            \"include_labels\", \"exclude_labels\"\n        )\n\n        # get the spike times for each merge_id\n        spike_times = []\n        for merge_id in merge_ids:\n            nwb_file = SpikeSortingOutput().fetch_nwb({\"merge_id\": merge_id})[0]\n            nwb_field_name = (\n                \"object_id\"\n                if \"object_id\" in nwb_file\n                else \"units\" if \"units\" in nwb_file else None\n            )\n            if nwb_field_name is None:\n                # case where no units found or curation removed all units\n                continue\n            sorting_spike_times = nwb_file[nwb_field_name][\n                \"spike_times\"\n            ].to_list()\n\n            # filter the spike times based on the labels if present\n            if \"label\" in nwb_file[nwb_field_name]:\n                group_label_list = nwb_file[nwb_field_name][\"label\"].to_list()\n                include_unit = SortedSpikesGroup.filter_units(\n                    group_label_list, include_labels, exclude_labels\n                )\n\n                sorting_spike_times = list(\n                    compress(sorting_spike_times, include_unit)\n                )\n\n            # filter the spike times based on the time slice if provided\n            if time_slice is not None:\n                sorting_spike_times = [\n                    times[\n                        np.logical_and(\n                            times &gt;= time_slice.start, times &lt;= time_slice.stop\n                        )\n                    ]\n                    for times in sorting_spike_times\n                ]\n\n            # append the approved spike times to the list\n            spike_times.extend(sorting_spike_times)\n\n        return spike_times\n\n    @classmethod\n    def get_spike_indicator(cls, key: dict, time: np.ndarray) -&gt; np.ndarray:\n        time = np.asarray(time)\n        min_time, max_time = time[[0, -1]]\n        spike_times = cls.fetch_spike_data(key)\n        spike_indicator = np.zeros((len(time), len(spike_times)))\n\n        for ind, times in enumerate(spike_times):\n            times = times[np.logical_and(times &gt;= min_time, times &lt;= max_time)]\n            spike_indicator[:, ind] = np.bincount(\n                np.digitize(times, time[1:-1]),\n                minlength=time.shape[0],\n            )\n\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        return spike_indicator\n\n    @classmethod\n    def get_firing_rate(\n        cls, key: dict, time: np.ndarray, multiunit: bool = False\n    ) -&gt; np.ndarray:\n        spike_indicator = cls.get_spike_indicator(key, time)\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        if multiunit:\n            spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n        return np.stack(\n            [\n                get_multiunit_population_firing_rate(\n                    indicator[:, np.newaxis], sampling_frequency\n                )\n                for indicator in spike_indicator.T\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/spikesorting/analysis/v1/group/#src.spyglass.spikesorting.analysis.v1.group.SortedSpikesGroup.filter_units", "title": "<code>filter_units(labels, include_labels, exclude_labels)</code>  <code>staticmethod</code>", "text": "<p>Filter units based on labels</p> Source code in <code>src/spyglass/spikesorting/analysis/v1/group.py</code> <pre><code>@staticmethod\ndef filter_units(\n    labels: list[list[str]],\n    include_labels: list[str],\n    exclude_labels: list[str],\n) -&gt; np.ndarray:\n    \"\"\"\n    Filter units based on labels\n    \"\"\"\n    include_labels = np.unique(include_labels)\n    exclude_labels = np.unique(exclude_labels)\n\n    if include_labels.size == 0 and exclude_labels.size == 0:\n        # if no labels are provided, include all units\n        return np.ones(len(labels), dtype=bool)\n\n    include_mask = np.zeros(len(labels), dtype=bool)\n    for ind, unit_labels in enumerate(labels):\n        if isinstance(unit_labels, str):\n            unit_labels = [unit_labels]\n        if (\n            include_labels.size &gt; 0\n            and np.all(~np.isin(unit_labels, include_labels))\n        ) or np.any(np.isin(unit_labels, exclude_labels)):\n            # if the unit does not have any of the include labels\n            # or has any of the exclude labels, skip\n            continue\n        include_mask[ind] = True\n    return include_mask\n</code></pre>"}, {"location": "api/spikesorting/v0/curation_figurl/", "title": "curation_figurl.py", "text": ""}, {"location": "api/spikesorting/v0/curation_figurl/#src.spyglass.spikesorting.v0.curation_figurl.CurationFigurl", "title": "<code>CurationFigurl</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/curation_figurl.py</code> <pre><code>@schema\nclass CurationFigurl(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; CurationFigurlSelection\n    ---\n    url: varchar(2000)\n    initial_curation_uri: varchar(2000)\n    new_curation_uri: varchar(2000)\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Create a Curation Figurl\n        Parameters\n        ----------\n        key : dict\n            primary key of an entry from CurationFigurlSelection table\n        \"\"\"\n\n        # get new_curation_uri from selection table\n        new_curation_uri = (CurationFigurlSelection &amp; key).fetch1(\n            \"new_curation_uri\"\n        )\n\n        # fetch\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        recording_label = SpikeSortingRecording._get_recording_name(key)\n        sorting_label = SpikeSorting._get_sorting_name(key)\n        unit_metrics = _reformat_metrics(\n            (Curation &amp; key).fetch1(\"quality_metrics\")\n        )\n        initial_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        initial_merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n\n        # new_curation_uri = key[\"new_curation_uri\"]\n\n        # Create the initial curation and store it in kachery\n        for k, v in initial_labels.items():\n            new_list = []\n            for item in v:\n                if item not in new_list:\n                    new_list.append(item)\n            initial_labels[k] = new_list\n        initial_curation = {\n            \"labelsByUnit\": initial_labels,\n            \"mergeGroups\": initial_merge_groups,\n        }\n        initial_curation_uri = kcl.store_json(initial_curation)\n\n        # Get the recording/sorting extractors\n        R = si.load_extractor(recording_path)\n        if R.get_num_segments() &gt; 1:\n            R = si.concatenate_recordings([R])\n        S = si.load_extractor(sorting_path)\n\n        # Generate the figURL\n        url = _generate_the_figurl(\n            R=R,\n            S=S,\n            initial_curation_uri=initial_curation_uri,\n            new_curation_uri=new_curation_uri,\n            recording_label=recording_label,\n            sorting_label=sorting_label,\n            unit_metrics=unit_metrics,\n        )\n\n        # insert\n        key[\"url\"] = url\n        key[\"initial_curation_uri\"] = initial_curation_uri\n        key[\"new_curation_uri\"] = new_curation_uri\n        self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/curation_figurl/#src.spyglass.spikesorting.v0.curation_figurl.CurationFigurl.make", "title": "<code>make(key)</code>", "text": "<p>Create a Curation Figurl</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of an entry from CurationFigurlSelection table</p> required Source code in <code>src/spyglass/spikesorting/v0/curation_figurl.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Create a Curation Figurl\n    Parameters\n    ----------\n    key : dict\n        primary key of an entry from CurationFigurlSelection table\n    \"\"\"\n\n    # get new_curation_uri from selection table\n    new_curation_uri = (CurationFigurlSelection &amp; key).fetch1(\n        \"new_curation_uri\"\n    )\n\n    # fetch\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    recording_label = SpikeSortingRecording._get_recording_name(key)\n    sorting_label = SpikeSorting._get_sorting_name(key)\n    unit_metrics = _reformat_metrics(\n        (Curation &amp; key).fetch1(\"quality_metrics\")\n    )\n    initial_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n    initial_merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n\n    # new_curation_uri = key[\"new_curation_uri\"]\n\n    # Create the initial curation and store it in kachery\n    for k, v in initial_labels.items():\n        new_list = []\n        for item in v:\n            if item not in new_list:\n                new_list.append(item)\n        initial_labels[k] = new_list\n    initial_curation = {\n        \"labelsByUnit\": initial_labels,\n        \"mergeGroups\": initial_merge_groups,\n    }\n    initial_curation_uri = kcl.store_json(initial_curation)\n\n    # Get the recording/sorting extractors\n    R = si.load_extractor(recording_path)\n    if R.get_num_segments() &gt; 1:\n        R = si.concatenate_recordings([R])\n    S = si.load_extractor(sorting_path)\n\n    # Generate the figURL\n    url = _generate_the_figurl(\n        R=R,\n        S=S,\n        initial_curation_uri=initial_curation_uri,\n        new_curation_uri=new_curation_uri,\n        recording_label=recording_label,\n        sorting_label=sorting_label,\n        unit_metrics=unit_metrics,\n    )\n\n    # insert\n    key[\"url\"] = url\n    key[\"initial_curation_uri\"] = initial_curation_uri\n    key[\"new_curation_uri\"] = new_curation_uri\n    self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/merged_sorting_extractor/", "title": "merged_sorting_extractor.py", "text": ""}, {"location": "api/spikesorting/v0/sortingview/", "title": "sortingview.py", "text": ""}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace", "title": "<code>SortingviewWorkspace</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>@schema\nclass SortingviewWorkspace(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SortingviewWorkspaceSelection\n    ---\n    workspace_uri: varchar(1000)\n    sortingview_recording_id: varchar(30)\n    sortingview_sorting_id: varchar(30)\n    channel = NULL : varchar(80)        # the name of kachery channel for data sharing (for kachery daemon, deprecated)\n    \"\"\"\n\n    # make class for parts table to hold URLs\n    class URL(SpyglassMixin, dj.Part):\n        # Table for holding URLs\n        definition = \"\"\"\n        -&gt; SortingviewWorkspace\n        ---\n        curation_url: varchar(1000)   # URL with sortingview data\n        curation_jot: varchar(200)   # URI for saving manual curation tags\n        \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Create a Sortingview workspace\n\n        Parameters\n        ----------\n        key : dict\n            primary key of an entry from SortingviewWorkspaceSelection table\n        \"\"\"\n\n        # fetch\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n        workspace_label = SpikeSortingRecording._get_recording_name(key)\n        recording_label = SpikeSortingRecording._get_recording_name(key)\n        sorting_label = SpikeSorting._get_sorting_name(key)\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n        curation_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        team_name = (SpikeSortingRecording &amp; key).fetch1()[\"team_name\"]\n        team_members = (LabTeam.LabTeamMember &amp; {\"team_name\": team_name}).fetch(\n            \"lab_member_name\"\n        )\n        google_user_ids = []\n        for team_member in team_members:\n            google_user_id = (\n                LabMember.LabMemberInfo &amp; {\"lab_member_name\": team_member}\n            ).fetch(\"google_user_name\")\n            if len(google_user_id) != 1:\n                logger.warn(\n                    f\"Google user ID for {team_member} does not exist or more than one ID detected;\\\n                        permission not given to {team_member}, skipping...\"\n                )\n                continue\n            google_user_ids.append(google_user_id[0])\n\n        # do\n        (\n            workspace_uri,\n            recording_id,\n            sorting_id,\n        ) = _create_spikesortingview_workspace(\n            recording_path=recording_path,\n            sorting_path=sorting_path,\n            merge_groups=merge_groups,\n            workspace_label=workspace_label,\n            recording_label=recording_label,\n            sorting_label=sorting_label,\n            metrics=metrics,\n            curation_labels=curation_labels,\n            google_user_ids=google_user_ids,\n        )\n\n        # insert\n        key[\"workspace_uri\"] = workspace_uri\n        key[\"sortingview_recording_id\"] = recording_id\n        key[\"sortingview_sorting_id\"] = sorting_id\n        self.insert1(key)\n\n        # insert URLs\n        # remove non-primary keys\n        del key[\"workspace_uri\"]\n        del key[\"sortingview_recording_id\"]\n        del key[\"sortingview_sorting_id\"]\n\n        # generate URLs and add to key\n        url = self.url_trythis(key)\n        key[\"curation_url\"] = url\n        key[\"curation_jot\"] = \"not ready yet\"\n\n        SortingviewWorkspace.URL.insert1(key)\n\n    def remove_sorting_from_workspace(self, key):\n        return NotImplementedError\n\n    def url_trythis(self, key: dict, sortingview_sorting_id: str = None):\n        \"\"\"Generate a URL for visualizing and curating a sorting on the web.\n        Will print instructions on how to do the curation.\n\n        Parameters\n        ----------\n        key : dict\n            An entry from SortingviewWorkspace table\n        sortingview_sorting_id : str, optional\n            sortingview sorting ID to visualize. If None then chooses the first one\n\n        Returns\n        -------\n        url : str\n        \"\"\"\n        workspace_uri = (self &amp; key).fetch1(\"workspace_uri\")\n        workspace = sv.load_workspace(workspace_uri)\n        recording_id = workspace.recording_ids[0]\n        if sortingview_sorting_id is None:\n            sortingview_sorting_id = workspace.sorting_ids[0]\n\n        R = workspace.get_recording_extractor(recording_id)\n        S = workspace.get_sorting_extractor(sortingview_sorting_id)\n\n        initial_labels = (Curation &amp; key).fetch(\"curation_labels\")[0]\n        for k, v in initial_labels.items():\n            new_list = []\n            for item in v:\n                if item not in new_list:\n                    new_list.append(item)\n            initial_labels[k] = new_list\n        initial_curation = {\"labelsByUnit\": initial_labels}\n\n        # custom metrics\n        unit_metrics = workspace.get_unit_metrics_for_sorting(\n            sortingview_sorting_id\n        )\n\n        # This will print some instructions on how to do the curation\n        # old: sv.trythis_start_sorting_curation\n        url = _generate_url(\n            recording=R,\n            sorting=S,\n            label=workspace.label,\n            initial_curation=initial_curation,\n            raster_plot_subsample_max_firing_rate=50,\n            spike_amplitudes_subsample_max_firing_rate=50,\n            unit_metrics=unit_metrics,\n        )\n        return url\n\n    def insert_manual_curation(\n        self, key: dict, url: str, description=\"manually curated\"\n    ):\n        \"\"\"Based on information in key for an SortingviewWorkspace, loads the\n        curated sorting from sortingview, saves it (with labels and the\n        optional description) and inserts it to CuratedSorting\n\n        Assumes that the workspace corresponding to the recording and (original) sorting exists\n\n        Parameters\n        ----------\n        key : dict\n            primary key of AutomaticCuration\n        description: str, optional\n            description of curated sorting\n        \"\"\"\n\n        # get the labels and remove the non-primary merged units\n        # labels = workspace.get_sorting_curation(sorting_id=sortingview_sorting_id)\n        # labels = sv.trythis_load_sorting_curation('jot://xTzzyDieQPkW')\n        labels = sv.trythis_load_sorting_curation(url)\n\n        # turn labels to list of str, only including accepted units.\n        # if bool(labels[\"mergeGroups\"]):\n        if bool(labels.get(\"mergeGroups\", [])):\n            # clusters were merged, so we empty out metrics\n            metrics = {}\n        else:\n            # get the metrics from the parent curation\n            metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n\n        # insert this curation into the  Table\n        return Curation.insert_curation(\n            key,\n            parent_curation_id=key[\"curation_id\"],\n            labels=labels[\"labelsByUnit\"],\n            merge_groups=labels.get(\"mergeGroups\", []),\n            metrics=metrics,\n            description=description,\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.make", "title": "<code>make(key)</code>", "text": "<p>Create a Sortingview workspace</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of an entry from SortingviewWorkspaceSelection table</p> required Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Create a Sortingview workspace\n\n    Parameters\n    ----------\n    key : dict\n        primary key of an entry from SortingviewWorkspaceSelection table\n    \"\"\"\n\n    # fetch\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n    workspace_label = SpikeSortingRecording._get_recording_name(key)\n    recording_label = SpikeSortingRecording._get_recording_name(key)\n    sorting_label = SpikeSorting._get_sorting_name(key)\n    metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n    curation_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n    team_name = (SpikeSortingRecording &amp; key).fetch1()[\"team_name\"]\n    team_members = (LabTeam.LabTeamMember &amp; {\"team_name\": team_name}).fetch(\n        \"lab_member_name\"\n    )\n    google_user_ids = []\n    for team_member in team_members:\n        google_user_id = (\n            LabMember.LabMemberInfo &amp; {\"lab_member_name\": team_member}\n        ).fetch(\"google_user_name\")\n        if len(google_user_id) != 1:\n            logger.warn(\n                f\"Google user ID for {team_member} does not exist or more than one ID detected;\\\n                    permission not given to {team_member}, skipping...\"\n            )\n            continue\n        google_user_ids.append(google_user_id[0])\n\n    # do\n    (\n        workspace_uri,\n        recording_id,\n        sorting_id,\n    ) = _create_spikesortingview_workspace(\n        recording_path=recording_path,\n        sorting_path=sorting_path,\n        merge_groups=merge_groups,\n        workspace_label=workspace_label,\n        recording_label=recording_label,\n        sorting_label=sorting_label,\n        metrics=metrics,\n        curation_labels=curation_labels,\n        google_user_ids=google_user_ids,\n    )\n\n    # insert\n    key[\"workspace_uri\"] = workspace_uri\n    key[\"sortingview_recording_id\"] = recording_id\n    key[\"sortingview_sorting_id\"] = sorting_id\n    self.insert1(key)\n\n    # insert URLs\n    # remove non-primary keys\n    del key[\"workspace_uri\"]\n    del key[\"sortingview_recording_id\"]\n    del key[\"sortingview_sorting_id\"]\n\n    # generate URLs and add to key\n    url = self.url_trythis(key)\n    key[\"curation_url\"] = url\n    key[\"curation_jot\"] = \"not ready yet\"\n\n    SortingviewWorkspace.URL.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.url_trythis", "title": "<code>url_trythis(key, sortingview_sorting_id=None)</code>", "text": "<p>Generate a URL for visualizing and curating a sorting on the web. Will print instructions on how to do the curation.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>An entry from SortingviewWorkspace table</p> required <code>sortingview_sorting_id</code> <code>str</code> <p>sortingview sorting ID to visualize. If None then chooses the first one</p> <code>None</code> <p>Returns:</p> Name Type Description <code>url</code> <code>str</code> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def url_trythis(self, key: dict, sortingview_sorting_id: str = None):\n    \"\"\"Generate a URL for visualizing and curating a sorting on the web.\n    Will print instructions on how to do the curation.\n\n    Parameters\n    ----------\n    key : dict\n        An entry from SortingviewWorkspace table\n    sortingview_sorting_id : str, optional\n        sortingview sorting ID to visualize. If None then chooses the first one\n\n    Returns\n    -------\n    url : str\n    \"\"\"\n    workspace_uri = (self &amp; key).fetch1(\"workspace_uri\")\n    workspace = sv.load_workspace(workspace_uri)\n    recording_id = workspace.recording_ids[0]\n    if sortingview_sorting_id is None:\n        sortingview_sorting_id = workspace.sorting_ids[0]\n\n    R = workspace.get_recording_extractor(recording_id)\n    S = workspace.get_sorting_extractor(sortingview_sorting_id)\n\n    initial_labels = (Curation &amp; key).fetch(\"curation_labels\")[0]\n    for k, v in initial_labels.items():\n        new_list = []\n        for item in v:\n            if item not in new_list:\n                new_list.append(item)\n        initial_labels[k] = new_list\n    initial_curation = {\"labelsByUnit\": initial_labels}\n\n    # custom metrics\n    unit_metrics = workspace.get_unit_metrics_for_sorting(\n        sortingview_sorting_id\n    )\n\n    # This will print some instructions on how to do the curation\n    # old: sv.trythis_start_sorting_curation\n    url = _generate_url(\n        recording=R,\n        sorting=S,\n        label=workspace.label,\n        initial_curation=initial_curation,\n        raster_plot_subsample_max_firing_rate=50,\n        spike_amplitudes_subsample_max_firing_rate=50,\n        unit_metrics=unit_metrics,\n    )\n    return url\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.insert_manual_curation", "title": "<code>insert_manual_curation(key, url, description='manually curated')</code>", "text": "<p>Based on information in key for an SortingviewWorkspace, loads the curated sorting from sortingview, saves it (with labels and the optional description) and inserts it to CuratedSorting</p> <p>Assumes that the workspace corresponding to the recording and (original) sorting exists</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of AutomaticCuration</p> required <code>description</code> <p>description of curated sorting</p> <code>'manually curated'</code> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def insert_manual_curation(\n    self, key: dict, url: str, description=\"manually curated\"\n):\n    \"\"\"Based on information in key for an SortingviewWorkspace, loads the\n    curated sorting from sortingview, saves it (with labels and the\n    optional description) and inserts it to CuratedSorting\n\n    Assumes that the workspace corresponding to the recording and (original) sorting exists\n\n    Parameters\n    ----------\n    key : dict\n        primary key of AutomaticCuration\n    description: str, optional\n        description of curated sorting\n    \"\"\"\n\n    # get the labels and remove the non-primary merged units\n    # labels = workspace.get_sorting_curation(sorting_id=sortingview_sorting_id)\n    # labels = sv.trythis_load_sorting_curation('jot://xTzzyDieQPkW')\n    labels = sv.trythis_load_sorting_curation(url)\n\n    # turn labels to list of str, only including accepted units.\n    # if bool(labels[\"mergeGroups\"]):\n    if bool(labels.get(\"mergeGroups\", [])):\n        # clusters were merged, so we empty out metrics\n        metrics = {}\n    else:\n        # get the metrics from the parent curation\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n\n    # insert this curation into the  Table\n    return Curation.insert_curation(\n        key,\n        parent_curation_id=key[\"curation_id\"],\n        labels=labels[\"labelsByUnit\"],\n        merge_groups=labels.get(\"mergeGroups\", []),\n        metrics=metrics,\n        description=description,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview_helper_fn/", "title": "sortingview_helper_fn.py", "text": "<p>Sortingview helper functions</p>"}, {"location": "api/spikesorting/v0/spikesorting_artifact/", "title": "spikesorting_artifact.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_artifact/#src.spyglass.spikesorting.v0.spikesorting_artifact.ArtifactDetectionParameters", "title": "<code>ArtifactDetectionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_artifact.py</code> <pre><code>@schema\nclass ArtifactDetectionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for detecting artifact times within a sort group.\n    artifact_params_name: varchar(200)\n    ---\n    artifact_params: blob  # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default artifact parameters with an appropriate parameter dict.\"\"\"\n        artifact_params = {}\n        artifact_params[\"zscore_thresh\"] = None  # must be None or &gt;= 0\n        artifact_params[\"amplitude_thresh\"] = 3000  # must be None or &gt;= 0\n        # all electrodes of sort group\n        artifact_params[\"proportion_above_thresh\"] = 1.0\n        artifact_params[\"removal_window_ms\"] = 1.0  # in milliseconds\n        self.insert1([\"default\", artifact_params], skip_duplicates=True)\n\n        artifact_params_none = {}\n        artifact_params_none[\"zscore_thresh\"] = None\n        artifact_params_none[\"amplitude_thresh\"] = None\n        self.insert1([\"none\", artifact_params_none], skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_artifact/#src.spyglass.spikesorting.v0.spikesorting_artifact.ArtifactDetectionParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default artifact parameters with an appropriate parameter dict.</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_artifact.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default artifact parameters with an appropriate parameter dict.\"\"\"\n    artifact_params = {}\n    artifact_params[\"zscore_thresh\"] = None  # must be None or &gt;= 0\n    artifact_params[\"amplitude_thresh\"] = 3000  # must be None or &gt;= 0\n    # all electrodes of sort group\n    artifact_params[\"proportion_above_thresh\"] = 1.0\n    artifact_params[\"removal_window_ms\"] = 1.0  # in milliseconds\n    self.insert1([\"default\", artifact_params], skip_duplicates=True)\n\n    artifact_params_none = {}\n    artifact_params_none[\"zscore_thresh\"] = None\n    artifact_params_none[\"amplitude_thresh\"] = None\n    self.insert1([\"none\", artifact_params_none], skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/", "title": "spikesorting_curation.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation", "title": "<code>Curation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass Curation(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Stores each spike sorting; similar to IntervalList\n    curation_id: int # a number corresponding to the index of this curation\n    -&gt; SpikeSorting\n    ---\n    parent_curation_id=-1: int\n    curation_labels: blob # a dictionary of labels for the units\n    merge_groups: blob # a list of merge groups for the units\n    quality_metrics: blob # a list of quality metrics for the units (if available)\n    description='': varchar(1000) #optional description for this curated sort\n    time_of_creation: int   # in Unix time, to the nearest second\n    \"\"\"\n\n    _nwb_table = AnalysisNwbfile\n\n    @staticmethod\n    def insert_curation(\n        sorting_key: dict,\n        parent_curation_id: int = -1,\n        labels=None,\n        merge_groups=None,\n        metrics=None,\n        description=\"\",\n    ):\n        \"\"\"Given a SpikeSorting key and the parent_sorting_id (and optional\n        arguments) insert an entry into Curation.\n\n\n        Parameters\n        ----------\n        sorting_key : dict\n            The key for the original SpikeSorting\n        parent_curation_id : int, optional\n            The id of the parent sorting\n        labels : dict or None, optional\n        merge_groups : dict or None, optional\n        metrics : dict or None, optional\n            Computed metrics for sorting\n        description : str, optional\n            text description of this sort\n\n        Returns\n        -------\n        curation_key : dict\n\n        \"\"\"\n        if parent_curation_id == -1:\n            # check to see if this sorting with a parent of -1 has already been\n            # inserted and if so, warn the user\n            inserted_curation = (Curation &amp; sorting_key).fetch(\"KEY\")\n            if len(inserted_curation) &gt; 0:\n                Warning(\n                    \"Sorting has already been inserted, returning key to previously\"\n                    \"inserted curation\"\n                )\n                return inserted_curation[0]\n\n        if labels is None:\n            labels = {}\n        if merge_groups is None:\n            merge_groups = []\n        if metrics is None:\n            metrics = {}\n\n        # generate a unique number for this curation\n        id = (Curation &amp; sorting_key).fetch(\"curation_id\")\n        if len(id) &gt; 0:\n            curation_id = max(id) + 1\n        else:\n            curation_id = 0\n\n        # convert unit_ids in labels to integers for labels from sortingview.\n        new_labels = {int(unit_id): labels[unit_id] for unit_id in labels}\n\n        sorting_key[\"curation_id\"] = curation_id\n        sorting_key[\"parent_curation_id\"] = parent_curation_id\n        sorting_key[\"description\"] = description\n        sorting_key[\"curation_labels\"] = new_labels\n        sorting_key[\"merge_groups\"] = merge_groups\n        sorting_key[\"quality_metrics\"] = metrics\n        sorting_key[\"time_of_creation\"] = int(time.time())\n\n        # mike: added skip duplicates\n        Curation.insert1(sorting_key, skip_duplicates=True)\n\n        # get the primary key for this curation\n        c_key = Curation.fetch(\"KEY\")[0]\n        curation_key = {item: sorting_key[item] for item in c_key}\n\n        return curation_key\n\n    @staticmethod\n    def get_recording(key: dict):\n        \"\"\"Returns the recording extractor for the recording related to this curation\n\n        Parameters\n        ----------\n        key : dict\n            SpikeSortingRecording key\n\n        Returns\n        -------\n        recording_extractor : spike interface recording extractor\n\n        \"\"\"\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        return si.load_extractor(recording_path)\n\n    @staticmethod\n    def get_curated_sorting(key: dict):\n        \"\"\"Returns the sorting extractor related to this curation,\n        with merges applied.\n\n        Parameters\n        ----------\n        key : dict\n            Curation key\n\n        Returns\n        -------\n        sorting_extractor: spike interface sorting extractor\n\n        \"\"\"\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        sorting = si.load_extractor(sorting_path)\n        merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n        # TODO: write code to get merged sorting extractor\n        if len(merge_groups) != 0:\n            return MergedSortingExtractor(\n                parent_sorting=sorting, merge_groups=merge_groups\n            )\n        else:\n            return sorting\n\n    @staticmethod\n    def save_sorting_nwb(\n        key,\n        sorting,\n        timestamps,\n        sort_interval_list_name,\n        sort_interval,\n        labels=None,\n        metrics=None,\n        unit_ids=None,\n    ):\n        \"\"\"Store a sorting in a new AnalysisNwbfile\n\n        Parameters\n        ----------\n        key : dict\n            key to SpikeSorting table\n        sorting : si.Sorting\n            sorting\n        timestamps : array_like\n            Time stamps of the sorted recoridng;\n            used to convert the spike timings from index to real time\n        sort_interval_list_name : str\n            name of sort interval\n        sort_interval : list\n            interval for start and end of sort\n        labels : dict, optional\n            curation labels, by default None\n        metrics : dict, optional\n            quality metrics, by default None\n        unit_ids : list, optional\n            IDs of units whose spiketrains to save, by default None\n\n        Returns\n        -------\n        analysis_file_name : str\n        units_object_id : str\n\n        \"\"\"\n\n        sort_interval_valid_times = (\n            IntervalList &amp; {\"interval_list_name\": sort_interval_list_name}\n        ).fetch1(\"valid_times\")\n\n        units = dict()\n        units_valid_times = dict()\n        units_sort_interval = dict()\n\n        if unit_ids is None:\n            unit_ids = sorting.get_unit_ids()\n\n        for unit_id in unit_ids:\n            spike_times_in_samples = sorting.get_unit_spike_train(\n                unit_id=unit_id\n            )\n            units[unit_id] = timestamps[spike_times_in_samples]\n            units_valid_times[unit_id] = sort_interval_valid_times\n            units_sort_interval[unit_id] = [sort_interval]\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n        object_ids = AnalysisNwbfile().add_units(\n            analysis_file_name,\n            units,\n            units_valid_times,\n            units_sort_interval,\n            metrics=metrics,\n            labels=labels,\n        )\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        if object_ids == \"\":\n            logger.warn(\n                \"Sorting contains no units.\"\n                \"Created an empty analysis nwb file anyway.\"\n            )\n            units_object_id = \"\"\n        else:\n            units_object_id = object_ids[0]\n\n        return analysis_file_name, units_object_id\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.insert_curation", "title": "<code>insert_curation(sorting_key, parent_curation_id=-1, labels=None, merge_groups=None, metrics=None, description='')</code>  <code>staticmethod</code>", "text": "<p>Given a SpikeSorting key and the parent_sorting_id (and optional arguments) insert an entry into Curation.</p> <p>Parameters:</p> Name Type Description Default <code>sorting_key</code> <code>dict</code> <p>The key for the original SpikeSorting</p> required <code>parent_curation_id</code> <code>int</code> <p>The id of the parent sorting</p> <code>-1</code> <code>labels</code> <code>dict or None</code> <code>None</code> <code>merge_groups</code> <code>dict or None</code> <code>None</code> <code>metrics</code> <code>dict or None</code> <p>Computed metrics for sorting</p> <code>None</code> <code>description</code> <code>str</code> <p>text description of this sort</p> <code>''</code> <p>Returns:</p> Name Type Description <code>curation_key</code> <code>dict</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef insert_curation(\n    sorting_key: dict,\n    parent_curation_id: int = -1,\n    labels=None,\n    merge_groups=None,\n    metrics=None,\n    description=\"\",\n):\n    \"\"\"Given a SpikeSorting key and the parent_sorting_id (and optional\n    arguments) insert an entry into Curation.\n\n\n    Parameters\n    ----------\n    sorting_key : dict\n        The key for the original SpikeSorting\n    parent_curation_id : int, optional\n        The id of the parent sorting\n    labels : dict or None, optional\n    merge_groups : dict or None, optional\n    metrics : dict or None, optional\n        Computed metrics for sorting\n    description : str, optional\n        text description of this sort\n\n    Returns\n    -------\n    curation_key : dict\n\n    \"\"\"\n    if parent_curation_id == -1:\n        # check to see if this sorting with a parent of -1 has already been\n        # inserted and if so, warn the user\n        inserted_curation = (Curation &amp; sorting_key).fetch(\"KEY\")\n        if len(inserted_curation) &gt; 0:\n            Warning(\n                \"Sorting has already been inserted, returning key to previously\"\n                \"inserted curation\"\n            )\n            return inserted_curation[0]\n\n    if labels is None:\n        labels = {}\n    if merge_groups is None:\n        merge_groups = []\n    if metrics is None:\n        metrics = {}\n\n    # generate a unique number for this curation\n    id = (Curation &amp; sorting_key).fetch(\"curation_id\")\n    if len(id) &gt; 0:\n        curation_id = max(id) + 1\n    else:\n        curation_id = 0\n\n    # convert unit_ids in labels to integers for labels from sortingview.\n    new_labels = {int(unit_id): labels[unit_id] for unit_id in labels}\n\n    sorting_key[\"curation_id\"] = curation_id\n    sorting_key[\"parent_curation_id\"] = parent_curation_id\n    sorting_key[\"description\"] = description\n    sorting_key[\"curation_labels\"] = new_labels\n    sorting_key[\"merge_groups\"] = merge_groups\n    sorting_key[\"quality_metrics\"] = metrics\n    sorting_key[\"time_of_creation\"] = int(time.time())\n\n    # mike: added skip duplicates\n    Curation.insert1(sorting_key, skip_duplicates=True)\n\n    # get the primary key for this curation\n    c_key = Curation.fetch(\"KEY\")[0]\n    curation_key = {item: sorting_key[item] for item in c_key}\n\n    return curation_key\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.get_recording", "title": "<code>get_recording(key)</code>  <code>staticmethod</code>", "text": "<p>Returns the recording extractor for the recording related to this curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>SpikeSortingRecording key</p> required <p>Returns:</p> Name Type Description <code>recording_extractor</code> <code>spike interface recording extractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_recording(key: dict):\n    \"\"\"Returns the recording extractor for the recording related to this curation\n\n    Parameters\n    ----------\n    key : dict\n        SpikeSortingRecording key\n\n    Returns\n    -------\n    recording_extractor : spike interface recording extractor\n\n    \"\"\"\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    return si.load_extractor(recording_path)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.get_curated_sorting", "title": "<code>get_curated_sorting(key)</code>  <code>staticmethod</code>", "text": "<p>Returns the sorting extractor related to this curation, with merges applied.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Curation key</p> required <p>Returns:</p> Name Type Description <code>sorting_extractor</code> <code>spike interface sorting extractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_curated_sorting(key: dict):\n    \"\"\"Returns the sorting extractor related to this curation,\n    with merges applied.\n\n    Parameters\n    ----------\n    key : dict\n        Curation key\n\n    Returns\n    -------\n    sorting_extractor: spike interface sorting extractor\n\n    \"\"\"\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    sorting = si.load_extractor(sorting_path)\n    merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n    # TODO: write code to get merged sorting extractor\n    if len(merge_groups) != 0:\n        return MergedSortingExtractor(\n            parent_sorting=sorting, merge_groups=merge_groups\n        )\n    else:\n        return sorting\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.save_sorting_nwb", "title": "<code>save_sorting_nwb(key, sorting, timestamps, sort_interval_list_name, sort_interval, labels=None, metrics=None, unit_ids=None)</code>  <code>staticmethod</code>", "text": "<p>Store a sorting in a new AnalysisNwbfile</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key to SpikeSorting table</p> required <code>sorting</code> <code>Sorting</code> <p>sorting</p> required <code>timestamps</code> <code>array_like</code> <p>Time stamps of the sorted recoridng; used to convert the spike timings from index to real time</p> required <code>sort_interval_list_name</code> <code>str</code> <p>name of sort interval</p> required <code>sort_interval</code> <code>list</code> <p>interval for start and end of sort</p> required <code>labels</code> <code>dict</code> <p>curation labels, by default None</p> <code>None</code> <code>metrics</code> <code>dict</code> <p>quality metrics, by default None</p> <code>None</code> <code>unit_ids</code> <code>list</code> <p>IDs of units whose spiketrains to save, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <code>units_object_id</code> <code>str</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef save_sorting_nwb(\n    key,\n    sorting,\n    timestamps,\n    sort_interval_list_name,\n    sort_interval,\n    labels=None,\n    metrics=None,\n    unit_ids=None,\n):\n    \"\"\"Store a sorting in a new AnalysisNwbfile\n\n    Parameters\n    ----------\n    key : dict\n        key to SpikeSorting table\n    sorting : si.Sorting\n        sorting\n    timestamps : array_like\n        Time stamps of the sorted recoridng;\n        used to convert the spike timings from index to real time\n    sort_interval_list_name : str\n        name of sort interval\n    sort_interval : list\n        interval for start and end of sort\n    labels : dict, optional\n        curation labels, by default None\n    metrics : dict, optional\n        quality metrics, by default None\n    unit_ids : list, optional\n        IDs of units whose spiketrains to save, by default None\n\n    Returns\n    -------\n    analysis_file_name : str\n    units_object_id : str\n\n    \"\"\"\n\n    sort_interval_valid_times = (\n        IntervalList &amp; {\"interval_list_name\": sort_interval_list_name}\n    ).fetch1(\"valid_times\")\n\n    units = dict()\n    units_valid_times = dict()\n    units_sort_interval = dict()\n\n    if unit_ids is None:\n        unit_ids = sorting.get_unit_ids()\n\n    for unit_id in unit_ids:\n        spike_times_in_samples = sorting.get_unit_spike_train(\n            unit_id=unit_id\n        )\n        units[unit_id] = timestamps[spike_times_in_samples]\n        units_valid_times[unit_id] = sort_interval_valid_times\n        units_sort_interval[unit_id] = [sort_interval]\n\n    analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n    object_ids = AnalysisNwbfile().add_units(\n        analysis_file_name,\n        units,\n        units_valid_times,\n        units_sort_interval,\n        metrics=metrics,\n        labels=labels,\n    )\n    AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n    if object_ids == \"\":\n        logger.warn(\n            \"Sorting contains no units.\"\n            \"Created an empty analysis nwb file anyway.\"\n        )\n        units_object_id = \"\"\n    else:\n        units_object_id = object_ids[0]\n\n    return analysis_file_name, units_object_id\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Waveforms", "title": "<code>Waveforms</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass Waveforms(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; WaveformSelection\n    ---\n    waveform_extractor_path: varchar(400)\n    -&gt; AnalysisNwbfile\n    waveforms_object_id: varchar(40)   # Object ID for the waveforms in NWB file\n    \"\"\"\n\n    def make(self, key):\n        recording = Curation.get_recording(key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n\n        sorting = Curation.get_curated_sorting(key)\n\n        logger.info(\"Extracting waveforms...\")\n        waveform_params = (WaveformParameters &amp; key).fetch1(\"waveform_params\")\n        if \"whiten\" in waveform_params:\n            if waveform_params.pop(\"whiten\"):\n                recording = sip.whiten(recording, dtype=\"float32\")\n\n        waveform_extractor_name = self._get_waveform_extractor_name(key)\n        key[\"waveform_extractor_path\"] = str(\n            Path(waveforms_dir) / Path(waveform_extractor_name)\n        )\n        if os.path.exists(key[\"waveform_extractor_path\"]):\n            shutil.rmtree(key[\"waveform_extractor_path\"])\n        waveforms = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=key[\"waveform_extractor_path\"],\n            **waveform_params,\n        )\n\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        object_id = AnalysisNwbfile().add_units_waveforms(\n            key[\"analysis_file_name\"], waveform_extractor=waveforms\n        )\n        key[\"waveforms_object_id\"] = object_id\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], key[\"analysis_file_name\"])\n\n        self.insert1(key)\n\n    def load_waveforms(self, key: dict):\n        \"\"\"Returns a spikeinterface waveform extractor specified by key\n\n        Parameters\n        ----------\n        key : dict\n            Could be an entry in Waveforms, or some other key that uniquely defines\n            an entry in Waveforms\n\n        Returns\n        -------\n        we : spikeinterface.WaveformExtractor\n        \"\"\"\n        we_path = (self &amp; key).fetch1(\"waveform_extractor_path\")\n        we = si.WaveformExtractor.load_from_folder(we_path)\n        return we\n\n    def fetch_nwb(self, key):\n        # TODO: implement fetching waveforms from NWB\n        return NotImplementedError\n\n    def _get_waveform_extractor_name(self, key):\n        waveform_params_name = (WaveformParameters &amp; key).fetch1(\n            \"waveform_params_name\"\n        )\n\n        return (\n            f'{key[\"nwb_file_name\"]}_{str(uuid.uuid4())[0:8]}_'\n            f'{key[\"curation_id\"]}_{waveform_params_name}_waveforms'\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Waveforms.load_waveforms", "title": "<code>load_waveforms(key)</code>", "text": "<p>Returns a spikeinterface waveform extractor specified by key</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Could be an entry in Waveforms, or some other key that uniquely defines an entry in Waveforms</p> required <p>Returns:</p> Name Type Description <code>we</code> <code>WaveformExtractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def load_waveforms(self, key: dict):\n    \"\"\"Returns a spikeinterface waveform extractor specified by key\n\n    Parameters\n    ----------\n    key : dict\n        Could be an entry in Waveforms, or some other key that uniquely defines\n        an entry in Waveforms\n\n    Returns\n    -------\n    we : spikeinterface.WaveformExtractor\n    \"\"\"\n    we_path = (self &amp; key).fetch1(\"waveform_extractor_path\")\n    we = si.WaveformExtractor.load_from_folder(we_path)\n    return we\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.MetricParameters", "title": "<code>MetricParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass MetricParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for computing quality metrics of sorted units\n    metric_params_name: varchar(64)\n    ---\n    metric_params: blob\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    metric_default_params = {\n        \"snr\": {\n            \"peak_sign\": \"neg\",\n            \"random_chunk_kwargs_dict\": {\n                \"num_chunks_per_segment\": 20,\n                \"chunk_size\": 10000,\n                \"seed\": 0,\n            },\n        },\n        \"isi_violation\": {\"isi_threshold_ms\": 1.5, \"min_isi_ms\": 0.0},\n        \"nn_isolation\": {\n            \"max_spikes\": 1000,\n            \"min_spikes\": 10,\n            \"n_neighbors\": 5,\n            \"n_components\": 7,\n            \"radius_um\": 100,\n            \"seed\": 0,\n        },\n        \"nn_noise_overlap\": {\n            \"max_spikes\": 1000,\n            \"min_spikes\": 10,\n            \"n_neighbors\": 5,\n            \"n_components\": 7,\n            \"radius_um\": 100,\n            \"seed\": 0,\n        },\n        \"peak_channel\": {\"peak_sign\": \"neg\"},\n        \"num_spikes\": {},\n    }\n    # Example of peak_offset parameters 'peak_offset': {'peak_sign': 'neg'}\n    available_metrics = [\n        \"snr\",\n        \"isi_violation\",\n        \"nn_isolation\",\n        \"nn_noise_overlap\",\n        \"peak_offset\",\n        \"peak_channel\",\n        \"num_spikes\",\n    ]\n\n    def get_metric_default_params(self, metric: str):\n        \"Returns default params for the given metric\"\n        return self.metric_default_params(metric)\n\n    def insert_default(self):\n        self.insert1(\n            [\"franklab_default3\", self.metric_default_params],\n            skip_duplicates=True,\n        )\n\n    def get_available_metrics(self):\n        for metric in _metric_name_to_func:\n            if metric in self.available_metrics:\n                metric_doc = _metric_name_to_func[metric].__doc__.split(\"\\n\")[0]\n                metric_string = (\"{metric_name} : {metric_doc}\").format(\n                    metric_name=metric, metric_doc=metric_doc\n                )\n                logger.info(metric_string + \"\\n\")\n\n    # TODO\n    def _validate_metrics_list(self, key):\n        \"\"\"Checks whether a row to be inserted contains only the available metrics\"\"\"\n        # get available metrics list\n        # get metric list from key\n        # compare\n        return NotImplementedError\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.MetricParameters.get_metric_default_params", "title": "<code>get_metric_default_params(metric)</code>", "text": "<p>Returns default params for the given metric</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def get_metric_default_params(self, metric: str):\n    \"Returns default params for the given metric\"\n    return self.metric_default_params(metric)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration", "title": "<code>AutomaticCuration</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass AutomaticCuration(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; AutomaticCurationSelection\n    ---\n    auto_curation_key: blob # the key to the curation inserted by make\n    \"\"\"\n\n    def make(self, key):\n        metrics_path = (QualityMetrics &amp; key).fetch1(\"quality_metrics_path\")\n        with open(metrics_path) as f:\n            quality_metrics = json.load(f)\n\n        # get the curation information and the curated sorting\n        parent_curation = (Curation &amp; key).fetch(as_dict=True)[0]\n        parent_merge_groups = parent_curation[\"merge_groups\"]\n        parent_labels = parent_curation[\"curation_labels\"]\n        parent_curation_id = parent_curation[\"curation_id\"]\n        parent_sorting = Curation.get_curated_sorting(key)\n\n        merge_params = (AutomaticCurationParameters &amp; key).fetch1(\n            \"merge_params\"\n        )\n        merge_groups, units_merged = self.get_merge_groups(\n            parent_sorting, parent_merge_groups, quality_metrics, merge_params\n        )\n\n        label_params = (AutomaticCurationParameters &amp; key).fetch1(\n            \"label_params\"\n        )\n        labels = self.get_labels(\n            parent_sorting, parent_labels, quality_metrics, label_params\n        )\n\n        # keep the quality metrics only if no merging occurred.\n        metrics = quality_metrics if not units_merged else None\n\n        # insert this sorting into the CuratedSpikeSorting Table\n        # first remove keys that aren't part of the Sorting (the primary key of curation)\n        c_key = (SpikeSorting &amp; key).fetch(\"KEY\")[0]\n        curation_key = {item: key[item] for item in key if item in c_key}\n        key[\"auto_curation_key\"] = Curation.insert_curation(\n            curation_key,\n            parent_curation_id=parent_curation_id,\n            labels=labels,\n            merge_groups=merge_groups,\n            metrics=metrics,\n            description=\"auto curated\",\n        )\n\n        self.insert1(key)\n\n    @staticmethod\n    def get_merge_groups(\n        sorting, parent_merge_groups, quality_metrics, merge_params\n    ):\n        \"\"\"Identifies units to be merged based on the quality_metrics and\n        merge parameters and returns an updated list of merges for the curation.\n\n        Parameters\n        ---------\n        sorting : spikeinterface.sorting\n        parent_merge_groups : list\n            Information about previous merges\n        quality_metrics : list\n        merge_params : dict\n\n        Returns\n        -------\n        merge_groups : list of lists\n        merge_occurred : bool\n\n        \"\"\"\n\n        # overview:\n        # 1. Use quality metrics to determine merge groups for units\n        # 2. Combine merge groups with current merge groups to produce union of merges\n\n        if not merge_params:\n            return parent_merge_groups, False\n        else:\n            # TODO: use the metrics to identify clusters that should be merged\n            # new_merges should then reflect those merges and the line below should be deleted.\n            new_merges = []\n            # append these merges to the parent merge_groups\n            for new_merge in new_merges:\n                # check to see if the first cluster listed is in a current merge group\n                for previous_merge in parent_merge_groups:\n                    if new_merge[0] == previous_merge[0]:\n                        # add the additional units in new_merge to the identified merge group.\n                        previous_merge.extend(new_merge[1:])\n                        previous_merge.sort()\n                        break\n                else:\n                    # append this merge group to the list if no previous merge\n                    parent_merge_groups.append(new_merge)\n            return parent_merge_groups.sort(), True\n\n    @staticmethod\n    def get_labels(sorting, parent_labels, quality_metrics, label_params):\n        \"\"\"Returns a dictionary of labels using quality_metrics and label\n        parameters.\n\n        Parameters\n        ---------\n        sorting : spikeinterface.sorting\n        parent_labels : list\n            Information about previous merges\n        quality_metrics : list\n        label_params : dict\n\n        Returns\n        -------\n        parent_labels : list\n\n        \"\"\"\n        # overview:\n        # 1. Use quality metrics to determine labels for units\n        # 2. Append labels to current labels, checking for inconsistencies\n        if not label_params:\n            return parent_labels\n        else:\n            for metric in label_params:\n                if metric not in quality_metrics:\n                    Warning(f\"{metric} not found in quality metrics; skipping\")\n                else:\n                    compare = _comparison_to_function[label_params[metric][0]]\n\n                    for unit_id in quality_metrics[metric].keys():\n                        # compare the quality metric to the threshold with the specified operator\n                        # note that label_params[metric] is a three element list with a comparison operator as a string,\n                        # the threshold value, and a list of labels to be applied if the comparison is true\n                        if compare(\n                            quality_metrics[metric][unit_id],\n                            label_params[metric][1],\n                        ):\n                            if unit_id not in parent_labels:\n                                parent_labels[unit_id] = label_params[metric][2]\n                            # check if the label is already there, and if not, add it\n                            elif (\n                                label_params[metric][2]\n                                not in parent_labels[unit_id]\n                            ):\n                                parent_labels[unit_id].extend(\n                                    label_params[metric][2]\n                                )\n            return parent_labels\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration.get_merge_groups", "title": "<code>get_merge_groups(sorting, parent_merge_groups, quality_metrics, merge_params)</code>  <code>staticmethod</code>", "text": "<p>Identifies units to be merged based on the quality_metrics and merge parameters and returns an updated list of merges for the curation.</p> <p>Parameters:</p> Name Type Description Default <code>sorting</code> <code>sorting</code> required <code>parent_merge_groups</code> <code>list</code> <p>Information about previous merges</p> required <code>quality_metrics</code> <code>list</code> required <code>merge_params</code> <code>dict</code> required <p>Returns:</p> Name Type Description <code>merge_groups</code> <code>list of lists</code> <code>merge_occurred</code> <code>bool</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_merge_groups(\n    sorting, parent_merge_groups, quality_metrics, merge_params\n):\n    \"\"\"Identifies units to be merged based on the quality_metrics and\n    merge parameters and returns an updated list of merges for the curation.\n\n    Parameters\n    ---------\n    sorting : spikeinterface.sorting\n    parent_merge_groups : list\n        Information about previous merges\n    quality_metrics : list\n    merge_params : dict\n\n    Returns\n    -------\n    merge_groups : list of lists\n    merge_occurred : bool\n\n    \"\"\"\n\n    # overview:\n    # 1. Use quality metrics to determine merge groups for units\n    # 2. Combine merge groups with current merge groups to produce union of merges\n\n    if not merge_params:\n        return parent_merge_groups, False\n    else:\n        # TODO: use the metrics to identify clusters that should be merged\n        # new_merges should then reflect those merges and the line below should be deleted.\n        new_merges = []\n        # append these merges to the parent merge_groups\n        for new_merge in new_merges:\n            # check to see if the first cluster listed is in a current merge group\n            for previous_merge in parent_merge_groups:\n                if new_merge[0] == previous_merge[0]:\n                    # add the additional units in new_merge to the identified merge group.\n                    previous_merge.extend(new_merge[1:])\n                    previous_merge.sort()\n                    break\n            else:\n                # append this merge group to the list if no previous merge\n                parent_merge_groups.append(new_merge)\n        return parent_merge_groups.sort(), True\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration.get_labels", "title": "<code>get_labels(sorting, parent_labels, quality_metrics, label_params)</code>  <code>staticmethod</code>", "text": "<p>Returns a dictionary of labels using quality_metrics and label parameters.</p> <p>Parameters:</p> Name Type Description Default <code>sorting</code> <code>sorting</code> required <code>parent_labels</code> <code>list</code> <p>Information about previous merges</p> required <code>quality_metrics</code> <code>list</code> required <code>label_params</code> <code>dict</code> required <p>Returns:</p> Name Type Description <code>parent_labels</code> <code>list</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_labels(sorting, parent_labels, quality_metrics, label_params):\n    \"\"\"Returns a dictionary of labels using quality_metrics and label\n    parameters.\n\n    Parameters\n    ---------\n    sorting : spikeinterface.sorting\n    parent_labels : list\n        Information about previous merges\n    quality_metrics : list\n    label_params : dict\n\n    Returns\n    -------\n    parent_labels : list\n\n    \"\"\"\n    # overview:\n    # 1. Use quality metrics to determine labels for units\n    # 2. Append labels to current labels, checking for inconsistencies\n    if not label_params:\n        return parent_labels\n    else:\n        for metric in label_params:\n            if metric not in quality_metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n            else:\n                compare = _comparison_to_function[label_params[metric][0]]\n\n                for unit_id in quality_metrics[metric].keys():\n                    # compare the quality metric to the threshold with the specified operator\n                    # note that label_params[metric] is a three element list with a comparison operator as a string,\n                    # the threshold value, and a list of labels to be applied if the comparison is true\n                    if compare(\n                        quality_metrics[metric][unit_id],\n                        label_params[metric][1],\n                    ):\n                        if unit_id not in parent_labels:\n                            parent_labels[unit_id] = label_params[metric][2]\n                        # check if the label is already there, and if not, add it\n                        elif (\n                            label_params[metric][2]\n                            not in parent_labels[unit_id]\n                        ):\n                            parent_labels[unit_id].extend(\n                                label_params[metric][2]\n                            )\n        return parent_labels\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting", "title": "<code>CuratedSpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass CuratedSpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; CuratedSpikeSortingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    units_object_id: varchar(40)\n    \"\"\"\n\n    class Unit(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Table for holding sorted units\n        -&gt; CuratedSpikeSorting\n        unit_id: int   # ID for each unit\n        ---\n        label='': varchar(200)   # optional set of labels for each unit\n        nn_noise_overlap=-1: float   # noise overlap metric for each unit\n        nn_isolation=-1: float   # isolation score metric for each unit\n        isi_violation=-1: float   # ISI violation score for each unit\n        snr=0: float            # SNR for each unit\n        firing_rate=-1: float   # firing rate\n        num_spikes=-1: int   # total number of spikes\n        peak_channel=null: int # channel of maximum amplitude for each unit\n        \"\"\"\n\n    def make(self, key):\n        unit_labels_to_remove = [\"reject\"]\n        # check that the Curation has metrics\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n        if metrics == {}:\n            Warning(\n                f\"Metrics for Curation {key} should normally be calculated before insertion here\"\n            )\n\n        sorting = Curation.get_curated_sorting(key)\n        unit_ids = sorting.get_unit_ids()\n        # Get the labels for the units, add only those units that do not have 'reject' or 'noise' labels\n        unit_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        accepted_units = []\n        for unit_id in unit_ids:\n            if unit_id in unit_labels:\n                if (\n                    len(set(unit_labels_to_remove) &amp; set(unit_labels[unit_id]))\n                    == 0\n                ):\n                    accepted_units.append(unit_id)\n            else:\n                accepted_units.append(unit_id)\n\n        # get the labels for the accepted units\n        labels = {}\n        for unit_id in accepted_units:\n            if unit_id in unit_labels:\n                labels[unit_id] = \",\".join(unit_labels[unit_id])\n\n        # convert unit_ids in metrics to integers, including only accepted units.\n        #  TODO: convert to int this somewhere else\n        final_metrics = {}\n        for metric in metrics:\n            final_metrics[metric] = {\n                int(unit_id): metrics[metric][unit_id]\n                for unit_id in metrics[metric]\n                if int(unit_id) in accepted_units\n            }\n\n        logger.info(f\"Found {len(accepted_units)} accepted units\")\n\n        # get the sorting and save it in the NWB file\n        sorting = Curation.get_curated_sorting(key)\n        recording = Curation.get_recording(key)\n\n        # get the sort_interval and sorting interval list\n        sort_interval = (SortInterval &amp; key).fetch1(\"sort_interval\")\n        sort_interval_list_name = (SpikeSorting &amp; key).fetch1(\n            \"artifact_removed_interval_list_name\"\n        )\n\n        timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n\n        (\n            key[\"analysis_file_name\"],\n            key[\"units_object_id\"],\n        ) = Curation().save_sorting_nwb(\n            key,\n            sorting,\n            timestamps,\n            sort_interval_list_name,\n            sort_interval,\n            metrics=final_metrics,\n            unit_ids=accepted_units,\n            labels=labels,\n        )\n        self.insert1(key)\n\n        # now add the units\n        # Remove the non primary key entries.\n        del key[\"units_object_id\"]\n        del key[\"analysis_file_name\"]\n\n        metric_fields = self.metrics_fields()\n        for unit_id in accepted_units:\n            key[\"unit_id\"] = unit_id\n            if unit_id in labels:\n                key[\"label\"] = labels[unit_id]\n            for field in metric_fields:\n                if field in final_metrics:\n                    key[field] = final_metrics[field][unit_id]\n                else:\n                    Warning(\n                        f\"No metric named {field} in computed unit quality metrics; skipping\"\n                    )\n            CuratedSpikeSorting.Unit.insert1(key)\n\n    def metrics_fields(self):\n        \"\"\"Returns a list of the metrics that are currently in the Units table.\"\"\"\n        unit_info = self.Unit().fetch(limit=1, format=\"frame\")\n        unit_fields = [column for column in unit_info.columns]\n        unit_fields.remove(\"label\")\n        return unit_fields\n\n    @classmethod\n    def get_recording(cls, key):\n        \"\"\"Returns the recording related to this curation. Useful for operations downstream of merge table\"\"\"\n        # expand the key\n        recording_key = (cls &amp; key).fetch1(\"KEY\")\n        return SpikeSortingRecording()._get_filtered_recording(recording_key)\n\n    @classmethod\n    def get_sorting(cls, key):\n        \"\"\"Returns the sorting related to this curation. Useful for operations downstream of merge table\"\"\"\n        # expand the key\n        sorting_key = (cls &amp; key).fetch1(\"KEY\")\n        return Curation.get_curated_sorting(sorting_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.metrics_fields", "title": "<code>metrics_fields()</code>", "text": "<p>Returns a list of the metrics that are currently in the Units table.</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def metrics_fields(self):\n    \"\"\"Returns a list of the metrics that are currently in the Units table.\"\"\"\n    unit_info = self.Unit().fetch(limit=1, format=\"frame\")\n    unit_fields = [column for column in unit_info.columns]\n    unit_fields.remove(\"label\")\n    return unit_fields\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Returns the recording related to this curation. Useful for operations downstream of merge table</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@classmethod\ndef get_recording(cls, key):\n    \"\"\"Returns the recording related to this curation. Useful for operations downstream of merge table\"\"\"\n    # expand the key\n    recording_key = (cls &amp; key).fetch1(\"KEY\")\n    return SpikeSortingRecording()._get_filtered_recording(recording_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Returns the sorting related to this curation. Useful for operations downstream of merge table</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key):\n    \"\"\"Returns the sorting related to this curation. Useful for operations downstream of merge table\"\"\"\n    # expand the key\n    sorting_key = (cls &amp; key).fetch1(\"KEY\")\n    return Curation.get_curated_sorting(sorting_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.UnitInclusionParameters", "title": "<code>UnitInclusionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass UnitInclusionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    unit_inclusion_param_name: varchar(80) # the name of the list of thresholds for unit inclusion\n    ---\n    inclusion_param_dict: blob # the dictionary of inclusion / exclusion parameters\n    \"\"\"\n\n    def insert1(self, key, **kwargs):\n        # check to see that the dictionary fits the specifications\n        # The inclusion parameter dict has the following form:\n        # param_dict['metric_name'] = (operator, value)\n        #    where operator is '&lt;', '&gt;', &lt;=', '&gt;=', or '==' and value is the comparison (float) value to be used ()\n        # param_dict['exclude_labels'] = [list of labels to exclude]\n        pdict = key[\"inclusion_param_dict\"]\n        metrics_list = CuratedSpikeSorting().metrics_fields()\n\n        for k in pdict:\n            if k not in metrics_list and k != \"exclude_labels\":\n                raise Exception(\n                    f\"key {k} is not a valid element of the inclusion_param_dict\"\n                )\n            if k in metrics_list:\n                if pdict[k][0] not in _comparison_to_function:\n                    raise Exception(\n                        f\"operator {pdict[k][0]} for metric {k} is not in the valid operators list: {_comparison_to_function.keys()}\"\n                    )\n            if k == \"exclude_labels\":\n                for label in pdict[k]:\n                    if label not in valid_labels:\n                        raise Exception(\n                            f\"exclude label {label} is not in the valid_labels list: {valid_labels}\"\n                        )\n        super().insert1(key, **kwargs)\n\n    def get_included_units(\n        self, curated_sorting_key, unit_inclusion_param_name\n    ):\n        \"\"\"Given a reference to a set of curated sorting units and the name of\n        a unit inclusion parameter list, returns unit key\n\n        Parameters\n        ----------\n        curated_sorting_key : dict\n            key to select a set of curated sorting\n        unit_inclusion_param_name : str\n            name of a unit inclusion parameter entry\n\n        Returns\n        -------\n        dict\n            key to select all of the included units\n        \"\"\"\n        curated_sortings = (CuratedSpikeSorting() &amp; curated_sorting_key).fetch()\n        inc_param_dict = (\n            UnitInclusionParameters\n            &amp; {\"unit_inclusion_param_name\": unit_inclusion_param_name}\n        ).fetch1(\"inclusion_param_dict\")\n        units = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch()\n        units_key = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch(\n            \"KEY\"\n        )\n        # get the list of labels to exclude if there is one\n        if \"exclude_labels\" in inc_param_dict:\n            exclude_labels = inc_param_dict[\"exclude_labels\"]\n            del inc_param_dict[\"exclude_labels\"]\n        else:\n            exclude_labels = []\n\n        # create a list of the units to kepp.\n        keep = np.asarray([True] * len(units))\n        for metric in inc_param_dict:\n            # for all units, go through each metric, compare it to the value\n            # specified, and update the list to be kept\n            keep = np.logical_and(\n                keep,\n                _comparison_to_function[inc_param_dict[metric][0]](\n                    units[metric], inc_param_dict[metric][1]\n                ),\n            )\n\n        # now exclude by label if it is specified\n        if len(exclude_labels):\n            for unit_ind in np.ravel(np.argwhere(keep)):\n                labels = units[unit_ind][\"label\"].split(\",\")\n                for label in labels:\n                    if label in exclude_labels:\n                        keep[unit_ind] = False\n                        break\n\n        # return units that passed all of the tests\n        # TODO: Make this more efficient\n        return {i: units_key[i] for i in np.ravel(np.argwhere(keep))}\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.UnitInclusionParameters.get_included_units", "title": "<code>get_included_units(curated_sorting_key, unit_inclusion_param_name)</code>", "text": "<p>Given a reference to a set of curated sorting units and the name of a unit inclusion parameter list, returns unit key</p> <p>Parameters:</p> Name Type Description Default <code>curated_sorting_key</code> <code>dict</code> <p>key to select a set of curated sorting</p> required <code>unit_inclusion_param_name</code> <code>str</code> <p>name of a unit inclusion parameter entry</p> required <p>Returns:</p> Type Description <code>dict</code> <p>key to select all of the included units</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def get_included_units(\n    self, curated_sorting_key, unit_inclusion_param_name\n):\n    \"\"\"Given a reference to a set of curated sorting units and the name of\n    a unit inclusion parameter list, returns unit key\n\n    Parameters\n    ----------\n    curated_sorting_key : dict\n        key to select a set of curated sorting\n    unit_inclusion_param_name : str\n        name of a unit inclusion parameter entry\n\n    Returns\n    -------\n    dict\n        key to select all of the included units\n    \"\"\"\n    curated_sortings = (CuratedSpikeSorting() &amp; curated_sorting_key).fetch()\n    inc_param_dict = (\n        UnitInclusionParameters\n        &amp; {\"unit_inclusion_param_name\": unit_inclusion_param_name}\n    ).fetch1(\"inclusion_param_dict\")\n    units = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch()\n    units_key = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch(\n        \"KEY\"\n    )\n    # get the list of labels to exclude if there is one\n    if \"exclude_labels\" in inc_param_dict:\n        exclude_labels = inc_param_dict[\"exclude_labels\"]\n        del inc_param_dict[\"exclude_labels\"]\n    else:\n        exclude_labels = []\n\n    # create a list of the units to kepp.\n    keep = np.asarray([True] * len(units))\n    for metric in inc_param_dict:\n        # for all units, go through each metric, compare it to the value\n        # specified, and update the list to be kept\n        keep = np.logical_and(\n            keep,\n            _comparison_to_function[inc_param_dict[metric][0]](\n                units[metric], inc_param_dict[metric][1]\n            ),\n        )\n\n    # now exclude by label if it is specified\n    if len(exclude_labels):\n        for unit_ind in np.ravel(np.argwhere(keep)):\n            labels = units[unit_ind][\"label\"].split(\",\")\n            for label in labels:\n                if label in exclude_labels:\n                    keep[unit_ind] = False\n                    break\n\n    # return units that passed all of the tests\n    # TODO: Make this more efficient\n    return {i: units_key[i] for i in np.ravel(np.argwhere(keep))}\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_populator/", "title": "spikesorting_populator.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_populator/#src.spyglass.spikesorting.v0.spikesorting_populator.spikesorting_pipeline_populator", "title": "<code>spikesorting_pipeline_populator(nwb_file_name, team_name, fig_url_repo=None, interval_list_name=None, sort_interval_name=None, pipeline_parameters_name=None, probe_restriction={}, artifact_parameters='ampl_2000_prop_75', preproc_params_name='franklab_tetrode_hippocampus', sorter='mountainsort4', sorter_params_name='franklab_tetrode_hippocampus_30KHz_tmp', waveform_params_name='default_whitened', metric_params_name='peak_offest_num_spikes_2', auto_curation_params_name='mike_noise_03_offset_2_isi_0025_mua')</code>", "text": "<p>Automatically populate the spike sorting pipeline for a given epoch</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>Session ID</p> required <code>team_name</code> <code>str</code> <p>Which team to assign the spike sorting to</p> required <code>fig_url_repo</code> <code>str</code> <p>Where to store the curation figurl json files (e.g., 'gh://LorenFrankLab/sorting-curations/main/user/'). Default None to skip figurl</p> <code>None</code> <code>interval_list_name</code> <code>(str)</code> <p>if sort_interval_name not provided, will create a sort interval for the given interval with the same name</p> <code>None</code> <code>sort_interval_name</code> <code>str</code> <p>if provided, will use the given sort interval, requires making this interval yourself</p> <code>None</code> <code>pipeline_parameters_name</code> <code>str</code> <p>If provided, will lookup pipeline parameters from the SpikeSortingPipelineParameters table, supersedes other values provided, by default None</p> <code>None</code> <code>probe_restriction</code> <code>dict</code> <p>Restricts analysis to sort groups with matching keys. Can use keys from the SortGroup and ElectrodeGroup Tables (e.g. electrode_group_name, probe_id, target_hemisphere), by default {}</p> <code>{}</code> <code>artifact_parameters</code> <code>str</code> <p>parameter set for artifact detection, by default \"ampl_2000_prop_75\"</p> <code>'ampl_2000_prop_75'</code> <code>preproc_params_name</code> <code>str</code> <p>parameter set for spikesorting recording, by default \"franklab_tetrode_hippocampus\"</p> <code>'franklab_tetrode_hippocampus'</code> <code>sorter</code> <code>str</code> <p>which spikesorting algorithm to use, by default \"mountainsort4\"</p> <code>'mountainsort4'</code> <code>sorter_params_name</code> <code>str</code> <p>parameters for the spike sorting algorithm, by default \"franklab_tetrode_hippocampus_30KHz_tmp\"</p> <code>'franklab_tetrode_hippocampus_30KHz_tmp'</code> <code>waveform_params_name</code> <code>str</code> <p>Parameters for spike waveform extraction. If empty string, will skip automatic curation steps, by default \"default_whitened\"</p> <code>'default_whitened'</code> <code>metric_params_name</code> <code>str</code> <p>Parameters defining which QualityMetrics to calculate and how. If empty string, will skip automatic curation steps, by default \"peak_offest_num_spikes_2\"</p> <code>'peak_offest_num_spikes_2'</code> <code>auto_curation_params_name</code> <code>str</code> <p>Thresholds applied to Quality metrics for automatic unit curation. If empty string, will skip automatic curation steps, by default \"mike_noise_03_offset_2_isi_0025_mua\"</p> <code>'mike_noise_03_offset_2_isi_0025_mua'</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_populator.py</code> <pre><code>def spikesorting_pipeline_populator(\n    nwb_file_name: str,\n    team_name: str,\n    fig_url_repo: str = None,\n    interval_list_name: str = None,\n    sort_interval_name: str = None,\n    pipeline_parameters_name: str = None,\n    probe_restriction: dict = {},\n    artifact_parameters: str = \"ampl_2000_prop_75\",\n    preproc_params_name: str = \"franklab_tetrode_hippocampus\",\n    sorter: str = \"mountainsort4\",\n    sorter_params_name: str = \"franklab_tetrode_hippocampus_30KHz_tmp\",\n    waveform_params_name: str = \"default_whitened\",\n    metric_params_name: str = \"peak_offest_num_spikes_2\",\n    auto_curation_params_name: str = \"mike_noise_03_offset_2_isi_0025_mua\",\n):\n    \"\"\"Automatically populate the spike sorting pipeline for a given epoch\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        Session ID\n    team_name : str\n        Which team to assign the spike sorting to\n    fig_url_repo : str, optional\n        Where to store the curation figurl json files (e.g.,\n        'gh://LorenFrankLab/sorting-curations/main/user/'). Default None to\n        skip figurl\n    interval_list_name : str,\n        if sort_interval_name not provided, will create a sort interval for the\n        given interval with the same name\n    sort_interval_name : str, default None\n        if provided, will use the given sort interval, requires making this\n        interval yourself\n    pipeline_parameters_name : str, optional\n        If provided, will lookup pipeline parameters from the\n        SpikeSortingPipelineParameters table, supersedes other values provided,\n        by default None\n    probe_restriction : dict, optional\n        Restricts analysis to sort groups with matching keys. Can use keys from\n        the SortGroup and ElectrodeGroup Tables (e.g. electrode_group_name,\n        probe_id, target_hemisphere), by default {}\n    artifact_parameters : str, optional\n        parameter set for artifact detection, by default \"ampl_2000_prop_75\"\n    preproc_params_name : str, optional\n        parameter set for spikesorting recording, by default\n        \"franklab_tetrode_hippocampus\"\n    sorter : str, optional\n        which spikesorting algorithm to use, by default \"mountainsort4\"\n    sorter_params_name : str, optional\n        parameters for the spike sorting algorithm, by default\n        \"franklab_tetrode_hippocampus_30KHz_tmp\"\n    waveform_params_name : str, optional\n        Parameters for spike waveform extraction. If empty string, will skip\n        automatic curation steps, by default \"default_whitened\"\n    metric_params_name : str, optional\n        Parameters defining which QualityMetrics to calculate and how. If empty\n        string, will skip automatic curation steps, by default\n        \"peak_offest_num_spikes_2\"\n    auto_curation_params_name : str, optional\n        Thresholds applied to Quality metrics for automatic unit curation. If\n        empty string, will skip automatic curation steps, by default\n        \"mike_noise_03_offset_2_isi_0025_mua\"\n    \"\"\"\n    nwbf_dict = dict(nwb_file_name=nwb_file_name)\n    # Define pipeline parameters\n    if pipeline_parameters_name is not None:\n        logger.info(f\"Using pipeline parameters {pipeline_parameters_name}\")\n        (\n            artifact_parameters,\n            preproc_params_name,\n            sorter,\n            sorter_params_name,\n            waveform_params_name,\n            metric_params_name,\n            auto_curation_params_name,\n        ) = (\n            SpikeSortingPipelineParameters\n            &amp; {\"pipeline_parameters_name\": pipeline_parameters_name}\n        ).fetch1(\n            \"artifact_parameters\",\n            \"preproc_params_name\",\n            \"sorter\",\n            \"sorter_params_name\",\n            \"waveform_params_name\",\n            \"metric_params_name\",\n            \"auto_curation_params_name\",\n        )\n\n    # make sort groups only if not currently available\n    # don't overwrite existing ones!\n    if not SortGroup() &amp; nwbf_dict:\n        logger.info(\"Generating sort groups\")\n        SortGroup().set_group_by_shank(nwb_file_name)\n\n    # Define sort interval\n    interval_dict = dict(**nwbf_dict, interval_list_name=interval_list_name)\n\n    if sort_interval_name is not None:\n        logger.info(f\"Using sort interval {sort_interval_name}\")\n        if not (\n            SortInterval\n            &amp; nwbf_dict\n            &amp; {\"sort_interval_name\": sort_interval_name}\n        ):\n            raise KeyError(f\"Sort interval {sort_interval_name} not found\")\n    else:\n        logger.info(f\"Generating sort interval from {interval_list_name}\")\n        interval_list = (IntervalList &amp; interval_dict).fetch1(\"valid_times\")[0]\n\n        sort_interval_name = interval_list_name\n        sort_interval = interval_list\n\n        SortInterval.insert1(\n            {\n                **nwbf_dict,\n                \"sort_interval_name\": sort_interval_name,\n                \"sort_interval\": sort_interval,\n            },\n            skip_duplicates=True,\n        )\n\n    sort_dict = dict(**nwbf_dict, sort_interval_name=sort_interval_name)\n\n    # find desired sort group(s) for these settings\n    sort_group_id_list = (\n        (SortGroup.SortGroupElectrode * ElectrodeGroup)\n        &amp; nwbf_dict\n        &amp; probe_restriction\n    ).fetch(\"sort_group_id\")\n\n    # make spike sorting recording\n    logger.info(\"Generating spike sorting recording\")\n    for sort_group_id in sort_group_id_list:\n        ssr_key = dict(\n            **sort_dict,\n            sort_group_id=sort_group_id,  # See SortGroup\n            preproc_params_name=preproc_params_name,  # See preproc_params\n            interval_list_name=interval_list_name,\n            team_name=team_name,\n        )\n        SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True)\n\n    SpikeSortingRecording.populate(interval_dict)\n\n    # Artifact detection\n    logger.info(\"Running artifact detection\")\n    artifact_keys = [\n        {**k, \"artifact_params_name\": artifact_parameters}\n        for k in (SpikeSortingRecordingSelection() &amp; interval_dict).fetch(\"KEY\")\n    ]\n    ArtifactDetectionSelection().insert(artifact_keys, skip_duplicates=True)\n    ArtifactDetection.populate(interval_dict)\n\n    # Spike sorting\n    logger.info(\"Running spike sorting\")\n    for artifact_key in artifact_keys:\n        ss_key = dict(\n            **(ArtifactDetection &amp; artifact_key).fetch1(\"KEY\"),\n            **(ArtifactRemovedIntervalList() &amp; artifact_key).fetch1(\"KEY\"),\n            sorter=sorter,\n            sorter_params_name=sorter_params_name,\n        )\n        ss_key.pop(\"artifact_params_name\")\n        SpikeSortingSelection.insert1(ss_key, skip_duplicates=True)\n    SpikeSorting.populate(sort_dict)\n\n    # initial curation\n    logger.info(\"Beginning curation\")\n    for sorting_key in (SpikeSorting() &amp; sort_dict).fetch(\"KEY\"):\n        if not (Curation() &amp; sorting_key):\n            Curation.insert_curation(sorting_key)\n\n    # Calculate quality metrics and perform automatic curation if specified\n    if (\n        len(waveform_params_name) &gt; 0\n        and len(metric_params_name) &gt; 0\n        and len(auto_curation_params_name) &gt; 0\n    ):\n        # Extract waveforms\n        logger.info(\"Extracting waveforms\")\n        curation_keys = [\n            {**k, \"waveform_params_name\": waveform_params_name}\n            for k in (Curation() &amp; sort_dict &amp; {\"curation_id\": 0}).fetch(\"KEY\")\n        ]\n        WaveformSelection.insert(curation_keys, skip_duplicates=True)\n        Waveforms.populate(sort_dict)\n\n        # Quality Metrics\n        logger.info(\"Calculating quality metrics\")\n        waveform_keys = [\n            {**k, \"metric_params_name\": metric_params_name}\n            for k in (Waveforms() &amp; sort_dict).fetch(\"KEY\")\n        ]\n        MetricSelection.insert(waveform_keys, skip_duplicates=True)\n        QualityMetrics().populate(sort_dict)\n\n        # Automatic Curation\n        logger.info(\"Creating automatic curation\")\n        metric_keys = [\n            {**k, \"auto_curation_params_name\": auto_curation_params_name}\n            for k in (QualityMetrics() &amp; sort_dict).fetch(\"KEY\")\n        ]\n        AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True)\n        AutomaticCuration().populate(sort_dict)\n\n        # Curated Spike Sorting\n        # get curation keys of the automatic curation to populate into curated\n        # spike sorting selection\n        logger.info(\"Creating curated spike sorting\")\n        auto_key_list = (AutomaticCuration() &amp; sort_dict).fetch(\n            \"auto_curation_key\"\n        )\n        for auto_key in auto_key_list:\n            curation_auto_key = (Curation() &amp; auto_key).fetch1(\"KEY\")\n            CuratedSpikeSortingSelection.insert1(\n                curation_auto_key, skip_duplicates=True\n            )\n\n    else:\n        # Perform no automatic curation, just populate curated spike sorting\n        # selection with the initial curation. Used in case of clusterless\n        # decoding\n        logger.info(\"Creating curated spike sorting\")\n        curation_keys = (Curation() &amp; sort_dict).fetch(\"KEY\")\n        for curation_key in curation_keys:\n            CuratedSpikeSortingSelection.insert1(\n                curation_key, skip_duplicates=True\n            )\n\n    # Populate curated spike sorting\n    CuratedSpikeSorting.populate(sort_dict)\n\n    if fig_url_repo:\n        # Curation Figurl\n        logger.info(\"Creating curation figurl\")\n        sort_interval_name = interval_list_name + \"_entire\"\n        gh_url = (\n            fig_url_repo\n            + str(nwb_file_name + \"_\" + sort_interval_name)  # session id\n            + \"/{}\"  # tetrode using auto_id['sort_group_id']\n            + \"/curation.json\"\n        )\n\n        for auto_id in (AutomaticCuration() &amp; sort_dict).fetch(\n            \"auto_curation_key\"\n        ):\n            auto_curation_out_key = dict(\n                **(Curation() &amp; auto_id).fetch1(\"KEY\"),\n                new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),\n            )\n            CurationFigurlSelection.insert1(\n                auto_curation_out_key, skip_duplicates=True\n            )\n            CurationFigurl.populate(auto_curation_out_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/", "title": "spikesorting_recording.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup", "title": "<code>SortGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>@schema\nclass SortGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Set of electrodes that will be sorted together\n    -&gt; Session\n    sort_group_id: int  # identifier for a group of electrodes\n    ---\n    sort_reference_electrode_id = -1: int  # the electrode to use for reference. -1: no reference, -2: common median\n    \"\"\"\n\n    class SortGroupElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; SortGroup\n        -&gt; Electrode\n        \"\"\"\n\n    def set_group_by_shank(\n        self,\n        nwb_file_name: str,\n        references: dict = None,\n        omit_ref_electrode_group=False,\n        omit_unitrode=True,\n    ):\n        \"\"\"Divides electrodes into groups based on their shank position.\n\n        * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n          single group\n        * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n          placed in one group per shank\n        * Bad channels are omitted\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            the name of the NWB file whose electrodes should be put into\n            sorting groups\n        references : dict, optional\n            If passed, used to set references. Otherwise, references set using\n            original reference electrodes from config. Keys: electrode groups.\n            Values: reference electrode.\n        omit_ref_electrode_group : bool\n            Optional. If True, no sort group is defined for electrode group of\n            reference.\n        omit_unitrode : bool\n            Optional. If True, no sort groups are defined for unitrodes.\n        \"\"\"\n        # delete any current groups\n        (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n        e_groups.sort(key=int)  # sort electrode groups numerically\n        sort_group = 0\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        for e_group in e_groups:\n            # for each electrode group, get a list of the unique shank numbers\n            shank_list = np.unique(\n                electrodes[\"probe_shank\"][\n                    electrodes[\"electrode_group_name\"] == e_group\n                ]\n            )\n            sge_key[\"electrode_group_name\"] = e_group\n            # get the indices of all electrodes in this group / shank and set their sorting group\n            for shank in shank_list:\n                sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n                # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n                if not references:\n                    shank_elect_ref = electrodes[\n                        \"original_reference_electrode\"\n                    ][\n                        np.logical_and(\n                            electrodes[\"electrode_group_name\"] == e_group,\n                            electrodes[\"probe_shank\"] == shank,\n                        )\n                    ]\n                    if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                        sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                            0\n                        ]\n                    else:\n                        ValueError(\n                            f\"Error in electrode group {e_group}: reference \"\n                            + \"electrodes are not all the same\"\n                        )\n                else:\n                    if e_group not in references.keys():\n                        raise Exception(\n                            f\"electrode group {e_group} not a key in \"\n                            + \"references, so cannot set reference\"\n                        )\n                    else:\n                        sg_key[\"sort_reference_electrode_id\"] = references[\n                            e_group\n                        ]\n                # Insert sort group and sort group electrodes\n                reference_electrode_group = electrodes[\n                    electrodes[\"electrode_id\"]\n                    == sg_key[\"sort_reference_electrode_id\"]\n                ][\n                    \"electrode_group_name\"\n                ]  # reference for this electrode group\n                if (\n                    len(reference_electrode_group) == 1\n                ):  # unpack single reference\n                    reference_electrode_group = reference_electrode_group[0]\n                elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                    len(reference_electrode_group) != 1\n                ):\n                    raise Exception(\n                        \"Should have found exactly one electrode group for \"\n                        + \"reference electrode, but found \"\n                        + f\"{len(reference_electrode_group)}.\"\n                    )\n                if omit_ref_electrode_group and (\n                    str(e_group) == str(reference_electrode_group)\n                ):\n                    logger.warn(\n                        f\"Omitting electrode group {e_group} from sort groups \"\n                        + \"because contains reference.\"\n                    )\n                    continue\n                shank_elect = electrodes[\"electrode_id\"][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if (\n                    omit_unitrode and len(shank_elect) == 1\n                ):  # omit unitrodes if indicated\n                    logger.warn(\n                        f\"Omitting electrode group {e_group}, shank {shank} \"\n                        + \"from sort groups because unitrode.\"\n                    )\n                    continue\n                self.insert1(sg_key)\n                for elect in shank_elect:\n                    sge_key[\"electrode_id\"] = elect\n                    self.SortGroupElectrode().insert1(sge_key)\n                sort_group += 1\n\n    def set_group_by_electrode_group(self, nwb_file_name: str):\n        \"\"\"Assign groups to all non-bad channel electrodes based on their electrode group\n        and sets the reference for each group to the reference for the first channel of the group.\n\n        Parameters\n        ----------\n        nwb_file_name: str\n            the name of the nwb whose electrodes should be put into sorting groups\n        \"\"\"\n        # delete any current groups\n        (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = np.unique(electrodes[\"electrode_group_name\"])\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        sort_group = 0\n        for e_group in e_groups:\n            sge_key[\"electrode_group_name\"] = e_group\n            # sg_key['sort_group_id'] = sge_key['sort_group_id'] = sort_group\n            # TEST\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = int(e_group)\n            # get the list of references and make sure they are all the same\n            shank_elect_ref = electrodes[\"original_reference_electrode\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n            if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[0]\n            else:\n                ValueError(\n                    f\"Error in electrode group {e_group}: reference electrodes are not all the same\"\n                )\n            self.insert1(sg_key)\n\n            shank_elect = electrodes[\"electrode_id\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                self.SortGroupElectrode().insert1(sge_key)\n            sort_group += 1\n\n    def set_reference_from_list(self, nwb_file_name, sort_group_ref_list):\n        \"\"\"\n        Set the reference electrode from a list containing sort groups and reference electrodes\n        :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode]\n        :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated\n        :return: Null\n        \"\"\"\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        sort_group_list = (SortGroup() &amp; key).fetch1()\n        for sort_group in sort_group_list:\n            key[\"sort_group_id\"] = sort_group\n            self.insert(\n                dj_replace(\n                    sort_group_list,\n                    sort_group_ref_list,\n                    \"sort_group_id\",\n                    \"sort_reference_electrode_id\",\n                ),\n                replace=\"True\",\n            )\n\n    def get_geometry(self, sort_group_id, nwb_file_name):\n        \"\"\"\n        Returns a list with the x,y coordinates of the electrodes in the sort group\n        for use with the SpikeInterface package.\n\n        Converts z locations to y where appropriate.\n\n        Parameters\n        ----------\n        sort_group_id : int\n        nwb_file_name : str\n\n        Returns\n        -------\n        geometry : list\n            List of coordinate pairs, one per electrode\n        \"\"\"\n\n        # create the channel_groups dictiorary\n        channel_group = dict()\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        electrodes = (Electrode() &amp; key).fetch()\n\n        key[\"sort_group_id\"] = sort_group_id\n        sort_group_electrodes = (SortGroup.SortGroupElectrode() &amp; key).fetch()\n        electrode_group_name = sort_group_electrodes[\"electrode_group_name\"][0]\n        probe_id = (\n            ElectrodeGroup\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_group_name\": electrode_group_name,\n            }\n        ).fetch1(\"probe_id\")\n        channel_group[sort_group_id] = dict()\n        channel_group[sort_group_id][\"channels\"] = sort_group_electrodes[\n            \"electrode_id\"\n        ].tolist()\n\n        n_chan = len(channel_group[sort_group_id][\"channels\"])\n\n        geometry = np.zeros((n_chan, 2), dtype=\"float\")\n        tmp_geom = np.zeros((n_chan, 3), dtype=\"float\")\n        for i, electrode_id in enumerate(\n            channel_group[sort_group_id][\"channels\"]\n        ):\n            # get the relative x and y locations of this channel from the probe table\n            probe_electrode = int(\n                electrodes[\"probe_electrode\"][\n                    electrodes[\"electrode_id\"] == electrode_id\n                ]\n            )\n            rel_x, rel_y, rel_z = (\n                Probe().Electrode()\n                &amp; {\"probe_id\": probe_id, \"probe_electrode\": probe_electrode}\n            ).fetch(\"rel_x\", \"rel_y\", \"rel_z\")\n            # TODO: Fix this HACK when we can use probeinterface:\n            rel_x = float(rel_x)\n            rel_y = float(rel_y)\n            rel_z = float(rel_z)\n            tmp_geom[i, :] = [rel_x, rel_y, rel_z]\n\n        # figure out which columns have coordinates\n        n_found = 0\n        for i in range(3):\n            if np.any(np.nonzero(tmp_geom[:, i])):\n                if n_found &lt; 2:\n                    geometry[:, n_found] = tmp_geom[:, i]\n                    n_found += 1\n                else:\n                    Warning(\n                        \"Relative electrode locations have three coordinates; only two are currently supported\"\n                    )\n        return np.ndarray.tolist(geometry)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_group_by_shank", "title": "<code>set_group_by_shank(nwb_file_name, references=None, omit_ref_electrode_group=False, omit_unitrode=True)</code>", "text": "<p>Divides electrodes into groups based on their shank position.</p> <ul> <li>Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a   single group</li> <li>Electrodes from probes with multiple shanks (e.g. polymer probes) are   placed in one group per shank</li> <li>Bad channels are omitted</li> </ul> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the NWB file whose electrodes should be put into sorting groups</p> required <code>references</code> <code>dict</code> <p>If passed, used to set references. Otherwise, references set using original reference electrodes from config. Keys: electrode groups. Values: reference electrode.</p> <code>None</code> <code>omit_ref_electrode_group</code> <code>bool</code> <p>Optional. If True, no sort group is defined for electrode group of reference.</p> <code>False</code> <code>omit_unitrode</code> <code>bool</code> <p>Optional. If True, no sort groups are defined for unitrodes.</p> <code>True</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_group_by_shank(\n    self,\n    nwb_file_name: str,\n    references: dict = None,\n    omit_ref_electrode_group=False,\n    omit_unitrode=True,\n):\n    \"\"\"Divides electrodes into groups based on their shank position.\n\n    * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n      single group\n    * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n      placed in one group per shank\n    * Bad channels are omitted\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        the name of the NWB file whose electrodes should be put into\n        sorting groups\n    references : dict, optional\n        If passed, used to set references. Otherwise, references set using\n        original reference electrodes from config. Keys: electrode groups.\n        Values: reference electrode.\n    omit_ref_electrode_group : bool\n        Optional. If True, no sort group is defined for electrode group of\n        reference.\n    omit_unitrode : bool\n        Optional. If True, no sort groups are defined for unitrodes.\n    \"\"\"\n    # delete any current groups\n    (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n    e_groups.sort(key=int)  # sort electrode groups numerically\n    sort_group = 0\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    for e_group in e_groups:\n        # for each electrode group, get a list of the unique shank numbers\n        shank_list = np.unique(\n            electrodes[\"probe_shank\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n        )\n        sge_key[\"electrode_group_name\"] = e_group\n        # get the indices of all electrodes in this group / shank and set their sorting group\n        for shank in shank_list:\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n            # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n            if not references:\n                shank_elect_ref = electrodes[\n                    \"original_reference_electrode\"\n                ][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                    sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                        0\n                    ]\n                else:\n                    ValueError(\n                        f\"Error in electrode group {e_group}: reference \"\n                        + \"electrodes are not all the same\"\n                    )\n            else:\n                if e_group not in references.keys():\n                    raise Exception(\n                        f\"electrode group {e_group} not a key in \"\n                        + \"references, so cannot set reference\"\n                    )\n                else:\n                    sg_key[\"sort_reference_electrode_id\"] = references[\n                        e_group\n                    ]\n            # Insert sort group and sort group electrodes\n            reference_electrode_group = electrodes[\n                electrodes[\"electrode_id\"]\n                == sg_key[\"sort_reference_electrode_id\"]\n            ][\n                \"electrode_group_name\"\n            ]  # reference for this electrode group\n            if (\n                len(reference_electrode_group) == 1\n            ):  # unpack single reference\n                reference_electrode_group = reference_electrode_group[0]\n            elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                len(reference_electrode_group) != 1\n            ):\n                raise Exception(\n                    \"Should have found exactly one electrode group for \"\n                    + \"reference electrode, but found \"\n                    + f\"{len(reference_electrode_group)}.\"\n                )\n            if omit_ref_electrode_group and (\n                str(e_group) == str(reference_electrode_group)\n            ):\n                logger.warn(\n                    f\"Omitting electrode group {e_group} from sort groups \"\n                    + \"because contains reference.\"\n                )\n                continue\n            shank_elect = electrodes[\"electrode_id\"][\n                np.logical_and(\n                    electrodes[\"electrode_group_name\"] == e_group,\n                    electrodes[\"probe_shank\"] == shank,\n                )\n            ]\n            if (\n                omit_unitrode and len(shank_elect) == 1\n            ):  # omit unitrodes if indicated\n                logger.warn(\n                    f\"Omitting electrode group {e_group}, shank {shank} \"\n                    + \"from sort groups because unitrode.\"\n                )\n                continue\n            self.insert1(sg_key)\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                self.SortGroupElectrode().insert1(sge_key)\n            sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_group_by_electrode_group", "title": "<code>set_group_by_electrode_group(nwb_file_name)</code>", "text": "<p>Assign groups to all non-bad channel electrodes based on their electrode group and sets the reference for each group to the reference for the first channel of the group.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the nwb whose electrodes should be put into sorting groups</p> required Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_group_by_electrode_group(self, nwb_file_name: str):\n    \"\"\"Assign groups to all non-bad channel electrodes based on their electrode group\n    and sets the reference for each group to the reference for the first channel of the group.\n\n    Parameters\n    ----------\n    nwb_file_name: str\n        the name of the nwb whose electrodes should be put into sorting groups\n    \"\"\"\n    # delete any current groups\n    (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = np.unique(electrodes[\"electrode_group_name\"])\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    sort_group = 0\n    for e_group in e_groups:\n        sge_key[\"electrode_group_name\"] = e_group\n        # sg_key['sort_group_id'] = sge_key['sort_group_id'] = sort_group\n        # TEST\n        sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = int(e_group)\n        # get the list of references and make sure they are all the same\n        shank_elect_ref = electrodes[\"original_reference_electrode\"][\n            electrodes[\"electrode_group_name\"] == e_group\n        ]\n        if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n            sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[0]\n        else:\n            ValueError(\n                f\"Error in electrode group {e_group}: reference electrodes are not all the same\"\n            )\n        self.insert1(sg_key)\n\n        shank_elect = electrodes[\"electrode_id\"][\n            electrodes[\"electrode_group_name\"] == e_group\n        ]\n        for elect in shank_elect:\n            sge_key[\"electrode_id\"] = elect\n            self.SortGroupElectrode().insert1(sge_key)\n        sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_reference_from_list", "title": "<code>set_reference_from_list(nwb_file_name, sort_group_ref_list)</code>", "text": "<p>Set the reference electrode from a list containing sort groups and reference electrodes :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode] :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated :return: Null</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_reference_from_list(self, nwb_file_name, sort_group_ref_list):\n    \"\"\"\n    Set the reference electrode from a list containing sort groups and reference electrodes\n    :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode]\n    :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated\n    :return: Null\n    \"\"\"\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    sort_group_list = (SortGroup() &amp; key).fetch1()\n    for sort_group in sort_group_list:\n        key[\"sort_group_id\"] = sort_group\n        self.insert(\n            dj_replace(\n                sort_group_list,\n                sort_group_ref_list,\n                \"sort_group_id\",\n                \"sort_reference_electrode_id\",\n            ),\n            replace=\"True\",\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.get_geometry", "title": "<code>get_geometry(sort_group_id, nwb_file_name)</code>", "text": "<p>Returns a list with the x,y coordinates of the electrodes in the sort group for use with the SpikeInterface package.</p> <p>Converts z locations to y where appropriate.</p> <p>Parameters:</p> Name Type Description Default <code>sort_group_id</code> <code>int</code> required <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>geometry</code> <code>list</code> <p>List of coordinate pairs, one per electrode</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def get_geometry(self, sort_group_id, nwb_file_name):\n    \"\"\"\n    Returns a list with the x,y coordinates of the electrodes in the sort group\n    for use with the SpikeInterface package.\n\n    Converts z locations to y where appropriate.\n\n    Parameters\n    ----------\n    sort_group_id : int\n    nwb_file_name : str\n\n    Returns\n    -------\n    geometry : list\n        List of coordinate pairs, one per electrode\n    \"\"\"\n\n    # create the channel_groups dictiorary\n    channel_group = dict()\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    electrodes = (Electrode() &amp; key).fetch()\n\n    key[\"sort_group_id\"] = sort_group_id\n    sort_group_electrodes = (SortGroup.SortGroupElectrode() &amp; key).fetch()\n    electrode_group_name = sort_group_electrodes[\"electrode_group_name\"][0]\n    probe_id = (\n        ElectrodeGroup\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_group_name\": electrode_group_name,\n        }\n    ).fetch1(\"probe_id\")\n    channel_group[sort_group_id] = dict()\n    channel_group[sort_group_id][\"channels\"] = sort_group_electrodes[\n        \"electrode_id\"\n    ].tolist()\n\n    n_chan = len(channel_group[sort_group_id][\"channels\"])\n\n    geometry = np.zeros((n_chan, 2), dtype=\"float\")\n    tmp_geom = np.zeros((n_chan, 3), dtype=\"float\")\n    for i, electrode_id in enumerate(\n        channel_group[sort_group_id][\"channels\"]\n    ):\n        # get the relative x and y locations of this channel from the probe table\n        probe_electrode = int(\n            electrodes[\"probe_electrode\"][\n                electrodes[\"electrode_id\"] == electrode_id\n            ]\n        )\n        rel_x, rel_y, rel_z = (\n            Probe().Electrode()\n            &amp; {\"probe_id\": probe_id, \"probe_electrode\": probe_electrode}\n        ).fetch(\"rel_x\", \"rel_y\", \"rel_z\")\n        # TODO: Fix this HACK when we can use probeinterface:\n        rel_x = float(rel_x)\n        rel_y = float(rel_y)\n        rel_z = float(rel_z)\n        tmp_geom[i, :] = [rel_x, rel_y, rel_z]\n\n    # figure out which columns have coordinates\n    n_found = 0\n    for i in range(3):\n        if np.any(np.nonzero(tmp_geom[:, i])):\n            if n_found &lt; 2:\n                geometry[:, n_found] = tmp_geom[:, i]\n                n_found += 1\n            else:\n                Warning(\n                    \"Relative electrode locations have three coordinates; only two are currently supported\"\n                )\n    return np.ndarray.tolist(geometry)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SpikeSortingRecording", "title": "<code>SpikeSortingRecording</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>@schema\nclass SpikeSortingRecording(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingRecordingSelection\n    ---\n    recording_path: varchar(1000)\n    -&gt; IntervalList.proj(sort_interval_list_name='interval_list_name')\n    \"\"\"\n\n    def make(self, key):\n        sort_interval_valid_times = self._get_sort_interval_valid_times(key)\n        recording = self._get_filtered_recording(key)\n        recording_name = self._get_recording_name(key)\n\n        # Path to files that will hold the recording extractors\n        recording_path = str(recording_dir / Path(recording_name))\n        if os.path.exists(recording_path):\n            shutil.rmtree(recording_path)\n\n        recording.save(\n            folder=recording_path, chunk_duration=\"10000ms\", n_jobs=8\n        )\n\n        IntervalList.insert1(\n            {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": recording_name,\n                \"valid_times\": sort_interval_valid_times,\n                \"pipeline\": \"spikesorting_recording_v0\",\n            },\n            replace=True,\n        )\n\n        self.insert1(\n            {\n                **key,\n                # store the list of valid times for the sort\n                \"sort_interval_list_name\": recording_name,\n                \"recording_path\": recording_path,\n            }\n        )\n\n    @staticmethod\n    def _get_recording_name(key):\n        return \"_\".join(\n            [\n                key[\"nwb_file_name\"],\n                key[\"sort_interval_name\"],\n                str(key[\"sort_group_id\"]),\n                key[\"preproc_params_name\"],\n            ]\n        )\n\n    @staticmethod\n    def _get_recording_timestamps(recording):\n        num_segments = recording.get_num_segments()\n\n        if num_segments &lt;= 1:\n            return recording.get_times()\n\n        frames_per_segment = [0] + [\n            recording.get_num_frames(segment_index=i)\n            for i in range(num_segments)\n        ]\n\n        cumsum_frames = np.cumsum(frames_per_segment)\n        total_frames = np.sum(frames_per_segment)\n\n        timestamps = np.zeros((total_frames,))\n        for i in range(num_segments):\n            start_index = cumsum_frames[i]\n            end_index = cumsum_frames[i + 1]\n            timestamps[start_index:end_index] = recording.get_times(\n                segment_index=i\n            )\n\n        return timestamps\n\n    def _get_sort_interval_valid_times(self, key):\n        \"\"\"Identifies the intersection between sort interval specified by the user\n        and the valid times (times for which neural data exist)\n\n        Parameters\n        ----------\n        key: dict\n            specifies a (partially filled) entry of SpikeSorting table\n\n        Returns\n        -------\n        sort_interval_valid_times: ndarray of tuples\n            (start, end) times for valid stretches of the sorting interval\n\n        \"\"\"\n        sort_interval = (\n            SortInterval\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_interval_name\": key[\"sort_interval_name\"],\n            }\n        ).fetch1(\"sort_interval\")\n\n        interval_list_name = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"interval_list_name\"\n        )\n\n        valid_interval_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        valid_sort_times = interval_list_intersect(\n            sort_interval, valid_interval_times\n        )\n        # Exclude intervals shorter than specified length\n        params = (SpikeSortingPreprocessingParameters &amp; key).fetch1(\n            \"preproc_params\"\n        )\n        if \"min_segment_length\" in params:\n            valid_sort_times = intervals_by_length(\n                valid_sort_times, min_length=params[\"min_segment_length\"]\n            )\n        return valid_sort_times\n\n    def _get_filtered_recording(self, key: dict):\n        \"\"\"Filters and references a recording\n        * Loads the NWB file created during insertion as a spikeinterface Recording\n        * Slices recording in time (interval) and space (channels);\n          recording chunks from disjoint intervals are concatenated\n        * Applies referencing and bandpass filtering\n\n        Parameters\n        ----------\n        key: dict,\n            primary key of SpikeSortingRecording table\n\n        Returns\n        -------\n        recording: si.Recording\n        \"\"\"\n\n        nwb_file_abs_path = Nwbfile().get_abs_path(key[\"nwb_file_name\"])\n        recording = se.read_nwb_recording(\n            nwb_file_abs_path, load_time_vector=True\n        )\n\n        valid_sort_times = self._get_sort_interval_valid_times(key)\n        # shape is (N, 2)\n        valid_sort_times_indices = np.array(\n            [\n                np.searchsorted(recording.get_times(), interval)\n                for interval in valid_sort_times\n            ]\n        )\n        # join intervals of indices that are adjacent\n        valid_sort_times_indices = reduce(\n            union_adjacent_index, valid_sort_times_indices\n        )\n        if valid_sort_times_indices.ndim == 1:\n            valid_sort_times_indices = np.expand_dims(\n                valid_sort_times_indices, 0\n            )\n\n        # create an AppendRecording if there is more than one disjoint sort interval\n        if len(valid_sort_times_indices) &gt; 1:\n            recordings_list = []\n            for interval_indices in valid_sort_times_indices:\n                recording_single = recording.frame_slice(\n                    start_frame=interval_indices[0],\n                    end_frame=interval_indices[1],\n                )\n                recordings_list.append(recording_single)\n            recording = si.append_recordings(recordings_list)\n        else:\n            recording = recording.frame_slice(\n                start_frame=valid_sort_times_indices[0][0],\n                end_frame=valid_sort_times_indices[0][1],\n            )\n\n        channel_ids = (\n            SortGroup.SortGroupElectrode\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_group_id\": key[\"sort_group_id\"],\n            }\n        ).fetch(\"electrode_id\")\n        ref_channel_id = (\n            SortGroup\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_group_id\": key[\"sort_group_id\"],\n            }\n        ).fetch1(\"sort_reference_electrode_id\")\n        channel_ids = np.setdiff1d(channel_ids, ref_channel_id)\n\n        # include ref channel in first slice, then exclude it in second slice\n        if ref_channel_id &gt;= 0:\n            channel_ids_ref = np.append(channel_ids, ref_channel_id)\n            recording = recording.channel_slice(channel_ids=channel_ids_ref)\n\n            recording = si.preprocessing.common_reference(\n                recording, reference=\"single\", ref_channel_ids=ref_channel_id\n            )\n            recording = recording.channel_slice(channel_ids=channel_ids)\n        elif ref_channel_id == -2:\n            recording = recording.channel_slice(channel_ids=channel_ids)\n            recording = si.preprocessing.common_reference(\n                recording, reference=\"global\", operator=\"median\"\n            )\n        else:\n            raise ValueError(\"Invalid reference channel ID\")\n        filter_params = (SpikeSortingPreprocessingParameters &amp; key).fetch1(\n            \"preproc_params\"\n        )\n        recording = si.preprocessing.bandpass_filter(\n            recording,\n            freq_min=filter_params[\"frequency_min\"],\n            freq_max=filter_params[\"frequency_max\"],\n        )\n\n        # if the sort group is a tetrode, change the channel location\n        # note that this is a workaround that would be deprecated when spikeinterface uses 3D probe locations\n        probe_type = []\n        electrode_group = []\n        for channel_id in channel_ids:\n            probe_type.append(\n                (\n                    Electrode * Probe\n                    &amp; {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"probe_type\")\n            )\n            electrode_group.append(\n                (\n                    Electrode\n                    &amp; {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"electrode_group_name\")\n            )\n        if (\n            all(p == \"tetrode_12.5\" for p in probe_type)\n            and len(probe_type) == 4\n            and all(eg == electrode_group[0] for eg in electrode_group)\n        ):\n            tetrode = pi.Probe(ndim=2)\n            position = [[0, 0], [0, 12.5], [12.5, 0], [12.5, 12.5]]\n            tetrode.set_contacts(\n                position, shapes=\"circle\", shape_params={\"radius\": 6.25}\n            )\n            tetrode.set_contact_ids(channel_ids)\n            tetrode.set_device_channel_indices(np.arange(4))\n            recording = recording.set_probe(tetrode, in_place=True)\n\n        return recording\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/", "title": "spikesorting_sorting.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorterParameters", "title": "<code>SpikeSorterParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>@schema\nclass SpikeSorterParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    sorter: varchar(32)\n    sorter_params_name: varchar(64)\n    ---\n    sorter_params: blob\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Default params from spike sorters available via spikeinterface\"\"\"\n        sorters = sis.available_sorters()\n        for sorter in sorters:\n            sorter_params = sis.get_default_sorter_params(sorter)\n            self.insert1(\n                [sorter, \"default\", sorter_params], skip_duplicates=True\n            )\n\n        # Insert Frank lab defaults\n        # Hippocampus tetrode default\n        sorter = \"mountainsort4\"\n        sorter_params_name = \"franklab_tetrode_hippocampus_30KHz\"\n        sorter_params = {\n            \"detect_sign\": -1,\n            \"adjacency_radius\": 100,\n            \"freq_min\": 600,\n            \"freq_max\": 6000,\n            \"filter\": False,\n            \"whiten\": True,\n            \"num_workers\": 1,\n            \"clip_size\": 40,\n            \"detect_threshold\": 3,\n            \"detect_interval\": 10,\n        }\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n\n        # Cortical probe default\n        sorter = \"mountainsort4\"\n        sorter_params_name = \"franklab_probe_ctx_30KHz\"\n        sorter_params = {\n            \"detect_sign\": -1,\n            \"adjacency_radius\": 100,\n            \"freq_min\": 300,\n            \"freq_max\": 6000,\n            \"filter\": False,\n            \"whiten\": True,\n            \"num_workers\": 1,\n            \"clip_size\": 40,\n            \"detect_threshold\": 3,\n            \"detect_interval\": 10,\n        }\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n\n        # clusterless defaults\n        sorter = \"clusterless_thresholder\"\n        sorter_params_name = \"default_clusterless\"\n        sorter_params = dict(\n            detect_threshold=100.0,  # uV\n            # Locally exclusive means one unit per spike detected\n            method=\"locally_exclusive\",\n            peak_sign=\"neg\",\n            exclude_sweep_ms=0.1,\n            local_radius_um=100,\n            # noise levels needs to be 1.0 so the units are in uV and not MAD\n            noise_levels=np.asarray([1.0]),\n            random_chunk_kwargs={},\n            # output needs to be set to sorting for the rest of the pipeline\n            outputs=\"sorting\",\n        )\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorterParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Default params from spike sorters available via spikeinterface</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def insert_default(self):\n    \"\"\"Default params from spike sorters available via spikeinterface\"\"\"\n    sorters = sis.available_sorters()\n    for sorter in sorters:\n        sorter_params = sis.get_default_sorter_params(sorter)\n        self.insert1(\n            [sorter, \"default\", sorter_params], skip_duplicates=True\n        )\n\n    # Insert Frank lab defaults\n    # Hippocampus tetrode default\n    sorter = \"mountainsort4\"\n    sorter_params_name = \"franklab_tetrode_hippocampus_30KHz\"\n    sorter_params = {\n        \"detect_sign\": -1,\n        \"adjacency_radius\": 100,\n        \"freq_min\": 600,\n        \"freq_max\": 6000,\n        \"filter\": False,\n        \"whiten\": True,\n        \"num_workers\": 1,\n        \"clip_size\": 40,\n        \"detect_threshold\": 3,\n        \"detect_interval\": 10,\n    }\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n\n    # Cortical probe default\n    sorter = \"mountainsort4\"\n    sorter_params_name = \"franklab_probe_ctx_30KHz\"\n    sorter_params = {\n        \"detect_sign\": -1,\n        \"adjacency_radius\": 100,\n        \"freq_min\": 300,\n        \"freq_max\": 6000,\n        \"filter\": False,\n        \"whiten\": True,\n        \"num_workers\": 1,\n        \"clip_size\": 40,\n        \"detect_threshold\": 3,\n        \"detect_interval\": 10,\n    }\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n\n    # clusterless defaults\n    sorter = \"clusterless_thresholder\"\n    sorter_params_name = \"default_clusterless\"\n    sorter_params = dict(\n        detect_threshold=100.0,  # uV\n        # Locally exclusive means one unit per spike detected\n        method=\"locally_exclusive\",\n        peak_sign=\"neg\",\n        exclude_sweep_ms=0.1,\n        local_radius_um=100,\n        # noise levels needs to be 1.0 so the units are in uV and not MAD\n        noise_levels=np.asarray([1.0]),\n        random_chunk_kwargs={},\n        # output needs to be set to sorting for the rest of the pipeline\n        outputs=\"sorting\",\n    )\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting", "title": "<code>SpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>@schema\nclass SpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingSelection\n    ---\n    sorting_path: varchar(1000)\n    time_of_sort: int   # in Unix time, to the nearest second\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Runs spike sorting on the data and parameters specified by the\n        SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n\n        Specifically,\n        1. Loads saved recording and runs the sort on it with spikeinterface\n        2. Saves the sorting with spikeinterface\n        3. Creates an analysis NWB file and saves the sorting there\n           (this is redundant with 2; will change in the future)\n\n        \"\"\"\n        # CBroz: does this not work w/o arg? as .populate() ?\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        recording = si.load_extractor(recording_path)\n\n        # first, get the timestamps\n        timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n        _ = recording.get_sampling_frequency()\n        # then concatenate the recordings\n        # Note: the timestamps are lost upon concatenation,\n        # i.e. concat_recording.get_times() doesn't return true timestamps anymore.\n        # but concat_recording.recoring_list[i].get_times() will return correct\n        # timestamps for ith recording.\n        if recording.get_num_segments() &gt; 1 and isinstance(\n            recording, si.AppendSegmentRecording\n        ):\n            recording = si.concatenate_recordings(recording.recording_list)\n        elif recording.get_num_segments() &gt; 1 and isinstance(\n            recording, si.BinaryRecordingExtractor\n        ):\n            recording = si.concatenate_recordings([recording])\n\n        # load artifact intervals\n        artifact_times = (\n            ArtifactRemovedIntervalList\n            &amp; {\n                \"artifact_removed_interval_list_name\": key[\n                    \"artifact_removed_interval_list_name\"\n                ]\n            }\n        ).fetch1(\"artifact_times\")\n        if len(artifact_times):\n            if artifact_times.ndim == 1:\n                artifact_times = np.expand_dims(artifact_times, 0)\n\n            # convert artifact intervals to indices\n            list_triggers = []\n            for interval in artifact_times:\n                list_triggers.append(\n                    np.arange(\n                        np.searchsorted(timestamps, interval[0]),\n                        np.searchsorted(timestamps, interval[1]),\n                    )\n                )\n            list_triggers = [list(np.concatenate(list_triggers))]\n            recording = sip.remove_artifacts(\n                recording=recording,\n                list_triggers=list_triggers,\n                ms_before=None,\n                ms_after=None,\n                mode=\"zeros\",\n            )\n\n        logger.info(f\"Running spike sorting on {key}...\")\n        sorter, sorter_params = (SpikeSorterParameters &amp; key).fetch1(\n            \"sorter\", \"sorter_params\"\n        )\n\n        sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n        # add tempdir option for mountainsort\n        sorter_params[\"tempdir\"] = sorter_temp_dir.name\n\n        if sorter == \"clusterless_thresholder\":\n            # need to remove tempdir and whiten from sorter_params\n            sorter_params.pop(\"tempdir\", None)\n            sorter_params.pop(\"whiten\", None)\n            sorter_params.pop(\"outputs\", None)\n\n            # Detect peaks for clusterless decoding\n            detected_spikes = detect_peaks(recording, **sorter_params)\n            sorting = si.NumpySorting.from_times_labels(\n                times_list=detected_spikes[\"sample_index\"],\n                labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n                sampling_frequency=recording.get_sampling_frequency(),\n            )\n        else:\n            if \"whiten\" in sorter_params.keys():\n                if sorter_params[\"whiten\"]:\n                    sorter_params[\"whiten\"] = False  # set whiten to False\n            # whiten recording separately; make sure dtype is float32\n            # to avoid downstream error with svd\n            recording = sip.whiten(recording, dtype=\"float32\")\n            sorting = sis.run_sorter(\n                sorter,\n                recording,\n                output_folder=sorter_temp_dir.name,\n                remove_existing_folder=True,\n                delete_output_folder=True,\n                **sorter_params,\n            )\n        key[\"time_of_sort\"] = int(time.time())\n\n        logger.info(\"Saving sorting results...\")\n\n        sorting_folder = Path(sorting_dir)\n\n        sorting_name = self._get_sorting_name(key)\n        key[\"sorting_path\"] = str(sorting_folder / Path(sorting_name))\n        if os.path.exists(key[\"sorting_path\"]):\n            shutil.rmtree(key[\"sorting_path\"])\n        sorting = sorting.save(folder=key[\"sorting_path\"])\n        self.insert1(key)\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        raise NotImplementedError\n        return None\n\n    def nightly_cleanup(self):\n        \"\"\"Clean up spike sorting directories that are not in the SpikeSorting table.\n        This should be run after AnalysisNwbFile().nightly_cleanup()\n        \"\"\"\n        # get a list of the files in the spike sorting storage directory\n        dir_names = next(os.walk(sorting_dir))[1]\n        # now retrieve a list of the currently used analysis nwb files\n        analysis_file_names = self.fetch(\"analysis_file_name\")\n        for dir in dir_names:\n            if dir not in analysis_file_names:\n                full_path = str(Path(sorting_dir) / dir)\n                logger.info(f\"removing {full_path}\")\n                shutil.rmtree(str(Path(sorting_dir) / dir))\n\n    @staticmethod\n    def _get_sorting_name(key):\n        recording_name = SpikeSortingRecording._get_recording_name(key)\n        sorting_name = (\n            recording_name + \"_\" + str(uuid.uuid4())[0:8] + \"_spikesorting\"\n        )\n        return sorting_name\n\n    # TODO: write a function to import sorting done outside of dj\n\n    def _import_sorting(self, key):\n        raise NotImplementedError\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting.make", "title": "<code>make(key)</code>", "text": "<p>Runs spike sorting on the data and parameters specified by the SpikeSortingSelection table and inserts a new entry to SpikeSorting table.</p> <p>Specifically, 1. Loads saved recording and runs the sort on it with spikeinterface 2. Saves the sorting with spikeinterface 3. Creates an analysis NWB file and saves the sorting there    (this is redundant with 2; will change in the future)</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Runs spike sorting on the data and parameters specified by the\n    SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n\n    Specifically,\n    1. Loads saved recording and runs the sort on it with spikeinterface\n    2. Saves the sorting with spikeinterface\n    3. Creates an analysis NWB file and saves the sorting there\n       (this is redundant with 2; will change in the future)\n\n    \"\"\"\n    # CBroz: does this not work w/o arg? as .populate() ?\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    recording = si.load_extractor(recording_path)\n\n    # first, get the timestamps\n    timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n    _ = recording.get_sampling_frequency()\n    # then concatenate the recordings\n    # Note: the timestamps are lost upon concatenation,\n    # i.e. concat_recording.get_times() doesn't return true timestamps anymore.\n    # but concat_recording.recoring_list[i].get_times() will return correct\n    # timestamps for ith recording.\n    if recording.get_num_segments() &gt; 1 and isinstance(\n        recording, si.AppendSegmentRecording\n    ):\n        recording = si.concatenate_recordings(recording.recording_list)\n    elif recording.get_num_segments() &gt; 1 and isinstance(\n        recording, si.BinaryRecordingExtractor\n    ):\n        recording = si.concatenate_recordings([recording])\n\n    # load artifact intervals\n    artifact_times = (\n        ArtifactRemovedIntervalList\n        &amp; {\n            \"artifact_removed_interval_list_name\": key[\n                \"artifact_removed_interval_list_name\"\n            ]\n        }\n    ).fetch1(\"artifact_times\")\n    if len(artifact_times):\n        if artifact_times.ndim == 1:\n            artifact_times = np.expand_dims(artifact_times, 0)\n\n        # convert artifact intervals to indices\n        list_triggers = []\n        for interval in artifact_times:\n            list_triggers.append(\n                np.arange(\n                    np.searchsorted(timestamps, interval[0]),\n                    np.searchsorted(timestamps, interval[1]),\n                )\n            )\n        list_triggers = [list(np.concatenate(list_triggers))]\n        recording = sip.remove_artifacts(\n            recording=recording,\n            list_triggers=list_triggers,\n            ms_before=None,\n            ms_after=None,\n            mode=\"zeros\",\n        )\n\n    logger.info(f\"Running spike sorting on {key}...\")\n    sorter, sorter_params = (SpikeSorterParameters &amp; key).fetch1(\n        \"sorter\", \"sorter_params\"\n    )\n\n    sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n    # add tempdir option for mountainsort\n    sorter_params[\"tempdir\"] = sorter_temp_dir.name\n\n    if sorter == \"clusterless_thresholder\":\n        # need to remove tempdir and whiten from sorter_params\n        sorter_params.pop(\"tempdir\", None)\n        sorter_params.pop(\"whiten\", None)\n        sorter_params.pop(\"outputs\", None)\n\n        # Detect peaks for clusterless decoding\n        detected_spikes = detect_peaks(recording, **sorter_params)\n        sorting = si.NumpySorting.from_times_labels(\n            times_list=detected_spikes[\"sample_index\"],\n            labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n    else:\n        if \"whiten\" in sorter_params.keys():\n            if sorter_params[\"whiten\"]:\n                sorter_params[\"whiten\"] = False  # set whiten to False\n        # whiten recording separately; make sure dtype is float32\n        # to avoid downstream error with svd\n        recording = sip.whiten(recording, dtype=\"float32\")\n        sorting = sis.run_sorter(\n            sorter,\n            recording,\n            output_folder=sorter_temp_dir.name,\n            remove_existing_folder=True,\n            delete_output_folder=True,\n            **sorter_params,\n        )\n    key[\"time_of_sort\"] = int(time.time())\n\n    logger.info(\"Saving sorting results...\")\n\n    sorting_folder = Path(sorting_dir)\n\n    sorting_name = self._get_sorting_name(key)\n    key[\"sorting_path\"] = str(sorting_folder / Path(sorting_name))\n    if os.path.exists(key[\"sorting_path\"]):\n        shutil.rmtree(key[\"sorting_path\"])\n    sorting = sorting.save(folder=key[\"sorting_path\"])\n    self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting.nightly_cleanup", "title": "<code>nightly_cleanup()</code>", "text": "<p>Clean up spike sorting directories that are not in the SpikeSorting table. This should be run after AnalysisNwbFile().nightly_cleanup()</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def nightly_cleanup(self):\n    \"\"\"Clean up spike sorting directories that are not in the SpikeSorting table.\n    This should be run after AnalysisNwbFile().nightly_cleanup()\n    \"\"\"\n    # get a list of the files in the spike sorting storage directory\n    dir_names = next(os.walk(sorting_dir))[1]\n    # now retrieve a list of the currently used analysis nwb files\n    analysis_file_names = self.fetch(\"analysis_file_name\")\n    for dir in dir_names:\n        if dir not in analysis_file_names:\n            full_path = str(Path(sorting_dir) / dir)\n            logger.info(f\"removing {full_path}\")\n            shutil.rmtree(str(Path(sorting_dir) / dir))\n</code></pre>"}, {"location": "api/spikesorting/v0/figurl_views/SpikeSortingRecordingView/", "title": "SpikeSortingRecordingView.py", "text": ""}, {"location": "api/spikesorting/v0/figurl_views/SpikeSortingView/", "title": "SpikeSortingView.py", "text": ""}, {"location": "api/spikesorting/v0/figurl_views/prepare_spikesortingview_data/", "title": "prepare_spikesortingview_data.py", "text": ""}, {"location": "api/spikesorting/v1/artifact/", "title": "artifact.py", "text": ""}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.ArtifactDetectionSelection", "title": "<code>ArtifactDetectionSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>@schema\nclass ArtifactDetectionSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Processed recording and artifact detection parameters. Use `insert_selection` method to insert new rows.\n    artifact_id: uuid\n    ---\n    -&gt; SpikeSortingRecording\n    -&gt; ArtifactDetectionParameters\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into ArtifactDetectionSelection with an\n        automatically generated unique artifact ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSortingRecording and ArtifactDetectionParameters\n\n        Returns\n        -------\n        artifact_id : str\n            the unique artifact ID serving as primary key for ArtifactDetectionSelection\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"artifact_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.ArtifactDetectionSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into ArtifactDetectionSelection with an automatically generated unique artifact ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSortingRecording and ArtifactDetectionParameters</p> required <p>Returns:</p> Name Type Description <code>artifact_id</code> <code>str</code> <p>the unique artifact ID serving as primary key for ArtifactDetectionSelection</p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into ArtifactDetectionSelection with an\n    automatically generated unique artifact ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSortingRecording and ArtifactDetectionParameters\n\n    Returns\n    -------\n    artifact_id : str\n        the unique artifact ID serving as primary key for ArtifactDetectionSelection\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.warn(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"artifact_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.merge_intervals", "title": "<code>merge_intervals(intervals)</code>", "text": "<p>Takes a list of intervals each of which is [start_time, stop_time] and takes union over intervals that are intersecting</p> <p>Parameters:</p> Name Type Description Default <code>intervals</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>def merge_intervals(intervals):\n    \"\"\"Takes a list of intervals each of which is [start_time, stop_time]\n    and takes union over intervals that are intersecting\n\n    Parameters\n    ----------\n    intervals : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    if len(intervals) == 0:\n        return []\n\n    # Sort the intervals based on their start times\n    intervals.sort(key=lambda x: x[0])\n\n    merged = [intervals[0]]\n\n    for i in range(1, len(intervals)):\n        current_start, current_stop = intervals[i]\n        last_merged_start, last_merged_stop = merged[-1]\n\n        if current_start &lt;= last_merged_stop:\n            # Overlapping intervals, merge them\n            merged[-1] = [\n                last_merged_start,\n                max(last_merged_stop, current_stop),\n            ]\n        else:\n            # Non-overlapping intervals, add the current one to the list\n            merged.append([current_start, current_stop])\n\n    return np.asarray(merged)\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/", "title": "curation.py", "text": ""}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1", "title": "<code>CurationV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@schema\nclass CurationV1(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Curation of a SpikeSorting. Use `insert_curation` to insert rows.\n    -&gt; SpikeSorting\n    curation_id=0: int\n    ---\n    parent_curation_id=-1: int\n    -&gt; AnalysisNwbfile\n    object_id: varchar(72)\n    merges_applied: bool\n    description: varchar(100)\n    \"\"\"\n\n    @classmethod\n    def insert_curation(\n        cls,\n        sorting_id: str,\n        parent_curation_id: int = -1,\n        labels: Union[None, Dict[str, List[str]]] = None,\n        merge_groups: Union[None, List[List[str]]] = None,\n        apply_merge: bool = False,\n        metrics: Union[None, Dict[str, Dict[str, float]]] = None,\n        description: str = \"\",\n    ):\n        \"\"\"Insert a row into CurationV1.\n\n        Parameters\n        ----------\n        sorting_id : str\n            The key for the original SpikeSorting\n        parent_curation_id : int, optional\n            The curation id of the parent curation\n        labels : dict or None, optional\n            curation labels (e.g. good, noise, mua)\n        merge_groups : dict or None, optional\n            groups of unit IDs to be merged\n        metrics : dict or None, optional\n            Computed quality metrics, one for each neuron\n        description : str, optional\n            description of this curation or where it originates; e.g. FigURL\n\n        Note\n        ----\n        Example curation.json (output of figurl):\n        {\n         \"labelsByUnit\":\n            {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},\n         \"mergeGroups\":\n            [[11,12],[46,48],[51,54],[50,53]]\n        }\n\n        Returns\n        -------\n        curation_key : dict\n        \"\"\"\n        sort_query = cls &amp; {\"sorting_id\": sorting_id}\n        parent_curation_id = max(parent_curation_id, -1)\n        if parent_curation_id == -1:\n            parent_curation_id = -1\n            # check to see if this sorting with a parent of -1\n            # has already been inserted and if so, warn the user\n            query = sort_query &amp; {\"parent_curation_id\": -1}\n            if query:\n                Warning(\"Sorting has already been inserted.\")\n                return query.fetch(\"KEY\")\n\n        # generate curation ID\n        existing_curation_ids = sort_query.fetch(\"curation_id\")\n        curation_id = max(existing_curation_ids, default=-1) + 1\n\n        # write the curation labels, merge groups,\n        # and metrics as columns in the units table of NWB\n        analysis_file_name, object_id = _write_sorting_to_nwb_with_curation(\n            sorting_id=sorting_id,\n            labels=labels,\n            merge_groups=merge_groups,\n            metrics=metrics,\n            apply_merge=apply_merge,\n        )\n\n        # INSERT\n        AnalysisNwbfile().add(\n            (SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}).fetch1(\n                \"nwb_file_name\"\n            ),\n            analysis_file_name,\n        )\n\n        key = {\n            \"sorting_id\": sorting_id,\n            \"curation_id\": curation_id,\n            \"parent_curation_id\": parent_curation_id,\n            \"analysis_file_name\": analysis_file_name,\n            \"object_id\": object_id,\n            \"merges_applied\": apply_merge,\n            \"description\": description,\n        }\n        cls.insert1(\n            key,\n            skip_duplicates=True,\n        )\n\n        return key\n\n    @classmethod\n    def insert_metric_curation(cls, key: Dict, apply_merge=False):\n        \"\"\"Insert a row into CurationV1.\n\n        Parameters\n        ----------\n        key : Dict\n            primary key of MetricCuration\n\n        Returns\n        -------\n        curation_key : Dict\n        \"\"\"\n        from spyglass.spikesorting.v1.metric_curation import (\n            MetricCuration,\n            MetricCurationSelection,\n        )\n\n        sorting_id, parent_curation_id = (MetricCurationSelection &amp; key).fetch1(\n            \"sorting_id\", \"curation_id\"\n        )\n\n        curation_key = cls.insert_curation(\n            sorting_id=sorting_id,\n            parent_curation_id=parent_curation_id,\n            labels=MetricCuration.get_labels(key) or None,\n            merge_groups=MetricCuration.get_merge_groups(key) or None,\n            apply_merge=apply_merge,\n            description=(f\"metric_curation_id: {key['metric_curation_id']}\"),\n        )\n\n        return curation_key\n\n    @classmethod\n    def get_recording(cls, key: dict) -&gt; si.BaseRecording:\n        \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1 table\n        \"\"\"\n\n        analysis_file_name = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        recording = se.read_nwb_recording(\n            analysis_file_abs_path, load_time_vector=True\n        )\n        recording.annotate(is_filtered=True)\n\n        return recording\n\n    @classmethod\n    def get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1 table\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n        recording = cls.get_recording(key)\n        sampling_frequency = recording.get_sampling_frequency()\n        analysis_file_name = (CurationV1 &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.units.to_dataframe()\n        units_dict_list = [\n            {\n                unit_id: np.searchsorted(recording.get_times(), spike_times)\n                for unit_id, spike_times in zip(\n                    units.index, units[\"spike_times\"]\n                )\n            }\n        ]\n\n        sorting = si.NumpySorting.from_unit_dict(\n            units_dict_list, sampling_frequency=sampling_frequency\n        )\n\n        return sorting\n\n    @classmethod\n    def get_merged_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting with merges applied.\n\n        Parameters\n        ----------\n        key : dict\n            CurationV1 key\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n        recording = cls.get_recording(key)\n\n        curation_key = (cls &amp; key).fetch1()\n\n        sorting_analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            curation_key[\"analysis_file_name\"]\n        )\n        si_sorting = se.read_nwb_sorting(\n            sorting_analysis_file_abs_path,\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n\n        with pynwb.NWBHDF5IO(\n            sorting_analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            nwb_sorting = nwbfile.objects[curation_key[\"object_id\"]]\n            merge_groups = nwb_sorting[\"merge_groups\"][:]\n\n        if merge_groups:\n            units_to_merge = _merge_dict_to_list(merge_groups)\n            return sc.MergeUnitsSorting(\n                parent_sorting=si_sorting, units_to_merge=units_to_merge\n            )\n        else:\n            return si_sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_curation", "title": "<code>insert_curation(sorting_id, parent_curation_id=-1, labels=None, merge_groups=None, apply_merge=False, metrics=None, description='')</code>  <code>classmethod</code>", "text": "<p>Insert a row into CurationV1.</p> <p>Parameters:</p> Name Type Description Default <code>sorting_id</code> <code>str</code> <p>The key for the original SpikeSorting</p> required <code>parent_curation_id</code> <code>int</code> <p>The curation id of the parent curation</p> <code>-1</code> <code>labels</code> <code>dict or None</code> <p>curation labels (e.g. good, noise, mua)</p> <code>None</code> <code>merge_groups</code> <code>dict or None</code> <p>groups of unit IDs to be merged</p> <code>None</code> <code>metrics</code> <code>dict or None</code> <p>Computed quality metrics, one for each neuron</p> <code>None</code> <code>description</code> <code>str</code> <p>description of this curation or where it originates; e.g. FigURL</p> <code>''</code>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_curation--note", "title": "Note", "text": "<p>Example curation.json (output of figurl): {  \"labelsByUnit\":     {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},  \"mergeGroups\":     [[11,12],[46,48],[51,54],[50,53]] }</p> <p>Returns:</p> Name Type Description <code>curation_key</code> <code>dict</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef insert_curation(\n    cls,\n    sorting_id: str,\n    parent_curation_id: int = -1,\n    labels: Union[None, Dict[str, List[str]]] = None,\n    merge_groups: Union[None, List[List[str]]] = None,\n    apply_merge: bool = False,\n    metrics: Union[None, Dict[str, Dict[str, float]]] = None,\n    description: str = \"\",\n):\n    \"\"\"Insert a row into CurationV1.\n\n    Parameters\n    ----------\n    sorting_id : str\n        The key for the original SpikeSorting\n    parent_curation_id : int, optional\n        The curation id of the parent curation\n    labels : dict or None, optional\n        curation labels (e.g. good, noise, mua)\n    merge_groups : dict or None, optional\n        groups of unit IDs to be merged\n    metrics : dict or None, optional\n        Computed quality metrics, one for each neuron\n    description : str, optional\n        description of this curation or where it originates; e.g. FigURL\n\n    Note\n    ----\n    Example curation.json (output of figurl):\n    {\n     \"labelsByUnit\":\n        {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},\n     \"mergeGroups\":\n        [[11,12],[46,48],[51,54],[50,53]]\n    }\n\n    Returns\n    -------\n    curation_key : dict\n    \"\"\"\n    sort_query = cls &amp; {\"sorting_id\": sorting_id}\n    parent_curation_id = max(parent_curation_id, -1)\n    if parent_curation_id == -1:\n        parent_curation_id = -1\n        # check to see if this sorting with a parent of -1\n        # has already been inserted and if so, warn the user\n        query = sort_query &amp; {\"parent_curation_id\": -1}\n        if query:\n            Warning(\"Sorting has already been inserted.\")\n            return query.fetch(\"KEY\")\n\n    # generate curation ID\n    existing_curation_ids = sort_query.fetch(\"curation_id\")\n    curation_id = max(existing_curation_ids, default=-1) + 1\n\n    # write the curation labels, merge groups,\n    # and metrics as columns in the units table of NWB\n    analysis_file_name, object_id = _write_sorting_to_nwb_with_curation(\n        sorting_id=sorting_id,\n        labels=labels,\n        merge_groups=merge_groups,\n        metrics=metrics,\n        apply_merge=apply_merge,\n    )\n\n    # INSERT\n    AnalysisNwbfile().add(\n        (SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}).fetch1(\n            \"nwb_file_name\"\n        ),\n        analysis_file_name,\n    )\n\n    key = {\n        \"sorting_id\": sorting_id,\n        \"curation_id\": curation_id,\n        \"parent_curation_id\": parent_curation_id,\n        \"analysis_file_name\": analysis_file_name,\n        \"object_id\": object_id,\n        \"merges_applied\": apply_merge,\n        \"description\": description,\n    }\n    cls.insert1(\n        key,\n        skip_duplicates=True,\n    )\n\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_metric_curation", "title": "<code>insert_metric_curation(key, apply_merge=False)</code>  <code>classmethod</code>", "text": "<p>Insert a row into CurationV1.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>Dict</code> <p>primary key of MetricCuration</p> required <p>Returns:</p> Name Type Description <code>curation_key</code> <code>Dict</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef insert_metric_curation(cls, key: Dict, apply_merge=False):\n    \"\"\"Insert a row into CurationV1.\n\n    Parameters\n    ----------\n    key : Dict\n        primary key of MetricCuration\n\n    Returns\n    -------\n    curation_key : Dict\n    \"\"\"\n    from spyglass.spikesorting.v1.metric_curation import (\n        MetricCuration,\n        MetricCurationSelection,\n    )\n\n    sorting_id, parent_curation_id = (MetricCurationSelection &amp; key).fetch1(\n        \"sorting_id\", \"curation_id\"\n    )\n\n    curation_key = cls.insert_curation(\n        sorting_id=sorting_id,\n        parent_curation_id=parent_curation_id,\n        labels=MetricCuration.get_labels(key) or None,\n        merge_groups=MetricCuration.get_merge_groups(key) or None,\n        apply_merge=apply_merge,\n        description=(f\"metric_curation_id: {key['metric_curation_id']}\"),\n    )\n\n    return curation_key\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Get recording related to this curation as spikeinterface BaseRecording</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1 table</p> required Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_recording(cls, key: dict) -&gt; si.BaseRecording:\n    \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1 table\n    \"\"\"\n\n    analysis_file_name = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    recording = se.read_nwb_recording(\n        analysis_file_abs_path, load_time_vector=True\n    )\n    recording.annotate(is_filtered=True)\n\n    return recording\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting in the analysis NWB file as spikeinterface BaseSorting</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1 table</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1 table\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n    recording = cls.get_recording(key)\n    sampling_frequency = recording.get_sampling_frequency()\n    analysis_file_name = (CurationV1 &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.units.to_dataframe()\n    units_dict_list = [\n        {\n            unit_id: np.searchsorted(recording.get_times(), spike_times)\n            for unit_id, spike_times in zip(\n                units.index, units[\"spike_times\"]\n            )\n        }\n    ]\n\n    sorting = si.NumpySorting.from_unit_dict(\n        units_dict_list, sampling_frequency=sampling_frequency\n    )\n\n    return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_merged_sorting", "title": "<code>get_merged_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting with merges applied.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>CurationV1 key</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_merged_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting with merges applied.\n\n    Parameters\n    ----------\n    key : dict\n        CurationV1 key\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n    recording = cls.get_recording(key)\n\n    curation_key = (cls &amp; key).fetch1()\n\n    sorting_analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        curation_key[\"analysis_file_name\"]\n    )\n    si_sorting = se.read_nwb_sorting(\n        sorting_analysis_file_abs_path,\n        sampling_frequency=recording.get_sampling_frequency(),\n    )\n\n    with pynwb.NWBHDF5IO(\n        sorting_analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbfile = io.read()\n        nwb_sorting = nwbfile.objects[curation_key[\"object_id\"]]\n        merge_groups = nwb_sorting[\"merge_groups\"][:]\n\n    if merge_groups:\n        units_to_merge = _merge_dict_to_list(merge_groups)\n        return sc.MergeUnitsSorting(\n            parent_sorting=si_sorting, units_to_merge=units_to_merge\n        )\n    else:\n        return si_sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/", "title": "figurl_curation.py", "text": ""}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection", "title": "<code>FigURLCurationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@schema\nclass FigURLCurationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Use `insert_selection` method to insert a row. Use `generate_curation_uri` method to generate a curation uri.\n    figurl_curation_id: uuid\n    ---\n    -&gt; CurationV1\n    curation_uri: varchar(1000)     # GitHub-based URI to a file to which the manual curation will be saved\n    metrics_figurl: blob            # metrics to display in the figURL\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into FigURLCurationSelection.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of `CurationV1`, `curation_uri`, and `metrics_figurl`.\n            - If `curation_uri` is not provided, it will be generated from `generate_curation_uri` method.\n            - If `metrics_figurl` is not provided, it will be set to [].\n\n        Returns\n        -------\n        key : dict\n            primary key of `FigURLCurationSelection` table.\n        \"\"\"\n        if \"curation_uri\" not in key:\n            key[\"curation_uri\"] = cls.generate_curation_uri(key)\n        if \"metrics_figurl\" not in key:\n            key[\"metrics_figurl\"] = []\n        if \"figurl_curation_id\" in key:\n            query = cls &amp; {\"figurl_curation_id\": key[\"figurl_curation_id\"]}\n            if query:\n                logger.warn(\"Similar row(s) already inserted.\")\n                return query.fetch(as_dict=True)\n        key[\"figurl_curation_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n\n    @staticmethod\n    def generate_curation_uri(key: Dict) -&gt; str:\n        \"\"\"Generates a kachery-cloud URI from a row in CurationV1 table\n\n        Parameters\n        ----------\n        key : dict\n            primary key from CurationV1\n        \"\"\"\n        curation_key = (CurationV1 &amp; key).fetch1()\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            curation_key[\"analysis_file_name\"]\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            nwb_sorting = nwbfile.objects[\n                curation_key[\"object_id\"]\n            ].to_dataframe()\n            unit_ids = list(nwb_sorting.index)\n            labels = list(nwb_sorting[\"curation_label\"])\n            merge_groups = list(nwb_sorting[\"merge_groups\"])\n\n        unit_ids = [str(unit_id) for unit_id in unit_ids]\n\n        labels_dict = (\n            {unit_id: list(label) for unit_id, label in zip(unit_ids, labels)}\n            if labels\n            else {}\n        )\n\n        merge_groups_list = (\n            [\n                [str(unit_id) for unit_id in merge_group]\n                for merge_group in _merge_dict_to_list(\n                    dict(zip(unit_ids, merge_groups))\n                )\n            ]\n            if merge_groups\n            else []\n        )\n\n        return kcl.store_json(\n            {\n                \"labelsByUnit\": labels_dict,\n                \"mergeGroups\": merge_groups_list,\n            }\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into FigURLCurationSelection.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of <code>CurationV1</code>, <code>curation_uri</code>, and <code>metrics_figurl</code>. - If <code>curation_uri</code> is not provided, it will be generated from <code>generate_curation_uri</code> method. - If <code>metrics_figurl</code> is not provided, it will be set to [].</p> required <p>Returns:</p> Name Type Description <code>key</code> <code>dict</code> <p>primary key of <code>FigURLCurationSelection</code> table.</p> Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into FigURLCurationSelection.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of `CurationV1`, `curation_uri`, and `metrics_figurl`.\n        - If `curation_uri` is not provided, it will be generated from `generate_curation_uri` method.\n        - If `metrics_figurl` is not provided, it will be set to [].\n\n    Returns\n    -------\n    key : dict\n        primary key of `FigURLCurationSelection` table.\n    \"\"\"\n    if \"curation_uri\" not in key:\n        key[\"curation_uri\"] = cls.generate_curation_uri(key)\n    if \"metrics_figurl\" not in key:\n        key[\"metrics_figurl\"] = []\n    if \"figurl_curation_id\" in key:\n        query = cls &amp; {\"figurl_curation_id\": key[\"figurl_curation_id\"]}\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n    key[\"figurl_curation_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection.generate_curation_uri", "title": "<code>generate_curation_uri(key)</code>  <code>staticmethod</code>", "text": "<p>Generates a kachery-cloud URI from a row in CurationV1 table</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key from CurationV1</p> required Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@staticmethod\ndef generate_curation_uri(key: Dict) -&gt; str:\n    \"\"\"Generates a kachery-cloud URI from a row in CurationV1 table\n\n    Parameters\n    ----------\n    key : dict\n        primary key from CurationV1\n    \"\"\"\n    curation_key = (CurationV1 &amp; key).fetch1()\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        curation_key[\"analysis_file_name\"]\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbfile = io.read()\n        nwb_sorting = nwbfile.objects[\n            curation_key[\"object_id\"]\n        ].to_dataframe()\n        unit_ids = list(nwb_sorting.index)\n        labels = list(nwb_sorting[\"curation_label\"])\n        merge_groups = list(nwb_sorting[\"merge_groups\"])\n\n    unit_ids = [str(unit_id) for unit_id in unit_ids]\n\n    labels_dict = (\n        {unit_id: list(label) for unit_id, label in zip(unit_ids, labels)}\n        if labels\n        else {}\n    )\n\n    merge_groups_list = (\n        [\n            [str(unit_id) for unit_id in merge_group]\n            for merge_group in _merge_dict_to_list(\n                dict(zip(unit_ids, merge_groups))\n            )\n        ]\n        if merge_groups\n        else []\n    )\n\n    return kcl.store_json(\n        {\n            \"labelsByUnit\": labels_dict,\n            \"mergeGroups\": merge_groups_list,\n        }\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/", "title": "metric_curation.py", "text": ""}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCurationSelection", "title": "<code>MetricCurationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@schema\nclass MetricCurationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Spike sorting and parameters for metric curation. Use `insert_selection` to insert a row into this table.\n    metric_curation_id: uuid\n    ---\n    -&gt; CurationV1\n    -&gt; WaveformParameters\n    -&gt; MetricParameters\n    -&gt; MetricCurationParameters\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into MetricCurationSelection with an\n        automatically generated unique metric curation ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters\n\n        Returns\n        -------\n        key : dict\n            key for the inserted row\n        \"\"\"\n        if cls &amp; key:\n            logger.warn(\"This row has already been inserted.\")\n            return (cls &amp; key).fetch1()\n        key[\"metric_curation_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCurationSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into MetricCurationSelection with an automatically generated unique metric curation ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters</p> required <p>Returns:</p> Name Type Description <code>key</code> <code>dict</code> <p>key for the inserted row</p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into MetricCurationSelection with an\n    automatically generated unique metric curation ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters\n\n    Returns\n    -------\n    key : dict\n        key for the inserted row\n    \"\"\"\n    if cls &amp; key:\n        logger.warn(\"This row has already been inserted.\")\n        return (cls &amp; key).fetch1()\n    key[\"metric_curation_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration", "title": "<code>MetricCuration</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@schema\nclass MetricCuration(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    # Results of applying curation based on quality metrics. To do additional curation, insert another row in `CurationV1`\n    -&gt; MetricCurationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # Object ID for the metrics in NWB file\n    \"\"\"\n\n    def make(self, key):\n        # FETCH\n        nwb_file_name = (\n            SpikeSortingSelection * MetricCurationSelection &amp; key\n        ).fetch1(\"nwb_file_name\")\n\n        waveform_params = (\n            WaveformParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"waveform_params\")\n        metric_params = (\n            MetricParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"metric_params\")\n        label_params, merge_params = (\n            MetricCurationParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"label_params\", \"merge_params\")\n        sorting_id, curation_id = (MetricCurationSelection &amp; key).fetch1(\n            \"sorting_id\", \"curation_id\"\n        )\n        # DO\n        # load recording and sorting\n        recording = CurationV1.get_recording(\n            {\"sorting_id\": sorting_id, \"curation_id\": curation_id}\n        )\n        sorting = CurationV1.get_sorting(\n            {\"sorting_id\": sorting_id, \"curation_id\": curation_id}\n        )\n        # extract waveforms\n        if \"whiten\" in waveform_params:\n            if waveform_params.pop(\"whiten\"):\n                recording = sp.whiten(recording, dtype=np.float64)\n\n        waveforms_dir = temp_dir + \"/\" + str(key[\"metric_curation_id\"])\n        os.makedirs(waveforms_dir, exist_ok=True)\n\n        logger.info(\"Extracting waveforms...\")\n        waveforms = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveforms_dir,\n            overwrite=True,\n            **waveform_params,\n        )\n        # compute metrics\n        logger.info(\"Computing metrics...\")\n        metrics = {}\n        for metric_name, metric_param_dict in metric_params.items():\n            metrics[metric_name] = self._compute_metric(\n                waveforms, metric_name, **metric_param_dict\n            )\n        if metrics[\"nn_isolation\"]:\n            metrics[\"nn_isolation\"] = {\n                unit_id: value[0]\n                for unit_id, value in metrics[\"nn_isolation\"].items()\n            }\n\n        logger.info(\"Applying curation...\")\n        labels = self._compute_labels(metrics, label_params)\n        merge_groups = self._compute_merge_groups(metrics, merge_params)\n\n        logger.info(\"Saving to NWB...\")\n        (\n            key[\"analysis_file_name\"],\n            key[\"object_id\"],\n        ) = _write_metric_curation_to_nwb(\n            nwb_file_name, waveforms, metrics, labels, merge_groups\n        )\n\n        # INSERT\n        AnalysisNwbfile().add(\n            nwb_file_name,\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @classmethod\n    def get_waveforms(cls):\n        return NotImplementedError\n\n    @classmethod\n    def get_metrics(cls, key: dict):\n        \"\"\"Returns metrics identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id, metric_param_name, metric_params = (\n            cls * MetricCurationSelection * MetricParameters &amp; key\n        ).fetch1(\n            \"analysis_file_name\",\n            \"object_id\",\n            \"metric_param_name\",\n            \"metric_params\",\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        return {\n            name: dict(zip(units.index, units[name])) for name in metric_params\n        }\n\n    @classmethod\n    def get_labels(cls, key: dict):\n        \"\"\"Returns curation labels identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id = (cls &amp; key).fetch1(\n            \"analysis_file_name\", \"object_id\"\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        return dict(zip(units.index, units[\"curation_label\"]))\n\n    @classmethod\n    def get_merge_groups(cls, key: dict):\n        \"\"\"Returns merge groups identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id = (cls &amp; key).fetch1(\n            \"analysis_file_name\", \"object_id\"\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        merge_group_dict = dict(zip(units.index, units[\"merge_groups\"]))\n\n        return _merge_dict_to_list(merge_group_dict)\n\n    @staticmethod\n    def _compute_metric(waveform_extractor, metric_name, **metric_params):\n        metric_func = _metric_name_to_func[metric_name]\n\n        peak_sign_metrics = [\"snr\", \"peak_offset\", \"peak_channel\"]\n        if metric_name in peak_sign_metrics:\n            if \"peak_sign\" not in metric_params:\n                raise Exception(\n                    f\"{peak_sign_metrics} metrics require peak_sign\",\n                    \"to be defined in the metric parameters\",\n                )\n            return metric_func(\n                waveform_extractor,\n                peak_sign=metric_params.pop(\"peak_sign\"),\n                **metric_params,\n            )\n\n        return {\n            unit_id: metric_func(waveform_extractor, this_unit_id=unit_id)\n            for unit_id in waveform_extractor.sorting.get_unit_ids()\n        }\n\n    @staticmethod\n    def _compute_labels(\n        metrics: Dict[str, Dict[str, Union[float, List[float]]]],\n        label_params: Dict[str, List[Any]],\n    ) -&gt; Dict[str, List[str]]:\n        \"\"\"Computes the labels based on the metric and label parameters.\n\n        Parameters\n        ----------\n        quality_metrics : dict\n            Example: {\"snr\" : {\"1\" : 2, \"2\" : 0.1, \"3\" : 2.3}}\n            This indicates that the values of the \"snr\" quality metric\n            for the units \"1\", \"2\", \"3\" are 2, 0.1, and 2.3, respectively.\n\n        label_params : dict\n            Example: {\n                        \"snr\" : [(\"&gt;\", 1, [\"good\", \"mua\"]),\n                                 (\"&lt;\", 1, [\"noise\"])]\n                     }\n            This indicates that units with values of the \"snr\" quality metric\n            greater than 1 should be given the labels \"good\" and \"mua\" and values\n            less than 1 should be given the label \"noise\".\n\n        Returns\n        -------\n        labels : dict\n            Example: {\"1\" : [\"good\", \"mua\"], \"2\" : [\"noise\"], \"3\" : [\"good\", \"mua\"]}\n\n        \"\"\"\n        if not label_params:\n            return {}\n\n        unit_ids = [\n            unit_id for unit_id in metrics[list(metrics.keys())[0]].keys()\n        ]\n        labels = {unit_id: [] for unit_id in unit_ids}\n\n        for metric in label_params:\n            if metric not in metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n                continue\n\n            condition = label_params[metric]\n            if not len(condition) == 3:\n                raise ValueError(f\"Condition {condition} must be of length 3\")\n\n            compare = _comparison_to_function[condition[0]]\n            for unit_id in unit_ids:\n                if compare(\n                    metrics[metric][unit_id],\n                    condition[1],\n                ):\n                    labels[unit_id].extend(label_params[metric][2])\n        return labels\n\n    @staticmethod\n    def _compute_merge_groups(\n        metrics: Dict[str, Dict[str, Union[float, List[float]]]],\n        merge_params: Dict[str, List[Any]],\n    ) -&gt; Dict[str, List[str]]:\n        \"\"\"Identifies units to be merged based on the metrics and merge parameters.\n\n        Parameters\n        ---------\n        quality_metrics : dict\n            Example: {\"cosine_similarity\" : {\n                                             \"1\" : {\"1\" : 1.00, \"2\" : 0.10, \"3\": 0.95},\n                                             \"2\" : {\"1\" : 0.10, \"2\" : 1.00, \"3\": 0.70},\n                                             \"3\" : {\"1\" : 0.95, \"2\" : 0.70, \"3\": 1.00}\n                                            }}\n            This shows the pairwise values of the \"cosine_similarity\" quality metric\n            for the units \"1\", \"2\", \"3\" as a nested dict.\n\n        merge_params : dict\n            Example: {\"cosine_similarity\" : [\"&gt;\", 0.9]}\n            This indicates that units with values of the \"cosine_similarity\" quality metric\n            greater than 0.9 should be placed in the same merge group.\n\n\n        Returns\n        -------\n        merge_groups : dict\n            Example: {\"1\" : [\"3\"], \"2\" : [], \"3\" : [\"1\"]}\n\n        \"\"\"\n\n        if not merge_params:\n            return []\n\n        unit_ids = list(metrics[list(metrics.keys())[0]].keys())\n        merge_groups = {unit_id: [] for unit_id in unit_ids}\n        for metric in merge_params:\n            if metric not in metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n                continue\n            compare = _comparison_to_function[merge_params[metric][0]]\n            for unit_id in unit_ids:\n                other_unit_ids = [\n                    other_unit_id\n                    for other_unit_id in unit_ids\n                    if other_unit_id != unit_id\n                ]\n                for other_unit_id in other_unit_ids:\n                    if compare(\n                        metrics[metric][unit_id][other_unit_id],\n                        merge_params[metric][1],\n                    ):\n                        merge_groups[unit_id].extend(other_unit_id)\n        return merge_groups\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_metrics", "title": "<code>get_metrics(key)</code>  <code>classmethod</code>", "text": "<p>Returns metrics identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_metrics(cls, key: dict):\n    \"\"\"Returns metrics identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id, metric_param_name, metric_params = (\n        cls * MetricCurationSelection * MetricParameters &amp; key\n    ).fetch1(\n        \"analysis_file_name\",\n        \"object_id\",\n        \"metric_param_name\",\n        \"metric_params\",\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    return {\n        name: dict(zip(units.index, units[name])) for name in metric_params\n    }\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_labels", "title": "<code>get_labels(key)</code>  <code>classmethod</code>", "text": "<p>Returns curation labels identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_labels(cls, key: dict):\n    \"\"\"Returns curation labels identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id = (cls &amp; key).fetch1(\n        \"analysis_file_name\", \"object_id\"\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    return dict(zip(units.index, units[\"curation_label\"]))\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_merge_groups", "title": "<code>get_merge_groups(key)</code>  <code>classmethod</code>", "text": "<p>Returns merge groups identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_merge_groups(cls, key: dict):\n    \"\"\"Returns merge groups identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id = (cls &amp; key).fetch1(\n        \"analysis_file_name\", \"object_id\"\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    merge_group_dict = dict(zip(units.index, units[\"merge_groups\"]))\n\n    return _merge_dict_to_list(merge_group_dict)\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/", "title": "metric_utils.py", "text": ""}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.compute_isi_violation_fractions", "title": "<code>compute_isi_violation_fractions(waveform_extractor, this_unit_id, isi_threshold_ms=2.0, min_isi_ms=0.0)</code>", "text": "<p>Computes the fraction of interspike interval violations.</p> <p>Parameters:</p> Name Type Description Default <code>waveform_extractor</code> <code>WaveformExtractor</code> <p>The extractor object for the recording.</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def compute_isi_violation_fractions(\n    waveform_extractor: si.WaveformExtractor,\n    this_unit_id: str,\n    isi_threshold_ms: float = 2.0,\n    min_isi_ms: float = 0.0,\n):\n    \"\"\"Computes the fraction of interspike interval violations.\n\n    Parameters\n    ----------\n    waveform_extractor: si.WaveformExtractor\n        The extractor object for the recording.\n\n    \"\"\"\n\n    # Extract the total number of spikes that violated the isi_threshold for each unit\n    _, isi_violation_counts = sq.compute_isi_violations(\n        waveform_extractor,\n        isi_threshold_ms=isi_threshold_ms,\n        min_isi_ms=min_isi_ms,\n    )\n    num_spikes = sq.compute_num_spikes(waveform_extractor)\n    return isi_violation_counts[this_unit_id] / (num_spikes[this_unit_id] - 1)\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_peak_offset", "title": "<code>get_peak_offset(waveform_extractor, peak_sign, **metric_params)</code>", "text": "<p>Computes the shift of the waveform peak from center of window.</p> <p>Parameters:</p> Name Type Description Default <code>waveform_extractor</code> <code>WaveformExtractor</code> <p>The extractor object for the recording.</p> required <code>peak_sign</code> <code>str</code> <p>The sign of the peak to compute. ('neg', 'pos', 'both')</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_peak_offset(\n    waveform_extractor: si.WaveformExtractor, peak_sign: str, **metric_params\n):\n    \"\"\"Computes the shift of the waveform peak from center of window.\n\n    Parameters\n    ----------\n    waveform_extractor: si.WaveformExtractor\n        The extractor object for the recording.\n    peak_sign: str\n        The sign of the peak to compute. ('neg', 'pos', 'both')\n    \"\"\"\n    if \"peak_sign\" in metric_params:\n        del metric_params[\"peak_sign\"]\n    peak_offset_inds = si.get_template_extremum_channel_peak_shift(\n        waveform_extractor=waveform_extractor,\n        peak_sign=peak_sign,\n        **metric_params,\n    )\n    return {key: int(abs(val)) for key, val in peak_offset_inds.items()}\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_peak_channel", "title": "<code>get_peak_channel(waveform_extractor, peak_sign, **metric_params)</code>", "text": "<p>Computes the electrode_id of the channel with the extremum peak for each unit.</p> Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_peak_channel(\n    waveform_extractor: si.WaveformExtractor, peak_sign: str, **metric_params\n):\n    \"\"\"Computes the electrode_id of the channel with the extremum peak for each unit.\"\"\"\n    if \"peak_sign\" in metric_params:\n        del metric_params[\"peak_sign\"]\n    peak_channel_dict = si.get_template_extremum_channel(\n        waveform_extractor=waveform_extractor,\n        peak_sign=peak_sign,\n        **metric_params,\n    )\n    return {key: int(val) for key, val in peak_channel_dict.items()}\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_num_spikes", "title": "<code>get_num_spikes(waveform_extractor, this_unit_id)</code>", "text": "<p>Computes the number of spikes for each unit.</p> Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_num_spikes(waveform_extractor: si.WaveformExtractor, this_unit_id: str):\n    \"\"\"Computes the number of spikes for each unit.\"\"\"\n    return sq.compute_num_spikes(waveform_extractor)[this_unit_id]\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/", "title": "recording.py", "text": ""}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SortGroup", "title": "<code>SortGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SortGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Set of electrodes to spike sort together\n    -&gt; Session\n    sort_group_id: int\n    ---\n    sort_reference_electrode_id = -1: int  # the electrode to use for referencing\n                                           # -1: no reference, -2: common median\n    \"\"\"\n\n    class SortGroupElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; SortGroup\n        -&gt; Electrode\n        \"\"\"\n\n    @classmethod\n    def set_group_by_shank(\n        cls,\n        nwb_file_name: str,\n        references: dict = None,\n        omit_ref_electrode_group=False,\n        omit_unitrode=True,\n    ):\n        \"\"\"Divides electrodes into groups based on their shank position.\n\n        * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n          single group\n        * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n          placed in one group per shank\n        * Bad channels are omitted\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            the name of the NWB file whose electrodes should be put into\n            sorting groups\n        references : dict, optional\n            If passed, used to set references. Otherwise, references set using\n            original reference electrodes from config. Keys: electrode groups.\n            Values: reference electrode.\n        omit_ref_electrode_group : bool\n            Optional. If True, no sort group is defined for electrode group of\n            reference.\n        omit_unitrode : bool\n            Optional. If True, no sort groups are defined for unitrodes.\n        \"\"\"\n        # delete any current groups\n        # (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n        e_groups.sort(key=int)  # sort electrode groups numerically\n        sort_group = 0\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        for e_group in e_groups:\n            # for each electrode group, get a list of the unique shank numbers\n            shank_list = np.unique(\n                electrodes[\"probe_shank\"][\n                    electrodes[\"electrode_group_name\"] == e_group\n                ]\n            )\n            sge_key[\"electrode_group_name\"] = e_group\n            # get the indices of all electrodes in this group / shank and set their sorting group\n            for shank in shank_list:\n                sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n                # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n                if not references:\n                    shank_elect_ref = electrodes[\n                        \"original_reference_electrode\"\n                    ][\n                        np.logical_and(\n                            electrodes[\"electrode_group_name\"] == e_group,\n                            electrodes[\"probe_shank\"] == shank,\n                        )\n                    ]\n                    if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                        sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                            0\n                        ]\n                    else:\n                        ValueError(\n                            f\"Error in electrode group {e_group}: reference \"\n                            + \"electrodes are not all the same\"\n                        )\n                else:\n                    if e_group not in references.keys():\n                        raise Exception(\n                            f\"electrode group {e_group} not a key in \"\n                            + \"references, so cannot set reference\"\n                        )\n                    else:\n                        sg_key[\"sort_reference_electrode_id\"] = references[\n                            e_group\n                        ]\n                # Insert sort group and sort group electrodes\n                reference_electrode_group = electrodes[\n                    electrodes[\"electrode_id\"]\n                    == sg_key[\"sort_reference_electrode_id\"]\n                ][\n                    \"electrode_group_name\"\n                ]  # reference for this electrode group\n                if (\n                    len(reference_electrode_group) == 1\n                ):  # unpack single reference\n                    reference_electrode_group = reference_electrode_group[0]\n                elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                    len(reference_electrode_group) != 1\n                ):\n                    raise Exception(\n                        \"Should have found exactly one electrode group for \"\n                        + \"reference electrode, but found \"\n                        + f\"{len(reference_electrode_group)}.\"\n                    )\n                if omit_ref_electrode_group and (\n                    str(e_group) == str(reference_electrode_group)\n                ):\n                    logger.warn(\n                        f\"Omitting electrode group {e_group} from sort groups \"\n                        + \"because contains reference.\"\n                    )\n                    continue\n                shank_elect = electrodes[\"electrode_id\"][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if (\n                    omit_unitrode and len(shank_elect) == 1\n                ):  # omit unitrodes if indicated\n                    logger.warn(\n                        f\"Omitting electrode group {e_group}, shank {shank} \"\n                        + \"from sort groups because unitrode.\"\n                    )\n                    continue\n                cls.insert1(sg_key, skip_duplicates=True)\n                for elect in shank_elect:\n                    sge_key[\"electrode_id\"] = elect\n                    cls.SortGroupElectrode().insert1(\n                        sge_key, skip_duplicates=True\n                    )\n                sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SortGroup.set_group_by_shank", "title": "<code>set_group_by_shank(nwb_file_name, references=None, omit_ref_electrode_group=False, omit_unitrode=True)</code>  <code>classmethod</code>", "text": "<p>Divides electrodes into groups based on their shank position.</p> <ul> <li>Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a   single group</li> <li>Electrodes from probes with multiple shanks (e.g. polymer probes) are   placed in one group per shank</li> <li>Bad channels are omitted</li> </ul> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the NWB file whose electrodes should be put into sorting groups</p> required <code>references</code> <code>dict</code> <p>If passed, used to set references. Otherwise, references set using original reference electrodes from config. Keys: electrode groups. Values: reference electrode.</p> <code>None</code> <code>omit_ref_electrode_group</code> <code>bool</code> <p>Optional. If True, no sort group is defined for electrode group of reference.</p> <code>False</code> <code>omit_unitrode</code> <code>bool</code> <p>Optional. If True, no sort groups are defined for unitrodes.</p> <code>True</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef set_group_by_shank(\n    cls,\n    nwb_file_name: str,\n    references: dict = None,\n    omit_ref_electrode_group=False,\n    omit_unitrode=True,\n):\n    \"\"\"Divides electrodes into groups based on their shank position.\n\n    * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n      single group\n    * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n      placed in one group per shank\n    * Bad channels are omitted\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        the name of the NWB file whose electrodes should be put into\n        sorting groups\n    references : dict, optional\n        If passed, used to set references. Otherwise, references set using\n        original reference electrodes from config. Keys: electrode groups.\n        Values: reference electrode.\n    omit_ref_electrode_group : bool\n        Optional. If True, no sort group is defined for electrode group of\n        reference.\n    omit_unitrode : bool\n        Optional. If True, no sort groups are defined for unitrodes.\n    \"\"\"\n    # delete any current groups\n    # (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n    e_groups.sort(key=int)  # sort electrode groups numerically\n    sort_group = 0\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    for e_group in e_groups:\n        # for each electrode group, get a list of the unique shank numbers\n        shank_list = np.unique(\n            electrodes[\"probe_shank\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n        )\n        sge_key[\"electrode_group_name\"] = e_group\n        # get the indices of all electrodes in this group / shank and set their sorting group\n        for shank in shank_list:\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n            # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n            if not references:\n                shank_elect_ref = electrodes[\n                    \"original_reference_electrode\"\n                ][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                    sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                        0\n                    ]\n                else:\n                    ValueError(\n                        f\"Error in electrode group {e_group}: reference \"\n                        + \"electrodes are not all the same\"\n                    )\n            else:\n                if e_group not in references.keys():\n                    raise Exception(\n                        f\"electrode group {e_group} not a key in \"\n                        + \"references, so cannot set reference\"\n                    )\n                else:\n                    sg_key[\"sort_reference_electrode_id\"] = references[\n                        e_group\n                    ]\n            # Insert sort group and sort group electrodes\n            reference_electrode_group = electrodes[\n                electrodes[\"electrode_id\"]\n                == sg_key[\"sort_reference_electrode_id\"]\n            ][\n                \"electrode_group_name\"\n            ]  # reference for this electrode group\n            if (\n                len(reference_electrode_group) == 1\n            ):  # unpack single reference\n                reference_electrode_group = reference_electrode_group[0]\n            elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                len(reference_electrode_group) != 1\n            ):\n                raise Exception(\n                    \"Should have found exactly one electrode group for \"\n                    + \"reference electrode, but found \"\n                    + f\"{len(reference_electrode_group)}.\"\n                )\n            if omit_ref_electrode_group and (\n                str(e_group) == str(reference_electrode_group)\n            ):\n                logger.warn(\n                    f\"Omitting electrode group {e_group} from sort groups \"\n                    + \"because contains reference.\"\n                )\n                continue\n            shank_elect = electrodes[\"electrode_id\"][\n                np.logical_and(\n                    electrodes[\"electrode_group_name\"] == e_group,\n                    electrodes[\"probe_shank\"] == shank,\n                )\n            ]\n            if (\n                omit_unitrode and len(shank_elect) == 1\n            ):  # omit unitrodes if indicated\n                logger.warn(\n                    f\"Omitting electrode group {e_group}, shank {shank} \"\n                    + \"from sort groups because unitrode.\"\n                )\n                continue\n            cls.insert1(sg_key, skip_duplicates=True)\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                cls.SortGroupElectrode().insert1(\n                    sge_key, skip_duplicates=True\n                )\n            sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecordingSelection", "title": "<code>SpikeSortingRecordingSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SpikeSortingRecordingSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Raw voltage traces and parameters. Use `insert_selection` method to insert rows.\n    recording_id: uuid\n    ---\n    -&gt; Raw\n    -&gt; SortGroup\n    -&gt; IntervalList\n    -&gt; SpikeSortingPreprocessingParameters\n    -&gt; LabTeam\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into SpikeSortingRecordingSelection with an\n        automatically generated unique recording ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of Raw, SortGroup, IntervalList,\n            SpikeSortingPreprocessingParameters, LabTeam tables\n\n        Returns\n        -------\n            primary key of SpikeSortingRecordingSelection table\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"recording_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecordingSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into SpikeSortingRecordingSelection with an automatically generated unique recording ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of Raw, SortGroup, IntervalList, SpikeSortingPreprocessingParameters, LabTeam tables</p> required <p>Returns:</p> Type Description <code>    primary key of SpikeSortingRecordingSelection table</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into SpikeSortingRecordingSelection with an\n    automatically generated unique recording ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of Raw, SortGroup, IntervalList,\n        SpikeSortingPreprocessingParameters, LabTeam tables\n\n    Returns\n    -------\n        primary key of SpikeSortingRecordingSelection table\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.warn(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"recording_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecording", "title": "<code>SpikeSortingRecording</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SpikeSortingRecording(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    # Processed recording.\n    -&gt; SpikeSortingRecordingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # Object ID for the processed recording in NWB file\n    \"\"\"\n\n    def make(self, key):\n        # DO:\n        # - get valid times for sort interval\n        # - proprocess recording\n        # - write recording to NWB file\n        sort_interval_valid_times = self._get_sort_interval_valid_times(key)\n        recording, timestamps = self._get_preprocessed_recording(key)\n        recording_nwb_file_name, recording_object_id = _write_recording_to_nwb(\n            recording,\n            timestamps,\n            (SpikeSortingRecordingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        )\n        key[\"analysis_file_name\"] = recording_nwb_file_name\n        key[\"object_id\"] = recording_object_id\n\n        # INSERT:\n        # - valid times into IntervalList\n        # - analysis NWB file holding processed recording into AnalysisNwbfile\n        # - entry into SpikeSortingRecording\n        IntervalList.insert1(\n            {\n                \"nwb_file_name\": (SpikeSortingRecordingSelection &amp; key).fetch1(\n                    \"nwb_file_name\"\n                ),\n                \"interval_list_name\": key[\"recording_id\"],\n                \"valid_times\": sort_interval_valid_times,\n                \"pipeline\": \"spikesorting_recording_v1\",\n            }\n        )\n        AnalysisNwbfile().add(\n            (SpikeSortingRecordingSelection &amp; key).fetch1(\"nwb_file_name\"),\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @classmethod\n    def get_recording(cls, key: dict) -&gt; si.BaseRecording:\n        \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSorting table\n        \"\"\"\n\n        analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        recording = se.read_nwb_recording(\n            analysis_file_abs_path, load_time_vector=True\n        )\n\n        return recording\n\n    @staticmethod\n    def _get_recording_timestamps(recording):\n        if recording.get_num_segments() &gt; 1:\n            frames_per_segment = [0]\n            for i in range(recording.get_num_segments()):\n                frames_per_segment.append(\n                    recording.get_num_frames(segment_index=i)\n                )\n\n            cumsum_frames = np.cumsum(frames_per_segment)\n            total_frames = np.sum(frames_per_segment)\n\n            timestamps = np.zeros((total_frames,))\n            for i in range(recording.get_num_segments()):\n                timestamps[cumsum_frames[i] : cumsum_frames[i + 1]] = (\n                    recording.get_times(segment_index=i)\n                )\n        else:\n            timestamps = recording.get_times()\n        return timestamps\n\n    def _get_sort_interval_valid_times(self, key: dict):\n        \"\"\"Identifies the intersection between sort interval specified by the user\n        and the valid times (times for which neural data exist, excluding e.g. dropped packets).\n\n        Parameters\n        ----------\n        key: dict\n            primary key of SpikeSortingRecordingSelection table\n\n        Returns\n        -------\n        sort_interval_valid_times: ndarray of tuples\n            (start, end) times for valid intervals in the sort interval\n\n        \"\"\"\n        # FETCH: - sort interval - valid times - preprocessing parameters\n        nwb_file_name, sort_interval_name, params = (\n            SpikeSortingPreprocessingParameters * SpikeSortingRecordingSelection\n            &amp; key\n        ).fetch1(\"nwb_file_name\", \"interval_list_name\", \"preproc_params\")\n\n        sort_interval = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": sort_interval_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        valid_interval_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": \"raw data valid times\",\n            }\n        ).fetch1(\"valid_times\")\n\n        # DO: - take intersection between sort interval and valid times\n        return interval_list_intersect(\n            sort_interval,\n            valid_interval_times,\n            min_length=params[\"min_segment_length\"],\n        )\n\n    def _get_preprocessed_recording(self, key: dict):\n        \"\"\"Filters and references a recording.\n\n        - Loads the NWB file created during insertion as a spikeinterface Recording\n        - Slices recording in time (interval) and space (channels);\n          recording chunks from disjoint intervals are concatenated\n        - Applies referencing and bandpass filtering\n\n        Parameters\n        ----------\n        key: dict\n            primary key of SpikeSortingRecordingSelection table\n\n        Returns\n        -------\n        recording: si.Recording\n        \"\"\"\n        # FETCH:\n        # - full path to NWB file\n        # - channels to be included in the sort\n        # - the reference channel\n        # - probe type\n        # - filter parameters\n        nwb_file_name = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"nwb_file_name\"\n        )\n        sort_group_id = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"sort_group_id\"\n        )\n        nwb_file_abs_path = Nwbfile().get_abs_path(nwb_file_name)\n        channel_ids = (\n            SortGroup.SortGroupElectrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"sort_group_id\": sort_group_id,\n            }\n        ).fetch(\"electrode_id\")\n        ref_channel_id = (\n            SortGroup\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"sort_group_id\": sort_group_id,\n            }\n        ).fetch1(\"sort_reference_electrode_id\")\n        recording_channel_ids = np.setdiff1d(channel_ids, ref_channel_id)\n        all_channel_ids = np.unique(np.append(channel_ids, ref_channel_id))\n\n        probe_type_by_channel = []\n        electrode_group_by_channel = []\n        for channel_id in channel_ids:\n            probe_type_by_channel.append(\n                (\n                    Electrode * Probe\n                    &amp; {\n                        \"nwb_file_name\": nwb_file_name,\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"probe_type\")\n            )\n            electrode_group_by_channel.append(\n                (\n                    Electrode\n                    &amp; {\n                        \"nwb_file_name\": nwb_file_name,\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"electrode_group_name\")\n            )\n        probe_type = np.unique(probe_type_by_channel)\n        filter_params = (\n            SpikeSortingPreprocessingParameters * SpikeSortingRecordingSelection\n            &amp; key\n        ).fetch1(\"preproc_params\")\n\n        # DO:\n        # - load NWB file as a spikeinterface Recording\n        # - slice the recording object in time and channels\n        # - apply referencing depending on the option chosen by the user\n        # - apply bandpass filter\n        # - set probe to recording\n        recording = se.read_nwb_recording(\n            nwb_file_abs_path, load_time_vector=True\n        )\n        all_timestamps = recording.get_times()\n\n        # TODO: make sure the following works for recordings that don't have explicit timestamps\n        valid_sort_times = self._get_sort_interval_valid_times(key)\n        valid_sort_times_indices = _consolidate_intervals(\n            valid_sort_times, all_timestamps\n        )\n\n        # slice in time; concatenate disjoint sort intervals\n        if len(valid_sort_times_indices) &gt; 1:\n            recordings_list = []\n            timestamps = []\n            for interval_indices in valid_sort_times_indices:\n                recording_single = recording.frame_slice(\n                    start_frame=interval_indices[0],\n                    end_frame=interval_indices[1],\n                )\n                recordings_list.append(recording_single)\n                timestamps.extend(\n                    all_timestamps[interval_indices[0] : interval_indices[1]]\n                )\n            recording = si.concatenate_recordings(recordings_list)\n        else:\n            recording = recording.frame_slice(\n                start_frame=valid_sort_times_indices[0][0],\n                end_frame=valid_sort_times_indices[0][1],\n            )\n            timestamps = all_timestamps[\n                valid_sort_times_indices[0][0] : valid_sort_times_indices[0][1]\n            ]\n\n        # slice in channels; include ref channel in first slice, then exclude it in second slice\n        if ref_channel_id &gt;= 0:\n            recording = recording.channel_slice(channel_ids=all_channel_ids)\n            recording = si.preprocessing.common_reference(\n                recording,\n                reference=\"single\",\n                ref_channel_ids=ref_channel_id,\n                dtype=np.float64,\n            )\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n        elif ref_channel_id == -2:\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n            recording = si.preprocessing.common_reference(\n                recording,\n                reference=\"global\",\n                operator=\"median\",\n                dtype=np.float64,\n            )\n        elif ref_channel_id == -1:\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n        else:\n            raise ValueError(\n                \"Invalid reference channel ID. Use -1 to skip referencing. Use \"\n                + \"-2 to reference via global median. Use positive integer to \"\n                + \"reference to a specific channel.\"\n            )\n\n        recording = si.preprocessing.bandpass_filter(\n            recording,\n            freq_min=filter_params[\"frequency_min\"],\n            freq_max=filter_params[\"frequency_max\"],\n            dtype=np.float64,\n        )\n\n        # if the sort group is a tetrode, change the channel location\n        # (necessary because the channel location for tetrodes are not set properly)\n        if (\n            len(probe_type) == 1\n            and probe_type[0] == \"tetrode_12.5\"\n            and len(recording_channel_ids) == 4\n            and len(np.unique(electrode_group_by_channel)) == 1\n        ):\n            tetrode = pi.Probe(ndim=2)\n            position = [[0, 0], [0, 12.5], [12.5, 0], [12.5, 12.5]]\n            tetrode.set_contacts(\n                position, shapes=\"circle\", shape_params={\"radius\": 6.25}\n            )\n            tetrode.set_contact_ids(channel_ids)\n            tetrode.set_device_channel_indices(np.arange(4))\n            recording = recording.set_probe(tetrode, in_place=True)\n\n        return recording, np.asarray(timestamps)\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecording.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Get recording related to this curation as spikeinterface BaseRecording</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSorting table</p> required Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef get_recording(cls, key: dict) -&gt; si.BaseRecording:\n    \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSorting table\n    \"\"\"\n\n    analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    recording = se.read_nwb_recording(\n        analysis_file_abs_path, load_time_vector=True\n    )\n\n    return recording\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeInterfaceRecordingDataChunkIterator", "title": "<code>SpikeInterfaceRecordingDataChunkIterator</code>", "text": "<p>             Bases: <code>GenericDataChunkIterator</code></p> <p>DataChunkIterator specifically for use on RecordingExtractor objects.</p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>class SpikeInterfaceRecordingDataChunkIterator(GenericDataChunkIterator):\n    \"\"\"DataChunkIterator specifically for use on RecordingExtractor objects.\"\"\"\n\n    def __init__(\n        self,\n        recording: si.BaseRecording,\n        segment_index: int = 0,\n        return_scaled: bool = False,\n        buffer_gb: Optional[float] = None,\n        buffer_shape: Optional[tuple] = None,\n        chunk_mb: Optional[float] = None,\n        chunk_shape: Optional[tuple] = None,\n        display_progress: bool = False,\n        progress_bar_options: Optional[dict] = None,\n    ):\n        \"\"\"\n        Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n        Parameters\n        ----------\n        recording : si.BaseRecording\n            The SpikeInterfaceRecording object which handles the data access.\n        segment_index : int, optional\n            The recording segment to iterate on.\n            Defaults to 0.\n        return_scaled : bool, optional\n            Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n            Defaults to False.\n        buffer_gb : float, optional\n            The upper bound on size in gigabytes (GB) of each selection from the iteration.\n            The buffer_shape will be set implicitly by this argument.\n            Cannot be set if `buffer_shape` is also specified.\n            The default is 1GB.\n        buffer_shape : tuple, optional\n            Manual specification of buffer shape to return on each iteration.\n            Must be a multiple of chunk_shape along each axis.\n            Cannot be set if `buffer_gb` is also specified.\n            The default is None.\n        chunk_mb : float, optional\n            The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n            The chunk_shape will be set implicitly by this argument.\n            Cannot be set if `chunk_shape` is also specified.\n            The default is 1MB, as recommended by the HDF5 group. For more details, see\n            https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n        chunk_shape : tuple, optional\n            Manual specification of the internal chunk shape for the HDF5 dataset.\n            Cannot be set if `chunk_mb` is also specified.\n            The default is None.\n        display_progress : bool, optional\n            Display a progress bar with iteration rate and estimated completion time.\n        progress_bar_options : dict, optional\n            Dictionary of keyword arguments to be passed directly to tqdm.\n            See https://github.com/tqdm/tqdm#parameters for options.\n        \"\"\"\n        self.recording = recording\n        self.segment_index = segment_index\n        self.return_scaled = return_scaled\n        self.channel_ids = recording.get_channel_ids()\n        super().__init__(\n            buffer_gb=buffer_gb,\n            buffer_shape=buffer_shape,\n            chunk_mb=chunk_mb,\n            chunk_shape=chunk_shape,\n            display_progress=display_progress,\n            progress_bar_options=progress_bar_options,\n        )\n\n    def _get_data(self, selection: Tuple[slice]) -&gt; Iterable:\n        return self.recording.get_traces(\n            segment_index=self.segment_index,\n            channel_ids=self.channel_ids[selection[1]],\n            start_frame=selection[0].start,\n            end_frame=selection[0].stop,\n            return_scaled=self.return_scaled,\n        )\n\n    def _get_dtype(self):\n        return self.recording.get_dtype()\n\n    def _get_maxshape(self):\n        return (\n            self.recording.get_num_samples(segment_index=self.segment_index),\n            self.recording.get_num_channels(),\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeInterfaceRecordingDataChunkIterator.__init__", "title": "<code>__init__(recording, segment_index=0, return_scaled=False, buffer_gb=None, buffer_shape=None, chunk_mb=None, chunk_shape=None, display_progress=False, progress_bar_options=None)</code>", "text": "<p>Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>BaseRecording</code> <p>The SpikeInterfaceRecording object which handles the data access.</p> required <code>segment_index</code> <code>int</code> <p>The recording segment to iterate on. Defaults to 0.</p> <code>0</code> <code>return_scaled</code> <code>bool</code> <p>Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False). Defaults to False.</p> <code>False</code> <code>buffer_gb</code> <code>float</code> <p>The upper bound on size in gigabytes (GB) of each selection from the iteration. The buffer_shape will be set implicitly by this argument. Cannot be set if <code>buffer_shape</code> is also specified. The default is 1GB.</p> <code>None</code> <code>buffer_shape</code> <code>tuple</code> <p>Manual specification of buffer shape to return on each iteration. Must be a multiple of chunk_shape along each axis. Cannot be set if <code>buffer_gb</code> is also specified. The default is None.</p> <code>None</code> <code>chunk_mb</code> <code>float</code> <p>The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset. The chunk_shape will be set implicitly by this argument. Cannot be set if <code>chunk_shape</code> is also specified. The default is 1MB, as recommended by the HDF5 group. For more details, see https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf</p> <code>None</code> <code>chunk_shape</code> <code>tuple</code> <p>Manual specification of the internal chunk shape for the HDF5 dataset. Cannot be set if <code>chunk_mb</code> is also specified. The default is None.</p> <code>None</code> <code>display_progress</code> <code>bool</code> <p>Display a progress bar with iteration rate and estimated completion time.</p> <code>False</code> <code>progress_bar_options</code> <code>dict</code> <p>Dictionary of keyword arguments to be passed directly to tqdm. See https://github.com/tqdm/tqdm#parameters for options.</p> <code>None</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>def __init__(\n    self,\n    recording: si.BaseRecording,\n    segment_index: int = 0,\n    return_scaled: bool = False,\n    buffer_gb: Optional[float] = None,\n    buffer_shape: Optional[tuple] = None,\n    chunk_mb: Optional[float] = None,\n    chunk_shape: Optional[tuple] = None,\n    display_progress: bool = False,\n    progress_bar_options: Optional[dict] = None,\n):\n    \"\"\"\n    Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n    Parameters\n    ----------\n    recording : si.BaseRecording\n        The SpikeInterfaceRecording object which handles the data access.\n    segment_index : int, optional\n        The recording segment to iterate on.\n        Defaults to 0.\n    return_scaled : bool, optional\n        Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n        Defaults to False.\n    buffer_gb : float, optional\n        The upper bound on size in gigabytes (GB) of each selection from the iteration.\n        The buffer_shape will be set implicitly by this argument.\n        Cannot be set if `buffer_shape` is also specified.\n        The default is 1GB.\n    buffer_shape : tuple, optional\n        Manual specification of buffer shape to return on each iteration.\n        Must be a multiple of chunk_shape along each axis.\n        Cannot be set if `buffer_gb` is also specified.\n        The default is None.\n    chunk_mb : float, optional\n        The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n        The chunk_shape will be set implicitly by this argument.\n        Cannot be set if `chunk_shape` is also specified.\n        The default is 1MB, as recommended by the HDF5 group. For more details, see\n        https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n    chunk_shape : tuple, optional\n        Manual specification of the internal chunk shape for the HDF5 dataset.\n        Cannot be set if `chunk_mb` is also specified.\n        The default is None.\n    display_progress : bool, optional\n        Display a progress bar with iteration rate and estimated completion time.\n    progress_bar_options : dict, optional\n        Dictionary of keyword arguments to be passed directly to tqdm.\n        See https://github.com/tqdm/tqdm#parameters for options.\n    \"\"\"\n    self.recording = recording\n    self.segment_index = segment_index\n    self.return_scaled = return_scaled\n    self.channel_ids = recording.get_channel_ids()\n    super().__init__(\n        buffer_gb=buffer_gb,\n        buffer_shape=buffer_shape,\n        chunk_mb=chunk_mb,\n        chunk_shape=chunk_shape,\n        display_progress=display_progress,\n        progress_bar_options=progress_bar_options,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.TimestampsDataChunkIterator", "title": "<code>TimestampsDataChunkIterator</code>", "text": "<p>             Bases: <code>GenericDataChunkIterator</code></p> <p>DataChunkIterator specifically for use on RecordingExtractor objects.</p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>class TimestampsDataChunkIterator(GenericDataChunkIterator):\n    \"\"\"DataChunkIterator specifically for use on RecordingExtractor objects.\"\"\"\n\n    def __init__(\n        self,\n        recording: si.BaseRecording,\n        segment_index: int = 0,\n        return_scaled: bool = False,\n        buffer_gb: Optional[float] = None,\n        buffer_shape: Optional[tuple] = None,\n        chunk_mb: Optional[float] = None,\n        chunk_shape: Optional[tuple] = None,\n        display_progress: bool = False,\n        progress_bar_options: Optional[dict] = None,\n    ):\n        \"\"\"\n        Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n        Parameters\n        ----------\n        recording : SpikeInterfaceRecording\n            The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.\n        segment_index : int, optional\n            The recording segment to iterate on.\n            Defaults to 0.\n        return_scaled : bool, optional\n            Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n            Defaults to False.\n        buffer_gb : float, optional\n            The upper bound on size in gigabytes (GB) of each selection from the iteration.\n            The buffer_shape will be set implicitly by this argument.\n            Cannot be set if `buffer_shape` is also specified.\n            The default is 1GB.\n        buffer_shape : tuple, optional\n            Manual specification of buffer shape to return on each iteration.\n            Must be a multiple of chunk_shape along each axis.\n            Cannot be set if `buffer_gb` is also specified.\n            The default is None.\n        chunk_mb : float, optional\n            The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n            The chunk_shape will be set implicitly by this argument.\n            Cannot be set if `chunk_shape` is also specified.\n            The default is 1MB, as recommended by the HDF5 group. For more details, see\n            https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n        chunk_shape : tuple, optional\n            Manual specification of the internal chunk shape for the HDF5 dataset.\n            Cannot be set if `chunk_mb` is also specified.\n            The default is None.\n        display_progress : bool, optional\n            Display a progress bar with iteration rate and estimated completion time.\n        progress_bar_options : dict, optional\n            Dictionary of keyword arguments to be passed directly to tqdm.\n            See https://github.com/tqdm/tqdm#parameters for options.\n        \"\"\"\n        self.recording = recording\n        self.segment_index = segment_index\n        self.return_scaled = return_scaled\n        self.channel_ids = recording.get_channel_ids()\n        super().__init__(\n            buffer_gb=buffer_gb,\n            buffer_shape=buffer_shape,\n            chunk_mb=chunk_mb,\n            chunk_shape=chunk_shape,\n            display_progress=display_progress,\n            progress_bar_options=progress_bar_options,\n        )\n\n    # change channel id to always be first channel\n    def _get_data(self, selection: Tuple[slice]) -&gt; Iterable:\n        return self.recording.get_traces(\n            segment_index=self.segment_index,\n            channel_ids=[0],\n            start_frame=selection[0].start,\n            end_frame=selection[0].stop,\n            return_scaled=self.return_scaled,\n        )\n\n    def _get_dtype(self):\n        return self.recording.get_dtype()\n\n    # remove the last dim for the timestamps since it is always just a 1D vector\n    def _get_maxshape(self):\n        return (\n            self.recording.get_num_samples(segment_index=self.segment_index),\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.TimestampsDataChunkIterator.__init__", "title": "<code>__init__(recording, segment_index=0, return_scaled=False, buffer_gb=None, buffer_shape=None, chunk_mb=None, chunk_shape=None, display_progress=False, progress_bar_options=None)</code>", "text": "<p>Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>SpikeInterfaceRecording</code> <p>The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.</p> required <code>segment_index</code> <code>int</code> <p>The recording segment to iterate on. Defaults to 0.</p> <code>0</code> <code>return_scaled</code> <code>bool</code> <p>Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False). Defaults to False.</p> <code>False</code> <code>buffer_gb</code> <code>float</code> <p>The upper bound on size in gigabytes (GB) of each selection from the iteration. The buffer_shape will be set implicitly by this argument. Cannot be set if <code>buffer_shape</code> is also specified. The default is 1GB.</p> <code>None</code> <code>buffer_shape</code> <code>tuple</code> <p>Manual specification of buffer shape to return on each iteration. Must be a multiple of chunk_shape along each axis. Cannot be set if <code>buffer_gb</code> is also specified. The default is None.</p> <code>None</code> <code>chunk_mb</code> <code>float</code> <p>The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset. The chunk_shape will be set implicitly by this argument. Cannot be set if <code>chunk_shape</code> is also specified. The default is 1MB, as recommended by the HDF5 group. For more details, see https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf</p> <code>None</code> <code>chunk_shape</code> <code>tuple</code> <p>Manual specification of the internal chunk shape for the HDF5 dataset. Cannot be set if <code>chunk_mb</code> is also specified. The default is None.</p> <code>None</code> <code>display_progress</code> <code>bool</code> <p>Display a progress bar with iteration rate and estimated completion time.</p> <code>False</code> <code>progress_bar_options</code> <code>dict</code> <p>Dictionary of keyword arguments to be passed directly to tqdm. See https://github.com/tqdm/tqdm#parameters for options.</p> <code>None</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>def __init__(\n    self,\n    recording: si.BaseRecording,\n    segment_index: int = 0,\n    return_scaled: bool = False,\n    buffer_gb: Optional[float] = None,\n    buffer_shape: Optional[tuple] = None,\n    chunk_mb: Optional[float] = None,\n    chunk_shape: Optional[tuple] = None,\n    display_progress: bool = False,\n    progress_bar_options: Optional[dict] = None,\n):\n    \"\"\"\n    Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n    Parameters\n    ----------\n    recording : SpikeInterfaceRecording\n        The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.\n    segment_index : int, optional\n        The recording segment to iterate on.\n        Defaults to 0.\n    return_scaled : bool, optional\n        Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n        Defaults to False.\n    buffer_gb : float, optional\n        The upper bound on size in gigabytes (GB) of each selection from the iteration.\n        The buffer_shape will be set implicitly by this argument.\n        Cannot be set if `buffer_shape` is also specified.\n        The default is 1GB.\n    buffer_shape : tuple, optional\n        Manual specification of buffer shape to return on each iteration.\n        Must be a multiple of chunk_shape along each axis.\n        Cannot be set if `buffer_gb` is also specified.\n        The default is None.\n    chunk_mb : float, optional\n        The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n        The chunk_shape will be set implicitly by this argument.\n        Cannot be set if `chunk_shape` is also specified.\n        The default is 1MB, as recommended by the HDF5 group. For more details, see\n        https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n    chunk_shape : tuple, optional\n        Manual specification of the internal chunk shape for the HDF5 dataset.\n        Cannot be set if `chunk_mb` is also specified.\n        The default is None.\n    display_progress : bool, optional\n        Display a progress bar with iteration rate and estimated completion time.\n    progress_bar_options : dict, optional\n        Dictionary of keyword arguments to be passed directly to tqdm.\n        See https://github.com/tqdm/tqdm#parameters for options.\n    \"\"\"\n    self.recording = recording\n    self.segment_index = segment_index\n    self.return_scaled = return_scaled\n    self.channel_ids = recording.get_channel_ids()\n    super().__init__(\n        buffer_gb=buffer_gb,\n        buffer_shape=buffer_shape,\n        chunk_mb=chunk_mb,\n        chunk_shape=chunk_shape,\n        display_progress=display_progress,\n        progress_bar_options=progress_bar_options,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/", "title": "sorting.py", "text": ""}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSortingSelection", "title": "<code>SpikeSortingSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@schema\nclass SpikeSortingSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Processed recording and spike sorting parameters. Use `insert_selection` method to insert rows.\n    sorting_id: uuid\n    ---\n    -&gt; SpikeSortingRecording\n    -&gt; SpikeSorterParameters\n    -&gt; IntervalList\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into SpikeSortingSelection with an\n        automatically generated unique sorting ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables\n\n        Returns\n        -------\n        sorting_id : uuid\n            the unique sorting ID serving as primary key for SpikeSorting\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.info(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"sorting_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSortingSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into SpikeSortingSelection with an automatically generated unique sorting ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables</p> required <p>Returns:</p> Name Type Description <code>sorting_id</code> <code>uuid</code> <p>the unique sorting ID serving as primary key for SpikeSorting</p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into SpikeSortingSelection with an\n    automatically generated unique sorting ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables\n\n    Returns\n    -------\n    sorting_id : uuid\n        the unique sorting ID serving as primary key for SpikeSorting\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.info(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"sorting_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting", "title": "<code>SpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@schema\nclass SpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40)          # Object ID for the sorting in NWB file\n    time_of_sort: int               # in Unix time, to the nearest second\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Runs spike sorting on the data and parameters specified by the\n        SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n        \"\"\"\n        # FETCH:\n        # - information about the recording\n        # - artifact free intervals\n        # - spike sorter and sorter params\n\n        recording_key = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1()\n        artifact_removed_intervals = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": (SpikeSortingSelection &amp; key).fetch1(\n                    \"nwb_file_name\"\n                ),\n                \"interval_list_name\": (SpikeSortingSelection &amp; key).fetch1(\n                    \"interval_list_name\"\n                ),\n            }\n        ).fetch1(\"valid_times\")\n        sorter, sorter_params = (\n            SpikeSorterParameters * SpikeSortingSelection &amp; key\n        ).fetch1(\"sorter\", \"sorter_params\")\n\n        # DO:\n        # - load recording\n        # - concatenate artifact removed intervals\n        # - run spike sorting\n        # - save output to NWB file\n        recording_analysis_nwb_file_abs_path = AnalysisNwbfile.get_abs_path(\n            recording_key[\"analysis_file_name\"]\n        )\n        recording = se.read_nwb_recording(\n            recording_analysis_nwb_file_abs_path, load_time_vector=True\n        )\n\n        timestamps = recording.get_times()\n\n        artifact_removed_intervals_ind = _consolidate_intervals(\n            artifact_removed_intervals, timestamps\n        )\n\n        # if the artifact removed intervals do not span the entire time range\n        if (\n            (len(artifact_removed_intervals_ind) &gt; 1)\n            or (artifact_removed_intervals_ind[0][0] &gt; 0)\n            or (artifact_removed_intervals_ind[-1][1] &lt; len(timestamps))\n        ):\n            # set the artifact intervals to zero\n            list_triggers = []\n            if artifact_removed_intervals_ind[0][0] &gt; 0:\n                list_triggers.append(\n                    np.array([0, artifact_removed_intervals_ind[0][0]])\n                )\n            for interval_ind in range(len(artifact_removed_intervals_ind) - 1):\n                list_triggers.append(\n                    np.arange(\n                        (artifact_removed_intervals_ind[interval_ind][1] + 1),\n                        artifact_removed_intervals_ind[interval_ind + 1][0],\n                    )\n                )\n            if artifact_removed_intervals_ind[-1][1] &lt; len(timestamps):\n                list_triggers.append(\n                    np.array(\n                        [\n                            artifact_removed_intervals_ind[-1][1],\n                            len(timestamps) - 1,\n                        ]\n                    )\n                )\n\n            list_triggers = [list(np.concatenate(list_triggers))]\n            recording = sip.remove_artifacts(\n                recording=recording,\n                list_triggers=list_triggers,\n                ms_before=None,\n                ms_after=None,\n                mode=\"zeros\",\n            )\n\n        if sorter == \"clusterless_thresholder\":\n            # need to remove tempdir and whiten from sorter_params\n            sorter_params.pop(\"tempdir\", None)\n            sorter_params.pop(\"whiten\", None)\n            sorter_params.pop(\"outputs\", None)\n\n            # Detect peaks for clusterless decoding\n            detected_spikes = detect_peaks(recording, **sorter_params)\n            sorting = si.NumpySorting.from_times_labels(\n                times_list=detected_spikes[\"sample_index\"],\n                labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n                sampling_frequency=recording.get_sampling_frequency(),\n            )\n        else:\n            # Specify tempdir (expected by some sorters like mountainsort4)\n            sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n            sorter_params[\"tempdir\"] = sorter_temp_dir.name\n            os.chmod(sorter_params[\"tempdir\"], 0o777)\n\n            if sorter == \"mountainsort5\":\n                _ = sorter_params.pop(\"tempdir\", None)\n\n            # if whitening is specified in sorter params, apply whitening separately\n            # prior to sorting and turn off \"sorter whitening\"\n            if sorter_params.get(\"whiten\", False):\n                recording = sip.whiten(recording, dtype=np.float64)\n                sorter_params[\"whiten\"] = False\n\n            common_sorter_items = {\n                \"sorter_name\": sorter,\n                \"recording\": recording,\n                \"output_folder\": sorter_temp_dir.name,\n                \"remove_existing_folder\": True,\n            }\n\n            if sorter.lower() in [\"kilosort2_5\", \"kilosort3\", \"ironclust\"]:\n                sorter_params = {\n                    k: v\n                    for k, v in sorter_params.items()\n                    if k\n                    not in [\"tempdir\", \"mp_context\", \"max_threads_per_process\"]\n                }\n                sorting = sis.run_sorter(\n                    **common_sorter_items,\n                    singularity_image=True,\n                    **sorter_params,\n                )\n            else:\n                sorting = sis.run_sorter(\n                    **common_sorter_items,\n                    **sorter_params,\n                )\n        key[\"time_of_sort\"] = int(time.time())\n        sorting = sic.remove_excess_spikes(sorting, recording)\n        key[\"analysis_file_name\"], key[\"object_id\"] = _write_sorting_to_nwb(\n            sorting,\n            timestamps,\n            artifact_removed_intervals,\n            (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        )\n\n        # INSERT\n        # - new entry to AnalysisNwbfile\n        # - new entry to SpikeSorting\n        AnalysisNwbfile().add(\n            (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSorting\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n\n        recording_id = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1(\"recording_id\")\n        recording = SpikeSortingRecording.get_recording(\n            {\"recording_id\": recording_id}\n        )\n        sampling_frequency = recording.get_sampling_frequency()\n        analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.units.to_dataframe()\n        units_dict_list = [\n            {\n                unit_id: np.searchsorted(recording.get_times(), spike_times)\n                for unit_id, spike_times in zip(\n                    units.index, units[\"spike_times\"]\n                )\n            }\n        ]\n\n        sorting = si.NumpySorting.from_unit_dict(\n            units_dict_list, sampling_frequency=sampling_frequency\n        )\n\n        return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting.make", "title": "<code>make(key)</code>", "text": "<p>Runs spike sorting on the data and parameters specified by the SpikeSortingSelection table and inserts a new entry to SpikeSorting table.</p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Runs spike sorting on the data and parameters specified by the\n    SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n    \"\"\"\n    # FETCH:\n    # - information about the recording\n    # - artifact free intervals\n    # - spike sorter and sorter params\n\n    recording_key = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1()\n    artifact_removed_intervals = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": (SpikeSortingSelection &amp; key).fetch1(\n                \"nwb_file_name\"\n            ),\n            \"interval_list_name\": (SpikeSortingSelection &amp; key).fetch1(\n                \"interval_list_name\"\n            ),\n        }\n    ).fetch1(\"valid_times\")\n    sorter, sorter_params = (\n        SpikeSorterParameters * SpikeSortingSelection &amp; key\n    ).fetch1(\"sorter\", \"sorter_params\")\n\n    # DO:\n    # - load recording\n    # - concatenate artifact removed intervals\n    # - run spike sorting\n    # - save output to NWB file\n    recording_analysis_nwb_file_abs_path = AnalysisNwbfile.get_abs_path(\n        recording_key[\"analysis_file_name\"]\n    )\n    recording = se.read_nwb_recording(\n        recording_analysis_nwb_file_abs_path, load_time_vector=True\n    )\n\n    timestamps = recording.get_times()\n\n    artifact_removed_intervals_ind = _consolidate_intervals(\n        artifact_removed_intervals, timestamps\n    )\n\n    # if the artifact removed intervals do not span the entire time range\n    if (\n        (len(artifact_removed_intervals_ind) &gt; 1)\n        or (artifact_removed_intervals_ind[0][0] &gt; 0)\n        or (artifact_removed_intervals_ind[-1][1] &lt; len(timestamps))\n    ):\n        # set the artifact intervals to zero\n        list_triggers = []\n        if artifact_removed_intervals_ind[0][0] &gt; 0:\n            list_triggers.append(\n                np.array([0, artifact_removed_intervals_ind[0][0]])\n            )\n        for interval_ind in range(len(artifact_removed_intervals_ind) - 1):\n            list_triggers.append(\n                np.arange(\n                    (artifact_removed_intervals_ind[interval_ind][1] + 1),\n                    artifact_removed_intervals_ind[interval_ind + 1][0],\n                )\n            )\n        if artifact_removed_intervals_ind[-1][1] &lt; len(timestamps):\n            list_triggers.append(\n                np.array(\n                    [\n                        artifact_removed_intervals_ind[-1][1],\n                        len(timestamps) - 1,\n                    ]\n                )\n            )\n\n        list_triggers = [list(np.concatenate(list_triggers))]\n        recording = sip.remove_artifacts(\n            recording=recording,\n            list_triggers=list_triggers,\n            ms_before=None,\n            ms_after=None,\n            mode=\"zeros\",\n        )\n\n    if sorter == \"clusterless_thresholder\":\n        # need to remove tempdir and whiten from sorter_params\n        sorter_params.pop(\"tempdir\", None)\n        sorter_params.pop(\"whiten\", None)\n        sorter_params.pop(\"outputs\", None)\n\n        # Detect peaks for clusterless decoding\n        detected_spikes = detect_peaks(recording, **sorter_params)\n        sorting = si.NumpySorting.from_times_labels(\n            times_list=detected_spikes[\"sample_index\"],\n            labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n    else:\n        # Specify tempdir (expected by some sorters like mountainsort4)\n        sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n        sorter_params[\"tempdir\"] = sorter_temp_dir.name\n        os.chmod(sorter_params[\"tempdir\"], 0o777)\n\n        if sorter == \"mountainsort5\":\n            _ = sorter_params.pop(\"tempdir\", None)\n\n        # if whitening is specified in sorter params, apply whitening separately\n        # prior to sorting and turn off \"sorter whitening\"\n        if sorter_params.get(\"whiten\", False):\n            recording = sip.whiten(recording, dtype=np.float64)\n            sorter_params[\"whiten\"] = False\n\n        common_sorter_items = {\n            \"sorter_name\": sorter,\n            \"recording\": recording,\n            \"output_folder\": sorter_temp_dir.name,\n            \"remove_existing_folder\": True,\n        }\n\n        if sorter.lower() in [\"kilosort2_5\", \"kilosort3\", \"ironclust\"]:\n            sorter_params = {\n                k: v\n                for k, v in sorter_params.items()\n                if k\n                not in [\"tempdir\", \"mp_context\", \"max_threads_per_process\"]\n            }\n            sorting = sis.run_sorter(\n                **common_sorter_items,\n                singularity_image=True,\n                **sorter_params,\n            )\n        else:\n            sorting = sis.run_sorter(\n                **common_sorter_items,\n                **sorter_params,\n            )\n    key[\"time_of_sort\"] = int(time.time())\n    sorting = sic.remove_excess_spikes(sorting, recording)\n    key[\"analysis_file_name\"], key[\"object_id\"] = _write_sorting_to_nwb(\n        sorting,\n        timestamps,\n        artifact_removed_intervals,\n        (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n    )\n\n    # INSERT\n    # - new entry to AnalysisNwbfile\n    # - new entry to SpikeSorting\n    AnalysisNwbfile().add(\n        (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        key[\"analysis_file_name\"],\n    )\n    self.insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting in the analysis NWB file as spikeinterface BaseSorting</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSorting</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSorting\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n\n    recording_id = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1(\"recording_id\")\n    recording = SpikeSortingRecording.get_recording(\n        {\"recording_id\": recording_id}\n    )\n    sampling_frequency = recording.get_sampling_frequency()\n    analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.units.to_dataframe()\n    units_dict_list = [\n        {\n            unit_id: np.searchsorted(recording.get_times(), spike_times)\n            for unit_id, spike_times in zip(\n                units.index, units[\"spike_times\"]\n            )\n        }\n    ]\n\n    sorting = si.NumpySorting.from_unit_dict(\n        units_dict_list, sampling_frequency=sampling_frequency\n    )\n\n    return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/utils/", "title": "utils.py", "text": ""}, {"location": "api/spikesorting/v1/utils/#src.spyglass.spikesorting.v1.utils.generate_nwb_uuid", "title": "<code>generate_nwb_uuid(nwb_file_name, initial, len_uuid=6)</code>", "text": "<p>Generates a unique identifier related to an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>description</p> required <code>initial</code> <code>str</code> <p>R if recording; A if artifact; S if sorting etc</p> required <code>len_uuid</code> <code>int</code> <p>how many digits of uuid4 to keep</p> <code>6</code> Source code in <code>src/spyglass/spikesorting/v1/utils.py</code> <pre><code>def generate_nwb_uuid(nwb_file_name: str, initial: str, len_uuid: int = 6):\n    \"\"\"Generates a unique identifier related to an NWB file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        _description_\n    initial : str\n        R if recording; A if artifact; S if sorting etc\n    len_uuid : int\n        how many digits of uuid4 to keep\n    \"\"\"\n    uuid4 = str(uuid.uuid4())\n    nwb_uuid = nwb_file_name + \"_\" + initial + \"_\" + uuid4[:len_uuid]\n    return nwb_uuid\n</code></pre>"}, {"location": "api/spikesorting/v1/utils/#src.spyglass.spikesorting.v1.utils.get_spiking_sorting_v1_merge_ids", "title": "<code>get_spiking_sorting_v1_merge_ids(restriction)</code>", "text": "<p>Parses the SpikingSorting V1 pipeline to get a list of merge ids for a given restriction.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>dict</code> <p>A dictionary containing some or all of the following key-value pairs: nwb_file_name : str     name of the nwb file interval_list_name : str     name of the interval list sort_group_name : str     name of the sort group artifact_param_name : str     name of the artifact parameter curation_id : int, optional     id of the curation (if not specified, uses the latest curation)</p> required <p>Returns:</p> Name Type Description <code>merge_id_list</code> <code>list</code> <p>list of merge ids for the given restriction</p> Source code in <code>src/spyglass/spikesorting/v1/utils.py</code> <pre><code>def get_spiking_sorting_v1_merge_ids(restriction: dict):\n    \"\"\"\n    Parses the SpikingSorting V1 pipeline to get a list of merge ids for a given restriction.\n\n    Parameters\n    ----------\n    restriction : dict\n        A dictionary containing some or all of the following key-value pairs:\n        nwb_file_name : str\n            name of the nwb file\n        interval_list_name : str\n            name of the interval list\n        sort_group_name : str\n            name of the sort group\n        artifact_param_name : str\n            name of the artifact parameter\n        curation_id : int, optional\n            id of the curation (if not specified, uses the latest curation)\n    Returns\n    -------\n    merge_id_list : list\n        list of merge ids for the given restriction\n    \"\"\"\n    # list of recording ids\n    recording_id_list = (SpikeSortingRecordingSelection() &amp; restriction).fetch(\n        \"recording_id\"\n    )\n    # list of artifact ids for each recording\n    artifact_id_list = [\n        (\n            ArtifactDetectionSelection() &amp; restriction &amp; {\"recording_id\": id}\n        ).fetch1(\"artifact_id\")\n        for id in recording_id_list\n    ]\n    # list of sorting ids for each recording\n    sorting_restriction = restriction.copy()\n    del sorting_restriction[\"interval_list_name\"]\n    sorting_id_list = []\n    for r_id, a_id in zip(recording_id_list, artifact_id_list):\n        # if sorted with artifact detection\n        if (\n            SpikeSortingSelection()\n            &amp; sorting_restriction\n            &amp; {\"recording_id\": r_id, \"interval_list_name\": a_id}\n        ):\n            sorting_id_list.append(\n                (\n                    SpikeSortingSelection()\n                    &amp; sorting_restriction\n                    &amp; {\"recording_id\": r_id, \"interval_list_name\": a_id}\n                ).fetch1(\"sorting_id\")\n            )\n        # if sorted without artifact detection\n        else:\n            sorting_id_list.append(\n                (\n                    SpikeSortingSelection()\n                    &amp; sorting_restriction\n                    &amp; {\"recording_id\": r_id, \"interval_list_name\": r_id}\n                ).fetch1(\"sorting_id\")\n            )\n    # if curation_id is specified, use that id for each sorting_id\n    if \"curation_id\" in restriction:\n        curation_id = [restriction[\"curation_id\"] for _ in sorting_id_list]\n    # if curation_id is not specified, use the latest curation_id for each sorting_id\n    else:\n        curation_id = [\n            np.max((CurationV1 &amp; {\"sorting_id\": id}).fetch(\"curation_id\"))\n            for id in sorting_id_list\n        ]\n    # list of merge ids for the desired curation(s)\n    merge_id_list = [\n        (\n            SpikeSortingOutput.CurationV1()\n            &amp; {\"sorting_id\": id, \"curation_id\": c_id}\n        ).fetch1(\"merge_id\")\n        for id, c_id in zip(sorting_id_list, curation_id)\n    ]\n    return merge_id_list\n</code></pre>"}, {"location": "api/utils/database_settings/", "title": "database_settings.py", "text": ""}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings", "title": "<code>DatabaseSettings</code>", "text": "Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>class DatabaseSettings:\n    def __init__(\n        self,\n        user_name=None,\n        host_name=None,\n        debug=False,\n        target_database=None,\n        exec_user=None,\n        exec_pass=None,\n    ):\n        \"\"\"Class to manage common database settings\n\n        Roles:\n        - dj_guest:  select for all prefix\n        - dj_collab: select for all prefix, all for user prefix\n        - dj_user:   select for all prefix, all for user prefix, all for shared\n        - dj_admin:     all for all prefix\n\n        Note: To add dj_user role to all those with common access, run:\n            query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"\n            users = dj.conn().query(query).fetchall()\n            for user in users:\n                dj.conn().query(f\"GRANT dj_user TO '{user[0][0]}'@'%';\")\n\n        Parameters\n        ----------\n        user_name : str, optional\n            The name of the user to add to the database. Default from dj.config\n        host_name : str, optional\n            The name of the host to add to the database. Default from dj.config\n        debug : bool, optional\n            Default False. If True, pprint sql instead of running\n        target_database : str, optional\n            Default is mysql. Can also be docker container id\n        exec_user : str, optional\n            User for executing commands. If None, use dj.config\n        exec_pass : str, optional\n            Password for executing commands. If None, use dj.config\n        \"\"\"\n        self.shared_modules = [f\"{m}{ESC}\" for m in SHARED_MODULES]\n        self.user = user_name or dj.config[\"database.user\"]\n        self.host = (\n            host_name or dj.config[\"database.host\"] or \"lmf-db.cin.ucsf.edu\"\n        )\n        self.debug = debug\n        self.target_database = target_database or \"mysql\"\n        self.exec_user = exec_user or dj.config[\"database.user\"]\n        self.exec_pass = exec_pass or dj.config[\"database.password\"]\n\n    @property\n    def _create_roles_dict(self):\n        return dict(\n            guest=[\n                f\"{CREATE_ROLE}`dj_guest`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_guest`;\\n\",\n            ],\n            collab=[\n                f\"{CREATE_ROLE}`dj_collab`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_collab`;\\n\",\n            ],  # also gets own prefix below\n            user=[\n                f\"{CREATE_ROLE}`dj_user`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_user`;\\n\",\n            ]\n            + [\n                f\"{GRANT_ALL}`{module}`.* TO `dj_user`;\\n\"\n                for module in self.shared_modules\n            ],  # also gets own prefix below\n            admin=[\n                f\"{CREATE_ROLE}`dj_admin`;\\n\",\n                f\"{GRANT_ALL}`%`.* TO `dj_admin`;\\n\",\n            ],\n        )\n\n    @property\n    def _create_roles_sql(self):\n        return sum(self._create_roles_dict.values(), [])\n\n    def _create_user_sql(self, role):\n        \"\"\"Create user and grant role\"\"\"\n        return [\n            f\"{CREATE_USR}'{self.user}'@'%'{TEMP_PASS}\\n\",  # create user\n            f\"GRANT {role} TO '{self.user}'@'%';\\n\",  # grant role\n        ]\n\n    @property\n    def _user_prefix_sql(self):\n        \"\"\"Grant user all permissions for user prefix\"\"\"\n        return [\n            f\"{GRANT_ALL}`{self.user}{ESC}`.* TO '{self.user}'@'%';\\n\",\n        ]\n\n    @property\n    def _add_guest_sql(self):\n        return self._create_user_sql(\"dj_guest\")\n\n    @property\n    def _add_collab_sql(self):\n        return self._create_user_sql(\"dj_collab\") + self._user_prefix_sql\n\n    @property\n    def _add_user_sql(self):\n        return self._create_user_sql(\"dj_user\") + self._user_prefix_sql\n\n    @property\n    def _add_admin_sql(self):\n        return self._create_user_sql(\"dj_admin\") + self._user_prefix_sql\n\n    def _add_module_sql(self, module_name):\n        return [f\"{GRANT_ALL}`{module_name}{ESC}`.* TO dj_user;\\n\"]\n\n    def add_guest(self, *args, **kwargs):\n        \"\"\"Add guest user with select permissions to shared modules\"\"\"\n        file = self.write_temp_file(self._add_guest_sql)\n        self.exec(file)\n\n    def add_collab(self, *args, **kwargs):\n        \"\"\"Add collaborator user with full permissions to shared modules\"\"\"\n        file = self.write_temp_file(self._add_collab_sql)\n        self.exec(file)\n\n    def add_user(self, check_exists=False, *args, **kwargs):\n        \"\"\"Add user to database with permissions to shared modules\"\"\"\n        if check_exists:\n            self.check_user_exists()\n        file = self.write_temp_file(self._add_user_sql)\n        self.exec(file)\n\n    def add_admin(self, *args, **kwargs):\n        \"\"\"Add admin user with full permissions to all modules\"\"\"\n        file = self.write_temp_file(self._add_admin_sql)\n        self.exec(file)\n\n    def add_module(self, module_name):\n        \"\"\"Add module to database. Grant permissions to all users in group\"\"\"\n        logger.info(f\"Granting everyone permissions to module {module_name}\")\n        file = self.write_temp_file(self._add_module_sql(module_name))\n        self.exec(file)\n\n    def check_user_exists(self):\n        \"\"\"Add user to database with permissions to shared modules\"\"\"\n        user_home = Path.home().parent / self.user\n        if user_home.exists():\n            logger.info(\"Creating database user \", self.user)\n        else:\n            sys.exit(\n                f\"Error: couldn't find {self.user} in home dir: {user_home}\"\n            )\n\n    def add_user_by_role(self, role, check_exists=False):\n        add_func = {\n            \"guest\": self.add_guest,\n            \"user\": self.add_user,\n            \"collab\": self.add_collab,\n            \"admin\": self.add_admin,\n        }\n        if role not in add_func:\n            raise ValueError(f\"Role {role} not recognized\")\n        add_func[role]()\n\n    def add_roles(self):\n        \"\"\"Add roles to database\"\"\"\n        file = self.write_temp_file(self._create_roles_sql)\n        self.exec(file)\n\n    def write_temp_file(self, content: list) -&gt; tempfile.NamedTemporaryFile:\n        \"\"\"Write content to a temporary file and return the file object\"\"\"\n        file = tempfile.NamedTemporaryFile(mode=\"w\")\n        for line in content:\n            file.write(line)\n        file.flush()\n\n        if self.debug:\n            from pprint import pprint  # noqa F401\n\n            pprint(file.name)\n            pprint(content)\n\n        return file\n\n    def exec(self, file):\n        \"\"\"Run commands saved to file in sql\"\"\"\n\n        if self.debug:\n            return\n\n        cmd = (\n            f\"mysql -p -h {self.host} &lt; {file.name}\"\n            if self.target_database == \"mysql\"\n            else f\"docker exec -i {self.target_database} mysql -u \"\n            + f\"{self.exec_user} --password={self.exec_pass} &lt; {file.name}\"\n        )\n        os.system(cmd)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.__init__", "title": "<code>__init__(user_name=None, host_name=None, debug=False, target_database=None, exec_user=None, exec_pass=None)</code>", "text": "<p>Class to manage common database settings</p> <p>Roles: - dj_guest:  select for all prefix - dj_collab: select for all prefix, all for user prefix - dj_user:   select for all prefix, all for user prefix, all for shared - dj_admin:     all for all prefix</p> <p>Note: To add dj_user role to all those with common access, run:     query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"     users = dj.conn().query(query).fetchall()     for user in users:         dj.conn().query(f\"GRANT dj_user TO '{user0}'@'%';\")</p> <p>Parameters:</p> Name Type Description Default <code>user_name</code> <code>str</code> <p>The name of the user to add to the database. Default from dj.config</p> <code>None</code> <code>host_name</code> <code>str</code> <p>The name of the host to add to the database. Default from dj.config</p> <code>None</code> <code>debug</code> <code>bool</code> <p>Default False. If True, pprint sql instead of running</p> <code>False</code> <code>target_database</code> <code>str</code> <p>Default is mysql. Can also be docker container id</p> <code>None</code> <code>exec_user</code> <code>str</code> <p>User for executing commands. If None, use dj.config</p> <code>None</code> <code>exec_pass</code> <code>str</code> <p>Password for executing commands. If None, use dj.config</p> <code>None</code> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def __init__(\n    self,\n    user_name=None,\n    host_name=None,\n    debug=False,\n    target_database=None,\n    exec_user=None,\n    exec_pass=None,\n):\n    \"\"\"Class to manage common database settings\n\n    Roles:\n    - dj_guest:  select for all prefix\n    - dj_collab: select for all prefix, all for user prefix\n    - dj_user:   select for all prefix, all for user prefix, all for shared\n    - dj_admin:     all for all prefix\n\n    Note: To add dj_user role to all those with common access, run:\n        query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"\n        users = dj.conn().query(query).fetchall()\n        for user in users:\n            dj.conn().query(f\"GRANT dj_user TO '{user[0][0]}'@'%';\")\n\n    Parameters\n    ----------\n    user_name : str, optional\n        The name of the user to add to the database. Default from dj.config\n    host_name : str, optional\n        The name of the host to add to the database. Default from dj.config\n    debug : bool, optional\n        Default False. If True, pprint sql instead of running\n    target_database : str, optional\n        Default is mysql. Can also be docker container id\n    exec_user : str, optional\n        User for executing commands. If None, use dj.config\n    exec_pass : str, optional\n        Password for executing commands. If None, use dj.config\n    \"\"\"\n    self.shared_modules = [f\"{m}{ESC}\" for m in SHARED_MODULES]\n    self.user = user_name or dj.config[\"database.user\"]\n    self.host = (\n        host_name or dj.config[\"database.host\"] or \"lmf-db.cin.ucsf.edu\"\n    )\n    self.debug = debug\n    self.target_database = target_database or \"mysql\"\n    self.exec_user = exec_user or dj.config[\"database.user\"]\n    self.exec_pass = exec_pass or dj.config[\"database.password\"]\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_guest", "title": "<code>add_guest(*args, **kwargs)</code>", "text": "<p>Add guest user with select permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_guest(self, *args, **kwargs):\n    \"\"\"Add guest user with select permissions to shared modules\"\"\"\n    file = self.write_temp_file(self._add_guest_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_collab", "title": "<code>add_collab(*args, **kwargs)</code>", "text": "<p>Add collaborator user with full permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_collab(self, *args, **kwargs):\n    \"\"\"Add collaborator user with full permissions to shared modules\"\"\"\n    file = self.write_temp_file(self._add_collab_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_user", "title": "<code>add_user(check_exists=False, *args, **kwargs)</code>", "text": "<p>Add user to database with permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_user(self, check_exists=False, *args, **kwargs):\n    \"\"\"Add user to database with permissions to shared modules\"\"\"\n    if check_exists:\n        self.check_user_exists()\n    file = self.write_temp_file(self._add_user_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_admin", "title": "<code>add_admin(*args, **kwargs)</code>", "text": "<p>Add admin user with full permissions to all modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_admin(self, *args, **kwargs):\n    \"\"\"Add admin user with full permissions to all modules\"\"\"\n    file = self.write_temp_file(self._add_admin_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_module", "title": "<code>add_module(module_name)</code>", "text": "<p>Add module to database. Grant permissions to all users in group</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_module(self, module_name):\n    \"\"\"Add module to database. Grant permissions to all users in group\"\"\"\n    logger.info(f\"Granting everyone permissions to module {module_name}\")\n    file = self.write_temp_file(self._add_module_sql(module_name))\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.check_user_exists", "title": "<code>check_user_exists()</code>", "text": "<p>Add user to database with permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def check_user_exists(self):\n    \"\"\"Add user to database with permissions to shared modules\"\"\"\n    user_home = Path.home().parent / self.user\n    if user_home.exists():\n        logger.info(\"Creating database user \", self.user)\n    else:\n        sys.exit(\n            f\"Error: couldn't find {self.user} in home dir: {user_home}\"\n        )\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_roles", "title": "<code>add_roles()</code>", "text": "<p>Add roles to database</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_roles(self):\n    \"\"\"Add roles to database\"\"\"\n    file = self.write_temp_file(self._create_roles_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.write_temp_file", "title": "<code>write_temp_file(content)</code>", "text": "<p>Write content to a temporary file and return the file object</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def write_temp_file(self, content: list) -&gt; tempfile.NamedTemporaryFile:\n    \"\"\"Write content to a temporary file and return the file object\"\"\"\n    file = tempfile.NamedTemporaryFile(mode=\"w\")\n    for line in content:\n        file.write(line)\n    file.flush()\n\n    if self.debug:\n        from pprint import pprint  # noqa F401\n\n        pprint(file.name)\n        pprint(content)\n\n    return file\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.exec", "title": "<code>exec(file)</code>", "text": "<p>Run commands saved to file in sql</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def exec(self, file):\n    \"\"\"Run commands saved to file in sql\"\"\"\n\n    if self.debug:\n        return\n\n    cmd = (\n        f\"mysql -p -h {self.host} &lt; {file.name}\"\n        if self.target_database == \"mysql\"\n        else f\"docker exec -i {self.target_database} mysql -u \"\n        + f\"{self.exec_user} --password={self.exec_pass} &lt; {file.name}\"\n    )\n    os.system(cmd)\n</code></pre>"}, {"location": "api/utils/dj_chains/", "title": "dj_chains.py", "text": ""}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains", "title": "<code>TableChains</code>", "text": "<p>Class for representing chains from parent to Merge table via parts.</p> <p>Functions as a plural version of TableChain, allowing a single <code>join</code> call across all chains from parent -&gt; Merge table.</p> <p>Attributes:</p> Name Type Description <code>parent</code> <code>Table</code> <p>Parent or origin of chains.</p> <code>child</code> <code>Table</code> <p>Merge table or destination of chains.</p> <code>connection</code> <code>(Connection, optional)</code> <p>Connection to database used to create FreeTable objects. Defaults to parent.connection.</p> <code>part_names</code> <code>List[str]</code> <p>List of full table names of child parts.</p> <code>chains</code> <code>List[TableChain]</code> <p>List of TableChain objects for each part in child.</p> <code>has_link</code> <code>bool</code> <p>Cached attribute to store whether parent is linked to child via any of child parts. False if (a) child is not in parent.descendants or (b) nx.NetworkXNoPath is raised by nx.shortest_path for all chains.</p> <p>Methods:</p> Name Description <code>__init__</code> <p>Initialize TableChains with parent and child tables.</p> <code>__repr__</code> <p>Return full representation of chains. Multiline parent -&gt; child for each chain.</p> <code>__len__</code> <p>Return number of chains with links.</p> <code>__getitem__</code> <p>Return TableChain object at index, or use substring of table name.</p> <code>join</code> <p>Return list of joins for each chain in self.chains.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>class TableChains:\n    \"\"\"Class for representing chains from parent to Merge table via parts.\n\n    Functions as a plural version of TableChain, allowing a single `join`\n    call across all chains from parent -&gt; Merge table.\n\n    Attributes\n    ----------\n    parent : Table\n        Parent or origin of chains.\n    child : Table\n        Merge table or destination of chains.\n    connection : datajoint.Connection, optional\n        Connection to database used to create FreeTable objects. Defaults to\n        parent.connection.\n    part_names : List[str]\n        List of full table names of child parts.\n    chains : List[TableChain]\n        List of TableChain objects for each part in child.\n    has_link : bool\n        Cached attribute to store whether parent is linked to child via any of\n        child parts. False if (a) child is not in parent.descendants or (b)\n        nx.NetworkXNoPath is raised by nx.shortest_path for all chains.\n\n    Methods\n    -------\n    __init__(parent, child, connection=None)\n        Initialize TableChains with parent and child tables.\n    __repr__()\n        Return full representation of chains.\n        Multiline parent -&gt; child for each chain.\n    __len__()\n        Return number of chains with links.\n    __getitem__(index: Union[int, str])\n        Return TableChain object at index, or use substring of table name.\n    join(restriction: str = None)\n        Return list of joins for each chain in self.chains.\n    \"\"\"\n\n    def __init__(self, parent, child, connection=None):\n        self.parent = parent\n        self.child = child\n        self.connection = connection or parent.connection\n        parts = child.parts(as_objects=True)\n        self.part_names = [part.full_table_name for part in parts]\n        self.chains = [TableChain(parent, part) for part in parts]\n        self.has_link = any([chain.has_link for chain in self.chains])\n\n    def __repr__(self):\n        return \"\\n\".join([str(chain) for chain in self.chains])\n\n    def __len__(self):\n        return len([c for c in self.chains if c.has_link])\n\n    @property\n    def max_len(self):\n        \"\"\"Return length of longest chain.\"\"\"\n        return max([len(chain) for chain in self.chains])\n\n    def __getitem__(self, index: Union[int, str]):\n        \"\"\"Return FreeTable object at index.\"\"\"\n        if isinstance(index, str):\n            for i, part in enumerate(self.part_names):\n                if index in part:\n                    return self.chains[i]\n        return self.chains[index]\n\n    def join(self, restriction=None) -&gt; List[QueryExpression]:\n        \"\"\"Return list of joins for each chain in self.chains.\"\"\"\n        restriction = restriction or self.parent.restriction or True\n        joins = []\n        for chain in self.chains:\n            if joined := chain.join(restriction):\n                joins.append(joined)\n        return joins\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.max_len", "title": "<code>max_len</code>  <code>property</code>", "text": "<p>Return length of longest chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.__getitem__", "title": "<code>__getitem__(index)</code>", "text": "<p>Return FreeTable object at index.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __getitem__(self, index: Union[int, str]):\n    \"\"\"Return FreeTable object at index.\"\"\"\n    if isinstance(index, str):\n        for i, part in enumerate(self.part_names):\n            if index in part:\n                return self.chains[i]\n    return self.chains[index]\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.join", "title": "<code>join(restriction=None)</code>", "text": "<p>Return list of joins for each chain in self.chains.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def join(self, restriction=None) -&gt; List[QueryExpression]:\n    \"\"\"Return list of joins for each chain in self.chains.\"\"\"\n    restriction = restriction or self.parent.restriction or True\n    joins = []\n    for chain in self.chains:\n        if joined := chain.join(restriction):\n            joins.append(joined)\n    return joins\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain", "title": "<code>TableChain</code>", "text": "<p>Class for representing a chain of tables.</p> <p>A chain is a sequence of tables from parent to child identified by networkx.shortest_path. Parent -&gt; Merge should use TableChains instead to handle multiple paths to the respective parts of the Merge table.</p> <p>Attributes:</p> Name Type Description <code>parent</code> <code>Table</code> <p>Parent or origin of chain.</p> <code>child</code> <code>Table</code> <p>Child or destination of chain.</p> <code>_connection</code> <code>(Connection, optional)</code> <p>Connection to database used to create FreeTable objects. Defaults to parent.connection.</p> <code>_link_symbol</code> <code>str</code> <p>Symbol used to represent the link between parent and child. Hardcoded to \" -&gt; \".</p> <code>has_link</code> <code>bool</code> <p>Cached attribute to store whether parent is linked to child. False if child is not in parent.descendants or nx.NetworkXNoPath is raised by nx.shortest_path.</p> <code>link_type</code> <code>str</code> <p>'directed' or 'undirected' based on whether path is found with directed or undirected graph. None if no path is found.</p> <code>graph</code> <code>DiGraph</code> <p>Directed graph of parent's dependencies from datajoint.connection.</p> <code>names</code> <code>List[str]</code> <p>List of full table names in chain.</p> <code>objects</code> <code>List[FreeTable]</code> <p>List of FreeTable objects for each table in chain.</p> <code>attr_maps</code> <code>List[dict]</code> <p>List of attribute maps for each link in chain.</p> <code>path</code> <code>OrderedDict[str, Dict[str, Union[FreeTable, dict]]]</code> <p>Dictionary of full table names in chain. Keys are self.names Values are a dict of free_table (self.objects) and attr_map (dict of new_name: old_name, self.attr_map).</p> <p>Methods:</p> Name Description <code>__str__</code> <p>Return string representation of chain: parent -&gt; child.</p> <code>__repr__</code> <p>Return full representation of chain: parent -&gt; {links} -&gt; child.</p> <code>__len__</code> <p>Return number of tables in chain.</p> <code>__getitem__</code> <p>Return FreeTable object at index, or use substring of table name.</p> <code>find_path</code> <p>Returns path OrderedDict of full table names in chain. If directed is True, uses directed graph. If False, uses undirected graph. Undirected excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc. to maintain valid joins.</p> <code>join</code> <p>Return join of tables in chain with restriction applied to parent.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>class TableChain:\n    \"\"\"Class for representing a chain of tables.\n\n    A chain is a sequence of tables from parent to child identified by\n    networkx.shortest_path. Parent -&gt; Merge should use TableChains instead to\n    handle multiple paths to the respective parts of the Merge table.\n\n    Attributes\n    ----------\n    parent : Table\n        Parent or origin of chain.\n    child : Table\n        Child or destination of chain.\n    _connection : datajoint.Connection, optional\n        Connection to database used to create FreeTable objects. Defaults to\n        parent.connection.\n    _link_symbol : str\n        Symbol used to represent the link between parent and child. Hardcoded\n        to \" -&gt; \".\n    has_link : bool\n        Cached attribute to store whether parent is linked to child. False if\n        child is not in parent.descendants or nx.NetworkXNoPath is raised by\n        nx.shortest_path.\n    link_type : str\n        'directed' or 'undirected' based on whether path is found with directed\n        or undirected graph. None if no path is found.\n    graph : nx.DiGraph\n        Directed graph of parent's dependencies from datajoint.connection.\n    names : List[str]\n        List of full table names in chain.\n    objects : List[dj.FreeTable]\n        List of FreeTable objects for each table in chain.\n    attr_maps : List[dict]\n        List of attribute maps for each link in chain.\n    path : OrderedDict[str, Dict[str, Union[dj.FreeTable,dict]]]\n        Dictionary of full table names in chain. Keys are self.names\n        Values are a dict of free_table (self.objects) and\n        attr_map (dict of new_name: old_name, self.attr_map).\n\n    Methods\n    -------\n    __str__()\n        Return string representation of chain: parent -&gt; child.\n    __repr__()\n        Return full representation of chain: parent -&gt; {links} -&gt; child.\n    __len__()\n        Return number of tables in chain.\n    __getitem__(index: Union[int, str])\n        Return FreeTable object at index, or use substring of table name.\n    find_path(directed=True)\n        Returns path OrderedDict of full table names in chain. If directed is\n        True, uses directed graph. If False, uses undirected graph. Undirected\n        excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc. to maintain\n        valid joins.\n    join(restriction: str = None)\n        Return join of tables in chain with restriction applied to parent.\n    \"\"\"\n\n    def __init__(self, parent: Table, child: Table, connection=None):\n        self._connection = connection or parent.connection\n        self.graph = self._connection.dependencies\n        self.graph.load()\n\n        if (  # if child is a merge table\n            get_master(child.full_table_name) == \"\"\n            and MERGE_PK in child.heading.names\n        ):\n            raise TypeError(\"Child is a merge table. Use TableChains instead.\")\n\n        self._link_symbol = \" -&gt; \"\n        self.parent = parent\n        self.child = child\n        self.link_type = None\n        self._searched = False\n\n        if child.full_table_name not in self.graph.nodes:\n            logger.warning(\n                \"Can't find item in graph. Try importing: \"\n                + f\"{child.full_table_name}\"\n            )\n            self._searched = True\n\n    def __str__(self):\n        \"\"\"Return string representation of chain: parent -&gt; child.\"\"\"\n        if not self.has_link:\n            return \"No link\"\n        return (\n            to_camel_case(self.parent.table_name)\n            + self._link_symbol\n            + to_camel_case(self.child.table_name)\n        )\n\n    def __repr__(self):\n        \"\"\"Return full representation of chain: parent -&gt; {links} -&gt; child.\"\"\"\n        if not self.has_link:\n            return \"No link\"\n        return \"Chain: \" + self._link_symbol.join(\n            [t.table_name for t in self.objects]\n        )\n\n    def __len__(self):\n        \"\"\"Return number of tables in chain.\"\"\"\n        if not self.has_link:\n            return 0\n        return len(self.names)\n\n    def __getitem__(self, index: Union[int, str]) -&gt; dj.FreeTable:\n        \"\"\"Return FreeTable object at index.\"\"\"\n        if not self.has_link:\n            return None\n        if isinstance(index, str):\n            for i, name in enumerate(self.names):\n                if index in name:\n                    return self.objects[i]\n        return self.objects[index]\n\n    @property\n    def has_link(self) -&gt; bool:\n        \"\"\"Return True if parent is linked to child.\n\n        If not searched, search for path. If searched and no link is found,\n        return False. If searched and link is found, return True.\n        \"\"\"\n        if not self._searched:\n            _ = self.path\n        return self.link_type is not None\n\n    def pk_link(self, src, trg, data) -&gt; float:\n        \"\"\"Return 1 if data[\"primary\"] else float(\"inf\").\n\n        Currently unused. Preserved for future debugging. shortest_path accepts\n        an option weight callable parameter.\n        nx.shortest_path(G, source, target,weight=pk_link)\n        \"\"\"\n        return 1 if data[\"primary\"] else float(\"inf\")\n\n    def find_path(self, directed=True) -&gt; OrderedDict:\n        \"\"\"Return list of full table names in chain.\n\n        Parameters\n        ----------\n        directed : bool, optional\n            If True, use directed graph. If False, use undirected graph.\n            Defaults to True. Undirected permits paths to traverse from merge\n            part-parent -&gt; merge part -&gt; merge table. Undirected excludes\n            PERIPHERAL_TABLES like interval_list, nwbfile, etc.\n\n        Returns\n        -------\n        OrderedDict\n            Dictionary of full table names in chain. Keys are full table names.\n            Values are free_table (dj.FreeTable representation) and attr_map\n            (dict of new_name: old_name). Attribute maps on the table upstream\n            of an alias node that can be used in .proj(). Returns None if no\n            path is found.\n\n        Ignores numeric table names in paths, which are\n        'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph\n        source code for comments on alias nodes.\n        \"\"\"\n        source, target = self.parent.full_table_name, self.child.full_table_name\n        if not directed:\n            self.graph = self.graph.to_undirected()\n            self.graph.remove_nodes_from(PERIPHERAL_TABLES)\n        try:\n            path = nx.shortest_path(self.graph, source, target)\n        except nx.NetworkXNoPath:\n            return None\n        except nx.NodeNotFound:\n            self._searched = True\n            return None\n\n        ret = OrderedDict()\n        prev_table = None\n        for i, table in enumerate(path):\n            if table.isnumeric():  # get proj() attribute map for alias node\n                if not prev_table:\n                    raise ValueError(\"Alias node found without prev table.\")\n                attr_map = self.graph[table][prev_table][\"attr_map\"]\n                ret[prev_table][\"attr_map\"] = attr_map\n            else:\n                free_table = dj.FreeTable(self._connection, table)\n                ret[table] = {\"free_table\": free_table, \"attr_map\": {}}\n                prev_table = table\n        return ret\n\n    @cached_property\n    def path(self) -&gt; OrderedDict:\n        \"\"\"Return list of full table names in chain.\"\"\"\n        if self._searched and not self.has_link:\n            return None\n\n        link = None\n        if link := self.find_path(directed=True):\n            self.link_type = \"directed\"\n        elif link := self.find_path(directed=False):\n            self.link_type = \"undirected\"\n        self._searched = True\n\n        return link\n\n    @cached_property\n    def names(self) -&gt; List[str]:\n        \"\"\"Return list of full table names in chain.\"\"\"\n        if not self.has_link:\n            return None\n        return list(self.path.keys())\n\n    @cached_property\n    def objects(self) -&gt; List[dj.FreeTable]:\n        \"\"\"Return list of FreeTable objects for each table in chain.\n\n        Unused. Preserved for future debugging.\n        \"\"\"\n        if not self.has_link:\n            return None\n        return [v[\"free_table\"] for v in self.path.values()]\n\n    @cached_property\n    def attr_maps(self) -&gt; List[dict]:\n        \"\"\"Return list of attribute maps for each table in chain.\n\n        Unused. Preserved for future debugging.\n        \"\"\"\n        if not self.has_link:\n            return None\n        return [v[\"attr_map\"] for v in self.path.values()]\n\n    def join(\n        self, restriction: str = None, reverse_order: bool = False\n    ) -&gt; dj.expression.QueryExpression:\n        \"\"\"Return join of tables in chain with restriction applied to parent.\n\n        Parameters\n        ----------\n        restriction : str, optional\n            Restriction to apply to first table in the order.\n            Defaults to self.parent.restriction.\n        reverse_order : bool, optional\n            If True, join tables in reverse order. Defaults to False.\n        \"\"\"\n        if not self.has_link:\n            return None\n\n        restriction = restriction or self.parent.restriction or True\n        path = (\n            OrderedDict(reversed(self.path.items()))\n            if reverse_order\n            else self.path\n        ).copy()\n\n        _, first_val = path.popitem(last=False)\n        join = first_val[\"free_table\"] &amp; restriction\n        for i, val in enumerate(path.values()):\n            attr_map, free_table = val[\"attr_map\"], val[\"free_table\"]\n            try:\n                join = (join.proj() * free_table).proj(**attr_map)\n            except dj.DataJointError as e:\n                attribute = str(e).split(\"attribute \")[-1]\n                logger.error(\n                    f\"{str(self)} at {free_table.table_name} with {attribute}\"\n                )\n                return None\n        return join\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__str__", "title": "<code>__str__()</code>", "text": "<p>Return string representation of chain: parent -&gt; child.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __str__(self):\n    \"\"\"Return string representation of chain: parent -&gt; child.\"\"\"\n    if not self.has_link:\n        return \"No link\"\n    return (\n        to_camel_case(self.parent.table_name)\n        + self._link_symbol\n        + to_camel_case(self.child.table_name)\n    )\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__repr__", "title": "<code>__repr__()</code>", "text": "<p>Return full representation of chain: parent -&gt; {links} -&gt; child.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __repr__(self):\n    \"\"\"Return full representation of chain: parent -&gt; {links} -&gt; child.\"\"\"\n    if not self.has_link:\n        return \"No link\"\n    return \"Chain: \" + self._link_symbol.join(\n        [t.table_name for t in self.objects]\n    )\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__len__", "title": "<code>__len__()</code>", "text": "<p>Return number of tables in chain.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __len__(self):\n    \"\"\"Return number of tables in chain.\"\"\"\n    if not self.has_link:\n        return 0\n    return len(self.names)\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__getitem__", "title": "<code>__getitem__(index)</code>", "text": "<p>Return FreeTable object at index.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __getitem__(self, index: Union[int, str]) -&gt; dj.FreeTable:\n    \"\"\"Return FreeTable object at index.\"\"\"\n    if not self.has_link:\n        return None\n    if isinstance(index, str):\n        for i, name in enumerate(self.names):\n            if index in name:\n                return self.objects[i]\n    return self.objects[index]\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.has_link", "title": "<code>has_link: bool</code>  <code>property</code>", "text": "<p>Return True if parent is linked to child.</p> <p>If not searched, search for path. If searched and no link is found, return False. If searched and link is found, return True.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.pk_link", "title": "<code>pk_link(src, trg, data)</code>", "text": "<p>Return 1 if data[\"primary\"] else float(\"inf\").</p> <p>Currently unused. Preserved for future debugging. shortest_path accepts an option weight callable parameter. nx.shortest_path(G, source, target,weight=pk_link)</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def pk_link(self, src, trg, data) -&gt; float:\n    \"\"\"Return 1 if data[\"primary\"] else float(\"inf\").\n\n    Currently unused. Preserved for future debugging. shortest_path accepts\n    an option weight callable parameter.\n    nx.shortest_path(G, source, target,weight=pk_link)\n    \"\"\"\n    return 1 if data[\"primary\"] else float(\"inf\")\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.find_path", "title": "<code>find_path(directed=True)</code>", "text": "<p>Return list of full table names in chain.</p> <p>Parameters:</p> Name Type Description Default <code>directed</code> <code>bool</code> <p>If True, use directed graph. If False, use undirected graph. Defaults to True. Undirected permits paths to traverse from merge part-parent -&gt; merge part -&gt; merge table. Undirected excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc.</p> <code>True</code> <p>Returns:</p> Type Description <code>OrderedDict</code> <p>Dictionary of full table names in chain. Keys are full table names. Values are free_table (dj.FreeTable representation) and attr_map (dict of new_name: old_name). Attribute maps on the table upstream of an alias node that can be used in .proj(). Returns None if no path is found.</p> <p>Ignores numeric table names in paths, which are 'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph source code for comments on alias nodes.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def find_path(self, directed=True) -&gt; OrderedDict:\n    \"\"\"Return list of full table names in chain.\n\n    Parameters\n    ----------\n    directed : bool, optional\n        If True, use directed graph. If False, use undirected graph.\n        Defaults to True. Undirected permits paths to traverse from merge\n        part-parent -&gt; merge part -&gt; merge table. Undirected excludes\n        PERIPHERAL_TABLES like interval_list, nwbfile, etc.\n\n    Returns\n    -------\n    OrderedDict\n        Dictionary of full table names in chain. Keys are full table names.\n        Values are free_table (dj.FreeTable representation) and attr_map\n        (dict of new_name: old_name). Attribute maps on the table upstream\n        of an alias node that can be used in .proj(). Returns None if no\n        path is found.\n\n    Ignores numeric table names in paths, which are\n    'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph\n    source code for comments on alias nodes.\n    \"\"\"\n    source, target = self.parent.full_table_name, self.child.full_table_name\n    if not directed:\n        self.graph = self.graph.to_undirected()\n        self.graph.remove_nodes_from(PERIPHERAL_TABLES)\n    try:\n        path = nx.shortest_path(self.graph, source, target)\n    except nx.NetworkXNoPath:\n        return None\n    except nx.NodeNotFound:\n        self._searched = True\n        return None\n\n    ret = OrderedDict()\n    prev_table = None\n    for i, table in enumerate(path):\n        if table.isnumeric():  # get proj() attribute map for alias node\n            if not prev_table:\n                raise ValueError(\"Alias node found without prev table.\")\n            attr_map = self.graph[table][prev_table][\"attr_map\"]\n            ret[prev_table][\"attr_map\"] = attr_map\n        else:\n            free_table = dj.FreeTable(self._connection, table)\n            ret[table] = {\"free_table\": free_table, \"attr_map\": {}}\n            prev_table = table\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.path", "title": "<code>path: OrderedDict</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of full table names in chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.names", "title": "<code>names: List[str]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of full table names in chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.objects", "title": "<code>objects: List[dj.FreeTable]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of FreeTable objects for each table in chain.</p> <p>Unused. Preserved for future debugging.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.attr_maps", "title": "<code>attr_maps: List[dict]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of attribute maps for each table in chain.</p> <p>Unused. Preserved for future debugging.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.join", "title": "<code>join(restriction=None, reverse_order=False)</code>", "text": "<p>Return join of tables in chain with restriction applied to parent.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to first table in the order. Defaults to self.parent.restriction.</p> <code>None</code> <code>reverse_order</code> <code>bool</code> <p>If True, join tables in reverse order. Defaults to False.</p> <code>False</code> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def join(\n    self, restriction: str = None, reverse_order: bool = False\n) -&gt; dj.expression.QueryExpression:\n    \"\"\"Return join of tables in chain with restriction applied to parent.\n\n    Parameters\n    ----------\n    restriction : str, optional\n        Restriction to apply to first table in the order.\n        Defaults to self.parent.restriction.\n    reverse_order : bool, optional\n        If True, join tables in reverse order. Defaults to False.\n    \"\"\"\n    if not self.has_link:\n        return None\n\n    restriction = restriction or self.parent.restriction or True\n    path = (\n        OrderedDict(reversed(self.path.items()))\n        if reverse_order\n        else self.path\n    ).copy()\n\n    _, first_val = path.popitem(last=False)\n    join = first_val[\"free_table\"] &amp; restriction\n    for i, val in enumerate(path.values()):\n        attr_map, free_table = val[\"attr_map\"], val[\"free_table\"]\n        try:\n            join = (join.proj() * free_table).proj(**attr_map)\n        except dj.DataJointError as e:\n            attribute = str(e).split(\"attribute \")[-1]\n            logger.error(\n                f\"{str(self)} at {free_table.table_name} with {attribute}\"\n            )\n            return None\n    return join\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/", "title": "dj_helper_fn.py", "text": "<p>Helper functions for manipulating information from DataJoint fetch calls.</p>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.deprecated_factory", "title": "<code>deprecated_factory(classes, old_module='')</code>", "text": "<p>Creates a list of classes and logs a warning when instantiated</p> <p>Parameters:</p> Name Type Description Default <code>classes</code> <code>list</code> <p>list of tuples containing old_class, new_class</p> required <p>Returns:</p> Type Description <code>list</code> <p>list of classes that will log a warning when instantiated</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def deprecated_factory(classes: list, old_module: str = \"\") -&gt; list:\n    \"\"\"Creates a list of classes and logs a warning when instantiated\n\n    Parameters\n    ---------\n    classes : list\n        list of tuples containing old_class, new_class\n\n    Returns\n    ------\n    list\n        list of classes that will log a warning when instantiated\n    \"\"\"\n\n    if not isinstance(classes, list):\n        classes = [classes]\n\n    ret = [\n        _subclass_factory(old_name=c[0], new_class=c[1], old_module=old_module)\n        for c in classes\n    ]\n\n    return ret[0] if len(ret) == 1 else ret\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.dj_replace", "title": "<code>dj_replace(original_table, new_values, key_column, replace_column)</code>", "text": "<p>Given the output of a fetch() call from a schema and a 2D array made up of (key_value, replace_value) tuples, find each instance of key_value in the key_column of the original table and replace the specified replace_column with the associated replace_value. Key values must be unique.</p> <p>Parameters:</p> Name Type Description Default <code>original_table</code> <p>Result of a datajoint .fetch() call on a schema query.</p> required <code>new_values</code> <code>list</code> <p>List of tuples, each containing (key_value, replace_value).</p> required <code>replace_column</code> <code>str</code> <p>The name of the column where to-be-replaced values are located.</p> required <p>Returns:</p> Type Description <code>original_table</code> <p>Structured array of new table entries that can be inserted back into the schema</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def dj_replace(original_table, new_values, key_column, replace_column):\n    \"\"\"Given the output of a fetch() call from a schema and a 2D array made up\n    of (key_value, replace_value) tuples, find each instance of key_value in\n    the key_column of the original table and replace the specified\n    replace_column with the associated replace_value. Key values must be\n    unique.\n\n    Parameters\n    ----------\n    original_table\n        Result of a datajoint .fetch() call on a schema query.\n    new_values : list\n        List of tuples, each containing (key_value, replace_value).\n    replace_column : str\n        The name of the column where to-be-replaced values are located.\n\n    Returns\n    -------\n    original_table\n        Structured array of new table entries that can be inserted back into the schema\n    \"\"\"\n\n    # check to make sure the new_values are a list or array of tuples and fix if not\n    if isinstance(new_values, tuple):\n        tmp = list()\n        tmp.append(new_values)\n        new_values = tmp\n\n    new_val_array = np.asarray(new_values)\n    replace_ind = np.where(\n        np.isin(original_table[key_column], new_val_array[:, 0])\n    )\n    original_table[replace_column][replace_ind] = new_val_array[:, 1]\n    return original_table\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.fetch_nwb", "title": "<code>fetch_nwb(query_expression, nwb_master, *attrs, **kwargs)</code>", "text": "<p>Get an NWB object from the given DataJoint query.</p> <p>Parameters:</p> Name Type Description Default <code>query_expression</code> <code>query</code> <p>A DataJoint query expression (e.g., join, restrict) or a table to call fetch on.</p> required <code>nwb_master</code> <code>tuple</code> <p>Tuple (table, attr) to get the NWB filepath from. i.e. absolute path to NWB file can be obtained by looking up attr column of table table is usually Nwbfile or AnalysisNwbfile; attr is usually 'nwb_file_abs_path' or 'analysis_file_abs_path'</p> required <code>*attrs</code> <code>list</code> <p>Attributes from normal DataJoint fetch call.</p> <code>()</code> <code>**kwargs</code> <code>dict</code> <p>Keyword arguments from normal DataJoint fetch call.</p> <code>{}</code> <p>Returns:</p> Name Type Description <code>nwb_objects</code> <code>list</code> <p>List of dicts containing fetch results and NWB objects.</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def fetch_nwb(query_expression, nwb_master, *attrs, **kwargs):\n    \"\"\"Get an NWB object from the given DataJoint query.\n\n    Parameters\n    ----------\n    query_expression : query\n        A DataJoint query expression (e.g., join, restrict) or a table to call fetch on.\n    nwb_master : tuple\n        Tuple (table, attr) to get the NWB filepath from.\n        i.e. absolute path to NWB file can be obtained by looking up attr column of table\n        table is usually Nwbfile or AnalysisNwbfile;\n        attr is usually 'nwb_file_abs_path' or 'analysis_file_abs_path'\n    *attrs : list\n        Attributes from normal DataJoint fetch call.\n    **kwargs : dict\n        Keyword arguments from normal DataJoint fetch call.\n\n    Returns\n    -------\n    nwb_objects : list\n        List of dicts containing fetch results and NWB objects.\n    \"\"\"\n    kwargs[\"as_dict\"] = True  # force return as dictionary\n    tbl, attr_name = nwb_master\n\n    if not attrs:\n        attrs = query_expression.heading.names\n\n    # get the list of analysis or nwb files\n    file_name_str = (\n        \"analysis_file_name\" if \"analysis\" in nwb_master[1] else \"nwb_file_name\"\n    )\n    # TODO: avoid this import?\n    from ..common.common_nwbfile import AnalysisNwbfile, Nwbfile\n\n    file_path_fn = (\n        AnalysisNwbfile.get_abs_path\n        if \"analysis\" in nwb_master[1]\n        else Nwbfile.get_abs_path\n    )\n\n    # TODO: check that the query_expression restricts tbl - CBroz\n    nwb_files = (\n        query_expression * tbl.proj(nwb2load_filepath=attr_name)\n    ).fetch(file_name_str)\n    for file_name in nwb_files:\n        file_path = file_path_fn(file_name)\n        if not os.path.exists(file_path):\n            # retrieve the file from kachery. This also opens the file and stores the file object\n            get_nwb_file(file_path)\n\n    rec_dicts = (\n        query_expression * tbl.proj(nwb2load_filepath=attr_name)\n    ).fetch(*attrs, \"nwb2load_filepath\", **kwargs)\n\n    if not rec_dicts or not np.any(\n        [\"object_id\" in key for key in rec_dicts[0]]\n    ):\n        return rec_dicts\n\n    ret = []\n    for rec_dict in rec_dicts:\n        nwbf = get_nwb_file(rec_dict.pop(\"nwb2load_filepath\"))\n        # for each attr that contains substring 'object_id', store key-value: attr name to NWB object\n        # remove '_object_id' from attr name\n        nwb_objs = {\n            id_attr.replace(\"_object_id\", \"\"): _get_nwb_object(\n                nwbf.objects, rec_dict[id_attr]\n            )\n            for id_attr in attrs\n            if \"object_id\" in id_attr and rec_dict[id_attr] != \"\"\n        }\n        ret.append({**rec_dict, **nwb_objs})\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/", "title": "dj_merge_tables.py", "text": ""}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge", "title": "<code>Merge</code>", "text": "<p>             Bases: <code>Manual</code></p> <p>Adds funcs to support standard Merge table operations.</p> <p>Many methods have the @classmethod decorator to permit MergeTable.method() symtax. This makes access to instance attributes (e.g., (MergeTable &amp; \"example='restriction'\").restriction) harder, but these attributes have limited utility when the user wants to, for example, restrict the merged view rather than the master table itself.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>class Merge(dj.Manual):\n    \"\"\"Adds funcs to support standard Merge table operations.\n\n    Many methods have the @classmethod decorator to permit MergeTable.method()\n    symtax. This makes access to instance attributes (e.g., (MergeTable &amp;\n    \"example='restriction'\").restriction) harder, but these attributes have\n    limited utility when the user wants to, for example, restrict the merged\n    view rather than the master table itself.\n    \"\"\"\n\n    def __init__(self):\n        super().__init__()\n        self._reserved_pk = RESERVED_PRIMARY_KEY\n        self._reserved_sk = RESERVED_SECONDARY_KEY\n        merge_def = (\n            f\"\\n    {self._reserved_pk}: uuid\\n    ---\\n\"\n            + f\"    {self._reserved_sk}: varchar({RESERVED_SK_LENGTH})\\n    \"\n        )\n        if not self.is_declared:\n            # remove comments after # from each line of definition\n            if self._remove_comments(self.definition) != merge_def:\n                logger.warn(\n                    \"Merge table with non-default definition\\n\\t\"\n                    + f\"Expected: {merge_def.strip()}\\n\\t\"\n                    + f\"Actual  : {self.definition.strip()}\"\n                )\n            for part in self.parts(as_objects=True):\n                if part.primary_key != self.primary_key:\n                    logger.warn(\n                        f\"Unexpected primary key in {part.table_name}\"\n                        + f\"\\n\\tExpected: {self.primary_key}\"\n                        + f\"\\n\\tActual  : {part.primary_key}\"\n                    )\n        self._source_class_dict = {}\n\n    def _remove_comments(self, definition):\n        \"\"\"Use regular expressions to remove comments and blank lines\"\"\"\n        return re.sub(  # First remove comments, then blank lines\n            r\"\\n\\s*\\n\", \"\\n\", re.sub(r\"#.*\\n\", \"\\n\", definition)\n        )\n\n    @staticmethod\n    def _part_name(part=None):\n        \"\"\"Return the CamelCase name of a part table\"\"\"\n        if not isinstance(part, str):\n            part = part.table_name\n        return to_camel_case(part.split(\"__\")[-1].strip(\"`\"))\n\n    def get_source_from_key(self, key: dict) -&gt; str:\n        \"\"\"Return the source of a given key\"\"\"\n        return self._normalize_source(key)\n\n    def parts(self, camel_case=False, *args, **kwargs) -&gt; list:\n        \"\"\"Return a list of part tables, add option for CamelCase names.\n\n        See DataJoint `parts` for additional arguments. If camel_case is True,\n        forces return of strings rather than objects.\n        \"\"\"\n        self._ensure_dependencies_loaded()\n\n        if camel_case and kwargs.get(\"as_objects\"):\n            logger.warning(\n                \"Overriding as_objects=True to return CamelCase part names.\"\n            )\n            kwargs[\"as_objects\"] = False\n\n        parts = super().parts(*args, **kwargs)\n\n        if camel_case:\n            parts = [self._part_name(part) for part in parts]\n\n        return parts\n\n    @classmethod\n    def _merge_restrict_parts(\n        cls,\n        restriction: str = True,\n        as_objects: bool = True,\n        return_empties: bool = True,\n        add_invalid_restrict: bool = True,\n    ) -&gt; list:\n        \"\"\"Returns a list of parts with restrictions applied.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the parts. Default True, no restrictions.\n        as_objects: bool, optional\n            Default True. Return part tables as objects\n        return_empties: bool, optional\n            Default True. Return empty part tables\n        add_invalid_restrict: bool, optional\n            Default True. Include part for which the restriction is invalid.\n\n        Returns\n        ------\n        list\n            list of datajoint tables, parts of Merge Table\n        \"\"\"\n\n        cls._ensure_dependencies_loaded()\n\n        if not restriction:\n            restriction = True\n\n        # Normalize restriction to sql string\n        restr_str = make_condition(cls(), restriction, set())\n\n        parts_all = cls.parts(as_objects=True)\n        # If the restriction makes ref to a source, we only want that part\n        if (\n            not return_empties\n            and isinstance(restr_str, str)\n            and f\"`{cls()._reserved_sk}`\" in restr_str\n        ):\n            parts_all = [\n                part\n                for part in parts_all\n                if from_camel_case(\n                    restr_str.split(f'`{cls()._reserved_sk}`=\"')[-1].split('\"')[\n                        0\n                    ]\n                )  # Only look at source part table\n                in part.full_table_name\n            ]\n        if isinstance(restriction, dict):  # restr by source already done above\n            _ = restriction.pop(cls()._reserved_sk, None)  # won't work for str\n            # If a dict restriction has all invalid keys, it is treated as True\n            if not add_invalid_restrict:\n                parts_all = [  # so exclude tables w/ nonmatching attrs\n                    p\n                    for p in parts_all\n                    if all([k in p.heading.names for k in restriction.keys()])\n                ]\n\n        parts = []\n        for part in parts_all:\n            try:\n                parts.append(part.restrict(restriction))\n            except DataJointError:  # If restriction not valid on given part\n                if add_invalid_restrict:\n                    parts.append(part)\n\n        if not return_empties:\n            parts = [p for p in parts if len(p)]\n        if not as_objects:\n            parts = [p.full_table_name for p in parts]\n\n        return parts\n\n    @classmethod\n    def _merge_restrict_parents(\n        cls,\n        restriction: str = True,\n        parent_name: str = None,\n        as_objects: bool = True,\n        return_empties: bool = True,\n        add_invalid_restrict: bool = True,\n    ) -&gt; list:\n        \"\"\"Returns a list of part parents with restrictions applied.\n\n        Rather than part tables, we look at parents of those parts, the source\n        of the data.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the returned parent. Default True, no\n            restrictions.\n        parent_name: str, optional\n            CamelCase name of the parent.\n        as_objects: bool, optional\n            Default True. Return part tables as objects\n        return_empties: bool, optional\n            Default True. Return empty part tables\n        add_invalid_restrict: bool, optional\n            Default True. Include part for which the restriction is invalid.\n\n        Returns\n        ------\n        list\n            list of datajoint tables, parents of parts of Merge Table\n        \"\"\"\n        # .restrict(restriction) does not work on returned part FreeTable\n        # &amp; part.fetch below restricts parent to entries in merge table\n        part_parents = [\n            parent\n            &amp; part.fetch(*part.heading.secondary_attributes, as_dict=True)\n            for part in cls()._merge_restrict_parts(\n                restriction=restriction,\n                return_empties=return_empties,\n                add_invalid_restrict=add_invalid_restrict,\n            )\n            for parent in part.parents(as_objects=True)  # ID respective parents\n            if cls().table_name not in parent.full_table_name  # Not merge table\n        ]\n        if parent_name:\n            part_parents = [\n                p\n                for p in part_parents\n                if from_camel_case(parent_name) in p.full_table_name\n            ]\n        if not as_objects:\n            part_parents = [p.full_table_name for p in part_parents]\n\n        return part_parents\n\n    @classmethod\n    def _merge_repr(\n        cls, restriction: str = True, include_empties=False\n    ) -&gt; dj.expression.Union:\n        \"\"\"Merged view, including null entries for columns unique to one part.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the merged view\n        include_empties: bool, optional\n            Default False. Add columns for empty parts.\n\n        Returns\n        ------\n        datajoint.expression.Union\n        \"\"\"\n\n        parts = [\n            cls() * p  # join with master to include sec key (i.e., 'source')\n            for p in cls._merge_restrict_parts(\n                restriction=restriction,\n                add_invalid_restrict=False,\n                return_empties=include_empties,\n            )\n        ]\n        if not parts:\n            logger.warning(\"No parts found. Try adjusting restriction.\")\n            return\n\n        attr_dict = {  # NULL for non-numeric, 0 for numeric\n            attr.name: \"0\" if attr.numeric else \"NULL\"\n            for attr in iter_chain.from_iterable(\n                part.heading.attributes.values() for part in parts\n            )\n        }\n\n        def _proj_part(part):\n            \"\"\"Project part, adding NULL/0 for missing attributes\"\"\"\n            return dj.U(*attr_dict.keys()) * part.proj(\n                ...,  # include all attributes from part\n                **{\n                    k: v\n                    for k, v in attr_dict.items()\n                    if k not in part.heading.names\n                },\n            )\n\n        query = _proj_part(parts[0])  # start with first part\n        for part in parts[1:]:  # add remaining parts\n            query += _proj_part(part)\n\n        return query\n\n    @classmethod\n    def _merge_insert(cls, rows: list, part_name: str = None, **kwargs) -&gt; None:\n        \"\"\"Insert rows into merge, ensuring data exists in part parent(s).\n\n        Parameters\n        ---------\n        rows: List[dict]\n            An iterable where an element is a dictionary.\n        part: str, optional\n            CamelCase name of the part table\n\n        Raises\n        ------\n        TypeError\n            If rows is not a list of dicts\n        ValueError\n            If data doesn't exist in part parents, integrity error\n        \"\"\"\n        cls._ensure_dependencies_loaded()\n\n        type_err_msg = \"Input `rows` must be a list of dictionaries\"\n        try:\n            for r in iter(rows):\n                if not isinstance(r, dict):\n                    raise TypeError(type_err_msg)\n        except TypeError:\n            raise TypeError(type_err_msg)\n\n        parts = cls._merge_restrict_parts(as_objects=True)\n        if part_name:\n            parts = [\n                p\n                for p in parts\n                if from_camel_case(part_name) in p.full_table_name\n            ]\n\n        master_entries = []\n        parts_entries = {p: [] for p in parts}\n        for row in rows:\n            keys = []  # empty to-be-inserted keys\n            for part in parts:  # check each part\n                part_name = cls._part_name(part)\n                part_parent = part.parents(as_objects=True)[-1]\n                if part_parent &amp; row:  # if row is in part parent\n                    keys = (part_parent &amp; row).fetch(\"KEY\")  # get pk\n                    if len(keys) &gt; 1:\n                        raise ValueError(\n                            \"Ambiguous entry. Data has mult rows in \"\n                            + f\"{part_name}:\\n\\tData:{row}\\n\\t{keys}\"\n                        )\n                    key = keys[0]\n                    master_sk = {cls()._reserved_sk: part_name}\n                    uuid = dj.hash.key_hash(key | master_sk)\n                    master_pk = {cls()._reserved_pk: uuid}\n\n                    master_entries.append({**master_pk, **master_sk})\n                    parts_entries[part].append({**master_pk, **key})\n\n            if not keys:\n                raise ValueError(\n                    \"Non-existing entry in any of the parent tables - Entry: \"\n                    + f\"{row}\"\n                )\n\n        with cls._safe_context():\n            super().insert(cls(), master_entries, **kwargs)\n            for part, part_entries in parts_entries.items():\n                part.insert(part_entries, **kwargs)\n\n    @classmethod\n    def _safe_context(cls):\n        \"\"\"Return transaction if not already in one.\"\"\"\n        return (\n            cls.connection.transaction\n            if not cls.connection.in_transaction\n            else nullcontext()\n        )\n\n    @classmethod\n    def _ensure_dependencies_loaded(cls) -&gt; None:\n        \"\"\"Ensure connection dependencies loaded.\n\n        Otherwise parts returns none\n        \"\"\"\n        if not dj.conn.connection.dependencies._loaded:\n            dj.conn.connection.dependencies.load()\n\n    def insert(self, rows: list, **kwargs):\n        \"\"\"Merges table specific insert, ensuring data exists in part parents.\n\n        Parameters\n        ---------\n        rows: List[dict]\n            An iterable where an element is a dictionary.\n\n        Raises\n        ------\n        TypeError\n            If rows is not a list of dicts\n        ValueError\n            If data doesn't exist in part parents, integrity error\n        \"\"\"\n        self._merge_insert(rows, **kwargs)\n\n    @classmethod\n    def merge_view(cls, restriction: str = True):\n        \"\"\"Prints merged view, including null entries for unique columns.\n\n        Note: To handle this Union as a table-like object, use `merge_resrict`\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the merged view\n        \"\"\"\n\n        # If we overwrite `preview`, we then encounter issues with operators\n        # getting passed a `Union`, which doesn't have a method we can\n        # intercept to manage master/parts\n\n        return pprint(cls._merge_repr(restriction=restriction))\n\n    @classmethod\n    def merge_html(cls, restriction: str = True):\n        \"\"\"Displays HTML in notebooks.\"\"\"\n\n        return HTML(repr_html(cls._merge_repr(restriction=restriction)))\n\n    @classmethod\n    def merge_restrict(cls, restriction: str = True) -&gt; dj.U:\n        \"\"\"Given a restriction, return a merged view with restriction applied.\n\n        Example\n        -------\n            &gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n\n        Parameters\n        ----------\n        restriction: str\n            Restriction one would apply if `merge_view` was a real table.\n\n        Returns\n        -------\n        datajoint.Union\n            Merged view with restriction applied.\n        \"\"\"\n        return cls._merge_repr(restriction=restriction)\n\n    @classmethod\n    def merge_delete(cls, restriction: str = True, **kwargs):\n        \"\"\"Given a restriction string, delete corresponding entries.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before deletion from master/part\n            tables. If not provided, delete all entries.\n        kwargs: dict\n            Additional keyword arguments for DataJoint delete.\n\n        Example\n        -------\n            &gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n        \"\"\"\n        uuids = [\n            {k: v}\n            for entry in cls.merge_restrict(restriction).fetch(\"KEY\")\n            for k, v in entry.items()\n            if k == cls()._reserved_pk\n        ]\n        (cls() &amp; uuids).delete(**kwargs)\n\n    @classmethod\n    def merge_delete_parent(\n        cls, restriction: str = True, dry_run=True, **kwargs\n    ) -&gt; list:\n        \"\"\"Delete entries from merge master, part, and respective part parents\n\n        Note: Clears merge entries from their respective parents.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before deletion from parents. If not\n            provided, delete all entries present in Merge Table.\n        dry_run: bool\n            Default True. If true, return list of tables with entries that would\n            be deleted. Otherwise, table entries.\n        kwargs: dict\n            Additional keyword arguments for DataJoint delete.\n        \"\"\"\n        part_parents = cls._merge_restrict_parents(\n            restriction=restriction, as_objects=True, return_empties=False\n        )\n\n        if dry_run:\n            return part_parents\n\n        merge_ids = cls.merge_restrict(restriction).fetch(\n            RESERVED_PRIMARY_KEY, as_dict=True\n        )\n\n        # CB: Removed transaction protection here bc 'no' confirmation resp\n        # still resulted in deletes. If re-add, consider transaction=False\n        super().delete((cls &amp; merge_ids), **kwargs)\n\n        if cls &amp; merge_ids:  # If 'no' on del prompt from above, skip below\n            return  # User can still abort del below, but yes/no is unlikly\n\n        for part_parent in part_parents:\n            super().delete(part_parent, **kwargs)\n\n    def fetch_nwb(\n        self,\n        restriction: str = None,\n        multi_source=False,\n        disable_warning=False,\n        *attrs,\n        **kwargs,\n    ):\n        \"\"\"Return the (Analysis)Nwbfile file linked in the source.\n\n        Relies on SpyglassMixin._nwb_table_tuple to determine the table to\n        fetch from and the appropriate path attribute to return.\n\n        Parameters\n        ----------\n        restriction: str, optional\n            Restriction to apply to parents before running fetch. Default True.\n        multi_source: bool\n            Return from multiple parents. Default False.\n        \"\"\"\n        if isinstance(self, dict):\n            raise ValueError(\"Try replacing Merge.method with Merge().method\")\n        restriction = restriction or self.restriction or True\n\n        return self.merge_restrict_class(restriction).fetch_nwb()\n\n    @classmethod\n    def merge_get_part(\n        cls,\n        restriction: str = True,\n        join_master: bool = False,\n        restrict_part=True,\n        multi_source=False,\n        return_empties=False,\n    ) -&gt; dj.Table:\n        \"\"\"Retrieve part table from a restricted Merge table.\n\n        Note: unlike other Merge Table methods, returns the native table, not\n        a FreeTable\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining part to return.\n            Default True.\n        join_master: bool\n            Join part with Merge master to show source field. Default False.\n        restrict_part: bool\n            Apply restriction to part. Default True. If False, return the\n            native part table.\n        multi_source: bool\n            Return multiple parts. Default False.\n        return_empties: bool\n            Default False. Return empty part tables.\n\n        Returns\n        ------\n        Union[dj.Table, List[dj.Table]]\n            Native part table(s) of Merge. If `multi_source`, returns list.\n\n        Example\n        -------\n            &gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n            &gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n\n        Raises\n        ------\n        ValueError\n            If multiple sources are found, but not expected lists and suggests\n            restricting\n        \"\"\"\n        sources = [\n            cls._part_name(part)  # friendly part name\n            for part in cls._merge_restrict_parts(\n                restriction=restriction,\n                as_objects=False,\n                return_empties=return_empties,\n                add_invalid_restrict=False,\n            )\n        ]\n\n        if not multi_source and len(sources) != 1:\n            raise ValueError(\n                f\"Found {len(sources)} potential parts: {sources}\\n\\t\"\n                + \"Try adding a restriction before invoking `get_part`.\\n\\t\"\n                + \"Or permitting multiple sources with `multi_source=True`.\"\n            )\n        if len(sources) == 0:\n            return None\n\n        parts = [\n            (\n                getattr(cls, source)().restrict(restriction)\n                if restrict_part  # Re-apply restriction or don't\n                else getattr(cls, source)()\n            )\n            for source in sources\n        ]\n        if join_master:\n            parts = [cls * part for part in parts]\n\n        return parts if multi_source else parts[0]\n\n    @classmethod\n    def merge_get_parent(\n        cls,\n        restriction: str = True,\n        join_master: bool = False,\n        multi_source: bool = False,\n        return_empties: bool = False,\n        add_invalid_restrict: bool = True,\n    ) -&gt; dj.FreeTable:\n        \"\"\"Returns a list of part parents with restrictions applied.\n\n        Rather than part tables, we look at parents of those parts, the source\n        of the data, and only the rows that have keys inserted in the merge\n        table.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining parent to return.\n            Default True.\n        join_master: bool\n            Default False. Join part with Merge master to show uuid and source\n        multi_source: bool\n            Return multiple parents. Default False.\n        return_empties: bool\n            Default False. Return empty parent tables.\n        add_invalid_restrict: bool\n            Default True. Include parent for which the restriction is invalid.\n\n        Returns\n        ------\n        dj.FreeTable\n            Parent of parts of Merge Table as FreeTable.\n        \"\"\"\n\n        part_parents = cls._merge_restrict_parents(\n            restriction=restriction,\n            as_objects=True,\n            return_empties=return_empties,\n            add_invalid_restrict=add_invalid_restrict,\n        )\n\n        if not multi_source and len(part_parents) != 1:\n            raise ValueError(\n                f\"Found  {len(part_parents)} potential parents: {part_parents}\"\n                + \"\\n\\tTry adding a string restriction when invoking \"\n                + \"`get_parent`. Or permitting multiple sources with \"\n                + \"`multi_source=True`.\"\n            )\n\n        if join_master:\n            part_parents = [cls * part for part in part_parents]\n\n        return part_parents if multi_source else part_parents[0]\n\n    @property\n    def source_class_dict(self) -&gt; dict:\n        # NOTE: fails if table is aliased in dj.Part but not merge script\n        # i.e., must import aliased table as part name\n        if not self._source_class_dict:\n            module = getmodule(self)\n            self._source_class_dict = {\n                part_name: getattr(module, part_name)\n                for part_name in self.parts(camel_case=True)\n            }\n        return self._source_class_dict\n\n    def _normalize_source(\n        self, source: Union[str, dj.Table, dj.condition.AndList, dict]\n    ) -&gt; str:\n        fetched_source = None\n        if isinstance(source, (Merge, dj.condition.AndList)):\n            try:\n                fetched_source = (self &amp; source).fetch(self._reserved_sk)\n            except DataJointError:\n                raise ValueError(f\"Unable to find source for {source}\")\n            source = fetched_source[0]\n            if len(fetched_source) &gt; 1:\n                logger.warn(f\"Multiple sources. Selecting first: {source}.\")\n        if isinstance(source, dj.Table):\n            source = self._part_name(source)\n        if isinstance(source, dict):\n            source = self._part_name(self.merge_get_parent(source))\n\n        return source\n\n    def merge_get_parent_class(self, source: str) -&gt; dj.Table:\n        \"\"\"Return the class of the parent table for a given CamelCase source.\n\n        Parameters\n        ----------\n        source: Union[str, dict, dj.Table]\n            Accepts a CamelCase name of the source, or key as a dict, or a part\n            table.\n\n        Returns\n        -------\n        dj.Table\n            Class instance of the parent table, including class methods.\n        \"\"\"\n\n        ret = self.source_class_dict.get(self._normalize_source(source))\n        if not ret:\n            logger.error(\n                f\"No source class found for {source}: \\n\\t\"\n                + f\"{self.parts(camel_case=True)}\"\n            )\n        return ret\n\n    def merge_restrict_class(self, key: dict) -&gt; dj.Table:\n        \"\"\"Returns native parent class, restricted with key.\"\"\"\n        parent_key = self.merge_get_parent(key).fetch(\"KEY\", as_dict=True)\n\n        if len(parent_key) &gt; 1:\n            raise ValueError(\n                f\"Ambiguous entry. Data has mult rows in parent:\\n\\tData:{key}\"\n                + f\"\\n\\t{parent_key}\"\n            )\n\n        parent_class = self.merge_get_parent_class(key)\n        return parent_class &amp; parent_key\n\n    @classmethod\n    def merge_fetch(self, restriction: str = True, *attrs, **kwargs) -&gt; list:\n        \"\"\"Perform a fetch across all parts. If &gt;1 result, return as a list.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining parent to return.\n            Default True.\n        attrs, kwargs\n            arguments passed to DataJoint `fetch` call\n\n        Returns\n        -------\n        Union[ List[np.array], List[dict], List[pd.DataFrame] ]\n            Table contents, with type determined by kwargs\n        \"\"\"\n        results = []\n        parts = self()._merge_restrict_parts(\n            restriction=restriction,\n            as_objects=True,\n            return_empties=False,\n            add_invalid_restrict=False,\n        )\n\n        for part in parts:\n            try:\n                results.extend(part.fetch(*attrs, **kwargs))\n            except DataJointError as e:\n                logger.warn(\n                    f\"{e.args[0]} Skipping \"\n                    + to_camel_case(part.table_name.split(\"__\")[-1])\n                )\n\n        # Note: this could collapse results like merge_view, but user may call\n        # for recarray, pd.DataFrame, or dict, and fetched contents differ if\n        # attrs or \"KEY\" called. Intercept format, merge, and then transform?\n\n        if not results:\n            logger.info(\n                \"No merge_fetch results.\\n\\t\"\n                + \"If not restricting, try: `M.merge_fetch(True,'attr')\\n\\t\"\n                + \"If restricting by source, use dict: \"\n                + \"`M.merge_fetch({'source':'X'})\"\n            )\n        return results[0] if len(results) == 1 else results\n\n    def merge_populate(self, source: str, keys=None):\n        \"\"\"Populate the merge table with entries from the source table.\"\"\"\n        logger.warning(\"CBroz: Not fully tested. Use with caution.\")\n        parent_class = self.merge_get_parent_class(source)\n        if not keys:\n            keys = parent_class.key_source\n        parent_class.populate(keys)\n        successes = (parent_class &amp; keys).fetch(\"KEY\", as_dict=True)\n        self.insert(successes)\n\n    def delete(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n        for part in self.merge_get_part(\n            restriction=self.restriction,\n            multi_source=True,\n            return_empties=False,\n        ):\n            part.delete(force_permission=force_permission, *args, **kwargs)\n\n    def super_delete(self, *args, **kwargs):\n        \"\"\"Alias for datajoint.table.Table.delete.\n\n        Added to support MRO of SpyglassMixin\"\"\"\n        logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n\n        self._log_use(start=time(), super_delete=True)\n        super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.get_source_from_key", "title": "<code>get_source_from_key(key)</code>", "text": "<p>Return the source of a given key</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def get_source_from_key(self, key: dict) -&gt; str:\n    \"\"\"Return the source of a given key\"\"\"\n    return self._normalize_source(key)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.parts", "title": "<code>parts(camel_case=False, *args, **kwargs)</code>", "text": "<p>Return a list of part tables, add option for CamelCase names.</p> <p>See DataJoint <code>parts</code> for additional arguments. If camel_case is True, forces return of strings rather than objects.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def parts(self, camel_case=False, *args, **kwargs) -&gt; list:\n    \"\"\"Return a list of part tables, add option for CamelCase names.\n\n    See DataJoint `parts` for additional arguments. If camel_case is True,\n    forces return of strings rather than objects.\n    \"\"\"\n    self._ensure_dependencies_loaded()\n\n    if camel_case and kwargs.get(\"as_objects\"):\n        logger.warning(\n            \"Overriding as_objects=True to return CamelCase part names.\"\n        )\n        kwargs[\"as_objects\"] = False\n\n    parts = super().parts(*args, **kwargs)\n\n    if camel_case:\n        parts = [self._part_name(part) for part in parts]\n\n    return parts\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.insert", "title": "<code>insert(rows, **kwargs)</code>", "text": "<p>Merges table specific insert, ensuring data exists in part parents.</p> <p>Parameters:</p> Name Type Description Default <code>rows</code> <code>list</code> <p>An iterable where an element is a dictionary.</p> required <p>Raises:</p> Type Description <code>TypeError</code> <p>If rows is not a list of dicts</p> <code>ValueError</code> <p>If data doesn't exist in part parents, integrity error</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def insert(self, rows: list, **kwargs):\n    \"\"\"Merges table specific insert, ensuring data exists in part parents.\n\n    Parameters\n    ---------\n    rows: List[dict]\n        An iterable where an element is a dictionary.\n\n    Raises\n    ------\n    TypeError\n        If rows is not a list of dicts\n    ValueError\n        If data doesn't exist in part parents, integrity error\n    \"\"\"\n    self._merge_insert(rows, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_view", "title": "<code>merge_view(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Prints merged view, including null entries for unique columns.</p> <p>Note: To handle this Union as a table-like object, use <code>merge_resrict</code></p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to the merged view</p> <code>True</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_view(cls, restriction: str = True):\n    \"\"\"Prints merged view, including null entries for unique columns.\n\n    Note: To handle this Union as a table-like object, use `merge_resrict`\n\n    Parameters\n    ---------\n    restriction: str, optional\n        Restriction to apply to the merged view\n    \"\"\"\n\n    # If we overwrite `preview`, we then encounter issues with operators\n    # getting passed a `Union`, which doesn't have a method we can\n    # intercept to manage master/parts\n\n    return pprint(cls._merge_repr(restriction=restriction))\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_html", "title": "<code>merge_html(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Displays HTML in notebooks.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_html(cls, restriction: str = True):\n    \"\"\"Displays HTML in notebooks.\"\"\"\n\n    return HTML(repr_html(cls._merge_repr(restriction=restriction)))\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict", "title": "<code>merge_restrict(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Given a restriction, return a merged view with restriction applied.</p>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n</code></pre> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction one would apply if <code>merge_view</code> was a real table.</p> <code>True</code> <p>Returns:</p> Type Description <code>Union</code> <p>Merged view with restriction applied.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_restrict(cls, restriction: str = True) -&gt; dj.U:\n    \"\"\"Given a restriction, return a merged view with restriction applied.\n\n    Example\n    -------\n        &gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n\n    Parameters\n    ----------\n    restriction: str\n        Restriction one would apply if `merge_view` was a real table.\n\n    Returns\n    -------\n    datajoint.Union\n        Merged view with restriction applied.\n    \"\"\"\n    return cls._merge_repr(restriction=restriction)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete", "title": "<code>merge_delete(restriction=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Given a restriction string, delete corresponding entries.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from master/part tables. If not provided, delete all entries.</p> <code>True</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n</code></pre> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_delete(cls, restriction: str = True, **kwargs):\n    \"\"\"Given a restriction string, delete corresponding entries.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before deletion from master/part\n        tables. If not provided, delete all entries.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n\n    Example\n    -------\n        &gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n    \"\"\"\n    uuids = [\n        {k: v}\n        for entry in cls.merge_restrict(restriction).fetch(\"KEY\")\n        for k, v in entry.items()\n        if k == cls()._reserved_pk\n    ]\n    (cls() &amp; uuids).delete(**kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete_parent", "title": "<code>merge_delete_parent(restriction=True, dry_run=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Delete entries from merge master, part, and respective part parents</p> <p>Note: Clears merge entries from their respective parents.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from parents. If not provided, delete all entries present in Merge Table.</p> <code>True</code> <code>dry_run</code> <p>Default True. If true, return list of tables with entries that would be deleted. Otherwise, table entries.</p> <code>True</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_delete_parent(\n    cls, restriction: str = True, dry_run=True, **kwargs\n) -&gt; list:\n    \"\"\"Delete entries from merge master, part, and respective part parents\n\n    Note: Clears merge entries from their respective parents.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before deletion from parents. If not\n        provided, delete all entries present in Merge Table.\n    dry_run: bool\n        Default True. If true, return list of tables with entries that would\n        be deleted. Otherwise, table entries.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n    \"\"\"\n    part_parents = cls._merge_restrict_parents(\n        restriction=restriction, as_objects=True, return_empties=False\n    )\n\n    if dry_run:\n        return part_parents\n\n    merge_ids = cls.merge_restrict(restriction).fetch(\n        RESERVED_PRIMARY_KEY, as_dict=True\n    )\n\n    # CB: Removed transaction protection here bc 'no' confirmation resp\n    # still resulted in deletes. If re-add, consider transaction=False\n    super().delete((cls &amp; merge_ids), **kwargs)\n\n    if cls &amp; merge_ids:  # If 'no' on del prompt from above, skip below\n        return  # User can still abort del below, but yes/no is unlikly\n\n    for part_parent in part_parents:\n        super().delete(part_parent, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.fetch_nwb", "title": "<code>fetch_nwb(restriction=None, multi_source=False, disable_warning=False, *attrs, **kwargs)</code>", "text": "<p>Return the (Analysis)Nwbfile file linked in the source.</p> <p>Relies on SpyglassMixin._nwb_table_tuple to determine the table to fetch from and the appropriate path attribute to return.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to parents before running fetch. Default True.</p> <code>None</code> <code>multi_source</code> <p>Return from multiple parents. Default False.</p> <code>False</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def fetch_nwb(\n    self,\n    restriction: str = None,\n    multi_source=False,\n    disable_warning=False,\n    *attrs,\n    **kwargs,\n):\n    \"\"\"Return the (Analysis)Nwbfile file linked in the source.\n\n    Relies on SpyglassMixin._nwb_table_tuple to determine the table to\n    fetch from and the appropriate path attribute to return.\n\n    Parameters\n    ----------\n    restriction: str, optional\n        Restriction to apply to parents before running fetch. Default True.\n    multi_source: bool\n        Return from multiple parents. Default False.\n    \"\"\"\n    if isinstance(self, dict):\n        raise ValueError(\"Try replacing Merge.method with Merge().method\")\n    restriction = restriction or self.restriction or True\n\n    return self.merge_restrict_class(restriction).fetch_nwb()\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_part", "title": "<code>merge_get_part(restriction=True, join_master=False, restrict_part=True, multi_source=False, return_empties=False)</code>  <code>classmethod</code>", "text": "<p>Retrieve part table from a restricted Merge table.</p> <p>Note: unlike other Merge Table methods, returns the native table, not a FreeTable</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining part to return. Default True.</p> <code>True</code> <code>join_master</code> <code>bool</code> <p>Join part with Merge master to show source field. Default False.</p> <code>False</code> <code>restrict_part</code> <p>Apply restriction to part. Default True. If False, return the native part table.</p> <code>True</code> <code>multi_source</code> <p>Return multiple parts. Default False.</p> <code>False</code> <code>return_empties</code> <p>Default False. Return empty part tables.</p> <code>False</code> <p>Returns:</p> Type Description <code>Union[Table, List[Table]]</code> <p>Native part table(s) of Merge. If <code>multi_source</code>, returns list.</p>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_part--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n&gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n</code></pre> <p>Raises:</p> Type Description <code>ValueError</code> <p>If multiple sources are found, but not expected lists and suggests restricting</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_get_part(\n    cls,\n    restriction: str = True,\n    join_master: bool = False,\n    restrict_part=True,\n    multi_source=False,\n    return_empties=False,\n) -&gt; dj.Table:\n    \"\"\"Retrieve part table from a restricted Merge table.\n\n    Note: unlike other Merge Table methods, returns the native table, not\n    a FreeTable\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining part to return.\n        Default True.\n    join_master: bool\n        Join part with Merge master to show source field. Default False.\n    restrict_part: bool\n        Apply restriction to part. Default True. If False, return the\n        native part table.\n    multi_source: bool\n        Return multiple parts. Default False.\n    return_empties: bool\n        Default False. Return empty part tables.\n\n    Returns\n    ------\n    Union[dj.Table, List[dj.Table]]\n        Native part table(s) of Merge. If `multi_source`, returns list.\n\n    Example\n    -------\n        &gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n        &gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n\n    Raises\n    ------\n    ValueError\n        If multiple sources are found, but not expected lists and suggests\n        restricting\n    \"\"\"\n    sources = [\n        cls._part_name(part)  # friendly part name\n        for part in cls._merge_restrict_parts(\n            restriction=restriction,\n            as_objects=False,\n            return_empties=return_empties,\n            add_invalid_restrict=False,\n        )\n    ]\n\n    if not multi_source and len(sources) != 1:\n        raise ValueError(\n            f\"Found {len(sources)} potential parts: {sources}\\n\\t\"\n            + \"Try adding a restriction before invoking `get_part`.\\n\\t\"\n            + \"Or permitting multiple sources with `multi_source=True`.\"\n        )\n    if len(sources) == 0:\n        return None\n\n    parts = [\n        (\n            getattr(cls, source)().restrict(restriction)\n            if restrict_part  # Re-apply restriction or don't\n            else getattr(cls, source)()\n        )\n        for source in sources\n    ]\n    if join_master:\n        parts = [cls * part for part in parts]\n\n    return parts if multi_source else parts[0]\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_parent", "title": "<code>merge_get_parent(restriction=True, join_master=False, multi_source=False, return_empties=False, add_invalid_restrict=True)</code>  <code>classmethod</code>", "text": "<p>Returns a list of part parents with restrictions applied.</p> <p>Rather than part tables, we look at parents of those parts, the source of the data, and only the rows that have keys inserted in the merge table.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining parent to return. Default True.</p> <code>True</code> <code>join_master</code> <code>bool</code> <p>Default False. Join part with Merge master to show uuid and source</p> <code>False</code> <code>multi_source</code> <code>bool</code> <p>Return multiple parents. Default False.</p> <code>False</code> <code>return_empties</code> <code>bool</code> <p>Default False. Return empty parent tables.</p> <code>False</code> <code>add_invalid_restrict</code> <code>bool</code> <p>Default True. Include parent for which the restriction is invalid.</p> <code>True</code> <p>Returns:</p> Type Description <code>FreeTable</code> <p>Parent of parts of Merge Table as FreeTable.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_get_parent(\n    cls,\n    restriction: str = True,\n    join_master: bool = False,\n    multi_source: bool = False,\n    return_empties: bool = False,\n    add_invalid_restrict: bool = True,\n) -&gt; dj.FreeTable:\n    \"\"\"Returns a list of part parents with restrictions applied.\n\n    Rather than part tables, we look at parents of those parts, the source\n    of the data, and only the rows that have keys inserted in the merge\n    table.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining parent to return.\n        Default True.\n    join_master: bool\n        Default False. Join part with Merge master to show uuid and source\n    multi_source: bool\n        Return multiple parents. Default False.\n    return_empties: bool\n        Default False. Return empty parent tables.\n    add_invalid_restrict: bool\n        Default True. Include parent for which the restriction is invalid.\n\n    Returns\n    ------\n    dj.FreeTable\n        Parent of parts of Merge Table as FreeTable.\n    \"\"\"\n\n    part_parents = cls._merge_restrict_parents(\n        restriction=restriction,\n        as_objects=True,\n        return_empties=return_empties,\n        add_invalid_restrict=add_invalid_restrict,\n    )\n\n    if not multi_source and len(part_parents) != 1:\n        raise ValueError(\n            f\"Found  {len(part_parents)} potential parents: {part_parents}\"\n            + \"\\n\\tTry adding a string restriction when invoking \"\n            + \"`get_parent`. Or permitting multiple sources with \"\n            + \"`multi_source=True`.\"\n        )\n\n    if join_master:\n        part_parents = [cls * part for part in part_parents]\n\n    return part_parents if multi_source else part_parents[0]\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_parent_class", "title": "<code>merge_get_parent_class(source)</code>", "text": "<p>Return the class of the parent table for a given CamelCase source.</p> <p>Parameters:</p> Name Type Description Default <code>source</code> <code>str</code> <p>Accepts a CamelCase name of the source, or key as a dict, or a part table.</p> required <p>Returns:</p> Type Description <code>Table</code> <p>Class instance of the parent table, including class methods.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_get_parent_class(self, source: str) -&gt; dj.Table:\n    \"\"\"Return the class of the parent table for a given CamelCase source.\n\n    Parameters\n    ----------\n    source: Union[str, dict, dj.Table]\n        Accepts a CamelCase name of the source, or key as a dict, or a part\n        table.\n\n    Returns\n    -------\n    dj.Table\n        Class instance of the parent table, including class methods.\n    \"\"\"\n\n    ret = self.source_class_dict.get(self._normalize_source(source))\n    if not ret:\n        logger.error(\n            f\"No source class found for {source}: \\n\\t\"\n            + f\"{self.parts(camel_case=True)}\"\n        )\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict_class", "title": "<code>merge_restrict_class(key)</code>", "text": "<p>Returns native parent class, restricted with key.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_restrict_class(self, key: dict) -&gt; dj.Table:\n    \"\"\"Returns native parent class, restricted with key.\"\"\"\n    parent_key = self.merge_get_parent(key).fetch(\"KEY\", as_dict=True)\n\n    if len(parent_key) &gt; 1:\n        raise ValueError(\n            f\"Ambiguous entry. Data has mult rows in parent:\\n\\tData:{key}\"\n            + f\"\\n\\t{parent_key}\"\n        )\n\n    parent_class = self.merge_get_parent_class(key)\n    return parent_class &amp; parent_key\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_fetch", "title": "<code>merge_fetch(restriction=True, *attrs, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Perform a fetch across all parts. If &gt;1 result, return as a list.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining parent to return. Default True.</p> <code>True</code> <code>attrs</code> <p>arguments passed to DataJoint <code>fetch</code> call</p> <code>()</code> <code>kwargs</code> <p>arguments passed to DataJoint <code>fetch</code> call</p> <code>()</code> <p>Returns:</p> Type Description <code>Union[List[array], List[dict], List[DataFrame]]</code> <p>Table contents, with type determined by kwargs</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_fetch(self, restriction: str = True, *attrs, **kwargs) -&gt; list:\n    \"\"\"Perform a fetch across all parts. If &gt;1 result, return as a list.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining parent to return.\n        Default True.\n    attrs, kwargs\n        arguments passed to DataJoint `fetch` call\n\n    Returns\n    -------\n    Union[ List[np.array], List[dict], List[pd.DataFrame] ]\n        Table contents, with type determined by kwargs\n    \"\"\"\n    results = []\n    parts = self()._merge_restrict_parts(\n        restriction=restriction,\n        as_objects=True,\n        return_empties=False,\n        add_invalid_restrict=False,\n    )\n\n    for part in parts:\n        try:\n            results.extend(part.fetch(*attrs, **kwargs))\n        except DataJointError as e:\n            logger.warn(\n                f\"{e.args[0]} Skipping \"\n                + to_camel_case(part.table_name.split(\"__\")[-1])\n            )\n\n    # Note: this could collapse results like merge_view, but user may call\n    # for recarray, pd.DataFrame, or dict, and fetched contents differ if\n    # attrs or \"KEY\" called. Intercept format, merge, and then transform?\n\n    if not results:\n        logger.info(\n            \"No merge_fetch results.\\n\\t\"\n            + \"If not restricting, try: `M.merge_fetch(True,'attr')\\n\\t\"\n            + \"If restricting by source, use dict: \"\n            + \"`M.merge_fetch({'source':'X'})\"\n        )\n    return results[0] if len(results) == 1 else results\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_populate", "title": "<code>merge_populate(source, keys=None)</code>", "text": "<p>Populate the merge table with entries from the source table.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_populate(self, source: str, keys=None):\n    \"\"\"Populate the merge table with entries from the source table.\"\"\"\n    logger.warning(\"CBroz: Not fully tested. Use with caution.\")\n    parent_class = self.merge_get_parent_class(source)\n    if not keys:\n        keys = parent_class.key_source\n    parent_class.populate(keys)\n    successes = (parent_class &amp; keys).fetch(\"KEY\", as_dict=True)\n    self.insert(successes)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.delete", "title": "<code>delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete, overwrites datajoint.table.Table.delete</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def delete(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n    for part in self.merge_get_part(\n        restriction=self.restriction,\n        multi_source=True,\n        return_empties=False,\n    ):\n        part.delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.super_delete", "title": "<code>super_delete(*args, **kwargs)</code>", "text": "<p>Alias for datajoint.table.Table.delete.</p> <p>Added to support MRO of SpyglassMixin</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def super_delete(self, *args, **kwargs):\n    \"\"\"Alias for datajoint.table.Table.delete.\n\n    Added to support MRO of SpyglassMixin\"\"\"\n    logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n\n    self._log_use(start=time(), super_delete=True)\n    super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.delete_downstream_merge", "title": "<code>delete_downstream_merge(table, restriction=None, dry_run=True, recurse_level=2, disable_warning=False, **kwargs)</code>", "text": "<p>Given a table/restriction, id or delete relevant downstream merge entries</p> <p>Parameters:</p> Name Type Description Default <code>table</code> <code>Table</code> <p>DataJoint table or restriction thereof</p> required <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from merge/part tables. If not provided, delete all downstream entries.</p> <code>None</code> <code>dry_run</code> <p>Default True. If true, return list of tuples, merge/part tables downstream of table input. Otherwise, delete merge/part table entries.</p> <code>True</code> <code>disable_warning</code> <p>Default False. If True, don't warn about restrictions on table object.</p> <code>False</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code> <p>Returns:</p> Type Description <code>List[Tuple[Table, Table]]</code> <p>Entries in merge/part tables downstream of table input.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def delete_downstream_merge(\n    table: dj.Table,\n    restriction: str = None,\n    dry_run=True,\n    recurse_level=2,\n    disable_warning=False,\n    **kwargs,\n) -&gt; list:\n    \"\"\"Given a table/restriction, id or delete relevant downstream merge entries\n\n    Parameters\n    ----------\n    table: dj.Table\n        DataJoint table or restriction thereof\n    restriction: str\n        Optional restriction to apply before deletion from merge/part\n        tables. If not provided, delete all downstream entries.\n    dry_run: bool\n        Default True. If true, return list of tuples, merge/part tables\n        downstream of table input. Otherwise, delete merge/part table entries.\n    disable_warning: bool\n        Default False. If True, don't warn about restrictions on table object.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n\n    Returns\n    -------\n    List[Tuple[dj.Table, dj.Table]]\n        Entries in merge/part tables downstream of table input.\n    \"\"\"\n    from spyglass.utils.dj_mixin import SpyglassMixin\n\n    if not isinstance(table, SpyglassMixin):\n        raise ValueError(\"Input must be a Spyglass Table.\")\n    table = table if isinstance(table, dj.Table) else table()\n\n    return table.delete_downstream_merge(\n        restriction=restriction, dry_run=dry_run, **kwargs\n    )\n</code></pre>"}, {"location": "api/utils/dj_mixin/", "title": "dj_mixin.py", "text": ""}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin", "title": "<code>SpyglassMixin</code>", "text": "<p>Mixin for Spyglass DataJoint tables.</p> <p>Provides methods for fetching NWBFile objects and checking user permission prior to deleting. As a mixin class, all Spyglass tables can inherit custom methods from a central location.</p> <p>Methods:</p> Name Description <code>fetch_nwb</code> <p>Fetch NWBFile object from relevant table. Uses either a foreign key to a NWBFile table (including AnalysisNwbfile) or a _nwb_table attribute to determine which table to use.</p> <code>delte_downstream_merge</code> <p>Delete downstream merge table entries associated with restriction. Requires caching of merge tables and links, which is slow on first call. <code>restriction</code> can be set to a string to restrict the delete. <code>dry_run</code> can be set to False to commit the delete. <code>reload_cache</code> can be set to True to reload the merge cache.</p> <code>ddm</code> <p>Alias for delete_downstream_merge.</p> <code>cautious_delete</code> <p>Check user permissions before deleting table rows. Permission is granted to users listed as admin in LabMember table or to users on a team with with the Session experimenter(s). If the table where the delete is executed cannot be linked to a Session, a warning is logged and the delete continues. If the Session has no experimenter, or if the user is not on a team with the Session experimenter(s), a PermissionError is raised. <code>force_permission</code> can be set to True to bypass permission check.</p> <code>cdel</code> <p>Alias for cautious_delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>class SpyglassMixin:\n    \"\"\"Mixin for Spyglass DataJoint tables.\n\n    Provides methods for fetching NWBFile objects and checking user permission\n    prior to deleting. As a mixin class, all Spyglass tables can inherit custom\n    methods from a central location.\n\n    Methods\n    -------\n    fetch_nwb(*attrs, **kwargs)\n        Fetch NWBFile object from relevant table. Uses either a foreign key to\n        a NWBFile table (including AnalysisNwbfile) or a _nwb_table attribute to\n        determine which table to use.\n    delte_downstream_merge(restriction=None, dry_run=True, reload_cache=False)\n        Delete downstream merge table entries associated with restriction.\n        Requires caching of merge tables and links, which is slow on first call.\n        `restriction` can be set to a string to restrict the delete. `dry_run`\n        can be set to False to commit the delete. `reload_cache` can be set to\n        True to reload the merge cache.\n    ddm(*args, **kwargs)\n        Alias for delete_downstream_merge.\n    cautious_delete(force_permission=False, *args, **kwargs)\n        Check user permissions before deleting table rows. Permission is granted\n        to users listed as admin in LabMember table or to users on a team with\n        with the Session experimenter(s). If the table where the delete is\n        executed cannot be linked to a Session, a warning is logged and the\n        delete continues. If the Session has no experimenter, or if the user is\n        not on a team with the Session experimenter(s), a PermissionError is\n        raised. `force_permission` can be set to True to bypass permission check.\n    cdel(*args, **kwargs)\n        Alias for cautious_delete.\n    \"\"\"\n\n    # _nwb_table = None # NWBFile table class, defined at the table level\n\n    # pks for delete permission check, assumed to be one field for each\n    _session_pk = None  # Session primary key. Mixin is ambivalent to Session pk\n    _member_pk = None  # LabMember primary key. Mixin ambivalent table structure\n\n    def __init__(self, *args, **kwargs):\n        \"\"\"Initialize SpyglassMixin.\n\n        Checks that schema prefix is in SHARED_MODULES.\n        \"\"\"\n        if (\n            self.database  # Connected to a database\n            and not self.is_declared  # New table\n            and self.database.split(\"_\")[0]  # Prefix\n            not in [\n                *SHARED_MODULES,  # Shared modules\n                dj.config[\"database.user\"],  # User schema\n                \"temp\",\n                \"test\",\n            ]\n        ):\n            logger.error(\n                f\"Schema prefix not in SHARED_MODULES: {self.database}\"\n            )\n\n    # ------------------------------- fetch_nwb -------------------------------\n\n    @cached_property\n    def _nwb_table_tuple(self) -&gt; tuple:\n        \"\"\"NWBFile table class.\n\n        Used to determine fetch_nwb behavior. Also used in Merge.fetch_nwb.\n        Implemented as a cached_property to avoid circular imports.\"\"\"\n        from spyglass.common.common_nwbfile import (\n            AnalysisNwbfile,\n            Nwbfile,\n        )  # noqa F401\n\n        table_dict = {\n            AnalysisNwbfile: \"analysis_file_abs_path\",\n            Nwbfile: \"nwb_file_abs_path\",\n        }\n\n        resolved = getattr(self, \"_nwb_table\", None) or (\n            AnalysisNwbfile\n            if \"-&gt; AnalysisNwbfile\" in self.definition\n            else Nwbfile if \"-&gt; Nwbfile\" in self.definition else None\n        )\n\n        if not resolved:\n            raise NotImplementedError(\n                f\"{self.__class__.__name__} does not have a \"\n                \"(Analysis)Nwbfile foreign key or _nwb_table attribute.\"\n            )\n\n        return (\n            resolved,\n            table_dict[resolved],\n        )\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        \"\"\"Fetch NWBFile object from relevant table.\n\n        Implementing class must have a foreign key reference to Nwbfile or\n        AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition)\n        or a _nwb_table attribute. If both are present, the attribute takes\n        precedence.\n        \"\"\"\n        return fetch_nwb(self, self._nwb_table_tuple, *attrs, **kwargs)\n\n    # ------------------------ delete_downstream_merge ------------------------\n\n    @cached_property\n    def _merge_tables(self) -&gt; Dict[str, dj.FreeTable]:\n        \"\"\"Dict of merge tables downstream of self: {full_table_name: FreeTable}.\n\n        Cache of items in parents of self.descendants(as_objects=True). Both\n        descendant and parent must have the reserved primary key 'merge_id'.\n        \"\"\"\n        self.connection.dependencies.load()\n        merge_tables = {}\n\n        def search_descendants(parent):\n            for desc in parent.descendants(as_objects=True):\n                if (\n                    MERGE_PK not in desc.heading.names\n                    or not (master_name := get_master(desc.full_table_name))\n                    or master_name in merge_tables\n                ):\n                    continue\n                master = dj.FreeTable(self.connection, master_name)\n                if MERGE_PK in master.heading.names:\n                    merge_tables[master_name] = master\n                    search_descendants(master)\n\n        _ = search_descendants(self)\n\n        logger.info(\n            f\"Building merge cache for {self.table_name}.\\n\\t\"\n            + f\"Found {len(merge_tables)} downstream merge tables\"\n        )\n\n        return merge_tables\n\n    @cached_property\n    def _merge_chains(self) -&gt; OrderedDict[str, List[dj.FreeTable]]:\n        \"\"\"Dict of chains to merges downstream of self\n\n        Format: {full_table_name: TableChains}.\n\n        For each merge table found in _merge_tables, find the path from self to\n        merge via merge parts. If the path is valid, add it to the dict. Cache\n        prevents need to recompute whenever delete_downstream_merge is called\n        with a new restriction. To recompute, add `reload_cache=True` to\n        delete_downstream_merge call.\n        \"\"\"\n        merge_chains = {}\n        for name, merge_table in self._merge_tables.items():\n            chains = TableChains(self, merge_table, connection=self.connection)\n            if len(chains):\n                merge_chains[name] = chains\n\n        # This is ordered by max_len of chain from self to merge, which assumes\n        # that the merge table with the longest chain is the most downstream.\n        # A more sophisticated approach would order by length from self to\n        # each merge part independently, but this is a good first approximation.\n        return OrderedDict(\n            sorted(\n                merge_chains.items(), key=lambda x: x[1].max_len, reverse=True\n            )\n        )\n\n    def _get_chain(self, substring) -&gt; TableChains:\n        \"\"\"Return chain from self to merge table with substring in name.\"\"\"\n        for name, chain in self._merge_chains.items():\n            if substring.lower() in name:\n                return chain\n\n    def _commit_merge_deletes(\n        self, merge_join_dict: Dict[str, List[QueryExpression]], **kwargs\n    ) -&gt; None:\n        \"\"\"Commit merge deletes.\n\n        Parameters\n        ----------\n        merge_join_dict : Dict[str, List[QueryExpression]]\n            Dictionary of merge tables and their joins. Uses 'merge_id' primary\n            key to restrict delete.\n\n        Extracted for use in cautious_delete and delete_downstream_merge.\"\"\"\n        for table_name, part_restr in merge_join_dict.items():\n            table = self._merge_tables[table_name]\n            keys = [part.fetch(MERGE_PK, as_dict=True) for part in part_restr]\n            (table &amp; keys).delete(**kwargs)\n\n    def delete_downstream_merge(\n        self,\n        restriction: str = None,\n        dry_run: bool = True,\n        reload_cache: bool = False,\n        disable_warning: bool = False,\n        return_parts: bool = True,\n        **kwargs,\n    ) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n        \"\"\"Delete downstream merge table entries associated with restriction.\n\n        Requires caching of merge tables and links, which is slow on first call.\n\n        Parameters\n        ----------\n        restriction : str, optional\n            Restriction to apply to merge tables. Default None. Will attempt to\n            use table restriction if None.\n        dry_run : bool, optional\n            If True, return list of merge part entries to be deleted. Default\n            True.\n        reload_cache : bool, optional\n            If True, reload merge cache. Default False.\n        disable_warning : bool, optional\n            If True, do not warn if no merge tables found. Default False.\n        return_parts : bool, optional\n            If True, return list of merge part entries to be deleted. Default\n            True. If False, return dictionary of merge tables and their joins.\n        **kwargs : Any\n            Passed to datajoint.table.Table.delete.\n        \"\"\"\n        if reload_cache:\n            del self._merge_tables\n            del self._merge_chains\n\n        restriction = restriction or self.restriction or True\n\n        merge_join_dict = {}\n        for name, chain in self._merge_chains.items():\n            join = chain.join(restriction)\n            if join:\n                merge_join_dict[name] = join\n\n        if not merge_join_dict and not disable_warning:\n            logger.warning(\n                f\"No merge deletes found w/ {self.table_name} &amp; \"\n                + f\"{restriction}.\\n\\tIf this is unexpected, try importing \"\n                + \" Merge table(s) and running with `reload_cache`.\"\n            )\n\n        if dry_run:\n            return merge_join_dict.values() if return_parts else merge_join_dict\n\n        self._commit_merge_deletes(merge_join_dict, **kwargs)\n\n    def ddm(\n        self,\n        restriction: str = None,\n        dry_run: bool = True,\n        reload_cache: bool = False,\n        disable_warning: bool = False,\n        return_parts: bool = True,\n        *args,\n        **kwargs,\n    ) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n        \"\"\"Alias for delete_downstream_merge.\"\"\"\n        return self.delete_downstream_merge(\n            restriction=restriction,\n            dry_run=dry_run,\n            reload_cache=reload_cache,\n            disable_warning=disable_warning,\n            return_parts=return_parts,\n            *args,\n            **kwargs,\n        )\n\n    # ---------------------------- cautious_delete ----------------------------\n\n    @cached_property\n    def _delete_deps(self) -&gt; List[Table]:\n        \"\"\"List of tables required for delete permission check.\n\n        LabMember, LabTeam, and Session are required for delete permission.\n\n        Used to delay import of tables until needed, avoiding circular imports.\n        Each of these tables inheits SpyglassMixin.\n        \"\"\"\n        from spyglass.common import LabMember, LabTeam, Session  # noqa F401\n\n        self._session_pk = Session.primary_key[0]\n        self._member_pk = LabMember.primary_key[0]\n        return [LabMember, LabTeam, Session]\n\n    def _get_exp_summary(self):\n        \"\"\"Get summary of experimenters for session(s), including NULL.\n\n        Parameters\n        ----------\n        sess_link : datajoint.expression.QueryExpression\n            Join of table link with Session table.\n\n        Returns\n        -------\n        str\n            Summary of experimenters for session(s).\n        \"\"\"\n        Session = self._delete_deps[-1]\n        SesExp = Session.Experimenter\n        empty_pk = {self._member_pk: \"NULL\"}\n\n        format = dj.U(self._session_pk, self._member_pk)\n        sess_link = self._session_connection.join(\n            self.restriction, reverse_order=True\n        )\n\n        exp_missing = format &amp; (sess_link - SesExp).proj(**empty_pk)\n        exp_present = format &amp; (sess_link * SesExp - exp_missing).proj()\n\n        return exp_missing + exp_present\n\n    @cached_property\n    def _session_connection(self) -&gt; Union[TableChain, bool]:\n        \"\"\"Path from Session table to self. False if no connection found.\"\"\"\n        connection = TableChain(parent=self._delete_deps[-1], child=self)\n        return connection if connection.has_link else False\n\n    @cached_property\n    def _test_mode(self) -&gt; bool:\n        \"\"\"Return True if in test mode.\n\n        Avoids circular import. Prevents prompt on delete.\"\"\"\n        from spyglass.settings import test_mode\n\n        return test_mode\n\n    def _check_delete_permission(self) -&gt; None:\n        \"\"\"Check user name against lab team assoc. w/ self -&gt; Session.\n\n        Returns\n        -------\n        None\n            Permission granted.\n\n        Raises\n        ------\n        PermissionError\n            Permission denied because (a) Session has no experimenter, or (b)\n            user is not on a team with Session experimenter(s).\n        \"\"\"\n        LabMember, LabTeam, Session = self._delete_deps\n\n        dj_user = dj.config[\"database.user\"]\n        if dj_user in LabMember().admin:  # bypass permission check for admin\n            return\n\n        if not self._session_connection:\n            logger.warn(  # Permit delete if no session connection\n                \"Could not find lab team associated with \"\n                + f\"{self.__class__.__name__}.\"\n                + \"\\nBe careful not to delete others' data.\"\n            )\n            return\n\n        sess_summary = self._get_exp_summary()\n        experimenters = sess_summary.fetch(self._member_pk)\n        if None in experimenters:\n            # TODO: Check if allow delete of remainder?\n            raise PermissionError(\n                \"Please ensure all Sessions have an experimenter in \"\n                + f\"SessionExperimenter:\\n{sess_summary}\"\n            )\n\n        user_name = LabMember().get_djuser_name(dj_user)\n        for experimenter in set(experimenters):\n            # Check once with cache, if fails, reload and check again\n            # On eval as set, reload will only be called once\n            if user_name not in LabTeam().get_team_members(\n                experimenter\n            ) and user_name not in LabTeam().get_team_members(\n                experimenter, reload=True\n            ):\n                sess_w_exp = sess_summary &amp; {self._member_pk: experimenter}\n                raise PermissionError(\n                    f\"User '{user_name}' is not on a team with '{experimenter}'\"\n                    + \", an experimenter for session(s):\\n\"\n                    + f\"{sess_w_exp}\"\n                )\n        logger.info(f\"Queueing delete for session(s):\\n{sess_summary}\")\n\n    @cached_property\n    def _usage_table(self):\n        \"\"\"Temporary inclusion for usage tracking.\"\"\"\n        from spyglass.common.common_usage import CautiousDelete\n\n        return CautiousDelete()\n\n    def _log_use(self, start, merge_deletes=None, super_delete=False):\n        \"\"\"Log use of cautious_delete.\"\"\"\n        if isinstance(merge_deletes, QueryExpression):\n            merge_deletes = merge_deletes.fetch(as_dict=True)\n        safe_insert = dict(\n            duration=time() - start,\n            dj_user=dj.config[\"database.user\"],\n            origin=self.full_table_name,\n        )\n        restr_str = \"Super delete: \" if super_delete else \"\"\n        restr_str += \"\".join(self.restriction) if self.restriction else \"None\"\n        try:\n            self._usage_table.insert1(\n                dict(\n                    **safe_insert,\n                    restriction=restr_str[:255],\n                    merge_deletes=merge_deletes,\n                )\n            )\n        except (DataJointError, DataError):\n            self._usage_table.insert1(\n                dict(**safe_insert, restriction=\"Unknown\")\n            )\n\n    # TODO: Intercept datajoint delete confirmation prompt for merge deletes\n    def cautious_delete(self, force_permission: bool = False, *args, **kwargs):\n        \"\"\"Delete table rows after checking user permission.\n\n        Permission is granted to users listed as admin in LabMember table or to\n        users on a team with with the Session experimenter(s). If the table\n        cannot be linked to Session, a warning is logged and the delete\n        continues. If the Session has no experimenter, or if the user is not on\n        a team with the Session experimenter(s), a PermissionError is raised.\n\n        Parameters\n        ----------\n        force_permission : bool, optional\n            Bypass permission check. Default False.\n        *args, **kwargs : Any\n            Passed to datajoint.table.Table.delete.\n        \"\"\"\n        start = time()\n\n        if not force_permission:\n            self._check_delete_permission()\n\n        merge_deletes = self.delete_downstream_merge(\n            dry_run=True,\n            disable_warning=True,\n            return_parts=False,\n        )\n\n        safemode = (\n            dj.config.get(\"safemode\", True)\n            if kwargs.get(\"safemode\") is None\n            else kwargs[\"safemode\"]\n        )\n\n        if merge_deletes:\n            for table, content in merge_deletes.items():\n                count = sum([len(part) for part in content])\n                dj_logger.info(f\"Merge: Deleting {count} rows from {table}\")\n            if (\n                not self._test_mode\n                or not safemode\n                or user_choice(\"Commit deletes?\", default=\"no\") == \"yes\"\n            ):\n                self._commit_merge_deletes(merge_deletes, **kwargs)\n            else:\n                logger.info(\"Delete aborted.\")\n                self._log_use(start)\n                return\n\n        super().delete(*args, **kwargs)  # Additional confirm here\n\n        self._log_use(start=start, merge_deletes=merge_deletes)\n\n    def cdel(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete.\"\"\"\n        self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n\n    def delete(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n        self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n\n    def super_delete(self, *args, **kwargs):\n        \"\"\"Alias for datajoint.table.Table.delete.\"\"\"\n        logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n        self._log_use(start=time(), super_delete=True)\n        super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.__init__", "title": "<code>__init__(*args, **kwargs)</code>", "text": "<p>Initialize SpyglassMixin.</p> <p>Checks that schema prefix is in SHARED_MODULES.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def __init__(self, *args, **kwargs):\n    \"\"\"Initialize SpyglassMixin.\n\n    Checks that schema prefix is in SHARED_MODULES.\n    \"\"\"\n    if (\n        self.database  # Connected to a database\n        and not self.is_declared  # New table\n        and self.database.split(\"_\")[0]  # Prefix\n        not in [\n            *SHARED_MODULES,  # Shared modules\n            dj.config[\"database.user\"],  # User schema\n            \"temp\",\n            \"test\",\n        ]\n    ):\n        logger.error(\n            f\"Schema prefix not in SHARED_MODULES: {self.database}\"\n        )\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>Fetch NWBFile object from relevant table.</p> <p>Implementing class must have a foreign key reference to Nwbfile or AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition) or a _nwb_table attribute. If both are present, the attribute takes precedence.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs):\n    \"\"\"Fetch NWBFile object from relevant table.\n\n    Implementing class must have a foreign key reference to Nwbfile or\n    AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition)\n    or a _nwb_table attribute. If both are present, the attribute takes\n    precedence.\n    \"\"\"\n    return fetch_nwb(self, self._nwb_table_tuple, *attrs, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.delete_downstream_merge", "title": "<code>delete_downstream_merge(restriction=None, dry_run=True, reload_cache=False, disable_warning=False, return_parts=True, **kwargs)</code>", "text": "<p>Delete downstream merge table entries associated with restriction.</p> <p>Requires caching of merge tables and links, which is slow on first call.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to merge tables. Default None. Will attempt to use table restriction if None.</p> <code>None</code> <code>dry_run</code> <code>bool</code> <p>If True, return list of merge part entries to be deleted. Default True.</p> <code>True</code> <code>reload_cache</code> <code>bool</code> <p>If True, reload merge cache. Default False.</p> <code>False</code> <code>disable_warning</code> <code>bool</code> <p>If True, do not warn if no merge tables found. Default False.</p> <code>False</code> <code>return_parts</code> <code>bool</code> <p>If True, return list of merge part entries to be deleted. Default True. If False, return dictionary of merge tables and their joins.</p> <code>True</code> <code>**kwargs</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>{}</code> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def delete_downstream_merge(\n    self,\n    restriction: str = None,\n    dry_run: bool = True,\n    reload_cache: bool = False,\n    disable_warning: bool = False,\n    return_parts: bool = True,\n    **kwargs,\n) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n    \"\"\"Delete downstream merge table entries associated with restriction.\n\n    Requires caching of merge tables and links, which is slow on first call.\n\n    Parameters\n    ----------\n    restriction : str, optional\n        Restriction to apply to merge tables. Default None. Will attempt to\n        use table restriction if None.\n    dry_run : bool, optional\n        If True, return list of merge part entries to be deleted. Default\n        True.\n    reload_cache : bool, optional\n        If True, reload merge cache. Default False.\n    disable_warning : bool, optional\n        If True, do not warn if no merge tables found. Default False.\n    return_parts : bool, optional\n        If True, return list of merge part entries to be deleted. Default\n        True. If False, return dictionary of merge tables and their joins.\n    **kwargs : Any\n        Passed to datajoint.table.Table.delete.\n    \"\"\"\n    if reload_cache:\n        del self._merge_tables\n        del self._merge_chains\n\n    restriction = restriction or self.restriction or True\n\n    merge_join_dict = {}\n    for name, chain in self._merge_chains.items():\n        join = chain.join(restriction)\n        if join:\n            merge_join_dict[name] = join\n\n    if not merge_join_dict and not disable_warning:\n        logger.warning(\n            f\"No merge deletes found w/ {self.table_name} &amp; \"\n            + f\"{restriction}.\\n\\tIf this is unexpected, try importing \"\n            + \" Merge table(s) and running with `reload_cache`.\"\n        )\n\n    if dry_run:\n        return merge_join_dict.values() if return_parts else merge_join_dict\n\n    self._commit_merge_deletes(merge_join_dict, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.ddm", "title": "<code>ddm(restriction=None, dry_run=True, reload_cache=False, disable_warning=False, return_parts=True, *args, **kwargs)</code>", "text": "<p>Alias for delete_downstream_merge.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def ddm(\n    self,\n    restriction: str = None,\n    dry_run: bool = True,\n    reload_cache: bool = False,\n    disable_warning: bool = False,\n    return_parts: bool = True,\n    *args,\n    **kwargs,\n) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n    \"\"\"Alias for delete_downstream_merge.\"\"\"\n    return self.delete_downstream_merge(\n        restriction=restriction,\n        dry_run=dry_run,\n        reload_cache=reload_cache,\n        disable_warning=disable_warning,\n        return_parts=return_parts,\n        *args,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.cautious_delete", "title": "<code>cautious_delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Delete table rows after checking user permission.</p> <p>Permission is granted to users listed as admin in LabMember table or to users on a team with with the Session experimenter(s). If the table cannot be linked to Session, a warning is logged and the delete continues. If the Session has no experimenter, or if the user is not on a team with the Session experimenter(s), a PermissionError is raised.</p> <p>Parameters:</p> Name Type Description Default <code>force_permission</code> <code>bool</code> <p>Bypass permission check. Default False.</p> <code>False</code> <code>*args</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>()</code> <code>**kwargs</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>()</code> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def cautious_delete(self, force_permission: bool = False, *args, **kwargs):\n    \"\"\"Delete table rows after checking user permission.\n\n    Permission is granted to users listed as admin in LabMember table or to\n    users on a team with with the Session experimenter(s). If the table\n    cannot be linked to Session, a warning is logged and the delete\n    continues. If the Session has no experimenter, or if the user is not on\n    a team with the Session experimenter(s), a PermissionError is raised.\n\n    Parameters\n    ----------\n    force_permission : bool, optional\n        Bypass permission check. Default False.\n    *args, **kwargs : Any\n        Passed to datajoint.table.Table.delete.\n    \"\"\"\n    start = time()\n\n    if not force_permission:\n        self._check_delete_permission()\n\n    merge_deletes = self.delete_downstream_merge(\n        dry_run=True,\n        disable_warning=True,\n        return_parts=False,\n    )\n\n    safemode = (\n        dj.config.get(\"safemode\", True)\n        if kwargs.get(\"safemode\") is None\n        else kwargs[\"safemode\"]\n    )\n\n    if merge_deletes:\n        for table, content in merge_deletes.items():\n            count = sum([len(part) for part in content])\n            dj_logger.info(f\"Merge: Deleting {count} rows from {table}\")\n        if (\n            not self._test_mode\n            or not safemode\n            or user_choice(\"Commit deletes?\", default=\"no\") == \"yes\"\n        ):\n            self._commit_merge_deletes(merge_deletes, **kwargs)\n        else:\n            logger.info(\"Delete aborted.\")\n            self._log_use(start)\n            return\n\n    super().delete(*args, **kwargs)  # Additional confirm here\n\n    self._log_use(start=start, merge_deletes=merge_deletes)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.cdel", "title": "<code>cdel(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def cdel(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete.\"\"\"\n    self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.delete", "title": "<code>delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete, overwrites datajoint.table.Table.delete</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def delete(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n    self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.super_delete", "title": "<code>super_delete(*args, **kwargs)</code>", "text": "<p>Alias for datajoint.table.Table.delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def super_delete(self, *args, **kwargs):\n    \"\"\"Alias for datajoint.table.Table.delete.\"\"\"\n    logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n    self._log_use(start=time(), super_delete=True)\n    super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/logging/", "title": "logging.py", "text": "<p>Logging configuration based on datajoint/logging.py</p>"}, {"location": "api/utils/nwb_helper_fn/", "title": "nwb_helper_fn.py", "text": "<p>NWB helper functions for finding processing modules and data interfaces.</p>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_nwb_file", "title": "<code>get_nwb_file(nwb_file_path)</code>", "text": "<p>Return an NWBFile object with the given file path in read mode.</p> <p>If the file is not found locally, this will check if it has been shared with kachery and if so, download it and open it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_path</code> <code>str</code> <p>Path to the NWB file or NWB file name. If it does not start with a \"/\", get path with Nwbfile.get_abs_path</p> required <p>Returns:</p> Name Type Description <code>nwbfile</code> <code>NWBFile</code> <p>NWB file object for the given path opened in read mode.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_nwb_file(nwb_file_path):\n    \"\"\"Return an NWBFile object with the given file path in read mode.\n\n    If the file is not found locally, this will check if it has been shared\n    with kachery and if so, download it and open it.\n\n\n    Parameters\n    ----------\n    nwb_file_path : str\n        Path to the NWB file or NWB file name. If it does not start with a \"/\",\n        get path with Nwbfile.get_abs_path\n\n    Returns\n    -------\n    nwbfile : pynwb.NWBFile\n        NWB file object for the given path opened in read mode.\n    \"\"\"\n    if not nwb_file_path.startswith(\"/\"):\n        from ..common import Nwbfile\n\n        nwb_file_path = Nwbfile.get_abs_path(nwb_file_path)\n\n    _, nwbfile = __open_nwb_files.get(nwb_file_path, (None, None))\n\n    if nwbfile is None:\n        # check to see if the file exists\n        if not os.path.exists(nwb_file_path):\n            logger.info(\n                \"NWB file not found locally; checking kachery for \"\n                + f\"{nwb_file_path}\"\n            )\n            # first try the analysis files\n            from ..sharing.sharing_kachery import AnalysisNwbfileKachery\n\n            # the download functions assume just the filename, so we need to\n            # get that from the path\n            if not AnalysisNwbfileKachery.download_file(\n                os.path.basename(nwb_file_path)\n            ):\n                return None\n        # now open the file\n        io = pynwb.NWBHDF5IO(\n            path=nwb_file_path, mode=\"r\", load_namespaces=True\n        )  # keep file open\n        nwbfile = io.read()\n        __open_nwb_files[nwb_file_path] = (io, nwbfile)\n\n    return nwbfile\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_config", "title": "<code>get_config(nwb_file_path)</code>", "text": "<p>Return a dictionary of config settings for the given NWB file. If the file does not exist, return an empty dict.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_path</code> <code>str</code> <p>Absolute path to the NWB file.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Dictionary of configuration settings loaded from the corresponding YAML file</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_config(nwb_file_path):\n    \"\"\"Return a dictionary of config settings for the given NWB file.\n    If the file does not exist, return an empty dict.\n\n    Parameters\n    ----------\n    nwb_file_path : str\n        Absolute path to the NWB file.\n\n    Returns\n    -------\n    dict\n        Dictionary of configuration settings loaded from the corresponding YAML file\n    \"\"\"\n    if nwb_file_path in __configs:  # load from cache if exists\n        return __configs[nwb_file_path]\n\n    p = Path(nwb_file_path)\n    # NOTE use p.stem[:-1] to remove the underscore that was added to the file\n    config_path = p.parent / (p.stem[:-1] + \"_spyglass_config.yaml\")\n    if not os.path.exists(config_path):\n        logger.info(f\"No config found at file path {config_path}\")\n        return dict()\n    with open(config_path, \"r\") as stream:\n        d = yaml.safe_load(stream)\n\n    # TODO write a JSON schema for the yaml file and validate the yaml file\n    __configs[nwb_file_path] = d  # store in cache\n    return d\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_data_interface", "title": "<code>get_data_interface(nwbfile, data_interface_name, data_interface_class=None)</code>", "text": "<p>Search for NWBDataInterface or DynamicTable in processing modules of an NWB.</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object to search in.</p> required <code>data_interface_name</code> <code>str</code> <p>The name of the NWBDataInterface or DynamicTable to search for.</p> required <code>data_interface_class</code> <code>type</code> <p>The class (or superclass) to search for. This argument helps to prevent accessing an object with the same name but the incorrect type. Default: no restriction.</p> <code>None</code> <p>Warns:</p> Type Description <code>LoggerWarning</code> <p>If multiple NWBDataInterface and DynamicTable objects with the matching name are found.</p> <p>Returns:</p> Name Type Description <code>data_interface</code> <code>NWBDataInterface</code> <p>The data interface object with the given name, or None if not found.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_data_interface(nwbfile, data_interface_name, data_interface_class=None):\n    \"\"\"\n    Search for NWBDataInterface or DynamicTable in processing modules of an NWB.\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object to search in.\n    data_interface_name : str\n        The name of the NWBDataInterface or DynamicTable to search for.\n    data_interface_class : type, optional\n        The class (or superclass) to search for. This argument helps to prevent\n        accessing an object with the same name but the incorrect type. Default:\n        no restriction.\n\n    Warns\n    -----\n    LoggerWarning\n        If multiple NWBDataInterface and DynamicTable objects with the matching\n        name are found.\n\n    Returns\n    -------\n    data_interface : NWBDataInterface\n        The data interface object with the given name, or None if not found.\n    \"\"\"\n    ret = []\n    for module in nwbfile.processing.values():\n        match = module.data_interfaces.get(data_interface_name, None)\n        if match is not None:\n            if data_interface_class is not None and not isinstance(\n                match, data_interface_class\n            ):\n                continue\n            ret.append(match)\n    if len(ret) &gt; 1:\n        logger.warning(\n            f\"Multiple data interfaces with name '{data_interface_name}' \"\n            f\"found in NWBFile with identifier {nwbfile.identifier}. \"\n            + \"Using the first one found. \"\n            \"Use the data_interface_class argument to restrict the search.\"\n        )\n    if len(ret) &gt;= 1:\n        return ret[0]\n\n    return None\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_position_obj", "title": "<code>get_position_obj(nwbfile)</code>", "text": "<p>Return the Position object from the behavior processing module. Meant to find position spatial series that are not found by <code>get_data_interface(nwbfile, 'position', pynwb.behavior.Position)</code>. The code returns the first <code>pynwb.behavior.Position</code> object (technically there should only be one).</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object.</p> required <p>Returns:</p> Type Description <code>pynwb.behavior.Position object</code> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_position_obj(nwbfile):\n    \"\"\"Return the Position object from the behavior processing module.\n    Meant to find position spatial series that are not found by\n    `get_data_interface(nwbfile, 'position', pynwb.behavior.Position)`.\n    The code returns the first `pynwb.behavior.Position` object (technically\n    there should only be one).\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object.\n\n    Returns\n    -------\n    pynwb.behavior.Position object\n    \"\"\"\n    ret = []\n    for obj in nwbfile.processing[\"behavior\"].data_interfaces.values():\n        if isinstance(obj, pynwb.behavior.Position):\n            ret.append(obj)\n    if len(ret) &gt; 1:\n        raise ValueError(f\"Found more than one position object in {nwbfile}\")\n    return ret[0] if ret and len(ret) else None\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_raw_eseries", "title": "<code>get_raw_eseries(nwbfile)</code>", "text": "<p>Return all ElectricalSeries in the acquisition group of an NWB file.</p> <p>ElectricalSeries found within LFP objects in the acquisition will also be returned.</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object to search in.</p> required <p>Returns:</p> Name Type Description <code>ret</code> <code>list</code> <p>A list of all ElectricalSeries in the acquisition group of an NWB file</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_raw_eseries(nwbfile):\n    \"\"\"Return all ElectricalSeries in the acquisition group of an NWB file.\n\n    ElectricalSeries found within LFP objects in the acquisition will also be\n    returned.\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object to search in.\n\n    Returns\n    -------\n    ret : list\n        A list of all ElectricalSeries in the acquisition group of an NWB file\n    \"\"\"\n    ret = []\n    for nwb_object in nwbfile.acquisition.values():\n        if isinstance(nwb_object, pynwb.ecephys.ElectricalSeries):\n            ret.append(nwb_object)\n        elif isinstance(nwb_object, pynwb.ecephys.LFP):\n            ret.extend(nwb_object.electrical_series.values())\n    return ret\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.estimate_sampling_rate", "title": "<code>estimate_sampling_rate(timestamps, multiplier=1.75, verbose=False, filename='file')</code>", "text": "<p>Estimate the sampling rate given a list of timestamps.</p> <p>Assumes that the most common temporal differences between timestamps approximate the sampling rate. Note that this can fail for very high sampling rates and irregular timestamps.</p> <p>Parameters:</p> Name Type Description Default <code>timestamps</code> <code>ndarray</code> <p>1D numpy array of timestamp values.</p> required <code>multiplier</code> <code>float or int</code> <p>Deft</p> <code>1.75</code> <code>verbose</code> <code>bool</code> <p>Log sampling rate to stdout. Default, False</p> <code>False</code> <code>filename</code> <code>str</code> <p>Filename to reference when logging or err. Default, \"file\"</p> <code>'file'</code> <p>Returns:</p> Name Type Description <code>estimated_rate</code> <code>float</code> <p>The estimated sampling rate.</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>If estimated rate is less than 0.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def estimate_sampling_rate(\n    timestamps, multiplier=1.75, verbose=False, filename=\"file\"\n):\n    \"\"\"Estimate the sampling rate given a list of timestamps.\n\n    Assumes that the most common temporal differences between timestamps\n    approximate the sampling rate. Note that this can fail for very high\n    sampling rates and irregular timestamps.\n\n    Parameters\n    ----------\n    timestamps : numpy.ndarray\n        1D numpy array of timestamp values.\n    multiplier : float or int, optional\n        Deft\n    verbose : bool, optional\n        Log sampling rate to stdout. Default, False\n    filename : str, optional\n        Filename to reference when logging or err. Default, \"file\"\n\n    Returns\n    -------\n    estimated_rate : float\n        The estimated sampling rate.\n\n    Raises\n    ------\n    ValueError\n        If estimated rate is less than 0.\n    \"\"\"\n\n    # approach:\n    # 1. use a box car smoother and a histogram to get the modal value\n    # 2. identify adjacent samples as those that have a\n    #    time difference &lt; the multiplier * the modal value\n    # 3. average the time differences between adjacent samples\n\n    sample_diff = np.diff(timestamps[~np.isnan(timestamps)])\n\n    if len(sample_diff) &lt; 10:\n        raise ValueError(\n            f\"Only {len(sample_diff)} timestamps are valid. Check the data.\"\n        )\n\n    smooth_diff = np.convolve(sample_diff, np.ones(10) / 10, mode=\"same\")\n\n    # we histogram with 100 bins out to 3 * mean, which should be fine for any\n    # reasonable number of samples\n    hist, bins = np.histogram(\n        smooth_diff, bins=100, range=[0, 3 * np.mean(smooth_diff)]\n    )\n    mode = bins[np.where(hist == np.max(hist))][0]\n\n    adjacent = sample_diff &lt; mode * multiplier\n\n    sampling_rate = np.round(1.0 / np.mean(sample_diff[adjacent]))\n\n    if sampling_rate &lt; 0:\n        raise ValueError(f\"Error calculating sampling rate. For {filename}\")\n    if verbose:\n        logger.info(\n            f\"Estimated sampling rate for {filename}: {sampling_rate} Hz\"\n        )\n\n    return sampling_rate\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_valid_intervals", "title": "<code>get_valid_intervals(timestamps, sampling_rate, gap_proportion=2.5, min_valid_len=0)</code>", "text": "<p>Finds the set of all valid intervals in a list of timestamps. Valid interval: (start time, stop time) during which there are no gaps (i.e. missing samples).</p> <p>Parameters:</p> Name Type Description Default <code>timestamps</code> <code>ndarray</code> <p>1D numpy array of timestamp values.</p> required <code>sampling_rate</code> <code>float</code> <p>Sampling rate of the data.</p> required <code>gap_proportion</code> <code>float</code> <p>Threshold for detecting a gap; i.e. if the difference (in samples) between consecutive timestamps exceeds gap_proportion, it is considered a gap. Must be &gt; 1. Default to 2.5</p> <code>2.5</code> <code>min_valid_len</code> <code>float</code> <p>Length of smallest valid interval. Default to 0. If greater than interval duration, log warning and use half the total time.</p> <code>0</code> <p>Returns:</p> Name Type Description <code>valid_times</code> <code>ndarray</code> <p>Array of start and stop times of shape (N, 2) for valid data.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_valid_intervals(\n    timestamps, sampling_rate, gap_proportion=2.5, min_valid_len=0\n):\n    \"\"\"Finds the set of all valid intervals in a list of timestamps.\n    Valid interval: (start time, stop time) during which there are\n    no gaps (i.e. missing samples).\n\n    Parameters\n    ----------\n    timestamps : numpy.ndarray\n        1D numpy array of timestamp values.\n    sampling_rate : float\n        Sampling rate of the data.\n    gap_proportion : float, optional\n        Threshold for detecting a gap; i.e. if the difference (in samples)\n        between consecutive timestamps exceeds gap_proportion, it is considered\n        a gap. Must be &gt; 1. Default to 2.5\n    min_valid_len : float, optional\n        Length of smallest valid interval. Default to 0. If greater\n        than interval duration, log warning and use half the total time.\n\n    Returns\n    -------\n    valid_times : np.ndarray\n        Array of start and stop times of shape (N, 2) for valid data.\n    \"\"\"\n\n    eps = 0.0000001\n\n    total_time = timestamps[-1] - timestamps[0]\n\n    if total_time &lt; min_valid_len:\n        half_total_time = total_time / 2\n        logger.warn(f\"Setting minimum valid interval to {half_total_time}\")\n        min_valid_len = half_total_time\n\n    # get rid of NaN elements\n    timestamps = timestamps[~np.isnan(timestamps)]\n    # find gaps\n    gap = np.diff(timestamps) &gt; 1.0 / sampling_rate * gap_proportion\n\n    # all true entries of gap represent gaps. Get the times bounding these intervals.\n    gapind = np.asarray(np.where(gap))\n    # The end of each valid interval are the indices of the gaps and the final value\n    valid_end = np.append(gapind, np.asarray(len(timestamps) - 1))\n\n    # the beginning of the gaps are the first element and gapind+1\n    valid_start = np.insert(gapind + 1, 0, 0)\n\n    valid_indices = np.vstack([valid_start, valid_end]).transpose()\n\n    valid_times = timestamps[valid_indices]\n    # adjust the times to deal with single valid samples\n    valid_times[:, 0] = valid_times[:, 0] - eps\n    valid_times[:, 1] = valid_times[:, 1] + eps\n\n    valid_intervals = (valid_times[:, 1] - valid_times[:, 0]) &gt; min_valid_len\n\n    return valid_times[valid_intervals, :]\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_electrode_indices", "title": "<code>get_electrode_indices(nwb_object, electrode_ids)</code>", "text": "<p>Return indices of the specified electrode_ids given an NWB file.</p> <p>Also accepts electrical series object. If an ElectricalSeries is given, then the indices returned are relative to the selected rows in ElectricalSeries.electrodes. For example, if electricalseries.electrodes = [5], and row index 5 of nwbfile.electrodes has ID 10, then calling get_electrode_indices(electricalseries, 10) will return 0, the index of the matching electrode in electricalseries.electrodes.</p> <p>Indices for electrode_ids that are not in the electrical series are returned as np.nan</p> <p>If an NWBFile is given, then the row indices with the matching IDs in the file's electrodes table are returned.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_object</code> <code>NWBFile or ElectricalSeries</code> <p>The NWB file object or NWB electrical series object.</p> required <code>electrode_ids</code> <code>ndarray or list</code> <p>Array or list of electrode IDs.</p> required <p>Returns:</p> Name Type Description <code>electrode_indices</code> <code>list</code> <p>Array of indices of the specified electrode IDs.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_electrode_indices(nwb_object, electrode_ids):\n    \"\"\"Return indices of the specified electrode_ids given an NWB file.\n\n    Also accepts electrical series object. If an ElectricalSeries is given,\n    then the indices returned are relative to the selected rows in\n    ElectricalSeries.electrodes. For example, if electricalseries.electrodes =\n    [5], and row index 5 of nwbfile.electrodes has ID 10, then calling\n    get_electrode_indices(electricalseries, 10) will return 0, the index of the\n    matching electrode in electricalseries.electrodes.\n\n    Indices for electrode_ids that are not in the electrical series are\n    returned as np.nan\n\n    If an NWBFile is given, then the row indices with the matching IDs in the\n    file's electrodes table are returned.\n\n    Parameters\n    ----------\n    nwb_object : pynwb.NWBFile or pynwb.ecephys.ElectricalSeries\n        The NWB file object or NWB electrical series object.\n    electrode_ids : np.ndarray or list\n        Array or list of electrode IDs.\n\n    Returns\n    -------\n    electrode_indices : list\n        Array of indices of the specified electrode IDs.\n    \"\"\"\n    if isinstance(nwb_object, pynwb.ecephys.ElectricalSeries):\n        # electrodes is a DynamicTableRegion which may contain a subset of the\n        # rows in NWBFile.electrodes match against only the subset of\n        # electrodes referenced by this ElectricalSeries\n        electrode_table_indices = nwb_object.electrodes.data[:]\n        selected_elect_ids = [\n            nwb_object.electrodes.table.id[x] for x in electrode_table_indices\n        ]\n    elif isinstance(nwb_object, pynwb.NWBFile):\n        # electrodes is a DynamicTable that contains all electrodes\n        selected_elect_ids = list(nwb_object.electrodes.id[:])\n    else:\n        raise ValueError(\n            \"nwb_object must be of type ElectricalSeries or NWBFile\"\n        )\n\n    # for each electrode_id, find its index in selected_elect_ids and return\n    # that if it's there and invalid_electrode_index if not.\n\n    return [\n        (\n            selected_elect_ids.index(elect_id)\n            if elect_id in selected_elect_ids\n            else invalid_electrode_index\n        )\n        for elect_id in electrode_ids\n    ]\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_all_spatial_series", "title": "<code>get_all_spatial_series(nwbf, verbose=False, incl_times=True)</code>", "text": "<p>Given an NWB, get the spatial series and return a dictionary by epoch.</p> <p>If incl_times is True, then the valid intervals are included in the output.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>verbose</code> <code>bool</code> <p>Flag representing whether to log the sampling rate.</p> <code>False</code> <code>incl_times</code> <code>bool</code> <p>Include valid times in the output. Default, True. Set to False for only spatial series object IDs.</p> <code>True</code> <p>Returns:</p> Name Type Description <code>pos_data_dict</code> <code>dict</code> <p>Dict mapping indices to a dict with keys 'valid_times' and 'raw_position_object_id'. Returns None if there is no position data in the file. The 'raw_position_object_id' is the object ID of the SpatialSeries object.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_all_spatial_series(nwbf, verbose=False, incl_times=True) -&gt; dict:\n    \"\"\"\n    Given an NWB, get the spatial series and return a dictionary by epoch.\n\n    If incl_times is True, then the valid intervals are included in the output.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    verbose : bool\n        Flag representing whether to log the sampling rate.\n    incl_times : bool\n        Include valid times in the output. Default, True. Set to False for only\n        spatial series object IDs.\n\n    Returns\n    -------\n    pos_data_dict : dict\n        Dict mapping indices to a dict with keys 'valid_times' and\n        'raw_position_object_id'. Returns None if there is no position data in\n        the file. The 'raw_position_object_id' is the object ID of the\n        SpatialSeries object.\n    \"\"\"\n    pos_interface = get_position_obj(nwbf)\n\n    if pos_interface is None:\n        return None\n\n    return _get_pos_dict(\n        position=pos_interface.spatial_series,\n        epoch_groups=_get_epoch_groups(pos_interface),\n        session_id=nwbf.session_id,\n        verbose=verbose,\n        incl_times=incl_times,\n    )\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_nwb_copy_filename", "title": "<code>get_nwb_copy_filename(nwb_file_name)</code>", "text": "<p>Get file name of copy of nwb file without the electrophys data</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_nwb_copy_filename(nwb_file_name):\n    \"\"\"Get file name of copy of nwb file without the electrophys data\"\"\"\n\n    filename, file_extension = os.path.splitext(nwb_file_name)\n\n    if filename.endswith(\"_\"):\n        logger.warning(f\"File may already be a copy: {nwb_file_name}\")\n\n    return f\"{filename}_{file_extension}\"\n</code></pre>"}, {"location": "misc/", "title": "Misc Docs", "text": "<p>This folder contains miscellaneous supporting files documentation.</p> <ul> <li>Database Management</li> <li>figurl Views</li> <li>insert Data</li> <li>Merge Tables</li> <li>Session Groups</li> </ul>"}, {"location": "misc/database_management/", "title": "Database Management", "text": "<p>While Spyglass can help you organize your data, there are a number of things you'll need to do to manage users, database backups, and file cleanup.</p> <p>Some these tasks should be set to run regularly. Cron jobs can help with automation.</p>"}, {"location": "misc/database_management/#mysql-version", "title": "MySQL Version", "text": "<p>The Frank Lab's database is running MySQL 8.0 with a number of custom configurations set by our system admin to reflect UCSF's IT security requirements.</p> <p>DataJoint's default docker container for MySQL is version 5.7. As the Spyglass team has hit select compatibility issues, we've worked with the DataJoint team to update the open source package to support MySQL 8.0.</p> <p>While the Spyglass team won't be able to support earlier versions, if you run into any issues declaring Spyglass tables with an 8.0 instance, please let us know.</p>"}, {"location": "misc/database_management/#user-management", "title": "User Management", "text": "<p>The DatabaseSettings class provides a number of methods to help you manage users. By default, it will write out a temporary <code>.sql</code> file and execute it on the database.</p>"}, {"location": "misc/database_management/#privileges", "title": "Privileges", "text": "<p>DataJoint schemas correspond to MySQL databases. Privileges are managed by schema/database prefix.</p> <ul> <li><code>SELECT</code> privileges allow users to read, write, and delete data.</li> <li><code>ALL</code> privileges allow users to create, alter, or drop tables and schemas in     addition to operations above.</li> </ul> <p>In practice, DataJoint only permits alterations of secondary keys on existing tables, and more derstructive operations would require using DataJoint to execeute MySQL commands.</p> <p>Shared schema prefixes are those defined in the Spyglass package (e.g., <code>common</code>, <code>lfp</code>, etc.). A 'user schema' is any schema with the username as prefix. User types differ in the privileges they are granted on these prifixes. Declaring a table with the SpyglassMixin on a schema other than a shared module or the user's own prefix will raise a warning.</p>"}, {"location": "misc/database_management/#users-roles", "title": "Users roles", "text": "<p>When a database is first initialized, the team should run <code>add_roles</code> to create the following roles:</p> <ul> <li><code>dj_guest</code>: <code>SELECT</code> on all schemas.</li> <li><code>dj_collab</code>: <code>ALL</code> on user schema, <code>SELECT</code> on all other schemas.</li> <li><code>dj_user</code>: <code>ALL</code> on shared and user schema, <code>SELECT</code> on all other schemas.</li> <li><code>dj_admin</code>: <code>ALL</code> on all schemas.</li> </ul> <p>If new shared modules are introduced, the <code>add_module</code> method should be used to expand the privileges of the <code>dj_user</code> role.</p>"}, {"location": "misc/database_management/#setting-passwords", "title": "Setting Passwords", "text": "<p>New users are generated with the password <code>temppass</code>. In order to change this, we recommend downloading DataJoint <code>0.14.2</code> (currently pre-release).</p> <pre><code>git clone https://github.com/datajoint/datajoint-python/\npip install ./datajoint-python\n</code></pre> <p>Then, you the user can reset within Python:</p> <pre><code>import datajoint as dj\n\ndj.set_password()\n</code></pre>"}, {"location": "misc/database_management/#database-backups", "title": "Database Backups", "text": "<p>The following codeblockes are a series of files used to back up our database and migrate the contents to another server. Some conventions to note:</p> <ul> <li><code>.host</code>: files used in the host's context</li> <li><code>.container</code>: files used inside the database Docker container</li> <li><code>.env</code>: files used to set environment variables used by the scripts for     database name, backup name, and backup credentials</li> </ul> <p>This backup process uses a dedicated backup user, that an admin would need to criate with the relevant permissions.</p>"}, {"location": "misc/database_management/#mysqlenvhost", "title": "mysql.env.host", "text": "MySQL host environment variables  Values may be adjusted as needed for different building images.  <pre><code>ROOT_PATH=/usr/local/containers/mysql # path to this container's working area\n\n# variables for building image\nSRC=ubuntu\nVER=20.04\nDOCKERFILE=Dockerfile.base\n\n# variables for referencing image\nIMAGE=mysql8\nTAG=u20\n# variables for running the container\nCNAME=mysql-datajoint\nMACADDR=4e:b0:3d:42:e0:70\nRPORT=3306\n\n# variables for initializing/relaunching the container\n# - where the mysql data and backups will live - these values\n# are examples\nDB_PATH=/data/db\nDB_DATA=mysql\nDB_BACKUP=/data/mysql-backups\n\n# backup info\nBACK_USER=mysql-backup\nBACK_PW={password}\nBACK_DBNAME={database}\n# mysql root password - make sure to remove this AFTER the container\n# is initialized - and this file will be replicated inside the container\n# on initialization, so remove it from there: /opt/bin/mysql.env\n</code></pre>"}, {"location": "misc/database_management/#backup-databaseshhost", "title": "backup-database.sh.host", "text": "<p>This script runs the mysql-backup container script (exec inside the container) that dumps the database contents for each database as well as the entire database. Use cron to set this to run on your desired schedule.</p> MySQL host docker exec <pre><code>#!/bin/bash\n\nPRIOR_DIR=$(pwd)\ncd /usr/local/containers/mysql || exit\n. mysql.env\ncd \"$(dirname ${ROOT_PATH})\"\n#\ndocker exec ${CNAME} /opt/bin/mysql-backup.csh\n#\ncd \"$(dirname ${DB_BACKUP})\"\n#\ncd ${PRIOR_DIR}\n</code></pre>"}, {"location": "misc/database_management/#mysql-backup-xfercshhost", "title": "mysql-backup-xfer.csh.host", "text": "<p>This script transfers the backup to another server 'X' and is specific for us as it uses passwordless ssh keys to a local unprivileged user on X that has the mysql backup area on X as that user's home.</p> MySQL host transfer script <pre><code>#!/bin/csh\nset td=`date +\"%Y%m%d\"`\ncd /data/mysql-backups\nscp -P {port} -i ~/mysql-backup -r ${database}-${td} mysql-backup@${X}:~/\n/bin/rm -r lmf-db-${td}\n</code></pre>"}, {"location": "misc/database_management/#myenvcshcontainer", "title": "myenv.csh.container", "text": "Docker container environment variables <pre><code>set db_backup=mysql-backups\nset back_user=mysql-backup\nset back_pw={password}\nset back_dbname={database}\n</code></pre>"}, {"location": "misc/database_management/#mysql-backupcshcontainer", "title": "mysql-backup.csh.container", "text": "Generate backups from within container <pre><code>#!/bin/csh\nsource /opt/bin/myenv.csh\nset td=`date +\"%Y%m%d\"`\ncd /${db_backup}\nmkdir ${back_dbname}-${td}\n\nset list=`echo \"show databases;\" | mysql --user=${back_user} --password=${back_pw}`\nset cnt=0\n\nforeach db ($list)\n  if ($cnt == 0) then\n    echo \"dumping mysql databases on $td\"\n  else\n    echo \"dumping MySQL database : $db\"\n    # Per-schema backups\n    mysqldump $db --max_allowed_packet=512M --user=${back_user} --password=${back_pw} &gt; /${db_backup}/${back_dbname}-${td}/mysql.${db}.sql\n  endif\n@ cnt = $cnt + 1\nend\n# Full database backup\nmysqldump --all-databases --max_allowed_packet=512M --user=${back_user} --password=${back_pw} &gt; /${db_backup}/${back_dbname}-${td}/mysql-all.sql\n</code></pre>"}, {"location": "misc/database_management/#file-cleanup", "title": "File Cleanup", "text": "<p>Spyglass is designed to hold metadata for analyses that reference NWB files on disk. There are several tables that retain lists of files that have been generated during analyses. If someone deletes analysis entries, files will still be on disk.</p> <p>To remove orphaned files, we run the following commands in our cron jobs:</p> <pre><code>from spyglass.common import AnalysisNwbfile\nfrom spyglass.spikesorting import SpikeSorting\nfrom spyglass.common.common_nwbfile import schema as nwbfile_schema\nfrom spyglass.decoding.v1.sorted_spikes import schema as spikes_schema\nfrom spyglass.decoding.v1.clusterless import schema as clusterless_schema\n\n\ndef main():\n    AnalysisNwbfile().nightly_cleanup()\n    SpikeSorting().nightly_cleanup()\n    nwbfile_schema.external['analysis'].delete(delete_external_files=True))\n    nwbfile_schema.external['raw'].delete(delete_external_files=True))\n    spikes_schema.external['analysis'].delete(delete_external_files=True))\n    clusterless_schema.external['analysis'].delete(delete_external_files=True))\n</code></pre> <p>The <code>delete</code> calls above use DataJoint's <code>ExternalTable.delete</code> method, which will remove files from disk that are no longer referenced in the database.</p>"}, {"location": "misc/figurl_views/", "title": "Creating figurl views", "text": ""}, {"location": "misc/figurl_views/#spike-sorting-recording-view", "title": "Spike sorting recording view", "text": "<pre><code>import spyglass.common as ndc\nimport spyglass.figurl_views as ndf\n\nquery = ...\n\n# To replace:\n# (ndf.SpikeSortingRecordingView &amp; query).delete()\n\nndf.SpikeSortingRecordingView.populate([(ndc.SpikeSortingRecording &amp; query).proj()])\n</code></pre>"}, {"location": "misc/figurl_views/#spike-sorting-view", "title": "Spike sorting view", "text": "<pre><code>import spyglass.common as ndc\nimport spyglass.figurl_views as ndf\n\nquery = ...\n\n# To replace:\n# (ndf.SpikeSortingView &amp; query).delete()\n\nndf.SpikeSortingView.populate([(ndc.SpikeSorting &amp; query).proj()])\n</code></pre>"}, {"location": "misc/insert_data/", "title": "How to insert data into <code>spyglass</code>", "text": "<p>In <code>spyglass</code>, every table corresponds to an object. An experimental session is defined as a collection of such objects. When an NWB file is ingested into <code>spyglass</code>, the information about these objects is first read and inserted into tables in the <code>common</code> module (e.g. <code>Institution</code>, <code>Lab</code>, <code>Electrode</code>, etc). However, not every NWB file has all the information required by <code>spyglass</code>. For example, many NWB files do not contain any information about the <code>DataAcquisitionDevice</code> or <code>Probe</code> because NWB does not yet have an official standard for specifying them. In addition, one might find that the information contained in the NWB file is incorrect and would like to modify it before inserting it into <code>spyglass</code> without having to go through the time-consuming process of re-generating the NWB file. For these cases, we provide an alternative approach to inserting data to <code>spyglass</code>.</p> <p>This alternate approach consists of two steps. First, the user must identify entries that they would like to add to the <code>spyglass</code> database that exist independently of any particular NWB file. For example, information about a particular probe is stored in the <code>ProbeType</code> and <code>Probe</code> tables of <code>spyglass.common</code>. The user can either:</p> <ol> <li> <p>create these entries programmatically using DataJoint <code>insert</code> commands, for     example:</p> <pre><code>sgc.ProbeType.insert1(\n    {\n        \"probe_type\": \"128c-4s6mm6cm-15um-26um-sl\",\n        \"probe_description\": \"A Livermore flexible probe with 128 channels, 4 shanks, 6 mm shank length, 6 cm ribbon length. 15 um contact diameter, 26 um center-to-center distance (pitch), single-line configuration.\",\n        \"manufacturer\": \"Lawrence Livermore National Lab\",\n        \"num_shanks\": 4,\n    },\n    skip_duplicates=True,\n)\n</code></pre> </li> <li> <p>define these entries in a special YAML file called <code>entries.yaml</code> that is     processed when <code>spyglass</code> is imported. One can think of <code>entries.yaml</code> as a     place to define information that the database should come pre-equipped     prior to ingesting any NWB files. The <code>entries.yaml</code> file should be placed     in the <code>spyglass</code> base directory. An example can be found in     <code>examples/config_yaml/entries.yaml</code>. It has the following structure:</p> <pre><code>TableName:\n    - TableEntry1Field1: Value\n\nTableEntry1Field2:\n    - TableEntry2Field1: Value\n\nTableEntry2Field2: Value\n</code></pre> <p>For example,</p> <pre><code>ProbeType:\n    - probe_type: 128c-4s6mm6cm-15um-26um-sl\nprobe_description: A Livermore flexible probe with 128 channels, 4 shanks, 6 mm shank\n    length, 6 cm ribbon length. 15 um contact diameter, 26 um center-to-center distance\n    (pitch), single-line configuration.\nmanufacturer: Lawrence Livermore National Lab\nnum_shanks: 4\n</code></pre> </li> </ol> <p>Using a YAML file over programmatically creating these entries in a notebook or script has the advantages that the YAML file maintains a record of what entries have been added that is easy to access, and the file is portable and can be shared alongside an NWB file or set of NWB files from a given experiment.</p> <p>Next, the user must associate the NWB file with entries defined in the database. This is done by cresqating a configuration file, which must: be in the same directory as the NWB file that it configures be in YAML format have the following naming convention: <code>&lt;name_of_nwb_file&gt;_spyglass_config.yaml</code>.</p> <p>Users can programmatically generate this configuration file. It is then read by spyglass when calling <code>insert_session</code> on the associated NWB file.</p> <p>An example of this can be found at <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code>. This file is associated with the NWB file <code>sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb</code>.</p> <p>This is the general format for the config entry:</p> <pre><code>TableName:\n  - primary_key1: value1\n</code></pre> <p>For example:</p> <pre><code>DataAcquisitionDevice:\n  - data_acquisition_device_name: Neuropixels Recording Device\n</code></pre> <p>In this example, the NWB file that corresponds to this config YAML will become associated with the DataAcquisitionDevice with primary key data_acquisition_device_name: Neuropixels Recording Device. This entry must exist.</p>"}, {"location": "misc/merge_tables/", "title": "Merge Tables", "text": ""}, {"location": "misc/merge_tables/#why", "title": "Why", "text": "<p>A pipeline may diverge when we want to process the same data in different ways. Merge Tables allow us to join divergent pipelines together, and unify downstream processing steps. For a more in depth discussion, please refer to this notebook and related discussions here and here.</p> <p>Note: Deleting entries upstream of Merge Tables will throw errors related to deleting a part entry before the master. To circumvent this, you can add <code>force_parts=True</code> to the <code>delete</code> function call, but this will leave and orphaned primary key in the master. Instead, use <code>(YourTable &amp; restriction).delete_downstream_merge()</code> to delete master/part pairs. If errors persist, identify and import the offending part table and rerun <code>delete_downstream_merge</code> with <code>reload_cache=True</code>. This process will be faster for subsequent calls if you reassign the your table after importing.</p> <pre><code>from spyglass.common import Nwbfile\n\nnwbfile = Nwbfile()\n(nwbfile &amp; \"nwb_file_name LIKE 'Name%'\").delete_downstream_merge()\n</code></pre>"}, {"location": "misc/merge_tables/#what", "title": "What", "text": "<p>A Merge Table is fundamentally a master table with one part for each divergent pipeline. By convention...</p> <ol> <li> <p>The master table has one primary key, <code>merge_id</code>, a     UUID, and     one secondary attribute, <code>source</code>, which gives the part table name. Both     are managed with the custom <code>insert</code> function of this class.</p> </li> <li> <p>Each part table has inherits the final table in its respective pipeline, and     shares the same name as this table.</p> </li> </ol> <pre><code>from spyglass.utils.dj_merge_tables import _Merge\n\n\n@schema\nclass MergeOutput(_Merge):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class One(dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; One\n        \"\"\"\n\n    class Two(dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; Two\n        \"\"\"\n</code></pre> <p></p> <p>By convention, Merge Tables have been named with the pipeline name plus <code>Output</code> (e.g., <code>LFPOutput</code>, <code>PositionOutput</code>). Using the underscore alias for this class allows us to circumvent a DataJoint protection that interprets the class as a table itself.</p>"}, {"location": "misc/merge_tables/#how", "title": "How", "text": ""}, {"location": "misc/merge_tables/#merging", "title": "Merging", "text": "<p>The Merge class in Spyglass's utils is a subclass of DataJoint's Manual Table and adds functions to make the awkwardness of part tables more manageable. These functions are described in the API section, under <code>utils.dj_merge_tables</code>.</p>"}, {"location": "misc/merge_tables/#restricting", "title": "Restricting", "text": "<p>In short: restrict Merge Tables with arguments, not the <code>&amp;</code> operator.</p> <ul> <li>Normally: <code>Table &amp; \"field='value'\"</code></li> <li>Instead: <code>MergeTable.merge_view(restriction=\"field='value'\"</code>).</li> </ul> <p>Caution. The <code>&amp;</code> operator may look like it's working when using <code>dict</code>, but this is because invalid keys will be ignored. <code>Master &amp; {'part_field':'value'}</code> is equivalent to <code>Master</code> alone (source).</p> <p>When provided as arguments, methods like <code>merge_get_part</code> and <code>merge_get_parent</code> will override the permissive treatment of mappings described above to only return relevant tables.</p>"}, {"location": "misc/merge_tables/#building-downstream", "title": "Building Downstream", "text": "<p>A downstream analysis will ideally be able to use all diverget pipelines interchangeably. If there are parameters that may be required for downstream processing, they should be included in the final table of the pipeline. In the example above, both <code>One</code> and <code>Two</code> might have a secondary key <code>params</code>. A downstream Computed table could do the following:</p> <pre><code>def make(self, key):\n    try:\n        params = MergeTable.merge_get_parent(restriction=key).fetch(\"params\")\n    except DataJointError:\n        params = default_params\n    processed_data = self.processing_func(key, params)\n</code></pre> <p>Note that the <code>try/except</code> above catches a possible error in the event <code>params</code> is not present in the parent.</p>"}, {"location": "misc/merge_tables/#example", "title": "Example", "text": "<p>For example usage, see our Merge Table notebook.</p>"}, {"location": "misc/session_groups/", "title": "Session groups", "text": "<p>A session group is a collection of sessions. Each group has a name (primary key) and a description.</p> <pre><code>from spyglass.common import SessionGroup\n\n# Create a new session group\nSessionGroup.add_group(\"test_group_1\", \"Description of test group 1\")\n\n# Get the table of session groups\nSessionGroup()\n\n# Add a session to the group\nSessionGroup.add_session_to_group(\"RN2_20191110_.nwb\", \"test_group_1\")\n\n# Remove a session from a group\n# SessionGroup.remove_session_from_group('RN2_20191110_.nwb', 'test_group_1')\n\n# Get all sessions in group\nSessionGroup.get_group_sessions(\"test_group_1\")\n\n# Update the description of a session group\nSessionGroup.update_session_group_description(\"test_group_1\", \"Test description\")\n</code></pre>"}, {"location": "notebooks/", "title": "Tutorial Notebooks", "text": "<p>There are several paths one can take to these notebooks. The notebooks have two-digits in their names, the first of which indicates it's 'batch', as described in the categories below.</p>"}, {"location": "notebooks/#0-intro", "title": "0. Intro", "text": "<p>Everyone should complete the Setup and Insert Data notebooks.</p> <p>Data Sync is an optional additional tool for collaborators that want to share analysis files.</p> <p>The Merge Tables notebook explains details on a new table tier unique to Spyglass that allows the user to use different versions of pipelines on the same data. This is important for understanding the later notebooks.</p>"}, {"location": "notebooks/#1-spike-sorting-pipeline", "title": "1. Spike Sorting Pipeline", "text": "<p>This series of notebooks covers the process of spike sorting, from automated spike sorting to optional manual curation of the output of the automated sorting.</p>"}, {"location": "notebooks/#2-position-pipeline", "title": "2. Position Pipeline", "text": "<p>This series of notebooks covers tracking the position(s) of the animal. The user can employ two different methods:</p> <ol> <li>the simple Trodes methods of tracking LEDs on the     animal's headstage</li> <li>DLC (DeepLabCut) which uses a neural network to track the     animal's body parts.</li> </ol> <p>Either case can be followed by the Linearization notebook if the user wants to linearize the position data for later use.</p>"}, {"location": "notebooks/#3-lfp-pipeline", "title": "3. LFP Pipeline", "text": "<p>This series of notebooks covers the process of LFP analysis. The LFP covers the extraction of the LFP in specific bands from the raw data. The Theta notebook shows specifically how to extract the theta band power and phase from the LFP data. Finally the Ripple Detection notebook shows how to detect ripples in the LFP data.</p>"}, {"location": "notebooks/#4-decoding-pipeline", "title": "4. Decoding Pipeline", "text": "<p>This series of notebooks covers the process of decoding the position of the animal from spiking data. It relies on the position data from the Position pipeline and the output of spike sorting from the Spike Sorting pipeline. Decoding can be from sorted or from unsorted data using spike waveform features (so-called clusterless decoding).</p> <p>The first notebook (Extracting Clusterless Waveform Features) in this series shows how to retrieve the spike waveform features used for clusterless decoding.</p> <p>The second notebook (Clusterless Decoding) shows a detailed example of how to decode the position of the animal from the spike waveform features. The third notebook (Decoding) shows how to decode the position of the animal from the sorted spikes.</p>"}, {"location": "notebooks/#developer-note", "title": "Developer note", "text": "<p>The <code>py_scripts</code> directory contains the same notebook data in <code>.py</code> form to facilitate GitHub PR reviews. To update them, run the following from the root Spyglass directory</p> <pre><code>pip install jupytext\njupytext --to py notebooks/*ipynb\nmv notebooks/*py notebooks/py_scripts\nblack .\n</code></pre> <p>Unfortunately, jupytext-generated py script are not black-compliant by default.</p>"}, {"location": "notebooks/00_Setup/", "title": "Setup", "text": "<p>Welcome to Spyglass, a DataJoint pipeline maintained by the Frank Lab at UCSF.</p> <p>Spyglass will help you take an NWB file from raw data to analysis-ready preprocessed formats using DataJoint to (a) connect to a relational database (here, MySQL), and (b) automate processing steps. To use Spyglass, you'll need to ...</p> <ol> <li>Set up your local environment</li> <li>Connect to a database</li> </ol> <p>JupyterHub users can skip this step. Frank Lab members should first follow 'rec to nwb overview' steps on Google Drive to set up an ssh connection.</p> <p>For local use, download and install ...</p> <ol> <li>Python 3.9.</li> <li>mamba as a replacement for conda. Spyglass installation is significantly faster with mamba.<pre>wget \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\nbash Miniforge3-$(uname)-$(uname -m).sh\n</pre> </li> <li>VS Code with relevant python extensions, including Jupyter. Hold off on selecting your interpreter until after you make the environment with <code>mamba</code>.</li> <li>git for downloading the repository, including notebooks.</li> </ol> <p>In a terminal, ...</p> <ol> <li>navigate to your project directory.</li> <li>use <code>git</code> to download the Spyglass repository.</li> <li>navigate to the newly downloaded directory.</li> <li>create a <code>mamba</code> environment with either the standard <code>environment.yml</code> or the <code>environment_position.yml</code>, if you intend to use the full position pipeline. The latter will take longer to install.</li> <li>open this notebook with VSCode</li> </ol> <p>Commands for the steps above ...</p> <pre>cd /your/project/directory/ # 1\ngit clone https://github.com/LorenFrankLab/spyglass/ # 2\ncd spyglass # 3\nmamba env create -f environment.yml # 4\ncode notebooks/00_Setup.ipynb # 5\n</pre> <p>Note: Spyglass is also installable via pip and pypi with <code>pip install spyglass-neuro</code>, but downloading from GitHub will also download other files.</p> <p>Next, within VSCode, select the kernel that matches your spyglass environment created with <code>mamba</code>. To use other Python interfaces, be sure to activate the environment: <code>conda activate spyglass</code></p> <p>See this guide for additional details on each of these programs and the role they play in using the pipeline.</p> <p>You have a few options for databases.</p> <ol> <li>Connect to an existing database.</li> <li>Run your own database with Docker</li> <li>JupyterHub (coming soon...)</li> </ol> <p>Your choice above should result in a set of credentials, including host name, host port, user name, and password. Note these for the next step.</p> Note for MySQL 8 users, including Frank Lab members <p>Using a MySQL 8 server, like the server hosted by the Frank Lab, will require the pre-release version of DataJoint to change one's password.</p> <pre>cd /location/for/datajoint/source/files/\ngit clone https://github.com/datajoint/datajoint-python\npip install ./datajoint-python\n</pre> <p>Connecting to an existing database will require a user name and password. Please contact your database administrator for this information.</p> <p>Frank Lab members should contact Chris.</p> <ul> <li><p>First, install Docker.</p> </li> <li><p>Add yourself to the <code>docker</code> group so that you don't have to be sudo to run docker.</p> </li> <li><p>Download the docker image for <code>datajoint/mysql:8.0</code>.</p> <pre>docker pull datajoint/mysql:8.0\n</pre> </li> <li><p>When run, this is referred to as a 'Docker container'</p> </li> <li><p>Next start the container with a couple additional pieces of info...</p> <ul> <li>Root password. We use <code>tutorial</code>.</li> <li>Database name. Here, we use <code>spyglass-db</code>.</li> <li>Port mapping. Here, we map 3306 across the local machine and container.</li> </ul> <pre>docker run --name spyglass-db -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql:8.0\n</pre> </li> <li><p>For data to persist after terminating the container, attach a volume when running:</p> <pre>docker volume create dj-vol\ndocker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n</pre> </li> </ul> <p>Docker credentials are as follows:</p> <ul> <li>Host: localhost</li> <li>Password: tutorial</li> <li>User: root</li> <li>Port: 3306</li> </ul> <p>Spyglass can load settings from either a DataJoint config file (recommended) or environmental variables. The code below will generate a config file, but we first need to decide a 'base path'. This is generally the parent directory where the data will be stored, with subdirectories for <code>raw</code>, <code>analysis</code>, and other data folders. If they don't exist already, they will be created.</p> <p>The function below will create a config file (<code>~/.datajoint.config</code> if global, <code>./dj_local_conf.json</code> if local). Local is recommended for the notebooks, as each will start by loading this file. Custom json configs can be saved elsewhere, but will need to be loaded in startup with <code>dj.config.load('your-path')</code>.</p> <p>To point spyglass to a folder elsewhere (e.g., an external drive for waveform data), simply edit the json file. Note that the <code>raw</code> and <code>analysis</code> paths appear under both <code>stores</code> and <code>custom</code>.</p> In\u00a0[\u00a0]: Copied! <pre>import os\nfrom spyglass.settings import SpyglassConfig\n\n# change to the root directory of the project\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\n\nSpyglassConfig().save_dj_config(\n    save_method=\"local\",  # global or local\n    base_dir=\"/path/like/stelmo/nwb/\",\n    database_user=\"your username\",\n    database_password=\"your password\",  # remove this line for shared machines\n    database_host=\"localhost or lmf-db.cin.ucsf.edu\",\n    database_port=3306,\n    set_password=False,\n)\n</pre> import os from spyglass.settings import SpyglassConfig  # change to the root directory of the project if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\")  SpyglassConfig().save_dj_config(     save_method=\"local\",  # global or local     base_dir=\"/path/like/stelmo/nwb/\",     database_user=\"your username\",     database_password=\"your password\",  # remove this line for shared machines     database_host=\"localhost or lmf-db.cin.ucsf.edu\",     database_port=3306,     set_password=False, ) <p>If you used either a local or global save method, we can check the connection to the database with ...</p> In\u00a0[\u00a0]: Copied! <pre>import datajoint as dj\n\ndj.conn()  # test connection\ndj.config  # check config\n\nfrom spyglass.common import Nwbfile\n\nNwbfile()\n</pre> import datajoint as dj  dj.conn()  # test connection dj.config  # check config  from spyglass.common import Nwbfile  Nwbfile() <p>If you see an error saying <code>Could not find SPYGLASS_BASE_DIR</code>, try loading your config before importing Spyglass, try setting this as an environmental variable before importing Spyglass.</p> <pre>os.environ['SPYGLASS_BASE_DIR'] = '/your/base/path'\n\nimport spyglass\nfrom spyglass.settings import SpyglassConfig\nimport datajoint as dj\nprint(SpyglassConfig().config)\ndj.config.save_local() # or global\n</pre> <p>Next, we'll try inserting data</p>"}, {"location": "notebooks/00_Setup/#setup", "title": "Setup\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#local-environment", "title": "Local environment\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#database", "title": "Database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#existing-database", "title": "Existing Database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#running-your-own-database-with-docker", "title": "Running your own database with Docker\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#config-and-connecting-to-the-database", "title": "Config and Connecting to the database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/", "title": "Insert Data", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> </ul> <p>Let's start by importing the <code>spyglass</code> package, along with a few others.</p> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\nwarnings.simplefilter(\"ignore\", category=UserWarning)\n\n# spyglass.common has the most frequently used tables\nimport spyglass.common as sgc\n\n# spyglass.data_import has tools for inserting NWB files into the database\nimport spyglass.data_import as sgi\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) warnings.simplefilter(\"ignore\", category=UserWarning)  # spyglass.common has the most frequently used tables import spyglass.common as sgc  # spyglass.data_import has tools for inserting NWB files into the database import spyglass.data_import as sgi <pre>[2024-01-29 16:24:30,933][INFO]: Connecting root@localhost:3309\n[2024-01-29 16:24:30,942][INFO]: Connected root@localhost:3309\n</pre> <p>Datajoint enables users to use Python to build and interact with a Relational Database. In a Relational Data Model, each table is an object that can reference information in other tables to avoid redundancy.</p> <p>DataJoint has built-in tools for generating/saving a Diagram of the relationships between tables. This page describes the notation used.</p> <p>Polygons are tables, colors reference table type:</p> <ul> <li>Green rectangle: tables whose entries are entered manually.</li> <li>Blue oval: tables whose entries are imported from external files (e.g. NWB file).</li> <li>Red circle: tables whose entries are computed from entries of other tables.</li> <li>No shape (only text): tables whose entries are part of the table upstream</li> </ul> <p>Lines are dependencies between tables. An upstream table is connected to a downstream table via inheritance of the primary key. This is the set of attributes (i.e., column names) used to uniquely define an entry (i.e., a row)</p> <ul> <li>Bold lines: the upstream primary key is the sole downstream primary key</li> <li>Solid lines: the upstream table as part of the downstream primary key</li> <li>Dashed lines: the primary key of upstream table as non-primary key</li> </ul> In\u00a0[2]: Copied! <pre># Draw tables that are two levels below and one level above Session\ndj.Diagram(sgc.Session) - 1 + 2\n</pre> # Draw tables that are two levels below and one level above Session dj.Diagram(sgc.Session) - 1 + 2 Out[2]: <p>By adding diagrams together, of adding and subtracting levels, we can visualize key parts of Spyglass.</p> <p>Note: Notice the Selection tables. This is a design pattern that selects a subset of upstream items for further processing. In some cases, these also pair the selected data with processing parameters.</p> <p>After exploring the pipeline's structure, we'll now grab some example data. Spyglass will assume that the data is a neural recording with relevant auxiliary in NWB.</p> <p>We offer a few examples:</p> <ul> <li><code>minirec20230622.nwb</code>, .3 GB: minimal recording, Link</li> <li><code>mediumnwb20230802.nwb</code>, 32 GB: full-featured dataset, Link</li> <li><code>montague20200802.nwb</code>, 8 GB: full experimental recording, Link</li> <li>For those in the UCSF network, these and many others on <code>/stelmo/nwb/raw</code></li> </ul> <p>If you are connected to the Frank lab database, please rename any downloaded files (e.g., <code>example20200101_yourname.nwb</code>) to avoid naming collisions, as the file name acts as the primary key across key tables.</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\n# Define the name of the file that you copied and renamed\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  # Define the name of the file that you copied and renamed nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) <p>Spyglass will create a copy with this name.</p> In\u00a0[4]: Copied! <pre>nwb_copy_file_name\n</pre> nwb_copy_file_name Out[4]: <pre>'minirec20230622_.nwb'</pre> <p>Let's start small by inserting personnel information.</p> <p>The <code>LabMember</code> table lists all lab members, with an additional part table for <code>LabMemberInfo</code>. This holds Google account and DataJoint username info for each member, for authentication purposes.</p> <p>We can insert lab member information using the NWB file <code>experimenter</code> field as follows...</p> In\u00a0[5]: Copied! <pre># take a look at the lab members\nsgc.LabMember.insert_from_nwbfile(nwb_file_name)\n</pre> # take a look at the lab members sgc.LabMember.insert_from_nwbfile(nwb_file_name) <pre>Please add the Google user ID for Firstname Lastname in the LabMember.LabMemberInfo table to help manage permissions.\nPlease add the Google user ID for Firstname2 Lastname2 in the LabMember.LabMemberInfo table to help manage permissions.\n</pre> <p>We can insert into <code>LabMemberInfo</code> directly with a list of lists that reflect the order of the fields present in the table. See this notebook for examples of inserting with <code>dicts</code>.</p> In\u00a0[6]: Copied! <pre>sgc.LabMember.LabMemberInfo.insert(\n    [  # Full name, Google email address, DataJoint username, admin\n        [\"Firstname Lastname\", \"example1@gmail.com\", \"example1\", 0],\n        [\"Firstname2 Lastname2\", \"example2@gmail.com\", \"example2\", 0],\n    ],\n    skip_duplicates=True,\n)\nsgc.LabMember.LabMemberInfo()\n</pre> sgc.LabMember.LabMemberInfo.insert(     [  # Full name, Google email address, DataJoint username, admin         [\"Firstname Lastname\", \"example1@gmail.com\", \"example1\", 0],         [\"Firstname2 Lastname2\", \"example2@gmail.com\", \"example2\", 0],     ],     skip_duplicates=True, ) sgc.LabMember.LabMemberInfo() Out[6]: Information about lab member in the context of Frank lab network <p>lab_member_name</p> <p>google_user_name</p> used for permission to curate <p>datajoint_user_name</p> used for permission to delete entries Firstname Lastname example1@gmail.com example1Firstname2 Lastname2 example2@gmail.com example2 <p>Total: 2</p> <p>A <code>LabTeam</code> is a set of lab members who own a set of NWB files and the associated information in the database. This is often a subgroup that collaborates on the same projects. Data is associated with a given team, granting members analysis (e.g., curation) and deletion (coming soon) privileges.</p> In\u00a0[7]: Copied! <pre>team_name = \"My Team\"\nsgc.LabTeam().create_new_team(\n    team_name=team_name,  # Should be unique\n    team_members=[\"Firstname Lastname\", \"Firstname2 Lastname2\"],\n    team_description=\"test\",  # Optional\n)\n</pre> team_name = \"My Team\" sgc.LabTeam().create_new_team(     team_name=team_name,  # Should be unique     team_members=[\"Firstname Lastname\", \"Firstname2 Lastname2\"],     team_description=\"test\",  # Optional ) <p>By default, each member is part of their own team. We can see all teams and members by looking at the part table <code>LabTeam.LabTeamMember</code>.</p> In\u00a0[8]: Copied! <pre>sgc.LabTeam.LabTeamMember()\n</pre> sgc.LabTeam.LabTeamMember() Out[8]: <p>team_name</p> <p>lab_member_name</p> Firstname Lastname Firstname LastnameMy Team Firstname LastnameFirstname2 Lastname2 Firstname2 Lastname2My Team Firstname2 Lastname2 <p>Total: 4</p> <p><code>spyglass.data_import.insert_sessions</code> helps take the many fields of data present in an NWB file and insert them into various tables across Spyglass. If the NWB file is properly composed, this includes...</p> <ul> <li>the experimenter (replicating part of the process above)</li> <li>animal behavior (e.g. video recording of position)</li> <li>neural activity (extracellular recording of multiple brain areas)</li> <li>etc.</li> </ul> <p>Note: this may take time as Spyglass creates the copy. You may see a prompt about inserting device information.</p> In\u00a0[3]: Copied! <pre>sgi.insert_sessions(nwb_file_name)\n</pre> sgi.insert_sessions(nwb_file_name) <pre>Creating a copy of NWB file minirec20230622.nwb with link to raw ephys data: minirec20230622_.nwb\nPopulate Session...\nNo config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\nInstitution...\nLab...\nLabMember...\nPlease add the Google user ID for Firstname2 Lastname2 in the LabMember.LabMemberInfo table to help manage permissions.\nSubject...\nPopulate CameraDevice...\nInserted camera devices ['test camera 1']\n\nPopulate Probe...\nProbe ID '128c-4s6mm6cm-15um-26um-sl' already exists in the database. Spyglass will use that and not create a new Probe, Shanks, or Electrodes.\nInserted probes {'128c-4s6mm6cm-15um-26um-sl'}\n\nSkipping Apparatus for now...\nIntervalList...\nLabMember with name lastname, firstname does not exist. Cannot link Session with LabMember in Session.Experimenter.\nLabMember with name lastname2, firstname2 does not exist. Cannot link Session with LabMember in Session.Experimenter.\nPopulate ElectrodeGroup...\nPopulate Electrode...\nNo config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\nPopulate Raw...\nEstimating sampling rate...\nEstimated sampling rate for file: 30000.0 Hz\nImporting raw data: Sampling rate:\t30000.0 Hz\nNumber of valid intervals:\t2\nPopulate SampleCount...\nPopulate DIOEvents...\nPopulate TaskEpochs\nPopulate StateScriptFile\nPopulate VideoFile\nNo video found corresponding to file minirec20230622_.nwb, epoch 01_s1\nNo video found corresponding to file minirec20230622_.nwb, epoch 02_s2\nRawPosition...\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.1912336349487305\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.1912336349487305\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.339195609092712\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.339195609092712\nPopulated PosIntervalMap for minirec20230622_.nwb, 01_s1\nPopulated PosIntervalMap for minirec20230622_.nwb, 02_s2\n</pre> <p>To look at data, we can query a table with <code>Table()</code> syntax.</p> In\u00a0[11]: Copied! <pre>sgc.Lab()\n</pre> sgc.Lab() Out[11]: <p>lab_name</p> Loren Frank Lab <p>Total: 1</p> <p>The <code>Session</code> table has considerably more fields</p> In\u00a0[12]: Copied! <pre>sgc.Session.heading.names\n</pre> sgc.Session.heading.names Out[12]: <pre>['nwb_file_name',\n 'subject_id',\n 'institution_name',\n 'lab_name',\n 'session_id',\n 'session_description',\n 'session_start_time',\n 'timestamps_reference_time',\n 'experiment_description']</pre> <p>But a short primary key</p> In\u00a0[13]: Copied! <pre>sgc.Session.heading.primary_key\n</pre> sgc.Session.heading.primary_key Out[13]: <pre>['nwb_file_name']</pre> <p>The primary key is shown in bold in the html</p> In\u00a0[14]: Copied! <pre>sgc.Session()\n</pre> sgc.Session() Out[14]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p>Text only interfaces designate the primary key fields with <code>*</code></p> In\u00a0[15]: Copied! <pre>print(sgc.Session())\n</pre> print(sgc.Session()) <pre>*nwb_file_name subject_id     institution_na lab_name       session_id     session_descri session_start_ timestamps_ref experiment_des\n+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\nminirec2023062 54321          UCSF           Loren Frank La 12345          test yaml inse 2023-06-22 15: 1970-01-01 00: Test Conversio\n (Total: 1)\n\n</pre> <p>To see a the table definition, including data types, use <code>describe</code>.</p> <ul> <li><code>---</code> separates the primary key</li> <li><code>:</code> are used to separate field name from data type</li> <li><code>#</code> can be used to add comments to a field</li> </ul> In\u00a0[16]: Copied! <pre>from pprint import pprint  # adds line breaks\n\npprint(sgc.Session.describe())\n</pre> from pprint import pprint  # adds line breaks  pprint(sgc.Session.describe()) <pre>('# Table for holding experimental sessions.\\n'\n '-&gt; sgc.Nwbfile\\n'\n '---\\n'\n '-&gt; [nullable] sgc.Subject\\n'\n '-&gt; [nullable] sgc.Institution\\n'\n '-&gt; [nullable] sgc.Lab\\n'\n 'session_id=null      : varchar(200)                 \\n'\n 'session_description  : varchar(2000)                \\n'\n 'session_start_time   : datetime                     \\n'\n 'timestamps_reference_time : datetime                     \\n'\n 'experiment_description=null : varchar(2000)                \\n')\n</pre> <p>To look at specific entries in a table, we can use the <code>&amp;</code> operator. Below, we restrict based on a <code>dict</code>, but you can also use a string.</p> In\u00a0[17]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[17]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p><code>Raw</code> is connected to <code>Session</code> with a bold line, so it has the same primary key.</p> In\u00a0[18]: Copied! <pre>dj.Diagram(sgc.Session) + dj.Diagram(sgc.Raw)\n</pre> dj.Diagram(sgc.Session) + dj.Diagram(sgc.Raw) Out[18]: In\u00a0[19]: Copied! <pre>sgc.Raw &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Raw &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[19]: Raw voltage timeseries data, ElectricalSeries in NWB. <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>raw_object_id</p> the NWB object ID for loading this object from the file <p>sampling_rate</p> Sampling rate calculated from data, in Hz <p>comments</p> <p>description</p> minirec20230622_.nwb raw data valid times 4e756642-9203-4f00-b9d0-0e9747c14493 30000.0 No comment Recording of extracellular voltage <p>Total: 1</p> <p><code>IntervalList</code> is connected to <code>Session</code> with a solid line because it has additional primary key attributes. Here, you need to know both <code>nwb_file_name</code> and <code>interval_list_name</code> to uniquely identify an entry.</p> In\u00a0[20]: Copied! <pre># join/split condenses long spaces before field comments\npprint(\"\".join(sgc.IntervalList.describe().split(\"  \")))\n</pre> # join/split condenses long spaces before field comments pprint(\"\".join(sgc.IntervalList.describe().split(\"  \"))) <pre>('# Time intervals used for analysis\\n'\n '-&gt; sgc.Session\\n'\n 'interval_list_name : varchar(200) # descriptive name of this interval list\\n'\n '---\\n'\n 'valid_times: longblob # numpy array with start and end times for each '\n 'interval\\n')\n</pre> In\u00a0[21]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[21]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start and end times for each interval minirec20230622_.nwb 01_s1 =BLOB=minirec20230622_.nwb 02_s2 =BLOB=minirec20230622_.nwb pos 0 valid times =BLOB=minirec20230622_.nwb pos 1 valid times =BLOB=minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 5</p> <p>Raw data types like <code>valid_times</code> are shown as <code>=BLOB=</code>. We can inspect these with <code>fetch</code></p> <p>Note: like <code>insert</code>/<code>insert1</code>, <code>fetch</code> can be uses as <code>fetch1</code> to raise an error when many (or no) entries are retrieved. To limit to one entry when there may be many, use <code>query.fetch(limit=1)[0]</code></p> In\u00a0[22]: Copied! <pre>(\n    sgc.IntervalList\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": \"pos 1 valid times\",\n    }\n).fetch1(\"valid_times\")\n</pre> (     sgc.IntervalList     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": \"pos 1 valid times\",     } ).fetch1(\"valid_times\") Out[22]: <pre>array([[1.68747483e+09, 1.68747484e+09]])</pre> <p>In DataJoint operators, <code>&amp;</code> selects by a condition and <code>-</code> removes a condition.</p> In\u00a0[23]: Copied! <pre>(\n    (\n        (sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name})\n        - {\"interval_list_name\": \"pos 1 valid times\"}\n    )\n    - {\"interval_list_name\": \"pos 2 valid times\"}\n).fetch(\"interval_list_name\")\n</pre> (     (         (sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name})         - {\"interval_list_name\": \"pos 1 valid times\"}     )     - {\"interval_list_name\": \"pos 2 valid times\"} ).fetch(\"interval_list_name\") Out[23]: <pre>array(['01_s1', '02_s2', 'pos 0 valid times', 'raw data valid times'],\n      dtype=object)</pre> <p><code>IntervalList</code> has an additional secondary key <code>pipeline</code> which can describe the origin of the data. Because it is a secondary key, it is not required to uniquely identify an entry. Current values for this key from spyglass pipelines are:</p> pipeline Source position sg.common.PositionSource lfp_v0 sg.common.LFP lfp_v1 sg.lfp.v1.LFPV1 lfp_band sg.common.LFPBand, sg.lfp.analysis.v1.LFPBandV1 lfp_artifact sg.lfp.v1.LFPArtifactDetection spikesorting_artifact_v0 sg.spikesorting.ArtifactDetection spikesorting_artifact_v1 sg.spikesorting.v1.ArtifactDetection spikesorting_recording_v0 sg.spikesorting.SpikeSortingRecording spikesorting_recording_v1 sg.spikesorting.v1.SpikeSortingRecording <p>Another neat feature of DataJoint is that it automatically maintains data integrity with cascading deletes. For example, if we delete our <code>Session</code> entry, all associated downstream entries are also deleted (e.g. <code>Raw</code>, <code>IntervalList</code>).</p> <p>Note: The deletion process can be complicated by Merge Tables when the entry is referenced by a part table. To demo deletion in these cases, run the hidden code below.</p> Quick Merge Insert <pre>import spyglass.lfp as lfp\n\nsgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"lfp_electrode_group_name\": \"test\",\n    \"target_interval_list_name\": \"01_s1\",\n    \"filter_name\": \"LFP 0-400 Hz\",\n    \"filter_sampling_rate\": 30_000,\n}\nlfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True)\nlfp.v1.LFPV1().populate(lfp_key)\n</pre> Deleting Merge Entries <pre>nwbfile = sgc.Nwbfile()\n\n(nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete_downstream_merge(\n    dry_run=False, # True will show Merge Table entries that would be deleted\n)\n</pre> <p>Please see the next notebook for a more detailed explanation.</p> In\u00a0[24]: Copied! <pre>session_entry = sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\nsession_entry\n</pre> session_entry = sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} session_entry Out[24]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p><code>Session.Experimenter</code> is used for permissions checks when deleting. The session will need to have an experimenter in order avoid an error being raised during this check.</p> In\u00a0[\u00a0]: Copied! <pre>sess_key = (sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(\n    \"KEY\", as_dict=True\n)[0]\nexp_key = (sgc.LabMember).fetch(\"KEY\", as_dict=True)[0]\nsgc.Session.Experimenter.insert1(\n    dict(**sess_key, **exp_key), skip_duplicates=True\n)\n</pre> sess_key = (sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(     \"KEY\", as_dict=True )[0] exp_key = (sgc.LabMember).fetch(\"KEY\", as_dict=True)[0] sgc.Session.Experimenter.insert1(     dict(**sess_key, **exp_key), skip_duplicates=True ) <p>Even with the experimenter specified, there are still delete protections in place. To see an example, uncomment the cell below.</p> In\u00a0[\u00a0]: Copied! <pre># session_entry.delete()\n</pre> # session_entry.delete() <p>To delete, you'll need to share a team with the session experimenter.</p> In\u00a0[\u00a0]: Copied! <pre>your_name = \"YourFirst YourLast\"\nparts = your_name.split(\" \")\nsgc.LabMember.insert1([your_name, parts[0], parts[1]])\nsgc.LabMember.LabMemberInfo.insert1(\n    [your_name, \"your_gmail\", dj.config[\"database.user\"], 0]\n)\nsgc.LabTeam.LabTeamMember.insert1([team_name, your_name])\n</pre> your_name = \"YourFirst YourLast\" parts = your_name.split(\" \") sgc.LabMember.insert1([your_name, parts[0], parts[1]]) sgc.LabMember.LabMemberInfo.insert1(     [your_name, \"your_gmail\", dj.config[\"database.user\"], 0] ) sgc.LabTeam.LabTeamMember.insert1([team_name, your_name]) <p>By default, DataJoint is cautious about deletes and will prompt before deleting. To delete, uncomment the cell below and respond <code>yes</code> in the prompt.</p> In\u00a0[26]: Copied! <pre># session_entry.delete()\n</pre> # session_entry.delete() <pre>[2023-09-28 08:29:15,814][INFO]: Deleting 4 rows from `common_behav`.`_raw_position__pos_object`\nINFO:datajoint:Deleting 4 rows from `common_behav`.`_raw_position__pos_object`\n[2023-09-28 08:29:15,822][INFO]: Deleting 2 rows from `common_behav`.`_raw_position`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`_raw_position`\n[2023-09-28 08:29:15,834][INFO]: Deleting 4 rows from `common_behav`.`position_source__spatial_series`\nINFO:datajoint:Deleting 4 rows from `common_behav`.`position_source__spatial_series`\n[2023-09-28 08:29:15,841][INFO]: Deleting 2 rows from `common_behav`.`position_source`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`position_source`\n[2023-09-28 08:29:15,851][INFO]: Deleting 7 rows from `common_dio`.`_d_i_o_events`\nINFO:datajoint:Deleting 7 rows from `common_dio`.`_d_i_o_events`\n[2023-09-28 08:29:15,871][INFO]: Deleting 128 rows from `common_ephys`.`_electrode`\nINFO:datajoint:Deleting 128 rows from `common_ephys`.`_electrode`\n[2023-09-28 08:29:15,879][INFO]: Deleting 1 rows from `common_ephys`.`_electrode_group`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_electrode_group`\n[2023-09-28 08:29:15,887][INFO]: Deleting 1 rows from `common_ephys`.`_raw`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_raw`\n[2023-09-28 08:29:15,896][INFO]: Deleting 1 rows from `common_ephys`.`_sample_count`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_sample_count`\n[2023-09-28 08:29:15,908][INFO]: Deleting 2 rows from `common_behav`.`__position_interval_map`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`__position_interval_map`\n[2023-09-28 08:29:15,918][INFO]: Deleting 2 rows from `common_task`.`_task_epoch`\nINFO:datajoint:Deleting 2 rows from `common_task`.`_task_epoch`\n[2023-09-28 08:29:15,924][INFO]: Deleting 5 rows from `common_interval`.`interval_list`\nINFO:datajoint:Deleting 5 rows from `common_interval`.`interval_list`\n[2023-09-28 08:29:15,931][INFO]: Deleting 1 rows from `common_session`.`_session`\nINFO:datajoint:Deleting 1 rows from `common_session`.`_session`\n[2023-09-28 08:29:18,765][INFO]: Deletes committed.\nINFO:datajoint:Deletes committed.\n</pre> Out[26]: <pre>1</pre> <p>We can check that delete worked, both for <code>Session</code> and <code>IntervalList</code></p> In\u00a0[27]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[27]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> <p>Total: 0</p> In\u00a0[28]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[28]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start and end times for each interval <p>Total: 0</p> <p><code>delete</code> is useful for re-running something. Editing entries is possible, but discouraged because it can lead to integrity issues. Instead, re-enter and let the automation handle the rest.</p> <p>Spyglass falls short, however, in that deleting from <code>Session</code> doesn't also delete the associated entry in <code>Nwbfile</code>, which has to be removed separately (for now). This table offers a <code>cleanup</code> method to remove the added files (with the <code>delete_files</code> argument as <code>True</code>).</p> <p>Note: this also applies to deleting files from <code>AnalysisNwbfile</code> table.</p> In\u00a0[53]: Copied! <pre># Uncomment to delete\n# (sgc.Nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete()\n</pre> # Uncomment to delete # (sgc.Nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete() <pre>[2023-07-18 19:01:15,343][INFO]: Deleting 1 rows from `common_nwbfile`.`nwbfile`\nINFO:datajoint:Deleting 1 rows from `common_nwbfile`.`nwbfile`\n[2023-07-18 19:01:17,130][INFO]: Deletes committed.\nINFO:datajoint:Deletes committed.\n</pre> Out[53]: <pre>1</pre> <p>Note that the file (ends with <code>_.nwb</code>) has not been deleted, even if the entry was deleted above.</p> In\u00a0[\u00a0]: Copied! <pre>!ls $SPYGLASS_BASE_DIR/raw\n</pre> !ls $SPYGLASS_BASE_DIR/raw <pre>minirec20230622.nwb\nminirec20230622_.nwb\nmontague20200802_tutorial.nwb\nmontague20200802_tutorial_.nwb\nmontague20200802_tutorial__.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim_.nwb\ntonks20211103_.nwb\n</pre> <p>We can clean these files with the <code>cleanup</code> method</p> In\u00a0[55]: Copied! <pre>sgc.Nwbfile().cleanup(delete_files=True)\n</pre> sgc.Nwbfile().cleanup(delete_files=True) <pre>100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1/1 [00:00&lt;00:00, 304.24it/s]\n</pre> In\u00a0[56]: Copied! <pre>!ls $SPYGLASS_BASE_DIR/raw\n</pre> !ls $SPYGLASS_BASE_DIR/raw <pre>minirec20230622.nwb\nmontague20200802_tutorial.nwb\nmontague20200802_tutorial_.nwb\nmontague20200802_tutorial__.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim_.nwb\ntonks20211103_.nwb\n</pre> <p>In the next notebook, we'll explore tools for syncing.</p>"}, {"location": "notebooks/01_Insert_Data/#insert-data", "title": "Insert Data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#visualizing-the-database", "title": "Visualizing the database\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#example-data", "title": "Example data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#basic-inserts-lab-team", "title": "Basic Inserts: Lab Team\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#inserting-from-nwb", "title": "Inserting from NWB\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#inspecting-the-data", "title": "Inspecting the data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#deleting-data", "title": "Deleting data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/", "title": "Data Sync", "text": "<p>This notebook will cover ...</p> <ol> <li>General Kachery information</li> <li>Setting up Kachery as a host. If you'll use an existing host, skip this.</li> <li>Setting up Kachery in your database. If you're using an existing database, skip this.</li> <li>Adding Kachery data.</li> </ol> <p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>To fully demonstrate syncing features, we'll need to run some basic analyses. This can either be done with code in this notebook or by running another notebook (e.g., LFP)</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> </ul> <p>Let's start by importing the <code>spyglass</code> package and testing that your environment is properly configured for kachery sharing</p> <p>If you haven't already done so, be sure to set up your Spyglass base directory and Kachery sharing directory with Setup</p> In\u00a0[\u00a0]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection\n\nimport spyglass.common as sgc\nimport spyglass.sharing as sgs\nfrom spyglass.settings import config\n\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection  import spyglass.common as sgc import spyglass.sharing as sgs from spyglass.settings import config  import warnings  warnings.filterwarnings(\"ignore\") <pre>[2023-12-22 08:22:32,189][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2023-12-22 08:22:32,244][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n</pre> <p>For example analysis files, run the code hidden below.</p> Quick Analysis <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nimport spyglass.data_import as sgi\nimport spyglass.lfp as lfp\n\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n\nsgi.insert_sessions(nwb_file_name)\nsgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp.v1.LFPSelection.insert1(\n    {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"lfp_electrode_group_name\": \"test\",\n        \"target_interval_list_name\": \"01_s1\",\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,\n    },\n    skip_duplicates=True,\n)\nlfp.v1.LFPV1().populate()\n</pre> <p>This notebook contains instructions for setting up data sharing/syncing through Kachery Cloud, which makes it possible to share analysis results, stored in NWB files. When a user tries to access a file, Spyglass does the following:</p> <ol> <li>Try to load from the local file system/store.</li> <li>If unavailable, check if it is in the relevant sharing table (i.e., <code>NwbKachery</code> or <code>AnalysisNWBKachery</code>).</li> <li>If present, attempt to download from the associated Kachery Resource to the user's spyglass analysis directory.</li> </ol> <p>Note: large file downloads may take a long time, so downloading raw data is not supported. We suggest direct transfer with globus or a similar service.</p> <p>A Kachery Zone is a cloud storage host. The Frank laboratory has three separate Kachery zones:</p> <ol> <li><code>franklab.default</code>: Internal file sharing, including figurls</li> <li><code>franklab.collaborator</code>: File sharing with collaborating labs.</li> <li><code>franklab.public</code>: Public file sharing (not yet active)</li> </ol> <p>Setting your zone can either be done as as an environment variable or an item in a DataJoint config. Spyglass will automatically handle setting the appropriate zone when downloading database files through kachery</p> <ul> <li><p>Environment variable:</p> <pre>export KACHERY_ZONE=franklab.default\nexport KACHERY_CLOUD_DIR=/stelmo/nwb/.kachery-cloud\n</pre> </li> <li><p>DataJoint Config:</p> <pre>\"custom\": {\n   \"kachery_zone\": \"franklab.default\",\n   \"kachery_dirs\": {\n      \"cloud\": \"/your/base/path/.kachery-cloud\"\n   }\n}\n</pre> </li> </ul> <p>See instructions for setting up new Kachery Zones, including creating a cloud bucket and registering it with the Kachery team.</p> <p>Notes:</p> <ul> <li>Bucket names cannot include periods, so we substitute a dash, as in <code>franklab-default</code>.</li> <li>You only need to create an API token for your first zone.</li> </ul> <p>See instructions for setting up zone resources. This allows for sharing files on demand. We suggest using the same name for the zone and resource.</p> <p>Note: For each zone, you need to run the local daemon that listens for requests from that zone and uploads data to the bucket for client download when requested. An example of the bash script we use is</p> <pre>export KACHERY_ZONE=franklab.collaborators\n    export KACHERY_CLOUD_DIR=/stelmo/nwb/.kachery-cloud\n    cd /stelmo/nwb/franklab_collaborators_resource\n    npx kachery-resource@latest share\n</pre> <p>For convenience, we recommend saving this code as a bash script which can be executed by the local daemon. For franklab member, these scripts can be found in the directory <code>/home/loren/bin/</code>:</p> <ul> <li>run_restart_kachery_collab.sh</li> <li>run_restart_kachery_default.sh</li> </ul> <p>Once you have a hosted zone running, we need to add its information to the Spyglass database. This will allow spyglass to manage linking files from our analysis tables to kachery. First, we'll check existing Zones.</p> In\u00a0[2]: Copied! <pre>sgs.KacheryZone()\n</pre> sgs.KacheryZone() Out[2]: <p>kachery_zone_name</p> the name of the kachery zone. Note that this is the same as the name of the kachery resource. <p>description</p> description of this zone <p>kachery_cloud_dir</p> kachery cloud directory on local machine where files are linked <p>kachery_proxy</p> kachery sharing proxy <p>lab_name</p> franklab.collaborators franklab collaborator zone /stelmo/nwb/.kachery-cloud https://kachery-resource-proxy.herokuapp.com Loren Frankfranklab.default internal franklab kachery zone /stelmo/nwb/.kachery-cloud https://kachery-resource-proxy.herokuapp.com Loren Frank <p>Total: 2</p> <p>To add a new hosted Zone, we need to prepare an entry for the <code>KacheryZone</code> table. Note that the <code>kacherycloud_dir</code> key should be the path for the server daemon hosting the zone, and is not required to be present on the client machine:</p> In\u00a0[38]: Copied! <pre>zone_name = config.get(\"KACHERY_ZONE\")\ncloud_dir = config.get(\"KACHERY_CLOUD_DIR\")\n\nzone_key = {\n    \"kachery_zone_name\": zone_name,\n    \"description\": \" \".join(zone_name.split(\".\")) + \" zone\",\n    \"kachery_cloud_dir\": cloud_dir,\n    \"kachery_proxy\": \"https://kachery-resource-proxy.herokuapp.com\",\n    \"lab_name\": sgc.Lab.fetch(\"lab_name\", limit=1)[0],\n}\n</pre> zone_name = config.get(\"KACHERY_ZONE\") cloud_dir = config.get(\"KACHERY_CLOUD_DIR\")  zone_key = {     \"kachery_zone_name\": zone_name,     \"description\": \" \".join(zone_name.split(\".\")) + \" zone\",     \"kachery_cloud_dir\": cloud_dir,     \"kachery_proxy\": \"https://kachery-resource-proxy.herokuapp.com\",     \"lab_name\": sgc.Lab.fetch(\"lab_name\", limit=1)[0], } <p>Use caution when inserting into an active database, as it could interfere with ongoing work.</p> In\u00a0[39]: Copied! <pre>sgs.KacheryZone().insert1(zone_key, skip_duplicates=True)\n</pre> sgs.KacheryZone().insert1(zone_key, skip_duplicates=True) <p>Once the zone exists, we can add <code>AnalysisNWB</code> files we want to share with members of the zone.</p> <p>The <code>AnalysisNwbFileKachery</code> table links analysis files made within other spyglass tables with a <code>uri</code> used by kachery. We can view files already made available through kachery here:</p> In\u00a0[8]: Copied! <pre>sgs.AnalysisNwbfileKachery()\n</pre> sgs.AnalysisNwbfileKachery() Out[8]: <p>kachery_zone_name</p> the name of the kachery zone. Note that this is the same as the name of the kachery resource. <p>analysis_file_name</p> name of the file <p>analysis_file_uri</p> the uri of the file franklab.collaborators Banner20220224_18NJSA2B42.nwb sha1://562b488936e5288eb89e7c480ae5c10b31c9cf2ffranklab.collaborators Frodo20230810_0F936W4B9Z.nwb sha1://b38d2b0fc1e9cde91cc239e1a0b50e3211b976fcfranklab.collaborators Frodo20230810_2MJ374GSJX.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_4L35OWMGHQ.nwb sha1://a8452cf8cf6e596b44569eb9189612d2dcd4c7d6franklab.collaborators Frodo20230810_63PWL1N0VS.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_7LYW2MK0C9.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_998JNA1VBF.nwb sha1://aa0e06028d52f5195cf24d61922ace233d8da783franklab.collaborators Frodo20230810_CFKWZTGXX0.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_GMCOCDSJ54.nwb sha1://2889b68d7aa2b30561e62be519c19759facad2d3franklab.collaborators Frodo20230810_I25NQSZQ5O.nwb sha1://973ea71d97aef91e050117bf860ea2ed83950b10franklab.collaborators Frodo20230810_JS06HC1RLC.nwb sha1://088a345c5eadfa3adea021de3f158aa86a527d4efranklab.collaborators Frodo20230810_KEEEEBDUNE.nwb sha1://4aa3199011b1405e745bbe96b62b825cd93bdacd <p>...</p> <p>Total: 298</p> <p>We can share additional results by populating new entries in this table.</p> <p>To do so we first add these entries to the <code>AnalysisNwbfileKacherySelection</code> table.</p> <p>Note: This step depends on having previously run an analysis on the example file.</p> In\u00a0[40]: Copied! <pre>nwb_copy_filename = \"minirec20230622_.nwb\"\n\nanalysis_file_list = (  # Grab all analysis files for this nwb file\n    sgc.AnalysisNwbfile() &amp; {\"nwb_file_name\": nwb_copy_filename}\n).fetch(\"analysis_file_name\")\n\nkachery_selection_key = {\"kachery_zone_name\": zone_name}\n\nfor file in analysis_file_list:  # Add all analysis to shared list\n    kachery_selection_key[\"analysis_file_name\"] = file\n    sgs.AnalysisNwbfileKacherySelection.insert1(\n        kachery_selection_key, skip_duplicates=True\n    )\n</pre> nwb_copy_filename = \"minirec20230622_.nwb\"  analysis_file_list = (  # Grab all analysis files for this nwb file     sgc.AnalysisNwbfile() &amp; {\"nwb_file_name\": nwb_copy_filename} ).fetch(\"analysis_file_name\")  kachery_selection_key = {\"kachery_zone_name\": zone_name}  for file in analysis_file_list:  # Add all analysis to shared list     kachery_selection_key[\"analysis_file_name\"] = file     sgs.AnalysisNwbfileKacherySelection.insert1(         kachery_selection_key, skip_duplicates=True     ) <p>With those files in the selection table, we can add them as links to the zone by populating the <code>AnalysisNwbfileKachery</code> table:</p> In\u00a0[\u00a0]: Copied! <pre>sgs.AnalysisNwbfileKachery.populate()\n</pre> sgs.AnalysisNwbfileKachery.populate() <p>Alternatively, we can share data based on its source table in the database using the helper function <code>share_data_to_kachery()</code></p> <p>This will take a list of tables and add all associated analysis files for entries corresponding with a passed restriction. Here, we are sharing LFP and position data for the Session \"minirec20230622_.nwb\"</p> In\u00a0[4]: Copied! <pre>from spyglass.sharing import share_data_to_kachery\nfrom spyglass.lfp.v1 import LFPV1\nfrom spyglass.position.v1 import TrodesPosV1\n\ntables = [LFPV1, TrodesPosV1]\nrestriction = {\"nwb_file_name\": \"minirec20230622_.nwb\"}\nshare_data_to_kachery(\n    table_list=tables,\n    restriction=restriction,\n    zone_name=zone_name,\n)\n</pre> from spyglass.sharing import share_data_to_kachery from spyglass.lfp.v1 import LFPV1 from spyglass.position.v1 import TrodesPosV1  tables = [LFPV1, TrodesPosV1] restriction = {\"nwb_file_name\": \"minirec20230622_.nwb\"} share_data_to_kachery(     table_list=tables,     restriction=restriction,     zone_name=zone_name, ) <p>If all of that worked,</p> <ol> <li>Go to https://kachery-gateway.figurl.org/admin?zone=your_zone (changing your_zone to the name of your zone)</li> <li>Go to the Admin/Authorization Settings tab</li> <li>Add the GitHub login names and permissions for the users you want to share with.</li> </ol> <p>If those users can connect to your database, they should now be able to use the <code>.fetch_nwb()</code> method to download any <code>AnalysisNwbfiles</code> that have been shared through Kachery.</p> <p>For example:</p> <pre>from spyglass.spikesorting import CuratedSpikeSorting\n\ntest_sort = (\n    CuratedSpikeSorting &amp; {\"nwb_file_name\": \"minirec20230622_.nwb\"}\n).fetch()[0]\nsort = (CuratedSpikeSorting &amp; test_sort).fetch_nwb()\n</pre> <p>If you are a collaborator accessing datasets, you first need to be given access to the zone by a collaborator admin (see above).</p> <p>If you know the uri for the dataset you are accessing you can test this process below (example is for members of <code>franklab.collaborators</code>)</p> In\u00a0[\u00a0]: Copied! <pre>import kachery_cloud as kcl\n\npath = \"/path/to/save/file/to/test\"\nzone_name = \"franklab.collaborators\"\nuri = \"sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4\"\n\nos.environ[\"KACHERY_ZONE\"] = zone_name\nkcl.load_file(uri=uri, dest=path, verbose=True)\nassert os.path.exists(path), f\"File not downloaded to {path}\"\n</pre> import kachery_cloud as kcl  path = \"/path/to/save/file/to/test\" zone_name = \"franklab.collaborators\" uri = \"sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4\"  os.environ[\"KACHERY_ZONE\"] = zone_name kcl.load_file(uri=uri, dest=path, verbose=True) assert os.path.exists(path), f\"File not downloaded to {path}\" <p>In normal use, spyglass will manage setting the zone and uri when accessing files. In general, the easiest way to access data valueswill be through the <code>fetch1_dataframe()</code> function part of many of the spyglass tables. In brief this will check for the appropriate nwb analysis file in your local directory, and if not found, attempt to download it from the appropriate kachery zone. It will then parse the relevant information from that nwb file into a pandas dataframe.</p> <p>We will look at an example with data from the <code>LFPV1</code> table:</p> In\u00a0[9]: Copied! <pre>from spyglass.lfp.v1 import LFPV1\n\n# Here is the data we are going to access\nLFPV1 &amp; {\n    \"nwb_file_name\": \"Winnie20220713_.nwb\",\n    \"target_interval_list_name\": \"pos 0 valid times\",\n}\n</pre> from spyglass.lfp.v1 import LFPV1  # Here is the data we are going to access LFPV1 &amp; {     \"nwb_file_name\": \"Winnie20220713_.nwb\",     \"target_interval_list_name\": \"pos 0 valid times\", } Out[9]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ Winnie20220713_.nwb tetrode_sample_Winnie pos 0 valid times LFP 0-400 Hz 30000 Winnie20220713_C52XDICU6D.nwb lfp_tetrode_sample_Winnie_pos 0 valid times_valid times a89c590f-290b-4f9c-a568-b9ae67eee96d 1000.0 <p>Total: 1</p> <p>We can access the data using <code>fetch1_dataframe()</code></p> In\u00a0[10]: Copied! <pre>(\n    LFPV1\n    &amp; {\n        \"nwb_file_name\": \"Winnie20220713_.nwb\",\n        \"target_interval_list_name\": \"pos 0 valid times\",\n    }\n).fetch1_dataframe()\n</pre> (     LFPV1     &amp; {         \"nwb_file_name\": \"Winnie20220713_.nwb\",         \"target_interval_list_name\": \"pos 0 valid times\",     } ).fetch1_dataframe() Out[10]: 0 1 2 3 4 5 6 7 8 9 ... 18 19 20 21 22 23 24 25 26 27 time 1.657741e+09 -90 -65 -104 -89 -31 -68 -27 -26 -32 -92 ... -91 -99 -87 -117 -123 -85 -73 -74 -62 13 1.657741e+09 -202 -145 -227 -220 -57 -130 -84 -68 -30 -191 ... -168 -199 -176 -250 -238 -172 -158 -140 -127 54 1.657741e+09 -218 -150 -224 -216 -84 -154 -84 -93 -29 -206 ... -125 -153 -158 -219 -206 -137 -132 -129 -120 69 1.657741e+09 -226 -151 -240 -230 -97 -144 -71 -95 -38 -236 ... -105 -136 -149 -183 -210 -111 -83 -129 -92 116 1.657741e+09 -235 -154 -250 -231 -54 -91 -81 -89 -30 -247 ... -85 -107 -116 -140 -190 -68 -28 -114 -36 193 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1.657742e+09 -3 -27 -6 29 -227 -442 -1 67 25 -15 ... -83 -217 -61 -248 -196 -63 -111 -211 -52 166 1.657742e+09 44 19 44 82 -175 -407 13 95 62 38 ... 3 -112 32 -177 -123 22 -5 -147 54 285 1.657742e+09 94 63 92 129 -121 -341 61 132 88 88 ... 62 -28 104 -99 -53 82 61 -62 125 347 1.657742e+09 142 107 135 179 -106 -370 88 178 120 148 ... 113 48 199 -44 7 145 108 -13 213 453 1.657742e+09 108 84 95 130 -82 -281 52 134 73 105 ... 97 46 169 -16 22 118 94 -3 175 348 <p>901529 rows \u00d7 28 columns</p> <p>In the next notebook, we'll explore the details of a table tier unique to Spyglass, Merge Tables.</p>"}, {"location": "notebooks/02_Data_Sync/#sync-data", "title": "Sync Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#kachery", "title": "Kachery\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#cloud", "title": "Cloud\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#zone", "title": "Zone\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#host-setup", "title": "Host Setup\u00b6", "text": "<ul> <li><p>If you are a member of a team with a pre-existing database and zone who will be sharing data, please skip to <code>Sharing Data</code></p> </li> <li><p>If you are a collaborator outside your team's network and need to access files shared with you, please skip to <code>Accessing Shared Data</code></p> </li> </ul>"}, {"location": "notebooks/02_Data_Sync/#zones", "title": "Zones\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#resources", "title": "Resources\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#database-setup", "title": "Database Setup\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#sharing-data", "title": "Sharing Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#managing-access", "title": "Managing access\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#accessing-shared-data", "title": "Accessing Shared Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/", "title": "Merge Tables", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>To insert data, see the Insert Data notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> <li>For information on why we use merge tables, and how to make one, see our documentation</li> </ul> <p>In short, merge tables represent the end processing point of a given way of processing the data in our pipelines. Merge Tables allow us to build new processing pipeline, or a new version of an existing pipeline, without having to drop or migrate the old tables. They allow data to be processed in different ways, but with a unified end result that downstream pipelines can all access.</p> <p>Let's start by importing the <code>spyglass</code> package, along with a few others.</p> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\nwarnings.simplefilter(\"ignore\", category=UserWarning)\n\nimport spyglass.common as sgc\nimport spyglass.lfp as lfp\nfrom spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nfrom spyglass.utils.dj_merge_tables import delete_downstream_merge, Merge\nfrom spyglass.common.common_ephys import LFP as CommonLFP  # Upstream 1\nfrom spyglass.lfp.lfp_merge import LFPOutput  # Merge Table\nfrom spyglass.lfp.v1.lfp import LFPV1  # Upstream 2\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) warnings.simplefilter(\"ignore\", category=UserWarning)  import spyglass.common as sgc import spyglass.lfp as lfp from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename from spyglass.utils.dj_merge_tables import delete_downstream_merge, Merge from spyglass.common.common_ephys import LFP as CommonLFP  # Upstream 1 from spyglass.lfp.lfp_merge import LFPOutput  # Merge Table from spyglass.lfp.v1.lfp import LFPV1  # Upstream 2 <pre>[2024-01-29 16:15:00,903][INFO]: Connecting root@localhost:3309\n[2024-01-29 16:15:00,912][INFO]: Connected root@localhost:3309\n</pre> <p>Check to make sure the data inserted in the previour notebook is still there.</p> In\u00a0[2]: Copied! <pre>nwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nnwb_file_dict = {\"nwb_file_name\": nwb_copy_file_name}\nsgc.Session &amp; nwb_file_dict\n</pre> nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) nwb_file_dict = {\"nwb_file_name\": nwb_copy_file_name} sgc.Session &amp; nwb_file_dict Out[2]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p>If you haven't already done so, insert data into a Merge Table.</p> <p>Note: Some existing parents of Merge Tables perform the Merge Table insert as part of the populate methods. This practice will be revised in the future.</p> In\u00a0[3]: Copied! <pre>sgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"lfp_electrode_group_name\": \"test\",\n    \"target_interval_list_name\": \"01_s1\",\n    \"filter_name\": \"LFP 0-400 Hz\",\n    \"filter_sampling_rate\": 30_000,\n}\nlfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True)\nlfp.v1.LFPV1().populate(lfp_key)\nLFPOutput.insert([lfp_key], skip_duplicates=True)\n</pre> sgc.FirFilterParameters().create_standard_filters() lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test\",     electrode_list=[0], ) lfp_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"lfp_electrode_group_name\": \"test\",     \"target_interval_list_name\": \"01_s1\",     \"filter_name\": \"LFP 0-400 Hz\",     \"filter_sampling_rate\": 30_000, } lfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True) lfp.v1.LFPV1().populate(lfp_key) LFPOutput.insert([lfp_key], skip_duplicates=True) <p>Merge Tables have multiple custom methods that begin with <code>merge</code>.</p> <p><code>help</code> can show us the docstring of each</p> In\u00a0[4]: Copied! <pre>merge_methods = [d for d in dir(Merge) if d.startswith(\"merge\")]\nprint(merge_methods)\n</pre> merge_methods = [d for d in dir(Merge) if d.startswith(\"merge\")] print(merge_methods) <pre>['merge_delete', 'merge_delete_parent', 'merge_fetch', 'merge_get_parent', 'merge_get_parent_class', 'merge_get_part', 'merge_html', 'merge_populate', 'merge_restrict', 'merge_restrict_class', 'merge_view']\n</pre> In\u00a0[5]: Copied! <pre>help(getattr(Merge, merge_methods[-1]))\n</pre> help(getattr(Merge, merge_methods[-1])) <pre>Help on method merge_view in module spyglass.utils.dj_merge_tables:\n\nmerge_view(restriction: str = True) method of datajoint.user_tables.TableMeta instance\n    Prints merged view, including null entries for unique columns.\n    \n    Note: To handle this Union as a table-like object, use `merge_resrict`\n    \n    Parameters\n    ---------\n    restriction: str, optional\n        Restriction to apply to the merged view\n\n</pre> <p><code>merge_view</code> shows a union of the master and all part tables.</p> <p>Note: Restrict Merge Tables with arguments, not the <code>&amp;</code> operator.</p> <ul> <li>Normally: <code>Table &amp; \"field='value'\"</code></li> <li>Instead: <code>MergeTable.merge_view(restriction=\"field='value'\"</code>).</li> </ul> In\u00a0[6]: Copied! <pre>LFPOutput.merge_view()\n</pre> LFPOutput.merge_view() <pre>*merge_id      *source    *nwb_file_name *lfp_electrode *target_interv *filter_name   *filter_sampli\n+------------+ +--------+ +------------+ +------------+ +------------+ +------------+ +------------+\nc34f98c5-7de7- LFPV1      minirec2023062 test           01_s1          LFP 0-400 Hz   30000         \n (Total: 1)\n\n</pre> <p>UUIDs help retain unique entries across all part tables. We can fetch NWB file by referencing this or other features.</p> In\u00a0[\u00a0]: Copied! <pre>uuid_key = (LFPOutput &amp; nwb_file_dict).fetch(limit=1, as_dict=True)[-1]\nrestrict = LFPOutput &amp; uuid_key\nrestrict\n</pre> uuid_key = (LFPOutput &amp; nwb_file_dict).fetch(limit=1, as_dict=True)[-1] restrict = LFPOutput &amp; uuid_key restrict Out[\u00a0]: <p>merge_id</p> <p>source</p> c34f98c5-7de7-1daf-6eaf-1e15981def44 LFPV1 <p>Total: 1</p> In\u00a0[8]: Copied! <pre>result1 = restrict.fetch_nwb(restrict.fetch1(\"KEY\"))\nresult1\n</pre> result1 = restrict.fetch_nwb(restrict.fetch1(\"KEY\")) result1 Out[8]: <pre>[{'nwb_file_name': 'minirec20230622_.nwb',\n  'lfp_electrode_group_name': 'test',\n  'target_interval_list_name': '01_s1',\n  'filter_name': 'LFP 0-400 Hz',\n  'filter_sampling_rate': 30000,\n  'analysis_file_name': 'minirec20230622_R5DWQ6S53S.nwb',\n  'interval_list_name': 'lfp_test_01_s1_valid times',\n  'lfp_object_id': 'ffb893d1-a31e-41d3-aec7-8dc8936c8898',\n  'lfp_sampling_rate': 1000.0,\n  'lfp': filtered data pynwb.ecephys.ElectricalSeries at 0x129602752674544\n  Fields:\n    comments: no comments\n    conversion: 1.0\n    data: &lt;HDF5 dataset \"data\": shape (10476, 1), type \"&lt;i2\"&gt;\n    description: filtered data\n    electrodes: electrodes &lt;class 'hdmf.common.table.DynamicTableRegion'&gt;\n    interval: 1\n    offset: 0.0\n    resolution: -1.0\n    timestamps: &lt;HDF5 dataset \"timestamps\": shape (10476,), type \"&lt;f8\"&gt;\n    timestamps_unit: seconds\n    unit: volts}]</pre> In\u00a0[9]: Copied! <pre>nwb_key = LFPOutput.merge_restrict(nwb_file_dict).fetch(as_dict=True)[0]\nnwb_key\n</pre> nwb_key = LFPOutput.merge_restrict(nwb_file_dict).fetch(as_dict=True)[0] nwb_key Out[9]: <pre>{'merge_id': UUID('c34f98c5-7de7-1daf-6eaf-1e15981def44'),\n 'source': 'LFPV1',\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'lfp_electrode_group_name': 'test',\n 'target_interval_list_name': '01_s1',\n 'filter_name': 'LFP 0-400 Hz',\n 'filter_sampling_rate': 30000}</pre> In\u00a0[10]: Copied! <pre>result2 = LFPOutput().fetch_nwb(nwb_key)\nresult2 == result1\n</pre> result2 = LFPOutput().fetch_nwb(nwb_key) result2 == result1 Out[10]: <pre>True</pre> <p>There are also functions for retrieving part/parent table(s) and fetching data.</p> <p>These <code>get</code> functions will either return the part table of the Merge table or the parent table with the source information for that part.</p> In\u00a0[11]: Copied! <pre>result4 = LFPOutput.merge_get_part(restriction=nwb_file_dict, join_master=True)\nresult4\n</pre> result4 = LFPOutput.merge_get_part(restriction=nwb_file_dict, join_master=True) result4 Out[11]: <p>merge_id</p> <p>source</p> <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter c34f98c5-7de7-1daf-6eaf-1e15981def44 LFPV1 minirec20230622_.nwb test 01_s1 LFP 0-400 Hz 30000 <p>Total: 1</p> In\u00a0[12]: Copied! <pre>result5 = LFPOutput.merge_get_parent(restriction='nwb_file_name LIKE \"mini%\"')\nresult5\n</pre> result5 = LFPOutput.merge_get_parent(restriction='nwb_file_name LIKE \"mini%\"') result5 Out[12]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ minirec20230622_.nwb test 01_s1 LFP 0-400 Hz 30000 minirec20230622_R5DWQ6S53S.nwb lfp_test_01_s1_valid times ffb893d1-a31e-41d3-aec7-8dc8936c8898 1000.0 <p>Total: 1</p> <p><code>fetch</code> will collect all relevant entries and return them as a list in the format specified by keyword arguments and one's DataJoint config.</p> In\u00a0[13]: Copied! <pre>result6 = result5.fetch(\"lfp_sampling_rate\")  # Sample rate for all mini* files\nresult6\n</pre> result6 = result5.fetch(\"lfp_sampling_rate\")  # Sample rate for all mini* files result6 Out[13]: <pre>array([1000.])</pre> <p><code>merge_fetch</code> requires a restriction as the first argument. For no restriction, use <code>True</code>.</p> In\u00a0[14]: Copied! <pre>result7 = LFPOutput.merge_fetch(True, \"filter_name\", \"nwb_file_name\")\nresult7\n</pre> result7 = LFPOutput.merge_fetch(True, \"filter_name\", \"nwb_file_name\") result7 Out[14]: <pre>[array(['LFP 0-400 Hz'], dtype=object),\n array(['minirec20230622_.nwb'], dtype=object)]</pre> In\u00a0[15]: Copied! <pre>result8 = LFPOutput.merge_fetch(as_dict=True)\nresult8\n</pre> result8 = LFPOutput.merge_fetch(as_dict=True) result8 Out[15]: <pre>{'merge_id': UUID('c34f98c5-7de7-1daf-6eaf-1e15981def44'),\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'lfp_electrode_group_name': 'test',\n 'target_interval_list_name': '01_s1',\n 'filter_name': 'LFP 0-400 Hz',\n 'filter_sampling_rate': 30000}</pre> <p>When deleting from Merge Tables, we can either...</p> <ol> <li><p>delete from the Merge Table itself with <code>merge_delete</code>, deleting both the master and part.</p> </li> <li><p>use <code>merge_delete_parent</code> to delete from the parent sources, getting rid of the entries in the source table they came from.</p> </li> <li><p>use <code>delete_downstream_merge</code> to find Merge Tables downstream of any other table and get rid full entries, avoiding orphaned master table entries.</p> </li> </ol> <p>The two latter cases can be destructive, so we include an extra layer of protection with <code>dry_run</code>. When true (by default), these functions return a list of tables with the entries that would otherwise be deleted.</p> In\u00a0[16]: Copied! <pre>LFPOutput.merge_delete(nwb_file_dict)  # Delete from merge table\n</pre> LFPOutput.merge_delete(nwb_file_dict)  # Delete from merge table <pre>[2024-01-29 16:15:23,054][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output__l_f_p_v1`\n[2024-01-29 16:15:23,058][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output`\n</pre> <pre>[2024-01-29 16:15:24,953][WARNING]: Deletes cancelled\n</pre> In\u00a0[17]: Copied! <pre>LFPOutput.merge_delete_parent(restriction=nwb_file_dict, dry_run=True)\n</pre> LFPOutput.merge_delete_parent(restriction=nwb_file_dict, dry_run=True) Out[17]: <pre>[FreeTable(`lfp_v1`.`__l_f_p_v1`)\n *nwb_file_name *lfp_electrode *target_interv *filter_name   *filter_sampli analysis_file_ interval_list_ lfp_object_id  lfp_sampling_r\n +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\n minirec2023062 test           01_s1          LFP 0-400 Hz   30000          minirec2023062 lfp_test_01_s1 ffb893d1-a31e- 1000.0        \n  (Total: 1)]</pre> <p><code>delete_downstream_merge</code> is available from any other table in the pipeline, but it does take some time to find the links downstream. If you're using this, you can save time by reassigning your table to a variable, which will preserve a copy of the previous search.</p> <p>Because the copy is stored, this function may not see additional merge tables you've imported. To refresh this copy, set <code>reload_cache=True</code></p> In\u00a0[18]: Copied! <pre>nwbfile = sgc.Nwbfile()\n\n(nwbfile &amp; nwb_file_dict).delete_downstream_merge(\n    dry_run=True,\n    reload_cache=False,  # if still encountering errors, try setting this to True\n)\n</pre> nwbfile = sgc.Nwbfile()  (nwbfile &amp; nwb_file_dict).delete_downstream_merge(     dry_run=True,     reload_cache=False,  # if still encountering errors, try setting this to True ) <pre>[16:15:37][INFO] Spyglass: Building merge cache for nwbfile.\n\tFound 3 downstream merge tables\n</pre> Out[18]: <pre>dict_values([[*nwb_file_name *analysis_file *lfp_electrode *target_interv *filter_name   *filter_sampli *merge_id      nwb_file_a analysis_f analysis_file_ analysis_p interval_list_ lfp_object_id  lfp_sampling_r\n+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +--------+ +--------+ +------------+ +--------+ +------------+ +------------+ +------------+\nminirec2023062 minirec2023062 test           01_s1          LFP 0-400 Hz   30000          c34f98c5-7de7- =BLOB=     =BLOB=                    =BLOB=     lfp_test_01_s1 ffb893d1-a31e- 1000.0        \n (Total: 1)\n]])</pre> <p>This function is run automatically whin you use <code>cautious_delete</code>, which checks team permissions before deleting.</p> In\u00a0[\u00a0]: Copied! <pre>(nwbfile &amp; nwb_file_dict).cautious_delete()\n</pre> (nwbfile &amp; nwb_file_dict).cautious_delete() <p>In the next notebook, we'll start working with ephys data with spike sorting.</p>"}, {"location": "notebooks/03_Merge_Tables/#merge-tables", "title": "Merge Tables\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#example-data", "title": "Example data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#helper-functions", "title": "Helper functions\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#showing-data", "title": "Showing data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#selecting-data", "title": "Selecting data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#deletion-from-merge-tables", "title": "Deletion from Merge Tables\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/04_PopulateConfigFile/", "title": "Config Populate", "text": "In\u00a0[\u00a0]: Copied! <pre>nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys_rly.nwb\"\n</pre> nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys_rly.nwb\" In\u00a0[\u00a0]: Copied! <pre># this configuration yaml file should be placed next to the downloaded NWB file\nyaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_rly_spyglass_config.yaml\"\nwith open(yaml_config_path, \"w\") as config_file:\n    lines = [\n        \"DataAcquisitionDevice\",\n        \"- data_acquisition_device_name: data_acq_device0\",\n    ]\n    config_file.writelines(line + \"\\n\" for line in lines)\n</pre> # this configuration yaml file should be placed next to the downloaded NWB file yaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_rly_spyglass_config.yaml\" with open(yaml_config_path, \"w\") as config_file:     lines = [         \"DataAcquisitionDevice\",         \"- data_acquisition_device_name: data_acq_device0\",     ]     config_file.writelines(line + \"\\n\" for line in lines) <p>Then call <code>insert_sessions</code> as usual.</p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.data_import as sgi\n\nsgi.insert_sessions(nwb_file_name)\n</pre> import spyglass.data_import as sgi  sgi.insert_sessions(nwb_file_name) <p>Confirm the session was inserted with the correct <code>DataAcquisitionDevice</code></p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.common as sgc\nfrom spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n\nsgc.Session.DataAcquisitionDevice &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> import spyglass.common as sgc from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)  sgc.Session.DataAcquisitionDevice &amp; {\"nwb_file_name\": nwb_copy_file_name}"}, {"location": "notebooks/04_PopulateConfigFile/#customizing-data-insertion-into-spyglass", "title": "Customizing Data Insertion into Spyglass\u00b6", "text": "<p>If you would like to insert data into Spyglass that does not follow the naming or organizational format expected by Spyglass,  or you would like to override what values are ingested into Spyglass from  your NWB files, including missing values, follow this guide.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#general-approach", "title": "General Approach\u00b6", "text": "<p>When an NWB file is ingested into Spyglass, metadata about the session is first read from the NWB file and inserted into tables in the <code>common</code> module (e.g. <code>Institution</code>, <code>Lab</code>, <code>Electrode</code>, etc). However, not every NWB file has all the information required by Spyglass or the information in the NWB file is not in a format that Spyglass expects. For example, many NWB files do not contain information about the <code>DataAcquisitionDevice</code> or <code>Probe</code> because the NWB data standard does not yet have an official standard for specifying them. For these cases, we provide a way to customize how data is ingested into Spyglass.</p> <p>Let's say that you want to ingest an NWB file into Spyglass where the lab name in the file is written as \"Loren Frank Lab\" or it is not specified, but you  know the data comes from the Loren Frank Lab. Let's say that in Spyglass, the lab name that is associated with sessions from the Loren Frank Lab is  \"Frank Lab\" and you would like to use the same name in order to facilitate data search in Spyglass. To change the lab name when you insert your new data  to Spyglass, you could either 1) edit the NWB file directly and then  insert it into Spyglass, or 2) define an override value \"Frank Lab\" to be  used instead of the value specified in the NWB file (or lack thereof).</p> <p>Note that if this is your own data and you want to make changes to information about how the data is interpreted, e.g., the units of measurement are incorrect, we recommend that you edit the NWB file directly because the  file or derivatives of it might eventually be shared outside of Spyglass and  they will not reflect any modifications that you have made to  the data only in Spyglass.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#define-a-configuration-yaml-file", "title": "Define a Configuration YAML File\u00b6", "text": "<p>To override values in the NWB file during insertion into Spyglass,  you will need to create a configuration  YAML file that lives in the same directory as your NWB file, named:  <code>&lt;name_of_nwb_file&gt;_spyglass_config.yaml</code></p> <p>An example configuration YAML file can be found at <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code>. This file is associated with the NWB file <code>sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb</code>.</p> <p>This is the general format for entries in this configuration file:</p> <pre>TableName:\n- primary_key1: value1\n</pre> <p>For example:</p> <pre>Lab:\n- lab_name: Frank Lab\n</pre> <p>In this example, the NWB file that corresponds to this config YAML will become associated with the entry in the <code>Lab</code> table with the value <code>Frank Lab</code> for  the primary key <code>lab_name</code>. This entry must already exist. More specifically, when the NWB file is ingested into Spyglass,  a new <code>Session</code> entry will be created for the NWB file that has a foreign key to the <code>Lab</code> entry with <code>lab_name</code> = \"Frank Lab\", ignoring whatever lab value is  in the NWB file, even if one does not exist.</p> <p>TODO implement this for <code>Lab</code>.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#create-entries-to-reference-in-the-configuration-yaml", "title": "Create Entries to Reference in the Configuration YAML\u00b6", "text": "<p>As mentioned earlier, the table entry that you want to associate with your NWB file must already exist in the database. This entry would typically be a value that is independent of any particular NWB file, such as <code>DataAcquisitionDevice</code>, <code>Lab</code>, <code>Probe</code>, and <code>BrainRegion</code>.</p> <p>If the entry does not already exist, you can either:</p> <ol> <li><p>create the entry programmatically using DataJoint <code>insert</code> commands, or</p> </li> <li><p>define the entry in a YAML file called <code>entries.yaml</code> that is automatically processed when Spyglass is imported. You can think of <code>entries.yaml</code> as a place to define information that the database should come pre-equipped prior to ingesting your NWB files. The <code>entries.yaml</code> file should be placed in the <code>spyglass</code> base directory (next to <code>README.md</code>). An example can be found in <code>examples/config_yaml/entries.yaml</code>. This file should have the following structure:</p> <pre>TableName:\n- TableEntry1Field1: Value\n  TableEntry1Field2: Value\n- TableEntry2Field1: Value\n  TableEntry2Field2: Value\n</pre> <p>For example,</p> <pre>DataAcquisitionDeviceSystem:\n  data_acquisition_device_system: SpikeGLX\nDataAcquisitionDevice:\n- data_acquisition_device_name: Neuropixels_SpikeGLX\n  data_acquisition_device_system: SpikeGLX\n  data_acquisition_device_amplifier: Intan\n</pre> <p>Only <code>dj.Manual</code>, <code>dj.Lookup</code>, and <code>dj.Part</code> tables can be populated  using this approach.</p> </li> </ol> <p>Once the entry that you want to associate with your NWB file exists in the database, you can write the configuration YAML file and then ingest your NWB file. As an another example, let's say that you want to associate your NWB file with the <code>DataAcquisitionDevice</code> entry with <code>data_acquisition_device_name</code> = \"Neuropixels_SpikeGLX\" that was defined above. You would write the following configuration YAML file:</p> <pre>DataAcquisitionDevice:\n- data_acquisition_device_name: Neuropixels_SpikeGLX\n</pre> <p>The example in <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code> includes additional examples.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#example-ingestion-with-real-data", "title": "Example Ingestion with Real Data\u00b6", "text": "<p>For this example, you will need to download the 5 GB NWB file  <code>sub-JDS-NFN-AM2_behavior+ecephys.nwb</code> from dandiset 000447 here:  https://dandiarchive.org/dandiset/000447/0.230316.2133/files?location=sub-JDS-NFN-AM2&amp;page=1</p> <p>Click the download arrow button to download the file to your computer. Add it to the folder containing your raw NWB data to be ingested into Spyglass.</p> <p>This file does not specify a data acquisition device. Let's say that the data was collected from a SpikeGadgets system with an Intan amplifier. This matches an existing entry in the <code>DataAcquisitionDevice</code> table with name \"data_acq_device0\". We will create a configuration YAML file to associate this entry with the NWB file.</p> <p>If you are connected to the Frank lab database, please rename any downloaded files (e.g., <code>example20200101_yourname.nwb</code>) to avoid naming collisions, as the file name acts as the primary key across key tables.</p>"}, {"location": "notebooks/10_Spike_SortingV0/", "title": "Spike Sorting V0", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> Extract the recording <ol> <li>Specifying your NWB file.</li> <li>Specifying which electrodes involved in the recording to sort data from. - <code>SortGroup</code></li> <li>Specifying the time segment of the recording we want to sort. - <code>IntervalList</code>, <code>SortInterval</code></li> <li>Specifying the parameters to use for filtering the recording. - <code>SpikeSortingPreprocessingParameters</code></li> <li>Combining these parameters. - <code>SpikeSortingRecordingSelection</code></li> <li>Extracting the recording. - <code>SpikeSortingRecording</code></li> <li>Specifying the parameters to apply for artifact detection/removal. -<code>ArtifactDetectionParameters</code></li> </ol> Spike sorting the recording <ol> <li>Specify the spike sorter and parameters to use. - <code>SpikeSorterParameters</code></li> <li>Combine these parameters. - <code>SpikeSortingSelection</code></li> <li>Spike sort the extracted recording according to chose parameter set. - <code>SpikeSorting</code></li> </ol> <p> </p> In\u00a0[10]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.spikesorting.v0 as sgs\nfrom spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.spikesorting.v0 as sgs from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <p>If you haven't already done so, add yourself to <code>LabTeam</code></p> In\u00a0[6]: Copied! <pre># Full name, Google email address, DataJoint username, admin\nname, email, dj_user, admin = (\n    \"Firstname Lastname\",\n    \"example@gmail.com\",\n    \"user\",\n    0,\n)\nsgc.LabMember.insert_from_name(name)\nsgc.LabMember.LabMemberInfo.insert1(\n    [\n        name,\n        email,\n        dj_user,\n        admin,\n    ],\n    skip_duplicates=True,\n)\nsgc.LabMember.LabMemberInfo() &amp; {\n    \"team_name\": \"My Team\",\n    \"lab_member_name\": \"Firstname Lastname\",\n}\n</pre> # Full name, Google email address, DataJoint username, admin name, email, dj_user, admin = (     \"Firstname Lastname\",     \"example@gmail.com\",     \"user\",     0, ) sgc.LabMember.insert_from_name(name) sgc.LabMember.LabMemberInfo.insert1(     [         name,         email,         dj_user,         admin,     ],     skip_duplicates=True, ) sgc.LabMember.LabMemberInfo() &amp; {     \"team_name\": \"My Team\",     \"lab_member_name\": \"Firstname Lastname\", } Out[6]: Information about lab member in the context of Frank lab network <p>lab_member_name</p> <p>google_user_name</p> For permission to curate <p>datajoint_user_name</p> For permission to delete <p>admin</p> Ignore permission checks FirstName LastName gmail@gmail.com user 0 <p>Total: 1</p> <p>We can try <code>fetch</code> to confirm.</p> <p>Exercise: Try to write a fer lines to generate a dictionary with team names as keys and lists of members as values. It may be helpful to add more data with the code above and use <code>fetch(as_dict=True)</code>.</p> In\u00a0[7]: Copied! <pre>my_team_members = (\n    (sgc.LabTeam.LabTeamMember &amp; {\"team_name\": \"My Team\"})\n    .fetch(\"lab_member_name\")\n    .tolist()\n)\nif name in my_team_members:\n    print(\"You made it in!\")\n</pre> my_team_members = (     (sgc.LabTeam.LabTeamMember &amp; {\"team_name\": \"My Team\"})     .fetch(\"lab_member_name\")     .tolist() ) if name in my_team_members:     print(\"You made it in!\") <pre>You made it in!\n</pre> Code hidden here <pre>members = sgc.LabTeam.LabTeamMember.fetch(as_dict=True)\nteams_dict = {member[\"team_name\"]: [] for member in members}\nfor member in members:\n    teams_dict[member[\"team_name\"]].append(member[\"lab_member_name\"])\nprint(teams_dict)\n</pre> <p>If you haven't already, load an NWB file. For more details on downloading and importing data, see this notebook.</p> In\u00a0[8]: Copied! <pre>import spyglass.data_import as sdi\n\nsdi.insert_sessions(\"minirec20230622.nwb\")\nnwb_file_name = \"minirec20230622_.nwb\"\n</pre> import spyglass.data_import as sdi  sdi.insert_sessions(\"minirec20230622.nwb\") nwb_file_name = \"minirec20230622_.nwb\" <pre>/home/sambray/Documents/spyglass/src/spyglass/data_import/insert_sessions.py:58: UserWarning: Cannot insert data from minirec20230622.nwb: minirec20230622_.nwb is already in Nwbfile table.\n  warnings.warn(\n</pre> <p>Each NWB file will have multiple electrodes we can use for spike sorting. We commonly use multiple electrodes in a <code>SortGroup</code> selected by what tetrode or shank of a probe they were on.</p> <p>Note: This will delete any existing entries. Answer 'yes' when prompted, or skip running this cell to leave data in place.</p> In\u00a0[12]: Copied! <pre>sgs.SortGroup().set_group_by_shank(nwb_file_name)\n</pre> sgs.SortGroup().set_group_by_shank(nwb_file_name) <p>Each electrode has an <code>electrode_id</code> and is associated with an <code>electrode_group_name</code>, which corresponds with a <code>sort_group_id</code>.</p> <p>For example, data recorded from a 32 tetrode (128 channel) drive results in 128 unique <code>electrode_id</code>. This could result in 32 unique <code>electrode_group_name</code> and 32 unique <code>sort_group_id</code>.</p> In\u00a0[13]: Copied! <pre>sgs.SortGroup.SortGroupElectrode &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgs.SortGroup.SortGroupElectrode &amp; {\"nwb_file_name\": nwb_file_name} Out[13]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode minirec20230622_.nwb 0 0 0minirec20230622_.nwb 0 0 1minirec20230622_.nwb 0 0 2minirec20230622_.nwb 0 0 3minirec20230622_.nwb 1 1 4minirec20230622_.nwb 1 1 5minirec20230622_.nwb 1 1 6minirec20230622_.nwb 1 1 7minirec20230622_.nwb 10 10 40minirec20230622_.nwb 10 10 41minirec20230622_.nwb 10 10 42minirec20230622_.nwb 10 10 43 <p>...</p> <p>Total: 128</p> In\u00a0[14]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name} Out[14]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1 =BLOB= minirec20230622_.nwb 01_s1_first9 =BLOB= minirec20230622_.nwb 01_s1_first9 lfp band 100Hz =BLOB= lfp bandminirec20230622_.nwb 02_s2 =BLOB= minirec20230622_.nwb lfp_test_01_s1_first9_valid times =BLOB= lfp_v1minirec20230622_.nwb lfp_test_01_s1_valid times =BLOB= lfp_v1minirec20230622_.nwb minirec20230622_.nwb_01_s1_first9_0_default_hippocampus =BLOB= spikesorting_recording_v0minirec20230622_.nwb minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times =BLOB= spikesorting_artifact_v0minirec20230622_.nwb pos 0 valid times =BLOB= minirec20230622_.nwb pos 1 valid times =BLOB= minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 11</p> <p>Let's start with the first run interval (<code>01_s1</code>) and fetch corresponding <code>valid_times</code>. For the <code>minirec</code> example, this is relatively short.</p> In\u00a0[15]: Copied! <pre>interval_list_name = \"01_s1\"\ninterval_list = (\n    sgc.IntervalList\n    &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": interval_list_name}\n).fetch1(\"valid_times\")[0]\n\n\ndef print_interval_duration(interval_list: np.ndarray):\n    duration = np.round((interval_list[1] - interval_list[0]))\n    print(f\"This interval list is {duration:g} seconds long\")\n\n\nprint_interval_duration(interval_list)\n</pre> interval_list_name = \"01_s1\" interval_list = (     sgc.IntervalList     &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": interval_list_name} ).fetch1(\"valid_times\")[0]   def print_interval_duration(interval_list: np.ndarray):     duration = np.round((interval_list[1] - interval_list[0]))     print(f\"This interval list is {duration:g} seconds long\")   print_interval_duration(interval_list) <pre>This interval list is 10 seconds long\n</pre> In\u00a0[16]: Copied! <pre>n = 9\nsort_interval_name = interval_list_name + f\"_first{n}\"\nsort_interval = np.array([interval_list[0], interval_list[0] + n])\n</pre> n = 9 sort_interval_name = interval_list_name + f\"_first{n}\" sort_interval = np.array([interval_list[0], interval_list[0] + n]) <p>With the above, we can insert into <code>SortInterval</code></p> In\u00a0[17]: Copied! <pre>sgs.SortInterval.insert1(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"sort_interval_name\": sort_interval_name,\n        \"sort_interval\": sort_interval,\n    },\n    skip_duplicates=True,\n)\n</pre> sgs.SortInterval.insert1(     {         \"nwb_file_name\": nwb_file_name,         \"sort_interval_name\": sort_interval_name,         \"sort_interval\": sort_interval,     },     skip_duplicates=True, ) <p>And verify the entry</p> In\u00a0[18]: Copied! <pre>print_interval_duration(\n    (\n        sgs.SortInterval\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"sort_interval_name\": sort_interval_name,\n        }\n    ).fetch1(\"sort_interval\")\n)\n</pre> print_interval_duration(     (         sgs.SortInterval         &amp; {             \"nwb_file_name\": nwb_file_name,             \"sort_interval_name\": sort_interval_name,         }     ).fetch1(\"sort_interval\") ) <pre>This interval list is 9 seconds long\n</pre> <p><code>SpikeSortingPreprocessingParameters</code> contains the parameters used to filter the recorded data in the spike band prior to sorting.</p> In\u00a0[19]: Copied! <pre>sgs.SpikeSortingPreprocessingParameters()\n</pre> sgs.SpikeSortingPreprocessingParameters() Out[19]: <p>preproc_params_name</p> <p>preproc_params</p> default =BLOB=default_hippocampus =BLOB=default_min_seg =BLOB=franklab_default_hippocampus =BLOB=franklab_default_hippocampus_min_segment_length =BLOB=franklab_tetrode_hippocampus =BLOB=franklab_tetrode_hippocampus_min_seg =BLOB=lf_test =BLOB= <p>Total: 8</p> <p>Here, we insert the default parameters and then fetch them.</p> In\u00a0[20]: Copied! <pre>sgs.SpikeSortingPreprocessingParameters().insert_default()\npreproc_params = (\n    sgs.SpikeSortingPreprocessingParameters()\n    &amp; {\"preproc_params_name\": \"default\"}\n).fetch1(\"preproc_params\")\nprint(preproc_params)\n</pre> sgs.SpikeSortingPreprocessingParameters().insert_default() preproc_params = (     sgs.SpikeSortingPreprocessingParameters()     &amp; {\"preproc_params_name\": \"default\"} ).fetch1(\"preproc_params\") print(preproc_params) <pre>{'frequency_min': 300, 'frequency_max': 6000, 'margin_ms': 5, 'seed': 0}\n</pre> <p>Let's adjust the <code>frequency_min</code> to 600, the preference for hippocampal data, and insert that into the table as a new set of parameters for hippocampal data.</p> In\u00a0[21]: Copied! <pre>preproc_params[\"frequency_min\"] = 600\nsgs.SpikeSortingPreprocessingParameters().insert1(\n    {\n        \"preproc_params_name\": \"default_hippocampus\",\n        \"preproc_params\": preproc_params,\n    },\n    skip_duplicates=True,\n)\n</pre> preproc_params[\"frequency_min\"] = 600 sgs.SpikeSortingPreprocessingParameters().insert1(     {         \"preproc_params_name\": \"default_hippocampus\",         \"preproc_params\": preproc_params,     },     skip_duplicates=True, ) In\u00a0[22]: Copied! <pre>interval_list_name\n</pre> interval_list_name Out[22]: <pre>'01_s1'</pre> In\u00a0[23]: Copied! <pre>ssr_key = dict(\n    nwb_file_name=nwb_file_name,\n    sort_group_id=0,  # See SortGroup\n    sort_interval_name=sort_interval_name,  # First N seconds above\n    preproc_params_name=\"default_hippocampus\",  # See preproc_params\n    interval_list_name=interval_list_name,\n    team_name=\"My Team\",\n)\n</pre> ssr_key = dict(     nwb_file_name=nwb_file_name,     sort_group_id=0,  # See SortGroup     sort_interval_name=sort_interval_name,  # First N seconds above     preproc_params_name=\"default_hippocampus\",  # See preproc_params     interval_list_name=interval_list_name,     team_name=\"My Team\", ) In\u00a0[24]: Copied! <pre>sgs.SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True)\nsgs.SpikeSortingRecordingSelection() &amp; ssr_key\n</pre> sgs.SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True) sgs.SpikeSortingRecordingSelection() &amp; ssr_key Out[24]: Defines recordings to be sorted <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team 01_s1 <p>Total: 1</p> In\u00a0[25]: Copied! <pre>ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; ssr_key).proj()\nsgs.SpikeSortingRecording.populate([ssr_pk])\n</pre> ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; ssr_key).proj() sgs.SpikeSortingRecording.populate([ssr_pk]) <p>Now we can see our recording in the table. E x c i t i n g !</p> In\u00a0[26]: Copied! <pre>sgs.SpikeSortingRecording() &amp; ssr_key\n</pre> sgs.SpikeSortingRecording() &amp; ssr_key Out[26]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>recording_path</p> <p>sort_interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team /stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus minirec20230622_.nwb_01_s1_first9_0_default_hippocampus <p>Total: 1</p> In\u00a0[27]: Copied! <pre>sgs.ArtifactDetectionParameters().insert_default()\nartifact_key = (sgs.SpikeSortingRecording() &amp; ssr_key).fetch1(\"KEY\")\nartifact_key[\"artifact_params_name\"] = \"none\"\n</pre> sgs.ArtifactDetectionParameters().insert_default() artifact_key = (sgs.SpikeSortingRecording() &amp; ssr_key).fetch1(\"KEY\") artifact_key[\"artifact_params_name\"] = \"none\" <p>We then pair artifact detection parameters in <code>ArtifactParameters</code> with a recording extracted through population of <code>SpikeSortingRecording</code> and insert into <code>ArtifactDetectionSelection</code>.</p> In\u00a0[29]: Copied! <pre>sgs.ArtifactDetectionSelection().insert1(artifact_key, skip_duplicates=True)\nsgs.ArtifactDetectionSelection() &amp; artifact_key\n</pre> sgs.ArtifactDetectionSelection().insert1(artifact_key, skip_duplicates=True) sgs.ArtifactDetectionSelection() &amp; artifact_key Out[29]: Specifies artifact detection parameters to apply to a sort group's recording. <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>artifact_params_name</p> <p>custom_artifact_detection</p> minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team none 0 <p>Total: 1</p> <p>Then, we can populate <code>ArtifactDetection</code>, which will find periods where there are artifacts, as specified by the parameters.</p> In\u00a0[30]: Copied! <pre>sgs.ArtifactDetection.populate(artifact_key)\n</pre> sgs.ArtifactDetection.populate(artifact_key) <p>Populating <code>ArtifactDetection</code> also inserts an entry into <code>ArtifactRemovedIntervalList</code>, which stores the interval without detected artifacts.</p> In\u00a0[31]: Copied! <pre>sgs.ArtifactRemovedIntervalList() &amp; artifact_key\n</pre> sgs.ArtifactRemovedIntervalList() &amp; artifact_key Out[31]: Stores intervals without detected artifacts. <p>artifact_removed_interval_list_name</p> <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>artifact_params_name</p> <p>artifact_removed_valid_times</p> <p>artifact_times</p> np array of artifact intervals minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team none =BLOB= =BLOB= <p>Total: 1</p> In\u00a0[32]: Copied! <pre>sgs.SpikeSorterParameters().insert_default()\n\n# Let's look at the default params\nsorter_name = \"mountainsort4\"\nms4_default_params = (\n    sgs.SpikeSorterParameters\n    &amp; {\"sorter\": sorter_name, \"sorter_params_name\": \"default\"}\n).fetch1()\nprint(ms4_default_params)\n</pre> sgs.SpikeSorterParameters().insert_default()  # Let's look at the default params sorter_name = \"mountainsort4\" ms4_default_params = (     sgs.SpikeSorterParameters     &amp; {\"sorter\": sorter_name, \"sorter_params_name\": \"default\"} ).fetch1() print(ms4_default_params) <pre>{'sorter': 'mountainsort4', 'sorter_params_name': 'default', 'sorter_params': {'detect_sign': -1, 'adjacency_radius': -1, 'freq_min': 300, 'freq_max': 6000, 'filter': True, 'whiten': True, 'num_workers': 1, 'clip_size': 50, 'detect_threshold': 3, 'detect_interval': 10}}\n</pre> <p>Now we can change these default parameters to line up more closely with our preferences.</p> In\u00a0[33]: Copied! <pre>sorter_params = {\n    **ms4_default_params[\"sorter_params\"],  # start with defaults\n    \"detect_sign\": -1,  # downward going spikes (1 for upward, 0 for both)\n    \"adjacency_radius\": 100,  # Sort electrodes together within 100 microns\n    \"filter\": False,  # No filter, since we filter prior to starting sort\n    \"freq_min\": 0,\n    \"freq_max\": 0,\n    \"whiten\": False,  # Turn whiten, since we whiten it prior to starting sort\n    \"num_workers\": 4,  #  same number as number of electrodes\n    \"verbose\": True,\n    \"clip_size\": np.int64(\n        1.33e-3  # same as # of samples for 1.33 ms based on the sampling rate\n        * (sgc.Raw &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\"sampling_rate\")\n    ),\n}\nfrom pprint import pprint\n\npprint(sorter_params)\n</pre> sorter_params = {     **ms4_default_params[\"sorter_params\"],  # start with defaults     \"detect_sign\": -1,  # downward going spikes (1 for upward, 0 for both)     \"adjacency_radius\": 100,  # Sort electrodes together within 100 microns     \"filter\": False,  # No filter, since we filter prior to starting sort     \"freq_min\": 0,     \"freq_max\": 0,     \"whiten\": False,  # Turn whiten, since we whiten it prior to starting sort     \"num_workers\": 4,  #  same number as number of electrodes     \"verbose\": True,     \"clip_size\": np.int64(         1.33e-3  # same as # of samples for 1.33 ms based on the sampling rate         * (sgc.Raw &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\"sampling_rate\")     ), } from pprint import pprint  pprint(sorter_params) <pre>{'adjacency_radius': 100,\n 'clip_size': 39,\n 'detect_interval': 10,\n 'detect_sign': -1,\n 'detect_threshold': 3,\n 'filter': False,\n 'freq_max': 0,\n 'freq_min': 0,\n 'num_workers': 4,\n 'verbose': True,\n 'whiten': False}\n</pre> <p>We can give these <code>sorter_params</code> a <code>sorter_params_name</code> and insert into <code>SpikeSorterParameters</code>.</p> In\u00a0[34]: Copied! <pre>sorter_params_name = \"hippocampus_tutorial\"\nsgs.SpikeSorterParameters.insert1(\n    {\n        \"sorter\": sorter_name,\n        \"sorter_params_name\": sorter_params_name,\n        \"sorter_params\": sorter_params,\n    },\n    skip_duplicates=True,\n)\n(\n    sgs.SpikeSorterParameters\n    &amp; {\"sorter\": sorter_name, \"sorter_params_name\": sorter_params_name}\n).fetch1()\n</pre> sorter_params_name = \"hippocampus_tutorial\" sgs.SpikeSorterParameters.insert1(     {         \"sorter\": sorter_name,         \"sorter_params_name\": sorter_params_name,         \"sorter_params\": sorter_params,     },     skip_duplicates=True, ) (     sgs.SpikeSorterParameters     &amp; {\"sorter\": sorter_name, \"sorter_params_name\": sorter_params_name} ).fetch1() Out[34]: <pre>{'sorter': 'mountainsort4',\n 'sorter_params_name': 'hippocampus_tutorial',\n 'sorter_params': {'detect_sign': -1,\n  'adjacency_radius': 100,\n  'freq_min': 0,\n  'freq_max': 0,\n  'filter': False,\n  'whiten': False,\n  'num_workers': 4,\n  'clip_size': 39,\n  'detect_threshold': 3,\n  'detect_interval': 10,\n  'verbose': True}}</pre> In\u00a0[35]: Copied! <pre>ss_key = dict(\n    **(sgs.ArtifactDetection &amp; ssr_key).fetch1(\"KEY\"),\n    **(sgs.ArtifactRemovedIntervalList() &amp; ssr_key).fetch1(\"KEY\"),\n    sorter=sorter_name,\n    sorter_params_name=sorter_params_name,\n)\nss_key.pop(\"artifact_params_name\")\nss_key\n</pre> ss_key = dict(     **(sgs.ArtifactDetection &amp; ssr_key).fetch1(\"KEY\"),     **(sgs.ArtifactRemovedIntervalList() &amp; ssr_key).fetch1(\"KEY\"),     sorter=sorter_name,     sorter_params_name=sorter_params_name, ) ss_key.pop(\"artifact_params_name\") ss_key Out[35]: <pre>{'nwb_file_name': 'minirec20230622_.nwb',\n 'sort_group_id': 0,\n 'sort_interval_name': '01_s1_first9',\n 'preproc_params_name': 'default_hippocampus',\n 'team_name': 'My Team',\n 'artifact_removed_interval_list_name': 'minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times',\n 'sorter': 'mountainsort4',\n 'sorter_params_name': 'hippocampus_tutorial'}</pre> In\u00a0[36]: Copied! <pre>sgs.SpikeSortingSelection.insert1(ss_key, skip_duplicates=True)\n(sgs.SpikeSortingSelection &amp; ss_key)\n</pre> sgs.SpikeSortingSelection.insert1(ss_key, skip_duplicates=True) (sgs.SpikeSortingSelection &amp; ss_key) Out[36]: Table for holding selection of recording and parameters for each spike sorting run <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>import_path</p> optional path to previous curated sorting output minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times <p>Total: 1</p> In\u00a0[39]: Copied! <pre># [(sgs.SpikeSortingSelection &amp; ss_key).proj()]\nsgs.SpikeSorting.populate(ss_key)\n</pre> # [(sgs.SpikeSortingSelection &amp; ss_key).proj()] sgs.SpikeSorting.populate(ss_key) <pre>[11:33:09][INFO] Spyglass: Running spike sorting on {'nwb_file_name': 'minirec20230622_.nwb', 'sort_group_id': 0, 'sort_interval_name': '01_s1_first9', 'preproc_params_name': 'default_hippocampus', 'team_name': 'My Team', 'sorter': 'mountainsort4', 'sorter_params_name': 'hippocampus_tutorial', 'artifact_removed_interval_list_name': 'minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times'}...\n</pre> <pre>Mountainsort4 use the OLD spikeextractors mapped with NewToOldRecording\nUsing temporary directory /stelmo/nwb/tmp/tmpo38gkza9\nUsing 4 workers.\nUsing tempdir: /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06\nNum. workers = 4\nPreparing /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06/timeseries.hdf5...\nPreparing neighborhood sorters (M=3, N=269997)...\nNeighboorhood of channel 1 has 3 channels.\nDetecting events on channel 2 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.035859\nNum events detected on channel 2 (phase1): 334\nComputing PCA features for channel 2 (phase1)...\nClustering for channel 2 (phase1)...\nFound 1 clusters for channel 2 (phase1)...\nComputing templates for channel 2 (phase1)...\nRe-assigning events for channel 2 (phase1)...\nNeighboorhood of channel 2 has 3 channels.\nDetecting events on channel 3 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.036157\nNum events detected on channel 3 (phase1): 463\nComputing PCA features for channel 3 (phase1)...\nClustering for channel 3 (phase1)...\nFound 1 clusters for channel 3 (phase1)...\nComputing templates for channel 3 (phase1)...\nRe-assigning events for channel 3 (phase1)...\nNeighboorhood of channel 0 has 3 channels.\nDetecting events on channel 1 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.033028\nNum events detected on channel 1 (phase1): 341\nComputing PCA features for channel 1 (phase1)...\nClustering for channel 1 (phase1)...\nFound 2 clusters for channel 1 (phase1)...\nComputing templates for channel 1 (phase1)...\nRe-assigning events for channel 1 (phase1)...\nNeighboorhood of channel 1 has 3 channels.\nComputing PCA features for channel 2 (phase2)...\nNo duplicate events found for channel 1 in phase2\nClustering for channel 2 (phase2)...\nFound 1 clusters for channel 2 (phase2)...\nNeighboorhood of channel 2 has 3 channels.\nComputing PCA features for channel 3 (phase2)...\nNo duplicate events found for channel 2 in phase2\nClustering for channel 3 (phase2)...\nFound 1 clusters for channel 3 (phase2)...\nNeighboorhood of channel 0 has 3 channels.\nComputing PCA features for channel 1 (phase2)...\nNo duplicate events found for channel 0 in phase2\nClustering for channel 1 (phase2)...\nFound 1 clusters for channel 1 (phase2)...\nPreparing output...\nDone with ms4alg.\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py:234: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n[11:33:24][INFO] Spyglass: Saving sorting results...\n</pre> <pre>Cleaning tempdir::::: /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06\nmountainsort4 run time 12.62s\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/basesorting.py:212: UserWarning: The registered recording will not be persistent on disk, but only available in memory\n  warnings.warn(\"The registered recording will not be persistent on disk, but only available in memory\")\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up &lt;TemporaryDirectory '/stelmo/nwb/tmp/tmpo38gkza9'&gt;\n  _warnings.warn(warn_message, ResourceWarning)\n</pre> In\u00a0[40]: Copied! <pre>sgs.SpikeSorting() &amp; ss_key\n</pre> sgs.SpikeSorting() &amp; ss_key Out[40]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>sorting_path</p> <p>time_of_sort</p> in Unix time, to the nearest second minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times /stelmo/nwb/spikesorting/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_d318c3f1_spikesorting 1710873204 <p>Total: 1</p> In\u00a0[42]: Copied! <pre>for sorting_key in (sgs.SpikeSorting() &amp; ss_key).fetch(\"KEY\"):\n    # insert_curation will make an entry with a new curation_id regardless of whether it already exists\n    # to avoid this, we check if the curation already exists\n    if not (sgs.Curation() &amp; sorting_key):\n        sgs.Curation.insert_curation(sorting_key)\n\nsgs.Curation() &amp; ss_key\n</pre> for sorting_key in (sgs.SpikeSorting() &amp; ss_key).fetch(\"KEY\"):     # insert_curation will make an entry with a new curation_id regardless of whether it already exists     # to avoid this, we check if the curation already exists     if not (sgs.Curation() &amp; sorting_key):         sgs.Curation.insert_curation(sorting_key)  sgs.Curation() &amp; ss_key Out[42]: Stores each spike sorting; similar to IntervalList <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>parent_curation_id</p> <p>curation_labels</p> a dictionary of labels for the units <p>merge_groups</p> a list of merge groups for the units <p>quality_metrics</p> a list of quality metrics for the units (if available) <p>description</p> optional description for this curated sort <p>time_of_creation</p> in Unix time, to the nearest second 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times -1 =BLOB= =BLOB= =BLOB= 1710873795 <p>Total: 1</p> In\u00a0[45]: Copied! <pre># Parameters used for waveform extraction from the recording\nwaveform_params_name = \"default_whitened\"\n(\n    sgs.WaveformParameters() &amp; {\"waveform_params_name\": waveform_params_name}\n).fetch(as_dict=True)[0]\n</pre> # Parameters used for waveform extraction from the recording waveform_params_name = \"default_whitened\" (     sgs.WaveformParameters() &amp; {\"waveform_params_name\": waveform_params_name} ).fetch(as_dict=True)[0] Out[45]: <pre>{'waveform_params_name': 'default_whitened',\n 'waveform_params': {'ms_before': 0.5,\n  'ms_after': 0.5,\n  'max_spikes_per_unit': 5000,\n  'n_jobs': 5,\n  'total_memory': '5G',\n  'whiten': True}}</pre> In\u00a0[46]: Copied! <pre># extract waveforms\ncuration_keys = [\n    {**k, \"waveform_params_name\": waveform_params_name}\n    for k in (sgs.Curation() &amp; ss_key &amp; {\"curation_id\": 0}).fetch(\"KEY\")\n]\nsgs.WaveformSelection.insert(curation_keys, skip_duplicates=True)\nsgs.Waveforms.populate(ss_key)\n</pre> # extract waveforms curation_keys = [     {**k, \"waveform_params_name\": waveform_params_name}     for k in (sgs.Curation() &amp; ss_key &amp; {\"curation_id\": 0}).fetch(\"KEY\") ] sgs.WaveformSelection.insert(curation_keys, skip_duplicates=True) sgs.Waveforms.populate(ss_key) <pre>[11:48:56][INFO] Spyglass: Extracting waveforms...\n</pre> <pre>extract waveforms memmap:   0%|          | 0/1 [00:00&lt;?, ?it/s]</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f76c8677d00&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f752b912d00&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f76c8677760&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n[11:48:57][INFO] Spyglass: Writing new NWB file minirec20230622_4ZZBN5G9DY.nwb\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/hdmf/build/objectmapper.py:260: DtypeConversionWarning: Spec 'Units/spike_times': Value with data type int64 is being converted to data type float64 as specified.\n  warnings.warn(full_warning_msg, DtypeConversionWarning)\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_4ZZBN5G9DY.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:276: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size &gt; 0` to check that an array is not empty.\n  if check_hash:\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> In\u00a0[58]: Copied! <pre># parameters which define what quality metrics are calculated and how\nmetric_params_name = \"franklab_default3\"\n(sgs.MetricParameters() &amp; {\"metric_params_name\": metric_params_name}).fetch(\n    \"metric_params\"\n)[0]\n</pre> # parameters which define what quality metrics are calculated and how metric_params_name = \"franklab_default3\" (sgs.MetricParameters() &amp; {\"metric_params_name\": metric_params_name}).fetch(     \"metric_params\" )[0] Out[58]: <pre>{'snr': {'peak_sign': 'neg',\n  'random_chunk_kwargs_dict': {'num_chunks_per_segment': 20,\n   'chunk_size': 10000,\n   'seed': 0}},\n 'isi_violation': {'isi_threshold_ms': 1.5, 'min_isi_ms': 0.0},\n 'nn_isolation': {'max_spikes': 1000,\n  'min_spikes': 10,\n  'n_neighbors': 5,\n  'n_components': 7,\n  'radius_um': 100,\n  'seed': 0},\n 'nn_noise_overlap': {'max_spikes': 1000,\n  'min_spikes': 10,\n  'n_neighbors': 5,\n  'n_components': 7,\n  'radius_um': 100,\n  'seed': 0},\n 'peak_channel': {'peak_sign': 'neg'},\n 'num_spikes': {}}</pre> In\u00a0[59]: Copied! <pre>waveform_keys = [\n    {**k, \"metric_params_name\": metric_params_name}\n    for k in (sgs.Waveforms() &amp; ss_key).fetch(\"KEY\")\n]\nsgs.MetricSelection.insert(waveform_keys, skip_duplicates=True)\nsgs.QualityMetrics().populate(ss_key)\nsgs.QualityMetrics() &amp; ss_key\n</pre> waveform_keys = [     {**k, \"metric_params_name\": metric_params_name}     for k in (sgs.Waveforms() &amp; ss_key).fetch(\"KEY\") ] sgs.MetricSelection.insert(waveform_keys, skip_duplicates=True) sgs.QualityMetrics().populate(ss_key) sgs.QualityMetrics() &amp; ss_key <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/postprocessing/template_tools.py:23: DeprecationWarning: The spikeinterface.postprocessing.template_tools is submodule is deprecated.Use spikeinterface.core.template_tools instead\n  _warn()\n[12:17:37][INFO] Spyglass: Computed all metrics: {'snr': {1: 3.627503, 2: 3.598743, 3: 3.6419973}, 'isi_violation': {'1': 0.03896103896103896, '2': 0.036065573770491806, '3': 0.03488372093023256}, 'nn_isolation': {'1': 0.9591503267973855, '2': 0.9594771241830065, '3': 0.9872549019607844}, 'nn_noise_overlap': {'1': 0.49642857142857144, '2': 0.44738562091503264, '3': 0.4}, 'peak_channel': {1: 1, 2: 2, 3: 3}, 'num_spikes': {'1': 309, '2': 306, '3': 431}}\n[12:17:37][INFO] Spyglass: Writing new NWB file minirec20230622_L3O536PHYB.nwb\n[12:17:38][INFO] Spyglass: Adding metric snr : {1: 3.627503, 2: 3.598743, 3: 3.6419973}\n[12:17:38][INFO] Spyglass: Adding metric isi_violation : {'1': 0.03896103896103896, '2': 0.036065573770491806, '3': 0.03488372093023256}\n[12:17:38][INFO] Spyglass: Adding metric nn_isolation : {'1': 0.9591503267973855, '2': 0.9594771241830065, '3': 0.9872549019607844}\n[12:17:38][INFO] Spyglass: Adding metric nn_noise_overlap : {'1': 0.49642857142857144, '2': 0.44738562091503264, '3': 0.4}\n[12:17:38][INFO] Spyglass: Adding metric peak_channel : {1: 1, 2: 2, 3: 3}\n[12:17:38][INFO] Spyglass: Adding metric num_spikes : {'1': 309, '2': 306, '3': 431}\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_L3O536PHYB.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:276: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size &gt; 0` to check that an array is not empty.\n  if check_hash:\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> Out[59]: <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>waveform_params_name</p> name of waveform extraction parameters <p>metric_params_name</p> <p>quality_metrics_path</p> <p>analysis_file_name</p> name of the file <p>object_id</p> Object ID for the metrics in NWB file 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times default_whitened franklab_default3 /stelmo/nwb/waveforms/minirec20230622_.nwb_0105557c_0_default_whitened_waveforms_qm.json minirec20230622_L3O536PHYB.nwb 4b1512bc-861f-4710-8fff-55aad7fbb6ba <p>Total: 1</p> In\u00a0[64]: Copied! <pre># Look at the quality metrics for the first curation\n(sgs.QualityMetrics() &amp; ss_key).fetch_nwb()[0][\"object_id\"]\n</pre> # Look at the quality metrics for the first curation (sgs.QualityMetrics() &amp; ss_key).fetch_nwb()[0][\"object_id\"] <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_L3O536PHYB.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> Out[64]: snr isi_violation nn_isolation nn_noise_overlap peak_channel num_spikes id 1 3.627503 0.038961 0.959150 0.496429 1 309 2 3.598743 0.036066 0.959477 0.447386 2 306 3 3.641997 0.034884 0.987255 0.400000 3 431 In\u00a0[67]: Copied! <pre># We can select our criteria for unit labeling here\nauto_curation_params_name = \"default\"\n(\n    sgs.AutomaticCurationParameters()\n    &amp; {\"auto_curation_params_name\": auto_curation_params_name}\n).fetch1()\n</pre> # We can select our criteria for unit labeling here auto_curation_params_name = \"default\" (     sgs.AutomaticCurationParameters()     &amp; {\"auto_curation_params_name\": auto_curation_params_name} ).fetch1() Out[67]: <pre>{'auto_curation_params_name': 'default',\n 'merge_params': {},\n 'label_params': {'nn_noise_overlap': ['&gt;', 0.1, ['noise', 'reject']]}}</pre> In\u00a0[72]: Copied! <pre># We can now apply the automatic curation criteria to the quality metrics\nmetric_keys = [\n    {**k, \"auto_curation_params_name\": auto_curation_params_name}\n    for k in (sgs.QualityMetrics() &amp; ss_key).fetch(\"KEY\")\n]\nsgs.AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True)\n# populating this table will make a new entry in the curation table\nsgs.AutomaticCuration().populate(ss_key)\nsgs.Curation() &amp; ss_key\n</pre> # We can now apply the automatic curation criteria to the quality metrics metric_keys = [     {**k, \"auto_curation_params_name\": auto_curation_params_name}     for k in (sgs.QualityMetrics() &amp; ss_key).fetch(\"KEY\") ] sgs.AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True) # populating this table will make a new entry in the curation table sgs.AutomaticCuration().populate(ss_key) sgs.Curation() &amp; ss_key Out[72]: Stores each spike sorting; similar to IntervalList <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>parent_curation_id</p> <p>curation_labels</p> a dictionary of labels for the units <p>merge_groups</p> a list of merge groups for the units <p>quality_metrics</p> a list of quality metrics for the units (if available) <p>description</p> optional description for this curated sort <p>time_of_creation</p> in Unix time, to the nearest second 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times -1 =BLOB= =BLOB= =BLOB= 17108737951 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times 0 =BLOB= =BLOB= =BLOB= auto curated 1710876397 <p>Total: 2</p> In\u00a0[82]: Copied! <pre># get the curation keys corresponding to the automatic curation\nauto_curation_key_list = (sgs.AutomaticCuration() &amp; ss_key).fetch(\n    \"auto_curation_key\"\n)\n\n# insert into CuratedSpikeSorting\nfor auto_key in auto_curation_key_list:\n    # get the full key information needed\n    curation_auto_key = (sgs.Curation() &amp; auto_key).fetch1(\"KEY\")\n    sgs.CuratedSpikeSortingSelection.insert1(\n        curation_auto_key, skip_duplicates=True\n    )\nsgs.CuratedSpikeSorting.populate(ss_key)\n\n# Add the curated spike sorting to the SpikeSortingOutput merge table\nkeys_for_merge_tables = (\n    sgs.CuratedSpikeSorting &amp; auto_curation_key_list\n).fetch(\"KEY\")\nSpikeSortingOutput.insert(\n    keys_for_merge_tables,\n    skip_duplicates=True,\n    part_name=\"CuratedSpikeSorting\",\n)\n# Here's our result!\nSpikeSortingOutput.CuratedSpikeSorting() &amp; ss_key\n</pre> # get the curation keys corresponding to the automatic curation auto_curation_key_list = (sgs.AutomaticCuration() &amp; ss_key).fetch(     \"auto_curation_key\" )  # insert into CuratedSpikeSorting for auto_key in auto_curation_key_list:     # get the full key information needed     curation_auto_key = (sgs.Curation() &amp; auto_key).fetch1(\"KEY\")     sgs.CuratedSpikeSortingSelection.insert1(         curation_auto_key, skip_duplicates=True     ) sgs.CuratedSpikeSorting.populate(ss_key)  # Add the curated spike sorting to the SpikeSortingOutput merge table keys_for_merge_tables = (     sgs.CuratedSpikeSorting &amp; auto_curation_key_list ).fetch(\"KEY\") SpikeSortingOutput.insert(     keys_for_merge_tables,     skip_duplicates=True,     part_name=\"CuratedSpikeSorting\", ) # Here's our result! SpikeSortingOutput.CuratedSpikeSorting() &amp; ss_key Out[82]: <p>merge_id</p> <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> 662f3e35-c81e-546c-69c3-b3a2f5ed2776 1 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times <p>Total: 1</p> <p>As of June 2021, members of the Frank Lab can use the <code>sortingview</code> web app for manual curation. To make use of this, we need to populate the <code>CurationFigurl</code> table.</p> <p>We begin by selecting a starting point from the curation entries. In this case we will use the AutomaticCuration populated above as a starting point for manual curation, though you could also  start from the opriginal curation entry by selecting the proper key from the <code>Curation</code> table</p> <p>Note: This step requires setting up your kachery sharing through the sharing notebook</p> In\u00a0[107]: Copied! <pre>starting_curations = (sgs.AutomaticCuration() &amp; ss_key).fetch(\n    \"auto_curation_key\"\n)  # you could also select any key from the sgs.Curation table here\n\nusername = \"username\"\nfig_url_repo = f\"gh://LorenFrankLab/sorting-curations/main/{username}/\"  # settings for franklab members\n\nsort_interval_name = interval_list_name\ngh_url = (\n    fig_url_repo\n    + str(nwb_file_name + \"_\" + sort_interval_name)  # session id\n    + \"/{}\"  # tetrode using auto_id['sort_group_id']\n    + \"/curation.json\"\n)  # url where the curation is stored\n\nfor auto_id in starting_curations:\n    auto_curation_out_key = dict(\n        **(sgs.Curation() &amp; auto_id).fetch1(\"KEY\"),\n        new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),\n    )\n    sgs.CurationFigurlSelection.insert1(\n        auto_curation_out_key, skip_duplicates=True\n    )\n    sgs.CurationFigurl.populate(auto_curation_out_key)\n</pre> starting_curations = (sgs.AutomaticCuration() &amp; ss_key).fetch(     \"auto_curation_key\" )  # you could also select any key from the sgs.Curation table here  username = \"username\" fig_url_repo = f\"gh://LorenFrankLab/sorting-curations/main/{username}/\"  # settings for franklab members  sort_interval_name = interval_list_name gh_url = (     fig_url_repo     + str(nwb_file_name + \"_\" + sort_interval_name)  # session id     + \"/{}\"  # tetrode using auto_id['sort_group_id']     + \"/curation.json\" )  # url where the curation is stored  for auto_id in starting_curations:     auto_curation_out_key = dict(         **(sgs.Curation() &amp; auto_id).fetch1(\"KEY\"),         new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),     )     sgs.CurationFigurlSelection.insert1(         auto_curation_out_key, skip_duplicates=True     )     sgs.CurationFigurl.populate(auto_curation_out_key) <pre>[16:04:49][INFO] Spyglass: Preparing spikesortingview data\n</pre> <pre>Initial pass: segment 0\nSegment 0 of 1\n/stelmo/nwb/kachery-cloud/sha1/de/41/7a/de417af585eda5dc274c1389ad1b28ef1a0580ab\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> <p>We can then access the url for the curation figurl like so:</p> In\u00a0[108]: Copied! <pre>print((sgs.CurationFigurl &amp; ss_key).fetch(\"url\")[0])\n</pre> print((sgs.CurationFigurl &amp; ss_key).fetch(\"url\")[0]) <pre>https://figurl.org/f?v=npm://@fi-sci/figurl-sortingview@12/dist&amp;d=sha1://b0d9355ba302bcbcb7005822796fc850c06b6d3d&amp;s={\"initialSortingCuration\":\"sha1://2800ea072728fd141d8e5bc88525ac0c6c137d04\",\"sortingCuration\":\"gh://LorenFrankLab/sorting-curations/main/sambray/minirec20230622_.nwb_01_s1/0/curation.json\"}&amp;label=minirec20230622_.nwb_01_s1_first9_0_default_hippocampus%20minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_42be1215_spikesorting&amp;zone=franklab.collaborators\n</pre> <p>This will take you to a workspace on the <code>sortingview</code> app. The workspace, which you can think of as a list of recording and associated sorting objects, was created at the end of spike sorting. On the workspace view, you will see a set of recordings that have been added to the workspace.</p> <p></p> <p>Clicking on a recording then takes you to a page that gives you information about the recording as well as the associated sorting objects.</p> <p></p> <p>Click on a sorting to see the curation view. Try exploring the many visualization widgets.</p> <p></p> <p>The most important is the <code>Units Table</code> and the <code>Curation</code> menu, which allows you to give labels to the units. The curation labels will persist even if you suddenly lose connection to the app; this is because the curation actions are appended to the workspace as soon as they are created. Note that if you are not logged in with your Google account, <code>Curation</code> menu may not be visible. Log in and refresh the page to access this feature.</p> <p></p>"}, {"location": "notebooks/10_Spike_SortingV0/#spike-sorting", "title": "Spike Sorting\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#imports", "title": "Imports\u00b6", "text": "<p>Let's start by importing tables from Spyglass and quieting warnings caused by some dependencies.</p> <p>Note: It the imports below throw a <code>FileNotFoundError</code>, make a cell with <code>!env | grep X</code> where X is part of the problematic directory. This will show the variable causing issues. Make another cell that sets this variable elsewhere with <code>%env VAR=\"/your/path/\"</code></p>"}, {"location": "notebooks/10_Spike_SortingV0/#fetch-exercise", "title": "Fetch Exercise\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#adding-an-nwb-file", "title": "Adding an NWB file\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#import-data", "title": "Import Data\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#extracting-the-recording", "title": "Extracting the recording\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#sortgroup", "title": "<code>SortGroup</code>\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#intervallist", "title": "<code>IntervalList</code>\u00b6", "text": "<p>Next, we make a decision about the time interval for our spike sorting using <code>IntervalList</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#sortinterval", "title": "<code>SortInterval</code>\u00b6", "text": "<p>For longer recordings, Spyglass subsets this interval with <code>SortInterval</code>. Below, we select the first <code>n</code> seconds of this interval.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#preprocessing-parameters", "title": "Preprocessing Parameters\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#processing-a-key", "title": "Processing a key\u00b6", "text": "<p>key is often used to describe an entry we want to move through the pipeline, and keys are often managed as dictionaries. Here, we'll manage the spike sort recording key, <code>ssr_key</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#recording-selection", "title": "Recording Selection\u00b6", "text": "<p>We now insert this key <code>SpikeSortingRecordingSelection</code> table to specify what time/tetrode/etc. of the recording we want to extract.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesortingrecording", "title": "<code>SpikeSortingRecording</code>\u00b6", "text": "<p>And now we're ready to extract the recording! The <code>populate</code> command will automatically process data in Computed or Imported table tiers.</p> <p>If we only want to process certain entries, we can grab their primary key with the <code>.proj()</code> command and use a list of primary keys when calling <code>populate</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#artifact-detection", "title": "Artifact Detection\u00b6", "text": "<p><code>ArtifactDetectionParameters</code> establishes the parameters for removing artifacts from the data. We may want to target artifact signal that is within the frequency band of our filter (600Hz-6KHz), and thus will not get removed by filtering.</p> <p>For this demo, we'll use a parameter set to skip this step.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spike-sorting", "title": "Spike sorting\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#spikesorterparameters", "title": "<code>SpikeSorterParameters</code>\u00b6", "text": "<p>For our example, we will be using <code>mountainsort4</code>. There are already some default parameters in the <code>SpikeSorterParameters</code> table we'll <code>fetch</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesortingselection", "title": "<code>SpikeSortingSelection</code>\u00b6", "text": "<p>Gearing up to Spike Sort!</p> <p>We now collect our various keys to insert into <code>SpikeSortingSelection</code>, which is specific to this recording and eventual sorting segment.</p> <p>Note: the spike sorter parameters defined above are specific to <code>mountainsort4</code> and may not work for other sorters.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesorting", "title": "<code>SpikeSorting</code>\u00b6", "text": "<p>After adding to <code>SpikeSortingSelection</code>, we can simply populate <code>SpikeSorting</code>.</p> <p>Note: This may take time with longer data sets. Be sure to <code>pip install mountainsort4</code> if this is your first time spike sorting.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#check-to-make-sure-the-table-populated", "title": "Check to make sure the table populated\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#automatic-curation", "title": "Automatic Curation\u00b6", "text": "<p>Spikesorting algorithms can sometimes identify noise or other undesired features as spiking units. Spyglass provides a curation pipeline to detect and label such features to exclude them from downstream analysis.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#initial-curation", "title": "Initial Curation\u00b6", "text": "<p>The <code>Curation</code> table keeps track of rounds of spikesorting curations in the spikesorting v0 pipeline. Before we begin, we first insert an initial curation entry with the spiking results.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#waveform-extraction", "title": "Waveform Extraction\u00b6", "text": "<p>Some metrics used for curating units are dependent on features of the spike waveform. We extract these for each unit's initial curation here</p>"}, {"location": "notebooks/10_Spike_SortingV0/#quality-metrics", "title": "Quality Metrics\u00b6", "text": "<p>With these waveforms, we can calculate the metrics used to determine the quality of each unit.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#automatic-curation-labeling", "title": "Automatic Curation Labeling\u00b6", "text": "<p>With these metrics, we can assign labels to the sorted units using the <code>AutomaticCuration</code> table</p>"}, {"location": "notebooks/10_Spike_SortingV0/#insert-desired-curation-into-downstream-and-merge-tables-for-future-analysis", "title": "Insert desired curation into downstream and merge tables for future analysis\u00b6", "text": "<p>Now that we've performed auto-curation, we can insert the results of our chosen curation into  <code>CuratedSpikeSorting</code> (the final table of this pipeline), and the merge table <code>SpikeSortingOutput</code>. Downstream analyses such as decoding will access the spiking data from there</p>"}, {"location": "notebooks/10_Spike_SortingV0/#manual-curation-with-figurl", "title": "Manual Curation with figurl\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV1/", "title": "Spike Sorting V1", "text": "<p>This is a tutorial for Spyglass spike sorting pipeline version 1 (V1). This pipeline coexists with version 0 but differs in that:</p> <ul> <li>it stores more of the intermediate results (e.g. filtered and referenced recording) in the NWB format</li> <li>it has more streamlined curation pipelines</li> <li>it uses UUIDs as the primary key for important tables (e.g. <code>SpikeSorting</code>) to reduce the number of keys that make up the composite primary key</li> </ul> <p>The output of both versions of the pipeline are saved in a merge table called <code>SpikeSortingOutput</code>.</p> <p>To start, connect to the database. See instructions in Setup.</p> In\u00a0[\u00a0]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config[\"enable_python_native_blobs\"] = True\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n%load_ext autoreload\n%autoreload 2\n</pre> import os import datajoint as dj import numpy as np  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config[\"enable_python_native_blobs\"] = True dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  %load_ext autoreload %autoreload 2 <p>First, import the pipeline and other necessary modules.</p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.common as sgc\nimport spyglass.spikesorting.v1 as sgs\nimport spyglass.data_import as sgi\n</pre> import spyglass.common as sgc import spyglass.spikesorting.v1 as sgs import spyglass.data_import as sgi <p>We will be using <code>mediumnwb20230802.nwb</code> as our example. As usual, first insert the NWB file into <code>Session</code> (can skip if you have already done so).</p> In\u00a0[\u00a0]: Copied! <pre>nwb_file_name = \"mediumnwb20230802.nwb\"\nnwb_file_name2 = \"mediumnwb20230802_.nwb\"\n</pre> nwb_file_name = \"mediumnwb20230802.nwb\" nwb_file_name2 = \"mediumnwb20230802_.nwb\" In\u00a0[\u00a0]: Copied! <pre>sgi.insert_sessions(nwb_file_name)\n</pre> sgi.insert_sessions(nwb_file_name) In\u00a0[\u00a0]: Copied! <pre>sgc.Session()\n</pre> sgc.Session() <p>Next, we will define the groups of electrodes to sort together. These are stored in <code>SortGroup</code> table.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_file_name2)\n</pre> sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_file_name2) <p>The next step is to filter and reference the recording so that we isolate the spike band data. This is done by combining the data with the parameters in <code>SpikeSortingRecordingSelection</code>. For inserting into this table, use <code>insert_selection</code> method. This automatically generates a UUID for a recording.</p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"nwb_file_name\": nwb_file_name2,\n    \"sort_group_id\": 0,\n    \"preproc_param_name\": \"default\",\n}\n</pre> key = {     \"nwb_file_name\": nwb_file_name2,     \"sort_group_id\": 0,     \"preproc_param_name\": \"default\", } In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecordingSelection.insert_selection(key)\n</pre> sgs.SpikeSortingRecordingSelection.insert_selection(key) <p>Next we will call <code>populate</code> method of <code>SpikeSortingRecording</code>.</p> In\u00a0[\u00a0]: Copied! <pre># Assuming 'key' is a dictionary with fields that you want to include in 'ssr_key'\nssr_key = {\n    \"recording_id\": (sgs.SpikeSortingRecordingSelection() &amp; key).fetch1('recording_id'),\n} | key\n</pre> # Assuming 'key' is a dictionary with fields that you want to include in 'ssr_key' ssr_key = {     \"recording_id\": (sgs.SpikeSortingRecordingSelection() &amp; key).fetch1('recording_id'), } | key In\u00a0[\u00a0]: Copied! <pre>#sgs.SpikeSortingRecording.populate()\nssr_pk = (sgs.SpikeSortingRecordingSelection &amp; key).proj()\n\n\nsgs.SpikeSortingRecording.populate(ssr_pk)\n</pre> #sgs.SpikeSortingRecording.populate() ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; key).proj()   sgs.SpikeSortingRecording.populate(ssr_pk) In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecording() &amp; ssr_key\n</pre> sgs.SpikeSortingRecording() &amp; ssr_key In\u00a0[\u00a0]: Copied! <pre>key = (\n    sgs.SpikeSortingRecordingSelection &amp; key\n).fetch1()\n</pre> key = (     sgs.SpikeSortingRecordingSelection &amp; key ).fetch1() <p>Sometimes the recording may contain artifacts that can confound spike sorting. For example, we often have artifacts when the animal licks the reward well for milk during behavior. These appear as sharp transients across all channels, and sometimes they are not adequately removed by filtering and referencing. We will identify the periods during which this type of artifact appears and set them to zero so that they won't interfere with spike sorting.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetectionSelection.insert_selection(\n    {\"recording_id\": key[\"recording_id\"], \"artifact_param_name\": \"default\"}\n)\n</pre> sgs.ArtifactDetectionSelection.insert_selection(     {\"recording_id\": key[\"recording_id\"], \"artifact_param_name\": \"default\"} ) In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetection.populate()\n</pre> sgs.ArtifactDetection.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetection()\n</pre> sgs.ArtifactDetection() <p>The output of <code>ArtifactDetection</code> is actually stored in <code>IntervalList</code> because it is another type of interval. The UUID however can be found in both.</p> <p>Now that we have prepared the recording, we will pair this with a spike sorting algorithm and associated parameters. This will be inserted to <code>SpikeSortingSelection</code>, again via <code>insert_selection</code> method.</p> <p>The spike sorting pipeline is powered by <code>spikeinterface</code>, a community-developed Python package that enables one to easily apply multiple spike sorters to a single recording. Some spike sorters have special requirements, such as GPU. Others need to be installed separately from spyglass. In the Frank lab, we have been using <code>mountainsort4</code>, though the pipeline have been tested with <code>mountainsort5</code>, <code>kilosort2_5</code>, <code>kilosort3</code>, and <code>ironclust</code> as well.</p> <p>When using <code>mountainsort5</code>, make sure to run <code>pip install mountainsort5</code>. <code>kilosort2_5</code>, <code>kilosort3</code>, and <code>ironclust</code> are MATLAB-based, but we can run these without having to install MATLAB thanks to <code>spikeinterface</code>. It does require downloading additional files (as singularity containers) so make sure to do <code>pip install spython</code>. These sorters also require GPU access, so also do <code>pip install cuda-python</code> (and make sure your computer does have a GPU).</p> In\u00a0[\u00a0]: Copied! <pre>sorter = 'mountainsort4'\n\ncommon_key = {\n    \"recording_id\": key[\"recording_id\"],\n    \"sorter\": sorter,\n    \"nwb_file_name\": nwb_file_name2,\n    \"interval_list_name\": str(\n        (\n            sgs.ArtifactDetectionSelection\n            &amp; {\"recording_id\": key[\"recording_id\"]}\n        ).fetch1(\"artifact_id\")\n    ),\n}\n\nif sorter == 'mountainsort4':\n    key = {\n        **common_key,\n        \"sorter_param_name\": \"franklab_tetrode_hippocampus_30KHz\",\n    }\nelse:\n    key = {\n        **common_key,\n        \"sorter_param_name\": \"default\",\n    }\n</pre> sorter = 'mountainsort4'  common_key = {     \"recording_id\": key[\"recording_id\"],     \"sorter\": sorter,     \"nwb_file_name\": nwb_file_name2,     \"interval_list_name\": str(         (             sgs.ArtifactDetectionSelection             &amp; {\"recording_id\": key[\"recording_id\"]}         ).fetch1(\"artifact_id\")     ), }  if sorter == 'mountainsort4':     key = {         **common_key,         \"sorter_param_name\": \"franklab_tetrode_hippocampus_30KHz\",     } else:     key = {         **common_key,         \"sorter_param_name\": \"default\",     } In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingSelection.insert_selection(key)\n</pre> sgs.SpikeSortingSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingSelection() &amp; key\n</pre> sgs.SpikeSortingSelection() &amp; key <p>Once <code>SpikeSortingSelection</code> is populated, let's run <code>SpikeSorting.populate</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sss_pk = (sgs.SpikeSortingSelection &amp; key).proj()\n\nsgs.SpikeSorting.populate(sss_pk)\n</pre> sss_pk = (sgs.SpikeSortingSelection &amp; key).proj()  sgs.SpikeSorting.populate(sss_pk) <p>The spike sorting results (spike times of detected units) are saved in an NWB file. We can access this in two ways. First, we can access it via the <code>fetch_nwb</code> method, which allows us to directly access the spike times saved in the <code>units</code> table of the NWB file. Second, we can access it as a <code>spikeinterface.NWBSorting</code> object. Ths allows us to take advantage of the rich APIs of <code>spikeinterface</code> to further analyze the sorting.</p> In\u00a0[\u00a0]: Copied! <pre>sorting_nwb = (sgs.SpikeSorting &amp; key).fetch_nwb()\nsorting_si = sgs.SpikeSorting.get_sorting(key)\n</pre> sorting_nwb = (sgs.SpikeSorting &amp; key).fetch_nwb() sorting_si = sgs.SpikeSorting.get_sorting(key) <p>Note that the spike times of <code>fetch_nwb</code> is in units of seconds aligned with the timestamps of the recording. The spike times of the <code>spikeinterface.NWBSorting</code> object is in units of samples (as is generally true for sorting objects in <code>spikeinterface</code>).</p> <p>Next step is to curate the results of spike sorting. This is often necessary because spike sorting algorithms are not perfect; they often return clusters that are clearly not biological in origin, and sometimes oversplit clusters that should have been merged. We have two main ways of curating spike sorting: by computing quality metrics followed by thresholding, and manually applying curation labels. To do either, we first insert the spike sorting to <code>CurationV1</code> using <code>insert_curation</code> method.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecording &amp; key\n</pre> sgs.SpikeSortingRecording &amp; key In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=(\n        sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}\n    ).fetch1(\"sorting_id\"),\n    description=\"testing sort\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=(         sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}     ).fetch1(\"sorting_id\"),     description=\"testing sort\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>We will first do an automatic curation based on quality metrics. Under the hood, this part again makes use of <code>spikeinterface</code>. Some of the quality metrics that we often compute are the nearest neighbor isolation and noise overlap metrics, as well as SNR and ISI violation rate. For computing some of these metrics, the waveforms must be extracted and projected onto a feature space. Thus here we set the parameters for waveform extraction as well as how to curate the units based on these metrics (e.g. if <code>nn_noise_overlap</code> is greater than 0.1, mark as <code>noise</code>).</p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"sorting_id\": (\n        sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}\n    ).fetch1(\"sorting_id\"),\n    \"curation_id\": 0,\n    \"waveform_param_name\": \"default_not_whitened\",\n    \"metric_param_name\": \"franklab_default\",\n    \"metric_curation_param_name\": \"default\",\n}\n</pre> key = {     \"sorting_id\": (         sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}     ).fetch1(\"sorting_id\"),     \"curation_id\": 0,     \"waveform_param_name\": \"default_not_whitened\",     \"metric_param_name\": \"franklab_default\",     \"metric_curation_param_name\": \"default\", } In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCurationSelection.insert_selection(key)\n</pre> sgs.MetricCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCurationSelection()\n</pre> sgs.MetricCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCuration.populate()\n</pre> sgs.MetricCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCuration()\n</pre> sgs.MetricCuration() <p>to do another round of curation, fetch the relevant info and insert back into CurationV1 using <code>insert_curation</code></p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"metric_curation_id\": (\n        sgs.MetricCurationSelection &amp; {\"sorting_id\": key[\"sorting_id\"]}\n    ).fetch1(\"metric_curation_id\")\n}\n</pre> key = {     \"metric_curation_id\": (         sgs.MetricCurationSelection &amp; {\"sorting_id\": key[\"sorting_id\"]}     ).fetch1(\"metric_curation_id\") } In\u00a0[\u00a0]: Copied! <pre>labels = sgs.MetricCuration.get_labels(key)\n</pre> labels = sgs.MetricCuration.get_labels(key) In\u00a0[\u00a0]: Copied! <pre>merge_groups = sgs.MetricCuration.get_merge_groups(key)\n</pre> merge_groups = sgs.MetricCuration.get_merge_groups(key) In\u00a0[\u00a0]: Copied! <pre>metrics = sgs.MetricCuration.get_metrics(key)\n</pre> metrics = sgs.MetricCuration.get_metrics(key) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=(\n        sgs.MetricCurationSelection\n        &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n    ).fetch1(\"sorting_id\"),\n    parent_curation_id=0,\n    labels=labels,\n    merge_groups=merge_groups,\n    metrics=metrics,\n    description=\"after metric curation\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=(         sgs.MetricCurationSelection         &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}     ).fetch1(\"sorting_id\"),     parent_curation_id=0,     labels=labels,     merge_groups=merge_groups,     metrics=metrics,     description=\"after metric curation\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>next we will do manual curation. this is done with figurl. to incorporate info from other stages of processing (e.g. metrics) we have to store that with kachery cloud and get curation uri referring to it. it can be done with <code>generate_curation_uri</code>.</p> In\u00a0[\u00a0]: Copied! <pre>curation_uri = sgs.FigURLCurationSelection.generate_curation_uri(\n    {\n        \"sorting_id\": (\n            sgs.MetricCurationSelection\n            &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n        ).fetch1(\"sorting_id\"),\n        \"curation_id\": 1,\n    }\n)\n</pre> curation_uri = sgs.FigURLCurationSelection.generate_curation_uri(     {         \"sorting_id\": (             sgs.MetricCurationSelection             &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}         ).fetch1(\"sorting_id\"),         \"curation_id\": 1,     } ) In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"sorting_id\": (\n        sgs.MetricCurationSelection\n        &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n    ).fetch1(\"sorting_id\"),\n    \"curation_id\": 1,\n    \"curation_uri\": curation_uri,\n    \"metrics_figurl\": list(metrics.keys()),\n}\n</pre> key = {     \"sorting_id\": (         sgs.MetricCurationSelection         &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}     ).fetch1(\"sorting_id\"),     \"curation_id\": 1,     \"curation_uri\": curation_uri,     \"metrics_figurl\": list(metrics.keys()), } In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection()\n</pre> sgs.FigURLCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection.insert_selection(key)\n</pre> sgs.FigURLCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection()\n</pre> sgs.FigURLCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration.populate()\n</pre> sgs.FigURLCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration()\n</pre> sgs.FigURLCuration() <p>or you can manually specify it if you already have a <code>curation.json</code></p> In\u00a0[\u00a0]: Copied! <pre>gh_curation_uri = (\n    \"gh://LorenFrankLab/sorting-curations/main/khl02007/test/curation.json\"\n)\n\nkey = {\n    \"sorting_id\": key[\"sorting_id\"],\n    \"curation_id\": 1,\n    \"curation_uri\": gh_curation_uri,\n    \"metrics_figurl\": [],\n}\n</pre> gh_curation_uri = (     \"gh://LorenFrankLab/sorting-curations/main/khl02007/test/curation.json\" )  key = {     \"sorting_id\": key[\"sorting_id\"],     \"curation_id\": 1,     \"curation_uri\": gh_curation_uri,     \"metrics_figurl\": [], } In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection.insert_selection(key)\n</pre> sgs.FigURLCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration.populate()\n</pre> sgs.FigURLCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration()\n</pre> sgs.FigURLCuration() <p>once you apply manual curation (curation labels and merge groups) you can store them as nwb by inserting another row in CurationV1. And then you can do more rounds of curation if you want.</p> In\u00a0[\u00a0]: Copied! <pre>labels = sgs.FigURLCuration.get_labels(gh_curation_uri)\n</pre> labels = sgs.FigURLCuration.get_labels(gh_curation_uri) In\u00a0[\u00a0]: Copied! <pre>merge_groups = sgs.FigURLCuration.get_merge_groups(gh_curation_uri)\n</pre> merge_groups = sgs.FigURLCuration.get_merge_groups(gh_curation_uri) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=key[\"sorting_id\"],\n    parent_curation_id=1,\n    labels=labels,\n    merge_groups=merge_groups,\n    metrics=metrics,\n    description=\"after figurl curation\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=key[\"sorting_id\"],     parent_curation_id=1,     labels=labels,     merge_groups=merge_groups,     metrics=metrics,     description=\"after figurl curation\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>We now insert the curated spike sorting to a <code>Merge</code> table for feeding into downstream processing pipelines.</p> In\u00a0[\u00a0]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput()\n</pre> SpikeSortingOutput() In\u00a0[\u00a0]: Copied! <pre>key\n</pre> key In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.insert([key], part_name=\"CurationV1\")\n</pre> SpikeSortingOutput.insert([key], part_name=\"CurationV1\") In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.merge_view()\n</pre> SpikeSortingOutput.merge_view() In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.CurationV1()\n</pre> SpikeSortingOutput.CurationV1() In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.CuratedSpikeSorting()\n</pre> SpikeSortingOutput.CuratedSpikeSorting() In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/10_Spike_SortingV1/#spike-sorting-pipeline-version-1", "title": "Spike Sorting: pipeline version 1\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/", "title": "Position Trodes", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>In this tutorial, we'll process position data extracted with Trodes Tracking by</p> <ul> <li>Defining parameters</li> <li>Processing raw position</li> <li>Extracting centroid and orientation</li> <li>Insert the results into the <code>TrodesPosV1</code> table</li> <li>Plotting the head position/direction results for quality assurance</li> </ul> <p>The pipeline takes the 2D video pixel data of green/red LEDs, and computes:</p> <ul> <li>head position (in cm)</li> <li>head orientation (in radians)</li> <li>head velocity (in cm/s)</li> <li>head speed (in cm/s)</li> </ul> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position as sgp\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position as sgp  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-12 13:47:50,578][INFO]: Connecting root@localhost:3306\n[2024-01-12 13:47:50,652][INFO]: Connected root@localhost:3306\n</pre> <p>First, we'll grab let us make sure that the session we want to analyze is inserted into the <code>RawPosition</code> table</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\n# Define the name of the file that you copied and renamed\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  # Define the name of the file that you copied and renamed nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) In\u00a0[3]: Copied! <pre>sgc.common_behav.RawPosition() &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.common_behav.RawPosition() &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[3]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb pos 0 valid timesminirec20230622_.nwb pos 1 valid times <p>Total: 2</p> <p>Parameters are set by the <code>TrodesPosParams</code> table, with a <code>default</code> set available. To adjust the default, insert a new set into this table. The parameters are...</p> <ul> <li><code>max_separation</code>, default 9 cm: maximum acceptable distance between red and green LEDs.<ul> <li>If exceeded, the times are marked as NaNs and inferred by interpolation.</li> <li>Useful when the inferred LED position tracks a reflection instead of the true position.</li> </ul> </li> <li><code>max_speed</code>, default 300.0 cm/s: maximum speed the animal can move.<ul> <li>If exceeded, times are marked as NaNs and inferred by interpolation.</li> <li>Useful to prevent big jumps in position.</li> </ul> </li> <li><code>position_smoothing_duration</code>, default 0.100 s: LED position smoothing before computing average position to get head position.</li> <li><code>speed_smoothing_std_dev</code>, default 0.100 s: standard deviation of the Gaussian kernel used to smooth the head speed.</li> <li><code>front_led1</code>, default 1 (True), use <code>xloc</code>/<code>yloc</code>: Which LED is the front LED for calculating the head direction.<ul> <li>1: LED corresponding to <code>xloc</code>, <code>yloc</code> in the <code>RawPosition</code> table is the front, <code>xloc2</code>, <code>yloc2</code> as the back.</li> <li>0: LED corresponding to <code>xloc2</code>, <code>yloc2</code> in the <code>RawPosition</code> table is the front, <code>xloc</code>, <code>yloc</code> as the back.</li> </ul> </li> </ul> <p>We can see these defaults with <code>TrodesPosParams().default_params</code>.</p> In\u00a0[4]: Copied! <pre>from pprint import pprint\n\nparameters = sgp.v1.TrodesPosParams().default_params\npprint(parameters)\n</pre> from pprint import pprint  parameters = sgp.v1.TrodesPosParams().default_params pprint(parameters) <pre>{'is_upsampled': 0,\n 'led1_is_front': 1,\n 'max_LED_separation': 9.0,\n 'max_plausible_speed': 300.0,\n 'orient_smoothing_std_dev': 0.001,\n 'position_smoothing_duration': 0.125,\n 'speed_smoothing_std_dev': 0.1,\n 'upsampling_interpolation_method': 'linear',\n 'upsampling_sampling_rate': None}\n</pre> <p>For the <code>minirec</code> demo file, only one LED is moving. The following paramset will allow us to process this data.</p> In\u00a0[5]: Copied! <pre>trodes_params_name = \"single_led\"\ntrodes_params = {\n    \"max_separation\": 10000.0,\n    \"max_speed\": 300.0,\n    \"position_smoothing_duration\": 0.125,\n    \"speed_smoothing_std_dev\": 0.1,\n    \"orient_smoothing_std_dev\": 0.001,\n    \"led1_is_front\": 1,\n    \"is_upsampled\": 0,\n    \"upsampling_sampling_rate\": None,\n    \"upsampling_interpolation_method\": \"linear\",\n}\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": trodes_params_name,\n        \"params\": trodes_params,\n    },\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosParams()\n</pre> trodes_params_name = \"single_led\" trodes_params = {     \"max_separation\": 10000.0,     \"max_speed\": 300.0,     \"position_smoothing_duration\": 0.125,     \"speed_smoothing_std_dev\": 0.1,     \"orient_smoothing_std_dev\": 0.001,     \"led1_is_front\": 1,     \"is_upsampled\": 0,     \"upsampling_sampling_rate\": None,     \"upsampling_interpolation_method\": \"linear\", } sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": trodes_params_name,         \"params\": trodes_params,     },     skip_duplicates=True, ) sgp.v1.TrodesPosParams() Out[5]: <p>trodes_pos_params_name</p> name for this set of parameters <p>params</p> default =BLOB=single_led =BLOB=single_led_upsampled =BLOB= <p>Total: 3</p> <p>Later, we'll pair the above parameters with an interval from our NWB file and insert into <code>TrodesPosSelection</code>.</p> <p>First, let's select an interval from the <code>IntervalList</code> table.</p> In\u00a0[6]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[6]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval minirec20230622_.nwb 01_s1 =BLOB=minirec20230622_.nwb 02_s2 =BLOB=minirec20230622_.nwb pos 0 valid times =BLOB=minirec20230622_.nwb pos 1 valid times =BLOB=minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 5</p> <p>The raw position in pixels is in the <code>RawPosition</code> table is extracted from the video data by the algorithm in Trodes. We have timepoints available for the duration when position tracking was turned on and off, which may be a subset of the video itself.</p> <p><code>fetch1_dataframe</code> returns the position of the LEDs as a pandas dataframe where time is the index.</p> In\u00a0[7]: Copied! <pre>interval_list_name = \"pos 0 valid times\"  # pos # is epoch # minus 1\nraw_position_df = (\n    sgc.RawPosition()\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": interval_list_name,\n    }\n).fetch1_dataframe()\nraw_position_df\n</pre> interval_list_name = \"pos 0 valid times\"  # pos # is epoch # minus 1 raw_position_df = (     sgc.RawPosition()     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": interval_list_name,     } ).fetch1_dataframe() raw_position_df <pre>[2024-01-12 13:47:54,251][WARNING]: Skipped checksum for file with hash: cce88743-51b1-5ad9-836a-260c938383dd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/minirec20230622_.nwb\n[2024-01-12 13:47:54,254][WARNING]: Skipped checksum for file with hash: cce88743-51b1-5ad9-836a-260c938383dd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/minirec20230622_.nwb\n</pre> Out[7]: xloc1 yloc1 xloc2 yloc2 time 1.687475e+09 445 567 0 0 1.687475e+09 445 567 0 0 1.687475e+09 445 567 0 0 1.687475e+09 444 568 0 0 1.687475e+09 444 568 0 0 ... ... ... ... ... 1.687475e+09 479 536 0 0 1.687475e+09 480 534 0 0 1.687475e+09 480 533 0 0 1.687475e+09 481 530 0 0 1.687475e+09 481 530 0 0 <p>267 rows \u00d7 4 columns</p> <p>Let's just quickly plot the two LEDs to get a sense of the inputs to the pipeline:</p> In\u00a0[8]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 10))\nax.plot(raw_position_df.xloc1, raw_position_df.yloc1, color=\"green\")\n# Uncomment for multiple LEDs\n# ax.plot(raw_position_df.xloc2, raw_position_df.yloc2, color=\"red\")\nax.set_xlabel(\"x-position [pixels]\", fontsize=18)\nax.set_ylabel(\"y-position [pixels]\", fontsize=18)\nax.set_title(\"Raw Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 10)) ax.plot(raw_position_df.xloc1, raw_position_df.yloc1, color=\"green\") # Uncomment for multiple LEDs # ax.plot(raw_position_df.xloc2, raw_position_df.yloc2, color=\"red\") ax.set_xlabel(\"x-position [pixels]\", fontsize=18) ax.set_ylabel(\"y-position [pixels]\", fontsize=18) ax.set_title(\"Raw Position\", fontsize=28) Out[8]: <pre>Text(0.5, 1.0, 'Raw Position')</pre> <p>To associate a set of parameters with a given interval, insert them into the <code>TrodesPosSelection</code> table.</p> In\u00a0[9]: Copied! <pre>trodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": trodes_params_name,\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\n</pre> trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": trodes_params_name, } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) <p>Now let's check to see if we've inserted the parameters correctly:</p> In\u00a0[10]: Copied! <pre>trodes_key = (sgp.v1.TrodesPosSelection() &amp; trodes_s_key).fetch1(\"KEY\")\n</pre> trodes_key = (sgp.v1.TrodesPosSelection() &amp; trodes_s_key).fetch1(\"KEY\") <p>We can run the pipeline for our chosen interval/parameters by using the <code>TrodesPosV1.populate</code>.</p> In\u00a0[11]: Copied! <pre>sgp.v1.TrodesPosV1.populate(trodes_key)\n</pre> sgp.v1.TrodesPosV1.populate(trodes_key) <p>Each NWB file, interval, and parameter set is now associated with a new analysis file and object ID.</p> In\u00a0[12]: Copied! <pre>sgp.v1.TrodesPosV1 &amp; trodes_key\n</pre> sgp.v1.TrodesPosV1 &amp; trodes_key Out[12]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>position_object_id</p> <p>orientation_object_id</p> <p>velocity_object_id</p> minirec20230622_.nwb pos 0 valid times single_led minirec20230622_AQQP7U6Y24.nwb f519d1e4-0919-492a-85a4-6730cce26c10 6ae01b40-f5d9-4dd1-9203-76881d7bd339 002ce1f0-50a6-40b5-8b97-4b5a80140193 <p>Total: 1</p> <p>When we populatethe <code>TrodesPosV1</code> table, we automatically create an entry in the <code>PositionOutput</code> merge table. Since this table supports position information from multiple methods, it's best practive to access data through here.</p> <p>We can view the entry in this table:</p> In\u00a0[13]: Copied! <pre>from spyglass.position import PositionOutput\n\nPositionOutput.TrodesPosV1 &amp; trodes_key\n</pre> from spyglass.position import PositionOutput  PositionOutput.TrodesPosV1 &amp; trodes_key Out[13]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters 15055247-04ba-ab0f-3b67-522f0ecc17e3 minirec20230622_.nwb pos 0 valid times single_led <p>Total: 1</p> <p>To retrieve the results as a pandas DataFrame with time as the index, we use <code>PositionOutput.fetch1_dataframe</code>. When doing so, we need to restric the merge table by the</p> <p>This dataframe has the following columns:</p> <ul> <li><code>position_{x,y}</code>: X or Y position of the head in cm.</li> <li><code>orientation</code>: Direction of the head relative to the bottom left corner in radians</li> <li><code>velocity_{x,y}</code>: Directional change in head position over time in cm/s</li> <li><code>speed</code>: the magnitude of the change in head position over time in cm/s</li> </ul> In\u00a0[14]: Copied! <pre># get the merge id corresponding to our inserted trodes_key\nmerge_key = (PositionOutput.merge_get_part(trodes_key)).fetch1(\"KEY\")\n# use this to restrict PositionOutput and fetch the data\nposition_info = (PositionOutput &amp; merge_key).fetch1_dataframe()\nposition_info\n</pre> # get the merge id corresponding to our inserted trodes_key merge_key = (PositionOutput.merge_get_part(trodes_key)).fetch1(\"KEY\") # use this to restrict PositionOutput and fetch the data position_info = (PositionOutput &amp; merge_key).fetch1_dataframe() position_info <pre>[2024-01-12 13:47:55,378][WARNING]: Skipped checksum for file with hash: c87c4027-855f-0181-d477-cf78242a7c20, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/minirec20230622/minirec20230622_AQQP7U6Y24.nwb\n</pre> Out[14]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.687475e+09 0 22.250000 28.350000 0.905373 -0.148627 0.116866 0.189071 1.687475e+09 1 22.250000 28.350000 0.905373 -0.219948 0.162667 0.273565 1.687475e+09 2 22.250000 28.350000 0.905373 -0.298575 0.205042 0.362201 1.687475e+09 3 22.233333 28.366667 0.906022 -0.371837 0.233439 0.439041 1.687475e+09 4 22.216667 28.383333 0.906671 -0.424639 0.238911 0.487234 ... ... ... ... ... ... ... ... 1.687475e+09 262 23.766667 27.066667 0.850225 2.738364 -3.998619 4.846400 1.687475e+09 263 23.933333 26.800000 0.841843 2.347307 -3.708857 4.389245 1.687475e+09 264 23.983333 26.716667 0.839258 1.907429 -3.237540 3.757652 1.687475e+09 265 24.016667 26.616667 0.836703 1.457185 -2.647347 3.021893 1.687475e+09 266 24.033333 26.550000 0.835110 1.040384 -2.021305 2.273340 <p>267 rows \u00d7 7 columns</p> <p><code>.index</code> on the pandas dataframe gives us timestamps.</p> In\u00a0[15]: Copied! <pre>position_info.index\n</pre> position_info.index Out[15]: <pre>Index([1687474800.1833298,  1687474800.216676,  1687474800.250001,\n        1687474800.283326,  1687474800.316672, 1687474801.8658931,\n       1687474801.8992178, 1687474801.9325643,  1687474801.965889,\n        1687474801.999214,\n       ...\n        1687474810.265809,  1687474810.299134, 1687474810.3324802,\n       1687474810.3658051,   1687474810.39913,  1687474810.432476,\n        1687474810.465801,  1687474810.499126, 1687474810.5324721,\n        1687474810.565797],\n      dtype='float64', name='time', length=267)</pre> <p>We should always spot check our results to verify that the pipeline worked correctly.</p> <p>Let's plot some of the variables first:</p> In\u00a0[16]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(8, 8))\nax.plot(position_info.position_x, position_info.position_y)\nax.set_xlabel(\"x-position [cm]\", fontsize=16)\nax.set_ylabel(\"y-position [cm]\", fontsize=16)\nax.set_title(\"Position\", fontsize=20)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(8, 8)) ax.plot(position_info.position_x, position_info.position_y) ax.set_xlabel(\"x-position [cm]\", fontsize=16) ax.set_ylabel(\"y-position [cm]\", fontsize=16) ax.set_title(\"Position\", fontsize=20) Out[16]: <pre>Text(0.5, 1.0, 'Position')</pre> In\u00a0[17]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(8, 8))\nax.plot(position_info.velocity_x, position_info.velocity_y)\nax.set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\nax.set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\nax.set_title(\"Velocity\", fontsize=20)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(8, 8)) ax.plot(position_info.velocity_x, position_info.velocity_y) ax.set_xlabel(\"x-velocity [cm/s]\", fontsize=16) ax.set_ylabel(\"y-velocity [cm/s]\", fontsize=16) ax.set_title(\"Velocity\", fontsize=20) Out[17]: <pre>Text(0.5, 1.0, 'Velocity')</pre> In\u00a0[18]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(16, 3))\nax.plot(position_info.index, position_info.speed)\nax.set_xlabel(\"Time\", fontsize=16)\nax.set_ylabel(\"Speed [cm/s]\", fontsize=16)\nax.set_title(\"Head Speed\", fontsize=20)\nax.set_xlim((position_info.index.min(), position_info.index.max()))\n</pre> fig, ax = plt.subplots(1, 1, figsize=(16, 3)) ax.plot(position_info.index, position_info.speed) ax.set_xlabel(\"Time\", fontsize=16) ax.set_ylabel(\"Speed [cm/s]\", fontsize=16) ax.set_title(\"Head Speed\", fontsize=20) ax.set_xlim((position_info.index.min(), position_info.index.max())) Out[18]: <pre>(1687474800.1833298, 1687474810.565797)</pre> In\u00a0[19]: Copied! <pre># sgp.v1.TrodesPosVideo().populate(\n#     {\n#         \"nwb_file_name\": nwb_copy_file_name,\n#         \"interval_list_name\": interval_list_name,\n#         \"position_info_param_name\": trodes_params_name,\n#     }\n# )\n</pre> # sgp.v1.TrodesPosVideo().populate( #     { #         \"nwb_file_name\": nwb_copy_file_name, #         \"interval_list_name\": interval_list_name, #         \"position_info_param_name\": trodes_params_name, #     } # ) In\u00a0[20]: Copied! <pre># sgp.v1.TrodesPosVideo()\n</pre> # sgp.v1.TrodesPosVideo() In\u00a0[21]: Copied! <pre>trodes_params_up_name = trodes_params_name + \"_upsampled\"\ntrodes_params_up = {\n    **trodes_params,\n    \"is_upsampled\": 1,\n    \"upsampling_sampling_rate\": 500,\n}\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": trodes_params_up_name,\n        \"params\": trodes_params_up,\n    },\n    skip_duplicates=True,\n)\n\nsgp.v1.TrodesPosParams()\n</pre> trodes_params_up_name = trodes_params_name + \"_upsampled\" trodes_params_up = {     **trodes_params,     \"is_upsampled\": 1,     \"upsampling_sampling_rate\": 500, } sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": trodes_params_up_name,         \"params\": trodes_params_up,     },     skip_duplicates=True, )  sgp.v1.TrodesPosParams() Out[21]: <p>trodes_pos_params_name</p> name for this set of parameters <p>params</p> default =BLOB=single_led =BLOB=single_led_upsampled =BLOB= <p>Total: 3</p> In\u00a0[22]: Copied! <pre>trodes_s_up_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": trodes_params_up_name,\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_up_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_up_key)\n</pre> trodes_s_up_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": trodes_params_up_name, } sgp.v1.TrodesPosSelection.insert1(     trodes_s_up_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_up_key) In\u00a0[23]: Copied! <pre>merge_key = (PositionOutput.merge_get_part(trodes_s_up_key)).fetch1(\"KEY\")\nupsampled_position_info = (PositionOutput &amp; merge_key).fetch1_dataframe()\nupsampled_position_info\n</pre> merge_key = (PositionOutput.merge_get_part(trodes_s_up_key)).fetch1(\"KEY\") upsampled_position_info = (PositionOutput &amp; merge_key).fetch1_dataframe() upsampled_position_info <pre>[13:47:56][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[2024-01-12 13:47:56,476][WARNING]: Skipped checksum for file with hash: 119a4889-1117-30a9-c774-3c7db7048f02, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/minirec20230622/minirec20230622_PBPM9HN98Y.nwb\n</pre> Out[23]: position_x position_y orientation velocity_x velocity_y speed time 1.687475e+09 22.250000 28.350000 0.808756 -0.083265 0.082262 0.117047 1.687475e+09 22.250000 28.350000 0.905134 -0.084928 0.083888 0.119373 1.687475e+09 22.250000 28.350000 0.905373 -0.086591 0.085514 0.121699 1.687475e+09 22.250000 28.350000 0.905373 -0.088254 0.087138 0.124023 1.687475e+09 22.250000 28.350000 0.905373 -0.089915 0.088760 0.126345 ... ... ... ... ... ... ... 1.687475e+09 24.029412 26.565686 0.835485 1.095035 -2.036613 2.312335 1.687475e+09 24.030392 26.561765 0.835391 1.070802 -1.998650 2.267425 1.687475e+09 24.031373 26.557843 0.835298 1.046820 -1.960860 2.222792 1.687475e+09 24.032353 26.553922 0.834983 1.023096 -1.923260 2.178452 1.687475e+09 24.033333 26.550000 0.746002 0.999636 -1.885865 2.134422 <p>5193 rows \u00d7 6 columns</p> In\u00a0[24]: Copied! <pre>fig, axes = plt.subplots(\n    1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.position_x, position_info.position_y)\naxes[0].set_xlabel(\"x-position [cm]\", fontsize=16)\naxes[0].set_ylabel(\"y-position [cm]\", fontsize=16)\naxes[0].set_title(\"Position\", fontsize=20)\n\naxes[1].plot(\n    upsampled_position_info.position_x,\n    upsampled_position_info.position_y,\n)\naxes[1].set_xlabel(\"x-position [cm]\", fontsize=16)\naxes[1].set_ylabel(\"y-position [cm]\", fontsize=16)\naxes[1].set_title(\"Upsampled Position\", fontsize=20)\n</pre> fig, axes = plt.subplots(     1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.position_x, position_info.position_y) axes[0].set_xlabel(\"x-position [cm]\", fontsize=16) axes[0].set_ylabel(\"y-position [cm]\", fontsize=16) axes[0].set_title(\"Position\", fontsize=20)  axes[1].plot(     upsampled_position_info.position_x,     upsampled_position_info.position_y, ) axes[1].set_xlabel(\"x-position [cm]\", fontsize=16) axes[1].set_ylabel(\"y-position [cm]\", fontsize=16) axes[1].set_title(\"Upsampled Position\", fontsize=20) Out[24]: <pre>Text(0.5, 1.0, 'Upsampled Position')</pre> In\u00a0[25]: Copied! <pre>fig, axes = plt.subplots(\n    2, 1, figsize=(16, 6), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.index, position_info.speed)\naxes[0].set_xlabel(\"Time\", fontsize=16)\naxes[0].set_ylabel(\"Speed [cm/s]\", fontsize=16)\naxes[0].set_title(\"Speed\", fontsize=20)\naxes[0].set_xlim((position_info.index.min(), position_info.index.max()))\n\naxes[1].plot(upsampled_position_info.index, upsampled_position_info.speed)\naxes[1].set_xlabel(\"Time\", fontsize=16)\naxes[1].set_ylabel(\"Speed [cm/s]\", fontsize=16)\naxes[1].set_title(\"Upsampled Speed\", fontsize=20)\n</pre> fig, axes = plt.subplots(     2, 1, figsize=(16, 6), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.index, position_info.speed) axes[0].set_xlabel(\"Time\", fontsize=16) axes[0].set_ylabel(\"Speed [cm/s]\", fontsize=16) axes[0].set_title(\"Speed\", fontsize=20) axes[0].set_xlim((position_info.index.min(), position_info.index.max()))  axes[1].plot(upsampled_position_info.index, upsampled_position_info.speed) axes[1].set_xlabel(\"Time\", fontsize=16) axes[1].set_ylabel(\"Speed [cm/s]\", fontsize=16) axes[1].set_title(\"Upsampled Speed\", fontsize=20) Out[25]: <pre>Text(0.5, 1.0, 'Upsampled Speed')</pre> In\u00a0[26]: Copied! <pre>fig, axes = plt.subplots(\n    1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.velocity_x, position_info.velocity_y)\naxes[0].set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\naxes[0].set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\naxes[0].set_title(\"Velocity\", fontsize=20)\n\naxes[1].plot(\n    upsampled_position_info.velocity_x,\n    upsampled_position_info.velocity_y,\n)\naxes[1].set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\naxes[1].set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\naxes[1].set_title(\"Upsampled Velocity\", fontsize=20)\n</pre> fig, axes = plt.subplots(     1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.velocity_x, position_info.velocity_y) axes[0].set_xlabel(\"x-velocity [cm/s]\", fontsize=16) axes[0].set_ylabel(\"y-velocity [cm/s]\", fontsize=16) axes[0].set_title(\"Velocity\", fontsize=20)  axes[1].plot(     upsampled_position_info.velocity_x,     upsampled_position_info.velocity_y, ) axes[1].set_xlabel(\"x-velocity [cm/s]\", fontsize=16) axes[1].set_ylabel(\"y-velocity [cm/s]\", fontsize=16) axes[1].set_title(\"Upsampled Velocity\", fontsize=20) Out[26]: <pre>Text(0.5, 1.0, 'Upsampled Velocity')</pre> <p>In the next notebook, we'll explore using DeepLabCut to generate position data from video.</p>"}, {"location": "notebooks/20_Position_Trodes/#trodes-position", "title": "Trodes Position\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#loading-the-data", "title": "Loading the data\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#setting-parameters", "title": "Setting parameters\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#select-interval", "title": "Select interval\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#pairing-interval-and-parameters", "title": "Pairing interval and parameters\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#running-the-pipeline", "title": "Running the pipeline\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#examine-results", "title": "Examine results\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#plots", "title": "Plots\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#video", "title": "Video\u00b6", "text": "<p>To keep <code>minirec</code> small, the download link does not include videos by default.</p> <p>If it is available, you can uncomment the code, populate the  <code>TrodesPosVideo</code> table, and plot the results on the video using the <code>make_video</code> function, which will appear in the current working directory.</p>"}, {"location": "notebooks/20_Position_Trodes/#upsampling-position", "title": "Upsampling position\u00b6", "text": "<p>To get position data in smaller in time bins, we can upsample using the  following parameters</p> <ul> <li><code>is_upsampled</code>, default 0 (False): If 1, perform upsampling.</li> <li><code>upsampling_sampling_rate</code>, default None: the rate to upsample to (e.g., 33 Hz video might be upsampled to 500 Hz).</li> <li><code>upsampling_interpolation_method</code>, default linear: interpolation method. See pandas.DataFrame.interpolate for alternate methods.</li> </ul>"}, {"location": "notebooks/20_Position_Trodes/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/21_DLC/", "title": "DLC Models", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This tutorial will extract position via DeepLabCut (DLC). It will walk through...</p> <ul> <li>creating a DLC project</li> <li>extracting and labeling frames</li> <li>training your model</li> <li>executing pose estimation on a novel behavioral video</li> <li>processing the pose estimation output to extract a centroid and orientation</li> <li>inserting the resulting information into the <code>PositionOutput</code> table</li> </ul> <p>Note 2: Make sure you are running this within the spyglass-position Conda environment (instructions for install are in the environment_position.yml)</p> <p>Here is a schematic showing the tables used in this pipeline.</p> <p></p> <p>You can click on any header to return to the Table of Contents</p> In\u00a0[12]: Copied! <pre>%load_ext autoreload\n%autoreload 2\n</pre> %load_ext autoreload %autoreload 2 In\u00a0[13]: Copied! <pre>import os\nimport datajoint as dj\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\n\nimport numpy as np\nimport pandas as pd\nimport pynwb\nfrom spyglass.position import PositionOutput\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj  import spyglass.common as sgc import spyglass.position.v1 as sgp  import numpy as np import pandas as pd import pynwb from spyglass.position import PositionOutput  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) Notes:<ul> <li>         The cells within this <code>DLCProject</code> step need to be performed          in a local Jupyter notebook to allow for use of the frame labeling GUI.     </li> <li>         Please do not add to the <code>BodyPart</code> table in the production          database unless necessary.     </li> </ul> <p>We'll begin by looking at the <code>BodyPart</code> table, which stores standard names of body parts used in DLC models throughout the lab with a concise description.</p> In\u00a0[14]: Copied! <pre>sgp.BodyPart()\n</pre> sgp.BodyPart() Out[14]: <p>bodypart</p> <p>bodypart_description</p> back middle of the rat's backdriveBack back of drivedriveFront front of driveearL left ear of the ratearR right ear of the ratforelimbL left forelimb of the ratforelimbR right forelimb of the ratgreenLED greenLEDhindlimbL left hindlimb of the rathindlimbR right hindlimb of the ratnose tip of the nose of the ratredLED_C redLED_C <p>...</p> <p>Total: 23</p> <p>If the bodyparts you plan to use in your model are not yet in the table, here is code to add bodyparts:</p> <pre>sgp.BodyPart.insert(\n    [\n        {\"bodypart\": \"bp_1\", \"bodypart_description\": \"concise descrip\"},\n        {\"bodypart\": \"bp_2\", \"bodypart_description\": \"concise descrip\"},\n    ],\n    skip_duplicates=True,\n)\n</pre> <p>To train a model, we'll need to extract frames, which we can label as training data. We can construct a list of videos from which we'll extract frames.</p> <p>The list can either contain dictionaries identifying behavioral videos for NWB files that have already been added to Spyglass, or absolute file paths to the videos you want to use.</p> <p>For this tutorial, we'll use two videos for which we already have frames labeled.</p> <p>Defining camera name is optional: it should be done in cases where there are multiple cameras streaming per epoch, but not necessary otherwise.  example: <code>camera_name = \"HomeBox_camera\"</code></p> <p>NOTE: The official release of Spyglass does not yet support multicamera projects. You can monitor progress on the effort to add this feature by checking this PR or use this experimental branch, which takes the keys nwb_file_name and epoch, and camera_name in the video_list variable.</p> In\u00a0[15]: Copied! <pre>video_list = [\n    {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},\n    {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4},\n]\n</pre> video_list = [     {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},     {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4}, ] In\u00a0[16]: Copied! <pre>from spyglass.settings import config\n\nconfig\n</pre> from spyglass.settings import config  config Out[16]: <pre>{'debug_mode': False,\n 'prepopulate': True,\n 'SPYGLASS_BASE_DIR': '/stelmo/nwb',\n 'SPYGLASS_RAW_DIR': '/stelmo/nwb/raw',\n 'SPYGLASS_ANALYSIS_DIR': '/stelmo/nwb/analysis',\n 'SPYGLASS_RECORDING_DIR': '/stelmo/nwb/recording',\n 'SPYGLASS_SORTING_DIR': '/stelmo/nwb/sorting',\n 'SPYGLASS_WAVEFORMS_DIR': '/stelmo/nwb/waveforms',\n 'SPYGLASS_TEMP_DIR': '/stelmo/nwb/tmp/spyglass',\n 'SPYGLASS_VIDEO_DIR': '/stelmo/nwb/video',\n 'KACHERY_CLOUD_DIR': '/stelmo/nwb/.kachery-cloud',\n 'KACHERY_STORAGE_DIR': '/stelmo/nwb/kachery_storage',\n 'KACHERY_TEMP_DIR': '/stelmo/nwb/tmp',\n 'DLC_PROJECT_DIR': '/nimbus/deeplabcut/projects',\n 'DLC_VIDEO_DIR': '/nimbus/deeplabcut/video',\n 'DLC_OUTPUT_DIR': '/nimbus/deeplabcut/output',\n 'KACHERY_ZONE': 'franklab.default',\n 'FIGURL_CHANNEL': 'franklab2',\n 'DJ_SUPPORT_FILEPATH_MANAGEMENT': 'TRUE',\n 'KACHERY_CLOUD_EPHEMERAL': 'TRUE',\n 'HD5_USE_FILE_LOCKING': 'FALSE'}</pre> <p>Before creating our project, we need to define a few variables.</p> <ul> <li>A team name, as shown in <code>LabTeam</code> for setting permissions. Here, we'll use \"LorenLab\".</li> <li>A <code>project_name</code>, as a unique identifier for this DLC project. Here, we'll use \"tutorial_scratch_yourinitials\"</li> <li><code>bodyparts</code> is a list of body parts for which we want to extract position. The pre-labeled frames we're using include the bodyparts listed below.</li> <li>Number of frames to extract/label as <code>frames_per_video</code>. Note that the DLC creators recommend having 200 frames as the minimum total number for each project.</li> </ul> In\u00a0[17]: Copied! <pre>team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\"\nproject_name = \"tutorial_scratch_DG\"\nframes_per_video = 100\nbodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"]\nproject_key = sgp.DLCProject.insert_new_project(\n    project_name=project_name,\n    bodyparts=bodyparts,\n    lab_team=team_name,\n    frames_per_video=frames_per_video,\n    video_list=video_list,\n    skip_duplicates=True,\n)\n</pre> team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\" project_name = \"tutorial_scratch_DG\" frames_per_video = 100 bodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"] project_key = sgp.DLCProject.insert_new_project(     project_name=project_name,     bodyparts=bodyparts,     lab_team=team_name,     frames_per_video=frames_per_video,     video_list=video_list,     skip_duplicates=True, ) <pre>project name: tutorial_scratch_DG is already in use.\n</pre> <p>Now that we've initialized our project we'll need to extract frames which we will then label.</p> In\u00a0[\u00a0]: Copied! <pre># comment this line out after you finish frame extraction for each project\nsgp.DLCProject().run_extract_frames(project_key)\n</pre> # comment this line out after you finish frame extraction for each project sgp.DLCProject().run_extract_frames(project_key) <p>This is the line used to label the frames you extracted, if you wish to use the DLC GUI on the computer you are currently using.</p> <pre><code>#comment\nsgp.DLCProject().run_label_frames(project_key)\n</code></pre> <p>Otherwise, it is best/easiest practice to label the frames on your local computer (like a MacBook) that can run DeepLabCut's GUI well. Instructions: </p> <ol> <li>Install DLC on your local (preferably into a 'Src' folder): https://deeplabcut.github.io/DeepLabCut/docs/installation.html</li> <li>Upload frames extracted and saved in nimbus (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/labeled-data</code>) AND the project's associated config file (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/config.yaml</code>) to Box (we get free with UCSF)</li> <li>Download labeled-data and config files on your local from Box</li> <li>Create a 'projects' folder where you installed DeepLabCut; create a new folder with your complete project name there; save the downloaded files there.</li> <li>Edit the config.yaml file: line 9 defining <code>project_path</code> needs to be the file path where it is saved on your local (ex: <code>/Users/lorenlab/Src/DeepLabCut/projects/tutorial_sratch_DG-LorenLab-2023-08-16</code>)</li> <li>Open the DLC GUI through terminal (ex: <code>conda activate miniconda/envs/DEEPLABCUT_M1</code> <code>pythonw -m deeplabcut</code>)</li> <li>Load an existing project; choose the config.yaml file</li> <li>Label frames; labeling tutorial: https://www.youtube.com/watch?v=hsA9IB5r73E.</li> <li>Once all frames are labeled, you should re-upload labeled-data folder back to Box and overwrite it in the original nimbus location so that your completed frames are ready to be used in the model.</li> </ol> <p>Now we can check the <code>DLCProject.File</code> part table and see all of our training files and videos there!</p> In\u00a0[18]: Copied! <pre>sgp.DLCProject.File &amp; project_key\n</pre> sgp.DLCProject.File &amp; project_key Out[18]: Paths of training files (e.g., labeled pngs, CSV or video) <p>project_name</p> name of DLC project <p>file_name</p> Concise name to describe file <p>file_ext</p> extension of file <p>file_path</p> tutorial_scratch_DG 20201103_peanut_04_r2 mp4 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20201103_peanut_04_r2.mp4tutorial_scratch_DG 20201103_peanut_04_r2_labeled_data h5 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20201103_peanut_04_r2/CollectedData_LorenLab.h5tutorial_scratch_DG 20210529_J16_02_r1 mp4 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20210529_J16_02_r1.mp4tutorial_scratch_DG 20210529_J16_02_r1_labeled_data h5 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20210529_J16_02_r1/CollectedData_LorenLab.h5 <p>Total: 4</p>      This step and beyond should be run on a GPU-enabled machine.  In\u00a0[19]: Copied! <pre>sgp.dlc_utils.get_gpu_memory()\n</pre> sgp.dlc_utils.get_gpu_memory() Out[19]: <pre>{0: 305}</pre> <p>Set GPU core:</p> In\u00a0[20]: Copied! <pre>gputouse = 1  # 1-9\n</pre> gputouse = 1  # 1-9 <p>Now we'll define the rest of our parameters and insert the entry.</p> <p>To see all possible parameters, try:</p> <pre>sgp.DLCModelTrainingParams.get_accepted_params()\n</pre> In\u00a0[21]: Copied! <pre>training_params_name = \"tutorial\"\nsgp.DLCModelTrainingParams.insert_new_params(\n    paramset_name=training_params_name,\n    params={\n        \"trainingsetindex\": 0,\n        \"shuffle\": 1,\n        \"gputouse\": gputouse,\n        \"net_type\": \"resnet_50\",\n        \"augmenter_type\": \"imgaug\",\n    },\n    skip_duplicates=True,\n)\n</pre> training_params_name = \"tutorial\" sgp.DLCModelTrainingParams.insert_new_params(     paramset_name=training_params_name,     params={         \"trainingsetindex\": 0,         \"shuffle\": 1,         \"gputouse\": gputouse,         \"net_type\": \"resnet_50\",         \"augmenter_type\": \"imgaug\",     },     skip_duplicates=True, ) <pre>New param set not added\nA param set with name: tutorial already exists\n</pre> <p>Next we'll modify the <code>project_key</code> from above to include the necessary entries for <code>DLCModelTraining</code></p> In\u00a0[22]: Copied! <pre># project_key['project_path'] = os.path.dirname(project_key['config_path'])\nif \"config_path\" in project_key:\n    del project_key[\"config_path\"]\n</pre> # project_key['project_path'] = os.path.dirname(project_key['config_path']) if \"config_path\" in project_key:     del project_key[\"config_path\"] <p>We can insert an entry into <code>DLCModelTrainingSelection</code> and populate <code>DLCModelTraining</code>.</p> <p>Note: You can stop training at any point using <code>I + I</code> or interrupt the Kernel.</p> <p>The maximum total number of training iterations is 1030000; you can end training before this amount if the loss rate (lr) and total loss plateau and are very close to 0.</p> In\u00a0[23]: Copied! <pre>sgp.DLCModelTrainingSelection.heading\n</pre> sgp.DLCModelTrainingSelection.heading Out[23]: <pre>project_name         : varchar(100)                 # name of DLC project\ndlc_training_params_name : varchar(50)                  # descriptive name of parameter set\ntraining_id          : int                          # unique integer,\n---\nmodel_prefix=\"\"      : varchar(32)                  # </pre> In\u00a0[24]: Copied! <pre>sgp.DLCModelTrainingSelection().insert1(\n    {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n        \"training_id\": 0,\n        \"model_prefix\": \"\",\n    }\n)\nmodel_training_key = (\n    sgp.DLCModelTrainingSelection\n    &amp; {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n    }\n).fetch1(\"KEY\")\nsgp.DLCModelTraining.populate(model_training_key)\n</pre> sgp.DLCModelTrainingSelection().insert1(     {         **project_key,         \"dlc_training_params_name\": training_params_name,         \"training_id\": 0,         \"model_prefix\": \"\",     } ) model_training_key = (     sgp.DLCModelTrainingSelection     &amp; {         **project_key,         \"dlc_training_params_name\": training_params_name,     } ).fetch1(\"KEY\") sgp.DLCModelTraining.populate(model_training_key) <pre>2024-01-18 10:23:30.406102: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n</pre> <pre>Loading DLC 2.2.3...\nOpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py\n</pre> <pre>\n---------------------------------------------------------------------------\nPermissionError                           Traceback (most recent call last)\nCell In[24], line 16\n      1 sgp.DLCModelTrainingSelection().insert1(\n      2     {\n      3         **project_key,\n   (...)\n      7     }\n      8 )\n      9 model_training_key = (\n     10     sgp.DLCModelTrainingSelection\n     11     &amp; {\n   (...)\n     14     }\n     15 ).fetch1(\"KEY\")\n---&gt; 16 sgp.DLCModelTraining.populate(model_training_key)\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:241, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)\n    237 if processes == 1:\n    238     for key in (\n    239         tqdm(keys, desc=self.__class__.__name__) if display_progress else keys\n    240     ):\n--&gt; 241         error = self._populate1(key, jobs, **populate_kwargs)\n    242         if error is not None:\n    243             error_list.append(error)\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:292, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)\n    290 self.__class__._allow_insert = True\n    291 try:\n--&gt; 292     make(dict(key), **(make_kwargs or {}))\n    293 except (KeyboardInterrupt, SystemExit, Exception) as error:\n    294     try:\n\nFile ~/Src/spyglass/src/spyglass/position/v1/position_dlc_training.py:150, in DLCModelTraining.make(self, key)\n    144     from deeplabcut.utils.auxiliaryfunctions import (\n    145         GetModelFolder as get_model_folder,\n    146     )\n    147 config_path, project_name = (DLCProject() &amp; key).fetch1(\n    148     \"config_path\", \"project_name\"\n    149 )\n--&gt; 150 with OutputLogger(\n    151     name=\"DLC_project_{project_name}_training\",\n    152     path=f\"{os.path.dirname(config_path)}/log.log\",\n    153     print_console=True,\n    154 ) as logger:\n    155     dlc_config = read_config(config_path)\n    156     project_path = dlc_config[\"project_path\"]\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:192, in OutputLogger.__init__(self, name, path, level, **kwargs)\n    191 def __init__(self, name, path, level=\"INFO\", **kwargs):\n--&gt; 192     self.logger = self.setup_logger(name, path, **kwargs)\n    193     self.name = self.logger.name\n    194     self.level = getattr(logging, level)\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:244, in OutputLogger.setup_logger(self, name_logfile, path_logfile, print_console)\n    241         logger.addHandler(self._get_stream_handler())\n    243 else:\n--&gt; 244     file_handler = self._get_file_handler(path_logfile)\n    245     logger.addHandler(file_handler)\n    246     if print_console:\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:255, in OutputLogger._get_file_handler(self, path)\n    253 if not os.path.exists(output_dir):\n    254     output_dir.mkdir(parents=True, exist_ok=True)\n--&gt; 255 file_handler = logging.FileHandler(path, mode=\"a\")\n    256 file_handler.setFormatter(self._get_formatter())\n    257 return file_handler\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1146, in FileHandler.__init__(self, filename, mode, encoding, delay, errors)\n   1144     self.stream = None\n   1145 else:\n-&gt; 1146     StreamHandler.__init__(self, self._open())\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1175, in FileHandler._open(self)\n   1170 def _open(self):\n   1171     \"\"\"\n   1172     Open the current base file with the (original) mode and encoding.\n   1173     Return the resulting stream.\n   1174     \"\"\"\n-&gt; 1175     return open(self.baseFilename, self.mode, encoding=self.encoding,\n   1176                 errors=self.errors)\n\nPermissionError: [Errno 13] Permission denied: '/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/log.log'</pre> <p>Here we'll make sure that the entry made it into the table properly!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTraining() &amp; model_training_key\n</pre> sgp.DLCModelTraining() &amp; model_training_key <p>Populating <code>DLCModelTraining</code> automatically inserts the entry into <code>DLCModelSource</code>, which is used to select between models trained using Spyglass vs. other tools.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource() &amp; model_training_key\n</pre> sgp.DLCModelSource() &amp; model_training_key <p>The <code>source</code> field will only accept \"FromImport\" or \"FromUpstream\" as entries. Let's checkout the <code>FromUpstream</code> part table attached to <code>DLCModelSource</code> below.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource.FromUpstream() &amp; model_training_key\n</pre> sgp.DLCModelSource.FromUpstream() &amp; model_training_key In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelParams.get_default()\n</pre> sgp.DLCModelParams.get_default() <p>Here is the syntax to add your own parameter set:</p> <pre>dlc_model_params_name = \"make_this_yours\"\nparams = {\n    \"params\": {},\n    \"shuffle\": 1,\n    \"trainingsetindex\": 0,\n    \"model_prefix\": \"\",\n}\nsgp.DLCModelParams.insert1(\n    {\"dlc_model_params_name\": dlc_model_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We can insert sets of parameters into <code>DLCModelSelection</code> and populate <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\")\n</pre> temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\") In\u00a0[\u00a0]: Copied! <pre># comment these lines out after successfully inserting, for each project\nsgp.DLCModelSelection().insert1(\n    {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True\n)\n</pre> # comment these lines out after successfully inserting, for each project sgp.DLCModelSelection().insert1(     {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True ) In\u00a0[\u00a0]: Copied! <pre>model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\")\nsgp.DLCModel.populate(model_key)\n</pre> model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\") sgp.DLCModel.populate(model_key) <p>Again, let's make sure that everything looks correct in <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModel() &amp; model_key\n</pre> sgp.DLCModel() &amp; model_key In\u00a0[3]: Copied! <pre>nwb_file_name = \"J1620210604_.nwb\"\nsgc.VideoFile() &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> nwb_file_name = \"J1620210604_.nwb\" sgc.VideoFile() &amp; {\"nwb_file_name\": nwb_file_name} Out[3]: <p>nwb_file_name</p> name of the NWB file <p>epoch</p> the session epoch for this task and apparatus(1 based) <p>video_file_num</p> <p>camera_name</p> <p>video_file_object_id</p> the object id of the file object J1620210604_.nwb 1 0 178f5746-30e3-4957-891e-8024e23522dcJ1620210604_.nwb 2 0 d64ec979-326b-429f-b3fe-1bbfbf806293J1620210604_.nwb 3 0 cf14bcd2-c0a9-457b-8791-42f3f28dd912J1620210604_.nwb 4 0 183c9910-36fd-46c1-a24c-8d1c306d7248J1620210604_.nwb 5 0 4677c7cd-8cd8-4801-8f6e-5b7bb14a6d6bJ1620210604_.nwb 6 0 0e46532b-483f-43af-ba6e-ba75ccf340eaJ1620210604_.nwb 7 0 c6d1d037-44ec-4d91-99d1-172d371bf82aJ1620210604_.nwb 8 0 4d7e070c-6220-47de-8173-993f013fafa8J1620210604_.nwb 9 0 b50108ec-f587-46df-b1c8-3ca23091bde0J1620210604_.nwb 10 0 b9b5da20-da39-4274-9be2-55610cfd1b5bJ1620210604_.nwb 11 0 6c827b8d-513c-4dba-ae75-0b36dcf4811fJ1620210604_.nwb 12 0 41bd2344-1b41-4737-8dfb-7c860d089155 <p>...</p> <p>Total: 20</p> In\u00a0[5]: Copied! <pre>epoch = 14  # change based on VideoFile entry\nvideo_file_num = 0  # change based on VideoFile entry\n</pre> epoch = 14  # change based on VideoFile entry video_file_num = 0  # change based on VideoFile entry <p>Using <code>insert_estimation_task</code> will convert out video to be in .mp4 format (DLC struggles with .h264) and determine the directory in which we'll store the pose estimation results.</p> <ul> <li><code>task_mode</code> (trigger or load) determines whether or not populating <code>DLCPoseEstimation</code> triggers a new pose estimation, or loads an existing.</li> <li><code>video_file_num</code> will be 0 in almost all cases.</li> <li><code>gputouse</code> was already set during training. It may be a good idea to make sure that core is still free before moving forward.</li> </ul> <p>The <code>DLCPoseEstimationSelection</code> insertion step will convert your .h264 video to an .mp4 first and save it in <code>/nimbus/deeplabcut/video</code>. If this video already exists here, the insertion will never complete.</p> <p>We first delete any .mp4 that exists for this video from the nimbus folder. Remove the <code>#</code> to run this line. The <code>!</code> tells the notebook that this is a system command to be run with a shell script instead of python. Be sure to change the string based on date and rat with which you are training the model</p> In\u00a0[\u00a0]: Copied! <pre>#! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete\n</pre> #! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete In\u00a0[6]: Copied! <pre>pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    },\n    task_mode=\"trigger\",  # trigger or load\n    params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n)\n</pre> pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(     {         \"nwb_file_name\": nwb_file_name,         \"epoch\": epoch,         \"video_file_num\": video_file_num,         **model_key,     },     task_mode=\"trigger\",  # trigger or load     params={\"gputouse\": gputouse, \"videotype\": \"mp4\"}, ) <pre>\n---------------------------------------------------------------------------\nNameError                                 Traceback (most recent call last)\nCell In[6], line 6\n      1 pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n      2     {\n      3         \"nwb_file_name\": nwb_file_name,\n      4         \"epoch\": epoch,\n      5         \"video_file_num\": video_file_num,\n----&gt; 6         **model_key,\n      7     },\n      8     task_mode=\"trigger\", #trigger or load\n      9     params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n     10 )\n\nNameError: name 'model_key' is not defined</pre> <p>If the above insertion step fails in either trigger or load mode for an epoch, run the following lines:</p> <pre><code>(pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    }).delete()\n</code></pre> <p>And now we populate <code>DLCPoseEstimation</code>! This might take some time for full datasets.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPoseEstimation().populate(pose_estimation_key)\n</pre> sgp.DLCPoseEstimation().populate(pose_estimation_key) <p>Let's visualize the output from Pose Estimation</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPoseEstimation() &amp; pose_estimation_key).fetch_dataframe()\n</pre> (sgp.DLCPoseEstimation() &amp; pose_estimation_key).fetch_dataframe() <p>Now that we've completed pose estimation, it's time to identify NaNs and optionally interpolate over low likelihood periods and smooth the resulting positions.First we need to define some parameters for smoothing and interpolation. We can see the default parameter set below.Note: it is recommended to use the <code>just_nan</code> parameters here and save interpolation and smoothing for the centroid step as this provides for a better end result.</p> In\u00a0[\u00a0]: Copied! <pre># The default parameter set to interpolate and smooth over each LED individually\nprint(sgp.DLCSmoothInterpParams.get_default())\n</pre> # The default parameter set to interpolate and smooth over each LED individually print(sgp.DLCSmoothInterpParams.get_default()) In\u00a0[\u00a0]: Copied! <pre># The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step\nprint(sgp.DLCSmoothInterpParams.get_nan_params())\nsi_params_name = \"just_nan\"  # could also use \"default\"\n</pre> # The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step print(sgp.DLCSmoothInterpParams.get_nan_params()) si_params_name = \"just_nan\"  # could also use \"default\" <p>To change any of these parameters, one would do the following:</p> <pre>si_params_name = \"your_unique_param_name\"\nparams = {\n    \"smoothing_params\": {\n        \"smoothing_duration\": 0.00,\n        \"smooth_method\": \"moving_avg\",\n    },\n    \"interp_params\": {\"likelihood_thresh\": 0.00},\n    \"max_plausible_speed\": 0,\n    \"speed_smoothing_std_dev\": 0.000,\n}\nsgp.DLCSmoothInterpParams().insert1(\n    {\"dlc_si_params_name\": si_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We'll create a dictionary with the correct set of keys for the <code>DLCSmoothInterpSelection</code> table</p> In\u00a0[\u00a0]: Copied! <pre>si_key = pose_estimation_key.copy()\nfields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())\nsi_key = {key: val for key, val in si_key.items() if key in fields}\nsi_key\n</pre> si_key = pose_estimation_key.copy() fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys()) si_key = {key: val for key, val in si_key.items() if key in fields} si_key <p>We can insert all of the bodyparts we want to process into <code>DLCSmoothInterpSelection</code> First lets visualize the bodyparts we have available to us.</p> In\u00a0[\u00a0]: Copied! <pre>print((sgp.DLCPoseEstimation.BodyPart &amp; pose_estimation_key).fetch(\"bodypart\"))\n</pre> print((sgp.DLCPoseEstimation.BodyPart &amp; pose_estimation_key).fetch(\"bodypart\")) <p>We can use <code>insert1</code> to insert a single bodypart, but would suggest using <code>insert</code> to insert a list of keys with different bodyparts.</p> <p>To insert a single bodypart, one would do the following:</p> <pre>sgp.DLCSmoothInterpSelection.insert1(\n    {\n        **si_key,\n        'bodypart': 'greenLED',\n        'dlc_si_params_name': si_params_name,\n    },\n    skip_duplicates=True)\n</pre> <p>We'll see a list of bodyparts and then insert them into <code>DLCSmoothInterpSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>bodyparts = [\"greenLED\", \"redLED_C\"]\nsgp.DLCSmoothInterpSelection.insert(\n    [\n        {\n            **si_key,\n            \"bodypart\": bodypart,\n            \"dlc_si_params_name\": si_params_name,\n        }\n        for bodypart in bodyparts\n    ],\n    skip_duplicates=True,\n)\n</pre> bodyparts = [\"greenLED\", \"redLED_C\"] sgp.DLCSmoothInterpSelection.insert(     [         {             **si_key,             \"bodypart\": bodypart,             \"dlc_si_params_name\": si_params_name,         }         for bodypart in bodyparts     ],     skip_duplicates=True, ) <p>And verify the entry:</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpSelection() &amp; si_key\n</pre> sgp.DLCSmoothInterpSelection() &amp; si_key <p>Now, we populate <code>DLCSmoothInterp</code>, which will perform smoothing and interpolation on all of the bodyparts specified.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterp().populate(si_key)\n</pre> sgp.DLCSmoothInterp().populate(si_key) <p>And let's visualize the resulting position data using a scatter plot</p> In\u00a0[\u00a0]: Copied! <pre>(\n    sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}\n).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))\n</pre> (     sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]} ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5)) <p>After smoothing/interpolation, we need to select bodyparts from which we want to derive a centroid and orientation, which is performed by the <code>DLCSmoothInterpCohort</code> table.</p> <p>First, let's make a key that represents the 'cohort', using <code>dlc_si_cohort_selection_name</code>. We'll need a bodypart dictionary using bodypart keys and smoothing/interpolation parameters used as value.</p> In\u00a0[\u00a0]: Copied! <pre>cohort_key = si_key.copy()\nif \"bodypart\" in cohort_key:\n    del cohort_key[\"bodypart\"]\nif \"dlc_si_params_name\" in cohort_key:\n    del cohort_key[\"dlc_si_params_name\"]\ncohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"\ncohort_key[\"bodyparts_params_dict\"] = {\n    \"greenLED\": si_params_name,\n    \"redLED_C\": si_params_name,\n}\nprint(cohort_key)\n</pre> cohort_key = si_key.copy() if \"bodypart\" in cohort_key:     del cohort_key[\"bodypart\"] if \"dlc_si_params_name\" in cohort_key:     del cohort_key[\"dlc_si_params_name\"] cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\" cohort_key[\"bodyparts_params_dict\"] = {     \"greenLED\": si_params_name,     \"redLED_C\": si_params_name, } print(cohort_key) <p>We'll insert the cohort into <code>DLCSmoothInterpCohortSelection</code> and populate <code>DLCSmoothInterpCohort</code>, which collates the separately smoothed and interpolated bodyparts into a single entry.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True)\nsgp.DLCSmoothInterpCohort.populate(cohort_key)\n</pre> sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True) sgp.DLCSmoothInterpCohort.populate(cohort_key) <p>And verify the entry:</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpCohort.BodyPart() &amp; cohort_key\n</pre> sgp.DLCSmoothInterpCohort.BodyPart() &amp; cohort_key <p>With this cohort, we can determine a centroid using another set of parameters.</p> In\u00a0[\u00a0]: Copied! <pre># Here is the default set\nprint(sgp.DLCCentroidParams.get_default())\ncentroid_params_name = \"default\"\n</pre> # Here is the default set print(sgp.DLCCentroidParams.get_default()) centroid_params_name = \"default\" <p>Here is the syntax to add your own parameters:</p> <pre>centroid_params = {\n    \"centroid_method\": \"two_pt_centroid\",\n    \"points\": {\n        \"greenLED\": \"greenLED\",\n        \"redLED_C\": \"redLED_C\",\n    },\n    \"speed_smoothing_std_dev\": 0.100,\n}\ncentroid_params_name = \"your_unique_param_name\"\nsgp.DLCCentroidParams.insert1(\n    {\n        \"dlc_centroid_params_name\": centroid_params_name,\n        \"params\": centroid_params,\n    },\n    skip_duplicates=True,\n)\n</pre> <p>We'll make a key to insert into <code>DLCCentroidSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>centroid_key = cohort_key.copy()\nfields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())\ncentroid_key = {key: val for key, val in centroid_key.items() if key in fields}\ncentroid_key[\"dlc_centroid_params_name\"] = centroid_params_name\nprint(centroid_key)\n</pre> centroid_key = cohort_key.copy() fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys()) centroid_key = {key: val for key, val in centroid_key.items() if key in fields} centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name print(centroid_key) <p>After inserting into the selection table, we can populate <code>DLCCentroid</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)\nsgp.DLCCentroid.populate(centroid_key)\n</pre> sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True) sgp.DLCCentroid.populate(centroid_key) <p>Here we can visualize the resulting centroid position</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(\n    x=\"position_x\",\n    y=\"position_y\",\n    c=\"speed\",\n    colormap=\"viridis\",\n    alpha=0.5,\n    s=0.5,\n    figsize=(10, 10),\n)\n</pre> (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(     x=\"position_x\",     y=\"position_y\",     c=\"speed\",     colormap=\"viridis\",     alpha=0.5,     s=0.5,     figsize=(10, 10), ) <p>We'll now go through a similar process to identify the orientation.</p> In\u00a0[\u00a0]: Copied! <pre>print(sgp.DLCOrientationParams.get_default())\ndlc_orientation_params_name = \"default\"\n</pre> print(sgp.DLCOrientationParams.get_default()) dlc_orientation_params_name = \"default\" <p>We'll prune the <code>cohort_key</code> we used above and add our <code>dlc_orientation_params_name</code> to make it suitable for <code>DLCOrientationSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())\norient_key = {key: val for key, val in cohort_key.items() if key in fields}\norient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name\nprint(orient_key)\n</pre> fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys()) orient_key = {key: val for key, val in cohort_key.items() if key in fields} orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name print(orient_key) <p>We'll insert into <code>DLCOrientationSelection</code> and populate <code>DLCOrientation</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)\nsgp.DLCOrientation().populate(orient_key)\n</pre> sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True) sgp.DLCOrientation().populate(orient_key) <p>We can fetch the orientation as a dataframe as quality assurance.</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCOrientation() &amp; orient_key).fetch1_dataframe()\n</pre> (sgp.DLCOrientation() &amp; orient_key).fetch1_dataframe() <p>After processing the position data, we have to do a few table manipulations to standardize various outputs.</p> <p>To summarize, we brought in a pretrained DLC project, used that model to run pose estimation on a new behavioral video, smoothed and interpolated the result, formed a cohort of bodyparts, and determined the centroid and orientation of this cohort.</p> <p>Now we'll populate <code>DLCPos</code> with our centroid/orientation entries above.</p> In\u00a0[\u00a0]: Copied! <pre>fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())\ndlc_key = {key: val for key, val in centroid_key.items() if key in fields}\ndlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\"dlc_si_cohort_selection_name\"]\ndlc_key[\"dlc_si_cohort_orientation\"] = orient_key[\n    \"dlc_si_cohort_selection_name\"\n]\ndlc_key[\"dlc_orientation_params_name\"] = orient_key[\n    \"dlc_orientation_params_name\"\n]\nprint(dlc_key)\n</pre> fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys()) dlc_key = {key: val for key, val in centroid_key.items() if key in fields} dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\"dlc_si_cohort_selection_name\"] dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[     \"dlc_si_cohort_selection_name\" ] dlc_key[\"dlc_orientation_params_name\"] = orient_key[     \"dlc_orientation_params_name\" ] print(dlc_key) <p>Now we can insert into <code>DLCPosSelection</code> and populate <code>DLCPos</code> with our <code>dlc_key</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)\nsgp.DLCPosV1().populate(dlc_key)\n</pre> sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True) sgp.DLCPosV1().populate(dlc_key) <p>We can also make sure that all of our data made it through by fetching the dataframe attached to this entry.We should expect 8 columns:</p> <p>timevideo_frame_indposition_xposition_yorientationvelocity_xvelocity_yspeed</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPosV1() &amp; dlc_key).fetch1_dataframe()\n</pre> (sgp.DLCPosV1() &amp; dlc_key).fetch1_dataframe() <p>And even more, we can fetch the <code>pose_eval_result</code> that is calculated during this step. This field contains the percentage of frames that each bodypart was below the likelihood threshold of 0.95 as a means of assessing the quality of the pose estimation.</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPosV1() &amp; dlc_key).fetch1(\"pose_eval_result\")\n</pre> (sgp.DLCPosV1() &amp; dlc_key).fetch1(\"pose_eval_result\") <p>We can create a video with the centroid and orientation overlaid on the original video. This will also plot the likelihood of each bodypart used in the cohort. This is optional, but a good quality assurance step.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideoParams.insert_default()\n</pre> sgp.DLCPosVideoParams.insert_default() In\u00a0[\u00a0]: Copied! <pre>params = {\n    \"percent_frames\": 0.05,\n    \"incl_likelihood\": True,\n}\nsgp.DLCPosVideoParams.insert1(\n    {\"dlc_pos_video_params_name\": \"five_percent\", \"params\": params},\n    skip_duplicates=True,\n)\n</pre> params = {     \"percent_frames\": 0.05,     \"incl_likelihood\": True, } sgp.DLCPosVideoParams.insert1(     {\"dlc_pos_video_params_name\": \"five_percent\", \"params\": params},     skip_duplicates=True, ) In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideoSelection.insert1(\n    {**dlc_key, \"dlc_pos_video_params_name\": \"five_percent\"},\n    skip_duplicates=True,\n)\n</pre> sgp.DLCPosVideoSelection.insert1(     {**dlc_key, \"dlc_pos_video_params_name\": \"five_percent\"},     skip_duplicates=True, ) In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideo().populate(dlc_key)\n</pre> sgp.DLCPosVideo().populate(dlc_key) <p><code>PositionOutput</code> is the final table of the pipeline and is automatically populated when we populate <code>DLCPosV1</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.PositionOutput.merge_get_part(dlc_key)\n</pre> sgp.PositionOutput.merge_get_part(dlc_key) <p><code>PositionOutput</code> also has a part table, similar to the <code>DLCModelSource</code> table above. Let's check that out as well.</p> In\u00a0[\u00a0]: Copied! <pre>PositionOutput.DLCPosV1() &amp; dlc_key\n</pre> PositionOutput.DLCPosV1() &amp; dlc_key In\u00a0[\u00a0]: Copied! <pre>(PositionOutput.DLCPosV1() &amp; dlc_key).fetch1_dataframe()\n</pre> (PositionOutput.DLCPosV1() &amp; dlc_key).fetch1_dataframe() <p>We can use the <code>PositionVideo</code> table to create a video that overlays just the centroid and orientation on the video. This table uses the parameter <code>plot</code> to determine whether to plot the entry deriving from the DLC arm or from the Trodes arm of the position pipeline. This parameter also accepts 'all', which will plot both (if they exist) in order to compare results.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.PositionVideoSelection().insert1(\n    {\n        \"nwb_file_name\": \"J1620210604_.nwb\",\n        \"interval_list_name\": \"pos 13 valid times\",\n        \"trodes_position_id\": 0,\n        \"dlc_position_id\": 1,\n        \"plot\": \"DLC\",\n        \"output_dir\": \"/home/dgramling/Src/\",\n    }\n)\n</pre> sgp.PositionVideoSelection().insert1(     {         \"nwb_file_name\": \"J1620210604_.nwb\",         \"interval_list_name\": \"pos 13 valid times\",         \"trodes_position_id\": 0,         \"dlc_position_id\": 1,         \"plot\": \"DLC\",         \"output_dir\": \"/home/dgramling/Src/\",     } ) In\u00a0[\u00a0]: Copied! <pre>sgp.PositionVideo.populate({\"plot\": \"DLC\"})\n</pre> sgp.PositionVideo.populate({\"plot\": \"DLC\"}) Return To Table of Contents"}, {"location": "notebooks/21_DLC/#position-deeplabcut-from-scratch", "title": "Position- DeepLabCut from Scratch\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#table-of-contents", "title": "Table of Contents\u00b6", "text": "<p><code>DLCProject</code> <code>DLCModelTraining</code> <code>DLCModel</code> <code>DLCPoseEstimation</code> <code>DLCSmoothInterp</code> <code>DLCCentroid</code> <code>DLCOrientation</code> <code>DLCPosV1</code> <code>DLCPosVideo</code> <code>PositionOutput</code></p>"}, {"location": "notebooks/21_DLC/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcproject", "title": "DLCProject \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#body-parts", "title": "Body Parts\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#define-videos-and-camera-name-optional-for-training-set", "title": "Define videos and camera name (optional) for training set\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#path-variables", "title": "Path variables\u00b6", "text": "<p>The position pipeline also keeps track of paths for project, video, and output. Just like we saw in Setup, you can manage these either with environmental variables...</p> <pre>export DLC_PROJECT_DIR=\"/nimbus/deeplabcut/projects\"\nexport DLC_VIDEO_DIR=\"/nimbus/deeplabcut/video\"\nexport DLC_OUTPUT_DIR=\"/nimbus/deeplabcut/output\"\n</pre> <p>Or these can be set in your datajoint config:</p> <pre>{\n  \"custom\": {\n    \"dlc_dirs\": {\n      \"base\": \"/nimbus/deeplabcut/\",\n      \"project\": \"/nimbus/deeplabcut/projects\",\n      \"video\": \"/nimbus/deeplabcut/video\",\n      \"output\": \"/nimbus/deeplabcut/output\"\n    }\n  }\n}\n</pre> <p>NOTE: If only <code>base</code> is specified as shown above, spyglass will assume the relative directories shown.</p> <p>You can check the result of this setup process with...</p>"}, {"location": "notebooks/21_DLC/#dlcmodeltraining", "title": "DLCModelTraining\u00b6", "text": "<p>Please make sure you're running this notebook on a GPU-enabled machine.</p> <p>Now that we've imported existing frames, we can get ready to train our model.</p> <p>First, we'll need to define a set of parameters for <code>DLCModelTrainingParams</code>, which will get used by DeepLabCut during training. Let's start with <code>gputouse</code>, which determines which GPU core to use.</p> <p>The cell below determines which core has space and set the <code>gputouse</code> variable accordingly.</p>"}, {"location": "notebooks/21_DLC/#dlcmodel", "title": "DLCModel \u00b6", "text": "<p>Next we'll populate the <code>DLCModel</code> table, which holds all the relevant information for all trained models.</p> <p>First, we'll need to determine a set of parameters for our model to select the correct model file. Here is the default:</p>"}, {"location": "notebooks/21_DLC/#dlcposeestimation", "title": "DLCPoseEstimation \u00b6", "text": "<p>Alright, now that we've trained model and populated the <code>DLCModel</code> table, we're ready to set-up Pose Estimation on a behavioral video of your choice.For this tutorial, you can choose to use an epoch of your choice, we can also use the one specified below. If you'd like to use your own video, just specify the <code>nwb_file_name</code> and <code>epoch</code> number and make sure it's in the <code>VideoFile</code> table!</p>"}, {"location": "notebooks/21_DLC/#dlcsmoothinterp", "title": "DLCSmoothInterp \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcsmoothinterpcohort", "title": "DLCSmoothInterpCohort \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlccentroid", "title": "DLCCentroid \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcorientation", "title": "DLCOrientation \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcposv1", "title": "DLCPosV1 \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcposvideo", "title": "DLCPosVideo \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#positionoutput", "title": "PositionOutput \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#positionvideo", "title": "PositionVideo\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#congratulations", "title": "CONGRATULATIONS!!\u00b6", "text": "<p>Please treat yourself to a nice tea break :-)</p>"}, {"location": "notebooks/22_DLC_Loop/", "title": "Looping DLC", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This tutorial will extract position via DeepLabCut (DLC). It will walk through...</p> <ul> <li>creating a DLC project</li> <li>extracting and labeling frames</li> <li>training your model</li> <li>executing pose estimation on a novel behavioral video</li> <li>processing the pose estimation output to extract a centroid and orientation</li> <li>inserting the resulting information into the <code>PositionOutput</code> table</li> </ul> <p>Note 2: Make sure you are running this within the spyglass-position Conda environment (instructions for install are in the environment_position.yml)</p> <p>Here is a schematic showing the tables used in this pipeline.</p> <p></p> <p>You can click on any header to return to the Table of Contents</p> In\u00a0[1]: Copied! <pre>%load_ext autoreload\n%autoreload 2\n</pre> %load_ext autoreload %autoreload 2 In\u00a0[2]: Copied! <pre>import os\nimport datajoint as dj\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\n\nimport numpy as np\nimport pandas as pd\nimport pynwb\nfrom spyglass.position import PositionOutput\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj  import spyglass.common as sgc import spyglass.position.v1 as sgp  import numpy as np import pandas as pd import pynwb from spyglass.position import PositionOutput  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-18 10:12:13,219][INFO]: Connecting ebroyles@lmf-db.cin.ucsf.edu:3306\n[2024-01-18 10:12:13,255][INFO]: Connected ebroyles@lmf-db.cin.ucsf.edu:3306\nOMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n</pre> Notes:<ul> <li>         The cells within this <code>DLCProject</code> step need to be performed          in a local Jupyter notebook to allow for use of the frame labeling GUI.     </li> <li>         Please do not add to the <code>BodyPart</code> table in the production          database unless necessary.     </li> </ul> <p>We'll begin by looking at the <code>BodyPart</code> table, which stores standard names of body parts used in DLC models throughout the lab with a concise description.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.BodyPart()\n</pre> sgp.BodyPart() <p>If the bodyparts you plan to use in your model are not yet in the table, here is code to add bodyparts:</p> <pre>sgp.BodyPart.insert(\n    [\n        {\"bodypart\": \"bp_1\", \"bodypart_description\": \"concise descrip\"},\n        {\"bodypart\": \"bp_2\", \"bodypart_description\": \"concise descrip\"},\n    ],\n    skip_duplicates=True,\n)\n</pre> <p>To train a model, we'll need to extract frames, which we can label as training data. We can construct a list of videos from which we'll extract frames.</p> <p>The list can either contain dictionaries identifying behavioral videos for NWB files that have already been added to Spyglass, or absolute file paths to the videos you want to use.</p> <p>For this tutorial, we'll use two videos for which we already have frames labeled.</p> <p>Defining camera name is optional: it should be done in cases where there are multiple cameras streaming per epoch, but not necessary otherwise.  example: <code>camera_name = \"HomeBox_camera\"</code></p> <p>NOTE: The official release of Spyglass does not yet support multicamera projects. You can monitor progress on the effort to add this feature by checking this PR or use this experimental branch, which takes the keys nwb_file_name and epoch, and camera_name in the video_list variable.</p> In\u00a0[\u00a0]: Copied! <pre>video_list = [\n    {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},\n    {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4},\n]\n</pre> video_list = [     {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},     {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4}, ] In\u00a0[\u00a0]: Copied! <pre>from spyglass.settings import config\n\nconfig\n</pre> from spyglass.settings import config  config <p>Before creating our project, we need to define a few variables.</p> <ul> <li>A team name, as shown in <code>LabTeam</code> for setting permissions. Here, we'll use \"LorenLab\".</li> <li>A <code>project_name</code>, as a unique identifier for this DLC project. Here, we'll use \"tutorial_scratch_yourinitials\"</li> <li><code>bodyparts</code> is a list of body parts for which we want to extract position. The pre-labeled frames we're using include the bodyparts listed below.</li> <li>Number of frames to extract/label as <code>frames_per_video</code>. Note that the DLC creators recommend having 200 frames as the minimum total number for each project.</li> </ul> In\u00a0[\u00a0]: Copied! <pre>team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\"\nproject_name = \"tutorial_scratch_DG\"\nframes_per_video = 100\nbodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"]\nproject_key = sgp.DLCProject.insert_new_project(\n    project_name=project_name,\n    bodyparts=bodyparts,\n    lab_team=team_name,\n    frames_per_video=frames_per_video,\n    video_list=video_list,\n    skip_duplicates=True,\n)\n</pre> team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\" project_name = \"tutorial_scratch_DG\" frames_per_video = 100 bodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"] project_key = sgp.DLCProject.insert_new_project(     project_name=project_name,     bodyparts=bodyparts,     lab_team=team_name,     frames_per_video=frames_per_video,     video_list=video_list,     skip_duplicates=True, ) <p>Now that we've initialized our project we'll need to extract frames which we will then label.</p> In\u00a0[\u00a0]: Copied! <pre># comment this line out after you finish frame extraction for each project\nsgp.DLCProject().run_extract_frames(project_key)\n</pre> # comment this line out after you finish frame extraction for each project sgp.DLCProject().run_extract_frames(project_key) <p>This is the line used to label the frames you extracted, if you wish to use the DLC GUI on the computer you are currently using.</p> <pre><code>#comment\nsgp.DLCProject().run_label_frames(project_key)\n</code></pre> <p>Otherwise, it is best/easiest practice to label the frames on your local computer (like a MacBook) that can run DeepLabCut's GUI well. Instructions: </p> <ol> <li>Install DLC on your local (preferably into a 'Src' folder): https://deeplabcut.github.io/DeepLabCut/docs/installation.html</li> <li>Upload frames extracted and saved in nimbus (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/labeled-data</code>) AND the project's associated config file (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/config.yaml</code>) to Box (we get free with UCSF)</li> <li>Download labeled-data and config files on your local from Box</li> <li>Create a 'projects' folder where you installed DeepLabCut; create a new folder with your complete project name there; save the downloaded files there.</li> <li>Edit the config.yaml file: line 9 defining <code>project_path</code> needs to be the file path where it is saved on your local (ex: <code>/Users/lorenlab/Src/DeepLabCut/projects/tutorial_sratch_DG-LorenLab-2023-08-16</code>)</li> <li>Open the DLC GUI through terminal (ex: <code>conda activate miniconda/envs/DEEPLABCUT_M1</code> <code>pythonw -m deeplabcut</code>)</li> <li>Load an existing project; choose the config.yaml file</li> <li>Label frames; labeling tutorial: https://www.youtube.com/watch?v=hsA9IB5r73E.</li> <li>Once all frames are labeled, you should re-upload labeled-data folder back to Box and overwrite it in the original nimbus location so that your completed frames are ready to be used in the model.</li> </ol> <p>Now we can check the <code>DLCProject.File</code> part table and see all of our training files and videos there!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCProject.File &amp; project_key\n</pre> sgp.DLCProject.File &amp; project_key      This step and beyond should be run on a GPU-enabled machine.  In\u00a0[\u00a0]: Copied! <pre>sgp.dlc_utils.get_gpu_memory()\n</pre> sgp.dlc_utils.get_gpu_memory() <p>Set GPU core:</p> In\u00a0[\u00a0]: Copied! <pre>gputouse = 1  # 1-9\n</pre> gputouse = 1  # 1-9 <p>Now we'll define the rest of our parameters and insert the entry.</p> <p>To see all possible parameters, try:</p> <pre>sgp.DLCModelTrainingParams.get_accepted_params()\n</pre> In\u00a0[\u00a0]: Copied! <pre>training_params_name = \"tutorial\"\nsgp.DLCModelTrainingParams.insert_new_params(\n    paramset_name=training_params_name,\n    params={\n        \"trainingsetindex\": 0,\n        \"shuffle\": 1,\n        \"gputouse\": gputouse,\n        \"net_type\": \"resnet_50\",\n        \"augmenter_type\": \"imgaug\",\n    },\n    skip_duplicates=True,\n)\n</pre> training_params_name = \"tutorial\" sgp.DLCModelTrainingParams.insert_new_params(     paramset_name=training_params_name,     params={         \"trainingsetindex\": 0,         \"shuffle\": 1,         \"gputouse\": gputouse,         \"net_type\": \"resnet_50\",         \"augmenter_type\": \"imgaug\",     },     skip_duplicates=True, ) <p>Next we'll modify the <code>project_key</code> from above to include the necessary entries for <code>DLCModelTraining</code></p> In\u00a0[\u00a0]: Copied! <pre># project_key['project_path'] = os.path.dirname(project_key['config_path'])\nif \"config_path\" in project_key:\n    del project_key[\"config_path\"]\n</pre> # project_key['project_path'] = os.path.dirname(project_key['config_path']) if \"config_path\" in project_key:     del project_key[\"config_path\"] <p>We can insert an entry into <code>DLCModelTrainingSelection</code> and populate <code>DLCModelTraining</code>.</p> <p>Note: You can stop training at any point using <code>I + I</code> or interrupt the Kernel.</p> <p>The maximum total number of training iterations is 1030000; you can end training before this amount if the loss rate (lr) and total loss plateau and are very close to 0.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTrainingSelection.heading\n</pre> sgp.DLCModelTrainingSelection.heading In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTrainingSelection().insert1(\n    {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n        \"training_id\": 0,\n        \"model_prefix\": \"\",\n    }\n)\nmodel_training_key = (\n    sgp.DLCModelTrainingSelection\n    &amp; {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n    }\n).fetch1(\"KEY\")\nsgp.DLCModelTraining.populate(model_training_key)\n</pre> sgp.DLCModelTrainingSelection().insert1(     {         **project_key,         \"dlc_training_params_name\": training_params_name,         \"training_id\": 0,         \"model_prefix\": \"\",     } ) model_training_key = (     sgp.DLCModelTrainingSelection     &amp; {         **project_key,         \"dlc_training_params_name\": training_params_name,     } ).fetch1(\"KEY\") sgp.DLCModelTraining.populate(model_training_key) <p>Here we'll make sure that the entry made it into the table properly!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTraining() &amp; model_training_key\n</pre> sgp.DLCModelTraining() &amp; model_training_key <p>Populating <code>DLCModelTraining</code> automatically inserts the entry into <code>DLCModelSource</code>, which is used to select between models trained using Spyglass vs. other tools.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource() &amp; model_training_key\n</pre> sgp.DLCModelSource() &amp; model_training_key <p>The <code>source</code> field will only accept \"FromImport\" or \"FromUpstream\" as entries. Let's checkout the <code>FromUpstream</code> part table attached to <code>DLCModelSource</code> below.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource.FromUpstream() &amp; model_training_key\n</pre> sgp.DLCModelSource.FromUpstream() &amp; model_training_key In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelParams.get_default()\n</pre> sgp.DLCModelParams.get_default() <p>Here is the syntax to add your own parameter set:</p> <pre>dlc_model_params_name = \"make_this_yours\"\nparams = {\n    \"params\": {},\n    \"shuffle\": 1,\n    \"trainingsetindex\": 0,\n    \"model_prefix\": \"\",\n}\nsgp.DLCModelParams.insert1(\n    {\"dlc_model_params_name\": dlc_model_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We can insert sets of parameters into <code>DLCModelSelection</code> and populate <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\")\n</pre> temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\") In\u00a0[\u00a0]: Copied! <pre># comment these lines out after successfully inserting, for each project\nsgp.DLCModelSelection().insert1(\n    {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True\n)\n</pre> # comment these lines out after successfully inserting, for each project sgp.DLCModelSelection().insert1(     {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True ) In\u00a0[\u00a0]: Copied! <pre>model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\")\nsgp.DLCModel.populate(model_key)\n</pre> model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\") sgp.DLCModel.populate(model_key) <p>Again, let's make sure that everything looks correct in <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModel() &amp; model_key\n</pre> sgp.DLCModel() &amp; model_key <p>We can view all <code>VideoFile</code> entries with the specidied <code>camera_ name</code> for this project to ensure the rat whose position you wish to model is in this table <code>matching_rows</code></p> In\u00a0[\u00a0]: Copied! <pre>camera_name = \"SleepBox_camera\"\nmatching_rows = sgc.VideoFile() &amp; {\"camera_name\": camera_name}\nmatching_rows\n</pre> camera_name = \"SleepBox_camera\" matching_rows = sgc.VideoFile() &amp; {\"camera_name\": camera_name} matching_rows <p>The <code>DLCPoseEstimationSelection</code> insertion step will convert your .h264 video to an .mp4 first and save it in <code>/nimbus/deeplabcut/video</code>. If this video already exists here, the insertion will never complete.</p> <p>We first delete any .mp4 that exists for this video from the nimbus folder:</p> In\u00a0[\u00a0]: Copied! <pre>! find /nimbus/deeplabcut/video -type f -name '*20230606_SC38*' -delete # change based on date and rat with which you are training the model\n</pre> ! find /nimbus/deeplabcut/video -type f -name '*20230606_SC38*' -delete # change based on date and rat with which you are training the model <p>If the first insertion step (for pose estimation task) fails in either trigger or load mode for an epoch, run the following lines:</p> <pre><code>(pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    }).delete()\n</code></pre> <p>This loop will generate posiiton data for all epochs associated with the pre-defined camera in one day, for one rat (based on the NWB file; see ***) The output should print Pose Estimation and Centroid plots for each epoch.</p> <ul> <li>It defines <code>col1val</code> as each <code>nwb_file_name</code> entry in the table, one at a time.</li> <li>Next, it sees if the trial on which you are testing this model is in the string for the current <code>col1val</code>; if not, it re-defines <code>col1val</code> as the next <code>nwb_file_name</code> entry and re-tries this step.</li> <li>If the previous step works, it then saves <code>col2val</code> and <code>col3val</code> as the <code>epoch</code> and the <code>video_file_num</code>, respectively, based on the nwb_file_name. From there, it iterates through the insertion and population steps required to extract position data, which we see laid out in notebook 05_DLC.ipynb.</li> </ul> In\u00a0[\u00a0]: Copied! <pre>for row in matching_rows:\n    col1val = row[\"nwb_file_name\"]\n    if \"SC3820230606\" in col1val:  # *** change depending on rat/day!!!\n        col2val = row[\"epoch\"]\n        col3val = row[\"video_file_num\"]\n\n        ##insert pose estimation task\n        pose_estimation_key = (\n            sgp.DLCPoseEstimationSelection.insert_estimation_task(\n                {\n                    \"nwb_file_name\": col1val,\n                    \"epoch\": col2val,\n                    \"video_file_num\": col3val,\n                    **model_key,\n                },\n                task_mode=\"trigger\",  # load or trigger\n                params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n            )\n        )\n\n        ##populate DLC Pose Estimation\n        sgp.DLCPoseEstimation().populate(pose_estimation_key)\n\n        ##start smooth interpolation\n        si_params_name = \"just_nan\"\n        si_key = pose_estimation_key.copy()\n        fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())\n        si_key = {key: val for key, val in si_key.items() if key in fields}\n        bodyparts = [\"greenLED\", \"redLED_C\"]\n        sgp.DLCSmoothInterpSelection.insert(\n            [\n                {\n                    **si_key,\n                    \"bodypart\": bodypart,\n                    \"dlc_si_params_name\": si_params_name,\n                }\n                for bodypart in bodyparts\n            ],\n            skip_duplicates=True,\n        )\n        sgp.DLCSmoothInterp().populate(si_key)\n        (\n            sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}\n        ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))\n\n        ##smoothinterpcohort\n        cohort_key = si_key.copy()\n        if \"bodypart\" in cohort_key:\n            del cohort_key[\"bodypart\"]\n        if \"dlc_si_params_name\" in cohort_key:\n            del cohort_key[\"dlc_si_params_name\"]\n        cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"\n        cohort_key[\"bodyparts_params_dict\"] = {\n            \"greenLED\": si_params_name,\n            \"redLED_C\": si_params_name,\n        }\n        sgp.DLCSmoothInterpCohortSelection().insert1(\n            cohort_key, skip_duplicates=True\n        )\n        sgp.DLCSmoothInterpCohort.populate(cohort_key)\n\n        ##DLC Centroid\n        centroid_params_name = \"default\"\n        centroid_key = cohort_key.copy()\n        fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())\n        centroid_key = {\n            key: val for key, val in centroid_key.items() if key in fields\n        }\n        centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name\n        sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)\n        sgp.DLCCentroid.populate(centroid_key)\n        (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(\n            x=\"position_x\",\n            y=\"position_y\",\n            c=\"speed\",\n            colormap=\"viridis\",\n            alpha=0.5,\n            s=0.5,\n            figsize=(10, 10),\n        )\n\n        ##DLC Orientation\n        dlc_orientation_params_name = \"default\"\n        fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())\n        orient_key = {\n            key: val for key, val in cohort_key.items() if key in fields\n        }\n        orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name\n        sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)\n        sgp.DLCOrientation().populate(orient_key)\n\n        ##DLCPosV1\n        fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())\n        dlc_key = {\n            key: val for key, val in centroid_key.items() if key in fields\n        }\n        dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\n            \"dlc_si_cohort_selection_name\"\n        ]\n        dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[\n            \"dlc_si_cohort_selection_name\"\n        ]\n        dlc_key[\"dlc_orientation_params_name\"] = orient_key[\n            \"dlc_orientation_params_name\"\n        ]\n        sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)\n        sgp.DLCPosV1().populate(dlc_key)\n\n    else:\n        continue\n</pre> for row in matching_rows:     col1val = row[\"nwb_file_name\"]     if \"SC3820230606\" in col1val:  # *** change depending on rat/day!!!         col2val = row[\"epoch\"]         col3val = row[\"video_file_num\"]          ##insert pose estimation task         pose_estimation_key = (             sgp.DLCPoseEstimationSelection.insert_estimation_task(                 {                     \"nwb_file_name\": col1val,                     \"epoch\": col2val,                     \"video_file_num\": col3val,                     **model_key,                 },                 task_mode=\"trigger\",  # load or trigger                 params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},             )         )          ##populate DLC Pose Estimation         sgp.DLCPoseEstimation().populate(pose_estimation_key)          ##start smooth interpolation         si_params_name = \"just_nan\"         si_key = pose_estimation_key.copy()         fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())         si_key = {key: val for key, val in si_key.items() if key in fields}         bodyparts = [\"greenLED\", \"redLED_C\"]         sgp.DLCSmoothInterpSelection.insert(             [                 {                     **si_key,                     \"bodypart\": bodypart,                     \"dlc_si_params_name\": si_params_name,                 }                 for bodypart in bodyparts             ],             skip_duplicates=True,         )         sgp.DLCSmoothInterp().populate(si_key)         (             sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}         ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))          ##smoothinterpcohort         cohort_key = si_key.copy()         if \"bodypart\" in cohort_key:             del cohort_key[\"bodypart\"]         if \"dlc_si_params_name\" in cohort_key:             del cohort_key[\"dlc_si_params_name\"]         cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"         cohort_key[\"bodyparts_params_dict\"] = {             \"greenLED\": si_params_name,             \"redLED_C\": si_params_name,         }         sgp.DLCSmoothInterpCohortSelection().insert1(             cohort_key, skip_duplicates=True         )         sgp.DLCSmoothInterpCohort.populate(cohort_key)          ##DLC Centroid         centroid_params_name = \"default\"         centroid_key = cohort_key.copy()         fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())         centroid_key = {             key: val for key, val in centroid_key.items() if key in fields         }         centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name         sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)         sgp.DLCCentroid.populate(centroid_key)         (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(             x=\"position_x\",             y=\"position_y\",             c=\"speed\",             colormap=\"viridis\",             alpha=0.5,             s=0.5,             figsize=(10, 10),         )          ##DLC Orientation         dlc_orientation_params_name = \"default\"         fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())         orient_key = {             key: val for key, val in cohort_key.items() if key in fields         }         orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name         sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)         sgp.DLCOrientation().populate(orient_key)          ##DLCPosV1         fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())         dlc_key = {             key: val for key, val in centroid_key.items() if key in fields         }         dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[             \"dlc_si_cohort_selection_name\"         ]         dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[             \"dlc_si_cohort_selection_name\"         ]         dlc_key[\"dlc_orientation_params_name\"] = orient_key[             \"dlc_orientation_params_name\"         ]         sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)         sgp.DLCPosV1().populate(dlc_key)      else:         continue Return To Table of Contents"}, {"location": "notebooks/22_DLC_Loop/#position-deeplabcut-from-scratch", "title": "Position- DeepLabCut from Scratch\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#table-of-contents", "title": "Table of Contents\u00b6", "text": "<p><code>DLCProject</code> <code>DLCModelTraining</code> <code>DLCModel</code> <code>DLCPoseEstimation</code> <code>DLCSmoothInterp</code> <code>DLCCentroid</code> <code>DLCOrientation</code> <code>DLCPosV1</code> <code>DLCPosVideo</code> <code>PositionOutput</code></p>"}, {"location": "notebooks/22_DLC_Loop/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#dlcproject", "title": "DLCProject \u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#body-parts", "title": "Body Parts\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#define-videos-and-camera-name-optional-for-training-set", "title": "Define videos and camera name (optional) for training set\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#path-variables", "title": "Path variables\u00b6", "text": "<p>The position pipeline also keeps track of paths for project, video, and output. Just like we saw in Setup, you can manage these either with environmental variables...</p> <pre>export DLC_PROJECT_DIR=\"/nimbus/deeplabcut/projects\"\nexport DLC_VIDEO_DIR=\"/nimbus/deeplabcut/video\"\nexport DLC_OUTPUT_DIR=\"/nimbus/deeplabcut/output\"\n</pre> <p>Or these can be set in your datajoint config:</p> <pre>{\n  \"custom\": {\n    \"dlc_dirs\": {\n      \"base\": \"/nimbus/deeplabcut/\",\n      \"project\": \"/nimbus/deeplabcut/projects\",\n      \"video\": \"/nimbus/deeplabcut/video\",\n      \"output\": \"/nimbus/deeplabcut/output\"\n    }\n  }\n}\n</pre> <p>NOTE: If only <code>base</code> is specified as shown above, spyglass will assume the relative directories shown.</p> <p>You can check the result of this setup process with...</p>"}, {"location": "notebooks/22_DLC_Loop/#dlcmodeltraining", "title": "DLCModelTraining\u00b6", "text": "<p>Please make sure you're running this notebook on a GPU-enabled machine.</p> <p>Now that we've imported existing frames, we can get ready to train our model.</p> <p>First, we'll need to define a set of parameters for <code>DLCModelTrainingParams</code>, which will get used by DeepLabCut during training. Let's start with <code>gputouse</code>, which determines which GPU core to use.</p> <p>The cell below determines which core has space and set the <code>gputouse</code> variable accordingly.</p>"}, {"location": "notebooks/22_DLC_Loop/#dlcmodel", "title": "DLCModel \u00b6", "text": "<p>Next we'll populate the <code>DLCModel</code> table, which holds all the relevant information for all trained models.</p> <p>First, we'll need to determine a set of parameters for our model to select the correct model file. Here is the default:</p>"}, {"location": "notebooks/22_DLC_Loop/#loop-begins", "title": "Loop Begins\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#congratulations", "title": "CONGRATULATIONS!!\u00b6", "text": "<p>Please treat yourself to a nice tea break :-)</p>"}, {"location": "notebooks/24_Linearization/", "title": "Linearization", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This pipeline takes 2D position data from the <code>PositionOutput</code> table and \"linearizes\" it to 1D position. If you haven't already done so, please generate input data with either the Trodes or DLC notebooks (1, 2, 3).</p> In\u00a0[1]: Copied! <pre>%reload_ext autoreload\n%autoreload 2\n</pre> %reload_ext autoreload %autoreload 2 In\u00a0[2]: Copied! <pre>import os\nimport pynwb\nimport numpy as np\nimport datajoint as dj\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\nimport spyglass.linearization.v1 as sgpl\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import pynwb import numpy as np import datajoint as dj import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position.v1 as sgp import spyglass.linearization.v1 as sgpl  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-24 09:19:28,736][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-01-24 09:19:28,767][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n[09:19:30][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> <p>To retrieve 2D position data, we'll specify an nwb file, a position time interval, and the set of parameters used to compute the position info.</p> In\u00a0[3]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\nnwb_file_name = \"minirec20230622.nwb\"  # detailed data: chimi20200216_new.nwb\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nnwb_copy_file_name\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  nwb_file_name = \"minirec20230622.nwb\"  # detailed data: chimi20200216_new.nwb nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) nwb_copy_file_name Out[3]: <pre>'chimi20200216_new_.nwb'</pre> <p>We will fetch the pandas dataframe from the <code>PositionOutput</code> table.</p> In\u00a0[4]: Copied! <pre>from spyglass.position import PositionOutput\nimport pandas as pd\n\npos_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",  # For chimi, \"pos 1 valid times\"\n    \"trodes_pos_params_name\": \"single_led_upsampled\",  # For chimi, \"default\"\n}\n\n# Note: You'll have to change the part table to the one where your data came from\nmerge_id = (PositionOutput.TrodesPosV1() &amp; pos_key).fetch1(\"merge_id\")\nposition_info = (PositionOutput &amp; {\"merge_id\": merge_id}).fetch1_dataframe()\nposition_info\n</pre> from spyglass.position import PositionOutput import pandas as pd  pos_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",  # For chimi, \"pos 1 valid times\"     \"trodes_pos_params_name\": \"single_led_upsampled\",  # For chimi, \"default\" }  # Note: You'll have to change the part table to the one where your data came from merge_id = (PositionOutput.TrodesPosV1() &amp; pos_key).fetch1(\"merge_id\") position_info = (PositionOutput &amp; {\"merge_id\": merge_id}).fetch1_dataframe() position_info Out[4]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.581887e+09 0 91.051650 211.127050 2.999696 1.387074 2.848838 3.168573 1.581887e+09 1 90.844337 211.417287 3.078386 3.123201 3.411111 4.624939 1.581887e+09 2 90.637025 211.707525 -3.114572 5.431643 4.089597 6.799085 1.581887e+09 3 90.802875 211.596958 -3.033109 8.097753 4.979262 9.506138 1.581887e+09 4 91.288579 211.482443 -3.062550 10.840482 6.071373 12.424880 ... ... ... ... ... ... ... ... 1.581888e+09 39335 182.158583 201.452467 -0.986926 0.348276 0.218575 0.411182 1.581888e+09 39336 182.158583 201.397183 -0.978610 0.279135 -0.058413 0.285182 1.581888e+09 39337 182.213867 201.341900 -0.957589 0.193798 -0.283200 0.343162 1.581888e+09 39338 182.158583 201.341900 -0.970083 0.110838 -0.417380 0.431846 1.581888e+09 39339 182.158583 201.286617 -0.936414 0.045190 -0.453966 0.456209 <p>39340 rows \u00d7 7 columns</p> <p>Before linearizing, plotting the head position will help us understand the data.</p> In\u00a0[5]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(5, 5))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\nax.set_title(\"Head Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(5, 5)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\", ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) ax.set_title(\"Head Position\", fontsize=28) Out[5]: <pre>Text(0.5, 1.0, 'Head Position')</pre> <p>To linearize, we need a graph of nodes and edges to represent track geometry in the <code>TrackGraph</code> table with four variables:</p> <ul> <li><code>node_positions</code> (cm): the 2D positions of the graph</li> <li><code>edges</code>: how the nodes are connected, as pairs of node indices, labeled by their respective index in <code>node_positions</code>.</li> <li><code>linear_edge_order</code>: layout of edges in linear space in order, as tuples.</li> <li><code>linear_edge_spacing</code>: spacing between each edge, as either a single number for all gaps or an array with a number for each gap. Gaps may be important for edges not connected in 2D space.</li> </ul> <p>For example, (79.910, 216.720) is the 2D position of node 0 and (183.784, 45.375) is the 2D position of node 8. Edge (0, 8) means there is an edge between node 0 and node 8. Nodes order controls order in 1D space. Edge (0, 1) connects from node 0 to 1. Edge (1, 0) would connect from node 1 to 0, reversing the linear positions for that edge.</p> <p>For more examples, see this notebook.</p> In\u00a0[6]: Copied! <pre>node_positions = np.array(\n    [\n        (79.910, 216.720),  # top left well 0\n        (132.031, 187.806),  # top middle intersection 1\n        (183.718, 217.713),  # top right well 2\n        (132.544, 132.158),  # middle intersection 3\n        (87.202, 101.397),  # bottom left intersection 4\n        (31.340, 126.110),  # middle left well 5\n        (180.337, 104.799),  # middle right intersection 6\n        (92.693, 42.345),  # bottom left well 7\n        (183.784, 45.375),  # bottom right well 8\n        (231.338, 136.281),  # middle right well 9\n    ]\n)\n\nedges = np.array(\n    [\n        (0, 1),\n        (1, 2),\n        (1, 3),\n        (3, 4),\n        (4, 5),\n        (3, 6),\n        (6, 9),\n        (4, 7),\n        (6, 8),\n    ]\n)\n\nlinear_edge_order = [\n    (3, 6),\n    (6, 8),\n    (6, 9),\n    (3, 1),\n    (1, 2),\n    (1, 0),\n    (3, 4),\n    (4, 5),\n    (4, 7),\n]\nlinear_edge_spacing = 15\n</pre> node_positions = np.array(     [         (79.910, 216.720),  # top left well 0         (132.031, 187.806),  # top middle intersection 1         (183.718, 217.713),  # top right well 2         (132.544, 132.158),  # middle intersection 3         (87.202, 101.397),  # bottom left intersection 4         (31.340, 126.110),  # middle left well 5         (180.337, 104.799),  # middle right intersection 6         (92.693, 42.345),  # bottom left well 7         (183.784, 45.375),  # bottom right well 8         (231.338, 136.281),  # middle right well 9     ] )  edges = np.array(     [         (0, 1),         (1, 2),         (1, 3),         (3, 4),         (4, 5),         (3, 6),         (6, 9),         (4, 7),         (6, 8),     ] )  linear_edge_order = [     (3, 6),     (6, 8),     (6, 9),     (3, 1),     (1, 2),     (1, 0),     (3, 4),     (4, 5),     (4, 7), ] linear_edge_spacing = 15 <p>With these variables, we then add a <code>track_graph_name</code> and the corresponding <code>environment</code>.</p> In\u00a0[7]: Copied! <pre>sgpl.TrackGraph.insert1(\n    {\n        \"track_graph_name\": \"6 arm\",\n        \"environment\": \"6 arm\",\n        \"node_positions\": node_positions,\n        \"edges\": edges,\n        \"linear_edge_order\": linear_edge_order,\n        \"linear_edge_spacing\": linear_edge_spacing,\n    },\n    skip_duplicates=True,\n)\n\ngraph = sgpl.TrackGraph &amp; {\"track_graph_name\": \"6 arm\"}\ngraph\n</pre> sgpl.TrackGraph.insert1(     {         \"track_graph_name\": \"6 arm\",         \"environment\": \"6 arm\",         \"node_positions\": node_positions,         \"edges\": edges,         \"linear_edge_order\": linear_edge_order,         \"linear_edge_spacing\": linear_edge_spacing,     },     skip_duplicates=True, )  graph = sgpl.TrackGraph &amp; {\"track_graph_name\": \"6 arm\"} graph Out[7]: <p>track_graph_name</p> <p>environment</p> Type of Environment <p>node_positions</p> 2D position of track_graph nodes, shape (n_nodes, 2) <p>edges</p> shape (n_edges, 2) <p>linear_edge_order</p> order of track graph edges in the linear space, shape (n_edges, 2) <p>linear_edge_spacing</p> amount of space between edges in the linear space, shape (n_edges,) 6 arm 6 arm =BLOB= =BLOB= =BLOB= =BLOB= <p>Total: 1</p> <p><code>TrackGraph</code> has several methods for visualizing in 2D and 1D space. <code>plot_track_graph</code> plots in 2D to make sure our layout makes sense.</p> In\u00a0[8]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(5, 5))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n    alpha=0.7,\n    zorder=0,\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\ngraph.plot_track_graph(ax=ax)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(5, 5)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\",     alpha=0.7,     zorder=0, ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) graph.plot_track_graph(ax=ax) <p><code>plot_track_graph_as_1D</code> shows what this looks like in 1D.</p> In\u00a0[9]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 1))\ngraph.plot_track_graph_as_1D(ax=ax)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 1)) graph.plot_track_graph_as_1D(ax=ax) <p>By default, linearization assigns each 2D position to its nearest point on the track graph. This is then translated into 1D space.</p> <p>If <code>use_hmm</code> is set to <code>true</code>, a Hidden Markov model is used to assign points. The HMM takes into account the prior position and edge, and can keep the position from suddenly jumping to another. Position jumping like this may occur at intersections or the head position swings closer to a non-target reward well while on another edge.</p> In\u00a0[10]: Copied! <pre>sgpl.LinearizationParameters.insert1(\n    {\"linearization_param_name\": \"default\"}, skip_duplicates=True\n)\nsgpl.LinearizationParameters()\n</pre> sgpl.LinearizationParameters.insert1(     {\"linearization_param_name\": \"default\"}, skip_duplicates=True ) sgpl.LinearizationParameters() Out[10]: <p>linearization_param_name</p> name for this set of parameters <p>use_hmm</p> use HMM to determine linearization <p>route_euclidean_distance_scaling</p> <p>sensor_std_dev</p> Uncertainty of position sensor (in cm). <p>diagonal_bias</p> default 0 1.0 5.0 0.5 <p>Total: 1</p> <p>With linearization parameters, we specify the position interval we wish to linearize from the <code>PositionOutput</code> table and create an entry in <code>LinearizationSelection</code></p> In\u00a0[11]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[11]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> chimi20200216_new_.nwb chimi University of California, San Francisco Loren Frank chimi_20200216 Spatial bandit task (regular) 2020-02-16 12:15:09 1970-01-01 00:00:00 Memory and value guided decision making <p>Total: 1</p> In\u00a0[12]: Copied! <pre>sgpl.LinearizationSelection.insert1(\n    {\n        \"pos_merge_id\": merge_id,\n        \"track_graph_name\": \"6 arm\",\n        \"linearization_param_name\": \"default\",\n    },\n    skip_duplicates=True,\n)\n\nsgpl.LinearizationSelection()\n</pre> sgpl.LinearizationSelection.insert1(     {         \"pos_merge_id\": merge_id,         \"track_graph_name\": \"6 arm\",         \"linearization_param_name\": \"default\",     },     skip_duplicates=True, )  sgpl.LinearizationSelection() Out[12]: <p>pos_merge_id</p> <p>track_graph_name</p> <p>linearization_param_name</p> name for this set of parameters af2e637c-47d9-8e27-11f1-3ac12431f417 6 arm default001b5af0-489e-5437-9e34-f2130727ebc5 ms_lineartrack default002476a5-c344-0236-d86c-89b995e22227 ms_lineartrack default00551a35-a28e-c884-0c33-8f3817cf59c2 ms_lineartrack default006bfebf-d986-e309-06ed-d7ea7c871c46 ms_lineartrack default00a9088c-6746-7892-d2ca-2e86e9047a60 ms_lineartrack default00afca31-a7b3-7d44-4752-531de37263a3 ms_lineartrack default00c9698e-146e-3bdc-61e4-c19ad4c8840d ms_lineartrack default00eaf8e9-168b-9c46-1621-e11b3bee558d ms_lineartrack default01717e02-3d2c-14ad-0a74-d1bd7f8417f0 ms_lineartrack default022c034d-da79-ca8c-a758-6c2e78b65fcf ms_lineartrack default025149ec-2d7e-4e52-3748-2efdb7f100a7 ms_lineartrack default <p>...</p> <p>Total: 1970</p> <p>And then run linearization by populating <code>LinearizedPositionV1</code>.</p> In\u00a0[13]: Copied! <pre>sgpl.LinearizedPositionV1().populate()\nsgpl.LinearizedPositionV1()\n</pre> sgpl.LinearizedPositionV1().populate() sgpl.LinearizedPositionV1() Out[13]: <p>pos_merge_id</p> <p>track_graph_name</p> <p>linearization_param_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>linearized_position_object_id</p> 001b5af0-489e-5437-9e34-f2130727ebc5 ms_lineartrack default Winnie20220714_8FOZKCPEXW.nwb 9b7b03d8-1df8-4a04-9566-ada9ad7d935e002476a5-c344-0236-d86c-89b995e22227 ms_lineartrack default Totoro20220614_BSOZ7EAWQU.nwb ddb91b90-8d36-4908-aa7c-2d117461d7a9002476a5-c344-0236-d86c-89b995e22227 ms_wtrack default Totoro20220614_GPQ1PDBI6S.nwb ee0a93ba-1233-4b1a-bffc-c810a8ba46b800551a35-a28e-c884-0c33-8f3817cf59c2 ms_lineartrack default Olive20220707_FIDPJQ458I.nwb 71ab61e9-cf9b-436c-b489-9f09b6f989da006bfebf-d986-e309-06ed-d7ea7c871c46 ms_lineartrack default Totoro20220530_X9K189ZS6H.nwb dff301c8-eb66-4b11-bca0-d5e05fd922c100a9088c-6746-7892-d2ca-2e86e9047a60 ms_lineartrack default Olive20220711_6QS37PFOW3.nwb 70a78fcd-7e17-43ba-8cb5-7f918047d68400a9088c-6746-7892-d2ca-2e86e9047a60 ms_wtrack default Olive20220711_8NMXZVVD5L.nwb 27b1097a-d612-4967-ac42-3699cc1de5b200afca31-a7b3-7d44-4752-531de37263a3 ms_lineartrack default Wallie20220922_EP1QE0SBJR.nwb b2797c39-606f-43cb-8d8f-652870b9db6700c9698e-146e-3bdc-61e4-c19ad4c8840d ms_lineartrack default Olive20220708_XXXT7RYLRX.nwb 8bbd3fb5-6cab-491a-8d74-c0681867824800eaf8e9-168b-9c46-1621-e11b3bee558d ms_lineartrack default Winnie20220717_2BIU1TYFMB.nwb 2b590d50-445a-4a5a-965e-1d64f3d8eafb01717e02-3d2c-14ad-0a74-d1bd7f8417f0 ms_lineartrack default Olive20220707_4WHPR9WFOK.nwb 8a31926a-132f-4957-b2ce-d5a2647f480f022c034d-da79-ca8c-a758-6c2e78b65fcf ms_lineartrack default Winnie20220720_F3YE2F28QX.nwb 2fc84409-b55f-44b0-9d72-740fdf5269c4 <p>...</p> <p>Total: 1970</p> <p>Populating <code>LinearizedPositionV1</code> also creates a corresponding entry in the <code>LinearizedPositionOutput</code> merge table. For downstream compatibility with alternate versions of the Linearization pipeline, we should fetch our data from here</p> <p>Running <code>fetch1_dataframe</code> will retrieve the linear position data, including...</p> <ul> <li><code>time</code>: dataframe index</li> <li><code>linear_position</code>: 1D linearized position</li> <li><code>track_segment_id</code>: index number of the edges given to track graph</li> <li><code>projected_{x,y}_position</code>: 2D position projected to the track graph</li> </ul> In\u00a0[18]: Copied! <pre>linear_key = {\n    \"pos_merge_id\": merge_id,\n    \"track_graph_name\": \"6 arm\",\n    \"linearization_param_name\": \"default\",\n}\n\nfrom spyglass.linearization.merge import LinearizedPositionOutput\n\nlinear_merge_key = LinearizedPositionOutput.merge_restrict(linear_key).fetch1(\n    \"KEY\"\n)\nlinear_position_df = (\n    LinearizedPositionOutput &amp; linear_merge_key\n).fetch1_dataframe()\nlinear_position_df\n</pre> linear_key = {     \"pos_merge_id\": merge_id,     \"track_graph_name\": \"6 arm\",     \"linearization_param_name\": \"default\", }  from spyglass.linearization.merge import LinearizedPositionOutput  linear_merge_key = LinearizedPositionOutput.merge_restrict(linear_key).fetch1(     \"KEY\" ) linear_position_df = (     LinearizedPositionOutput &amp; linear_merge_key ).fetch1_dataframe() linear_position_df Out[18]: linear_position track_segment_id projected_x_position projected_y_position time 1.581887e+09 412.042773 0 90.802281 210.677533 1.581887e+09 412.364853 0 90.520636 210.833775 1.581887e+09 412.686934 0 90.238990 210.990018 1.581887e+09 412.488270 0 90.412714 210.893645 1.581887e+09 412.007991 0 90.832697 210.660660 ... ... ... ... ... 1.581888e+09 340.401589 1 175.500994 212.958497 1.581888e+09 340.373902 1 175.477029 212.944630 1.581888e+09 340.394065 1 175.494481 212.954729 1.581888e+09 340.346214 1 175.453064 212.930764 1.581888e+09 340.318527 1 175.429100 212.916898 <p>39340 rows \u00d7 4 columns</p> <p>We'll plot the 1D position over time, colored by edge, and use the 1D track graph layout on the y-axis.</p> In\u00a0[19]: Copied! <pre>fig, ax = plt.subplots(figsize=(20, 13))\nax.scatter(\n    linear_position_df.index,\n    linear_position_df.linear_position,\n    c=linear_position_df.track_segment_id,\n    s=1,\n)\ngraph.plot_track_graph_as_1D(\n    ax=ax, axis=\"y\", other_axis_start=linear_position_df.index[-1] + 10\n)\n\nax.set_xlabel(\"Time [s]\", fontsize=18)\nax.set_ylabel(\"Linear Position [cm]\", fontsize=18)\nax.set_title(\"Linear Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(figsize=(20, 13)) ax.scatter(     linear_position_df.index,     linear_position_df.linear_position,     c=linear_position_df.track_segment_id,     s=1, ) graph.plot_track_graph_as_1D(     ax=ax, axis=\"y\", other_axis_start=linear_position_df.index[-1] + 10 )  ax.set_xlabel(\"Time [s]\", fontsize=18) ax.set_ylabel(\"Linear Position [cm]\", fontsize=18) ax.set_title(\"Linear Position\", fontsize=28) Out[19]: <pre>Text(0.5, 1.0, 'Linear Position')</pre> <p>We can also plot the 2D position projected to the track graph</p> In\u00a0[20]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 10))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n    alpha=0.7,\n    zorder=0,\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\nax.plot(\n    linear_position_df.projected_x_position,\n    linear_position_df.projected_y_position,\n)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 10)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\",     alpha=0.7,     zorder=0, ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) ax.plot(     linear_position_df.projected_x_position,     linear_position_df.projected_y_position, ) Out[20]: <pre>[&lt;matplotlib.lines.Line2D at 0x7fc2ffb63040&gt;]</pre>"}, {"location": "notebooks/24_Linearization/#position-linearization", "title": "Position - Linearization\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#retrieve-2d-position", "title": "Retrieve 2D position\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#specifying-the-track", "title": "Specifying the track\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#parameters", "title": "Parameters\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#linearization", "title": "Linearization\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#examine-data", "title": "Examine data\u00b6", "text": ""}, {"location": "notebooks/30_LFP/", "title": "LFP", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> In\u00a0[2]: Copied! <pre>import os\nimport copy\nimport datajoint as dj\nimport numpy as np\nimport pandas as pd\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.lfp as lfp\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import copy import datajoint as dj import numpy as np import pandas as pd  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.lfp as lfp  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-12 08:39:21,871][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-01-12 08:39:21,915][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\nWARNING:root:Populate: Entry in DataAcquisitionDeviceSystem with primary keys {'data_acquisition_device_system': 'SpikeGadgets'} already exists.\nWARNING:root:Populate: Entry in DataAcquisitionDeviceAmplifier with primary keys {'data_acquisition_device_amplifier': 'Intan'} already exists.\n</pre> <p>First, we select the NWB file, which corresponds to the dataset we want to extract LFP from.</p> In\u00a0[3]: Copied! <pre>nwb_file_name = \"minirec20230622_.nwb\"\n</pre> nwb_file_name = \"minirec20230622_.nwb\" <p>Next, we create the standard LFP Filters. This only needs to be done once.</p> In\u00a0[4]: Copied! <pre>sgc.FirFilterParameters().create_standard_filters()\nsgc.FirFilterParameters()\n</pre> sgc.FirFilterParameters().create_standard_filters() sgc.FirFilterParameters() Out[4]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Delta 0.5-4 Hz 1000 lowpass 0.25 0.5 4.0 5.0 delta filter for 1 KHz data =BLOB= =BLOB=Delta 0.5-4 Hz pass, 0.25-4.5 Hz stop 1000 lowpass 0.25 0.5 4.0 4.5 revised delta filter for 1 KHz data =BLOB= =BLOB=Epilepsy project 0.2-40 Hz 2000 bandpass 0.1 0.2 40.0 41.0 Sharp wave filter =BLOB= =BLOB=Epilepsy project 140-800 Hz 2000 bandpass 135.0 140.0 800.0 825.0 Extended ripple filter =BLOB= =BLOB=Epilepsy project 140-800 Hz ripple 2000 bandpass 135.0 140.0 800.0 825.0 Extended ripple filter =BLOB= =BLOB=Epilepsy project 40-600 Hz 2000 bandpass 39.0 40.0 600.0 625.0 HFE filter =BLOB= =BLOB=Fast Gamma 65-100 Hz 1000 bandpass 55.0 65.0 100.0 110.0 slow gamma filter for 1 Khz data =BLOB= =BLOB=LFP 0-400 Hz 1000 lowpass 0.0 0.0 400.0 425.0 LFP filter for referencing =BLOB= =BLOB=LFP 0-400 Hz 20000 lowpass 0.0 0.0 400.0 425.0 standard LFP filter for 20 KHz data =BLOB= =BLOB=LFP 0-400 Hz 29995 lowpass 0.0 0.0 400.0 425.0 adapted LFP filter for Banner20220105 =BLOB= =BLOB=LFP 0-400 Hz 30000 lowpass 0.0 0.0 400.0 425.0 standard LFP filter for 20 KHz data =BLOB= =BLOB=LFP 0-800 Hz 30000 lowpass 0.0 0.0 800.0 825.0 LFP filter 0-800 Hz =BLOB= =BLOB= <p>...</p> <p>Total: 31</p> <p>Now, we create an LFP electrode group, or the set of electrodes we want to filter for LFP data. We can grab all electrodes and brain regions as a data frame.</p> In\u00a0[5]: Copied! <pre>electrodes_df = (\n    pd.DataFrame(\n        (sgc.Electrode &amp; {\"nwb_file_name\": nwb_file_name, \"probe_electrode\": 0})\n        * sgc.BrainRegion\n    )\n    .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]\n    .sort_values(by=\"electrode_id\")\n)\nelectrodes_df\n</pre> electrodes_df = (     pd.DataFrame(         (sgc.Electrode &amp; {\"nwb_file_name\": nwb_file_name, \"probe_electrode\": 0})         * sgc.BrainRegion     )     .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]     .sort_values(by=\"electrode_id\") ) electrodes_df Out[5]: nwb_file_name electrode_id region_name 0 minirec20230622_.nwb 0 corpus callosum and associated subcortical whi... 1 minirec20230622_.nwb 4 corpus callosum and associated subcortical whi... 12 minirec20230622_.nwb 8 corpus callosum and associated subcortical whi... 23 minirec20230622_.nwb 12 corpus callosum and associated subcortical whi... 26 minirec20230622_.nwb 16 corpus callosum and associated subcortical whi... 27 minirec20230622_.nwb 20 corpus callosum and associated subcortical whi... 28 minirec20230622_.nwb 24 corpus callosum and associated subcortical whi... 29 minirec20230622_.nwb 28 corpus callosum and associated subcortical whi... 30 minirec20230622_.nwb 32 corpus callosum and associated subcortical whi... 31 minirec20230622_.nwb 36 corpus callosum and associated subcortical whi... 2 minirec20230622_.nwb 40 corpus callosum and associated subcortical whi... 3 minirec20230622_.nwb 44 corpus callosum and associated subcortical whi... 4 minirec20230622_.nwb 48 corpus callosum and associated subcortical whi... 5 minirec20230622_.nwb 52 corpus callosum and associated subcortical whi... 6 minirec20230622_.nwb 56 corpus callosum and associated subcortical whi... 7 minirec20230622_.nwb 60 corpus callosum and associated subcortical whi... 8 minirec20230622_.nwb 64 corpus callosum and associated subcortical whi... 9 minirec20230622_.nwb 68 corpus callosum and associated subcortical whi... 10 minirec20230622_.nwb 72 corpus callosum and associated subcortical whi... 11 minirec20230622_.nwb 76 corpus callosum and associated subcortical whi... 13 minirec20230622_.nwb 80 corpus callosum and associated subcortical whi... 14 minirec20230622_.nwb 84 corpus callosum and associated subcortical whi... 15 minirec20230622_.nwb 88 corpus callosum and associated subcortical whi... 16 minirec20230622_.nwb 92 corpus callosum and associated subcortical whi... 17 minirec20230622_.nwb 96 corpus callosum and associated subcortical whi... 18 minirec20230622_.nwb 100 corpus callosum and associated subcortical whi... 19 minirec20230622_.nwb 104 corpus callosum and associated subcortical whi... 20 minirec20230622_.nwb 108 corpus callosum and associated subcortical whi... 21 minirec20230622_.nwb 112 corpus callosum and associated subcortical whi... 22 minirec20230622_.nwb 116 corpus callosum and associated subcortical whi... 24 minirec20230622_.nwb 120 corpus callosum and associated subcortical whi... 25 minirec20230622_.nwb 124 corpus callosum and associated subcortical whi... <p>For a larger dataset, we might want to filter by region, but our example data only has one electrode.</p> <pre>lfp_electrode_ids = electrodes_df.loc[\n    electrodes_df.region_name == \"ca1\"\n].electrode_id\n</pre> In\u00a0[6]: Copied! <pre>lfp_electrode_ids = [0]\nlfp_electrode_group_name = \"test\"\nlfp_eg_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"lfp_electrode_group_name\": lfp_electrode_group_name,\n}\n\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_file_name,\n    group_name=lfp_electrode_group_name,\n    electrode_list=lfp_electrode_ids,\n)\n</pre> lfp_electrode_ids = [0] lfp_electrode_group_name = \"test\" lfp_eg_key = {     \"nwb_file_name\": nwb_file_name,     \"lfp_electrode_group_name\": lfp_electrode_group_name, }  lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_file_name,     group_name=lfp_electrode_group_name,     electrode_list=lfp_electrode_ids, ) <p>We can verify the electrode list as follows</p> In\u00a0[7]: Copied! <pre>lfp.lfp_electrode.LFPElectrodeGroup.LFPElectrode() &amp; {\n    \"nwb_file_name\": nwb_file_name\n}\n</pre> lfp.lfp_electrode.LFPElectrodeGroup.LFPElectrode() &amp; {     \"nwb_file_name\": nwb_file_name } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode minirec20230622_.nwb test 0 0 <p>Total: 1</p> <p>Recall from the previous notebook that <code>IntervalList</code> selects time frames from the experiment. We can select the interval and subset to the first <code>n</code> seconds...</p> In\u00a0[8]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name} Out[8]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1 =BLOB= minirec20230622_.nwb 01_s1_first9 =BLOB= minirec20230622_.nwb 02_s2 =BLOB= minirec20230622_.nwb pos 0 valid times =BLOB= minirec20230622_.nwb pos 1 valid times =BLOB= minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 6</p> In\u00a0[9]: Copied! <pre>n = 9\norig_interval_list_name = \"01_s1\"\ninterval_list_name = orig_interval_list_name + f\"_first{n}\"\n\nvalid_times = (\n    sgc.IntervalList\n    &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_list_name\": orig_interval_list_name,\n    }\n).fetch1(\"valid_times\")\n\ninterval_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"valid_times\": np.asarray([[valid_times[0, 0], valid_times[0, 0] + n]]),\n}\n\nsgc.IntervalList.insert1(\n    interval_key,\n    skip_duplicates=True,\n)\n</pre> n = 9 orig_interval_list_name = \"01_s1\" interval_list_name = orig_interval_list_name + f\"_first{n}\"  valid_times = (     sgc.IntervalList     &amp; {         \"nwb_file_name\": nwb_file_name,         \"interval_list_name\": orig_interval_list_name,     } ).fetch1(\"valid_times\")  interval_key = {     \"nwb_file_name\": nwb_file_name,     \"interval_list_name\": interval_list_name,     \"valid_times\": np.asarray([[valid_times[0, 0], valid_times[0, 0] + n]]), }  sgc.IntervalList.insert1(     interval_key,     skip_duplicates=True, ) In\u00a0[10]: Copied! <pre>sgc.IntervalList() &amp; {\n    \"nwb_file_name\": nwb_file_name,\n    \"interval_list_name\": interval_list_name,\n}\n</pre> sgc.IntervalList() &amp; {     \"nwb_file_name\": nwb_file_name,     \"interval_list_name\": interval_list_name, } Out[10]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1_first9 =BLOB= <p>Total: 1</p> <p>LFPSelection combines the data, interval and filter</p> In\u00a0[23]: Copied! <pre>lfp_s_key = copy.deepcopy(lfp_eg_key)\n\nlfp_s_key.update(\n    {\n        \"target_interval_list_name\": interval_list_name,\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)\n        \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)\n    }\n)\n\nlfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)\n</pre> lfp_s_key = copy.deepcopy(lfp_eg_key)  lfp_s_key.update(     {         \"target_interval_list_name\": interval_list_name,         \"filter_name\": \"LFP 0-400 Hz\",         \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)         \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)     } )  lfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True) In\u00a0[24]: Copied! <pre>lfp.v1.LFPSelection() &amp; lfp_s_key\n</pre> lfp.v1.LFPSelection() &amp; lfp_s_key Out[24]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>target_sampling_rate</p> the desired output sampling rate, in HZ minirec20230622_.nwb test 01_s1_first9 LFP 0-400 Hz 30000 1000.0 <p>Total: 1</p> <p><code>LFPV1</code> has a similar <code>populate</code> command as we've seen before.</p> <p>Notes:</p> <ul> <li>For full recordings, this takes ~2h when done locally, for all electrodes</li> <li>This <code>populate</code> also inserts into the LFP Merge Table, <code>LFPOutput</code>. For more on Merge Tables, see our documentation. In short, this collects different LFP processing streams into one table.</li> </ul> In\u00a0[25]: Copied! <pre>lfp.v1.LFPV1().populate(lfp_s_key)\n</pre> lfp.v1.LFPV1().populate(lfp_s_key) <p>We can now look at the LFP table to see the data we've extracted</p> In\u00a0[26]: Copied! <pre>lfp.LFPOutput.LFPV1() &amp; lfp_s_key\n</pre> lfp.LFPOutput.LFPV1() &amp; lfp_s_key Out[26]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 minirec20230622_.nwb test 01_s1_first9 LFP 0-400 Hz 30000 <p>Total: 1</p> In\u00a0[27]: Copied! <pre>lfp_key = {\"merge_id\": (lfp.LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")}\nlfp_key\n</pre> lfp_key = {\"merge_id\": (lfp.LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")} lfp_key Out[27]: <pre>{'merge_id': UUID('1e7fbe35-034b-ed8e-7965-a0467ae5c0a4')}</pre> In\u00a0[28]: Copied! <pre>lfp.LFPOutput &amp; lfp_key\n</pre> lfp.LFPOutput &amp; lfp_key Out[28]: <p>merge_id</p> <p>source</p> 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 LFPV1 <p>Total: 1</p> <p>From the Merge Table, we can get the keys for the LFP data we want to see</p> In\u00a0[31]: Copied! <pre>lfp_df = (lfp.LFPOutput &amp; lfp_key).fetch1_dataframe()\nlfp_df\n</pre> lfp_df = (lfp.LFPOutput &amp; lfp_key).fetch1_dataframe() lfp_df Out[31]: 0 time 1.687475e+09 21 1.687475e+09 10 1.687475e+09 37 1.687475e+09 81 1.687475e+09 61 ... ... 1.687475e+09 -125 1.687475e+09 -126 1.687475e+09 -123 1.687475e+09 -129 1.687475e+09 -122 <p>9000 rows \u00d7 1 columns</p> In\u00a0[32]: Copied! <pre>lfp_sampling_rate = lfp.LFPOutput.merge_get_parent(lfp_key).fetch1(\n    \"lfp_sampling_rate\"\n)\n\nfilter_name = \"Theta 5-11 Hz\"\n\nsgc.common_filter.FirFilterParameters().add_filter(\n    filter_name,\n    lfp_sampling_rate,\n    \"bandpass\",\n    [4, 5, 11, 12],\n    \"theta filter for 1 Khz data\",\n)\n\nsgc.common_filter.FirFilterParameters() &amp; {\n    \"filter_name\": filter_name,\n    \"filter_sampling_rate\": lfp_sampling_rate,\n}\n</pre> lfp_sampling_rate = lfp.LFPOutput.merge_get_parent(lfp_key).fetch1(     \"lfp_sampling_rate\" )  filter_name = \"Theta 5-11 Hz\"  sgc.common_filter.FirFilterParameters().add_filter(     filter_name,     lfp_sampling_rate,     \"bandpass\",     [4, 5, 11, 12],     \"theta filter for 1 Khz data\", )  sgc.common_filter.FirFilterParameters() &amp; {     \"filter_name\": filter_name,     \"filter_sampling_rate\": lfp_sampling_rate, } Out[32]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Theta 5-11 Hz 1000 lowpass 4.0 5.0 11.0 12.0 theta filter for 1 KHz data =BLOB= =BLOB= <p>Total: 1</p> In\u00a0[33]: Copied! <pre>sgc.IntervalList()\n</pre> sgc.IntervalList() Out[33]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') aj8020230628_.nwb 02_lineartrack =BLOB= aj8020230628_.nwb 04_lineartrack =BLOB= aj8020230628_.nwb 06_lineartrack =BLOB= aj8020230628_.nwb pos 0 valid times =BLOB= aj8020230628_.nwb pos 1 valid times =BLOB= aj8020230628_.nwb pos 2 valid times =BLOB= aj8020230628_.nwb raw data valid times =BLOB= aj8420230703_.nwb 01_lineartrack =BLOB= aj8420230703_.nwb pos 0 valid times =BLOB= aj8420230703_.nwb raw data valid times =BLOB= aj8620230628_.nwb 02_lineartrack =BLOB= aj8620230628_.nwb 04_lineartrack =BLOB= <p>...</p> <p>Total: 130046</p> <p>We can specify electrodes of interest, and desired sampling rate.</p> In\u00a0[34]: Copied! <pre>from spyglass.lfp.analysis.v1 import lfp_band\n\nlfp_band_electrode_ids = [0]  # assumes we've filtered these electrodes\nlfp_band_sampling_rate = 100  # desired sampling rate\n\nlfp_band.LFPBandSelection().set_lfp_band_electrodes(\n    nwb_file_name=nwb_file_name,\n    lfp_merge_id=lfp_key[\"merge_id\"],\n    electrode_list=lfp_band_electrode_ids,\n    filter_name=filter_name,  # defined above\n    interval_list_name=interval_list_name,  # Defined in IntervalList above\n    reference_electrode_list=[-1],  # -1 means no ref electrode for all channels\n    lfp_band_sampling_rate=lfp_band_sampling_rate,\n)\n\nlfp_band.LFPBandSelection()\n</pre> from spyglass.lfp.analysis.v1 import lfp_band  lfp_band_electrode_ids = [0]  # assumes we've filtered these electrodes lfp_band_sampling_rate = 100  # desired sampling rate  lfp_band.LFPBandSelection().set_lfp_band_electrodes(     nwb_file_name=nwb_file_name,     lfp_merge_id=lfp_key[\"merge_id\"],     electrode_list=lfp_band_electrode_ids,     filter_name=filter_name,  # defined above     interval_list_name=interval_list_name,  # Defined in IntervalList above     reference_electrode_list=[-1],  # -1 means no ref electrode for all channels     lfp_band_sampling_rate=lfp_band_sampling_rate, )  lfp_band.LFPBandSelection() Out[34]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>min_interval_len</p> the minimum length of a valid interval to filter 001d08a1-5365-3c20-716b-f431ab136a55 Theta 5-11 Hz 1000 Totoro20220602_.nwb pos 4 valid times 100 1.000213375-550b-ae22-394b-82da3cdfddb3 Theta 5-11 Hz 1000 Yoshi20220518_.nwb pos 15 valid times 100 1.0004560d5-417b-ff99-5a13-99a37a3eb998 ripple_150_250 1000 j1620210715_.nwb 09_s5 1000 1.0004560d5-417b-ff99-5a13-99a37a3eb998 theta_5_11 1000 j1620210715_.nwb 09_s5 1000 1.0006b9516-88d1-dd58-75e6-e519fb7dfea8 ripple_150_250 1000 senor20201103_.nwb 15_s8 1000 1.0006b9516-88d1-dd58-75e6-e519fb7dfea8 theta_5_11 1000 senor20201103_.nwb 15_s8 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Fast Gamma 65-100 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b ms_stim_125ms_period 1000 Winnie20220714_.nwb pos 9 valid times 200 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Slow Gamma 25-55 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Theta 5-11 Hz 1000 Winnie20220714_.nwb pos 9 valid times 100 1.00093a2c5-d720-d0c4-6cc4-35f1f34a0a01 ripple_150_250 1000 senor20201118_.nwb 08_r4 noPrePostTrialTimes 1000 1.0 <p>...</p> <p>Total: 8457</p> <p>Next we add an entry for the LFP Band and the electrodes we want to filter</p> In\u00a0[39]: Copied! <pre>lfp_band_key = (\n    lfp_band.LFPBandSelection\n    &amp; {\n        \"lfp_merge_id\": lfp_key[\"merge_id\"],\n        \"filter_name\": filter_name,\n        \"lfp_band_sampling_rate\": lfp_band_sampling_rate,\n    }\n).fetch1(\"KEY\")\nlfp_band_key\n</pre> lfp_band_key = (     lfp_band.LFPBandSelection     &amp; {         \"lfp_merge_id\": lfp_key[\"merge_id\"],         \"filter_name\": filter_name,         \"lfp_band_sampling_rate\": lfp_band_sampling_rate,     } ).fetch1(\"KEY\") lfp_band_key Out[39]: <pre>{'lfp_merge_id': UUID('1e7fbe35-034b-ed8e-7965-a0467ae5c0a4'),\n 'filter_name': 'Theta 5-11 Hz',\n 'filter_sampling_rate': 1000,\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'target_interval_list_name': '01_s1_first9',\n 'lfp_band_sampling_rate': 100}</pre> <p>Check to make sure it worked</p> In\u00a0[40]: Copied! <pre>lfp_band.LFPBandSelection() &amp; lfp_band_key\n</pre> lfp_band.LFPBandSelection() &amp; lfp_band_key Out[40]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>min_interval_len</p> the minimum length of a valid interval to filter 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 1.0 <p>Total: 1</p> In\u00a0[42]: Copied! <pre>lfp_band.LFPBandV1().populate(lfp_band.LFPBandSelection() &amp; lfp_band_key)\nlfp_band.LFPBandV1() &amp; lfp_band_key\n</pre> lfp_band.LFPBandV1().populate(lfp_band.LFPBandSelection() &amp; lfp_band_key) lfp_band.LFPBandV1() &amp; lfp_band_key Out[42]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_DCP0O105KG.nwb 01_s1_first9 lfp band 100Hz 277bdc87-98b9-4eff-a33e-00b64b4c3bb2 <p>Total: 1</p> <p>Now we can plot the original signal, the LFP filtered trace, and the theta filtered trace together. Get the three electrical series objects and the indices of the electrodes we band pass filtered</p> <p>Note: Much of the code below could be replaced by a function calls that would return the data from each electrical series.</p> <p>Note: If you see an error <code>Qt: Session Management Error</code>, try running the following unix command: <code>export -n SESSION_MANAGER</code>. See also</p> In\u00a0[43]: Copied! <pre>orig_eseries = (sgc.Raw() &amp; {\"nwb_file_name\": nwb_file_name}).fetch_nwb()[0][\n    \"raw\"\n]\norig_elect_indices = sgc.get_electrode_indices(\n    orig_eseries, lfp_band_electrode_ids\n)\norig_timestamps = np.asarray(orig_eseries.timestamps)\n</pre> orig_eseries = (sgc.Raw() &amp; {\"nwb_file_name\": nwb_file_name}).fetch_nwb()[0][     \"raw\" ] orig_elect_indices = sgc.get_electrode_indices(     orig_eseries, lfp_band_electrode_ids ) orig_timestamps = np.asarray(orig_eseries.timestamps) In\u00a0[44]: Copied! <pre>lfp_eseries = lfp.LFPOutput().fetch_nwb(lfp_key)[0][\"lfp\"]\nlfp_elect_indices = sgc.get_electrode_indices(\n    lfp_eseries, lfp_band_electrode_ids\n)\nlfp_timestamps = np.asarray(lfp_eseries.timestamps)\n</pre> lfp_eseries = lfp.LFPOutput().fetch_nwb(lfp_key)[0][\"lfp\"] lfp_elect_indices = sgc.get_electrode_indices(     lfp_eseries, lfp_band_electrode_ids ) lfp_timestamps = np.asarray(lfp_eseries.timestamps) In\u00a0[45]: Copied! <pre>lfp_band_eseries = (lfp_band.LFPBandV1 &amp; lfp_band_key).fetch_nwb()[0][\n    \"lfp_band\"\n]\nlfp_band_elect_indices = sgc.get_electrode_indices(\n    lfp_band_eseries, lfp_band_electrode_ids\n)\nlfp_band_timestamps = np.asarray(lfp_band_eseries.timestamps)\n</pre> lfp_band_eseries = (lfp_band.LFPBandV1 &amp; lfp_band_key).fetch_nwb()[0][     \"lfp_band\" ] lfp_band_elect_indices = sgc.get_electrode_indices(     lfp_band_eseries, lfp_band_electrode_ids ) lfp_band_timestamps = np.asarray(lfp_band_eseries.timestamps) <p>Get a list of times for the first run epoch and then select a 2 second interval 100 seconds from the beginning</p> In\u00a0[46]: Copied! <pre>plottimes = [valid_times[0][0] + 1, valid_times[0][0] + 8]\n</pre> plottimes = [valid_times[0][0] + 1, valid_times[0][0] + 8] In\u00a0[47]: Copied! <pre># get the time indices for each dataset\norig_time_ind = np.where(\n    np.logical_and(\n        orig_timestamps &gt; plottimes[0], orig_timestamps &lt; plottimes[1]\n    )\n)[0]\n\nlfp_time_ind = np.where(\n    np.logical_and(lfp_timestamps &gt; plottimes[0], lfp_timestamps &lt; plottimes[1])\n)[0]\nlfp_band_time_ind = np.where(\n    np.logical_and(\n        lfp_band_timestamps &gt; plottimes[0],\n        lfp_band_timestamps &lt; plottimes[1],\n    )\n)[0]\n</pre> # get the time indices for each dataset orig_time_ind = np.where(     np.logical_and(         orig_timestamps &gt; plottimes[0], orig_timestamps &lt; plottimes[1]     ) )[0]  lfp_time_ind = np.where(     np.logical_and(lfp_timestamps &gt; plottimes[0], lfp_timestamps &lt; plottimes[1]) )[0] lfp_band_time_ind = np.where(     np.logical_and(         lfp_band_timestamps &gt; plottimes[0],         lfp_band_timestamps &lt; plottimes[1],     ) )[0] In\u00a0[48]: Copied! <pre>import matplotlib.pyplot as plt\n\nplt.plot(\n    orig_eseries.timestamps[orig_time_ind],\n    orig_eseries.data[orig_time_ind, orig_elect_indices[0]],\n    \"k-\",\n)\nplt.plot(\n    lfp_eseries.timestamps[lfp_time_ind],\n    lfp_eseries.data[lfp_time_ind, lfp_elect_indices[0]],\n    \"b-\",\n)\nplt.plot(\n    lfp_band_eseries.timestamps[lfp_band_time_ind],\n    lfp_band_eseries.data[lfp_band_time_ind, lfp_band_elect_indices[0]],\n    \"r-\",\n)\nplt.xlabel(\"Time (sec)\")\nplt.ylabel(\"Amplitude (AD units)\")\n\n# Uncomment to see plot\n# plt.show()\n</pre> import matplotlib.pyplot as plt  plt.plot(     orig_eseries.timestamps[orig_time_ind],     orig_eseries.data[orig_time_ind, orig_elect_indices[0]],     \"k-\", ) plt.plot(     lfp_eseries.timestamps[lfp_time_ind],     lfp_eseries.data[lfp_time_ind, lfp_elect_indices[0]],     \"b-\", ) plt.plot(     lfp_band_eseries.timestamps[lfp_band_time_ind],     lfp_band_eseries.data[lfp_band_time_ind, lfp_band_elect_indices[0]],     \"r-\", ) plt.xlabel(\"Time (sec)\") plt.ylabel(\"Amplitude (AD units)\")  # Uncomment to see plot # plt.show() Out[48]: <pre>Text(0, 0.5, 'Amplitude (AD units)')</pre>"}, {"location": "notebooks/30_LFP/#lfp-extraction", "title": "LFP Extraction\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#select-data", "title": "Select data\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#create-filters", "title": "Create Filters\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#electrode-group", "title": "Electrode Group\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#intervallist", "title": "<code>IntervalList</code>\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#lfpselection", "title": "<code>LFPSelection</code>\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#populate-lfp", "title": "Populate LFP\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#lfp-band", "title": "LFP Band\u00b6", "text": "<p>Now that we've created the LFP object we can perform a second level of filtering for a band of interest, in this case the theta band. We first need to create the filter.</p>"}, {"location": "notebooks/30_LFP/#plotting", "title": "Plotting\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#next-steps", "title": "Next Steps\u00b6", "text": "<p>Next, we'll use look at Theta bands within LFP data.</p>"}, {"location": "notebooks/31_Theta/", "title": "Theta", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> <li>To run this notebook, you should have already completed the  LFP notebook and populated the <code>LFPBand</code> table.</li> </ul> <p>In this tutorial, we demonstrate how to generate analytic signals from the LFP data, as well as how to compute theta phases and power.</p> In\u00a0[4]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.lfp.analysis.v1.lfp_band as lfp_band\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.lfp.analysis.v1.lfp_band as lfp_band  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) In\u00a0[5]: Copied! <pre>lfp_band.LFPBandV1()\n</pre> lfp_band.LFPBandV1() Out[5]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_H89T89A0CK.nwb 01_s1_first9 lfp band 100Hz 71bbb1a8-409e-48eb-a336-8c70fd5d6b74 <p>Total: 1</p> <p>Now, we create a keys to reference the theta band data.</p> In\u00a0[6]: Copied! <pre>nwb_file_name = \"minirec20230622_.nwb\"\nlfp_key = dict(\n    nwb_file_name=nwb_file_name,\n    filter_name=\"Theta 5-11 Hz\",\n)\nlfp_band.LFPBandV1() &amp; lfp_key\n</pre> nwb_file_name = \"minirec20230622_.nwb\" lfp_key = dict(     nwb_file_name=nwb_file_name,     filter_name=\"Theta 5-11 Hz\", ) lfp_band.LFPBandV1() &amp; lfp_key Out[6]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_H89T89A0CK.nwb 01_s1_first9 lfp band 100Hz 71bbb1a8-409e-48eb-a336-8c70fd5d6b74 <p>Total: 1</p> <p>We do not need all electrodes for theta phase/power, so we define a list for analyses. When working with full data, this list might limit to hippocampal reference electrodes.</p> <p>Make sure that the chosen electrodes already exist in the LFPBand data; if not, go to the LFP tutorial to generate them.</p> In\u00a0[7]: Copied! <pre>electrode_list = [0]\n\nall_electrodes = (  # All available electrode ids\n    (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"]\n).electrodes.data[:]\n\nnp.isin(electrode_list, all_electrodes)  # Check if our list is in 'all'\n</pre> electrode_list = [0]  all_electrodes = (  # All available electrode ids     (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"] ).electrodes.data[:]  np.isin(electrode_list, all_electrodes)  # Check if our list is in 'all' Out[7]: <pre>array([ True])</pre> <p>Next, we'll compute the theta analytic signal.</p> In\u00a0[9]: Copied! <pre>theta_analytic_signal = (\n    lfp_band.LFPBandV1() &amp; lfp_key\n).compute_analytic_signal(electrode_list=electrode_list)\n\ntheta_analytic_signal\n</pre> theta_analytic_signal = (     lfp_band.LFPBandV1() &amp; lfp_key ).compute_analytic_signal(electrode_list=electrode_list)  theta_analytic_signal Out[9]: electrode 0 time 1.687475e+09 -124.000000-133.973800j 1.687475e+09 -30.000000-308.834746j 1.687475e+09 56.000000-267.135234j 1.687475e+09 139.000000-293.141114j 1.687475e+09 227.000000-236.059129j ... ... 1.687475e+09 -548.000000+29.032951j 1.687475e+09 -464.000000-225.303955j 1.687475e+09 -291.000000-329.028626j 1.687475e+09 -86.000000-404.908506j 1.687475e+09 92.000000-178.143737j <p>900 rows \u00d7 1 columns</p> <p>In the dataframe above, the index is the timestamps, and the columns are the analytic signals of theta band (complex numbers) for each electrode.</p> <p>Using a similar method, we can compute theta phase and power from the LFPBand table.</p> In\u00a0[10]: Copied! <pre>theta_phase = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_phase(\n    electrode_list=electrode_list\n)\ntheta_power = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_power(\n    electrode_list=electrode_list\n)\n</pre> theta_phase = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_phase(     electrode_list=electrode_list ) theta_power = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_power(     electrode_list=electrode_list ) <p>We can get the theta band data to plot with the theta phase results.</p> In\u00a0[11]: Copied! <pre>theta_band = (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"]\nelectrode_index = np.isin(theta_band.electrodes.data[:], electrode_list)\ntheta_band_selected = theta_band.data[:, electrode_index]\n</pre> theta_band = (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"] electrode_index = np.isin(theta_band.electrodes.data[:], electrode_list) theta_band_selected = theta_band.data[:, electrode_index] In\u00a0[13]: Copied! <pre>electrode_id = electrode_list[0]  # the electrode for which we want to plot\nplot_start, plot_end = 0, 5000  # start/end time of plotting\n\nfig, ax1 = plt.subplots(figsize=(20, 6))\nax1.set_xlabel(\"Time (sec)\", fontsize=15)\nax1.set_ylabel(\"Amplitude (AD units)\", fontsize=15)\nax1.plot(\n    theta_phase.index[plot_start:plot_end],\n    theta_band_selected[\n        plot_start:plot_end, np.where(np.array(electrode_list) == 0)[0][0]\n    ],\n    \"k-\",\n    linewidth=1,\n    alpha=0.9,\n)\nax1.tick_params(axis=\"y\", labelcolor=\"k\")\n\nax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis\nax2.set_ylabel(\"Phase(deg)\", color=\"b\", fontsize=15)\nax2.plot(\n    theta_phase.index[plot_start:plot_end],\n    theta_phase[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],\n    \"b\",\n)\nax2.tick_params(axis=\"y\", labelcolor=\"b\")\nax2.axhline(y=0, color=\"r\", linestyle=\"-\")\n\nfig.tight_layout()\nax1.set_title(\n    f\"Theta band amplitude and phase, electrode {electrode_id}\",\n    fontsize=20,\n)\n</pre> electrode_id = electrode_list[0]  # the electrode for which we want to plot plot_start, plot_end = 0, 5000  # start/end time of plotting  fig, ax1 = plt.subplots(figsize=(20, 6)) ax1.set_xlabel(\"Time (sec)\", fontsize=15) ax1.set_ylabel(\"Amplitude (AD units)\", fontsize=15) ax1.plot(     theta_phase.index[plot_start:plot_end],     theta_band_selected[         plot_start:plot_end, np.where(np.array(electrode_list) == 0)[0][0]     ],     \"k-\",     linewidth=1,     alpha=0.9, ) ax1.tick_params(axis=\"y\", labelcolor=\"k\")  ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis ax2.set_ylabel(\"Phase(deg)\", color=\"b\", fontsize=15) ax2.plot(     theta_phase.index[plot_start:plot_end],     theta_phase[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],     \"b\", ) ax2.tick_params(axis=\"y\", labelcolor=\"b\") ax2.axhline(y=0, color=\"r\", linestyle=\"-\")  fig.tight_layout() ax1.set_title(     f\"Theta band amplitude and phase, electrode {electrode_id}\",     fontsize=20, ) <p>We can also plot the theta power.</p> In\u00a0[14]: Copied! <pre>electrode_id = electrode_list[0]  # the electrode for which we want to plot\nplot_start, plot_end = 0, 5000  # start/end time of plotting\n\nfig, ax = plt.subplots(figsize=(20, 6))\nax.set_xlabel(\"Time (sec)\", fontsize=15)\nax.set_ylabel(\"Theta power\", fontsize=15)\nax.plot(\n    theta_power.index[plot_start:plot_end],\n    theta_power[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],\n    \"k-\",\n    linewidth=1,\n)\nax.tick_params(axis=\"y\", labelcolor=\"k\")\nax.set_title(\n    f\"Theta band power, electrode {electrode_id}\",\n    fontsize=20,\n)\n</pre> electrode_id = electrode_list[0]  # the electrode for which we want to plot plot_start, plot_end = 0, 5000  # start/end time of plotting  fig, ax = plt.subplots(figsize=(20, 6)) ax.set_xlabel(\"Time (sec)\", fontsize=15) ax.set_ylabel(\"Theta power\", fontsize=15) ax.plot(     theta_power.index[plot_start:plot_end],     theta_power[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],     \"k-\",     linewidth=1, ) ax.tick_params(axis=\"y\", labelcolor=\"k\") ax.set_title(     f\"Theta band power, electrode {electrode_id}\",     fontsize=20, )"}, {"location": "notebooks/31_Theta/#theta-phase-and-power", "title": "Theta phase and power\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#acquire-signal", "title": "Acquire Signal\u00b6", "text": "<p>First, we'll acquire the theta band analytic signal from the <code>LFPBand</code> data for electrodes of interest. We can grab keys from this table.</p>"}, {"location": "notebooks/31_Theta/#compute-phase-and-power", "title": "Compute phase and power\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#plot-results", "title": "Plot results\u00b6", "text": "<p>We can overlay theta and detected phase for each electrode.</p> <p>Note: The red horizontal line indicates phase 0, corresponding to the through of theta.</p>"}, {"location": "notebooks/31_Theta/#up-next", "title": "Up Next\u00b6", "text": "<p>Next, we'll turn our attention to position data.</p>"}, {"location": "notebooks/32_Ripple_Detection/", "title": "Ripple Detection", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>Ripple detection depends on a set of LFPs, the parameters used for detection and the speed of the animal. You will need <code>RippleLFPSelection</code>, <code>RippleParameters</code>, and <code>PositionOutput</code> to be populated accordingly.</p> In\u00a0[32]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\nimport pandas as pd\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\nimport spyglass.lfp.analysis.v1 as lfp_analysis\nfrom spyglass.lfp import LFPOutput\nimport spyglass.lfp as sglfp\nfrom spyglass.position import PositionOutput\nimport spyglass.ripple.v1 as sgrip\nimport spyglass.ripple.v1 as sgr\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np import pandas as pd  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position.v1 as sgp import spyglass.lfp.analysis.v1 as lfp_analysis from spyglass.lfp import LFPOutput import spyglass.lfp as sglfp from spyglass.position import PositionOutput import spyglass.ripple.v1 as sgrip import spyglass.ripple.v1 as sgr  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <p>First, we need to generate a filter band from the LFP data at the ripple frequency.  This process is analogous to that in 30_LFP.ipynb.</p> <p>If you have already populated the LFP table for your data you may skip this step. Here, we will begin by creating a lfp group of just electrodes in the hippocampus region, and populate the lfp on a subsetted interval:</p> In\u00a0[9]: Copied! <pre>nwb_file_name = \"mediumnwb20230802_.nwb\"\nlfp_electrode_group_name = \"test_hippocampus\"\ninterval_list_name = \"02_r1_ripple_demo\"\n\n# select hippocampus electrodes\nelectrodes_df = (\n    pd.DataFrame(\n        (\n            sgc.Electrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n            }\n        )\n        * (sgc.BrainRegion &amp; {\"region_name\": \"hippocampus\"})\n    )\n    .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]\n    .sort_values(by=\"electrode_id\")\n)\n# create lfp_electrode_group\nlfp_eg_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"lfp_electrode_group_name\": lfp_electrode_group_name,\n}\nsglfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_file_name,\n    group_name=lfp_electrode_group_name,\n    electrode_list=electrodes_df.electrode_id.tolist(),\n)\n\n# make a shorter interval to run this demo on\ninterval_start = (\n    sgc.IntervalList\n    &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": \"02_r1\"}\n).fetch1(\"valid_times\")[0][0]\ntruncated_interval = np.array(\n    [[interval_start, interval_start + 120]]\n)  # first 2 minutes of epoch\nsgc.IntervalList.insert1(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_list_name\": \"02_r1_ripple_demo\",\n        \"valid_times\": truncated_interval,\n    },\n    skip_duplicates=True,\n)\n\n# make the lfp selection\nlfp_s_key = lfp_eg_key.copy()\nlfp_s_key.update(\n    {\n        \"target_interval_list_name\": interval_list_name,\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)\n        \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)\n    }\n)\nsglfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)\n\n# populate the lfp\nsglfp.v1.LFPV1.populate(lfp_s_key, display_progress=True)\nsglfp.v1.LFPV1 &amp; lfp_s_key\n</pre> nwb_file_name = \"mediumnwb20230802_.nwb\" lfp_electrode_group_name = \"test_hippocampus\" interval_list_name = \"02_r1_ripple_demo\"  # select hippocampus electrodes electrodes_df = (     pd.DataFrame(         (             sgc.Electrode             &amp; {                 \"nwb_file_name\": nwb_file_name,             }         )         * (sgc.BrainRegion &amp; {\"region_name\": \"hippocampus\"})     )     .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]     .sort_values(by=\"electrode_id\") ) # create lfp_electrode_group lfp_eg_key = {     \"nwb_file_name\": nwb_file_name,     \"lfp_electrode_group_name\": lfp_electrode_group_name, } sglfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_file_name,     group_name=lfp_electrode_group_name,     electrode_list=electrodes_df.electrode_id.tolist(), )  # make a shorter interval to run this demo on interval_start = (     sgc.IntervalList     &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": \"02_r1\"} ).fetch1(\"valid_times\")[0][0] truncated_interval = np.array(     [[interval_start, interval_start + 120]] )  # first 2 minutes of epoch sgc.IntervalList.insert1(     {         \"nwb_file_name\": nwb_file_name,         \"interval_list_name\": \"02_r1_ripple_demo\",         \"valid_times\": truncated_interval,     },     skip_duplicates=True, )  # make the lfp selection lfp_s_key = lfp_eg_key.copy() lfp_s_key.update(     {         \"target_interval_list_name\": interval_list_name,         \"filter_name\": \"LFP 0-400 Hz\",         \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)         \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)     } ) sglfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)  # populate the lfp sglfp.v1.LFPV1.populate(lfp_s_key, display_progress=True) sglfp.v1.LFPV1 &amp; lfp_s_key Out[9]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ mediumnwb20230802_.nwb test_hippocampus 02_r1_ripple_demo LFP 0-400 Hz 30000 mediumnwb20230802_7625J294O4.nwb lfp_test_hippocampus_02_r1_ripple_demo_valid times 95ac8100-eca8-4dff-a504-b1c139a2a3af 1000.0 <p>Total: 1</p> In\u00a0[10]: Copied! <pre>sgc.FirFilterParameters().add_filter(\n    filter_name=\"Ripple 150-250 Hz\",\n    fs=1000.0,\n    filter_type=\"bandpass\",\n    band_edges=[140, 150, 250, 260],\n    comments=\"ripple band filter for 1 kHz data\",\n)\n\nsgc.FirFilterParameters() &amp; \"filter_name='Ripple 150-250 Hz'\"\n</pre> sgc.FirFilterParameters().add_filter(     filter_name=\"Ripple 150-250 Hz\",     fs=1000.0,     filter_type=\"bandpass\",     band_edges=[140, 150, 250, 260],     comments=\"ripple band filter for 1 kHz data\", )  sgc.FirFilterParameters() &amp; \"filter_name='Ripple 150-250 Hz'\" Out[10]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Ripple 150-250 Hz 1000 lowpass 140.0 150.0 250.0 260.0 ripple band filter for 1 kHz data =BLOB= =BLOB= <p>Total: 1</p> <p>We can then populate the ripple band</p> In\u00a0[18]: Copied! <pre>from spyglass.lfp.analysis.v1 import lfp_band\n\nfilter_name = \"Ripple 150-250 Hz\"\nlfp_band_electrode_ids = (\n    electrodes_df.electrode_id.tolist()\n)  # assumes we've filtered these electrodes\nlfp_band_sampling_rate = 1000  # desired sampling rate\n\nlfp_merge_id = (LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")\nlfp_band.LFPBandSelection().set_lfp_band_electrodes(\n    nwb_file_name=nwb_file_name,\n    lfp_merge_id=lfp_merge_id,\n    electrode_list=lfp_band_electrode_ids,\n    filter_name=filter_name,\n    interval_list_name=interval_list_name,\n    reference_electrode_list=[-1],  # -1 means no ref electrode for all channels\n    lfp_band_sampling_rate=lfp_band_sampling_rate,\n)\n\nlfp_band.LFPBandV1.populate(\n    {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name},\n    display_progress=True,\n)\nlfp_band.LFPBandV1 &amp; {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name}\n</pre> from spyglass.lfp.analysis.v1 import lfp_band  filter_name = \"Ripple 150-250 Hz\" lfp_band_electrode_ids = (     electrodes_df.electrode_id.tolist() )  # assumes we've filtered these electrodes lfp_band_sampling_rate = 1000  # desired sampling rate  lfp_merge_id = (LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\") lfp_band.LFPBandSelection().set_lfp_band_electrodes(     nwb_file_name=nwb_file_name,     lfp_merge_id=lfp_merge_id,     electrode_list=lfp_band_electrode_ids,     filter_name=filter_name,     interval_list_name=interval_list_name,     reference_electrode_list=[-1],  # -1 means no ref electrode for all channels     lfp_band_sampling_rate=lfp_band_sampling_rate, )  lfp_band.LFPBandV1.populate(     {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name},     display_progress=True, ) lfp_band.LFPBandV1 &amp; {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name} Out[18]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 mediumnwb20230802_.nwb 02_r1_ripple_demo 1000 mediumnwb20230802_XVQHZZVY3B.nwb 02_r1_ripple_demo lfp band 1000Hz eca68871-9a8d-4fa4-b265-95dafdfdda29 <p>Total: 1</p> <p>Next, we'll pick the electrodes on which we'll run ripple detection on, using <code>RippleLFPSelection.set_lfp_electrodes</code></p> In\u00a0[19]: Copied! <pre>?sgr.RippleLFPSelection.set_lfp_electrodes\n</pre> ?sgr.RippleLFPSelection.set_lfp_electrodes <pre>Signature:\nsgr.RippleLFPSelection.set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name='CA1',\n    **kwargs,\n)\nDocstring:\nRemoves all electrodes for the specified nwb file and then\nadds back the electrodes in the list\n\nParameters\n----------\nkey : dict\n    dictionary corresponding to the LFPBand entry to use for\n    ripple detection\nelectrode_list : list\n    list of electrodes from LFPBandSelection.LFPBandElectrode\n    to be used as the ripple LFP during detection\ngroup_name : str, optional\n    description of the electrode group, by default \"CA1\"\nFile:      ~/Documents/spyglass/src/spyglass/ripple/v1/ripple.py\nType:      function</pre> <p>We'll need the <code>nwb_file_name</code>, an <code>electrode_list</code>, and to a <code>group_name</code>.</p> <ul> <li>By default, <code>group_name</code> is set to CA1 for ripple detection, but we could alternatively use PFC.</li> <li>We use <code>nwb_file_name</code> to explore which electrodes are available for the <code>electrode_list</code>.</li> </ul> <p>Now we can look at <code>electrode_id</code> in the <code>Electrode</code> table:</p> In\u00a0[20]: Copied! <pre>electrodes = (\n    (sgc.Electrode() &amp; {\"nwb_file_name\": nwb_file_name})\n    * (\n        lfp_analysis.LFPBandSelection.LFPBandElectrode()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"filter_name\": filter_name,\n            \"target_interval_list_name\": interval_list_name,\n        }\n    )\n    * sgc.BrainRegion\n).fetch(format=\"frame\")\nelectrodes\n</pre> electrodes = (     (sgc.Electrode() &amp; {\"nwb_file_name\": nwb_file_name})     * (         lfp_analysis.LFPBandSelection.LFPBandElectrode()         &amp; {             \"nwb_file_name\": nwb_file_name,             \"filter_name\": filter_name,             \"target_interval_list_name\": interval_list_name,         }     )     * sgc.BrainRegion ).fetch(format=\"frame\") electrodes Out[20]: probe_id probe_shank probe_electrode name original_reference_electrode x y z filtering impedance bad_channel x_warped y_warped z_warped contacts region_name subregion_name subsubregion_name nwb_file_name electrode_group_name electrode_id lfp_merge_id filter_name filter_sampling_rate target_interval_list_name lfp_band_sampling_rate lfp_electrode_group_name reference_elect_id region_id mediumnwb20230802_.nwb 0 0 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 0 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 1 1 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 2 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 2 2 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 3 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 3 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 4 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 4 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 8 35 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 35 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 9 36 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 36 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 37 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 1 37 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 38 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 2 38 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 39 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 39 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None <p>88 rows \u00d7 18 columns</p> <p>For ripple detection, we want only tetrodes, and only the first good wire on each tetrode. We will assume that is the first wire on each tetrode. I will do this using pandas syntax but you could use datajoint to filter this table as well. Here is the filtered table.</p> In\u00a0[21]: Copied! <pre>hpc_names = [\"ca1\", \"hippocampus\", \"CA1\", \"Hippocampus\"]\nelectrodes.loc[\n    (electrodes.region_name.isin(hpc_names)) &amp; (electrodes.probe_electrode == 0)\n]\n</pre> hpc_names = [\"ca1\", \"hippocampus\", \"CA1\", \"Hippocampus\"] electrodes.loc[     (electrodes.region_name.isin(hpc_names)) &amp; (electrodes.probe_electrode == 0) ] Out[21]: probe_id probe_shank probe_electrode name original_reference_electrode x y z filtering impedance bad_channel x_warped y_warped z_warped contacts region_name subregion_name subsubregion_name nwb_file_name electrode_group_name electrode_id lfp_merge_id filter_name filter_sampling_rate target_interval_list_name lfp_band_sampling_rate lfp_electrode_group_name reference_elect_id region_id mediumnwb20230802_.nwb 0 0 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 0 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 4 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 4 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 10 40 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 40 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 11 44 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 44 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 13 52 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 52 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 14 56 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 56 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 15 60 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 60 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 16 64 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 64 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 17 68 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 68 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 18 72 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 72 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 19 76 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 76 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 20 80 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 80 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 21 84 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 84 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 22 88 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 88 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 23 92 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 92 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 3 12 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 12 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 4 16 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 16 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 5 20 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 20 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 6 24 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 24 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 7 28 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 28 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 8 32 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 32 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 9 36 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 36 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None <p>We only want the electrode_id to put in the <code>electrode_list</code>:</p> In\u00a0[22]: Copied! <pre>electrode_list = np.unique(\n    (\n        electrodes.loc[\n            (electrodes.region_name.isin(hpc_names))\n            &amp; (electrodes.probe_electrode == 0)\n        ]\n        .reset_index()\n        .electrode_id\n    ).tolist()\n)\n\nelectrode_list.sort()\n</pre> electrode_list = np.unique(     (         electrodes.loc[             (electrodes.region_name.isin(hpc_names))             &amp; (electrodes.probe_electrode == 0)         ]         .reset_index()         .electrode_id     ).tolist() )  electrode_list.sort() <p>By default, <code>set_lfp_electrodes</code> will use all the available electrodes from <code>LFPBandV1</code>.</p> <p>We can insert into <code>RippleLFPSelection</code> and the <code>RippleLFPElectrode</code> part table, passing the key for the entry from <code>LFPBandV1</code>, our <code>electrode_list</code>, and the <code>group_name</code> into <code>set_lfp_electrodes</code></p> In\u00a0[24]: Copied! <pre>group_name = \"CA1_test\"\n\nlfp_band_key = (\n    lfp_analysis.LFPBandV1()\n    &amp; {\n        \"filter_name\": filter_name,\n        \"nwb_file_name\": nwb_file_name,\n        \"lfp_band_sampling_rate\": 1000,\n    }\n).fetch1(\"KEY\")\n\nsgr.RippleLFPSelection.set_lfp_electrodes(\n    lfp_band_key,\n    electrode_list=electrode_list,\n    group_name=group_name,\n)\n</pre> group_name = \"CA1_test\"  lfp_band_key = (     lfp_analysis.LFPBandV1()     &amp; {         \"filter_name\": filter_name,         \"nwb_file_name\": nwb_file_name,         \"lfp_band_sampling_rate\": 1000,     } ).fetch1(\"KEY\")  sgr.RippleLFPSelection.set_lfp_electrodes(     lfp_band_key,     electrode_list=electrode_list,     group_name=group_name, ) In\u00a0[25]: Copied! <pre>sgr.RippleLFPSelection.RippleLFPElectrode()\n</pre> sgr.RippleLFPSelection.RippleLFPElectrode() Out[25]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>group_name</p> <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode <p>reference_elect_id</p> the reference electrode to use; -1 for no reference 0087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 0 0 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 1 4 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 11 44 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 12 49 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 13 52 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 14 56 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 16 64 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 17 68 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 18 72 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 19 76 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 2 8 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 20 80 -1 <p>...</p> <p>Total: 60159</p> <p>Here's the ripple selection key we'll use downstream</p> In\u00a0[26]: Copied! <pre>rip_sel_key = (sgrip.RippleLFPSelection &amp; lfp_band_key).fetch1(\"KEY\")\n</pre> rip_sel_key = (sgrip.RippleLFPSelection &amp; lfp_band_key).fetch1(\"KEY\") In\u00a0[27]: Copied! <pre>sgr.RippleParameters().insert_default\n</pre> sgr.RippleParameters().insert_default Out[27]: <p>ripple_param_name</p> a name for this set of parameters <p>ripple_param_dict</p> dictionary of parameters default =BLOB=default_ms =BLOB=default_sharon =BLOB= <p>Total: 3</p> <p>Here are the default ripple parameters:</p> In\u00a0[49]: Copied! <pre>(sgrip.RippleParameters() &amp; {\"ripple_param_name\": \"default_trodes\"}).fetch1()\n</pre> (sgrip.RippleParameters() &amp; {\"ripple_param_name\": \"default_trodes\"}).fetch1() Out[49]: <pre>{'ripple_param_name': 'default_trodes',\n 'ripple_param_dict': {'speed_name': 'speed',\n  'ripple_detection_algorithm': 'Kay_ripple_detector',\n  'ripple_detection_params': {'speed_threshold': 4.0,\n   'minimum_duration': 0.015,\n   'zscore_threshold': 2.0,\n   'smoothing_sigma': 0.004,\n   'close_ripple_threshold': 0.0}}}</pre> <ul> <li><code>filter_name</code>: which bandpass filter is used</li> <li><code>speed_name</code>: the name of the speed parameters in <code>IntervalPositionInfo</code></li> </ul> <p>For the <code>Kay_ripple_detector</code> (options are currently Kay and Karlsson, see <code>ripple_detection</code> package for specifics) the parameters are:</p> <ul> <li><code>speed_threshold</code> (cm/s): maximum speed the animal can move</li> <li><code>minimum_duration</code> (s): minimum time above threshold</li> <li><code>zscore_threshold</code> (std): minimum value to be considered a ripple, in standard deviations from mean</li> <li><code>smoothing_sigma</code> (s): how much to smooth the signal in time</li> <li><code>close_ripple_threshold</code> (s): exclude ripples closer than this amount</li> </ul> In\u00a0[41]: Copied! <pre>pos_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"trodes_pos_params_name\": \"single_led\",\n    \"interval_list_name\": \"pos 0 valid times\",\n}\nsgp.TrodesPosSelection().insert1(pos_key, skip_duplicates=True)\nsgp.TrodesPosV1.populate(pos_key, display_progress=True)\nsgp.TrodesPosV1 &amp; pos_key\n</pre> pos_key = {     \"nwb_file_name\": nwb_file_name,     \"trodes_pos_params_name\": \"single_led\",     \"interval_list_name\": \"pos 0 valid times\", } sgp.TrodesPosSelection().insert1(pos_key, skip_duplicates=True) sgp.TrodesPosV1.populate(pos_key, display_progress=True) sgp.TrodesPosV1 &amp; pos_key Out[41]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>position_object_id</p> <p>orientation_object_id</p> <p>velocity_object_id</p> mediumnwb20230802_.nwb pos 0 valid times single_led mediumnwb20230802_9GTXMUKTK1.nwb 6d725947-3ba0-4cbe-9483-e77b897ba1ab 848f5b77-cf49-41c2-906d-06b58a731086 41cf08d3-f114-4201-b7d8-d6e541015b42 <p>Total: 1</p> In\u00a0[46]: Copied! <pre>from spyglass.position import PositionOutput\n\npos_key = PositionOutput.merge_get_part(pos_key).fetch1(\"KEY\")\n(PositionOutput &amp; pos_key).fetch1_dataframe()\n</pre> from spyglass.position import PositionOutput  pos_key = PositionOutput.merge_get_part(pos_key).fetch1(\"KEY\") (PositionOutput &amp; pos_key).fetch1_dataframe() Out[46]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.625936e+09 0 180.251195 162.885335 1.584011 -1.062553 1.052022 1.495249 1.625936e+09 1 180.090400 163.287322 1.579705 -1.162356 0.674062 1.343663 1.625936e+09 2 180.143998 163.421318 1.581228 -1.218606 0.243100 1.242618 1.625936e+09 3 179.983203 162.938933 1.576679 -1.243190 -0.085662 1.246138 1.625936e+09 4 180.197597 162.670942 1.582475 -1.230759 -0.224637 1.251091 ... ... ... ... ... ... ... ... 1.625937e+09 44186 38.483603 113.574868 -1.402199 1.008364 -0.865117 1.328618 1.625937e+09 44187 38.483603 113.521270 -1.398606 0.603148 -0.534938 0.806192 1.625937e+09 44188 38.430005 113.574868 -1.416478 0.256839 -0.219871 0.338096 1.625937e+09 44189 38.376407 113.574868 -1.432157 0.017772 0.025349 0.030958 1.625937e+09 44190 38.376407 113.628467 -1.435269 -0.107231 0.174792 0.205063 <p>44191 rows \u00d7 7 columns</p> <p>We'll use the <code>speed</code> above as part of <code>RippleParameters</code>. Ensure your selected ripple parameters value for <code>speed_name</code> matches for your data.</p> <p>Now we can put everything together.</p> In\u00a0[52]: Copied! <pre>key = {\n    \"ripple_param_name\": \"default_trodes\",\n    **rip_sel_key,\n    \"pos_merge_id\": pos_key[\"merge_id\"],\n}\nsgrip.RippleTimesV1().populate(key)\n</pre> key = {     \"ripple_param_name\": \"default_trodes\",     **rip_sel_key,     \"pos_merge_id\": pos_key[\"merge_id\"], } sgrip.RippleTimesV1().populate(key) <p>And then <code>fetch1_dataframe</code> for ripple times</p> In\u00a0[53]: Copied! <pre>ripple_times = (sgrip.RippleTimesV1() &amp; key).fetch1_dataframe()\nripple_times\n</pre> ripple_times = (sgrip.RippleTimesV1() &amp; key).fetch1_dataframe() ripple_times Out[53]: start_time end_time id 0 1.625936e+09 1.625936e+09 1 1.625936e+09 1.625936e+09 2 1.625936e+09 1.625936e+09 3 1.625936e+09 1.625936e+09 4 1.625936e+09 1.625936e+09 5 1.625936e+09 1.625936e+09 6 1.625936e+09 1.625936e+09 7 1.625936e+09 1.625936e+09 8 1.625936e+09 1.625936e+09 9 1.625936e+09 1.625936e+09 10 1.625936e+09 1.625936e+09 11 1.625936e+09 1.625936e+09 12 1.625936e+09 1.625936e+09 13 1.625936e+09 1.625936e+09 14 1.625936e+09 1.625936e+09 15 1.625936e+09 1.625936e+09 16 1.625936e+09 1.625936e+09 17 1.625936e+09 1.625936e+09 18 1.625936e+09 1.625936e+09 19 1.625936e+09 1.625936e+09 20 1.625936e+09 1.625936e+09 21 1.625936e+09 1.625936e+09 22 1.625936e+09 1.625936e+09 23 1.625936e+09 1.625936e+09 24 1.625936e+09 1.625936e+09 25 1.625936e+09 1.625936e+09 26 1.625936e+09 1.625936e+09 27 1.625936e+09 1.625936e+09 28 1.625936e+09 1.625936e+09 29 1.625936e+09 1.625936e+09 30 1.625936e+09 1.625936e+09 31 1.625936e+09 1.625936e+09 32 1.625936e+09 1.625936e+09 33 1.625936e+09 1.625936e+09 34 1.625936e+09 1.625936e+09 35 1.625936e+09 1.625936e+09 36 1.625936e+09 1.625936e+09 37 1.625936e+09 1.625936e+09 38 1.625936e+09 1.625936e+09 <p>We can also inspect the lfp trace at these ripple times.</p> <ul> <li>Note: The ripple detection algorithm depends on estimates of the standard deviation of power in the ripple band. Running analysis on longer intervals will lead to better estimates of this value, and thereby better segmentation of ripple events</li> </ul> In\u00a0[103]: Copied! <pre>import matplotlib.pyplot as plt\n\nripple_band_df = (lfp_band.LFPBandV1() &amp; lfp_band_key).fetch1_dataframe()\n\nwindow = 0.1\ni = -1\nripple_start = ripple_times.iloc[i].start_time\nripple_end = ripple_times.iloc[i].end_time\nplt.plot(\n    ripple_band_df.loc[ripple_start - window : ripple_end + window].index,\n    ripple_band_df.loc[ripple_start - window : ripple_end + window].iloc[\n        :, ::15\n    ],\n)\nplt.axvline(ripple_start, color=\"r\")\nplt.axvline(ripple_end, color=\"r\")\n\nplt.xlabel(\"Time (s)\")\nplt.ylabel(\"Voltage (uV)\")\n</pre> import matplotlib.pyplot as plt  ripple_band_df = (lfp_band.LFPBandV1() &amp; lfp_band_key).fetch1_dataframe()  window = 0.1 i = -1 ripple_start = ripple_times.iloc[i].start_time ripple_end = ripple_times.iloc[i].end_time plt.plot(     ripple_band_df.loc[ripple_start - window : ripple_end + window].index,     ripple_band_df.loc[ripple_start - window : ripple_end + window].iloc[         :, ::15     ], ) plt.axvline(ripple_start, color=\"r\") plt.axvline(ripple_end, color=\"r\")  plt.xlabel(\"Time (s)\") plt.ylabel(\"Voltage (uV)\") Out[103]: <pre>Text(0, 0.5, 'Voltage (uV)')</pre>"}, {"location": "notebooks/32_Ripple_Detection/#ripple-detection", "title": "Ripple Detection\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#generate-lfp-ripple-band", "title": "Generate LFP Ripple Band\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#make-lfp", "title": "Make LFP\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#populate-ripple-band", "title": "Populate Ripple Band\u00b6", "text": "<p>We now create a filter for this frequency band</p>"}, {"location": "notebooks/32_Ripple_Detection/#selecting-ripple-analysis-electrodes", "title": "Selecting Ripple Analysis Electrodes\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#setting-ripple-parameters", "title": "Setting Ripple Parameters\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#check-interval-speed", "title": "Check interval speed\u00b6", "text": "<p>The speed for this interval should exist under the default position parameter set and for a given interval. We can quickly populate this here</p>"}, {"location": "notebooks/32_Ripple_Detection/#run-ripple-detection", "title": "Run Ripple Detection\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#up-next", "title": "Up Next\u00b6", "text": "<p>Next, we'll extract mark indicator.</p>"}, {"location": "notebooks/40_Extracting_Clusterless_Waveform_Features/", "title": "Extracting Waveforms", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on clusterless decoding in Spyglass</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> <li>Prior to running, please familiarize yourself with the spike sorting pipeline and generate input position data with either the Trodes or DLC notebooks (1, 2, 3).</li> </ul> <p>The goal of this notebook is to populate the <code>UnitWaveformFeatures</code> table, which depends <code>SpikeSortingOutput</code>. This table contains the features of the waveforms of each unit.</p> <p>While clusterless decoding avoids actual spike sorting, we need to pass through these tables to maintain (relative) pipeline simplicity. Pass-through tables keep spike sorting and clusterless waveform extraction as similar as possible, by using shared steps. Here, \"spike sorting\" involves simple thresholding (sorter: clusterless_thresholder).</p> In\u00a0[1]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info <p>First, if you haven't inserted the the <code>mediumnwb20230802.wnb</code> file into the database, you should do so now. This is the file that we will use for the decoding tutorials.</p> <p>It is a truncated version of the full NWB file, so it will run faster, but bigger than the minirec file we used in the previous tutorials so that decoding makes sense.</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nimport spyglass.data_import as sgi\nimport spyglass.position as sgp\n\n# Insert the nwb file\nnwb_file_name = \"mediumnwb20230802.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nsgi.insert_sessions(nwb_file_name)\n\n# Position\nsgp.v1.TrodesPosParams.insert_default()\n\ninterval_list_name = \"pos 0 valid times\"\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": \"default\",\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_key)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename import spyglass.data_import as sgi import spyglass.position as sgp  # Insert the nwb file nwb_file_name = \"mediumnwb20230802.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) sgi.insert_sessions(nwb_file_name)  # Position sgp.v1.TrodesPosParams.insert_default()  interval_list_name = \"pos 0 valid times\"  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": \"default\", } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_key) <pre>[2024-01-17 22:14:51,194][INFO]: Connecting root@localhost:3306\n[2024-01-17 22:14:51,274][INFO]: Connected root@localhost:3306\n/Users/edeno/Documents/GitHub/spyglass/src/spyglass/data_import/insert_sessions.py:58: UserWarning: Cannot insert data from mediumnwb20230802.nwb: mediumnwb20230802_.nwb is already in Nwbfile table.\n  warnings.warn(\n</pre> <p>These next steps are the same as in the Spike Sorting notebook, but we'll repeat them here for clarity. These are pre-processing steps that are shared between spike sorting and clusterless decoding.</p> <p>We first set the <code>SortGroup</code> to define which contacts are sorted together.</p> <p>We then setup for spike sorting by bandpass filtering and whitening the data via the <code>SpikeSortingRecording</code> table.</p> In\u00a0[3]: Copied! <pre>import spyglass.spikesorting.v1 as sgs\n\nsgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_copy_file_name)\n\nsort_group_ids = (sgs.SortGroup &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(\n    \"sort_group_id\"\n)\n\ngroup_keys = []\nfor sort_group_id in sort_group_ids:\n    key = {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"sort_group_id\": sort_group_id,\n        \"interval_list_name\": interval_list_name,\n        \"preproc_param_name\": \"default\",\n        \"team_name\": \"Alison Comrie\",\n    }\n    group_keys.append(key)\n    sgs.SpikeSortingRecordingSelection.insert_selection(key)\n\nsgs.SpikeSortingRecording.populate(group_keys)\n</pre> import spyglass.spikesorting.v1 as sgs  sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_copy_file_name)  sort_group_ids = (sgs.SortGroup &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(     \"sort_group_id\" )  group_keys = [] for sort_group_id in sort_group_ids:     key = {         \"nwb_file_name\": nwb_copy_file_name,         \"sort_group_id\": sort_group_id,         \"interval_list_name\": interval_list_name,         \"preproc_param_name\": \"default\",         \"team_name\": \"Alison Comrie\",     }     group_keys.append(key)     sgs.SpikeSortingRecordingSelection.insert_selection(key)  sgs.SpikeSortingRecording.populate(group_keys) <pre>[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n</pre> <p>Next we do artifact detection. Here we skip it by setting the <code>artifact_param_name</code> to <code>None</code>, but in practice you should detect artifacts as it will affect the decoding.</p> In\u00a0[4]: Copied! <pre>recording_ids = (\n    sgs.SpikeSortingRecordingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name}\n).fetch(\"recording_id\")\n\ngroup_keys = []\nfor recording_id in recording_ids:\n    key = {\n        \"recording_id\": recording_id,\n        \"artifact_param_name\": \"none\",\n    }\n    group_keys.append(key)\n    sgs.ArtifactDetectionSelection.insert_selection(key)\n\nsgs.ArtifactDetection.populate(group_keys)\n</pre> recording_ids = (     sgs.SpikeSortingRecordingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name} ).fetch(\"recording_id\")  group_keys = [] for recording_id in recording_ids:     key = {         \"recording_id\": recording_id,         \"artifact_param_name\": \"none\",     }     group_keys.append(key)     sgs.ArtifactDetectionSelection.insert_selection(key)  sgs.ArtifactDetection.populate(group_keys) <pre>[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n</pre> <p>Now we run the \"spike sorting\", which in our case is simply thresholding the signal to find spikes. We use the <code>SpikeSorting</code> table to store the results. Note that <code>sorter_param_name</code> defines the parameters for thresholding the signal.</p> In\u00a0[5]: Copied! <pre>group_keys = []\nfor recording_id in recording_ids:\n    key = {\n        \"recording_id\": recording_id,\n        \"sorter\": \"clusterless_thresholder\",\n        \"sorter_param_name\": \"default_clusterless\",\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": str(\n            (\n                sgs.ArtifactDetectionSelection &amp; {\"recording_id\": recording_id}\n            ).fetch1(\"artifact_id\")\n        ),\n    }\n    group_keys.append(key)\n    sgs.SpikeSortingSelection.insert_selection(key)\n\nsgs.SpikeSorting.populate(group_keys)\n</pre> group_keys = [] for recording_id in recording_ids:     key = {         \"recording_id\": recording_id,         \"sorter\": \"clusterless_thresholder\",         \"sorter_param_name\": \"default_clusterless\",         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": str(             (                 sgs.ArtifactDetectionSelection &amp; {\"recording_id\": recording_id}             ).fetch1(\"artifact_id\")         ),     }     group_keys.append(key)     sgs.SpikeSortingSelection.insert_selection(key)  sgs.SpikeSorting.populate(group_keys) <pre>[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n</pre> <p>For clusterless decoding we do not need any manual curation, but for the sake of the pipeline, we need to store the output of the thresholding in the <code>CurationV1</code> table and insert this into the <code>SpikeSortingOutput</code> table.</p> In\u00a0[6]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\nsorting_ids = (\n    sgs.SpikeSortingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name}\n).fetch(\"sorting_id\")\n\nfor sorting_id in sorting_ids:\n    try:\n        sgs.CurationV1.insert_curation(sorting_id=sorting_id)\n    except KeyError as e:\n        pass\n\nSpikeSortingOutput.insert(\n    sgs.CurationV1().fetch(\"KEY\"),\n    part_name=\"CurationV1\",\n    skip_duplicates=True,\n)\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  sorting_ids = (     sgs.SpikeSortingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name} ).fetch(\"sorting_id\")  for sorting_id in sorting_ids:     try:         sgs.CurationV1.insert_curation(sorting_id=sorting_id)     except KeyError as e:         pass  SpikeSortingOutput.insert(     sgs.CurationV1().fetch(\"KEY\"),     part_name=\"CurationV1\",     skip_duplicates=True, ) <p>Finally, we extract the waveform features of each SortGroup. This is done by the <code>UnitWaveformFeatures</code> table.</p> <p>To set this up, we use the <code>WaveformFeaturesParams</code> to define the time around the spike that we want to use for feature extraction, and which features to extract. Here is an example of the parameters used for extraction the amplitude of the negative peak of the waveform:</p> <pre>waveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    }\n}\n</pre> <p>We see that we want 0.5 ms of time before and after the peak of the negative spike. We also see that we want to extract the amplitude of the negative peak, and that we do not want to estimate the peak time (since we know it is at 0 ms).</p> <p>You can define other features to extract such as spatial location of the spike:</p> <pre>waveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    },\n    \"spike location\": {}\n}\n</pre> <p>Note: Members of the Frank Lab can use \"ampl_10_jobs_v2\" instead of \"amplitude\" for significant speed improvements.</p> In\u00a0[7]: Copied! <pre>from spyglass.decoding.v1.waveform_features import WaveformFeaturesParams\n\nwaveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    }\n}\n\nWaveformFeaturesParams.insert1(\n    {\n        \"features_param_name\": \"amplitude\",\n        \"params\": {\n            \"waveform_extraction_params\": waveform_extraction_params,\n            \"waveform_feature_params\": waveform_feature_params,\n        },\n    },\n    skip_duplicates=True,\n)\n\nWaveformFeaturesParams()\n</pre> from spyglass.decoding.v1.waveform_features import WaveformFeaturesParams  waveform_extraction_params = {     \"ms_before\": 0.5,     \"ms_after\": 0.5,     \"max_spikes_per_unit\": None,     \"n_jobs\": 5,     \"total_memory\": \"5G\", } waveform_feature_params = {     \"amplitude\": {         \"peak_sign\": \"neg\",         \"estimate_peak_time\": False,     } }  WaveformFeaturesParams.insert1(     {         \"features_param_name\": \"amplitude\",         \"params\": {             \"waveform_extraction_params\": waveform_extraction_params,             \"waveform_feature_params\": waveform_feature_params,         },     },     skip_duplicates=True, )  WaveformFeaturesParams() Out[7]: <p>features_param_name</p> a name for this set of parameters <p>params</p> the parameters for the waveform features amplitude =BLOB=amplitude, spike_location =BLOB= <p>Total: 2</p> <p>Now that we've inserted the waveform features parameters, we need to define which parameters to use for each SortGroup. This is done by the <code>UnitWaveformFeaturesSelection</code> table. We need to link the primary key <code>merge_id</code> from the <code>SpikeSortingOutput</code> table to a features parameter set.</p> In\u00a0[8]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\nUnitWaveformFeaturesSelection()\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection  UnitWaveformFeaturesSelection() Out[8]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>First we find the units we need:</p> In\u00a0[9]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\nmerge_ids = (\n    (SpikeSortingOutput.CurationV1 * sgs.SpikeSortingSelection)\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"sorter\": \"clusterless_thresholder\",\n        \"sorter_param_name\": \"default_clusterless\",\n    }\n).fetch(\"merge_id\")\nmerge_ids\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  merge_ids = (     (SpikeSortingOutput.CurationV1 * sgs.SpikeSortingSelection)     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"sorter\": \"clusterless_thresholder\",         \"sorter_param_name\": \"default_clusterless\",     } ).fetch(\"merge_id\") merge_ids Out[9]: <pre>array([UUID('485a4ddf-332d-35b5-3ad4-0561736c1844'),\n       UUID('6acb99b8-6a0c-eb83-1141-5f603c5895e0'),\n       UUID('f7237e18-4e73-4aee-805b-90735e9147de'),\n       UUID('7e3fa66e-727e-1541-819a-b01309bb30ae'),\n       UUID('6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5'),\n       UUID('e0e9133a-7a4e-1321-a43a-e8afcb2f25da'),\n       UUID('9959b614-2318-f597-6651-a3a82124d28a'),\n       UUID('c0eb6455-fc41-c200-b62e-e3ca81b9a3f7'),\n       UUID('912e250e-56d8-ee33-4525-c844d810971b'),\n       UUID('d7d2c97a-0e6e-d1b8-735c-d55dc66a30e1'),\n       UUID('abb92dce-4410-8f17-a501-a4104bda0dcf'),\n       UUID('74e10781-1228-4075-0870-af224024ffdc'),\n       UUID('8bbddc0f-d6ae-6260-9400-f884a6e25ae8'),\n       UUID('614d796c-0b95-6364-aaa0-b6cb1e7bbb83'),\n       UUID('b332482b-e430-169d-8ac0-0a73ce968ed7'),\n       UUID('86897349-ff68-ac72-02eb-739dd88936e6'),\n       UUID('4a712103-c223-864f-82e0-6c23de79cc14'),\n       UUID('cf858380-e8a3-49de-c2a9-1a277e307a68'),\n       UUID('cc4ee561-f974-f8e5-0ea4-83185263ac67'),\n       UUID('4a72c253-b3ca-8c13-e615-736a7ebff35c'),\n       UUID('b92a94d8-ee1e-2097-a81f-5c1e1556ed24'),\n       UUID('5c53bd33-d57c-fbba-e0fb-55e0bcb85d03'),\n       UUID('0751a1e1-a406-7f87-ae6f-ce4ffc60621c')], dtype=object)</pre> <p>Then we link them with the features parameters:</p> In\u00a0[10]: Copied! <pre>selection_keys = [\n    {\n        \"spikesorting_merge_id\": merge_id,\n        \"features_param_name\": \"amplitude\",\n    }\n    for merge_id in merge_ids\n]\nUnitWaveformFeaturesSelection.insert(selection_keys, skip_duplicates=True)\n\nUnitWaveformFeaturesSelection &amp; selection_keys\n</pre> selection_keys = [     {         \"spikesorting_merge_id\": merge_id,         \"features_param_name\": \"amplitude\",     }     for merge_id in merge_ids ] UnitWaveformFeaturesSelection.insert(selection_keys, skip_duplicates=True)  UnitWaveformFeaturesSelection &amp; selection_keys Out[10]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>Finally, we extract the waveform features, by populating the <code>UnitWaveformFeatures</code> table:</p> In\u00a0[11]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeatures\n\nUnitWaveformFeatures.populate(selection_keys)\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeatures  UnitWaveformFeatures.populate(selection_keys) <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:08,494][WARNING]: Skipped checksum for file with hash: 6629fd95-636a-4ad4-c9af-cee507de2130, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_AMBBKQ9RIY.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:08][INFO] Spyglass: Writing new NWB file mediumnwb20230802_NQEPSMKPK0.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:19,450][WARNING]: Skipped checksum for file with hash: 6d04cbdb-e1e4-f44f-7274-0e1ab0356d75, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_W1MLF0Q86S.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:19][INFO] Spyglass: Writing new NWB file mediumnwb20230802_F02UG5Z5FR.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:30,787][WARNING]: Skipped checksum for file with hash: 8993754e-7dbe-94a1-403d-8c55aa9c6c42, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_JN4A4GSLZB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:31][INFO] Spyglass: Writing new NWB file mediumnwb20230802_OTV91MLKDT.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:41,633][WARNING]: Skipped checksum for file with hash: 9e24661c-b021-6ad4-f224-89e331334f18, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T2DBO3EMZ8.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:41][INFO] Spyglass: Writing new NWB file mediumnwb20230802_TSPNTCGNN1.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:52,561][WARNING]: Skipped checksum for file with hash: f64f34ee-e72d-e566-a048-65f2ea31708a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_USMRXAAV8I.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:52][INFO] Spyglass: Writing new NWB file mediumnwb20230802_QSK70WFDJH.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:03,559][WARNING]: Skipped checksum for file with hash: 6d13e338-41bd-b011-beb5-4de53d9d467b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_JA2OA12RPN.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:03][INFO] Spyglass: Writing new NWB file mediumnwb20230802_DO45HKXYTB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:14,288][WARNING]: Skipped checksum for file with hash: d740eb7d-ce29-e140-06a2-c56655e0842a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_L92EE1VRPB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:14][INFO] Spyglass: Writing new NWB file mediumnwb20230802_KFIYRJ4HFO.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:24,130][WARNING]: Skipped checksum for file with hash: 1f386cd3-89da-0233-03ff-76ba94e91a3a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TX2ZX3DAP4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:24][INFO] Spyglass: Writing new NWB file mediumnwb20230802_0YIM5K3H47.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:35,048][WARNING]: Skipped checksum for file with hash: fa76d419-77a4-697a-325d-5c2ddbe517f9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0R6AWXMC6G.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:35][INFO] Spyglass: Writing new NWB file mediumnwb20230802_CTLEGE2TWZ.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:46,009][WARNING]: Skipped checksum for file with hash: ce4cb0c3-3dd0-70fd-8ea0-98a8b84592d9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7UIA2ILMG6.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:46][INFO] Spyglass: Writing new NWB file mediumnwb20230802_7EN0N1U4U1.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:56,814][WARNING]: Skipped checksum for file with hash: e43f95ff-9779-b980-00a3-99e104864462, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_AKOI7OTASI.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:57][INFO] Spyglass: Writing new NWB file mediumnwb20230802_DHKWBWWAMC.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:05,013][WARNING]: Skipped checksum for file with hash: ff81d274-17f7-702d-a2b4-92ac43c29316, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Y2YF504C5D.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:05][INFO] Spyglass: Writing new NWB file mediumnwb20230802_PEN0D79Q0B.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:15,903][WARNING]: Skipped checksum for file with hash: e282a8e5-844b-20f6-345c-cded12e761a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DUNM1TZUGR.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:16][INFO] Spyglass: Writing new NWB file mediumnwb20230802_WP7SIXDJ2A.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:26,609][WARNING]: Skipped checksum for file with hash: 7d05460d-7366-27c9-2ba7-de2ad5d402f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_4JXWFJ3JRI.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:26][INFO] Spyglass: Writing new NWB file mediumnwb20230802_B82OS6W1QA.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:37,652][WARNING]: Skipped checksum for file with hash: c202eb9e-ca43-0a72-4086-57a5bb6eb937, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_5TY04H3B5T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:37][INFO] Spyglass: Writing new NWB file mediumnwb20230802_XO17FQLN6T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:47,269][WARNING]: Skipped checksum for file with hash: 4357905c-c6b9-3990-4d62-740a54cfc667, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_X84BYVM2B0.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:47][INFO] Spyglass: Writing new NWB file mediumnwb20230802_OCFI0GFLZ9.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:58,240][WARNING]: Skipped checksum for file with hash: 4c1103ac-eaca-b282-e5ff-aa2194e65a43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_2R6VQ8EDL4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:58][INFO] Spyglass: Writing new NWB file mediumnwb20230802_60M9VSZX0W.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:09,119][WARNING]: Skipped checksum for file with hash: 023c874f-8114-3ef6-7fcf-813844787d5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_L7HDY9IDHO.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:09][INFO] Spyglass: Writing new NWB file mediumnwb20230802_Z5HJ68LHYW.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:20,605][WARNING]: Skipped checksum for file with hash: fde8b240-6adc-86f0-6391-f3f6fad72ee9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_HWU3E4EKP4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:20][INFO] Spyglass: Writing new NWB file mediumnwb20230802_U5U5JVGY4F.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:31,780][WARNING]: Skipped checksum for file with hash: c592e63b-4db1-40be-632e-0180e6fa02d7, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_SGAU9PX7US.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:32][INFO] Spyglass: Writing new NWB file mediumnwb20230802_0D5Z0NSIP8.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:42,644][WARNING]: Skipped checksum for file with hash: 148d9058-e6dc-e959-4c4d-75db9aa0b6e4, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EF6N6XI3AH.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:42][INFO] Spyglass: Writing new NWB file mediumnwb20230802_EYV2NARUKU.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:54,570][WARNING]: Skipped checksum for file with hash: b4b6404f-aaf8-c4cc-9abe-ceea56e103f3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_O7ZZ0F1XN7.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:54][INFO] Spyglass: Writing new NWB file mediumnwb20230802_T4XBCIW44T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:19:05,568][WARNING]: Skipped checksum for file with hash: 26f7bdc7-da8d-6ad5-3f4a-554ceb48755e, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0TKF5589B7.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:19:05][INFO] Spyglass: Writing new NWB file mediumnwb20230802_UD55CR8LZK.nwb\n</pre> In\u00a0[12]: Copied! <pre>UnitWaveformFeatures &amp; selection_keys\n</pre> UnitWaveformFeatures &amp; selection_keys Out[12]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters <p>analysis_file_name</p> name of the file <p>object_id</p> the NWB object that stores the waveforms 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude mediumnwb20230802_NQEPSMKPK0.nwb 8607d6a6-213c-431d-ab99-70196b6cf0bf485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude mediumnwb20230802_F02UG5Z5FR.nwb 9f693a74-a203-4628-b3ec-50a32b3549d84a712103-c223-864f-82e0-6c23de79cc14 amplitude mediumnwb20230802_OTV91MLKDT.nwb 648953e8-1891-4c90-9756-d6b7cc2b7c3d4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude mediumnwb20230802_TSPNTCGNN1.nwb 6d0af664-f811-4781-9a23-cac437fb2d155c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude mediumnwb20230802_QSK70WFDJH.nwb a67ed5bb-3edd-465e-8737-ee08f3e7d7d5614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude mediumnwb20230802_DO45HKXYTB.nwb 13218b00-bf34-455c-9c38-c3b3174d40096acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude mediumnwb20230802_KFIYRJ4HFO.nwb d892bb47-94fc-4c29-acab-d5b3d9565c976d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude mediumnwb20230802_0YIM5K3H47.nwb 60f4d280-a42a-4a77-9c35-9bd4d2c7699174e10781-1228-4075-0870-af224024ffdc amplitude mediumnwb20230802_CTLEGE2TWZ.nwb 99f51e3d-54b5-41d7-a61e-7013b22fb0667e3fa66e-727e-1541-819a-b01309bb30ae amplitude mediumnwb20230802_7EN0N1U4U1.nwb 535b28d1-c9b5-4d7d-a4f5-4d80508542b486897349-ff68-ac72-02eb-739dd88936e6 amplitude mediumnwb20230802_DHKWBWWAMC.nwb 67ee1547-c570-4746-b886-748d96075b548bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude mediumnwb20230802_PEN0D79Q0B.nwb 5dd7b87f-4cf0-4a91-a281-15c6f6c86d61 <p>...</p> <p>Total: 23</p> <p>Now that we've extracted the data, we can inspect the results. Let's fetch the data:</p> In\u00a0[13]: Copied! <pre>spike_times, spike_waveform_features = (\n    UnitWaveformFeatures &amp; selection_keys\n).fetch_data()\n</pre> spike_times, spike_waveform_features = (     UnitWaveformFeatures &amp; selection_keys ).fetch_data() <pre>[2024-01-17 22:19:07,354][WARNING]: Skipped checksum for file with hash: a7c9b1d9-d1a2-7f40-9127-206e83a87006, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_NQEPSMKPK0.nwb\n[2024-01-17 22:19:07,359][WARNING]: Skipped checksum for file with hash: ec7faa5b-3847-6649-1a93-74ebd50dcfb9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_F02UG5Z5FR.nwb\n[2024-01-17 22:19:07,369][WARNING]: Skipped checksum for file with hash: 8e964932-96ab-e1c9-2133-edce8eacab5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OTV91MLKDT.nwb\n[2024-01-17 22:19:07,379][WARNING]: Skipped checksum for file with hash: 895bac7b-bfd6-b4f2-b2ad-460362aaafa8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TSPNTCGNN1.nwb\n[2024-01-17 22:19:07,382][WARNING]: Skipped checksum for file with hash: 58713583-cf49-4527-7707-105f9c9ee477, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_QSK70WFDJH.nwb\n[2024-01-17 22:19:07,385][WARNING]: Skipped checksum for file with hash: a64829f8-ab12-fecc-eda9-a22b90b20d43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DO45HKXYTB.nwb\n[2024-01-17 22:19:07,391][WARNING]: Skipped checksum for file with hash: 3a580271-9126-8e57-048e-a7bbb3f917b9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_KFIYRJ4HFO.nwb\n[2024-01-17 22:19:07,395][WARNING]: Skipped checksum for file with hash: 13cf8ad9-023c-c9b7-05c3-eaa3330304f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0YIM5K3H47.nwb\n[2024-01-17 22:19:07,397][WARNING]: Skipped checksum for file with hash: 7ce8a640-0a25-4866-6d5a-aa2c65f0aca5, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_CTLEGE2TWZ.nwb\n[2024-01-17 22:19:07,399][WARNING]: Skipped checksum for file with hash: aa657f4f-f409-d444-8b32-31d37abe0797, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7EN0N1U4U1.nwb\n[2024-01-17 22:19:07,401][WARNING]: Skipped checksum for file with hash: f3b4bd22-1439-e6d2-4e15-aa3650143fdf, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DHKWBWWAMC.nwb\n[2024-01-17 22:19:07,404][WARNING]: Skipped checksum for file with hash: 68eac0b2-e5be-e0c5-9eae-cd8dbe6676a8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_PEN0D79Q0B.nwb\n[2024-01-17 22:19:07,407][WARNING]: Skipped checksum for file with hash: c8b95099-2cb3-df0b-5ab1-7a5e120a8e2f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_WP7SIXDJ2A.nwb\n[2024-01-17 22:19:07,409][WARNING]: Skipped checksum for file with hash: 8fae8089-f683-5f0a-4e59-c71d6ee14f38, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_B82OS6W1QA.nwb\n[2024-01-17 22:19:07,412][WARNING]: Skipped checksum for file with hash: dd9d0f51-6445-b368-32bd-b1f142bf6ed3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_XO17FQLN6T.nwb\n[2024-01-17 22:19:07,414][WARNING]: Skipped checksum for file with hash: 4e2cf5f5-ff7c-1a2b-db85-2d1c4f036fbd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OCFI0GFLZ9.nwb\n[2024-01-17 22:19:07,416][WARNING]: Skipped checksum for file with hash: 8691c252-0bd1-122b-8cf3-b89c4d0fdee0, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_60M9VSZX0W.nwb\n[2024-01-17 22:19:07,419][WARNING]: Skipped checksum for file with hash: 57b89835-8edb-e91d-0798-09d22fb4fbc9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Z5HJ68LHYW.nwb\n[2024-01-17 22:19:07,422][WARNING]: Skipped checksum for file with hash: 54401121-4426-86c9-72f7-e056bc16e99d, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_U5U5JVGY4F.nwb\n[2024-01-17 22:19:07,424][WARNING]: Skipped checksum for file with hash: 0ff21e84-2214-6911-2575-a9c92a541407, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0D5Z0NSIP8.nwb\n[2024-01-17 22:19:07,426][WARNING]: Skipped checksum for file with hash: 0949b006-5309-93c8-fd8b-1308e8130869, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EYV2NARUKU.nwb\n[2024-01-17 22:19:07,428][WARNING]: Skipped checksum for file with hash: b4b31e50-dfa2-0d02-514a-525782a81255, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T4XBCIW44T.nwb\n[2024-01-17 22:19:07,430][WARNING]: Skipped checksum for file with hash: c18a9ac4-06bc-4249-2bad-439d4f618421, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_UD55CR8LZK.nwb\n</pre> <p>Let's look at the features shape. This is a list corresponding to tetrodes, with each element being a numpy array of shape (n_spikes, n_features). The features in this case are the amplitude of each tetrode wire at the negative peak of the waveform.</p> In\u00a0[14]: Copied! <pre>for features in spike_waveform_features:\n    print(features.shape)\n</pre> for features in spike_waveform_features:     print(features.shape) <pre>(49808, 4)\n(21675, 4)\n(21024, 4)\n(51330, 4)\n(43804, 4)\n(6348, 4)\n(12188, 4)\n(2654, 4)\n(99400, 4)\n(8952, 4)\n(39886, 4)\n(18, 4)\n(44284, 4)\n(8283, 4)\n(36687, 4)\n(803, 4)\n(76353, 4)\n(11367, 4)\n(41622, 4)\n(106549, 4)\n(57394, 4)\n(30772, 4)\n(4198, 4)\n</pre> <p>We can plot the amplitudes to see if there is anything that looks neural and to look for outliers:</p> In\u00a0[15]: Copied! <pre>import matplotlib.pyplot as plt\n\ntetrode_ind = 1\nplt.scatter(\n    spike_waveform_features[tetrode_ind][:, 0],\n    spike_waveform_features[tetrode_ind][:, 1],\n    s=1,\n)\n</pre> import matplotlib.pyplot as plt  tetrode_ind = 1 plt.scatter(     spike_waveform_features[tetrode_ind][:, 0],     spike_waveform_features[tetrode_ind][:, 1],     s=1, ) Out[15]: <pre>&lt;matplotlib.collections.PathCollection at 0x1c61bf940&gt;</pre> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/41_Decoding_Clusterless/", "title": "Decoding Clusterless", "text": "In\u00a0[1]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info In\u00a0[2]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\nimport spyglass.spikesorting.v1 as sgs\nfrom spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\nsorter_keys = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorter\": \"clusterless_thresholder\",\n    \"sorter_param_name\": \"default_clusterless\",\n}\n\nfeature_key = {\"features_param_name\": \"amplitude\"}\n\n(sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 * (\n    UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")\n    &amp; feature_key\n)\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput import spyglass.spikesorting.v1 as sgs from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection   nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  sorter_keys = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorter\": \"clusterless_thresholder\",     \"sorter_param_name\": \"default_clusterless\", }  feature_key = {\"features_param_name\": \"amplitude\"}  (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 * (     UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")     &amp; feature_key ) <pre>[2024-01-29 10:24:06,266][INFO]: Connecting root@localhost:3306\n[2024-01-29 10:24:06,337][INFO]: Connected root@localhost:3306\n[10:24:07][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> Out[2]: <p>sorting_id</p> <p>merge_id</p> <p>features_param_name</p> a name for this set of parameters <p>recording_id</p> <p>sorter</p> <p>sorter_param_name</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>curation_id</p> 08a302b6-5505-40fa-b4d5-62162f8eef58 485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude 449b64e3-db0b-437e-a1b9-0d29928aa2dd clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 45f6b9a1-eef3-46eb-866d-d0999afebda6 00ca508ee-af4c-4a89-8181-d48bd209bfd4 6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude 328da21c-1d9c-41e2-9800-76b3484b707b clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 686d9951-1c0f-4d5e-9f5c-09e6fd8bdd4c 0209dc048-6fae-4315-b293-c06fff29f947 f7237e18-4e73-4aee-805b-90735e9147de amplitude aff78f2f-2ba0-412a-95cc-447c3a2f4683 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 719e8a86-fcf1-4ffc-8c1f-ea912f67ad5d 021a9a593-f6f3-4b82-99d7-8fc46556eff3 7e3fa66e-727e-1541-819a-b01309bb30ae amplitude 2402805a-04f9-4a88-9ccf-071376c8de19 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb d581b117-160e-4311-b096-7781a4de4394 0406a20e3-5a9f-4fec-b046-a6561f72461e 6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude f1427e00-2974-4301-b2ac-b4dc29277c51 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 0e848c38-9105-4ea4-b6ba-dbdd5b46a088 04131c51b-c56d-41fa-b046-46635fc17fd9 e0e9133a-7a4e-1321-a43a-e8afcb2f25da amplitude 9e332d82-1daf-4e92-bb50-12e4f9430875 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 9ed11db5-c42e-491a-8caf-7d9a37a65f13 04c5a629a-71d9-481d-ab11-a4cb0fc16087 9959b614-2318-f597-6651-a3a82124d28a amplitude 3a2c3eed-413a-452a-83c8-0e4648141bde clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 2b9fbf14-74a0-4294-a805-26702340aac9 04d629c07-1931-4e1f-a3a8-cbf1b72161e3 c0eb6455-fc41-c200-b62e-e3ca81b9a3f7 amplitude f07bc0b0-de6b-4424-8ef9-766213aaca26 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 5c68f0f0-f577-4905-8a09-e4d171d0a22d 0554a9a3c-0461-48be-8435-123eed59c228 912e250e-56d8-ee33-4525-c844d810971b amplitude 7f128981-6868-4976-ba20-248655dcac21 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb f4b9301f-bc91-455b-9474-c801093f3856 07bb007f2-26d3-463f-b7dc-7bd4d271725e d7d2c97a-0e6e-d1b8-735c-d55dc66a30e1 amplitude a9b7cec0-1256-49cf-abf0-8c45fd155379 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 74270cba-36ee-4afb-ab50-2a6cc948e68c 080e1f37f-48a7-4087-bd37-7a37b6a2c160 abb92dce-4410-8f17-a501-a4104bda0dcf amplitude 3c40ebdc-0b61-4105-9971-e1348bd49bc7 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 0f91197e-bebb-4dc6-ad41-5bf89c3eed28 08848c4a8-a2f2-4f3d-82cd-51b13b8bae3c 74e10781-1228-4075-0870-af224024ffdc amplitude 257c077b-8f3b-4abb-a631-6b8084d6a1ea clusterless_thresholder default_clusterless mediumnwb20230802_.nwb e289e03d-32ad-461a-a1cc-c88537343149 0 <p>...</p> <p>Total: 23</p> In\u00a0[3]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\nspikesorting_merge_id = (\n    (sgs.SpikeSortingSelection &amp; sorter_keys)\n    * SpikeSortingOutput.CurationV1\n    * (\n        UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")\n        &amp; feature_key\n    )\n).fetch(\"merge_id\")\n\nwaveform_selection_keys = [\n    {\"spikesorting_merge_id\": merge_id, \"features_param_name\": \"amplitude\"}\n    for merge_id in spikesorting_merge_id\n]\n\nUnitWaveformFeaturesSelection &amp; waveform_selection_keys\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection  spikesorting_merge_id = (     (sgs.SpikeSortingSelection &amp; sorter_keys)     * SpikeSortingOutput.CurationV1     * (         UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")         &amp; feature_key     ) ).fetch(\"merge_id\")  waveform_selection_keys = [     {\"spikesorting_merge_id\": merge_id, \"features_param_name\": \"amplitude\"}     for merge_id in spikesorting_merge_id ]  UnitWaveformFeaturesSelection &amp; waveform_selection_keys Out[3]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>We will create a group called <code>test_group</code> that contains all of the tetrodes that we want to decode from. We will use the <code>create_group</code> function to create this group. This function takes two arguments: the name of the group, and the keys of the tables that we want to include in the group.</p> In\u00a0[4]: Copied! <pre>from spyglass.decoding.v1.clusterless import UnitWaveformFeaturesGroup\n\nUnitWaveformFeaturesGroup().create_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test_group\",\n    keys=waveform_selection_keys,\n)\nUnitWaveformFeaturesGroup &amp; {\"waveform_features_group_name\": \"test_group\"}\n</pre> from spyglass.decoding.v1.clusterless import UnitWaveformFeaturesGroup  UnitWaveformFeaturesGroup().create_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test_group\",     keys=waveform_selection_keys, ) UnitWaveformFeaturesGroup &amp; {\"waveform_features_group_name\": \"test_group\"} Out[4]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> mediumnwb20230802_.nwb test_group <p>Total: 1</p> <p>We can see that we successfully associated \"test_group\" with the tetrodes that we want to decode from by using the <code>get_group</code> function.</p> In\u00a0[5]: Copied! <pre>UnitWaveformFeaturesGroup.UnitFeatures &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"waveform_features_group_name\": \"test_group\",\n}\n</pre> UnitWaveformFeaturesGroup.UnitFeatures &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"waveform_features_group_name\": \"test_group\", } Out[5]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters mediumnwb20230802_.nwb test_group 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitudemediumnwb20230802_.nwb test_group 485a4ddf-332d-35b5-3ad4-0561736c1844 amplitudemediumnwb20230802_.nwb test_group 4a712103-c223-864f-82e0-6c23de79cc14 amplitudemediumnwb20230802_.nwb test_group 4a72c253-b3ca-8c13-e615-736a7ebff35c amplitudemediumnwb20230802_.nwb test_group 5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitudemediumnwb20230802_.nwb test_group 614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitudemediumnwb20230802_.nwb test_group 6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitudemediumnwb20230802_.nwb test_group 6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitudemediumnwb20230802_.nwb test_group 74e10781-1228-4075-0870-af224024ffdc amplitudemediumnwb20230802_.nwb test_group 7e3fa66e-727e-1541-819a-b01309bb30ae amplitudemediumnwb20230802_.nwb test_group 86897349-ff68-ac72-02eb-739dd88936e6 amplitudemediumnwb20230802_.nwb test_group 8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> In\u00a0[6]: Copied! <pre>from spyglass.position import PositionOutput\nimport spyglass.position as sgp\n\n\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": \"default_decoding\",\n        \"params\": {\n            \"max_LED_separation\": 9.0,\n            \"max_plausible_speed\": 300.0,\n            \"position_smoothing_duration\": 0.125,\n            \"speed_smoothing_std_dev\": 0.100,\n            \"orient_smoothing_std_dev\": 0.001,\n            \"led1_is_front\": 1,\n            \"is_upsampled\": 1,\n            \"upsampling_sampling_rate\": 250,\n            \"upsampling_interpolation_method\": \"linear\",\n        },\n    },\n    skip_duplicates=True,\n)\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",\n    \"trodes_pos_params_name\": \"default_decoding\",\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_key)\n\nPositionOutput.TrodesPosV1 &amp; trodes_s_key\n</pre> from spyglass.position import PositionOutput import spyglass.position as sgp   sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": \"default_decoding\",         \"params\": {             \"max_LED_separation\": 9.0,             \"max_plausible_speed\": 300.0,             \"position_smoothing_duration\": 0.125,             \"speed_smoothing_std_dev\": 0.100,             \"orient_smoothing_std_dev\": 0.001,             \"led1_is_front\": 1,             \"is_upsampled\": 1,             \"upsampling_sampling_rate\": 250,             \"upsampling_interpolation_method\": \"linear\",         },     },     skip_duplicates=True, )  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",     \"trodes_pos_params_name\": \"default_decoding\", } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_key)  PositionOutput.TrodesPosV1 &amp; trodes_s_key <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[10:24:13][INFO] Spyglass: Writing new NWB file mediumnwb20230802_FUSH604NQA.nwb\n</pre> <pre>Computing position for: {'nwb_file_name': 'mediumnwb20230802_.nwb', 'interval_list_name': 'pos 0 valid times', 'trodes_pos_params_name': 'default_decoding'}\n</pre> <pre>[2024-01-29 10:24:13,819][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[2024-01-29 10:24:13,821][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[2024-01-29 10:24:13,996][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[2024-01-29 10:24:13,998][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[10:24:14][INFO] Spyglass: No video frame index found. Assuming all camera frames are present.\n</pre> Out[6]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters 6dfae23d-6034-e483-06e7-28ab4c29282f mediumnwb20230802_.nwb pos 0 valid times default_decoding <p>Total: 1</p> In\u00a0[7]: Copied! <pre>from spyglass.decoding.v1.core import PositionGroup\n\nposition_merge_ids = (\n    PositionOutput.TrodesPosV1\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": \"pos 0 valid times\",\n        \"trodes_pos_params_name\": \"default_decoding\",\n    }\n).fetch(\"merge_id\")\n\nPositionGroup().create_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test_group\",\n    keys=[{\"pos_merge_id\": merge_id} for merge_id in position_merge_ids],\n)\n\nPositionGroup &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"position_group_name\": \"test_group\",\n}\n</pre> from spyglass.decoding.v1.core import PositionGroup  position_merge_ids = (     PositionOutput.TrodesPosV1     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": \"pos 0 valid times\",         \"trodes_pos_params_name\": \"default_decoding\",     } ).fetch(\"merge_id\")  PositionGroup().create_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test_group\",     keys=[{\"pos_merge_id\": merge_id} for merge_id in position_merge_ids], )  PositionGroup &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"position_group_name\": \"test_group\", } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>position_group_name</p> <p>position_variables</p> list of position variables to decode mediumnwb20230802_.nwb test_group =BLOB= <p>Total: 1</p> In\u00a0[8]: Copied! <pre>(\n    PositionGroup\n    &amp; {\"nwb_file_name\": nwb_copy_file_name, \"position_group_name\": \"test_group\"}\n).fetch1(\"position_variables\")\n</pre> (     PositionGroup     &amp; {\"nwb_file_name\": nwb_copy_file_name, \"position_group_name\": \"test_group\"} ).fetch1(\"position_variables\") Out[8]: <pre>['position_x', 'position_y']</pre> In\u00a0[9]: Copied! <pre>PositionGroup.Position &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"position_group_name\": \"test_group\",\n}\n</pre> PositionGroup.Position &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"position_group_name\": \"test_group\", } Out[9]: <p>nwb_file_name</p> name of the NWB file <p>position_group_name</p> <p>pos_merge_id</p> mediumnwb20230802_.nwb test_group 6dfae23d-6034-e483-06e7-28ab4c29282f <p>Total: 1</p> In\u00a0[10]: Copied! <pre>from non_local_detector.models import ContFragClusterlessClassifier\n\nContFragClusterlessClassifier()\n</pre> from non_local_detector.models import ContFragClusterlessClassifier  ContFragClusterlessClassifier() Out[10]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> <p>You can change these parameters like so:</p> In\u00a0[11]: Copied! <pre>from non_local_detector.models import ContFragClusterlessClassifier\n\nContFragClusterlessClassifier(\n    clusterless_algorithm_params={\n        \"block_size\": 10000,\n        \"position_std\": 12.0,\n        \"waveform_std\": 24.0,\n    },\n)\n</pre> from non_local_detector.models import ContFragClusterlessClassifier  ContFragClusterlessClassifier(     clusterless_algorithm_params={         \"block_size\": 10000,         \"position_std\": 12.0,         \"waveform_std\": 24.0,     }, ) Out[11]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 12.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, us...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 12.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, us...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> <p>This is how to insert the model parameters into the database:</p> In\u00a0[12]: Copied! <pre>from spyglass.decoding.v1.core import DecodingParameters\n\n\nDecodingParameters.insert1(\n    {\n        \"decoding_param_name\": \"contfrag_clusterless\",\n        \"decoding_params\": ContFragClusterlessClassifier(),\n        \"decoding_kwargs\": dict(),\n    },\n    skip_duplicates=True,\n)\n\nDecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"}\n</pre> from spyglass.decoding.v1.core import DecodingParameters   DecodingParameters.insert1(     {         \"decoding_param_name\": \"contfrag_clusterless\",         \"decoding_params\": ContFragClusterlessClassifier(),         \"decoding_kwargs\": dict(),     },     skip_duplicates=True, )  DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"} Out[12]: <p>decoding_param_name</p> a name for this set of parameters <p>decoding_params</p> initialization parameters for model <p>decoding_kwargs</p> additional keyword arguments contfrag_clusterless =BLOB= =BLOB= <p>Total: 1</p> <p>We can retrieve these parameters and rebuild the model like so:</p> In\u00a0[13]: Copied! <pre>model_params = (\n    DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"}\n).fetch1()\n\nContFragClusterlessClassifier(**model_params[\"decoding_params\"])\n</pre> model_params = (     DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"} ).fetch1()  ContFragClusterlessClassifier(**model_params[\"decoding_params\"]) Out[13]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=[Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0)],\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=[Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0)],\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> In\u00a0[14]: Copied! <pre>from non_local_detector.environment import Environment\n\n?Environment\n</pre> from non_local_detector.environment import Environment  ?Environment <pre>Init signature:\nEnvironment(\n    environment_name: str = '',\n    place_bin_size: Union[float, Tuple[float]] = 2.0,\n    track_graph: Optional[networkx.classes.graph.Graph] = None,\n    edge_order: Optional[tuple] = None,\n    edge_spacing: Optional[tuple] = None,\n    is_track_interior: Optional[numpy.ndarray] = None,\n    position_range: Optional[numpy.ndarray] = None,\n    infer_track_interior: bool = True,\n    fill_holes: bool = False,\n    dilate: bool = False,\n    bin_count_threshold: int = 0,\n) -&gt; None\nDocstring:     \nRepresent the spatial environment with a discrete grid.\n\nParameters\n----------\nenvironment_name : str, optional\nplace_bin_size : float, optional\n    Approximate size of the position bins.\ntrack_graph : networkx.Graph, optional\n    Graph representing the 1D spatial topology\nedge_order : tuple of 2-tuples, optional\n    The order of the edges in 1D space\nedge_spacing : None or int or tuples of len n_edges-1, optional\n    Any gapes between the edges in 1D space\nis_track_interior : np.ndarray or None, optional\n    If given, this will be used to define the valid areas of the track.\n    Must be of type boolean.\nposition_range : sequence, optional\n    A sequence of `n_position_dims`, each an optional (lower, upper)\n    tuple giving the outer bin edges for position.\n    An entry of None in the sequence results in the minimum and maximum\n    values being used for the corresponding dimension.\n    The default, None, is equivalent to passing a tuple of\n    `n_position_dims` None values.\ninfer_track_interior : bool, optional\n    If True, then use the given positions to figure out the valid track\n    areas.\nfill_holes : bool, optional\n    Fill holes when inferring the track\ndilate : bool, optional\n    Inflate the available track area with binary dilation\nbin_count_threshold : int, optional\n    Greater than this number of samples should be in the bin for it to\n    be considered on the track.\nFile:           ~/miniconda3/envs/spyglass/lib/python3.9/site-packages/non_local_detector/environment.py\nType:           type\nSubclasses:     </pre> In\u00a0[15]: Copied! <pre>from spyglass.decoding.v1.clusterless import ClusterlessDecodingSelection\n\nClusterlessDecodingSelection()\n</pre> from spyglass.decoding.v1.clusterless import ClusterlessDecodingSelection  ClusterlessDecodingSelection() Out[15]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters <p>Total: 0</p> In\u00a0[16]: Copied! <pre>from spyglass.common import IntervalList\n\nIntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> from spyglass.common import IntervalList  IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[16]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') mediumnwb20230802_.nwb 02_r1 =BLOB= mediumnwb20230802_.nwb 04f3ecb4-a18c-4ffb-85d8-2f5f62d4d6d4 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 0e848c38-9105-4ea4-b6ba-dbdd5b46a088 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 0f91197e-bebb-4dc6-ad41-5bf89c3eed28 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 15c8a3e8-5ce9-4654-891e-6ee4109d6f1a =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 1d2b5966-415a-4c65-955a-0e422d8b5b00 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 1e3f3707-613e-4a44-93f1-c7e5484112cd =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 2402805a-04f9-4a88-9ccf-071376c8de19 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 24107d8c-ce26-4c77-8f6a-bf6955d8a3c7 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 257c077b-8f3b-4abb-a631-6b8084d6a1ea =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 2b93bcd0-7b05-457c-8aab-c41ef543ecf2 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 2b9fbf14-74a0-4294-a805-26702340aac9 =BLOB= spikesorting_artifact_v1 <p>...</p> <p>Total: 52</p> In\u00a0[17]: Copied! <pre>decoding_interval_valid_times = [\n    [1625935714.6359036, 1625935714.6359036 + 15.0]\n]\n\nIntervalList.insert1(\n    {\n        \"nwb_file_name\": \"mediumnwb20230802_.nwb\",\n        \"interval_list_name\": \"test decoding interval\",\n        \"valid_times\": decoding_interval_valid_times,\n    },\n    skip_duplicates=True,\n)\n</pre> decoding_interval_valid_times = [     [1625935714.6359036, 1625935714.6359036 + 15.0] ]  IntervalList.insert1(     {         \"nwb_file_name\": \"mediumnwb20230802_.nwb\",         \"interval_list_name\": \"test decoding interval\",         \"valid_times\": decoding_interval_valid_times,     },     skip_duplicates=True, ) <p>Once we have figured out the keys that we need, we can insert the <code>ClusterlessDecodingSelection</code> into the database.</p> In\u00a0[18]: Copied! <pre>selection_key = {\n    \"waveform_features_group_name\": \"test_group\",\n    \"position_group_name\": \"test_group\",\n    \"decoding_param_name\": \"contfrag_clusterless\",\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"encoding_interval\": \"pos 0 valid times\",\n    \"decoding_interval\": \"test decoding interval\",\n    \"estimate_decoding_params\": False,\n}\n\nClusterlessDecodingSelection.insert1(\n    selection_key,\n    skip_duplicates=True,\n)\n\nClusterlessDecodingSelection &amp; selection_key\n</pre> selection_key = {     \"waveform_features_group_name\": \"test_group\",     \"position_group_name\": \"test_group\",     \"decoding_param_name\": \"contfrag_clusterless\",     \"nwb_file_name\": nwb_copy_file_name,     \"encoding_interval\": \"pos 0 valid times\",     \"decoding_interval\": \"test decoding interval\",     \"estimate_decoding_params\": False, }  ClusterlessDecodingSelection.insert1(     selection_key,     skip_duplicates=True, )  ClusterlessDecodingSelection &amp; selection_key Out[18]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> In\u00a0[19]: Copied! <pre>ClusterlessDecodingSelection()\n</pre> ClusterlessDecodingSelection() Out[19]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>To run decoding, we simply populate the <code>ClusterlessDecodingOutput</code> table. This will run the decoding and insert the results into the database. We can then retrieve the results from the database.</p> In\u00a0[20]: Copied! <pre>from spyglass.decoding.v1.clusterless import ClusterlessDecodingV1\n\nClusterlessDecodingV1.populate(selection_key)\n</pre> from spyglass.decoding.v1.clusterless import ClusterlessDecodingV1  ClusterlessDecodingV1.populate(selection_key) <pre>[10:24:17][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[2024-01-29 10:24:17,234][WARNING]: Skipped checksum for file with hash: 0cd40383-03e0-44ec-5dac-36c66063796a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_FUSH604NQA.nwb\n[2024-01-29 10:24:17,409][WARNING]: Skipped checksum for file with hash: a7c9b1d9-d1a2-7f40-9127-206e83a87006, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_NQEPSMKPK0.nwb\n[2024-01-29 10:24:17,411][WARNING]: Skipped checksum for file with hash: ec7faa5b-3847-6649-1a93-74ebd50dcfb9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_F02UG5Z5FR.nwb\n[2024-01-29 10:24:17,413][WARNING]: Skipped checksum for file with hash: 8e964932-96ab-e1c9-2133-edce8eacab5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OTV91MLKDT.nwb\n[2024-01-29 10:24:17,415][WARNING]: Skipped checksum for file with hash: 895bac7b-bfd6-b4f2-b2ad-460362aaafa8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TSPNTCGNN1.nwb\n[2024-01-29 10:24:17,417][WARNING]: Skipped checksum for file with hash: 58713583-cf49-4527-7707-105f9c9ee477, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_QSK70WFDJH.nwb\n[2024-01-29 10:24:17,419][WARNING]: Skipped checksum for file with hash: a64829f8-ab12-fecc-eda9-a22b90b20d43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DO45HKXYTB.nwb\n[2024-01-29 10:24:17,420][WARNING]: Skipped checksum for file with hash: 3a580271-9126-8e57-048e-a7bbb3f917b9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_KFIYRJ4HFO.nwb\n[2024-01-29 10:24:17,423][WARNING]: Skipped checksum for file with hash: 13cf8ad9-023c-c9b7-05c3-eaa3330304f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0YIM5K3H47.nwb\n[2024-01-29 10:24:17,425][WARNING]: Skipped checksum for file with hash: 7ce8a640-0a25-4866-6d5a-aa2c65f0aca5, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_CTLEGE2TWZ.nwb\n[2024-01-29 10:24:17,427][WARNING]: Skipped checksum for file with hash: aa657f4f-f409-d444-8b32-31d37abe0797, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7EN0N1U4U1.nwb\n[2024-01-29 10:24:17,429][WARNING]: Skipped checksum for file with hash: f3b4bd22-1439-e6d2-4e15-aa3650143fdf, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DHKWBWWAMC.nwb\n[2024-01-29 10:24:17,430][WARNING]: Skipped checksum for file with hash: 68eac0b2-e5be-e0c5-9eae-cd8dbe6676a8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_PEN0D79Q0B.nwb\n[2024-01-29 10:24:17,432][WARNING]: Skipped checksum for file with hash: c8b95099-2cb3-df0b-5ab1-7a5e120a8e2f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_WP7SIXDJ2A.nwb\n[2024-01-29 10:24:17,434][WARNING]: Skipped checksum for file with hash: 8fae8089-f683-5f0a-4e59-c71d6ee14f38, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_B82OS6W1QA.nwb\n[2024-01-29 10:24:17,437][WARNING]: Skipped checksum for file with hash: dd9d0f51-6445-b368-32bd-b1f142bf6ed3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_XO17FQLN6T.nwb\n[2024-01-29 10:24:17,439][WARNING]: Skipped checksum for file with hash: 4e2cf5f5-ff7c-1a2b-db85-2d1c4f036fbd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OCFI0GFLZ9.nwb\n[2024-01-29 10:24:17,441][WARNING]: Skipped checksum for file with hash: 8691c252-0bd1-122b-8cf3-b89c4d0fdee0, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_60M9VSZX0W.nwb\n[2024-01-29 10:24:17,443][WARNING]: Skipped checksum for file with hash: 57b89835-8edb-e91d-0798-09d22fb4fbc9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Z5HJ68LHYW.nwb\n[2024-01-29 10:24:17,445][WARNING]: Skipped checksum for file with hash: 54401121-4426-86c9-72f7-e056bc16e99d, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_U5U5JVGY4F.nwb\n[2024-01-29 10:24:17,447][WARNING]: Skipped checksum for file with hash: 0ff21e84-2214-6911-2575-a9c92a541407, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0D5Z0NSIP8.nwb\n[2024-01-29 10:24:17,449][WARNING]: Skipped checksum for file with hash: 0949b006-5309-93c8-fd8b-1308e8130869, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EYV2NARUKU.nwb\n[2024-01-29 10:24:17,451][WARNING]: Skipped checksum for file with hash: b4b31e50-dfa2-0d02-514a-525782a81255, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T4XBCIW44T.nwb\n[2024-01-29 10:24:17,453][WARNING]: Skipped checksum for file with hash: c18a9ac4-06bc-4249-2bad-439d4f618421, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_UD55CR8LZK.nwb\n</pre> <pre>Encoding models:   0%|          | 0/23 [00:00&lt;?, ?electrode/s]</pre> <pre>Non-Local Likelihood:   0%|          | 0/23 [00:00&lt;?, ?electrode/s]</pre> <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/non_local_detector/models/base.py:780: FutureWarning: the `pandas.MultiIndex` object(s) passed as 'state_bins' coordinate(s) or data variable(s) will no longer be implicitly promoted and wrapped into multiple indexed coordinates in the future (i.e., one coordinate for each multi-index level + one dimension coordinate). If you want to keep this behavior, you need to first wrap it explicitly using `mindex_coords = xarray.Coordinates.from_pandas_multiindex(mindex_obj, 'dim')` and pass it as coordinates, e.g., `xarray.Dataset(coords=mindex_coords)`, `dataset.assign_coords(mindex_coords)` or `dataarray.assign_coords(mindex_coords)`.\n  results = xr.Dataset(\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n</pre> <p>We can now see it as an entry in the <code>DecodingOutput</code> table.</p> In\u00a0[21]: Copied! <pre>from spyglass.decoding.decoding_merge import DecodingOutput\n\nDecodingOutput.ClusterlessDecodingV1 &amp; selection_key\n</pre> from spyglass.decoding.decoding_merge import DecodingOutput  DecodingOutput.ClusterlessDecodingV1 &amp; selection_key Out[21]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters b63395dd-402e-270a-a8d1-7aabaf83d452 mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>We can load the results of the decoding:</p> In\u00a0[22]: Copied! <pre>decoding_results = (ClusterlessDecodingV1 &amp; selection_key).fetch_results()\ndecoding_results\n</pre> decoding_results = (ClusterlessDecodingV1 &amp; selection_key).fetch_results() decoding_results <pre>[2024-01-29 10:26:49,467][WARNING]: Skipped checksum for file with hash: 10c77056-5508-ace0-bd84-5a4d7497f7a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.nc\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n</pre> Out[22]: <pre>&lt;xarray.Dataset&gt;\nDimensions:                      (state_ind: 26668, dim_0: 26668, time: 3750,\n                                  states: 2, intervals: 1, state_bins: 26668)\nCoordinates:\n  * state_ind                    (state_ind) int32 0 0 0 0 0 0 0 ... 1 1 1 1 1 1\n  * time                         (time) float64 1.626e+09 ... 1.626e+09\n  * states                       (states) object 'Continuous' 'Fragmented'\n    environments                 (states) object ...\n    encoding_groups              (states) int32 ...\n  * state_bins                   (state_bins) object MultiIndex\n  * state                        (state_bins) object 'Continuous' ... 'Fragme...\n  * x_position                   (state_bins) float64 29.02 29.02 ... 262.7\n  * y_position                   (state_bins) float64 0.5211 2.516 ... 224.0\nDimensions without coordinates: dim_0, intervals\nData variables:\n    initial_conditions           (dim_0) float64 ...\n    discrete_state_transitions   (states, states) float64 ...\n    acausal_posterior            (intervals, time, state_bins) float32 ...\n    acausal_state_probabilities  (intervals, time, states) float64 ...\nAttributes:\n    marginal_log_likelihoods:  -159596.89</pre>xarray.Dataset<ul><li>Dimensions:<ul><li>state_ind: 26668</li><li>dim_0: 26668</li><li>time: 3750</li><li>states: 2</li><li>intervals: 1</li><li>state_bins: 26668</li></ul></li><li>Coordinates: (9)<ul><li>state_ind(state_ind)int320 0 0 0 0 0 0 0 ... 1 1 1 1 1 1 1 1<pre>array([0, 0, 0, ..., 1, 1, 1], dtype=int32)</pre></li><li>time(time)float641.626e+09 1.626e+09 ... 1.626e+09<pre>array([1.625936e+09, 1.625936e+09, 1.625936e+09, ..., 1.625936e+09,\n       1.625936e+09, 1.625936e+09])</pre></li><li>states(states)object'Continuous' 'Fragmented'<pre>array(['Continuous', 'Fragmented'], dtype=object)</pre></li><li>environments(states)object...<pre>[2 values with dtype=object]</pre></li><li>encoding_groups(states)int32...<pre>[2 values with dtype=int32]</pre></li><li>state_bins(state_bins)objectMultiIndex<pre>array([('Continuous', 29.022380454030234, 0.5210949074118973),\n       ('Continuous', 29.022380454030234, 2.5161439814859774),\n       ('Continuous', 29.022380454030234, 4.5111930555600575), ...,\n       ('Fragmented', 262.70451087811205, 219.9764930555607),\n       ('Fragmented', 262.70451087811205, 221.97154212963477),\n       ('Fragmented', 262.70451087811205, 223.96659120370884)], dtype=object)</pre></li><li>state(state_bins)object'Continuous' ... 'Fragmented'<pre>array(['Continuous', 'Continuous', 'Continuous', ..., 'Fragmented',\n       'Fragmented', 'Fragmented'], dtype=object)</pre></li><li>x_position(state_bins)float6429.02 29.02 29.02 ... 262.7 262.7<pre>array([ 29.02238 ,  29.02238 ,  29.02238 , ..., 262.704511, 262.704511,\n       262.704511])</pre></li><li>y_position(state_bins)float640.5211 2.516 4.511 ... 222.0 224.0<pre>array([  0.521095,   2.516144,   4.511193, ..., 219.976493, 221.971542,\n       223.966591])</pre></li></ul></li><li>Data variables: (4)<ul><li>initial_conditions(dim_0)float64...<pre>[26668 values with dtype=float64]</pre></li><li>discrete_state_transitions(states, states)float64...<pre>[4 values with dtype=float64]</pre></li><li>acausal_posterior(intervals, time, state_bins)float32...<pre>[100005000 values with dtype=float32]</pre></li><li>acausal_state_probabilities(intervals, time, states)float64...<pre>[7500 values with dtype=float64]</pre></li></ul></li><li>Indexes: (4)<ul><li>state_indPandasIndex<pre>PandasIndex(Index([0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n       ...\n       1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n      dtype='int32', name='state_ind', length=26668))</pre></li><li>timePandasIndex<pre>PandasIndex(Index([1625935714.6373355, 1625935714.6413357, 1625935714.6453357,\n       1625935714.6493356, 1625935714.6533356, 1625935714.6573355,\n       1625935714.6613357, 1625935714.6653357, 1625935714.6693356,\n       1625935714.6733356,\n       ...\n       1625935729.5973077, 1625935729.6013076, 1625935729.6053078,\n       1625935729.6093078, 1625935729.6133077, 1625935729.6173077,\n       1625935729.6213076, 1625935729.6253078, 1625935729.6293077,\n       1625935729.6333077],\n      dtype='float64', name='time', length=3750))</pre></li><li>statesPandasIndex<pre>PandasIndex(Index(['Continuous', 'Fragmented'], dtype='object', name='states'))</pre></li><li>state_binsstatex_positiony_positionPandasMultiIndex<pre>PandasIndex(MultiIndex([('Continuous', 29.022380454030234, 0.5210949074118973),\n            ('Continuous', 29.022380454030234, 2.5161439814859774),\n            ('Continuous', 29.022380454030234, 4.5111930555600575),\n            ('Continuous', 29.022380454030234,  6.506242129634137),\n            ('Continuous', 29.022380454030234,  8.501291203708217),\n            ('Continuous', 29.022380454030234, 10.496340277782297),\n            ('Continuous', 29.022380454030234, 12.491389351856377),\n            ('Continuous', 29.022380454030234, 14.486438425930455),\n            ('Continuous', 29.022380454030234, 16.481487500004537),\n            ('Continuous', 29.022380454030234, 18.476536574078615),\n            ...\n            ('Fragmented', 262.70451087811205, 206.01114953704212),\n            ('Fragmented', 262.70451087811205, 208.00619861111622),\n            ('Fragmented', 262.70451087811205,  210.0012476851903),\n            ('Fragmented', 262.70451087811205, 211.99629675926437),\n            ('Fragmented', 262.70451087811205, 213.99134583333844),\n            ('Fragmented', 262.70451087811205, 215.98639490741252),\n            ('Fragmented', 262.70451087811205, 217.98144398148662),\n            ('Fragmented', 262.70451087811205,  219.9764930555607),\n            ('Fragmented', 262.70451087811205, 221.97154212963477),\n            ('Fragmented', 262.70451087811205, 223.96659120370884)],\n           name='state_bins', length=26668))</pre></li></ul></li><li>Attributes: (1)marginal_log_likelihoods :-159596.89</li></ul> <p>Finally, if we deleted the results, we can use the <code>cleanup</code> function to delete the results from the file system:</p> In\u00a0[23]: Copied! <pre>DecodingOutput().cleanup()\n</pre> DecodingOutput().cleanup() <pre>[10:26:49][INFO] Spyglass: Cleaning up decoding outputs\n[2024-01-29 10:26:49,699][WARNING]: Skipped checksum for file with hash: 10c77056-5508-ace0-bd84-5a4d7497f7a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.nc\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_259e8498-1eb6-4e84-a0f6-7575c4ab9b87.nc\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_8404dc49-081c-48c7-b448-34767512e8ed.nc\n[2024-01-29 10:26:49,774][WARNING]: Skipped checksum for file with hash: c4a577dc-6f11-bd71-cc5f-e131c6eaa39f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.pkl\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_259e8498-1eb6-4e84-a0f6-7575c4ab9b87.pkl\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_8404dc49-081c-48c7-b448-34767512e8ed.pkl\n</pre> In\u00a0[24]: Copied! <pre># from non_local_detector.visualization import (\n#     create_interactive_2D_decoding_figurl,\n# )\n\n# (\n#     position_info,\n#     position_variable_names,\n# ) = ClusterlessDecodingV1.fetch_position_info(selection_key)\n# results_time = decoding_results.acausal_posterior.isel(intervals=0).time.values\n# position_info = position_info.loc[results_time[0] : results_time[-1]]\n\n# env = ClusterlessDecodingV1.fetch_environments(selection_key)[0]\n# spike_times, _ = ClusterlessDecodingV1.fetch_spike_data(selection_key)\n\n\n# create_interactive_2D_decoding_figurl(\n#     position_time=position_info.index.to_numpy(),\n#     position=position_info[position_variable_names],\n#     env=env,\n#     results=decoding_results,\n#     posterior=decoding_results.acausal_posterior.isel(intervals=0)\n#     .unstack(\"state_bins\")\n#     .sum(\"state\"),\n#     spike_times=spike_times,\n#     head_dir=position_info[\"orientation\"],\n#     speed=position_info[\"speed\"],\n# )\n</pre> # from non_local_detector.visualization import ( #     create_interactive_2D_decoding_figurl, # )  # ( #     position_info, #     position_variable_names, # ) = ClusterlessDecodingV1.fetch_position_info(selection_key) # results_time = decoding_results.acausal_posterior.isel(intervals=0).time.values # position_info = position_info.loc[results_time[0] : results_time[-1]]  # env = ClusterlessDecodingV1.fetch_environments(selection_key)[0] # spike_times, _ = ClusterlessDecodingV1.fetch_spike_data(selection_key)   # create_interactive_2D_decoding_figurl( #     position_time=position_info.index.to_numpy(), #     position=position_info[position_variable_names], #     env=env, #     results=decoding_results, #     posterior=decoding_results.acausal_posterior.isel(intervals=0) #     .unstack(\"state_bins\") #     .sum(\"state\"), #     spike_times=spike_times, #     head_dir=position_info[\"orientation\"], #     speed=position_info[\"speed\"], # ) In\u00a0[25]: Copied! <pre>import jax\n\njax.devices()\n</pre> import jax  jax.devices() Out[25]: <pre>[CpuDevice(id=0)]</pre> In\u00a0[26]: Copied! <pre># device_id = 2\n# device = jax.devices()[device_id]\n# jax.config.update(\"jax_default_device\", device)\n# device\n</pre> # device_id = 2 # device = jax.devices()[device_id] # jax.config.update(\"jax_default_device\", device) # device"}, {"location": "notebooks/41_Decoding_Clusterless/#clusterless-decoding", "title": "Clusterless Decoding\u00b6", "text": ""}, {"location": "notebooks/41_Decoding_Clusterless/#overview", "title": "Overview\u00b6", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>This tutorial assumes you've already  extracted waveforms, as well as loaded  position data. If 1D decoding, this data should also be linearized.</li> </ul> <p>Clusterless decoding can be performed on either 1D or 2D data. We will start with 2D data.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#elements-of-clusterless-decoding", "title": "Elements of Clusterless Decoding\u00b6", "text": "<ul> <li>Position Data: This is the data that we want to decode. It can be 1D or 2D.</li> <li>Spike Waveform Features: These are the features that we will use to decode the position data.</li> <li>Decoding Model Parameters: This is how we define the model that we will use to decode the position data.</li> </ul>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-data", "title": "Grouping Data\u00b6", "text": "<p>An important concept will be groups. Groups are tables that allow use to specify collections of data. We will use groups in two situations here:</p> <ol> <li>Because we want to decode from more than one tetrode (or probe), so we will create a group that contains all of the tetrodes that we want to decode from.</li> <li>Similarly, we will create a group for the position data that we want to decode, so that we can decode from position data from multiple sessions.</li> </ol>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-waveform-features", "title": "Grouping Waveform Features\u00b6", "text": "<p>Let's start with grouping the Waveform Features. We will first inspect the waveform features that we have extracted to figure out the primary keys of the data that we want to decode from. We need to use the tables <code>SpikeSortingSelection</code> and <code>SpikeSortingOutput</code> to figure out the <code>merge_id</code> associated with <code>nwb_file_name</code> to get the waveform features associated with the NWB file of interest.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-position-data", "title": "Grouping Position Data\u00b6", "text": "<p>We will now create a group called <code>02_r1</code> that contains all of the position data that we want to decode from. As before, we will use the <code>create_group</code> function to create this group. This function takes two arguments: the name of the group, and the keys of the tables that we want to include in the group.</p> <p>We use the the <code>PositionOutput</code> table to figure out the <code>merge_id</code> associated with <code>nwb_file_name</code> to get the position data associated with the NWB file of interest. In this case, we only have one position to insert, but we could insert multiple positions if we wanted to decode from multiple sessions.</p> <p>Note that the position data sampling frequency is what determines the time step of the decoding. In this case, the position data sampling frequency is 30 Hz, so the time step of the decoding will be 1/30 seconds. In practice, you will want to use a smaller time step such as 500 Hz. This will allow you to decode at a finer time scale. To do this, you will want to interpolate the position data to a higher sampling frequency as shown in the position trodes notebook.</p> <p>You will also want to specify the name of the position variables if they are different from the default names. The default names are <code>position_x</code> and <code>position_y</code>.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding-model-parameters", "title": "Decoding Model Parameters\u00b6", "text": "<p>We will use the <code>non_local_detector</code> package to decode the data. This package is highly flexible and allows several different types of models to be used. In this case, we will use the <code>ContFragClusterlessClassifier</code> to decode the data. This has two discrete states: Continuous and Fragmented, which correspond to different types of movement models. To read more about this model, see:</p> <p>Denovellis, E.L., Gillespie, A.K., Coulter, M.E., Sosa, M., Chung, J.E., Eden, U.T., and Frank, L.M. (2021). Hippocampal replay of experience at real-world speeds. eLife 10, e64505. 10.7554/eLife.64505.</p> <p>Let's first look at the model and the default parameters:</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#1d-decoding", "title": "1D Decoding\u00b6", "text": "<p>If you want to do 1D decoding, you will need to specify the <code>track_graph</code>, <code>edge_order</code>, and <code>edge_spacing</code> in the <code>environments</code> parameter. You can read more about these parameters in the linearization notebook. You can retrieve these parameters from the <code>TrackGraph</code> table if you have stored them there. These will then go into the <code>environments</code> parameter of the <code>ContFragClusterlessClassifier</code> model.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding", "title": "Decoding\u00b6", "text": "<p>Now that we have grouped the data and defined the model parameters, we have finally set up the elements in tables that we need to decode the data. We now need to use the <code>ClusterlessDecodingSelection</code> to fully specify all the parameters and data that we want.</p> <p>This has:</p> <ul> <li><code>waveform_features_group_name</code>: the name of the group that contains the waveform features that we want to decode from</li> <li><code>position_group_name</code>: the name of the group that contains the position data that we want to decode from</li> <li><code>decoding_param_name</code>: the name of the decoding parameters that we want to use</li> <li><code>nwb_file_name</code>: the name of the NWB file that we want to decode from</li> <li><code>encoding_interval</code>: the interval of time that we want to train the initial model on</li> <li><code>decoding_interval</code>: the interval of time that we want to decode from</li> <li><code>estimate_decoding_params</code>: whether or not we want to estimate the decoding parameters</li> </ul> <p>The first three parameters should be familiar to you.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding-and-encoding-intervals", "title": "Decoding and Encoding Intervals\u00b6", "text": "<p>The <code>encoding_interval</code> is the interval of time that we want to train the initial model on. The <code>decoding_interval</code> is the interval of time that we want to decode from. These two intervals can be the same, but they do not have to be. For example, we may want to train the model on a long interval of time, but only decode from a short interval of time. This is useful if we want to decode from a short interval of time that is not representative of the entire session. In this case, we will train the model on a longer interval of time that is representative of the entire session.</p> <p>These keys come from the <code>IntervalList</code> table. We can see that the <code>IntervalList</code> table contains the <code>nwb_file_name</code> and <code>interval_name</code> that we need to specify the <code>encoding_interval</code> and <code>decoding_interval</code>. We will specify a short decoding interval called <code>test decoding interval</code> and use that to decode from.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#estimating-decoding-parameters", "title": "Estimating Decoding Parameters\u00b6", "text": "<p>The last parameter is <code>estimate_decoding_params</code>. This is a boolean that specifies whether or not we want to estimate the decoding parameters. If this is <code>True</code>, then we will estimate the initial conditions and discrete transition matrix from the data.</p> <p>NOTE: If estimating parameters, then we need to treat times outside decoding interval as missing. this means that times outside the decoding interval will not use the spiking data and only the state transition matrix and previous time step will be used. This may or may not be desired depending on the length of this missing interval.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#visualization-of-decoding-output", "title": "Visualization of decoding output.\u00b6", "text": "<p>The output of decoding can be challenging to visualize with static graphs, especially if the decoding is performed on 2D data.</p> <p>We can interactively visualize the output of decoding using the figurl package. This package allows to create a visualization of the decoding output that can be viewed in a web browser. This is useful for exploring the decoding output over time and sharing the results with others.</p> <p>NOTE: You will need a kachery cloud instance to use this feature. If you are a member of the Frank lab, you should have access to the Frank lab kachery cloud instance. If you are not a member of the Frank lab, you can create your own kachery cloud instance by following the instructions here.</p> <p>For each user, you will need to run <code>kachery-cloud-init</code> in the terminal and follow the instructions to associate your computer with your GitHub user on the kachery-cloud network.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#gpus", "title": "GPUs\u00b6", "text": "<p>We can use GPUs for decoding which will result in a significant speedup. This is achieved using the jax package.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#ensuring-jax-can-find-a-gpu", "title": "Ensuring jax can find a GPU\u00b6", "text": "<p>Assuming you've set up a GPU, we can use <code>jax.devices()</code> to make sure the decoding code can see the GPU. If a GPU is available, it will be listed.</p> <p>In the following instance, we do not have a GPU:</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#selecting-a-gpu", "title": "Selecting a GPU\u00b6", "text": "<p>If you do have multiple GPUs, you can use the <code>jax</code> package to set the device (GPU) that you want to use. For example, if you want to use the second GPU, you can use the following code (uncomment first):</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#monitoring-gpu-usage", "title": "Monitoring GPU Usage\u00b6", "text": "<p>You can see which GPUs are occupied (if you have multiple GPUs) by running the command <code>nvidia-smi</code> in a terminal (or <code>!nvidia-smi</code> in a notebook). Pick a GPU with low memory usage.</p> <p>We can monitor GPU use with the terminal command <code>watch -n 0.1 nvidia-smi</code>, will update <code>nvidia-smi</code> every 100 ms. This won't work in a notebook, as it won't display the updates.</p> <p>Other ways to monitor GPU usage are:</p> <ul> <li>A  jupyter widget by nvidia to monitor GPU usage in the notebook</li> <li>A terminal program like nvidia-smi with more information about  which GPUs are being utilized and by whom.</li> </ul>"}, {"location": "notebooks/42_Decoding_SortedSpikes/", "title": "Decoding Sorted Spikes", "text": "In\u00a0[9]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info In\u00a0[1]: Copied! <pre>from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams\n\nUnitSelectionParams().insert_default()\n\n# look at the filter set we'll use here\nunit_filter_params_name = \"default_exclusion\"\nprint(\n    (\n        UnitSelectionParams()\n        &amp; {\"unit_filter_params_name\": unit_filter_params_name}\n    ).fetch1()\n)\n# look at full table\nUnitSelectionParams()\n</pre> from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams  UnitSelectionParams().insert_default()  # look at the filter set we'll use here unit_filter_params_name = \"default_exclusion\" print(     (         UnitSelectionParams()         &amp; {\"unit_filter_params_name\": unit_filter_params_name}     ).fetch1() ) # look at full table UnitSelectionParams() <pre>[2024-02-02 12:06:04,725][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-02-02 12:06:04,762][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n[12:06:05][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> <pre>{'unit_filter_params_name': 'default_exclusion', 'include_labels': [], 'exclude_labels': ['noise', 'mua']}\n</pre> Out[1]: <p>unit_filter_params_name</p> <p>include_labels</p> <p>exclude_labels</p> all_units =BLOB= =BLOB=default_exclusion =BLOB= =BLOB=exclude_noise =BLOB= =BLOB= <p>Total: 3</p> <p>Now we can make our sorted spikes group with this unit selection parameter</p> In\u00a0[2]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\nimport spyglass.spikesorting.v1 as sgs\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\nsorter_keys = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorter\": \"mountainsort4\",\n    \"curation_id\": 1,\n}\n# check the set of sorting we'll use\n(sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput import spyglass.spikesorting.v1 as sgs  nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  sorter_keys = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorter\": \"mountainsort4\",     \"curation_id\": 1, } # check the set of sorting we'll use (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 Out[2]: <p>sorting_id</p> <p>merge_id</p> <p>recording_id</p> <p>sorter</p> <p>sorter_param_name</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>curation_id</p> 642242ff-5f0e-45a2-bcc1-ca681f37b4a3 75286bf3-f876-4550-f235-321f2a7badef 01c5b8e9-933d-4f1e-9a5d-c494276edb3a mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 0a6611b3-c593-4900-a715-66bb1396940e 1a4b5a94d-ba41-4634-92d0-1d31c9daa913 143dff79-3779-c0d2-46fe-7c5040404219 a8a1d29d-ffdf-4370-8b3d-909fef57f9d4 mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 3d782852-a56b-4a9d-89ca-be9e1a15c957 1874775be-df0f-4850-8f88-59ba1bbead89 a900c1c8-909d-e583-c377-e98c4f0deebf 747f4eea-6df3-422b-941e-b5aaad7ec607 mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 9cf9e3cd-7115-4b59-a718-3633725d4738 1 <p>Total: 3</p> In\u00a0[6]: Copied! <pre>from spyglass.decoding.v1.sorted_spikes import SortedSpikesGroup\n\nSortedSpikesGroup()\n</pre> from spyglass.decoding.v1.sorted_spikes import SortedSpikesGroup  SortedSpikesGroup() Out[6]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb all_units test_group <p>Total: 1</p> In\u00a0[7]: Copied! <pre># get the merge_ids for the selected sorting\nspikesorting_merge_ids = (\n    (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1\n).fetch(\"merge_id\")\n\n# create a new sorted spikes group\nunit_filter_params_name = \"default_exclusion\"\nSortedSpikesGroup().create_group(\n    group_name=\"test_group\",\n    nwb_file_name=nwb_copy_file_name,\n    keys=[\n        {\"spikesorting_merge_id\": merge_id}\n        for merge_id in spikesorting_merge_ids\n    ],\n    unit_filter_params_name=unit_filter_params_name,\n)\n# check the new group\nSortedSpikesGroup &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n}\n</pre> # get the merge_ids for the selected sorting spikesorting_merge_ids = (     (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 ).fetch(\"merge_id\")  # create a new sorted spikes group unit_filter_params_name = \"default_exclusion\" SortedSpikesGroup().create_group(     group_name=\"test_group\",     nwb_file_name=nwb_copy_file_name,     keys=[         {\"spikesorting_merge_id\": merge_id}         for merge_id in spikesorting_merge_ids     ],     unit_filter_params_name=unit_filter_params_name, ) # check the new group SortedSpikesGroup &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\", } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb all_units test_groupmediumnwb20230802_.nwb default_exclusion test_group <p>Total: 2</p> In\u00a0[8]: Copied! <pre># look at the sorting within the group we just made\nSortedSpikesGroup.Units &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": unit_filter_params_name,\n}\n</pre> # look at the sorting within the group we just made SortedSpikesGroup.Units &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": unit_filter_params_name, } Out[8]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>spikesorting_merge_id</p> mediumnwb20230802_.nwb default_exclusion test_group 143dff79-3779-c0d2-46fe-7c5040404219mediumnwb20230802_.nwb default_exclusion test_group 75286bf3-f876-4550-f235-321f2a7badefmediumnwb20230802_.nwb default_exclusion test_group a900c1c8-909d-e583-c377-e98c4f0deebf <p>Total: 3</p> In\u00a0[9]: Copied! <pre>from spyglass.decoding.v1.core import DecodingParameters\nfrom non_local_detector.models import ContFragSortedSpikesClassifier\n\n\nDecodingParameters.insert1(\n    {\n        \"decoding_param_name\": \"contfrag_sorted\",\n        \"decoding_params\": ContFragSortedSpikesClassifier(),\n        \"decoding_kwargs\": dict(),\n    },\n    skip_duplicates=True,\n)\n\nDecodingParameters()\n</pre> from spyglass.decoding.v1.core import DecodingParameters from non_local_detector.models import ContFragSortedSpikesClassifier   DecodingParameters.insert1(     {         \"decoding_param_name\": \"contfrag_sorted\",         \"decoding_params\": ContFragSortedSpikesClassifier(),         \"decoding_kwargs\": dict(),     },     skip_duplicates=True, )  DecodingParameters() Out[9]: <p>decoding_param_name</p> a name for this set of parameters <p>decoding_params</p> initialization parameters for model <p>decoding_kwargs</p> additional keyword arguments contfrag_clusterless =BLOB= =BLOB=contfrag_clusterless_0.5.13 =BLOB= =BLOB=contfrag_clusterless_6track =BLOB= =BLOB=contfrag_sorted =BLOB= =BLOB=contfrag_sorted_0.5.13 =BLOB= =BLOB=j1620210710_contfrag_clusterless_1D =BLOB= =BLOB=j1620210710_test_contfrag_clusterless =BLOB= =BLOB=MS2220180629_contfrag_sorted =BLOB= =BLOB=ms_lineartrack_2023_contfrag_sorted =BLOB= =BLOB=ms_lineartrack_contfrag_clusterless =BLOB= =BLOB=ms_lineartrack_contfrag_sorted =BLOB= =BLOB=ms_wtrack_2023_contfrag_sorted =BLOB= =BLOB= <p>...</p> <p>Total: 15</p> In\u00a0[2]: Copied! <pre>selection_key = {\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": \"default_exclusion\",\n    \"position_group_name\": \"test_group\",\n    \"decoding_param_name\": \"contfrag_sorted\",\n    \"nwb_file_name\": \"mediumnwb20230802_.nwb\",\n    \"encoding_interval\": \"pos 0 valid times\",\n    \"decoding_interval\": \"test decoding interval\",\n    \"estimate_decoding_params\": False,\n}\n\nfrom spyglass.decoding import SortedSpikesDecodingSelection\n\nSortedSpikesDecodingSelection.insert1(\n    selection_key,\n    skip_duplicates=True,\n)\n</pre> selection_key = {     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": \"default_exclusion\",     \"position_group_name\": \"test_group\",     \"decoding_param_name\": \"contfrag_sorted\",     \"nwb_file_name\": \"mediumnwb20230802_.nwb\",     \"encoding_interval\": \"pos 0 valid times\",     \"decoding_interval\": \"test decoding interval\",     \"estimate_decoding_params\": False, }  from spyglass.decoding import SortedSpikesDecodingSelection  SortedSpikesDecodingSelection.insert1(     selection_key,     skip_duplicates=True, ) In\u00a0[14]: Copied! <pre>from spyglass.decoding.v1.sorted_spikes import SortedSpikesDecodingV1\n\nSortedSpikesDecodingV1.populate(selection_key)\n</pre> from spyglass.decoding.v1.sorted_spikes import SortedSpikesDecodingV1  SortedSpikesDecodingV1.populate(selection_key) <pre>[12:19:30][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\nNo GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n</pre> <pre>Encoding models:   0%|          | 0/54 [00:00&lt;?, ?cell/s]</pre> <pre>Non-Local Likelihood:   0%|          | 0/54 [00:00&lt;?, ?cell/s]</pre> <p>We verify that the results have been inserted into the <code>DecodingOutput</code> merge table.</p> In\u00a0[3]: Copied! <pre>from spyglass.decoding.decoding_merge import DecodingOutput\n\nDecodingOutput.SortedSpikesDecodingV1 &amp; selection_key\n</pre> from spyglass.decoding.decoding_merge import DecodingOutput  DecodingOutput.SortedSpikesDecodingV1 &amp; selection_key Out[3]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters 42e9e7f9-a6f2-9242-63ce-94228bc72743 mediumnwb20230802_.nwb default_exclusion test_group test_group contfrag_sorted pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>We can load the results as before:</p> In\u00a0[6]: Copied! <pre>results = (SortedSpikesDecodingV1 &amp; selection_key).fetch_results()\nresults\n</pre> results = (SortedSpikesDecodingV1 &amp; selection_key).fetch_results() results Out[6]: <pre>&lt;xarray.Dataset&gt;\nDimensions:                      (state_ind: 25752, dim_0: 25752, time: 5001,\n                                  states: 2, intervals: 1, state_bins: 25752)\nCoordinates:\n  * state_ind                    (state_ind) int32 0 0 0 0 0 0 0 ... 1 1 1 1 1 1\n  * time                         (time) float64 1.626e+09 ... 1.626e+09\n  * states                       (states) object 'Continuous' 'Fragmented'\n    environments                 (states) object ...\n    encoding_groups              (states) int32 ...\n  * state_bins                   (state_bins) object MultiIndex\n  * state                        (state_bins) object 'Continuous' ... 'Fragme...\n  * x_position                   (state_bins) float64 29.02 29.02 ... 258.8\n  * y_position                   (state_bins) float64 5.828 7.811 ... 224.0\nDimensions without coordinates: dim_0, intervals\nData variables:\n    initial_conditions           (dim_0) float64 ...\n    discrete_state_transitions   (states, states) float64 ...\n    acausal_posterior            (intervals, time, state_bins) float32 ...\n    acausal_state_probabilities  (intervals, time, states) float64 ...\nAttributes:\n    marginal_log_likelihoods:  -39514.59</pre>xarray.Dataset<ul><li>Dimensions:<ul><li>state_ind: 25752</li><li>dim_0: 25752</li><li>time: 5001</li><li>states: 2</li><li>intervals: 1</li><li>state_bins: 25752</li></ul></li><li>Coordinates: (9)<ul><li>state_ind(state_ind)int320 0 0 0 0 0 0 0 ... 1 1 1 1 1 1 1 1<pre>array([0, 0, 0, ..., 1, 1, 1], dtype=int32)</pre></li><li>time(time)float641.626e+09 1.626e+09 ... 1.626e+09<pre>array([1.625936e+09, 1.625936e+09, 1.625936e+09, ..., 1.625936e+09,\n       1.625936e+09, 1.625936e+09])</pre></li><li>states(states)object'Continuous' 'Fragmented'<pre>array(['Continuous', 'Fragmented'], dtype=object)</pre></li><li>environments(states)object...<pre>[2 values with dtype=object]</pre></li><li>encoding_groups(states)int32...<pre>[2 values with dtype=int32]</pre></li><li>state_bins(state_bins)objectMultiIndex<pre>array([('Continuous', 29.019020985146526, 5.828088389548259),\n       ('Continuous', 29.019020985146526, 7.811111022216012),\n       ('Continuous', 29.019020985146526, 9.794133654883765), ...,\n       ('Fragmented', 258.79439532749024, 219.99453271766555),\n       ('Fragmented', 258.79439532749024, 221.9775553503333),\n       ('Fragmented', 258.79439532749024, 223.96057798300103)], dtype=object)</pre></li><li>state(state_bins)object'Continuous' ... 'Fragmented'<pre>array(['Continuous', 'Continuous', 'Continuous', ..., 'Fragmented',\n       'Fragmented', 'Fragmented'], dtype=object)</pre></li><li>x_position(state_bins)float6429.02 29.02 29.02 ... 258.8 258.8<pre>array([ 29.019021,  29.019021,  29.019021, ..., 258.794395, 258.794395,\n       258.794395])</pre></li><li>y_position(state_bins)float645.828 7.811 9.794 ... 222.0 224.0<pre>array([  5.828088,   7.811111,   9.794134, ..., 219.994533, 221.977555,\n       223.960578])</pre></li></ul></li><li>Data variables: (4)<ul><li>initial_conditions(dim_0)float64...<pre>[25752 values with dtype=float64]</pre></li><li>discrete_state_transitions(states, states)float64...<pre>[4 values with dtype=float64]</pre></li><li>acausal_posterior(intervals, time, state_bins)float32...<pre>[128785752 values with dtype=float32]</pre></li><li>acausal_state_probabilities(intervals, time, states)float64...<pre>[10002 values with dtype=float64]</pre></li></ul></li><li>Indexes: (4)<ul><li>state_indPandasIndex<pre>PandasIndex(Index([0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n       ...\n       1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n      dtype='int32', name='state_ind', length=25752))</pre></li><li>timePandasIndex<pre>PandasIndex(Index([1625935703.3693566, 1625935703.3713567, 1625935703.3733566,\n       1625935703.3753567, 1625935703.3773565, 1625935703.3793566,\n       1625935703.3813567, 1625935703.3833566, 1625935703.3853567,\n       1625935703.3873565,\n       ...\n       1625935713.3513515, 1625935713.3533516, 1625935713.3553517,\n       1625935713.3573515, 1625935713.3593516, 1625935713.3613515,\n       1625935713.3633516, 1625935713.3653517, 1625935713.3673515,\n       1625935713.3693516],\n      dtype='float64', name='time', length=5001))</pre></li><li>statesPandasIndex<pre>PandasIndex(Index(['Continuous', 'Fragmented'], dtype='object', name='states'))</pre></li><li>state_binsstatex_positiony_positionPandasMultiIndex<pre>PandasIndex(MultiIndex([('Continuous', 29.019020985146526,  5.828088389548259),\n            ('Continuous', 29.019020985146526,  7.811111022216012),\n            ('Continuous', 29.019020985146526,  9.794133654883765),\n            ('Continuous', 29.019020985146526, 11.777156287551517),\n            ('Continuous', 29.019020985146526,  13.76017892021927),\n            ('Continuous', 29.019020985146526, 15.743201552887022),\n            ('Continuous', 29.019020985146526, 17.726224185554777),\n            ('Continuous', 29.019020985146526, 19.709246818222525),\n            ('Continuous', 29.019020985146526,  21.69226945089028),\n            ('Continuous', 29.019020985146526, 23.675292083558034),\n            ...\n            ('Fragmented', 258.79439532749024, 206.11337428899128),\n            ('Fragmented', 258.79439532749024, 208.09639692165902),\n            ('Fragmented', 258.79439532749024, 210.07941955432676),\n            ('Fragmented', 258.79439532749024, 212.06244218699453),\n            ('Fragmented', 258.79439532749024,  214.0454648196623),\n            ('Fragmented', 258.79439532749024, 216.02848745233004),\n            ('Fragmented', 258.79439532749024, 218.01151008499778),\n            ('Fragmented', 258.79439532749024, 219.99453271766555),\n            ('Fragmented', 258.79439532749024,  221.9775553503333),\n            ('Fragmented', 258.79439532749024, 223.96057798300103)],\n           name='state_bins', length=25752))</pre></li></ul></li><li>Attributes: (1)marginal_log_likelihoods :-39514.59</li></ul> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#sorted-spikes-decoding", "title": "Sorted Spikes Decoding\u00b6", "text": "<p>The mechanics of decoding with sorted spikes are largely similar to those of decoding with unsorted spikes. You should familiarize yourself with the clusterless decoding tutorial before proceeding with this one.</p> <p>The elements we will need to decode with sorted spikes are:</p> <ul> <li><code>PositionGroup</code></li> <li><code>SortedSpikesGroup</code></li> <li><code>DecodingParameters</code></li> <li><code>encoding_interval</code></li> <li><code>decoding_interval</code></li> </ul> <p>This time, instead of extracting waveform features, we can proceed directly from the SpikeSortingOutput table to specify which units we want to decode. The rest of the decoding process is the same as before.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#sortedspikesgroup", "title": "SortedSpikesGroup\u00b6", "text": "<p><code>SortedSpikesGroup</code> is a child table of <code>SpikeSortingOutput</code> in the spikesorting pipeline. It allows us to group the spikesorting results from multiple  sources (e.g. multiple terode groups or intervals) into a single entry. Here we will group together the spiking of multiple tetrode groups to use for decoding.</p> <p>This table allows us filter units by their annotation labels from curation (e.g only include units labeled \"good\", exclude units labeled \"noise\") by defining parameters from <code>UnitSelectionParams</code>. When accessing data through <code>SortedSpikesGroup</code> the table will include only units with at least one label in <code>include_labels</code> and no labels in <code>exclude_labels</code>. We can look at those here:</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#model-parameters", "title": "Model parameters\u00b6", "text": "<p>As before we can specify the model parameters. The only difference is that we will use the <code>ContFragSortedSpikesClassifier</code> instead of the <code>ContFragClusterlessClassifier</code>.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#1d-decoding", "title": "1D Decoding\u00b6", "text": "<p>As in the clusterless notebook, we can decode 1D position if we specify the <code>track_graph</code>, <code>edge_order</code>, and <code>edge_spacing</code> parameters in the <code>Environment</code> class constructor. See the clusterless decoding tutorial for more details.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#decoding", "title": "Decoding\u00b6", "text": "<p>Now we can decode the position using the sorted spikes using the <code>SortedSpikesDecodingSelection</code> table. Here we assume that <code>PositionGroup</code> has been specified as in the clusterless decoding tutorial.</p>"}, {"location": "notebooks/50_MUA_Detection/", "title": "MUA Detection", "text": "In\u00a0[1]: Copied! <pre>import datajoint as dj\nfrom pathlib import Path\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n\nfrom spyglass.mua.v1.mua import MuaEventsV1, MuaEventsParameters\n</pre> import datajoint as dj from pathlib import Path  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info  from spyglass.mua.v1.mua import MuaEventsV1, MuaEventsParameters <pre>[2024-02-07 14:44:16,324][INFO]: Connecting edeno@lmf-db.cin.ucsf.edu:3306\n[2024-02-07 14:44:16,357][INFO]: Connected edeno@lmf-db.cin.ucsf.edu:3306\n</pre> In\u00a0[2]: Copied! <pre>MuaEventsParameters()\n</pre> MuaEventsParameters() Out[2]: <p>mua_param_name</p> a name for this set of parameters <p>mua_param_dict</p> dictionary of parameters default =BLOB= <p>Total: 1</p> In\u00a0[3]: Copied! <pre>MuaEventsV1()\n</pre> MuaEventsV1() Out[3]: <p>mua_param_name</p> a name for this set of parameters <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>pos_merge_id</p> <p>detection_interval</p> descriptive name of this interval list <p>analysis_file_name</p> name of the file <p>mua_times_object_id</p> default mediumnwb20230802_.nwb default_exclusion test_group 4eb59a18-045a-5768-d12e-b6473415ae1c pos 0 valid times mediumnwb20230802_235KSEV39O.nwb a93532eb-2947-4552-8877-19ea0d2dcc4f <p>Total: 1</p> In\u00a0[4]: Copied! <pre>from spyglass.position import PositionOutput\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",\n    \"trodes_pos_params_name\": \"single_led_upsampled\",\n}\n\npos_merge_id = (PositionOutput.TrodesPosV1 &amp; trodes_s_key).fetch1(\"merge_id\")\npos_merge_id\n</pre> from spyglass.position import PositionOutput  nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",     \"trodes_pos_params_name\": \"single_led_upsampled\", }  pos_merge_id = (PositionOutput.TrodesPosV1 &amp; trodes_s_key).fetch1(\"merge_id\") pos_merge_id Out[4]: <pre>UUID('4eb59a18-045a-5768-d12e-b6473415ae1c')</pre> In\u00a0[5]: Copied! <pre>from spyglass.spikesorting.analysis.v1.group import (\n    SortedSpikesGroup,\n)\n\nsorted_spikes_group_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": \"default_exclusion\",\n}\n\nSortedSpikesGroup &amp; sorted_spikes_group_key\n</pre> from spyglass.spikesorting.analysis.v1.group import (     SortedSpikesGroup, )  sorted_spikes_group_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": \"default_exclusion\", }  SortedSpikesGroup &amp; sorted_spikes_group_key Out[5]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb default_exclusion test_group <p>Total: 1</p> In\u00a0[6]: Copied! <pre>mua_key = {\n    \"mua_param_name\": \"default\",\n    **sorted_spikes_group_key,\n    \"pos_merge_id\": pos_merge_id,\n    \"detection_interval\": \"pos 0 valid times\",\n}\n\nMuaEventsV1().populate(mua_key)\nMuaEventsV1 &amp; mua_key\n</pre> mua_key = {     \"mua_param_name\": \"default\",     **sorted_spikes_group_key,     \"pos_merge_id\": pos_merge_id,     \"detection_interval\": \"pos 0 valid times\", }  MuaEventsV1().populate(mua_key) MuaEventsV1 &amp; mua_key Out[6]: <p>mua_param_name</p> a name for this set of parameters <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>pos_merge_id</p> <p>detection_interval</p> descriptive name of this interval list <p>analysis_file_name</p> name of the file <p>mua_times_object_id</p> default mediumnwb20230802_.nwb default_exclusion test_group 4eb59a18-045a-5768-d12e-b6473415ae1c pos 0 valid times mediumnwb20230802_235KSEV39O.nwb a93532eb-2947-4552-8877-19ea0d2dcc4f <p>Total: 1</p> In\u00a0[7]: Copied! <pre>mua_times = (MuaEventsV1 &amp; mua_key).fetch1_dataframe()\nmua_times\n</pre> mua_times = (MuaEventsV1 &amp; mua_key).fetch1_dataframe() mua_times Out[7]: start_time end_time duration mean_zscore median_zscore max_zscore min_zscore speed_at_start speed_at_end max_speed min_speed median_speed mean_speed id 0 1.625936e+09 1.625936e+09 0.238 1.204139 1.220925 2.258331 0.028349 2.760825 1.517296 3.576624 1.427160 2.851154 2.616020 1 1.625936e+09 1.625936e+09 0.120 1.364368 1.349665 2.412096 0.015503 1.803533 0.497099 1.803533 0.464203 0.553494 0.722099 2 1.625936e+09 1.625936e+09 0.082 1.350428 1.490812 2.229001 0.016666 0.639882 0.830598 0.838116 0.639882 0.791716 0.772154 3 1.625936e+09 1.625936e+09 0.074 1.609420 1.803123 2.578417 0.013637 2.690052 0.684351 2.690052 0.684351 1.327693 1.451886 4 1.625936e+09 1.625936e+09 0.208 1.459232 1.441996 2.610289 0.040791 0.208621 1.136773 1.298535 0.162133 1.088990 0.935486 ... ... ... ... ... ... ... ... ... ... ... ... ... ... 160 1.625937e+09 1.625937e+09 0.212 3.099559 3.296528 5.312823 0.013042 1.183369 0.500407 1.183369 0.386452 0.827476 0.791367 161 1.625937e+09 1.625937e+09 0.406 2.234536 2.555549 4.133360 0.013230 0.483370 0.415434 0.967244 0.170981 0.615845 0.603776 162 1.625937e+09 1.625937e+09 0.106 1.272785 1.090846 2.567647 0.029594 1.173970 1.531232 1.586072 1.173970 1.533011 1.472277 163 1.625937e+09 1.625937e+09 0.130 2.206783 2.288842 3.916084 0.000061 1.843658 2.197606 2.263417 1.843658 2.212691 2.156476 164 1.625937e+09 1.625937e+09 0.088 1.682973 1.689252 3.048358 0.021588 1.288769 1.058330 1.334759 1.058330 1.296841 1.260912 <p>165 rows \u00d7 13 columns</p> In\u00a0[8]: Copied! <pre>import matplotlib.pyplot as plt\nimport numpy as np\n\nfig, axes = plt.subplots(2, 1, sharex=True, figsize=(15, 4))\nspeed = MuaEventsV1.get_speed(mua_key).to_numpy()\ntime = speed.index.to_numpy()\nmultiunit_firing_rate = MuaEventsV1.get_firing_rate(mua_key, time)\n\ntime_slice = slice(\n    np.searchsorted(time, mua_times.loc[10].start_time) - 1_000,\n    np.searchsorted(time, mua_times.loc[10].start_time) + 5_000,\n)\n\naxes[0].plot(\n    time[time_slice],\n    multiunit_firing_rate[time_slice],\n    color=\"black\",\n)\naxes[0].set_ylabel(\"firing rate (Hz)\")\naxes[0].set_title(\"multiunit\")\naxes[1].fill_between(time[time_slice], speed[time_slice], color=\"lightgrey\")\naxes[1].set_ylabel(\"speed (cm/s)\")\naxes[1].set_xlabel(\"time (s)\")\n\nfor id, mua_time in mua_times.loc[\n    np.logical_and(\n        mua_times[\"start_time\"] &gt; time[time_slice].min(),\n        mua_times[\"end_time\"] &lt; time[time_slice].max(),\n    )\n].iterrows():\n    axes[0].axvspan(\n        mua_time[\"start_time\"], mua_time[\"end_time\"], color=\"red\", alpha=0.5\n    )\n</pre> import matplotlib.pyplot as plt import numpy as np  fig, axes = plt.subplots(2, 1, sharex=True, figsize=(15, 4)) speed = MuaEventsV1.get_speed(mua_key).to_numpy() time = speed.index.to_numpy() multiunit_firing_rate = MuaEventsV1.get_firing_rate(mua_key, time)  time_slice = slice(     np.searchsorted(time, mua_times.loc[10].start_time) - 1_000,     np.searchsorted(time, mua_times.loc[10].start_time) + 5_000, )  axes[0].plot(     time[time_slice],     multiunit_firing_rate[time_slice],     color=\"black\", ) axes[0].set_ylabel(\"firing rate (Hz)\") axes[0].set_title(\"multiunit\") axes[1].fill_between(time[time_slice], speed[time_slice], color=\"lightgrey\") axes[1].set_ylabel(\"speed (cm/s)\") axes[1].set_xlabel(\"time (s)\")  for id, mua_time in mua_times.loc[     np.logical_and(         mua_times[\"start_time\"] &gt; time[time_slice].min(),         mua_times[\"end_time\"] &lt; time[time_slice].max(),     ) ].iterrows():     axes[0].axvspan(         mua_time[\"start_time\"], mua_time[\"end_time\"], color=\"red\", alpha=0.5     ) <pre>[14:44:19][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n[14:44:24][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n</pre> In\u00a0[9]: Copied! <pre>(MuaEventsV1 &amp; mua_key).create_figurl(\n    zscore_mua=True,\n)\n</pre> (MuaEventsV1 &amp; mua_key).create_figurl(     zscore_mua=True, ) <pre>[14:44:25][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[14:44:29][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n</pre> <pre>WARNING: TimeseriesGraph::_add_series y argument is not 1D array. Using squeeze.\n</pre> Out[9]: <pre>'https://figurl.org/f?v=gs://figurl/spikesortingview-10&amp;d=sha1://08d7c35434e8469686fca8aff3ec9319bccb44f9&amp;label=Multiunit%20Detection&amp;zone=franklab.default'</pre> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}]}
\ No newline at end of file
+{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Spyglass", "text": "<p>Spyglass is an open-source software framework designed to offer reliable and reproducible analysis of neuroscience data and sharing of the results with collaborators and the broader community.</p> <p>Features of Spyglass include:</p> <ul> <li>Standardized data storage - Spyglass uses the open-source     Neurodata Without Borders: Neurophysiology (NWB:N)     format to ingest and store processed data. NWB:N is a standard set by the     BRAIN Initiative for neurophysiological data     (R\u00fcbel et al., 2022).</li> <li>Reproducible analysis - Spyglass uses DataJoint     to ensure that all analysis is reproducible. DataJoint is a data management     system that automatically tracks dependencies between data and analysis     code. This ensures that all analysis is reproducible and that the results     are automatically updated when the data or analysis code changes.</li> <li>Common analysis tools - Spyglass provides easy usage of the open-source     packages SpikeInterface,     Ghostipy, and     DeepLabCut for common analysis     tasks. These packages are well-documented and have active developer     communities.</li> <li>Interactive data visualization - Spyglass uses     figurl to create interactive     data visualizations that can be shared with collaborators and the broader     community. These visualizations are hosted on the web and can be viewed in     any modern web browser. The interactivity allows users to explore the data     and analysis results in detail.</li> <li>Sharing results - Spyglass enables sharing of data and analysis results     via Kachery, a     decentralized content addressable data sharing platform. Kachery Cloud     allows users to access the database and pull data and analysis results     directly to their local machine.</li> <li>Pipeline versioning - Processing and analysis of data in neuroscience is     often dynamic, requiring new features. Spyglass uses Merge tables to     ensure that analysis pipelines can be versioned. This allows users to easily     use and compare results from different versions of the analysis pipeline     while retaining the ability to access previously generated results.</li> <li>Cautious Delete - Spyglass uses a <code>cautious delete</code> feature to ensure that     data is not accidentally deleted by other users. When a user deletes data,     Spyglass will first check to see if the data belongs to another team of     users. This enables teams of users to work collaboratively on the same     database without worrying about accidentally deleting each other's data.</li> </ul>"}, {"location": "#getting-started", "title": "Getting Started", "text": "<p>This site hosts both installation instructions and tutorials to help you get started with Spyglass. We recommend running the notebooks yourself. They can be downloaded from GitHub here.</p>"}, {"location": "#diving-deeper", "title": "Diving Deeper", "text": "<p>The API Reference provides a detailed description of all the tables and class functions in Spyglass via python docstrings. Potential contributors should also read the Developer Guide. Those interested in in hosting a Spyglass instance for their own data should read the database management guide.</p> <p>We have a series of additional docs under the misc folder that may be helpful. Our changelog highlights the changes that have been made to Spyglass over time and the copyright page contains license information.</p>"}, {"location": "#citing-spyglass", "title": "Citing Spyglass", "text": "<p>Lee, K.H.*, Denovellis, E.L.*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Tritt, A., R\u00fcbel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. 10.1101/2024.01.25.577295.</p> <p>* Equal contribution</p> <p>See paper related code here.</p>"}, {"location": "CHANGELOG/", "title": "Change Log", "text": ""}, {"location": "CHANGELOG/#052-unreleased", "title": "[0.5.2] (Unreleased)", "text": "<ul> <li>Refactor <code>TableChain</code> to include <code>_searched</code> attribute. #867</li> <li>Fix errors in config import #882</li> </ul>"}, {"location": "CHANGELOG/#051-march-7-2024", "title": "[0.5.1] (March 7, 2024)", "text": ""}, {"location": "CHANGELOG/#infrastructure", "title": "Infrastructure", "text": "<ul> <li>Add user roles to <code>database_settings.py</code>. #832</li> <li>Fix redundancy in <code>waveforms_dir</code> #857</li> <li>Revise <code>dj_chains</code> to permit undirected paths for paths with multiple Merge     Tables. #846</li> </ul>"}, {"location": "CHANGELOG/#pipelines", "title": "Pipelines", "text": "<ul> <li>Position:<ul> <li>Fixes to <code>environment-dlc.yml</code> restricting tensortflow #834</li> <li>Video restriction for multicamera epochs #834</li> <li>Fixes to <code>_convert_mp4</code> #834</li> <li>Replace deprecated calls to <code>yaml.safe_load()</code> #834</li> </ul> </li> <li>Spikesorting:<ul> <li>Increase<code>spikeinterface</code> version to &gt;=0.99.1, &lt;0.100 #852</li> <li>Bug fix in single artifact interval edge case #859</li> <li>Bug fix in FigURL #871</li> </ul> </li> <li>LFP<ul> <li>In LFPArtifactDetection, only apply referencing if explicitly selected #863</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#050-february-9-2024", "title": "0.5.0 (February 9, 2024)", "text": ""}, {"location": "CHANGELOG/#infrastructure_1", "title": "Infrastructure", "text": "<ul> <li>Docs:<ul> <li>Additional documentation. #690</li> <li>Add overview of Spyglass to docs. #779</li> <li>Update docs to reflect new notebooks. #776</li> </ul> </li> <li>Mixin:<ul> <li>Add Mixin class to centralize <code>fetch_nwb</code> functionality. #692, #734</li> <li>Refactor restriction use in <code>delete_downstream_merge</code> #703</li> <li>Add <code>cautious_delete</code> to Mixin class<ul> <li>Initial implementation. #711, #762</li> <li>More robust caching of join to downstream tables. #806</li> <li>Overwrite datajoint <code>delete</code> method to use <code>cautious_delete</code>. #806</li> <li>Reverse join order for session summary. #821</li> <li>Add temporary logging of use to <code>common_usage</code>. #811, #821</li> </ul> </li> </ul> </li> <li>Merge Tables:<ul> <li>UUIDs: Revise Merge table uuid generation to include source. #824</li> <li>UUIDs: Remove mutual exclusivity logic due to new UUID generation. #824</li> <li>Add method for <code>merge_populate</code>. #824</li> </ul> </li> <li>Linting:<ul> <li>Clean up following pre-commit checks. #688</li> <li>Update linting for Black 24. #808</li> </ul> </li> <li>Misc:<ul> <li>Add <code>deprecation_factory</code> to facilitate table migration. #717</li> <li>Add Spyglass logger. #730</li> <li>Increase pytest coverage for <code>common</code>, <code>lfp</code>, and <code>utils</code>. #743</li> <li>Steamline dependency management. #822</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#pipelines_1", "title": "Pipelines", "text": "<ul> <li>Common:<ul> <li><code>IntervalList</code>: Add secondary key <code>pipeline</code> #742</li> <li>Add <code>common_usage</code> table. #811, #821, #824</li> <li>Add catch errors during <code>populate_all_common</code>. #824</li> </ul> </li> <li>Spike sorting:<ul> <li>Add SpikeSorting V1 pipeline. #651</li> <li>Move modules into spikesorting.v0 #807</li> </ul> </li> <li>LFP:<ul> <li>Minor fixes to LFPBandV1 populator and <code>make</code>. #706, #795</li> <li>LFPV1: Fix error for multiple lfp settings on same data #775</li> </ul> </li> <li>Linearization:<ul> <li>Minor fixes to LinearizedPositionV1 pipeline #695</li> <li>Rename <code>position_linearization</code> -&gt; <code>linearization</code>. #717</li> <li>Migrate tables: <code>common_position</code> -&gt; <code>linearization.v0</code>. #717</li> </ul> </li> <li>Position:<ul> <li>Refactor input validation in DLC pipeline. #688</li> <li>DLC path handling from config, and normalize naming convention. #722</li> <li>Fix in place column bug #752</li> </ul> </li> <li>Decoding:<ul> <li>Add <code>decoding</code> pipeline V1. #731, #769, #819</li> <li>Add a table to store the decoding results #731</li> <li>Use the new <code>non_local_detector</code> package for decoding #731</li> <li>Allow multiple spike waveform features for clusterless decoding #731</li> <li>Reorder notebooks #731</li> <li>Add fetch class functionality to <code>Merge</code> table. #783, #786</li> <li>Add ability to filter sorted units in decoding #807</li> <li>Rename SortedSpikesGroup.SortGroup to SortedSpikesGroup.Units #807</li> <li>Change methods with load_... to fetch_... for consistency #807</li> <li>Use merge table methods to access part methods #807</li> </ul> </li> <li>MUA<ul> <li>Add MUA pipeline V1. #731, #819</li> </ul> </li> <li>Ripple<ul> <li>Add figurl to Ripple pipeline #819</li> </ul> </li> </ul>"}, {"location": "CHANGELOG/#043-november-7-2023", "title": "0.4.3 (November 7, 2023)", "text": "<ul> <li>Migrate <code>config</code> helper scripts to Spyglass codebase. #662</li> <li>Revise contribution guidelines. #655</li> <li>Minor bug fixes. #656, #657, #659, #651, #671</li> <li>Add setup instruction specificity.</li> <li>Reduce primary key varchar allocation aross may tables. #664</li> </ul>"}, {"location": "CHANGELOG/#042-october-10-2023", "title": "0.4.2 (October 10, 2023)", "text": ""}, {"location": "CHANGELOG/#infrastructure-support", "title": "Infrastructure / Support", "text": "<ul> <li>Bumped Python version to 3.9. #583</li> <li>Updated user management helper scripts for MySQL 8. #650</li> <li>Centralized config/path handling to permit setting via datajoint config. #593</li> <li>Fixed Merge Table deletes: error specificity and transaction context. #617</li> </ul>"}, {"location": "CHANGELOG/#pipelines_2", "title": "Pipelines", "text": "<ul> <li>Common:<ul> <li>Added support multiple cameras per epoch. #557</li> <li>Removed <code>common_backup</code> schema. #631</li> <li>Added support for multiple position objects per NWB in <code>common_behav</code> via     PositionSource.SpatialSeries and RawPosition.PosObject #628, #616. Note:     Existing functions have been made compatible, but column labels for     <code>RawPosition.fetch1_dataframe</code> may change.</li> </ul> </li> <li>Spike sorting:<ul> <li>Added pipeline populator. #637, #646, #647</li> <li>Fixed curation functionality for <code>nn_isolation</code>. #597, #598</li> </ul> </li> <li>Position: Added position interval/epoch mapping via PositionIntervalMap. #620,     #621, #627</li> <li>LFP: Refactored pipeline. #594, #588, #605, #606, #607, #608, #615, #629</li> </ul>"}, {"location": "CHANGELOG/#041-june-30-2023", "title": "0.4.1 (June 30, 2023)", "text": "<ul> <li>Add mkdocs automated deployment. #527, #537, #549, #551</li> <li>Add class for Merge Tables. #556, #564, #565</li> </ul>"}, {"location": "CHANGELOG/#040-may-22-2023", "title": "0.4.0 (May 22, 2023)", "text": "<ul> <li>Updated call to <code>spikeinterface.preprocessing.whiten</code> to use dtype np.float16.     #446,</li> <li>Updated default spike sorting metric parameters. #447</li> <li>Updated whitening to be compatible with recent changes in spikeinterface when     using mountainsort. #449</li> <li>Moved LFP pipeline to <code>src/spyglass/lfp/v1</code> and addressed related usability     issues. #468, #478, #482, #484, #504</li> <li>Removed whiten parameter for clusterless thresholder. #454</li> <li>Added plot to plot all DIO events in a session. #457</li> <li>Added file sharing functionality through kachery_cloud. #458, #460</li> <li>Pinned numpy version to <code>numpy&lt;1.24</code></li> <li>Added scripts to add guests and collaborators as users. #463</li> <li>Cleaned up installation instructions in repo README. #467</li> <li>Added checks in decoding visualization to ensure time dimensions are the     correct length.</li> <li>Fixed artifact removed valid times. #472</li> <li>Added codespell workflow for spell checking and fixed typos. #471</li> <li>Updated LFP code to save LFP as <code>pynwb.ecephys.LFP</code> type. #475</li> <li>Added artifact detection to LFP pipeline. #473</li> <li>Replaced calls to <code>spikeinterface.sorters.get_default_params</code> with     <code>spikeinterface.sorters.get_default_sorter_params</code>. #486</li> <li>Updated position pipeline and added functionality to handle pose estimation     through DeepLabCut. #367, #505</li> <li>Updated <code>environment_position.yml</code>. #502</li> <li>Renamed <code>FirFilter</code> class to <code>FirFilterParameters</code>. #512</li> </ul>"}, {"location": "CHANGELOG/#034-march-30-2023", "title": "0.3.4 (March 30, 2023)", "text": "<ul> <li>Fixed error in spike sorting pipeline referencing the \"probe_type\" column     which is no longer accessible from the <code>Electrode</code> table. #437</li> <li>Fixed error when inserting an NWB file that does not have a probe     manufacturer. #433, #436</li> <li>Fixed error when adding a new <code>DataAcquisitionDevice</code> and a new <code>ProbeType</code>.     #436</li> <li>Fixed inconsistency between capitalized/uncapitalized versions of \"Intan\" for     DataAcquisitionAmplifier and DataAcquisitionDevice.adc_circuit. #430, #438</li> </ul>"}, {"location": "CHANGELOG/#033-march-29-2023", "title": "0.3.3 (March 29, 2023)", "text": "<ul> <li>Fixed errors from referencing the changed primary key for <code>Probe</code>. #429</li> </ul>"}, {"location": "CHANGELOG/#032-march-28-2023", "title": "0.3.2 (March 28, 2023)", "text": "<ul> <li>Fixed import of <code>common_nwbfile</code>. #424</li> </ul>"}, {"location": "CHANGELOG/#031-march-24-2023", "title": "0.3.1 (March 24, 2023)", "text": "<ul> <li>Fixed import error due to <code>sortingview.Workspace</code>. #421</li> </ul>"}, {"location": "CHANGELOG/#030-march-24-2023", "title": "0.3.0 (March 24, 2023)", "text": "<ul> <li>Refactor common for non Frank Lab data, allow file-based mods #420</li> <li>Allow creation and linkage of device metadata from YAML #400</li> <li>Move helper functions to utils directory #386</li> </ul>"}, {"location": "LICENSE/", "title": "Copyright", "text": "<p>Copyright (c) 2020-present Loren Frank</p> <p>Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:</p> <p>The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.</p> <p>THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BEsq LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.</p>"}, {"location": "contribute/", "title": "Developer notes", "text": "<p>Notes on how the repo / database is organized, intended for a new developer.</p>"}, {"location": "contribute/#development-workflow", "title": "Development workflow", "text": "<p>New contributors should follow the Fork-and-Branch workflow. See GitHub instructions here.</p> <p>Regular contributors may choose to follow the Feature Branch Workflow for features that will involve multiple contributors.</p>"}, {"location": "contribute/#code-organization", "title": "Code organization", "text": "<ul> <li>Tables are grouped into schemas by topic (e.g., <code>common_metrics</code>)</li> <li>Schemas<ul> <li>Are defined in a <code>py</code> pile.</li> <li>Correspond to MySQL 'databases'.</li> <li>Are organized into modules (e.g., <code>common</code>) by folders.</li> </ul> </li> <li>The common module<ul> <li>In principle, contains schema that are shared across all projects.</li> <li>In practice, contains shared tables (e.g., Session) and the first draft of     schemas that have since been split into their own     modality-specific\\     modules (e.g., <code>lfp</code>)</li> <li>Should not be added to without discussion.</li> </ul> </li> <li>A pipeline<ul> <li>Refers to a set of tables used for processing data of a particular modality     (e.g., LFP, spike sorting, position tracking).</li> <li>May span multiple schema.</li> </ul> </li> <li>For analysis that will be only useful to you, create your own schema.</li> </ul>"}, {"location": "contribute/#types-of-tables", "title": "Types of tables", "text": "<p>Spyglass uses DataJoint's default table tiers.</p> <p>By convention, an individual pipeline has one or more the following table types:</p> <ul> <li>Common/Multi-pipeline table</li> <li>NWB ingestion table</li> <li>Parameters table</li> <li>Selection table</li> <li>Data table</li> <li>Merge Table (see also doc</li> </ul>"}, {"location": "contribute/#commonmulti-pipeline", "title": "Common/Multi-pipeline", "text": "<p>Tables shared across multiple pipelines for shared data types.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Manual</code></li> <li>Examples: <code>IntervalList</code> (time interval for any analysis), <code>AnalysisNwbfile</code>     (analysis NWB files)</li> </ul> <p>Note: Because these are stand-alone tables not part of the dependency structure, developers should include enough information to link entries back to the pipeline where the data is used.</p>"}, {"location": "contribute/#nwb-ingestion", "title": "NWB ingestion", "text": "<p>Automatically populated when an NWB file is ingested (i.e., <code>dj.Imported</code>) to keep track of object hashes (i.e., <code>object_id</code>) in the NWB file. All such tables should be included in the <code>make</code> method of <code>Session</code>.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Imported</code></li> <li>Primary key: foreign key from <code>Session</code></li> <li>Non-primary key: <code>object_id</code>, the unique hash of an object in the NWB file.</li> <li>Examples: <code>Raw</code>, <code>Institution</code>, etc.</li> <li>Required methods:<ul> <li><code>make</code>: must read information from an NWB file and insert it to the table.</li> <li><code>fetch_nwb</code>: retrieve the data specified by the object ID.</li> </ul> </li> </ul>"}, {"location": "contribute/#parameters", "title": "Parameters", "text": "<p>Stores the set of values that may be used in an analysis.</p> <ul> <li>Naming convention: end with <code>Parameters</code> or <code>Params</code></li> <li>Data tier: <code>dj.Manual</code>, or <code>dj.Lookup</code></li> <li>Primary key: <code>{pipeline}_params_name</code>, <code>varchar</code></li> <li>Non-primary key: <code>{pipeline}_params</code>, <code>blob</code> - dict of parameters</li> <li>Examples: <code>RippleParameters</code>, <code>DLCModelParams</code></li> <li>Possible method: if <code>dj.Manual</code>, include <code>insert_default</code></li> </ul> <p>Notes: Some early instances of Parameter tables (a) used non-primary keys for each individual parameter, and (b) use the Manual rather than Lookup tier, requiring a class method to insert defaults.</p>"}, {"location": "contribute/#selection", "title": "Selection", "text": "<p>A staging area to pair sessions with parameter sets, allowing us to be selective in the analyses we run. It may not make sense to pair every paramset with every session.</p> <ul> <li>Naming convention: end with <code>Selection</code></li> <li>Data tier: <code>dj.Manual</code></li> <li>Primary key(s): Foreign key references to<ul> <li>one or more NWB or data tables</li> <li>optionally, one or more parameter tables</li> </ul> </li> <li>Non-primary key: None</li> <li>Examples: <code>MetricSelection</code>, <code>LFPSelection</code></li> </ul> <p>It is possible for a Selection table to collect information from more than one Parameter table. For example, the Selection table for spike sorting holds information about both the interval (<code>SortInterval</code>) and the group of electrodes (<code>SortGroup</code>) to be sorted.</p>"}, {"location": "contribute/#data", "title": "Data", "text": "<p>The output of processing steps associated with a selection table. Has a <code>make</code> method that carries out the computation specified in the Selection table when <code>populate</code> is called.</p> <ul> <li>Naming convention: None</li> <li>Data tier: <code>dj.Computed</code></li> <li>Primary key: Foreign key reference to a Selection table.</li> <li>Non-primary key: <code>analysis_file_name</code> inherited from <code>AnalysisNwbfile</code> table     (i.e., name of the analysis NWB file that will hold the output of the     computation).</li> <li>Required methods:<ul> <li><code>make</code>: carries out the computation and insert a new entry; must also create     an analysis NWB file and insert it to the <code>AnalysisNwbfile</code> table. Note     that this method is never called directly; it is called via <code>populate</code>.     Multiple entries can be run in parallel when called with     <code>reserve_jobs=True</code>.</li> <li><code>delete</code>: extension of the <code>delete</code> method that checks user privilege before     deleting entries as a way to prevent accidental deletion of computations     that take a long time (see below).</li> </ul> </li> <li>Example: <code>QualityMetrics</code>, <code>LFPV1</code></li> </ul>"}, {"location": "contribute/#merge", "title": "Merge", "text": "<p>Following a convention outlined in the dedicated doc, merges the output of different pipelines dedicated to the same modality as part tables (e.g., common LFP, LFP v1, imported LFP) to permit unified downstream processing.</p> <ul> <li>Naming convention: <code>{Pipeline}Output</code></li> <li>Data tier: custom <code>_Merge</code> class</li> <li>Primary key: <code>merge_id</code>, <code>uuid</code></li> <li>Non-primary key: <code>source</code>, <code>varchar</code> table name associated with that entry</li> <li>Required methods: None - see custom class methods with <code>merge_</code> prefix</li> <li>Example: <code>LFPOutput</code>, <code>PositionOutput</code></li> </ul>"}, {"location": "contribute/#integration-with-nwb", "title": "Integration with NWB", "text": ""}, {"location": "contribute/#nwb-files", "title": "NWB files", "text": "<p>NWB files contain everything about the experiment and form the starting point of all analyses.</p> <ul> <li>Naming: <code>{animal name}YYYYMMDD.nwb</code></li> <li>Storage:<ul> <li>On disk, directory identified by <code>settings.py</code> as <code>raw_dir</code> (e.g.,     <code>/stelmo/nwb/raw</code>)</li> <li>In database, in the <code>Nwbfile</code> table</li> </ul> </li> <li>Copies:<ul> <li>made with an underscore <code>{animal name}YYYYMMDD_.nwb</code></li> <li>stored in the same <code>raw_dir</code></li> <li>contain pointers to objects in original file</li> <li>permit adding new parts to the NWB file without risk of corrupting the     original data</li> </ul> </li> </ul>"}, {"location": "contribute/#analysis-files", "title": "Analysis files", "text": "<p>Hold the results of intermediate steps in the analysis.</p> <ul> <li>Naming: <code>{animal name}YYYYMMDD_{10-character random string}.nwb</code></li> <li>Storage:<ul> <li>On disk, directory identified by <code>settings.py</code> as <code>analysis_dir</code> (e.g.,     <code>/stelmo/nwb/analysis</code>). Items are further sorted into folders matching     original NWB file name</li> <li>In database, in the <code>AnalysisNwbfile</code> table.</li> </ul> </li> <li>Examples: filtered recordings, spike times of putative units after sorting, or     waveform snippets.</li> </ul> <p>Note: Because NWB files and analysis files exist both on disk and listed in tables, these can become out of sync. You can 'equalize' the database table lists and the set of files on disk by running <code>cleanup</code> method, which deletes any files not listed in the table from disk.</p>"}, {"location": "contribute/#reading-and-writing-recordings", "title": "Reading and writing recordings", "text": "<p>Recordings start out as an NWB file, which is opened as a <code>NwbRecordingExtractor</code>, a class in <code>spikeinterface</code>. When using <code>sortingview</code> for visualizing the results of spike sorting, this recording is saved again in HDF5 format. This duplication should be resolved in the future.</p>"}, {"location": "contribute/#naming-convention", "title": "Naming convention", "text": "<p>The following objects should be uniquely named.</p> <ul> <li>Recordings: Underscore-separated concatenations of uniquely defining     features,     <code>NWBFileName_IntervalName_ElectrodeGroupName_PreprocessingParamsName</code>.</li> <li>SpikeSorting: Adds <code>SpikeSorter_SorterParamName</code> to the name of the     recording.</li> <li>Waveforms: Adds <code>_WaveformParamName</code> to the name of the sorting.</li> <li>Quality metrics: Adds <code>_MetricParamName</code> to the name of the waveform.</li> <li>Analysis NWB files:     <code>NWBFileName_IntervalName_ElectrodeGroupName_PreprocessingParamsName.nwb</code></li> <li>Each recording and sorting is given truncated UUID strings as part of     concatenations.</li> </ul> <p>Following broader Python conventions, methods a method that will not be explicitly called by the user should start with <code>_</code></p>"}, {"location": "contribute/#time", "title": "Time", "text": "<p>The <code>IntervalList</code> table stores all time intervals in the following format: <code>[start_time, stop_time]</code>, which represents a contiguous time of valid data. These are used to exclude any invalid timepoints, such as missing data from a faulty connection.</p> <ul> <li>Intervals can be nested for a set of disjoint intervals.</li> <li>Some recordings have explicit     PTP timestamps     associated with each sample. Some older recordings are missing PTP times,     and times must be inferred from the TTL pulses from the camera.</li> </ul>"}, {"location": "contribute/#misc", "title": "Misc", "text": "<ul> <li>During development, we suggest using a Docker container. See     example.</li> <li>DataJoint is unable to set delete permissions on a per-table basis. If a user     is able to delete entries in a given table, she can delete entries in any     table in the schema. The <code>SpikeSorting</code> table extends the built-in <code>delete</code>     method to check if the username matches a list of allowed users when     <code>delete</code> is called. Issues #226 and #586 track the progress of generalizing     this feature.</li> <li><code>numpy</code> style docstrings will be interpreted by API docs. To check for     compliance, monitor the std out when building docs (see <code>docs/README.md</code>)</li> <li><code>fetch_nwb</code> is currently reperated across many tables. For progress on a fix,     follow issue #530</li> </ul>"}, {"location": "contribute/#making-a-release", "title": "Making a release", "text": "<p>Spyglass follows Semantic Versioning with versioning of the form <code>X.Y.Z</code> (e.g., <code>0.4.2</code>).</p> <ol> <li>In <code>CITATION.cff</code>, update the <code>version</code> key.</li> <li>Make a pull request with changes.</li> <li>After the pull request is merged, pull this merge commit and tag it with     <code>git tag {version}</code></li> <li>Publish the new release tag. Run <code>git push origin {version}</code>. This will     rebuild docs and push updates to PyPI.</li> <li>Make a new     release on GitHub.</li> </ol>"}, {"location": "installation/", "title": "Installation", "text": "<p>Note: Developers, or those who wish to add features or otherwise work on the codebase should follow the same steps below, but install Spyglass as editable with the <code>-e</code> flag: <code>pip install -e /path/to/spyglass</code></p>"}, {"location": "installation/#basic-installation", "title": "Basic Installation", "text": "<p>For basic installation steps, see the Setup notebook 'local installation' section, including python, mamba (for managing a virtual environment), VSCode, Jupyter, and git. This notebook also covers database access.</p>"}, {"location": "installation/#additional-packages", "title": "Additional Packages", "text": "<p>Some pipelines require installation of additional packages.</p> <p>The spike sorting pipeline relies on <code>spikeinterface</code> and optionally <code>mountainsort4</code>.</p> <pre><code>pip install spikeinterface[full,widgets]\npip install mountainsort4\n</code></pre> <p>WARNING: If you are on an M1 Mac, you need to install <code>pyfftw</code> via <code>conda</code> BEFORE installing <code>ghostipy</code>:</p> <pre><code>conda install -c conda-forge pyfftw\n</code></pre> <p>The LFP pipeline uses <code>ghostipy</code>:</p> <pre><code>pip install ghostipy\n</code></pre>"}, {"location": "installation/#database-access", "title": "Database access", "text": "<p>For basic installation steps, see the Setup notebook 'database connection' section. For additional details, see the DataJoint documentation.</p>"}, {"location": "installation/#config", "title": "Config", "text": ""}, {"location": "installation/#via-file-recommended", "title": "Via File (Recommended)", "text": "<p>A <code>dj_local_conf.json</code> file in your current directory when launching python can hold all the specifics needed to connect to a database. This can include different directories for different pipelines. If only the Spyglass <code>base</code> is specified, other subfolder names are assumed from defaults. See <code>dj_local_conf_example.json</code> for the full set of options. This example can be copied and saved as <code>dj_local_conf.json</code> to set the configuration for a given folder. Alternatively, it can be saved as <code>.datajoint_config.json</code> in a user's home directory to be accessed globally. See DataJoint docs for more details.</p> <p>Note that raw and analysis folder locations should be specified under both <code>stores</code> and <code>custom</code> sections of the config file. The <code>stores</code> section is used by DataJoint to store the location of files referenced in database, while the <code>custom</code> section is used by Spyglass. Spyglass will check that these sections match on startup.</p>"}, {"location": "installation/#via-environment-variables", "title": "Via Environment Variables", "text": "<p>Older versions of Spyglass relied exclusively on environment for config. If <code>spyglass_dirs</code> is not found in the config file, Spyglass will look for environment variables. These can be set either once in a terminal session, or permanently in a unix settings file (e.g., <code>.bashrc</code> or <code>.bash_profile</code>) in your home directory.</p> <pre><code>export SPYGLASS_BASE_DIR=\"/stelmo/nwb\"\nexport SPYGLASS_RECORDING_DIR=\"$SPYGLASS_BASE_DIR/recording\"\nexport SPYGLASS_SORTING_DIR=\"$SPYGLASS_BASE_DIR/sorting\"\nexport SPYGLASS_VIDEO_DIR=\"$SPYGLASS_BASE_DIR/video\"\nexport SPYGLASS_WAVEFORMS_DIR=\"$SPYGLASS_BASE_DIR/waveforms\"\nexport SPYGLASS_TEMP_DIR=\"$SPYGLASS_BASE_DIR/tmp\"\nexport DJ_SUPPORT_FILEPATH_MANAGEMENT=\"TRUE\"\n</code></pre> <p>To load variables from a <code>.bashrc</code> file, run <code>source ~/.bashrc</code> in a terminal.</p>"}, {"location": "installation/#temporary-directory", "title": "Temporary directory", "text": "<p>A temporary directory will speed up spike sorting. If unspecified by either method above, it will be assumed as a <code>tmp</code> subfolder relative to the base path. Be sure it has enough free space (ideally at least 500GB).</p>"}, {"location": "installation/#subfolders", "title": "Subfolders", "text": "<p>If subfolders do not exist, they will be created automatically. If unspecified by either method above, they will be assumed as <code>recording</code>, <code>sorting</code>, <code>video</code>, etc. subfolders relative to the base path.</p>"}, {"location": "installation/#file-manager", "title": "File manager", "text": "<p><code>kachery-cloud</code> is a file manager for Frank Lab collaborators who do not have access to the lab's production database.</p> <p>To customize <code>kachery</code> file paths, see <code>dj_local_conf_example.json</code> or set the following variables in your unix settings file (e.g., <code>.bashrc</code>). If unspecified, the defaults below are assumed.</p> <pre><code>export KACHERY_CLOUD_DIR=\"$SPYGLASS_BASE_DIR/.kachery-cloud\"\nexport KACHERY_TEMP_DIR=\"$SPYGLASS_BASE_DIR/tmp\"\n</code></pre> <p>Be sure to load these with <code>source ~/.bashrc</code> to persist changes.</p>"}, {"location": "installation/#test-connection", "title": "Test connection", "text": "<p>Finally, open up a python console (e.g., run <code>ipython</code> from terminal) and import <code>spyglass</code> to check that the installation has worked.</p> <pre><code>from spyglass.common import Nwbfile\n\nNwbfile()\n</code></pre>"}, {"location": "api/", "title": "API Docs", "text": "<p>The files in this directory are automatically generated from the docstrings in the source code. They include descriptions of each of the DataJoint tables and other classes/methods within Spyglass.</p> <p>These docs are updated any time a new release is made or a tag is pushed to the repository.</p>"}, {"location": "api/settings/", "title": "settings.py", "text": ""}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig", "title": "<code>SpyglassConfig</code>", "text": "<p>Gets Spyglass dirs from dj.config or environment variables.</p> <p>Uses SpyglassConfig.relative_dirs to (a) gather user settings from dj.config or os environment variables or defaults relative to base, in that order (b) set environment variables, and (c) make dirs that don't exist. NOTE: when passed a base_dir, it will ignore env vars to facilitate testing.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>class SpyglassConfig:\n    \"\"\"Gets Spyglass dirs from dj.config or environment variables.\n\n    Uses SpyglassConfig.relative_dirs to (a) gather user\n    settings from dj.config or os environment variables or defaults relative to\n    base, in that order (b) set environment variables, and (c) make dirs that\n    don't exist. NOTE: when passed a base_dir, it will ignore env vars to\n    facilitate testing.\n    \"\"\"\n\n    def __init__(self, base_dir: str = None, **kwargs):\n        \"\"\"\n        Initializes a new instance of the class.\n\n        Parameters\n        ----------\n        base_dir (str)\n            The base directory.\n\n        Attributes\n        ----------\n        supplied_base_dir (str)\n            The base directory passed to the class.\n        config_defaults (dict)\n            Default settings for the config.\n        relative_dirs (dict)\n            Relative dirs for each prefix (spyglass, kachery, dlc). Relative\n            to respective base_dir. Created on init.\n        dj_defaults (dict)\n            Default settings for datajoint.\n        env_defaults (dict)\n            Default settings for environment variables.\n        _config (dict)\n            Cached config settings.\n        _debug_mode (bool)\n            True if debug_mode is set. Supports skipping known bugs in test env.\n        _test_mode (bool)\n            True if test_mode is set. Required for pytests to run without\n            prompts.\n        \"\"\"\n        self.supplied_base_dir = base_dir\n        self._config = dict()\n        self.config_defaults = dict(prepopulate=True)\n        self._debug_mode = kwargs.get(\"debug_mode\", False)\n        self._test_mode = kwargs.get(\"test_mode\", False)\n        self._dlc_base = None\n        self.load_failed = False\n\n        self.relative_dirs = {\n            # {PREFIX}_{KEY}_DIR, default dir relative to base_dir\n            \"spyglass\": {\n                \"raw\": \"raw\",\n                \"analysis\": \"analysis\",\n                \"recording\": \"recording\",\n                \"sorting\": \"spikesorting\",\n                \"waveforms\": \"waveforms\",\n                \"temp\": \"tmp\",\n                \"video\": \"video\",\n            },\n            \"kachery\": {\n                \"cloud\": \".kachery-cloud\",\n                \"storage\": \"kachery_storage\",\n                \"temp\": \"tmp\",\n            },\n            \"dlc\": {\n                \"project\": \"projects\",\n                \"video\": \"video\",\n                \"output\": \"output\",\n            },\n        }\n        self.dj_defaults = {\n            \"database.host\": kwargs.get(\"database_host\", \"lmf-db.cin.ucsf.edu\"),\n            \"database.user\": kwargs.get(\"database_user\"),\n            \"database.port\": kwargs.get(\"database_port\", 3306),\n            \"database.use_tls\": kwargs.get(\"database_use_tls\", True),\n            \"filepath_checksum_size_limit\": 1 * 1024**3,\n            \"enable_python_native_blobs\": True,\n        }\n        self.env_defaults = {\n            \"FIGURL_CHANNEL\": \"franklab2\",\n            \"DJ_SUPPORT_FILEPATH_MANAGEMENT\": \"TRUE\",\n            \"KACHERY_CLOUD_EPHEMERAL\": \"TRUE\",\n            \"HD5_USE_FILE_LOCKING\": \"FALSE\",\n        }\n\n    def load_config(\n        self,\n        base_dir=None,\n        force_reload=False,\n        on_startup: bool = False,\n        **kwargs,\n    ):\n        \"\"\"\n        Loads the configuration settings for the object.\n\n        Order of precedence, where X is base, raw, analysis, etc.:\n            1. SpyglassConfig(base_dir=\"string\") for base dir only\n            2. dj.config['custom']['{spyglass/kachery}_dirs']['X']\n            3. os.environ['{SPYGLASS/KACHERY}_{X}_DIR']\n            4. resolved_base_dir/X for non-base dirs\n\n        Parameters\n        ----------\n        base_dir: str\n            Optional. Default None. The base directory. If not provided, will\n            use the env variable or existing config.\n        force_reload: bool\n            Optional. Default False. Default skip load if already completed.\n\n        Raises\n        ------\n        ValueError\n            If base_dir is not set in either dj.config or os.environ.\n\n        Returns\n        -------\n        dict\n            list of relative_dirs and other settings (e.g., prepopulate).\n        \"\"\"\n        if not force_reload and self._config:\n            return self._config\n\n        dj_custom = dj.config.get(\"custom\", {})\n        dj_spyglass = dj_custom.get(\"spyglass_dirs\", {})\n        dj_kachery = dj_custom.get(\"kachery_dirs\", {})\n        dj_dlc = dj_custom.get(\"dlc_dirs\", {})\n\n        self._debug_mode = dj_custom.get(\"debug_mode\", False)\n        self._test_mode = kwargs.get(\"test_mode\") or dj_custom.get(\n            \"test_mode\", False\n        )\n\n        resolved_base = (\n            base_dir\n            or self.supplied_base_dir\n            or dj_spyglass.get(\"base\")\n            or os.environ.get(\"SPYGLASS_BASE_DIR\")\n        )\n\n        if resolved_base and not Path(resolved_base).exists():\n            resolved_base = Path(resolved_base).expanduser()\n        if not resolved_base or not Path(resolved_base).exists():\n            if not on_startup:  # Only warn if not on startup\n                logger.error(\n                    f\"Could not find SPYGLASS_BASE_DIR: {resolved_base}\"\n                    + \"\\n\\tCheck dj.config['custom']['spyglass_dirs']['base']\"\n                    + \"\\n\\tand os.environ['SPYGLASS_BASE_DIR']\"\n                )\n            self.load_failed = True\n            return\n\n        self._dlc_base = (\n            dj_dlc.get(\"base\")\n            or os.environ.get(\"DLC_BASE_DIR\")\n            or os.environ.get(\"DLC_PROJECT_PATH\", \"\").split(\"projects\")[0]\n            or str(Path(resolved_base) / \"deeplabcut\")\n        )\n        Path(self._dlc_base).mkdir(exist_ok=True)\n\n        config_dirs = {\"SPYGLASS_BASE_DIR\": str(resolved_base)}\n        for prefix, dirs in self.relative_dirs.items():\n            this_base = self._dlc_base if prefix == \"dlc\" else resolved_base\n            for dir, dir_str in dirs.items():\n                dir_env_fmt = self.dir_to_var(dir=dir, dir_type=prefix)\n\n                env_loc = (  # Ignore env vars if base was passed to func\n                    os.environ.get(dir_env_fmt)\n                    if not self.supplied_base_dir\n                    else None\n                )\n\n                source_config = (\n                    dj_dlc\n                    if prefix == \"dlc\"\n                    else dj_kachery if prefix == \"kachery\" else dj_spyglass\n                )\n                dir_location = (\n                    source_config.get(dir)\n                    or env_loc\n                    or str(Path(this_base) / dir_str)\n                ).replace('\"', \"\")\n\n                config_dirs.update({dir_env_fmt: str(dir_location)})\n\n        kachery_zone_dict = {\n            \"KACHERY_ZONE\": (\n                os.environ.get(\"KACHERY_ZONE\")\n                or dj.config.get(\"custom\", {}).get(\"kachery_zone\")\n                or \"franklab.default\"\n            )\n        }\n\n        loaded_env = self._load_env_vars()\n        self._set_env_with_dict(\n            {**config_dirs, **kachery_zone_dict, **loaded_env}\n        )\n        self._mkdirs_from_dict_vals(config_dirs)\n\n        self._config = dict(\n            debug_mode=self._debug_mode,\n            test_mode=self._test_mode,\n            **self.config_defaults,\n            **config_dirs,\n            **kachery_zone_dict,\n            **loaded_env,\n        )\n\n        self._set_dj_config_stores()\n\n        return self._config\n\n    def _load_env_vars(self):\n        loaded_dict = {}\n        for var, val in self.env_defaults.items():\n            loaded_dict[var] = os.getenv(var, val)\n        return loaded_dict\n\n    def _set_env_with_dict(self, env_dict):\n        # NOTE: Kept for backwards compatibility. Should be removed in future\n        # for custom paths. Keep self.env_defaults.\n        for var, val in env_dict.items():\n            os.environ[var] = str(val)\n\n    def _mkdirs_from_dict_vals(self, dir_dict):\n        if self._debug_mode:\n            return\n        for dir_str in dir_dict.values():\n            Path(dir_str).mkdir(exist_ok=True)\n\n    def _set_dj_config_stores(self, check_match=True, set_stores=True):\n        \"\"\"\n        Checks dj.config['stores'] match resolved dirs. Ensures stores set.\n\n        Parameters\n        ----------\n        dir_dict: dict\n            Dictionary of resolved dirs.\n        check_match: bool\n            Optional. Default True. Check that dj.config['stores'] match\n            resolved dirs.\n        set_stores: bool\n            Optional. Default True. Set dj.config['stores'] to resolved dirs.\n        \"\"\"\n\n        mismatch_analysis = False\n        mismatch_raw = False\n\n        if check_match:\n            dj_stores = dj.config.get(\"stores\", {})\n            store_r = dj_stores.get(\"raw\", {}).get(\"location\")\n            store_a = dj_stores.get(\"analysis\", {}).get(\"location\")\n            mismatch_raw = store_r and Path(store_r) != Path(self.raw_dir)\n            mismatch_analysis = store_a and Path(store_a) != Path(\n                self.analysis_dir\n            )\n\n        if set_stores:\n            if mismatch_raw or mismatch_analysis:\n                logger.warning(\n                    \"Setting config DJ stores to resolve mismatch.\\n\\t\"\n                    + f\"raw     : {self.raw_dir}\\n\\t\"\n                    + f\"analysis: {self.analysis_dir}\"\n                )\n            dj.config.update(self._dj_stores)\n            return\n\n        if mismatch_raw or mismatch_analysis:\n            raise ValueError(\n                \"dj.config['stores'] does not match resolved dirs.\"\n                + f\"\\n\\tdj.config['stores']: {dj_stores}\"\n                + f\"\\n\\tResolved dirs: {self._dj_stores}\"\n            )\n\n        return\n\n    def dir_to_var(self, dir: str, dir_type: str = \"spyglass\"):\n        \"\"\"Converts a dir string to an env variable name.\"\"\"\n        return f\"{dir_type.upper()}_{dir.upper()}_DIR\"\n\n    def _generate_dj_config(\n        self,\n        base_dir: str = None,\n        database_user: str = None,\n        database_password: str = None,\n        database_host: str = \"lmf-db.cin.ucsf.edu\",\n        database_port: int = 3306,\n        database_use_tls: bool = True,\n        **kwargs,\n    ):\n        \"\"\"Generate a datajoint configuration file.\n\n        Parameters\n        ----------\n        database_user : str, optional\n            The database user. If not provided, resulting config will not\n            specify.\n        database_password : str, optional\n            The database password. If not provided, resulting config will not\n            specify.\n        database_host : str, optional\n            Default lmf-db.cin.ucsf.edu. MySQL host name.\n        dapabase_port : int, optional\n            Default 3306. Port number for MySQL server.\n        database_use_tls : bool, optional\n            Default True. Use TLS encryption.\n        **kwargs: dict, optional\n            Any other valid datajoint configuration parameters.\n            Note: python will raise error for params with `.` in name.\n        \"\"\"\n\n        if database_user:\n            kwargs.update({\"database.user\": database_user})\n        if database_password:\n            kwargs.update({\"database.password\": database_password})\n\n        kwargs.update(\n            {\n                \"database.host\": database_host,\n                \"database.port\": database_port,\n                \"database.use_tls\": database_use_tls,\n            }\n        )\n\n        # `|` merges dictionaries\n        return self.dj_defaults | self._dj_stores | self._dj_custom | kwargs\n\n    def save_dj_config(\n        self,\n        save_method: str = \"global\",\n        output_filename: str = None,\n        base_dir=None,\n        set_password=True,\n        **kwargs,\n    ):\n        \"\"\"Set the dj.config parameters, set password, and save config to file.\n\n        Parameters\n        ----------\n        save_method : {'local', 'global', 'custom'}, optional\n            The method to use to save the config. If either 'local' or 'global',\n            datajoint builtins will be used to save.\n        output_filename : str or Path, optional\n            Default to datajoint global config. If save_method = 'custom', name\n            of file to generate. Must end in either be either yaml or json.\n        base_dir : str, optional\n            The base directory. If not provided, will default to the env var\n        set_password : bool, optional\n            Default True. Set the database password.\n        kwargs: dict, optional\n            Any other valid datajoint configuration parameters, including\n            database_user, database_password, database_host, database_port, etc.\n            Note: python will raise error for params with `.` in name, so use\n            underscores instead.\n        \"\"\"\n        if base_dir:\n            self.load_config(\n                base_dir=base_dir, force_reload=True, on_startup=False\n            )\n\n        if output_filename:\n            save_method = \"custom\"\n            path = Path(output_filename).expanduser()  # Expand ~\n            filepath = path if path.is_absolute() else path.absolute()\n            filepath.parent.mkdir(exist_ok=True, parents=True)\n            filepath = (\n                filepath.with_suffix(\".json\")  # ensure suffix, default json\n                if filepath.suffix not in [\".json\", \".yaml\"]\n                else filepath\n            )\n        elif save_method == \"local\":\n            filepath = Path(\".\") / dj.settings.LOCALCONFIG\n        elif save_method == \"global\":\n            filepath = Path(\"~\").expanduser() / dj.settings.GLOBALCONFIG\n        else:\n            raise ValueError(\n                \"For save_dj_config, either (a) save_method must be 'local' \"\n                + \" or 'global' or (b) must provide custom output_filename.\"\n            )\n\n        dj.config.update(self._generate_dj_config(**kwargs))\n\n        if set_password:\n            try:\n                dj.set_password()\n            except OperationalError as e:\n                warnings.warn(f\"Database connection issues. Wrong pass?\\n\\t{e}\")\n\n        user_warn = (\n            f\"Replace existing file? {filepath.resolve()}\\n\\t\"\n            + \"\\n\\t\".join([f\"{k}: {v}\" for k, v in config.items()])\n            + \"\\n\"\n        )\n\n        if (\n            not self.test_mode\n            and filepath.exists()\n            and dj.utils.user_choice(user_warn)[0] != \"y\"\n        ):\n            return\n\n        if save_method == \"global\":\n            dj.config.save_global(verbose=True)\n            return\n\n        if save_method == \"local\":\n            dj.config.save_local(verbose=True)\n            return\n\n        with open(filepath, \"w\") as outfile:\n            if filepath.suffix == \".yaml\":\n                yaml.dump(dj.config._conf, outfile, default_flow_style=False)\n            else:\n                json.dump(dj.config._conf, outfile, indent=2)\n            logger.info(f\"Saved config to {filepath}\")\n\n    @property\n    def _dj_stores(self) -&gt; dict:\n        self.load_config()\n        return {\n            \"stores\": {\n                \"raw\": {\n                    \"protocol\": \"file\",\n                    \"location\": self.raw_dir,\n                    \"stage\": self.raw_dir,\n                },\n                \"analysis\": {\n                    \"protocol\": \"file\",\n                    \"location\": self.analysis_dir,\n                    \"stage\": self.analysis_dir,\n                },\n            }\n        }\n\n    @property\n    def _dj_custom(self) -&gt; dict:\n        self.load_config()\n        return {\n            \"custom\": {\n                \"debug_mode\": str(self.debug_mode).lower(),\n                \"test_mode\": str(self._test_mode).lower(),\n                \"spyglass_dirs\": {\n                    \"base\": self.base_dir,\n                    \"raw\": self.raw_dir,\n                    \"analysis\": self.analysis_dir,\n                    \"recording\": self.recording_dir,\n                    \"sorting\": self.sorting_dir,\n                    \"waveforms\": self.waveforms_dir,\n                    \"temp\": self.temp_dir,\n                    \"video\": self.video_dir,\n                },\n                \"kachery_dirs\": {\n                    \"cloud\": self.config.get(\n                        self.dir_to_var(\"cloud\", \"kachery\")\n                    ),\n                    \"storage\": self.config.get(\n                        self.dir_to_var(\"storage\", \"kachery\")\n                    ),\n                    \"temp\": self.config.get(self.dir_to_var(\"temp\", \"kachery\")),\n                },\n                \"dlc_dirs\": {\n                    \"base\": self._dlc_base,\n                    \"project\": self.dlc_project_dir,\n                    \"video\": self.dlc_video_dir,\n                    \"output\": self.dlc_output_dir,\n                },\n                \"kachery_zone\": \"franklab.default\",\n            }\n        }\n\n    @property\n    def config(self) -&gt; dict:\n        self.load_config()\n        return self._config\n\n    @property\n    def base_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"base\"))\n\n    @property\n    def raw_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"raw\"))\n\n    @property\n    def analysis_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"analysis\"))\n\n    @property\n    def recording_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"recording\"))\n\n    @property\n    def sorting_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"sorting\"))\n\n    @property\n    def waveforms_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"waveforms\"))\n\n    @property\n    def temp_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"temp\"))\n\n    @property\n    def video_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"video\"))\n\n    @property\n    def debug_mode(self) -&gt; bool:\n        \"\"\"Returns True if debug_mode is set.\n\n        Supports skipping inserts for Dockerized development.\n        \"\"\"\n        return self._debug_mode\n\n    @property\n    def test_mode(self) -&gt; bool:\n        \"\"\"Returns True if test_mode is set.\n\n        Required for pytests to run without prompts.\"\"\"\n        return self._test_mode\n\n    @property\n    def dlc_project_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"project\", \"dlc\"))\n\n    @property\n    def dlc_video_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"video\", \"dlc\"))\n\n    @property\n    def dlc_output_dir(self) -&gt; str:\n        return self.config.get(self.dir_to_var(\"output\", \"dlc\"))\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.__init__", "title": "<code>__init__(base_dir=None, **kwargs)</code>", "text": "<p>Initializes a new instance of the class.</p> <p>Parameters:</p> Name Type Description Default <code>base_dir</code> <code>str</code> <p>The base directory.</p> <code>None</code> <p>Attributes:</p> Name Type Description <code>supplied_base_dir (str)</code> <p>The base directory passed to the class.</p> <code>config_defaults (dict)</code> <p>Default settings for the config.</p> <code>relative_dirs (dict)</code> <p>Relative dirs for each prefix (spyglass, kachery, dlc). Relative to respective base_dir. Created on init.</p> <code>dj_defaults (dict)</code> <p>Default settings for datajoint.</p> <code>env_defaults (dict)</code> <p>Default settings for environment variables.</p> <code>_config (dict)</code> <p>Cached config settings.</p> <code>_debug_mode (bool)</code> <p>True if debug_mode is set. Supports skipping known bugs in test env.</p> <code>_test_mode (bool)</code> <p>True if test_mode is set. Required for pytests to run without prompts.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def __init__(self, base_dir: str = None, **kwargs):\n    \"\"\"\n    Initializes a new instance of the class.\n\n    Parameters\n    ----------\n    base_dir (str)\n        The base directory.\n\n    Attributes\n    ----------\n    supplied_base_dir (str)\n        The base directory passed to the class.\n    config_defaults (dict)\n        Default settings for the config.\n    relative_dirs (dict)\n        Relative dirs for each prefix (spyglass, kachery, dlc). Relative\n        to respective base_dir. Created on init.\n    dj_defaults (dict)\n        Default settings for datajoint.\n    env_defaults (dict)\n        Default settings for environment variables.\n    _config (dict)\n        Cached config settings.\n    _debug_mode (bool)\n        True if debug_mode is set. Supports skipping known bugs in test env.\n    _test_mode (bool)\n        True if test_mode is set. Required for pytests to run without\n        prompts.\n    \"\"\"\n    self.supplied_base_dir = base_dir\n    self._config = dict()\n    self.config_defaults = dict(prepopulate=True)\n    self._debug_mode = kwargs.get(\"debug_mode\", False)\n    self._test_mode = kwargs.get(\"test_mode\", False)\n    self._dlc_base = None\n    self.load_failed = False\n\n    self.relative_dirs = {\n        # {PREFIX}_{KEY}_DIR, default dir relative to base_dir\n        \"spyglass\": {\n            \"raw\": \"raw\",\n            \"analysis\": \"analysis\",\n            \"recording\": \"recording\",\n            \"sorting\": \"spikesorting\",\n            \"waveforms\": \"waveforms\",\n            \"temp\": \"tmp\",\n            \"video\": \"video\",\n        },\n        \"kachery\": {\n            \"cloud\": \".kachery-cloud\",\n            \"storage\": \"kachery_storage\",\n            \"temp\": \"tmp\",\n        },\n        \"dlc\": {\n            \"project\": \"projects\",\n            \"video\": \"video\",\n            \"output\": \"output\",\n        },\n    }\n    self.dj_defaults = {\n        \"database.host\": kwargs.get(\"database_host\", \"lmf-db.cin.ucsf.edu\"),\n        \"database.user\": kwargs.get(\"database_user\"),\n        \"database.port\": kwargs.get(\"database_port\", 3306),\n        \"database.use_tls\": kwargs.get(\"database_use_tls\", True),\n        \"filepath_checksum_size_limit\": 1 * 1024**3,\n        \"enable_python_native_blobs\": True,\n    }\n    self.env_defaults = {\n        \"FIGURL_CHANNEL\": \"franklab2\",\n        \"DJ_SUPPORT_FILEPATH_MANAGEMENT\": \"TRUE\",\n        \"KACHERY_CLOUD_EPHEMERAL\": \"TRUE\",\n        \"HD5_USE_FILE_LOCKING\": \"FALSE\",\n    }\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.load_config", "title": "<code>load_config(base_dir=None, force_reload=False, on_startup=False, **kwargs)</code>", "text": "<p>Loads the configuration settings for the object.</p> <p>Order of precedence, where X is base, raw, analysis, etc.:     1. SpyglassConfig(base_dir=\"string\") for base dir only     2. dj.config['custom']['{spyglass/kachery}dirs']['X']     3. os.environ['{SPYGLASS/KACHERY}_DIR']     4. resolved_base_dir/X for non-base dirs</p> <p>Parameters:</p> Name Type Description Default <code>base_dir</code> <p>Optional. Default None. The base directory. If not provided, will use the env variable or existing config.</p> <code>None</code> <code>force_reload</code> <p>Optional. Default False. Default skip load if already completed.</p> <code>False</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>If base_dir is not set in either dj.config or os.environ.</p> <p>Returns:</p> Type Description <code>dict</code> <p>list of relative_dirs and other settings (e.g., prepopulate).</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def load_config(\n    self,\n    base_dir=None,\n    force_reload=False,\n    on_startup: bool = False,\n    **kwargs,\n):\n    \"\"\"\n    Loads the configuration settings for the object.\n\n    Order of precedence, where X is base, raw, analysis, etc.:\n        1. SpyglassConfig(base_dir=\"string\") for base dir only\n        2. dj.config['custom']['{spyglass/kachery}_dirs']['X']\n        3. os.environ['{SPYGLASS/KACHERY}_{X}_DIR']\n        4. resolved_base_dir/X for non-base dirs\n\n    Parameters\n    ----------\n    base_dir: str\n        Optional. Default None. The base directory. If not provided, will\n        use the env variable or existing config.\n    force_reload: bool\n        Optional. Default False. Default skip load if already completed.\n\n    Raises\n    ------\n    ValueError\n        If base_dir is not set in either dj.config or os.environ.\n\n    Returns\n    -------\n    dict\n        list of relative_dirs and other settings (e.g., prepopulate).\n    \"\"\"\n    if not force_reload and self._config:\n        return self._config\n\n    dj_custom = dj.config.get(\"custom\", {})\n    dj_spyglass = dj_custom.get(\"spyglass_dirs\", {})\n    dj_kachery = dj_custom.get(\"kachery_dirs\", {})\n    dj_dlc = dj_custom.get(\"dlc_dirs\", {})\n\n    self._debug_mode = dj_custom.get(\"debug_mode\", False)\n    self._test_mode = kwargs.get(\"test_mode\") or dj_custom.get(\n        \"test_mode\", False\n    )\n\n    resolved_base = (\n        base_dir\n        or self.supplied_base_dir\n        or dj_spyglass.get(\"base\")\n        or os.environ.get(\"SPYGLASS_BASE_DIR\")\n    )\n\n    if resolved_base and not Path(resolved_base).exists():\n        resolved_base = Path(resolved_base).expanduser()\n    if not resolved_base or not Path(resolved_base).exists():\n        if not on_startup:  # Only warn if not on startup\n            logger.error(\n                f\"Could not find SPYGLASS_BASE_DIR: {resolved_base}\"\n                + \"\\n\\tCheck dj.config['custom']['spyglass_dirs']['base']\"\n                + \"\\n\\tand os.environ['SPYGLASS_BASE_DIR']\"\n            )\n        self.load_failed = True\n        return\n\n    self._dlc_base = (\n        dj_dlc.get(\"base\")\n        or os.environ.get(\"DLC_BASE_DIR\")\n        or os.environ.get(\"DLC_PROJECT_PATH\", \"\").split(\"projects\")[0]\n        or str(Path(resolved_base) / \"deeplabcut\")\n    )\n    Path(self._dlc_base).mkdir(exist_ok=True)\n\n    config_dirs = {\"SPYGLASS_BASE_DIR\": str(resolved_base)}\n    for prefix, dirs in self.relative_dirs.items():\n        this_base = self._dlc_base if prefix == \"dlc\" else resolved_base\n        for dir, dir_str in dirs.items():\n            dir_env_fmt = self.dir_to_var(dir=dir, dir_type=prefix)\n\n            env_loc = (  # Ignore env vars if base was passed to func\n                os.environ.get(dir_env_fmt)\n                if not self.supplied_base_dir\n                else None\n            )\n\n            source_config = (\n                dj_dlc\n                if prefix == \"dlc\"\n                else dj_kachery if prefix == \"kachery\" else dj_spyglass\n            )\n            dir_location = (\n                source_config.get(dir)\n                or env_loc\n                or str(Path(this_base) / dir_str)\n            ).replace('\"', \"\")\n\n            config_dirs.update({dir_env_fmt: str(dir_location)})\n\n    kachery_zone_dict = {\n        \"KACHERY_ZONE\": (\n            os.environ.get(\"KACHERY_ZONE\")\n            or dj.config.get(\"custom\", {}).get(\"kachery_zone\")\n            or \"franklab.default\"\n        )\n    }\n\n    loaded_env = self._load_env_vars()\n    self._set_env_with_dict(\n        {**config_dirs, **kachery_zone_dict, **loaded_env}\n    )\n    self._mkdirs_from_dict_vals(config_dirs)\n\n    self._config = dict(\n        debug_mode=self._debug_mode,\n        test_mode=self._test_mode,\n        **self.config_defaults,\n        **config_dirs,\n        **kachery_zone_dict,\n        **loaded_env,\n    )\n\n    self._set_dj_config_stores()\n\n    return self._config\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.dir_to_var", "title": "<code>dir_to_var(dir, dir_type='spyglass')</code>", "text": "<p>Converts a dir string to an env variable name.</p> Source code in <code>src/spyglass/settings.py</code> <pre><code>def dir_to_var(self, dir: str, dir_type: str = \"spyglass\"):\n    \"\"\"Converts a dir string to an env variable name.\"\"\"\n    return f\"{dir_type.upper()}_{dir.upper()}_DIR\"\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.save_dj_config", "title": "<code>save_dj_config(save_method='global', output_filename=None, base_dir=None, set_password=True, **kwargs)</code>", "text": "<p>Set the dj.config parameters, set password, and save config to file.</p> <p>Parameters:</p> Name Type Description Default <code>save_method</code> <code>(local, 'global', custom)</code> <p>The method to use to save the config. If either 'local' or 'global', datajoint builtins will be used to save.</p> <code>'local'</code> <code>output_filename</code> <code>str or Path</code> <p>Default to datajoint global config. If save_method = 'custom', name of file to generate. Must end in either be either yaml or json.</p> <code>None</code> <code>base_dir</code> <code>str</code> <p>The base directory. If not provided, will default to the env var</p> <code>None</code> <code>set_password</code> <code>bool</code> <p>Default True. Set the database password.</p> <code>True</code> <code>kwargs</code> <p>Any other valid datajoint configuration parameters, including database_user, database_password, database_host, database_port, etc. Note: python will raise error for params with <code>.</code> in name, so use underscores instead.</p> <code>{}</code> Source code in <code>src/spyglass/settings.py</code> <pre><code>def save_dj_config(\n    self,\n    save_method: str = \"global\",\n    output_filename: str = None,\n    base_dir=None,\n    set_password=True,\n    **kwargs,\n):\n    \"\"\"Set the dj.config parameters, set password, and save config to file.\n\n    Parameters\n    ----------\n    save_method : {'local', 'global', 'custom'}, optional\n        The method to use to save the config. If either 'local' or 'global',\n        datajoint builtins will be used to save.\n    output_filename : str or Path, optional\n        Default to datajoint global config. If save_method = 'custom', name\n        of file to generate. Must end in either be either yaml or json.\n    base_dir : str, optional\n        The base directory. If not provided, will default to the env var\n    set_password : bool, optional\n        Default True. Set the database password.\n    kwargs: dict, optional\n        Any other valid datajoint configuration parameters, including\n        database_user, database_password, database_host, database_port, etc.\n        Note: python will raise error for params with `.` in name, so use\n        underscores instead.\n    \"\"\"\n    if base_dir:\n        self.load_config(\n            base_dir=base_dir, force_reload=True, on_startup=False\n        )\n\n    if output_filename:\n        save_method = \"custom\"\n        path = Path(output_filename).expanduser()  # Expand ~\n        filepath = path if path.is_absolute() else path.absolute()\n        filepath.parent.mkdir(exist_ok=True, parents=True)\n        filepath = (\n            filepath.with_suffix(\".json\")  # ensure suffix, default json\n            if filepath.suffix not in [\".json\", \".yaml\"]\n            else filepath\n        )\n    elif save_method == \"local\":\n        filepath = Path(\".\") / dj.settings.LOCALCONFIG\n    elif save_method == \"global\":\n        filepath = Path(\"~\").expanduser() / dj.settings.GLOBALCONFIG\n    else:\n        raise ValueError(\n            \"For save_dj_config, either (a) save_method must be 'local' \"\n            + \" or 'global' or (b) must provide custom output_filename.\"\n        )\n\n    dj.config.update(self._generate_dj_config(**kwargs))\n\n    if set_password:\n        try:\n            dj.set_password()\n        except OperationalError as e:\n            warnings.warn(f\"Database connection issues. Wrong pass?\\n\\t{e}\")\n\n    user_warn = (\n        f\"Replace existing file? {filepath.resolve()}\\n\\t\"\n        + \"\\n\\t\".join([f\"{k}: {v}\" for k, v in config.items()])\n        + \"\\n\"\n    )\n\n    if (\n        not self.test_mode\n        and filepath.exists()\n        and dj.utils.user_choice(user_warn)[0] != \"y\"\n    ):\n        return\n\n    if save_method == \"global\":\n        dj.config.save_global(verbose=True)\n        return\n\n    if save_method == \"local\":\n        dj.config.save_local(verbose=True)\n        return\n\n    with open(filepath, \"w\") as outfile:\n        if filepath.suffix == \".yaml\":\n            yaml.dump(dj.config._conf, outfile, default_flow_style=False)\n        else:\n            json.dump(dj.config._conf, outfile, indent=2)\n        logger.info(f\"Saved config to {filepath}\")\n</code></pre>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.debug_mode", "title": "<code>debug_mode: bool</code>  <code>property</code>", "text": "<p>Returns True if debug_mode is set.</p> <p>Supports skipping inserts for Dockerized development.</p>"}, {"location": "api/settings/#src.spyglass.settings.SpyglassConfig.test_mode", "title": "<code>test_mode: bool</code>  <code>property</code>", "text": "<p>Returns True if test_mode is set.</p> <p>Required for pytests to run without prompts.</p>"}, {"location": "api/cli/cli/", "title": "cli.py", "text": ""}, {"location": "api/common/common_behav/", "title": "common_behav.py", "text": ""}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource", "title": "<code>PositionSource</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass PositionSource(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Session\n    -&gt; IntervalList\n    ---\n    source: varchar(200)             # source of data (e.g., trodes, dlc)\n    import_file_name: varchar(2000)  # path to import file if importing\n    \"\"\"\n\n    class SpatialSeries(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        id = 0 : int unsigned            # index of spatial series\n        ---\n        name=null: varchar(32)       # name of spatial series\n        \"\"\"\n\n    def populate(self, keys=None):\n        \"\"\"Insert position source data from NWB file.\n\n        WARNING: populate method on Manual table is not protected by transaction\n                protections like other DataJoint tables.\n        \"\"\"\n        if not isinstance(keys, list):\n            keys = [keys]\n        if isinstance(keys[0], (dj.Table, dj.expression.QueryExpression)):\n            keys = [k for tbl in keys for k in tbl.fetch(\"KEY\", as_dict=True)]\n        for key in keys:\n            nwb_file_name = key.get(\"nwb_file_name\")\n            if not nwb_file_name:\n                raise ValueError(\n                    \"PositionSource.populate is an alias for a non-computed table \"\n                    + \"and must be passed a key with nwb_file_name\"\n                )\n            self.insert_from_nwbfile(nwb_file_name, skip_duplicates=True)\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwb_file_name, skip_duplicates=False) -&gt; None:\n        \"\"\"Add intervals to ItervalList and PositionSource.\n\n        Given an NWB file name, get the spatial series and interval lists from\n        the file, add the interval lists to the IntervalList table, and\n        populate the RawPosition table if possible.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        \"\"\"\n        nwbf = get_nwb_file(nwb_file_name)\n        all_pos = get_all_spatial_series(nwbf, verbose=True)\n        sess_key = Nwbfile.get_file_key(nwb_file_name)\n        src_key = dict(**sess_key, source=\"trodes\", import_file_name=\"\")\n\n        if all_pos is None:\n            return\n\n        sources = []\n        intervals = []\n        spat_series = []\n\n        for epoch, epoch_list in all_pos.items():\n            ind_key = dict(interval_list_name=cls.get_pos_interval_name(epoch))\n\n            sources.append(dict(**src_key, **ind_key))\n            intervals.append(\n                dict(\n                    **sess_key,\n                    **ind_key,\n                    valid_times=epoch_list[0][\"valid_times\"],\n                    pipeline=\"position\",\n                )\n            )\n\n            for index, pdict in enumerate(epoch_list):\n                spat_series.append(\n                    dict(\n                        **sess_key,\n                        **ind_key,\n                        id=index,\n                        name=pdict.get(\"name\"),\n                    )\n                )\n\n        with cls.connection.transaction:\n            IntervalList.insert(intervals, skip_duplicates=skip_duplicates)\n            cls.insert(sources, skip_duplicates=skip_duplicates)\n            cls.SpatialSeries.insert(\n                spat_series, skip_duplicates=skip_duplicates\n            )\n\n        # make map from epoch intervals to position intervals\n        populate_position_interval_map_session(nwb_file_name)\n\n    @staticmethod\n    def get_pos_interval_name(epoch_num: int) -&gt; str:\n        \"\"\"Return string of the interval name from the epoch number.\n\n        Parameters\n        ----------\n        epoch_num : int\n            Input epoch number\n\n        Returns\n        -------\n        str\n            Position interval name (e.g., pos 2 valid times)\n        \"\"\"\n        try:\n            int(epoch_num)\n        except ValueError:\n            raise ValueError(\n                f\"Epoch number must must be an integer. Received: {epoch_num}\"\n            )\n        return f\"pos {epoch_num} valid times\"\n\n    @staticmethod\n    def _is_valid_name(name) -&gt; bool:\n        return name.startswith(\"pos \") and name.endswith(\" valid times\")\n\n    @staticmethod\n    def get_epoch_num(name: str) -&gt; int:\n        \"\"\"Return the epoch number from the interval name.\n\n        Parameters\n        ----------\n        name : str\n            Name of position interval (e.g., pos epoch 1 index 2 valid times)\n\n        Returns\n        -------\n        int\n            epoch number\n        \"\"\"\n        if not PositionSource._is_valid_name(name):\n            raise ValueError(f\"Invalid interval name: {name}\")\n        return int(name.replace(\"pos \", \"\").replace(\" valid times\", \"\"))\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.populate", "title": "<code>populate(keys=None)</code>", "text": "<p>Insert position source data from NWB file.</p> <p>WARNING: populate method on Manual table is not protected by transaction         protections like other DataJoint tables.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def populate(self, keys=None):\n    \"\"\"Insert position source data from NWB file.\n\n    WARNING: populate method on Manual table is not protected by transaction\n            protections like other DataJoint tables.\n    \"\"\"\n    if not isinstance(keys, list):\n        keys = [keys]\n    if isinstance(keys[0], (dj.Table, dj.expression.QueryExpression)):\n        keys = [k for tbl in keys for k in tbl.fetch(\"KEY\", as_dict=True)]\n    for key in keys:\n        nwb_file_name = key.get(\"nwb_file_name\")\n        if not nwb_file_name:\n            raise ValueError(\n                \"PositionSource.populate is an alias for a non-computed table \"\n                + \"and must be passed a key with nwb_file_name\"\n            )\n        self.insert_from_nwbfile(nwb_file_name, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwb_file_name, skip_duplicates=False)</code>  <code>classmethod</code>", "text": "<p>Add intervals to ItervalList and PositionSource.</p> <p>Given an NWB file name, get the spatial series and interval lists from the file, add the interval lists to the IntervalList table, and populate the RawPosition table if possible.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwb_file_name, skip_duplicates=False) -&gt; None:\n    \"\"\"Add intervals to ItervalList and PositionSource.\n\n    Given an NWB file name, get the spatial series and interval lists from\n    the file, add the interval lists to the IntervalList table, and\n    populate the RawPosition table if possible.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    \"\"\"\n    nwbf = get_nwb_file(nwb_file_name)\n    all_pos = get_all_spatial_series(nwbf, verbose=True)\n    sess_key = Nwbfile.get_file_key(nwb_file_name)\n    src_key = dict(**sess_key, source=\"trodes\", import_file_name=\"\")\n\n    if all_pos is None:\n        return\n\n    sources = []\n    intervals = []\n    spat_series = []\n\n    for epoch, epoch_list in all_pos.items():\n        ind_key = dict(interval_list_name=cls.get_pos_interval_name(epoch))\n\n        sources.append(dict(**src_key, **ind_key))\n        intervals.append(\n            dict(\n                **sess_key,\n                **ind_key,\n                valid_times=epoch_list[0][\"valid_times\"],\n                pipeline=\"position\",\n            )\n        )\n\n        for index, pdict in enumerate(epoch_list):\n            spat_series.append(\n                dict(\n                    **sess_key,\n                    **ind_key,\n                    id=index,\n                    name=pdict.get(\"name\"),\n                )\n            )\n\n    with cls.connection.transaction:\n        IntervalList.insert(intervals, skip_duplicates=skip_duplicates)\n        cls.insert(sources, skip_duplicates=skip_duplicates)\n        cls.SpatialSeries.insert(\n            spat_series, skip_duplicates=skip_duplicates\n        )\n\n    # make map from epoch intervals to position intervals\n    populate_position_interval_map_session(nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.get_pos_interval_name", "title": "<code>get_pos_interval_name(epoch_num)</code>  <code>staticmethod</code>", "text": "<p>Return string of the interval name from the epoch number.</p> <p>Parameters:</p> Name Type Description Default <code>epoch_num</code> <code>int</code> <p>Input epoch number</p> required <p>Returns:</p> Type Description <code>str</code> <p>Position interval name (e.g., pos 2 valid times)</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@staticmethod\ndef get_pos_interval_name(epoch_num: int) -&gt; str:\n    \"\"\"Return string of the interval name from the epoch number.\n\n    Parameters\n    ----------\n    epoch_num : int\n        Input epoch number\n\n    Returns\n    -------\n    str\n        Position interval name (e.g., pos 2 valid times)\n    \"\"\"\n    try:\n        int(epoch_num)\n    except ValueError:\n        raise ValueError(\n            f\"Epoch number must must be an integer. Received: {epoch_num}\"\n        )\n    return f\"pos {epoch_num} valid times\"\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.PositionSource.get_epoch_num", "title": "<code>get_epoch_num(name)</code>  <code>staticmethod</code>", "text": "<p>Return the epoch number from the interval name.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>Name of position interval (e.g., pos epoch 1 index 2 valid times)</p> required <p>Returns:</p> Type Description <code>int</code> <p>epoch number</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@staticmethod\ndef get_epoch_num(name: str) -&gt; int:\n    \"\"\"Return the epoch number from the interval name.\n\n    Parameters\n    ----------\n    name : str\n        Name of position interval (e.g., pos epoch 1 index 2 valid times)\n\n    Returns\n    -------\n    int\n        epoch number\n    \"\"\"\n    if not PositionSource._is_valid_name(name):\n        raise ValueError(f\"Invalid interval name: {name}\")\n    return int(name.replace(\"pos \", \"\").replace(\" valid times\", \"\"))\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition", "title": "<code>RawPosition</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition--notes", "title": "Notes", "text": "<p>The position timestamps come from: .pos_cameraHWSync.dat. If PTP is not used, the position timestamps are inferred by finding the closest timestamps from the neural recording via the trodes time.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass RawPosition(SpyglassMixin, dj.Imported):\n    \"\"\"\n\n    Notes\n    -----\n    The position timestamps come from: .pos_cameraHWSync.dat.\n    If PTP is not used, the position timestamps are inferred by finding the\n    closest timestamps from the neural recording via the trodes time.\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionSource\n    \"\"\"\n\n    class PosObject(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        -&gt; PositionSource.SpatialSeries.proj('id')\n        ---\n        raw_position_object_id: varchar(40) # id of spatial series in NWB file\n        \"\"\"\n\n        _nwb_table = Nwbfile\n\n        def fetch1_dataframe(self):\n            id_rp = [(n[\"id\"], n[\"raw_position\"]) for n in self.fetch_nwb()]\n\n            if len(set(rp.interval for _, rp in id_rp)) &gt; 1:\n                logger.warn(\"Loading DataFrame with multiple intervals.\")\n\n            df_list = [\n                pd.DataFrame(\n                    data=rp.data,\n                    index=pd.Index(rp.timestamps, name=\"time\"),\n                    columns=self._get_column_names(rp, pos_id),\n                )\n                for pos_id, rp in id_rp\n            ]\n\n            return reduce(lambda x, y: pd.merge(x, y, on=\"time\"), df_list)\n\n        @staticmethod\n        def _get_column_names(rp, pos_id):\n            INDEX_ADJUST = 1  # adjust 0-index to 1-index (e.g., xloc0 -&gt; xloc1)\n            n_pos_dims = rp.data.shape[1]\n            column_names = [\n                (\n                    col  # use existing columns if already numbered\n                    if \"1\" in rp.description or \"2\" in rp.description\n                    # else number them by id\n                    else col + str(pos_id + INDEX_ADJUST)\n                )\n                for col in rp.description.split(\", \")\n            ]\n            if len(column_names) != n_pos_dims:\n                # if the string split didn't work, use default names\n                column_names = [\"x\", \"y\", \"z\"][:n_pos_dims]\n            return column_names\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        interval_list_name = key[\"interval_list_name\"]\n\n        nwbf = get_nwb_file(nwb_file_name)\n        indices = (PositionSource.SpatialSeries &amp; key).fetch(\"id\")\n\n        # incl_times = False -&gt; don't do extra processing for valid_times\n        spat_objs = get_all_spatial_series(nwbf, incl_times=False)[\n            PositionSource.get_epoch_num(interval_list_name)\n        ]\n\n        self.insert1(key)\n        self.PosObject.insert(\n            [\n                dict(\n                    nwb_file_name=nwb_file_name,\n                    interval_list_name=interval_list_name,\n                    id=index,\n                    raw_position_object_id=obj[\"raw_position_object_id\"],\n                )\n                for index, obj in enumerate(spat_objs)\n                if index in indices\n            ]\n        )\n\n    def fetch_nwb(self, *attrs, **kwargs) -&gt; list:\n        \"\"\"\n        Returns a condatenated list of nwb objects from RawPosition.PosObject\n        \"\"\"\n        return (\n            self.PosObject()\n            .restrict(self.restriction)  # Avoids fetch_nwb on whole table\n            .fetch_nwb(*attrs, **kwargs)\n        )\n\n    def fetch1_dataframe(self):\n        \"\"\"Returns a dataframe with all RawPosition.PosObject items.\n\n        Uses interval_list_name as column index.\n        \"\"\"\n        ret = {}\n\n        pos_obj_set = self.PosObject &amp; self.restriction\n        unique_intervals = set(pos_obj_set.fetch(\"interval_list_name\"))\n\n        for interval in unique_intervals:\n            ret[interval] = (\n                pos_obj_set &amp; {\"interval_list_name\": interval}\n            ).fetch1_dataframe()\n\n        if len(unique_intervals) == 1:\n            return next(iter(ret.values()))\n\n        return pd.concat(ret, axis=1)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>Returns a condatenated list of nwb objects from RawPosition.PosObject</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs) -&gt; list:\n    \"\"\"\n    Returns a condatenated list of nwb objects from RawPosition.PosObject\n    \"\"\"\n    return (\n        self.PosObject()\n        .restrict(self.restriction)  # Avoids fetch_nwb on whole table\n        .fetch_nwb(*attrs, **kwargs)\n    )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.RawPosition.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Returns a dataframe with all RawPosition.PosObject items.</p> <p>Uses interval_list_name as column index.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Returns a dataframe with all RawPosition.PosObject items.\n\n    Uses interval_list_name as column index.\n    \"\"\"\n    ret = {}\n\n    pos_obj_set = self.PosObject &amp; self.restriction\n    unique_intervals = set(pos_obj_set.fetch(\"interval_list_name\"))\n\n    for interval in unique_intervals:\n        ret[interval] = (\n            pos_obj_set &amp; {\"interval_list_name\": interval}\n        ).fetch1_dataframe()\n\n    if len(unique_intervals) == 1:\n        return next(iter(ret.values()))\n\n    return pd.concat(ret, axis=1)\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.StateScriptFile", "title": "<code>StateScriptFile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass StateScriptFile(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; TaskEpoch\n    ---\n    file_object_id: varchar(40)  # the object id of the file object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        \"\"\"Add a new row to the StateScriptFile table.\"\"\"\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n        associated_files = nwbf.processing.get(\n            \"associated_files\"\n        ) or nwbf.processing.get(\"associated files\")\n        if associated_files is None:\n            logger.info(\n                \"Unable to import StateScriptFile: no processing module named \"\n                + '\"associated_files\" found in {nwb_file_name}.'\n            )\n            return  # See #849\n\n        for associated_file_obj in associated_files.data_interfaces.values():\n            if not isinstance(\n                associated_file_obj, ndx_franklab_novela.AssociatedFiles\n            ):\n                logger.info(\n                    f\"Data interface {associated_file_obj.name} within \"\n                    + '\"associated_files\" processing module is not '\n                    + \"of expected type ndx_franklab_novela.AssociatedFiles\\n\"\n                )\n                return\n\n            # parse the task_epochs string\n            # TODO: update associated_file_obj.task_epochs to be an array of\n            # 1-based ints, not a comma-separated string of ints\n\n            epoch_list = associated_file_obj.task_epochs.split(\",\")\n            # only insert if this is the statescript file\n            logger.info(associated_file_obj.description)\n            if (\n                \"statescript\".upper() in associated_file_obj.description.upper()\n                or \"state_script\".upper()\n                in associated_file_obj.description.upper()\n                or \"state script\".upper()\n                in associated_file_obj.description.upper()\n            ):\n                # find the file associated with this epoch\n                if str(key[\"epoch\"]) in epoch_list:\n                    key[\"file_object_id\"] = associated_file_obj.object_id\n                    self.insert1(key)\n            else:\n                logger.info(\"not a statescript file\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.StateScriptFile.make", "title": "<code>make(key)</code>", "text": "<p>Add a new row to the StateScriptFile table.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def make(self, key):\n    \"\"\"Add a new row to the StateScriptFile table.\"\"\"\n    nwb_file_name = key[\"nwb_file_name\"]\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    nwbf = get_nwb_file(nwb_file_abspath)\n\n    associated_files = nwbf.processing.get(\n        \"associated_files\"\n    ) or nwbf.processing.get(\"associated files\")\n    if associated_files is None:\n        logger.info(\n            \"Unable to import StateScriptFile: no processing module named \"\n            + '\"associated_files\" found in {nwb_file_name}.'\n        )\n        return  # See #849\n\n    for associated_file_obj in associated_files.data_interfaces.values():\n        if not isinstance(\n            associated_file_obj, ndx_franklab_novela.AssociatedFiles\n        ):\n            logger.info(\n                f\"Data interface {associated_file_obj.name} within \"\n                + '\"associated_files\" processing module is not '\n                + \"of expected type ndx_franklab_novela.AssociatedFiles\\n\"\n            )\n            return\n\n        # parse the task_epochs string\n        # TODO: update associated_file_obj.task_epochs to be an array of\n        # 1-based ints, not a comma-separated string of ints\n\n        epoch_list = associated_file_obj.task_epochs.split(\",\")\n        # only insert if this is the statescript file\n        logger.info(associated_file_obj.description)\n        if (\n            \"statescript\".upper() in associated_file_obj.description.upper()\n            or \"state_script\".upper()\n            in associated_file_obj.description.upper()\n            or \"state script\".upper()\n            in associated_file_obj.description.upper()\n        ):\n            # find the file associated with this epoch\n            if str(key[\"epoch\"]) in epoch_list:\n                key[\"file_object_id\"] = associated_file_obj.object_id\n                self.insert1(key)\n        else:\n            logger.info(\"not a statescript file\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile", "title": "<code>VideoFile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile--notes", "title": "Notes", "text": "<p>The video timestamps come from: videoTimeStamps.cameraHWSync if PTP is used. If PTP is not used, the video timestamps come from videoTimeStamps.cameraHWFrameCount .</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@schema\nclass VideoFile(SpyglassMixin, dj.Imported):\n    \"\"\"\n\n    Notes\n    -----\n    The video timestamps come from: videoTimeStamps.cameraHWSync if PTP is\n    used. If PTP is not used, the video timestamps come from\n    videoTimeStamps.cameraHWFrameCount .\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; TaskEpoch\n    video_file_num = 0: int\n    ---\n    camera_name: varchar(80)\n    video_file_object_id: varchar(40)  # the object id of the file object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        self._no_transaction_make(key)\n\n    def _no_transaction_make(self, key, verbose=True):\n        if not self.connection.in_transaction:\n            self.populate(key)\n            return\n\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        videos = get_data_interface(\n            nwbf, \"video\", pynwb.behavior.BehavioralEvents\n        )\n\n        if videos is None:\n            logger.warn(f\"No video data interface found in {nwb_file_name}\\n\")\n            return\n        else:\n            videos = videos.time_series\n\n        # get the interval for the current TaskEpoch\n        interval_list_name = (TaskEpoch() &amp; key).fetch1(\"interval_list_name\")\n        valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        cam_device_str = r\"camera_device (\\d+)\"\n        is_found = False\n        for ind, video in enumerate(videos.values()):\n            if isinstance(video, pynwb.image.ImageSeries):\n                video = [video]\n            for video_obj in video:\n                # check to see if the times for this video_object are largely\n                # overlapping with the task epoch times\n\n                if len(\n                    interval_list_contains(valid_times, video_obj.timestamps)\n                    &gt; 0.9 * len(video_obj.timestamps)\n                ):\n                    nwb_cam_device = video_obj.device.name\n                    # returns whatever was captured in the first group (within the parentheses) of the regular expression -- in this case, 0\n                    key[\"video_file_num\"] = int(\n                        re.match(cam_device_str, nwb_cam_device)[1]\n                    )\n                    camera_name = video_obj.device.camera_name\n                    if CameraDevice &amp; {\"camera_name\": camera_name}:\n                        key[\"camera_name\"] = video_obj.device.camera_name\n                    else:\n                        raise KeyError(\n                            f\"No camera with camera_name: {camera_name} found \"\n                            + \"in CameraDevice table.\"\n                        )\n                    key[\"video_file_object_id\"] = video_obj.object_id\n                    self.insert1(key)\n                    is_found = True\n\n        if not is_found and verbose:\n            logger.info(\n                f\"No video found corresponding to file {nwb_file_name}, \"\n                + f\"epoch {interval_list_name}\"\n            )\n\n    @classmethod\n    def update_entries(cls, restrict={}):\n        existing_entries = (cls &amp; restrict).fetch(\"KEY\")\n        for row in existing_entries:\n            if (cls &amp; row).fetch1(\"camera_name\"):\n                continue\n            video_nwb = (cls &amp; row).fetch_nwb()[0]\n            if len(video_nwb) != 1:\n                raise ValueError(\n                    f\"Expecting 1 video file per entry. {len(video_nwb)} found\"\n                )\n            row[\"camera_name\"] = video_nwb[0][\"video_file\"].device.camera_name\n            cls.update1(row=row)\n\n    @classmethod\n    def get_abs_path(cls, key: Dict):\n        \"\"\"Return the absolute path for a stored video file given a key.\n\n        Key must include the nwb_file_name and epoch number. The\n        SPYGLASS_VIDEO_DIR environment variable must be set.\n\n        Parameters\n        ----------\n        key : dict\n            dictionary with nwb_file_name and epoch as keys\n\n        Returns\n        -------\n        nwb_video_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        video_path_obj = pathlib.Path(video_dir)\n        video_info = (cls &amp; key).fetch1()\n        nwb_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n        nwbf = get_nwb_file(nwb_path)\n        nwb_video = nwbf.objects[video_info[\"video_file_object_id\"]]\n        video_filename = nwb_video.name\n        # see if the file exists and is stored in the base analysis dir\n        nwb_video_file_abspath = pathlib.Path(video_path_obj / video_filename)\n        if nwb_video_file_abspath.exists():\n            return nwb_video_file_abspath.as_posix()\n        else:\n            raise FileNotFoundError(\n                f\"video file with filename: {video_filename} \"\n                f\"does not exist in {video_path_obj}/\"\n            )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.VideoFile.get_abs_path", "title": "<code>get_abs_path(key)</code>  <code>classmethod</code>", "text": "<p>Return the absolute path for a stored video file given a key.</p> <p>Key must include the nwb_file_name and epoch number. The SPYGLASS_VIDEO_DIR environment variable must be set.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary with nwb_file_name and epoch as keys</p> required <p>Returns:</p> Name Type Description <code>nwb_video_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>@classmethod\ndef get_abs_path(cls, key: Dict):\n    \"\"\"Return the absolute path for a stored video file given a key.\n\n    Key must include the nwb_file_name and epoch number. The\n    SPYGLASS_VIDEO_DIR environment variable must be set.\n\n    Parameters\n    ----------\n    key : dict\n        dictionary with nwb_file_name and epoch as keys\n\n    Returns\n    -------\n    nwb_video_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    video_path_obj = pathlib.Path(video_dir)\n    video_info = (cls &amp; key).fetch1()\n    nwb_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n    nwbf = get_nwb_file(nwb_path)\n    nwb_video = nwbf.objects[video_info[\"video_file_object_id\"]]\n    video_filename = nwb_video.name\n    # see if the file exists and is stored in the base analysis dir\n    nwb_video_file_abspath = pathlib.Path(video_path_obj / video_filename)\n    if nwb_video_file_abspath.exists():\n        return nwb_video_file_abspath.as_posix()\n    else:\n        raise FileNotFoundError(\n            f\"video file with filename: {video_filename} \"\n            f\"does not exist in {video_path_obj}/\"\n        )\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.convert_epoch_interval_name_to_position_interval_name", "title": "<code>convert_epoch_interval_name_to_position_interval_name(key, populate_missing=True)</code>", "text": "<p>Converts IntervalList key to the corresponding position interval name.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Lookup key</p> required <code>populate_missing</code> <code>bool</code> <p>Whether to populate PositionIntervalMap for the key if missing. Should be False if this function is used inside of another populate call. Defaults to True</p> <code>True</code> <p>Returns:</p> Name Type Description <code>position_interval_name</code> <code>str</code> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def convert_epoch_interval_name_to_position_interval_name(\n    key: dict, populate_missing: bool = True\n) -&gt; str:\n    \"\"\"Converts IntervalList key to the corresponding position interval name.\n\n    Parameters\n    ----------\n    key : dict\n        Lookup key\n    populate_missing: bool\n        Whether to populate PositionIntervalMap for the key if missing. Should\n        be False if this function is used inside of another populate call.\n        Defaults to True\n\n    Returns\n    -------\n    position_interval_name : str\n    \"\"\"\n    # get the interval list name if given epoch but not interval list name\n    if \"interval_list_name\" not in key and \"epoch\" in key:\n        key[\"interval_list_name\"] = get_interval_list_name_from_epoch(\n            key[\"nwb_file_name\"], key[\"epoch\"]\n        )\n\n    pos_query = PositionIntervalMap &amp; key\n\n    if len(pos_query) == 0:\n        if populate_missing:\n            PositionIntervalMap()._no_transaction_make(key)\n        else:\n            raise KeyError(\n                f\"{key} must be populated in the PositionIntervalMap table \"\n                + \"prior to your current populate call\"\n            )\n\n    if len(pos_query) == 0:\n        logger.info(f\"No position intervals found for {key}\")\n        return []\n\n    if len(pos_query) == 1:\n        return pos_query.fetch1(\"position_interval_name\")\n</code></pre>"}, {"location": "api/common/common_behav/#src.spyglass.common.common_behav.get_interval_list_name_from_epoch", "title": "<code>get_interval_list_name_from_epoch(nwb_file_name, epoch)</code>", "text": "<p>Returns the interval list name for the given epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required <code>epoch</code> <code>int</code> <p>The epoch number.</p> required <p>Returns:</p> Name Type Description <code>interval_list_name</code> <code>str</code> <p>The interval list name.</p> Source code in <code>src/spyglass/common/common_behav.py</code> <pre><code>def get_interval_list_name_from_epoch(nwb_file_name: str, epoch: int) -&gt; str:\n    \"\"\"Returns the interval list name for the given epoch.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    epoch : int\n        The epoch number.\n\n    Returns\n    -------\n    interval_list_name : str\n        The interval list name.\n    \"\"\"\n    interval_names = (\n        TaskEpoch &amp; {\"nwb_file_name\": nwb_file_name, \"epoch\": epoch}\n    ).fetch(\"interval_list_name\")\n\n    if len(interval_names) != 1:\n        logger.info(\n            f\"Found {len(interval_names)} interval list names found for \"\n            + f\"{nwb_file_name} epoch {epoch}\"\n        )\n        return None\n\n    return interval_names[0]\n</code></pre>"}, {"location": "api/common/common_device/", "title": "common_device.py", "text": ""}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice", "title": "<code>DataAcquisitionDevice</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass DataAcquisitionDevice(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    data_acquisition_device_name: varchar(80)\n    ---\n    -&gt; DataAcquisitionDeviceSystem\n    -&gt; DataAcquisitionDeviceAmplifier\n    adc_circuit = NULL: varchar(2000)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, config):\n        \"\"\"Insert data acquisition devices from an NWB file.\n\n        Note that this does not link the DataAcquisitionDevices with a Session.\n        For that, see DataAcquisitionDeviceList.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n        \"\"\"\n        _, ndx_devices, _ = cls.get_all_device_names(nwbf, config)\n\n        for device_name in ndx_devices:\n            new_device_dict = dict()\n\n            # read device properties into new_device_dict from PyNWB extension\n            # device object\n            nwb_device_obj = ndx_devices[device_name]\n\n            name = nwb_device_obj.name\n            adc_circuit = nwb_device_obj.adc_circuit\n\n            # transform system value. check if value is in DB. if not, prompt\n            # user to add an entry or cancel.\n            system = cls._add_system(nwb_device_obj.system)\n\n            # transform amplifier value. check if value is in DB. if not, prompt\n            # user to add an entry or cancel.\n            amplifier = cls._add_amplifier(nwb_device_obj.amplifier)\n\n            # standardize how Intan is represented in the database\n            if adc_circuit.title() == \"Intan\":\n                adc_circuit = \"Intan\"\n\n            new_device_dict[\"data_acquisition_device_name\"] = name\n            new_device_dict[\"data_acquisition_device_system\"] = system\n            new_device_dict[\"data_acquisition_device_amplifier\"] = amplifier\n            new_device_dict[\"adc_circuit\"] = adc_circuit\n\n            cls._add_device(new_device_dict)\n\n        if ndx_devices:\n            logger.info(\n                \"Inserted or referenced data acquisition device(s): \"\n                + f\"{ndx_devices.keys()}\"\n            )\n        else:\n            logger.warn(\"No conforming data acquisition device metadata found.\")\n\n    @classmethod\n    def get_all_device_names(cls, nwbf, config) -&gt; tuple:\n        \"\"\"\n        Return device names in the NWB file, after appending and overwriting by\n        the config file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : tuple\n            List of data acquisition object names found in the NWB file.\n        \"\"\"\n        # make a dict mapping device name to PyNWB device object for all devices\n        # in the NWB file that are of type ndx_franklab_novela.DataAcqDevice and\n        # thus have the required metadata\n        ndx_devices = {\n            device_obj.name: device_obj\n            for device_obj in nwbf.devices.values()\n            if isinstance(device_obj, ndx_franklab_novela.DataAcqDevice)\n        }\n\n        # make a list of device names that are associated with this NWB file\n        if \"DataAcquisitionDevice\" in config:\n            config_devices = [\n                device_dict[\"data_acquisition_device_name\"]\n                for device_dict in config[\"DataAcquisitionDevice\"]\n            ]\n        else:\n            config_devices = list()\n\n        all_device_names = set(ndx_devices.keys()).union(set(config_devices))\n\n        return all_device_names, ndx_devices, config_devices\n\n    @classmethod\n    def _add_device(cls, new_device_dict):\n        \"\"\"Ensure match between NWB file info &amp; database entry.\n\n        If no DataAcquisitionDevice with the given name exists in the database,\n        check whether the user wants to add a new entry instead of referencing\n        an existing entry. If so, return. If not, raise an exception.\n\n        Parameters\n        ----------\n        new_device_dict : dict\n            Dict of new device properties\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device to the database when prompted or\n            if the device properties from the NWB file do not match the\n            properties of the corresponding database entry.\n        \"\"\"\n        name = new_device_dict[\"data_acquisition_device_name\"]\n        all_values = DataAcquisitionDevice.fetch(\n            \"data_acquisition_device_name\"\n        ).tolist()\n        if prompt_insert(name=name, all_values=all_values):\n            cls.insert1(new_device_dict, skip_duplicates=True)\n            return\n\n        # Check if values provided match the values stored in the database\n        db_dict = (\n            DataAcquisitionDevice &amp; {\"data_acquisition_device_name\": name}\n        ).fetch1()\n        if db_dict != new_device_dict:\n            raise PopulateException(\n                \"Data acquisition device properties of PyNWB Device object \"\n                + f\"with name '{name}': {new_device_dict} do not match \"\n                f\"properties of the corresponding database entry: {db_dict}.\"\n            )\n\n    @classmethod\n    def _add_system(cls, system):\n        \"\"\"Check the system value. If not in the db, prompt user to add it.\n\n        This method also renames the system value \"MCU\" to \"SpikeGadgets\".\n\n        Parameters\n        ----------\n        system : str\n            The system value to check.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device system value to the database\n            when prompted.\n\n        Returns\n        -------\n        system : str\n            The system value that was added to the database.\n        \"\"\"\n        if system == \"MCU\":\n            system = \"SpikeGadgets\"\n\n        all_values = DataAcquisitionDeviceSystem.fetch(\n            \"data_acquisition_device_system\"\n        ).tolist()\n        if prompt_insert(\n            name=system, all_values=all_values, table_type=\"system\"\n        ):\n            key = {\"data_acquisition_device_system\": system}\n            DataAcquisitionDeviceSystem.insert1(key, skip_duplicates=True)\n        return system\n\n    @classmethod\n    def _add_amplifier(cls, amplifier):\n        \"\"\"Check amplifier value. If not in db, prompt user to add.\n\n        Parameters\n        ----------\n        amplifier : str\n            The amplifier value to check.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a device amplifier value to the database\n            when prompted.\n\n        Returns\n        -------\n        amplifier : str\n            The amplifier value that was added to the database.\n        \"\"\"\n        # standardize how Intan is represented in the database\n        if amplifier.title() == \"Intan\":\n            amplifier = \"Intan\"\n\n        all_values = DataAcquisitionDeviceAmplifier.fetch(\n            \"data_acquisition_device_amplifier\"\n        ).tolist()\n        if prompt_insert(\n            name=amplifier, all_values=all_values, table_type=\"amplifier\"\n        ):\n            key = {\"data_acquisition_device_amplifier\": amplifier}\n            DataAcquisitionDeviceAmplifier.insert1(key, skip_duplicates=True)\n        return amplifier\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Insert data acquisition devices from an NWB file.</p> <p>Note that this does not link the DataAcquisitionDevices with a Session. For that, see DataAcquisitionDeviceList.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, config):\n    \"\"\"Insert data acquisition devices from an NWB file.\n\n    Note that this does not link the DataAcquisitionDevices with a Session.\n    For that, see DataAcquisitionDeviceList.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n    \"\"\"\n    _, ndx_devices, _ = cls.get_all_device_names(nwbf, config)\n\n    for device_name in ndx_devices:\n        new_device_dict = dict()\n\n        # read device properties into new_device_dict from PyNWB extension\n        # device object\n        nwb_device_obj = ndx_devices[device_name]\n\n        name = nwb_device_obj.name\n        adc_circuit = nwb_device_obj.adc_circuit\n\n        # transform system value. check if value is in DB. if not, prompt\n        # user to add an entry or cancel.\n        system = cls._add_system(nwb_device_obj.system)\n\n        # transform amplifier value. check if value is in DB. if not, prompt\n        # user to add an entry or cancel.\n        amplifier = cls._add_amplifier(nwb_device_obj.amplifier)\n\n        # standardize how Intan is represented in the database\n        if adc_circuit.title() == \"Intan\":\n            adc_circuit = \"Intan\"\n\n        new_device_dict[\"data_acquisition_device_name\"] = name\n        new_device_dict[\"data_acquisition_device_system\"] = system\n        new_device_dict[\"data_acquisition_device_amplifier\"] = amplifier\n        new_device_dict[\"adc_circuit\"] = adc_circuit\n\n        cls._add_device(new_device_dict)\n\n    if ndx_devices:\n        logger.info(\n            \"Inserted or referenced data acquisition device(s): \"\n            + f\"{ndx_devices.keys()}\"\n        )\n    else:\n        logger.warn(\"No conforming data acquisition device metadata found.\")\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.DataAcquisitionDevice.get_all_device_names", "title": "<code>get_all_device_names(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Return device names in the NWB file, after appending and overwriting by the config file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>tuple</code> <p>List of data acquisition object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef get_all_device_names(cls, nwbf, config) -&gt; tuple:\n    \"\"\"\n    Return device names in the NWB file, after appending and overwriting by\n    the config file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : tuple\n        List of data acquisition object names found in the NWB file.\n    \"\"\"\n    # make a dict mapping device name to PyNWB device object for all devices\n    # in the NWB file that are of type ndx_franklab_novela.DataAcqDevice and\n    # thus have the required metadata\n    ndx_devices = {\n        device_obj.name: device_obj\n        for device_obj in nwbf.devices.values()\n        if isinstance(device_obj, ndx_franklab_novela.DataAcqDevice)\n    }\n\n    # make a list of device names that are associated with this NWB file\n    if \"DataAcquisitionDevice\" in config:\n        config_devices = [\n            device_dict[\"data_acquisition_device_name\"]\n            for device_dict in config[\"DataAcquisitionDevice\"]\n        ]\n    else:\n        config_devices = list()\n\n    all_device_names = set(ndx_devices.keys()).union(set(config_devices))\n\n    return all_device_names, ndx_devices, config_devices\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.CameraDevice", "title": "<code>CameraDevice</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass CameraDevice(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    camera_name: varchar(80)\n    ---\n    meters_per_pixel = 0: float  # height / width of pixel in meters\n    manufacturer = \"\": varchar(2000)\n    model = \"\": varchar(2000)\n    lens = \"\": varchar(2000)\n    camera_id = -1: int\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert camera devices from an NWB file\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n\n        Returns\n        -------\n        device_name_list : list\n            List of camera device object names found in the NWB file.\n        \"\"\"\n        device_name_list = list()\n        for device in nwbf.devices.values():\n            if isinstance(device, ndx_franklab_novela.CameraDevice):\n                device_dict = dict()\n                # TODO ideally the ID is not encoded in the name formatted in a\n                # particular way device.name must have the form \"[any string\n                # without a space, usually camera] [int]\"\n                device_dict = {\n                    \"camera_id\": int(device.name.split()[1]),\n                    \"camera_name\": device.camera_name,\n                    \"manufacturer\": device.manufacturer,\n                    \"model\": device.model,\n                    \"lens\": device.lens,\n                    \"meters_per_pixel\": device.meters_per_pixel,\n                }\n                cls.insert1(device_dict, skip_duplicates=True)\n                device_name_list.append(device_dict[\"camera_name\"])\n        if device_name_list:\n            logger.info(f\"Inserted camera devices {device_name_list}\")\n        else:\n            logger.warn(\"No conforming camera device metadata found.\")\n        return device_name_list\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.CameraDevice.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert camera devices from an NWB file</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of camera device object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert camera devices from an NWB file\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n\n    Returns\n    -------\n    device_name_list : list\n        List of camera device object names found in the NWB file.\n    \"\"\"\n    device_name_list = list()\n    for device in nwbf.devices.values():\n        if isinstance(device, ndx_franklab_novela.CameraDevice):\n            device_dict = dict()\n            # TODO ideally the ID is not encoded in the name formatted in a\n            # particular way device.name must have the form \"[any string\n            # without a space, usually camera] [int]\"\n            device_dict = {\n                \"camera_id\": int(device.name.split()[1]),\n                \"camera_name\": device.camera_name,\n                \"manufacturer\": device.manufacturer,\n                \"model\": device.model,\n                \"lens\": device.lens,\n                \"meters_per_pixel\": device.meters_per_pixel,\n            }\n            cls.insert1(device_dict, skip_duplicates=True)\n            device_name_list.append(device_dict[\"camera_name\"])\n    if device_name_list:\n        logger.info(f\"Inserted camera devices {device_name_list}\")\n    else:\n        logger.warn(\"No conforming camera device metadata found.\")\n    return device_name_list\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe", "title": "<code>Probe</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@schema\nclass Probe(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # A configuration of a ProbeType. For most probe types, there is only one,\n    # which should always be used. For Neuropixels, the channel map (which\n    # electrodes used, where they are, and in what order) can differ between\n    # users and sessions. Each config should have a different ProbeType.\n    probe_id: varchar(80)     # a unique ID for this probe &amp; dynamic config\n    ---\n    -&gt; ProbeType              # Type of probe, selected from a controlled list\n    -&gt; [nullable] DataAcquisitionDevice  # the data acquisition device used\n    contact_side_numbering: enum(\"True\", \"False\")  # Facing you when numbering\n    \"\"\"\n\n    class Shank(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; Probe\n        probe_shank: int              # unique shank number within probe.\n        \"\"\"\n\n    class Electrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Electrode configuration, with ID, contact size, X/Y/Z coordinates\n        -&gt; Probe.Shank\n        probe_electrode: int          # electrode ID, output from acquisition\n                                      # system. Unique within a Probe\n        ---\n        contact_size = NULL: float    # (um) contact size\n        rel_x = NULL: float           # (um) x coordinate of electrode\n        rel_y = NULL: float           # (um) y coordinate of electrode\n        rel_z = NULL: float           # (um) z coordinate of electrode\n        \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, config):\n        \"\"\"Insert probe devices from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : list\n            List of probe device types found in the NWB file.\n        \"\"\"\n        all_probes_types, ndx_probes, _ = cls.get_all_probe_names(nwbf, config)\n\n        for probe_type in all_probes_types:\n            new_probe_type_dict = dict()\n            new_probe_dict = dict()\n            shank_dict = dict()\n            elect_dict = dict()\n            num_shanks = 0\n\n            if probe_type in ndx_probes:\n                # read probe properties into new_probe_dict from PyNWB extension\n                # probe object\n                nwb_probe_obj = ndx_probes[probe_type]\n                cls.__read_ndx_probe_data(\n                    nwb_probe_obj,\n                    new_probe_type_dict,\n                    new_probe_dict,\n                    shank_dict,\n                    elect_dict,\n                )\n\n            # check that number of shanks is consistent\n            num_shanks = new_probe_type_dict[\"num_shanks\"]\n            assert num_shanks == 0 or num_shanks == len(\n                shank_dict\n            ), \"`num_shanks` is not equal to the number of shanks.\"\n\n            # if probe id already exists, do not overwrite anything or create\n            # new Shanks and Electrodes\n            # TODO: test whether the Shanks and Electrodes in the NWB file match\n            # the ones in the database\n            query = Probe &amp; {\"probe_id\": new_probe_dict[\"probe_id\"]}\n            if len(query) &gt; 0:\n                logger.info(\n                    f\"Probe ID '{new_probe_dict['probe_id']}' already exists in\"\n                    \" the database. Spyglass will use that and not create a new\"\n                    \" Probe, Shanks, or Electrodes.\"\n                )\n                continue\n\n            cls.insert1(new_probe_dict, skip_duplicates=True)\n\n            for shank in shank_dict.values():\n                cls.Shank.insert1(shank, skip_duplicates=True)\n            for electrode in elect_dict.values():\n                cls.Electrode.insert1(electrode, skip_duplicates=True)\n\n        if all_probes_types:\n            logger.info(f\"Inserted probes {all_probes_types}\")\n        else:\n            logger.warn(\"No conforming probe metadata found.\")\n\n        return all_probes_types\n\n    @classmethod\n    def get_all_probe_names(cls, nwbf, config):\n        \"\"\"Get a list of all device names in the NWB.\n\n        Includes all devices, after appending/overwriting by the config file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        config : dict\n            Dictionary read from a user-defined YAML file containing values to\n            replace in the NWB file.\n\n        Returns\n        -------\n        device_name_list : list\n            List of data acquisition object names found in the NWB file.\n        \"\"\"\n\n        # make a dict mapping probe type to PyNWB object for all devices in the\n        # NWB file that are of type ndx_franklab_novela.Probe and thus have the\n        # required metadata\n        ndx_probes = {\n            device_obj.probe_type: device_obj\n            for device_obj in nwbf.devices.values()\n            if isinstance(device_obj, ndx_franklab_novela.Probe)\n        }\n\n        # make a dict mapping probe type to dict of device metadata from the\n        # config YAML if exists\n        config_probes = (\n            [probe_dict[\"probe_type\"] for probe_dict in config[\"Probe\"]]\n            if \"Probe\" in config\n            else list()\n        )\n\n        # get all the probe types from the NWB file plus the config YAML\n        all_probes_types = set(ndx_probes.keys()).union(set(config_probes))\n\n        return all_probes_types, ndx_probes, config_probes\n\n    @classmethod\n    def __read_ndx_probe_data(\n        cls,\n        nwb_probe_obj: ndx_franklab_novela.Probe,\n        new_probe_type_dict: dict,\n        new_probe_dict: dict,\n        shank_dict: dict,\n        elect_dict: dict,\n    ):\n        # construct dictionary of values to add to ProbeType\n        new_probe_type_dict.update(\n            {\n                \"manufacturer\": getattr(nwb_probe_obj, \"manufacturer\") or \"\",\n                \"probe_type\": nwb_probe_obj.probe_type,\n                \"probe_description\": nwb_probe_obj.probe_description,\n                \"num_shanks\": len(nwb_probe_obj.shanks),\n            }\n        )\n\n        cls._add_probe_type(new_probe_type_dict)\n\n        new_probe_dict.update(\n            {\n                \"probe_id\": nwb_probe_obj.probe_type,\n                \"probe_type\": nwb_probe_obj.probe_type,\n                \"contact_side_numbering\": (\n                    \"True\" if nwb_probe_obj.contact_side_numbering else \"False\"\n                ),\n            }\n        )\n        # go through the shanks and add each one to the Shank table\n        for shank in nwb_probe_obj.shanks.values():\n            shank_dict[shank.name] = {\n                \"probe_id\": new_probe_dict[\"probe_type\"],\n                \"probe_shank\": int(shank.name),\n            }\n\n            # go through the electrodes and add each one to the Electrode table\n            for electrode in shank.shanks_electrodes.values():\n                # the next line will need to be fixed if we have different sized\n                # contacts on a shank\n                elect_dict[electrode.name] = {\n                    \"probe_id\": new_probe_dict[\"probe_type\"],\n                    \"probe_shank\": shank_dict[shank.name][\"probe_shank\"],\n                    \"contact_size\": nwb_probe_obj.contact_size,\n                    \"probe_electrode\": int(electrode.name),\n                    \"rel_x\": electrode.rel_x,\n                    \"rel_y\": electrode.rel_y,\n                    \"rel_z\": electrode.rel_z,\n                }\n\n    @classmethod\n    def _add_probe_type(cls, new_probe_type_dict):\n        \"\"\"Check the probe type value against the values in the database.\n\n        Parameters\n        ----------\n        new_probe_type_dict : dict\n            Dictionary of probe type properties. See ProbeType for keys.\n\n        Raises\n        ------\n        PopulateException\n            If user chooses not to add a probe type to the database when\n            prompted.\n\n        Returns\n        -------\n        probe_type : str\n            The probe type value that was added to the database.\n        \"\"\"\n        probe_type = new_probe_type_dict[\"probe_type\"]\n        all_values = ProbeType.fetch(\"probe_type\").tolist()\n        if prompt_insert(probe_type, all_values, table=\"probe type\"):\n            ProbeType.insert1(new_probe_type_dict, skip_duplicates=True)\n            return\n\n        # else / entry exists: check whether the values provided match the\n        # values stored in the database\n        db_dict = (ProbeType &amp; {\"probe_type\": probe_type}).fetch1()\n        if db_dict != new_probe_type_dict:\n            raise PopulateException(\n                \"\\nProbe type properties of PyNWB Probe object with name \"\n                f\"'{probe_type}': {new_probe_type_dict} do not match properties\"\n                f\" of the corresponding database entry: {db_dict}.\"\n            )\n        return probe_type\n\n    @classmethod\n    def create_from_nwbfile(\n        cls,\n        nwb_file_name: str,\n        nwb_device_name: str,\n        probe_id: str,\n        probe_type: str,\n        contact_side_numbering: bool,\n    ):\n        \"\"\"Create master/part Probe entry from the NWB file.\n\n        This method will parse the electrodes in the electrodes table, electrode\n        groups (as shanks), and devices (as probes) in the NWB file, but only\n        ones that are associated with the device that matches the given\n        `nwb_device_name`.\n\n        Note that this code assumes the relatively standard convention where the\n        NWB device corresponds to a Probe, the NWB electrode group corresponds\n        to a Shank, and the NWB electrode corresponds to an Electrode.\n\n        Example usage: ``` sgc.Probe.create_from_nwbfile(\n            nwbfile=nwb_file_name, nwb_device_name=\"Device\",\n            probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",\n            probe_type=\"Neuropixels 1.0\", contact_side_numbering=True\n        )\n        ```\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        nwb_device_name : str\n            The name of the PyNWB Device object that represents the probe to\n            read in the NWB file.\n        probe_id : str\n            A unique ID for the probe and its configuration, to be used as the\n            primary key for the new Probe entry.\n        probe_type : str\n            The existing ProbeType entry that represents the type of probe being\n            created. It must exist.\n        contact_side_numbering : bool\n            Whether the electrode contacts are facing you when numbering them.\n            Stored in the new Probe entry.\n        \"\"\"\n\n        from .common_nwbfile import Nwbfile\n\n        nwb_file_path = Nwbfile.get_abs_path(nwb_file_name)\n        nwbfile = get_nwb_file(nwb_file_path)\n\n        query = ProbeType &amp; {\"probe_type\": probe_type}\n        if len(query) == 0:\n            logger.warn(\n                f\"No ProbeType found with probe_type '{probe_type}'. Aborting.\"\n            )\n            return\n\n        new_probe_dict = {\n            \"probe_id\": probe_id,\n            \"probe_type\": probe_type,\n            \"contact_side_numbering\": (\n                \"True\" if contact_side_numbering else \"False\"\n            ),\n        }\n        shank_dict = {}\n        elect_dict = {}\n\n        # iterate through the electrodes table in the NWB file\n        # and use the group column (ElectrodeGroup) to create shanks\n        # and use the device attribute of each ElectrodeGroup to create a probe\n        created_shanks = {}  # map device name to shank_index (int)\n        device_found = False\n        for elec_index in range(len(nwbfile.electrodes)):\n            electrode_group = nwbfile.electrodes[elec_index, \"group\"]\n            eg_device_name = electrode_group.device.name\n\n            # only look at electrodes where the associated device is the one\n            # specified\n            if eg_device_name == nwb_device_name:\n                device_found = True\n\n                # if a Shank has not yet been created from the electrode group,\n                # then create it\n                if electrode_group.name not in created_shanks:\n                    shank_index = len(created_shanks)\n                    created_shanks[electrode_group.name] = shank_index\n\n                    # build the dictionary of Probe.Shank data\n                    shank_dict[shank_index] = {\n                        \"probe_id\": new_probe_dict[\"probe_id\"],\n                        \"probe_shank\": shank_index,\n                    }\n\n                # get the probe shank index associated with this Electrode\n                probe_shank = created_shanks[electrode_group.name]\n\n                # build the dictionary of Probe.Electrode data\n                elect_dict[elec_index] = {\n                    \"probe_id\": new_probe_dict[\"probe_id\"],\n                    \"probe_shank\": probe_shank,\n                    \"probe_electrode\": elec_index,\n                }\n                if \"rel_x\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_x\"] = nwbfile.electrodes[\n                        elec_index, \"rel_x\"\n                    ]\n                if \"rel_y\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_y\"] = nwbfile.electrodes[\n                        elec_index, \"rel_y\"\n                    ]\n                if \"rel_z\" in nwbfile.electrodes[elec_index]:\n                    elect_dict[elec_index][\"rel_z\"] = nwbfile.electrodes[\n                        elec_index, \"rel_z\"\n                    ]\n\n        if not device_found:\n            logger.warn(\n                \"No electrodes in the NWB file were associated with a device \"\n                + f\"named '{nwb_device_name}'.\"\n            )\n            return\n\n        # insert the Probe, then the Shank parts, and then the Electrode parts\n        cls.insert1(new_probe_dict, skip_duplicates=True)\n\n        for shank in shank_dict.values():\n            cls.Shank.insert1(shank, skip_duplicates=True)\n        for electrode in elect_dict.values():\n            cls.Electrode.insert1(electrode, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Insert probe devices from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of probe device types found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, config):\n    \"\"\"Insert probe devices from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : list\n        List of probe device types found in the NWB file.\n    \"\"\"\n    all_probes_types, ndx_probes, _ = cls.get_all_probe_names(nwbf, config)\n\n    for probe_type in all_probes_types:\n        new_probe_type_dict = dict()\n        new_probe_dict = dict()\n        shank_dict = dict()\n        elect_dict = dict()\n        num_shanks = 0\n\n        if probe_type in ndx_probes:\n            # read probe properties into new_probe_dict from PyNWB extension\n            # probe object\n            nwb_probe_obj = ndx_probes[probe_type]\n            cls.__read_ndx_probe_data(\n                nwb_probe_obj,\n                new_probe_type_dict,\n                new_probe_dict,\n                shank_dict,\n                elect_dict,\n            )\n\n        # check that number of shanks is consistent\n        num_shanks = new_probe_type_dict[\"num_shanks\"]\n        assert num_shanks == 0 or num_shanks == len(\n            shank_dict\n        ), \"`num_shanks` is not equal to the number of shanks.\"\n\n        # if probe id already exists, do not overwrite anything or create\n        # new Shanks and Electrodes\n        # TODO: test whether the Shanks and Electrodes in the NWB file match\n        # the ones in the database\n        query = Probe &amp; {\"probe_id\": new_probe_dict[\"probe_id\"]}\n        if len(query) &gt; 0:\n            logger.info(\n                f\"Probe ID '{new_probe_dict['probe_id']}' already exists in\"\n                \" the database. Spyglass will use that and not create a new\"\n                \" Probe, Shanks, or Electrodes.\"\n            )\n            continue\n\n        cls.insert1(new_probe_dict, skip_duplicates=True)\n\n        for shank in shank_dict.values():\n            cls.Shank.insert1(shank, skip_duplicates=True)\n        for electrode in elect_dict.values():\n            cls.Electrode.insert1(electrode, skip_duplicates=True)\n\n    if all_probes_types:\n        logger.info(f\"Inserted probes {all_probes_types}\")\n    else:\n        logger.warn(\"No conforming probe metadata found.\")\n\n    return all_probes_types\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.get_all_probe_names", "title": "<code>get_all_probe_names(nwbf, config)</code>  <code>classmethod</code>", "text": "<p>Get a list of all device names in the NWB.</p> <p>Includes all devices, after appending/overwriting by the config file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>config</code> <code>dict</code> <p>Dictionary read from a user-defined YAML file containing values to replace in the NWB file.</p> required <p>Returns:</p> Name Type Description <code>device_name_list</code> <code>list</code> <p>List of data acquisition object names found in the NWB file.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef get_all_probe_names(cls, nwbf, config):\n    \"\"\"Get a list of all device names in the NWB.\n\n    Includes all devices, after appending/overwriting by the config file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    config : dict\n        Dictionary read from a user-defined YAML file containing values to\n        replace in the NWB file.\n\n    Returns\n    -------\n    device_name_list : list\n        List of data acquisition object names found in the NWB file.\n    \"\"\"\n\n    # make a dict mapping probe type to PyNWB object for all devices in the\n    # NWB file that are of type ndx_franklab_novela.Probe and thus have the\n    # required metadata\n    ndx_probes = {\n        device_obj.probe_type: device_obj\n        for device_obj in nwbf.devices.values()\n        if isinstance(device_obj, ndx_franklab_novela.Probe)\n    }\n\n    # make a dict mapping probe type to dict of device metadata from the\n    # config YAML if exists\n    config_probes = (\n        [probe_dict[\"probe_type\"] for probe_dict in config[\"Probe\"]]\n        if \"Probe\" in config\n        else list()\n    )\n\n    # get all the probe types from the NWB file plus the config YAML\n    all_probes_types = set(ndx_probes.keys()).union(set(config_probes))\n\n    return all_probes_types, ndx_probes, config_probes\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.create_from_nwbfile", "title": "<code>create_from_nwbfile(nwb_file_name, nwb_device_name, probe_id, probe_type, contact_side_numbering)</code>  <code>classmethod</code>", "text": "<p>Create master/part Probe entry from the NWB file.</p> <p>This method will parse the electrodes in the electrodes table, electrode groups (as shanks), and devices (as probes) in the NWB file, but only ones that are associated with the device that matches the given <code>nwb_device_name</code>.</p> <p>Note that this code assumes the relatively standard convention where the NWB device corresponds to a Probe, the NWB electrode group corresponds to a Shank, and the NWB electrode corresponds to an Electrode.</p> <p>Example usage: <code>sgc.Probe.create_from_nwbfile(     nwbfile=nwb_file_name, nwb_device_name=\"Device\",     probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",     probe_type=\"Neuropixels 1.0\", contact_side_numbering=True )</code></p>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.Probe.create_from_nwbfile--parameters", "title": "Parameters", "text": "<p>nwb_file_name : str     The name of the NWB file. nwb_device_name : str     The name of the PyNWB Device object that represents the probe to     read in the NWB file. probe_id : str     A unique ID for the probe and its configuration, to be used as the     primary key for the new Probe entry. probe_type : str     The existing ProbeType entry that represents the type of probe being     created. It must exist. contact_side_numbering : bool     Whether the electrode contacts are facing you when numbering them.     Stored in the new Probe entry.</p> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>@classmethod\ndef create_from_nwbfile(\n    cls,\n    nwb_file_name: str,\n    nwb_device_name: str,\n    probe_id: str,\n    probe_type: str,\n    contact_side_numbering: bool,\n):\n    \"\"\"Create master/part Probe entry from the NWB file.\n\n    This method will parse the electrodes in the electrodes table, electrode\n    groups (as shanks), and devices (as probes) in the NWB file, but only\n    ones that are associated with the device that matches the given\n    `nwb_device_name`.\n\n    Note that this code assumes the relatively standard convention where the\n    NWB device corresponds to a Probe, the NWB electrode group corresponds\n    to a Shank, and the NWB electrode corresponds to an Electrode.\n\n    Example usage: ``` sgc.Probe.create_from_nwbfile(\n        nwbfile=nwb_file_name, nwb_device_name=\"Device\",\n        probe_id=\"Neuropixels 1.0 Giocomo Lab Configuration\",\n        probe_type=\"Neuropixels 1.0\", contact_side_numbering=True\n    )\n    ```\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    nwb_device_name : str\n        The name of the PyNWB Device object that represents the probe to\n        read in the NWB file.\n    probe_id : str\n        A unique ID for the probe and its configuration, to be used as the\n        primary key for the new Probe entry.\n    probe_type : str\n        The existing ProbeType entry that represents the type of probe being\n        created. It must exist.\n    contact_side_numbering : bool\n        Whether the electrode contacts are facing you when numbering them.\n        Stored in the new Probe entry.\n    \"\"\"\n\n    from .common_nwbfile import Nwbfile\n\n    nwb_file_path = Nwbfile.get_abs_path(nwb_file_name)\n    nwbfile = get_nwb_file(nwb_file_path)\n\n    query = ProbeType &amp; {\"probe_type\": probe_type}\n    if len(query) == 0:\n        logger.warn(\n            f\"No ProbeType found with probe_type '{probe_type}'. Aborting.\"\n        )\n        return\n\n    new_probe_dict = {\n        \"probe_id\": probe_id,\n        \"probe_type\": probe_type,\n        \"contact_side_numbering\": (\n            \"True\" if contact_side_numbering else \"False\"\n        ),\n    }\n    shank_dict = {}\n    elect_dict = {}\n\n    # iterate through the electrodes table in the NWB file\n    # and use the group column (ElectrodeGroup) to create shanks\n    # and use the device attribute of each ElectrodeGroup to create a probe\n    created_shanks = {}  # map device name to shank_index (int)\n    device_found = False\n    for elec_index in range(len(nwbfile.electrodes)):\n        electrode_group = nwbfile.electrodes[elec_index, \"group\"]\n        eg_device_name = electrode_group.device.name\n\n        # only look at electrodes where the associated device is the one\n        # specified\n        if eg_device_name == nwb_device_name:\n            device_found = True\n\n            # if a Shank has not yet been created from the electrode group,\n            # then create it\n            if electrode_group.name not in created_shanks:\n                shank_index = len(created_shanks)\n                created_shanks[electrode_group.name] = shank_index\n\n                # build the dictionary of Probe.Shank data\n                shank_dict[shank_index] = {\n                    \"probe_id\": new_probe_dict[\"probe_id\"],\n                    \"probe_shank\": shank_index,\n                }\n\n            # get the probe shank index associated with this Electrode\n            probe_shank = created_shanks[electrode_group.name]\n\n            # build the dictionary of Probe.Electrode data\n            elect_dict[elec_index] = {\n                \"probe_id\": new_probe_dict[\"probe_id\"],\n                \"probe_shank\": probe_shank,\n                \"probe_electrode\": elec_index,\n            }\n            if \"rel_x\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_x\"] = nwbfile.electrodes[\n                    elec_index, \"rel_x\"\n                ]\n            if \"rel_y\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_y\"] = nwbfile.electrodes[\n                    elec_index, \"rel_y\"\n                ]\n            if \"rel_z\" in nwbfile.electrodes[elec_index]:\n                elect_dict[elec_index][\"rel_z\"] = nwbfile.electrodes[\n                    elec_index, \"rel_z\"\n                ]\n\n    if not device_found:\n        logger.warn(\n            \"No electrodes in the NWB file were associated with a device \"\n            + f\"named '{nwb_device_name}'.\"\n        )\n        return\n\n    # insert the Probe, then the Shank parts, and then the Electrode parts\n    cls.insert1(new_probe_dict, skip_duplicates=True)\n\n    for shank in shank_dict.values():\n        cls.Shank.insert1(shank, skip_duplicates=True)\n    for electrode in elect_dict.values():\n        cls.Electrode.insert1(electrode, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_device/#src.spyglass.common.common_device.prompt_insert", "title": "<code>prompt_insert(name, all_values, table='Data Acquisition Device', table_type=None)</code>", "text": "<p>Prompt user to add an item to the database. Return True if yes.</p> <p>Assume insert during test mode.</p> <p>Parameters:</p> Name Type Description Default <code>name</code> <code>str</code> <p>The name of the item to add.</p> required <code>all_values</code> <code>list</code> <p>List of all values in the database.</p> required <code>table</code> <code>str</code> <p>The name of the table to add to, by default Data Acquisition Device</p> <code>'Data Acquisition Device'</code> <code>table_type</code> <code>str</code> <p>The type of item to add, by default None. Data Acquisition Device X</p> <code>None</code> Source code in <code>src/spyglass/common/common_device.py</code> <pre><code>def prompt_insert(\n    name: str,\n    all_values: list,\n    table: str = \"Data Acquisition Device\",\n    table_type: str = None,\n) -&gt; bool:\n    \"\"\"Prompt user to add an item to the database. Return True if yes.\n\n    Assume insert during test mode.\n\n    Parameters\n    ----------\n    name : str\n        The name of the item to add.\n    all_values : list\n        List of all values in the database.\n    table : str, optional\n        The name of the table to add to, by default Data Acquisition Device\n    table_type : str, optional\n        The type of item to add, by default None. Data Acquisition Device X\n    \"\"\"\n    if name in all_values:\n        return False\n\n    if test_mode:\n        return True\n\n    if table_type:\n        table_type += \" \"\n\n    logger.info(\n        f\"{table}{table_type} '{name}' was not found in the\"\n        f\"database. The current values are: {all_values}.\\n\"\n        \"Please ensure that the device you want to add does not already\"\n        \"exist in the database under a different name or spelling. If you\"\n        \"want to use an existing device in the database, please change the\"\n        \"corresponding Device object in the NWB file.\\nEntering 'N' will \"\n        \"raise an exception.\"\n    )\n    msg = f\"Do you want to add {table}{table_type} '{name}' to the database?\"\n    if dj.utils.user_choice(msg).lower() in [\"y\", \"yes\"]:\n        return True\n\n    raise PopulateException(\n        f\"User chose not to add {table}{table_type} '{name}' to the database.\"\n    )\n</code></pre>"}, {"location": "api/common/common_dio/", "title": "common_dio.py", "text": ""}, {"location": "api/common/common_dio/#src.spyglass.common.common_dio.DIOEvents", "title": "<code>DIOEvents</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_dio.py</code> <pre><code>@schema\nclass DIOEvents(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; Session\n    dio_event_name: varchar(80)   # the name assigned to this DIO event\n    ---\n    dio_object_id: varchar(40)    # the object id of the data in the NWB file\n    -&gt; IntervalList               # the list of intervals for this object\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n        behav_events = get_data_interface(\n            nwbf, \"behavioral_events\", pynwb.behavior.BehavioralEvents\n        )\n        if behav_events is None:\n            logger.warn(\n                \"No conforming behavioral events data interface found in \"\n                + f\"{nwb_file_name}\\n\"\n            )\n            return  # See #849\n\n        # Times for these events correspond to the valid times for the raw data\n        key[\"interval_list_name\"] = (\n            Raw() &amp; {\"nwb_file_name\": nwb_file_name}\n        ).fetch1(\"interval_list_name\")\n        for event_series in behav_events.time_series.values():\n            key[\"dio_event_name\"] = event_series.name\n            key[\"dio_object_id\"] = event_series.object_id\n            self.insert1(key, skip_duplicates=True)\n\n    def plot_all_dio_events(self, return_fig=False):\n        \"\"\"Plot all DIO events in the session.\n\n        Examples\n        --------\n        &gt; restr1 = {'nwb_file_name': 'arthur20220314_.nwb'}\n        &gt; restr2 = {'nwb_file_name': 'arthur20220316_.nwb'}\n        &gt; (DIOEvents &amp; restr1).plot_all_dio_events()\n        &gt; (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()\n\n        \"\"\"\n        behavioral_events = self.fetch_nwb()\n        nwb_file_names = np.unique(\n            [event[\"nwb_file_name\"] for event in behavioral_events]\n        )\n        epoch_valid_times = (\n            pd.DataFrame(\n                IntervalList()\n                &amp; [\n                    {\"nwb_file_name\": nwb_file_name}\n                    for nwb_file_name in nwb_file_names\n                ]\n            )\n            .set_index(\"interval_list_name\")\n            .filter(regex=r\"^[0-9]\", axis=0)\n            .valid_times\n        )\n\n        n_events = len(behavioral_events)\n\n        _, axes = plt.subplots(\n            n_events,\n            1,\n            figsize=(15, n_events * 0.3),\n            dpi=100,\n            sharex=True,\n            constrained_layout=True,\n        )\n\n        for ind, (ax, event) in enumerate(zip(axes.flat, behavioral_events)):\n            for epoch_name, epoch in epoch_valid_times.items():\n                start_time, stop_time = epoch.squeeze()\n                ax.axvspan(start_time, stop_time, alpha=0.5)\n                if ind == 0:\n                    ax.text(\n                        start_time + (stop_time - start_time) / 2,\n                        1.001,\n                        epoch_name,\n                        ha=\"center\",\n                        va=\"bottom\",\n                    )\n            ax.step(\n                np.asarray(event[\"dio\"].timestamps),\n                np.asarray(event[\"dio\"].data),\n                where=\"post\",\n                color=\"black\",\n            )\n            ax.set_ylabel(\n                event[\"dio_event_name\"], rotation=0, ha=\"right\", va=\"center\"\n            )\n            ax.set_yticks([])\n        ax.set_xlabel(\"Time\")\n\n        if len(nwb_file_names) == 1:\n            plt.suptitle(f\"DIO events in {nwb_file_names[0]}\")\n        else:\n            plt.suptitle(f\"DIO events in {', '.join(nwb_file_names)}\")\n\n        if return_fig:\n            return plt.gcf()\n</code></pre>"}, {"location": "api/common/common_dio/#src.spyglass.common.common_dio.DIOEvents.plot_all_dio_events", "title": "<code>plot_all_dio_events(return_fig=False)</code>", "text": "<p>Plot all DIO events in the session.</p> <p>Examples:</p> <p>restr1 = {'nwb_file_name': 'arthur20220314_.nwb'} restr2 = {'nwb_file_name': 'arthur20220316_.nwb'} (DIOEvents &amp; restr1).plot_all_dio_events() (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()</p> Source code in <code>src/spyglass/common/common_dio.py</code> <pre><code>def plot_all_dio_events(self, return_fig=False):\n    \"\"\"Plot all DIO events in the session.\n\n    Examples\n    --------\n    &gt; restr1 = {'nwb_file_name': 'arthur20220314_.nwb'}\n    &gt; restr2 = {'nwb_file_name': 'arthur20220316_.nwb'}\n    &gt; (DIOEvents &amp; restr1).plot_all_dio_events()\n    &gt; (DIOEvents &amp; [restr1, restr2]).plot_all_dio_events()\n\n    \"\"\"\n    behavioral_events = self.fetch_nwb()\n    nwb_file_names = np.unique(\n        [event[\"nwb_file_name\"] for event in behavioral_events]\n    )\n    epoch_valid_times = (\n        pd.DataFrame(\n            IntervalList()\n            &amp; [\n                {\"nwb_file_name\": nwb_file_name}\n                for nwb_file_name in nwb_file_names\n            ]\n        )\n        .set_index(\"interval_list_name\")\n        .filter(regex=r\"^[0-9]\", axis=0)\n        .valid_times\n    )\n\n    n_events = len(behavioral_events)\n\n    _, axes = plt.subplots(\n        n_events,\n        1,\n        figsize=(15, n_events * 0.3),\n        dpi=100,\n        sharex=True,\n        constrained_layout=True,\n    )\n\n    for ind, (ax, event) in enumerate(zip(axes.flat, behavioral_events)):\n        for epoch_name, epoch in epoch_valid_times.items():\n            start_time, stop_time = epoch.squeeze()\n            ax.axvspan(start_time, stop_time, alpha=0.5)\n            if ind == 0:\n                ax.text(\n                    start_time + (stop_time - start_time) / 2,\n                    1.001,\n                    epoch_name,\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n        ax.step(\n            np.asarray(event[\"dio\"].timestamps),\n            np.asarray(event[\"dio\"].data),\n            where=\"post\",\n            color=\"black\",\n        )\n        ax.set_ylabel(\n            event[\"dio_event_name\"], rotation=0, ha=\"right\", va=\"center\"\n        )\n        ax.set_yticks([])\n    ax.set_xlabel(\"Time\")\n\n    if len(nwb_file_names) == 1:\n        plt.suptitle(f\"DIO events in {nwb_file_names[0]}\")\n    else:\n        plt.suptitle(f\"DIO events in {', '.join(nwb_file_names)}\")\n\n    if return_fig:\n        return plt.gcf()\n</code></pre>"}, {"location": "api/common/common_ephys/", "title": "common_ephys.py", "text": ""}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.Electrode", "title": "<code>Electrode</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass Electrode(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; ElectrodeGroup\n    electrode_id: int                      # the unique number for this electrode\n    ---\n    -&gt; [nullable] Probe.Electrode\n    -&gt; BrainRegion\n    name = \"\": varchar(200)                 # unique label for each contact\n    original_reference_electrode = -1: int  # the configured reference electrode for this electrode\n    x = NULL: float                         # the x coordinate of the electrode position in the brain\n    y = NULL: float                         # the y coordinate of the electrode position in the brain\n    z = NULL: float                         # the z coordinate of the electrode position in the brain\n    filtering: varchar(2000)                # description of the signal filtering\n    impedance = NULL: float                 # electrode impedance\n    bad_channel = \"False\": enum(\"True\", \"False\")  # if electrode is \"good\" or \"bad\" as observed during recording\n    x_warped = NULL: float                  # x coordinate of electrode position warped to common template brain\n    y_warped = NULL: float                  # y coordinate of electrode position warped to common template brain\n    z_warped = NULL: float                  # z coordinate of electrode position warped to common template brain\n    contacts: varchar(200)                  # label of electrode contacts used for a bipolar signal - current workaround\n    \"\"\"\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        config = get_config(nwb_file_abspath)\n\n        if \"Electrode\" in config:\n            electrode_config_dicts = {\n                electrode_dict[\"electrode_id\"]: electrode_dict\n                for electrode_dict in config[\"Electrode\"]\n            }\n        else:\n            electrode_config_dicts = dict()\n\n        electrodes = nwbf.electrodes.to_dataframe()\n        for elect_id, elect_data in electrodes.iterrows():\n            key[\"electrode_id\"] = elect_id\n            key[\"name\"] = str(elect_id)\n            key[\"electrode_group_name\"] = elect_data.group_name\n            key[\"region_id\"] = BrainRegion.fetch_add(\n                region_name=elect_data.group.location\n            )\n            key[\"x\"] = elect_data.x\n            key[\"y\"] = elect_data.y\n            key[\"z\"] = elect_data.z\n            key[\"x_warped\"] = 0\n            key[\"y_warped\"] = 0\n            key[\"z_warped\"] = 0\n            key[\"contacts\"] = \"\"\n            key[\"filtering\"] = elect_data.filtering\n            key[\"impedance\"] = elect_data.get(\"imp\")\n\n            # rough check of whether the electrodes table was created by\n            # rec_to_nwb and has the appropriate custom columns used by\n            # rec_to_nwb\n\n            # TODO this could be better resolved by making an extension for the\n            # electrodes table\n\n            if (\n                isinstance(elect_data.group.device, ndx_franklab_novela.Probe)\n                and \"probe_shank\" in elect_data\n                and \"probe_electrode\" in elect_data\n                and \"bad_channel\" in elect_data\n                and \"ref_elect_id\" in elect_data\n            ):\n                key[\"probe_id\"] = elect_data.group.device.probe_type\n                key[\"probe_shank\"] = elect_data.probe_shank\n                key[\"probe_electrode\"] = elect_data.probe_electrode\n                key[\"bad_channel\"] = (\n                    \"True\" if elect_data.bad_channel else \"False\"\n                )\n                key[\"original_reference_electrode\"] = elect_data.ref_elect_id\n\n            # override with information from the config YAML based on primary\n            # key (electrode id)\n\n            if elect_id in electrode_config_dicts:\n                # check whether the Probe.Electrode being referenced exists\n                query = Probe.Electrode &amp; electrode_config_dicts[elect_id]\n                if len(query) == 0:\n                    warnings.warn(\n                        \"No Probe.Electrode exists that matches the data: \"\n                        + f\"{electrode_config_dicts[elect_id]}. \"\n                        \"The config YAML for Electrode with electrode_id \"\n                        + f\"{elect_id} will be ignored.\"\n                    )\n                else:\n                    key.update(electrode_config_dicts[elect_id])\n\n            self.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def create_from_config(cls, nwb_file_name: str):\n        \"\"\"Create or update Electrode entries from what is specified in the config YAML file.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the NWB file.\n        \"\"\"\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        config = get_config(nwb_file_abspath)\n        if \"Electrode\" not in config:\n            return  # See #849\n\n        # map electrode id to dictof electrode information from config YAML\n        electrode_dicts = {\n            electrode_dict[\"electrode_id\"]: electrode_dict\n            for electrode_dict in config[\"Electrode\"]\n        }\n\n        electrodes = nwbf.electrodes.to_dataframe()\n        for nwbfile_elect_id, elect_data in electrodes.iterrows():\n            if nwbfile_elect_id in electrode_dicts:\n                # use the information in the electrodes table to start and then\n                # add (or overwrite) values from the config YAML\n\n                key = dict()\n                key[\"nwb_file_name\"] = nwb_file_name\n                key[\"name\"] = str(nwbfile_elect_id)\n                key[\"electrode_group_name\"] = elect_data.group_name\n                key[\"region_id\"] = BrainRegion.fetch_add(\n                    region_name=elect_data.group.location\n                )\n                key[\"x\"] = elect_data.x\n                key[\"y\"] = elect_data.y\n                key[\"z\"] = elect_data.z\n                key[\"x_warped\"] = 0\n                key[\"y_warped\"] = 0\n                key[\"z_warped\"] = 0\n                key[\"contacts\"] = \"\"\n                key[\"filtering\"] = elect_data.filtering\n                key[\"impedance\"] = elect_data.get(\"imp\")\n                key.update(electrode_dicts[nwbfile_elect_id])\n                query = Electrode &amp; {\"electrode_id\": nwbfile_elect_id}\n                if len(query):\n                    cls.update1(key)\n                    logger.info(\n                        f\"Updated Electrode with ID {nwbfile_elect_id}.\"\n                    )\n                else:\n                    cls.insert1(\n                        key, skip_duplicates=True, allow_direct_insert=True\n                    )\n                    logger.info(\n                        f\"Inserted Electrode with ID {nwbfile_elect_id}.\"\n                    )\n            else:\n                warnings.warn(\n                    f\"Electrode ID {nwbfile_elect_id} exists in the NWB file \"\n                    + \"but has no corresponding config YAML entry.\"\n                )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.Electrode.create_from_config", "title": "<code>create_from_config(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Create or update Electrode entries from what is specified in the config YAML file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file.</p> required Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@classmethod\ndef create_from_config(cls, nwb_file_name: str):\n    \"\"\"Create or update Electrode entries from what is specified in the config YAML file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file.\n    \"\"\"\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    nwbf = get_nwb_file(nwb_file_abspath)\n    config = get_config(nwb_file_abspath)\n    if \"Electrode\" not in config:\n        return  # See #849\n\n    # map electrode id to dictof electrode information from config YAML\n    electrode_dicts = {\n        electrode_dict[\"electrode_id\"]: electrode_dict\n        for electrode_dict in config[\"Electrode\"]\n    }\n\n    electrodes = nwbf.electrodes.to_dataframe()\n    for nwbfile_elect_id, elect_data in electrodes.iterrows():\n        if nwbfile_elect_id in electrode_dicts:\n            # use the information in the electrodes table to start and then\n            # add (or overwrite) values from the config YAML\n\n            key = dict()\n            key[\"nwb_file_name\"] = nwb_file_name\n            key[\"name\"] = str(nwbfile_elect_id)\n            key[\"electrode_group_name\"] = elect_data.group_name\n            key[\"region_id\"] = BrainRegion.fetch_add(\n                region_name=elect_data.group.location\n            )\n            key[\"x\"] = elect_data.x\n            key[\"y\"] = elect_data.y\n            key[\"z\"] = elect_data.z\n            key[\"x_warped\"] = 0\n            key[\"y_warped\"] = 0\n            key[\"z_warped\"] = 0\n            key[\"contacts\"] = \"\"\n            key[\"filtering\"] = elect_data.filtering\n            key[\"impedance\"] = elect_data.get(\"imp\")\n            key.update(electrode_dicts[nwbfile_elect_id])\n            query = Electrode &amp; {\"electrode_id\": nwbfile_elect_id}\n            if len(query):\n                cls.update1(key)\n                logger.info(\n                    f\"Updated Electrode with ID {nwbfile_elect_id}.\"\n                )\n            else:\n                cls.insert1(\n                    key, skip_duplicates=True, allow_direct_insert=True\n                )\n                logger.info(\n                    f\"Inserted Electrode with ID {nwbfile_elect_id}.\"\n                )\n        else:\n            warnings.warn(\n                f\"Electrode ID {nwbfile_elect_id} exists in the NWB file \"\n                + \"but has no corresponding config YAML entry.\"\n            )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPSelection", "title": "<code>LFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass LFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; Session\n     \"\"\"\n\n    class LFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPSelection\n        -&gt; Electrode\n        \"\"\"\n\n    def set_lfp_electrodes(self, nwb_file_name, electrode_list):\n        \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the nwb file for the desired session\n        electrode_list : list\n            list of electrodes to be used for LFP\n\n        \"\"\"\n        # remove the session and then recreate the session and Electrode list\n        (LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).delete(\n            safemode=not test_mode\n        )\n        # check to see if the user allowed the deletion\n        if (\n            len((LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).fetch())\n            == 0\n        ):\n            LFPSelection().insert1({\"nwb_file_name\": nwb_file_name})\n\n            # TODO: do this in a better way\n            all_electrodes = (\n                Electrode() &amp; {\"nwb_file_name\": nwb_file_name}\n            ).fetch(as_dict=True)\n            primary_key = Electrode.primary_key\n            for e in all_electrodes:\n                # create a dictionary so we can insert new elects\n                if e[\"electrode_id\"] in electrode_list:\n                    lfpelectdict = {\n                        k: v for k, v in e.items() if k in primary_key\n                    }\n                    LFPSelection().LFPElectrode.insert1(\n                        lfpelectdict, replace=True\n                    )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(nwb_file_name, electrode_list)</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the nwb file for the desired session</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes to be used for LFP</p> required Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>def set_lfp_electrodes(self, nwb_file_name, electrode_list):\n    \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the nwb file for the desired session\n    electrode_list : list\n        list of electrodes to be used for LFP\n\n    \"\"\"\n    # remove the session and then recreate the session and Electrode list\n    (LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).delete(\n        safemode=not test_mode\n    )\n    # check to see if the user allowed the deletion\n    if (\n        len((LFPSelection() &amp; {\"nwb_file_name\": nwb_file_name}).fetch())\n        == 0\n    ):\n        LFPSelection().insert1({\"nwb_file_name\": nwb_file_name})\n\n        # TODO: do this in a better way\n        all_electrodes = (\n            Electrode() &amp; {\"nwb_file_name\": nwb_file_name}\n        ).fetch(as_dict=True)\n        primary_key = Electrode.primary_key\n        for e in all_electrodes:\n            # create a dictionary so we can insert new elects\n            if e[\"electrode_id\"] in electrode_list:\n                lfpelectdict = {\n                    k: v for k, v in e.items() if k in primary_key\n                }\n                LFPSelection().LFPElectrode.insert1(\n                    lfpelectdict, replace=True\n                )\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPBandSelection", "title": "<code>LFPBandSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>@schema\nclass LFPBandSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; LFP\n    -&gt; FirFilterParameters                   # the filter to use for the data\n    -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n    lfp_band_sampling_rate: int    # the sampling rate for this band\n    ---\n    min_interval_len = 1: float  # the minimum length of a valid interval to filter\n    \"\"\"\n\n    class LFPBandElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPBandSelection\n        -&gt; LFPSelection.LFPElectrode  # the LFP electrode to be filtered\n        reference_elect_id = -1: int  # the reference electrode to use; -1 for no reference\n        ---\n        \"\"\"\n\n    def set_lfp_band_electrodes(\n        self,\n        nwb_file_name,\n        electrode_list,\n        filter_name,\n        interval_list_name,\n        reference_electrode_list,\n        lfp_band_sampling_rate,\n    ):\n        \"\"\"\n        Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and\n        reference electrode.\n        Also removes any entries that have the same filter, interval list and reference electrode but are not\n        in the electrode_list.\n        :param nwb_file_name: string - the name of the nwb file for the desired session\n        :param electrode_list: list of LFP electrodes to be filtered\n        :param filter_name: the name of the filter (from the FirFilterParameters schema)\n        :param interval_name: the name of the interval list (from the IntervalList schema)\n        :param reference_electrode_list: A single electrode id corresponding to the reference to use for all\n        electrodes or a list with one element per entry in the electrode_list\n        :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an\n        integer divisor of the LFP sampling rate\n        :return: none\n        \"\"\"\n        # Error checks on parameters\n        # electrode_list\n        query = LFPSelection().LFPElectrode() &amp; {\"nwb_file_name\": nwb_file_name}\n        available_electrodes = query.fetch(\"electrode_id\")\n        if not np.all(np.isin(electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in electrode_list must be valid electrode_ids in the LFPSelection table\"\n            )\n        # sampling rate\n        lfp_sampling_rate = (LFP() &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\n            \"lfp_sampling_rate\"\n        )\n        decimation = lfp_sampling_rate // lfp_band_sampling_rate\n        if lfp_sampling_rate // decimation != lfp_band_sampling_rate:\n            raise ValueError(\n                f\"lfp_band_sampling rate {lfp_band_sampling_rate} is not an integer divisor of lfp \"\n                f\"samping rate {lfp_sampling_rate}\"\n            )\n        # filter\n        query = FirFilterParameters() &amp; {\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": lfp_sampling_rate,\n        }\n        if not query:\n            raise ValueError(\n                f\"filter {filter_name}, sampling rate {lfp_sampling_rate} is not in the FirFilterParameters table\"\n            )\n        # interval_list\n        query = IntervalList() &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_name\": interval_list_name,\n        }\n        if not query:\n            raise ValueError(\n                f\"interval list {interval_list_name} is not in the IntervalList table; the list must be \"\n                \"added before this function is called\"\n            )\n        # reference_electrode_list\n        if len(reference_electrode_list) != 1 and len(\n            reference_electrode_list\n        ) != len(electrode_list):\n            raise ValueError(\n                \"reference_electrode_list must contain either 1 or len(electrode_list) elements\"\n            )\n        # add a -1 element to the list to allow for the no reference option\n        available_electrodes = np.append(available_electrodes, [-1])\n        if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in reference_electrode_list must be valid electrode_ids in the LFPSelection \"\n                \"table\"\n            )\n\n        # make a list of all the references\n        ref_list = np.zeros((len(electrode_list),))\n        ref_list[:] = reference_electrode_list\n\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"filter_name\"] = filter_name\n        key[\"filter_sampling_rate\"] = lfp_sampling_rate\n        key[\"target_interval_list_name\"] = interval_list_name\n        key[\"lfp_band_sampling_rate\"] = lfp_sampling_rate // decimation\n        # insert an entry into the main LFPBandSelectionTable\n        self.insert1(key, skip_duplicates=True)\n\n        # get all of the current entries and delete any that are not in the list\n        elect_id, ref_id = (self.LFPBandElectrode() &amp; key).fetch(\n            \"electrode_id\", \"reference_elect_id\"\n        )\n        for e, r in zip(elect_id, ref_id):\n            if not len(np.where((electrode_list == e) &amp; (ref_list == r))[0]):\n                key[\"electrode_id\"] = e\n                key[\"reference_elect_id\"] = r\n                (self.LFPBandElectrode() &amp; key).delete()\n\n        # iterate through all of the new elements and add them\n        for e, r in zip(electrode_list, ref_list):\n            key[\"electrode_id\"] = e\n            query = Electrode &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_id\": e,\n            }\n            key[\"electrode_group_name\"] = query.fetch1(\"electrode_group_name\")\n            key[\"reference_elect_id\"] = r\n            self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_ephys/#src.spyglass.common.common_ephys.LFPBandSelection.set_lfp_band_electrodes", "title": "<code>set_lfp_band_electrodes(nwb_file_name, electrode_list, filter_name, interval_list_name, reference_electrode_list, lfp_band_sampling_rate)</code>", "text": "<p>Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and reference electrode. Also removes any entries that have the same filter, interval list and reference electrode but are not in the electrode_list. :param nwb_file_name: string - the name of the nwb file for the desired session :param electrode_list: list of LFP electrodes to be filtered :param filter_name: the name of the filter (from the FirFilterParameters schema) :param interval_name: the name of the interval list (from the IntervalList schema) :param reference_electrode_list: A single electrode id corresponding to the reference to use for all electrodes or a list with one element per entry in the electrode_list :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an integer divisor of the LFP sampling rate :return: none</p> Source code in <code>src/spyglass/common/common_ephys.py</code> <pre><code>def set_lfp_band_electrodes(\n    self,\n    nwb_file_name,\n    electrode_list,\n    filter_name,\n    interval_list_name,\n    reference_electrode_list,\n    lfp_band_sampling_rate,\n):\n    \"\"\"\n    Adds an entry for each electrode in the electrode_list with the specified filter, interval_list, and\n    reference electrode.\n    Also removes any entries that have the same filter, interval list and reference electrode but are not\n    in the electrode_list.\n    :param nwb_file_name: string - the name of the nwb file for the desired session\n    :param electrode_list: list of LFP electrodes to be filtered\n    :param filter_name: the name of the filter (from the FirFilterParameters schema)\n    :param interval_name: the name of the interval list (from the IntervalList schema)\n    :param reference_electrode_list: A single electrode id corresponding to the reference to use for all\n    electrodes or a list with one element per entry in the electrode_list\n    :param lfp_band_sampling_rate: The output sampling rate to be used for the filtered data; must be an\n    integer divisor of the LFP sampling rate\n    :return: none\n    \"\"\"\n    # Error checks on parameters\n    # electrode_list\n    query = LFPSelection().LFPElectrode() &amp; {\"nwb_file_name\": nwb_file_name}\n    available_electrodes = query.fetch(\"electrode_id\")\n    if not np.all(np.isin(electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in electrode_list must be valid electrode_ids in the LFPSelection table\"\n        )\n    # sampling rate\n    lfp_sampling_rate = (LFP() &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\n        \"lfp_sampling_rate\"\n    )\n    decimation = lfp_sampling_rate // lfp_band_sampling_rate\n    if lfp_sampling_rate // decimation != lfp_band_sampling_rate:\n        raise ValueError(\n            f\"lfp_band_sampling rate {lfp_band_sampling_rate} is not an integer divisor of lfp \"\n            f\"samping rate {lfp_sampling_rate}\"\n        )\n    # filter\n    query = FirFilterParameters() &amp; {\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": lfp_sampling_rate,\n    }\n    if not query:\n        raise ValueError(\n            f\"filter {filter_name}, sampling rate {lfp_sampling_rate} is not in the FirFilterParameters table\"\n        )\n    # interval_list\n    query = IntervalList() &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_name\": interval_list_name,\n    }\n    if not query:\n        raise ValueError(\n            f\"interval list {interval_list_name} is not in the IntervalList table; the list must be \"\n            \"added before this function is called\"\n        )\n    # reference_electrode_list\n    if len(reference_electrode_list) != 1 and len(\n        reference_electrode_list\n    ) != len(electrode_list):\n        raise ValueError(\n            \"reference_electrode_list must contain either 1 or len(electrode_list) elements\"\n        )\n    # add a -1 element to the list to allow for the no reference option\n    available_electrodes = np.append(available_electrodes, [-1])\n    if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in reference_electrode_list must be valid electrode_ids in the LFPSelection \"\n            \"table\"\n        )\n\n    # make a list of all the references\n    ref_list = np.zeros((len(electrode_list),))\n    ref_list[:] = reference_electrode_list\n\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"filter_name\"] = filter_name\n    key[\"filter_sampling_rate\"] = lfp_sampling_rate\n    key[\"target_interval_list_name\"] = interval_list_name\n    key[\"lfp_band_sampling_rate\"] = lfp_sampling_rate // decimation\n    # insert an entry into the main LFPBandSelectionTable\n    self.insert1(key, skip_duplicates=True)\n\n    # get all of the current entries and delete any that are not in the list\n    elect_id, ref_id = (self.LFPBandElectrode() &amp; key).fetch(\n        \"electrode_id\", \"reference_elect_id\"\n    )\n    for e, r in zip(elect_id, ref_id):\n        if not len(np.where((electrode_list == e) &amp; (ref_list == r))[0]):\n            key[\"electrode_id\"] = e\n            key[\"reference_elect_id\"] = r\n            (self.LFPBandElectrode() &amp; key).delete()\n\n    # iterate through all of the new elements and add them\n    for e, r in zip(electrode_list, ref_list):\n        key[\"electrode_id\"] = e\n        query = Electrode &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_id\": e,\n        }\n        key[\"electrode_group_name\"] = query.fetch1(\"electrode_group_name\")\n        key[\"reference_elect_id\"] = r\n        self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_filter/", "title": "common_filter.py", "text": ""}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters", "title": "<code>FirFilterParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>@schema\nclass FirFilterParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    filter_name: varchar(80)           # descriptive name of this filter\n    filter_sampling_rate: int          # sampling rate for this filter\n    ---\n    filter_type: enum(\"lowpass\", \"highpass\", \"bandpass\")\n    filter_low_stop = 0: float     # lowest freq for stop band for low filt\n    filter_low_pass = 0: float     # lowest freq for pass band of low filt\n    filter_high_pass = 0: float    # highest freq for pass band for high filt\n    filter_high_stop = 0: float    # highest freq for stop band of high filt\n    filter_comments: varchar(2000) # comments about the filter\n    filter_band_edges: blob        # numpy array of filter bands\n                                   # redundant with individual parameters\n    filter_coeff: longblob         # numpy array of filter coefficients\n    \"\"\"\n\n    def add_filter(\n        self,\n        filter_name: str,\n        fs: float,\n        filter_type: str,\n        band_edges: list,\n        comments: str = \"\",\n    ) -&gt; None:\n        \"\"\"Add filter to the Filter table.\n\n        Parameters\n        ----------\n        filter_name: str\n            The name of the filter.\n        fs: float\n            The filter sampling rate.\n        filter_type: str\n            The type of the filter ('lowpass', 'highpass', or 'bandpass').\n        band_edges: List[float]\n            The band edges for the filter.\n        comments: str, optional)\n            Additional comments for the filter. Default \"\".\n\n        Returns\n        -------\n        None\n            Returns None if there is an error in the filter type or band\n            frequencies.\n\n        Raises\n        ------\n        Exception:\n            Raises an exception if an unexpected filter type is encountered.\n        \"\"\"\n        VALID_FILTERS = {\"lowpass\": 2, \"highpass\": 2, \"bandpass\": 4}\n        FILTER_ERR = \"Error in Filter.add_filter: \"\n        FILTER_N_ERR = FILTER_ERR + \"filter {} requires {} band_frequencies.\"\n\n        # add an FIR bandpass filter of the specified type.\n        # band_edges should be as follows:\n        #   low pass : [high_pass high_stop]\n        #   high pass: [low stop low pass]\n        #   band pass: [low_stop low_pass high_pass high_stop].\n        if filter_type not in VALID_FILTERS:\n            logger.error(\n                FILTER_ERR\n                + f\"{filter_type} not valid type: {VALID_FILTERS.keys()}\"\n            )\n            return None\n\n        if not len(band_edges) == VALID_FILTERS[filter_type]:\n            logger.error(\n                FILTER_N_ERR.format(filter_name, VALID_FILTERS[filter_type])\n            )\n            return None\n\n        gsp = _import_ghostipy()\n        TRANS_SPLINE = 2  # transition spline will be quadratic\n\n        if filter_type != \"bandpass\":\n            transition_width = band_edges[1] - band_edges[0]\n\n        else:\n            # transition width is mean of left and right transition regions\n            transition_width = (\n                (band_edges[1] - band_edges[0])\n                + (band_edges[3] - band_edges[2])\n            ) / 2.0\n\n        numtaps = gsp.estimate_taps(fs, transition_width)\n        filterdict = {\n            \"filter_type\": filter_type,\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": fs,\n            \"filter_comments\": comments,\n            \"filter_low_stop\": 0,\n            \"filter_low_pass\": 0,\n            \"filter_high_pass\": 0,\n            \"filter_high_stop\": 0,\n            \"filter_band_edges\": np.asarray(band_edges),\n        }\n\n        # set the desired frequency response\n        if filter_type == \"lowpass\":\n            desired = [1, 0]\n            pass_stop_dict = {\n                \"filter_high_pass\": band_edges[0],\n                \"filter_high_stop\": band_edges[1],\n            }\n        elif filter_type == \"highpass\":\n            desired = [0, 1]\n            pass_stop_dict = {\n                \"filter_low_stop\": band_edges[0],\n                \"filter_low_pass\": band_edges[1],\n            }\n        else:\n            desired = [0, 1, 1, 0]\n            pass_stop_dict = {\n                \"filter_low_stop\": band_edges[0],\n                \"filter_low_pass\": band_edges[1],\n                \"filter_high_pass\": band_edges[2],\n                \"filter_high_stop\": band_edges[3],\n            }\n\n        # create 1d array for coefficients\n        filterdict.update(\n            {\n                **pass_stop_dict,\n                \"filter_coeff\": np.array(\n                    gsp.firdesign(\n                        numtaps, band_edges, desired, fs=fs, p=TRANS_SPLINE\n                    ),\n                    ndmin=1,\n                ),\n            }\n        )\n\n        self.insert1(filterdict, skip_duplicates=True)\n\n    def _filter_restrict(self, filter_name, fs):\n        return (\n            self &amp; {\"filter_name\": filter_name} &amp; {\"filter_sampling_rate\": fs}\n        ).fetch1()\n\n    def plot_magnitude(self, filter_name, fs, return_fig=False):\n        filter_dict = self._filter_restrict(filter_name, fs)\n        plt.figure()\n        w, h = signal.freqz(filter_dict[\"filter_coeff\"], worN=65536)\n        magnitude = 20 * np.log10(np.abs(h))\n        plt.plot(w / np.pi * fs / 2, magnitude)\n        plt.xlabel(\"Frequency (Hz)\")\n        plt.ylabel(\"Magnitude\")\n        plt.title(\"Frequency Response\")\n        plt.xlim(0, np.max(filter_dict[\"filter_band_edges\"] * 2))\n        plt.ylim(np.min(magnitude), -1 * np.min(magnitude) * 0.1)\n        plt.grid(True)\n        if return_fig:\n            return plt.gcf()\n\n    def plot_fir_filter(self, filter_name, fs, return_fig=False):\n        filter_dict = self._filter_restrict(filter_name, fs)\n        plt.figure()\n        plt.clf()\n        plt.plot(filter_dict[\"filter_coeff\"], \"k\")\n        plt.xlabel(\"Coefficient\")\n        plt.ylabel(\"Magnitude\")\n        plt.title(\"Filter Taps\")\n        plt.grid(True)\n        if return_fig:\n            return plt.gcf()\n\n    def filter_delay(self, filter_name, fs):\n        return self.calc_filter_delay(\n            self._filter_restrict(filter_name, fs)[\"filter_coeff\"]\n        )\n\n    def _time_bound_check(self, start, stop, all, nsamples):\n        timestamp_warn = \"Interval time warning: \"\n        if start &lt; all[0]:\n            warnings.warn(\n                timestamp_warn\n                + \"start time smaller than first timestamp, \"\n                + f\"substituting first: {start} &lt; {all[0]}\"\n            )\n            start = all[0]\n\n        if stop &gt; all[-1]:\n            logger.warning(\n                timestamp_warn\n                + \"stop time larger than last timestamp, \"\n                + f\"substituting last: {stop} &lt; {all[-1]}\"\n            )\n            stop = all[-1]\n\n        frm, to = np.searchsorted(all, (start, stop))\n        to = min(to, nsamples)\n        return frm, to\n\n    def filter_data_nwb(\n        self,\n        analysis_file_abs_path: str,\n        eseries: pynwb.ecephys.ElectricalSeries,\n        filter_coeff: np.ndarray,\n        valid_times: np.ndarray,\n        electrode_ids: list,\n        decimation: int,\n        description: str = \"filtered data\",\n        data_type: Union[None, str] = None,\n    ):\n        \"\"\"\n        Filter data from an NWB electrical series using the ghostipy package,\n        and save the result as a new electrical series in the analysis NWB file.\n\n        Parameters\n        ----------\n        analysis_file_abs_path : str\n            Full path to the analysis NWB file.\n        eseries : pynwb.ecephys.ElectricalSeries\n            Electrical series with data to be filtered.\n        filter_coeff : np.ndarray\n            Array with filter coefficients for FIR filter.\n        valid_times : np.ndarray\n            Array with start and stop times of intervals to be filtered.\n        electrode_ids : list\n            List of electrode IDs to filter.\n        decimation : int\n            Decimation factor.\n        description : str\n            Description of the filtered data.\n        data_type : Union[None, str]\n            Type of data (e.g., \"LFP\").\n\n        Returns\n        -------\n        tuple\n            The NWB object ID of the filtered data and a list containing the\n            first and last timestamps.\n        \"\"\"\n        # Note: type -&gt; data_type to avoid conflict with builtin type\n        # All existing refs to this func use positional args, so no need to\n        # adjust elsewhere, but low probability of issues with custom scripts\n\n        MEM_USE_LIMIT = 0.9  # % of RAM use permitted\n\n        gsp = _import_ghostipy()\n\n        data_on_disk = eseries.data\n        timestamps_on_disk = eseries.timestamps\n\n        n_samples = len(timestamps_on_disk)\n        time_axis = 0 if data_on_disk.shape[0] == n_samples else 1\n        electrode_axis = 1 - time_axis\n\n        n_electrodes = data_on_disk.shape[electrode_axis]\n        input_dim_restrictions = [None] * len(data_on_disk.shape)\n\n        # Get input dimension restrictions\n        input_dim_restrictions[electrode_axis] = np.s_[\n            get_electrode_indices(eseries, electrode_ids)\n        ]\n\n        indices = []\n        output_shape_list = [0] * len(data_on_disk.shape)\n        output_shape_list[electrode_axis] = len(electrode_ids)\n        data_dtype = data_on_disk[0][0].dtype\n\n        filter_delay = self.calc_filter_delay(filter_coeff)\n\n        output_offsets = [0]\n\n        for a_start, a_stop in valid_times:\n            frm, to = self._time_bound_check(\n                a_start, a_stop, timestamps_on_disk, n_samples\n            )\n\n            indices.append((frm, to))\n\n            shape, _ = gsp.filter_data_fir(\n                data_on_disk,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[frm, to - 1],\n                output_index_bounds=[filter_delay, filter_delay + to - frm],\n                describe_dims=True,\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n            )\n            output_offsets.append(output_offsets[-1] + shape[time_axis])\n            output_shape_list[time_axis] += shape[time_axis]\n\n        # Create dynamic table region and electrode series, write/close file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n\n            # get the indices of the electrodes in the electrode table\n            elect_ind = get_electrode_indices(nwbf, electrode_ids)\n\n            electrode_table_region = nwbf.create_electrode_table_region(\n                elect_ind, \"filtered electrode table\"\n            )\n            es = pynwb.ecephys.ElectricalSeries(\n                name=\"filtered data\",\n                data=np.empty(tuple(output_shape_list), dtype=data_dtype),\n                electrodes=electrode_table_region,\n                timestamps=np.empty(output_shape_list[time_axis]),\n                description=description,\n            )\n            if data_type == \"LFP\":\n                ecephys_module = nwbf.create_processing_module(\n                    name=\"ecephys\", description=description\n                )\n                ecephys_module.add(pynwb.ecephys.LFP(electrical_series=es))\n            else:\n                nwbf.add_scratch(es)\n\n            io.write(nwbf)\n\n        # Reload NWB file to get h5py objects for data/timestamps\n        # NOTE: CBroz - why io context within io context? Unindenting\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            es = nwbf.objects[es.object_id]\n            filtered_data = es.data\n            new_timestamps = es.timestamps\n            indices = np.array(indices, ndmin=2)\n            # Filter and write the output dataset\n            ts_offset = 0\n\n            logger.info(\"Filtering data\")\n            for ii, (start, stop) in enumerate(indices):\n                # Calc size of timestamps + data, check if &lt; 90% of RAM\n                interval_samples = stop - start\n                req_mem = interval_samples * (\n                    timestamps_on_disk[0].itemsize\n                    + n_electrodes * data_on_disk[0][0].itemsize\n                )\n                if req_mem &lt; MEM_USE_LIMIT * psutil.virtual_memory().available:\n                    logger.info(f\"Interval {ii}: loading data into memory\")\n                    timestamps = np.asarray(\n                        timestamps_on_disk[start:stop],\n                        dtype=timestamps_on_disk[0].dtype,\n                    )\n                    if time_axis == 0:\n                        data = np.asarray(\n                            data_on_disk[start:stop, :], dtype=data_dtype\n                        )\n                    else:\n                        data = np.asarray(\n                            data_on_disk[:, start:stop], dtype=data_dtype\n                        )\n                    extracted_ts = timestamps[0::decimation]\n                    new_timestamps[\n                        ts_offset : ts_offset + len(extracted_ts)\n                    ] = extracted_ts\n                    ts_offset += len(extracted_ts)\n                    input_index_bounds = [0, interval_samples - 1]\n\n                else:\n                    logger.info(f\"Interval {ii}: leaving data on disk\")\n                    data = data_on_disk\n                    timestamps = timestamps_on_disk\n                    extracted_ts = timestamps[start:stop:decimation]\n                    new_timestamps[\n                        ts_offset : ts_offset + len(extracted_ts)\n                    ] = extracted_ts\n                    ts_offset += len(extracted_ts)\n                    input_index_bounds = [start, stop]\n\n                # filter the data\n                gsp.filter_data_fir(\n                    data,\n                    filter_coeff,\n                    axis=time_axis,\n                    input_index_bounds=input_index_bounds,\n                    output_index_bounds=[\n                        filter_delay,\n                        filter_delay + stop - start,\n                    ],\n                    ds=decimation,\n                    input_dim_restrictions=input_dim_restrictions,\n                    outarray=filtered_data,\n                    output_offset=output_offsets[ii],\n                )\n\n            start_end = [new_timestamps[0], new_timestamps[-1]]\n\n            io.write(nwbf)\n\n        return es.object_id, start_end\n\n    def filter_data(\n        self,\n        timestamps,\n        data,\n        filter_coeff,\n        valid_times,\n        electrodes,\n        decimation,\n    ):\n        \"\"\"\n        Parameters\n        ----------\n        timestamps: numpy array\n            List of timestamps for data\n        data:\n            original data array\n        filter_coeff: numpy array\n            Filter coefficients for FIR filter\n        valid_times: 2D numpy array\n            Start and stop times of intervals to be filtered\n        electrodes: list\n            Electrodes to filter\n        decimation:\n            decimation factor\n\n        Return\n        ------\n        filtered_data, timestamps\n        \"\"\"\n\n        gsp = _import_ghostipy()\n\n        n_dim = len(data.shape)\n        n_samples = len(timestamps)\n        time_axis = 0 if data.shape[0] == n_samples else 1\n        electrode_axis = 1 - time_axis\n        input_dim_restrictions = [None] * n_dim\n        input_dim_restrictions[electrode_axis] = np.s_[electrodes]\n\n        indices = []\n        output_shape_list = [0] * n_dim\n        output_shape_list[electrode_axis] = len(electrodes)\n        output_offsets = [0]\n\n        filter_delay = self.calc_filter_delay(filter_coeff)\n        for a_start, a_stop in valid_times:\n            frm, to = self._time_bound_check(\n                a_start, a_stop, timestamps, n_samples\n            )\n            if np.isclose(frm, to, rtol=0, atol=1e-8):\n                continue\n            indices.append((frm, to))\n\n            shape, _ = gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[frm, to],\n                output_index_bounds=[filter_delay, filter_delay + to - frm],\n                describe_dims=True,\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n            )\n            output_offsets.append(output_offsets[-1] + shape[time_axis])\n            output_shape_list[time_axis] += shape[time_axis]\n\n        # create the dataset and the timestamps array\n        filtered_data = np.empty(tuple(output_shape_list), dtype=data.dtype)\n\n        new_timestamps = np.empty(\n            (output_shape_list[time_axis],), timestamps.dtype\n        )\n\n        indices = np.array(indices, ndmin=2)\n\n        # Filter  the output dataset\n        ts_offset = 0\n\n        for ii, (start, stop) in enumerate(indices):\n            extracted_ts = timestamps[start:stop:decimation]\n\n            new_timestamps[ts_offset : ts_offset + len(extracted_ts)] = (\n                extracted_ts\n            )\n            ts_offset += len(extracted_ts)\n\n            # finally ready to filter data!\n            gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=[start, stop],\n                output_index_bounds=[filter_delay, filter_delay + stop - start],\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n                outarray=filtered_data,\n                output_offset=output_offsets[ii],\n            )\n\n        return filtered_data, new_timestamps\n\n    def calc_filter_delay(self, filter_coeff):\n        \"\"\"\n        Parameters\n        ----------\n        filter_coeff: numpy array\n\n        Return\n        ------\n        filter delay: int\n        \"\"\"\n        return (len(filter_coeff) - 1) // 2\n\n    def create_standard_filters(self):\n        \"\"\"Add standard filters to the Filter table\n\n        Includes 0-400 Hz low pass for continuous raw data -&gt; LFP\n        \"\"\"\n        self.add_filter(\n            \"LFP 0-400 Hz\",\n            20000,\n            \"lowpass\",\n            [400, 425],\n            \"standard LFP filter for 20 KHz data\",\n        )\n        self.add_filter(\n            \"LFP 0-400 Hz\",\n            30000,\n            \"lowpass\",\n            [400, 425],\n            \"standard LFP filter for 30 KHz data\",\n        )\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.add_filter", "title": "<code>add_filter(filter_name, fs, filter_type, band_edges, comments='')</code>", "text": "<p>Add filter to the Filter table.</p> <p>Parameters:</p> Name Type Description Default <code>filter_name</code> <code>str</code> <p>The name of the filter.</p> required <code>fs</code> <code>float</code> <p>The filter sampling rate.</p> required <code>filter_type</code> <code>str</code> <p>The type of the filter ('lowpass', 'highpass', or 'bandpass').</p> required <code>band_edges</code> <code>list</code> <p>The band edges for the filter.</p> required <code>comments</code> <code>str</code> <p>Additional comments for the filter. Default \"\".</p> <code>''</code> <p>Returns:</p> Type Description <code>None</code> <p>Returns None if there is an error in the filter type or band frequencies.</p> <p>Raises:</p> Type Description <code>Exception:</code> <p>Raises an exception if an unexpected filter type is encountered.</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def add_filter(\n    self,\n    filter_name: str,\n    fs: float,\n    filter_type: str,\n    band_edges: list,\n    comments: str = \"\",\n) -&gt; None:\n    \"\"\"Add filter to the Filter table.\n\n    Parameters\n    ----------\n    filter_name: str\n        The name of the filter.\n    fs: float\n        The filter sampling rate.\n    filter_type: str\n        The type of the filter ('lowpass', 'highpass', or 'bandpass').\n    band_edges: List[float]\n        The band edges for the filter.\n    comments: str, optional)\n        Additional comments for the filter. Default \"\".\n\n    Returns\n    -------\n    None\n        Returns None if there is an error in the filter type or band\n        frequencies.\n\n    Raises\n    ------\n    Exception:\n        Raises an exception if an unexpected filter type is encountered.\n    \"\"\"\n    VALID_FILTERS = {\"lowpass\": 2, \"highpass\": 2, \"bandpass\": 4}\n    FILTER_ERR = \"Error in Filter.add_filter: \"\n    FILTER_N_ERR = FILTER_ERR + \"filter {} requires {} band_frequencies.\"\n\n    # add an FIR bandpass filter of the specified type.\n    # band_edges should be as follows:\n    #   low pass : [high_pass high_stop]\n    #   high pass: [low stop low pass]\n    #   band pass: [low_stop low_pass high_pass high_stop].\n    if filter_type not in VALID_FILTERS:\n        logger.error(\n            FILTER_ERR\n            + f\"{filter_type} not valid type: {VALID_FILTERS.keys()}\"\n        )\n        return None\n\n    if not len(band_edges) == VALID_FILTERS[filter_type]:\n        logger.error(\n            FILTER_N_ERR.format(filter_name, VALID_FILTERS[filter_type])\n        )\n        return None\n\n    gsp = _import_ghostipy()\n    TRANS_SPLINE = 2  # transition spline will be quadratic\n\n    if filter_type != \"bandpass\":\n        transition_width = band_edges[1] - band_edges[0]\n\n    else:\n        # transition width is mean of left and right transition regions\n        transition_width = (\n            (band_edges[1] - band_edges[0])\n            + (band_edges[3] - band_edges[2])\n        ) / 2.0\n\n    numtaps = gsp.estimate_taps(fs, transition_width)\n    filterdict = {\n        \"filter_type\": filter_type,\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": fs,\n        \"filter_comments\": comments,\n        \"filter_low_stop\": 0,\n        \"filter_low_pass\": 0,\n        \"filter_high_pass\": 0,\n        \"filter_high_stop\": 0,\n        \"filter_band_edges\": np.asarray(band_edges),\n    }\n\n    # set the desired frequency response\n    if filter_type == \"lowpass\":\n        desired = [1, 0]\n        pass_stop_dict = {\n            \"filter_high_pass\": band_edges[0],\n            \"filter_high_stop\": band_edges[1],\n        }\n    elif filter_type == \"highpass\":\n        desired = [0, 1]\n        pass_stop_dict = {\n            \"filter_low_stop\": band_edges[0],\n            \"filter_low_pass\": band_edges[1],\n        }\n    else:\n        desired = [0, 1, 1, 0]\n        pass_stop_dict = {\n            \"filter_low_stop\": band_edges[0],\n            \"filter_low_pass\": band_edges[1],\n            \"filter_high_pass\": band_edges[2],\n            \"filter_high_stop\": band_edges[3],\n        }\n\n    # create 1d array for coefficients\n    filterdict.update(\n        {\n            **pass_stop_dict,\n            \"filter_coeff\": np.array(\n                gsp.firdesign(\n                    numtaps, band_edges, desired, fs=fs, p=TRANS_SPLINE\n                ),\n                ndmin=1,\n            ),\n        }\n    )\n\n    self.insert1(filterdict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data_nwb", "title": "<code>filter_data_nwb(analysis_file_abs_path, eseries, filter_coeff, valid_times, electrode_ids, decimation, description='filtered data', data_type=None)</code>", "text": "<p>Filter data from an NWB electrical series using the ghostipy package, and save the result as a new electrical series in the analysis NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_abs_path</code> <code>str</code> <p>Full path to the analysis NWB file.</p> required <code>eseries</code> <code>ElectricalSeries</code> <p>Electrical series with data to be filtered.</p> required <code>filter_coeff</code> <code>ndarray</code> <p>Array with filter coefficients for FIR filter.</p> required <code>valid_times</code> <code>ndarray</code> <p>Array with start and stop times of intervals to be filtered.</p> required <code>electrode_ids</code> <code>list</code> <p>List of electrode IDs to filter.</p> required <code>decimation</code> <code>int</code> <p>Decimation factor.</p> required <code>description</code> <code>str</code> <p>Description of the filtered data.</p> <code>'filtered data'</code> <code>data_type</code> <code>Union[None, str]</code> <p>Type of data (e.g., \"LFP\").</p> <code>None</code> <p>Returns:</p> Type Description <code>tuple</code> <p>The NWB object ID of the filtered data and a list containing the first and last timestamps.</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def filter_data_nwb(\n    self,\n    analysis_file_abs_path: str,\n    eseries: pynwb.ecephys.ElectricalSeries,\n    filter_coeff: np.ndarray,\n    valid_times: np.ndarray,\n    electrode_ids: list,\n    decimation: int,\n    description: str = \"filtered data\",\n    data_type: Union[None, str] = None,\n):\n    \"\"\"\n    Filter data from an NWB electrical series using the ghostipy package,\n    and save the result as a new electrical series in the analysis NWB file.\n\n    Parameters\n    ----------\n    analysis_file_abs_path : str\n        Full path to the analysis NWB file.\n    eseries : pynwb.ecephys.ElectricalSeries\n        Electrical series with data to be filtered.\n    filter_coeff : np.ndarray\n        Array with filter coefficients for FIR filter.\n    valid_times : np.ndarray\n        Array with start and stop times of intervals to be filtered.\n    electrode_ids : list\n        List of electrode IDs to filter.\n    decimation : int\n        Decimation factor.\n    description : str\n        Description of the filtered data.\n    data_type : Union[None, str]\n        Type of data (e.g., \"LFP\").\n\n    Returns\n    -------\n    tuple\n        The NWB object ID of the filtered data and a list containing the\n        first and last timestamps.\n    \"\"\"\n    # Note: type -&gt; data_type to avoid conflict with builtin type\n    # All existing refs to this func use positional args, so no need to\n    # adjust elsewhere, but low probability of issues with custom scripts\n\n    MEM_USE_LIMIT = 0.9  # % of RAM use permitted\n\n    gsp = _import_ghostipy()\n\n    data_on_disk = eseries.data\n    timestamps_on_disk = eseries.timestamps\n\n    n_samples = len(timestamps_on_disk)\n    time_axis = 0 if data_on_disk.shape[0] == n_samples else 1\n    electrode_axis = 1 - time_axis\n\n    n_electrodes = data_on_disk.shape[electrode_axis]\n    input_dim_restrictions = [None] * len(data_on_disk.shape)\n\n    # Get input dimension restrictions\n    input_dim_restrictions[electrode_axis] = np.s_[\n        get_electrode_indices(eseries, electrode_ids)\n    ]\n\n    indices = []\n    output_shape_list = [0] * len(data_on_disk.shape)\n    output_shape_list[electrode_axis] = len(electrode_ids)\n    data_dtype = data_on_disk[0][0].dtype\n\n    filter_delay = self.calc_filter_delay(filter_coeff)\n\n    output_offsets = [0]\n\n    for a_start, a_stop in valid_times:\n        frm, to = self._time_bound_check(\n            a_start, a_stop, timestamps_on_disk, n_samples\n        )\n\n        indices.append((frm, to))\n\n        shape, _ = gsp.filter_data_fir(\n            data_on_disk,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[frm, to - 1],\n            output_index_bounds=[filter_delay, filter_delay + to - frm],\n            describe_dims=True,\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n        )\n        output_offsets.append(output_offsets[-1] + shape[time_axis])\n        output_shape_list[time_axis] += shape[time_axis]\n\n    # Create dynamic table region and electrode series, write/close file\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n\n        # get the indices of the electrodes in the electrode table\n        elect_ind = get_electrode_indices(nwbf, electrode_ids)\n\n        electrode_table_region = nwbf.create_electrode_table_region(\n            elect_ind, \"filtered electrode table\"\n        )\n        es = pynwb.ecephys.ElectricalSeries(\n            name=\"filtered data\",\n            data=np.empty(tuple(output_shape_list), dtype=data_dtype),\n            electrodes=electrode_table_region,\n            timestamps=np.empty(output_shape_list[time_axis]),\n            description=description,\n        )\n        if data_type == \"LFP\":\n            ecephys_module = nwbf.create_processing_module(\n                name=\"ecephys\", description=description\n            )\n            ecephys_module.add(pynwb.ecephys.LFP(electrical_series=es))\n        else:\n            nwbf.add_scratch(es)\n\n        io.write(nwbf)\n\n    # Reload NWB file to get h5py objects for data/timestamps\n    # NOTE: CBroz - why io context within io context? Unindenting\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path, mode=\"a\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        es = nwbf.objects[es.object_id]\n        filtered_data = es.data\n        new_timestamps = es.timestamps\n        indices = np.array(indices, ndmin=2)\n        # Filter and write the output dataset\n        ts_offset = 0\n\n        logger.info(\"Filtering data\")\n        for ii, (start, stop) in enumerate(indices):\n            # Calc size of timestamps + data, check if &lt; 90% of RAM\n            interval_samples = stop - start\n            req_mem = interval_samples * (\n                timestamps_on_disk[0].itemsize\n                + n_electrodes * data_on_disk[0][0].itemsize\n            )\n            if req_mem &lt; MEM_USE_LIMIT * psutil.virtual_memory().available:\n                logger.info(f\"Interval {ii}: loading data into memory\")\n                timestamps = np.asarray(\n                    timestamps_on_disk[start:stop],\n                    dtype=timestamps_on_disk[0].dtype,\n                )\n                if time_axis == 0:\n                    data = np.asarray(\n                        data_on_disk[start:stop, :], dtype=data_dtype\n                    )\n                else:\n                    data = np.asarray(\n                        data_on_disk[:, start:stop], dtype=data_dtype\n                    )\n                extracted_ts = timestamps[0::decimation]\n                new_timestamps[\n                    ts_offset : ts_offset + len(extracted_ts)\n                ] = extracted_ts\n                ts_offset += len(extracted_ts)\n                input_index_bounds = [0, interval_samples - 1]\n\n            else:\n                logger.info(f\"Interval {ii}: leaving data on disk\")\n                data = data_on_disk\n                timestamps = timestamps_on_disk\n                extracted_ts = timestamps[start:stop:decimation]\n                new_timestamps[\n                    ts_offset : ts_offset + len(extracted_ts)\n                ] = extracted_ts\n                ts_offset += len(extracted_ts)\n                input_index_bounds = [start, stop]\n\n            # filter the data\n            gsp.filter_data_fir(\n                data,\n                filter_coeff,\n                axis=time_axis,\n                input_index_bounds=input_index_bounds,\n                output_index_bounds=[\n                    filter_delay,\n                    filter_delay + stop - start,\n                ],\n                ds=decimation,\n                input_dim_restrictions=input_dim_restrictions,\n                outarray=filtered_data,\n                output_offset=output_offsets[ii],\n            )\n\n        start_end = [new_timestamps[0], new_timestamps[-1]]\n\n        io.write(nwbf)\n\n    return es.object_id, start_end\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data", "title": "<code>filter_data(timestamps, data, filter_coeff, valid_times, electrodes, decimation)</code>", "text": "<p>Parameters:</p> Name Type Description Default <code>timestamps</code> <p>List of timestamps for data</p> required <code>data</code> <p>original data array</p> required <code>filter_coeff</code> <p>Filter coefficients for FIR filter</p> required <code>valid_times</code> <p>Start and stop times of intervals to be filtered</p> required <code>electrodes</code> <p>Electrodes to filter</p> required <code>decimation</code> <p>decimation factor</p> required"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.filter_data--return", "title": "Return", "text": "<p>filtered_data, timestamps</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def filter_data(\n    self,\n    timestamps,\n    data,\n    filter_coeff,\n    valid_times,\n    electrodes,\n    decimation,\n):\n    \"\"\"\n    Parameters\n    ----------\n    timestamps: numpy array\n        List of timestamps for data\n    data:\n        original data array\n    filter_coeff: numpy array\n        Filter coefficients for FIR filter\n    valid_times: 2D numpy array\n        Start and stop times of intervals to be filtered\n    electrodes: list\n        Electrodes to filter\n    decimation:\n        decimation factor\n\n    Return\n    ------\n    filtered_data, timestamps\n    \"\"\"\n\n    gsp = _import_ghostipy()\n\n    n_dim = len(data.shape)\n    n_samples = len(timestamps)\n    time_axis = 0 if data.shape[0] == n_samples else 1\n    electrode_axis = 1 - time_axis\n    input_dim_restrictions = [None] * n_dim\n    input_dim_restrictions[electrode_axis] = np.s_[electrodes]\n\n    indices = []\n    output_shape_list = [0] * n_dim\n    output_shape_list[electrode_axis] = len(electrodes)\n    output_offsets = [0]\n\n    filter_delay = self.calc_filter_delay(filter_coeff)\n    for a_start, a_stop in valid_times:\n        frm, to = self._time_bound_check(\n            a_start, a_stop, timestamps, n_samples\n        )\n        if np.isclose(frm, to, rtol=0, atol=1e-8):\n            continue\n        indices.append((frm, to))\n\n        shape, _ = gsp.filter_data_fir(\n            data,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[frm, to],\n            output_index_bounds=[filter_delay, filter_delay + to - frm],\n            describe_dims=True,\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n        )\n        output_offsets.append(output_offsets[-1] + shape[time_axis])\n        output_shape_list[time_axis] += shape[time_axis]\n\n    # create the dataset and the timestamps array\n    filtered_data = np.empty(tuple(output_shape_list), dtype=data.dtype)\n\n    new_timestamps = np.empty(\n        (output_shape_list[time_axis],), timestamps.dtype\n    )\n\n    indices = np.array(indices, ndmin=2)\n\n    # Filter  the output dataset\n    ts_offset = 0\n\n    for ii, (start, stop) in enumerate(indices):\n        extracted_ts = timestamps[start:stop:decimation]\n\n        new_timestamps[ts_offset : ts_offset + len(extracted_ts)] = (\n            extracted_ts\n        )\n        ts_offset += len(extracted_ts)\n\n        # finally ready to filter data!\n        gsp.filter_data_fir(\n            data,\n            filter_coeff,\n            axis=time_axis,\n            input_index_bounds=[start, stop],\n            output_index_bounds=[filter_delay, filter_delay + stop - start],\n            ds=decimation,\n            input_dim_restrictions=input_dim_restrictions,\n            outarray=filtered_data,\n            output_offset=output_offsets[ii],\n        )\n\n    return filtered_data, new_timestamps\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.calc_filter_delay", "title": "<code>calc_filter_delay(filter_coeff)</code>", "text": "<p>Parameters:</p> Name Type Description Default <code>filter_coeff</code> required"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.calc_filter_delay--return", "title": "Return", "text": "<p>filter delay: int</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def calc_filter_delay(self, filter_coeff):\n    \"\"\"\n    Parameters\n    ----------\n    filter_coeff: numpy array\n\n    Return\n    ------\n    filter delay: int\n    \"\"\"\n    return (len(filter_coeff) - 1) // 2\n</code></pre>"}, {"location": "api/common/common_filter/#src.spyglass.common.common_filter.FirFilterParameters.create_standard_filters", "title": "<code>create_standard_filters()</code>", "text": "<p>Add standard filters to the Filter table</p> <p>Includes 0-400 Hz low pass for continuous raw data -&gt; LFP</p> Source code in <code>src/spyglass/common/common_filter.py</code> <pre><code>def create_standard_filters(self):\n    \"\"\"Add standard filters to the Filter table\n\n    Includes 0-400 Hz low pass for continuous raw data -&gt; LFP\n    \"\"\"\n    self.add_filter(\n        \"LFP 0-400 Hz\",\n        20000,\n        \"lowpass\",\n        [400, 425],\n        \"standard LFP filter for 20 KHz data\",\n    )\n    self.add_filter(\n        \"LFP 0-400 Hz\",\n        30000,\n        \"lowpass\",\n        [400, 425],\n        \"standard LFP filter for 30 KHz data\",\n    )\n</code></pre>"}, {"location": "api/common/common_interval/", "title": "common_interval.py", "text": ""}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.IntervalList", "title": "<code>IntervalList</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>@schema\nclass IntervalList(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Time intervals used for analysis\n    -&gt; Session\n    interval_list_name: varchar(170)  # descriptive name of this interval list\n    ---\n    valid_times: longblob  # numpy array with start/end times for each interval\n    pipeline = \"\": varchar(64)  # type of interval list (e.g. 'position', 'spikesorting_recording_v1')\n    \"\"\"\n\n    # See #630, #664. Excessive key length.\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf, *, nwb_file_name):\n        \"\"\"Add each entry in the NWB file epochs table to the IntervalList.\n\n        The interval list name for each epoch is set to the first tag for the\n        epoch. If the epoch has no tags, then 'interval_x' will be used as the\n        interval list name, where x is the index (0-indexed) of the epoch in the\n        epochs table. The start time and stop time of the epoch are stored in\n        the valid_times field as a numpy array of [start time, stop time] for\n        each epoch.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        nwb_file_name : str\n            The file name of the NWB file, used as a primary key to the Session\n            table.\n        \"\"\"\n        if nwbf.epochs is None:\n            logger.info(\"No epochs found in NWB file.\")\n            return\n\n        epochs = nwbf.epochs.to_dataframe()\n\n        for _, epoch_data in epochs.iterrows():\n            epoch_dict = {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": (\n                    epoch_data.tags[0]\n                    if epoch_data.tags\n                    else f\"interval_{epoch_data[0]}\"\n                ),\n                \"valid_times\": np.asarray(\n                    [[epoch_data.start_time, epoch_data.stop_time]]\n                ),\n            }\n\n            cls.insert1(epoch_dict, skip_duplicates=True)\n\n    def plot_intervals(self, figsize=(20, 5), return_fig=False):\n        interval_list = pd.DataFrame(self)\n        fig, ax = plt.subplots(figsize=figsize)\n        interval_count = 0\n        for row in interval_list.itertuples(index=False):\n            for interval in row.valid_times:\n                ax.plot(interval, [interval_count, interval_count])\n                ax.scatter(\n                    interval,\n                    [interval_count, interval_count],\n                    alpha=0.8,\n                    zorder=2,\n                )\n            interval_count += 1\n        ax.set_yticks(np.arange(interval_list.shape[0]))\n        ax.set_yticklabels(interval_list.interval_list_name)\n        ax.set_xlabel(\"Time [s]\")\n        ax.grid(True)\n        if return_fig:\n            return fig\n\n    def plot_epoch_pos_raw_intervals(self, figsize=(20, 5), return_fig=False):\n        interval_list = pd.DataFrame(self)\n        fig, ax = plt.subplots(figsize=(30, 3))\n\n        raw_data_valid_times = interval_list.loc[\n            interval_list.interval_list_name == \"raw data valid times\"\n        ].valid_times\n        interval_y = 1\n\n        for interval in np.asarray(raw_data_valid_times)[0]:\n            ax.plot(interval, [interval_y, interval_y])\n            ax.scatter(interval, [interval_y, interval_y], alpha=0.8, zorder=2)\n\n        epoch_valid_times = (\n            interval_list.set_index(\"interval_list_name\")\n            .filter(regex=r\"^[0-9]\", axis=0)\n            .valid_times\n        )\n        interval_y = 2\n        for epoch, valid_times in zip(\n            epoch_valid_times.index, epoch_valid_times\n        ):\n            for interval in valid_times:\n                ax.plot(interval, [interval_y, interval_y])\n                ax.scatter(\n                    interval, [interval_y, interval_y], alpha=0.8, zorder=2\n                )\n                ax.text(\n                    interval[0] + np.diff(interval)[0] / 2,\n                    interval_y,\n                    epoch,\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n\n        pos_valid_times = (\n            interval_list.set_index(\"interval_list_name\")\n            .filter(regex=r\"^pos \\d+ valid times$\", axis=0)\n            .valid_times\n        ).sort_index(key=lambda index: [int(name.split()[1]) for name in index])\n        interval_y = 0\n        for epoch, valid_times in zip(pos_valid_times.index, pos_valid_times):\n            for interval in valid_times:\n                ax.plot(interval, [interval_y, interval_y])\n                ax.scatter(\n                    interval, [interval_y, interval_y], alpha=0.8, zorder=2\n                )\n                ax.text(\n                    interval[0] + np.diff(interval)[0] / 2,\n                    interval_y,\n                    epoch.strip(\" valid times\"),\n                    ha=\"center\",\n                    va=\"bottom\",\n                )\n\n        ax.set_ylim((-0.25, 2.25))\n        ax.set_yticks(np.arange(3))\n        ax.set_yticklabels([\"pos valid times\", \"raw data valid times\", \"epoch\"])\n        ax.set_xlabel(\"Time [s]\")\n        ax.grid(True)\n        if return_fig:\n            return fig\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.IntervalList.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf, *, nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Add each entry in the NWB file epochs table to the IntervalList.</p> <p>The interval list name for each epoch is set to the first tag for the epoch. If the epoch has no tags, then 'interval_x' will be used as the interval list name, where x is the index (0-indexed) of the epoch in the epochs table. The start time and stop time of the epoch are stored in the valid_times field as a numpy array of [start time, stop time] for each epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>nwb_file_name</code> <code>str</code> <p>The file name of the NWB file, used as a primary key to the Session table.</p> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf, *, nwb_file_name):\n    \"\"\"Add each entry in the NWB file epochs table to the IntervalList.\n\n    The interval list name for each epoch is set to the first tag for the\n    epoch. If the epoch has no tags, then 'interval_x' will be used as the\n    interval list name, where x is the index (0-indexed) of the epoch in the\n    epochs table. The start time and stop time of the epoch are stored in\n    the valid_times field as a numpy array of [start time, stop time] for\n    each epoch.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    nwb_file_name : str\n        The file name of the NWB file, used as a primary key to the Session\n        table.\n    \"\"\"\n    if nwbf.epochs is None:\n        logger.info(\"No epochs found in NWB file.\")\n        return\n\n    epochs = nwbf.epochs.to_dataframe()\n\n    for _, epoch_data in epochs.iterrows():\n        epoch_dict = {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": (\n                epoch_data.tags[0]\n                if epoch_data.tags\n                else f\"interval_{epoch_data[0]}\"\n            ),\n            \"valid_times\": np.asarray(\n                [[epoch_data.start_time, epoch_data.stop_time]]\n            ),\n        }\n\n        cls.insert1(epoch_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.intervals_by_length", "title": "<code>intervals_by_length(interval_list, min_length=0.0, max_length=10000000000.0)</code>", "text": "<p>Select intervals of certain lengths from an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>min_length</code> <code>float</code> <p>Minimum interval length in seconds. Defaults to 0.0.</p> <code>0.0</code> <code>max_length</code> <code>float</code> <p>Maximum interval length in seconds. Defaults to 1e10.</p> <code>10000000000.0</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def intervals_by_length(interval_list, min_length=0.0, max_length=1e10):\n    \"\"\"Select intervals of certain lengths from an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    min_length : float, optional\n        Minimum interval length in seconds. Defaults to 0.0.\n    max_length : float, optional\n        Maximum interval length in seconds. Defaults to 1e10.\n    \"\"\"\n    lengths = np.ravel(np.diff(interval_list))\n    return interval_list[\n        np.logical_and(lengths &gt; min_length, lengths &lt; max_length)\n    ]\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_contains_ind", "title": "<code>interval_list_contains_ind(interval_list, timestamps)</code>", "text": "<p>Find indices of list of timestamps contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_contains_ind(interval_list, timestamps):\n    \"\"\"Find indices of list of timestamps contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    ind = []\n    for interval in interval_list:\n        ind += np.ravel(\n            np.argwhere(\n                np.logical_and(\n                    timestamps &gt;= interval[0], timestamps &lt;= interval[1]\n                )\n            )\n        ).tolist()\n    return np.asarray(ind)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_contains", "title": "<code>interval_list_contains(interval_list, timestamps)</code>", "text": "<p>Find timestamps that are contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_contains(interval_list, timestamps):\n    \"\"\"Find timestamps that are contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    ind = []\n    for interval in interval_list:\n        ind += np.ravel(\n            np.argwhere(\n                np.logical_and(\n                    timestamps &gt;= interval[0], timestamps &lt;= interval[1]\n                )\n            )\n        ).tolist()\n    return timestamps[ind]\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_excludes_ind", "title": "<code>interval_list_excludes_ind(interval_list, timestamps)</code>", "text": "<p>Find indices of timestamps that are not contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_excludes_ind(interval_list, timestamps):\n    \"\"\"Find indices of timestamps that are not contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n\n    contained_inds = interval_list_contains_ind(interval_list, timestamps)\n    return np.setdiff1d(np.arange(len(timestamps)), contained_inds)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_excludes", "title": "<code>interval_list_excludes(interval_list, timestamps)</code>", "text": "<p>Find timestamps that are not contained in an interval list.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>array_like</code> <p>Each element is (start time, stop time), i.e. an interval in seconds.</p> required <code>timestamps</code> <code>array_like</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_excludes(interval_list, timestamps):\n    \"\"\"Find timestamps that are not contained in an interval list.\n\n    Parameters\n    ----------\n    interval_list : array_like\n        Each element is (start time, stop time), i.e. an interval in seconds.\n    timestamps : array_like\n    \"\"\"\n    contained_times = interval_list_contains(interval_list, timestamps)\n    return np.setdiff1d(timestamps, contained_times)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_intersect", "title": "<code>interval_list_intersect(interval_list1, interval_list2, min_length=0)</code>", "text": "<p>Finds the intersections between two interval lists</p> <p>Parameters:</p> Name Type Description Default <code>interval_list1</code> <code>np.array, (N,2) where N = number of intervals</code> required <code>interval_list2</code> <code>np.array, (N,2) where N = number of intervals</code> required <code>min_length</code> <code>float, optional.</code> <p>Minimum length of intervals to include, default 0</p> <code>0</code> <p>Each interval is (start time, stop time)</p> <p>Returns:</p> Name Type Description <code>interval_list</code> <code>(array, (N, 2))</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_intersect(interval_list1, interval_list2, min_length=0):\n    \"\"\"Finds the intersections between two interval lists\n\n    Parameters\n    ----------\n    interval_list1 : np.array, (N,2) where N = number of intervals\n    interval_list2 : np.array, (N,2) where N = number of intervals\n    min_length : float, optional.\n        Minimum length of intervals to include, default 0\n\n    Each interval is (start time, stop time)\n\n    Returns\n    -------\n    interval_list: np.array, (N,2)\n    \"\"\"\n\n    # Consolidate interval lists to disjoint int'ls by sorting &amp; applying union\n    interval_list1 = consolidate_intervals(interval_list1)\n    interval_list2 = consolidate_intervals(interval_list2)\n\n    # then do pairwise comparison and collect intersections\n    intersecting_intervals = [\n        _intersection(interval2, interval1)\n        for interval2 in interval_list2\n        for interval1 in interval_list1\n        if _intersection(interval2, interval1) is not None\n    ]\n\n    # if no intersection, then return an empty list\n    if not intersecting_intervals:\n        return []\n\n    intersecting_intervals = np.asarray(intersecting_intervals)\n    intersecting_intervals = intersecting_intervals[\n        np.argsort(intersecting_intervals[:, 0])\n    ]\n\n    return intervals_by_length(intersecting_intervals, min_length=min_length)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.union_adjacent_index", "title": "<code>union_adjacent_index(interval1, interval2)</code>", "text": "<p>Union index-adjacent intervals. If not adjacent, just concatenate.</p> <p>e.g. [a,b] and [b+1, c] is converted to [a,c]</p> <p>Parameters:</p> Name Type Description Default <code>interval1</code> <code>array</code> required <code>interval2</code> <code>array</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def union_adjacent_index(interval1, interval2):\n    \"\"\"Union index-adjacent intervals. If not adjacent, just concatenate.\n\n    e.g. [a,b] and [b+1, c] is converted to [a,c]\n\n    Parameters\n    ----------\n    interval1 : np.array\n    interval2 : np.array\n    \"\"\"\n    interval1 = np.atleast_2d(interval1)\n    interval2 = np.atleast_2d(interval2)\n\n    if (\n        interval1[-1][1] + 1 == interval2[0][0]\n        or interval2[0][1] + 1 == interval1[-1][0]\n    ):\n        x = np.array(\n            [\n                [\n                    np.min([interval1[-1][0], interval2[0][0]]),\n                    np.max([interval1[-1][1], interval2[0][1]]),\n                ]\n            ]\n        )\n        return np.concatenate((interval1[:-1], x), axis=0)\n    else:\n        return np.concatenate((interval1, interval2), axis=0)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_union", "title": "<code>interval_list_union(interval_list1, interval_list2, min_length=0.0, max_length=10000000000.0)</code>", "text": "<p>Finds the union (all times in one or both) for two interval lists</p> <p>Parameters:</p> Name Type Description Default <code>interval_list1</code> <code>ndarray</code> <p>The first interval list [start, stop]</p> required <code>interval_list2</code> <code>ndarray</code> <p>The second interval list [start, stop]</p> required <code>min_length</code> <code>float</code> <p>Minimum length of interval for inclusion in output, default 0.0</p> <code>0.0</code> <code>max_length</code> <code>float</code> <p>Maximum length of interval for inclusion in output, default 1e10</p> <code>10000000000.0</code> <p>Returns:</p> Type Description <code>ndarray</code> <p>Array of intervals [start, stop]</p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_union(\n    interval_list1: np.ndarray,\n    interval_list2: np.ndarray,\n    min_length: float = 0.0,\n    max_length: float = 1e10,\n) -&gt; np.ndarray:\n    \"\"\"Finds the union (all times in one or both) for two interval lists\n\n    Parameters\n    ----------\n    interval_list1 : np.ndarray\n        The first interval list [start, stop]\n    interval_list2 : np.ndarray\n        The second interval list [start, stop]\n    min_length : float, optional\n        Minimum length of interval for inclusion in output, default 0.0\n    max_length : float, optional\n        Maximum length of interval for inclusion in output, default 1e10\n\n    Returns\n    -------\n    np.ndarray\n        Array of intervals [start, stop]\n    \"\"\"\n\n    il1, il1_start_end = _parallel_union(interval_list1)\n    il2, il2_start_end = _parallel_union(interval_list2)\n\n    # Concatenate the two lists so we can resort the intervals and apply the\n    # same sorting to the start-end arrays\n    combined_intervals = np.concatenate((il1, il2))\n    ss = np.concatenate((il1_start_end, il2_start_end))\n    sort_ind = np.argsort(combined_intervals)\n    combined_intervals = combined_intervals[sort_ind]\n\n    # a cumulative sum of 1 indicates the beginning of a joint interval; a\n    # cumulative sum of 0 indicates the end\n    union_starts = np.ravel(np.array(np.where(np.cumsum(ss[sort_ind]) == 1)))\n    union_stops = np.ravel(np.array(np.where(np.cumsum(ss[sort_ind]) == 0)))\n    union = [\n        [combined_intervals[start], combined_intervals[stop]]\n        for start, stop in zip(union_starts, union_stops)\n    ]\n\n    return np.asarray(union)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_censor", "title": "<code>interval_list_censor(interval_list, timestamps)</code>", "text": "<p>Returns new interval list that starts/ends at first/last timestamp</p> <p>Parameters:</p> Name Type Description Default <code>interval_list</code> <code>numpy array of intervals [start, stop]</code> <p>interval list from IntervalList valid times</p> required <code>timestamps</code> <code>numpy array or list</code> required <p>Returns:</p> Type Description <code>interval_list (numpy array of intervals [start, stop])</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_censor(interval_list, timestamps):\n    \"\"\"Returns new interval list that starts/ends at first/last timestamp\n\n    Parameters\n    ----------\n    interval_list : numpy array of intervals [start, stop]\n        interval list from IntervalList valid times\n    timestamps : numpy array or list\n\n    Returns\n    -------\n    interval_list (numpy array of intervals [start, stop])\n    \"\"\"\n    # check that all timestamps are in the interval list\n    if len(interval_list_contains_ind(interval_list, timestamps)) != len(\n        timestamps\n    ):\n        raise ValueError(\"Interval_list must contain all timestamps\")\n\n    timestamps_interval = np.asarray([[timestamps[0], timestamps[-1]]])\n    return interval_list_intersect(interval_list, timestamps_interval)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_from_inds", "title": "<code>interval_from_inds(list_frames)</code>", "text": "<p>Converts a list of indices to a list of intervals.</p> <p>e.g. [2,3,4,6,7,8,9,10] -&gt; [[2,4],[6,10]]</p> <p>Parameters:</p> Name Type Description Default <code>list_frames</code> <code>array_like of int</code> required Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_from_inds(list_frames):\n    \"\"\"Converts a list of indices to a list of intervals.\n\n    e.g. [2,3,4,6,7,8,9,10] -&gt; [[2,4],[6,10]]\n\n    Parameters\n    ----------\n    list_frames : array_like of int\n    \"\"\"\n    list_frames = np.unique(list_frames)\n    interval_list = []\n    for key, group in itertools.groupby(\n        enumerate(list_frames), lambda t: t[1] - t[0]\n    ):\n        group = list(group)\n        interval_list.append([group[0][1], group[-1][1]])\n    return np.asarray(interval_list)\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_set_difference_inds", "title": "<code>interval_set_difference_inds(intervals1, intervals2)</code>", "text": "<p>e.g. intervals1 = [(0, 5), (8, 10)] intervals2 = [(1, 2), (3, 4), (6, 9)]</p> <p>result = [(0, 1), (4, 5), (9, 10)]</p> <p>Parameters:</p> Name Type Description Default <code>intervals1</code> <code>_type_</code> <p>description</p> required <code>intervals2</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_set_difference_inds(intervals1, intervals2):\n    \"\"\"\n    e.g.\n    intervals1 = [(0, 5), (8, 10)]\n    intervals2 = [(1, 2), (3, 4), (6, 9)]\n\n    result = [(0, 1), (4, 5), (9, 10)]\n\n    Parameters\n    ----------\n    intervals1 : _type_\n        _description_\n    intervals2 : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    result = []\n    i = j = 0\n    while i &lt; len(intervals1) and j &lt; len(intervals2):\n        if intervals1[i][1] &lt;= intervals2[j][0]:\n            result.append(intervals1[i])\n            i += 1\n        elif intervals2[j][1] &lt;= intervals1[i][0]:\n            j += 1\n        else:\n            if intervals1[i][0] &lt; intervals2[j][0]:\n                result.append((intervals1[i][0], intervals2[j][0]))\n            if intervals1[i][1] &gt; intervals2[j][1]:\n                intervals1[i] = (intervals2[j][1], intervals1[i][1])\n                j += 1\n            else:\n                i += 1\n    result += intervals1[i:]\n    return result\n</code></pre>"}, {"location": "api/common/common_interval/#src.spyglass.common.common_interval.interval_list_complement", "title": "<code>interval_list_complement(intervals1, intervals2, min_length=0.0)</code>", "text": "<p>Finds intervals in intervals1 that are not in intervals2</p> <p>Parameters:</p> Name Type Description Default <code>min_length</code> <code>float</code> <p>Minimum interval length in seconds. Defaults to 0.0.</p> <code>0.0</code> Source code in <code>src/spyglass/common/common_interval.py</code> <pre><code>def interval_list_complement(intervals1, intervals2, min_length=0.0):\n    \"\"\"\n    Finds intervals in intervals1 that are not in intervals2\n\n    Parameters\n    ----------\n    min_length : float, optional\n        Minimum interval length in seconds. Defaults to 0.0.\n    \"\"\"\n\n    result = []\n\n    for start1, end1 in intervals1:\n        subtracted = [(start1, end1)]\n\n        for start2, end2 in intervals2:\n            new_subtracted = []\n\n            for s, e in subtracted:\n                if start2 &lt;= s and e &lt;= end2:\n                    continue\n\n                if e &lt;= start2 or end2 &lt;= s:\n                    new_subtracted.append((s, e))\n                    continue\n\n                if start2 &gt; s:\n                    new_subtracted.append((s, start2))\n\n                if end2 &lt; e:\n                    new_subtracted.append((end2, e))\n\n            subtracted = new_subtracted\n\n        result.extend(subtracted)\n\n    return intervals_by_length(\n        np.asarray(result), min_length=min_length, max_length=1e100\n    )\n</code></pre>"}, {"location": "api/common/common_lab/", "title": "common_lab.py", "text": "<p>Schema for institution, lab team/name/members. Session-independent.</p>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember", "title": "<code>LabMember</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass LabMember(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    lab_member_name: varchar(80)\n    ---\n    first_name: varchar(200)\n    last_name: varchar(200)\n    \"\"\"\n\n    # NOTE that names must be unique here. If there are two neuroscientists\n    # named Jack Black that have data in this database, this will create an\n    # incorrect linkage. NWB does not yet provide unique IDs for names.\n\n    # NOTE: PR requires table alter.\n    class LabMemberInfo(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Information about lab member in the context of Frank lab network\n        -&gt; LabMember\n        ---\n        google_user_name         : varchar(200) # For permission to curate\n        datajoint_user_name = \"\" : varchar(200) # For permission to delete\n        admin = 0                : bool         # Ignore permission checks\n        unique index (datajoint_user_name)\n        unique index (google_user_name)\n        \"\"\"\n\n        # NOTE: index present for new instances. pending surgery, not enforced\n        # for existing instances.\n\n    _admin = []\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert lab member information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf: pynwb.NWBFile\n            The NWB file with experimenter information.\n        \"\"\"\n        if isinstance(nwbf, str):\n            nwb_file_abspath = Nwbfile.get_abs_path(nwbf, new_file=True)\n            nwbf = get_nwb_file(nwb_file_abspath)\n\n        if nwbf.experimenter is None:\n            logger.info(\"No experimenter metadata found.\\n\")\n            return\n\n        for experimenter in nwbf.experimenter:\n            cls.insert_from_name(experimenter)\n\n            # each person is by default the member of their own LabTeam\n            # (same as their name)\n\n            full_name, _, _ = decompose_name(experimenter)\n            LabTeam.create_new_team(\n                team_name=full_name, team_members=[full_name]\n            )\n\n    @classmethod\n    def insert_from_name(cls, full_name):\n        \"\"\"Insert a lab member by name.\n\n        Parameters\n        ----------\n        full_name : str\n            The name to be added.\n        \"\"\"\n        _, first, last = decompose_name(full_name)\n        cls.insert1(\n            dict(\n                lab_member_name=f\"{first} {last}\",\n                first_name=first,\n                last_name=last,\n            ),\n            skip_duplicates=True,\n        )\n\n    def _load_admin(cls):\n        \"\"\"Load admin list.\"\"\"\n        cls._admin = list(\n            (cls.LabMemberInfo &amp; {\"admin\": True}).fetch(\"datajoint_user_name\")\n        )\n\n    @property\n    def admin(cls) -&gt; list:\n        \"\"\"Return the list of admin users.\n\n        Note: This is cached. If adding a new admin, run _load_admin\n        \"\"\"\n        if not cls._admin:\n            cls._load_admin()\n        return cls._admin\n\n    def get_djuser_name(cls, dj_user) -&gt; str:\n        \"\"\"Return the lab member name associated with a datajoint user name.\n\n        Parameters\n        ----------\n        dj_user: str\n            The datajoint user name.\n\n        Returns\n        -------\n        str\n            The lab member name.\n        \"\"\"\n        query = (cls.LabMemberInfo &amp; {\"datajoint_user_name\": dj_user}).fetch(\n            \"lab_member_name\"\n        )\n\n        if len(query) != 1:\n            raise ValueError(\n                f\"Could not find name for datajoint user {dj_user}\"\n                + f\" in common.LabMember.LabMemberInfo: {query}\"\n            )\n\n        return query[0]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert lab member information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <p>The NWB file with experimenter information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert lab member information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf: pynwb.NWBFile\n        The NWB file with experimenter information.\n    \"\"\"\n    if isinstance(nwbf, str):\n        nwb_file_abspath = Nwbfile.get_abs_path(nwbf, new_file=True)\n        nwbf = get_nwb_file(nwb_file_abspath)\n\n    if nwbf.experimenter is None:\n        logger.info(\"No experimenter metadata found.\\n\")\n        return\n\n    for experimenter in nwbf.experimenter:\n        cls.insert_from_name(experimenter)\n\n        # each person is by default the member of their own LabTeam\n        # (same as their name)\n\n        full_name, _, _ = decompose_name(experimenter)\n        LabTeam.create_new_team(\n            team_name=full_name, team_members=[full_name]\n        )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.insert_from_name", "title": "<code>insert_from_name(full_name)</code>  <code>classmethod</code>", "text": "<p>Insert a lab member by name.</p> <p>Parameters:</p> Name Type Description Default <code>full_name</code> <code>str</code> <p>The name to be added.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_name(cls, full_name):\n    \"\"\"Insert a lab member by name.\n\n    Parameters\n    ----------\n    full_name : str\n        The name to be added.\n    \"\"\"\n    _, first, last = decompose_name(full_name)\n    cls.insert1(\n        dict(\n            lab_member_name=f\"{first} {last}\",\n            first_name=first,\n            last_name=last,\n        ),\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.admin", "title": "<code>admin: list</code>  <code>property</code>", "text": "<p>Return the list of admin users.</p> <p>Note: This is cached. If adding a new admin, run _load_admin</p>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabMember.get_djuser_name", "title": "<code>get_djuser_name(dj_user)</code>", "text": "<p>Return the lab member name associated with a datajoint user name.</p> <p>Parameters:</p> Name Type Description Default <code>dj_user</code> <p>The datajoint user name.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The lab member name.</p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def get_djuser_name(cls, dj_user) -&gt; str:\n    \"\"\"Return the lab member name associated with a datajoint user name.\n\n    Parameters\n    ----------\n    dj_user: str\n        The datajoint user name.\n\n    Returns\n    -------\n    str\n        The lab member name.\n    \"\"\"\n    query = (cls.LabMemberInfo &amp; {\"datajoint_user_name\": dj_user}).fetch(\n        \"lab_member_name\"\n    )\n\n    if len(query) != 1:\n        raise ValueError(\n            f\"Could not find name for datajoint user {dj_user}\"\n            + f\" in common.LabMember.LabMemberInfo: {query}\"\n        )\n\n    return query[0]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam", "title": "<code>LabTeam</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass LabTeam(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    team_name: varchar(80)\n    ---\n    team_description = \"\": varchar(2000)\n    \"\"\"\n\n    class LabTeamMember(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LabTeam\n        -&gt; LabMember\n        \"\"\"\n\n    _shared_teams = {}\n\n    def get_team_members(cls, member, reload=False):\n        \"\"\"Return the set of lab members that share a team with member.\n\n        Parameters\n        ----------\n        member : str\n            The lab member to check.\n        reload : bool\n            If True, reload the shared teams cache.\n        \"\"\"\n        if reload or member not in cls._shared_teams:\n            teams = (cls.LabTeamMember &amp; {\"lab_member_name\": member}).fetch(\n                \"team_name\"\n            )\n            cls._shared_teams[member] = set(\n                (\n                    LabTeam.LabTeamMember\n                    &amp; [{\"team_name\": team} for team in teams]\n                ).fetch(\"lab_member_name\")\n            )\n        return cls._shared_teams[member]\n\n    @classmethod\n    def create_new_team(\n        cls, team_name: str, team_members: list, team_description: str = \"\"\n    ):\n        \"\"\"Create a new team with a list of team members.\n\n        If the lab member does not exist in the database, they will be added.\n\n        Parameters\n        ----------\n        team_name : str\n            The name of the team.\n        team_members : str\n            The full names of the lab members that are part of the team.\n        team_description: str\n            The description of the team.\n        \"\"\"\n        labteam_dict = {\n            \"team_name\": team_name,\n            \"team_description\": team_description,\n        }\n\n        member_list = []\n        for team_member in team_members:\n            LabMember.insert_from_name(team_member)\n            query = (\n                LabMember.LabMemberInfo() &amp; {\"lab_member_name\": team_member}\n            ).fetch(\"google_user_name\")\n            if not query:\n                logger.info(\n                    f\"Please add the Google user ID for {team_member} in \"\n                    + \"LabMember.LabMemberInfo to help manage permissions.\"\n                )\n            labteammember_dict = {\n                \"team_name\": team_name,\n                \"lab_member_name\": team_member,\n            }\n            member_list.append(labteammember_dict)\n            # clear cache for this member\n            _ = cls._shared_teams.pop(team_member, None)\n\n        cls.insert1(labteam_dict, skip_duplicates=True)\n        cls.LabTeamMember.insert(member_list, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam.get_team_members", "title": "<code>get_team_members(member, reload=False)</code>", "text": "<p>Return the set of lab members that share a team with member.</p> <p>Parameters:</p> Name Type Description Default <code>member</code> <code>str</code> <p>The lab member to check.</p> required <code>reload</code> <code>bool</code> <p>If True, reload the shared teams cache.</p> <code>False</code> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def get_team_members(cls, member, reload=False):\n    \"\"\"Return the set of lab members that share a team with member.\n\n    Parameters\n    ----------\n    member : str\n        The lab member to check.\n    reload : bool\n        If True, reload the shared teams cache.\n    \"\"\"\n    if reload or member not in cls._shared_teams:\n        teams = (cls.LabTeamMember &amp; {\"lab_member_name\": member}).fetch(\n            \"team_name\"\n        )\n        cls._shared_teams[member] = set(\n            (\n                LabTeam.LabTeamMember\n                &amp; [{\"team_name\": team} for team in teams]\n            ).fetch(\"lab_member_name\")\n        )\n    return cls._shared_teams[member]\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.LabTeam.create_new_team", "title": "<code>create_new_team(team_name, team_members, team_description='')</code>  <code>classmethod</code>", "text": "<p>Create a new team with a list of team members.</p> <p>If the lab member does not exist in the database, they will be added.</p> <p>Parameters:</p> Name Type Description Default <code>team_name</code> <code>str</code> <p>The name of the team.</p> required <code>team_members</code> <code>str</code> <p>The full names of the lab members that are part of the team.</p> required <code>team_description</code> <code>str</code> <p>The description of the team.</p> <code>''</code> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef create_new_team(\n    cls, team_name: str, team_members: list, team_description: str = \"\"\n):\n    \"\"\"Create a new team with a list of team members.\n\n    If the lab member does not exist in the database, they will be added.\n\n    Parameters\n    ----------\n    team_name : str\n        The name of the team.\n    team_members : str\n        The full names of the lab members that are part of the team.\n    team_description: str\n        The description of the team.\n    \"\"\"\n    labteam_dict = {\n        \"team_name\": team_name,\n        \"team_description\": team_description,\n    }\n\n    member_list = []\n    for team_member in team_members:\n        LabMember.insert_from_name(team_member)\n        query = (\n            LabMember.LabMemberInfo() &amp; {\"lab_member_name\": team_member}\n        ).fetch(\"google_user_name\")\n        if not query:\n            logger.info(\n                f\"Please add the Google user ID for {team_member} in \"\n                + \"LabMember.LabMemberInfo to help manage permissions.\"\n            )\n        labteammember_dict = {\n            \"team_name\": team_name,\n            \"lab_member_name\": team_member,\n        }\n        member_list.append(labteammember_dict)\n        # clear cache for this member\n        _ = cls._shared_teams.pop(team_member, None)\n\n    cls.insert1(labteam_dict, skip_duplicates=True)\n    cls.LabTeamMember.insert(member_list, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Institution", "title": "<code>Institution</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass Institution(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    institution_name: varchar(80)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert institution information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The NWB file with institution information.\n        \"\"\"\n        if nwbf.institution is None:\n            logger.info(\"No institution metadata found.\\n\")\n            return\n\n        cls.insert1(\n            dict(institution_name=nwbf.institution), skip_duplicates=True\n        )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Institution.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert institution information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The NWB file with institution information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert institution information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The NWB file with institution information.\n    \"\"\"\n    if nwbf.institution is None:\n        logger.info(\"No institution metadata found.\\n\")\n        return\n\n    cls.insert1(\n        dict(institution_name=nwbf.institution), skip_duplicates=True\n    )\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Lab", "title": "<code>Lab</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@schema\nclass Lab(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    lab_name: varchar(80)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert lab name information from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The NWB file with lab name information.\n        \"\"\"\n        if nwbf.lab is None:\n            logger.info(\"No lab metadata found.\\n\")\n            return\n        cls.insert1(dict(lab_name=nwbf.lab), skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.Lab.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert lab name information from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The NWB file with lab name information.</p> required Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert lab name information from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The NWB file with lab name information.\n    \"\"\"\n    if nwbf.lab is None:\n        logger.info(\"No lab metadata found.\\n\")\n        return\n    cls.insert1(dict(lab_name=nwbf.lab), skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_lab/#src.spyglass.common.common_lab.decompose_name", "title": "<code>decompose_name(full_name)</code>", "text": "<p>Return the full, first and last name parts of a full name.</p> <p>Names capitalized. Decomposes either comma- or space-separated. If name includes spaces, must use exactly one comma.</p> <p>Parameters:</p> Name Type Description Default <code>full_name</code> <code>str</code> <p>The full name to decompose.</p> required <p>Returns:</p> Type Description <code>tuple</code> <p>A tuple of the full, first, last name parts.</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>When full_name is in an unsupported format</p> Source code in <code>src/spyglass/common/common_lab.py</code> <pre><code>def decompose_name(full_name: str) -&gt; tuple:\n    \"\"\"Return the full, first and last name parts of a full name.\n\n    Names capitalized. Decomposes either comma- or space-separated.\n    If name includes spaces, must use exactly one comma.\n\n    Parameters\n    ----------\n    full_name: str\n        The full name to decompose.\n\n    Returns\n    -------\n    tuple\n        A tuple of the full, first, last name parts.\n\n    Raises\n    ------\n    ValueError\n        When full_name is in an unsupported format\n    \"\"\"\n\n    full_trimmed = \"\" if not full_name else full_name.strip()\n\n    delim = (\n        \", \"\n        if full_trimmed.count(\", \") == 1\n        else \" \" if full_trimmed.count(\" \") == 1 else None\n    )\n\n    parts = full_trimmed.split(delim)\n\n    if delim is None or len(parts) != 2:  # catch unsupported format\n        raise ValueError(\n            f\"Name has unsupported format for {full_name}. \\n\\t\"\n            + \"Must use exactly one comma+space (i.e., ', ') or space.\\n\\t\"\n            + \"And provide exactly two name parts.\\n\\t\"\n            + \"For example, 'First Last' or 'Last1 Last2, First1 First2' \"\n            + \"if name(s) includes spaces.\"\n        )\n\n    first, last = parts if delim == \" \" else parts[::-1]\n    full = f\"{first} {last}\"\n\n    return full, first, last\n</code></pre>"}, {"location": "api/common/common_nwbfile/", "title": "common_nwbfile.py", "text": ""}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile", "title": "<code>Nwbfile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@schema\nclass Nwbfile(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Table for holding the NWB files.\n    nwb_file_name: varchar(64)   # name of the NWB file\n    ---\n    nwb_file_abs_path: filepath@raw\n    INDEX (nwb_file_abs_path)\n    \"\"\"\n    # NOTE the INDEX above is implicit from filepath@... above but needs to be explicit\n    # so that alter() can work\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    @classmethod\n    def insert_from_relative_file_name(cls, nwb_file_name):\n        \"\"\"Insert a new session from an existing NWB file.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The relative path to the NWB file.\n        \"\"\"\n        nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n        assert os.path.exists(\n            nwb_file_abs_path\n        ), f\"File does not exist: {nwb_file_abs_path}\"\n\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"nwb_file_abs_path\"] = nwb_file_abs_path\n        cls.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def _get_file_name(cls, nwb_file_name: str) -&gt; str:\n        \"\"\"Get valid nwb file name given substring.\"\"\"\n        query = cls &amp; f'nwb_file_name LIKE \"%{nwb_file_name}%\"'\n\n        if len(query) == 1:\n            return query.fetch1(\"nwb_file_name\")\n\n        raise ValueError(\n            f\"Found {len(query)} matches for {nwb_file_name} in Nwbfile table:\"\n            + f\" \\n{query}\"\n        )\n\n    @classmethod\n    def get_file_key(cls, nwb_file_name: str) -&gt; dict:\n        \"\"\"Return primary key using nwb_file_name substring.\"\"\"\n        return {\"nwb_file_name\": cls._get_file_name(nwb_file_name)}\n\n    @classmethod\n    def get_abs_path(cls, nwb_file_name, new_file=False) -&gt; str:\n        \"\"\"Return absolute path for a stored raw NWB file given file name.\n\n        The SPYGLASS_BASE_DIR must be set, either as an environment or part of\n        dj.config['custom']. See spyglass.settings.load_config\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file that has been inserted into the Nwbfile()\n            table. May be file substring. May include % wildcard(s).\n        new_file : bool, optional\n            Adding a new file to Nwbfile table. Defaults to False.\n\n        Returns\n        -------\n        nwb_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        if new_file:\n            return raw_dir + \"/\" + nwb_file_name\n\n        return raw_dir + \"/\" + cls._get_file_name(nwb_file_name)\n\n    @staticmethod\n    def add_to_lock(nwb_file_name):\n        \"\"\"Add the specified NWB file to the file with the list of NWB files to be locked.\n\n        The NWB_LOCK_FILE environment variable must be set.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file that has been inserted into the Nwbfile() schema.\n        \"\"\"\n        key = {\"nwb_file_name\": nwb_file_name}\n        # check to make sure the file exists\n        assert (\n            len((Nwbfile() &amp; key).fetch()) &gt; 0\n        ), f\"Error adding {nwb_file_name} to lock file, not in Nwbfile() schema\"\n\n        lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"a+\")\n        lock_file.write(f\"{nwb_file_name}\\n\")\n        lock_file.close()\n\n    @staticmethod\n    def cleanup(delete_files=False):\n        \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n        This does not delete the files themselves unless delete_files=True is specified\n        Run this after deleting the Nwbfile() entries themselves.\n        \"\"\"\n        schema.external[\"raw\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.insert_from_relative_file_name", "title": "<code>insert_from_relative_file_name(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Insert a new session from an existing NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The relative path to the NWB file.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef insert_from_relative_file_name(cls, nwb_file_name):\n    \"\"\"Insert a new session from an existing NWB file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The relative path to the NWB file.\n    \"\"\"\n    nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n    assert os.path.exists(\n        nwb_file_abs_path\n    ), f\"File does not exist: {nwb_file_abs_path}\"\n\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"nwb_file_abs_path\"] = nwb_file_abs_path\n    cls.insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.get_file_key", "title": "<code>get_file_key(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Return primary key using nwb_file_name substring.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_file_key(cls, nwb_file_name: str) -&gt; dict:\n    \"\"\"Return primary key using nwb_file_name substring.\"\"\"\n    return {\"nwb_file_name\": cls._get_file_name(nwb_file_name)}\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.get_abs_path", "title": "<code>get_abs_path(nwb_file_name, new_file=False)</code>  <code>classmethod</code>", "text": "<p>Return absolute path for a stored raw NWB file given file name.</p> <p>The SPYGLASS_BASE_DIR must be set, either as an environment or part of dj.config['custom']. See spyglass.settings.load_config</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file that has been inserted into the Nwbfile() table. May be file substring. May include % wildcard(s).</p> required <code>new_file</code> <code>bool</code> <p>Adding a new file to Nwbfile table. Defaults to False.</p> <code>False</code> <p>Returns:</p> Name Type Description <code>nwb_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_abs_path(cls, nwb_file_name, new_file=False) -&gt; str:\n    \"\"\"Return absolute path for a stored raw NWB file given file name.\n\n    The SPYGLASS_BASE_DIR must be set, either as an environment or part of\n    dj.config['custom']. See spyglass.settings.load_config\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file that has been inserted into the Nwbfile()\n        table. May be file substring. May include % wildcard(s).\n    new_file : bool, optional\n        Adding a new file to Nwbfile table. Defaults to False.\n\n    Returns\n    -------\n    nwb_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    if new_file:\n        return raw_dir + \"/\" + nwb_file_name\n\n    return raw_dir + \"/\" + cls._get_file_name(nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.add_to_lock", "title": "<code>add_to_lock(nwb_file_name)</code>  <code>staticmethod</code>", "text": "<p>Add the specified NWB file to the file with the list of NWB files to be locked.</p> <p>The NWB_LOCK_FILE environment variable must be set.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file that has been inserted into the Nwbfile() schema.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef add_to_lock(nwb_file_name):\n    \"\"\"Add the specified NWB file to the file with the list of NWB files to be locked.\n\n    The NWB_LOCK_FILE environment variable must be set.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file that has been inserted into the Nwbfile() schema.\n    \"\"\"\n    key = {\"nwb_file_name\": nwb_file_name}\n    # check to make sure the file exists\n    assert (\n        len((Nwbfile() &amp; key).fetch()) &gt; 0\n    ), f\"Error adding {nwb_file_name} to lock file, not in Nwbfile() schema\"\n\n    lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"a+\")\n    lock_file.write(f\"{nwb_file_name}\\n\")\n    lock_file.close()\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.Nwbfile.cleanup", "title": "<code>cleanup(delete_files=False)</code>  <code>staticmethod</code>", "text": "<p>Remove the filepath entries for NWB files that are not in use.</p> <p>This does not delete the files themselves unless delete_files=True is specified Run this after deleting the Nwbfile() entries themselves.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef cleanup(delete_files=False):\n    \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n    This does not delete the files themselves unless delete_files=True is specified\n    Run this after deleting the Nwbfile() entries themselves.\n    \"\"\"\n    schema.external[\"raw\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile", "title": "<code>AnalysisNwbfile</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@schema\nclass AnalysisNwbfile(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Table for holding the NWB files that contain results of analysis, such as spike sorting.\n    analysis_file_name: varchar(64)               # name of the file\n    ---\n    -&gt; Nwbfile                                     # name of the parent NWB file. Used for naming and metadata copy\n    analysis_file_abs_path: filepath@analysis      # the full path to the file\n    analysis_file_description = \"\": varchar(2000)  # an optional description of this analysis\n    analysis_parameters = NULL: blob               # additional relevant parameters. Currently used only for analyses\n                                                   # that span multiple NWB files\n    INDEX (analysis_file_abs_path)\n    \"\"\"\n    # NOTE the INDEX above is implicit from filepath@... above but needs to be explicit\n    # so that alter() can work\n\n    # See #630, #664. Excessive key length.\n\n    def create(self, nwb_file_name):\n        \"\"\"Open the NWB file, create a copy, write the copy to disk and return the name of the new file.\n\n        Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of an NWB file to be copied.\n\n        Returns\n        -------\n        analysis_file_name : str\n            The name of the new NWB file.\n        \"\"\"\n        nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n        with pynwb.NWBHDF5IO(\n            path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # pop off the unnecessary elements to save space\n            nwb_fields = nwbf.fields\n            for field in nwb_fields:\n                if field not in NWB_KEEP_FIELDS:\n                    nwb_object = getattr(nwbf, field)\n                    if isinstance(nwb_object, pynwb.core.LabelledDict):\n                        for module in list(nwb_object.keys()):\n                            nwb_object.pop(module)\n\n            analysis_file_name = self.__get_new_file_name(nwb_file_name)\n            # write the new file\n            logger.info(f\"Writing new NWB file {analysis_file_name}\")\n            analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n                analysis_file_name\n            )\n            # export the new NWB file\n            with pynwb.NWBHDF5IO(\n                path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n            ) as export_io:\n                export_io.export(io, nwbf)\n\n        # change the permissions to only allow owner to write\n        permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n        os.chmod(analysis_file_abs_path, permissions)\n\n        return analysis_file_name\n\n    @classmethod\n    def __get_new_file_name(cls, nwb_file_name):\n        # each file ends with a random string of 10 digits, so we generate that\n        # string and redo if by some miracle it's already there\n        file_in_table = True\n        while file_in_table:\n            analysis_file_name = (\n                os.path.splitext(nwb_file_name)[0]\n                + \"\".join(\n                    random.choices(string.ascii_uppercase + string.digits, k=10)\n                )\n                + \".nwb\"\n            )\n            file_in_table = AnalysisNwbfile &amp; {\n                \"analysis_file_name\": analysis_file_name\n            }\n\n        return analysis_file_name\n\n    @classmethod\n    def __get_analysis_file_dir(cls, analysis_file_name: str):\n        # strip off everything after and including the final underscore and return the result\n        return analysis_file_name[0 : analysis_file_name.rfind(\"_\")]\n\n    @classmethod\n    def copy(cls, nwb_file_name):\n        \"\"\"Make a copy of an analysis NWB file.\n\n        Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the analysis NWB file to be copied.\n\n        Returns\n        -------\n        analysis_file_name : str\n            The name of the new NWB file.\n        \"\"\"\n        nwb_file_abspath = AnalysisNwbfile.get_abs_path(nwb_file_name)\n        with pynwb.NWBHDF5IO(\n            path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # get the current number of analysis files related to this nwb file\n            query = AnalysisNwbfile &amp; {\"analysis_file_name\": nwb_file_name}\n            original_nwb_file_name = query.fetch(\"nwb_file_name\")[0]\n            analysis_file_name = cls.__get_new_file_name(original_nwb_file_name)\n            # write the new file\n            logger.info(f\"Writing new NWB file {analysis_file_name}...\")\n            analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n                analysis_file_name\n            )\n            # export the new NWB file\n            with pynwb.NWBHDF5IO(\n                path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n            ) as export_io:\n                export_io.export(io, nwbf)\n\n        return analysis_file_name\n\n    def add(self, nwb_file_name, analysis_file_name):\n        \"\"\"Add the specified file to AnalysisNWBfile table.\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the parent NWB file.\n        analysis_file_name : str\n            The name of the analysis NWB file that was created.\n        \"\"\"\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        key[\"analysis_file_name\"] = analysis_file_name\n        key[\"analysis_file_description\"] = \"\"\n        key[\"analysis_file_abs_path\"] = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        self.insert1(key)\n\n    @staticmethod\n    def get_abs_path(analysis_nwb_file_name):\n        \"\"\"Return the absolute path for a stored analysis NWB file given just the file name.\n\n        The spyglass config from settings.py must be set.\n\n        Parameters\n        ----------\n        analysis_nwb_file_name : str\n            The name of the NWB file that has been inserted into the AnalysisNwbfile() schema\n\n        Returns\n        -------\n        analysis_nwb_file_abspath : str\n            The absolute path for the given file name.\n        \"\"\"\n        # see if the file exists and is stored in the base analysis dir\n        test_path = f\"{analysis_dir}/{analysis_nwb_file_name}\"\n\n        if os.path.exists(test_path):\n            return test_path\n        else:\n            # use the new path\n            analysis_file_base_path = Path(\n                analysis_dir\n            ) / AnalysisNwbfile.__get_analysis_file_dir(analysis_nwb_file_name)\n            if not analysis_file_base_path.exists():\n                os.mkdir(str(analysis_file_base_path))\n            return str(analysis_file_base_path / analysis_nwb_file_name)\n\n    def add_nwb_object(\n        self, analysis_file_name, nwb_object, table_name=\"pandas_table\"\n    ):\n        # TODO: change to add_object with checks for object type and a name parameter, which should be specified if\n        # it is not an NWB container\n        \"\"\"Add an NWB object to the analysis file in the scratch area and returns the NWB object ID\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        nwb_object : pynwb.core.NWBDataInterface\n            The NWB object created by PyNWB.\n        table_name : str (optional, defaults to 'pandas_table')\n            The name of the DynamicTable made from a dataframe.\n\n        Returns\n        -------\n        nwb_object_id : str\n            The NWB object ID of the added object.\n        \"\"\"\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            if isinstance(nwb_object, pd.DataFrame):\n                dt_object = DynamicTable.from_dataframe(\n                    name=table_name, df=nwb_object\n                )\n                nwbf.add_scratch(dt_object)\n                io.write(nwbf)\n                return dt_object.object_id\n            else:\n                nwbf.add_scratch(nwb_object)\n                io.write(nwbf)\n                return nwb_object.object_id\n\n    def add_units(\n        self,\n        analysis_file_name,\n        units,\n        units_valid_times,\n        units_sort_interval,\n        metrics=None,\n        units_waveforms=None,\n        labels=None,\n    ):\n        \"\"\"Add units to analysis NWB file\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        units : dict\n            keys are unit ids, values are spike times\n        units_valid_times : dict\n            Dictionary of units and valid times with unit ids as keys.\n        units_sort_interval : dict\n            Dictionary of units and sort_interval with unit ids as keys.\n        units_waveforms : dict, optional\n            Dictionary of unit waveforms with unit ids as keys.\n        metrics : dict, optional\n            Cluster metrics.\n        labels : dict, optional\n            Curation labels for clusters\n\n        Returns\n        -------\n        units_object_id, waveforms_object_id : str, str\n            The NWB object id of the Units object and the object id of the waveforms object ('' if None)\n        \"\"\"\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            sort_intervals = list()\n            if len(units.keys()):\n                # Add spike times and valid time range for the sort\n                for id in units.keys():\n                    nwbf.add_unit(\n                        spike_times=units[id],\n                        id=id,\n                        # waveform_mean = units_templates[id],\n                        obs_intervals=units_valid_times[id],\n                    )\n                    sort_intervals.append(units_sort_interval[id])\n                # Add a column for the sort interval (subset of valid time)\n                nwbf.add_unit_column(\n                    name=\"sort_interval\",\n                    description=\"the interval used for spike sorting\",\n                    data=sort_intervals,\n                )\n                # If metrics were specified, add one column per metric\n                if metrics is not None:\n                    for metric in metrics:\n                        if metrics[metric]:\n                            unit_ids = np.array(list(metrics[metric].keys()))\n                            metric_values = np.array(\n                                list(metrics[metric].values())\n                            )\n                            # sort by unit_ids and apply that sorting to values to ensure that things go in the right order\n                            metric_values = metric_values[np.argsort(unit_ids)]\n                            logger.info(\n                                f\"Adding metric {metric} : {metric_values}\"\n                            )\n                            nwbf.add_unit_column(\n                                name=metric,\n                                description=f\"{metric} metric\",\n                                data=metric_values,\n                            )\n                if labels is not None:\n                    unit_ids = np.array(list(units.keys()))\n                    for unit in unit_ids:\n                        if unit not in labels:\n                            labels[unit] = \"\"\n                    label_values = np.array(list(labels.values()))\n                    label_values = label_values[np.argsort(unit_ids)].tolist()\n                    nwbf.add_unit_column(\n                        name=\"label\",\n                        description=\"label given during curation\",\n                        data=label_values,\n                    )\n                # If the waveforms were specified, add them as a dataframe to scratch\n                waveforms_object_id = \"\"\n                if units_waveforms is not None:\n                    waveforms_df = pd.DataFrame.from_dict(\n                        units_waveforms, orient=\"index\"\n                    )\n                    waveforms_df.columns = [\"waveforms\"]\n                    nwbf.add_scratch(\n                        waveforms_df,\n                        name=\"units_waveforms\",\n                        notes=\"spike waveforms for each unit\",\n                    )\n                    waveforms_object_id = nwbf.scratch[\n                        \"units_waveforms\"\n                    ].object_id\n\n                io.write(nwbf)\n                return nwbf.units.object_id, waveforms_object_id\n            else:\n                return \"\"\n\n    def add_units_waveforms(\n        self,\n        analysis_file_name,\n        waveform_extractor: si.WaveformExtractor,\n        metrics=None,\n        labels=None,\n    ):\n        \"\"\"Add units to analysis NWB file along with the waveforms\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        waveform_extractor : si.WaveformExtractor object\n        metrics : dict, optional\n            Cluster metrics.\n        labels : dict, optional\n            Curation labels for clusters\n\n        Returns\n        -------\n        units_object_id : str\n            The NWB object id of the Units object\n        \"\"\"\n\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            for id in waveform_extractor.sorting.get_unit_ids():\n                # (spikes, samples, channels)\n                waveforms = waveform_extractor.get_waveforms(unit_id=id)\n                # (channels, spikes, samples)\n                waveforms = np.moveaxis(waveforms, source=2, destination=0)\n                nwbf.add_unit(\n                    spike_times=waveform_extractor.sorting.get_unit_spike_train(\n                        unit_id=id\n                    ),\n                    id=id,\n                    electrodes=waveform_extractor.recording.get_channel_ids(),\n                    waveforms=waveforms,\n                )\n\n            # The following is a rough sketch of AnalysisNwbfile().add_waveforms\n            # analysis_file_name = AnalysisNwbfile().create(key['nwb_file_name'])\n            # or\n            # nwbfile = pynwb.NWBFile(...)\n            # (channels, spikes, samples)\n            # wfs = [\n            #         [     # elec 1\n            #             [1, 2, 3],  # spike 1, [sample 1, sample 2, sample 3]\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ], [  # elec 2\n            #             [1, 2, 3],  # spike 1\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ], [  # elec 3\n            #             [1, 2, 3],  # spike 1\n            #             [1, 2, 3],  # spike 2\n            #             [1, 2, 3],  # spike 3\n            #             [1, 2, 3]   # spike 4\n            #         ]\n            # ]\n            # elecs = ... # DynamicTableRegion referring to three electrodes (rows) of the electrodes table\n            # nwbfile.add_unit(spike_times=[1, 2, 3], electrodes=elecs, waveforms=wfs)\n\n            # If metrics were specified, add one column per metric\n            if metrics is not None:\n                for metric_name, metric_dict in metrics.items():\n                    logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                    metric_data = metric_dict.values().to_list()\n                    nwbf.add_unit_column(\n                        name=metric_name,\n                        description=metric_name,\n                        data=metric_data,\n                    )\n            if labels is not None:\n                nwbf.add_unit_column(\n                    name=\"label\",\n                    description=\"label given during curation\",\n                    data=labels,\n                )\n\n            io.write(nwbf)\n            return nwbf.units.object_id\n\n    def add_units_metrics(self, analysis_file_name, metrics):\n        \"\"\"Add units to analysis NWB file along with the waveforms\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        metrics : dict, optional\n            Cluster metrics.\n\n        Returns\n        -------\n        units_object_id : str\n            The NWB object id of the Units object\n        \"\"\"\n        metric_names = list(metrics.keys())\n        unit_ids = list(metrics[metric_names[0]].keys())\n        with pynwb.NWBHDF5IO(\n            path=self.get_abs_path(analysis_file_name),\n            mode=\"a\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            for id in unit_ids:\n                nwbf.add_unit(id=id)\n\n            for metric_name, metric_dict in metrics.items():\n                logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                metric_data = list(metric_dict.values())\n                nwbf.add_unit_column(\n                    name=metric_name, description=metric_name, data=metric_data\n                )\n\n            io.write(nwbf)\n            return nwbf.units.object_id\n\n    @classmethod\n    def get_electrode_indices(cls, analysis_file_name, electrode_ids):\n        \"\"\"Given an analysis NWB file name, returns the indices of the specified electrode_ids.\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis NWB file.\n        electrode_ids : numpy array or list\n            Array or list of electrode IDs.\n\n        Returns\n        -------\n        electrode_indices : numpy array\n            Array of indices in the electrodes table for the given electrode IDs.\n        \"\"\"\n        nwbf = get_nwb_file(cls.get_abs_path(analysis_file_name))\n        return get_electrode_indices(nwbf.electrodes, electrode_ids)\n\n    @staticmethod\n    def cleanup(delete_files=False):\n        \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n        Does not delete the files themselves unless delete_files=True is specified.\n        Run this after deleting the Nwbfile() entries themselves.\n\n        Parameters\n        ----------\n        delete_files : bool, optional\n            Whether the original files should be deleted (default False).\n        \"\"\"\n        schema.external[\"analysis\"].delete(delete_external_files=delete_files)\n\n    @staticmethod\n    def nightly_cleanup():\n        child_tables = get_child_tables(AnalysisNwbfile)\n        (AnalysisNwbfile - child_tables).delete_quick()\n\n        # a separate external files clean up required - this is to be done\n        # during times when no other transactions are in progress.\n        AnalysisNwbfile.cleanup(True)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.create", "title": "<code>create(nwb_file_name)</code>", "text": "<p>Open the NWB file, create a copy, write the copy to disk and return the name of the new file.</p> <p>Note that this does NOT add the file to the schema; that needs to be done after data are written to it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of an NWB file to be copied.</p> required <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <p>The name of the new NWB file.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def create(self, nwb_file_name):\n    \"\"\"Open the NWB file, create a copy, write the copy to disk and return the name of the new file.\n\n    Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of an NWB file to be copied.\n\n    Returns\n    -------\n    analysis_file_name : str\n        The name of the new NWB file.\n    \"\"\"\n    nwb_file_abspath = Nwbfile.get_abs_path(nwb_file_name)\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        # pop off the unnecessary elements to save space\n        nwb_fields = nwbf.fields\n        for field in nwb_fields:\n            if field not in NWB_KEEP_FIELDS:\n                nwb_object = getattr(nwbf, field)\n                if isinstance(nwb_object, pynwb.core.LabelledDict):\n                    for module in list(nwb_object.keys()):\n                        nwb_object.pop(module)\n\n        analysis_file_name = self.__get_new_file_name(nwb_file_name)\n        # write the new file\n        logger.info(f\"Writing new NWB file {analysis_file_name}\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n        ) as export_io:\n            export_io.export(io, nwbf)\n\n    # change the permissions to only allow owner to write\n    permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n    os.chmod(analysis_file_abs_path, permissions)\n\n    return analysis_file_name\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.copy", "title": "<code>copy(nwb_file_name)</code>  <code>classmethod</code>", "text": "<p>Make a copy of an analysis NWB file.</p> <p>Note that this does NOT add the file to the schema; that needs to be done after data are written to it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the analysis NWB file to be copied.</p> required <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <p>The name of the new NWB file.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef copy(cls, nwb_file_name):\n    \"\"\"Make a copy of an analysis NWB file.\n\n    Note that this does NOT add the file to the schema; that needs to be done after data are written to it.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the analysis NWB file to be copied.\n\n    Returns\n    -------\n    analysis_file_name : str\n        The name of the new NWB file.\n    \"\"\"\n    nwb_file_abspath = AnalysisNwbfile.get_abs_path(nwb_file_name)\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abspath, mode=\"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        # get the current number of analysis files related to this nwb file\n        query = AnalysisNwbfile &amp; {\"analysis_file_name\": nwb_file_name}\n        original_nwb_file_name = query.fetch(\"nwb_file_name\")[0]\n        analysis_file_name = cls.__get_new_file_name(original_nwb_file_name)\n        # write the new file\n        logger.info(f\"Writing new NWB file {analysis_file_name}...\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path, mode=\"w\", manager=io.manager\n        ) as export_io:\n            export_io.export(io, nwbf)\n\n    return analysis_file_name\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add", "title": "<code>add(nwb_file_name, analysis_file_name)</code>", "text": "<p>Add the specified file to AnalysisNWBfile table.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the parent NWB file.</p> required <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file that was created.</p> required Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add(self, nwb_file_name, analysis_file_name):\n    \"\"\"Add the specified file to AnalysisNWBfile table.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the parent NWB file.\n    analysis_file_name : str\n        The name of the analysis NWB file that was created.\n    \"\"\"\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    key[\"analysis_file_name\"] = analysis_file_name\n    key[\"analysis_file_description\"] = \"\"\n    key[\"analysis_file_abs_path\"] = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    self.insert1(key)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.get_abs_path", "title": "<code>get_abs_path(analysis_nwb_file_name)</code>  <code>staticmethod</code>", "text": "<p>Return the absolute path for a stored analysis NWB file given just the file name.</p> <p>The spyglass config from settings.py must be set.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_nwb_file_name</code> <code>str</code> <p>The name of the NWB file that has been inserted into the AnalysisNwbfile() schema</p> required <p>Returns:</p> Name Type Description <code>analysis_nwb_file_abspath</code> <code>str</code> <p>The absolute path for the given file name.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef get_abs_path(analysis_nwb_file_name):\n    \"\"\"Return the absolute path for a stored analysis NWB file given just the file name.\n\n    The spyglass config from settings.py must be set.\n\n    Parameters\n    ----------\n    analysis_nwb_file_name : str\n        The name of the NWB file that has been inserted into the AnalysisNwbfile() schema\n\n    Returns\n    -------\n    analysis_nwb_file_abspath : str\n        The absolute path for the given file name.\n    \"\"\"\n    # see if the file exists and is stored in the base analysis dir\n    test_path = f\"{analysis_dir}/{analysis_nwb_file_name}\"\n\n    if os.path.exists(test_path):\n        return test_path\n    else:\n        # use the new path\n        analysis_file_base_path = Path(\n            analysis_dir\n        ) / AnalysisNwbfile.__get_analysis_file_dir(analysis_nwb_file_name)\n        if not analysis_file_base_path.exists():\n            os.mkdir(str(analysis_file_base_path))\n        return str(analysis_file_base_path / analysis_nwb_file_name)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_nwb_object", "title": "<code>add_nwb_object(analysis_file_name, nwb_object, table_name='pandas_table')</code>", "text": "<p>Add an NWB object to the analysis file in the scratch area and returns the NWB object ID</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>nwb_object</code> <code>NWBDataInterface</code> <p>The NWB object created by PyNWB.</p> required <code>table_name</code> <code>str (optional, defaults to 'pandas_table')</code> <p>The name of the DynamicTable made from a dataframe.</p> <code>'pandas_table'</code> <p>Returns:</p> Name Type Description <code>nwb_object_id</code> <code>str</code> <p>The NWB object ID of the added object.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_nwb_object(\n    self, analysis_file_name, nwb_object, table_name=\"pandas_table\"\n):\n    # TODO: change to add_object with checks for object type and a name parameter, which should be specified if\n    # it is not an NWB container\n    \"\"\"Add an NWB object to the analysis file in the scratch area and returns the NWB object ID\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    nwb_object : pynwb.core.NWBDataInterface\n        The NWB object created by PyNWB.\n    table_name : str (optional, defaults to 'pandas_table')\n        The name of the DynamicTable made from a dataframe.\n\n    Returns\n    -------\n    nwb_object_id : str\n        The NWB object ID of the added object.\n    \"\"\"\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        if isinstance(nwb_object, pd.DataFrame):\n            dt_object = DynamicTable.from_dataframe(\n                name=table_name, df=nwb_object\n            )\n            nwbf.add_scratch(dt_object)\n            io.write(nwbf)\n            return dt_object.object_id\n        else:\n            nwbf.add_scratch(nwb_object)\n            io.write(nwbf)\n            return nwb_object.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units", "title": "<code>add_units(analysis_file_name, units, units_valid_times, units_sort_interval, metrics=None, units_waveforms=None, labels=None)</code>", "text": "<p>Add units to analysis NWB file</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>units</code> <code>dict</code> <p>keys are unit ids, values are spike times</p> required <code>units_valid_times</code> <code>dict</code> <p>Dictionary of units and valid times with unit ids as keys.</p> required <code>units_sort_interval</code> <code>dict</code> <p>Dictionary of units and sort_interval with unit ids as keys.</p> required <code>units_waveforms</code> <code>dict</code> <p>Dictionary of unit waveforms with unit ids as keys.</p> <code>None</code> <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> <code>None</code> <code>labels</code> <code>dict</code> <p>Curation labels for clusters</p> <code>None</code> <p>Returns:</p> Type Description <code>units_object_id, waveforms_object_id : str, str</code> <p>The NWB object id of the Units object and the object id of the waveforms object ('' if None)</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units(\n    self,\n    analysis_file_name,\n    units,\n    units_valid_times,\n    units_sort_interval,\n    metrics=None,\n    units_waveforms=None,\n    labels=None,\n):\n    \"\"\"Add units to analysis NWB file\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    units : dict\n        keys are unit ids, values are spike times\n    units_valid_times : dict\n        Dictionary of units and valid times with unit ids as keys.\n    units_sort_interval : dict\n        Dictionary of units and sort_interval with unit ids as keys.\n    units_waveforms : dict, optional\n        Dictionary of unit waveforms with unit ids as keys.\n    metrics : dict, optional\n        Cluster metrics.\n    labels : dict, optional\n        Curation labels for clusters\n\n    Returns\n    -------\n    units_object_id, waveforms_object_id : str, str\n        The NWB object id of the Units object and the object id of the waveforms object ('' if None)\n    \"\"\"\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        sort_intervals = list()\n        if len(units.keys()):\n            # Add spike times and valid time range for the sort\n            for id in units.keys():\n                nwbf.add_unit(\n                    spike_times=units[id],\n                    id=id,\n                    # waveform_mean = units_templates[id],\n                    obs_intervals=units_valid_times[id],\n                )\n                sort_intervals.append(units_sort_interval[id])\n            # Add a column for the sort interval (subset of valid time)\n            nwbf.add_unit_column(\n                name=\"sort_interval\",\n                description=\"the interval used for spike sorting\",\n                data=sort_intervals,\n            )\n            # If metrics were specified, add one column per metric\n            if metrics is not None:\n                for metric in metrics:\n                    if metrics[metric]:\n                        unit_ids = np.array(list(metrics[metric].keys()))\n                        metric_values = np.array(\n                            list(metrics[metric].values())\n                        )\n                        # sort by unit_ids and apply that sorting to values to ensure that things go in the right order\n                        metric_values = metric_values[np.argsort(unit_ids)]\n                        logger.info(\n                            f\"Adding metric {metric} : {metric_values}\"\n                        )\n                        nwbf.add_unit_column(\n                            name=metric,\n                            description=f\"{metric} metric\",\n                            data=metric_values,\n                        )\n            if labels is not None:\n                unit_ids = np.array(list(units.keys()))\n                for unit in unit_ids:\n                    if unit not in labels:\n                        labels[unit] = \"\"\n                label_values = np.array(list(labels.values()))\n                label_values = label_values[np.argsort(unit_ids)].tolist()\n                nwbf.add_unit_column(\n                    name=\"label\",\n                    description=\"label given during curation\",\n                    data=label_values,\n                )\n            # If the waveforms were specified, add them as a dataframe to scratch\n            waveforms_object_id = \"\"\n            if units_waveforms is not None:\n                waveforms_df = pd.DataFrame.from_dict(\n                    units_waveforms, orient=\"index\"\n                )\n                waveforms_df.columns = [\"waveforms\"]\n                nwbf.add_scratch(\n                    waveforms_df,\n                    name=\"units_waveforms\",\n                    notes=\"spike waveforms for each unit\",\n                )\n                waveforms_object_id = nwbf.scratch[\n                    \"units_waveforms\"\n                ].object_id\n\n            io.write(nwbf)\n            return nwbf.units.object_id, waveforms_object_id\n        else:\n            return \"\"\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units_waveforms", "title": "<code>add_units_waveforms(analysis_file_name, waveform_extractor, metrics=None, labels=None)</code>", "text": "<p>Add units to analysis NWB file along with the waveforms</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>waveform_extractor</code> <code>si.WaveformExtractor object</code> required <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> <code>None</code> <code>labels</code> <code>dict</code> <p>Curation labels for clusters</p> <code>None</code> <p>Returns:</p> Name Type Description <code>units_object_id</code> <code>str</code> <p>The NWB object id of the Units object</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units_waveforms(\n    self,\n    analysis_file_name,\n    waveform_extractor: si.WaveformExtractor,\n    metrics=None,\n    labels=None,\n):\n    \"\"\"Add units to analysis NWB file along with the waveforms\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    waveform_extractor : si.WaveformExtractor object\n    metrics : dict, optional\n        Cluster metrics.\n    labels : dict, optional\n        Curation labels for clusters\n\n    Returns\n    -------\n    units_object_id : str\n        The NWB object id of the Units object\n    \"\"\"\n\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        for id in waveform_extractor.sorting.get_unit_ids():\n            # (spikes, samples, channels)\n            waveforms = waveform_extractor.get_waveforms(unit_id=id)\n            # (channels, spikes, samples)\n            waveforms = np.moveaxis(waveforms, source=2, destination=0)\n            nwbf.add_unit(\n                spike_times=waveform_extractor.sorting.get_unit_spike_train(\n                    unit_id=id\n                ),\n                id=id,\n                electrodes=waveform_extractor.recording.get_channel_ids(),\n                waveforms=waveforms,\n            )\n\n        # The following is a rough sketch of AnalysisNwbfile().add_waveforms\n        # analysis_file_name = AnalysisNwbfile().create(key['nwb_file_name'])\n        # or\n        # nwbfile = pynwb.NWBFile(...)\n        # (channels, spikes, samples)\n        # wfs = [\n        #         [     # elec 1\n        #             [1, 2, 3],  # spike 1, [sample 1, sample 2, sample 3]\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ], [  # elec 2\n        #             [1, 2, 3],  # spike 1\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ], [  # elec 3\n        #             [1, 2, 3],  # spike 1\n        #             [1, 2, 3],  # spike 2\n        #             [1, 2, 3],  # spike 3\n        #             [1, 2, 3]   # spike 4\n        #         ]\n        # ]\n        # elecs = ... # DynamicTableRegion referring to three electrodes (rows) of the electrodes table\n        # nwbfile.add_unit(spike_times=[1, 2, 3], electrodes=elecs, waveforms=wfs)\n\n        # If metrics were specified, add one column per metric\n        if metrics is not None:\n            for metric_name, metric_dict in metrics.items():\n                logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n                metric_data = metric_dict.values().to_list()\n                nwbf.add_unit_column(\n                    name=metric_name,\n                    description=metric_name,\n                    data=metric_data,\n                )\n        if labels is not None:\n            nwbf.add_unit_column(\n                name=\"label\",\n                description=\"label given during curation\",\n                data=labels,\n            )\n\n        io.write(nwbf)\n        return nwbf.units.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.add_units_metrics", "title": "<code>add_units_metrics(analysis_file_name, metrics)</code>", "text": "<p>Add units to analysis NWB file along with the waveforms</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>metrics</code> <code>dict</code> <p>Cluster metrics.</p> required <p>Returns:</p> Name Type Description <code>units_object_id</code> <code>str</code> <p>The NWB object id of the Units object</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>def add_units_metrics(self, analysis_file_name, metrics):\n    \"\"\"Add units to analysis NWB file along with the waveforms\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    metrics : dict, optional\n        Cluster metrics.\n\n    Returns\n    -------\n    units_object_id : str\n        The NWB object id of the Units object\n    \"\"\"\n    metric_names = list(metrics.keys())\n    unit_ids = list(metrics[metric_names[0]].keys())\n    with pynwb.NWBHDF5IO(\n        path=self.get_abs_path(analysis_file_name),\n        mode=\"a\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        for id in unit_ids:\n            nwbf.add_unit(id=id)\n\n        for metric_name, metric_dict in metrics.items():\n            logger.info(f\"Adding metric {metric_name} : {metric_dict}\")\n            metric_data = list(metric_dict.values())\n            nwbf.add_unit_column(\n                name=metric_name, description=metric_name, data=metric_data\n            )\n\n        io.write(nwbf)\n        return nwbf.units.object_id\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.get_electrode_indices", "title": "<code>get_electrode_indices(analysis_file_name, electrode_ids)</code>  <code>classmethod</code>", "text": "<p>Given an analysis NWB file name, returns the indices of the specified electrode_ids.</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis NWB file.</p> required <code>electrode_ids</code> <code>numpy array or list</code> <p>Array or list of electrode IDs.</p> required <p>Returns:</p> Name Type Description <code>electrode_indices</code> <code>numpy array</code> <p>Array of indices in the electrodes table for the given electrode IDs.</p> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@classmethod\ndef get_electrode_indices(cls, analysis_file_name, electrode_ids):\n    \"\"\"Given an analysis NWB file name, returns the indices of the specified electrode_ids.\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis NWB file.\n    electrode_ids : numpy array or list\n        Array or list of electrode IDs.\n\n    Returns\n    -------\n    electrode_indices : numpy array\n        Array of indices in the electrodes table for the given electrode IDs.\n    \"\"\"\n    nwbf = get_nwb_file(cls.get_abs_path(analysis_file_name))\n    return get_electrode_indices(nwbf.electrodes, electrode_ids)\n</code></pre>"}, {"location": "api/common/common_nwbfile/#src.spyglass.common.common_nwbfile.AnalysisNwbfile.cleanup", "title": "<code>cleanup(delete_files=False)</code>  <code>staticmethod</code>", "text": "<p>Remove the filepath entries for NWB files that are not in use.</p> <p>Does not delete the files themselves unless delete_files=True is specified. Run this after deleting the Nwbfile() entries themselves.</p> <p>Parameters:</p> Name Type Description Default <code>delete_files</code> <code>bool</code> <p>Whether the original files should be deleted (default False).</p> <code>False</code> Source code in <code>src/spyglass/common/common_nwbfile.py</code> <pre><code>@staticmethod\ndef cleanup(delete_files=False):\n    \"\"\"Remove the filepath entries for NWB files that are not in use.\n\n    Does not delete the files themselves unless delete_files=True is specified.\n    Run this after deleting the Nwbfile() entries themselves.\n\n    Parameters\n    ----------\n    delete_files : bool, optional\n        Whether the original files should be deleted (default False).\n    \"\"\"\n    schema.external[\"analysis\"].delete(delete_external_files=delete_files)\n</code></pre>"}, {"location": "api/common/common_position/", "title": "common_position.py", "text": ""}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionInfoParameters", "title": "<code>PositionInfoParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Parameters for extracting the smoothed position, orientation and velocity.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass PositionInfoParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"\n    Parameters for extracting the smoothed position, orientation and velocity.\n    \"\"\"\n\n    definition = \"\"\"\n    position_info_param_name : varchar(32) # name for this set of parameters\n    ---\n    max_separation = 9.0  : float   # max distance (in cm) between head LEDs\n    max_speed = 300.0     : float   # max speed (in cm / s) of animal\n    position_smoothing_duration = 0.125 : float # size of moving window (s)\n    speed_smoothing_std_dev = 0.100 : float # smoothing standard deviation (s)\n    head_orient_smoothing_std_dev = 0.001 : float # smoothing std deviation (s)\n    led1_is_front = 1 : int # 1 if 1st LED is front LED, else 1st LED is back\n    is_upsampled = 0 : int # upsample the position to higher sampling rate\n    upsampling_sampling_rate = NULL : float # The rate to be upsampled to\n    upsampling_interpolation_method = linear : varchar(80) # see\n        # pandas.DataFrame.interpolation for list of methods\n    \"\"\"\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfoSelection", "title": "<code>IntervalPositionInfoSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Combines the parameters for position extraction and a time interval to extract the smoothed position on.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass IntervalPositionInfoSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Combines the parameters for position extraction and a time interval to\n    extract the smoothed position on.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionInfoParameters\n    -&gt; IntervalList\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfo", "title": "<code>IntervalPositionInfo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Computes the smoothed head position, orientation and velocity for a given interval.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass IntervalPositionInfo(SpyglassMixin, dj.Computed):\n    \"\"\"Computes the smoothed head position, orientation and velocity for a given\n    interval.\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalPositionInfoSelection\n    ---\n    -&gt; AnalysisNwbfile\n    head_position_object_id : varchar(40)\n    head_orientation_object_id : varchar(40)\n    head_velocity_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing position for: {key}\")\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        raw_position = RawPosition.PosObject &amp; key\n        spatial_series = raw_position.fetch_nwb()[0][\"raw_position\"]\n        spatial_df = raw_position.fetch1_dataframe()\n\n        position_info_parameters = (PositionInfoParameters() &amp; key).fetch1()\n\n        position_info = self.calculate_position_info(\n            spatial_df=spatial_df,\n            meters_to_pixels=spatial_series.conversion,\n            **position_info_parameters,\n        )\n\n        key.update(\n            dict(\n                analysis_file_name=analysis_file_name,\n                **self.generate_pos_components(\n                    spatial_series=spatial_series,\n                    position_info=position_info,\n                    analysis_fname=analysis_file_name,\n                ),\n            )\n        )\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        self.insert1(key)\n\n    @staticmethod\n    def generate_pos_components(\n        spatial_series,\n        position_info,\n        analysis_fname,\n        prefix=\"head_\",\n        add_frame_ind=False,\n        video_frame_ind=None,\n    ):\n        \"\"\"Generate position, orientation and velocity components.\"\"\"\n        METERS_PER_CM = 0.01\n\n        position = pynwb.behavior.Position()\n        orientation = pynwb.behavior.CompassDirection()\n        velocity = pynwb.behavior.BehavioralTimeSeries()\n\n        # NOTE: CBroz1 removed a try/except ValueError that surrounded all\n        #       .create_X_series methods. dpeg22 could not recall purpose\n\n        time_comments = dict(\n            comments=spatial_series.comments,\n            timestamps=position_info[\"time\"],\n        )\n        time_comments_ref = dict(\n            **time_comments,\n            reference_frame=spatial_series.reference_frame,\n        )\n\n        # create nwb objects for insertion into analysis nwb file\n        position.create_spatial_series(\n            name=f\"{prefix}position\",\n            conversion=METERS_PER_CM,\n            data=position_info[\"position\"],\n            description=f\"{prefix}x_position, {prefix}y_position\",\n            **time_comments_ref,\n        )\n\n        orientation.create_spatial_series(\n            name=f\"{prefix}orientation\",\n            conversion=1.0,\n            data=position_info[\"orientation\"],\n            description=f\"{prefix}orientation\",\n            **time_comments_ref,\n        )\n\n        velocity.create_timeseries(\n            name=f\"{prefix}velocity\",\n            conversion=METERS_PER_CM,\n            unit=\"m/s\",\n            data=np.concatenate(\n                (\n                    position_info[\"velocity\"],\n                    position_info[\"speed\"][:, np.newaxis],\n                ),\n                axis=1,\n            ),\n            description=f\"{prefix}x_velocity, {prefix}y_velocity, \"\n            + f\"{prefix}speed\",\n            **time_comments,\n        )\n\n        if add_frame_ind:\n            if video_frame_ind is not None:\n                velocity.create_timeseries(\n                    name=\"video_frame_ind\",\n                    unit=\"index\",\n                    data=video_frame_ind.to_numpy(),\n                    description=\"video_frame_ind\",\n                    **time_comments,\n                )\n            else:\n                logger.info(\n                    \"No video frame index found. Assuming all camera frames \"\n                    + \"are present.\"\n                )\n                velocity.create_timeseries(\n                    name=\"video_frame_ind\",\n                    unit=\"index\",\n                    data=np.arange(len(position_info[\"time\"])),\n                    description=\"video_frame_ind\",\n                    **time_comments,\n                )\n\n        # Insert into analysis nwb file\n        nwba = AnalysisNwbfile()\n\n        return {\n            f\"{prefix}position_object_id\": nwba.add_nwb_object(\n                analysis_fname, position\n            ),\n            f\"{prefix}orientation_object_id\": nwba.add_nwb_object(\n                analysis_fname, orientation\n            ),\n            f\"{prefix}velocity_object_id\": nwba.add_nwb_object(\n                analysis_fname, velocity\n            ),\n        }\n\n    @staticmethod\n    def _fix_kwargs(\n        orient_smoothing_std_dev=None,\n        speed_smoothing_std_dev=None,\n        max_LED_separation=None,\n        max_plausible_speed=None,\n        **kwargs,\n    ):\n        \"\"\"Handles discrepancies between common and v1 param names.\"\"\"\n        if not orient_smoothing_std_dev:\n            orient_smoothing_std_dev = kwargs.get(\n                \"head_orient_smoothing_std_dev\"\n            )\n        if not speed_smoothing_std_dev:\n            speed_smoothing_std_dev = kwargs.get(\"head_speed_smoothing_std_dev\")\n        if not max_LED_separation:\n            max_LED_separation = kwargs.get(\"max_separation\")\n        if not max_plausible_speed:\n            max_plausible_speed = kwargs.get(\"max_speed\")\n        if not all(\n            [speed_smoothing_std_dev, max_LED_separation, max_plausible_speed]\n        ):\n            raise ValueError(\n                \"Missing at least one required parameter:\\n\\t\"\n                + f\"speed_smoothing_std_dev: {speed_smoothing_std_dev}\\n\\t\"\n                + f\"max_LED_separation: {max_LED_separation}\\n\\t\"\n                + f\"max_plausible_speed: {max_plausible_speed}\"\n            )\n        return (\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n        )\n\n    @staticmethod\n    def _upsample(\n        front_LED,\n        back_LED,\n        time,\n        sampling_rate,\n        upsampling_sampling_rate,\n        upsampling_interpolation_method,\n        **kwargs,\n    ):\n        position_df = pd.DataFrame(\n            {\n                \"time\": time,\n                \"back_LED_x\": back_LED[:, 0],\n                \"back_LED_y\": back_LED[:, 1],\n                \"front_LED_x\": front_LED[:, 0],\n                \"front_LED_y\": front_LED[:, 1],\n            }\n        ).set_index(\"time\")\n\n        upsampling_start_time = time[0]\n        upsampling_end_time = time[-1]\n\n        n_samples = (\n            int(\n                np.ceil(\n                    (upsampling_end_time - upsampling_start_time)\n                    * upsampling_sampling_rate\n                )\n            )\n            + 1\n        )\n        new_time = np.linspace(\n            upsampling_start_time, upsampling_end_time, n_samples\n        )\n        new_index = pd.Index(\n            np.unique(np.concatenate((position_df.index, new_time))),\n            name=\"time\",\n        )\n        position_df = (\n            position_df.reindex(index=new_index)\n            .interpolate(method=upsampling_interpolation_method)\n            .reindex(index=new_time)\n        )\n\n        time = np.asarray(position_df.index)\n        back_LED = np.asarray(position_df.loc[:, [\"back_LED_x\", \"back_LED_y\"]])\n        front_LED = np.asarray(\n            position_df.loc[:, [\"front_LED_x\", \"front_LED_y\"]]\n        )\n\n        sampling_rate = upsampling_sampling_rate\n\n        return front_LED, back_LED, time, sampling_rate\n\n    def calculate_position_info(\n        self,\n        spatial_df: pd.DataFrame,\n        meters_to_pixels: float,\n        position_smoothing_duration,\n        led1_is_front,\n        is_upsampled,\n        upsampling_sampling_rate,\n        upsampling_interpolation_method,\n        orient_smoothing_std_dev=None,\n        speed_smoothing_std_dev=None,\n        max_LED_separation=None,\n        max_plausible_speed=None,\n        **kwargs,\n    ):\n        CM_TO_METERS = 100\n\n        (\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n        ) = self._fix_kwargs(\n            orient_smoothing_std_dev,\n            speed_smoothing_std_dev,\n            max_LED_separation,\n            max_plausible_speed,\n            **kwargs,\n        )\n\n        spatial_df = _fix_col_names(spatial_df)\n        # Get spatial series properties\n        time = np.asarray(spatial_df.index)  # seconds\n        position = np.asarray(spatial_df.iloc[:, :4])  # meters\n\n        # remove NaN times\n        is_nan_time = np.isnan(time)\n        position = position[~is_nan_time]\n        time = time[~is_nan_time]\n\n        dt = np.median(np.diff(time))\n        sampling_rate = 1 / dt\n\n        if position.shape[1] &lt; 4:\n            front_LED = position.astype(float)\n            back_LED = position.astype(float)\n        else:\n            # If there are 4 columns, then there are 2 LEDs\n            if led1_is_front:\n                front_LED = position[:, [0, 1]].astype(float)\n                back_LED = position[:, [2, 3]].astype(float)\n            else:\n                back_LED = position[:, [0, 1]].astype(float)\n                front_LED = position[:, [2, 3]].astype(float)\n\n        # Convert to cm\n        back_LED *= meters_to_pixels * CM_TO_METERS\n        front_LED *= meters_to_pixels * CM_TO_METERS\n\n        # Set points to NaN where the front and back LEDs are too separated\n        dist_between_LEDs = get_distance(back_LED, front_LED)\n        is_too_separated = dist_between_LEDs &gt;= max_LED_separation\n        if np.all(is_too_separated):\n            raise ValueError(\n                \"All points are too far apart. If this is single LED data,\"\n                + \"please check that using a parameter set with large max_LED_seperation.\"\n                + f\"Current max_LED_separation: {max_LED_separation}\"\n            )\n\n        back_LED[is_too_separated] = np.nan\n        front_LED[is_too_separated] = np.nan\n\n        # Calculate speed\n        front_LED_speed = get_speed(\n            front_LED,\n            time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )\n        back_LED_speed = get_speed(\n            back_LED,\n            time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )\n\n        # Set to points to NaN where the speed is too fast\n        is_too_fast = (front_LED_speed &gt; max_plausible_speed) | (\n            back_LED_speed &gt; max_plausible_speed\n        )\n        back_LED[is_too_fast] = np.nan\n        front_LED[is_too_fast] = np.nan\n\n        # Interpolate the NaN points\n        back_LED = interpolate_nan(back_LED)\n        front_LED = interpolate_nan(front_LED)\n\n        # Smooth\n        moving_average_window = int(position_smoothing_duration * sampling_rate)\n        back_LED = bottleneck.move_mean(\n            back_LED, window=moving_average_window, axis=0, min_count=1\n        )\n        front_LED = bottleneck.move_mean(\n            front_LED, window=moving_average_window, axis=0, min_count=1\n        )\n\n        if is_upsampled:\n            front_LED, back_LED, time, sampling_rate = self._upsample(\n                front_LED,\n                back_LED,\n                time,\n                sampling_rate,\n                upsampling_sampling_rate,\n                upsampling_interpolation_method,\n            )\n\n        # Calculate position, orientation, velocity, speed\n        position = get_centroid(back_LED, front_LED)  # cm\n\n        orientation = get_angle(back_LED, front_LED)  # radians\n        is_nan = np.isnan(orientation)\n\n        # Unwrap orientation before smoothing\n        orientation[~is_nan] = np.unwrap(orientation[~is_nan])\n        orientation[~is_nan] = gaussian_smooth(\n            orientation[~is_nan],\n            orient_smoothing_std_dev,\n            sampling_rate,\n            axis=0,\n            truncate=8,\n        )\n        # convert back to between -pi and pi\n        orientation[~is_nan] = np.angle(np.exp(1j * orientation[~is_nan]))\n\n        velocity = get_velocity(\n            position,\n            time=time,\n            sigma=speed_smoothing_std_dev,\n            sampling_frequency=sampling_rate,\n        )  # cm/s\n        speed = np.sqrt(np.sum(velocity**2, axis=1))  # cm/s\n\n        return {\n            \"time\": time,\n            \"position\": position,\n            \"orientation\": orientation,\n            \"velocity\": velocity,\n            \"speed\": speed,\n        }\n\n    def fetch1_dataframe(self):\n        return self._data_to_df(self.fetch_nwb()[0])\n\n    @staticmethod\n    def _data_to_df(data, prefix=\"head_\", add_frame_ind=False):\n        pos, ori, vel = [\n            prefix + c for c in [\"position\", \"orientation\", \"velocity\"]\n        ]\n\n        COLUMNS = [\n            f\"{pos}_x\",\n            f\"{pos}_y\",\n            ori,\n            f\"{vel}_x\",\n            f\"{vel}_y\",\n            f\"{prefix}speed\",\n        ]\n\n        df = pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(data[pos].get_spatial_series().data),\n                    np.asarray(data[ori].get_spatial_series().data)[\n                        :, np.newaxis\n                    ],\n                    np.asarray(data[vel].time_series[vel].data),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=pd.Index(\n                np.asarray(data[pos].get_spatial_series().timestamps),\n                name=\"time\",\n            ),\n        )\n\n        if add_frame_ind:\n            df.insert(\n                0,\n                \"video_frame_ind\",\n                np.asarray(\n                    data[vel].time_series[\"video_frame_ind\"].data,\n                    dtype=int,\n                ),\n            )\n\n        return df\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.IntervalPositionInfo.generate_pos_components", "title": "<code>generate_pos_components(spatial_series, position_info, analysis_fname, prefix='head_', add_frame_ind=False, video_frame_ind=None)</code>  <code>staticmethod</code>", "text": "<p>Generate position, orientation and velocity components.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@staticmethod\ndef generate_pos_components(\n    spatial_series,\n    position_info,\n    analysis_fname,\n    prefix=\"head_\",\n    add_frame_ind=False,\n    video_frame_ind=None,\n):\n    \"\"\"Generate position, orientation and velocity components.\"\"\"\n    METERS_PER_CM = 0.01\n\n    position = pynwb.behavior.Position()\n    orientation = pynwb.behavior.CompassDirection()\n    velocity = pynwb.behavior.BehavioralTimeSeries()\n\n    # NOTE: CBroz1 removed a try/except ValueError that surrounded all\n    #       .create_X_series methods. dpeg22 could not recall purpose\n\n    time_comments = dict(\n        comments=spatial_series.comments,\n        timestamps=position_info[\"time\"],\n    )\n    time_comments_ref = dict(\n        **time_comments,\n        reference_frame=spatial_series.reference_frame,\n    )\n\n    # create nwb objects for insertion into analysis nwb file\n    position.create_spatial_series(\n        name=f\"{prefix}position\",\n        conversion=METERS_PER_CM,\n        data=position_info[\"position\"],\n        description=f\"{prefix}x_position, {prefix}y_position\",\n        **time_comments_ref,\n    )\n\n    orientation.create_spatial_series(\n        name=f\"{prefix}orientation\",\n        conversion=1.0,\n        data=position_info[\"orientation\"],\n        description=f\"{prefix}orientation\",\n        **time_comments_ref,\n    )\n\n    velocity.create_timeseries(\n        name=f\"{prefix}velocity\",\n        conversion=METERS_PER_CM,\n        unit=\"m/s\",\n        data=np.concatenate(\n            (\n                position_info[\"velocity\"],\n                position_info[\"speed\"][:, np.newaxis],\n            ),\n            axis=1,\n        ),\n        description=f\"{prefix}x_velocity, {prefix}y_velocity, \"\n        + f\"{prefix}speed\",\n        **time_comments,\n    )\n\n    if add_frame_ind:\n        if video_frame_ind is not None:\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                data=video_frame_ind.to_numpy(),\n                description=\"video_frame_ind\",\n                **time_comments,\n            )\n        else:\n            logger.info(\n                \"No video frame index found. Assuming all camera frames \"\n                + \"are present.\"\n            )\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                data=np.arange(len(position_info[\"time\"])),\n                description=\"video_frame_ind\",\n                **time_comments,\n            )\n\n    # Insert into analysis nwb file\n    nwba = AnalysisNwbfile()\n\n    return {\n        f\"{prefix}position_object_id\": nwba.add_nwb_object(\n            analysis_fname, position\n        ),\n        f\"{prefix}orientation_object_id\": nwba.add_nwb_object(\n            analysis_fname, orientation\n        ),\n        f\"{prefix}velocity_object_id\": nwba.add_nwb_object(\n            analysis_fname, velocity\n        ),\n    }\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionVideo", "title": "<code>PositionVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@schema\nclass PositionVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalPositionInfo\n    \"\"\"\n\n    def make(self, key):\n        M_TO_CM = 100\n\n        logger.info(\"Loading position data...\")\n        raw_position_df = (\n            RawPosition()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch1_dataframe()\n        position_info_df = (\n            IntervalPositionInfo()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n                \"position_info_param_name\": key[\"position_info_param_name\"],\n            }\n        ).fetch1_dataframe()\n\n        logger.info(\"Loading video data...\")\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n        video_info = (\n            VideoFile()\n            &amp; {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        ).fetch1()\n        io = pynwb.NWBHDF5IO(raw_dir + \"/\" + video_info[\"nwb_file_name\"], \"r\")\n        nwb_file = io.read()\n        nwb_video = nwb_file.objects[video_info[\"video_file_object_id\"]]\n        video_filename = nwb_video.external_file[0]\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        output_video_filename = (\n            f\"{nwb_base_filename}_{epoch:02d}_\"\n            f'{key[\"position_info_param_name\"]}.mp4'\n        )\n\n        # ensure standardized column names\n        raw_position_df = _fix_col_names(raw_position_df)\n        # if IntervalPositionInfo supersampled position, downsample to video\n        if position_info_df.shape[0] &gt; raw_position_df.shape[0]:\n            ind = np.digitize(\n                raw_position_df.index, position_info_df.index, right=True\n            )\n            position_info_df = position_info_df.iloc[ind]\n\n        centroids = {\n            \"red\": np.asarray(raw_position_df[[\"xloc\", \"yloc\"]]),\n            \"green\": np.asarray(raw_position_df[[\"xloc2\", \"yloc2\"]]),\n        }\n        head_position_mean = np.asarray(\n            position_info_df[[\"head_position_x\", \"head_position_y\"]]\n        )\n        head_orientation_mean = np.asarray(\n            position_info_df[[\"head_orientation\"]]\n        )\n        video_time = np.asarray(nwb_video.timestamps)\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = nwb_video.device.meters_per_pixel * M_TO_CM\n\n        logger.info(\"Making video...\")\n        self.make_video(\n            f\"{video_dir}/{video_filename}\",\n            centroids,\n            head_position_mean,\n            head_orientation_mean,\n            video_time,\n            position_time,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n        )\n\n    @staticmethod\n    def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n        \"\"\"Converts from cm to pixels and flips the y-axis.\n        Parameters\n        ----------\n        data : ndarray, shape (n_time, 2)\n        frame_size : array_like, shape (2,)\n        cm_to_pixels : float\n\n        Returns\n        -------\n        converted_data : ndarray, shape (n_time, 2)\n        \"\"\"\n        return data / cm_to_pixels\n\n    @staticmethod\n    def fill_nan(variable, video_time, variable_time):\n        video_ind = np.digitize(variable_time, video_time[1:])\n\n        n_video_time = len(video_time)\n        try:\n            n_variable_dims = variable.shape[1]\n            filled_variable = np.full((n_video_time, n_variable_dims), np.nan)\n        except IndexError:\n            filled_variable = np.full((n_video_time,), np.nan)\n        filled_variable[video_ind] = variable\n\n        return filled_variable\n\n    def make_video(\n        self,\n        video_filename,\n        centroids,\n        head_position_mean,\n        head_orientation_mean,\n        video_time,\n        position_time,\n        output_video_filename=\"output.mp4\",\n        cm_to_pixels=1.0,\n        disable_progressbar=False,\n        arrow_radius=15,\n        circle_radius=8,\n    ):\n        import cv2  # noqa: F401\n\n        RGB_PINK = (234, 82, 111)\n        RGB_YELLOW = (253, 231, 76)\n        RGB_WHITE = (255, 255, 255)\n\n        video = cv2.VideoCapture(video_filename)\n        fourcc = cv2.VideoWriter_fourcc(*\"mp4v\")\n        frame_size = (int(video.get(3)), int(video.get(4)))\n        frame_rate = video.get(5)\n        n_frames = int(head_orientation_mean.shape[0])\n\n        out = cv2.VideoWriter(\n            output_video_filename, fourcc, frame_rate, frame_size, True\n        )\n\n        centroids = {\n            color: self.fill_nan(data, video_time, position_time)\n            for color, data in centroids.items()\n        }\n        head_position_mean = self.fill_nan(\n            head_position_mean, video_time, position_time\n        )\n        head_orientation_mean = self.fill_nan(\n            head_orientation_mean, video_time, position_time\n        )\n\n        for time_ind in tqdm(\n            range(n_frames - 1), desc=\"frames\", disable=disable_progressbar\n        ):\n            is_grabbed, frame = video.read()\n            if is_grabbed:\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n\n                red_centroid = centroids[\"red\"][time_ind]\n                green_centroid = centroids[\"green\"][time_ind]\n\n                head_position = head_position_mean[time_ind]\n                head_position = self.convert_to_pixels(\n                    head_position, frame_size, cm_to_pixels\n                )\n                head_orientation = head_orientation_mean[time_ind]\n\n                if np.all(~np.isnan(red_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(red_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_YELLOW,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(green_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(green_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_PINK,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(head_position)) &amp; np.all(\n                    ~np.isnan(head_orientation)\n                ):\n                    arrow_tip = (\n                        int(\n                            head_position[0]\n                            + arrow_radius * np.cos(head_orientation)\n                        ),\n                        int(\n                            head_position[1]\n                            + arrow_radius * np.sin(head_orientation)\n                        ),\n                    )\n                    cv2.arrowedLine(\n                        img=frame,\n                        pt1=tuple(head_position.astype(int)),\n                        pt2=arrow_tip,\n                        color=RGB_WHITE,\n                        thickness=4,\n                        line_type=8,\n                        shift=cv2.CV_8U,\n                        tipLength=0.25,\n                    )\n\n                if np.all(~np.isnan(head_position)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(head_position.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_WHITE,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)\n                out.write(frame)\n            else:\n                break\n\n        video.release()\n        out.release()\n        cv2.destroyAllWindows()\n</code></pre>"}, {"location": "api/common/common_position/#src.spyglass.common.common_position.PositionVideo.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>  <code>staticmethod</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/common/common_position.py</code> <pre><code>@staticmethod\ndef convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/common/common_region/", "title": "common_region.py", "text": ""}, {"location": "api/common/common_region/#src.spyglass.common.common_region.BrainRegion", "title": "<code>BrainRegion</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/common/common_region.py</code> <pre><code>@schema\nclass BrainRegion(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    region_id: smallint auto_increment\n    ---\n    region_name: varchar(200)             # the name of the brain region\n    subregion_name=NULL: varchar(200)     # subregion name\n    subsubregion_name=NULL: varchar(200)  # subregion within subregion\n    \"\"\"\n\n    # TODO consider making (region_name, subregion_name, subsubregion_name) a\n    # primary key subregion_name='' and subsubregion_name='' will be necessary\n    # but that seems OK\n\n    @classmethod\n    def fetch_add(\n        cls, region_name, subregion_name=None, subsubregion_name=None\n    ):\n        \"\"\"Return the region ID for names. If no match, add to the BrainRegion.\n\n        The combination of (region_name, subregion_name, subsubregion_name) is\n        effectively unique, then.\n\n        Parameters\n        ----------\n        region_name : str\n            The name of the brain region.\n        subregion_name : str, optional\n            The name of the subregion within the brain region.\n        subsubregion_name : str, optional\n            The name of the subregion within the subregion.\n\n        Returns\n        -------\n        region_id : int\n            The index of the region in the BrainRegion table.\n        \"\"\"\n        key = dict(\n            region_name=region_name,\n            subregion_name=subregion_name,\n            subsubregion_name=subsubregion_name,\n        )\n        query = BrainRegion &amp; key\n        if not query:\n            cls.insert1(key)\n            query = BrainRegion &amp; key\n        return query.fetch1(\"region_id\")\n</code></pre>"}, {"location": "api/common/common_region/#src.spyglass.common.common_region.BrainRegion.fetch_add", "title": "<code>fetch_add(region_name, subregion_name=None, subsubregion_name=None)</code>  <code>classmethod</code>", "text": "<p>Return the region ID for names. If no match, add to the BrainRegion.</p> <p>The combination of (region_name, subregion_name, subsubregion_name) is effectively unique, then.</p> <p>Parameters:</p> Name Type Description Default <code>region_name</code> <code>str</code> <p>The name of the brain region.</p> required <code>subregion_name</code> <code>str</code> <p>The name of the subregion within the brain region.</p> <code>None</code> <code>subsubregion_name</code> <code>str</code> <p>The name of the subregion within the subregion.</p> <code>None</code> <p>Returns:</p> Name Type Description <code>region_id</code> <code>int</code> <p>The index of the region in the BrainRegion table.</p> Source code in <code>src/spyglass/common/common_region.py</code> <pre><code>@classmethod\ndef fetch_add(\n    cls, region_name, subregion_name=None, subsubregion_name=None\n):\n    \"\"\"Return the region ID for names. If no match, add to the BrainRegion.\n\n    The combination of (region_name, subregion_name, subsubregion_name) is\n    effectively unique, then.\n\n    Parameters\n    ----------\n    region_name : str\n        The name of the brain region.\n    subregion_name : str, optional\n        The name of the subregion within the brain region.\n    subsubregion_name : str, optional\n        The name of the subregion within the subregion.\n\n    Returns\n    -------\n    region_id : int\n        The index of the region in the BrainRegion table.\n    \"\"\"\n    key = dict(\n        region_name=region_name,\n        subregion_name=subregion_name,\n        subsubregion_name=subsubregion_name,\n    )\n    query = BrainRegion &amp; key\n    if not query:\n        cls.insert1(key)\n        query = BrainRegion &amp; key\n    return query.fetch1(\"region_id\")\n</code></pre>"}, {"location": "api/common/common_ripple/", "title": "common_ripple.py", "text": ""}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleLFPSelection", "title": "<code>RippleLFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleLFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; LFPBand\n     group_name = 'CA1' : varchar(80)\n     \"\"\"\n\n    class RippleLFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; RippleLFPSelection\n        -&gt; LFPBandSelection.LFPBandElectrode\n        \"\"\"\n\n    def insert1(self, key, **kwargs):\n        filter_name = (LFPBand &amp; key).fetch1(\"filter_name\")\n        if \"ripple\" not in filter_name.lower():\n            logger.warning(\"Please use a ripple filter\")\n        super().insert1(key, **kwargs)\n\n    @staticmethod\n    def set_lfp_electrodes(\n        key,\n        electrode_list=None,\n        group_name=\"CA1\",\n        **kwargs,\n    ):\n        \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n        Parameters\n        ----------\n        key : dict\n            dictionary corresponding to the LFPBand entry to use for ripple detection\n        electrode_list : list\n            list of electrodes from LFPBandSelection.LFPBandElectrode\n            to be used as the ripple LFP during detection\n        group_name : str, optional\n            description of the electrode group, by default \"CA1\"\n        \"\"\"\n\n        RippleLFPSelection().insert1(\n            {**key, \"group_name\": group_name},\n            skip_duplicates=True,\n            **kwargs,\n        )\n        if not electrode_list:\n            electrode_list = (\n                (LFPBandSelection.LFPBandElectrode() &amp; key)\n                .fetch(\"electrode_id\")\n                .tolist()\n            )\n        electrode_list.sort()\n        electrode_keys = (\n            pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n            .set_index(\"electrode_id\")\n            .loc[electrode_list]\n            .reset_index()\n            .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n        )\n        electrode_keys[\"group_name\"] = group_name\n        electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n        RippleLFPSelection().RippleLFPElectrode.insert(\n            electrode_keys.to_dict(orient=\"records\"),\n            replace=True,\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleLFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(key, electrode_list=None, group_name='CA1', **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary corresponding to the LFPBand entry to use for ripple detection</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes from LFPBandSelection.LFPBandElectrode to be used as the ripple LFP during detection</p> <code>None</code> <code>group_name</code> <code>str</code> <p>description of the electrode group, by default \"CA1\"</p> <code>'CA1'</code> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@staticmethod\ndef set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name=\"CA1\",\n    **kwargs,\n):\n    \"\"\"Removes all electrodes for the specified nwb file and then adds back the electrodes in the list\n\n    Parameters\n    ----------\n    key : dict\n        dictionary corresponding to the LFPBand entry to use for ripple detection\n    electrode_list : list\n        list of electrodes from LFPBandSelection.LFPBandElectrode\n        to be used as the ripple LFP during detection\n    group_name : str, optional\n        description of the electrode group, by default \"CA1\"\n    \"\"\"\n\n    RippleLFPSelection().insert1(\n        {**key, \"group_name\": group_name},\n        skip_duplicates=True,\n        **kwargs,\n    )\n    if not electrode_list:\n        electrode_list = (\n            (LFPBandSelection.LFPBandElectrode() &amp; key)\n            .fetch(\"electrode_id\")\n            .tolist()\n        )\n    electrode_list.sort()\n    electrode_keys = (\n        pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n        .set_index(\"electrode_id\")\n        .loc[electrode_list]\n        .reset_index()\n        .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n    )\n    electrode_keys[\"group_name\"] = group_name\n    electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n    RippleLFPSelection().RippleLFPElectrode.insert(\n        electrode_keys.to_dict(orient=\"records\"),\n        replace=True,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleParameters", "title": "<code>RippleParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleParameters(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    ripple_param_name : varchar(80) # a name for this set of parameters\n    ----\n    ripple_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\"\"\"\n        default_dict = {\n            \"speed_name\": \"head_speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\"\"\"\n    default_dict = {\n        \"speed_name\": \"head_speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleTimes", "title": "<code>RippleTimes</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>@schema\nclass RippleTimes(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; RippleParameters\n    -&gt; RippleLFPSelection\n    -&gt; IntervalPositionInfo\n    ---\n    -&gt; AnalysisNwbfile\n    ripple_times_object_id : varchar(40)\n     \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing ripple times for: {key}\")\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        ripple_detection_algorithm = ripple_params[\"ripple_detection_algorithm\"]\n        ripple_detection_params = ripple_params[\"ripple_detection_params\"]\n\n        (\n            speed,\n            interval_ripple_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n\n        ripple_times = RIPPLE_DETECTION_ALGORITHMS[ripple_detection_algorithm](\n            time=np.asarray(interval_ripple_lfps.index),\n            filtered_lfps=np.asarray(interval_ripple_lfps),\n            speed=np.asarray(speed),\n            sampling_frequency=sampling_frequency,\n            **ripple_detection_params,\n        )\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(\n            key[\"nwb_file_name\"]\n        )\n        key[\"ripple_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=ripple_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"ripple_times\"] for data in self.fetch_nwb()]\n\n    @staticmethod\n    def get_ripple_lfps_and_position_info(key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        interval_list_name = key[\"target_interval_list_name\"]\n        position_info_param_name = key[\"position_info_param_name\"]\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        speed_name = ripple_params[\"speed_name\"]\n\n        electrode_keys = (RippleLFPSelection.RippleLFPElectrode() &amp; key).fetch(\n            \"electrode_id\"\n        )\n\n        # warn/validate that there is only one wire per electrode\n        lfp_key = key.copy()\n        del lfp_key[\"interval_list_name\"]\n        ripple_lfp_nwb = (LFPBand &amp; lfp_key).fetch_nwb()[0]\n        ripple_lfp_electrodes = ripple_lfp_nwb[\"filtered_data\"].electrodes.data[\n            :\n        ]\n        elec_mask = np.full_like(ripple_lfp_electrodes, 0, dtype=bool)\n        elec_mask[\n            [\n                ind\n                for ind, elec in enumerate(ripple_lfp_electrodes)\n                if elec in electrode_keys\n            ]\n        ] = True\n        ripple_lfp = pd.DataFrame(\n            ripple_lfp_nwb[\"filtered_data\"].data,\n            index=pd.Index(\n                ripple_lfp_nwb[\"filtered_data\"].timestamps, name=\"time\"\n            ),\n        )\n        sampling_frequency = ripple_lfp_nwb[\"lfp_band_sampling_rate\"]\n\n        ripple_lfp = ripple_lfp.loc[:, elec_mask]\n\n        position_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        position_info = (\n            IntervalPositionInfo()\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n                \"position_info_param_name\": position_info_param_name,\n            }\n        ).fetch1_dataframe()\n\n        position_info = pd.concat(\n            [\n                position_info.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=1,\n        )\n        interval_ripple_lfps = pd.concat(\n            [\n                ripple_lfp.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=1,\n        )\n\n        position_info = interpolate_to_new_time(\n            position_info, interval_ripple_lfps.index\n        )\n\n        return (\n            position_info[speed_name],\n            interval_ripple_lfps,\n            sampling_frequency,\n        )\n\n    @staticmethod\n    def get_Kay_ripple_consensus_trace(\n        ripple_filtered_lfps, sampling_frequency, smoothing_sigma=0.004\n    ):\n        ripple_consensus_trace = np.full_like(ripple_filtered_lfps, np.nan)\n        not_null = np.all(pd.notnull(ripple_filtered_lfps), axis=1)\n\n        ripple_consensus_trace[not_null] = get_envelope(\n            np.asarray(ripple_filtered_lfps)[not_null]\n        )\n        ripple_consensus_trace = np.sum(ripple_consensus_trace**2, axis=1)\n        ripple_consensus_trace[not_null] = gaussian_smooth(\n            ripple_consensus_trace[not_null],\n            smoothing_sigma,\n            sampling_frequency,\n        )\n        return pd.DataFrame(\n            np.sqrt(ripple_consensus_trace), index=ripple_filtered_lfps.index\n        )\n\n    @staticmethod\n    def plot_ripple_consensus_trace(\n        ripple_consensus_trace,\n        ripple_times,\n        ripple_label=1,\n        offset=0.100,\n        relative=True,\n        ax=None,\n    ):\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n\n        start_offset = ripple_start if relative else 0\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, 1))\n        ax.plot(\n            ripple_consensus_trace.loc[time_slice].index - start_offset,\n            ripple_consensus_trace.loc[time_slice],\n        )\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_xlabel(\"Time [s]\")\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n\n    @staticmethod\n    def plot_ripple(\n        lfps, ripple_times, ripple_label=1, offset=0.100, relative=True, ax=None\n    ):\n        lfp_labels = lfps.columns\n        n_lfps = len(lfp_labels)\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, n_lfps * 0.20))\n\n        start_offset = ripple_start if relative else 0\n\n        for lfp_ind, lfp_label in enumerate(lfp_labels):\n            lfp = lfps.loc[time_slice, lfp_label]\n            ax.plot(\n                lfp.index - start_offset,\n                lfp_ind + (lfp - lfp.mean()) / (lfp.max() - lfp.min()),\n                color=\"black\",\n            )\n\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_ylim((-1, n_lfps))\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n        ax.set_ylabel(\"LFPs\")\n        ax.set_xlabel(\"Time [s]\")\n</code></pre>"}, {"location": "api/common/common_ripple/#src.spyglass.common.common_ripple.RippleTimes.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/common/common_ripple.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/common/common_sensors/", "title": "common_sensors.py", "text": "<p>Schema for headstage or other environmental sensors.</p>"}, {"location": "api/common/common_session/", "title": "common_session.py", "text": ""}, {"location": "api/common/common_subject/", "title": "common_subject.py", "text": ""}, {"location": "api/common/common_subject/#src.spyglass.common.common_subject.Subject", "title": "<code>Subject</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_subject.py</code> <pre><code>@schema\nclass Subject(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    subject_id: varchar(80)\n    ---\n    age = NULL: varchar(200)\n    description = NULL: varchar(2000)\n    genotype = NULL: varchar(2000)\n    sex = \"U\": enum(\"M\", \"F\", \"U\")\n    species = NULL: varchar(200)\n    \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Get the subject info from the NWBFile, insert into the Subject.\"\"\"\n        sub = nwbf.subject\n        if sub is None:\n            logger.warn(\"No subject metadata found.\\n\")\n            return\n        subject_dict = dict()\n        subject_dict[\"subject_id\"] = sub.subject_id\n        subject_dict[\"age\"] = sub.age\n        subject_dict[\"description\"] = sub.description\n        subject_dict[\"genotype\"] = sub.genotype\n        if sub.sex in (\"Male\", \"male\", \"M\", \"m\"):\n            sex = \"M\"\n        elif sub.sex in (\"Female\", \"female\", \"F\", \"f\"):\n            sex = \"F\"\n        else:\n            sex = \"U\"\n        subject_dict[\"sex\"] = sex\n        subject_dict[\"species\"] = sub.species\n        cls.insert1(subject_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_subject/#src.spyglass.common.common_subject.Subject.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Get the subject info from the NWBFile, insert into the Subject.</p> Source code in <code>src/spyglass/common/common_subject.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Get the subject info from the NWBFile, insert into the Subject.\"\"\"\n    sub = nwbf.subject\n    if sub is None:\n        logger.warn(\"No subject metadata found.\\n\")\n        return\n    subject_dict = dict()\n    subject_dict[\"subject_id\"] = sub.subject_id\n    subject_dict[\"age\"] = sub.age\n    subject_dict[\"description\"] = sub.description\n    subject_dict[\"genotype\"] = sub.genotype\n    if sub.sex in (\"Male\", \"male\", \"M\", \"m\"):\n        sex = \"M\"\n    elif sub.sex in (\"Female\", \"female\", \"F\", \"f\"):\n        sex = \"F\"\n    else:\n        sex = \"U\"\n    subject_dict[\"sex\"] = sex\n    subject_dict[\"species\"] = sub.species\n    cls.insert1(subject_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_task/", "title": "common_task.py", "text": ""}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task", "title": "<code>Task</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@schema\nclass Task(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     task_name: varchar(80)\n     ---\n     task_description = NULL: varchar(2000)    # description of this task\n     task_type = NULL: varchar(2000)           # type of task\n     task_subtype = NULL: varchar(2000)        # subtype of task\n     \"\"\"\n\n    @classmethod\n    def insert_from_nwbfile(cls, nwbf):\n        \"\"\"Insert tasks from an NWB file.\n\n        Parameters\n        ----------\n        nwbf : pynwb.NWBFile\n            The source NWB file object.\n        \"\"\"\n        tasks_mod = nwbf.processing.get(\"tasks\")\n        if tasks_mod is None:\n            logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n            return\n        for task in tasks_mod.data_interfaces.values():\n            if cls.check_task_table(task):\n                cls.insert_from_task_table(task)\n\n    @classmethod\n    def insert_from_task_table(cls, task_table):\n        \"\"\"Insert tasks from a pynwb DynamicTable containing task metadata.\n\n        Duplicate tasks will not be added.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n        \"\"\"\n        taskdf = task_table.to_dataframe()\n        for task_entry in taskdf.iterrows():\n            task_dict = dict()\n            task_dict[\"task_name\"] = task_entry[1].task_name\n            task_dict[\"task_description\"] = task_entry[1].task_description\n            cls.insert1(task_dict, skip_duplicates=True)\n\n    @classmethod\n    def check_task_table(cls, task_table):\n        \"\"\"Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.\n\n\n        The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and\n        'task_description'.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n\n        Returns\n        -------\n        bool\n            Whether the DynamicTable conforms to the expected format for loading data into the Task table.\n        \"\"\"\n        return (\n            isinstance(task_table, pynwb.core.DynamicTable)\n            and hasattr(task_table, \"task_name\")\n            and hasattr(task_table, \"task_description\")\n        )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.insert_from_nwbfile", "title": "<code>insert_from_nwbfile(nwbf)</code>  <code>classmethod</code>", "text": "<p>Insert tasks from an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef insert_from_nwbfile(cls, nwbf):\n    \"\"\"Insert tasks from an NWB file.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    \"\"\"\n    tasks_mod = nwbf.processing.get(\"tasks\")\n    if tasks_mod is None:\n        logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n        return\n    for task in tasks_mod.data_interfaces.values():\n        if cls.check_task_table(task):\n            cls.insert_from_task_table(task)\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.insert_from_task_table", "title": "<code>insert_from_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Insert tasks from a pynwb DynamicTable containing task metadata.</p> <p>Duplicate tasks will not be added.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef insert_from_task_table(cls, task_table):\n    \"\"\"Insert tasks from a pynwb DynamicTable containing task metadata.\n\n    Duplicate tasks will not be added.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n    \"\"\"\n    taskdf = task_table.to_dataframe()\n    for task_entry in taskdf.iterrows():\n        task_dict = dict()\n        task_dict[\"task_name\"] = task_entry[1].task_name\n        task_dict[\"task_description\"] = task_entry[1].task_description\n        cls.insert1(task_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.Task.check_task_table", "title": "<code>check_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.</p> <p>The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and 'task_description'.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether the DynamicTable conforms to the expected format for loading data into the Task table.</p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef check_task_table(cls, task_table):\n    \"\"\"Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.\n\n\n    The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name' and\n    'task_description'.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n\n    Returns\n    -------\n    bool\n        Whether the DynamicTable conforms to the expected format for loading data into the Task table.\n    \"\"\"\n    return (\n        isinstance(task_table, pynwb.core.DynamicTable)\n        and hasattr(task_table, \"task_name\")\n        and hasattr(task_table, \"task_description\")\n    )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.TaskEpoch", "title": "<code>TaskEpoch</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@schema\nclass TaskEpoch(SpyglassMixin, dj.Imported):\n    # Tasks, session and time intervals\n    definition = \"\"\"\n     -&gt; Session\n     epoch: int  # the session epoch for this task and apparatus(1 based)\n     ---\n     -&gt; Task\n     -&gt; [nullable] CameraDevice\n     -&gt; IntervalList\n     task_environment = NULL: varchar(200)  # the environment the animal was in\n     camera_names : blob # list of keys corresponding to entry in CameraDevice\n     \"\"\"\n\n    def make(self, key):\n        nwb_file_name = key[\"nwb_file_name\"]\n        nwb_file_abspath = Nwbfile().get_abs_path(nwb_file_name)\n        nwbf = get_nwb_file(nwb_file_abspath)\n        camera_names = dict()\n\n        # the tasks refer to the camera_id which is unique for the NWB file but\n        # not for CameraDevice schema, so we need to look up the right camera\n        # map camera ID (in camera name) to camera_name\n\n        for device in nwbf.devices.values():\n            if isinstance(device, ndx_franklab_novela.CameraDevice):\n                # get the camera ID\n                camera_id = int(str.split(device.name)[1])\n                camera_names[camera_id] = device.camera_name\n\n        # find the task modules and for each one, add the task to the Task\n        # schema if it isn't there and then add an entry for each epoch\n\n        tasks_mod = nwbf.processing.get(\"tasks\")\n        if tasks_mod is None:\n            logger.warn(f\"No tasks processing module found in {nwbf}\\n\")\n            return\n\n        for task in tasks_mod.data_interfaces.values():\n            if self.check_task_table(task):\n                # check if the task is in the Task table and if not, add it\n                Task.insert_from_task_table(task)\n                key[\"task_name\"] = task.task_name[0]\n\n                # get the CameraDevice used for this task (primary key is\n                # camera name so we need to map from ID to name)\n\n                camera_ids = task.camera_id[0]\n                valid_camera_ids = [\n                    camera_id\n                    for camera_id in camera_ids\n                    if camera_id in camera_names.keys()\n                ]\n                if valid_camera_ids:\n                    key[\"camera_names\"] = [\n                        {\"camera_name\": camera_names[camera_id]}\n                        for camera_id in valid_camera_ids\n                    ]\n                else:\n                    logger.warn(\n                        f\"No camera device found with ID {camera_ids} in NWB \"\n                        + f\"file {nwbf}\\n\"\n                    )\n                # Add task environment\n                if hasattr(task, \"task_environment\"):\n                    key[\"task_environment\"] = task.task_environment[0]\n\n                # get the interval list for this task, which corresponds to the\n                # matching epoch for the raw data. Users should define more\n                # restrictive intervals as required for analyses\n\n                session_intervals = (\n                    IntervalList() &amp; {\"nwb_file_name\": nwb_file_name}\n                ).fetch(\"interval_list_name\")\n                for epoch in task.task_epochs[0]:\n                    # TODO in beans file, task_epochs[0] is 1x2 dset of ints,\n                    # so epoch would be an int\n\n                    key[\"epoch\"] = epoch\n                    target_interval = str(epoch).zfill(2)\n                    for interval in session_intervals:\n                        if (\n                            target_interval in interval\n                        ):  # TODO this is not true for the beans file\n                            break\n                    # TODO case when interval is not found is not handled\n                    key[\"interval_list_name\"] = interval\n                    self.insert1(key)\n\n    @classmethod\n    def update_entries(cls, restrict={}):\n        existing_entries = (cls &amp; restrict).fetch(\"KEY\")\n        for row in existing_entries:\n            if (cls &amp; row).fetch1(\"camera_names\"):\n                continue\n            row[\"camera_names\"] = [\n                {\"camera_name\": (cls &amp; row).fetch1(\"camera_name\")}\n            ]\n            cls.update1(row=row)\n\n    @classmethod\n    def check_task_table(cls, task_table) -&gt; bool:\n        \"\"\"Check whether the pynwb DynamicTable containing task metadata\n        conforms to the expected format.\n\n        The table should be an instance of pynwb.core.DynamicTable and contain\n        the columns 'task_name', 'task_description', 'camera_id', 'and\n        'task_epochs'.\n\n        Parameters\n        ----------\n        task_table : pynwb.core.DynamicTable\n            The table representing task metadata.\n\n        Returns\n        -------\n        bool\n            Whether the DynamicTable conforms to the expected format for\n            loading data into the TaskEpoch table.\n        \"\"\"\n\n        # TODO this could be more strict and check data types, but really it\n        # should be schematized\n        return (\n            Task.check_task_table(task_table)\n            and hasattr(task_table, \"camera_id\")\n            and hasattr(task_table, \"task_epochs\")\n        )\n</code></pre>"}, {"location": "api/common/common_task/#src.spyglass.common.common_task.TaskEpoch.check_task_table", "title": "<code>check_task_table(task_table)</code>  <code>classmethod</code>", "text": "<p>Check whether the pynwb DynamicTable containing task metadata conforms to the expected format.</p> <p>The table should be an instance of pynwb.core.DynamicTable and contain the columns 'task_name', 'task_description', 'camera_id', 'and 'task_epochs'.</p> <p>Parameters:</p> Name Type Description Default <code>task_table</code> <code>DynamicTable</code> <p>The table representing task metadata.</p> required <p>Returns:</p> Type Description <code>bool</code> <p>Whether the DynamicTable conforms to the expected format for loading data into the TaskEpoch table.</p> Source code in <code>src/spyglass/common/common_task.py</code> <pre><code>@classmethod\ndef check_task_table(cls, task_table) -&gt; bool:\n    \"\"\"Check whether the pynwb DynamicTable containing task metadata\n    conforms to the expected format.\n\n    The table should be an instance of pynwb.core.DynamicTable and contain\n    the columns 'task_name', 'task_description', 'camera_id', 'and\n    'task_epochs'.\n\n    Parameters\n    ----------\n    task_table : pynwb.core.DynamicTable\n        The table representing task metadata.\n\n    Returns\n    -------\n    bool\n        Whether the DynamicTable conforms to the expected format for\n        loading data into the TaskEpoch table.\n    \"\"\"\n\n    # TODO this could be more strict and check data types, but really it\n    # should be schematized\n    return (\n        Task.check_task_table(task_table)\n        and hasattr(task_table, \"camera_id\")\n        and hasattr(task_table, \"task_epochs\")\n    )\n</code></pre>"}, {"location": "api/common/common_usage/", "title": "common_usage.py", "text": "<p>A schema to store the usage of advanced Spyglass features.</p> <p>Records show usage of features such as cautious delete and fault-permitting insert, which will be used to determine which features are used, how often, and by whom. This will help plan future development of Spyglass.</p>"}, {"location": "api/common/errors/", "title": "errors.py", "text": ""}, {"location": "api/common/populate_all_common/", "title": "populate_all_common.py", "text": ""}, {"location": "api/common/populate_all_common/#src.spyglass.common.populate_all_common.populate_all_common", "title": "<code>populate_all_common(nwb_file_name)</code>", "text": "<p>Insert all common tables for a given NWB file.</p> Source code in <code>src/spyglass/common/populate_all_common.py</code> <pre><code>def populate_all_common(nwb_file_name):\n    \"\"\"Insert all common tables for a given NWB file.\"\"\"\n    from spyglass.spikesorting.imported import ImportedSpikeSorting\n\n    key = [(Nwbfile &amp; f\"nwb_file_name LIKE '{nwb_file_name}'\").proj()]\n    tables = [\n        Session,\n        # NwbfileKachery, # Not used by default\n        ElectrodeGroup,\n        Electrode,\n        Raw,\n        SampleCount,\n        DIOEvents,\n        # SensorData, # Not used by default. Generates large files\n        RawPosition,\n        TaskEpoch,\n        StateScriptFile,\n        VideoFile,\n        PositionSource,\n        RawPosition,\n        ImportedSpikeSorting,\n    ]\n    error_constants = dict(\n        dj_user=dj.config[\"database.user\"],\n        connection_id=dj.conn().connection_id,\n        nwb_file_name=nwb_file_name,\n    )\n\n    for table in tables:\n        logger.info(f\"Populating {table.__name__}...\")\n        try:\n            table.populate(key)\n        except Exception as e:\n            InsertError.insert1(\n                dict(\n                    **error_constants,\n                    table=table.__name__,\n                    error_type=type(e).__name__,\n                    error_message=str(e),\n                    error_raw=str(e),\n                )\n            )\n    query = InsertError &amp; error_constants\n    if query:\n        err_tables = query.fetch(\"table\")\n        logger.error(\n            f\"Errors occurred during population for {nwb_file_name}:\\n\\t\"\n            + f\"Failed tables {err_tables}\\n\\t\"\n            + \"See common_usage.InsertError for more details\"\n        )\n        return query.fetch(\"KEY\")\n</code></pre>"}, {"location": "api/common/signal_processing/", "title": "signal_processing.py", "text": ""}, {"location": "api/common/signal_processing/#src.spyglass.common.signal_processing.hilbert_decomp", "title": "<code>hilbert_decomp(lfp_band_object, sampling_rate=1)</code>", "text": "<p>Generates analytical decomposition of signals in the lfp_band_object</p> <p>NOTE: This function is not currently used in the pipeline.</p> <p>Parameters:</p> Name Type Description Default <code>lfp_band_object</code> <code>ElectricalSeries</code> <p>bandpass filtered LFP</p> required <code>sampling_rate</code> <code>int</code> <p>bandpass filtered LFP sampling rate (defaults to 1; only used for instantaneous frequency)</p> <code>1</code> <p>Returns:</p> Name Type Description <code>envelope</code> <code>ElectricalSeries</code> <p>envelope of the signal</p> Source code in <code>src/spyglass/common/signal_processing.py</code> <pre><code>def hilbert_decomp(lfp_band_object, sampling_rate=1):\n    \"\"\"Generates analytical decomposition of signals in the lfp_band_object\n\n    NOTE: This function is not currently used in the pipeline.\n\n    Parameters\n    ----------\n    lfp_band_object : pynwb.ecephys.ElectricalSeries\n        bandpass filtered LFP\n    sampling_rate : int, optional\n        bandpass filtered LFP sampling rate\n        (defaults to 1; only used for instantaneous frequency)\n\n    Returns\n    -------\n    envelope : pynwb.ecephys.ElectricalSeries\n        envelope of the signal\n    \"\"\"\n    analytical_signal = signal.hilbert(lfp_band_object.data, axis=0)\n\n    eseries_name = \"envelope\"\n    envelope = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=np.abs(analytical_signal),\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n\n    eseries_name = \"phase\"\n    instantaneous_phase = np.unwrap(np.angle(analytical_signal))\n    phase = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=instantaneous_phase,\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n\n    eseries_name = \"frequency\"\n    instantaneous_frequency = (\n        np.diff(instantaneous_phase) / (2.0 * np.pi) * sampling_rate\n    )\n    frequency = pynwb.ecephys.ElectricalSeries(\n        name=eseries_name,\n        data=instantaneous_frequency,\n        electrodes=lfp_band_object.electrodes,\n        timestamps=lfp_band_object.timestamps,\n    )\n    return envelope, phase, frequency\n</code></pre>"}, {"location": "api/common/prepopulate/prepopulate/", "title": "prepopulate.py", "text": ""}, {"location": "api/common/prepopulate/prepopulate/#src.spyglass.common.prepopulate.prepopulate.prepopulate_default", "title": "<code>prepopulate_default()</code>", "text": "<p>Populate the database with default values in SPYGLASS_BASE_DIR/entries.yaml</p> Source code in <code>src/spyglass/common/prepopulate/prepopulate.py</code> <pre><code>def prepopulate_default():\n    \"\"\"\n    Populate the database with default values in SPYGLASS_BASE_DIR/entries.yaml\n    \"\"\"\n    yaml_path = pathlib.Path(base_dir) / \"entries.yaml\"\n    if os.path.exists(yaml_path):\n        populate_from_yaml(yaml_path)\n</code></pre>"}, {"location": "api/common/prepopulate/prepopulate/#src.spyglass.common.prepopulate.prepopulate.populate_from_yaml", "title": "<code>populate_from_yaml(yaml_path)</code>", "text": "<p>Populate the database from specially formatted YAML files.</p> Source code in <code>src/spyglass/common/prepopulate/prepopulate.py</code> <pre><code>def populate_from_yaml(yaml_path: str):\n    \"\"\"Populate the database from specially formatted YAML files.\"\"\"\n    if not os.path.exists(yaml_path):\n        raise ValueError(f\"There is no file found with the path: {yaml_path}\")\n    with open(yaml_path, \"r\") as stream:\n        d = yaml.safe_load(stream)\n\n    for table_name, table_entries in d.items():\n        table_cls = _get_table_cls(table_name)\n        for entry_dict in table_entries:\n            # test whether an entity with the primary key(s) already exists\n            if not issubclass(table_cls, (dj.Manual, dj.Lookup, dj.Part)):\n                raise ValueError(\n                    f\"Prepopulate YAML ('{yaml_path}') contains table \"\n                    + f\"{table_name}' that cannot be prepopulated. Only Manual \"\n                    + \"and Lookup tables can be prepopulated.\"\n                )\n            if hasattr(table_cls, \"fetch_add\"):\n                # if the table has defined a fetch_add method, use that instead\n                # of insert1. this is useful for tables where the primary key\n                # is an ID that auto-increments. first check whether an entry\n                # exists with the same information.\n\n                query = table_cls &amp; entry_dict\n                if not query:\n                    logger.info(\n                        f\"Populate: Populating table {table_cls.__name__} with \"\n                        + f\"data {entry_dict} using fetch_add.\"\n                    )\n                    table_cls.fetch_add(**entry_dict)\n                continue\n\n            primary_key_values = {\n                k: v\n                for k, v in entry_dict.items()\n                if k in table_cls.primary_key\n            }\n            if not primary_key_values:\n                logger.warn(\n                    f\"Populate: No primary key provided in data {entry_dict} \"\n                    + f\"for table {table_cls.__name__}\"\n                )\n                continue\n            if primary_key_values not in table_cls.fetch(\n                *table_cls.primary_key, as_dict=True\n            ):\n                logger.info(\n                    f\"Populate: Populating table {table_cls.__name__} with data\"\n                    + f\" {entry_dict} using insert1.\"\n                )\n                table_cls.insert1(entry_dict)\n            else:\n                logging.warn(\n                    f\"Populate: Entry in {table_cls.__name__} with primary keys\"\n                    + f\" {primary_key_values} already exists.\"\n                )\n</code></pre>"}, {"location": "api/data_import/insert_sessions/", "title": "insert_sessions.py", "text": ""}, {"location": "api/data_import/insert_sessions/#src.spyglass.data_import.insert_sessions.insert_sessions", "title": "<code>insert_sessions(nwb_file_names)</code>", "text": "<p>Populate the dj database with new sessions.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_names</code> <code>str or List of str</code> <p>File names in raw directory ($SPYGLASS_RAW_DIR) pointing to existing .nwb files. Each file represents a session. Also accepts strings with glob wildcards (e.g., *) so long as the wildcard specifies exactly one file.</p> required Source code in <code>src/spyglass/data_import/insert_sessions.py</code> <pre><code>def insert_sessions(nwb_file_names: Union[str, List[str]]):\n    \"\"\"\n    Populate the dj database with new sessions.\n\n    Parameters\n    ----------\n    nwb_file_names : str or List of str\n        File names in raw directory ($SPYGLASS_RAW_DIR) pointing to\n        existing .nwb files. Each file represents a session. Also accepts\n        strings with glob wildcards (e.g., *) so long as the wildcard specifies\n        exactly one file.\n    \"\"\"\n\n    if not isinstance(nwb_file_names, list):\n        nwb_file_names = [nwb_file_names]\n\n    for nwb_file_name in nwb_file_names:\n        if \"/\" in nwb_file_name:\n            nwb_file_name = nwb_file_name.split(\"/\")[-1]\n\n        nwb_file_abs_path = Path(\n            Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n        )\n\n        if not nwb_file_abs_path.exists():\n            possible_matches = sorted(Path(raw_dir).glob(f\"*{nwb_file_name}*\"))\n\n            if len(possible_matches) == 1:\n                nwb_file_abs_path = possible_matches[0]\n                nwb_file_name = nwb_file_abs_path.name\n\n            else:\n                raise FileNotFoundError(\n                    f\"File not found: {nwb_file_abs_path}\\n\\t\"\n                    + f\"{len(possible_matches)} possible matches:\"\n                    + f\"{possible_matches}\"\n                )\n\n        # file name for the copied raw data\n        out_nwb_file_name = get_nwb_copy_filename(nwb_file_abs_path.name)\n\n        # Check whether the file already exists in the Nwbfile table\n        if len(Nwbfile() &amp; {\"nwb_file_name\": out_nwb_file_name}):\n            warnings.warn(\n                f\"Cannot insert data from {nwb_file_name}: {out_nwb_file_name}\"\n                + \" is already in Nwbfile table.\"\n            )\n            continue\n\n        # Make a copy of the NWB file that ends with '_'.\n        # This has everything except the raw data but has a link to\n        # the raw data in the original file\n        copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)\n        Nwbfile().insert_from_relative_file_name(out_nwb_file_name)\n        populate_all_common(out_nwb_file_name)\n</code></pre>"}, {"location": "api/data_import/insert_sessions/#src.spyglass.data_import.insert_sessions.copy_nwb_link_raw_ephys", "title": "<code>copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)</code>", "text": "<p>Copies an NWB file with a link to raw ephys data.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file to be copied.</p> required <code>out_nwb_file_name</code> <code>str</code> <p>The name of the new NWB file with the link to raw ephys data.</p> required <p>Returns:</p> Type Description <code>str</code> <p>The absolute path of the new NWB file.</p> Source code in <code>src/spyglass/data_import/insert_sessions.py</code> <pre><code>def copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name):\n    \"\"\"Copies an NWB file with a link to raw ephys data.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the NWB file to be copied.\n    out_nwb_file_name : str\n        The name of the new NWB file with the link to raw ephys data.\n\n    Returns\n    -------\n    str\n        The absolute path of the new NWB file.\n    \"\"\"\n    logger.info(\n        f\"Creating a copy of NWB file {nwb_file_name} \"\n        + f\"with link to raw ephys data: {out_nwb_file_name}\"\n    )\n\n    nwb_file_abs_path = Nwbfile.get_abs_path(nwb_file_name, new_file=True)\n\n    if not os.path.exists(nwb_file_abs_path):\n        raise FileNotFoundError(f\"Could not find raw file: {nwb_file_abs_path}\")\n\n    out_nwb_file_abs_path = Nwbfile.get_abs_path(\n        out_nwb_file_name, new_file=True\n    )\n\n    if os.path.exists(out_nwb_file_abs_path):\n        if debug_mode:\n            return out_nwb_file_abs_path\n        logger.warning(\n            f\"Output file {out_nwb_file_abs_path} exists and will be \"\n            + \"overwritten.\"\n        )\n\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abs_path, mode=\"r\", load_namespaces=True\n    ) as input_io:\n        nwbf = input_io.read()\n\n        # pop off acquisition electricalseries\n        eseries_list = get_raw_eseries(nwbf)\n        for eseries in eseries_list:\n            nwbf.acquisition.pop(eseries.name)\n\n        # pop off analog processing module\n        analog_processing = nwbf.processing.get(\"analog\")\n        if analog_processing:\n            nwbf.processing.pop(\"analog\")\n\n        # export the new NWB file\n        with pynwb.NWBHDF5IO(\n            path=out_nwb_file_abs_path, mode=\"w\", manager=input_io.manager\n        ) as export_io:\n            export_io.export(input_io, nwbf)\n\n    # add link from new file back to raw ephys data in raw data file using fresh build manager and container cache\n    # where the acquisition electricalseries objects have not been removed\n    with pynwb.NWBHDF5IO(\n        path=nwb_file_abs_path, mode=\"r\", load_namespaces=True\n    ) as input_io:\n        nwbf_raw = input_io.read()\n        eseries_list = get_raw_eseries(nwbf_raw)\n        analog_processing = nwbf_raw.processing.get(\"analog\")\n\n        with pynwb.NWBHDF5IO(\n            path=out_nwb_file_abs_path, mode=\"a\", manager=input_io.manager\n        ) as export_io:\n            nwbf_export = export_io.read()\n\n            # add link to raw ephys ElectricalSeries in raw data file\n            for eseries in eseries_list:\n                nwbf_export.add_acquisition(eseries)\n\n            # add link to processing module in raw data file\n            if analog_processing:\n                nwbf_export.add_processing_module(analog_processing)\n\n            nwbf_export.set_modified()\n            export_io.write(nwbf_export)\n\n    # change the permissions to only allow owner to write\n    permissions = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH\n    os.chmod(out_nwb_file_abs_path, permissions)\n\n    return out_nwb_file_abs_path\n</code></pre>"}, {"location": "api/decoding/decoding_merge/", "title": "decoding_merge.py", "text": ""}, {"location": "api/decoding/decoding_merge/#src.spyglass.decoding.decoding_merge.DecodingOutput", "title": "<code>DecodingOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/decoding/decoding_merge.py</code> <pre><code>@schema\nclass DecodingOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class ClusterlessDecodingV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ClusterlessDecodingV1\n        \"\"\"\n\n    class SortedSpikesDecodingV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; SortedSpikesDecodingV1\n        \"\"\"\n\n    def cleanup(self, dry_run=False):\n        \"\"\"Remove any decoding outputs that are not in the merge table\"\"\"\n        if dry_run:\n            logger.info(\"Dry run, not removing any files\")\n        else:\n            logger.info(\"Cleaning up decoding outputs\")\n        table_results_paths = list(\n            chain(\n                *[\n                    part_parent_table.fetch(\"results_path\").tolist()\n                    for part_parent_table in self.merge_get_parent(\n                        multi_source=True\n                    )\n                ]\n            )\n        )\n        for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.nc\"):\n            if str(path) not in table_results_paths:\n                logger.info(f\"Removing {path}\")\n                if not dry_run:\n                    try:\n                        path.unlink(missing_ok=True)  # Ignore FileNotFoundError\n                    except PermissionError:\n                        logger.warning(f\"Unable to remove {path}, skipping\")\n\n        table_model_paths = list(\n            chain(\n                *[\n                    part_parent_table.fetch(\"classifier_path\").tolist()\n                    for part_parent_table in self.merge_get_parent(\n                        multi_source=True\n                    )\n                ]\n            )\n        )\n        for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.pkl\"):\n            if str(path) not in table_model_paths:\n                logger.info(f\"Removing {path}\")\n                if not dry_run:\n                    try:\n                        path.unlink()\n                    except (PermissionError, FileNotFoundError):\n                        logger.warning(f\"Unable to remove {path}, skipping\")\n\n    @classmethod\n    def fetch_results(cls, key):\n        return cls().merge_get_parent_class(key).fetch_results()\n\n    @classmethod\n    def fetch_model(cls, key):\n        return cls().merge_get_parent_class(key).fetch_model()\n\n    @classmethod\n    def fetch_environments(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_environments(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_position_info(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_position_info(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_linear_position_info(cls, key):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_linear_position_info(decoding_selection_key)\n        )\n\n    @classmethod\n    def fetch_spike_data(cls, key, filter_by_interval=True):\n        decoding_selection_key = cls.merge_get_parent(key).fetch1(\"KEY\")\n        return (\n            cls()\n            .merge_get_parent_class(key)\n            .fetch_linear_position_info(\n                decoding_selection_key, filter_by_interval=filter_by_interval\n            )\n        )\n\n    @classmethod\n    def create_decoding_view(cls, key, head_direction_name=\"head_orientation\"):\n        results = cls.fetch_results(key)\n        posterior = (\n            results.squeeze()\n            .acausal_posterior.unstack(\"state_bins\")\n            .drop_sel(state=[\"Local\", \"No-Spike\"], errors=\"ignore\")\n            .sum(\"state\")\n        )\n        posterior /= posterior.sum(\"position\")\n        env = cls.fetch_environments(key)[0]\n\n        if \"x_position\" in results.coords:\n            position_info, position_variable_names = cls.fetch_position_info(\n                key\n            )\n            # Not 1D\n            bin_size = (\n                np.nanmedian(np.diff(np.unique(results.x_position.values))),\n                np.nanmedian(np.diff(np.unique(results.y_position.values))),\n            )\n            return create_2D_decode_view(\n                position_time=position_info.index,\n                position=position_info[position_variable_names],\n                interior_place_bin_centers=env.place_bin_centers_[\n                    env.is_track_interior_.ravel(order=\"C\")\n                ],\n                place_bin_size=bin_size,\n                posterior=posterior,\n                head_dir=position_info[head_direction_name],\n            )\n        else:\n            (\n                position_info,\n                position_variable_names,\n            ) = cls.fetch_linear_position_info(key)\n            return create_1D_decode_view(\n                posterior=posterior,\n                linear_position=position_info[\"linear_position\"],\n            )\n</code></pre>"}, {"location": "api/decoding/decoding_merge/#src.spyglass.decoding.decoding_merge.DecodingOutput.cleanup", "title": "<code>cleanup(dry_run=False)</code>", "text": "<p>Remove any decoding outputs that are not in the merge table</p> Source code in <code>src/spyglass/decoding/decoding_merge.py</code> <pre><code>def cleanup(self, dry_run=False):\n    \"\"\"Remove any decoding outputs that are not in the merge table\"\"\"\n    if dry_run:\n        logger.info(\"Dry run, not removing any files\")\n    else:\n        logger.info(\"Cleaning up decoding outputs\")\n    table_results_paths = list(\n        chain(\n            *[\n                part_parent_table.fetch(\"results_path\").tolist()\n                for part_parent_table in self.merge_get_parent(\n                    multi_source=True\n                )\n            ]\n        )\n    )\n    for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.nc\"):\n        if str(path) not in table_results_paths:\n            logger.info(f\"Removing {path}\")\n            if not dry_run:\n                try:\n                    path.unlink(missing_ok=True)  # Ignore FileNotFoundError\n                except PermissionError:\n                    logger.warning(f\"Unable to remove {path}, skipping\")\n\n    table_model_paths = list(\n        chain(\n            *[\n                part_parent_table.fetch(\"classifier_path\").tolist()\n                for part_parent_table in self.merge_get_parent(\n                    multi_source=True\n                )\n            ]\n        )\n    )\n    for path in Path(config[\"SPYGLASS_ANALYSIS_DIR\"]).glob(\"**/*.pkl\"):\n        if str(path) not in table_model_paths:\n            logger.info(f\"Removing {path}\")\n            if not dry_run:\n                try:\n                    path.unlink()\n                except (PermissionError, FileNotFoundError):\n                    logger.warning(f\"Unable to remove {path}, skipping\")\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/", "title": "clusterless.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from unclustered spikes and spike waveform features. See [1] for details.</p>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters", "title": "<code>MarkParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Defines the type of waveform feature computed for a given spike time.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass MarkParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Defines the type of waveform feature computed for a given spike time.\"\"\"\n\n    definition = \"\"\"\n    mark_param_name : varchar(32) # a name for this set of parameters\n    ---\n    # the type of mark. Currently only 'amplitude' is supported\n    mark_type = 'amplitude':  varchar(40)\n    mark_param_dict: BLOB # dict of parameters for the mark extraction function\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\n\n        Examples\n        --------\n        {'peak_sign': 'neg', 'threshold' : 100}\n        corresponds to negative going waveforms of at least 100 uV size\n        \"\"\"\n        default_dict = {}\n        self.insert1(\n            {\"mark_param_name\": \"default\", \"mark_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n\n    @staticmethod\n    def supported_mark_type(mark_type):\n        \"\"\"Checks whether the requested mark type is supported.\n\n        Currently only 'amplitude\" is supported.\n\n        Parameters\n        ----------\n        mark_type : str\n\n        \"\"\"\n        supported_types = [\"amplitude\"]\n        return mark_type in supported_types\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> <p>Examples:</p> <p>{'peak_sign': 'neg', 'threshold' : 100} corresponds to negative going waveforms of at least 100 uV size</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\n\n    Examples\n    --------\n    {'peak_sign': 'neg', 'threshold' : 100}\n    corresponds to negative going waveforms of at least 100 uV size\n    \"\"\"\n    default_dict = {}\n    self.insert1(\n        {\"mark_param_name\": \"default\", \"mark_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.MarkParameters.supported_mark_type", "title": "<code>supported_mark_type(mark_type)</code>  <code>staticmethod</code>", "text": "<p>Checks whether the requested mark type is supported.</p> <p>Currently only 'amplitude\" is supported.</p> <p>Parameters:</p> Name Type Description Default <code>mark_type</code> <code>str</code> required Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef supported_mark_type(mark_type):\n    \"\"\"Checks whether the requested mark type is supported.\n\n    Currently only 'amplitude\" is supported.\n\n    Parameters\n    ----------\n    mark_type : str\n\n    \"\"\"\n    supported_types = [\"amplitude\"]\n    return mark_type in supported_types\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarks", "title": "<code>UnitMarks</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Compute spike waveform features for each spike time.</p> <p>For each spike time, compute a spike waveform feature associated with that spike. Used for clusterless decoding.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarks(SpyglassMixin, dj.Computed):\n    \"\"\"Compute spike waveform features for each spike time.\n\n    For each spike time, compute a spike waveform feature associated with that\n    spike. Used for clusterless decoding.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarkParameters\n    ---\n    -&gt; AnalysisNwbfile\n    marks_object_id: varchar(40) # the NWB object that stores the marks\n    \"\"\"\n\n    def make(self, key):\n        # get the list of mark parameters\n        mark_param = (MarkParameters &amp; key).fetch1()\n\n        # check that the mark type is supported\n        if not MarkParameters().supported_mark_type(mark_param[\"mark_type\"]):\n            Warning(\n                f'Mark type {mark_param[\"mark_type\"]} not supported; skipping'\n            )\n            return\n\n        # retrieve the units from the NWB file\n        nwb_units = (CuratedSpikeSorting() &amp; key).fetch_nwb()[0][\"units\"]\n\n        recording = Curation.get_recording(key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n        sorting = Curation.get_curated_sorting(key)\n        waveform_extractor_name = (\n            f'{key[\"nwb_file_name\"]}_{str(uuid.uuid4())[0:8]}_'\n            f'{key[\"curation_id\"]}_clusterless_waveforms'\n        )\n        waveform_extractor_path = str(\n            Path(waveforms_dir) / Path(waveform_extractor_name)\n        )\n        if os.path.exists(waveform_extractor_path):\n            shutil.rmtree(waveform_extractor_path)\n\n        WAVEFORM_PARAMS = {\n            \"ms_before\": 0.5,\n            \"ms_after\": 0.5,\n            \"max_spikes_per_unit\": None,\n            \"n_jobs\": 5,\n            \"total_memory\": \"5G\",\n        }\n        waveform_extractor = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveform_extractor_path,\n            **WAVEFORM_PARAMS,\n        )\n\n        if mark_param[\"mark_type\"] == \"amplitude\":\n            sorter = (CuratedSpikeSorting() &amp; key).fetch1(\"sorter\")\n            if sorter == \"clusterless_thresholder\":\n                estimate_peak_time = False\n            else:\n                estimate_peak_time = True\n\n            peak_sign = mark_param[\"mark_param_dict\"].get(\"peak_sign\")\n\n            marks = np.concatenate(\n                [\n                    UnitMarks._get_peak_amplitude(\n                        waveform=waveform_extractor.get_waveforms(unit_id),\n                        peak_sign=peak_sign,\n                        estimate_peak_time=estimate_peak_time,\n                    )\n                    for unit_id in nwb_units.index\n                ],\n                axis=0,\n            )\n\n            timestamps = np.concatenate(np.asarray(nwb_units[\"spike_times\"]))\n            sorted_timestamp_ind = np.argsort(timestamps)\n            marks = marks[sorted_timestamp_ind]\n            timestamps = timestamps[sorted_timestamp_ind]\n\n        if \"threshold\" in mark_param[\"mark_param_dict\"]:\n            timestamps, marks = UnitMarks._threshold(\n                timestamps, marks, mark_param[\"mark_param_dict\"]\n            )\n\n        # create a new AnalysisNwbfile and a timeseries for the marks and save\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        nwb_object = pynwb.TimeSeries(\n            name=\"marks\",\n            data=marks,\n            unit=\"uV\",\n            timestamps=timestamps,\n            description=\"spike features for clusterless decoding\",\n        )\n        key[\"marks_object_id\"] = AnalysisNwbfile().add_nwb_object(\n            key[\"analysis_file_name\"], nwb_object\n        )\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], key[\"analysis_file_name\"])\n        self.insert1(key)\n\n    def fetch1_dataframe(self) -&gt; pd.DataFrame:\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n        return [self._convert_to_dataframe(data) for data in self.fetch_nwb()]\n\n    @staticmethod\n    def _convert_to_dataframe(nwb_data) -&gt; pd.DataFrame:\n        \"\"\"Converts the marks from an NWB object to a pandas dataframe\"\"\"\n        n_marks = nwb_data[\"marks\"].data.shape[1]\n        columns = [f\"amplitude_{ind:04d}\" for ind in range(n_marks)]\n        return pd.DataFrame(\n            nwb_data[\"marks\"].data,\n            index=pd.Index(nwb_data[\"marks\"].timestamps, name=\"time\"),\n            columns=columns,\n        )\n\n    @staticmethod\n    def _get_peak_amplitude(\n        waveform: np.array,\n        peak_sign: str = \"neg\",\n        estimate_peak_time: bool = False,\n    ) -&gt; np.array:\n        \"\"\"Returns the amplitudes of all channels at the time of the peak.\n\n        Amplitude across channels.\n\n        Parameters\n        ----------\n        waveform : np.array\n            array-like, shape (n_spikes, n_time, n_channels)\n        peak_sign : str, optional\n            One of 'pos', 'neg', 'both'. Direction of the peak in the waveform\n        estimate_peak_time : bool, optional\n            Find the peak times for each spike because some spikesorters do not\n            align the spike time (at index n_time // 2) to the peak\n\n        Returns\n        -------\n        peak_amplitudes : np.array\n            array-like, shape (n_spikes, n_channels)\n\n        \"\"\"\n        if estimate_peak_time:\n            if peak_sign == \"neg\":\n                peak_inds = np.argmin(np.min(waveform, axis=2), axis=1)\n            elif peak_sign == \"pos\":\n                peak_inds = np.argmax(np.max(waveform, axis=2), axis=1)\n            elif peak_sign == \"both\":\n                peak_inds = np.argmax(np.max(np.abs(waveform), axis=2), axis=1)\n\n            # Get mode of peaks to find the peak time\n            values, counts = np.unique(peak_inds, return_counts=True)\n            spike_peak_ind = values[counts.argmax()]\n        else:\n            spike_peak_ind = waveform.shape[1] // 2\n\n        return waveform[:, spike_peak_ind]\n\n    @staticmethod\n    def _threshold(\n        timestamps: np.array, marks: np.array, mark_param_dict: dict\n    ):\n        \"\"\"Filter the marks by an amplitude threshold\n\n        Parameters\n        ----------\n        timestamps : np.array\n            array-like, shape (n_time,)\n        marks : np.array\n            array-like, shape (n_time, n_channels)\n        mark_param_dict : dict\n\n        Returns\n        -------\n        filtered_timestamps : np.array\n            array-like, shape (n_filtered_time,)\n        filtered_marks : np.array\n            array-like, shape (n_filtered_time, n_channels)\n\n        \"\"\"\n        if mark_param_dict[\"peak_sign\"] == \"neg\":\n            include = np.min(marks, axis=1) &lt;= -1 * mark_param_dict[\"threshold\"]\n        elif mark_param_dict[\"peak_sign\"] == \"pos\":\n            include = np.max(marks, axis=1) &gt;= mark_param_dict[\"threshold\"]\n        elif mark_param_dict[\"peak_sign\"] == \"both\":\n            include = (\n                np.max(np.abs(marks), axis=1) &gt;= mark_param_dict[\"threshold\"]\n            )\n        return timestamps[include], marks[include]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarks.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1_dataframe(self) -&gt; pd.DataFrame:\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicatorSelection", "title": "<code>UnitMarksIndicatorSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Pairing of a UnitMarksIndicator with a time interval and sampling rate</p> <p>Bins the spike times and associated spike waveform features for a given time interval into regular time bins determined by the sampling rate.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarksIndicatorSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Pairing of a UnitMarksIndicator with a time interval and sampling rate\n\n    Bins the spike times and associated spike waveform features for a given\n    time interval into regular time bins determined by the sampling rate.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarks\n    -&gt; IntervalList\n    sampling_rate=500 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator", "title": "<code>UnitMarksIndicator</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Bins spike times and waveforms into regular time bins.</p> <p>Bins the spike times and associated spike waveform features into regular time bins according to the sampling rate. Features that fall into the same time bin are averaged.</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass UnitMarksIndicator(SpyglassMixin, dj.Computed):\n    \"\"\"Bins spike times and waveforms into regular time bins.\n\n    Bins the spike times and associated spike waveform features into regular\n    time bins according to the sampling rate. Features that fall into the same\n    time bin are averaged.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitMarks\n    -&gt; UnitMarksIndicatorSelection\n    ---\n    -&gt; AnalysisNwbfile\n    marks_indicator_object_id: varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        # TODO: intersection of sort interval and interval list\n        interval_times = (IntervalList &amp; key).fetch1(\"valid_times\")\n\n        sampling_rate = (UnitMarksIndicatorSelection &amp; key).fetch(\n            \"sampling_rate\"\n        )\n\n        marks_df = (UnitMarks &amp; key).fetch1_dataframe()\n\n        time = self.get_time_bins_from_interval(interval_times, sampling_rate)\n\n        # Bin marks into time bins. No spike bins will have NaN\n        marks_df = marks_df.loc[time.min() : time.max()]\n        time_index = np.digitize(marks_df.index, time[1:-1])\n        marks_indicator_df = (\n            marks_df.groupby(time[time_index])\n            .mean()\n            .reindex(index=pd.Index(time, name=\"time\"))\n        )\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(\n            key[\"nwb_file_name\"]\n        )\n\n        key[\"marks_indicator_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=marks_indicator_df.reset_index(),\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    @staticmethod\n    def get_time_bins_from_interval(\n        interval_times: np.array, sampling_rate: int\n    ) -&gt; np.array:\n        \"\"\"Picks the superset of the interval\"\"\"\n        start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n        n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n        return np.linspace(start_time, end_time, n_samples)\n\n    @staticmethod\n    def plot_all_marks(\n        marks_indicators: xr.DataArray,\n        plot_size: int = 5,\n        marker_size: int = 10,\n        plot_limit: int = None,\n    ):\n        \"\"\"Plot all marks for all electrodes.\n\n        Plots 2D slices of each of the spike features against each other\n        for all electrodes.\n\n        Parameters\n        ----------\n        marks_indicators : xr.DataArray, shape (n_time, n_electrodes, n_features)\n            Spike times and associated spike waveform features binned into\n        plot_size : int, optional\n            Default 5. Matplotlib figure size for each mark.\n        marker_size : int, optional\n            Default 10. Marker size\n        plot_limit : int, optional\n            Default None. Limits to first N electrodes.\n        \"\"\"\n        if not plot_limit:\n            plot_limit = len(marks_indicators.electrodes)\n\n        for electrode_ind in marks_indicators.electrodes[:plot_limit]:\n            marks = (\n                marks_indicators.sel(electrodes=electrode_ind)\n                .dropna(\"time\", how=\"all\")\n                .dropna(\"marks\")\n            )\n            n_features = len(marks.marks)\n            fig, axes = plt.subplots(\n                n_features,\n                n_features,\n                constrained_layout=True,\n                sharex=True,\n                sharey=True,\n                figsize=(plot_size * n_features, plot_size * n_features),\n            )\n            for ax_ind1, feature1 in enumerate(marks.marks):\n                for ax_ind2, feature2 in enumerate(marks.marks):\n                    try:\n                        axes[ax_ind1, ax_ind2].scatter(\n                            marks.sel(marks=feature1),\n                            marks.sel(marks=feature2),\n                            s=marker_size,\n                        )\n                    except TypeError:\n                        axes.scatter(\n                            marks.sel(marks=feature1),\n                            marks.sel(marks=feature2),\n                            s=marker_size,\n                        )\n\n    def fetch1_dataframe(self) -&gt; pd.DataFrame:\n        \"\"\"Convenience function for returning the first dataframe\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n        \"\"\"Fetches the marks indicators as a list of pandas dataframes\"\"\"\n        return [\n            data[\"marks_indicator\"].set_index(\"time\")\n            for data in self.fetch_nwb()\n        ]\n\n    def fetch_xarray(self):\n        \"\"\"Fetches the marks indicators as an xarray DataArray\"\"\"\n        # sort_group_electrodes = (\n        #     SortGroup.SortGroupElectrode() &amp;\n        #     pd.DataFrame(self).to_dict('records'))\n        # brain_region = (sort_group_electrodes * Electrode *\n        #                 BrainRegion).fetch('region_name')\n\n        marks_indicators = (\n            xr.concat(\n                [\n                    df.to_xarray().to_array(\"marks\")\n                    for df in self.fetch_dataframe()\n                ],\n                dim=\"electrodes\",\n            )\n            .transpose(\"time\", \"marks\", \"electrodes\")\n            .assign_coords({\"electrodes\": self.fetch(\"sort_group_id\")})\n            .sortby([\"electrodes\", \"marks\"])\n        )\n\n        # hacky way to keep the marks in order\n        def reformat_name(name):\n            mark_type, number = name.split(\"_\")\n            return f\"{mark_type}_{int(number):04d}\"\n\n        new_mark_names = [\n            reformat_name(name) for name in marks_indicators.marks.values\n        ]\n\n        return marks_indicators.assign_coords({\"marks\": new_mark_names}).sortby(\n            [\"electrodes\", \"marks\"]\n        )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.get_time_bins_from_interval", "title": "<code>get_time_bins_from_interval(interval_times, sampling_rate)</code>  <code>staticmethod</code>", "text": "<p>Picks the superset of the interval</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef get_time_bins_from_interval(\n    interval_times: np.array, sampling_rate: int\n) -&gt; np.array:\n    \"\"\"Picks the superset of the interval\"\"\"\n    start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n    return np.linspace(start_time, end_time, n_samples)\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.plot_all_marks", "title": "<code>plot_all_marks(marks_indicators, plot_size=5, marker_size=10, plot_limit=None)</code>  <code>staticmethod</code>", "text": "<p>Plot all marks for all electrodes.</p> <p>Plots 2D slices of each of the spike features against each other for all electrodes.</p> <p>Parameters:</p> Name Type Description Default <code>marks_indicators</code> <code>(DataArray, shape(n_time, n_electrodes, n_features))</code> <p>Spike times and associated spike waveform features binned into</p> required <code>plot_size</code> <code>int</code> <p>Default 5. Matplotlib figure size for each mark.</p> <code>5</code> <code>marker_size</code> <code>int</code> <p>Default 10. Marker size</p> <code>10</code> <code>plot_limit</code> <code>int</code> <p>Default None. Limits to first N electrodes.</p> <code>None</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@staticmethod\ndef plot_all_marks(\n    marks_indicators: xr.DataArray,\n    plot_size: int = 5,\n    marker_size: int = 10,\n    plot_limit: int = None,\n):\n    \"\"\"Plot all marks for all electrodes.\n\n    Plots 2D slices of each of the spike features against each other\n    for all electrodes.\n\n    Parameters\n    ----------\n    marks_indicators : xr.DataArray, shape (n_time, n_electrodes, n_features)\n        Spike times and associated spike waveform features binned into\n    plot_size : int, optional\n        Default 5. Matplotlib figure size for each mark.\n    marker_size : int, optional\n        Default 10. Marker size\n    plot_limit : int, optional\n        Default None. Limits to first N electrodes.\n    \"\"\"\n    if not plot_limit:\n        plot_limit = len(marks_indicators.electrodes)\n\n    for electrode_ind in marks_indicators.electrodes[:plot_limit]:\n        marks = (\n            marks_indicators.sel(electrodes=electrode_ind)\n            .dropna(\"time\", how=\"all\")\n            .dropna(\"marks\")\n        )\n        n_features = len(marks.marks)\n        fig, axes = plt.subplots(\n            n_features,\n            n_features,\n            constrained_layout=True,\n            sharex=True,\n            sharey=True,\n            figsize=(plot_size * n_features, plot_size * n_features),\n        )\n        for ax_ind1, feature1 in enumerate(marks.marks):\n            for ax_ind2, feature2 in enumerate(marks.marks):\n                try:\n                    axes[ax_ind1, ax_ind2].scatter(\n                        marks.sel(marks=feature1),\n                        marks.sel(marks=feature2),\n                        s=marker_size,\n                    )\n                except TypeError:\n                    axes.scatter(\n                        marks.sel(marks=feature1),\n                        marks.sel(marks=feature2),\n                        s=marker_size,\n                    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the first dataframe</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1_dataframe(self) -&gt; pd.DataFrame:\n    \"\"\"Convenience function for returning the first dataframe\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch_dataframe", "title": "<code>fetch_dataframe()</code>", "text": "<p>Fetches the marks indicators as a list of pandas dataframes</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch_dataframe(self) -&gt; list[pd.DataFrame]:\n    \"\"\"Fetches the marks indicators as a list of pandas dataframes\"\"\"\n    return [\n        data[\"marks_indicator\"].set_index(\"time\")\n        for data in self.fetch_nwb()\n    ]\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.UnitMarksIndicator.fetch_xarray", "title": "<code>fetch_xarray()</code>", "text": "<p>Fetches the marks indicators as an xarray DataArray</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch_xarray(self):\n    \"\"\"Fetches the marks indicators as an xarray DataArray\"\"\"\n    # sort_group_electrodes = (\n    #     SortGroup.SortGroupElectrode() &amp;\n    #     pd.DataFrame(self).to_dict('records'))\n    # brain_region = (sort_group_electrodes * Electrode *\n    #                 BrainRegion).fetch('region_name')\n\n    marks_indicators = (\n        xr.concat(\n            [\n                df.to_xarray().to_array(\"marks\")\n                for df in self.fetch_dataframe()\n            ],\n            dim=\"electrodes\",\n        )\n        .transpose(\"time\", \"marks\", \"electrodes\")\n        .assign_coords({\"electrodes\": self.fetch(\"sort_group_id\")})\n        .sortby([\"electrodes\", \"marks\"])\n    )\n\n    # hacky way to keep the marks in order\n    def reformat_name(name):\n        mark_type, number = name.split(\"_\")\n        return f\"{mark_type}_{int(number):04d}\"\n\n    new_mark_names = [\n        reformat_name(name) for name in marks_indicators.marks.values\n    ]\n\n    return marks_indicators.assign_coords({\"marks\": new_mark_names}).sortby(\n        [\"electrodes\", \"marks\"]\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.make_default_decoding_parameters_cpu", "title": "<code>make_default_decoding_parameters_cpu()</code>", "text": "<p>Default parameters for decoding on CPU</p> <p>Returns:</p> Name Type Description <code>classifier_parameters</code> <code>dict</code> <code>fit_parameters</code> <code>dict</code> <code>predict_parameters</code> <code>dict</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def make_default_decoding_parameters_cpu() -&gt; tuple[dict, dict, dict]:\n    \"\"\"Default parameters for decoding on CPU\n\n    Returns\n    -------\n    classifier_parameters : dict\n    fit_parameters : dict\n    predict_parameters : dict\n    \"\"\"\n\n    classifier_parameters = dict(\n        environments=[_DEFAULT_ENVIRONMENT],\n        observation_models=None,\n        continuous_transition_types=_DEFAULT_CONTINUOUS_TRANSITIONS,\n        discrete_transition_type=DiagonalDiscrete(0.98),\n        initial_conditions_type=UniformInitialConditions(),\n        infer_track_interior=True,\n        clusterless_algorithm=\"multiunit_likelihood_integer\",\n        clusterless_algorithm_params=_DEFAULT_CLUSTERLESS_MODEL_KWARGS,\n    )\n\n    predict_parameters = {\n        \"is_compute_acausal\": True,\n        \"use_gpu\": False,\n        \"state_names\": [\"Continuous\", \"Fragmented\"],\n    }\n    fit_parameters = dict()\n\n    return classifier_parameters, fit_parameters, predict_parameters\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.make_default_decoding_parameters_gpu", "title": "<code>make_default_decoding_parameters_gpu()</code>", "text": "<p>Default parameters for decoding on GPU</p> <p>Returns:</p> Name Type Description <code>classifier_parameters</code> <code>dict</code> <code>fit_parameters</code> <code>dict</code> <code>predict_parameters</code> <code>dict</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def make_default_decoding_parameters_gpu() -&gt; tuple[dict, dict, dict]:\n    \"\"\"Default parameters for decoding on GPU\n\n    Returns\n    -------\n    classifier_parameters : dict\n    fit_parameters : dict\n    predict_parameters : dict\n    \"\"\"\n\n    classifier_parameters = dict(\n        environments=[_DEFAULT_ENVIRONMENT],\n        observation_models=None,\n        continuous_transition_types=_DEFAULT_CONTINUOUS_TRANSITIONS,\n        discrete_transition_type=DiagonalDiscrete(0.98),\n        initial_conditions_type=UniformInitialConditions(),\n        infer_track_interior=True,\n        clusterless_algorithm=\"multiunit_likelihood_integer_gpu\",\n        clusterless_algorithm_params={\n            \"mark_std\": 24.0,\n            \"position_std\": 6.0,\n        },\n    )\n\n    predict_parameters = {\n        \"is_compute_acausal\": True,\n        \"use_gpu\": True,\n        \"state_names\": [\"Continuous\", \"Fragmented\"],\n    }\n\n    fit_parameters = dict()\n\n    return classifier_parameters, fit_parameters, predict_parameters\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters", "title": "<code>ClusterlessClassifierParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Decodes animal's mental position.</p> <p>Decodes the animal's mental position and some category of interest from unclustered spikes and spike waveform features</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>@schema\nclass ClusterlessClassifierParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Decodes animal's mental position.\n\n    Decodes the animal's mental position and some category of interest\n    from unclustered spikes and spike waveform features\n    \"\"\"\n\n    definition = \"\"\"\n    classifier_param_name : varchar(80) # a name for this set of parameters\n    ---\n    classifier_params :   BLOB    # initialization parameters\n    fit_params :          BLOB    # fit parameters\n    predict_params :      BLOB    # prediction parameters\n    \"\"\"\n\n    def insert_default(self) -&gt; None:\n        \"\"\"Insert the default parameter set\"\"\"\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_cpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_cpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_gpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_gpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n    def insert1(self, key, **kwargs) -&gt; None:\n        \"\"\"Custom insert1 to convert classes to dicts\"\"\"\n        super().insert1(convert_classes_to_dict(key), **kwargs)\n\n    def fetch1(self, *args, **kwargs) -&gt; dict:\n        \"\"\"Custom fetch1 to convert dicts to classes\"\"\"\n        return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert_default(self) -&gt; None:\n    \"\"\"Insert the default parameter set\"\"\"\n    (\n        classifier_parameters,\n        fit_parameters,\n        predict_parameters,\n    ) = make_default_decoding_parameters_cpu()\n    self.insert1(\n        {\n            \"classifier_param_name\": \"default_decoding_cpu\",\n            \"classifier_params\": classifier_parameters,\n            \"fit_params\": fit_parameters,\n            \"predict_params\": predict_parameters,\n        },\n        skip_duplicates=True,\n    )\n\n    (\n        classifier_parameters,\n        fit_parameters,\n        predict_parameters,\n    ) = make_default_decoding_parameters_gpu()\n    self.insert1(\n        {\n            \"classifier_param_name\": \"default_decoding_gpu\",\n            \"classifier_params\": classifier_parameters,\n            \"fit_params\": fit_parameters,\n            \"predict_params\": predict_parameters,\n        },\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.insert1", "title": "<code>insert1(key, **kwargs)</code>", "text": "<p>Custom insert1 to convert classes to dicts</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def insert1(self, key, **kwargs) -&gt; None:\n    \"\"\"Custom insert1 to convert classes to dicts\"\"\"\n    super().insert1(convert_classes_to_dict(key), **kwargs)\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.ClusterlessClassifierParameters.fetch1", "title": "<code>fetch1(*args, **kwargs)</code>", "text": "<p>Custom fetch1 to convert dicts to classes</p> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def fetch1(self, *args, **kwargs) -&gt; dict:\n    \"\"\"Custom fetch1 to convert dicts to classes\"\"\"\n    return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.get_decoding_data_for_epoch", "title": "<code>get_decoding_data_for_epoch(nwb_file_name, interval_list_name, position_info_param_name='default_decoding', additional_mark_keys={})</code>", "text": "<p>Collects necessary data for decoding.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>interval_list_name</code> <code>str</code> required <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> <code>additional_mark_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_columns))</code> <code>marks</code> <code>(DataArray, shape(n_time, n_marks, n_electrodes))</code> <code>valid_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def get_decoding_data_for_epoch(\n    nwb_file_name: str,\n    interval_list_name: str,\n    position_info_param_name: str = \"default_decoding\",\n    additional_mark_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, xr.DataArray, list[slice]]:\n    \"\"\"Collects necessary data for decoding.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    interval_list_name : str\n    position_info_param_name : str, optional\n    additional_mark_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_columns)\n    marks : xr.DataArray, shape (n_time, n_marks, n_electrodes)\n    valid_slices : list[slice]\n    \"\"\"\n\n    valid_ephys_position_times_by_epoch = (\n        get_valid_ephys_position_times_by_epoch(nwb_file_name)\n    )\n    valid_ephys_position_times = valid_ephys_position_times_by_epoch[\n        interval_list_name\n    ]\n    valid_slices = convert_valid_times_to_slice(valid_ephys_position_times)\n    position_interval_name = (\n        convert_epoch_interval_name_to_position_interval_name(\n            {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        )\n    )\n\n    position_info = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": position_interval_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch1_dataframe()\n\n    position_info = pd.concat(\n        [position_info.loc[times] for times in valid_slices]\n    )\n\n    marks = (\n        (\n            UnitMarksIndicator()\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": position_interval_name,\n                **additional_mark_keys,\n            }\n        )\n    ).fetch_xarray()\n\n    marks = xr.concat(\n        [marks.sel(time=times) for times in valid_slices], dim=\"time\"\n    )\n\n    return position_info, marks, valid_slices\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.get_data_for_multiple_epochs", "title": "<code>get_data_for_multiple_epochs(nwb_file_name, epoch_names, position_info_param_name='default_decoding', additional_mark_keys={})</code>", "text": "<p>Collects necessary data for decoding multiple environments</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>epoch_names</code> <code>list[str]</code> required <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> <code>additional_mark_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_columns))</code> <code>marks</code> <code>(DataArray, shape(n_time, n_marks, n_electrodes))</code> <code>valid_slices</code> <code>dict[str, list[slice]]</code> <code>environment_labels</code> <code>(ndarray, shape(n_time))</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def get_data_for_multiple_epochs(\n    nwb_file_name: str,\n    epoch_names: list[str],\n    position_info_param_name=\"default_decoding\",\n    additional_mark_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, xr.DataArray, dict[str, list[slice]], np.ndarray]:\n    \"\"\"Collects necessary data for decoding multiple environments\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    epoch_names : list[str]\n    position_info_param_name : str, optional\n    additional_mark_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_columns)\n    marks : xr.DataArray, shape (n_time, n_marks, n_electrodes)\n    valid_slices : dict[str, list[slice]]\n    environment_labels : np.ndarray, shape (n_time,)\n    \"\"\"\n    data = []\n    environment_labels = []\n\n    for epoch in epoch_names:\n        data.append(\n            get_decoding_data_for_epoch(\n                nwb_file_name,\n                epoch,\n                position_info_param_name=position_info_param_name,\n                additional_mark_keys=additional_mark_keys,\n            )\n        )\n        n_time = data[-1][0].shape[0]\n        environment_labels.append([epoch] * n_time)\n\n    environment_labels = np.concatenate(environment_labels, axis=0)\n    position_info, marks, valid_slices = list(zip(*data))\n    position_info = pd.concat(position_info, axis=0)\n    marks = xr.concat(marks, dim=\"time\")\n    valid_slices = {\n        epoch: valid_slice\n        for epoch, valid_slice in zip(epoch_names, valid_slices)\n    }\n\n    assert position_info.shape[0] == marks.shape[0]\n\n    return position_info, marks, valid_slices, environment_labels\n</code></pre>"}, {"location": "api/decoding/v0/clusterless/#src.spyglass.decoding.v0.clusterless.populate_mark_indicators", "title": "<code>populate_mark_indicators(spikesorting_selection_keys, mark_param_name='default', position_info_param_name='default_decoding')</code>", "text": "<p>Populate mark indicators</p> <p>Populates for all units in a given spike sorting selection.</p> <p>This function is a way to do several pipeline steps at once. It will: 1. Populate the SpikeSortingSelection table 2. Populate the SpikeSorting table 3. Populate the Curation table 4. Populate the CuratedSpikeSortingSelection table 5. Populate UnitMarks 6. Compute UnitMarksIndicator for each position epoch</p> <p>Parameters:</p> Name Type Description Default <code>spikesorting_selection_keys</code> <code>dict</code> required <code>mark_param_name</code> <code>str</code> <code>'default'</code> <code>position_info_param_name</code> <code>str</code> <code>'default_decoding'</code> Source code in <code>src/spyglass/decoding/v0/clusterless.py</code> <pre><code>def populate_mark_indicators(\n    spikesorting_selection_keys: dict,\n    mark_param_name: str = \"default\",\n    position_info_param_name: str = \"default_decoding\",\n):\n    \"\"\"Populate mark indicators\n\n    Populates for all units in a given spike sorting selection.\n\n    This function is a way to do several pipeline steps at once. It will:\n    1. Populate the SpikeSortingSelection table\n    2. Populate the SpikeSorting table\n    3. Populate the Curation table\n    4. Populate the CuratedSpikeSortingSelection table\n    5. Populate UnitMarks\n    6. Compute UnitMarksIndicator for each position epoch\n\n    Parameters\n    ----------\n    spikesorting_selection_keys : dict\n    mark_param_name : str, optional\n    position_info_param_name : str, optional\n    \"\"\"\n    spikesorting_selection_keys = deepcopy(spikesorting_selection_keys)\n    # Populate spike sorting\n    SpikeSortingSelection().insert(\n        spikesorting_selection_keys,\n        skip_duplicates=True,\n    )\n    SpikeSorting.populate(spikesorting_selection_keys)\n\n    # Skip any curation\n    curation_keys = [\n        Curation.insert_curation(key) for key in spikesorting_selection_keys\n    ]\n\n    CuratedSpikeSortingSelection().insert(curation_keys, skip_duplicates=True)\n    CuratedSpikeSorting.populate(CuratedSpikeSortingSelection() &amp; curation_keys)\n\n    # Populate marks\n    mark_parameters_keys = pd.DataFrame(CuratedSpikeSorting &amp; curation_keys)\n    mark_parameters_keys[\"mark_param_name\"] = mark_param_name\n    mark_parameters_keys = mark_parameters_keys.loc[\n        :, UnitMarkParameters.primary_key\n    ].to_dict(\"records\")\n    UnitMarkParameters().insert(mark_parameters_keys, skip_duplicates=True)\n    UnitMarks.populate(UnitMarkParameters &amp; mark_parameters_keys)\n\n    # Compute mark indicators for each position epoch\n    nwb_file_name = spikesorting_selection_keys[0][\"nwb_file_name\"]\n    position_interval_names = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch(\"interval_list_name\")\n\n    for interval_name in tqdm(position_interval_names):\n        position_interval = IntervalList &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": interval_name,\n        }\n\n        marks_selection = (UnitMarks &amp; mark_parameters_keys) * position_interval\n        marks_selection = (\n            pd.DataFrame(marks_selection)\n            .loc[:, marks_selection.primary_key]\n            .to_dict(\"records\")\n        )\n        UnitMarksIndicatorSelection.insert(\n            marks_selection, skip_duplicates=True\n        )\n        UnitMarksIndicator.populate(marks_selection)\n</code></pre>"}, {"location": "api/decoding/v0/core/", "title": "core.py", "text": ""}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_valid_ephys_position_times_from_interval", "title": "<code>get_valid_ephys_position_times_from_interval(interval_list_name, nwb_file_name)</code>", "text": "<p>Finds the intersection of the valid times for the interval list, the valid times for the ephys data, and the valid times for the position data.</p> <p>Parameters:</p> Name Type Description Default <code>interval_list_name</code> <code>str</code> required <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>valid_ephys_position_times</code> <code>(ndarray, shape(n_valid_times, 2))</code> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_valid_ephys_position_times_from_interval(\n    interval_list_name: str, nwb_file_name: str\n) -&gt; np.ndarray:\n    \"\"\"Finds the intersection of the valid times for the interval list, the valid times for the ephys data,\n    and the valid times for the position data.\n\n    Parameters\n    ----------\n    interval_list_name : str\n    nwb_file_name : str\n\n    Returns\n    -------\n    valid_ephys_position_times : np.ndarray, shape (n_valid_times, 2)\n\n    \"\"\"\n    interval_valid_times = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": interval_list_name,\n        }\n    ).fetch1(\"valid_times\")\n\n    position_interval_names = (\n        RawPosition\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n        }\n    ).fetch(\"interval_list_name\")\n    position_interval_names = position_interval_names[\n        np.argsort(\n            [\n                int(name.strip(\"pos valid time\"))\n                for name in position_interval_names\n            ]\n        )\n    ]\n    valid_pos_times = [\n        (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": pos_interval_name,\n            }\n        ).fetch1(\"valid_times\")\n        for pos_interval_name in position_interval_names\n    ]\n\n    valid_ephys_times = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": \"raw data valid times\",\n        }\n    ).fetch1(\"valid_times\")\n\n    return interval_list_intersect(\n        interval_list_intersect(interval_valid_times, valid_ephys_times),\n        np.concatenate(valid_pos_times),\n    )\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_epoch_interval_names", "title": "<code>get_epoch_interval_names(nwb_file_name)</code>", "text": "<p>Find the interval names that are epochs.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>epoch_names</code> <code>list[str]</code> <p>List of interval names that are epochs.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_epoch_interval_names(nwb_file_name: str) -&gt; list[str]:\n    \"\"\"Find the interval names that are epochs.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n\n    Returns\n    -------\n    epoch_names : list[str]\n        List of interval names that are epochs.\n    \"\"\"\n    interval_list = pd.DataFrame(\n        IntervalList() &amp; {\"nwb_file_name\": nwb_file_name}\n    )\n\n    interval_list = interval_list.loc[\n        interval_list.interval_list_name.str.contains(\n            r\"^(?:\\d+)_(?:\\w+)$\", regex=True, na=False\n        )\n    ]\n\n    return interval_list.interval_list_name.tolist()\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.get_valid_ephys_position_times_by_epoch", "title": "<code>get_valid_ephys_position_times_by_epoch(nwb_file_name)</code>", "text": "<p>Get the valid ephys position times for each epoch.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>valid_ephys_position_times_by_epoch</code> <code>dict[str, ndarray]</code> <p>Dictionary of epoch names and valid ephys position times.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def get_valid_ephys_position_times_by_epoch(\n    nwb_file_name: str,\n) -&gt; dict[str, np.ndarray]:\n    \"\"\"Get the valid ephys position times for each epoch.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n\n    Returns\n    -------\n    valid_ephys_position_times_by_epoch : dict[str, np.ndarray]\n        Dictionary of epoch names and valid ephys position times.\n\n    \"\"\"\n    return {\n        epoch: get_valid_ephys_position_times_from_interval(\n            epoch, nwb_file_name\n        )\n        for epoch in get_epoch_interval_names(nwb_file_name)\n    }\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.convert_valid_times_to_slice", "title": "<code>convert_valid_times_to_slice(valid_times)</code>", "text": "<p>Converts the valid times to a list of slices so that arrays can be indexed easily.</p> <p>Parameters:</p> Name Type Description Default <code>valid_times</code> <code>(ndarray, shape(n_valid_times, 2))</code> <p>Start and end times for each valid time.</p> required <p>Returns:</p> Name Type Description <code>valid_time_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def convert_valid_times_to_slice(valid_times: np.ndarray) -&gt; list[slice]:\n    \"\"\"Converts the valid times to a list of slices so that arrays can be indexed easily.\n\n    Parameters\n    ----------\n    valid_times : np.ndarray, shape (n_valid_times, 2)\n        Start and end times for each valid time.\n\n    Returns\n    -------\n    valid_time_slices : list[slice]\n\n    \"\"\"\n    return [slice(times[0], times[1]) for times in valid_times]\n</code></pre>"}, {"location": "api/decoding/v0/core/#src.spyglass.decoding.v0.core.create_model_for_multiple_epochs", "title": "<code>create_model_for_multiple_epochs(epoch_names, env_kwargs)</code>", "text": "<p>Creates the observation model, environment, and continuous transition types for multiple epochs for decoding</p> <p>Parameters:</p> Name Type Description Default <code>epoch_names</code> <code>(list[str], length(n_epochs))</code> required <code>env_kwargs</code> <code>dict</code> <p>Environment keyword arguments.</p> required <p>Returns:</p> Name Type Description <code>observation_models</code> <code>tuple[list[ObservationModel]</code> <p>Observation model for each epoch.</p> <code>environments</code> <code>list[Environment]</code> <p>Environment for each epoch.</p> <code>continuous_transition_types</code> <code>list[list[object]]]</code> <p>Continuous transition types for each epoch.</p> Source code in <code>src/spyglass/decoding/v0/core.py</code> <pre><code>def create_model_for_multiple_epochs(\n    epoch_names: list[str], env_kwargs: dict\n) -&gt; tuple[list[ObservationModel], list[Environment], list[list[object]]]:\n    \"\"\"Creates the observation model, environment, and continuous transition types for multiple epochs for decoding\n\n    Parameters\n    ----------\n    epoch_names : list[str], length (n_epochs)\n    env_kwargs : dict\n        Environment keyword arguments.\n\n    Returns\n    -------\n    observation_models: tuple[list[ObservationModel]\n        Observation model for each epoch.\n    environments : list[Environment]\n        Environment for each epoch.\n    continuous_transition_types : list[list[object]]]\n        Continuous transition types for each epoch.\n\n    \"\"\"\n    observation_models = []\n    environments = []\n    continuous_transition_types = []\n\n    for epoch in epoch_names:\n        observation_models.append(ObservationModel(epoch))\n        environments.append(Environment(epoch, **env_kwargs))\n\n    for epoch1 in epoch_names:\n        continuous_transition_types.append([])\n        for epoch2 in epoch_names:\n            if epoch1 == epoch2:\n                continuous_transition_types[-1].append(\n                    RandomWalk(epoch1, use_diffusion=False)\n                )\n            else:\n                continuous_transition_types[-1].append(Uniform(epoch1, epoch2))\n\n    return observation_models, environments, continuous_transition_types\n</code></pre>"}, {"location": "api/decoding/v0/dj_decoder_conversion/", "title": "dj_decoder_conversion.py", "text": "<p>Converts decoder classes into dictionaries and dictionaries into classes so that datajoint can store them in tables.</p>"}, {"location": "api/decoding/v0/dj_decoder_conversion/#src.spyglass.decoding.v0.dj_decoder_conversion.restore_classes", "title": "<code>restore_classes(params)</code>", "text": "<p>Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes</p> Source code in <code>src/spyglass/decoding/v0/dj_decoder_conversion.py</code> <pre><code>def restore_classes(params: dict) -&gt; dict:\n    \"\"\"Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes\"\"\"\n    continuous_state_transition_types = {\n        \"RandomWalk\": RandomWalk,\n        \"RandomWalkDirection1\": RandomWalkDirection1,\n        \"RandomWalkDirection2\": RandomWalkDirection2,\n        \"Uniform\": Uniform,\n        \"Identity\": Identity,\n    }\n\n    discrete_state_transition_types = {\n        \"DiagonalDiscrete\": DiagonalDiscrete,\n        \"UniformDiscrete\": UniformDiscrete,\n        \"RandomDiscrete\": RandomDiscrete,\n        \"UserDefinedDiscrete\": UserDefinedDiscrete,\n    }\n\n    initial_conditions_types = {\n        \"UniformInitialConditions\": UniformInitialConditions,\n        \"UniformOneEnvironmentInitialConditions\": UniformOneEnvironmentInitialConditions,\n    }\n\n    params[\"classifier_params\"][\"continuous_transition_types\"] = [\n        [\n            _convert_dict_to_class(st, continuous_state_transition_types)\n            for st in sts\n        ]\n        for sts in params[\"classifier_params\"][\"continuous_transition_types\"]\n    ]\n    params[\"classifier_params\"][\"environments\"] = [\n        _convert_env_dict(env_params)\n        for env_params in params[\"classifier_params\"][\"environments\"]\n    ]\n    params[\"classifier_params\"][\"discrete_transition_type\"] = (\n        _convert_dict_to_class(\n            params[\"classifier_params\"][\"discrete_transition_type\"],\n            discrete_state_transition_types,\n        )\n    )\n    params[\"classifier_params\"][\"initial_conditions_type\"] = (\n        _convert_dict_to_class(\n            params[\"classifier_params\"][\"initial_conditions_type\"],\n            initial_conditions_types,\n        )\n    )\n\n    if params[\"classifier_params\"].get(\"observation_models\"):\n        params[\"classifier_params\"][\"observation_models\"] = [\n            ObservationModel(obs)\n            for obs in params[\"classifier_params\"][\"observation_models\"]\n        ]\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v0/dj_decoder_conversion/#src.spyglass.decoding.v0.dj_decoder_conversion.convert_classes_to_dict", "title": "<code>convert_classes_to_dict(key)</code>", "text": "<p>Converts the classifier parameters into a dictionary so that datajoint can store it.</p> Source code in <code>src/spyglass/decoding/v0/dj_decoder_conversion.py</code> <pre><code>def convert_classes_to_dict(key: dict) -&gt; dict:\n    \"\"\"Converts the classifier parameters into a dictionary so that datajoint can store it.\"\"\"\n    try:\n        key[\"classifier_params\"][\"environments\"] = [\n            _convert_environment_to_dict(env)\n            for env in key[\"classifier_params\"][\"environments\"]\n        ]\n    except TypeError:\n        key[\"classifier_params\"][\"environments\"] = [\n            _convert_environment_to_dict(\n                key[\"classifier_params\"][\"environments\"]\n            )\n        ]\n    key[\"classifier_params\"][\"continuous_transition_types\"] = (\n        _convert_transitions_to_dict(\n            key[\"classifier_params\"][\"continuous_transition_types\"]\n        )\n    )\n    key[\"classifier_params\"][\"discrete_transition_type\"] = _to_dict(\n        key[\"classifier_params\"][\"discrete_transition_type\"]\n    )\n    key[\"classifier_params\"][\"initial_conditions_type\"] = _to_dict(\n        key[\"classifier_params\"][\"initial_conditions_type\"]\n    )\n\n    if key[\"classifier_params\"][\"observation_models\"] is not None:\n        key[\"classifier_params\"][\"observation_models\"] = [\n            vars(obs) for obs in key[\"classifier_params\"][\"observation_models\"]\n        ]\n\n    try:\n        key[\"classifier_params\"][\"clusterless_algorithm_params\"] = (\n            _convert_algorithm_params(\n                key[\"classifier_params\"][\"clusterless_algorithm_params\"]\n            )\n        )\n    except KeyError:\n        pass\n\n    return key\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/", "title": "sorted_spikes.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from clustered spikes times. See [1] for details.</p>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicatorSelection", "title": "<code>SortedSpikesIndicatorSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Bins spike times into regular intervals given by the sampling rate. Start and stop time of the interval are defined by the interval list.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesIndicatorSelection(SpyglassMixin, dj.Lookup):\n    \"\"\"Bins spike times into regular intervals given by the sampling rate.\n    Start and stop time of the interval are defined by the interval list.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; CuratedSpikeSorting\n    -&gt; IntervalList\n    sampling_rate=500 : float\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicator", "title": "<code>SortedSpikesIndicator</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Bins spike times into regular intervals given by the sampling rate. Useful for GLMs and for decoding.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesIndicator(SpyglassMixin, dj.Computed):\n    \"\"\"Bins spike times into regular intervals given by the sampling rate.\n    Useful for GLMs and for decoding.\n\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; SortedSpikesIndicatorSelection\n    ---\n    -&gt; AnalysisNwbfile\n    spike_indicator_object_id: varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        pprint.pprint(key)\n        # TODO: intersection of sort interval and interval list\n        interval_times = (IntervalList &amp; key).fetch1(\"valid_times\")\n\n        sampling_rate = (SortedSpikesIndicatorSelection &amp; key).fetch(\n            \"sampling_rate\"\n        )\n\n        time = self.get_time_bins_from_interval(interval_times, sampling_rate)\n\n        spikes_nwb = (CuratedSpikeSorting &amp; key).fetch_nwb()\n        # restrict to cases with units\n        spikes_nwb = [entry for entry in spikes_nwb if \"units\" in entry]\n        spike_times_list = [\n            np.asarray(n_trode[\"units\"][\"spike_times\"])\n            for n_trode in spikes_nwb\n        ]\n        if len(spike_times_list) &gt; 0:  # if units\n            spikes = np.concatenate(spike_times_list)\n\n            # Bin spikes into time bins\n            spike_indicator = []\n            for spike_times in spikes:\n                spike_times = spike_times[\n                    (spike_times &gt; time[0]) &amp; (spike_times &lt;= time[-1])\n                ]\n                spike_indicator.append(\n                    np.bincount(\n                        np.digitize(spike_times, time[1:-1]),\n                        minlength=time.shape[0],\n                    )\n                )\n\n            column_names = np.concatenate(\n                [\n                    [\n                        f'{n_trode[\"sort_group_id\"]:04d}_{unit_number:04d}'\n                        for unit_number in n_trode[\"units\"].index\n                    ]\n                    for n_trode in spikes_nwb\n                ]\n            )\n            spike_indicator = pd.DataFrame(\n                np.stack(spike_indicator, axis=1),\n                index=pd.Index(time, name=\"time\"),\n                columns=column_names,\n            )\n\n            # Insert into analysis nwb file\n            nwb_analysis_file = AnalysisNwbfile()\n            key[\"analysis_file_name\"] = nwb_analysis_file.create(\n                key[\"nwb_file_name\"]\n            )\n\n            key[\"spike_indicator_object_id\"] = nwb_analysis_file.add_nwb_object(\n                analysis_file_name=key[\"analysis_file_name\"],\n                nwb_object=spike_indicator.reset_index(),\n            )\n\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n\n            self.insert1(key)\n\n    @staticmethod\n    def get_time_bins_from_interval(interval_times, sampling_rate):\n        \"\"\"Gets the superset of the interval.\"\"\"\n        start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n        n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n        return np.linspace(start_time, end_time, n_samples)\n\n    def fetch1_dataframe(self):\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return pd.concat(\n            [\n                data[\"spike_indicator\"].set_index(\"time\")\n                for data in self.fetch_nwb()\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesIndicator.get_time_bins_from_interval", "title": "<code>get_time_bins_from_interval(interval_times, sampling_rate)</code>  <code>staticmethod</code>", "text": "<p>Gets the superset of the interval.</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@staticmethod\ndef get_time_bins_from_interval(interval_times, sampling_rate):\n    \"\"\"Gets the superset of the interval.\"\"\"\n    start_time, end_time = interval_times[0][0], interval_times[-1][-1]\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n\n    return np.linspace(start_time, end_time, n_samples)\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.SortedSpikesClassifierParameters", "title": "<code>SortedSpikesClassifierParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Stores parameters for decoding with sorted spikes</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>@schema\nclass SortedSpikesClassifierParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Stores parameters for decoding with sorted spikes\"\"\"\n\n    definition = \"\"\"\n    classifier_param_name : varchar(80) # a name for this set of parameters\n    ---\n    classifier_params :   BLOB    # initialization parameters\n    fit_params :          BLOB    # fit parameters\n    predict_params :      BLOB    # prediction parameters\n    \"\"\"\n\n    def insert_default(self):\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_cpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_cpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n        (\n            classifier_parameters,\n            fit_parameters,\n            predict_parameters,\n        ) = make_default_decoding_parameters_gpu()\n        self.insert1(\n            {\n                \"classifier_param_name\": \"default_decoding_gpu\",\n                \"classifier_params\": classifier_parameters,\n                \"fit_params\": fit_parameters,\n                \"predict_params\": predict_parameters,\n            },\n            skip_duplicates=True,\n        )\n\n    def insert1(self, key, **kwargs):\n        super().insert1(convert_classes_to_dict(key), **kwargs)\n\n    def fetch1(self, *args, **kwargs):\n        return restore_classes(super().fetch1(*args, **kwargs))\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_spike_indicator", "title": "<code>get_spike_indicator(key, time_range, sampling_rate=500.0)</code>", "text": "<p>For a given key, returns a dataframe with the spike indicator for each unit</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> required <code>time_range</code> <code>tuple[float, float]</code> <p>Start and end time of the spike indicator</p> required <code>sampling_rate</code> <code>float</code> <code>500.0</code> <p>Returns:</p> Name Type Description <code>spike_indicator</code> <code>(DataFrame, shape(n_time, n_units))</code> <p>A dataframe with the spike indicator for each unit</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_spike_indicator(\n    key: dict, time_range: tuple[float, float], sampling_rate: float = 500.0\n) -&gt; pd.DataFrame:\n    \"\"\"For a given key, returns a dataframe with the spike indicator for each unit\n\n    Parameters\n    ----------\n    key : dict\n    time_range : tuple[float, float]\n        Start and end time of the spike indicator\n    sampling_rate : float, optional\n\n    Returns\n    -------\n    spike_indicator : pd.DataFrame, shape (n_time, n_units)\n        A dataframe with the spike indicator for each unit\n    \"\"\"\n    start_time, end_time = time_range\n    n_samples = int(np.ceil((end_time - start_time) * sampling_rate)) + 1\n    time = np.linspace(start_time, end_time, n_samples)\n\n    spike_indicator = dict()\n    spikes_nwb_table = CuratedSpikeSorting() &amp; key\n\n    for n_trode in spikes_nwb_table.fetch_nwb():\n        try:\n            for unit_id, unit_spike_times in n_trode[\"units\"][\n                \"spike_times\"\n            ].items():\n                unit_spike_times = unit_spike_times[\n                    (unit_spike_times &gt; time[0])\n                    &amp; (unit_spike_times &lt;= time[-1])\n                ]\n                unit_name = f'{n_trode[\"sort_group_id\"]:04d}_{unit_id:04d}'\n                spike_indicator[unit_name] = np.bincount(\n                    np.digitize(unit_spike_times, time[1:-1]),\n                    minlength=time.shape[0],\n                )\n        except KeyError:\n            pass\n\n    return pd.DataFrame(\n        spike_indicator,\n        index=pd.Index(time, name=\"time\"),\n    )\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_decoding_data_for_epoch", "title": "<code>get_decoding_data_for_epoch(nwb_file_name, interval_list_name, position_info_param_name='default', additional_spike_keys={})</code>", "text": "<p>Collects the data needed for decoding</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>interval_list_name</code> <code>str</code> required <code>position_info_param_name</code> <code>str</code> <code>'default'</code> <code>additional_spike_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_position_features))</code> <code>spikes</code> <code>(DataFrame, shape(n_time, n_units))</code> <code>valid_slices</code> <code>list[slice]</code> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_decoding_data_for_epoch(\n    nwb_file_name: str,\n    interval_list_name: str,\n    position_info_param_name: str = \"default\",\n    additional_spike_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, pd.DataFrame, list[slice]]:\n    \"\"\"Collects the data needed for decoding\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    interval_list_name : str\n    position_info_param_name : str, optional\n    additional_spike_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_position_features)\n    spikes : pd.DataFrame, shape (n_time, n_units)\n    valid_slices : list[slice]\n\n    \"\"\"\n    # valid slices\n    valid_ephys_position_times_by_epoch = (\n        get_valid_ephys_position_times_by_epoch(nwb_file_name)\n    )\n    valid_ephys_position_times = valid_ephys_position_times_by_epoch[\n        interval_list_name\n    ]\n    valid_slices = convert_valid_times_to_slice(valid_ephys_position_times)\n\n    # position interval\n    position_interval_name = (\n        convert_epoch_interval_name_to_position_interval_name(\n            {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": interval_list_name,\n            }\n        )\n    )\n\n    # spikes\n    valid_times = np.asarray(\n        [(times.start, times.stop) for times in valid_slices]\n    )\n\n    curated_spikes_key = {\n        \"nwb_file_name\": nwb_file_name,\n        **additional_spike_keys,\n    }\n    spikes = get_spike_indicator(\n        curated_spikes_key,\n        (valid_times.min(), valid_times.max()),\n        sampling_rate=500,\n    )\n    spikes = pd.concat([spikes.loc[times] for times in valid_slices])\n\n    # position\n    position_info = (\n        IntervalPositionInfo()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_list_name\": position_interval_name,\n            \"position_info_param_name\": position_info_param_name,\n        }\n    ).fetch1_dataframe()\n    new_time = spikes.index.to_numpy()\n    new_index = pd.Index(\n        np.unique(np.concatenate((position_info.index, new_time))),\n        name=\"time\",\n    )\n    position_info = (\n        position_info.reindex(index=new_index)\n        .interpolate(method=\"linear\")\n        .reindex(index=new_time)\n    )\n\n    return position_info, spikes, valid_slices\n</code></pre>"}, {"location": "api/decoding/v0/sorted_spikes/#src.spyglass.decoding.v0.sorted_spikes.get_data_for_multiple_epochs", "title": "<code>get_data_for_multiple_epochs(nwb_file_name, epoch_names, position_info_param_name='decoding', additional_spike_keys={})</code>", "text": "<p>Collects the data needed for decoding for multiple epochs</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> required <code>epoch_names</code> <code>list</code> required <code>position_info_param_name</code> <code>str</code> <code>'decoding'</code> <code>additional_spike_keys</code> <code>dict</code> <code>{}</code> <p>Returns:</p> Name Type Description <code>position_info</code> <code>(DataFrame, shape(n_time, n_position_features))</code> <code>spikes</code> <code>(DataFrame, shape(n_time, n_units))</code> <code>valid_slices</code> <code>list[slice]</code> <code>environment_labels</code> <code>(ndarray, shape(n_time))</code> <p>The environment label for each time point</p> <code>sort_group_ids</code> <code>(ndarray, shape(n_units))</code> <p>The sort group of each unit</p> Source code in <code>src/spyglass/decoding/v0/sorted_spikes.py</code> <pre><code>def get_data_for_multiple_epochs(\n    nwb_file_name: str,\n    epoch_names: list,\n    position_info_param_name: str = \"decoding\",\n    additional_spike_keys: dict = {},\n) -&gt; tuple[pd.DataFrame, pd.DataFrame, list[slice], np.ndarray, np.ndarray]:\n    \"\"\"Collects the data needed for decoding for multiple epochs\n\n    Parameters\n    ----------\n    nwb_file_name : str\n    epoch_names : list\n    position_info_param_name : str, optional\n    additional_spike_keys : dict, optional\n\n    Returns\n    -------\n    position_info : pd.DataFrame, shape (n_time, n_position_features)\n    spikes : pd.DataFrame, shape (n_time, n_units)\n    valid_slices : list[slice]\n    environment_labels : np.ndarray, shape (n_time,)\n        The environment label for each time point\n    sort_group_ids : np.ndarray, shape (n_units,)\n        The sort group of each unit\n    \"\"\"\n    data = []\n    environment_labels = []\n\n    for epoch in epoch_names:\n        logger.info(epoch)\n        data.append(\n            get_decoding_data_for_epoch(\n                nwb_file_name,\n                epoch,\n                position_info_param_name=position_info_param_name,\n                additional_spike_keys=additional_spike_keys,\n            )\n        )\n        n_time = data[-1][0].shape[0]\n        environment_labels.append([epoch] * n_time)\n\n    environment_labels = np.concatenate(environment_labels, axis=0)\n    position_info, spikes, valid_slices = list(zip(*data))\n    position_info = pd.concat(position_info, axis=0)\n    spikes = pd.concat(spikes, axis=0)\n    valid_slices = {\n        epoch: valid_slice\n        for epoch, valid_slice in zip(epoch_names, valid_slices)\n    }\n\n    assert position_info.shape[0] == spikes.shape[0]\n\n    sort_group_ids = np.asarray(\n        [int(col.split(\"_\")[0]) for col in spikes.columns]\n    )\n\n    return (\n        position_info,\n        spikes,\n        valid_slices,\n        environment_labels,\n        sort_group_ids,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/visualization/", "title": "visualization.py", "text": ""}, {"location": "api/decoding/v0/visualization_1D_view/", "title": "visualization_1D_view.py", "text": ""}, {"location": "api/decoding/v0/visualization_1D_view/#src.spyglass.decoding.v0.visualization_1D_view.create_1D_decode_view", "title": "<code>create_1D_decode_view(posterior, linear_position=None, ref_time_sec=None)</code>", "text": "<p>Creates a view of an interactive heatmap of position vs. time.</p> <p>Parameters:</p> Name Type Description Default <code>posterior</code> <code>(DataArray, shape(n_time, n_position_bins))</code> required <code>linear_position</code> <code>(ndarray, shape(n_time))</code> <code>None</code> <code>ref_time_sec</code> <code>float64</code> <p>Reference time for the purpose of offsetting the start time</p> <code>None</code> <p>Returns:</p> Name Type Description <code>view</code> <code>DecodedLinearPositionData</code> Source code in <code>src/spyglass/decoding/v0/visualization_1D_view.py</code> <pre><code>def create_1D_decode_view(\n    posterior: xr.DataArray,\n    linear_position: np.ndarray = None,\n    ref_time_sec: Union[np.float64, None] = None,\n) -&gt; vvf.DecodedLinearPositionData:\n    \"\"\"Creates a view of an interactive heatmap of position vs. time.\n\n    Parameters\n    ----------\n    posterior : xr.DataArray, shape (n_time, n_position_bins)\n    linear_position : np.ndarray, shape (n_time, ), optional\n    ref_time_sec : np.float64, optional\n        Reference time for the purpose of offsetting the start time\n\n    Returns\n    -------\n    view : vvf.DecodedLinearPositionData\n\n    \"\"\"\n    if linear_position is not None:\n        linear_position = np.asarray(linear_position).squeeze()\n\n    trimmed_posterior = discretize_and_trim(posterior)\n    observations_per_time = get_observations_per_time(\n        trimmed_posterior, posterior\n    )\n    sampling_freq = get_sampling_freq(posterior.time)\n    start_time_sec = posterior.time.values[0]\n    if ref_time_sec is not None:\n        start_time_sec = start_time_sec - ref_time_sec\n\n    trimmed_bin_center_index = get_trimmed_bin_center_index(\n        posterior.position.values, trimmed_posterior.position.values\n    )\n\n    return vvf.DecodedLinearPositionData(\n        values=trimmed_posterior.values,\n        positions=trimmed_bin_center_index,\n        frame_bounds=observations_per_time,\n        positions_key=posterior.position.values.astype(np.float32),\n        observed_positions=linear_position,\n        start_time_sec=start_time_sec,\n        sampling_frequency=sampling_freq,\n    )\n</code></pre>"}, {"location": "api/decoding/v0/visualization_2D_view/", "title": "visualization_2D_view.py", "text": ""}, {"location": "api/decoding/v0/visualization_2D_view/#src.spyglass.decoding.v0.visualization_2D_view.create_2D_decode_view", "title": "<code>create_2D_decode_view(position_time, position, interior_place_bin_centers, place_bin_size, posterior, head_dir=None)</code>", "text": "<p>Creates a 2D decoding movie view</p> <p>Parameters:</p> Name Type Description Default <code>position_time</code> <code>(ndarray, shape(n_time))</code> required <code>position</code> <code>(ndarray, shape(n_time, 2))</code> required <code>interior_place_bin_centers</code> <code>ndarray</code> required <code>place_bin_size</code> <code>(ndarray, shape(2, 1))</code> required <code>posterior</code> <code>(DataArray, shape(n_time, n_position_bins))</code> required <code>head_dir</code> <code>ndarray</code> <code>None</code> <p>Returns:</p> Name Type Description <code>view</code> <code>TrackPositionAnimationV1</code> Source code in <code>src/spyglass/decoding/v0/visualization_2D_view.py</code> <pre><code>def create_2D_decode_view(\n    position_time: np.ndarray,\n    position: np.ndarray,\n    interior_place_bin_centers: np.ndarray,\n    place_bin_size: np.ndarray,\n    posterior: xr.DataArray,\n    head_dir: np.ndarray = None,\n) -&gt; vvf.TrackPositionAnimationV1:\n    \"\"\"Creates a 2D decoding movie view\n\n    Parameters\n    ----------\n    position_time : np.ndarray, shape (n_time,)\n    position : np.ndarray, shape (n_time, 2)\n    interior_place_bin_centers: np.ndarray, shape (n_track_bins, 2)\n    place_bin_size : np.ndarray, shape (2, 1)\n    posterior : xr.DataArray, shape (n_time, n_position_bins)\n    head_dir : np.ndarray, optional\n\n    Returns\n    -------\n    view : vvf.TrackPositionAnimationV1\n\n    \"\"\"\n    assert (\n        position_time.shape[0] == position.shape[0]\n    ), \"position_time and position must have the same length\"\n    assert (\n        posterior.shape[0] == position.shape[0]\n    ), \"posterior and position must have the same length\"\n\n    position_time = np.squeeze(np.asarray(position_time)).copy()\n    position = np.asarray(position)\n    if head_dir is not None:\n        head_dir = np.squeeze(np.asarray(head_dir))\n\n    track_bin_width = place_bin_size[0]\n    track_bin_height = place_bin_size[1]\n    # NOTE: We expect caller to have converted from fortran ordering already\n    # i.e. somewhere upstream, centers = env.place_bin_centers_[env.is_track_interior_.ravel(order=\"F\")]\n    upper_left_points = get_ul_corners(\n        track_bin_width, track_bin_height, interior_place_bin_centers\n    )\n\n    data = create_static_track_animation(\n        ul_corners=upper_left_points,\n        track_rect_height=track_bin_height,\n        track_rect_width=track_bin_width,\n        timestamps=position_time,\n        positions=position.T,\n        head_dir=head_dir,\n        compute_real_time_rate=True,\n    )\n    data[\"decodedData\"] = process_decoded_data(posterior)\n\n    return create_track_animation_object(static_track_animation=data)\n</code></pre>"}, {"location": "api/decoding/v1/clusterless/", "title": "clusterless.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from unclustered spikes and spike waveform features. See [1] for details.</p>"}, {"location": "api/decoding/v1/clusterless/#src.spyglass.decoding.v1.clusterless--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v1/core/", "title": "core.py", "text": ""}, {"location": "api/decoding/v1/core/#src.spyglass.decoding.v1.core.DecodingParameters", "title": "<code>DecodingParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Parameters for decoding the animal's mental position and some category of interest</p> Source code in <code>src/spyglass/decoding/v1/core.py</code> <pre><code>@schema\nclass DecodingParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Parameters for decoding the animal's mental position and some category of interest\"\"\"\n\n    definition = \"\"\"\n    decoding_param_name : varchar(80)  # a name for this set of parameters\n    ---\n    decoding_params : BLOB             # initialization parameters for model\n    decoding_kwargs = NULL : BLOB      # additional keyword arguments\n    \"\"\"\n\n    contents = [\n        {\n            \"decoding_param_name\": f\"contfrag_clusterless_{non_local_detector_version}\",\n            \"decoding_params\": ContFragClusterlessClassifier(),\n        },\n        {\n            \"decoding_param_name\": f\"nonlocal_clusterless_{non_local_detector_version}\",\n            \"decoding_params\": NonLocalClusterlessDetector(),\n        },\n        {\n            \"decoding_param_name\": f\"contfrag_sorted_{non_local_detector_version}\",\n            \"decoding_params\": ContFragSortedSpikesClassifier(),\n        },\n        {\n            \"decoding_param_name\": f\"nonlocal_sorted_{non_local_detector_version}\",\n            \"decoding_params\": NonLocalSortedSpikesDetector(),\n        },\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        cls.insert(cls.contents, skip_duplicates=True)\n\n    def insert(self, rows, *args, **kwargs):\n        for row in rows:\n            row[\"decoding_params\"] = convert_classes_to_dict(\n                vars(row[\"decoding_params\"])\n            )\n        super().insert(rows, *args, **kwargs)\n\n    def fetch(self, *args, **kwargs):\n        rows = super().fetch(*args, **kwargs)\n        if len(rows) &gt; 0 and len(rows[0]) &gt; 1:\n            content = []\n            for (\n                decoding_param_name,\n                decoding_params,\n                decoding_kwargs,\n            ) in rows:\n                content.append(\n                    (\n                        decoding_param_name,\n                        restore_classes(decoding_params),\n                        decoding_kwargs,\n                    )\n                )\n        else:\n            content = rows\n        return content\n\n    def fetch1(self, *args, **kwargs):\n        row = super().fetch1(*args, **kwargs)\n        row[\"decoding_params\"] = restore_classes(row[\"decoding_params\"])\n        return row\n</code></pre>"}, {"location": "api/decoding/v1/dj_decoder_conversion/", "title": "dj_decoder_conversion.py", "text": "<p>Converts decoder classes into dictionaries and dictionaries into classes so that datajoint can store them in tables.</p>"}, {"location": "api/decoding/v1/dj_decoder_conversion/#src.spyglass.decoding.v1.dj_decoder_conversion.restore_classes", "title": "<code>restore_classes(params)</code>", "text": "<p>Converts a dictionary of parameters into a dictionary of classes since datajoint cannot handle classes</p> <p>Parameters:</p> Name Type Description Default <code>params</code> <code>dict</code> <p>The parameters to convert</p> required <p>Returns:</p> Name Type Description <code>converted_params</code> <code>dict</code> <p>The converted parameters</p> Source code in <code>src/spyglass/decoding/v1/dj_decoder_conversion.py</code> <pre><code>def restore_classes(params: dict) -&gt; dict:\n    \"\"\"Converts a dictionary of parameters into a dictionary of classes\n    since datajoint cannot handle classes\n\n    Parameters\n    ----------\n    params : dict\n        The parameters to convert\n\n    Returns\n    -------\n    converted_params : dict\n        The converted parameters\n    \"\"\"\n\n    params = copy.deepcopy(params)\n\n    continuous_state_transition_types = _map_class_name_to_class(cst)\n    discrete_state_transition_types = _map_class_name_to_class(dst)\n    continuous_initial_conditions_types = _map_class_name_to_class(ic)\n\n    params[\"environments\"] = [\n        _convert_env_dict(env_params) for env_params in params[\"environments\"]\n    ]\n\n    params[\"continuous_transition_types\"] = [\n        [\n            _convert_dict_to_class(st, continuous_state_transition_types)\n            for st in sts\n        ]\n        for sts in params[\"continuous_transition_types\"]\n    ]\n    params[\"discrete_transition_type\"] = _convert_dict_to_class(\n        params[\"discrete_transition_type\"],\n        discrete_state_transition_types,\n    )\n    params[\"continuous_initial_conditions_types\"] = [\n        _convert_dict_to_class(cont_ic, continuous_initial_conditions_types)\n        for cont_ic in params[\"continuous_initial_conditions_types\"]\n    ]\n\n    if params[\"observation_models\"] is not None:\n        params[\"observation_models\"] = [\n            ObservationModel(**obs) for obs in params[\"observation_models\"]\n        ]\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v1/dj_decoder_conversion/#src.spyglass.decoding.v1.dj_decoder_conversion.convert_classes_to_dict", "title": "<code>convert_classes_to_dict(params)</code>", "text": "<p>Converts the classifier parameters into a dictionary so that datajoint can store it.</p> Source code in <code>src/spyglass/decoding/v1/dj_decoder_conversion.py</code> <pre><code>def convert_classes_to_dict(params: dict) -&gt; dict:\n    \"\"\"Converts the classifier parameters into a dictionary so that datajoint can store it.\"\"\"\n    params = copy.deepcopy(params)\n    try:\n        params[\"environments\"] = [\n            _convert_environment_to_dict(env) for env in params[\"environments\"]\n        ]\n    except TypeError:\n        params[\"environments\"] = [\n            _convert_environment_to_dict(params[\"environments\"])\n        ]\n    params[\"continuous_transition_types\"] = _convert_transitions_to_dict(\n        params[\"continuous_transition_types\"]\n    )\n    params[\"discrete_transition_type\"] = _to_dict(\n        params[\"discrete_transition_type\"]\n    )\n    params[\"continuous_initial_conditions_types\"] = [\n        _to_dict(cont_ic)\n        for cont_ic in params[\"continuous_initial_conditions_types\"]\n    ]\n\n    if params[\"observation_models\"] is not None:\n        params[\"observation_models\"] = [\n            vars(obs) for obs in params[\"observation_models\"]\n        ]\n\n    try:\n        params[\"clusterless_algorithm_params\"] = _convert_algorithm_params(\n            params[\"clusterless_algorithm_params\"]\n        )\n    except KeyError:\n        pass\n\n    return params\n</code></pre>"}, {"location": "api/decoding/v1/sorted_spikes/", "title": "sorted_spikes.py", "text": "<p>Pipeline for decoding the animal's mental position and some category of interest from clustered spikes times. See [1] for details.</p>"}, {"location": "api/decoding/v1/sorted_spikes/#src.spyglass.decoding.v1.sorted_spikes--references", "title": "References", "text": "<p>[1] Denovellis, E. L. et al. Hippocampal replay of experience at real-world speeds. eLife 10, e64505 (2021).</p>"}, {"location": "api/decoding/v1/temp/", "title": "temp.py", "text": ""}, {"location": "api/decoding/v1/waveform_features/", "title": "waveform_features.py", "text": ""}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams", "title": "<code>WaveformFeaturesParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Defines the types of spike waveform features computed for a given spike time.</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@schema\nclass WaveformFeaturesParams(SpyglassMixin, dj.Lookup):\n    \"\"\"Defines the types of spike waveform features computed for a given spike\n    time.\"\"\"\n\n    definition = \"\"\"\n    features_param_name : varchar(80) # a name for this set of parameters\n    ---\n    params : longblob # the parameters for the waveform features\n    \"\"\"\n    _default_waveform_feature_params = {\n        \"amplitude\": {\n            \"peak_sign\": \"neg\",\n            \"estimate_peak_time\": False,\n        }\n    }\n    _default_waveform_extraction_params = {\n        \"ms_before\": 0.5,\n        \"ms_after\": 0.5,\n        \"max_spikes_per_unit\": None,\n        \"n_jobs\": 5,\n        \"chunk_duration\": \"1000s\",\n    }\n    contents = [\n        [\n            \"amplitude\",\n            {\n                \"waveform_features_params\": _default_waveform_feature_params,\n                \"waveform_extraction_params\": _default_waveform_extraction_params,\n            },\n        ],\n        [\n            \"amplitude, spike_location\",\n            {\n                \"waveform_features_params\": {\n                    \"amplitude\": _default_waveform_feature_params[\"amplitude\"],\n                    \"spike_location\": {},\n                },\n                \"waveform_extraction_params\": _default_waveform_extraction_params,\n            },\n        ],\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        cls.insert(cls.contents, skip_duplicates=True)\n\n    @staticmethod\n    def check_supported_waveform_features(waveform_features: list[str]) -&gt; bool:\n        \"\"\"Checks whether the requested waveform features types are supported\n\n        Parameters\n        ----------\n        waveform_features : list\n        \"\"\"\n        supported_features = set(WAVEFORM_FEATURE_FUNCTIONS)\n        return set(waveform_features).issubset(supported_features)\n\n    @property\n    def supported_waveform_features(self) -&gt; list[str]:\n        \"\"\"Returns the list of supported waveform features\"\"\"\n        return list(WAVEFORM_FEATURE_FUNCTIONS)\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams.check_supported_waveform_features", "title": "<code>check_supported_waveform_features(waveform_features)</code>  <code>staticmethod</code>", "text": "<p>Checks whether the requested waveform features types are supported</p> <p>Parameters:</p> Name Type Description Default <code>waveform_features</code> <code>list</code> required Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@staticmethod\ndef check_supported_waveform_features(waveform_features: list[str]) -&gt; bool:\n    \"\"\"Checks whether the requested waveform features types are supported\n\n    Parameters\n    ----------\n    waveform_features : list\n    \"\"\"\n    supported_features = set(WAVEFORM_FEATURE_FUNCTIONS)\n    return set(waveform_features).issubset(supported_features)\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.WaveformFeaturesParams.supported_waveform_features", "title": "<code>supported_waveform_features: list[str]</code>  <code>property</code>", "text": "<p>Returns the list of supported waveform features</p>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.UnitWaveformFeatures", "title": "<code>UnitWaveformFeatures</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>For each spike time, compute a spike waveform feature associated with that spike. Used for clusterless decoding.</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>@schema\nclass UnitWaveformFeatures(SpyglassMixin, dj.Computed):\n    \"\"\"For each spike time, compute a spike waveform feature associated with that\n    spike. Used for clusterless decoding.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; UnitWaveformFeaturesSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # the NWB object that stores the waveforms\n    \"\"\"\n\n    def make(self, key):\n        # get the list of feature parameters\n        params = (WaveformFeaturesParams &amp; key).fetch1(\"params\")\n\n        # check that the feature type is supported\n        if not WaveformFeaturesParams.check_supported_waveform_features(\n            params[\"waveform_features_params\"]\n        ):\n            raise NotImplementedError(\n                f\"Features {set(params['waveform_features_params'])} are not supported\"\n            )\n\n        merge_key = {\"merge_id\": key[\"spikesorting_merge_id\"]}\n        waveform_extractor = self._fetch_waveform(\n            merge_key, params[\"waveform_extraction_params\"]\n        )\n\n        source_key = SpikeSortingOutput().merge_get_parent(merge_key).fetch1()\n        # v0 pipeline\n        if \"sorter\" in source_key and \"nwb_file_name\" in source_key:\n            sorter = source_key[\"sorter\"]\n            nwb_file_name = source_key[\"nwb_file_name\"]\n            analysis_nwb_key = \"units\"\n        # v1 pipeline\n        else:\n            sorting_id = (SpikeSortingOutput.CurationV1 &amp; merge_key).fetch1(\n                \"sorting_id\"\n            )\n            sorter, nwb_file_name = (\n                SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}\n            ).fetch1(\"sorter\", \"nwb_file_name\")\n            analysis_nwb_key = \"object_id\"\n\n        waveform_features = {}\n\n        for feature, feature_params in params[\n            \"waveform_features_params\"\n        ].items():\n            waveform_features[feature] = self._compute_waveform_features(\n                waveform_extractor,\n                feature,\n                feature_params,\n                sorter,\n            )\n\n        spike_times = SpikeSortingOutput().fetch_nwb(merge_key)[0][\n            analysis_nwb_key\n        ][\"spike_times\"]\n\n        (\n            key[\"analysis_file_name\"],\n            key[\"object_id\"],\n        ) = _write_waveform_features_to_nwb(\n            nwb_file_name,\n            waveform_extractor,\n            spike_times,\n            waveform_features,\n        )\n\n        AnalysisNwbfile().add(\n            nwb_file_name,\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @staticmethod\n    def _fetch_waveform(\n        merge_key: dict, waveform_extraction_params: dict\n    ) -&gt; si.WaveformExtractor:\n        # get the recording from the parent table\n        recording = SpikeSortingOutput().get_recording(merge_key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n        # get the sorting from the parent table\n        sorting = SpikeSortingOutput().get_sorting(merge_key)\n\n        waveforms_temp_dir = temp_dir + \"/\" + str(merge_key[\"merge_id\"])\n        os.makedirs(waveforms_temp_dir, exist_ok=True)\n\n        return si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveforms_temp_dir,\n            overwrite=True,\n            **waveform_extraction_params,\n        )\n\n    @staticmethod\n    def _compute_waveform_features(\n        waveform_extractor: si.WaveformExtractor,\n        feature: str,\n        feature_params: dict,\n        sorter: str,\n    ) -&gt; dict:\n        feature_func = WAVEFORM_FEATURE_FUNCTIONS[feature]\n        if sorter == \"clusterless_thresholder\" and feature == \"amplitude\":\n            feature_params[\"estimate_peak_time\"] = False\n\n        return {\n            unit_id: feature_func(waveform_extractor, unit_id, **feature_params)\n            for unit_id in waveform_extractor.sorting.get_unit_ids()\n        }\n\n    def fetch_data(self) -&gt; tuple[list[np.ndarray], list[np.ndarray]]:\n        \"\"\"Fetches the spike times and features for each unit.\n\n        Returns\n        -------\n        spike_times : list of np.ndarray\n            List of spike times for each unit\n        features : list of np.ndarray\n            List of features for each unit\n\n        \"\"\"\n        return tuple(\n            zip(\n                *list(\n                    chain(\n                        *[self._convert_data(data) for data in self.fetch_nwb()]\n                    )\n                )\n            )\n        )\n\n    @staticmethod\n    def _convert_data(nwb_data) -&gt; list[tuple[np.ndarray, np.ndarray]]:\n        feature_df = nwb_data[\"object_id\"]\n\n        feature_columns = [\n            column for column in feature_df.columns if column != \"spike_times\"\n        ]\n\n        return [\n            (\n                unit.spike_times,\n                np.concatenate(unit[feature_columns].to_numpy(), axis=1),\n            )\n            for _, unit in feature_df.iterrows()\n        ]\n</code></pre>"}, {"location": "api/decoding/v1/waveform_features/#src.spyglass.decoding.v1.waveform_features.UnitWaveformFeatures.fetch_data", "title": "<code>fetch_data()</code>", "text": "<p>Fetches the spike times and features for each unit.</p> <p>Returns:</p> Name Type Description <code>spike_times</code> <code>list of np.ndarray</code> <p>List of spike times for each unit</p> <code>features</code> <code>list of np.ndarray</code> <p>List of features for each unit</p> Source code in <code>src/spyglass/decoding/v1/waveform_features.py</code> <pre><code>def fetch_data(self) -&gt; tuple[list[np.ndarray], list[np.ndarray]]:\n    \"\"\"Fetches the spike times and features for each unit.\n\n    Returns\n    -------\n    spike_times : list of np.ndarray\n        List of spike times for each unit\n    features : list of np.ndarray\n        List of features for each unit\n\n    \"\"\"\n    return tuple(\n        zip(\n            *list(\n                chain(\n                    *[self._convert_data(data) for data in self.fetch_nwb()]\n                )\n            )\n        )\n    )\n</code></pre>"}, {"location": "api/lfp/lfp_electrode/", "title": "lfp_electrode.py", "text": ""}, {"location": "api/lfp/lfp_electrode/#src.spyglass.lfp.lfp_electrode.LFPElectrodeGroup", "title": "<code>LFPElectrodeGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lfp/lfp_electrode.py</code> <pre><code>@schema\nclass LFPElectrodeGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; Session                             # the session to which this LFP belongs\n     lfp_electrode_group_name: varchar(200) # the name of this group of electrodes\n     \"\"\"\n\n    class LFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPElectrodeGroup # the group of electrodes to be filtered\n        -&gt; Electrode        # the electrode to be filtered\n        \"\"\"\n\n    @staticmethod\n    def create_lfp_electrode_group(\n        nwb_file_name: str, group_name: str, electrode_list: list[int]\n    ):\n        \"\"\"Adds an LFPElectrodeGroup and the individual electrodes\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            The name of the nwb file (e.g. the session)\n        group_name : str\n            The name of this group (&lt; 200 char)\n        electrode_list : list\n            A list of the electrode ids to include in this group.\n        \"\"\"\n        # remove the session and then recreate the session and Electrode list\n        # check to see if the user allowed the deletion\n        key = {\n            \"nwb_file_name\": nwb_file_name,\n            \"lfp_electrode_group_name\": group_name,\n        }\n        LFPElectrodeGroup().insert1(key, skip_duplicates=True)\n\n        # TODO: do this in a better way\n        all_electrodes = (Electrode() &amp; {\"nwb_file_name\": nwb_file_name}).fetch(\n            as_dict=True\n        )\n        primary_key = Electrode.primary_key\n        if isinstance(electrode_list, ndarray):\n            # convert to list if it is an numpy array\n            electrode_list = list(electrode_list.astype(int).reshape(-1))\n        for e in all_electrodes:\n            # create a dictionary so we can insert the electrodes\n            if e[\"electrode_id\"] in electrode_list:\n                lfpelectdict = {k: v for k, v in e.items() if k in primary_key}\n                lfpelectdict[\"lfp_electrode_group_name\"] = group_name\n                LFPElectrodeGroup().LFPElectrode.insert1(\n                    lfpelectdict, skip_duplicates=True\n                )\n</code></pre>"}, {"location": "api/lfp/lfp_electrode/#src.spyglass.lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group", "title": "<code>create_lfp_electrode_group(nwb_file_name, group_name, electrode_list)</code>  <code>staticmethod</code>", "text": "<p>Adds an LFPElectrodeGroup and the individual electrodes</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the nwb file (e.g. the session)</p> required <code>group_name</code> <code>str</code> <p>The name of this group (&lt; 200 char)</p> required <code>electrode_list</code> <code>list</code> <p>A list of the electrode ids to include in this group.</p> required Source code in <code>src/spyglass/lfp/lfp_electrode.py</code> <pre><code>@staticmethod\ndef create_lfp_electrode_group(\n    nwb_file_name: str, group_name: str, electrode_list: list[int]\n):\n    \"\"\"Adds an LFPElectrodeGroup and the individual electrodes\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        The name of the nwb file (e.g. the session)\n    group_name : str\n        The name of this group (&lt; 200 char)\n    electrode_list : list\n        A list of the electrode ids to include in this group.\n    \"\"\"\n    # remove the session and then recreate the session and Electrode list\n    # check to see if the user allowed the deletion\n    key = {\n        \"nwb_file_name\": nwb_file_name,\n        \"lfp_electrode_group_name\": group_name,\n    }\n    LFPElectrodeGroup().insert1(key, skip_duplicates=True)\n\n    # TODO: do this in a better way\n    all_electrodes = (Electrode() &amp; {\"nwb_file_name\": nwb_file_name}).fetch(\n        as_dict=True\n    )\n    primary_key = Electrode.primary_key\n    if isinstance(electrode_list, ndarray):\n        # convert to list if it is an numpy array\n        electrode_list = list(electrode_list.astype(int).reshape(-1))\n    for e in all_electrodes:\n        # create a dictionary so we can insert the electrodes\n        if e[\"electrode_id\"] in electrode_list:\n            lfpelectdict = {k: v for k, v in e.items() if k in primary_key}\n            lfpelectdict[\"lfp_electrode_group_name\"] = group_name\n            LFPElectrodeGroup().LFPElectrode.insert1(\n                lfpelectdict, skip_duplicates=True\n            )\n</code></pre>"}, {"location": "api/lfp/lfp_imported/", "title": "lfp_imported.py", "text": ""}, {"location": "api/lfp/lfp_merge/", "title": "lfp_merge.py", "text": ""}, {"location": "api/lfp/lfp_merge/#src.spyglass.lfp.lfp_merge.LFPOutput", "title": "<code>LFPOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/lfp/lfp_merge.py</code> <pre><code>@schema\nclass LFPOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class LFPV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; LFPV1\n        \"\"\"\n\n    class ImportedLFP(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ImportedLFP\n        \"\"\"\n\n    class CommonLFP(SpyglassMixin, dj.Part):  # noqa: F811\n        \"\"\"Table to pass-through legacy LFP\"\"\"\n\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CommonLFP\n        \"\"\"\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        # Note: `proj` below facilitates operator syntax eg Table &amp; restrict\n        nwb_lfp = self.fetch_nwb(self.proj())[0]\n        return pd.DataFrame(\n            nwb_lfp[\"lfp\"].data,\n            index=pd.Index(nwb_lfp[\"lfp\"].timestamps, name=\"time\"),\n        )\n</code></pre>"}, {"location": "api/lfp/lfp_merge/#src.spyglass.lfp.lfp_merge.LFPOutput.CommonLFP", "title": "<code>CommonLFP</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through legacy LFP</p> Source code in <code>src/spyglass/lfp/lfp_merge.py</code> <pre><code>class CommonLFP(SpyglassMixin, dj.Part):  # noqa: F811\n    \"\"\"Table to pass-through legacy LFP\"\"\"\n\n    definition = \"\"\"\n    -&gt; master\n    ---\n    -&gt; CommonLFP\n    \"\"\"\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/", "title": "lfp_band.py", "text": ""}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandSelection", "title": "<code>LFPBandSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>The user's selection of LFP data to be filtered in a given frequency band.</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>@schema\nclass LFPBandSelection(SpyglassMixin, dj.Manual):\n    \"\"\"The user's selection of LFP data to be filtered in a given frequency band.\"\"\"\n\n    definition = \"\"\"\n    -&gt; LFPOutput.proj(lfp_merge_id='merge_id')                            # the LFP data to be filtered\n    -&gt; FirFilterParameters                                                # the filter to use for the data\n    -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n    lfp_band_sampling_rate: int                                           # the sampling rate for this band\n    ---\n    min_interval_len = 1.0: float  # the minimum length of a valid interval to filter\n    \"\"\"\n\n    class LFPBandElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; LFPBandSelection # the LFP band selection\n        -&gt; LFPElectrodeGroup.LFPElectrode  # the LFP electrode to be filtered\n        reference_elect_id = -1: int  # the reference electrode to use; -1 for no reference\n        ---\n        \"\"\"\n\n    def set_lfp_band_electrodes(\n        self,\n        nwb_file_name: str,\n        lfp_merge_id: int,\n        electrode_list: list[int],\n        filter_name: str,\n        interval_list_name: str,\n        reference_electrode_list: list[int],\n        lfp_band_sampling_rate: int,\n    ):\n        \"\"\"Sets the electrodes to be filtered for a given LFP\n\n        Parameters\n        ----------\n        nwb_file_name: str\n            The name of the NWB file containing the LFP data\n        lfp_merge_id: int\n            The uuid of the LFP data to be filtered\n        electrode_list: list\n            A list of the electrodes to be filtered\n        filter_name: str\n            The name of the filter to be used\n        interval_list_name: str\n            The name of the interval list to be used\n        reference_electrode_list: list\n            A list of the reference electrodes to be used\n        lfp_band_sampling_rate: int\n        \"\"\"\n        # Error checks on parameters\n        # electrode_list\n\n        lfp_key = {\"merge_id\": lfp_merge_id}\n        lfp_part_table = LFPOutput.merge_get_part(lfp_key)\n\n        query = LFPElectrodeGroup().LFPElectrode() &amp; lfp_key\n        available_electrodes = query.fetch(\"electrode_id\")\n        if not np.all(np.isin(electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in electrode_list must be valid electrode_ids in\"\n                + \" the LFPElectodeGroup table: \"\n                + f\"{electrode_list} not in {available_electrodes}\"\n            )\n        # sampling rate\n        lfp_sampling_rate = LFPOutput.merge_get_parent(lfp_key).fetch1(\n            \"lfp_sampling_rate\"\n        )\n        decimation = lfp_sampling_rate // lfp_band_sampling_rate\n        # filter\n        filter_query = FirFilterParameters() &amp; {\n            \"filter_name\": filter_name,\n            \"filter_sampling_rate\": lfp_sampling_rate,\n        }\n        if not filter_query:\n            raise ValueError(\n                f\"Filter {filter_name}, sampling rate {lfp_sampling_rate} is \"\n                + \"not in the FirFilterParameters table\"\n            )\n        # interval_list\n        interval_query = IntervalList() &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"interval_name\": interval_list_name,\n        }\n        if not interval_query:\n            raise ValueError(\n                f\"interval list {interval_list_name} is not in the IntervalList\"\n                \" table; the list must be added before this function is called\"\n            )\n        # reference_electrode_list\n        if len(reference_electrode_list) != 1 and len(\n            reference_electrode_list\n        ) != len(electrode_list):\n            raise ValueError(\n                \"reference_electrode_list must contain either 1 or \"\n                + \"len(electrode_list) elements\"\n            )\n        # add a -1 element to the list to allow for the no reference option\n        available_electrodes = np.append(available_electrodes, [-1])\n        if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n            raise ValueError(\n                \"All elements in reference_electrode_list must be valid \"\n                \"electrode_ids in the LFPSelection table\"\n            )\n\n        # make a list of all the references\n        ref_list = np.zeros((len(electrode_list),))\n        ref_list[:] = reference_electrode_list\n\n        key = dict(\n            nwb_file_name=nwb_file_name,\n            lfp_merge_id=lfp_merge_id,\n            filter_name=filter_name,\n            filter_sampling_rate=lfp_sampling_rate,\n            target_interval_list_name=interval_list_name,\n            lfp_band_sampling_rate=lfp_sampling_rate // decimation,\n        )\n        # insert an entry into the main LFPBandSelectionTable\n        self.insert1(key, skip_duplicates=True)\n\n        key[\"lfp_electrode_group_name\"] = lfp_part_table.fetch1(\n            \"lfp_electrode_group_name\"\n        )\n        # iterate through all of the new elements and add them\n        for e, r in zip(electrode_list, ref_list):\n            elect_key = (\n                LFPElectrodeGroup.LFPElectrode\n                &amp; {\n                    \"nwb_file_name\": nwb_file_name,\n                    \"lfp_electrode_group_name\": key[\"lfp_electrode_group_name\"],\n                    \"electrode_id\": e,\n                }\n            ).fetch1(\"KEY\")\n            for item in elect_key:\n                key[item] = elect_key[item]\n            query = Electrode &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_id\": e,\n            }\n            key[\"reference_elect_id\"] = r\n            self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandSelection.set_lfp_band_electrodes", "title": "<code>set_lfp_band_electrodes(nwb_file_name, lfp_merge_id, electrode_list, filter_name, interval_list_name, reference_electrode_list, lfp_band_sampling_rate)</code>", "text": "<p>Sets the electrodes to be filtered for a given LFP</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>The name of the NWB file containing the LFP data</p> required <code>lfp_merge_id</code> <code>int</code> <p>The uuid of the LFP data to be filtered</p> required <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to be filtered</p> required <code>filter_name</code> <code>str</code> <p>The name of the filter to be used</p> required <code>interval_list_name</code> <code>str</code> <p>The name of the interval list to be used</p> required <code>reference_electrode_list</code> <code>list[int]</code> <p>A list of the reference electrodes to be used</p> required <code>lfp_band_sampling_rate</code> <code>int</code> required Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def set_lfp_band_electrodes(\n    self,\n    nwb_file_name: str,\n    lfp_merge_id: int,\n    electrode_list: list[int],\n    filter_name: str,\n    interval_list_name: str,\n    reference_electrode_list: list[int],\n    lfp_band_sampling_rate: int,\n):\n    \"\"\"Sets the electrodes to be filtered for a given LFP\n\n    Parameters\n    ----------\n    nwb_file_name: str\n        The name of the NWB file containing the LFP data\n    lfp_merge_id: int\n        The uuid of the LFP data to be filtered\n    electrode_list: list\n        A list of the electrodes to be filtered\n    filter_name: str\n        The name of the filter to be used\n    interval_list_name: str\n        The name of the interval list to be used\n    reference_electrode_list: list\n        A list of the reference electrodes to be used\n    lfp_band_sampling_rate: int\n    \"\"\"\n    # Error checks on parameters\n    # electrode_list\n\n    lfp_key = {\"merge_id\": lfp_merge_id}\n    lfp_part_table = LFPOutput.merge_get_part(lfp_key)\n\n    query = LFPElectrodeGroup().LFPElectrode() &amp; lfp_key\n    available_electrodes = query.fetch(\"electrode_id\")\n    if not np.all(np.isin(electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in electrode_list must be valid electrode_ids in\"\n            + \" the LFPElectodeGroup table: \"\n            + f\"{electrode_list} not in {available_electrodes}\"\n        )\n    # sampling rate\n    lfp_sampling_rate = LFPOutput.merge_get_parent(lfp_key).fetch1(\n        \"lfp_sampling_rate\"\n    )\n    decimation = lfp_sampling_rate // lfp_band_sampling_rate\n    # filter\n    filter_query = FirFilterParameters() &amp; {\n        \"filter_name\": filter_name,\n        \"filter_sampling_rate\": lfp_sampling_rate,\n    }\n    if not filter_query:\n        raise ValueError(\n            f\"Filter {filter_name}, sampling rate {lfp_sampling_rate} is \"\n            + \"not in the FirFilterParameters table\"\n        )\n    # interval_list\n    interval_query = IntervalList() &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_name\": interval_list_name,\n    }\n    if not interval_query:\n        raise ValueError(\n            f\"interval list {interval_list_name} is not in the IntervalList\"\n            \" table; the list must be added before this function is called\"\n        )\n    # reference_electrode_list\n    if len(reference_electrode_list) != 1 and len(\n        reference_electrode_list\n    ) != len(electrode_list):\n        raise ValueError(\n            \"reference_electrode_list must contain either 1 or \"\n            + \"len(electrode_list) elements\"\n        )\n    # add a -1 element to the list to allow for the no reference option\n    available_electrodes = np.append(available_electrodes, [-1])\n    if not np.all(np.isin(reference_electrode_list, available_electrodes)):\n        raise ValueError(\n            \"All elements in reference_electrode_list must be valid \"\n            \"electrode_ids in the LFPSelection table\"\n        )\n\n    # make a list of all the references\n    ref_list = np.zeros((len(electrode_list),))\n    ref_list[:] = reference_electrode_list\n\n    key = dict(\n        nwb_file_name=nwb_file_name,\n        lfp_merge_id=lfp_merge_id,\n        filter_name=filter_name,\n        filter_sampling_rate=lfp_sampling_rate,\n        target_interval_list_name=interval_list_name,\n        lfp_band_sampling_rate=lfp_sampling_rate // decimation,\n    )\n    # insert an entry into the main LFPBandSelectionTable\n    self.insert1(key, skip_duplicates=True)\n\n    key[\"lfp_electrode_group_name\"] = lfp_part_table.fetch1(\n        \"lfp_electrode_group_name\"\n    )\n    # iterate through all of the new elements and add them\n    for e, r in zip(electrode_list, ref_list):\n        elect_key = (\n            LFPElectrodeGroup.LFPElectrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"lfp_electrode_group_name\": key[\"lfp_electrode_group_name\"],\n                \"electrode_id\": e,\n            }\n        ).fetch1(\"KEY\")\n        for item in elect_key:\n            key[item] = elect_key[item]\n        query = Electrode &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_id\": e,\n        }\n        key[\"reference_elect_id\"] = r\n        self.LFPBandElectrode().insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1", "title": "<code>LFPBandV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>@schema\nclass LFPBandV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; LFPBandSelection              # the LFP band selection\n    ---\n    -&gt; AnalysisNwbfile               # the name of the nwb file with the lfp data\n    -&gt; IntervalList                  # the final interval list of valid times for the data\n    lfp_band_object_id: varchar(40)  # the NWB object ID for loading this object from the file\n    \"\"\"\n\n    def make(self, key):\n        # get the NWB object with the lfp data; FIX: change to fetch with additional infrastructure\n        lfp_key = {\"merge_id\": key[\"lfp_merge_id\"]}\n        lfp_object = (LFPOutput &amp; lfp_key).fetch_nwb()[0][\"lfp\"]\n\n        # get the electrodes to be filtered and their references\n        lfp_band_elect_id, lfp_band_ref_id = (\n            LFPBandSelection().LFPBandElectrode() &amp; key\n        ).fetch(\"electrode_id\", \"reference_elect_id\")\n\n        # sort the electrodes to make sure they are in ascending order\n        lfp_band_elect_id = np.asarray(lfp_band_elect_id)\n        lfp_band_ref_id = np.asarray(lfp_band_ref_id)\n        lfp_sort_order = np.argsort(lfp_band_elect_id)\n        lfp_band_elect_id = lfp_band_elect_id[lfp_sort_order]\n        lfp_band_ref_id = lfp_band_ref_id[lfp_sort_order]\n\n        lfp_sampling_rate, lfp_interval_list = LFPOutput.merge_get_parent(\n            lfp_key\n        ).fetch1(\"lfp_sampling_rate\", \"interval_list_name\")\n        interval_list_name, lfp_band_sampling_rate = (\n            LFPBandSelection() &amp; key\n        ).fetch1(\"target_interval_list_name\", \"lfp_band_sampling_rate\")\n        valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        # the valid_times for this interval may be slightly beyond the valid\n        # times for the lfp itself, so we have to intersect the two lists\n        lfp_valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": lfp_interval_list,\n            }\n        ).fetch1(\"valid_times\")\n        min_length = (LFPBandSelection &amp; key).fetch1(\"min_interval_len\")\n        lfp_band_valid_times = interval_list_intersect(\n            valid_times, lfp_valid_times, min_length=min_length\n        )\n\n        filter_name, filter_sampling_rate, lfp_band_sampling_rate = (\n            LFPBandSelection() &amp; key\n        ).fetch1(\n            \"filter_name\", \"filter_sampling_rate\", \"lfp_band_sampling_rate\"\n        )\n\n        decimation = int(lfp_sampling_rate) // lfp_band_sampling_rate\n\n        # load in the timestamps\n        timestamps = np.asarray(lfp_object.timestamps)\n        # get the indices of the first timestamp and the last timestamp that\n        # are within the valid times\n        included_indices = interval_list_contains_ind(\n            lfp_band_valid_times, timestamps\n        )\n        # pad the indices by 1 on each side to avoid message in filter_data\n        if included_indices[0] &gt; 0:\n            included_indices[0] -= 1\n        if included_indices[-1] != len(timestamps) - 1:\n            included_indices[-1] += 1\n\n        timestamps = timestamps[included_indices[0] : included_indices[-1]]\n\n        # load all the data to speed filtering\n        lfp_data = np.asarray(\n            lfp_object.data[included_indices[0] : included_indices[-1], :],\n            dtype=type(lfp_object.data[0][0]),\n        )\n\n        # get the indices of the electrodes to be filtered and the references\n        lfp_band_elect_index = get_electrode_indices(\n            lfp_object, lfp_band_elect_id\n        )\n        lfp_band_ref_index = get_electrode_indices(lfp_object, lfp_band_ref_id)\n\n        # subtract off the references for the selected channels\n        lfp_data_original = lfp_data.copy()\n        for index, elect_index in enumerate(lfp_band_elect_index):\n            if lfp_band_ref_id[index] != -1:\n                lfp_data[:, elect_index] = (\n                    lfp_data_original[:, elect_index]\n                    - lfp_data_original[:, lfp_band_ref_index[index]]\n                )\n\n        # get the LFP filter that matches the raw data\n        filter = (\n            FirFilterParameters()\n            &amp; {\"filter_name\": filter_name}\n            &amp; {\"filter_sampling_rate\": filter_sampling_rate}\n        ).fetch(as_dict=True)\n\n        filter_coeff = filter[0][\"filter_coeff\"]\n        if len(filter_coeff) == 0:\n            logger.error(\n                \"LFPBand: no filter found with data \"\n                + f\"sampling rate of {lfp_band_sampling_rate}\"\n            )\n            return None\n\n        # create the analysis nwb file to store the results.\n        lfp_band_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n        lfp_band_file_abspath = AnalysisNwbfile().get_abs_path(\n            lfp_band_file_name\n        )\n        # filter the data and write to an the nwb file\n        filtered_data, new_timestamps = FirFilterParameters().filter_data(\n            timestamps,\n            lfp_data,\n            filter_coeff,\n            lfp_band_valid_times,\n            lfp_band_elect_index,\n            decimation,\n        )\n\n        # now that the LFP is filtered, we create an electrical series for it and add it to the file\n        with pynwb.NWBHDF5IO(\n            path=lfp_band_file_abspath, mode=\"a\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            # get the indices of the electrodes in the electrode table of the file to get the right values\n            elect_index = get_electrode_indices(nwbf, lfp_band_elect_id)\n            electrode_table_region = nwbf.create_electrode_table_region(\n                elect_index, \"filtered electrode table\"\n            )\n            eseries_name = \"filtered data\"\n            # TODO: use datatype of data\n            es = pynwb.ecephys.ElectricalSeries(\n                name=eseries_name,\n                data=filtered_data,\n                electrodes=electrode_table_region,\n                timestamps=new_timestamps,\n            )\n            lfp = pynwb.ecephys.LFP(electrical_series=es)\n            ecephys_module = nwbf.create_processing_module(\n                name=\"ecephys\",\n                description=f\"LFP data processed with {filter_name}\",\n            )\n            ecephys_module.add(lfp)\n            io.write(nwbf)\n            lfp_band_object_id = es.object_id\n        #\n        # add the file to the AnalysisNwbfile table\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], lfp_band_file_name)\n        key[\"analysis_file_name\"] = lfp_band_file_name\n        key[\"lfp_band_object_id\"] = lfp_band_object_id\n\n        # finally, we need to censor the valid times to account for the downsampling if this is the first time we've\n        # downsampled these data\n        key[\"interval_list_name\"] = (\n            interval_list_name\n            + \" lfp band \"\n            + str(lfp_band_sampling_rate)\n            + \"Hz\"\n        )\n        tmp_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch(\"valid_times\")\n        if len(tmp_valid_times) == 0:\n            lfp_band_valid_times = interval_list_censor(\n                lfp_band_valid_times, new_timestamps\n            )\n            # add an interval list for the LFP valid times\n            IntervalList.insert1(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"interval_list_name\": key[\"interval_list_name\"],\n                    \"valid_times\": lfp_band_valid_times,\n                    \"pipeline\": \"lfp band\",\n                }\n            )\n        else:\n            lfp_band_valid_times = interval_list_censor(\n                lfp_band_valid_times, new_timestamps\n            )\n            # check that the valid times are the same\n            assert np.isclose(\n                tmp_valid_times[0], lfp_band_valid_times\n            ).all(), (\n                \"previously saved lfp band times do not match current times\"\n            )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        \"\"\"Fetches the filtered data as a dataframe\"\"\"\n        filtered_nwb = self.fetch_nwb()[0]\n        return pd.DataFrame(\n            filtered_nwb[\"lfp_band\"].data,\n            index=pd.Index(filtered_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n        )\n\n    def compute_analytic_signal(self, electrode_list: list[int], **kwargs):\n        \"\"\"Computes the hilbert transform of a given LFPBand signal\n\n        Uses scipy.signal.hilbert to compute the hilbert transform of the signal\n\n        Parameters\n        ----------\n        electrode_list: list[int]\n            A list of the electrodes to compute the hilbert transform of\n\n        Returns\n        -------\n        analytic_signal_df: pd.DataFrame\n            DataFrame containing hilbert transform of signal\n\n        Raises\n        ------\n        ValueError\n            If items in electrode_list are invalid for the dataset\n        \"\"\"\n\n        filtered_band = self.fetch_nwb()[0][\"lfp_band\"]\n        electrode_index = np.isin(\n            filtered_band.electrodes.data[:], electrode_list\n        )\n        if len(electrode_list) != np.sum(electrode_index):\n            raise ValueError(\n                \"Some of the electrodes specified in electrode_list are missing\"\n                + \" in the current LFPBand table.\"\n            )\n        analytic_signal_df = pd.DataFrame(\n            hilbert(filtered_band.data[:, electrode_index], axis=0),\n            index=pd.Index(filtered_band.timestamps, name=\"time\"),\n            columns=[f\"electrode {e}\" for e in electrode_list],\n        )\n        return analytic_signal_df\n\n    def compute_signal_phase(\n        self, electrode_list: list[int] = None, **kwargs\n    ) -&gt; pd.DataFrame:\n        \"\"\"Computes the phase of a given LFPBand signals using the hilbert transform\n\n        Parameters\n        ----------\n        electrode_list : list[int], optional\n            A list of the electrodes to compute the phase of, by default None\n\n        Returns\n        -------\n        signal_phase_df : pd.DataFrame\n            DataFrame containing the phase of the signals\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = []\n\n        analytic_signal_df = self.compute_analytic_signal(\n            electrode_list, **kwargs\n        )\n\n        return pd.DataFrame(\n            np.angle(analytic_signal_df) + np.pi,\n            columns=analytic_signal_df.columns,\n            index=analytic_signal_df.index,\n        )\n\n    def compute_signal_power(\n        self, electrode_list: list[int] = None, **kwargs\n    ) -&gt; pd.DataFrame:\n        \"\"\"Computes the power of a given LFPBand signals using the hilbert transform\n\n        Parameters\n        ----------\n        electrode_list : list[int], optional\n            A list of the electrodes to compute the power of, by default None\n\n        Returns\n        -------\n        signal_power_df : pd.DataFrame\n            DataFrame containing the power of the signals\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = []\n\n        analytic_signal_df = self.compute_analytic_signal(\n            electrode_list, **kwargs\n        )\n\n        return pd.DataFrame(\n            np.abs(analytic_signal_df) ** 2,\n            columns=analytic_signal_df.columns,\n            index=analytic_signal_df.index,\n        )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.fetch1_dataframe", "title": "<code>fetch1_dataframe(*attrs, **kwargs)</code>", "text": "<p>Fetches the filtered data as a dataframe</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def fetch1_dataframe(self, *attrs, **kwargs):\n    \"\"\"Fetches the filtered data as a dataframe\"\"\"\n    filtered_nwb = self.fetch_nwb()[0]\n    return pd.DataFrame(\n        filtered_nwb[\"lfp_band\"].data,\n        index=pd.Index(filtered_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n    )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_analytic_signal", "title": "<code>compute_analytic_signal(electrode_list, **kwargs)</code>", "text": "<p>Computes the hilbert transform of a given LFPBand signal</p> <p>Uses scipy.signal.hilbert to compute the hilbert transform of the signal</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the hilbert transform of</p> required <p>Returns:</p> Name Type Description <code>analytic_signal_df</code> <code>DataFrame</code> <p>DataFrame containing hilbert transform of signal</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>If items in electrode_list are invalid for the dataset</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_analytic_signal(self, electrode_list: list[int], **kwargs):\n    \"\"\"Computes the hilbert transform of a given LFPBand signal\n\n    Uses scipy.signal.hilbert to compute the hilbert transform of the signal\n\n    Parameters\n    ----------\n    electrode_list: list[int]\n        A list of the electrodes to compute the hilbert transform of\n\n    Returns\n    -------\n    analytic_signal_df: pd.DataFrame\n        DataFrame containing hilbert transform of signal\n\n    Raises\n    ------\n    ValueError\n        If items in electrode_list are invalid for the dataset\n    \"\"\"\n\n    filtered_band = self.fetch_nwb()[0][\"lfp_band\"]\n    electrode_index = np.isin(\n        filtered_band.electrodes.data[:], electrode_list\n    )\n    if len(electrode_list) != np.sum(electrode_index):\n        raise ValueError(\n            \"Some of the electrodes specified in electrode_list are missing\"\n            + \" in the current LFPBand table.\"\n        )\n    analytic_signal_df = pd.DataFrame(\n        hilbert(filtered_band.data[:, electrode_index], axis=0),\n        index=pd.Index(filtered_band.timestamps, name=\"time\"),\n        columns=[f\"electrode {e}\" for e in electrode_list],\n    )\n    return analytic_signal_df\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_signal_phase", "title": "<code>compute_signal_phase(electrode_list=None, **kwargs)</code>", "text": "<p>Computes the phase of a given LFPBand signals using the hilbert transform</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the phase of, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>signal_phase_df</code> <code>DataFrame</code> <p>DataFrame containing the phase of the signals</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_signal_phase(\n    self, electrode_list: list[int] = None, **kwargs\n) -&gt; pd.DataFrame:\n    \"\"\"Computes the phase of a given LFPBand signals using the hilbert transform\n\n    Parameters\n    ----------\n    electrode_list : list[int], optional\n        A list of the electrodes to compute the phase of, by default None\n\n    Returns\n    -------\n    signal_phase_df : pd.DataFrame\n        DataFrame containing the phase of the signals\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = []\n\n    analytic_signal_df = self.compute_analytic_signal(\n        electrode_list, **kwargs\n    )\n\n    return pd.DataFrame(\n        np.angle(analytic_signal_df) + np.pi,\n        columns=analytic_signal_df.columns,\n        index=analytic_signal_df.index,\n    )\n</code></pre>"}, {"location": "api/lfp/analysis/v1/lfp_band/#src.spyglass.lfp.analysis.v1.lfp_band.LFPBandV1.compute_signal_power", "title": "<code>compute_signal_power(electrode_list=None, **kwargs)</code>", "text": "<p>Computes the power of a given LFPBand signals using the hilbert transform</p> <p>Parameters:</p> Name Type Description Default <code>electrode_list</code> <code>list[int]</code> <p>A list of the electrodes to compute the power of, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>signal_power_df</code> <code>DataFrame</code> <p>DataFrame containing the power of the signals</p> Source code in <code>src/spyglass/lfp/analysis/v1/lfp_band.py</code> <pre><code>def compute_signal_power(\n    self, electrode_list: list[int] = None, **kwargs\n) -&gt; pd.DataFrame:\n    \"\"\"Computes the power of a given LFPBand signals using the hilbert transform\n\n    Parameters\n    ----------\n    electrode_list : list[int], optional\n        A list of the electrodes to compute the power of, by default None\n\n    Returns\n    -------\n    signal_power_df : pd.DataFrame\n        DataFrame containing the power of the signals\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = []\n\n    analytic_signal_df = self.compute_analytic_signal(\n        electrode_list, **kwargs\n    )\n\n    return pd.DataFrame(\n        np.abs(analytic_signal_df) ** 2,\n        columns=analytic_signal_df.columns,\n        index=analytic_signal_df.index,\n    )\n</code></pre>"}, {"location": "api/lfp/v1/lfp/", "title": "lfp.py", "text": ""}, {"location": "api/lfp/v1/lfp/#src.spyglass.lfp.v1.lfp.LFPSelection", "title": "<code>LFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>The user's selection of LFP data to be filtered</p> <p>This table is used to select the LFP data to be filtered.  The user can select the LFP data by specifying the electrode group and the interval list to be used. The interval list is used to select the times from the raw data that will be filtered.  The user can also specify the filter to be used.</p> <p>The LFP data is filtered and downsampled to the user-defined sampling rate, specified as lfp_sampling_rate.  The filtered data is stored in the AnalysisNwbfile table. The valid times for the filtered data are stored in the IntervalList table.</p> Source code in <code>src/spyglass/lfp/v1/lfp.py</code> <pre><code>@schema\nclass LFPSelection(SpyglassMixin, dj.Manual):\n    \"\"\"The user's selection of LFP data to be filtered\n\n    This table is used to select the LFP data to be filtered.  The user can select\n    the LFP data by specifying the electrode group and the interval list to be used.\n    The interval list is used to select the times from the raw data that will be\n    filtered.  The user can also specify the filter to be used.\n\n    The LFP data is filtered and downsampled to the user-defined sampling rate, specified\n    as lfp_sampling_rate.  The filtered data is stored in the AnalysisNwbfile table.\n    The valid times for the filtered data are stored in the IntervalList table.\n    \"\"\"\n\n    definition = \"\"\"\n     -&gt; LFPElectrodeGroup                                                  # the group of electrodes to be filtered\n     -&gt; IntervalList.proj(target_interval_list_name='interval_list_name')  # the original set of times to be filtered\n     -&gt; FirFilterParameters                                                # the filter to be used\n     ---\n     target_sampling_rate = 1000 : float                                     # the desired output sampling rate, in HZ\n     \"\"\"\n</code></pre>"}, {"location": "api/lfp/v1/lfp/#src.spyglass.lfp.v1.lfp.LFPV1", "title": "<code>LFPV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>The filtered LFP data</p> Source code in <code>src/spyglass/lfp/v1/lfp.py</code> <pre><code>@schema\nclass LFPV1(SpyglassMixin, dj.Computed):\n    \"\"\"The filtered LFP data\"\"\"\n\n    definition = \"\"\"\n    -&gt; LFPSelection             # the user's selection of LFP data to be filtered\n    ---\n    -&gt; AnalysisNwbfile          # the name of the nwb file with the lfp data\n    -&gt; IntervalList             # the final interval list of valid times for the data\n    lfp_object_id: varchar(40)  # the NWB object ID for loading this object from the file\n    lfp_sampling_rate: float    # the sampling rate, in HZ\n    \"\"\"\n\n    def make(self, key):\n        # get the NWB object with the data\n        nwbf_key = {\"nwb_file_name\": key[\"nwb_file_name\"]}\n        rawdata = (Raw &amp; nwbf_key).fetch_nwb()[0][\"raw\"]\n\n        # CBroz: assumes Raw sampling rate matches FirFilterParameters set?\n        #        if we just pull rate from Raw, why include in Param table?\n        sampling_rate, raw_interval_list_name = (Raw &amp; nwbf_key).fetch1(\n            \"sampling_rate\", \"interval_list_name\"\n        )\n        sampling_rate = int(np.round(sampling_rate))\n        target_sampling_rate = (LFPSelection &amp; key).fetch1(\n            \"target_sampling_rate\"\n        )\n\n        # to get the list of valid times, we need to combine those from the\n        # user with those from the raw data\n        orig_key = copy.deepcopy(key)\n        orig_key[\"interval_list_name\"] = key[\"target_interval_list_name\"]\n        user_valid_times = (IntervalList() &amp; orig_key).fetch1(\"valid_times\")\n        # we remove the extra entry so we can insert this into the LFPOutput table.\n        del orig_key[\"interval_list_name\"]\n\n        raw_valid_times = (\n            IntervalList()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": raw_interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n        valid_times = interval_list_intersect(\n            user_valid_times,\n            raw_valid_times,\n            min_length=MIN_LFP_INTERVAL_DURATION,\n        )\n        logger.info(\n            f\"LFP: found {len(valid_times)} intervals &gt; \"\n            + f\"{MIN_LFP_INTERVAL_DURATION} sec long.\"\n        )\n        # target user-specified sampling rate\n        decimation = int(sampling_rate // target_sampling_rate)\n\n        # get the LFP filter that matches the raw data\n        filter = (\n            FirFilterParameters()\n            &amp; {\n                \"filter_name\": key[\"filter_name\"],\n                \"filter_sampling_rate\": sampling_rate,\n            }  # not key['filter_sampling_rate']?\n        ).fetch(as_dict=True)[0]\n\n        # there should only be one filter that matches, so we take the first of\n        # the dictionaries\n\n        key[\"filter_name\"] = filter[\"filter_name\"]\n        key[\"filter_sampling_rate\"] = filter[\"filter_sampling_rate\"]\n\n        filter_coeff = filter[\"filter_coeff\"]\n        if len(filter_coeff) == 0:\n            logger.error(\n                \"LFP: no filter found with data sampling rate of \"\n                + f\"{sampling_rate}\"\n            )\n            return None  # See #849\n\n        # get the list of selected LFP Channels from LFPElectrode\n        electrode_keys = (LFPElectrodeGroup.LFPElectrode &amp; key).fetch(\"KEY\")\n        electrode_id_list = list(k[\"electrode_id\"] for k in electrode_keys)\n        electrode_id_list.sort()\n\n        lfp_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        lfp_file_abspath = AnalysisNwbfile().get_abs_path(lfp_file_name)\n        (\n            lfp_object_id,\n            timestamp_interval,\n        ) = FirFilterParameters().filter_data_nwb(\n            lfp_file_abspath,\n            rawdata,\n            filter_coeff,\n            valid_times,\n            electrode_id_list,\n            decimation,\n        )\n\n        # now that the LFP is filtered and in the file, add the file to the\n        # AnalysisNwbfile table\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], lfp_file_name)\n\n        key[\"analysis_file_name\"] = lfp_file_name\n        key[\"lfp_object_id\"] = lfp_object_id\n        key[\"lfp_sampling_rate\"] = sampling_rate // decimation\n\n        # need to censor the valid times to account for the downsampling\n        lfp_valid_times = interval_list_censor(valid_times, timestamp_interval)\n\n        # add an interval list for the LFP valid times, or check that it matches the existing one\n        key[\"interval_list_name\"] = \"_\".join(\n            (\n                \"lfp\",\n                key[\"lfp_electrode_group_name\"],\n                key[\"target_interval_list_name\"],\n                \"valid times\",\n            )\n        )\n\n        tmp_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch(\"valid_times\")\n        if len(tmp_valid_times) == 0:\n            IntervalList.insert1(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"interval_list_name\": key[\"interval_list_name\"],\n                    \"valid_times\": lfp_valid_times,\n                    \"pipeline\": \"lfp_v1\",\n                },\n                replace=True,\n            )\n        elif not np.allclose(tmp_valid_times[0], lfp_valid_times):\n            raise ValueError(\n                \"previously saved lfp times do not match current times\"\n            )\n        self.insert1(key)\n\n        # finally, we insert this into the LFP output table.\n        from spyglass.lfp.lfp_merge import LFPOutput\n\n        orig_key[\"analysis_file_name\"] = lfp_file_name\n        orig_key[\"lfp_object_id\"] = lfp_object_id\n        LFPOutput.insert1(orig_key)\n\n    def fetch1_dataframe(self, *attrs, **kwargs):\n        nwb_lfp = self.fetch_nwb()[0]\n        return pd.DataFrame(\n            nwb_lfp[\"lfp\"].data,\n            index=pd.Index(nwb_lfp[\"lfp\"].timestamps, name=\"time\"),\n        )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact/", "title": "lfp_artifact.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact/#src.spyglass.lfp.v1.lfp_artifact.LFPArtifactDetectionParameters", "title": "<code>LFPArtifactDetectionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact.py</code> <pre><code>@schema\nclass LFPArtifactDetectionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for detecting LFP artifact times within a LFP group.\n    artifact_params_name: varchar(64)\n    ---\n    artifact_params: blob  # dictionary of parameters\n    \"\"\"\n\n    # See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Insert the default artifact parameters.\"\"\"\n        diff_params = [\n            \"default_difference\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": 0.1,\n                    \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": 0.05,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                    \"local_window_ms\": 40.0,  # in milliseconds\n                },\n            },\n        ]\n\n        diff_ref_params = [\n            \"default_difference_ref\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": 0.1,\n                    \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": 0.05,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                    \"local_window_ms\": 40.0,  # in milliseconds\n                },\n                \"referencing\": {\n                    \"ref_on\": 1,\n                    \"reference_list\": [0, 0, 0, 0, 0],\n                    \"electrode_list\": [0, 0],\n                },\n            },\n        ]\n\n        no_params = [\n            \"none\",\n            {\n                \"artifact_detection_algorithm\": \"difference\",\n                \"artifact_detection_algorithm_params\": {\n                    \"amplitude_thresh_1st\": None,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_1st\": None,\n                    \"amplitude_thresh_2nd\": None,  # must be None or &gt;= 0\n                    \"proportion_above_thresh_2nd\": None,\n                    \"removal_window_ms\": None,  # in milliseconds\n                    \"local_window_ms\": None,  # in milliseconds\n                },\n            },\n        ]\n\n        mad_params = [\n            \"default_mad\",\n            {\n                \"artifact_detection_algorithm\": \"mad\",\n                \"artifact_detection_algorithm_params\": {\n                    # akin to z-score std dev if the distribution is normal\n                    \"mad_thresh\": 6.0,\n                    \"proportion_above_thresh\": 0.1,\n                    \"removal_window_ms\": 10.0,  # in milliseconds\n                },\n            },\n        ]\n\n        self.insert(\n            [diff_params, diff_ref_params, no_params, mad_params],\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact/#src.spyglass.lfp.v1.lfp_artifact.LFPArtifactDetectionParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default artifact parameters.</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default artifact parameters.\"\"\"\n    diff_params = [\n        \"default_difference\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": 0.1,\n                \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": 0.05,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n                \"local_window_ms\": 40.0,  # in milliseconds\n            },\n        },\n    ]\n\n    diff_ref_params = [\n        \"default_difference_ref\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": 500,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": 0.1,\n                \"amplitude_thresh_2nd\": 1000,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": 0.05,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n                \"local_window_ms\": 40.0,  # in milliseconds\n            },\n            \"referencing\": {\n                \"ref_on\": 1,\n                \"reference_list\": [0, 0, 0, 0, 0],\n                \"electrode_list\": [0, 0],\n            },\n        },\n    ]\n\n    no_params = [\n        \"none\",\n        {\n            \"artifact_detection_algorithm\": \"difference\",\n            \"artifact_detection_algorithm_params\": {\n                \"amplitude_thresh_1st\": None,  # must be None or &gt;= 0\n                \"proportion_above_thresh_1st\": None,\n                \"amplitude_thresh_2nd\": None,  # must be None or &gt;= 0\n                \"proportion_above_thresh_2nd\": None,\n                \"removal_window_ms\": None,  # in milliseconds\n                \"local_window_ms\": None,  # in milliseconds\n            },\n        },\n    ]\n\n    mad_params = [\n        \"default_mad\",\n        {\n            \"artifact_detection_algorithm\": \"mad\",\n            \"artifact_detection_algorithm_params\": {\n                # akin to z-score std dev if the distribution is normal\n                \"mad_thresh\": 6.0,\n                \"proportion_above_thresh\": 0.1,\n                \"removal_window_ms\": 10.0,  # in milliseconds\n            },\n        },\n    ]\n\n    self.insert(\n        [diff_params, diff_ref_params, no_params, mad_params],\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact_MAD_detection/", "title": "lfp_artifact_MAD_detection.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact_MAD_detection/#src.spyglass.lfp.v1.lfp_artifact_MAD_detection.mad_artifact_detector", "title": "<code>mad_artifact_detector(recording, mad_thresh=6.0, proportion_above_thresh=0.1, removal_window_ms=10.0, sampling_frequency=1000.0, *args, **kwargs)</code>", "text": "<p>Detect LFP artifacts using the median absolute deviation method.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>RecordingExtractor</code> <p>The recording extractor object</p> required <code>mad_thresh</code> <code>float</code> <p>Threshold on the median absolute deviation scaled LFPs, defaults to 6.0</p> <code>6.0</code> <code>proportion_above_thresh</code> <code>float</code> <p>Proportion of electrodes that need to be above the threshold, defaults to 1.0</p> <code>0.1</code> <code>removal_window_ms</code> <code>float</code> <p>Width of the window in milliseconds to mask out per artifact (window/2 removed on each side of threshold crossing), defaults to 1 ms</p> <code>10.0</code> <code>sampling_frequency</code> <code>float</code> <p>Sampling frequency of the recording extractor, defaults to 1000.0</p> <code>1000.0</code> <p>Returns:</p> Name Type Description <code>artifact_removed_valid_times</code> <code>ndarray</code> <p>Intervals of valid times where artifacts were not detected, unit: seconds</p> <code>artifact_intervals</code> <code>ndarray</code> <p>Intervals in which artifacts are detected (including removal windows), unit: seconds</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact_MAD_detection.py</code> <pre><code>def mad_artifact_detector(\n    recording: None,\n    mad_thresh: float = 6.0,\n    proportion_above_thresh: float = 0.1,\n    removal_window_ms: float = 10.0,\n    sampling_frequency: float = 1000.0,\n    *args,\n    **kwargs,\n) -&gt; tuple[np.ndarray, np.ndarray]:\n    \"\"\"Detect LFP artifacts using the median absolute deviation method.\n\n    Parameters\n    ----------\n    recording : RecordingExtractor\n        The recording extractor object\n    mad_thresh : float, optional\n        Threshold on the median absolute deviation scaled LFPs, defaults to 6.0\n    proportion_above_thresh : float, optional\n        Proportion of electrodes that need to be above the threshold, defaults to 1.0\n    removal_window_ms : float, optional\n        Width of the window in milliseconds to mask out per artifact\n        (window/2 removed on each side of threshold crossing), defaults to 1 ms\n    sampling_frequency : float, optional\n        Sampling frequency of the recording extractor, defaults to 1000.0\n\n    Returns\n    -------\n    artifact_removed_valid_times : np.ndarray\n        Intervals of valid times where artifacts were not detected, unit: seconds\n    artifact_intervals : np.ndarray\n        Intervals in which artifacts are detected (including removal windows), unit: seconds\n\n    \"\"\"\n\n    timestamps = np.asarray(recording.timestamps)\n    lfps = np.asarray(recording.data)\n\n    mad = median_abs_deviation(lfps, axis=0, nan_policy=\"omit\", scale=\"normal\")\n    is_artifact = _is_above_proportion_thresh(\n        _mad_scale_lfps(lfps, mad), mad_thresh, proportion_above_thresh\n    )\n\n    MILLISECONDS_PER_SECOND = 1000.0\n    half_removal_window_s = (removal_window_ms / MILLISECONDS_PER_SECOND) * 0.5\n    half_removal_window_idx = int(half_removal_window_s * sampling_frequency)\n    is_artifact = _extend_array_by_window(is_artifact, half_removal_window_idx)\n\n    artifact_intervals_s = _get_time_intervals_from_bool_array(\n        is_artifact, timestamps\n    )\n\n    valid_times = _get_time_intervals_from_bool_array(~is_artifact, timestamps)\n\n    return valid_times, artifact_intervals_s\n</code></pre>"}, {"location": "api/lfp/v1/lfp_artifact_difference_detection/", "title": "lfp_artifact_difference_detection.py", "text": ""}, {"location": "api/lfp/v1/lfp_artifact_difference_detection/#src.spyglass.lfp.v1.lfp_artifact_difference_detection.difference_artifact_detector", "title": "<code>difference_artifact_detector(recording, timestamps, amplitude_thresh_1st=None, amplitude_thresh_2nd=None, proportion_above_thresh_1st=1.0, proportion_above_thresh_2nd=1.0, removal_window_ms=1.0, local_window_ms=1.0, sampling_frequency=1000.0, referencing=False)</code>", "text": "<p>Detects times during which artifacts do and do not occur.</p> <p>Artifacts are defined as periods where the absolute value of the change in LFP exceeds amplitude change thresholds on the proportion of channels specified, with the period extended by the removal_window_ms/2 on each side. amplitude change threshold values of None are ignored.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>lfp eseries zscore_thresh : float</code> <p>Stdev threshold for exclusion, should be &gt;=0, defaults to None</p> required <code>amplitude_thresh_1st</code> <code>float</code> <p>Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to None</p> <code>None</code> <code>amplitude_thresh_2nd</code> <code>float</code> <p>Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to None</p> <code>None</code> <code>proportion_above_thresh_1st</code> <code>float, optional, should be&gt;0 and &lt;=1</code> <p>Proportion of electrodes that need to have threshold crossings, defaults to 1</p> <code>1.0</code> <code>proportion_above_thresh_2nd</code> <code>float, optional, should be&gt;0 and &lt;=1</code> <p>Proportion of electrodes that need to have threshold crossings, defaults to 1</p> <code>1.0</code> <code>removal_window_ms</code> <code>float</code> <p>Width of the window in milliseconds to mask out per artifact (window/2 removed on each side of threshold crossing), defaults to 1 ms</p> <code>1.0</code> <code>referencing</code> <code>bool</code> <p>Whether or not the data passed to this function is referenced, defaults to False</p> <code>False</code> <p>Returns:</p> Name Type Description <code>artifact_removed_valid_times</code> <code>ndarray</code> <p>Intervals of valid times where artifacts were not detected, unit: seconds</p> <code>artifact_intervals</code> <code>ndarray</code> <p>Intervals in which artifacts are detected (including removal windows), unit: seconds</p> Source code in <code>src/spyglass/lfp/v1/lfp_artifact_difference_detection.py</code> <pre><code>def difference_artifact_detector(\n    recording: None,\n    timestamps: None,\n    amplitude_thresh_1st: Union[float, None] = None,\n    amplitude_thresh_2nd: Union[float, None] = None,\n    proportion_above_thresh_1st: float = 1.0,\n    proportion_above_thresh_2nd: float = 1.0,\n    removal_window_ms: float = 1.0,\n    local_window_ms: float = 1.0,\n    sampling_frequency: float = 1000.0,\n    referencing: bool = False,\n):\n    \"\"\"Detects times during which artifacts do and do not occur.\n\n    Artifacts are defined as periods where the absolute value of the change in\n    LFP exceeds amplitude change thresholds on the proportion of channels\n    specified, with the period extended by the removal_window_ms/2 on each side.\n    amplitude change threshold values of None are ignored.\n\n    Parameters\n    ----------\n    recording : lfp eseries zscore_thresh : float, optional\n        Stdev threshold for exclusion, should be &gt;=0, defaults to None\n    amplitude_thresh_1st : float, optional\n        Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to\n        None\n    amplitude_thresh_2nd : float, optional\n        Amplitude (ad units) threshold for exclusion, should be &gt;=0, defaults to\n        None\n    proportion_above_thresh_1st : float, optional, should be&gt;0 and &lt;=1\n        Proportion of electrodes that need to have threshold crossings, defaults\n        to 1\n    proportion_above_thresh_2nd : float, optional, should be&gt;0 and &lt;=1\n        Proportion of electrodes that need to have threshold crossings, defaults\n        to 1\n    removal_window_ms : float, optional\n        Width of the window in milliseconds to mask out per artifact (window/2\n        removed on each side of threshold crossing), defaults to 1 ms\n    referencing : bool, optional\n        Whether or not the data passed to this function is referenced, defaults to False\n\n    Returns\n    -------\n    artifact_removed_valid_times : np.ndarray\n        Intervals of valid times where artifacts were not detected, unit:\n        seconds\n    artifact_intervals : np.ndarray\n        Intervals in which artifacts are detected (including removal windows),\n        unit: seconds\n    \"\"\"\n\n    # NOTE: 7-17-23 updated to remove recording.data, since it will converted to\n    # numpy array before referencing check for referencing flag\n\n    if referencing:\n        logger.info(\"referencing activated. may be set to -1\")\n\n    # valid_timestamps = recording.timestamps\n    valid_timestamps = timestamps\n\n    local_window = int(local_window_ms / 2)\n\n    # if both thresholds are None, we skip artifact detection\n    if amplitude_thresh_1st is None:\n        recording_interval = np.asarray(\n            [valid_timestamps[0], valid_timestamps[-1]]\n        )\n        artifact_times_empty = np.asarray([])\n        logger.info(\"Amplitude threshold is None, skipping artifact detection\")\n        return recording_interval, artifact_times_empty\n\n    # verify threshold parameters\n    (\n        amplitude_thresh_1st,\n        amplitude_thresh_2nd,\n        proportion_above_thresh_1st,\n        proportion_above_thresh_2nd,\n    ) = _check_artifact_thresholds(\n        amplitude_thresh_1st,\n        amplitude_thresh_2nd,\n        proportion_above_thresh_1st,\n        proportion_above_thresh_2nd,\n    )\n\n    # want to detect frames without parallel processing\n    # compute the number of electrodes that have to be above threshold\n    nelect_above_1st = np.ceil(proportion_above_thresh_1st * recording.shape[1])\n    nelect_above_2nd = np.ceil(proportion_above_thresh_2nd * recording.shape[1])\n    logger.info(\"num tets 1\", nelect_above_1st, \"num tets 2\", nelect_above_2nd)\n    logger.info(\"data shape\", recording.shape)\n\n    # find the artifact occurrences using one or both thresholds, across\n    # channels\n\n    if amplitude_thresh_1st is not None:\n        # first find times with large amp change: sum diff over several timebins\n\n        diff_array = np.diff(recording, axis=0)\n        window = np.ones((3, 1)) if referencing else np.ones((15, 1))\n\n        # sum differences over bins using convolution for speed\n        width = int((window.size - 1) / 2)\n        diff_array = np.pad(\n            diff_array,\n            pad_width=((width, width), (0, 0)),\n            mode=\"constant\",\n        )\n        diff_array_5 = scipy.signal.convolve(diff_array, window, mode=\"valid\")\n\n        artifact_times_all_5 = np.sum(\n            (np.abs(diff_array_5) &gt; amplitude_thresh_1st), axis=1\n        )\n\n        above_thresh_1st = np.where(artifact_times_all_5 &gt;= nelect_above_1st)[0]\n\n        # second, find artifacts with large baseline change\n        logger.info(\"thresh\", amplitude_thresh_2nd, \"window\", local_window)\n\n        big_artifacts = np.zeros(\n            (recording.shape[1], above_thresh_1st.shape[0])\n        )\n        for art_count in np.arange(above_thresh_1st.shape[0]):\n            if above_thresh_1st[art_count] &lt;= local_window:\n                local_min = local_max = above_thresh_1st[art_count]\n            else:\n                local_max = np.max(\n                    recording[\n                        above_thresh_1st[art_count]\n                        - local_window : above_thresh_1st[art_count]\n                        + local_window,\n                        :,\n                    ],\n                    axis=0,\n                )\n                local_min = np.min(\n                    recording[\n                        above_thresh_1st[art_count]\n                        - local_window : above_thresh_1st[art_count]\n                        + local_window,\n                        :,\n                    ],\n                    axis=0,\n                )\n            big_artifacts[:, art_count] = (\n                np.abs(local_max - local_min) &gt; amplitude_thresh_2nd\n            )\n\n        # sum columns in big artficat, then compare to nelect_above_2nd\n        above_thresh = above_thresh_1st[\n            np.sum(big_artifacts, axis=0) &gt;= nelect_above_2nd\n        ]\n\n    artifact_frames = above_thresh.copy()\n    logger.info(\"detected \", artifact_frames.shape[0], \" artifacts\")\n\n    # Convert to s to remove from either side of each detected artifact\n    half_removal_window_s = removal_window_ms / 1000 * 0.5\n\n    if len(artifact_frames) == 0:\n        recording_interval = np.asarray(\n            [[valid_timestamps[0], valid_timestamps[-1]]]\n        )\n        artifact_times_empty = np.asarray([])\n        logger.info(\"No artifacts detected.\")\n        return recording_interval, artifact_times_empty\n\n    artifact_intervals = interval_from_inds(artifact_frames)\n\n    artifact_intervals_s = np.zeros(\n        (len(artifact_intervals), 2), dtype=np.float64\n    )\n\n    for interval_idx, interval in enumerate(artifact_intervals):\n        artifact_intervals_s[interval_idx] = [\n            valid_timestamps[interval[0]] - half_removal_window_s,\n            valid_timestamps[interval[1]] + half_removal_window_s,\n        ]\n    artifact_intervals_s = reduce(_union_concat, artifact_intervals_s)\n\n    valid_intervals = get_valid_intervals(\n        valid_timestamps, sampling_frequency, 1.5, 0.000001\n    )\n\n    # these are artifact times - need to subtract these from valid timestamps\n    artifact_valid_times = interval_list_intersect(\n        valid_intervals, artifact_intervals_s\n    )\n\n    # note: this is a slow step\n    list_triggers = []\n    for interval in artifact_valid_times:\n        list_triggers.append(\n            np.arange(\n                np.searchsorted(valid_timestamps, interval[0]),\n                np.searchsorted(valid_timestamps, interval[1]),\n            )\n        )\n\n    new_array = np.array(np.concatenate(list_triggers))\n\n    new_timestamps = np.delete(valid_timestamps, new_array)\n\n    artifact_removed_valid_times = get_valid_intervals(\n        new_timestamps, sampling_frequency, 1.5, 0.000001\n    )\n\n    return artifact_removed_valid_times, artifact_intervals_s\n</code></pre>"}, {"location": "api/linearization/merge/", "title": "merge.py", "text": ""}, {"location": "api/linearization/v0/main/", "title": "main.py", "text": ""}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.LinearizationParameters", "title": "<code>LinearizationParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Choose whether to use an HMM to linearize position.</p> <p>This can help when the euclidean distances between separate arms are too close and the previous position has some information about which arm the animal is on.</p> <p>route_euclidean_distance_scaling: How much to prefer route distances between successive time points that are closer to the euclidean distance. Smaller numbers mean the route distance is more likely to be close to the euclidean distance.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass LinearizationParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Choose whether to use an HMM to linearize position.\n\n    This can help when the euclidean distances between separate arms are too\n    close and the previous position has some information about which arm the\n    animal is on.\n\n    route_euclidean_distance_scaling: How much to prefer route distances between\n    successive time points that are closer to the euclidean distance. Smaller\n    numbers mean the route distance is more likely to be close to the euclidean\n    distance.\n    \"\"\"\n\n    definition = \"\"\"\n    linearization_param_name : varchar(80)   # name for this set of parameters\n    ---\n    use_hmm = 0 : int   # use HMM to determine linearization\n    route_euclidean_distance_scaling = 1.0 : float # Preference for euclidean.\n    sensor_std_dev = 5.0 : float   # Uncertainty of position sensor (in cm).\n    # Biases the transition matrix to prefer the current track segment.\n    diagonal_bias = 0.5 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph", "title": "<code>TrackGraph</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Graph representation of track representing the spatial environment.</p> <p>Used for linearizing position.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass TrackGraph(SpyglassMixin, dj.Manual):\n    \"\"\"Graph representation of track representing the spatial environment.\n\n    Used for linearizing position.\n    \"\"\"\n\n    definition = \"\"\"\n    track_graph_name : varchar(80)\n    ----\n    environment : varchar(80)  # Type of Environment\n    node_positions : blob      # 2D position of nodes, (n_nodes, 2)\n    edges: blob                # shape (n_edges, 2)\n    linear_edge_order : blob   # order of edges in linear space, (n_edges, 2)\n    linear_edge_spacing : blob # space btwn edges in linear space, (n_edges,)\n    \"\"\"\n\n    def get_networkx_track_graph(self, track_graph_parameters=None):\n        if track_graph_parameters is None:\n            track_graph_parameters = self.fetch1()\n        return make_track_graph(\n            node_positions=track_graph_parameters[\"node_positions\"],\n            edges=track_graph_parameters[\"edges\"],\n        )\n\n    def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n        \"\"\"Plot the track graph in 2D position space.\"\"\"\n        track_graph = self.get_networkx_track_graph()\n        plot_track_graph(\n            track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n        )\n\n    def plot_track_graph_as_1D(\n        self,\n        ax=None,\n        axis=\"x\",\n        other_axis_start=0.0,\n        draw_edge_labels=False,\n        node_size=300,\n        node_color=\"#1f77b4\",\n    ):\n        \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n        track_graph_parameters = self.fetch1()\n        track_graph = self.get_networkx_track_graph(\n            track_graph_parameters=track_graph_parameters\n        )\n        plot_graph_as_1D(\n            track_graph,\n            edge_order=track_graph_parameters[\"linear_edge_order\"],\n            edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n            ax=ax,\n            axis=axis,\n            other_axis_start=other_axis_start,\n            draw_edge_labels=draw_edge_labels,\n            node_size=node_size,\n            node_color=node_color,\n        )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph.plot_track_graph", "title": "<code>plot_track_graph(ax=None, draw_edge_labels=False, **kwds)</code>", "text": "<p>Plot the track graph in 2D position space.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n    \"\"\"Plot the track graph in 2D position space.\"\"\"\n    track_graph = self.get_networkx_track_graph()\n    plot_track_graph(\n        track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n    )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.TrackGraph.plot_track_graph_as_1D", "title": "<code>plot_track_graph_as_1D(ax=None, axis='x', other_axis_start=0.0, draw_edge_labels=False, node_size=300, node_color='#1f77b4')</code>", "text": "<p>Plot the track graph in 1D to see how the linearization is set up.</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>def plot_track_graph_as_1D(\n    self,\n    ax=None,\n    axis=\"x\",\n    other_axis_start=0.0,\n    draw_edge_labels=False,\n    node_size=300,\n    node_color=\"#1f77b4\",\n):\n    \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n    track_graph_parameters = self.fetch1()\n    track_graph = self.get_networkx_track_graph(\n        track_graph_parameters=track_graph_parameters\n    )\n    plot_graph_as_1D(\n        track_graph,\n        edge_order=track_graph_parameters[\"linear_edge_order\"],\n        edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n        ax=ax,\n        axis=axis,\n        other_axis_start=other_axis_start,\n        draw_edge_labels=draw_edge_labels,\n        node_size=node_size,\n        node_color=node_color,\n    )\n</code></pre>"}, {"location": "api/linearization/v0/main/#src.spyglass.linearization.v0.main.IntervalLinearizedPosition", "title": "<code>IntervalLinearizedPosition</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Linearized position for a given interval</p> Source code in <code>src/spyglass/linearization/v0/main.py</code> <pre><code>@schema\nclass IntervalLinearizedPosition(SpyglassMixin, dj.Computed):\n    \"\"\"Linearized position for a given interval\"\"\"\n\n    definition = \"\"\"\n    -&gt; IntervalLinearizationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    linearized_position_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing linear position for: {key}\")\n\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n\n        position_nwb = (\n            IntervalPositionInfo\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n                \"position_info_param_name\": key[\"position_info_param_name\"],\n            }\n        ).fetch_nwb()[0]\n\n        position = np.asarray(\n            position_nwb[\"head_position\"].get_spatial_series().data\n        )\n        time = np.asarray(\n            position_nwb[\"head_position\"].get_spatial_series().timestamps\n        )\n\n        linearization_parameters = (\n            LinearizationParameters()\n            &amp; {\"linearization_param_name\": key[\"linearization_param_name\"]}\n        ).fetch1()\n        track_graph_info = (\n            TrackGraph() &amp; {\"track_graph_name\": key[\"track_graph_name\"]}\n        ).fetch1()\n\n        track_graph = make_track_graph(\n            node_positions=track_graph_info[\"node_positions\"],\n            edges=track_graph_info[\"edges\"],\n        )\n\n        linear_position_df = get_linearized_position(\n            position=position,\n            track_graph=track_graph,\n            edge_spacing=track_graph_info[\"linear_edge_spacing\"],\n            edge_order=track_graph_info[\"linear_edge_order\"],\n            use_HMM=linearization_parameters[\"use_hmm\"],\n            route_euclidean_distance_scaling=linearization_parameters[\n                \"route_euclidean_distance_scaling\"\n            ],\n            sensor_std_dev=linearization_parameters[\"sensor_std_dev\"],\n            diagonal_bias=linearization_parameters[\"diagonal_bias\"],\n        )\n\n        linear_position_df[\"time\"] = time\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n\n        key[\"linearized_position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=linear_position_df,\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        return self.fetch_nwb()[0][\"linearized_position\"].set_index(\"time\")\n</code></pre>"}, {"location": "api/linearization/v1/main/", "title": "main.py", "text": ""}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.LinearizationParameters", "title": "<code>LinearizationParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> <p>Choose whether to use an HMM to linearize position. This can help when the eucledian distances between separate arms are too close and the previous position has some information about which arm the animal is on.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass LinearizationParameters(SpyglassMixin, dj.Lookup):\n    \"\"\"Choose whether to use an HMM to linearize position. This can help when\n    the eucledian distances between separate arms are too close and the previous\n    position has some information about which arm the animal is on.\"\"\"\n\n    definition = \"\"\"\n    linearization_param_name : varchar(80)   # name for this set of parameters\n    ---\n    use_hmm = 0 : int   # use HMM to determine linearization\n    # How much to prefer route distances between successive time points that are closer to the euclidean distance. Smaller numbers mean the route distance is more likely to be close to the euclidean distance.\n    route_euclidean_distance_scaling = 1.0 : float\n    sensor_std_dev = 5.0 : float   # Uncertainty of position sensor (in cm).\n    # Biases the transition matrix to prefer the current track segment.\n    diagonal_bias = 0.5 : float\n    \"\"\"\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph", "title": "<code>TrackGraph</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Graph representation of track representing the spatial environment. Used for linearizing position.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass TrackGraph(SpyglassMixin, dj.Manual):\n    \"\"\"Graph representation of track representing the spatial environment.\n    Used for linearizing position.\"\"\"\n\n    definition = \"\"\"\n    track_graph_name : varchar(80)\n    ----\n    environment : varchar(80)    # Type of Environment\n    node_positions : blob  # 2D position of track_graph nodes, shape (n_nodes, 2)\n    edges: blob                  # shape (n_edges, 2)\n    linear_edge_order : blob  # order of track graph edges in the linear space, shape (n_edges, 2)\n    linear_edge_spacing : blob  # amount of space between edges in the linear space, shape (n_edges,)\n    \"\"\"\n\n    def get_networkx_track_graph(self, track_graph_parameters=None):\n        if track_graph_parameters is None:\n            track_graph_parameters = self.fetch1()\n        return make_track_graph(\n            node_positions=track_graph_parameters[\"node_positions\"],\n            edges=track_graph_parameters[\"edges\"],\n        )\n\n    def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n        \"\"\"Plot the track graph in 2D position space.\"\"\"\n        track_graph = self.get_networkx_track_graph()\n        plot_track_graph(\n            track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n        )\n\n    def plot_track_graph_as_1D(\n        self,\n        ax=None,\n        axis=\"x\",\n        other_axis_start=0.0,\n        draw_edge_labels=False,\n        node_size=300,\n        node_color=\"#1f77b4\",\n    ):\n        \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n        track_graph_parameters = self.fetch1()\n        track_graph = self.get_networkx_track_graph(\n            track_graph_parameters=track_graph_parameters\n        )\n        plot_graph_as_1D(\n            track_graph,\n            edge_order=track_graph_parameters[\"linear_edge_order\"],\n            edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n            ax=ax,\n            axis=axis,\n            other_axis_start=other_axis_start,\n            draw_edge_labels=draw_edge_labels,\n            node_size=node_size,\n            node_color=node_color,\n        )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph.plot_track_graph", "title": "<code>plot_track_graph(ax=None, draw_edge_labels=False, **kwds)</code>", "text": "<p>Plot the track graph in 2D position space.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>def plot_track_graph(self, ax=None, draw_edge_labels=False, **kwds):\n    \"\"\"Plot the track graph in 2D position space.\"\"\"\n    track_graph = self.get_networkx_track_graph()\n    plot_track_graph(\n        track_graph, ax=ax, draw_edge_labels=draw_edge_labels, **kwds\n    )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.TrackGraph.plot_track_graph_as_1D", "title": "<code>plot_track_graph_as_1D(ax=None, axis='x', other_axis_start=0.0, draw_edge_labels=False, node_size=300, node_color='#1f77b4')</code>", "text": "<p>Plot the track graph in 1D to see how the linearization is set up.</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>def plot_track_graph_as_1D(\n    self,\n    ax=None,\n    axis=\"x\",\n    other_axis_start=0.0,\n    draw_edge_labels=False,\n    node_size=300,\n    node_color=\"#1f77b4\",\n):\n    \"\"\"Plot the track graph in 1D to see how the linearization is set up.\"\"\"\n    track_graph_parameters = self.fetch1()\n    track_graph = self.get_networkx_track_graph(\n        track_graph_parameters=track_graph_parameters\n    )\n    plot_graph_as_1D(\n        track_graph,\n        edge_order=track_graph_parameters[\"linear_edge_order\"],\n        edge_spacing=track_graph_parameters[\"linear_edge_spacing\"],\n        ax=ax,\n        axis=axis,\n        other_axis_start=other_axis_start,\n        draw_edge_labels=draw_edge_labels,\n        node_size=node_size,\n        node_color=node_color,\n    )\n</code></pre>"}, {"location": "api/linearization/v1/main/#src.spyglass.linearization.v1.main.LinearizedPositionV1", "title": "<code>LinearizedPositionV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Linearized position for a given interval</p> Source code in <code>src/spyglass/linearization/v1/main.py</code> <pre><code>@schema\nclass LinearizedPositionV1(SpyglassMixin, dj.Computed):\n    \"\"\"Linearized position for a given interval\"\"\"\n\n    definition = \"\"\"\n    -&gt; LinearizationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    linearized_position_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n        logger.info(f\"Computing linear position for: {key}\")\n\n        position_nwb = PositionOutput().fetch_nwb(\n            {\"merge_id\": key[\"pos_merge_id\"]}\n        )[0]\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            position_nwb[\"nwb_file_name\"]\n        )\n        position = np.asarray(\n            position_nwb[\"position\"].get_spatial_series().data\n        )\n        time = np.asarray(\n            position_nwb[\"position\"].get_spatial_series().timestamps\n        )\n\n        linearization_parameters = (\n            LinearizationParameters()\n            &amp; {\"linearization_param_name\": key[\"linearization_param_name\"]}\n        ).fetch1()\n        track_graph_info = (\n            TrackGraph() &amp; {\"track_graph_name\": key[\"track_graph_name\"]}\n        ).fetch1()\n\n        track_graph = make_track_graph(\n            node_positions=track_graph_info[\"node_positions\"],\n            edges=track_graph_info[\"edges\"],\n        )\n\n        linear_position_df = get_linearized_position(\n            position=position,\n            track_graph=track_graph,\n            edge_spacing=track_graph_info[\"linear_edge_spacing\"],\n            edge_order=track_graph_info[\"linear_edge_order\"],\n            use_HMM=linearization_parameters[\"use_hmm\"],\n            route_euclidean_distance_scaling=linearization_parameters[\n                \"route_euclidean_distance_scaling\"\n            ],\n            sensor_std_dev=linearization_parameters[\"sensor_std_dev\"],\n            diagonal_bias=linearization_parameters[\"diagonal_bias\"],\n        )\n\n        linear_position_df[\"time\"] = time\n\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n\n        key[\"linearized_position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=linear_position_df,\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=position_nwb[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n        from spyglass.linearization.merge import LinearizedPositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n\n        LinearizedPositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    def fetch1_dataframe(self):\n        return self.fetch_nwb()[0][\"linearized_position\"].set_index(\"time\")\n</code></pre>"}, {"location": "api/lock/file_lock/", "title": "file_lock.py", "text": ""}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.NwbfileLock", "title": "<code>NwbfileLock</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>@schema\nclass NwbfileLock(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Nwbfile\n    \"\"\"\n\n    def populate_from_lock_file(self):\n        \"\"\"\n        Reads from the NWB_LOCK_FILE (defined by an environment variable),\n        adds the entries to this schema, and then removes the file\n        \"\"\"\n        if os.path.exists(os.getenv(\"NWB_LOCK_FILE\")):\n            lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"r\")\n            for line in lock_file:\n                logger.info(line)\n                key = {\"nwb_file_name\": line.strip()}\n                self.insert1(key, skip_duplicates=\"True\")\n            lock_file.close()\n            os.remove(os.getenv(\"NWB_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.NwbfileLock.populate_from_lock_file", "title": "<code>populate_from_lock_file()</code>", "text": "<p>Reads from the NWB_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and then removes the file</p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>def populate_from_lock_file(self):\n    \"\"\"\n    Reads from the NWB_LOCK_FILE (defined by an environment variable),\n    adds the entries to this schema, and then removes the file\n    \"\"\"\n    if os.path.exists(os.getenv(\"NWB_LOCK_FILE\")):\n        lock_file = open(os.getenv(\"NWB_LOCK_FILE\"), \"r\")\n        for line in lock_file:\n            logger.info(line)\n            key = {\"nwb_file_name\": line.strip()}\n            self.insert1(key, skip_duplicates=\"True\")\n        lock_file.close()\n        os.remove(os.getenv(\"NWB_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.AnalysisNwbfileLock", "title": "<code>AnalysisNwbfileLock</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>@schema\nclass AnalysisNwbfileLock(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; AnalysisNwbfile\n    \"\"\"\n\n    def populate_from_lock_file(self):\n        \"\"\"Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and\n        then removes the file\n        \"\"\"\n        if os.path.exists(os.getenv(\"ANALYSIS_LOCK_FILE\")):\n            lock_file = open(os.getenv(\"ANALYSIS_LOCK_FILE\"), \"r\")\n            for line in lock_file:\n                key = {\"analysis_file_name\": line.strip()}\n                self.insert1(key, skip_duplicates=\"True\")\n            os.remove(os.getenv(\"ANALYSIS_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/lock/file_lock/#src.spyglass.lock.file_lock.AnalysisNwbfileLock.populate_from_lock_file", "title": "<code>populate_from_lock_file()</code>", "text": "<p>Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and then removes the file</p> Source code in <code>src/spyglass/lock/file_lock.py</code> <pre><code>def populate_from_lock_file(self):\n    \"\"\"Reads from the ANALYSIS_LOCK_FILE (defined by an environment variable), adds the entries to this schema, and\n    then removes the file\n    \"\"\"\n    if os.path.exists(os.getenv(\"ANALYSIS_LOCK_FILE\")):\n        lock_file = open(os.getenv(\"ANALYSIS_LOCK_FILE\"), \"r\")\n        for line in lock_file:\n            key = {\"analysis_file_name\": line.strip()}\n            self.insert1(key, skip_duplicates=\"True\")\n        os.remove(os.getenv(\"ANALYSIS_LOCK_FILE\"))\n</code></pre>"}, {"location": "api/mua/v1/mua/", "title": "mua.py", "text": ""}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsParameters", "title": "<code>MuaEventsParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Params to extract times of high mulitunit activity during immobility.</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@schema\nclass MuaEventsParameters(SpyglassMixin, dj.Manual):\n    \"\"\"Params to extract times of high mulitunit activity during immobility.\"\"\"\n\n    definition = \"\"\"\n    mua_param_name : varchar(80) # a name for this set of parameters\n    ----\n    mua_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n    contents = [\n        {\n            \"mua_param_name\": \"default\",\n            \"mua_param_dict\": {\n                \"minimum_duration\": 0.015,  # seconds\n                \"zscore_threshold\": 2.0,\n                \"close_event_threshold\": 0.0,  # seconds\n                \"speed_threshold\": 4.0,  # cm/s\n            },\n        },\n    ]\n\n    @classmethod\n    def insert_default(cls):\n        \"\"\"Insert the default parameter set\"\"\"\n        cls.insert(cls.contents, skip_duplicates=True)\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsParameters.insert_default", "title": "<code>insert_default()</code>  <code>classmethod</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@classmethod\ndef insert_default(cls):\n    \"\"\"Insert the default parameter set\"\"\"\n    cls.insert(cls.contents, skip_duplicates=True)\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsV1", "title": "<code>MuaEventsV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>@schema\nclass MuaEventsV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; MuaEventsParameters\n    -&gt; SortedSpikesGroup\n    -&gt; PositionOutput.proj(pos_merge_id='merge_id')\n    -&gt; IntervalList.proj(detection_interval='interval_list_name')\n    ---\n    -&gt; AnalysisNwbfile\n    mua_times_object_id : varchar(40)\n    \"\"\"\n\n    def make(self, key):\n        speed = self.get_speed(key)\n        time = speed.index.to_numpy()\n        speed = speed.to_numpy()\n\n        spike_indicator = SortedSpikesGroup.get_spike_indicator(key, time)\n        spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        mua_params = (MuaEventsParameters &amp; key).fetch1(\"mua_param_dict\")\n\n        valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"detection_interval\"],\n            }\n        ).fetch1(\"valid_times\")\n        mask = np.zeros_like(time, dtype=bool)\n        for start, end in valid_times:\n            mask = mask | ((time &gt;= start) &amp; (time &lt;= end))\n\n        time = time[mask]\n        speed = speed[mask]\n        spike_indicator = spike_indicator[mask]\n\n        mua_times = multiunit_HSE_detector(\n            time, spike_indicator, speed, sampling_frequency, **mua_params\n        )\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        nwb_file_name = (SortedSpikesGroup &amp; key).fetch1(\"nwb_file_name\")\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(nwb_file_name)\n        key[\"mua_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=mua_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=nwb_file_name,\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"mua_times\"] for data in self.fetch_nwb()]\n\n    @classmethod\n    def get_firing_rate(cls, key, time):\n        return SortedSpikesGroup.get_firing_rate(key, time, multiunit=True)\n\n    @staticmethod\n    def get_speed(key):\n        position_info = (\n            PositionOutput &amp; {\"merge_id\": key[\"pos_merge_id\"]}\n        ).fetch1_dataframe()\n        speed_name = (\n            \"speed\" if \"speed\" in position_info.columns else \"head_speed\"\n        )\n        return position_info[speed_name]\n\n    def create_figurl(\n        self,\n        zscore_mua=True,\n        mua_times_color=\"red\",\n        speed_color=\"black\",\n        mua_color=\"black\",\n        view_height=800,\n    ):\n        key = self.fetch1(\"KEY\")\n        speed = self.get_speed(key)\n        time = speed.index.to_numpy()\n        multiunit_firing_rate = self.get_firing_rate(key, time)\n        if zscore_mua:\n            multiunit_firing_rate = zscore(multiunit_firing_rate)\n\n        mua_times = self.fetch1_dataframe()\n\n        multiunit_firing_rate_view = vv.TimeseriesGraph()\n        multiunit_firing_rate_view.add_interval_series(\n            name=\"MUA Events\",\n            t_start=mua_times.start_time.to_numpy(),\n            t_end=mua_times.end_time.to_numpy(),\n            color=mua_times_color,\n        )\n        name = \"Z-Scored Multiunit Rate\" if zscore_mua else \"Multiunit Rate\"\n        multiunit_firing_rate_view.add_line_series(\n            name=name,\n            t=np.asarray(time),\n            y=np.asarray(multiunit_firing_rate, dtype=np.float32),\n            color=mua_color,\n            width=1,\n        )\n        if zscore_mua:\n            mua_params = (MuaEventsParameters &amp; key).fetch1(\"mua_param_dict\")\n            zscore_threshold = mua_params.get(\"zscore_threshold\")\n            multiunit_firing_rate_view.add_line_series(\n                name=\"Z-Score Threshold\",\n                t=np.asarray(time).squeeze(),\n                y=np.ones_like(\n                    multiunit_firing_rate, dtype=np.float32\n                ).squeeze()\n                * zscore_threshold,\n                color=mua_times_color,\n                width=1,\n            )\n        speed_view = vv.TimeseriesGraph().add_line_series(\n            name=\"Speed [cm/s]\",\n            t=np.asarray(time),\n            y=np.asarray(speed, dtype=np.float32),\n            color=speed_color,\n            width=1,\n        )\n        speed_view.add_interval_series(\n            name=\"MUA Events\",\n            t_start=mua_times.start_time.to_numpy(),\n            t_end=mua_times.end_time.to_numpy(),\n            color=mua_times_color,\n        )\n        vertical_panel_content = [\n            vv.LayoutItem(\n                multiunit_firing_rate_view, stretch=2, title=\"Multiunit\"\n            ),\n            vv.LayoutItem(speed_view, stretch=2, title=\"Speed\"),\n        ]\n\n        view = vv.Box(\n            direction=\"horizontal\",\n            show_titles=True,\n            height=view_height,\n            items=[\n                vv.LayoutItem(\n                    vv.Box(\n                        direction=\"vertical\",\n                        show_titles=True,\n                        items=vertical_panel_content,\n                    )\n                ),\n            ],\n        )\n\n        return view.url(label=\"Multiunit Detection\")\n</code></pre>"}, {"location": "api/mua/v1/mua/#src.spyglass.mua.v1.mua.MuaEventsV1.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/mua/v1/mua.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/position/position_merge/", "title": "position_merge.py", "text": ""}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput", "title": "<code>PositionOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> <p>Table to identify source of Position Information from upstream options (e.g. DLC, Trodes, etc...) To add another upstream option, a new Part table should be added in the same syntax as DLCPos and TrodesPos.</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>@schema\nclass PositionOutput(_Merge, SpyglassMixin):\n    \"\"\"\n    Table to identify source of Position Information from upstream options\n    (e.g. DLC, Trodes, etc...) To add another upstream option, a new Part table\n    should be added in the same syntax as DLCPos and TrodesPos.\n    \"\"\"\n\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class DLCPosV1(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream DLC Pose Estimation information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; DLCPosV1\n        \"\"\"\n\n    class TrodesPosV1(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream Trodes Position Tracking information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; TrodesPosV1\n        \"\"\"\n\n    class CommonPos(SpyglassMixin, dj.Part):\n        \"\"\"\n        Table to pass-through upstream Trodes Position Tracking information\n        \"\"\"\n\n        definition = \"\"\"\n        -&gt; PositionOutput\n        ---\n        -&gt; CommonPos\n        \"\"\"\n\n    def fetch1_dataframe(self):\n        # proj replaces operator restriction to enable\n        # (TableName &amp; restriction).fetch1_dataframe()\n        key = self.merge_restrict(self.proj()).proj()\n        query = (\n            source_class_dict[\n                to_camel_case(self.merge_get_parent(self.proj()).table_name)\n            ]\n            &amp; key\n        )\n        return query.fetch1_dataframe()\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.DLCPosV1", "title": "<code>DLCPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream DLC Pose Estimation information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class DLCPosV1(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream DLC Pose Estimation information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; DLCPosV1\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.TrodesPosV1", "title": "<code>TrodesPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream Trodes Position Tracking information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class TrodesPosV1(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream Trodes Position Tracking information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; TrodesPosV1\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionOutput.CommonPos", "title": "<code>CommonPos</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Table to pass-through upstream Trodes Position Tracking information</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>class CommonPos(SpyglassMixin, dj.Part):\n    \"\"\"\n    Table to pass-through upstream Trodes Position Tracking information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionOutput\n    ---\n    -&gt; CommonPos\n    \"\"\"\n</code></pre>"}, {"location": "api/position/position_merge/#src.spyglass.position.position_merge.PositionVideo", "title": "<code>PositionVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/position_merge.py</code> <pre><code>@schema\nclass PositionVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; PositionVideoSelection\n    ---\n    \"\"\"\n\n    def make(self, key):\n        raise NotImplementedError(\"work in progress -DPG\")\n\n        plot = key.get(\"plot\")\n        if plot not in [\"DLC\", \"Trodes\", \"Common\", \"All\"]:\n            raise ValueError(f\"Plot {key['plot']} not supported\")\n            # CBroz: I was told only tests should `assert`, code should `raise`\n\n        M_TO_CM = 100\n        output_dir = (PositionVideoSelection &amp; key).fetch1(\"output_dir\")\n\n        logger.info(\"Loading position data...\")\n        # raw_position_df = (\n        #     RawPosition()\n        #     &amp; {\n        #         \"nwb_file_name\": key[\"nwb_file_name\"],\n        #         \"interval_list_name\": key[\"interval_list_name\"],\n        #     }\n        # ).fetch1_dataframe()\n\n        query = {\n            \"nwb_file_name\": key[\"nwb_file_name\"],\n            \"interval_list_name\": key[\"interval_list_name\"],\n        }\n        merge_entries = {\n            \"DLC\": PositionOutput.DLCPosV1 &amp; query,\n            \"Trodes\": PositionOutput.TrodesPosV1 &amp; query,\n            \"Common\": PositionOutput.CommonPos &amp; query,\n        }\n\n        position_mean_dict = {}\n        if plot == \"All\":\n            # Check which entries exist in PositionOutput\n            merge_dict = {}\n            for source, entries in merge_entries.items():\n                if entries:\n                    merge_dict[source] = entries.fetch1_dataframe().drop(\n                        columns=[\"velocity_x\", \"velocity_y\", \"speed\"]\n                    )\n\n            pos_df = ft.reduce(\n                lambda left, right,: pd.merge(\n                    left[1],\n                    right[1],\n                    left_index=True,\n                    right_index=True,\n                    suffixes=[f\"_{left[0]}\", f\"_{right[0]}\"],\n                ),\n                merge_dict.items(),\n            )\n            position_mean_dict = {\n                source: {\n                    \"position\": np.asarray(\n                        pos_df[[f\"position_x_{source}\", f\"position_y_{source}\"]]\n                    ),\n                    \"orientation\": np.asarray(\n                        pos_df[[f\"orientation_{source}\"]]\n                    ),\n                }\n                for source in merge_dict.keys()\n            }\n        else:\n            if plot == \"DLC\":\n                # CBroz - why is this extra step needed for DLC?\n                pos_df_key = merge_entries[plot].fetch1(as_dict=True)\n                pos_df = (PositionOutput &amp; pos_df_key).fetch1_dataframe()\n            elif plot in [\"Trodes\", \"Common\"]:\n                pos_df = merge_entries[plot].fetch1_dataframe()\n\n            position_mean_dict[plot][\"position\"] = np.asarray(\n                pos_df[[\"position_x\", \"position_y\"]]\n            )\n            position_mean_dict[plot][\"orientation\"] = np.asarray(\n                pos_df[[\"orientation\"]]\n            )\n\n        logger.info(\"Loading video data...\")\n        epoch = int(\"\".join(filter(str.isdigit, key[\"interval_list_name\"]))) + 1\n\n        (\n            video_path,\n            video_filename,\n            meters_per_pixel,\n            video_time,\n        ) = get_video_path(\n            {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        )\n        video_dir = os.path.dirname(video_path) + \"/\"\n        video_frame_col_name = [\n            col for col in pos_df.columns if \"video_frame_ind\" in col\n        ][0]\n        video_frame_inds = pos_df[video_frame_col_name].astype(int).to_numpy()\n        if plot in [\"DLC\", \"All\"]:\n            video_path = (\n                DLCPoseEstimationSelection\n                &amp; (PositionOutput.DLCPosV1 &amp; key).fetch1(\"KEY\")\n            ).fetch1(\"video_path\")\n        else:\n            video_path = check_videofile(\n                video_dir, key[\"output_dir\"], video_filename\n            )[0]\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        output_video_filename = Path(\n            f\"{Path(output_dir).as_posix()}/{nwb_base_filename}{epoch:02d}_\"\n            f\"{key['plot']}_pos_overlay.mp4\"\n        ).as_posix()\n\n        # centroids = {'red': np.asarray(raw_position_df[['xloc', 'yloc']]),\n        #              'green':  np.asarray(raw_position_df[['xloc2', 'yloc2']])}\n\n        logger.info(\"Making video...\")\n\n        make_video(\n            video_path,\n            video_frame_inds,\n            position_mean_dict,\n            video_time,\n            np.asarray(pos_df.index),\n            processor=\"opencv\",\n            output_video_filename=output_video_filename,\n            cm_to_pixels=meters_per_pixel * M_TO_CM,\n            disable_progressbar=False,\n        )\n        self.insert1(key)\n</code></pre>"}, {"location": "api/position/v1/dlc_decorators/", "title": "dlc_decorators.py", "text": ""}, {"location": "api/position/v1/dlc_reader/", "title": "dlc_reader.py", "text": ""}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.read_yaml", "title": "<code>read_yaml(fullpath, filename='*')</code>", "text": "<p>Return contents of yml in fullpath. If available, defer to DJ-saved version</p> <p>Parameters:</p> Name Type Description Default <code>fullpath</code> required <code>filename</code> <code>'*'</code> <p>Returns filepath and contents as dict</p> Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def read_yaml(fullpath, filename=\"*\"):\n    \"\"\"Return contents of yml in fullpath. If available, defer to DJ-saved version\n\n    Parameters\n    ----------\n    fullpath: String or pathlib path. Directory with yaml files\n    filename: String. Filename, no extension. Permits wildcards.\n\n    Returns filepath and contents as dict\n    \"\"\"\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    # Take the DJ-saved if there. If not, return list of available\n    yml_paths = list(Path(fullpath).glob(\"dj_dlc_config.yaml\")) or sorted(\n        list(Path(fullpath).glob(f\"{filename}.y*ml\"))\n    )\n\n    assert (  # If more than 1 and not DJ-saved,\n        len(yml_paths) == 1\n    ), f\"Found more yaml files than expected: {len(yml_paths)}\\n{fullpath}\"\n\n    return yml_paths[0], read_config(yml_paths[0])\n</code></pre>"}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.save_yaml", "title": "<code>save_yaml(output_dir, config_dict, filename='dj_dlc_config', mkdir=True)</code>", "text": "<p>Save config_dict to output_path as filename.yaml. By default, preserves original.</p> <p>Parameters:</p> Name Type Description Default <code>output_dir</code> required <code>config_dict</code> required <code>filename</code> <pre><code>  Set to 'config' to overwrite original file.\n  If extension is included, removed and replaced with \"yaml\".\n</code></pre> <code>'dj_dlc_config'</code> <code>mkdir</code> <code>True</code> <p>Returns:</p> Type Description <code>str</code> <p>path of saved file as string - due to DLC func preference for strings</p> Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def save_yaml(output_dir, config_dict, filename=\"dj_dlc_config\", mkdir=True):\n    \"\"\"Save config_dict to output_path as filename.yaml. By default, preserves original.\n\n    Parameters\n    ----------\n    output_dir: where to save yaml file\n    config_dict: dict of config params or element-deeplabcut model.Model dict\n    filename: Optional, default 'dj_dlc_config' or preserve original 'config'\n              Set to 'config' to overwrite original file.\n              If extension is included, removed and replaced with \"yaml\".\n    mkdir (bool): Optional, True. Make new directory if output_dir not exist\n\n    Returns\n    -------\n    str\n        path of saved file as string - due to DLC func preference for strings\n    \"\"\"\n    from deeplabcut.utils.auxiliaryfunctions import write_config\n\n    if \"config_template\" in config_dict:  # if passed full model.Model dict\n        config_dict = config_dict[\"config_template\"]\n    if mkdir:\n        Path(output_dir).mkdir(exist_ok=True)\n    if \".\" in filename:  # if user provided extension, remove\n        filename = filename.split(\".\")[0]\n\n    output_filepath = Path(output_dir) / f\"{filename}.yaml\"\n    write_config(output_filepath, config_dict)\n    return str(output_filepath)\n</code></pre>"}, {"location": "api/position/v1/dlc_reader/#src.spyglass.position.v1.dlc_reader.do_pose_estimation", "title": "<code>do_pose_estimation(video_filepaths, dlc_model, project_path, output_dir, videotype='', gputouse=None, save_as_csv=False, batchsize=None, cropping=None, TFGPUinference=True, dynamic=(False, 0.5, 10), robust_nframes=False, allow_growth=False, use_shelve=False)</code>", "text": "<p>Launch DLC's analyze_videos within element-deeplabcut</p> <p>Other optional parameters may be set other than those described below. See deeplabcut.analyze_videos parameters for descriptions/defaults.</p> <p>Parameters:</p> Name Type Description Default <code>video_filepaths</code> required <code>dlc_model</code> required <code>project_path</code> required <code>output_dir</code> required Source code in <code>src/spyglass/position/v1/dlc_reader.py</code> <pre><code>def do_pose_estimation(\n    video_filepaths,\n    dlc_model,\n    project_path,\n    output_dir,\n    videotype=\"\",\n    gputouse=None,\n    save_as_csv=False,\n    batchsize=None,\n    cropping=None,\n    TFGPUinference=True,\n    dynamic=(False, 0.5, 10),\n    robust_nframes=False,\n    allow_growth=False,\n    use_shelve=False,\n):\n    \"\"\"Launch DLC's analyze_videos within element-deeplabcut\n\n    Other optional parameters may be set other than those described below. See\n    deeplabcut.analyze_videos parameters for descriptions/defaults.\n\n    Parameters\n    ----------\n    video_filepaths: list of videos to analyze\n    dlc_model: element-deeplabcut dlc.Model dict\n    project_path: path to project config.yml\n    output_dir: where to save output\n    \"\"\"\n    from deeplabcut.pose_estimation_tensorflow import analyze_videos\n\n    # ---- Build and save DLC configuration (yaml) file ----\n    dlc_config = dlc_model[\"config_template\"]\n    dlc_project_path = Path(project_path)\n    dlc_config[\"project_path\"] = dlc_project_path.as_posix()\n\n    # ---- Write config files ----\n    # To output dir: Important for loading/parsing output in datajoint\n    _ = save_yaml(output_dir, dlc_config)\n    # To project dir: Required by DLC to run the analyze_videos\n    if dlc_project_path != output_dir:\n        config_filepath = save_yaml(dlc_project_path, dlc_config)\n    # ---- Trigger DLC prediction job ----\n    analyze_videos(\n        config=config_filepath,\n        videos=video_filepaths,\n        shuffle=dlc_model[\"shuffle\"],\n        trainingsetindex=dlc_model[\"trainingsetindex\"],\n        destfolder=output_dir,\n        modelprefix=dlc_model[\"model_prefix\"],\n        videotype=videotype,\n        gputouse=gputouse,\n        save_as_csv=save_as_csv,\n        batchsize=batchsize,\n        cropping=cropping,\n        TFGPUinference=TFGPUinference,\n        dynamic=dynamic,\n        robust_nframes=robust_nframes,\n        allow_growth=allow_growth,\n        use_shelve=use_shelve,\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/", "title": "dlc_utils.py", "text": ""}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_option", "title": "<code>validate_option(option=None, options=None, name='option', types=None, val_range=None, permit_none=False)</code>", "text": "<p>Validate that option is in a list options or a list of types.</p> <p>Parameters:</p> Name Type Description Default <code>option</code> <code>str</code> <p>If none, runs no checks.</p> <code>None</code> <code>options</code> <code>lis</code> <p>If provided, option must be in options.</p> <code>None</code> <code>name</code> <code>st</code> <p>If provided, name of option to use in error message.</p> <code>'option'</code> <code>types</code> <code>tuple</code> <p>If provided, option must be an instance of one of the types in types.</p> <code>None</code> <code>val_range</code> <code>tuple</code> <p>If provided, option must be in range (min, max)</p> <code>None</code> <code>permit_none</code> <code>bool</code> <p>If True, permit option to be None. Default False.</p> <code>False</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>If option is not in options.</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_option(\n    option=None,\n    options: list = None,\n    name=\"option\",\n    types: tuple = None,\n    val_range: tuple = None,\n    permit_none=False,\n):\n    \"\"\"Validate that option is in a list options or a list of types.\n\n    Parameters\n    ----------\n    option : str, optional\n        If none, runs no checks.\n    options : lis, optional\n        If provided, option must be in options.\n    name : st, optional\n        If provided, name of option to use in error message.\n    types : tuple, optional\n        If provided, option must be an instance of one of the types in types.\n    val_range : tuple, optional\n        If provided, option must be in range (min, max)\n    permit_none : bool, optional\n        If True, permit option to be None. Default False.\n\n    Raises\n    ------\n    ValueError\n        If option is not in options.\n    \"\"\"\n    if option is None and not permit_none:\n        raise ValueError(f\"{name} cannot be None\")\n\n    if options and option not in options:\n        raise KeyError(\n            f\"Unknown {name}: {option} \" f\"Available options: {options}\"\n        )\n\n    if types and not isinstance(option, tuple(types)):\n        raise TypeError(f\"{name} is {type(option)}. Available types {types}\")\n\n    if val_range and not (val_range[0] &lt;= option &lt;= val_range[1]):\n        raise ValueError(f\"{name} must be in range {val_range}\")\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_list", "title": "<code>validate_list(required_items, option_list=None, name='List', condition='', permit_none=False)</code>", "text": "<p>Validate that option_list contains all items in required_items.</p> <p>Parameters:</p> Name Type Description Default <code>required_items</code> <code>list</code> required <code>option_list</code> <code>list</code> <p>If provided, option_list must contain all items in required_items.</p> <code>None</code> <code>name</code> <code>str</code> <p>If provided, name of option_list to use in error message.</p> <code>'List'</code> <code>condition</code> <code>str</code> <p>If provided, condition in error message as 'when using X'.</p> <code>''</code> <code>permit_none</code> <code>bool</code> <p>If True, permit option_list to be None. Default False.</p> <code>False</code> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_list(\n    required_items: list,\n    option_list: list = None,\n    name=\"List\",\n    condition=\"\",\n    permit_none=False,\n):\n    \"\"\"Validate that option_list contains all items in required_items.\n\n    Parameters\n    ---------\n    required_items : list\n    option_list : list, optional\n        If provided, option_list must contain all items in required_items.\n    name : str, optional\n        If provided, name of option_list to use in error message.\n    condition : str, optional\n        If provided, condition in error message as 'when using X'.\n    permit_none : bool, optional\n        If True, permit option_list to be None. Default False.\n    \"\"\"\n    if option_list is None:\n        if permit_none:\n            return\n        else:\n            raise ValueError(f\"{name} cannot be None\")\n    if condition:\n        condition = f\" when using {condition}\"\n    if any(x not in required_items for x in option_list):\n        raise KeyError(\n            f\"{name} must contain all items in {required_items}{condition}.\"\n        )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.validate_smooth_params", "title": "<code>validate_smooth_params(params)</code>", "text": "<p>If params['smooth'], validate method is in list and duration type</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def validate_smooth_params(params):\n    \"\"\"If params['smooth'], validate method is in list and duration type\"\"\"\n    if not params.get(\"smooth\"):\n        return\n    smoothing_params = params.get(\"smoothing_params\")\n    validate_option(smoother=smoothing_params, name=\"smoothing_params\")\n    validate_option(\n        option=smoothing_params.get(\"smooth_method\"),\n        name=\"smooth_method\",\n        options=_key_to_smooth_func_dict,\n    )\n    validate_option(\n        option=smoothing_params.get(\"smoothing_duration\"),\n        name=\"smoothing_duration\",\n        types=(int, float),\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.OutputLogger", "title": "<code>OutputLogger</code>", "text": "<p>A class to wrap a logging.Logger object in order to provide context manager capabilities.</p> <p>This class uses contextlib.redirect_stdout to temporarily redirect sys.stdout and thus print statements to the log file instead of, or as well as the console.</p> <p>Attributes:</p> Name Type Description <code>logger</code> <code>Logger</code> <p>logger object</p> <code>name</code> <code>str</code> <p>name of logger</p> <code>level</code> <code>int</code> <p>level of logging that the logger is set to handle</p> <p>Methods:</p> Name Description <code>setup_logger</code> <p>initialize or get logger object with name_logfile that writes to path_logfile</p> <p>Examples:</p> <pre><code>&gt;&gt;&gt; with OutputLogger(name, path, print_console=True) as logger:\n...    print(\"this will print to logfile\")\n...    logger.logger.info(\"this will log to the logfile\")\n... print(\"this will print to the console\")\n... logger.logger.info(\"this will log to the logfile\")\n</code></pre> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>class OutputLogger:  # TODO: migrate to spyglass.utils.logger\n    \"\"\"\n    A class to wrap a logging.Logger object in order to provide context manager capabilities.\n\n    This class uses contextlib.redirect_stdout to temporarily redirect sys.stdout and thus\n    print statements to the log file instead of, or as well as the console.\n\n    Attributes\n    ----------\n    logger : logging.Logger\n        logger object\n    name : str\n        name of logger\n    level : int\n        level of logging that the logger is set to handle\n\n    Methods\n    -------\n    setup_logger(name_logfile, path_logfile, print_console=False)\n        initialize or get logger object with name_logfile\n        that writes to path_logfile\n\n    Examples\n    --------\n    &gt;&gt;&gt; with OutputLogger(name, path, print_console=True) as logger:\n    ...    print(\"this will print to logfile\")\n    ...    logger.logger.info(\"this will log to the logfile\")\n    ... print(\"this will print to the console\")\n    ... logger.logger.info(\"this will log to the logfile\")\n\n    \"\"\"\n\n    def __init__(self, name, path, level=\"INFO\", **kwargs):\n        self.logger = self.setup_logger(name, path, **kwargs)\n        self.name = self.logger.name\n        self.level = getattr(logging, level)\n\n    def setup_logger(\n        self, name_logfile, path_logfile, print_console=False\n    ) -&gt; logging.Logger:\n        \"\"\"\n        Sets up a logger for that outputs to a file, and optionally, the console\n\n        Parameters\n        ----------\n        name_logfile : str\n            name of the logfile to use\n        path_logfile : str\n            path to the file that should be used as the file handler\n        print_console : bool, default-False\n            if True, prints to console as well as log file.\n\n        Returns\n        -------\n        logger : logging.Logger\n            the logger object with specified handlers\n        \"\"\"\n\n        logger = logging.getLogger(name_logfile)\n        # check to see if handlers already exist for this logger\n        if logger.handlers:\n            for handler in logger.handlers:\n                # if it's a file handler\n                # type is used instead of isinstance,\n                # which doesn't work properly with logging.StreamHandler\n                if type(handler) == logging.FileHandler:\n                    # if paths don't match, change file handler path\n                    if not os.path.samefile(handler.baseFilename, path_logfile):\n                        handler.close()\n                        logger.removeHandler(handler)\n                        file_handler = self._get_file_handler(path_logfile)\n                        logger.addHandler(file_handler)\n                # if a stream handler exists and\n                # if print_console is False remove streamHandler\n                if type(handler) == logging.StreamHandler:\n                    if not print_console:\n                        handler.close()\n                        logger.removeHandler(handler)\n            if print_console and not any(\n                type(handler) == logging.StreamHandler\n                for handler in logger.handlers\n            ):\n                logger.addHandler(self._get_stream_handler())\n\n        else:\n            file_handler = self._get_file_handler(path_logfile)\n            logger.addHandler(file_handler)\n            if print_console:\n                logger.addHandler(self._get_stream_handler())\n        logger.setLevel(logging.INFO)\n        return logger\n\n    def _get_file_handler(self, path):\n        output_dir = pathlib.Path(os.path.dirname(path))\n        if not os.path.exists(output_dir):\n            output_dir.mkdir(parents=True, exist_ok=True)\n        file_handler = logging.FileHandler(path, mode=\"a\")\n        file_handler.setFormatter(self._get_formatter())\n        return file_handler\n\n    def _get_stream_handler(self):\n        stream_handler = logging.StreamHandler()\n        stream_handler.setFormatter(self._get_formatter())\n        return stream_handler\n\n    def _get_formatter(self):\n        return logging.Formatter(\n            \"[%(asctime)s] in %(pathname)s, line %(lineno)d: %(message)s\",\n            datefmt=\"%d-%b-%y %H:%M:%S\",\n        )\n\n    def write(self, msg):\n        if msg and not msg.isspace():\n            self.logger.log(self.level, msg)\n\n    def flush(self):\n        pass\n\n    def __enter__(self):\n        self._redirector = redirect_stdout(self)\n        self._redirector.__enter__()\n        return self\n\n    def __exit__(self, exc_type, exc_value, traceback):\n        # let contextlib do any exception handling here\n        self._redirector.__exit__(exc_type, exc_value, traceback)\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.OutputLogger.setup_logger", "title": "<code>setup_logger(name_logfile, path_logfile, print_console=False)</code>", "text": "<p>Sets up a logger for that outputs to a file, and optionally, the console</p> <p>Parameters:</p> Name Type Description Default <code>name_logfile</code> <code>str</code> <p>name of the logfile to use</p> required <code>path_logfile</code> <code>str</code> <p>path to the file that should be used as the file handler</p> required <code>print_console</code> <code>(bool, default - False)</code> <p>if True, prints to console as well as log file.</p> <code>False</code> <p>Returns:</p> Name Type Description <code>logger</code> <code>Logger</code> <p>the logger object with specified handlers</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def setup_logger(\n    self, name_logfile, path_logfile, print_console=False\n) -&gt; logging.Logger:\n    \"\"\"\n    Sets up a logger for that outputs to a file, and optionally, the console\n\n    Parameters\n    ----------\n    name_logfile : str\n        name of the logfile to use\n    path_logfile : str\n        path to the file that should be used as the file handler\n    print_console : bool, default-False\n        if True, prints to console as well as log file.\n\n    Returns\n    -------\n    logger : logging.Logger\n        the logger object with specified handlers\n    \"\"\"\n\n    logger = logging.getLogger(name_logfile)\n    # check to see if handlers already exist for this logger\n    if logger.handlers:\n        for handler in logger.handlers:\n            # if it's a file handler\n            # type is used instead of isinstance,\n            # which doesn't work properly with logging.StreamHandler\n            if type(handler) == logging.FileHandler:\n                # if paths don't match, change file handler path\n                if not os.path.samefile(handler.baseFilename, path_logfile):\n                    handler.close()\n                    logger.removeHandler(handler)\n                    file_handler = self._get_file_handler(path_logfile)\n                    logger.addHandler(file_handler)\n            # if a stream handler exists and\n            # if print_console is False remove streamHandler\n            if type(handler) == logging.StreamHandler:\n                if not print_console:\n                    handler.close()\n                    logger.removeHandler(handler)\n        if print_console and not any(\n            type(handler) == logging.StreamHandler\n            for handler in logger.handlers\n        ):\n            logger.addHandler(self._get_stream_handler())\n\n    else:\n        file_handler = self._get_file_handler(path_logfile)\n        logger.addHandler(file_handler)\n        if print_console:\n            logger.addHandler(self._get_stream_handler())\n    logger.setLevel(logging.INFO)\n    return logger\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_dlc_processed_data_dir", "title": "<code>get_dlc_processed_data_dir()</code>", "text": "<p>Returns session_dir relative to custom 'dlc_output_dir' root</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_dlc_processed_data_dir() -&gt; str:\n    \"\"\"Returns session_dir relative to custom 'dlc_output_dir' root\"\"\"\n    if \"custom\" in dj.config:\n        if \"dlc_output_dir\" in dj.config[\"custom\"]:\n            dlc_output_dir = dj.config.get(\"custom\", {}).get(\"dlc_output_dir\")\n    if dlc_output_dir:\n        return pathlib.Path(dlc_output_dir)\n    else:\n        return pathlib.Path(\"/nimbus/deeplabcut/output/\")\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.find_full_path", "title": "<code>find_full_path(root_directories, relative_path)</code>", "text": "<p>from Datajoint Elements - unused Given a relative path, search and return the full-path  from provided potential root directories (in the given order)     :param root_directories: potential root directories     :param relative_path: the relative path to find the valid root directory     :return: full-path (pathlib.Path object)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def find_full_path(root_directories, relative_path):\n    \"\"\"\n    from Datajoint Elements - unused\n    Given a relative path, search and return the full-path\n     from provided potential root directories (in the given order)\n        :param root_directories: potential root directories\n        :param relative_path: the relative path to find the valid root directory\n        :return: full-path (pathlib.Path object)\n    \"\"\"\n    relative_path = _to_Path(relative_path)\n\n    if relative_path.exists():\n        return relative_path\n\n    # Turn to list if only a single root directory is provided\n    if isinstance(root_directories, (str, pathlib.Path)):\n        root_directories = [_to_Path(root_directories)]\n\n    for root_dir in root_directories:\n        if (_to_Path(root_dir) / relative_path).exists():\n            return _to_Path(root_dir) / relative_path\n\n    raise FileNotFoundError(\n        f\"No valid full-path found (from {root_directories})\"\n        f\" for {relative_path}\"\n    )\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.find_root_directory", "title": "<code>find_root_directory(root_directories, full_path)</code>", "text": "<p>From datajoint elements - unused Given multiple potential root directories and a full-path, search and return one directory that is the parent of the given path     :param root_directories: potential root directories     :param full_path: the full path to search the root directory     :return: root_directory (pathlib.Path object)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def find_root_directory(root_directories, full_path):\n    \"\"\"\n    From datajoint elements - unused\n    Given multiple potential root directories and a full-path,\n    search and return one directory that is the parent of the given path\n        :param root_directories: potential root directories\n        :param full_path: the full path to search the root directory\n        :return: root_directory (pathlib.Path object)\n    \"\"\"\n    full_path = _to_Path(full_path)\n\n    if not full_path.exists():\n        raise FileNotFoundError(f\"{full_path} does not exist!\")\n\n    # Turn to list if only a single root directory is provided\n    if isinstance(root_directories, (str, pathlib.Path)):\n        root_directories = [_to_Path(root_directories)]\n\n    try:\n        return next(\n            _to_Path(root_dir)\n            for root_dir in root_directories\n            if _to_Path(root_dir) in set(full_path.parents)\n        )\n\n    except StopIteration as exc:\n        raise FileNotFoundError(\n            f\"No valid root directory found (from {root_directories})\"\n            f\" for {full_path}\"\n        ) from exc\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.infer_output_dir", "title": "<code>infer_output_dir(key, makedir=True)</code>", "text": "<p>Return the expected pose_estimation_output_dir.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> required Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def infer_output_dir(key, makedir=True):\n    \"\"\"Return the expected pose_estimation_output_dir.\n\n    Parameters\n    ----------\n    key: DataJoint key specifying a pairing of VideoFile and Model.\n    \"\"\"\n    # TODO: add check to make sure interval_list_name refers to a single epoch\n    # Or make key include epoch in and of itself instead of interval_list_name\n    nwb_file_name = key[\"nwb_file_name\"].split(\"_.\")[0]\n    output_dir = pathlib.Path(dlc_output_dir) / pathlib.Path(\n        f\"{nwb_file_name}/{nwb_file_name}_{key['epoch']:02}\"\n        f\"_model_\" + key[\"dlc_model_name\"].replace(\" \", \"-\")\n    )\n    if makedir is True:\n        if not os.path.exists(output_dir):\n            output_dir.mkdir(parents=True, exist_ok=True)\n    return output_dir\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_video_path", "title": "<code>get_video_path(key)</code>", "text": "<p>Given nwb_file_name and interval_list_name returns specified video file filename and path</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Dictionary containing nwb_file_name and interval_list_name as keys</p> required <p>Returns:</p> Name Type Description <code>video_filepath</code> <code>str</code> <p>path to the video file, including video filename</p> <code>video_filename</code> <code>str</code> <p>filename of the video</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_video_path(key):\n    \"\"\"\n    Given nwb_file_name and interval_list_name returns specified\n    video file filename and path\n\n    Parameters\n    ----------\n    key : dict\n        Dictionary containing nwb_file_name and interval_list_name as keys\n\n    Returns\n    -------\n    video_filepath : str\n        path to the video file, including video filename\n    video_filename : str\n        filename of the video\n    \"\"\"\n    import pynwb\n\n    vf_key = {k: val for k, val in key.items() if k in VideoFile.heading.names}\n    if not VideoFile &amp; vf_key:\n        VideoFile()._no_transaction_make(vf_key, verbose=False)\n    video_query = VideoFile &amp; vf_key\n\n    if len(video_query) != 1:\n        print(f\"Found {len(video_query)} videos for {vf_key}\")\n        return None, None, None, None\n\n    video_info = video_query.fetch1()\n    nwb_path = f\"{raw_dir}/{video_info['nwb_file_name']}\"\n\n    with pynwb.NWBHDF5IO(path=nwb_path, mode=\"r\") as in_out:\n        nwb_file = in_out.read()\n        nwb_video = nwb_file.objects[video_info[\"video_file_object_id\"]]\n        video_filepath = VideoFile.get_abs_path(vf_key)\n        video_dir = os.path.dirname(video_filepath) + \"/\"\n        video_filename = video_filepath.split(video_dir)[-1]\n        meters_per_pixel = nwb_video.device.meters_per_pixel\n        timestamps = np.asarray(nwb_video.timestamps)\n\n    return video_dir, video_filename, meters_per_pixel, timestamps\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.check_videofile", "title": "<code>check_videofile(video_path, output_path=dlc_video_dir, video_filename=None, video_filetype='h264')</code>", "text": "<p>Checks the file extension of a video file to make sure it is .mp4 for DeepLabCut processes. Converts to MP4 if not already.</p> <p>Parameters:</p> Name Type Description Default <code>video_path</code> <code>str or PosixPath object</code> <p>path to directory of the existing video file without filename</p> required <code>output_path</code> <code>str or PosixPath object</code> <p>path to directory where converted video will be saved</p> <code>dlc_video_dir</code> <code>video_filename</code> <code>(str, Optional)</code> <p>filename of the video to convert, if not provided, video_filetype must be and all video files of video_filetype in the directory will be converted</p> <code>None</code> <code>video_filetype</code> <code>str or List, Default 'h264', Optional</code> <p>If video_filename is not provided, all videos of this filetype will be converted to .mp4</p> <code>'h264'</code> <p>Returns:</p> Name Type Description <code>output_files</code> <code>List of PosixPath objects</code> <p>paths to converted video file(s)</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def check_videofile(\n    video_path: Union[str, pathlib.PosixPath],\n    output_path: Union[str, pathlib.PosixPath] = dlc_video_dir,\n    video_filename: str = None,\n    video_filetype: str = \"h264\",\n):\n    \"\"\"\n    Checks the file extension of a video file to make sure it is .mp4 for\n    DeepLabCut processes. Converts to MP4 if not already.\n\n    Parameters\n    ----------\n    video_path : str or PosixPath object\n        path to directory of the existing video file without filename\n    output_path : str or PosixPath object\n        path to directory where converted video will be saved\n    video_filename : str, Optional\n        filename of the video to convert, if not provided, video_filetype must be\n        and all video files of video_filetype in the directory will be converted\n    video_filetype : str or List, Default 'h264', Optional\n        If video_filename is not provided,\n        all videos of this filetype will be converted to .mp4\n\n    Returns\n    -------\n    output_files : List of PosixPath objects\n        paths to converted video file(s)\n    \"\"\"\n\n    if not video_filename:\n        video_files = pathlib.Path(video_path).glob(f\"*.{video_filetype}\")\n    else:\n        video_files = [pathlib.Path(f\"{video_path}/{video_filename}\")]\n    output_files = []\n    for video_filepath in video_files:\n        if video_filepath.exists():\n            if video_filepath.suffix == \".mp4\":\n                output_files.append(video_filepath)\n                continue\n        video_file = (\n            video_filepath.as_posix()\n            .rsplit(video_filepath.parent.as_posix(), maxsplit=1)[-1]\n            .split(\"/\")[-1]\n        )\n        output_files.append(\n            _convert_mp4(video_file, video_path, output_path, videotype=\"mp4\")\n        )\n    return output_files\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_gpu_memory", "title": "<code>get_gpu_memory()</code>", "text": "<p>Queries the gpu cluster and returns the memory use for each core. This is used to evaluate which GPU cores are available to run jobs on (i.e. pose estimation, DLC model training)</p> <p>Returns:</p> Name Type Description <code>memory_use_values</code> <code>dict</code> <p>dictionary with core number as key and memory in use as value.</p> <p>Raises:</p> Type Description <code>RuntimeError</code> <p>if subproccess command errors.</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_gpu_memory():\n    \"\"\"Queries the gpu cluster and returns the memory use for each core.\n    This is used to evaluate which GPU cores are available to run jobs on\n    (i.e. pose estimation, DLC model training)\n\n    Returns\n    -------\n    memory_use_values : dict\n        dictionary with core number as key and memory in use as value.\n\n    Raises\n    ------\n    RuntimeError\n        if subproccess command errors.\n    \"\"\"\n\n    def output_to_list(x):\n        return x.decode(\"ascii\").split(\"\\n\")[:-1]\n\n    query_cmd = \"nvidia-smi --query-gpu=memory.used --format=csv\"\n    try:\n        memory_use_info = output_to_list(\n            subprocess.check_output(query_cmd.split(), stderr=subprocess.STDOUT)\n        )[1:]\n    except subprocess.CalledProcessError as err:\n        raise RuntimeError(\n            f\"command {err.cmd} return with error (code {err.returncode}): \"\n            + f\"{err.output}\"\n        ) from err\n    memory_use_values = {\n        i: int(x.split()[0]) for i, x in enumerate(memory_use_info)\n    }\n    return memory_use_values\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.get_span_start_stop", "title": "<code>get_span_start_stop(indices)</code>", "text": "<p>summary</p> <p>Parameters:</p> Name Type Description Default <code>indices</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def get_span_start_stop(indices):\n    \"\"\"_summary_\n\n    Parameters\n    ----------\n    indices : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    span_inds = []\n    # Get start and stop index of spans of consecutive indices\n    for k, g in groupby(enumerate(indices), lambda x: x[1] - x[0]):\n        group = list(map(itemgetter(1), g))\n        span_inds.append((group[0], group[-1]))\n    return span_inds\n</code></pre>"}, {"location": "api/position/v1/dlc_utils/#src.spyglass.position.v1.dlc_utils.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/position/v1/dlc_utils.py</code> <pre><code>def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/", "title": "position_dlc_centroid.py", "text": ""}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams", "title": "<code>DLCCentroidParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for calculating the centroid</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroidParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for calculating the centroid\n    \"\"\"\n\n    # TODO: whether to keep all params in a params dict\n    # or break out into individual secondary keys\n    definition = \"\"\"\n    dlc_centroid_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        \"\"\"\n        Inserts default centroid parameters. Assumes 2 LEDs tracked\n        \"\"\"\n        params = {\n            \"centroid_method\": \"two_pt_centroid\",\n            \"points\": {\n                \"point1\": \"greenLED\",\n                \"point2\": \"redLED_C\",\n            },\n            \"interpolate\": True,\n            \"interp_params\": {\"max_cm_to_interp\": 15},\n            \"smooth\": True,\n            \"smoothing_params\": {\n                \"smoothing_duration\": 0.05,\n                \"smooth_method\": \"moving_avg\",\n            },\n            \"max_LED_separation\": 12,\n            \"speed_smoothing_std_dev\": 0.100,\n        }\n        cls.insert1(\n            {\"dlc_centroid_params_name\": \"default\", \"params\": params}, **kwargs\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_centroid_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (cls &amp; {\"dlc_centroid_params_name\": \"default\"}).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n\n    def insert1(self, key, **kwargs):\n        \"\"\"\n        Check provided parameter dictionary to make sure\n        it contains all necessary items\n        \"\"\"\n        params = key[\"params\"]\n        centroid_method = params.get(\"centroid_method\")\n        validate_option(  # Ensure centroid method is valid\n            option=centroid_method,\n            options=_key_to_points.keys(),\n            name=\"centroid_method\",\n        )\n        validate_list(  # Ensure points are valid for centroid method\n            required_items=set(params[\"points\"].keys()),\n            option_list=params[\"points\"],\n            name=\"points\",\n            condition=centroid_method,\n        )\n\n        validate_option(\n            option=params.get(\"max_LED_separation\"),\n            name=\"max_LED_separation\",\n            types=(int, float),\n            permit_none=True,\n        )\n\n        validate_smooth_params(params)\n\n        super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams.insert_default", "title": "<code>insert_default(**kwargs)</code>  <code>classmethod</code>", "text": "<p>Inserts default centroid parameters. Assumes 2 LEDs tracked</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@classmethod\ndef insert_default(cls, **kwargs):\n    \"\"\"\n    Inserts default centroid parameters. Assumes 2 LEDs tracked\n    \"\"\"\n    params = {\n        \"centroid_method\": \"two_pt_centroid\",\n        \"points\": {\n            \"point1\": \"greenLED\",\n            \"point2\": \"redLED_C\",\n        },\n        \"interpolate\": True,\n        \"interp_params\": {\"max_cm_to_interp\": 15},\n        \"smooth\": True,\n        \"smoothing_params\": {\n            \"smoothing_duration\": 0.05,\n            \"smooth_method\": \"moving_avg\",\n        },\n        \"max_LED_separation\": 12,\n        \"speed_smoothing_std_dev\": 0.100,\n    }\n    cls.insert1(\n        {\"dlc_centroid_params_name\": \"default\", \"params\": params}, **kwargs\n    )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidParams.insert1", "title": "<code>insert1(key, **kwargs)</code>", "text": "<p>Check provided parameter dictionary to make sure it contains all necessary items</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def insert1(self, key, **kwargs):\n    \"\"\"\n    Check provided parameter dictionary to make sure\n    it contains all necessary items\n    \"\"\"\n    params = key[\"params\"]\n    centroid_method = params.get(\"centroid_method\")\n    validate_option(  # Ensure centroid method is valid\n        option=centroid_method,\n        options=_key_to_points.keys(),\n        name=\"centroid_method\",\n    )\n    validate_list(  # Ensure points are valid for centroid method\n        required_items=set(params[\"points\"].keys()),\n        option_list=params[\"points\"],\n        name=\"points\",\n        condition=centroid_method,\n    )\n\n    validate_option(\n        option=params.get(\"max_LED_separation\"),\n        name=\"max_LED_separation\",\n        types=(int, float),\n        permit_none=True,\n    )\n\n    validate_smooth_params(params)\n\n    super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroidSelection", "title": "<code>DLCCentroidSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to pair a cohort of bodypart entries with the parameters for calculating their centroid</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroidSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to pair a cohort of bodypart entries with\n    the parameters for calculating their centroid\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpCohort\n    -&gt; DLCCentroidParams\n    ---\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.DLCCentroid", "title": "<code>DLCCentroid</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to calculate the centroid of a group of bodyparts</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>@schema\nclass DLCCentroid(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to calculate the centroid of a group of bodyparts\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCCentroidSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_position_object_id : varchar(80)\n    dlc_velocity_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        idx = pd.IndexSlice\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            logger.logger.info(\"Centroid Calculation\")\n\n            # Get labels to smooth from Parameters table\n            cohort_entries = DLCSmoothInterpCohort.BodyPart &amp; key\n            params = (DLCCentroidParams() &amp; key).fetch1(\"params\")\n            centroid_method = params.pop(\"centroid_method\")\n            bodyparts_avail = cohort_entries.fetch(\"bodypart\")\n            speed_smoothing_std_dev = params.pop(\"speed_smoothing_std_dev\")\n\n            if not centroid_method:\n                raise ValueError(\"Please specify a centroid method to use.\")\n            validate_option(option=centroid_method, options=_key_to_func_dict)\n\n            points = params.get(\"points\")\n            required_points = _key_to_points.get(centroid_method)\n            validate_list(\n                required_items=required_points,\n                option_list=points,\n                name=\"params points\",\n                condition=centroid_method,\n            )\n            for point in required_points:\n                bodypart = points[point]\n                if bodypart not in bodyparts_avail:\n                    raise ValueError(\n                        \"Bodypart in points not in model.\"\n                        f\"\\tBodypart {bodypart}\"\n                        f\"\\tIn Model {bodyparts_avail}\"\n                    )\n            bodyparts_to_use = [points[point] for point in required_points]\n\n            pos_df = pd.concat(\n                {\n                    bodypart: (\n                        DLCSmoothInterpCohort.BodyPart\n                        &amp; {**key, **{\"bodypart\": bodypart}}\n                    ).fetch1_dataframe()\n                    for bodypart in bodyparts_to_use\n                },\n                axis=1,\n            )\n            dt = np.median(np.diff(pos_df.index.to_numpy()))\n            sampling_rate = 1 / dt\n            logger.logger.info(\n                \"Calculating centroid with %s\", str(centroid_method)\n            )\n            centroid_func = _key_to_func_dict.get(centroid_method)\n            centroid = centroid_func(pos_df, **params)\n            centroid_df = pd.DataFrame(\n                centroid,\n                columns=[\"x\", \"y\"],\n                index=pos_df.index.to_numpy(),\n            )\n            if params[\"interpolate\"]:\n                if np.any(np.isnan(centroid)):\n                    logger.logger.info(\"interpolating over NaNs\")\n                    nan_inds = (\n                        pd.isnull(centroid_df.loc[:, idx[(\"x\", \"y\")]])\n                        .any(axis=1)\n                        .to_numpy()\n                        .nonzero()[0]\n                    )\n                    nan_spans = get_span_start_stop(nan_inds)\n                    interp_df = interp_pos(\n                        centroid_df.copy(), nan_spans, **params[\"interp_params\"]\n                    )\n                else:\n                    logger.logger.info(\"no NaNs to interpolate over\")\n                    interp_df = centroid_df.copy()\n            else:\n                interp_df = centroid_df.copy()\n            if params[\"smooth\"]:\n                smoothing_duration = params[\"smoothing_params\"].get(\n                    \"smoothing_duration\"\n                )\n                if not smoothing_duration:\n                    raise KeyError(\n                        \"smoothing_duration needs to be passed within smoothing_params\"\n                    )\n                dt = np.median(np.diff(pos_df.index.to_numpy()))\n                sampling_rate = 1 / dt\n                logger.logger.info(\"smoothing position\")\n                smooth_func = _key_to_smooth_func_dict[\n                    params[\"smoothing_params\"][\"smooth_method\"]\n                ]\n                logger.logger.info(\n                    \"Smoothing using method: %s\",\n                    str(params[\"smoothing_params\"][\"smooth_method\"]),\n                )\n                final_df = smooth_func(\n                    interp_df,\n                    sampling_rate=sampling_rate,\n                    **params[\"smoothing_params\"],\n                )\n            else:\n                final_df = interp_df.copy()\n            logger.logger.info(\"getting velocity\")\n            velocity = get_velocity(\n                final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                time=pos_df.index.to_numpy(),\n                sigma=speed_smoothing_std_dev,\n                sampling_frequency=sampling_rate,\n            )  # cm/s\n            speed = np.sqrt(np.sum(velocity**2, axis=1))  # cm/s\n            # Create dataframe\n            velocity_df = pd.DataFrame(\n                np.concatenate((velocity, speed[:, np.newaxis]), axis=1),\n                columns=[\"velocity_x\", \"velocity_y\", \"speed\"],\n                index=pos_df.index.to_numpy(),\n            )\n            total_nan = np.sum(\n                final_df.loc[:, idx[(\"x\", \"y\")]].isna().any(axis=1)\n            )\n            pretrack_nan = np.sum(\n                final_df.iloc[:1000].loc[:, idx[(\"x\", \"y\")]].isna().any(axis=1)\n            )\n            logger.logger.info(\"total NaNs in centroid dataset: %d\", total_nan)\n            logger.logger.info(\n                \"NaNs in centroid dataset before ind 1000: %d\", pretrack_nan\n            )\n            position = pynwb.behavior.Position()\n            velocity = pynwb.behavior.BehavioralTimeSeries()\n            spatial_series = (RawPosition() &amp; key).fetch_nwb()[0][\n                \"raw_position\"\n            ]\n            METERS_PER_CM = 0.01\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=final_df.index.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=spatial_series.reference_frame,\n                comments=spatial_series.comments,\n                description=\"x_position, y_position\",\n            )\n            velocity.create_timeseries(\n                name=\"velocity\",\n                timestamps=velocity_df.index.to_numpy(),\n                conversion=METERS_PER_CM,\n                unit=\"m/s\",\n                data=velocity_df.loc[\n                    :, idx[(\"velocity_x\", \"velocity_y\", \"speed\")]\n                ].to_numpy(),\n                comments=spatial_series.comments,\n                description=\"x_velocity, y_velocity, speed\",\n            )\n            velocity.create_timeseries(\n                name=\"video_frame_ind\",\n                unit=\"index\",\n                timestamps=final_df.index.to_numpy(),\n                data=pos_df[\n                    pos_df.columns.levels[0][0]\n                ].video_frame_ind.to_numpy(),\n                description=\"video_frame_ind\",\n                comments=\"no comments\",\n            )\n            # Add to Analysis NWB file\n            analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n            nwb_analysis_file = AnalysisNwbfile()\n            key.update(\n                {\n                    \"analysis_file_name\": analysis_file_name,\n                    \"dlc_position_object_id\": nwb_analysis_file.add_nwb_object(\n                        analysis_file_name, position\n                    ),\n                    \"dlc_velocity_object_id\": nwb_analysis_file.add_nwb_object(\n                        analysis_file_name, velocity\n                    ),\n                }\n            )\n\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.insert1(key)\n            logger.logger.info(\"inserted entry into DLCCentroid\")\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_position\"].get_spatial_series().timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"position_x\",\n            \"position_y\",\n            \"velocity_x\",\n            \"velocity_y\",\n            \"speed\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"dlc_velocity\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"dlc_position\"].get_spatial_series().data\n                    ),\n                    np.asarray(\n                        nwb_data[\"dlc_velocity\"].time_series[\"velocity\"].data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.four_led_centroid", "title": "<code>four_led_centroid(pos_df, **params)</code>", "text": "<p>Determines the centroid of 4 LEDS on an implant LED ring. Assumed to be the Green LED, and 3 red LEDs called: redLED_C, redLED_L, redLED_R By default, uses (greenled + redLED_C) / 2 to calculate centroid If Green LED is NaN, but red center LED is not,     then the red center LED is called the centroid If green and red center LEDs are NaN, but red left and red right LEDs are not,     then the centroid is (redLED_L + redLED_R) / 2 If red center LED is NaN, but the other 3 LEDS are not,     then the centroid is (greenled + (redLED_L + redLED_R) / 2) / 2 If red center and left LEDs are NaN, but green and red right LEDs are not,     then the centroid is (greenled + redLED_R) / 2 If red center and right LEDs are NaN, but green and red left LEDs are not,     then the centroid is (greenled + redLED_L) / 2 If all red LEDs are NaN, but green LED is not,     then the green LED is called the centroid If all LEDs are NaN, then the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for each LED of interest, index is timestamps. Column names specified by params</p> required <code>**params</code> <code>dict</code> <p>contains 'greenLED' and 'redLED_C', 'redLED_R', 'redLED_L' keys, whose values specify the column names in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def four_led_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines the centroid of 4 LEDS on an implant LED ring.\n    Assumed to be the Green LED, and 3 red LEDs called: redLED_C, redLED_L, redLED_R\n    By default, uses (greenled + redLED_C) / 2 to calculate centroid\n    If Green LED is NaN, but red center LED is not,\n        then the red center LED is called the centroid\n    If green and red center LEDs are NaN, but red left and red right LEDs are not,\n        then the centroid is (redLED_L + redLED_R) / 2\n    If red center LED is NaN, but the other 3 LEDS are not,\n        then the centroid is (greenled + (redLED_L + redLED_R) / 2) / 2\n    If red center and left LEDs are NaN, but green and red right LEDs are not,\n        then the centroid is (greenled + redLED_R) / 2\n    If red center and right LEDs are NaN, but green and red left LEDs are not,\n        then the centroid is (greenled + redLED_L) / 2\n    If all red LEDs are NaN, but green LED is not,\n        then the green LED is called the centroid\n    If all LEDs are NaN, then the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for each LED of interest,\n        index is timestamps. Column names specified by params\n    **params : dict\n        contains 'greenLED' and 'redLED_C', 'redLED_R', 'redLED_L' keys,\n        whose values specify the column names in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n    centroid = np.zeros(shape=(len(pos_df), 2))\n    idx = pd.IndexSlice\n    # TODO: this feels messy, clean-up\n    green_led = params[\"points\"].pop(\"greenLED\", None)\n    red_led_C = params[\"points\"].pop(\"redLED_C\", None)\n    red_led_L = params[\"points\"].pop(\"redLED_L\", None)\n    red_led_R = params[\"points\"].pop(\"redLED_R\", None)\n    green_nans = pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].isna().any(axis=1)\n    red_C_nans = pos_df.loc[:, idx[red_led_C, (\"x\", \"y\")]].isna().any(axis=1)\n    red_L_nans = pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].isna().any(axis=1)\n    red_R_nans = pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].isna().any(axis=1)\n    # TODO: implement checks to make sure not rewriting previously set index in centroid\n    # If all given LEDs are not NaN\n    dist_between_green_red = get_distance(\n        pos_df.loc[:, idx[red_led_C, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    g_c_is_too_separated = (\n        dist_between_green_red &gt;= params[\"max_LED_separation\"]\n    )\n    all_good_mask = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            ~red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~g_c_is_too_separated,\n        ),\n    )\n    centroid[all_good_mask] = [\n        *zip(\n            (\n                pos_df.loc[idx[all_good_mask], idx[red_led_C, \"x\"]]\n                + pos_df.loc[idx[all_good_mask], idx[green_led, \"x\"]]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[all_good_mask], idx[red_led_C, \"y\"]]\n                + pos_df.loc[idx[all_good_mask], idx[green_led, \"y\"]]\n            )\n            / 2,\n        )\n    ]\n    # If green LED and red center LED are both not NaN\n    green_red_C = np.logical_and(\n        ~green_nans, ~red_C_nans, ~g_c_is_too_separated\n    )\n    if np.sum(green_red_C) &gt; 0:\n        centroid[green_red_C] = [\n            *zip(\n                (\n                    pos_df.loc[idx[green_red_C], idx[red_led_C, \"x\"]]\n                    + pos_df.loc[idx[green_red_C], idx[green_led, \"x\"]]\n                )\n                / 2,\n                (\n                    pos_df.loc[idx[green_red_C], idx[red_led_C, \"y\"]]\n                    + pos_df.loc[idx[green_red_C], idx[green_led, \"y\"]]\n                )\n                / 2,\n            )\n        ]\n    # If all given LEDs are NaN\n    all_bad_mask = reduce(\n        np.logical_and, (green_nans, red_C_nans, red_L_nans, red_R_nans)\n    )\n    centroid[all_bad_mask, :] = np.nan\n    # If green LED is NaN, but red center LED is not\n    no_green_red_C = np.logical_and(green_nans, ~red_C_nans)\n    if np.sum(no_green_red_C) &gt; 0:\n        centroid[no_green_red_C] = [\n            *zip(\n                pos_df.loc[idx[no_green_red_C], idx[red_led_C, \"x\"]],\n                pos_df.loc[idx[no_green_red_C], idx[red_led_C, \"y\"]],\n            )\n        ]\n    # If green and red center LEDs are NaN, but red left and red right LEDs are not\n    dist_between_left_right = get_distance(\n        pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].to_numpy(),\n    )\n    l_r_is_too_separated = (\n        dist_between_left_right &gt;= params[\"max_LED_separation\"]\n    )\n    no_green_no_red_C_red_L_red_R = reduce(\n        np.logical_and,\n        (\n            green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~l_r_is_too_separated,\n        ),\n    )\n    if np.sum(no_green_no_red_C_red_L_red_R) &gt; 0:\n        centroid[no_green_no_red_C_red_L_red_R] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_L, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_R, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_L, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[no_green_no_red_C_red_L_red_R], idx[red_led_R, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center LED is NaN, but green, red left, and right LEDs are not\n    dist_between_left_green = get_distance(\n        pos_df.loc[:, idx[red_led_L, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    dist_between_right_green = get_distance(\n        pos_df.loc[:, idx[red_led_R, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[green_led, (\"x\", \"y\")]].to_numpy(),\n    )\n    l_g_is_too_separated = (\n        dist_between_left_green &gt;= params[\"max_LED_separation\"]\n    )\n    r_g_is_too_separated = (\n        dist_between_right_green &gt;= params[\"max_LED_separation\"]\n    )\n    green_red_L_red_R_no_red_C = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            ~red_R_nans,\n            ~l_r_is_too_separated,\n            ~l_g_is_too_separated,\n            ~r_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_L_red_R_no_red_C) &gt; 0:\n        midpoint = (\n            (\n                pos_df.loc[idx[green_red_L_red_R_no_red_C], idx[red_led_L, \"x\"]]\n                + pos_df.loc[\n                    idx[green_red_L_red_R_no_red_C], idx[red_led_R, \"x\"]\n                ]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[green_red_L_red_R_no_red_C], idx[red_led_L, \"y\"]]\n                + pos_df.loc[\n                    idx[green_red_L_red_R_no_red_C], idx[red_led_R, \"y\"]\n                ]\n            )\n            / 2,\n        )\n        centroid[green_red_L_red_R_no_red_C] = [\n            *zip(\n                (\n                    midpoint[0]\n                    + pos_df.loc[\n                        idx[green_red_L_red_R_no_red_C], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    midpoint[1]\n                    + pos_df.loc[\n                        idx[green_red_L_red_R_no_red_C], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center and left LED is NaN, but green and red right LED are not\n    green_red_R_no_red_C_no_red_L = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            red_L_nans,\n            ~red_R_nans,\n            ~r_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_R_no_red_C_no_red_L) &gt; 0:\n        centroid[green_red_R_no_red_C_no_red_L] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[red_led_R, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[red_led_R, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_R_no_red_C_no_red_L], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If red center and right LED is NaN, but green and red left LED are not\n    green_red_L_no_red_C_no_red_R = reduce(\n        np.logical_and,\n        (\n            ~green_nans,\n            red_C_nans,\n            ~red_L_nans,\n            red_R_nans,\n            ~l_g_is_too_separated,\n        ),\n    )\n    if np.sum(green_red_L_no_red_C_no_red_R) &gt; 0:\n        centroid[green_red_L_no_red_C_no_red_R] = [\n            *zip(\n                (\n                    pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[red_led_L, \"x\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[green_led, \"x\"]\n                    ]\n                )\n                / 2,\n                (\n                    pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[red_led_L, \"y\"]\n                    ]\n                    + pos_df.loc[\n                        idx[green_red_L_no_red_C_no_red_R], idx[green_led, \"y\"]\n                    ]\n                )\n                / 2,\n            )\n        ]\n    # If all LEDS are NaN except red left LED\n    red_L_no_green_no_red_C_no_red_R = reduce(\n        np.logical_and, (green_nans, red_C_nans, ~red_L_nans, red_R_nans)\n    )\n    if np.sum(red_L_no_green_no_red_C_no_red_R) &gt; 0:\n        centroid[red_L_no_green_no_red_C_no_red_R] = [\n            *zip(\n                pos_df.loc[\n                    idx[red_L_no_green_no_red_C_no_red_R], idx[red_led_L, \"x\"]\n                ],\n                pos_df.loc[\n                    idx[red_L_no_green_no_red_C_no_red_R], idx[red_led_L, \"y\"]\n                ],\n            )\n        ]\n    # If all LEDS are NaN except red right LED\n    red_R_no_green_no_red_C_no_red_L = reduce(\n        np.logical_and, (green_nans, red_C_nans, red_L_nans, ~red_R_nans)\n    )\n    if np.sum(red_R_no_green_no_red_C_no_red_L) &gt; 0:\n        centroid[red_R_no_green_no_red_C_no_red_L] = [\n            *zip(\n                pos_df.loc[\n                    idx[red_R_no_green_no_red_C_no_red_L], idx[red_led_R, \"x\"]\n                ],\n                pos_df.loc[\n                    idx[red_R_no_green_no_red_C_no_red_L], idx[red_led_R, \"y\"]\n                ],\n            )\n        ]\n    # If all red LEDs are NaN, but green LED is not\n    green_no_red = reduce(\n        np.logical_and, (~green_nans, red_C_nans, red_L_nans, red_R_nans)\n    )\n    if np.sum(green_no_red) &gt; 0:\n        centroid[green_no_red] = [\n            *zip(\n                pos_df.loc[idx[green_no_red], idx[green_led, \"x\"]],\n                pos_df.loc[idx[green_no_red], idx[green_led, \"y\"]],\n            )\n        ]\n    too_separated_inds = reduce(\n        np.logical_or,\n        (\n            g_c_is_too_separated,\n            l_r_is_too_separated,\n            l_g_is_too_separated,\n            r_g_is_too_separated,\n        ),\n    )\n    if np.sum(too_separated_inds) &gt; 0:\n        centroid[too_separated_inds, :] = np.nan\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.two_pt_centroid", "title": "<code>two_pt_centroid(pos_df, **params)</code>", "text": "<p>Determines the centroid of 2 points using (point1 + point2) / 2 For a given timestamp, if one point is NaN, then the other point is assigned as the centroid. If both are NaN, the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for each point of interest, index is timestamps. Column names specified by params</p> required <code>**params</code> <code>dict</code> <p>contains 'point1' and 'point2' keys, whose values specify the column names in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def two_pt_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"\n    Determines the centroid of 2 points using (point1 + point2) / 2\n    For a given timestamp, if one point is NaN,\n    then the other point is assigned as the centroid.\n    If both are NaN, the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for each point of interest,\n        index is timestamps. Column names specified by params\n    **params : dict\n        contains 'point1' and 'point2' keys,\n        whose values specify the column names in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n\n    idx = pd.IndexSlice\n    centroid = np.zeros(shape=(len(pos_df), 2))\n    PT1 = params[\"points\"].pop(\"point1\", None)\n    PT2 = params[\"points\"].pop(\"point2\", None)\n    pt1_nans = pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].isna().any(axis=1)\n    pt2_nans = pos_df.loc[:, idx[PT2, (\"x\", \"y\")]].isna().any(axis=1)\n    dist_between_points = get_distance(\n        pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].to_numpy(),\n        pos_df.loc[:, idx[PT2, (\"x\", \"y\")]].to_numpy(),\n    )\n    is_too_separated = dist_between_points &gt;= params[\"max_LED_separation\"]\n    all_good_mask = np.logical_and(~pt1_nans, ~pt2_nans, ~is_too_separated)\n    centroid[all_good_mask] = [\n        *zip(\n            (\n                pos_df.loc[idx[all_good_mask], idx[PT1, \"x\"]]\n                + pos_df.loc[idx[all_good_mask], idx[PT2, \"x\"]]\n            )\n            / 2,\n            (\n                pos_df.loc[idx[all_good_mask], idx[PT1, \"y\"]]\n                + pos_df.loc[idx[all_good_mask], idx[PT2, \"y\"]]\n            )\n            / 2,\n        )\n    ]\n    # If only point1 is good\n    pt1_mask = np.logical_and(~pt1_nans, pt2_nans)\n    if np.sum(pt1_mask) &gt; 0:\n        centroid[pt1_mask] = [\n            *zip(\n                pos_df.loc[idx[pt1_mask], idx[PT1, \"x\"]],\n                pos_df.loc[idx[pt1_mask], idx[PT1, \"y\"]],\n            )\n        ]\n    # If only point2 is good\n    pt2_mask = np.logical_and(pt1_nans, ~pt2_nans)\n    if np.sum(pt2_mask) &gt; 0:\n        centroid[pt2_mask] = [\n            *zip(\n                pos_df.loc[idx[pt2_mask], idx[PT2, \"x\"]],\n                pos_df.loc[idx[pt2_mask], idx[PT2, \"y\"]],\n            )\n        ]\n    # If neither point is not NaN\n    all_bad_mask = np.logical_and(pt1_nans, pt2_nans)\n    centroid[all_bad_mask, :] = np.nan\n    # If LEDs are too far apart\n    centroid[is_too_separated, :] = np.nan\n\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_centroid/#src.spyglass.position.v1.position_dlc_centroid.one_pt_centroid", "title": "<code>one_pt_centroid(pos_df, **params)</code>", "text": "<p>Passes through the provided point as the centroid For a given timestamp, if the point is NaN, then the centroid is NaN</p> <p>Parameters:</p> Name Type Description Default <code>pos_df</code> <code>DataFrame</code> <p>dataframe containing x and y position for the point of interest, index is timestamps. Column name specified by params</p> required <code>**params</code> <code>dict</code> <p>contains a 'point1' key, whose value specifies the column name in <code>pos_df</code></p> <code>{}</code> <p>Returns:</p> Name Type Description <code>centroid</code> <code>ndarray</code> <p>numpy array with shape (n_time, 2) centroid[0] is the x coord and centroid[1] is the y coord</p> Source code in <code>src/spyglass/position/v1/position_dlc_centroid.py</code> <pre><code>def one_pt_centroid(pos_df: pd.DataFrame, **params):\n    \"\"\"\n    Passes through the provided point as the centroid\n    For a given timestamp, if the point is NaN,\n    then the centroid is NaN\n\n    Parameters\n    ----------\n    pos_df : pd.DataFrame\n        dataframe containing x and y position for the point of interest,\n        index is timestamps. Column name specified by params\n    **params : dict\n        contains a 'point1' key,\n        whose value specifies the column name in `pos_df`\n\n    Returns\n    -------\n    centroid : np.ndarray\n        numpy array with shape (n_time, 2)\n        centroid[0] is the x coord and centroid[1] is the y coord\n    \"\"\"\n    idx = pd.IndexSlice\n    PT1 = params[\"points\"].pop(\"point1\", None)\n    centroid = pos_df.loc[:, idx[PT1, (\"x\", \"y\")]].to_numpy()\n    return centroid\n</code></pre>"}, {"location": "api/position/v1/position_dlc_cohort/", "title": "position_dlc_cohort.py", "text": ""}, {"location": "api/position/v1/position_dlc_cohort/#src.spyglass.position.v1.position_dlc_cohort.DLCSmoothInterpCohortSelection", "title": "<code>DLCSmoothInterpCohortSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to specify which combination of bodyparts from DLCSmoothInterp get combined into a cohort</p> Source code in <code>src/spyglass/position/v1/position_dlc_cohort.py</code> <pre><code>@schema\nclass DLCSmoothInterpCohortSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to specify which combination of bodyparts from DLCSmoothInterp\n    get combined into a cohort\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_si_cohort_selection_name : varchar(120)\n    -&gt; DLCPoseEstimation\n    ---\n    bodyparts_params_dict   : blob      # Dict with bodypart as key and desired dlc_si_params_name as value\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_cohort/#src.spyglass.position.v1.position_dlc_cohort.DLCSmoothInterpCohort", "title": "<code>DLCSmoothInterpCohort</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to combine multiple bodyparts from DLCSmoothInterp to enable centroid/orientation calculations</p> Source code in <code>src/spyglass/position/v1/position_dlc_cohort.py</code> <pre><code>@schema\nclass DLCSmoothInterpCohort(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to combine multiple bodyparts from DLCSmoothInterp\n    to enable centroid/orientation calculations\n    \"\"\"\n\n    # Need to ensure that nwb_file_name/epoch/interval list name endure as primary keys\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpCohortSelection\n    ---\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCSmoothInterpCohort\n        -&gt; DLCSmoothInterp\n        ---\n        -&gt; AnalysisNwbfile\n        dlc_smooth_interp_position_object_id : varchar(80)\n        dlc_smooth_interp_info_object_id : varchar(80)\n        \"\"\"\n\n        def fetch1_dataframe(self):\n            nwb_data = self.fetch_nwb()[0]\n            index = pd.Index(\n                np.asarray(\n                    nwb_data[\"dlc_smooth_interp_position\"]\n                    .get_spatial_series()\n                    .timestamps\n                ),\n                name=\"time\",\n            )\n            COLUMNS = [\n                \"video_frame_ind\",\n                \"x\",\n                \"y\",\n            ]\n            return pd.DataFrame(\n                np.concatenate(\n                    (\n                        np.asarray(\n                            nwb_data[\"dlc_smooth_interp_info\"]\n                            .time_series[\"video_frame_ind\"]\n                            .data,\n                            dtype=int,\n                        )[:, np.newaxis],\n                        np.asarray(\n                            nwb_data[\"dlc_smooth_interp_position\"]\n                            .get_spatial_series()\n                            .data\n                        ),\n                    ),\n                    axis=1,\n                ),\n                columns=COLUMNS,\n                index=index,\n            )\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            logger.logger.info(\"Bodypart Cohort\")\n            # from Jen Guidera\n            self.insert1(key)\n            cohort_selection = (DLCSmoothInterpCohortSelection &amp; key).fetch1()\n            table_entries = []\n            bodyparts_params_dict = cohort_selection.pop(\n                \"bodyparts_params_dict\"\n            )\n            temp_key = cohort_selection.copy()\n            for bodypart, params in bodyparts_params_dict.items():\n                temp_key[\"bodypart\"] = bodypart\n                temp_key[\"dlc_si_params_name\"] = params\n                table_entries.append((DLCSmoothInterp &amp; temp_key).fetch())\n            assert len(table_entries) == len(\n                bodyparts_params_dict\n            ), \"more entries found in DLCSmoothInterp than specified in bodyparts_params_dict\"\n            table_column_names = list(table_entries[0].dtype.fields.keys())\n            for table_entry in table_entries:\n                entry_key = {\n                    **{\n                        k: v for k, v in zip(table_column_names, table_entry[0])\n                    },\n                    **key,\n                }\n                DLCSmoothInterpCohort.BodyPart.insert1(\n                    entry_key, skip_duplicates=True\n                )\n        logger.logger.info(\"Inserted entry into DLCSmoothInterpCohort\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/", "title": "position_dlc_model.py", "text": ""}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelInput", "title": "<code>DLCModelInput</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to hold model path if model is being input from local disk instead of Spyglass</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelInput(SpyglassMixin, dj.Manual):\n    \"\"\"Table to hold model path if model is being input\n    from local disk instead of Spyglass\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_model_name : varchar(64)  # Different than dlc_model_name in DLCModelSource... not great\n    -&gt; DLCProject\n    ---\n    project_path         : varchar(255) # Path to project directory\n    \"\"\"\n\n    def insert1(self, key, **kwargs):\n        # expects key from DLCProject with config_path\n        project_path = Path(key[\"config_path\"]).parent\n        assert project_path.exists(), \"project path does not exist\"\n        key[\"dlc_model_name\"] = f'{project_path.name.split(\"model\")[0]}model'\n        key[\"project_path\"] = project_path.as_posix()\n        del key[\"config_path\"]\n        super().insert1(key, **kwargs)\n        DLCModelSource.insert_entry(\n            dlc_model_name=key[\"dlc_model_name\"],\n            project_name=key[\"project_name\"],\n            source=\"FromImport\",\n            key=key,\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelSource", "title": "<code>DLCModelSource</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to determine whether model originates from upstream DLCModelTraining table, or from local directory</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelSource(SpyglassMixin, dj.Manual):\n    \"\"\"Table to determine whether model originates from\n    upstream DLCModelTraining table, or from local directory\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCProject\n    dlc_model_name : varchar(64)    # User-friendly model name\n    ---\n    source         : enum ('FromUpstream', 'FromImport')\n    \"\"\"\n\n    class FromImport(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCModelSource\n        -&gt; DLCModelInput\n        ---\n        project_path : varchar(255)\n        \"\"\"\n\n    class FromUpstream(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; DLCModelSource\n        -&gt; DLCModelTraining\n        ---\n        project_path : varchar(255)\n        \"\"\"\n\n    @classmethod\n    @accepts(None, None, (\"FromUpstream\", \"FromImport\"), None)\n    def insert_entry(\n        cls,\n        dlc_model_name: str,\n        project_name: str,\n        source: str = \"FromUpstream\",\n        key: dict = None,\n        **kwargs,\n    ):\n        cls.insert1(\n            {\n                \"dlc_model_name\": dlc_model_name,\n                \"project_name\": project_name,\n                \"source\": source,\n            },\n            **kwargs,\n        )\n        part_table = getattr(cls, source)\n        table_query = dj.FreeTable(\n            dj.conn(), full_table_name=part_table.parents()[-1]\n        ) &amp; {\"project_name\": project_name}\n        project_path = table_query.fetch1(\"project_path\")\n        part_table.insert1(\n            {\n                \"dlc_model_name\": dlc_model_name,\n                \"project_name\": project_name,\n                \"project_path\": project_path,\n                **key,\n            },\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelEvaluation", "title": "<code>DLCModelEvaluation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>@schema\nclass DLCModelEvaluation(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCModel\n    ---\n    train_iterations   : int   # Training iterations\n    train_error=null   : float # Train error (px)\n    test_error=null    : float # Test error (px)\n    p_cutoff=null      : float # p-cutoff used\n    train_error_p=null : float # Train error with p-cutoff\n    test_error_p=null  : float # Test error with p-cutoff\n    \"\"\"\n\n    def make(self, key):\n        \"\"\".populate() method will launch evaluation for each unique entry in Model.\"\"\"\n        import csv\n\n        from deeplabcut import evaluate_network\n        from deeplabcut.utils.auxiliaryfunctions import get_evaluation_folder\n\n        dlc_config, project_path, model_prefix, shuffle, trainingsetindex = (\n            DLCModel &amp; key\n        ).fetch1(\n            \"config_template\",\n            \"project_path\",\n            \"model_prefix\",\n            \"shuffle\",\n            \"trainingsetindex\",\n        )\n\n        yml_path, _ = dlc_reader.read_yaml(project_path)\n\n        evaluate_network(\n            yml_path,\n            Shuffles=[shuffle],  # this needs to be a list\n            trainingsetindex=trainingsetindex,\n            comparisonbodyparts=\"all\",\n        )\n\n        eval_folder = get_evaluation_folder(\n            trainFraction=dlc_config[\"TrainingFraction\"][trainingsetindex],\n            shuffle=shuffle,\n            cfg=dlc_config,\n            modelprefix=model_prefix,\n        )\n        eval_path = project_path / eval_folder\n        assert (\n            eval_path.exists()\n        ), f\"Couldn't find evaluation folder:\\n{eval_path}\"\n\n        eval_csvs = list(eval_path.glob(\"*csv\"))\n        max_modified_time = 0\n        for eval_csv in eval_csvs:\n            modified_time = os.path.getmtime(eval_csv)\n            if modified_time &gt; max_modified_time:\n                eval_csv_latest = eval_csv\n        with open(eval_csv_latest, newline=\"\") as f:\n            results = list(csv.DictReader(f, delimiter=\",\"))[0]\n        # in testing, test_error_p returned empty string\n        self.insert1(\n            dict(\n                key,\n                train_iterations=results[\"Training iterations:\"],\n                train_error=results[\" Train error(px)\"],\n                test_error=results[\" Test error(px)\"],\n                p_cutoff=results[\"p-cutoff used\"],\n                train_error_p=results[\"Train error with p-cutoff\"],\n                test_error_p=results[\"Test error with p-cutoff\"],\n            )\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.DLCModelEvaluation.make", "title": "<code>make(key)</code>", "text": "<p>.populate() method will launch evaluation for each unique entry in Model.</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>def make(self, key):\n    \"\"\".populate() method will launch evaluation for each unique entry in Model.\"\"\"\n    import csv\n\n    from deeplabcut import evaluate_network\n    from deeplabcut.utils.auxiliaryfunctions import get_evaluation_folder\n\n    dlc_config, project_path, model_prefix, shuffle, trainingsetindex = (\n        DLCModel &amp; key\n    ).fetch1(\n        \"config_template\",\n        \"project_path\",\n        \"model_prefix\",\n        \"shuffle\",\n        \"trainingsetindex\",\n    )\n\n    yml_path, _ = dlc_reader.read_yaml(project_path)\n\n    evaluate_network(\n        yml_path,\n        Shuffles=[shuffle],  # this needs to be a list\n        trainingsetindex=trainingsetindex,\n        comparisonbodyparts=\"all\",\n    )\n\n    eval_folder = get_evaluation_folder(\n        trainFraction=dlc_config[\"TrainingFraction\"][trainingsetindex],\n        shuffle=shuffle,\n        cfg=dlc_config,\n        modelprefix=model_prefix,\n    )\n    eval_path = project_path / eval_folder\n    assert (\n        eval_path.exists()\n    ), f\"Couldn't find evaluation folder:\\n{eval_path}\"\n\n    eval_csvs = list(eval_path.glob(\"*csv\"))\n    max_modified_time = 0\n    for eval_csv in eval_csvs:\n        modified_time = os.path.getmtime(eval_csv)\n        if modified_time &gt; max_modified_time:\n            eval_csv_latest = eval_csv\n    with open(eval_csv_latest, newline=\"\") as f:\n        results = list(csv.DictReader(f, delimiter=\",\"))[0]\n    # in testing, test_error_p returned empty string\n    self.insert1(\n        dict(\n            key,\n            train_iterations=results[\"Training iterations:\"],\n            train_error=results[\" Train error(px)\"],\n            test_error=results[\" Test error(px)\"],\n            p_cutoff=results[\"p-cutoff used\"],\n            train_error_p=results[\"Train error with p-cutoff\"],\n            test_error_p=results[\"Test error with p-cutoff\"],\n        )\n    )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_model/#src.spyglass.position.v1.position_dlc_model.str_to_bool", "title": "<code>str_to_bool(value)</code>", "text": "<p>Return whether the provided string represents true. Otherwise false.</p> Source code in <code>src/spyglass/position/v1/position_dlc_model.py</code> <pre><code>def str_to_bool(value) -&gt; bool:\n    \"\"\"Return whether the provided string represents true. Otherwise false.\"\"\"\n    # Due to distutils equivalent depreciation in 3.10\n    # Adopted from github.com/PostHog/posthog/blob/master/posthog/utils.py\n    if not value:\n        return False\n    return str(value).lower() in (\"y\", \"yes\", \"t\", \"true\", \"on\", \"1\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/", "title": "position_dlc_orient.py", "text": ""}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.DLCOrientationParams", "title": "<code>DLCOrientationParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for determining and smoothing the orientation of a set of BodyParts</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>@schema\nclass DLCOrientationParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for determining and smoothing the orientation of a set of BodyParts\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_orientation_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @classmethod\n    def insert_params(cls, params_name: str, params: dict, **kwargs):\n        cls.insert1(\n            {\"dlc_orientation_params_name\": params_name, \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        params = {\n            \"orient_method\": \"red_green_orientation\",\n            \"bodypart1\": \"greenLED\",\n            \"bodypart2\": \"redLED_C\",\n            \"orientation_smoothing_std_dev\": 0.001,\n        }\n        cls.insert1(\n            {\"dlc_orientation_params_name\": \"default\", \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_orientation_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (\n                cls &amp; {\"dlc_orientation_params_name\": \"default\"}\n            ).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.DLCOrientation", "title": "<code>DLCOrientation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Determines and smooths orientation of a set of bodyparts given a specified method</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>@schema\nclass DLCOrientation(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Determines and smooths orientation of a set of bodyparts given a specified method\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCOrientationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_orientation_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        # Get labels to smooth from Parameters table\n        cohort_entries = DLCSmoothInterpCohort.BodyPart &amp; key\n        pos_df = pd.concat(\n            {\n                bodypart: (\n                    DLCSmoothInterpCohort.BodyPart\n                    &amp; {**key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in cohort_entries.fetch(\"bodypart\")\n            },\n            axis=1,\n        )\n        params = (DLCOrientationParams() &amp; key).fetch1(\"params\")\n        orientation_smoothing_std_dev = params.pop(\n            \"orientation_smoothing_std_dev\", None\n        )\n        dt = np.median(np.diff(pos_df.index.to_numpy()))\n        sampling_rate = 1 / dt\n        orient_func = _key_to_func_dict[params[\"orient_method\"]]\n        orientation = orient_func(pos_df, **params)\n        if not params[\"orient_method\"] == \"none\":\n            # Smooth orientation\n            is_nan = np.isnan(orientation)\n            unwrap_orientation = orientation.copy()\n            # Only unwrap non nan values, while keeping nans in dataset for interpolation\n            unwrap_orientation[~is_nan] = np.unwrap(orientation[~is_nan])\n            unwrap_df = pd.DataFrame(\n                unwrap_orientation, columns=[\"orientation\"], index=pos_df.index\n            )\n            nan_spans = get_span_start_stop(np.where(is_nan)[0])\n            orient_df = interp_orientation(\n                unwrap_df,\n                nan_spans,\n            )\n            orientation = gaussian_smooth(\n                orient_df[\"orientation\"].to_numpy(),\n                orientation_smoothing_std_dev,\n                sampling_rate,\n                axis=0,\n                truncate=8,\n            )\n            # convert back to between -pi and pi\n            orientation = np.angle(np.exp(1j * orientation))\n        final_df = pd.DataFrame(\n            orientation, columns=[\"orientation\"], index=pos_df.index\n        )\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        spatial_series = (RawPosition() &amp; key).fetch_nwb()[0][\"raw_position\"]\n        orientation = pynwb.behavior.CompassDirection()\n        orientation.create_spatial_series(\n            name=\"orientation\",\n            timestamps=final_df.index.to_numpy(),\n            conversion=1.0,\n            data=final_df[\"orientation\"].to_numpy(),\n            reference_frame=spatial_series.reference_frame,\n            comments=spatial_series.comments,\n            description=\"orientation\",\n        )\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"dlc_orientation_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], orientation\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_orientation\"].get_spatial_series().timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"orientation\",\n        ]\n        return pd.DataFrame(\n            np.asarray(nwb_data[\"dlc_orientation\"].get_spatial_series().data)[\n                :, np.newaxis\n            ],\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.two_pt_head_orientation", "title": "<code>two_pt_head_orientation(pos_df, **params)</code>", "text": "<p>Determines orientation based on vector between two points</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>def two_pt_head_orientation(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines orientation based on vector between two points\"\"\"\n    BP1 = params.pop(\"bodypart1\", None)\n    BP2 = params.pop(\"bodypart2\", None)\n    orientation = np.arctan2(\n        (pos_df[BP1][\"y\"] - pos_df[BP2][\"y\"]),\n        (pos_df[BP1][\"x\"] - pos_df[BP2][\"x\"]),\n    )\n    return orientation\n</code></pre>"}, {"location": "api/position/v1/position_dlc_orient/#src.spyglass.position.v1.position_dlc_orient.red_led_bisector_orientation", "title": "<code>red_led_bisector_orientation(pos_df, **params)</code>", "text": "<p>Determines orientation based on 2 equally-spaced identifiers that are assumed to be perpendicular to the orientation direction. A third object is needed to determine forward/backward</p> Source code in <code>src/spyglass/position/v1/position_dlc_orient.py</code> <pre><code>def red_led_bisector_orientation(pos_df: pd.DataFrame, **params):\n    \"\"\"Determines orientation based on 2 equally-spaced identifiers\n    that are assumed to be perpendicular to the orientation direction.\n    A third object is needed to determine forward/backward\n    \"\"\"\n    LED1 = params.pop(\"led1\", None)\n    LED2 = params.pop(\"led2\", None)\n    LED3 = params.pop(\"led3\", None)\n    orientation = []\n    for index, row in pos_df.iterrows():\n        x_vec = row[LED1][\"x\"] - row[LED2][\"x\"]\n        y_vec = row[LED1][\"y\"] - row[LED2][\"y\"]\n        if y_vec == 0:\n            if (row[LED3][\"y\"] &gt; row[LED1][\"y\"]) &amp; (\n                row[LED3][\"y\"] &gt; row[LED2][\"y\"]\n            ):\n                orientation.append(np.pi / 2)\n            elif (row[LED3][\"y\"] &lt; row[LED1][\"y\"]) &amp; (\n                row[LED3][\"y\"] &lt; row[LED2][\"y\"]\n            ):\n                orientation.append(-(np.pi / 2))\n            else:\n                raise Exception(\"Cannot determine head direction from bisector\")\n        else:\n            length = np.sqrt(y_vec * y_vec + x_vec * x_vec)\n            norm = np.array([-y_vec / length, x_vec / length])\n            orientation.append(np.arctan2(norm[1], norm[0]))\n        if index + 1 == len(pos_df):\n            break\n    return np.array(orientation)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/", "title": "position_dlc_pose_estimation.py", "text": ""}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection", "title": "<code>DLCPoseEstimationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@schema\nclass DLCPoseEstimationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; VideoFile                           # Session -&gt; Recording + File part table\n    -&gt; DLCModel                                    # Must specify a DLC project_path\n    ---\n    task_mode='load' : enum('load', 'trigger')  # load results or trigger computation\n    video_path : varchar(120)                   # path to video file\n    pose_estimation_output_dir='': varchar(255) # output dir relative to the root dir\n    pose_estimation_params=null  : longblob     # analyze_videos params, if not default\n    \"\"\"\n\n    @classmethod\n    def get_video_crop(cls, video_path):\n        \"\"\"\n        Queries the user to determine the cropping parameters for a given video\n\n        Parameters\n        ----------\n        video_path : str\n            path to the video file\n\n        Returns\n        -------\n        crop_ints : list\n            list of 4 integers [x min, x max, y min, y max]\n        \"\"\"\n        import cv2\n\n        cap = cv2.VideoCapture(video_path)\n        _, frame = cap.read()\n        fig, ax = plt.subplots(figsize=(20, 10))\n        ax.imshow(frame)\n        xlims = ax.get_xlim()\n        ylims = ax.get_ylim()\n        ax.set_xticks(np.arange(xlims[0], xlims[-1], 50))\n        ax.set_yticks(np.arange(ylims[0], ylims[-1], -50))\n        ax.grid(visible=True, color=\"white\", lw=0.5, alpha=0.5)\n        display(fig)\n        crop_input = input(\n            \"Please enter the crop parameters for your video in format xmin, xmax, ymin, ymax, or 'none'\\n\"\n        )\n        plt.close()\n        if crop_input.lower() == \"none\":\n            return None\n        crop_ints = [int(val) for val in crop_input.split(\",\")]\n        assert all(isinstance(val, int) for val in crop_ints)\n        return crop_ints\n\n    @classmethod\n    def insert_estimation_task(\n        cls,\n        key,\n        task_mode=\"trigger\",  # load or trigger\n        params: dict = None,\n        check_crop=None,\n        skip_duplicates=True,\n    ):\n        \"\"\"\n        Insert PoseEstimationTask in inferred output dir.\n        From Datajoint Elements\n\n        Parameters\n        ----------\n        key: dict\n            DataJoint key specifying a pairing of VideoRecording and Model.\n        task_mode: bool, optional\n            Default 'trigger' computation. Or 'load' existing results.\n        params (dict): Optional. Parameters passed to DLC's analyze_videos:\n            videotype, gputouse, save_as_csv, batchsize, cropping,\n            TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve\n        \"\"\"\n        from .dlc_utils import check_videofile, get_video_path\n\n        video_path, video_filename, _, _ = get_video_path(key)\n        output_dir = infer_output_dir(key)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n        ) as logger:\n            logger.logger.info(\"Pose Estimation Selection\")\n            video_dir = os.path.dirname(video_path) + \"/\"\n            logger.logger.info(\"video_dir: %s\", video_dir)\n            video_path = check_videofile(\n                video_path=video_dir, video_filename=video_filename\n            )[0]\n            if check_crop is not None:\n                params[\"cropping\"] = cls.get_video_crop(\n                    video_path=video_path.as_posix()\n                )\n            cls.insert1(\n                {\n                    **key,\n                    \"task_mode\": task_mode,\n                    \"pose_estimation_params\": params,\n                    \"video_path\": video_path,\n                    \"pose_estimation_output_dir\": output_dir,\n                },\n                skip_duplicates=skip_duplicates,\n            )\n        logger.logger.info(\"inserted entry into Pose Estimation Selection\")\n        return {**key, \"task_mode\": task_mode}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection.get_video_crop", "title": "<code>get_video_crop(video_path)</code>  <code>classmethod</code>", "text": "<p>Queries the user to determine the cropping parameters for a given video</p> <p>Parameters:</p> Name Type Description Default <code>video_path</code> <code>str</code> <p>path to the video file</p> required <p>Returns:</p> Name Type Description <code>crop_ints</code> <code>list</code> <p>list of 4 integers [x min, x max, y min, y max]</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@classmethod\ndef get_video_crop(cls, video_path):\n    \"\"\"\n    Queries the user to determine the cropping parameters for a given video\n\n    Parameters\n    ----------\n    video_path : str\n        path to the video file\n\n    Returns\n    -------\n    crop_ints : list\n        list of 4 integers [x min, x max, y min, y max]\n    \"\"\"\n    import cv2\n\n    cap = cv2.VideoCapture(video_path)\n    _, frame = cap.read()\n    fig, ax = plt.subplots(figsize=(20, 10))\n    ax.imshow(frame)\n    xlims = ax.get_xlim()\n    ylims = ax.get_ylim()\n    ax.set_xticks(np.arange(xlims[0], xlims[-1], 50))\n    ax.set_yticks(np.arange(ylims[0], ylims[-1], -50))\n    ax.grid(visible=True, color=\"white\", lw=0.5, alpha=0.5)\n    display(fig)\n    crop_input = input(\n        \"Please enter the crop parameters for your video in format xmin, xmax, ymin, ymax, or 'none'\\n\"\n    )\n    plt.close()\n    if crop_input.lower() == \"none\":\n        return None\n    crop_ints = [int(val) for val in crop_input.split(\",\")]\n    assert all(isinstance(val, int) for val in crop_ints)\n    return crop_ints\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimationSelection.insert_estimation_task", "title": "<code>insert_estimation_task(key, task_mode='trigger', params=None, check_crop=None, skip_duplicates=True)</code>  <code>classmethod</code>", "text": "<p>Insert PoseEstimationTask in inferred output dir. From Datajoint Elements</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <p>DataJoint key specifying a pairing of VideoRecording and Model.</p> required <code>task_mode</code> <p>Default 'trigger' computation. Or 'load' existing results.</p> <code>'trigger'</code> <code>params</code> <code>dict</code> <p>videotype, gputouse, save_as_csv, batchsize, cropping, TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@classmethod\ndef insert_estimation_task(\n    cls,\n    key,\n    task_mode=\"trigger\",  # load or trigger\n    params: dict = None,\n    check_crop=None,\n    skip_duplicates=True,\n):\n    \"\"\"\n    Insert PoseEstimationTask in inferred output dir.\n    From Datajoint Elements\n\n    Parameters\n    ----------\n    key: dict\n        DataJoint key specifying a pairing of VideoRecording and Model.\n    task_mode: bool, optional\n        Default 'trigger' computation. Or 'load' existing results.\n    params (dict): Optional. Parameters passed to DLC's analyze_videos:\n        videotype, gputouse, save_as_csv, batchsize, cropping,\n        TFGPUinference, dynamic, robust_nframes, allow_growth, use_shelve\n    \"\"\"\n    from .dlc_utils import check_videofile, get_video_path\n\n    video_path, video_filename, _, _ = get_video_path(key)\n    output_dir = infer_output_dir(key)\n    with OutputLogger(\n        name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n        path=f\"{output_dir.as_posix()}/log.log\",\n    ) as logger:\n        logger.logger.info(\"Pose Estimation Selection\")\n        video_dir = os.path.dirname(video_path) + \"/\"\n        logger.logger.info(\"video_dir: %s\", video_dir)\n        video_path = check_videofile(\n            video_path=video_dir, video_filename=video_filename\n        )[0]\n        if check_crop is not None:\n            params[\"cropping\"] = cls.get_video_crop(\n                video_path=video_path.as_posix()\n            )\n        cls.insert1(\n            {\n                **key,\n                \"task_mode\": task_mode,\n                \"pose_estimation_params\": params,\n                \"video_path\": video_path,\n                \"pose_estimation_output_dir\": output_dir,\n            },\n            skip_duplicates=skip_duplicates,\n        )\n    logger.logger.info(\"inserted entry into Pose Estimation Selection\")\n    return {**key, \"task_mode\": task_mode}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimation", "title": "<code>DLCPoseEstimation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>@schema\nclass DLCPoseEstimation(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCPoseEstimationSelection\n    ---\n    pose_estimation_time: datetime  # time of generation of this set of DLC results\n    meters_per_pixel : double       # conversion of meters per pixel for analyzed video\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        definition = \"\"\" # uses DeepLabCut h5 output for body part position\n        -&gt; DLCPoseEstimation\n        -&gt; DLCModel.BodyPart\n        ---\n        -&gt; AnalysisNwbfile\n        dlc_pose_estimation_position_object_id : varchar(80)\n        dlc_pose_estimation_likelihood_object_id : varchar(80)\n        \"\"\"\n\n        _nwb_table = AnalysisNwbfile\n\n        def fetch1_dataframe(self):\n            nwb_data = self.fetch_nwb()[0]\n            index = pd.Index(\n                np.asarray(\n                    nwb_data[\"dlc_pose_estimation_position\"]\n                    .get_spatial_series()\n                    .timestamps\n                ),\n                name=\"time\",\n            )\n            COLUMNS = [\n                \"video_frame_ind\",\n                \"x\",\n                \"y\",\n                \"likelihood\",\n            ]\n            return pd.DataFrame(\n                np.concatenate(\n                    (\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_likelihood\"]\n                            .time_series[\"video_frame_ind\"]\n                            .data,\n                            dtype=int,\n                        )[:, np.newaxis],\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_position\"]\n                            .get_spatial_series()\n                            .data\n                        ),\n                        np.asarray(\n                            nwb_data[\"dlc_pose_estimation_likelihood\"]\n                            .time_series[\"likelihood\"]\n                            .data\n                        )[:, np.newaxis],\n                    ),\n                    axis=1,\n                ),\n                columns=COLUMNS,\n                index=index,\n            )\n\n    def make(self, key):\n        \"\"\".populate() method will launch training for each PoseEstimationTask\"\"\"\n        from . import dlc_reader\n        from .dlc_utils import get_video_path\n\n        METERS_PER_CM = 0.01\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n        ) as logger:\n            logger.logger.info(\"----------------------\")\n            logger.logger.info(\"Pose Estimation\")\n            # ID model and directories\n            dlc_model = (DLCModel &amp; key).fetch1()\n            bodyparts = (DLCModel.BodyPart &amp; key).fetch(\"bodypart\")\n            task_mode, analyze_video_params, video_path, output_dir = (\n                DLCPoseEstimationSelection &amp; key\n            ).fetch1(\n                \"task_mode\",\n                \"pose_estimation_params\",\n                \"video_path\",\n                \"pose_estimation_output_dir\",\n            )\n            analyze_video_params = analyze_video_params or {}\n\n            project_path = dlc_model[\"project_path\"]\n\n            # Trigger PoseEstimation\n            if task_mode == \"trigger\":\n                dlc_reader.do_pose_estimation(\n                    video_path,\n                    dlc_model,\n                    project_path,\n                    output_dir,\n                    **analyze_video_params,\n                )\n            dlc_result = dlc_reader.PoseEstimation(output_dir)\n            creation_time = datetime.fromtimestamp(\n                dlc_result.creation_time\n            ).strftime(\"%Y-%m-%d %H:%M:%S\")\n\n            logger.logger.info(\"getting raw position\")\n            interval_list_name = (\n                convert_epoch_interval_name_to_position_interval_name(\n                    {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"epoch\": key[\"epoch\"],\n                    },\n                    populate_missing=False,\n                )\n            )\n            spatial_series = (\n                RawPosition()\n                &amp; {**key, \"interval_list_name\": interval_list_name}\n            ).fetch_nwb()[0][\"raw_position\"]\n            _, _, _, video_time = get_video_path(key)\n            pos_time = spatial_series.timestamps\n            # TODO: should get timestamps from VideoFile, but need the video_frame_ind from RawPosition,\n            # which also has timestamps\n            key[\"meters_per_pixel\"] = spatial_series.conversion\n\n            # Insert entry into DLCPoseEstimation\n            logger.logger.info(\n                \"Inserting %s, epoch %02d into DLCPoseEsimation\",\n                key[\"nwb_file_name\"],\n                key[\"epoch\"],\n            )\n            self.insert1({**key, \"pose_estimation_time\": creation_time})\n            meters_per_pixel = key[\"meters_per_pixel\"]\n            del key[\"meters_per_pixel\"]\n            body_parts = dlc_result.df.columns.levels[0]\n            body_parts_df = {}\n            # Insert dlc pose estimation into analysis NWB file for\n            # each body part.\n            for body_part in bodyparts:\n                if body_part in body_parts:\n                    body_parts_df[body_part] = pd.DataFrame.from_dict(\n                        {\n                            c: dlc_result.df.get(body_part).get(c).values\n                            for c in dlc_result.df.get(body_part).columns\n                        }\n                    )\n            idx = pd.IndexSlice\n            for body_part, part_df in body_parts_df.items():\n                logger.logger.info(\"converting to cm\")\n                part_df = convert_to_cm(part_df, meters_per_pixel)\n                logger.logger.info(\"adding timestamps to DataFrame\")\n                part_df = add_timestamps(\n                    part_df, pos_time=pos_time, video_time=video_time\n                )\n                key[\"bodypart\"] = body_part\n                key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                    key[\"nwb_file_name\"]\n                )\n                position = pynwb.behavior.Position()\n                likelihood = pynwb.behavior.BehavioralTimeSeries()\n                position.create_spatial_series(\n                    name=\"position\",\n                    timestamps=part_df.time.to_numpy(),\n                    conversion=METERS_PER_CM,\n                    data=part_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                    reference_frame=spatial_series.reference_frame,\n                    comments=spatial_series.comments,\n                    description=\"x_position, y_position\",\n                )\n                likelihood.create_timeseries(\n                    name=\"likelihood\",\n                    timestamps=part_df.time.to_numpy(),\n                    data=part_df.loc[:, idx[\"likelihood\"]].to_numpy(),\n                    unit=\"likelihood\",\n                    comments=\"no comments\",\n                    description=\"likelihood\",\n                )\n                likelihood.create_timeseries(\n                    name=\"video_frame_ind\",\n                    timestamps=part_df.time.to_numpy(),\n                    data=part_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                    unit=\"index\",\n                    comments=\"no comments\",\n                    description=\"video_frame_ind\",\n                )\n                nwb_analysis_file = AnalysisNwbfile()\n                key[\"dlc_pose_estimation_position_object_id\"] = (\n                    nwb_analysis_file.add_nwb_object(\n                        analysis_file_name=key[\"analysis_file_name\"],\n                        nwb_object=position,\n                    )\n                )\n                key[\"dlc_pose_estimation_likelihood_object_id\"] = (\n                    nwb_analysis_file.add_nwb_object(\n                        analysis_file_name=key[\"analysis_file_name\"],\n                        nwb_object=likelihood,\n                    )\n                )\n                nwb_analysis_file.add(\n                    nwb_file_name=key[\"nwb_file_name\"],\n                    analysis_file_name=key[\"analysis_file_name\"],\n                )\n                self.BodyPart.insert1(key)\n\n    def fetch_dataframe(self, *attrs, **kwargs):\n        entries = (self.BodyPart &amp; self).fetch(\"KEY\")\n        nwb_data_dict = {\n            entry[\"bodypart\"]: (self.BodyPart() &amp; entry).fetch_nwb()[0]\n            for entry in entries\n        }\n        index = pd.Index(\n            np.asarray(\n                nwb_data_dict[entries[0][\"bodypart\"]][\n                    \"dlc_pose_estimation_position\"\n                ]\n                .get_spatial_series()\n                .timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"x\",\n            \"y\",\n            \"likelihood\",\n        ]\n        return pd.concat(\n            {\n                entry[\"bodypart\"]: pd.DataFrame(\n                    np.concatenate(\n                        (\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_likelihood\"\n                                ]\n                                .time_series[\"video_frame_ind\"]\n                                .data,\n                                dtype=int,\n                            )[:, np.newaxis],\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_position\"\n                                ]\n                                .get_spatial_series()\n                                .data\n                            ),\n                            np.asarray(\n                                nwb_data_dict[entry[\"bodypart\"]][\n                                    \"dlc_pose_estimation_likelihood\"\n                                ]\n                                .time_series[\"likelihood\"]\n                                .data\n                            )[:, np.newaxis],\n                        ),\n                        axis=1,\n                    ),\n                    columns=COLUMNS,\n                    index=index,\n                )\n                for entry in entries\n            },\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.DLCPoseEstimation.make", "title": "<code>make(key)</code>", "text": "<p>.populate() method will launch training for each PoseEstimationTask</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>def make(self, key):\n    \"\"\".populate() method will launch training for each PoseEstimationTask\"\"\"\n    from . import dlc_reader\n    from .dlc_utils import get_video_path\n\n    METERS_PER_CM = 0.01\n\n    output_dir = infer_output_dir(key=key, makedir=False)\n    with OutputLogger(\n        name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n        path=f\"{output_dir.as_posix()}/log.log\",\n    ) as logger:\n        logger.logger.info(\"----------------------\")\n        logger.logger.info(\"Pose Estimation\")\n        # ID model and directories\n        dlc_model = (DLCModel &amp; key).fetch1()\n        bodyparts = (DLCModel.BodyPart &amp; key).fetch(\"bodypart\")\n        task_mode, analyze_video_params, video_path, output_dir = (\n            DLCPoseEstimationSelection &amp; key\n        ).fetch1(\n            \"task_mode\",\n            \"pose_estimation_params\",\n            \"video_path\",\n            \"pose_estimation_output_dir\",\n        )\n        analyze_video_params = analyze_video_params or {}\n\n        project_path = dlc_model[\"project_path\"]\n\n        # Trigger PoseEstimation\n        if task_mode == \"trigger\":\n            dlc_reader.do_pose_estimation(\n                video_path,\n                dlc_model,\n                project_path,\n                output_dir,\n                **analyze_video_params,\n            )\n        dlc_result = dlc_reader.PoseEstimation(output_dir)\n        creation_time = datetime.fromtimestamp(\n            dlc_result.creation_time\n        ).strftime(\"%Y-%m-%d %H:%M:%S\")\n\n        logger.logger.info(\"getting raw position\")\n        interval_list_name = (\n            convert_epoch_interval_name_to_position_interval_name(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"epoch\": key[\"epoch\"],\n                },\n                populate_missing=False,\n            )\n        )\n        spatial_series = (\n            RawPosition()\n            &amp; {**key, \"interval_list_name\": interval_list_name}\n        ).fetch_nwb()[0][\"raw_position\"]\n        _, _, _, video_time = get_video_path(key)\n        pos_time = spatial_series.timestamps\n        # TODO: should get timestamps from VideoFile, but need the video_frame_ind from RawPosition,\n        # which also has timestamps\n        key[\"meters_per_pixel\"] = spatial_series.conversion\n\n        # Insert entry into DLCPoseEstimation\n        logger.logger.info(\n            \"Inserting %s, epoch %02d into DLCPoseEsimation\",\n            key[\"nwb_file_name\"],\n            key[\"epoch\"],\n        )\n        self.insert1({**key, \"pose_estimation_time\": creation_time})\n        meters_per_pixel = key[\"meters_per_pixel\"]\n        del key[\"meters_per_pixel\"]\n        body_parts = dlc_result.df.columns.levels[0]\n        body_parts_df = {}\n        # Insert dlc pose estimation into analysis NWB file for\n        # each body part.\n        for body_part in bodyparts:\n            if body_part in body_parts:\n                body_parts_df[body_part] = pd.DataFrame.from_dict(\n                    {\n                        c: dlc_result.df.get(body_part).get(c).values\n                        for c in dlc_result.df.get(body_part).columns\n                    }\n                )\n        idx = pd.IndexSlice\n        for body_part, part_df in body_parts_df.items():\n            logger.logger.info(\"converting to cm\")\n            part_df = convert_to_cm(part_df, meters_per_pixel)\n            logger.logger.info(\"adding timestamps to DataFrame\")\n            part_df = add_timestamps(\n                part_df, pos_time=pos_time, video_time=video_time\n            )\n            key[\"bodypart\"] = body_part\n            key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                key[\"nwb_file_name\"]\n            )\n            position = pynwb.behavior.Position()\n            likelihood = pynwb.behavior.BehavioralTimeSeries()\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=part_df.time.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=part_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=spatial_series.reference_frame,\n                comments=spatial_series.comments,\n                description=\"x_position, y_position\",\n            )\n            likelihood.create_timeseries(\n                name=\"likelihood\",\n                timestamps=part_df.time.to_numpy(),\n                data=part_df.loc[:, idx[\"likelihood\"]].to_numpy(),\n                unit=\"likelihood\",\n                comments=\"no comments\",\n                description=\"likelihood\",\n            )\n            likelihood.create_timeseries(\n                name=\"video_frame_ind\",\n                timestamps=part_df.time.to_numpy(),\n                data=part_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                unit=\"index\",\n                comments=\"no comments\",\n                description=\"video_frame_ind\",\n            )\n            nwb_analysis_file = AnalysisNwbfile()\n            key[\"dlc_pose_estimation_position_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=position,\n                )\n            )\n            key[\"dlc_pose_estimation_likelihood_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=likelihood,\n                )\n            )\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.BodyPart.insert1(key)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_pose_estimation/#src.spyglass.position.v1.position_dlc_pose_estimation.add_timestamps", "title": "<code>add_timestamps(df, pos_time, video_time)</code>", "text": "<p>Takes timestamps from raw_pos_df and adds to df, which is returned with timestamps and their matching video frame index</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>pose estimation dataframe to add timestamps</p> required <code>pos_time</code> <code>ndarray</code> <p>numpy array containing timestamps from the raw position object</p> required <code>video_time</code> <code>ndarray</code> <p>numpy array containing timestamps from the video file</p> required <p>Returns:</p> Type Description <code>DataFrame</code> <p>original df with timestamps and video_frame_ind as new columns</p> Source code in <code>src/spyglass/position/v1/position_dlc_pose_estimation.py</code> <pre><code>def add_timestamps(\n    df: pd.DataFrame, pos_time: np.ndarray, video_time: np.ndarray\n) -&gt; pd.DataFrame:\n    \"\"\"\n    Takes timestamps from raw_pos_df and adds to df,\n    which is returned with timestamps and their matching video frame index\n\n    Parameters\n    ----------\n    df : pd.DataFrame\n        pose estimation dataframe to add timestamps\n    pos_time : np.ndarray\n        numpy array containing timestamps from the raw position object\n    video_time: np.ndarray\n        numpy array containing timestamps from the video file\n\n    Returns\n    -------\n    pd.DataFrame\n        original df with timestamps and video_frame_ind as new columns\n    \"\"\"\n    first_video_frame = np.searchsorted(video_time, pos_time[0])\n    video_frame_ind = np.arange(first_video_frame, len(video_time))\n    time_df = pd.DataFrame(\n        index=video_frame_ind,\n        data=video_time[first_video_frame:],\n        columns=[\"time\"],\n    )\n    df = df.join(time_df)\n    # Drop indices where time is NaN\n    df = df.dropna(subset=[\"time\"])\n    # Add video_frame_ind as column\n    df = df.rename_axis(\"video_frame_ind\").reset_index()\n    return df\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/", "title": "position_dlc_position.py", "text": ""}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.DLCSmoothInterpParams", "title": "<code>DLCSmoothInterpParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for extracting the smoothed head position.</p> <p>Attributes:</p> Name Type Description <code>interpolate</code> <code>bool, default True</code> <p>whether to interpolate over NaN spans</p> <code>smooth</code> <code>bool, default True</code> <p>whether to smooth the dataset</p> <code>smoothing_params</code> <code>dict</code> <p>smoothing_duration : float, default 0.05     number of frames to smooth over: sampling_rate*smoothing_duration = num_frames</p> <code>interp_params</code> <code>dict</code> <p>max_cm_to_interp : int, default 20     maximum distance between high likelihood points on either side of a NaN span     to interpolate over</p> <code>likelihood_thresh</code> <code>float, default 0.95</code> <p>likelihood below which to NaN and interpolate over</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>@schema\nclass DLCSmoothInterpParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for extracting the smoothed head position.\n\n    Attributes\n    ----------\n    interpolate : bool, default True\n        whether to interpolate over NaN spans\n    smooth : bool, default True\n        whether to smooth the dataset\n    smoothing_params : dict\n        smoothing_duration : float, default 0.05\n            number of frames to smooth over: sampling_rate*smoothing_duration = num_frames\n    interp_params : dict\n        max_cm_to_interp : int, default 20\n            maximum distance between high likelihood points on either side of a NaN span\n            to interpolate over\n    likelihood_thresh : float, default 0.95\n        likelihood below which to NaN and interpolate over\n    \"\"\"\n\n    definition = \"\"\"\n    dlc_si_params_name : varchar(80) # name for this set of parameters\n    ---\n    params: longblob # dictionary of parameters\n    \"\"\"\n\n    @classmethod\n    def insert_params(cls, params_name: str, params: dict, **kwargs):\n        cls.insert1(\n            {\"dlc_si_params_name\": params_name, \"params\": params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        default_params = {\n            \"smooth\": True,\n            \"smoothing_params\": {\n                \"smoothing_duration\": 0.05,\n                \"smooth_method\": \"moving_avg\",\n            },\n            \"interpolate\": True,\n            \"likelihood_thresh\": 0.95,\n            \"interp_params\": {\"max_cm_to_interp\": 15},\n            \"max_cm_between_pts\": 20,\n            # This is for use when finding \"good spans\" and is how many indices\n            # to bridge in between good spans see inds_to_span in get_good_spans\n            \"num_inds_to_span\": 20,\n        }\n        cls.insert1(\n            {\"dlc_si_params_name\": \"default\", \"params\": default_params},\n            **kwargs,\n        )\n\n    @classmethod\n    def insert_nan_params(cls, **kwargs):\n        nan_params = {\n            \"smooth\": False,\n            \"interpolate\": False,\n            \"likelihood_thresh\": 0.95,\n            \"max_cm_between_pts\": 20,\n            \"num_inds_to_span\": 20,\n        }\n        cls.insert1(\n            {\"dlc_si_params_name\": \"just_nan\", \"params\": nan_params}, **kwargs\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; {\"dlc_si_params_name\": \"default\"}\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            default = (cls &amp; {\"dlc_si_params_name\": \"default\"}).fetch1()\n        else:\n            default = query.fetch1()\n        return default\n\n    @classmethod\n    def get_nan_params(cls):\n        query = cls &amp; {\"dlc_si_params_name\": \"just_nan\"}\n        if not len(query) &gt; 0:\n            cls().insert_nan_params(skip_duplicates=True)\n            nan_params = (cls &amp; {\"dlc_si_params_name\": \"just_nan\"}).fetch1()\n        else:\n            nan_params = query.fetch1()\n        return nan_params\n\n    @staticmethod\n    def get_available_methods():\n        return _key_to_smooth_func_dict.keys()\n\n    def insert1(self, key, **kwargs):\n        params = key.get(\"params\")\n        if not isinstance(params, dict):\n            raise KeyError(\"'params' must be a dict in key\")\n\n        validate_option(\n            option=params.get(\"max_cm_between_pts\"), name=\"max_cm_between_pts\"\n        )\n        validate_smooth_params(params)\n\n        validate_option(\n            params.get(\"likelihood_thresh\"),\n            name=\"likelihood_thresh\",\n            types=(float),\n            val_range=(0, 1),\n        )\n\n        super().insert1(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.DLCSmoothInterp", "title": "<code>DLCSmoothInterp</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Interpolates across low likelihood periods and smooths the position Can take a few minutes.</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>@schema\nclass DLCSmoothInterp(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Interpolates across low likelihood periods and smooths the position\n    Can take a few minutes.\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCSmoothInterpSelection\n    ---\n    -&gt; AnalysisNwbfile\n    dlc_smooth_interp_position_object_id : varchar(80)\n    dlc_smooth_interp_info_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        from .dlc_utils import OutputLogger, infer_output_dir\n\n        METERS_PER_CM = 0.01\n\n        output_dir = infer_output_dir(key=key, makedir=False)\n        with OutputLogger(\n            name=f\"{key['nwb_file_name']}_{key['epoch']}_{key['dlc_model_name']}_log\",\n            path=f\"{output_dir.as_posix()}/log.log\",\n            print_console=False,\n        ) as logger:\n            logger.logger.info(\"-----------------------\")\n            idx = pd.IndexSlice\n            # Get labels to smooth from Parameters table\n            params = (DLCSmoothInterpParams() &amp; key).fetch1(\"params\")\n            # Get DLC output dataframe\n            logger.logger.info(\"fetching Pose Estimation Dataframe\")\n            dlc_df = (DLCPoseEstimation.BodyPart() &amp; key).fetch1_dataframe()\n            dt = np.median(np.diff(dlc_df.index.to_numpy()))\n            sampling_rate = 1 / dt\n            logger.logger.info(\"Identifying indices to NaN\")\n            df_w_nans, bad_inds = nan_inds(\n                dlc_df.copy(),\n                params[\"max_cm_between_pts\"],\n                likelihood_thresh=params.pop(\"likelihood_thresh\"),\n                inds_to_span=params[\"num_inds_to_span\"],\n            )\n\n            nan_spans = get_span_start_stop(np.where(bad_inds)[0])\n            if params[\"interpolate\"]:\n                logger.logger.info(\"interpolating across low likelihood times\")\n                interp_df = interp_pos(\n                    df_w_nans.copy(), nan_spans, **params[\"interp_params\"]\n                )\n            else:\n                interp_df = df_w_nans.copy()\n                logger.logger.info(\"skipping interpolation\")\n            if params[\"smooth\"]:\n                if \"smoothing_duration\" in params[\"smoothing_params\"]:\n                    smoothing_duration = params[\"smoothing_params\"].pop(\n                        \"smoothing_duration\"\n                    )\n                dt = np.median(np.diff(dlc_df.index.to_numpy()))\n                sampling_rate = 1 / dt\n                logger.logger.info(\"smoothing position\")\n                smooth_func = _key_to_smooth_func_dict[\n                    params[\"smoothing_params\"][\"smooth_method\"]\n                ]\n                logger.logger.info(\n                    \"Smoothing using method: %s\",\n                    str(params[\"smoothing_params\"][\"smooth_method\"]),\n                )\n                smooth_df = smooth_func(\n                    interp_df,\n                    smoothing_duration=smoothing_duration,\n                    sampling_rate=sampling_rate,\n                    **params[\"smoothing_params\"],\n                )\n            else:\n                smooth_df = interp_df.copy()\n                logger.logger.info(\"skipping smoothing\")\n            final_df = smooth_df.drop([\"likelihood\"], axis=1)\n            final_df = final_df.rename_axis(\"time\").reset_index()\n            position_nwb_data = (\n                (DLCPoseEstimation.BodyPart() &amp; key)\n                .fetch_nwb()[0][\"dlc_pose_estimation_position\"]\n                .get_spatial_series()\n            )\n            key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n                key[\"nwb_file_name\"]\n            )\n            # Add dataframe to AnalysisNwbfile\n            nwb_analysis_file = AnalysisNwbfile()\n            position = pynwb.behavior.Position()\n            video_frame_ind = pynwb.behavior.BehavioralTimeSeries()\n            logger.logger.info(\"Creating NWB objects\")\n            position.create_spatial_series(\n                name=\"position\",\n                timestamps=final_df.time.to_numpy(),\n                conversion=METERS_PER_CM,\n                data=final_df.loc[:, idx[(\"x\", \"y\")]].to_numpy(),\n                reference_frame=position_nwb_data.reference_frame,\n                comments=position_nwb_data.comments,\n                description=\"x_position, y_position\",\n            )\n            video_frame_ind.create_timeseries(\n                name=\"video_frame_ind\",\n                timestamps=final_df.time.to_numpy(),\n                data=final_df.loc[:, idx[\"video_frame_ind\"]].to_numpy(),\n                unit=\"index\",\n                comments=\"no comments\",\n                description=\"video_frame_ind\",\n            )\n            key[\"dlc_smooth_interp_position_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=position,\n                )\n            )\n            key[\"dlc_smooth_interp_info_object_id\"] = (\n                nwb_analysis_file.add_nwb_object(\n                    analysis_file_name=key[\"analysis_file_name\"],\n                    nwb_object=video_frame_ind,\n                )\n            )\n            nwb_analysis_file.add(\n                nwb_file_name=key[\"nwb_file_name\"],\n                analysis_file_name=key[\"analysis_file_name\"],\n            )\n            self.insert1(key)\n            logger.logger.info(\"inserted entry into DLCSmoothInterp\")\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(\n                nwb_data[\"dlc_smooth_interp_position\"]\n                .get_spatial_series()\n                .timestamps\n            ),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"x\",\n            \"y\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"dlc_smooth_interp_info\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"dlc_smooth_interp_position\"]\n                        .get_spatial_series()\n                        .data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_position/#src.spyglass.position.v1.position_dlc_position.get_good_spans", "title": "<code>get_good_spans(bad_inds_mask, inds_to_span=50)</code>", "text": "<p>This function takes in a boolean mask of good and bad indices and determines spans of consecutive good indices. It combines two neighboring spans with a separation of less than inds_to_span and treats them as a single good span.</p> <p>Parameters:</p> Name Type Description Default <code>bad_inds_mask</code> <code>boolean mask</code> <p>A boolean mask where True is a bad index and False is a good index.</p> required <code>inds_to_span</code> <code>int</code> <p>This indicates how many indices between two good spans should be bridged to form a single good span. For instance if span A is (1500, 2350) and span B is (2370, 3700), then span A and span B would be combined into span A (1500, 3700) since one would want to identify potential jumps in the space in between the original A and B.</p> <code>50</code> <p>Returns:</p> Name Type Description <code>good_spans</code> <code>list</code> <p>List of spans of good indices, unmodified.</p> <code>modified_spans</code> <code>list</code> <p>spans that are amended to bridge up to inds_to_span consecutive bad indices</p> Source code in <code>src/spyglass/position/v1/position_dlc_position.py</code> <pre><code>def get_good_spans(bad_inds_mask, inds_to_span: int = 50):\n    \"\"\"\n    This function takes in a boolean mask of good and bad indices and\n    determines spans of consecutive good indices. It combines two neighboring\n    spans with a separation of less than inds_to_span and treats them as a\n    single good span.\n\n    Parameters\n    ----------\n    bad_inds_mask : boolean mask\n        A boolean mask where True is a bad index and False is a good index.\n    inds_to_span : int, default 50\n        This indicates how many indices between two good spans should\n        be bridged to form a single good span.\n        For instance if span A is (1500, 2350) and span B is (2370, 3700),\n        then span A and span B would be combined into span A (1500, 3700)\n        since one would want to identify potential jumps in the space in between\n        the original A and B.\n\n    Returns\n    -------\n    good_spans : list\n        List of spans of good indices, unmodified.\n    modified_spans : list\n        spans that are amended to bridge up to inds_to_span consecutive bad indices\n    \"\"\"\n    good_spans = get_span_start_stop(\n        np.arange(len(bad_inds_mask))[~bad_inds_mask]\n    )\n    if len(good_spans) &gt; 1:\n        modified_spans = []\n        for (start1, stop1), (start2, stop2) in zip(\n            good_spans[:-1], good_spans[1:]\n        ):\n            check_existing = [\n                entry\n                for entry in modified_spans\n                if start1\n                in range(entry[0] - inds_to_span, entry[1] + inds_to_span)\n            ]\n            if len(check_existing) &gt; 0:\n                modify_ind = modified_spans.index(check_existing[0])\n                if (start2 - stop1) &lt;= inds_to_span:\n                    modified_spans[modify_ind] = (check_existing[0][0], stop2)\n                else:\n                    modified_spans[modify_ind] = (check_existing[0][0], stop1)\n                    modified_spans.append((start2, stop2))\n                continue\n            if (start2 - stop1) &lt;= inds_to_span:\n                modified_spans.append((start1, stop2))\n            else:\n                modified_spans.append((start1, stop1))\n                modified_spans.append((start2, stop2))\n        return good_spans, modified_spans\n    else:\n        return None, good_spans\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/", "title": "position_dlc_project.py", "text": ""}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.BodyPart", "title": "<code>BodyPart</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Holds bodyparts for use in DeepLabCut models</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@schema\nclass BodyPart(SpyglassMixin, dj.Manual):\n    \"\"\"Holds bodyparts for use in DeepLabCut models\"\"\"\n\n    definition = \"\"\"\n    bodypart                : varchar(32)\n    ---\n    bodypart_description='' : varchar(80)\n    \"\"\"\n\n    @classmethod\n    def add_from_config(cls, bodyparts: List, descriptions: List = None):\n        \"\"\"Given a list of bodyparts from the config and\n        an optional list of descriptions, inserts into BodyPart table.\n\n        Parameters\n        ----------\n        bodyparts : List\n            list of bodyparts from config\n        descriptions : List, default None\n            optional list of descriptions for bodyparts.\n            If None, description is set to bodypart name\n        \"\"\"\n        if descriptions is not None:\n            bodyparts_dict = [\n                {\"bodypart\": bp, \"bodypart_description\": desc}\n                for (bp, desc) in zip(bodyparts, descriptions)\n            ]\n        else:\n            bodyparts_dict = [\n                {\"bodypart\": bp, \"bodypart_description\": bp} for bp in bodyparts\n            ]\n        cls.insert(bodyparts_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.BodyPart.add_from_config", "title": "<code>add_from_config(bodyparts, descriptions=None)</code>  <code>classmethod</code>", "text": "<p>Given a list of bodyparts from the config and an optional list of descriptions, inserts into BodyPart table.</p> <p>Parameters:</p> Name Type Description Default <code>bodyparts</code> <code>List</code> <p>list of bodyparts from config</p> required <code>descriptions</code> <code>List</code> <p>optional list of descriptions for bodyparts. If None, description is set to bodypart name</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef add_from_config(cls, bodyparts: List, descriptions: List = None):\n    \"\"\"Given a list of bodyparts from the config and\n    an optional list of descriptions, inserts into BodyPart table.\n\n    Parameters\n    ----------\n    bodyparts : List\n        list of bodyparts from config\n    descriptions : List, default None\n        optional list of descriptions for bodyparts.\n        If None, description is set to bodypart name\n    \"\"\"\n    if descriptions is not None:\n        bodyparts_dict = [\n            {\"bodypart\": bp, \"bodypart_description\": desc}\n            for (bp, desc) in zip(bodyparts, descriptions)\n        ]\n    else:\n        bodyparts_dict = [\n            {\"bodypart\": bp, \"bodypart_description\": bp} for bp in bodyparts\n        ]\n    cls.insert(bodyparts_dict, skip_duplicates=True)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject", "title": "<code>DLCProject</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to facilitate creation of a new DeepLabCut model. With ability to edit config, extract frames, label frames</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@schema\nclass DLCProject(SpyglassMixin, dj.Manual):\n    \"\"\"Table to facilitate creation of a new DeepLabCut model.\n    With ability to edit config, extract frames, label frames\n    \"\"\"\n\n    # Add more parameters as secondary keys...\n    # TODO: collapse params into blob dict\n    definition = \"\"\"\n    project_name     : varchar(100) # name of DLC project\n    ---\n    -&gt; LabTeam\n    bodyparts        : blob         # list of bodyparts to label\n    frames_per_video : int          # number of frames to extract from each video\n    config_path      : varchar(120) # path to config.yaml for model\n    \"\"\"\n\n    class BodyPart(SpyglassMixin, dj.Part):\n        \"\"\"Part table to hold bodyparts used in each project.\"\"\"\n\n        definition = \"\"\"\n        -&gt; DLCProject\n        -&gt; BodyPart\n        \"\"\"\n\n    class File(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Paths of training files (e.g., labeled pngs, CSV or video)\n        -&gt; DLCProject\n        file_name: varchar(200) # Concise name to describe file\n        file_ext : enum(\"mp4\", \"csv\", \"h5\") # extension of file\n        ---\n        file_path: varchar(255)\n        \"\"\"\n\n    def insert1(self, key, **kwargs):\n        assert isinstance(\n            key[\"project_name\"], str\n        ), \"project_name must be a string\"\n        assert isinstance(\n            key[\"frames_per_video\"], int\n        ), \"frames_per_video must be of type `int`\"\n        super().insert1(key, **kwargs)\n\n    @classmethod\n    def insert_existing_project(\n        cls,\n        project_name: str,\n        lab_team: str,\n        config_path: str,\n        bodyparts: List = None,\n        frames_per_video: int = None,\n        add_to_files: bool = True,\n        **kwargs,\n    ):\n        \"\"\"\n        insert an existing project into DLCProject table.\n        Parameters\n        ----------\n        project_name : str\n            user-friendly name of project\n        lab_team : str\n            name of lab team. Should match an entry in LabTeam table\n        config_path : str\n            path to project directory\n        bodyparts : list\n            optional list of bodyparts to label that\n            are not already in existing config\n        \"\"\"\n\n        # Read config\n        project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n        if project_name in project_names_in_use:\n            print(f\"project name: {project_name} is already in use.\")\n            return_key = {}\n            return_key[\"project_name\"], return_key[\"config_path\"] = (\n                cls &amp; {\"project_name\": project_name}\n            ).fetch1(\"project_name\", \"config_path\")\n            return return_key\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        cfg = read_config(config_path)\n        if bodyparts:\n            bodyparts_to_add = [\n                bodypart\n                for bodypart in bodyparts\n                if bodypart not in cfg[\"bodyparts\"]\n            ]\n            all_bodyparts = bodyparts_to_add + cfg[\"bodyparts\"]\n        else:\n            all_bodyparts = cfg[\"bodyparts\"]\n        BodyPart.add_from_config(cfg[\"bodyparts\"])\n        for bodypart in all_bodyparts:\n            if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n                raise ValueError(\n                    f\"bodypart: {bodypart} not found in BodyPart table\"\n                )\n        # check bodyparts are in config, if not add\n        if len(bodyparts_to_add) &gt; 0:\n            add_to_config(config_path, bodyparts=bodyparts_to_add)\n        # Get frames per video from config. If passed as arg, check match\n        if frames_per_video:\n            if frames_per_video != cfg[\"numframes2pick\"]:\n                add_to_config(\n                    config_path, **{\"numframes2pick\": frames_per_video}\n                )\n        config_path = Path(config_path)\n        project_path = config_path.parent\n        dlc_project_path = dlc_project_dir\n        if dlc_project_path not in project_path.as_posix():\n            project_dirname = project_path.name\n            dest_folder = Path(f\"{dlc_project_path}/{project_dirname}/\")\n            if dest_folder.exists():\n                new_proj_dir = dest_folder.as_posix()\n            else:\n                new_proj_dir = shutil.copytree(\n                    src=project_path,\n                    dst=f\"{dlc_project_path}/{project_dirname}/\",\n                )\n            new_config_path = Path(f\"{new_proj_dir}/config.yaml\")\n            assert (\n                new_config_path.exists()\n            ), \"config.yaml does not exist in new project directory\"\n            config_path = new_config_path\n            add_to_config(config_path, **{\"project_path\": new_proj_dir})\n        # TODO still need to copy videos over to video dir\n        key = {\n            \"project_name\": project_name,\n            \"team_name\": lab_team,\n            \"bodyparts\": bodyparts,\n            \"config_path\": config_path.as_posix(),\n            \"frames_per_video\": frames_per_video,\n        }\n        cls.insert1(key, **kwargs)\n        cls.BodyPart.insert(\n            [\n                {\"project_name\": project_name, \"bodypart\": bp}\n                for bp in all_bodyparts\n            ],\n            **kwargs,\n        )\n        if add_to_files:\n            del key[\"bodyparts\"]\n            del key[\"team_name\"]\n            del key[\"config_path\"]\n            del key[\"frames_per_video\"]\n            # Check for training files to add\n            cls.add_training_files(key, **kwargs)\n        return {\n            \"project_name\": project_name,\n            \"config_path\": config_path.as_posix(),\n        }\n\n    @classmethod\n    def insert_new_project(\n        cls,\n        project_name: str,\n        bodyparts: List,\n        lab_team: str,\n        frames_per_video: int,\n        video_list: List,\n        groupname: str = None,\n        project_directory: str = dlc_project_dir,\n        output_path: str = dlc_video_dir,\n        **kwargs,\n    ):\n        \"\"\"Insert a new project into DLCProject table.\n\n        Parameters\n        ----------\n        project_name : str\n            user-friendly name of project\n        groupname : str, optional\n            Name for project group. If None, defaults to username\n        bodyparts : list\n            list of bodyparts to label. Should match bodyparts in BodyPart table\n        lab_team : str\n            name of lab team. Should match an entry in LabTeam table\n        project_directory : str\n            directory where to create project.\n            (Default is '/cumulus/deeplabcut/')\n        frames_per_video : int\n            number of frames to extract from each video\n        video_list : list\n            list of (a) dicts of to query VideoFile table for or (b) absolute\n            paths to videos to train on. If dict, use format:\n            [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]\n        output_path : str\n            target path to output converted videos\n            (Default is '/nimbus/deeplabcut/videos/')\n        \"\"\"\n        project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n        if project_name in project_names_in_use:\n            print(f\"project name: {project_name} is already in use.\")\n            return_key = {}\n            return_key[\"project_name\"], return_key[\"config_path\"] = (\n                cls &amp; {\"project_name\": project_name}\n            ).fetch1(\"project_name\", \"config_path\")\n            return return_key\n\n        add_to_files = kwargs.pop(\"add_to_files\", True)\n        if not bool(LabTeam() &amp; {\"team_name\": lab_team}):\n            raise ValueError(f\"team_name: {lab_team} does not exist in LabTeam\")\n        skeleton_node = None\n        # If dict, assume of form {'nwb_file_name': nwb_file_name, 'epoch': epoch}\n        # and pass to get_video_path to reference VideoFile table for path\n\n        if all(isinstance(n, Dict) for n in video_list):\n            videos_to_convert = [\n                get_video_path(video_key) for video_key in video_list\n            ]\n            videos = [\n                check_videofile(\n                    video_path=video[0],\n                    output_path=output_path,\n                    video_filename=video[1],\n                )[0].as_posix()\n                for video in videos_to_convert\n                if video[0] is not None\n            ]\n            if len(videos) &lt; 1:\n                raise ValueError(\n                    f\"no .mp4 videos found in {videos_to_convert[0][0]}\"\n                    + f\" for key: {video_list[0]}\"\n                )\n\n        # If not dict, assume list of video file paths that may or may not need to be converted\n        else:\n            videos = []\n            if not all([Path(video).exists() for video in video_list]):\n                raise OSError(\"at least one file in video_list does not exist\")\n            for video in video_list:\n                video_path = Path(video).parent\n                video_filename = video.rsplit(\n                    video_path.as_posix(), maxsplit=1\n                )[-1].split(\"/\")[-1]\n                videos.extend(\n                    [\n                        check_videofile(\n                            video_path=video_path,\n                            output_path=output_path,\n                            video_filename=video_filename,\n                        )[0].as_posix()\n                    ]\n                )\n            if len(videos) &lt; 1:\n                raise ValueError(f\"no .mp4 videos found in{video_path}\")\n        from deeplabcut import create_new_project\n\n        config_path = create_new_project(\n            project_name,\n            lab_team,\n            videos,\n            working_directory=project_directory,\n            copy_videos=True,\n            multianimal=False,\n        )\n        for bodypart in bodyparts:\n            if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n                raise ValueError(\n                    f\"bodypart: {bodypart} not found in BodyPart table\"\n                )\n        kwargs_copy = copy.deepcopy(kwargs)\n        kwargs_copy.update({\"numframes2pick\": frames_per_video, \"dotsize\": 3})\n        add_to_config(\n            config_path, bodyparts, skeleton_node=skeleton_node, **kwargs_copy\n        )\n        key = {\n            \"project_name\": project_name,\n            \"team_name\": lab_team,\n            \"bodyparts\": bodyparts,\n            \"config_path\": config_path,\n            \"frames_per_video\": frames_per_video,\n        }\n        cls.insert1(key, **kwargs)\n        cls.BodyPart.insert(\n            [\n                {\"project_name\": project_name, \"bodypart\": bp}\n                for bp in bodyparts\n            ],\n            **kwargs,\n        )\n        if add_to_files:\n            del key[\"bodyparts\"]\n            del key[\"team_name\"]\n            del key[\"config_path\"]\n            del key[\"frames_per_video\"]\n            # Add videos to training files\n            cls.add_training_files(key, **kwargs)\n        if isinstance(config_path, PosixPath):\n            config_path = config_path.as_posix()\n        return {\"project_name\": project_name, \"config_path\": config_path}\n\n    @classmethod\n    def add_video_files(\n        cls,\n        video_list,\n        config_path=None,\n        key=None,\n        output_path: str = os.getenv(\"DLC_VIDEO_PATH\"),\n        add_new=False,\n        add_to_files=True,\n        **kwargs,\n    ):\n        has_config_or_key = bool(config_path) or bool(key)\n\n        if add_new and not has_config_or_key:\n            raise ValueError(\"If add_new, must provide key or config_path\")\n        config_path = config_path or (cls &amp; key).fetch1(\"config_path\")\n\n        if (\n            add_to_files\n            and not key\n            and len(cls &amp; {\"config_path\": config_path}) != 1\n        ):\n            raise ValueError(\"Cannot set add_to_files=True without passing key\")\n\n        if all(isinstance(n, Dict) for n in video_list):\n            videos_to_convert = [\n                get_video_path(video_key) for video_key in video_list\n            ]\n            videos = [\n                check_videofile(\n                    video_path=video[0],\n                    output_path=output_path,\n                    video_filename=video[1],\n                )[0].as_posix()\n                for video in videos_to_convert\n            ]\n        # If not dict, assume list of video file paths\n        # that may or may not need to be converted\n        else:\n            videos = []\n            if not all([Path(video).exists() for video in video_list]):\n                raise OSError(\"at least one file in video_list does not exist\")\n            for video in video_list:\n                video_path = Path(video).parent\n                video_filename = video.rsplit(\n                    video_path.as_posix(), maxsplit=1\n                )[-1].split(\"/\")[-1]\n                videos.append(\n                    check_videofile(\n                        video_path=video_path,\n                        output_path=output_path,\n                        video_filename=video_filename,\n                    )[0].as_posix()\n                )\n            if len(videos) &lt; 1:\n                raise ValueError(f\"no .mp4 videos found in{video_path}\")\n        if add_new:\n            from deeplabcut import add_new_videos\n\n            add_new_videos(config=config_path, videos=videos, copy_videos=True)\n        if add_to_files:\n            # Add videos to training files\n            cls.add_training_files(key, **kwargs)\n        return videos\n\n    @classmethod\n    def add_training_files(cls, key, **kwargs):\n        \"\"\"Add training videos and labeled frames .h5\n        and .csv to DLCProject.File\"\"\"\n        config_path = (cls &amp; {\"project_name\": key[\"project_name\"]}).fetch1(\n            \"config_path\"\n        )\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        if \"config_path\" in key:\n            del key[\"config_path\"]\n        cfg = read_config(config_path)\n        video_names = list(cfg[\"video_sets\"].keys())\n        training_files = []\n        for video in video_names:\n            video_name = os.path.splitext(\n                video.split(os.path.dirname(video) + \"/\")[-1]\n            )[0]\n            training_files.extend(\n                glob.glob(\n                    f\"{cfg['project_path']}/\"\n                    f\"labeled-data/{video_name}/*Collected*\"\n                )\n            )\n        for video in video_names:\n            key[\"file_name\"] = f'{os.path.splitext(video.split(\"/\")[-1])[0]}'\n            key[\"file_ext\"] = os.path.splitext(video.split(\"/\")[-1])[-1].split(\n                \".\"\n            )[-1]\n            key[\"file_path\"] = video\n            cls.File.insert1(key, **kwargs)\n        if len(training_files) &gt; 0:\n            for file in training_files:\n                video_name = os.path.dirname(file).split(\"/\")[-1]\n                file_type = os.path.splitext(\n                    file.split(os.path.dirname(file) + \"/\")[-1]\n                )[-1].split(\".\")[-1]\n                key[\"file_name\"] = f\"{video_name}_labeled_data\"\n                key[\"file_ext\"] = file_type\n                key[\"file_path\"] = file\n                cls.File.insert1(key, **kwargs)\n        else:\n            Warning(\"No training files to add\")\n\n    @classmethod\n    def run_extract_frames(cls, key, **kwargs):\n        \"\"\"Convenience function to launch DLC GUI for extracting frames.\n        Must be run on local machine to access GUI,\n        cannot be run through ssh tunnel\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import extract_frames\n\n        extract_frames(config_path, **kwargs)\n\n    @classmethod\n    def run_label_frames(cls, key):\n        \"\"\"Convenience function to launch DLC GUI for labeling frames.\n        Must be run on local machine to access GUI,\n        cannot be run through ssh tunnel\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import label_frames\n\n        label_frames(config_path)\n\n    @classmethod\n    def check_labels(cls, key, **kwargs):\n        \"\"\"Convenience function to check labels on\n        previously extracted and labeled frames\n        \"\"\"\n        config_path = (cls &amp; key).fetch1(\"config_path\")\n        from deeplabcut import check_labels\n\n        check_labels(config_path, **kwargs)\n\n    @classmethod\n    def import_labeled_frames(\n        cls,\n        key: Dict,\n        import_project_path: Union[str, PosixPath],\n        video_filenames: Union[str, List],\n        **kwargs,\n    ):\n        \"\"\"Function to import pre-labeled frames from an existing project\n        into a new project\n\n        Parameters\n        ----------\n        key : Dict\n            key to specify entry in DLCProject table to add labeled frames to\n        import_project_path : str\n            absolute path to project directory containing\n            labeled frames to import\n        video_filenames : str or List\n            filename or list of filenames of video(s)\n            from which to import frames.\n            without file extension\n        \"\"\"\n        project_entry = (cls &amp; key).fetch1()\n        team_name = project_entry[\"team_name\"]\n        current_project_path = Path(project_entry[\"config_path\"]).parent\n        current_labeled_data_path = Path(\n            f\"{current_project_path.as_posix()}/labeled-data\"\n        )\n        if isinstance(import_project_path, PosixPath):\n            assert import_project_path.exists(), (\n                \"import_project_path: \"\n                f\"{import_project_path.as_posix()} does not exist\"\n            )\n            import_labeled_data_path = Path(\n                f\"{import_project_path.as_posix()}/labeled-data\"\n            )\n        else:\n            assert Path(\n                import_project_path\n            ).exists(), (\n                f\"import_project_path: {import_project_path} does not exist\"\n            )\n            import_labeled_data_path = Path(\n                f\"{import_project_path}/labeled-data\"\n            )\n        assert (\n            import_labeled_data_path.exists()\n        ), \"import_project has no directory 'labeled-data'\"\n        if not isinstance(video_filenames, List):\n            video_filenames = [video_filenames]\n        for video_file in video_filenames:\n            h5_file = glob.glob(\n                f\"{import_labeled_data_path.as_posix()}/{video_file}/*.h5\"\n            )[0]\n            dlc_df = pd.read_hdf(h5_file)\n            dlc_df.columns = dlc_df.columns.set_levels([team_name], level=0)\n            dlc_df.to_hdf(\n                Path(\n                    f\"{current_labeled_data_path.as_posix()}/\"\n                    f\"{video_file}/CollectedData_{team_name}.h5\"\n                ).as_posix(),\n                \"df_with_missing\",\n            )\n        cls.add_training_files(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.BodyPart", "title": "<code>BodyPart</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Part</code></p> <p>Part table to hold bodyparts used in each project.</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>class BodyPart(SpyglassMixin, dj.Part):\n    \"\"\"Part table to hold bodyparts used in each project.\"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCProject\n    -&gt; BodyPart\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.insert_existing_project", "title": "<code>insert_existing_project(project_name, lab_team, config_path, bodyparts=None, frames_per_video=None, add_to_files=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>insert an existing project into DLCProject table.</p> <p>Parameters:</p> Name Type Description Default <code>project_name</code> <code>str</code> <p>user-friendly name of project</p> required <code>lab_team</code> <code>str</code> <p>name of lab team. Should match an entry in LabTeam table</p> required <code>config_path</code> <code>str</code> <p>path to project directory</p> required <code>bodyparts</code> <code>list</code> <p>optional list of bodyparts to label that are not already in existing config</p> <code>None</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef insert_existing_project(\n    cls,\n    project_name: str,\n    lab_team: str,\n    config_path: str,\n    bodyparts: List = None,\n    frames_per_video: int = None,\n    add_to_files: bool = True,\n    **kwargs,\n):\n    \"\"\"\n    insert an existing project into DLCProject table.\n    Parameters\n    ----------\n    project_name : str\n        user-friendly name of project\n    lab_team : str\n        name of lab team. Should match an entry in LabTeam table\n    config_path : str\n        path to project directory\n    bodyparts : list\n        optional list of bodyparts to label that\n        are not already in existing config\n    \"\"\"\n\n    # Read config\n    project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n    if project_name in project_names_in_use:\n        print(f\"project name: {project_name} is already in use.\")\n        return_key = {}\n        return_key[\"project_name\"], return_key[\"config_path\"] = (\n            cls &amp; {\"project_name\": project_name}\n        ).fetch1(\"project_name\", \"config_path\")\n        return return_key\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    cfg = read_config(config_path)\n    if bodyparts:\n        bodyparts_to_add = [\n            bodypart\n            for bodypart in bodyparts\n            if bodypart not in cfg[\"bodyparts\"]\n        ]\n        all_bodyparts = bodyparts_to_add + cfg[\"bodyparts\"]\n    else:\n        all_bodyparts = cfg[\"bodyparts\"]\n    BodyPart.add_from_config(cfg[\"bodyparts\"])\n    for bodypart in all_bodyparts:\n        if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n            raise ValueError(\n                f\"bodypart: {bodypart} not found in BodyPart table\"\n            )\n    # check bodyparts are in config, if not add\n    if len(bodyparts_to_add) &gt; 0:\n        add_to_config(config_path, bodyparts=bodyparts_to_add)\n    # Get frames per video from config. If passed as arg, check match\n    if frames_per_video:\n        if frames_per_video != cfg[\"numframes2pick\"]:\n            add_to_config(\n                config_path, **{\"numframes2pick\": frames_per_video}\n            )\n    config_path = Path(config_path)\n    project_path = config_path.parent\n    dlc_project_path = dlc_project_dir\n    if dlc_project_path not in project_path.as_posix():\n        project_dirname = project_path.name\n        dest_folder = Path(f\"{dlc_project_path}/{project_dirname}/\")\n        if dest_folder.exists():\n            new_proj_dir = dest_folder.as_posix()\n        else:\n            new_proj_dir = shutil.copytree(\n                src=project_path,\n                dst=f\"{dlc_project_path}/{project_dirname}/\",\n            )\n        new_config_path = Path(f\"{new_proj_dir}/config.yaml\")\n        assert (\n            new_config_path.exists()\n        ), \"config.yaml does not exist in new project directory\"\n        config_path = new_config_path\n        add_to_config(config_path, **{\"project_path\": new_proj_dir})\n    # TODO still need to copy videos over to video dir\n    key = {\n        \"project_name\": project_name,\n        \"team_name\": lab_team,\n        \"bodyparts\": bodyparts,\n        \"config_path\": config_path.as_posix(),\n        \"frames_per_video\": frames_per_video,\n    }\n    cls.insert1(key, **kwargs)\n    cls.BodyPart.insert(\n        [\n            {\"project_name\": project_name, \"bodypart\": bp}\n            for bp in all_bodyparts\n        ],\n        **kwargs,\n    )\n    if add_to_files:\n        del key[\"bodyparts\"]\n        del key[\"team_name\"]\n        del key[\"config_path\"]\n        del key[\"frames_per_video\"]\n        # Check for training files to add\n        cls.add_training_files(key, **kwargs)\n    return {\n        \"project_name\": project_name,\n        \"config_path\": config_path.as_posix(),\n    }\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.insert_new_project", "title": "<code>insert_new_project(project_name, bodyparts, lab_team, frames_per_video, video_list, groupname=None, project_directory=dlc_project_dir, output_path=dlc_video_dir, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert a new project into DLCProject table.</p> <p>Parameters:</p> Name Type Description Default <code>project_name</code> <code>str</code> <p>user-friendly name of project</p> required <code>groupname</code> <code>str</code> <p>Name for project group. If None, defaults to username</p> <code>None</code> <code>bodyparts</code> <code>list</code> <p>list of bodyparts to label. Should match bodyparts in BodyPart table</p> required <code>lab_team</code> <code>str</code> <p>name of lab team. Should match an entry in LabTeam table</p> required <code>project_directory</code> <code>str</code> <p>directory where to create project. (Default is '/cumulus/deeplabcut/')</p> <code>dlc_project_dir</code> <code>frames_per_video</code> <code>int</code> <p>number of frames to extract from each video</p> required <code>video_list</code> <code>list</code> <p>list of (a) dicts of to query VideoFile table for or (b) absolute paths to videos to train on. If dict, use format: [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]</p> required <code>output_path</code> <code>str</code> <p>target path to output converted videos (Default is '/nimbus/deeplabcut/videos/')</p> <code>dlc_video_dir</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef insert_new_project(\n    cls,\n    project_name: str,\n    bodyparts: List,\n    lab_team: str,\n    frames_per_video: int,\n    video_list: List,\n    groupname: str = None,\n    project_directory: str = dlc_project_dir,\n    output_path: str = dlc_video_dir,\n    **kwargs,\n):\n    \"\"\"Insert a new project into DLCProject table.\n\n    Parameters\n    ----------\n    project_name : str\n        user-friendly name of project\n    groupname : str, optional\n        Name for project group. If None, defaults to username\n    bodyparts : list\n        list of bodyparts to label. Should match bodyparts in BodyPart table\n    lab_team : str\n        name of lab team. Should match an entry in LabTeam table\n    project_directory : str\n        directory where to create project.\n        (Default is '/cumulus/deeplabcut/')\n    frames_per_video : int\n        number of frames to extract from each video\n    video_list : list\n        list of (a) dicts of to query VideoFile table for or (b) absolute\n        paths to videos to train on. If dict, use format:\n        [{'nwb_file_name': nwb_file_name, 'epoch': epoch #},...]\n    output_path : str\n        target path to output converted videos\n        (Default is '/nimbus/deeplabcut/videos/')\n    \"\"\"\n    project_names_in_use = np.unique(cls.fetch(\"project_name\"))\n    if project_name in project_names_in_use:\n        print(f\"project name: {project_name} is already in use.\")\n        return_key = {}\n        return_key[\"project_name\"], return_key[\"config_path\"] = (\n            cls &amp; {\"project_name\": project_name}\n        ).fetch1(\"project_name\", \"config_path\")\n        return return_key\n\n    add_to_files = kwargs.pop(\"add_to_files\", True)\n    if not bool(LabTeam() &amp; {\"team_name\": lab_team}):\n        raise ValueError(f\"team_name: {lab_team} does not exist in LabTeam\")\n    skeleton_node = None\n    # If dict, assume of form {'nwb_file_name': nwb_file_name, 'epoch': epoch}\n    # and pass to get_video_path to reference VideoFile table for path\n\n    if all(isinstance(n, Dict) for n in video_list):\n        videos_to_convert = [\n            get_video_path(video_key) for video_key in video_list\n        ]\n        videos = [\n            check_videofile(\n                video_path=video[0],\n                output_path=output_path,\n                video_filename=video[1],\n            )[0].as_posix()\n            for video in videos_to_convert\n            if video[0] is not None\n        ]\n        if len(videos) &lt; 1:\n            raise ValueError(\n                f\"no .mp4 videos found in {videos_to_convert[0][0]}\"\n                + f\" for key: {video_list[0]}\"\n            )\n\n    # If not dict, assume list of video file paths that may or may not need to be converted\n    else:\n        videos = []\n        if not all([Path(video).exists() for video in video_list]):\n            raise OSError(\"at least one file in video_list does not exist\")\n        for video in video_list:\n            video_path = Path(video).parent\n            video_filename = video.rsplit(\n                video_path.as_posix(), maxsplit=1\n            )[-1].split(\"/\")[-1]\n            videos.extend(\n                [\n                    check_videofile(\n                        video_path=video_path,\n                        output_path=output_path,\n                        video_filename=video_filename,\n                    )[0].as_posix()\n                ]\n            )\n        if len(videos) &lt; 1:\n            raise ValueError(f\"no .mp4 videos found in{video_path}\")\n    from deeplabcut import create_new_project\n\n    config_path = create_new_project(\n        project_name,\n        lab_team,\n        videos,\n        working_directory=project_directory,\n        copy_videos=True,\n        multianimal=False,\n    )\n    for bodypart in bodyparts:\n        if not bool(BodyPart() &amp; {\"bodypart\": bodypart}):\n            raise ValueError(\n                f\"bodypart: {bodypart} not found in BodyPart table\"\n            )\n    kwargs_copy = copy.deepcopy(kwargs)\n    kwargs_copy.update({\"numframes2pick\": frames_per_video, \"dotsize\": 3})\n    add_to_config(\n        config_path, bodyparts, skeleton_node=skeleton_node, **kwargs_copy\n    )\n    key = {\n        \"project_name\": project_name,\n        \"team_name\": lab_team,\n        \"bodyparts\": bodyparts,\n        \"config_path\": config_path,\n        \"frames_per_video\": frames_per_video,\n    }\n    cls.insert1(key, **kwargs)\n    cls.BodyPart.insert(\n        [\n            {\"project_name\": project_name, \"bodypart\": bp}\n            for bp in bodyparts\n        ],\n        **kwargs,\n    )\n    if add_to_files:\n        del key[\"bodyparts\"]\n        del key[\"team_name\"]\n        del key[\"config_path\"]\n        del key[\"frames_per_video\"]\n        # Add videos to training files\n        cls.add_training_files(key, **kwargs)\n    if isinstance(config_path, PosixPath):\n        config_path = config_path.as_posix()\n    return {\"project_name\": project_name, \"config_path\": config_path}\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.add_training_files", "title": "<code>add_training_files(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Add training videos and labeled frames .h5 and .csv to DLCProject.File</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef add_training_files(cls, key, **kwargs):\n    \"\"\"Add training videos and labeled frames .h5\n    and .csv to DLCProject.File\"\"\"\n    config_path = (cls &amp; {\"project_name\": key[\"project_name\"]}).fetch1(\n        \"config_path\"\n    )\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    if \"config_path\" in key:\n        del key[\"config_path\"]\n    cfg = read_config(config_path)\n    video_names = list(cfg[\"video_sets\"].keys())\n    training_files = []\n    for video in video_names:\n        video_name = os.path.splitext(\n            video.split(os.path.dirname(video) + \"/\")[-1]\n        )[0]\n        training_files.extend(\n            glob.glob(\n                f\"{cfg['project_path']}/\"\n                f\"labeled-data/{video_name}/*Collected*\"\n            )\n        )\n    for video in video_names:\n        key[\"file_name\"] = f'{os.path.splitext(video.split(\"/\")[-1])[0]}'\n        key[\"file_ext\"] = os.path.splitext(video.split(\"/\")[-1])[-1].split(\n            \".\"\n        )[-1]\n        key[\"file_path\"] = video\n        cls.File.insert1(key, **kwargs)\n    if len(training_files) &gt; 0:\n        for file in training_files:\n            video_name = os.path.dirname(file).split(\"/\")[-1]\n            file_type = os.path.splitext(\n                file.split(os.path.dirname(file) + \"/\")[-1]\n            )[-1].split(\".\")[-1]\n            key[\"file_name\"] = f\"{video_name}_labeled_data\"\n            key[\"file_ext\"] = file_type\n            key[\"file_path\"] = file\n            cls.File.insert1(key, **kwargs)\n    else:\n        Warning(\"No training files to add\")\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.run_extract_frames", "title": "<code>run_extract_frames(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Convenience function to launch DLC GUI for extracting frames. Must be run on local machine to access GUI, cannot be run through ssh tunnel</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef run_extract_frames(cls, key, **kwargs):\n    \"\"\"Convenience function to launch DLC GUI for extracting frames.\n    Must be run on local machine to access GUI,\n    cannot be run through ssh tunnel\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import extract_frames\n\n    extract_frames(config_path, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.run_label_frames", "title": "<code>run_label_frames(key)</code>  <code>classmethod</code>", "text": "<p>Convenience function to launch DLC GUI for labeling frames. Must be run on local machine to access GUI, cannot be run through ssh tunnel</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef run_label_frames(cls, key):\n    \"\"\"Convenience function to launch DLC GUI for labeling frames.\n    Must be run on local machine to access GUI,\n    cannot be run through ssh tunnel\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import label_frames\n\n    label_frames(config_path)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.check_labels", "title": "<code>check_labels(key, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Convenience function to check labels on previously extracted and labeled frames</p> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef check_labels(cls, key, **kwargs):\n    \"\"\"Convenience function to check labels on\n    previously extracted and labeled frames\n    \"\"\"\n    config_path = (cls &amp; key).fetch1(\"config_path\")\n    from deeplabcut import check_labels\n\n    check_labels(config_path, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.DLCProject.import_labeled_frames", "title": "<code>import_labeled_frames(key, import_project_path, video_filenames, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Function to import pre-labeled frames from an existing project into a new project</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>Dict</code> <p>key to specify entry in DLCProject table to add labeled frames to</p> required <code>import_project_path</code> <code>str</code> <p>absolute path to project directory containing labeled frames to import</p> required <code>video_filenames</code> <code>str or List</code> <p>filename or list of filenames of video(s) from which to import frames. without file extension</p> required Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>@classmethod\ndef import_labeled_frames(\n    cls,\n    key: Dict,\n    import_project_path: Union[str, PosixPath],\n    video_filenames: Union[str, List],\n    **kwargs,\n):\n    \"\"\"Function to import pre-labeled frames from an existing project\n    into a new project\n\n    Parameters\n    ----------\n    key : Dict\n        key to specify entry in DLCProject table to add labeled frames to\n    import_project_path : str\n        absolute path to project directory containing\n        labeled frames to import\n    video_filenames : str or List\n        filename or list of filenames of video(s)\n        from which to import frames.\n        without file extension\n    \"\"\"\n    project_entry = (cls &amp; key).fetch1()\n    team_name = project_entry[\"team_name\"]\n    current_project_path = Path(project_entry[\"config_path\"]).parent\n    current_labeled_data_path = Path(\n        f\"{current_project_path.as_posix()}/labeled-data\"\n    )\n    if isinstance(import_project_path, PosixPath):\n        assert import_project_path.exists(), (\n            \"import_project_path: \"\n            f\"{import_project_path.as_posix()} does not exist\"\n        )\n        import_labeled_data_path = Path(\n            f\"{import_project_path.as_posix()}/labeled-data\"\n        )\n    else:\n        assert Path(\n            import_project_path\n        ).exists(), (\n            f\"import_project_path: {import_project_path} does not exist\"\n        )\n        import_labeled_data_path = Path(\n            f\"{import_project_path}/labeled-data\"\n        )\n    assert (\n        import_labeled_data_path.exists()\n    ), \"import_project has no directory 'labeled-data'\"\n    if not isinstance(video_filenames, List):\n        video_filenames = [video_filenames]\n    for video_file in video_filenames:\n        h5_file = glob.glob(\n            f\"{import_labeled_data_path.as_posix()}/{video_file}/*.h5\"\n        )[0]\n        dlc_df = pd.read_hdf(h5_file)\n        dlc_df.columns = dlc_df.columns.set_levels([team_name], level=0)\n        dlc_df.to_hdf(\n            Path(\n                f\"{current_labeled_data_path.as_posix()}/\"\n                f\"{video_file}/CollectedData_{team_name}.h5\"\n            ).as_posix(),\n            \"df_with_missing\",\n        )\n    cls.add_training_files(key, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_project/#src.spyglass.position.v1.position_dlc_project.add_to_config", "title": "<code>add_to_config(config, bodyparts=None, skeleton_node=None, **kwargs)</code>", "text": "<p>Add necessary items to the config.yaml for the model</p> <p>Parameters:</p> Name Type Description Default <code>config</code> <code>str</code> <p>Path to config.yaml</p> required <code>bodyparts</code> <code>list</code> <p>list of bodyparts to add to model</p> <code>None</code> <code>skeleton_node</code> <code>str</code> <p>(default is None) node to link LEDs in skeleton</p> <code>None</code> <code>kwargs</code> <code>dict</code> <p>Other parameters of config to modify in key:value pairs</p> <code>{}</code> Source code in <code>src/spyglass/position/v1/position_dlc_project.py</code> <pre><code>def add_to_config(\n    config, bodyparts: List = None, skeleton_node: str = None, **kwargs\n):\n    \"\"\"\n    Add necessary items to the config.yaml for the model\n    Parameters\n    ----------\n    config : str\n        Path to config.yaml\n    bodyparts : list\n        list of bodyparts to add to model\n    skeleton_node : str\n        (default is None) node to link LEDs in skeleton\n    kwargs : dict\n        Other parameters of config to modify in key:value pairs\n    \"\"\"\n\n    yaml = ruamel.yaml.YAML()\n    with open(config) as fp:\n        data = yaml.load(fp)\n    if bodyparts:\n        data[\"bodyparts\"] = bodyparts\n        led_parts = [element for element in bodyparts if \"LED\" in element]\n        if skeleton_node is not None:\n            bodypart_skeleton = [\n                list(link)\n                for link in combinations(led_parts, 2)\n                if skeleton_node in link\n            ]\n        else:\n            bodypart_skeleton = list(combinations(led_parts, 2))\n        other_parts = list(set(bodyparts) - set(led_parts))\n        for ind, part in enumerate(other_parts):\n            other_parts[ind] = [part, part]\n        bodypart_skeleton.append(other_parts)\n        data[\"skeleton\"] = bodypart_skeleton\n    for kwarg, val in kwargs.items():\n        if not isinstance(kwarg, str):\n            kwarg = str(kwarg)\n        data[kwarg] = val\n    with open(config, \"w\") as fw:\n        yaml.dump(data, fw)\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/", "title": "position_dlc_selection.py", "text": ""}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosSelection", "title": "<code>DLCPosSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Specify collection of upstream DLCCentroid and DLCOrientation entries to combine into a set of position information</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Specify collection of upstream DLCCentroid and DLCOrientation entries\n    to combine into a set of position information\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCCentroid.proj(dlc_si_cohort_centroid='dlc_si_cohort_selection_name', centroid_analysis_file_name='analysis_file_name')\n    -&gt; DLCOrientation.proj(dlc_si_cohort_orientation='dlc_si_cohort_selection_name', orientation_analysis_file_name='analysis_file_name')\n    \"\"\"\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosV1", "title": "<code>DLCPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Combines upstream DLCCentroid and DLCOrientation entries into a single entry with a single Analysis NWB file</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosV1(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Combines upstream DLCCentroid and DLCOrientation\n    entries into a single entry with a single Analysis NWB file\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCPosSelection\n    ---\n    -&gt; AnalysisNwbfile\n    position_object_id      : varchar(80)\n    orientation_object_id   : varchar(80)\n    velocity_object_id      : varchar(80)\n    pose_eval_result        : longblob\n    \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n        key[\"pose_eval_result\"] = self.evaluate_pose_estimation(key)\n\n        pos_nwb = (DLCCentroid &amp; key).fetch_nwb()[0]\n        ori_nwb = (DLCOrientation &amp; key).fetch_nwb()[0]\n\n        pos_obj = pos_nwb[\"dlc_position\"].spatial_series[\"position\"]\n        vel_obj = pos_nwb[\"dlc_velocity\"].time_series[\"velocity\"]\n        vid_frame_obj = pos_nwb[\"dlc_velocity\"].time_series[\"video_frame_ind\"]\n        ori_obj = ori_nwb[\"dlc_orientation\"].spatial_series[\"orientation\"]\n\n        position = pynwb.behavior.Position()\n        orientation = pynwb.behavior.CompassDirection()\n        velocity = pynwb.behavior.BehavioralTimeSeries()\n\n        position.create_spatial_series(\n            name=pos_obj.name,\n            timestamps=np.asarray(pos_obj.timestamps),\n            conversion=pos_obj.conversion,\n            data=np.asarray(pos_obj.data),\n            reference_frame=pos_obj.reference_frame,\n            comments=pos_obj.comments,\n            description=pos_obj.description,\n        )\n\n        orientation.create_spatial_series(\n            name=ori_obj.name,\n            timestamps=np.asarray(ori_obj.timestamps),\n            conversion=ori_obj.conversion,\n            data=np.asarray(ori_obj.data),\n            reference_frame=ori_obj.reference_frame,\n            comments=ori_obj.comments,\n            description=ori_obj.description,\n        )\n\n        velocity.create_timeseries(\n            name=vel_obj.name,\n            timestamps=np.asarray(vel_obj.timestamps),\n            conversion=vel_obj.conversion,\n            unit=vel_obj.unit,\n            data=np.asarray(vel_obj.data),\n            comments=vel_obj.comments,\n            description=vel_obj.description,\n        )\n\n        velocity.create_timeseries(\n            name=vid_frame_obj.name,\n            unit=vid_frame_obj.unit,\n            timestamps=np.asarray(vid_frame_obj.timestamps),\n            data=np.asarray(vid_frame_obj.data),\n            description=vid_frame_obj.description,\n            comments=vid_frame_obj.comments,\n        )\n\n        # Add to Analysis NWB file\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"orientation_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], orientation\n        )\n        key[\"position_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], position\n        )\n        key[\"velocity_object_id\"] = nwb_analysis_file.add_nwb_object(\n            key[\"analysis_file_name\"], velocity\n        )\n\n        nwb_analysis_file.add(\n            nwb_file_name=key[\"nwb_file_name\"],\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n        from ..position_merge import PositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n        # TODO: The next line belongs in a merge table function\n        PositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    def fetch1_dataframe(self):\n        nwb_data = self.fetch_nwb()[0]\n        index = pd.Index(\n            np.asarray(nwb_data[\"position\"].get_spatial_series().timestamps),\n            name=\"time\",\n        )\n        COLUMNS = [\n            \"video_frame_ind\",\n            \"position_x\",\n            \"position_y\",\n            \"orientation\",\n            \"velocity_x\",\n            \"velocity_y\",\n            \"speed\",\n        ]\n        return pd.DataFrame(\n            np.concatenate(\n                (\n                    np.asarray(\n                        nwb_data[\"velocity\"]\n                        .time_series[\"video_frame_ind\"]\n                        .data,\n                        dtype=int,\n                    )[:, np.newaxis],\n                    np.asarray(nwb_data[\"position\"].get_spatial_series().data),\n                    np.asarray(\n                        nwb_data[\"orientation\"].get_spatial_series().data\n                    )[:, np.newaxis],\n                    np.asarray(\n                        nwb_data[\"velocity\"].time_series[\"velocity\"].data\n                    ),\n                ),\n                axis=1,\n            ),\n            columns=COLUMNS,\n            index=index,\n        )\n\n    @classmethod\n    def evaluate_pose_estimation(cls, key):\n        likelihood_thresh = []\n        valid_fields = (\n            DLCSmoothInterpCohort.BodyPart().fetch().dtype.fields.keys()\n        )\n        centroid_key = {k: val for k, val in key.items() if k in valid_fields}\n        centroid_key[\"dlc_si_cohort_selection_name\"] = key[\n            \"dlc_si_cohort_centroid\"\n        ]\n        orientation_key = centroid_key.copy()\n        orientation_key[\"dlc_si_cohort_selection_name\"] = key[\n            \"dlc_si_cohort_orientation\"\n        ]\n        centroid_bodyparts, centroid_si_params = (\n            DLCSmoothInterpCohort.BodyPart &amp; centroid_key\n        ).fetch(\"bodypart\", \"dlc_si_params_name\")\n        orientation_bodyparts, orientation_si_params = (\n            DLCSmoothInterpCohort.BodyPart &amp; orientation_key\n        ).fetch(\"bodypart\", \"dlc_si_params_name\")\n        for param in np.unique(\n            np.concatenate((centroid_si_params, orientation_si_params))\n        ):\n            likelihood_thresh.append(\n                (\n                    DLCSmoothInterpParams() &amp; {\"dlc_si_params_name\": param}\n                ).fetch1(\"params\")[\"likelihood_thresh\"]\n            )\n\n        if len(np.unique(likelihood_thresh)) &gt; 1:\n            raise ValueError(\"more than one likelihood threshold used\")\n        like_thresh = likelihood_thresh[0]\n        bodyparts = np.unique([*centroid_bodyparts, *orientation_bodyparts])\n        fields = list(DLCPoseEstimation.BodyPart.fetch().dtype.fields.keys())\n        pose_estimation_key = {k: v for k, v in key.items() if k in fields}\n        pose_estimation_df = pd.concat(\n            {\n                bodypart: (\n                    DLCPoseEstimation.BodyPart()\n                    &amp; {**pose_estimation_key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in bodyparts.tolist()\n            },\n            axis=1,\n        )\n        df_filter = {\n            bodypart: pose_estimation_df[bodypart][\"likelihood\"] &lt; like_thresh\n            for bodypart in bodyparts\n            if bodypart in pose_estimation_df.columns\n        }\n        sub_thresh_percent_dict = {\n            bodypart: (\n                len(\n                    np.where(\n                        ~np.isnan(\n                            pose_estimation_df[bodypart][\"likelihood\"].where(\n                                df_filter[bodypart]\n                            )\n                        )\n                    )[0]\n                )\n                / len(pose_estimation_df)\n            )\n            * 100\n            for bodypart in bodyparts\n        }\n        return sub_thresh_percent_dict\n</code></pre>"}, {"location": "api/position/v1/position_dlc_selection/#src.spyglass.position.v1.position_dlc_selection.DLCPosVideo", "title": "<code>DLCPosVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/v1/position_dlc_selection.py</code> <pre><code>@schema\nclass DLCPosVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; DLCPosVideoSelection\n    ---\n    \"\"\"\n\n    def make(self, key):\n        from tqdm import tqdm as tqdm\n\n        params = (DLCPosVideoParams &amp; key).fetch1(\"params\")\n        if \"video_params\" not in params:\n            params[\"video_params\"] = {}\n        M_TO_CM = 100\n        interval_list_name = (\n            convert_epoch_interval_name_to_position_interval_name(\n                {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"epoch\": key[\"epoch\"],\n                },\n                populate_missing=False,\n            )\n        )\n        key[\"interval_list_name\"] = interval_list_name\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n        pose_estimation_key = {\n            \"nwb_file_name\": key[\"nwb_file_name\"],\n            \"epoch\": epoch,\n            \"dlc_model_name\": key[\"dlc_model_name\"],\n            \"dlc_model_params_name\": key[\"dlc_model_params_name\"],\n        }\n        pose_estimation_params, video_filename, output_dir = (\n            DLCPoseEstimationSelection() &amp; pose_estimation_key\n        ).fetch1(\n            \"pose_estimation_params\", \"video_path\", \"pose_estimation_output_dir\"\n        )\n        print(f\"video filename: {video_filename}\")\n        meters_per_pixel = (DLCPoseEstimation() &amp; pose_estimation_key).fetch1(\n            \"meters_per_pixel\"\n        )\n        crop = None\n        if \"cropping\" in pose_estimation_params:\n            crop = pose_estimation_params[\"cropping\"]\n        print(\"Loading position data...\")\n        position_info_df = (\n            DLCPosV1()\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"epoch\": epoch,\n                \"dlc_si_cohort_centroid\": key[\"dlc_si_cohort_centroid\"],\n                \"dlc_centroid_params_name\": key[\"dlc_centroid_params_name\"],\n                \"dlc_si_cohort_orientation\": key[\"dlc_si_cohort_orientation\"],\n                \"dlc_orientation_params_name\": key[\n                    \"dlc_orientation_params_name\"\n                ],\n            }\n        ).fetch1_dataframe()\n        pose_estimation_df = pd.concat(\n            {\n                bodypart: (\n                    DLCPoseEstimation.BodyPart()\n                    &amp; {**pose_estimation_key, **{\"bodypart\": bodypart}}\n                ).fetch1_dataframe()\n                for bodypart in (\n                    DLCSmoothInterpCohort.BodyPart &amp; pose_estimation_key\n                )\n                .fetch(\"bodypart\")\n                .tolist()\n            },\n            axis=1,\n        )\n        assert len(pose_estimation_df) == len(position_info_df), (\n            f\"length of pose_estimation_df: {len(pose_estimation_df)} \"\n            f\"does not match the length of position_info_df: {len(position_info_df)}.\"\n        )\n\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        if Path(output_dir).exists():\n            output_video_filename = (\n                f\"{Path(output_dir).as_posix()}/\"\n                f\"{nwb_base_filename}_{epoch:02d}_\"\n                f'{key[\"dlc_si_cohort_centroid\"]}_'\n                f'{key[\"dlc_centroid_params_name\"]}'\n                f'{key[\"dlc_orientation_params_name\"]}.mp4'\n            )\n        else:\n            output_video_filename = (\n                f\"{nwb_base_filename}_{epoch:02d}_\"\n                f'{key[\"dlc_si_cohort_centroid\"]}_'\n                f'{key[\"dlc_centroid_params_name\"]}'\n                f'{key[\"dlc_orientation_params_name\"]}.mp4'\n            )\n        idx = pd.IndexSlice\n        video_frame_inds = (\n            position_info_df[\"video_frame_ind\"].astype(int).to_numpy()\n        )\n        centroids = {\n            bodypart: pose_estimation_df.loc[\n                :, idx[bodypart, (\"x\", \"y\")]\n            ].to_numpy()\n            for bodypart in pose_estimation_df.columns.levels[0]\n        }\n        if params.get(\"incl_likelihood\", None):\n            likelihoods = {\n                bodypart: pose_estimation_df.loc[\n                    :, idx[bodypart, (\"likelihood\")]\n                ].to_numpy()\n                for bodypart in pose_estimation_df.columns.levels[0]\n            }\n        else:\n            likelihoods = None\n        position_mean = {\n            \"DLC\": np.asarray(position_info_df[[\"position_x\", \"position_y\"]])\n        }\n        orientation_mean = {\n            \"DLC\": np.asarray(position_info_df[[\"orientation\"]])\n        }\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = meters_per_pixel * M_TO_CM\n        percent_frames = params.get(\"percent_frames\", None)\n        frames = params.get(\"frames\", None)\n        if frames is not None:\n            frames_arr = np.arange(frames[0], frames[1])\n        else:\n            frames_arr = frames\n\n        print(\"Making video...\")\n        make_video(\n            video_filename=video_filename,\n            video_frame_inds=video_frame_inds,\n            position_mean=position_mean,\n            orientation_mean=orientation_mean,\n            centroids=centroids,\n            likelihoods=likelihoods,\n            position_time=position_time,\n            video_time=None,\n            processor=params.get(\"processor\", \"matplotlib\"),\n            frames=frames_arr,\n            percent_frames=percent_frames,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n            crop=crop,\n            **params[\"video_params\"],\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/", "title": "position_dlc_training.py", "text": ""}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTrainingParams", "title": "<code>DLCModelTrainingParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@schema\nclass DLCModelTrainingParams(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    # Parameters to specify a DLC model training instance\n    # For DLC \u2264 v2.0, include scorer_lecacy = True in params\n    dlc_training_params_name      : varchar(50) # descriptive name of parameter set\n    ---\n    params                        : longblob    # dictionary of all applicable parameters\n    \"\"\"\n\n    required_parameters = (\n        \"shuffle\",\n        \"trainingsetindex\",\n        \"net_type\",\n        \"gputouse\",\n    )\n    skipped_parameters = (\"project_path\", \"video_sets\")\n\n    @classmethod\n    def insert_new_params(cls, paramset_name: str, params: dict, **kwargs):\n        \"\"\"\n        Insert a new set of training parameters into dlc.TrainingParamSet.\n\n        Parameters\n        ----------\n        paramset_name : str\n            Description of parameter set to be inserted\n        params : dict\n            Dictionary including all settings to specify model training.\n            Must include shuffle &amp; trainingsetindex b/c not in config.yaml.\n            project_path and video_sets will be overwritten by config.yaml.\n            Note that trainingsetindex is 0-indexed\n        \"\"\"\n\n        for required_param in cls.required_parameters:\n            assert required_param in params, (\n                \"Missing required parameter: \" + required_param\n            )\n        for skipped_param in cls.skipped_parameters:\n            if skipped_param in params:\n                params.pop(skipped_param)\n\n        param_dict = {\n            \"dlc_training_params_name\": paramset_name,\n            \"params\": params,\n        }\n        param_query = cls &amp; {\n            \"dlc_training_params_name\": param_dict[\"dlc_training_params_name\"]\n        }\n        # If the specified param-set already exists\n        # Not sure we need this part, as much just a check if the name is the same\n        if param_query:\n            existing_paramset_name = param_query.fetch1(\n                \"dlc_training_params_name\"\n            )\n            if (\n                existing_paramset_name == paramset_name\n            ):  # If existing name same:\n                return print(\n                    f\"New param set not added\\n\"\n                    f\"A param set with name: {paramset_name} already exists\"\n                )\n        else:\n            cls.insert1(\n                param_dict, **kwargs\n            )  # if duplicate, will raise duplicate error\n            # if this will raise duplicate error, why is above check needed? @datajoint\n\n    @classmethod\n    def get_accepted_params(cls):\n        from deeplabcut import create_training_dataset, train_network\n\n        return list(\n            set(\n                [\n                    *list(inspect.signature(train_network).parameters),\n                    *list(\n                        inspect.signature(create_training_dataset).parameters\n                    ),\n                ]\n            )\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTrainingParams.insert_new_params", "title": "<code>insert_new_params(paramset_name, params, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert a new set of training parameters into dlc.TrainingParamSet.</p> <p>Parameters:</p> Name Type Description Default <code>paramset_name</code> <code>str</code> <p>Description of parameter set to be inserted</p> required <code>params</code> <code>dict</code> <p>Dictionary including all settings to specify model training. Must include shuffle &amp; trainingsetindex b/c not in config.yaml. project_path and video_sets will be overwritten by config.yaml. Note that trainingsetindex is 0-indexed</p> required Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@classmethod\ndef insert_new_params(cls, paramset_name: str, params: dict, **kwargs):\n    \"\"\"\n    Insert a new set of training parameters into dlc.TrainingParamSet.\n\n    Parameters\n    ----------\n    paramset_name : str\n        Description of parameter set to be inserted\n    params : dict\n        Dictionary including all settings to specify model training.\n        Must include shuffle &amp; trainingsetindex b/c not in config.yaml.\n        project_path and video_sets will be overwritten by config.yaml.\n        Note that trainingsetindex is 0-indexed\n    \"\"\"\n\n    for required_param in cls.required_parameters:\n        assert required_param in params, (\n            \"Missing required parameter: \" + required_param\n        )\n    for skipped_param in cls.skipped_parameters:\n        if skipped_param in params:\n            params.pop(skipped_param)\n\n    param_dict = {\n        \"dlc_training_params_name\": paramset_name,\n        \"params\": params,\n    }\n    param_query = cls &amp; {\n        \"dlc_training_params_name\": param_dict[\"dlc_training_params_name\"]\n    }\n    # If the specified param-set already exists\n    # Not sure we need this part, as much just a check if the name is the same\n    if param_query:\n        existing_paramset_name = param_query.fetch1(\n            \"dlc_training_params_name\"\n        )\n        if (\n            existing_paramset_name == paramset_name\n        ):  # If existing name same:\n            return print(\n                f\"New param set not added\\n\"\n                f\"A param set with name: {paramset_name} already exists\"\n            )\n    else:\n        cls.insert1(\n            param_dict, **kwargs\n        )  # if duplicate, will raise duplicate error\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTraining", "title": "<code>DLCModelTraining</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>@schema\nclass DLCModelTraining(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; DLCModelTrainingSelection\n    ---\n    project_path         : varchar(255) # Path to project directory\n    latest_snapshot: int unsigned # latest exact snapshot index (i.e., never -1)\n    config_template: longblob     # stored full config file\n    \"\"\"\n\n    # To continue from previous training snapshot, devs suggest editing pose_cfg.yml\n    # https://github.com/DeepLabCut/DeepLabCut/issues/70\n\n    def make(self, key):\n        \"\"\"Launch training for each entry in DLCModelTrainingSelection via `.populate()`.\"\"\"\n        model_prefix = (DLCModelTrainingSelection &amp; key).fetch1(\"model_prefix\")\n        from deeplabcut import create_training_dataset, train_network\n        from deeplabcut.utils.auxiliaryfunctions import read_config\n\n        from . import dlc_reader\n\n        try:\n            from deeplabcut.utils.auxiliaryfunctions import get_model_folder\n        except ImportError:\n            from deeplabcut.utils.auxiliaryfunctions import (\n                GetModelFolder as get_model_folder,\n            )\n        config_path, project_name = (DLCProject() &amp; key).fetch1(\n            \"config_path\", \"project_name\"\n        )\n        with OutputLogger(\n            name=\"DLC_project_{project_name}_training\",\n            path=f\"{os.path.dirname(config_path)}/log.log\",\n            print_console=True,\n        ) as logger:\n            dlc_config = read_config(config_path)\n            project_path = dlc_config[\"project_path\"]\n            key[\"project_path\"] = project_path\n            # ---- Build and save DLC configuration (yaml) file ----\n            _, dlc_config = dlc_reader.read_yaml(project_path)\n            if not dlc_config:\n                dlc_config = read_config(config_path)\n            dlc_config.update((DLCModelTrainingParams &amp; key).fetch1(\"params\"))\n            dlc_config.update(\n                {\n                    \"project_path\": Path(project_path).as_posix(),\n                    \"modelprefix\": model_prefix,\n                    \"train_fraction\": dlc_config[\"TrainingFraction\"][\n                        int(dlc_config[\"trainingsetindex\"])\n                    ],\n                    \"training_filelist_datajoint\": [  # don't overwrite origin video_sets\n                        Path(fp).as_posix()\n                        for fp in (DLCProject.File &amp; key).fetch(\"file_path\")\n                    ],\n                }\n            )\n            # Write dlc config file to base project folder\n            # TODO: need to make sure this will work\n            dlc_cfg_filepath = dlc_reader.save_yaml(project_path, dlc_config)\n            # ---- create training dataset ----\n            training_dataset_input_args = list(\n                inspect.signature(create_training_dataset).parameters\n            )\n            training_dataset_kwargs = {\n                k: v\n                for k, v in dlc_config.items()\n                if k in training_dataset_input_args\n            }\n            logger.logger.info(\"creating training dataset\")\n            create_training_dataset(dlc_cfg_filepath, **training_dataset_kwargs)\n            # ---- Trigger DLC model training job ----\n            train_network_input_args = list(\n                inspect.signature(train_network).parameters\n            )\n            train_network_kwargs = {\n                k: v\n                for k, v in dlc_config.items()\n                if k in train_network_input_args\n            }\n            for k in [\"shuffle\", \"trainingsetindex\", \"maxiters\"]:\n                if k in train_network_kwargs:\n                    train_network_kwargs[k] = int(train_network_kwargs[k])\n            try:\n                train_network(dlc_cfg_filepath, **train_network_kwargs)\n            except (\n                KeyboardInterrupt\n            ):  # Instructions indicate to train until interrupt\n                logger.logger.info(\n                    \"DLC training stopped via Keyboard Interrupt\"\n                )\n\n            snapshots = list(\n                (\n                    project_path\n                    / get_model_folder(\n                        trainFraction=dlc_config[\"train_fraction\"],\n                        shuffle=dlc_config[\"shuffle\"],\n                        cfg=dlc_config,\n                        modelprefix=dlc_config[\"modelprefix\"],\n                    )\n                    / \"train\"\n                ).glob(\"*index*\")\n            )\n            max_modified_time = 0\n            # DLC goes by snapshot magnitude when judging 'latest' for evaluation\n            # Here, we mean most recently generated\n            for snapshot in snapshots:\n                modified_time = os.path.getmtime(snapshot)\n                if modified_time &gt; max_modified_time:\n                    latest_snapshot = int(snapshot.stem[9:])\n                    max_modified_time = modified_time\n\n            self.insert1(\n                {\n                    **key,\n                    \"latest_snapshot\": latest_snapshot,\n                    \"config_template\": dlc_config,\n                }\n            )\n            from .position_dlc_model import DLCModelSource\n\n            dlc_model_name = f\"{key['project_name']}_{key['dlc_training_params_name']}_{key['training_id']:02d}\"\n            DLCModelSource.insert_entry(\n                dlc_model_name=dlc_model_name,\n                project_name=key[\"project_name\"],\n                source=\"FromUpstream\",\n                key=key,\n                skip_duplicates=True,\n            )\n        print(\n            f\"Inserted {dlc_model_name} from {key['project_name']} into DLCModelSource\"\n        )\n</code></pre>"}, {"location": "api/position/v1/position_dlc_training/#src.spyglass.position.v1.position_dlc_training.DLCModelTraining.make", "title": "<code>make(key)</code>", "text": "<p>Launch training for each entry in DLCModelTrainingSelection via <code>.populate()</code>.</p> Source code in <code>src/spyglass/position/v1/position_dlc_training.py</code> <pre><code>def make(self, key):\n    \"\"\"Launch training for each entry in DLCModelTrainingSelection via `.populate()`.\"\"\"\n    model_prefix = (DLCModelTrainingSelection &amp; key).fetch1(\"model_prefix\")\n    from deeplabcut import create_training_dataset, train_network\n    from deeplabcut.utils.auxiliaryfunctions import read_config\n\n    from . import dlc_reader\n\n    try:\n        from deeplabcut.utils.auxiliaryfunctions import get_model_folder\n    except ImportError:\n        from deeplabcut.utils.auxiliaryfunctions import (\n            GetModelFolder as get_model_folder,\n        )\n    config_path, project_name = (DLCProject() &amp; key).fetch1(\n        \"config_path\", \"project_name\"\n    )\n    with OutputLogger(\n        name=\"DLC_project_{project_name}_training\",\n        path=f\"{os.path.dirname(config_path)}/log.log\",\n        print_console=True,\n    ) as logger:\n        dlc_config = read_config(config_path)\n        project_path = dlc_config[\"project_path\"]\n        key[\"project_path\"] = project_path\n        # ---- Build and save DLC configuration (yaml) file ----\n        _, dlc_config = dlc_reader.read_yaml(project_path)\n        if not dlc_config:\n            dlc_config = read_config(config_path)\n        dlc_config.update((DLCModelTrainingParams &amp; key).fetch1(\"params\"))\n        dlc_config.update(\n            {\n                \"project_path\": Path(project_path).as_posix(),\n                \"modelprefix\": model_prefix,\n                \"train_fraction\": dlc_config[\"TrainingFraction\"][\n                    int(dlc_config[\"trainingsetindex\"])\n                ],\n                \"training_filelist_datajoint\": [  # don't overwrite origin video_sets\n                    Path(fp).as_posix()\n                    for fp in (DLCProject.File &amp; key).fetch(\"file_path\")\n                ],\n            }\n        )\n        # Write dlc config file to base project folder\n        # TODO: need to make sure this will work\n        dlc_cfg_filepath = dlc_reader.save_yaml(project_path, dlc_config)\n        # ---- create training dataset ----\n        training_dataset_input_args = list(\n            inspect.signature(create_training_dataset).parameters\n        )\n        training_dataset_kwargs = {\n            k: v\n            for k, v in dlc_config.items()\n            if k in training_dataset_input_args\n        }\n        logger.logger.info(\"creating training dataset\")\n        create_training_dataset(dlc_cfg_filepath, **training_dataset_kwargs)\n        # ---- Trigger DLC model training job ----\n        train_network_input_args = list(\n            inspect.signature(train_network).parameters\n        )\n        train_network_kwargs = {\n            k: v\n            for k, v in dlc_config.items()\n            if k in train_network_input_args\n        }\n        for k in [\"shuffle\", \"trainingsetindex\", \"maxiters\"]:\n            if k in train_network_kwargs:\n                train_network_kwargs[k] = int(train_network_kwargs[k])\n        try:\n            train_network(dlc_cfg_filepath, **train_network_kwargs)\n        except (\n            KeyboardInterrupt\n        ):  # Instructions indicate to train until interrupt\n            logger.logger.info(\n                \"DLC training stopped via Keyboard Interrupt\"\n            )\n\n        snapshots = list(\n            (\n                project_path\n                / get_model_folder(\n                    trainFraction=dlc_config[\"train_fraction\"],\n                    shuffle=dlc_config[\"shuffle\"],\n                    cfg=dlc_config,\n                    modelprefix=dlc_config[\"modelprefix\"],\n                )\n                / \"train\"\n            ).glob(\"*index*\")\n        )\n        max_modified_time = 0\n        # DLC goes by snapshot magnitude when judging 'latest' for evaluation\n        # Here, we mean most recently generated\n        for snapshot in snapshots:\n            modified_time = os.path.getmtime(snapshot)\n            if modified_time &gt; max_modified_time:\n                latest_snapshot = int(snapshot.stem[9:])\n                max_modified_time = modified_time\n\n        self.insert1(\n            {\n                **key,\n                \"latest_snapshot\": latest_snapshot,\n                \"config_template\": dlc_config,\n            }\n        )\n        from .position_dlc_model import DLCModelSource\n\n        dlc_model_name = f\"{key['project_name']}_{key['dlc_training_params_name']}_{key['training_id']:02d}\"\n        DLCModelSource.insert_entry(\n            dlc_model_name=dlc_model_name,\n            project_name=key[\"project_name\"],\n            source=\"FromUpstream\",\n            key=key,\n            skip_duplicates=True,\n        )\n    print(\n        f\"Inserted {dlc_model_name} from {key['project_name']} into DLCModelSource\"\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/", "title": "position_trodes_position.py", "text": ""}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosParams", "title": "<code>TrodesPosParams</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Parameters for calculating the position (centroid, velocity, orientation)</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosParams(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Parameters for calculating the position (centroid, velocity, orientation)\n    \"\"\"\n\n    definition = \"\"\"\n    trodes_pos_params_name: varchar(80) # name for this set of parameters\n    ---\n    params: longblob\n    \"\"\"\n\n    @property\n    def default_pk(self):\n        return {\"trodes_pos_params_name\": \"default\"}\n\n    @property\n    def default_params(self):\n        return {\n            \"max_LED_separation\": 9.0,\n            \"max_plausible_speed\": 300.0,\n            \"position_smoothing_duration\": 0.125,\n            \"speed_smoothing_std_dev\": 0.100,\n            \"orient_smoothing_std_dev\": 0.001,\n            \"led1_is_front\": 1,\n            \"is_upsampled\": 0,\n            \"upsampling_sampling_rate\": None,\n            \"upsampling_interpolation_method\": \"linear\",\n        }\n\n    @classmethod\n    def insert_default(cls, **kwargs):\n        \"\"\"\n        Insert default parameter set for position determination\n        \"\"\"\n        cls.insert1(\n            {**cls().default_pk, \"params\": cls().default_params},\n            skip_duplicates=True,\n        )\n\n    @classmethod\n    def get_default(cls):\n        query = cls &amp; cls().default_pk\n        if not len(query) &gt; 0:\n            cls().insert_default(skip_duplicates=True)\n            return (cls &amp; cls().default_pk).fetch1()\n\n        return query.fetch1()\n\n    @classmethod\n    def get_accepted_params(cls):\n        return [k for k in cls().default_params.keys()]\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosParams.insert_default", "title": "<code>insert_default(**kwargs)</code>  <code>classmethod</code>", "text": "<p>Insert default parameter set for position determination</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@classmethod\ndef insert_default(cls, **kwargs):\n    \"\"\"\n    Insert default parameter set for position determination\n    \"\"\"\n    cls.insert1(\n        {**cls().default_pk, \"params\": cls().default_params},\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosSelection", "title": "<code>TrodesPosSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> <p>Table to pair an interval with position data and position determination parameters</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosSelection(SpyglassMixin, dj.Manual):\n    \"\"\"\n    Table to pair an interval with position data\n    and position determination parameters\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; RawPosition\n    -&gt; TrodesPosParams\n    \"\"\"\n\n    @classmethod\n    def insert_with_default(\n        cls,\n        key: dict,\n        skip_duplicates: bool = False,\n        edit_defaults: dict = {},\n        edit_name: str = None,\n    ) -&gt; None:\n        \"\"\"Insert key with default parameters.\n\n        To change defaults, supply a dict as edit_defaults with a name for\n        the new paramset as edit_name.\n\n        Parameters\n        ----------\n        key: Union[dict, str]\n            Restriction uniquely identifying entr(y/ies) in RawPosition.\n        skip_duplicates: bool, optional\n            Skip duplicate entries.\n        edit_defaults: dict, optional\n            Dictionary of overrides to default parameters.\n        edit_name: str, optional\n            If edit_defauts is passed, the name of the new entry\n\n        Raises\n        ------\n        ValueError\n            Key does not identify any entries in RawPosition.\n        \"\"\"\n        query = RawPosition &amp; key\n        if not query:\n            raise ValueError(f\"Found no entries found for {key}\")\n\n        param_pk, param_name = list(TrodesPosParams().default_pk.items())[0]\n\n        if bool(edit_defaults) ^ bool(edit_name):  # XOR: only one of them\n            raise ValueError(\"Must specify both edit_defauts and edit_name\")\n\n        elif edit_defaults and edit_name:\n            TrodesPosParams.insert1(\n                {\n                    param_pk: edit_name,\n                    \"params\": {\n                        **TrodesPosParams().default_params,\n                        **edit_defaults,\n                    },\n                },\n                skip_duplicates=skip_duplicates,\n            )\n\n        cls.insert(\n            [\n                {**k, param_pk: edit_name or param_name}\n                for k in query.fetch(\"KEY\", as_dict=True)\n            ],\n            skip_duplicates=skip_duplicates,\n        )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosSelection.insert_with_default", "title": "<code>insert_with_default(key, skip_duplicates=False, edit_defaults={}, edit_name=None)</code>  <code>classmethod</code>", "text": "<p>Insert key with default parameters.</p> <p>To change defaults, supply a dict as edit_defaults with a name for the new paramset as edit_name.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Restriction uniquely identifying entr(y/ies) in RawPosition.</p> required <code>skip_duplicates</code> <code>bool</code> <p>Skip duplicate entries.</p> <code>False</code> <code>edit_defaults</code> <code>dict</code> <p>Dictionary of overrides to default parameters.</p> <code>{}</code> <code>edit_name</code> <code>str</code> <p>If edit_defauts is passed, the name of the new entry</p> <code>None</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>Key does not identify any entries in RawPosition.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@classmethod\ndef insert_with_default(\n    cls,\n    key: dict,\n    skip_duplicates: bool = False,\n    edit_defaults: dict = {},\n    edit_name: str = None,\n) -&gt; None:\n    \"\"\"Insert key with default parameters.\n\n    To change defaults, supply a dict as edit_defaults with a name for\n    the new paramset as edit_name.\n\n    Parameters\n    ----------\n    key: Union[dict, str]\n        Restriction uniquely identifying entr(y/ies) in RawPosition.\n    skip_duplicates: bool, optional\n        Skip duplicate entries.\n    edit_defaults: dict, optional\n        Dictionary of overrides to default parameters.\n    edit_name: str, optional\n        If edit_defauts is passed, the name of the new entry\n\n    Raises\n    ------\n    ValueError\n        Key does not identify any entries in RawPosition.\n    \"\"\"\n    query = RawPosition &amp; key\n    if not query:\n        raise ValueError(f\"Found no entries found for {key}\")\n\n    param_pk, param_name = list(TrodesPosParams().default_pk.items())[0]\n\n    if bool(edit_defaults) ^ bool(edit_name):  # XOR: only one of them\n        raise ValueError(\"Must specify both edit_defauts and edit_name\")\n\n    elif edit_defaults and edit_name:\n        TrodesPosParams.insert1(\n            {\n                param_pk: edit_name,\n                \"params\": {\n                    **TrodesPosParams().default_params,\n                    **edit_defaults,\n                },\n            },\n            skip_duplicates=skip_duplicates,\n        )\n\n    cls.insert(\n        [\n            {**k, param_pk: edit_name or param_name}\n            for k in query.fetch(\"KEY\", as_dict=True)\n        ],\n        skip_duplicates=skip_duplicates,\n    )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosV1", "title": "<code>TrodesPosV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Table to calculate the position based on Trodes tracking</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosV1(SpyglassMixin, dj.Computed):\n    \"\"\"\n    Table to calculate the position based on Trodes tracking\n    \"\"\"\n\n    definition = \"\"\"\n    -&gt; TrodesPosSelection\n    ---\n    -&gt; AnalysisNwbfile\n    position_object_id : varchar(80)\n    orientation_object_id : varchar(80)\n    velocity_object_id : varchar(80)\n    \"\"\"\n\n    def make(self, key):\n        logger.info(f\"Computing position for: {key}\")\n        orig_key = copy.deepcopy(key)\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n\n        raw_position = RawPosition.PosObject &amp; key\n        spatial_series = raw_position.fetch_nwb()[0][\"raw_position\"]\n        spatial_df = raw_position.fetch1_dataframe()\n\n        position_info_parameters = (TrodesPosParams() &amp; key).fetch1(\"params\")\n        position_info = self.calculate_position_info(\n            spatial_df=spatial_df,\n            meters_to_pixels=spatial_series.conversion,\n            **position_info_parameters,\n        )\n\n        key.update(\n            dict(\n                analysis_file_name=analysis_file_name,\n                **self.generate_pos_components(\n                    spatial_series=spatial_series,\n                    position_info=position_info,\n                    analysis_fname=analysis_file_name,\n                    prefix=\"\",\n                    add_frame_ind=True,\n                    video_frame_ind=getattr(\n                        spatial_df, \"video_frame_ind\", None\n                    ),\n                ),\n            )\n        )\n\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        self.insert1(key)\n\n        from ..position_merge import PositionOutput\n\n        part_name = to_camel_case(self.table_name.split(\"__\")[-1])\n\n        # TODO: The next line belongs in a merge table function\n        PositionOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    @staticmethod\n    def generate_pos_components(*args, **kwargs):\n        return IntervalPositionInfo().generate_pos_components(*args, **kwargs)\n\n    @staticmethod\n    def calculate_position_info(*args, **kwargs):\n        \"\"\"Calculate position info from 2D spatial series.\"\"\"\n        return IntervalPositionInfo().calculate_position_info(*args, **kwargs)\n\n    def fetch1_dataframe(self, add_frame_ind=True):\n        pos_params = self.fetch1(\"trodes_pos_params_name\")\n        if (\n            add_frame_ind\n            and (\n                TrodesPosParams &amp; {\"trodes_pos_params_name\": pos_params}\n            ).fetch1(\"params\")[\"is_upsampled\"]\n        ):\n            logger.warning(\n                \"Upsampled position data, frame indices are invalid. \"\n                + \"Setting add_frame_ind=False\"\n            )\n            add_frame_ind = False\n        return IntervalPositionInfo._data_to_df(\n            self.fetch_nwb()[0], prefix=\"\", add_frame_ind=add_frame_ind\n        )\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosV1.calculate_position_info", "title": "<code>calculate_position_info(*args, **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Calculate position info from 2D spatial series.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@staticmethod\ndef calculate_position_info(*args, **kwargs):\n    \"\"\"Calculate position info from 2D spatial series.\"\"\"\n    return IntervalPositionInfo().calculate_position_info(*args, **kwargs)\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosVideo", "title": "<code>TrodesPosVideo</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> <p>Creates a video of the computed head position and orientation as well as the original LED positions overlaid on the video of the animal.</p> <p>Use for debugging the effect of position extraction parameters.</p> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@schema\nclass TrodesPosVideo(SpyglassMixin, dj.Computed):\n    \"\"\"Creates a video of the computed head position and orientation as well as\n    the original LED positions overlaid on the video of the animal.\n\n    Use for debugging the effect of position extraction parameters.\"\"\"\n\n    definition = \"\"\"\n    -&gt; TrodesPosV1\n    ---\n    has_video : bool\n    \"\"\"\n\n    def make(self, key):\n        M_TO_CM = 100\n\n        logger.info(\"Loading position data...\")\n        raw_position_df = (\n            RawPosition.PosObject\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"interval_list_name\"],\n            }\n        ).fetch1_dataframe()\n        position_info_df = (TrodesPosV1() &amp; key).fetch1_dataframe()\n\n        logger.info(\"Loading video data...\")\n        epoch = (\n            int(\n                key[\"interval_list_name\"]\n                .replace(\"pos \", \"\")\n                .replace(\" valid times\", \"\")\n            )\n            + 1\n        )\n\n        (\n            video_path,\n            video_filename,\n            meters_per_pixel,\n            video_time,\n        ) = get_video_path(\n            {\"nwb_file_name\": key[\"nwb_file_name\"], \"epoch\": epoch}\n        )\n\n        if not video_path:\n            self.insert1(dict(**key, has_video=False))\n            return\n\n        video_dir = os.path.dirname(video_path) + \"/\"\n        video_path = check_videofile(\n            video_path=video_dir, video_filename=video_filename\n        )[0].as_posix()\n        nwb_base_filename = key[\"nwb_file_name\"].replace(\".nwb\", \"\")\n        current_dir = Path(os.getcwd())\n        output_video_filename = (\n            f\"{current_dir.as_posix()}/{nwb_base_filename}_\"\n            f\"{epoch:02d}_{key['trodes_pos_params_name']}.mp4\"\n        )\n        centroids = {\n            \"red\": np.asarray(raw_position_df[[\"xloc\", \"yloc\"]]),\n            \"green\": np.asarray(raw_position_df[[\"xloc2\", \"yloc2\"]]),\n        }\n        position_mean = np.asarray(\n            position_info_df[[\"position_x\", \"position_y\"]]\n        )\n        orientation_mean = np.asarray(position_info_df[[\"orientation\"]])\n        position_time = np.asarray(position_info_df.index)\n        cm_per_pixel = meters_per_pixel * M_TO_CM\n\n        logger.info(\"Making video...\")\n        self.make_video(\n            video_path,\n            centroids,\n            position_mean,\n            orientation_mean,\n            video_time,\n            position_time,\n            output_video_filename=output_video_filename,\n            cm_to_pixels=cm_per_pixel,\n            disable_progressbar=False,\n        )\n        self.insert1(dict(**key, has_video=True))\n\n    @staticmethod\n    def convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n        \"\"\"Converts from cm to pixels and flips the y-axis.\n        Parameters\n        ----------\n        data : ndarray, shape (n_time, 2)\n        frame_size : array_like, shape (2,)\n        cm_to_pixels : float\n\n        Returns\n        -------\n        converted_data : ndarray, shape (n_time, 2)\n        \"\"\"\n        return data / cm_to_pixels\n\n    @staticmethod\n    def fill_nan(variable, video_time, variable_time):\n        video_ind = np.digitize(variable_time, video_time[1:])\n\n        n_video_time = len(video_time)\n        try:\n            n_variable_dims = variable.shape[1]\n            filled_variable = np.full((n_video_time, n_variable_dims), np.nan)\n        except IndexError:\n            filled_variable = np.full((n_video_time,), np.nan)\n        filled_variable[video_ind] = variable\n\n        return filled_variable\n\n    def make_video(\n        self,\n        video_filename,\n        centroids,\n        position_mean,\n        orientation_mean,\n        video_time,\n        position_time,\n        output_video_filename=\"output.mp4\",\n        cm_to_pixels=1.0,\n        disable_progressbar=False,\n        arrow_radius=15,\n        circle_radius=8,\n    ):\n        import cv2\n\n        RGB_PINK = (234, 82, 111)\n        RGB_YELLOW = (253, 231, 76)\n        RGB_WHITE = (255, 255, 255)\n\n        video = cv2.VideoCapture(video_filename)\n        fourcc = cv2.VideoWriter_fourcc(*\"mp4v\")\n        frame_size = (int(video.get(3)), int(video.get(4)))\n        frame_rate = video.get(5)\n        n_frames = int(orientation_mean.shape[0])\n        logger.info(f\"video filepath: {output_video_filename}\")\n        out = cv2.VideoWriter(\n            output_video_filename, fourcc, frame_rate, frame_size, True\n        )\n\n        centroids = {\n            color: self.fill_nan(data, video_time, position_time)\n            for color, data in centroids.items()\n        }\n        position_mean = self.fill_nan(position_mean, video_time, position_time)\n        orientation_mean = self.fill_nan(\n            orientation_mean, video_time, position_time\n        )\n\n        for time_ind in tqdm(\n            range(n_frames - 1), desc=\"frames\", disable=disable_progressbar\n        ):\n            is_grabbed, frame = video.read()\n            if is_grabbed:\n                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n\n                red_centroid = centroids[\"red\"][time_ind]\n                green_centroid = centroids[\"green\"][time_ind]\n\n                position = position_mean[time_ind]\n                position = self.convert_to_pixels(\n                    position, frame_size, cm_to_pixels\n                )\n                orientation = orientation_mean[time_ind]\n\n                if np.all(~np.isnan(red_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(red_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_YELLOW,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(green_centroid)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(green_centroid.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_PINK,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                if np.all(~np.isnan(position)) &amp; np.all(~np.isnan(orientation)):\n                    arrow_tip = (\n                        int(position[0] + arrow_radius * np.cos(orientation)),\n                        int(position[1] + arrow_radius * np.sin(orientation)),\n                    )\n                    cv2.arrowedLine(\n                        img=frame,\n                        pt1=tuple(position.astype(int)),\n                        pt2=arrow_tip,\n                        color=RGB_WHITE,\n                        thickness=4,\n                        line_type=8,\n                        shift=cv2.CV_8U,\n                        tipLength=0.25,\n                    )\n\n                if np.all(~np.isnan(position)):\n                    cv2.circle(\n                        img=frame,\n                        center=tuple(position.astype(int)),\n                        radius=circle_radius,\n                        color=RGB_WHITE,\n                        thickness=-1,\n                        shift=cv2.CV_8U,\n                    )\n\n                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)\n                out.write(frame)\n            else:\n                break\n\n        video.release()\n        out.release()\n        cv2.destroyAllWindows()\n</code></pre>"}, {"location": "api/position/v1/position_trodes_position/#src.spyglass.position.v1.position_trodes_position.TrodesPosVideo.convert_to_pixels", "title": "<code>convert_to_pixels(data, frame_size, cm_to_pixels=1.0)</code>  <code>staticmethod</code>", "text": "<p>Converts from cm to pixels and flips the y-axis.</p> <p>Parameters:</p> Name Type Description Default <code>data</code> <code>(ndarray, shape(n_time, 2))</code> required <code>frame_size</code> <code>(array_like, shape(2))</code> required <code>cm_to_pixels</code> <code>float</code> <code>1.0</code> <p>Returns:</p> Name Type Description <code>converted_data</code> <code>(ndarray, shape(n_time, 2))</code> Source code in <code>src/spyglass/position/v1/position_trodes_position.py</code> <pre><code>@staticmethod\ndef convert_to_pixels(data, frame_size, cm_to_pixels=1.0):\n    \"\"\"Converts from cm to pixels and flips the y-axis.\n    Parameters\n    ----------\n    data : ndarray, shape (n_time, 2)\n    frame_size : array_like, shape (2,)\n    cm_to_pixels : float\n\n    Returns\n    -------\n    converted_data : ndarray, shape (n_time, 2)\n    \"\"\"\n    return data / cm_to_pixels\n</code></pre>"}, {"location": "api/ripple/v1/ripple/", "title": "ripple.py", "text": ""}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleLFPSelection", "title": "<code>RippleLFPSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleLFPSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n     -&gt; LFPBandV1\n     group_name = 'CA1' : varchar(80)\n     \"\"\"\n\n    class RippleLFPElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; RippleLFPSelection\n        -&gt; LFPBandSelection.LFPBandElectrode\n        \"\"\"\n\n    @staticmethod\n    def validate_key(key):\n        filter_name = (LFPBandV1 &amp; key).fetch1(\"filter_name\")\n        if \"ripple\" not in filter_name.lower():\n            raise ValueError(\"Please use a ripple filter\")\n\n    @staticmethod\n    def set_lfp_electrodes(\n        key,\n        electrode_list=None,\n        group_name=\"CA1\",\n        **kwargs,\n    ):\n        \"\"\"Removes all electrodes for the specified nwb file and then\n        adds back the electrodes in the list\n\n        Parameters\n        ----------\n        key : dict\n            dictionary corresponding to the LFPBand entry to use for\n            ripple detection\n        electrode_list : list\n            list of electrodes from LFPBandSelection.LFPBandElectrode\n            to be used as the ripple LFP during detection\n        group_name : str, optional\n            description of the electrode group, by default \"CA1\"\n        \"\"\"\n        if electrode_list is None:\n            electrode_list = (\n                (LFPBandSelection.LFPBandElectrode &amp; key)\n                .fetch(\"electrode_id\")\n                .tolist()\n            )\n        electrode_list.sort()\n        try:\n            electrode_keys = (\n                pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n                .set_index(\"electrode_id\")\n                .loc[np.asarray(electrode_list)]\n                .reset_index()\n                .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n            )\n        except KeyError as err:\n            logger.debug(err)\n            raise KeyError(\n                \"Attempting to use electrode_ids that aren't in the associated\"\n                \" LFPBand filtered dataset.\"\n            ) from err\n        electrode_keys[\"group_name\"] = group_name\n        electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n        RippleLFPSelection.validate_key(key)\n        RippleLFPSelection().insert1(\n            {**key, \"group_name\": group_name},\n            skip_duplicates=True,\n            **kwargs,\n        )\n        RippleLFPSelection().RippleLFPElectrode.insert(\n            electrode_keys.to_dict(orient=\"records\"),\n            replace=True,\n            **kwargs,\n        )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleLFPSelection.set_lfp_electrodes", "title": "<code>set_lfp_electrodes(key, electrode_list=None, group_name='CA1', **kwargs)</code>  <code>staticmethod</code>", "text": "<p>Removes all electrodes for the specified nwb file and then adds back the electrodes in the list</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>dictionary corresponding to the LFPBand entry to use for ripple detection</p> required <code>electrode_list</code> <code>list</code> <p>list of electrodes from LFPBandSelection.LFPBandElectrode to be used as the ripple LFP during detection</p> <code>None</code> <code>group_name</code> <code>str</code> <p>description of the electrode group, by default \"CA1\"</p> <code>'CA1'</code> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@staticmethod\ndef set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name=\"CA1\",\n    **kwargs,\n):\n    \"\"\"Removes all electrodes for the specified nwb file and then\n    adds back the electrodes in the list\n\n    Parameters\n    ----------\n    key : dict\n        dictionary corresponding to the LFPBand entry to use for\n        ripple detection\n    electrode_list : list\n        list of electrodes from LFPBandSelection.LFPBandElectrode\n        to be used as the ripple LFP during detection\n    group_name : str, optional\n        description of the electrode group, by default \"CA1\"\n    \"\"\"\n    if electrode_list is None:\n        electrode_list = (\n            (LFPBandSelection.LFPBandElectrode &amp; key)\n            .fetch(\"electrode_id\")\n            .tolist()\n        )\n    electrode_list.sort()\n    try:\n        electrode_keys = (\n            pd.DataFrame(LFPBandSelection.LFPBandElectrode() &amp; key)\n            .set_index(\"electrode_id\")\n            .loc[np.asarray(electrode_list)]\n            .reset_index()\n            .loc[:, LFPBandSelection.LFPBandElectrode.primary_key]\n        )\n    except KeyError as err:\n        logger.debug(err)\n        raise KeyError(\n            \"Attempting to use electrode_ids that aren't in the associated\"\n            \" LFPBand filtered dataset.\"\n        ) from err\n    electrode_keys[\"group_name\"] = group_name\n    electrode_keys = electrode_keys.sort_values(by=[\"electrode_id\"])\n    RippleLFPSelection.validate_key(key)\n    RippleLFPSelection().insert1(\n        {**key, \"group_name\": group_name},\n        skip_duplicates=True,\n        **kwargs,\n    )\n    RippleLFPSelection().RippleLFPElectrode.insert(\n        electrode_keys.to_dict(orient=\"records\"),\n        replace=True,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleParameters", "title": "<code>RippleParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Lookup</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleParameters(SpyglassMixin, dj.Lookup):\n    definition = \"\"\"\n    ripple_param_name : varchar(80) # a name for this set of parameters\n    ----\n    ripple_param_dict : BLOB    # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default parameter set\"\"\"\n        default_dict = {\n            \"speed_name\": \"head_speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n            skip_duplicates=True,\n        )\n        default_dict_trodes = {\n            \"speed_name\": \"speed\",\n            \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n            \"ripple_detection_params\": dict(\n                speed_threshold=4.0,  # cm/s\n                minimum_duration=0.015,  # sec\n                zscore_threshold=2.0,  # std\n                smoothing_sigma=0.004,  # sec\n                close_ripple_threshold=0.0,  # sec\n            ),\n        }\n        self.insert1(\n            {\n                \"ripple_param_name\": \"default_trodes\",\n                \"ripple_param_dict\": default_dict_trodes,\n            },\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default parameter set</p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default parameter set\"\"\"\n    default_dict = {\n        \"speed_name\": \"head_speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\"ripple_param_name\": \"default\", \"ripple_param_dict\": default_dict},\n        skip_duplicates=True,\n    )\n    default_dict_trodes = {\n        \"speed_name\": \"speed\",\n        \"ripple_detection_algorithm\": \"Kay_ripple_detector\",\n        \"ripple_detection_params\": dict(\n            speed_threshold=4.0,  # cm/s\n            minimum_duration=0.015,  # sec\n            zscore_threshold=2.0,  # std\n            smoothing_sigma=0.004,  # sec\n            close_ripple_threshold=0.0,  # sec\n        ),\n    }\n    self.insert1(\n        {\n            \"ripple_param_name\": \"default_trodes\",\n            \"ripple_param_dict\": default_dict_trodes,\n        },\n        skip_duplicates=True,\n    )\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleTimesV1", "title": "<code>RippleTimesV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>@schema\nclass RippleTimesV1(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; RippleLFPSelection\n    -&gt; RippleParameters\n    -&gt; PositionOutput.proj(pos_merge_id='merge_id')\n    ---\n    -&gt; AnalysisNwbfile\n    ripple_times_object_id : varchar(40)\n     \"\"\"\n\n    def make(self, key):\n        nwb_file_name = (LFPBandV1 &amp; key).fetch1(\"nwb_file_name\")\n\n        logger.info(f\"Computing ripple times for: {key}\")\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        ripple_detection_algorithm = ripple_params[\"ripple_detection_algorithm\"]\n        ripple_detection_params = ripple_params[\"ripple_detection_params\"]\n\n        (\n            speed,\n            interval_ripple_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n        ripple_times = RIPPLE_DETECTION_ALGORITHMS[ripple_detection_algorithm](\n            time=np.asarray(interval_ripple_lfps.index),\n            filtered_lfps=np.asarray(interval_ripple_lfps),\n            speed=np.asarray(speed),\n            sampling_frequency=sampling_frequency,\n            **ripple_detection_params,\n        )\n        # Insert into analysis nwb file\n        nwb_analysis_file = AnalysisNwbfile()\n        key[\"analysis_file_name\"] = nwb_analysis_file.create(nwb_file_name)\n        key[\"ripple_times_object_id\"] = nwb_analysis_file.add_nwb_object(\n            analysis_file_name=key[\"analysis_file_name\"],\n            nwb_object=ripple_times,\n        )\n        nwb_analysis_file.add(\n            nwb_file_name=nwb_file_name,\n            analysis_file_name=key[\"analysis_file_name\"],\n        )\n\n        self.insert1(key)\n\n    def fetch1_dataframe(self):\n        \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n        return self.fetch_dataframe()[0]\n\n    def fetch_dataframe(self):\n        return [data[\"ripple_times\"] for data in self.fetch_nwb()]\n\n    @staticmethod\n    def get_ripple_lfps_and_position_info(key):\n        ripple_params = (\n            RippleParameters &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n        ).fetch1(\"ripple_param_dict\")\n\n        speed_name = ripple_params[\"speed_name\"]\n\n        electrode_keys = (RippleLFPSelection.RippleLFPElectrode() &amp; key).fetch(\n            \"electrode_id\"\n        )\n\n        # warn/validate that there is only one wire per electrode\n        ripple_lfp_nwb = (LFPBandV1 &amp; key).fetch_nwb()[0]\n        ripple_lfp_electrodes = ripple_lfp_nwb[\"lfp_band\"].electrodes.data[:]\n        elec_mask = np.zeros_like(ripple_lfp_electrodes, dtype=bool)\n        valid_elecs = [\n            elec for elec in electrode_keys if elec in ripple_lfp_electrodes\n        ]\n        lfp_indexed_elec_ids = get_electrode_indices(\n            ripple_lfp_nwb[\"lfp_band\"], valid_elecs\n        )\n        elec_mask[lfp_indexed_elec_ids] = True\n        ripple_lfp = pd.DataFrame(\n            ripple_lfp_nwb[\"lfp_band\"].data,\n            index=pd.Index(ripple_lfp_nwb[\"lfp_band\"].timestamps, name=\"time\"),\n        ).loc[:, elec_mask]\n        sampling_frequency = ripple_lfp_nwb[\"lfp_band_sampling_rate\"]\n\n        position_valid_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": key[\"target_interval_list_name\"],\n            }\n        ).fetch1(\"valid_times\")\n        position_info = (\n            PositionOutput() &amp; {\"merge_id\": key[\"pos_merge_id\"]}\n        ).fetch1_dataframe()\n\n        # restrict valid times to position time\n        valid_times_interval = np.array(\n            [position_info.index[0], position_info.index[-1]]\n        )\n        position_valid_times = interval_list_intersect(\n            position_valid_times, valid_times_interval\n        )\n\n        position_info = pd.concat(\n            [\n                position_info.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=0,\n        )\n        interval_ripple_lfps = pd.concat(\n            [\n                ripple_lfp.loc[slice(valid_time[0], valid_time[1])]\n                for valid_time in position_valid_times\n            ],\n            axis=0,\n        )\n\n        position_info = interpolate_to_new_time(\n            position_info, interval_ripple_lfps.index\n        )\n\n        return (\n            position_info[speed_name],\n            interval_ripple_lfps,\n            sampling_frequency,\n        )\n\n    @staticmethod\n    def get_Kay_ripple_consensus_trace(\n        ripple_filtered_lfps, sampling_frequency, smoothing_sigma=0.004\n    ):\n        ripple_consensus_trace = np.full_like(ripple_filtered_lfps, np.nan)\n        not_null = np.all(pd.notnull(ripple_filtered_lfps), axis=1)\n\n        ripple_consensus_trace[not_null] = get_envelope(\n            np.asarray(ripple_filtered_lfps)[not_null]\n        )\n        ripple_consensus_trace = np.sum(ripple_consensus_trace**2, axis=1)\n        ripple_consensus_trace[not_null] = gaussian_smooth(\n            ripple_consensus_trace[not_null],\n            smoothing_sigma,\n            sampling_frequency,\n        )\n        return pd.DataFrame(\n            np.sqrt(ripple_consensus_trace), index=ripple_filtered_lfps.index\n        )\n\n    @staticmethod\n    def plot_ripple_consensus_trace(\n        ripple_consensus_trace,\n        ripple_times,\n        ripple_label=1,\n        offset=0.100,\n        relative=True,\n        ax=None,\n    ):\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n\n        start_offset = ripple_start if relative else 0\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, 1))\n        ax.plot(\n            ripple_consensus_trace.loc[time_slice].index - start_offset,\n            ripple_consensus_trace.loc[time_slice],\n        )\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_xlabel(\"Time [s]\")\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n\n    @staticmethod\n    def plot_ripple(\n        lfps, ripple_times, ripple_label=1, offset=0.100, relative=True, ax=None\n    ):\n        lfp_labels = lfps.columns\n        n_lfps = len(lfp_labels)\n        ripple_start = ripple_times.loc[ripple_label].start_time\n        ripple_end = ripple_times.loc[ripple_label].end_time\n        time_slice = slice(ripple_start - offset, ripple_end + offset)\n        if ax is None:\n            fig, ax = plt.subplots(1, 1, figsize=(12, n_lfps * 0.20))\n\n        start_offset = ripple_start if relative else 0\n\n        for lfp_ind, lfp_label in enumerate(lfp_labels):\n            lfp = lfps.loc[time_slice, lfp_label]\n            ax.plot(\n                lfp.index - start_offset,\n                lfp_ind + (lfp - lfp.mean()) / (lfp.max() - lfp.min()),\n                color=\"black\",\n            )\n\n        ax.axvspan(\n            ripple_start - start_offset,\n            ripple_end - start_offset,\n            zorder=-1,\n            alpha=0.5,\n            color=\"lightgrey\",\n        )\n        ax.set_ylim((-1, n_lfps))\n        ax.set_xlim(\n            (time_slice.start - start_offset, time_slice.stop - start_offset)\n        )\n        ax.set_ylabel(\"LFPs\")\n        ax.set_xlabel(\"Time [s]\")\n\n    def create_figurl(\n        self,\n        zscore_ripple=False,\n        ripple_times_color=\"red\",\n        consensus_color=\"black\",\n        speed_color=\"black\",\n        view_height=800,\n        use_ripple_filtered_lfps=False,\n        lfp_offset=1,\n        lfp_channel_ind=None,\n    ):\n        ripple_times = self.fetch1_dataframe()\n\n        def _add_ripple_times(\n            view,\n            ripple_times=ripple_times,\n            ripple_times_color=ripple_times_color,\n        ):\n            return view.add_interval_series(\n                name=\"Ripple Events\",\n                t_start=ripple_times.start_time.to_numpy(),\n                t_end=ripple_times.end_time.to_numpy(),\n                color=ripple_times_color,\n            )\n\n        key = self.fetch1(\"KEY\")\n        (\n            speed,\n            ripple_filtered_lfps,\n            sampling_frequency,\n        ) = self.get_ripple_lfps_and_position_info(key)\n        ripple_consensus_trace = self.get_Kay_ripple_consensus_trace(\n            ripple_filtered_lfps, sampling_frequency\n        )\n\n        if zscore_ripple:\n            ripple_consensus_trace = zscore(ripple_consensus_trace)\n\n        consensus_view = _add_ripple_times(vv.TimeseriesGraph())\n        consensus_name = (\n            \"Z-Scored Consensus Trace\" if zscore_ripple else \"Consensus Trace\"\n        )\n        consensus_view.add_line_series(\n            name=consensus_name,\n            t=np.asarray(ripple_consensus_trace.index).squeeze(),\n            y=np.asarray(ripple_consensus_trace, dtype=np.float32).squeeze(),\n            color=consensus_color,\n            width=1,\n        )\n        if zscore_ripple:\n            ripple_params = (\n                RippleParameters\n                &amp; {\"ripple_param_name\": key[\"ripple_param_name\"]}\n            ).fetch1(\"ripple_param_dict\")\n\n            zscore_threshold = ripple_params[\"ripple_detection_params\"].get(\n                \"zscore_threshold\"\n            )\n            if zscore_threshold is not None:\n                consensus_view.add_line_series(\n                    name=\"Z-Score Threshold\",\n                    t=np.asarray(ripple_consensus_trace.index).squeeze(),\n                    y=np.ones_like(\n                        ripple_consensus_trace, dtype=np.float32\n                    ).squeeze()\n                    * zscore_threshold,\n                    color=ripple_times_color,\n                    width=1,\n                )\n\n        if use_ripple_filtered_lfps:\n            interval_ripple_lfps = ripple_filtered_lfps\n        else:\n            lfp_merge_id = (LFPBandSelection &amp; key).fetch1(\"lfp_merge_id\")\n            lfp_df = (LFPOutput &amp; {\"merge_id\": lfp_merge_id}).fetch1_dataframe()\n            interval_ripple_lfps = lfp_df.loc[speed.index[0] : speed.index[-1]]\n        if lfp_channel_ind is not None:\n            if lfp_channel_ind.max() &gt;= interval_ripple_lfps.shape[1]:\n                raise ValueError(\n                    \"lfp_channel_ind is out of range for the number of LFPs\"\n                )\n            interval_ripple_lfps = interval_ripple_lfps.iloc[:, lfp_channel_ind]\n\n        lfp_view = _add_ripple_times(vv.TimeseriesGraph())\n        max_lfp_value = interval_ripple_lfps.to_numpy().max()\n        lfp_offset *= max_lfp_value\n\n        for i, lfp in enumerate(interval_ripple_lfps.to_numpy().T):\n            lfp_view.add_line_series(\n                name=f\"LFP {i}\",\n                t=np.asarray(interval_ripple_lfps.index).squeeze(),\n                y=np.asarray(lfp + lfp_offset * i, dtype=np.int16).squeeze(),\n                color=\"black\",\n                width=1,\n            )\n        speed_view = _add_ripple_times(vv.TimeseriesGraph())\n        speed_view.add_line_series(\n            name=\"Speed [cm/s]\",\n            t=np.asarray(speed.index).squeeze(),\n            y=np.asarray(speed, dtype=np.float32).squeeze(),\n            color=speed_color,\n            width=1,\n        )\n        vertical_panel_content = [\n            vv.LayoutItem(consensus_view, stretch=2, title=\"Consensus\"),\n            vv.LayoutItem(lfp_view, stretch=8, title=\"LFPs\"),\n            vv.LayoutItem(speed_view, stretch=2, title=\"Speed\"),\n        ]\n\n        view = vv.Box(\n            direction=\"horizontal\",\n            show_titles=True,\n            height=view_height,\n            items=[\n                vv.LayoutItem(\n                    vv.Box(\n                        direction=\"vertical\",\n                        show_titles=True,\n                        items=vertical_panel_content,\n                    )\n                ),\n            ],\n        )\n\n        return view.url(label=\"Ripple Detection\")\n</code></pre>"}, {"location": "api/ripple/v1/ripple/#src.spyglass.ripple.v1.ripple.RippleTimesV1.fetch1_dataframe", "title": "<code>fetch1_dataframe()</code>", "text": "<p>Convenience function for returning the marks in a readable format</p> Source code in <code>src/spyglass/ripple/v1/ripple.py</code> <pre><code>def fetch1_dataframe(self):\n    \"\"\"Convenience function for returning the marks in a readable format\"\"\"\n    return self.fetch_dataframe()[0]\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/", "title": "sharing_kachery.py", "text": ""}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.kachery_download_file", "title": "<code>kachery_download_file(uri, dest, kachery_zone_name)</code>", "text": "<p>set the kachery resource url and attempt to down load the uri into the destination path</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>def kachery_download_file(uri: str, dest: str, kachery_zone_name: str) -&gt; str:\n    \"\"\"set the kachery resource url and attempt to down load the uri into the destination path\"\"\"\n    KacheryZone.set_resource_url({\"kachery_zone_name\": kachery_zone_name})\n    return kcl.load_file(uri, dest=dest)\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone", "title": "<code>KacheryZone</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@schema\nclass KacheryZone(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    kachery_zone_name: varchar(200) # the name of the kachery zone. Note that this is the same as the name of the kachery resource.\n    ---\n    description: varchar(200) # description of this zone\n    kachery_cloud_dir: varchar(200) # kachery cloud directory on local machine where files are linked\n    kachery_proxy: varchar(200) # kachery sharing proxy\n    -&gt; Lab\n    \"\"\"\n\n    @staticmethod\n    def set_zone(key: dict):\n        \"\"\"Set the kachery zone based on the key to KacheryZone\n\n        Parameters\n        ----------\n        key : dict\n            key defining a single KacheryZone\n\n        \"\"\"\n        try:\n            kachery_zone_name, kachery_cloud_dir = (KacheryZone &amp; key).fetch1(\n                \"kachery_zone_name\", \"kachery_cloud_dir\"\n            )\n        except DataJointError:\n            raise Exception(\n                f\"{key} does not correspond to a single entry in KacheryZone.\"\n            )\n            return None\n        # set the new zone and cloud directory\n        os.environ[kachery_zone_envar] = kachery_zone_name\n        os.environ[kachery_cloud_dir_envar] = kachery_cloud_dir\n\n    @staticmethod\n    def reset_zone():\n        \"\"\"Resets the kachery zone environment variable to the default values.\"\"\"\n        if default_kachery_zone is not None:\n            os.environ[kachery_zone_envar] = default_kachery_zone\n        if default_kachery_cloud_dir is not None:\n            os.environ[kachery_cloud_dir_envar] = default_kachery_cloud_dir\n\n    @staticmethod\n    def set_resource_url(key: dict):\n        \"\"\"Sets the KACHERY_RESOURCE_URL based on the key corresponding to a\n        single Kachery Zone\n\n        Parameters\n        ----------\n        key : dict\n            key to retrieve a single kachery zone\n        \"\"\"\n        try:\n            kachery_zone_name, kachery_proxy = (KacheryZone &amp; key).fetch1(\n                \"kachery_zone_name\", \"kachery_proxy\"\n            )\n        except DataJointError:\n            raise Exception(\n                f\"{key} does not correspond to a single entry in KacheryZone.\"\n            )\n        # set the new zone and cloud directory\n        os.environ[kachery_zone_envar] = kachery_zone_name\n        os.environ[kachery_resource_url_envar] = (\n            kachery_proxy + \"/r/\" + kachery_zone_name\n        )\n\n    @staticmethod\n    def reset_resource_url():\n        KacheryZone.reset_zone()\n        if default_kachery_resource_url is not None:\n            os.environ[kachery_resource_url_envar] = (\n                default_kachery_resource_url\n            )\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.set_zone", "title": "<code>set_zone(key)</code>  <code>staticmethod</code>", "text": "<p>Set the kachery zone based on the key to KacheryZone</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key defining a single KacheryZone</p> required Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef set_zone(key: dict):\n    \"\"\"Set the kachery zone based on the key to KacheryZone\n\n    Parameters\n    ----------\n    key : dict\n        key defining a single KacheryZone\n\n    \"\"\"\n    try:\n        kachery_zone_name, kachery_cloud_dir = (KacheryZone &amp; key).fetch1(\n            \"kachery_zone_name\", \"kachery_cloud_dir\"\n        )\n    except DataJointError:\n        raise Exception(\n            f\"{key} does not correspond to a single entry in KacheryZone.\"\n        )\n        return None\n    # set the new zone and cloud directory\n    os.environ[kachery_zone_envar] = kachery_zone_name\n    os.environ[kachery_cloud_dir_envar] = kachery_cloud_dir\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.reset_zone", "title": "<code>reset_zone()</code>  <code>staticmethod</code>", "text": "<p>Resets the kachery zone environment variable to the default values.</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef reset_zone():\n    \"\"\"Resets the kachery zone environment variable to the default values.\"\"\"\n    if default_kachery_zone is not None:\n        os.environ[kachery_zone_envar] = default_kachery_zone\n    if default_kachery_cloud_dir is not None:\n        os.environ[kachery_cloud_dir_envar] = default_kachery_cloud_dir\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.KacheryZone.set_resource_url", "title": "<code>set_resource_url(key)</code>  <code>staticmethod</code>", "text": "<p>Sets the KACHERY_RESOURCE_URL based on the key corresponding to a single Kachery Zone</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key to retrieve a single kachery zone</p> required Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef set_resource_url(key: dict):\n    \"\"\"Sets the KACHERY_RESOURCE_URL based on the key corresponding to a\n    single Kachery Zone\n\n    Parameters\n    ----------\n    key : dict\n        key to retrieve a single kachery zone\n    \"\"\"\n    try:\n        kachery_zone_name, kachery_proxy = (KacheryZone &amp; key).fetch1(\n            \"kachery_zone_name\", \"kachery_proxy\"\n        )\n    except DataJointError:\n        raise Exception(\n            f\"{key} does not correspond to a single entry in KacheryZone.\"\n        )\n    # set the new zone and cloud directory\n    os.environ[kachery_zone_envar] = kachery_zone_name\n    os.environ[kachery_resource_url_envar] = (\n        kachery_proxy + \"/r/\" + kachery_zone_name\n    )\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.AnalysisNwbfileKachery", "title": "<code>AnalysisNwbfileKachery</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@schema\nclass AnalysisNwbfileKachery(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; AnalysisNwbfileKacherySelection\n    ---\n    analysis_file_uri='': varchar(200)  # the uri of the file\n    \"\"\"\n\n    class LinkedFile(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; AnalysisNwbfileKachery\n        linked_file_rel_path: varchar(200) # the path for the linked file relative to the SPYGLASS_BASE_DIR environment variable\n        ---\n        linked_file_uri='': varchar(200) # the uri for the linked file\n        \"\"\"\n\n    def make(self, key):\n        # note that we're assuming that the user has initialized a kachery-cloud\n        # client with kachery-cloud-init. Uncomment the line below once we are\n        # sharing linked files as well.\n\n        # linked_key = copy.deepcopy(key)\n\n        logger.info(f'Linking {key[\"analysis_file_name\"]} in kachery-cloud...')\n        # set the kachery zone\n\n        KacheryZone.set_zone(key)\n\n        key[\"analysis_file_uri\"] = kcl.link_file(\n            AnalysisNwbfile().get_abs_path(key[\"analysis_file_name\"])\n        )\n        logger.info(\n            os.environ[kachery_zone_envar], os.environ[kachery_cloud_dir_envar]\n        )\n        logger.info(AnalysisNwbfile().get_abs_path(key[\"analysis_file_name\"]))\n        logger.info(kcl.load_file(key[\"analysis_file_uri\"]))\n        self.insert1(key)\n\n        # we also need to insert any linked files\n        # TODO: change this to automatically detect all linked files\n        # self.LinkedFile.insert1(key)\n\n        # reset the Kachery zone and cloud_dir to the defaults\n        KacheryZone.reset_zone()\n\n    @staticmethod\n    def download_file(analysis_file_name: str) -&gt; bool:\n        \"\"\"Download the specified analysis file and associated linked files\n        from kachery-cloud if possible\n\n        Parameters\n        ----------\n        analysis_file_name : str\n            The name of the analysis file\n\n        Returns\n        ----------\n        is_success : bool\n            True if the file was successfully downloaded, False otherwise\n        \"\"\"\n        fetched_list = (\n            AnalysisNwbfileKachery &amp; {\"analysis_file_name\": analysis_file_name}\n        ).fetch(\"analysis_file_uri\", \"kachery_zone_name\")\n        downloaded = False\n        for uri, kachery_zone_name in zip(fetched_list[0], fetched_list[1]):\n            if len(uri) == 0:\n                return False\n            logger.info(\"uri:\", uri)\n            if kachery_download_file(\n                uri=uri,\n                dest=AnalysisNwbfile.get_abs_path(analysis_file_name),\n                kachery_zone_name=kachery_zone_name,\n            ):\n                downloaded = True\n                # now download the linked file(s)\n                linked_files = (\n                    AnalysisNwbfileKachery.LinkedFile\n                    &amp; {\"analysis_file_name\": analysis_file_name}\n                ).fetch(as_dict=True)\n                for file in linked_files:\n                    uri = file[\"linked_file_uri\"]\n                    logger.info(f\"attempting to download linked file uri {uri}\")\n                    linked_file_path = (\n                        os.environ[\"SPYGLASS_BASE_DIR\"]\n                        + file[\"linked_file_rel_path\"]\n                    )\n                    if not kachery_download_file(\n                        uri=uri,\n                        dest=linked_file_path,\n                        kachery_zone_name=kachery_zone_name,\n                    ):\n                        raise Exception(\n                            f\"Linked file {linked_file_path} cannot be downloaded\"\n                        )\n        if not downloaded:\n            raise Exception(f\"{analysis_file_name} cannot be downloaded\")\n\n        return True\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.AnalysisNwbfileKachery.download_file", "title": "<code>download_file(analysis_file_name)</code>  <code>staticmethod</code>", "text": "<p>Download the specified analysis file and associated linked files from kachery-cloud if possible</p> <p>Parameters:</p> Name Type Description Default <code>analysis_file_name</code> <code>str</code> <p>The name of the analysis file</p> required <p>Returns:</p> Name Type Description <code>is_success</code> <code>bool</code> <p>True if the file was successfully downloaded, False otherwise</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>@staticmethod\ndef download_file(analysis_file_name: str) -&gt; bool:\n    \"\"\"Download the specified analysis file and associated linked files\n    from kachery-cloud if possible\n\n    Parameters\n    ----------\n    analysis_file_name : str\n        The name of the analysis file\n\n    Returns\n    ----------\n    is_success : bool\n        True if the file was successfully downloaded, False otherwise\n    \"\"\"\n    fetched_list = (\n        AnalysisNwbfileKachery &amp; {\"analysis_file_name\": analysis_file_name}\n    ).fetch(\"analysis_file_uri\", \"kachery_zone_name\")\n    downloaded = False\n    for uri, kachery_zone_name in zip(fetched_list[0], fetched_list[1]):\n        if len(uri) == 0:\n            return False\n        logger.info(\"uri:\", uri)\n        if kachery_download_file(\n            uri=uri,\n            dest=AnalysisNwbfile.get_abs_path(analysis_file_name),\n            kachery_zone_name=kachery_zone_name,\n        ):\n            downloaded = True\n            # now download the linked file(s)\n            linked_files = (\n                AnalysisNwbfileKachery.LinkedFile\n                &amp; {\"analysis_file_name\": analysis_file_name}\n            ).fetch(as_dict=True)\n            for file in linked_files:\n                uri = file[\"linked_file_uri\"]\n                logger.info(f\"attempting to download linked file uri {uri}\")\n                linked_file_path = (\n                    os.environ[\"SPYGLASS_BASE_DIR\"]\n                    + file[\"linked_file_rel_path\"]\n                )\n                if not kachery_download_file(\n                    uri=uri,\n                    dest=linked_file_path,\n                    kachery_zone_name=kachery_zone_name,\n                ):\n                    raise Exception(\n                        f\"Linked file {linked_file_path} cannot be downloaded\"\n                    )\n    if not downloaded:\n        raise Exception(f\"{analysis_file_name} cannot be downloaded\")\n\n    return True\n</code></pre>"}, {"location": "api/sharing/sharing_kachery/#src.spyglass.sharing.sharing_kachery.share_data_to_kachery", "title": "<code>share_data_to_kachery(restriction={}, table_list=[], zone_name=None)</code>", "text": "<p>Share data to kachery</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>dict</code> <p>restriction to select what data should be shared from table, by default {}</p> <code>{}</code> <code>table_list</code> <code>list</code> <p>List of tables to share data from, by default []</p> <code>[]</code> <code>zone_name</code> <code>str</code> <p>What kachery zone to share the data to, by default zone in spyglass.settings.config, which looks for <code>KACHERY_ZONE</code> environmental variable, but defaults to 'franklab.default'</p> <code>None</code> <p>Raises:</p> Type Description <code>ValueError</code> <p>Does not allow sharing of all data in table</p> Source code in <code>src/spyglass/sharing/sharing_kachery.py</code> <pre><code>def share_data_to_kachery(\n    restriction={},\n    table_list=[],\n    zone_name=None,\n):\n    \"\"\"Share data to kachery\n\n    Parameters\n    ----------\n    restriction : dict, optional\n        restriction to select what data should be shared from table, by default {}\n    table_list : list, optional\n        List of tables to share data from, by default []\n    zone_name : str, optional\n        What kachery zone to share the data to, by default zone in spyglass.settings.config,\n        which looks for `KACHERY_ZONE` environmental variable, but defaults to\n        'franklab.default'\n\n    Raises\n    ------\n    ValueError\n        Does not allow sharing of all data in table\n    \"\"\"\n    if not zone_name:\n        zone_name = config[\"KACHERY_ZONE\"]\n    kachery_selection_key = {\"kachery_zone_name\": zone_name}\n    if not restriction:\n        raise ValueError(\"Must provide a restriction to the table\")\n    selection_inserts = []\n    for table in table_list:\n        analysis_file_list = (table &amp; restriction).fetch(\"analysis_file_name\")\n        for file in analysis_file_list:  # Add all analysis to shared list\n            kachery_selection_key[\"analysis_file_name\"] = file\n            selection_inserts.append(kachery_selection_key)\n    AnalysisNwbfileKacherySelection.insert(\n        selection_inserts, skip_duplicates=True\n    )\n    AnalysisNwbfileKachery.populate()\n</code></pre>"}, {"location": "api/spikesorting/imported/", "title": "imported.py", "text": ""}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting", "title": "<code>ImportedSpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Imported</code></p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>@schema\nclass ImportedSpikeSorting(SpyglassMixin, dj.Imported):\n    definition = \"\"\"\n    -&gt; Session\n    ---\n    object_id: varchar(40)\n    \"\"\"\n\n    _nwb_table = Nwbfile\n\n    class Annotations(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; ImportedSpikeSorting\n        id: int # unit id, corresponds to dataframe index of unit in NWB file\n        ---\n        label = Null: longblob # list of string labels for the unit\n        annotations: longblob # dict of other annotations (e.g. metrics)\n        \"\"\"\n\n    def make(self, key):\n        orig_key = copy.deepcopy(key)\n\n        nwb_file_abs_path = Nwbfile.get_abs_path(key[\"nwb_file_name\"])\n\n        with pynwb.NWBHDF5IO(\n            nwb_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            if not nwbfile.units:\n                logger.warn(\"No units found in NWB file\")\n                return\n\n        from spyglass.spikesorting.spikesorting_merge import (  # noqa: F401\n            SpikeSortingOutput,\n        )\n\n        key[\"object_id\"] = nwbfile.units.object_id\n\n        self.insert1(key, skip_duplicates=True)\n\n        part_name = SpikeSortingOutput._part_name(self.table_name)\n        SpikeSortingOutput._merge_insert(\n            [orig_key], part_name=part_name, skip_duplicates=True\n        )\n\n    @classmethod\n    def get_recording(cls, key):\n        raise NotImplementedError(\n            \"Imported spike sorting does not have a `get_recording` method\"\n        )\n\n    @classmethod\n    def get_sorting(cls, key):\n        raise NotImplementedError(\n            \"Imported spike sorting does not have a `get_sorting` method\"\n        )\n\n    def add_annotation(\n        self, key, id, label=[], annotations={}, merge_annotations=False\n    ):\n        \"\"\"Manually add annotations to the spike sorting output\n\n        Parameters\n        ----------\n        key : dict\n            restriction key for ImportedSpikeSorting\n        id : int\n            unit id\n        label : List[str], optional\n            list of str labels for the unit, by default None\n        annotations : dict, optional\n            dictionary of other annotation values for unit, by default None\n        merge_annotations : bool, optional\n            whether to merge with existing annotations, by default False\n        \"\"\"\n        if isinstance(label, str):\n            label = [label]\n        query = self &amp; key\n        if not len(query) == 1:\n            raise ValueError(\n                f\"ImportedSpikeSorting key must be unique. Found: {query}\"\n            )\n        unit_key = {**key, \"id\": id}\n        annotation_query = ImportedSpikeSorting.Annotations &amp; unit_key\n        if annotation_query and not merge_annotations:\n            raise ValueError(\n                f\"Unit already has annotations: {annotation_query}\"\n            )\n        elif annotation_query:\n            existing_annotations = annotation_query.fetch1()\n            existing_annotations[\"label\"] += label\n            existing_annotations[\"annotations\"].update(annotations)\n            self.Annotations.update1(existing_annotations)\n        else:\n            self.Annotations.insert1(\n                dict(unit_key, label=label, annotations=annotations),\n                skip_duplicates=True,\n            )\n\n    def make_df_from_annotations(self):\n        \"\"\"Convert the annotations part table into a dataframe that can be\n        concatenated to the spikes dataframe in the nwb file.\"\"\"\n        df = []\n        for id, label, annotations in zip(\n            *self.Annotations.fetch(\"id\", \"label\", \"annotations\")\n        ):\n            df.append(\n                dict(\n                    id=id,\n                    label=label,\n                    **annotations,\n                )\n            )\n        df = pd.DataFrame(df)\n        df.set_index(\"id\", inplace=True)\n        return df\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        \"\"\"class method to fetch the nwb and add annotations to the spike dfs returned\"\"\"\n        # get the original nwbs\n        nwbs = super().fetch_nwb(*attrs, **kwargs)\n        # for each nwb, get the annotations and add them to the spikes dataframe\n        for i, key in enumerate(self.fetch(\"KEY\")):\n            if not ImportedSpikeSorting.Annotations &amp; key:\n                continue\n            # make the annotation_df\n            annotation_df = (self &amp; key).make_df_from_annotations()\n            # concatenate the annotations to the spikes dataframe in the returned nwb\n            nwbs[i][\"object_id\"] = pd.concat(\n                [nwbs[i][\"object_id\"], annotation_df], axis=\"columns\"\n            )\n        return nwbs\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.add_annotation", "title": "<code>add_annotation(key, id, label=[], annotations={}, merge_annotations=False)</code>", "text": "<p>Manually add annotations to the spike sorting output</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>restriction key for ImportedSpikeSorting</p> required <code>id</code> <code>int</code> <p>unit id</p> required <code>label</code> <code>List[str]</code> <p>list of str labels for the unit, by default None</p> <code>[]</code> <code>annotations</code> <code>dict</code> <p>dictionary of other annotation values for unit, by default None</p> <code>{}</code> <code>merge_annotations</code> <code>bool</code> <p>whether to merge with existing annotations, by default False</p> <code>False</code> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def add_annotation(\n    self, key, id, label=[], annotations={}, merge_annotations=False\n):\n    \"\"\"Manually add annotations to the spike sorting output\n\n    Parameters\n    ----------\n    key : dict\n        restriction key for ImportedSpikeSorting\n    id : int\n        unit id\n    label : List[str], optional\n        list of str labels for the unit, by default None\n    annotations : dict, optional\n        dictionary of other annotation values for unit, by default None\n    merge_annotations : bool, optional\n        whether to merge with existing annotations, by default False\n    \"\"\"\n    if isinstance(label, str):\n        label = [label]\n    query = self &amp; key\n    if not len(query) == 1:\n        raise ValueError(\n            f\"ImportedSpikeSorting key must be unique. Found: {query}\"\n        )\n    unit_key = {**key, \"id\": id}\n    annotation_query = ImportedSpikeSorting.Annotations &amp; unit_key\n    if annotation_query and not merge_annotations:\n        raise ValueError(\n            f\"Unit already has annotations: {annotation_query}\"\n        )\n    elif annotation_query:\n        existing_annotations = annotation_query.fetch1()\n        existing_annotations[\"label\"] += label\n        existing_annotations[\"annotations\"].update(annotations)\n        self.Annotations.update1(existing_annotations)\n    else:\n        self.Annotations.insert1(\n            dict(unit_key, label=label, annotations=annotations),\n            skip_duplicates=True,\n        )\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.make_df_from_annotations", "title": "<code>make_df_from_annotations()</code>", "text": "<p>Convert the annotations part table into a dataframe that can be concatenated to the spikes dataframe in the nwb file.</p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def make_df_from_annotations(self):\n    \"\"\"Convert the annotations part table into a dataframe that can be\n    concatenated to the spikes dataframe in the nwb file.\"\"\"\n    df = []\n    for id, label, annotations in zip(\n        *self.Annotations.fetch(\"id\", \"label\", \"annotations\")\n    ):\n        df.append(\n            dict(\n                id=id,\n                label=label,\n                **annotations,\n            )\n        )\n    df = pd.DataFrame(df)\n    df.set_index(\"id\", inplace=True)\n    return df\n</code></pre>"}, {"location": "api/spikesorting/imported/#src.spyglass.spikesorting.imported.ImportedSpikeSorting.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>class method to fetch the nwb and add annotations to the spike dfs returned</p> Source code in <code>src/spyglass/spikesorting/imported.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs):\n    \"\"\"class method to fetch the nwb and add annotations to the spike dfs returned\"\"\"\n    # get the original nwbs\n    nwbs = super().fetch_nwb(*attrs, **kwargs)\n    # for each nwb, get the annotations and add them to the spikes dataframe\n    for i, key in enumerate(self.fetch(\"KEY\")):\n        if not ImportedSpikeSorting.Annotations &amp; key:\n            continue\n        # make the annotation_df\n        annotation_df = (self &amp; key).make_df_from_annotations()\n        # concatenate the annotations to the spikes dataframe in the returned nwb\n        nwbs[i][\"object_id\"] = pd.concat(\n            [nwbs[i][\"object_id\"], annotation_df], axis=\"columns\"\n        )\n    return nwbs\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/", "title": "spikesorting_merge.py", "text": ""}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput", "title": "<code>SpikeSortingOutput</code>", "text": "<p>             Bases: <code>_Merge</code>, <code>SpyglassMixin</code></p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@schema\nclass SpikeSortingOutput(_Merge, SpyglassMixin):\n    definition = \"\"\"\n    # Output of spike sorting pipelines.\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class CurationV1(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CurationV1\n        \"\"\"\n\n    class ImportedSpikeSorting(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; ImportedSpikeSorting\n        \"\"\"\n\n    class CuratedSpikeSorting(SpyglassMixin, dj.Part):  # noqa: F811\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; CuratedSpikeSorting\n        \"\"\"\n\n    @classmethod\n    def get_recording(cls, key):\n        \"\"\"get the recording associated with a spike sorting output\"\"\"\n        source_table = source_class_dict[\n            to_camel_case(cls.merge_get_parent(key).table_name)\n        ]\n        query = source_table &amp; cls.merge_get_part(key)\n        return query.get_recording(query.fetch(\"KEY\"))\n\n    @classmethod\n    def get_sorting(cls, key):\n        \"\"\"get the sorting associated with a spike sorting output\"\"\"\n        source_table = source_class_dict[\n            to_camel_case(cls.merge_get_parent(key).table_name)\n        ]\n        query = source_table &amp; cls.merge_get_part(key)\n        return query.get_sorting(query.fetch(\"KEY\"))\n\n    @classmethod\n    def get_spike_times(cls, key):\n        spike_times = []\n        for nwb_file in cls.fetch_nwb(key):\n            # V1 uses 'object_id', V0 uses 'units'\n            file_loc = \"object_id\" if \"object_id\" in nwb_file else \"units\"\n            spike_times.extend(nwb_file[file_loc][\"spike_times\"].to_list())\n        return spike_times\n\n    @classmethod\n    def get_spike_indicator(cls, key, time):\n        time = np.asarray(time)\n        min_time, max_time = time[[0, -1]]\n        spike_times = cls.fetch_spike_data(key)\n        spike_indicator = np.zeros((len(time), len(spike_times)))\n\n        for ind, times in enumerate(spike_times):\n            times = times[np.logical_and(times &gt;= min_time, times &lt;= max_time)]\n            spike_indicator[:, ind] = np.bincount(\n                np.digitize(times, time[1:-1]),\n                minlength=time.shape[0],\n            )\n\n        return spike_indicator\n\n    @classmethod\n    def get_firing_rate(cls, key, time, multiunit=False):\n        spike_indicator = cls.get_spike_indicator(key, time)\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        if multiunit:\n            spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n        return np.stack(\n            [\n                get_multiunit_population_firing_rate(\n                    indicator[:, np.newaxis], sampling_frequency\n                )\n                for indicator in spike_indicator.T\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>get the recording associated with a spike sorting output</p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@classmethod\ndef get_recording(cls, key):\n    \"\"\"get the recording associated with a spike sorting output\"\"\"\n    source_table = source_class_dict[\n        to_camel_case(cls.merge_get_parent(key).table_name)\n    ]\n    query = source_table &amp; cls.merge_get_part(key)\n    return query.get_recording(query.fetch(\"KEY\"))\n</code></pre>"}, {"location": "api/spikesorting/spikesorting_merge/#src.spyglass.spikesorting.spikesorting_merge.SpikeSortingOutput.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>get the sorting associated with a spike sorting output</p> Source code in <code>src/spyglass/spikesorting/spikesorting_merge.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key):\n    \"\"\"get the sorting associated with a spike sorting output\"\"\"\n    source_table = source_class_dict[\n        to_camel_case(cls.merge_get_parent(key).table_name)\n    ]\n    query = source_table &amp; cls.merge_get_part(key)\n    return query.get_sorting(query.fetch(\"KEY\"))\n</code></pre>"}, {"location": "api/spikesorting/analysis/v1/group/", "title": "group.py", "text": ""}, {"location": "api/spikesorting/analysis/v1/group/#src.spyglass.spikesorting.analysis.v1.group.SortedSpikesGroup", "title": "<code>SortedSpikesGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/analysis/v1/group.py</code> <pre><code>@schema\nclass SortedSpikesGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    -&gt; Session\n    -&gt; UnitSelectionParams\n    sorted_spikes_group_name: varchar(80)\n    \"\"\"\n\n    class Units(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        -&gt; SpikeSortingOutput.proj(spikesorting_merge_id='merge_id')\n        \"\"\"\n\n    def create_group(\n        self,\n        group_name: str,\n        nwb_file_name: str,\n        unit_filter_params_name: str = \"all_units\",\n        keys: list[dict] = [],\n    ):\n        group_key = {\n            \"sorted_spikes_group_name\": group_name,\n            \"nwb_file_name\": nwb_file_name,\n            \"unit_filter_params_name\": unit_filter_params_name,\n        }\n        parts_insert = [{**key, **group_key} for key in keys]\n\n        self.insert1(\n            group_key,\n            skip_duplicates=True,\n        )\n        self.Units.insert(parts_insert, skip_duplicates=True)\n\n    @staticmethod\n    def filter_units(\n        labels: list[list[str]],\n        include_labels: list[str],\n        exclude_labels: list[str],\n    ) -&gt; np.ndarray:\n        \"\"\"\n        Filter units based on labels\n        \"\"\"\n        include_labels = np.unique(include_labels)\n        exclude_labels = np.unique(exclude_labels)\n\n        if include_labels.size == 0 and exclude_labels.size == 0:\n            # if no labels are provided, include all units\n            return np.ones(len(labels), dtype=bool)\n\n        include_mask = np.zeros(len(labels), dtype=bool)\n        for ind, unit_labels in enumerate(labels):\n            if isinstance(unit_labels, str):\n                unit_labels = [unit_labels]\n            if (\n                include_labels.size &gt; 0\n                and np.all(~np.isin(unit_labels, include_labels))\n            ) or np.any(np.isin(unit_labels, exclude_labels)):\n                # if the unit does not have any of the include labels\n                # or has any of the exclude labels, skip\n                continue\n            include_mask[ind] = True\n        return include_mask\n\n    @staticmethod\n    def fetch_spike_data(key, time_slice=None):\n        # get merge_ids for SpikeSortingOutput\n        merge_ids = (\n            (\n                SortedSpikesGroup.Units\n                &amp; {\n                    \"nwb_file_name\": key[\"nwb_file_name\"],\n                    \"sorted_spikes_group_name\": key[\"sorted_spikes_group_name\"],\n                }\n            )\n        ).fetch(\"spikesorting_merge_id\")\n\n        # get the filtering parameters\n        include_labels, exclude_labels = (UnitSelectionParams &amp; key).fetch1(\n            \"include_labels\", \"exclude_labels\"\n        )\n\n        # get the spike times for each merge_id\n        spike_times = []\n        for merge_id in merge_ids:\n            nwb_file = SpikeSortingOutput().fetch_nwb({\"merge_id\": merge_id})[0]\n            nwb_field_name = (\n                \"object_id\"\n                if \"object_id\" in nwb_file\n                else \"units\" if \"units\" in nwb_file else None\n            )\n            if nwb_field_name is None:\n                # case where no units found or curation removed all units\n                continue\n            sorting_spike_times = nwb_file[nwb_field_name][\n                \"spike_times\"\n            ].to_list()\n\n            # filter the spike times based on the labels if present\n            if \"label\" in nwb_file[nwb_field_name]:\n                group_label_list = nwb_file[nwb_field_name][\"label\"].to_list()\n                include_unit = SortedSpikesGroup.filter_units(\n                    group_label_list, include_labels, exclude_labels\n                )\n\n                sorting_spike_times = list(\n                    compress(sorting_spike_times, include_unit)\n                )\n\n            # filter the spike times based on the time slice if provided\n            if time_slice is not None:\n                sorting_spike_times = [\n                    times[\n                        np.logical_and(\n                            times &gt;= time_slice.start, times &lt;= time_slice.stop\n                        )\n                    ]\n                    for times in sorting_spike_times\n                ]\n\n            # append the approved spike times to the list\n            spike_times.extend(sorting_spike_times)\n\n        return spike_times\n\n    @classmethod\n    def get_spike_indicator(cls, key: dict, time: np.ndarray) -&gt; np.ndarray:\n        time = np.asarray(time)\n        min_time, max_time = time[[0, -1]]\n        spike_times = cls.fetch_spike_data(key)\n        spike_indicator = np.zeros((len(time), len(spike_times)))\n\n        for ind, times in enumerate(spike_times):\n            times = times[np.logical_and(times &gt;= min_time, times &lt;= max_time)]\n            spike_indicator[:, ind] = np.bincount(\n                np.digitize(times, time[1:-1]),\n                minlength=time.shape[0],\n            )\n\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        return spike_indicator\n\n    @classmethod\n    def get_firing_rate(\n        cls, key: dict, time: np.ndarray, multiunit: bool = False\n    ) -&gt; np.ndarray:\n        spike_indicator = cls.get_spike_indicator(key, time)\n        if spike_indicator.ndim == 1:\n            spike_indicator = spike_indicator[:, np.newaxis]\n\n        sampling_frequency = 1 / np.median(np.diff(time))\n\n        if multiunit:\n            spike_indicator = spike_indicator.sum(axis=1, keepdims=True)\n        return np.stack(\n            [\n                get_multiunit_population_firing_rate(\n                    indicator[:, np.newaxis], sampling_frequency\n                )\n                for indicator in spike_indicator.T\n            ],\n            axis=1,\n        )\n</code></pre>"}, {"location": "api/spikesorting/analysis/v1/group/#src.spyglass.spikesorting.analysis.v1.group.SortedSpikesGroup.filter_units", "title": "<code>filter_units(labels, include_labels, exclude_labels)</code>  <code>staticmethod</code>", "text": "<p>Filter units based on labels</p> Source code in <code>src/spyglass/spikesorting/analysis/v1/group.py</code> <pre><code>@staticmethod\ndef filter_units(\n    labels: list[list[str]],\n    include_labels: list[str],\n    exclude_labels: list[str],\n) -&gt; np.ndarray:\n    \"\"\"\n    Filter units based on labels\n    \"\"\"\n    include_labels = np.unique(include_labels)\n    exclude_labels = np.unique(exclude_labels)\n\n    if include_labels.size == 0 and exclude_labels.size == 0:\n        # if no labels are provided, include all units\n        return np.ones(len(labels), dtype=bool)\n\n    include_mask = np.zeros(len(labels), dtype=bool)\n    for ind, unit_labels in enumerate(labels):\n        if isinstance(unit_labels, str):\n            unit_labels = [unit_labels]\n        if (\n            include_labels.size &gt; 0\n            and np.all(~np.isin(unit_labels, include_labels))\n        ) or np.any(np.isin(unit_labels, exclude_labels)):\n            # if the unit does not have any of the include labels\n            # or has any of the exclude labels, skip\n            continue\n        include_mask[ind] = True\n    return include_mask\n</code></pre>"}, {"location": "api/spikesorting/v0/curation_figurl/", "title": "curation_figurl.py", "text": ""}, {"location": "api/spikesorting/v0/curation_figurl/#src.spyglass.spikesorting.v0.curation_figurl.CurationFigurl", "title": "<code>CurationFigurl</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/curation_figurl.py</code> <pre><code>@schema\nclass CurationFigurl(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; CurationFigurlSelection\n    ---\n    url: varchar(2000)\n    initial_curation_uri: varchar(2000)\n    new_curation_uri: varchar(2000)\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Create a Curation Figurl\n        Parameters\n        ----------\n        key : dict\n            primary key of an entry from CurationFigurlSelection table\n        \"\"\"\n\n        # get new_curation_uri from selection table\n        new_curation_uri = (CurationFigurlSelection &amp; key).fetch1(\n            \"new_curation_uri\"\n        )\n\n        # fetch\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        recording_label = SpikeSortingRecording._get_recording_name(key)\n        sorting_label = SpikeSorting._get_sorting_name(key)\n        unit_metrics = _reformat_metrics(\n            (Curation &amp; key).fetch1(\"quality_metrics\")\n        )\n        initial_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        initial_merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n\n        # new_curation_uri = key[\"new_curation_uri\"]\n\n        # Create the initial curation and store it in kachery\n        for k, v in initial_labels.items():\n            new_list = []\n            for item in v:\n                if item not in new_list:\n                    new_list.append(item)\n            initial_labels[k] = new_list\n        initial_curation = {\n            \"labelsByUnit\": initial_labels,\n            \"mergeGroups\": initial_merge_groups,\n        }\n        initial_curation_uri = kcl.store_json(initial_curation)\n\n        # Get the recording/sorting extractors\n        R = si.load_extractor(recording_path)\n        if R.get_num_segments() &gt; 1:\n            R = si.concatenate_recordings([R])\n        S = si.load_extractor(sorting_path)\n\n        # Generate the figURL\n        url = _generate_the_figurl(\n            R=R,\n            S=S,\n            initial_curation_uri=initial_curation_uri,\n            new_curation_uri=new_curation_uri,\n            recording_label=recording_label,\n            sorting_label=sorting_label,\n            unit_metrics=unit_metrics,\n        )\n\n        # insert\n        key[\"url\"] = url\n        key[\"initial_curation_uri\"] = initial_curation_uri\n        key[\"new_curation_uri\"] = new_curation_uri\n        self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/curation_figurl/#src.spyglass.spikesorting.v0.curation_figurl.CurationFigurl.make", "title": "<code>make(key)</code>", "text": "<p>Create a Curation Figurl</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of an entry from CurationFigurlSelection table</p> required Source code in <code>src/spyglass/spikesorting/v0/curation_figurl.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Create a Curation Figurl\n    Parameters\n    ----------\n    key : dict\n        primary key of an entry from CurationFigurlSelection table\n    \"\"\"\n\n    # get new_curation_uri from selection table\n    new_curation_uri = (CurationFigurlSelection &amp; key).fetch1(\n        \"new_curation_uri\"\n    )\n\n    # fetch\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    recording_label = SpikeSortingRecording._get_recording_name(key)\n    sorting_label = SpikeSorting._get_sorting_name(key)\n    unit_metrics = _reformat_metrics(\n        (Curation &amp; key).fetch1(\"quality_metrics\")\n    )\n    initial_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n    initial_merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n\n    # new_curation_uri = key[\"new_curation_uri\"]\n\n    # Create the initial curation and store it in kachery\n    for k, v in initial_labels.items():\n        new_list = []\n        for item in v:\n            if item not in new_list:\n                new_list.append(item)\n        initial_labels[k] = new_list\n    initial_curation = {\n        \"labelsByUnit\": initial_labels,\n        \"mergeGroups\": initial_merge_groups,\n    }\n    initial_curation_uri = kcl.store_json(initial_curation)\n\n    # Get the recording/sorting extractors\n    R = si.load_extractor(recording_path)\n    if R.get_num_segments() &gt; 1:\n        R = si.concatenate_recordings([R])\n    S = si.load_extractor(sorting_path)\n\n    # Generate the figURL\n    url = _generate_the_figurl(\n        R=R,\n        S=S,\n        initial_curation_uri=initial_curation_uri,\n        new_curation_uri=new_curation_uri,\n        recording_label=recording_label,\n        sorting_label=sorting_label,\n        unit_metrics=unit_metrics,\n    )\n\n    # insert\n    key[\"url\"] = url\n    key[\"initial_curation_uri\"] = initial_curation_uri\n    key[\"new_curation_uri\"] = new_curation_uri\n    self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/merged_sorting_extractor/", "title": "merged_sorting_extractor.py", "text": ""}, {"location": "api/spikesorting/v0/sortingview/", "title": "sortingview.py", "text": ""}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace", "title": "<code>SortingviewWorkspace</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>@schema\nclass SortingviewWorkspace(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SortingviewWorkspaceSelection\n    ---\n    workspace_uri: varchar(1000)\n    sortingview_recording_id: varchar(30)\n    sortingview_sorting_id: varchar(30)\n    channel = NULL : varchar(80)        # the name of kachery channel for data sharing (for kachery daemon, deprecated)\n    \"\"\"\n\n    # make class for parts table to hold URLs\n    class URL(SpyglassMixin, dj.Part):\n        # Table for holding URLs\n        definition = \"\"\"\n        -&gt; SortingviewWorkspace\n        ---\n        curation_url: varchar(1000)   # URL with sortingview data\n        curation_jot: varchar(200)   # URI for saving manual curation tags\n        \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Create a Sortingview workspace\n\n        Parameters\n        ----------\n        key : dict\n            primary key of an entry from SortingviewWorkspaceSelection table\n        \"\"\"\n\n        # fetch\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n        workspace_label = SpikeSortingRecording._get_recording_name(key)\n        recording_label = SpikeSortingRecording._get_recording_name(key)\n        sorting_label = SpikeSorting._get_sorting_name(key)\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n        curation_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        team_name = (SpikeSortingRecording &amp; key).fetch1()[\"team_name\"]\n        team_members = (LabTeam.LabTeamMember &amp; {\"team_name\": team_name}).fetch(\n            \"lab_member_name\"\n        )\n        google_user_ids = []\n        for team_member in team_members:\n            google_user_id = (\n                LabMember.LabMemberInfo &amp; {\"lab_member_name\": team_member}\n            ).fetch(\"google_user_name\")\n            if len(google_user_id) != 1:\n                logger.warn(\n                    f\"Google user ID for {team_member} does not exist or more than one ID detected;\\\n                        permission not given to {team_member}, skipping...\"\n                )\n                continue\n            google_user_ids.append(google_user_id[0])\n\n        # do\n        (\n            workspace_uri,\n            recording_id,\n            sorting_id,\n        ) = _create_spikesortingview_workspace(\n            recording_path=recording_path,\n            sorting_path=sorting_path,\n            merge_groups=merge_groups,\n            workspace_label=workspace_label,\n            recording_label=recording_label,\n            sorting_label=sorting_label,\n            metrics=metrics,\n            curation_labels=curation_labels,\n            google_user_ids=google_user_ids,\n        )\n\n        # insert\n        key[\"workspace_uri\"] = workspace_uri\n        key[\"sortingview_recording_id\"] = recording_id\n        key[\"sortingview_sorting_id\"] = sorting_id\n        self.insert1(key)\n\n        # insert URLs\n        # remove non-primary keys\n        del key[\"workspace_uri\"]\n        del key[\"sortingview_recording_id\"]\n        del key[\"sortingview_sorting_id\"]\n\n        # generate URLs and add to key\n        url = self.url_trythis(key)\n        key[\"curation_url\"] = url\n        key[\"curation_jot\"] = \"not ready yet\"\n\n        SortingviewWorkspace.URL.insert1(key)\n\n    def remove_sorting_from_workspace(self, key):\n        return NotImplementedError\n\n    def url_trythis(self, key: dict, sortingview_sorting_id: str = None):\n        \"\"\"Generate a URL for visualizing and curating a sorting on the web.\n        Will print instructions on how to do the curation.\n\n        Parameters\n        ----------\n        key : dict\n            An entry from SortingviewWorkspace table\n        sortingview_sorting_id : str, optional\n            sortingview sorting ID to visualize. If None then chooses the first one\n\n        Returns\n        -------\n        url : str\n        \"\"\"\n        workspace_uri = (self &amp; key).fetch1(\"workspace_uri\")\n        workspace = sv.load_workspace(workspace_uri)\n        recording_id = workspace.recording_ids[0]\n        if sortingview_sorting_id is None:\n            sortingview_sorting_id = workspace.sorting_ids[0]\n\n        R = workspace.get_recording_extractor(recording_id)\n        S = workspace.get_sorting_extractor(sortingview_sorting_id)\n\n        initial_labels = (Curation &amp; key).fetch(\"curation_labels\")[0]\n        for k, v in initial_labels.items():\n            new_list = []\n            for item in v:\n                if item not in new_list:\n                    new_list.append(item)\n            initial_labels[k] = new_list\n        initial_curation = {\"labelsByUnit\": initial_labels}\n\n        # custom metrics\n        unit_metrics = workspace.get_unit_metrics_for_sorting(\n            sortingview_sorting_id\n        )\n\n        # This will print some instructions on how to do the curation\n        # old: sv.trythis_start_sorting_curation\n        url = _generate_url(\n            recording=R,\n            sorting=S,\n            label=workspace.label,\n            initial_curation=initial_curation,\n            raster_plot_subsample_max_firing_rate=50,\n            spike_amplitudes_subsample_max_firing_rate=50,\n            unit_metrics=unit_metrics,\n        )\n        return url\n\n    def insert_manual_curation(\n        self, key: dict, url: str, description=\"manually curated\"\n    ):\n        \"\"\"Based on information in key for an SortingviewWorkspace, loads the\n        curated sorting from sortingview, saves it (with labels and the\n        optional description) and inserts it to CuratedSorting\n\n        Assumes that the workspace corresponding to the recording and (original) sorting exists\n\n        Parameters\n        ----------\n        key : dict\n            primary key of AutomaticCuration\n        description: str, optional\n            description of curated sorting\n        \"\"\"\n\n        # get the labels and remove the non-primary merged units\n        # labels = workspace.get_sorting_curation(sorting_id=sortingview_sorting_id)\n        # labels = sv.trythis_load_sorting_curation('jot://xTzzyDieQPkW')\n        labels = sv.trythis_load_sorting_curation(url)\n\n        # turn labels to list of str, only including accepted units.\n        # if bool(labels[\"mergeGroups\"]):\n        if bool(labels.get(\"mergeGroups\", [])):\n            # clusters were merged, so we empty out metrics\n            metrics = {}\n        else:\n            # get the metrics from the parent curation\n            metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n\n        # insert this curation into the  Table\n        return Curation.insert_curation(\n            key,\n            parent_curation_id=key[\"curation_id\"],\n            labels=labels[\"labelsByUnit\"],\n            merge_groups=labels.get(\"mergeGroups\", []),\n            metrics=metrics,\n            description=description,\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.make", "title": "<code>make(key)</code>", "text": "<p>Create a Sortingview workspace</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of an entry from SortingviewWorkspaceSelection table</p> required Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Create a Sortingview workspace\n\n    Parameters\n    ----------\n    key : dict\n        primary key of an entry from SortingviewWorkspaceSelection table\n    \"\"\"\n\n    # fetch\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n    workspace_label = SpikeSortingRecording._get_recording_name(key)\n    recording_label = SpikeSortingRecording._get_recording_name(key)\n    sorting_label = SpikeSorting._get_sorting_name(key)\n    metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n    curation_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n    team_name = (SpikeSortingRecording &amp; key).fetch1()[\"team_name\"]\n    team_members = (LabTeam.LabTeamMember &amp; {\"team_name\": team_name}).fetch(\n        \"lab_member_name\"\n    )\n    google_user_ids = []\n    for team_member in team_members:\n        google_user_id = (\n            LabMember.LabMemberInfo &amp; {\"lab_member_name\": team_member}\n        ).fetch(\"google_user_name\")\n        if len(google_user_id) != 1:\n            logger.warn(\n                f\"Google user ID for {team_member} does not exist or more than one ID detected;\\\n                    permission not given to {team_member}, skipping...\"\n            )\n            continue\n        google_user_ids.append(google_user_id[0])\n\n    # do\n    (\n        workspace_uri,\n        recording_id,\n        sorting_id,\n    ) = _create_spikesortingview_workspace(\n        recording_path=recording_path,\n        sorting_path=sorting_path,\n        merge_groups=merge_groups,\n        workspace_label=workspace_label,\n        recording_label=recording_label,\n        sorting_label=sorting_label,\n        metrics=metrics,\n        curation_labels=curation_labels,\n        google_user_ids=google_user_ids,\n    )\n\n    # insert\n    key[\"workspace_uri\"] = workspace_uri\n    key[\"sortingview_recording_id\"] = recording_id\n    key[\"sortingview_sorting_id\"] = sorting_id\n    self.insert1(key)\n\n    # insert URLs\n    # remove non-primary keys\n    del key[\"workspace_uri\"]\n    del key[\"sortingview_recording_id\"]\n    del key[\"sortingview_sorting_id\"]\n\n    # generate URLs and add to key\n    url = self.url_trythis(key)\n    key[\"curation_url\"] = url\n    key[\"curation_jot\"] = \"not ready yet\"\n\n    SortingviewWorkspace.URL.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.url_trythis", "title": "<code>url_trythis(key, sortingview_sorting_id=None)</code>", "text": "<p>Generate a URL for visualizing and curating a sorting on the web. Will print instructions on how to do the curation.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>An entry from SortingviewWorkspace table</p> required <code>sortingview_sorting_id</code> <code>str</code> <p>sortingview sorting ID to visualize. If None then chooses the first one</p> <code>None</code> <p>Returns:</p> Name Type Description <code>url</code> <code>str</code> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def url_trythis(self, key: dict, sortingview_sorting_id: str = None):\n    \"\"\"Generate a URL for visualizing and curating a sorting on the web.\n    Will print instructions on how to do the curation.\n\n    Parameters\n    ----------\n    key : dict\n        An entry from SortingviewWorkspace table\n    sortingview_sorting_id : str, optional\n        sortingview sorting ID to visualize. If None then chooses the first one\n\n    Returns\n    -------\n    url : str\n    \"\"\"\n    workspace_uri = (self &amp; key).fetch1(\"workspace_uri\")\n    workspace = sv.load_workspace(workspace_uri)\n    recording_id = workspace.recording_ids[0]\n    if sortingview_sorting_id is None:\n        sortingview_sorting_id = workspace.sorting_ids[0]\n\n    R = workspace.get_recording_extractor(recording_id)\n    S = workspace.get_sorting_extractor(sortingview_sorting_id)\n\n    initial_labels = (Curation &amp; key).fetch(\"curation_labels\")[0]\n    for k, v in initial_labels.items():\n        new_list = []\n        for item in v:\n            if item not in new_list:\n                new_list.append(item)\n        initial_labels[k] = new_list\n    initial_curation = {\"labelsByUnit\": initial_labels}\n\n    # custom metrics\n    unit_metrics = workspace.get_unit_metrics_for_sorting(\n        sortingview_sorting_id\n    )\n\n    # This will print some instructions on how to do the curation\n    # old: sv.trythis_start_sorting_curation\n    url = _generate_url(\n        recording=R,\n        sorting=S,\n        label=workspace.label,\n        initial_curation=initial_curation,\n        raster_plot_subsample_max_firing_rate=50,\n        spike_amplitudes_subsample_max_firing_rate=50,\n        unit_metrics=unit_metrics,\n    )\n    return url\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview/#src.spyglass.spikesorting.v0.sortingview.SortingviewWorkspace.insert_manual_curation", "title": "<code>insert_manual_curation(key, url, description='manually curated')</code>", "text": "<p>Based on information in key for an SortingviewWorkspace, loads the curated sorting from sortingview, saves it (with labels and the optional description) and inserts it to CuratedSorting</p> <p>Assumes that the workspace corresponding to the recording and (original) sorting exists</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of AutomaticCuration</p> required <code>description</code> <p>description of curated sorting</p> <code>'manually curated'</code> Source code in <code>src/spyglass/spikesorting/v0/sortingview.py</code> <pre><code>def insert_manual_curation(\n    self, key: dict, url: str, description=\"manually curated\"\n):\n    \"\"\"Based on information in key for an SortingviewWorkspace, loads the\n    curated sorting from sortingview, saves it (with labels and the\n    optional description) and inserts it to CuratedSorting\n\n    Assumes that the workspace corresponding to the recording and (original) sorting exists\n\n    Parameters\n    ----------\n    key : dict\n        primary key of AutomaticCuration\n    description: str, optional\n        description of curated sorting\n    \"\"\"\n\n    # get the labels and remove the non-primary merged units\n    # labels = workspace.get_sorting_curation(sorting_id=sortingview_sorting_id)\n    # labels = sv.trythis_load_sorting_curation('jot://xTzzyDieQPkW')\n    labels = sv.trythis_load_sorting_curation(url)\n\n    # turn labels to list of str, only including accepted units.\n    # if bool(labels[\"mergeGroups\"]):\n    if bool(labels.get(\"mergeGroups\", [])):\n        # clusters were merged, so we empty out metrics\n        metrics = {}\n    else:\n        # get the metrics from the parent curation\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n\n    # insert this curation into the  Table\n    return Curation.insert_curation(\n        key,\n        parent_curation_id=key[\"curation_id\"],\n        labels=labels[\"labelsByUnit\"],\n        merge_groups=labels.get(\"mergeGroups\", []),\n        metrics=metrics,\n        description=description,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v0/sortingview_helper_fn/", "title": "sortingview_helper_fn.py", "text": "<p>Sortingview helper functions</p>"}, {"location": "api/spikesorting/v0/spikesorting_artifact/", "title": "spikesorting_artifact.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_artifact/#src.spyglass.spikesorting.v0.spikesorting_artifact.ArtifactDetectionParameters", "title": "<code>ArtifactDetectionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_artifact.py</code> <pre><code>@schema\nclass ArtifactDetectionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for detecting artifact times within a sort group.\n    artifact_params_name: varchar(200)\n    ---\n    artifact_params: blob  # dictionary of parameters\n    \"\"\"\n\n    def insert_default(self):\n        \"\"\"Insert the default artifact parameters with an appropriate parameter dict.\"\"\"\n        artifact_params = {}\n        artifact_params[\"zscore_thresh\"] = None  # must be None or &gt;= 0\n        artifact_params[\"amplitude_thresh\"] = 3000  # must be None or &gt;= 0\n        # all electrodes of sort group\n        artifact_params[\"proportion_above_thresh\"] = 1.0\n        artifact_params[\"removal_window_ms\"] = 1.0  # in milliseconds\n        self.insert1([\"default\", artifact_params], skip_duplicates=True)\n\n        artifact_params_none = {}\n        artifact_params_none[\"zscore_thresh\"] = None\n        artifact_params_none[\"amplitude_thresh\"] = None\n        self.insert1([\"none\", artifact_params_none], skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_artifact/#src.spyglass.spikesorting.v0.spikesorting_artifact.ArtifactDetectionParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Insert the default artifact parameters with an appropriate parameter dict.</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_artifact.py</code> <pre><code>def insert_default(self):\n    \"\"\"Insert the default artifact parameters with an appropriate parameter dict.\"\"\"\n    artifact_params = {}\n    artifact_params[\"zscore_thresh\"] = None  # must be None or &gt;= 0\n    artifact_params[\"amplitude_thresh\"] = 3000  # must be None or &gt;= 0\n    # all electrodes of sort group\n    artifact_params[\"proportion_above_thresh\"] = 1.0\n    artifact_params[\"removal_window_ms\"] = 1.0  # in milliseconds\n    self.insert1([\"default\", artifact_params], skip_duplicates=True)\n\n    artifact_params_none = {}\n    artifact_params_none[\"zscore_thresh\"] = None\n    artifact_params_none[\"amplitude_thresh\"] = None\n    self.insert1([\"none\", artifact_params_none], skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/", "title": "spikesorting_curation.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation", "title": "<code>Curation</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass Curation(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Stores each spike sorting; similar to IntervalList\n    curation_id: int # a number corresponding to the index of this curation\n    -&gt; SpikeSorting\n    ---\n    parent_curation_id=-1: int\n    curation_labels: blob # a dictionary of labels for the units\n    merge_groups: blob # a list of merge groups for the units\n    quality_metrics: blob # a list of quality metrics for the units (if available)\n    description='': varchar(1000) #optional description for this curated sort\n    time_of_creation: int   # in Unix time, to the nearest second\n    \"\"\"\n\n    _nwb_table = AnalysisNwbfile\n\n    @staticmethod\n    def insert_curation(\n        sorting_key: dict,\n        parent_curation_id: int = -1,\n        labels=None,\n        merge_groups=None,\n        metrics=None,\n        description=\"\",\n    ):\n        \"\"\"Given a SpikeSorting key and the parent_sorting_id (and optional\n        arguments) insert an entry into Curation.\n\n\n        Parameters\n        ----------\n        sorting_key : dict\n            The key for the original SpikeSorting\n        parent_curation_id : int, optional\n            The id of the parent sorting\n        labels : dict or None, optional\n        merge_groups : dict or None, optional\n        metrics : dict or None, optional\n            Computed metrics for sorting\n        description : str, optional\n            text description of this sort\n\n        Returns\n        -------\n        curation_key : dict\n\n        \"\"\"\n        if parent_curation_id == -1:\n            # check to see if this sorting with a parent of -1 has already been\n            # inserted and if so, warn the user\n            inserted_curation = (Curation &amp; sorting_key).fetch(\"KEY\")\n            if len(inserted_curation) &gt; 0:\n                Warning(\n                    \"Sorting has already been inserted, returning key to previously\"\n                    \"inserted curation\"\n                )\n                return inserted_curation[0]\n\n        if labels is None:\n            labels = {}\n        if merge_groups is None:\n            merge_groups = []\n        if metrics is None:\n            metrics = {}\n\n        # generate a unique number for this curation\n        id = (Curation &amp; sorting_key).fetch(\"curation_id\")\n        if len(id) &gt; 0:\n            curation_id = max(id) + 1\n        else:\n            curation_id = 0\n\n        # convert unit_ids in labels to integers for labels from sortingview.\n        new_labels = {int(unit_id): labels[unit_id] for unit_id in labels}\n\n        sorting_key[\"curation_id\"] = curation_id\n        sorting_key[\"parent_curation_id\"] = parent_curation_id\n        sorting_key[\"description\"] = description\n        sorting_key[\"curation_labels\"] = new_labels\n        sorting_key[\"merge_groups\"] = merge_groups\n        sorting_key[\"quality_metrics\"] = metrics\n        sorting_key[\"time_of_creation\"] = int(time.time())\n\n        # mike: added skip duplicates\n        Curation.insert1(sorting_key, skip_duplicates=True)\n\n        # get the primary key for this curation\n        c_key = Curation.fetch(\"KEY\")[0]\n        curation_key = {item: sorting_key[item] for item in c_key}\n\n        return curation_key\n\n    @staticmethod\n    def get_recording(key: dict):\n        \"\"\"Returns the recording extractor for the recording related to this curation\n\n        Parameters\n        ----------\n        key : dict\n            SpikeSortingRecording key\n\n        Returns\n        -------\n        recording_extractor : spike interface recording extractor\n\n        \"\"\"\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        return si.load_extractor(recording_path)\n\n    @staticmethod\n    def get_curated_sorting(key: dict):\n        \"\"\"Returns the sorting extractor related to this curation,\n        with merges applied.\n\n        Parameters\n        ----------\n        key : dict\n            Curation key\n\n        Returns\n        -------\n        sorting_extractor: spike interface sorting extractor\n\n        \"\"\"\n        sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n        sorting = si.load_extractor(sorting_path)\n        merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n        # TODO: write code to get merged sorting extractor\n        if len(merge_groups) != 0:\n            return MergedSortingExtractor(\n                parent_sorting=sorting, merge_groups=merge_groups\n            )\n        else:\n            return sorting\n\n    @staticmethod\n    def save_sorting_nwb(\n        key,\n        sorting,\n        timestamps,\n        sort_interval_list_name,\n        sort_interval,\n        labels=None,\n        metrics=None,\n        unit_ids=None,\n    ):\n        \"\"\"Store a sorting in a new AnalysisNwbfile\n\n        Parameters\n        ----------\n        key : dict\n            key to SpikeSorting table\n        sorting : si.Sorting\n            sorting\n        timestamps : array_like\n            Time stamps of the sorted recoridng;\n            used to convert the spike timings from index to real time\n        sort_interval_list_name : str\n            name of sort interval\n        sort_interval : list\n            interval for start and end of sort\n        labels : dict, optional\n            curation labels, by default None\n        metrics : dict, optional\n            quality metrics, by default None\n        unit_ids : list, optional\n            IDs of units whose spiketrains to save, by default None\n\n        Returns\n        -------\n        analysis_file_name : str\n        units_object_id : str\n\n        \"\"\"\n\n        sort_interval_valid_times = (\n            IntervalList &amp; {\"interval_list_name\": sort_interval_list_name}\n        ).fetch1(\"valid_times\")\n\n        units = dict()\n        units_valid_times = dict()\n        units_sort_interval = dict()\n\n        if unit_ids is None:\n            unit_ids = sorting.get_unit_ids()\n\n        for unit_id in unit_ids:\n            spike_times_in_samples = sorting.get_unit_spike_train(\n                unit_id=unit_id\n            )\n            units[unit_id] = timestamps[spike_times_in_samples]\n            units_valid_times[unit_id] = sort_interval_valid_times\n            units_sort_interval[unit_id] = [sort_interval]\n\n        analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n        object_ids = AnalysisNwbfile().add_units(\n            analysis_file_name,\n            units,\n            units_valid_times,\n            units_sort_interval,\n            metrics=metrics,\n            labels=labels,\n        )\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n        if object_ids == \"\":\n            logger.warn(\n                \"Sorting contains no units.\"\n                \"Created an empty analysis nwb file anyway.\"\n            )\n            units_object_id = \"\"\n        else:\n            units_object_id = object_ids[0]\n\n        return analysis_file_name, units_object_id\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.insert_curation", "title": "<code>insert_curation(sorting_key, parent_curation_id=-1, labels=None, merge_groups=None, metrics=None, description='')</code>  <code>staticmethod</code>", "text": "<p>Given a SpikeSorting key and the parent_sorting_id (and optional arguments) insert an entry into Curation.</p> <p>Parameters:</p> Name Type Description Default <code>sorting_key</code> <code>dict</code> <p>The key for the original SpikeSorting</p> required <code>parent_curation_id</code> <code>int</code> <p>The id of the parent sorting</p> <code>-1</code> <code>labels</code> <code>dict or None</code> <code>None</code> <code>merge_groups</code> <code>dict or None</code> <code>None</code> <code>metrics</code> <code>dict or None</code> <p>Computed metrics for sorting</p> <code>None</code> <code>description</code> <code>str</code> <p>text description of this sort</p> <code>''</code> <p>Returns:</p> Name Type Description <code>curation_key</code> <code>dict</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef insert_curation(\n    sorting_key: dict,\n    parent_curation_id: int = -1,\n    labels=None,\n    merge_groups=None,\n    metrics=None,\n    description=\"\",\n):\n    \"\"\"Given a SpikeSorting key and the parent_sorting_id (and optional\n    arguments) insert an entry into Curation.\n\n\n    Parameters\n    ----------\n    sorting_key : dict\n        The key for the original SpikeSorting\n    parent_curation_id : int, optional\n        The id of the parent sorting\n    labels : dict or None, optional\n    merge_groups : dict or None, optional\n    metrics : dict or None, optional\n        Computed metrics for sorting\n    description : str, optional\n        text description of this sort\n\n    Returns\n    -------\n    curation_key : dict\n\n    \"\"\"\n    if parent_curation_id == -1:\n        # check to see if this sorting with a parent of -1 has already been\n        # inserted and if so, warn the user\n        inserted_curation = (Curation &amp; sorting_key).fetch(\"KEY\")\n        if len(inserted_curation) &gt; 0:\n            Warning(\n                \"Sorting has already been inserted, returning key to previously\"\n                \"inserted curation\"\n            )\n            return inserted_curation[0]\n\n    if labels is None:\n        labels = {}\n    if merge_groups is None:\n        merge_groups = []\n    if metrics is None:\n        metrics = {}\n\n    # generate a unique number for this curation\n    id = (Curation &amp; sorting_key).fetch(\"curation_id\")\n    if len(id) &gt; 0:\n        curation_id = max(id) + 1\n    else:\n        curation_id = 0\n\n    # convert unit_ids in labels to integers for labels from sortingview.\n    new_labels = {int(unit_id): labels[unit_id] for unit_id in labels}\n\n    sorting_key[\"curation_id\"] = curation_id\n    sorting_key[\"parent_curation_id\"] = parent_curation_id\n    sorting_key[\"description\"] = description\n    sorting_key[\"curation_labels\"] = new_labels\n    sorting_key[\"merge_groups\"] = merge_groups\n    sorting_key[\"quality_metrics\"] = metrics\n    sorting_key[\"time_of_creation\"] = int(time.time())\n\n    # mike: added skip duplicates\n    Curation.insert1(sorting_key, skip_duplicates=True)\n\n    # get the primary key for this curation\n    c_key = Curation.fetch(\"KEY\")[0]\n    curation_key = {item: sorting_key[item] for item in c_key}\n\n    return curation_key\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.get_recording", "title": "<code>get_recording(key)</code>  <code>staticmethod</code>", "text": "<p>Returns the recording extractor for the recording related to this curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>SpikeSortingRecording key</p> required <p>Returns:</p> Name Type Description <code>recording_extractor</code> <code>spike interface recording extractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_recording(key: dict):\n    \"\"\"Returns the recording extractor for the recording related to this curation\n\n    Parameters\n    ----------\n    key : dict\n        SpikeSortingRecording key\n\n    Returns\n    -------\n    recording_extractor : spike interface recording extractor\n\n    \"\"\"\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    return si.load_extractor(recording_path)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.get_curated_sorting", "title": "<code>get_curated_sorting(key)</code>  <code>staticmethod</code>", "text": "<p>Returns the sorting extractor related to this curation, with merges applied.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Curation key</p> required <p>Returns:</p> Name Type Description <code>sorting_extractor</code> <code>spike interface sorting extractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_curated_sorting(key: dict):\n    \"\"\"Returns the sorting extractor related to this curation,\n    with merges applied.\n\n    Parameters\n    ----------\n    key : dict\n        Curation key\n\n    Returns\n    -------\n    sorting_extractor: spike interface sorting extractor\n\n    \"\"\"\n    sorting_path = (SpikeSorting &amp; key).fetch1(\"sorting_path\")\n    sorting = si.load_extractor(sorting_path)\n    merge_groups = (Curation &amp; key).fetch1(\"merge_groups\")\n    # TODO: write code to get merged sorting extractor\n    if len(merge_groups) != 0:\n        return MergedSortingExtractor(\n            parent_sorting=sorting, merge_groups=merge_groups\n        )\n    else:\n        return sorting\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Curation.save_sorting_nwb", "title": "<code>save_sorting_nwb(key, sorting, timestamps, sort_interval_list_name, sort_interval, labels=None, metrics=None, unit_ids=None)</code>  <code>staticmethod</code>", "text": "<p>Store a sorting in a new AnalysisNwbfile</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>key to SpikeSorting table</p> required <code>sorting</code> <code>Sorting</code> <p>sorting</p> required <code>timestamps</code> <code>array_like</code> <p>Time stamps of the sorted recoridng; used to convert the spike timings from index to real time</p> required <code>sort_interval_list_name</code> <code>str</code> <p>name of sort interval</p> required <code>sort_interval</code> <code>list</code> <p>interval for start and end of sort</p> required <code>labels</code> <code>dict</code> <p>curation labels, by default None</p> <code>None</code> <code>metrics</code> <code>dict</code> <p>quality metrics, by default None</p> <code>None</code> <code>unit_ids</code> <code>list</code> <p>IDs of units whose spiketrains to save, by default None</p> <code>None</code> <p>Returns:</p> Name Type Description <code>analysis_file_name</code> <code>str</code> <code>units_object_id</code> <code>str</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef save_sorting_nwb(\n    key,\n    sorting,\n    timestamps,\n    sort_interval_list_name,\n    sort_interval,\n    labels=None,\n    metrics=None,\n    unit_ids=None,\n):\n    \"\"\"Store a sorting in a new AnalysisNwbfile\n\n    Parameters\n    ----------\n    key : dict\n        key to SpikeSorting table\n    sorting : si.Sorting\n        sorting\n    timestamps : array_like\n        Time stamps of the sorted recoridng;\n        used to convert the spike timings from index to real time\n    sort_interval_list_name : str\n        name of sort interval\n    sort_interval : list\n        interval for start and end of sort\n    labels : dict, optional\n        curation labels, by default None\n    metrics : dict, optional\n        quality metrics, by default None\n    unit_ids : list, optional\n        IDs of units whose spiketrains to save, by default None\n\n    Returns\n    -------\n    analysis_file_name : str\n    units_object_id : str\n\n    \"\"\"\n\n    sort_interval_valid_times = (\n        IntervalList &amp; {\"interval_list_name\": sort_interval_list_name}\n    ).fetch1(\"valid_times\")\n\n    units = dict()\n    units_valid_times = dict()\n    units_sort_interval = dict()\n\n    if unit_ids is None:\n        unit_ids = sorting.get_unit_ids()\n\n    for unit_id in unit_ids:\n        spike_times_in_samples = sorting.get_unit_spike_train(\n            unit_id=unit_id\n        )\n        units[unit_id] = timestamps[spike_times_in_samples]\n        units_valid_times[unit_id] = sort_interval_valid_times\n        units_sort_interval[unit_id] = [sort_interval]\n\n    analysis_file_name = AnalysisNwbfile().create(key[\"nwb_file_name\"])\n    object_ids = AnalysisNwbfile().add_units(\n        analysis_file_name,\n        units,\n        units_valid_times,\n        units_sort_interval,\n        metrics=metrics,\n        labels=labels,\n    )\n    AnalysisNwbfile().add(key[\"nwb_file_name\"], analysis_file_name)\n\n    if object_ids == \"\":\n        logger.warn(\n            \"Sorting contains no units.\"\n            \"Created an empty analysis nwb file anyway.\"\n        )\n        units_object_id = \"\"\n    else:\n        units_object_id = object_ids[0]\n\n    return analysis_file_name, units_object_id\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Waveforms", "title": "<code>Waveforms</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass Waveforms(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; WaveformSelection\n    ---\n    waveform_extractor_path: varchar(400)\n    -&gt; AnalysisNwbfile\n    waveforms_object_id: varchar(40)   # Object ID for the waveforms in NWB file\n    \"\"\"\n\n    def make(self, key):\n        recording = Curation.get_recording(key)\n        if recording.get_num_segments() &gt; 1:\n            recording = si.concatenate_recordings([recording])\n\n        sorting = Curation.get_curated_sorting(key)\n\n        logger.info(\"Extracting waveforms...\")\n        waveform_params = (WaveformParameters &amp; key).fetch1(\"waveform_params\")\n        if \"whiten\" in waveform_params:\n            if waveform_params.pop(\"whiten\"):\n                recording = sip.whiten(recording, dtype=\"float32\")\n\n        waveform_extractor_name = self._get_waveform_extractor_name(key)\n        key[\"waveform_extractor_path\"] = str(\n            Path(waveforms_dir) / Path(waveform_extractor_name)\n        )\n        if os.path.exists(key[\"waveform_extractor_path\"]):\n            shutil.rmtree(key[\"waveform_extractor_path\"])\n        waveforms = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=key[\"waveform_extractor_path\"],\n            **waveform_params,\n        )\n\n        key[\"analysis_file_name\"] = AnalysisNwbfile().create(\n            key[\"nwb_file_name\"]\n        )\n        object_id = AnalysisNwbfile().add_units_waveforms(\n            key[\"analysis_file_name\"], waveform_extractor=waveforms\n        )\n        key[\"waveforms_object_id\"] = object_id\n        AnalysisNwbfile().add(key[\"nwb_file_name\"], key[\"analysis_file_name\"])\n\n        self.insert1(key)\n\n    def load_waveforms(self, key: dict):\n        \"\"\"Returns a spikeinterface waveform extractor specified by key\n\n        Parameters\n        ----------\n        key : dict\n            Could be an entry in Waveforms, or some other key that uniquely defines\n            an entry in Waveforms\n\n        Returns\n        -------\n        we : spikeinterface.WaveformExtractor\n        \"\"\"\n        we_path = (self &amp; key).fetch1(\"waveform_extractor_path\")\n        we = si.WaveformExtractor.load_from_folder(we_path)\n        return we\n\n    def fetch_nwb(self, key):\n        # TODO: implement fetching waveforms from NWB\n        return NotImplementedError\n\n    def _get_waveform_extractor_name(self, key):\n        waveform_params_name = (WaveformParameters &amp; key).fetch1(\n            \"waveform_params_name\"\n        )\n\n        return (\n            f'{key[\"nwb_file_name\"]}_{str(uuid.uuid4())[0:8]}_'\n            f'{key[\"curation_id\"]}_{waveform_params_name}_waveforms'\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.Waveforms.load_waveforms", "title": "<code>load_waveforms(key)</code>", "text": "<p>Returns a spikeinterface waveform extractor specified by key</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>Could be an entry in Waveforms, or some other key that uniquely defines an entry in Waveforms</p> required <p>Returns:</p> Name Type Description <code>we</code> <code>WaveformExtractor</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def load_waveforms(self, key: dict):\n    \"\"\"Returns a spikeinterface waveform extractor specified by key\n\n    Parameters\n    ----------\n    key : dict\n        Could be an entry in Waveforms, or some other key that uniquely defines\n        an entry in Waveforms\n\n    Returns\n    -------\n    we : spikeinterface.WaveformExtractor\n    \"\"\"\n    we_path = (self &amp; key).fetch1(\"waveform_extractor_path\")\n    we = si.WaveformExtractor.load_from_folder(we_path)\n    return we\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.MetricParameters", "title": "<code>MetricParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass MetricParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Parameters for computing quality metrics of sorted units\n    metric_params_name: varchar(64)\n    ---\n    metric_params: blob\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    metric_default_params = {\n        \"snr\": {\n            \"peak_sign\": \"neg\",\n            \"random_chunk_kwargs_dict\": {\n                \"num_chunks_per_segment\": 20,\n                \"chunk_size\": 10000,\n                \"seed\": 0,\n            },\n        },\n        \"isi_violation\": {\"isi_threshold_ms\": 1.5, \"min_isi_ms\": 0.0},\n        \"nn_isolation\": {\n            \"max_spikes\": 1000,\n            \"min_spikes\": 10,\n            \"n_neighbors\": 5,\n            \"n_components\": 7,\n            \"radius_um\": 100,\n            \"seed\": 0,\n        },\n        \"nn_noise_overlap\": {\n            \"max_spikes\": 1000,\n            \"min_spikes\": 10,\n            \"n_neighbors\": 5,\n            \"n_components\": 7,\n            \"radius_um\": 100,\n            \"seed\": 0,\n        },\n        \"peak_channel\": {\"peak_sign\": \"neg\"},\n        \"num_spikes\": {},\n    }\n    # Example of peak_offset parameters 'peak_offset': {'peak_sign': 'neg'}\n    available_metrics = [\n        \"snr\",\n        \"isi_violation\",\n        \"nn_isolation\",\n        \"nn_noise_overlap\",\n        \"peak_offset\",\n        \"peak_channel\",\n        \"num_spikes\",\n    ]\n\n    def get_metric_default_params(self, metric: str):\n        \"Returns default params for the given metric\"\n        return self.metric_default_params(metric)\n\n    def insert_default(self):\n        self.insert1(\n            [\"franklab_default3\", self.metric_default_params],\n            skip_duplicates=True,\n        )\n\n    def get_available_metrics(self):\n        for metric in _metric_name_to_func:\n            if metric in self.available_metrics:\n                metric_doc = _metric_name_to_func[metric].__doc__.split(\"\\n\")[0]\n                metric_string = (\"{metric_name} : {metric_doc}\").format(\n                    metric_name=metric, metric_doc=metric_doc\n                )\n                logger.info(metric_string + \"\\n\")\n\n    # TODO\n    def _validate_metrics_list(self, key):\n        \"\"\"Checks whether a row to be inserted contains only the available metrics\"\"\"\n        # get available metrics list\n        # get metric list from key\n        # compare\n        return NotImplementedError\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.MetricParameters.get_metric_default_params", "title": "<code>get_metric_default_params(metric)</code>", "text": "<p>Returns default params for the given metric</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def get_metric_default_params(self, metric: str):\n    \"Returns default params for the given metric\"\n    return self.metric_default_params(metric)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration", "title": "<code>AutomaticCuration</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass AutomaticCuration(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; AutomaticCurationSelection\n    ---\n    auto_curation_key: blob # the key to the curation inserted by make\n    \"\"\"\n\n    def make(self, key):\n        metrics_path = (QualityMetrics &amp; key).fetch1(\"quality_metrics_path\")\n        with open(metrics_path) as f:\n            quality_metrics = json.load(f)\n\n        # get the curation information and the curated sorting\n        parent_curation = (Curation &amp; key).fetch(as_dict=True)[0]\n        parent_merge_groups = parent_curation[\"merge_groups\"]\n        parent_labels = parent_curation[\"curation_labels\"]\n        parent_curation_id = parent_curation[\"curation_id\"]\n        parent_sorting = Curation.get_curated_sorting(key)\n\n        merge_params = (AutomaticCurationParameters &amp; key).fetch1(\n            \"merge_params\"\n        )\n        merge_groups, units_merged = self.get_merge_groups(\n            parent_sorting, parent_merge_groups, quality_metrics, merge_params\n        )\n\n        label_params = (AutomaticCurationParameters &amp; key).fetch1(\n            \"label_params\"\n        )\n        labels = self.get_labels(\n            parent_sorting, parent_labels, quality_metrics, label_params\n        )\n\n        # keep the quality metrics only if no merging occurred.\n        metrics = quality_metrics if not units_merged else None\n\n        # insert this sorting into the CuratedSpikeSorting Table\n        # first remove keys that aren't part of the Sorting (the primary key of curation)\n        c_key = (SpikeSorting &amp; key).fetch(\"KEY\")[0]\n        curation_key = {item: key[item] for item in key if item in c_key}\n        key[\"auto_curation_key\"] = Curation.insert_curation(\n            curation_key,\n            parent_curation_id=parent_curation_id,\n            labels=labels,\n            merge_groups=merge_groups,\n            metrics=metrics,\n            description=\"auto curated\",\n        )\n\n        self.insert1(key)\n\n    @staticmethod\n    def get_merge_groups(\n        sorting, parent_merge_groups, quality_metrics, merge_params\n    ):\n        \"\"\"Identifies units to be merged based on the quality_metrics and\n        merge parameters and returns an updated list of merges for the curation.\n\n        Parameters\n        ---------\n        sorting : spikeinterface.sorting\n        parent_merge_groups : list\n            Information about previous merges\n        quality_metrics : list\n        merge_params : dict\n\n        Returns\n        -------\n        merge_groups : list of lists\n        merge_occurred : bool\n\n        \"\"\"\n\n        # overview:\n        # 1. Use quality metrics to determine merge groups for units\n        # 2. Combine merge groups with current merge groups to produce union of merges\n\n        if not merge_params:\n            return parent_merge_groups, False\n        else:\n            # TODO: use the metrics to identify clusters that should be merged\n            # new_merges should then reflect those merges and the line below should be deleted.\n            new_merges = []\n            # append these merges to the parent merge_groups\n            for new_merge in new_merges:\n                # check to see if the first cluster listed is in a current merge group\n                for previous_merge in parent_merge_groups:\n                    if new_merge[0] == previous_merge[0]:\n                        # add the additional units in new_merge to the identified merge group.\n                        previous_merge.extend(new_merge[1:])\n                        previous_merge.sort()\n                        break\n                else:\n                    # append this merge group to the list if no previous merge\n                    parent_merge_groups.append(new_merge)\n            return parent_merge_groups.sort(), True\n\n    @staticmethod\n    def get_labels(sorting, parent_labels, quality_metrics, label_params):\n        \"\"\"Returns a dictionary of labels using quality_metrics and label\n        parameters.\n\n        Parameters\n        ---------\n        sorting : spikeinterface.sorting\n        parent_labels : list\n            Information about previous merges\n        quality_metrics : list\n        label_params : dict\n\n        Returns\n        -------\n        parent_labels : list\n\n        \"\"\"\n        # overview:\n        # 1. Use quality metrics to determine labels for units\n        # 2. Append labels to current labels, checking for inconsistencies\n        if not label_params:\n            return parent_labels\n        else:\n            for metric in label_params:\n                if metric not in quality_metrics:\n                    Warning(f\"{metric} not found in quality metrics; skipping\")\n                else:\n                    compare = _comparison_to_function[label_params[metric][0]]\n\n                    for unit_id in quality_metrics[metric].keys():\n                        # compare the quality metric to the threshold with the specified operator\n                        # note that label_params[metric] is a three element list with a comparison operator as a string,\n                        # the threshold value, and a list of labels to be applied if the comparison is true\n                        if compare(\n                            quality_metrics[metric][unit_id],\n                            label_params[metric][1],\n                        ):\n                            if unit_id not in parent_labels:\n                                parent_labels[unit_id] = label_params[metric][2]\n                            # check if the label is already there, and if not, add it\n                            elif (\n                                label_params[metric][2]\n                                not in parent_labels[unit_id]\n                            ):\n                                parent_labels[unit_id].extend(\n                                    label_params[metric][2]\n                                )\n            return parent_labels\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration.get_merge_groups", "title": "<code>get_merge_groups(sorting, parent_merge_groups, quality_metrics, merge_params)</code>  <code>staticmethod</code>", "text": "<p>Identifies units to be merged based on the quality_metrics and merge parameters and returns an updated list of merges for the curation.</p> <p>Parameters:</p> Name Type Description Default <code>sorting</code> <code>sorting</code> required <code>parent_merge_groups</code> <code>list</code> <p>Information about previous merges</p> required <code>quality_metrics</code> <code>list</code> required <code>merge_params</code> <code>dict</code> required <p>Returns:</p> Name Type Description <code>merge_groups</code> <code>list of lists</code> <code>merge_occurred</code> <code>bool</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_merge_groups(\n    sorting, parent_merge_groups, quality_metrics, merge_params\n):\n    \"\"\"Identifies units to be merged based on the quality_metrics and\n    merge parameters and returns an updated list of merges for the curation.\n\n    Parameters\n    ---------\n    sorting : spikeinterface.sorting\n    parent_merge_groups : list\n        Information about previous merges\n    quality_metrics : list\n    merge_params : dict\n\n    Returns\n    -------\n    merge_groups : list of lists\n    merge_occurred : bool\n\n    \"\"\"\n\n    # overview:\n    # 1. Use quality metrics to determine merge groups for units\n    # 2. Combine merge groups with current merge groups to produce union of merges\n\n    if not merge_params:\n        return parent_merge_groups, False\n    else:\n        # TODO: use the metrics to identify clusters that should be merged\n        # new_merges should then reflect those merges and the line below should be deleted.\n        new_merges = []\n        # append these merges to the parent merge_groups\n        for new_merge in new_merges:\n            # check to see if the first cluster listed is in a current merge group\n            for previous_merge in parent_merge_groups:\n                if new_merge[0] == previous_merge[0]:\n                    # add the additional units in new_merge to the identified merge group.\n                    previous_merge.extend(new_merge[1:])\n                    previous_merge.sort()\n                    break\n            else:\n                # append this merge group to the list if no previous merge\n                parent_merge_groups.append(new_merge)\n        return parent_merge_groups.sort(), True\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.AutomaticCuration.get_labels", "title": "<code>get_labels(sorting, parent_labels, quality_metrics, label_params)</code>  <code>staticmethod</code>", "text": "<p>Returns a dictionary of labels using quality_metrics and label parameters.</p> <p>Parameters:</p> Name Type Description Default <code>sorting</code> <code>sorting</code> required <code>parent_labels</code> <code>list</code> <p>Information about previous merges</p> required <code>quality_metrics</code> <code>list</code> required <code>label_params</code> <code>dict</code> required <p>Returns:</p> Name Type Description <code>parent_labels</code> <code>list</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@staticmethod\ndef get_labels(sorting, parent_labels, quality_metrics, label_params):\n    \"\"\"Returns a dictionary of labels using quality_metrics and label\n    parameters.\n\n    Parameters\n    ---------\n    sorting : spikeinterface.sorting\n    parent_labels : list\n        Information about previous merges\n    quality_metrics : list\n    label_params : dict\n\n    Returns\n    -------\n    parent_labels : list\n\n    \"\"\"\n    # overview:\n    # 1. Use quality metrics to determine labels for units\n    # 2. Append labels to current labels, checking for inconsistencies\n    if not label_params:\n        return parent_labels\n    else:\n        for metric in label_params:\n            if metric not in quality_metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n            else:\n                compare = _comparison_to_function[label_params[metric][0]]\n\n                for unit_id in quality_metrics[metric].keys():\n                    # compare the quality metric to the threshold with the specified operator\n                    # note that label_params[metric] is a three element list with a comparison operator as a string,\n                    # the threshold value, and a list of labels to be applied if the comparison is true\n                    if compare(\n                        quality_metrics[metric][unit_id],\n                        label_params[metric][1],\n                    ):\n                        if unit_id not in parent_labels:\n                            parent_labels[unit_id] = label_params[metric][2]\n                        # check if the label is already there, and if not, add it\n                        elif (\n                            label_params[metric][2]\n                            not in parent_labels[unit_id]\n                        ):\n                            parent_labels[unit_id].extend(\n                                label_params[metric][2]\n                            )\n        return parent_labels\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting", "title": "<code>CuratedSpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass CuratedSpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; CuratedSpikeSortingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    units_object_id: varchar(40)\n    \"\"\"\n\n    class Unit(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        # Table for holding sorted units\n        -&gt; CuratedSpikeSorting\n        unit_id: int   # ID for each unit\n        ---\n        label='': varchar(200)   # optional set of labels for each unit\n        nn_noise_overlap=-1: float   # noise overlap metric for each unit\n        nn_isolation=-1: float   # isolation score metric for each unit\n        isi_violation=-1: float   # ISI violation score for each unit\n        snr=0: float            # SNR for each unit\n        firing_rate=-1: float   # firing rate\n        num_spikes=-1: int   # total number of spikes\n        peak_channel=null: int # channel of maximum amplitude for each unit\n        \"\"\"\n\n    def make(self, key):\n        unit_labels_to_remove = [\"reject\"]\n        # check that the Curation has metrics\n        metrics = (Curation &amp; key).fetch1(\"quality_metrics\")\n        if metrics == {}:\n            Warning(\n                f\"Metrics for Curation {key} should normally be calculated before insertion here\"\n            )\n\n        sorting = Curation.get_curated_sorting(key)\n        unit_ids = sorting.get_unit_ids()\n        # Get the labels for the units, add only those units that do not have 'reject' or 'noise' labels\n        unit_labels = (Curation &amp; key).fetch1(\"curation_labels\")\n        accepted_units = []\n        for unit_id in unit_ids:\n            if unit_id in unit_labels:\n                if (\n                    len(set(unit_labels_to_remove) &amp; set(unit_labels[unit_id]))\n                    == 0\n                ):\n                    accepted_units.append(unit_id)\n            else:\n                accepted_units.append(unit_id)\n\n        # get the labels for the accepted units\n        labels = {}\n        for unit_id in accepted_units:\n            if unit_id in unit_labels:\n                labels[unit_id] = \",\".join(unit_labels[unit_id])\n\n        # convert unit_ids in metrics to integers, including only accepted units.\n        #  TODO: convert to int this somewhere else\n        final_metrics = {}\n        for metric in metrics:\n            final_metrics[metric] = {\n                int(unit_id): metrics[metric][unit_id]\n                for unit_id in metrics[metric]\n                if int(unit_id) in accepted_units\n            }\n\n        logger.info(f\"Found {len(accepted_units)} accepted units\")\n\n        # get the sorting and save it in the NWB file\n        sorting = Curation.get_curated_sorting(key)\n        recording = Curation.get_recording(key)\n\n        # get the sort_interval and sorting interval list\n        sort_interval = (SortInterval &amp; key).fetch1(\"sort_interval\")\n        sort_interval_list_name = (SpikeSorting &amp; key).fetch1(\n            \"artifact_removed_interval_list_name\"\n        )\n\n        timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n\n        (\n            key[\"analysis_file_name\"],\n            key[\"units_object_id\"],\n        ) = Curation().save_sorting_nwb(\n            key,\n            sorting,\n            timestamps,\n            sort_interval_list_name,\n            sort_interval,\n            metrics=final_metrics,\n            unit_ids=accepted_units,\n            labels=labels,\n        )\n        self.insert1(key)\n\n        # now add the units\n        # Remove the non primary key entries.\n        del key[\"units_object_id\"]\n        del key[\"analysis_file_name\"]\n\n        metric_fields = self.metrics_fields()\n        for unit_id in accepted_units:\n            key[\"unit_id\"] = unit_id\n            if unit_id in labels:\n                key[\"label\"] = labels[unit_id]\n            for field in metric_fields:\n                if field in final_metrics:\n                    key[field] = final_metrics[field][unit_id]\n                else:\n                    Warning(\n                        f\"No metric named {field} in computed unit quality metrics; skipping\"\n                    )\n            CuratedSpikeSorting.Unit.insert1(key)\n\n    def metrics_fields(self):\n        \"\"\"Returns a list of the metrics that are currently in the Units table.\"\"\"\n        unit_info = self.Unit().fetch(limit=1, format=\"frame\")\n        unit_fields = [column for column in unit_info.columns]\n        unit_fields.remove(\"label\")\n        return unit_fields\n\n    @classmethod\n    def get_recording(cls, key):\n        \"\"\"Returns the recording related to this curation. Useful for operations downstream of merge table\"\"\"\n        # expand the key\n        recording_key = (cls &amp; key).fetch1(\"KEY\")\n        return SpikeSortingRecording()._get_filtered_recording(recording_key)\n\n    @classmethod\n    def get_sorting(cls, key):\n        \"\"\"Returns the sorting related to this curation. Useful for operations downstream of merge table\"\"\"\n        # expand the key\n        sorting_key = (cls &amp; key).fetch1(\"KEY\")\n        return Curation.get_curated_sorting(sorting_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.metrics_fields", "title": "<code>metrics_fields()</code>", "text": "<p>Returns a list of the metrics that are currently in the Units table.</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def metrics_fields(self):\n    \"\"\"Returns a list of the metrics that are currently in the Units table.\"\"\"\n    unit_info = self.Unit().fetch(limit=1, format=\"frame\")\n    unit_fields = [column for column in unit_info.columns]\n    unit_fields.remove(\"label\")\n    return unit_fields\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Returns the recording related to this curation. Useful for operations downstream of merge table</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@classmethod\ndef get_recording(cls, key):\n    \"\"\"Returns the recording related to this curation. Useful for operations downstream of merge table\"\"\"\n    # expand the key\n    recording_key = (cls &amp; key).fetch1(\"KEY\")\n    return SpikeSortingRecording()._get_filtered_recording(recording_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.CuratedSpikeSorting.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Returns the sorting related to this curation. Useful for operations downstream of merge table</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key):\n    \"\"\"Returns the sorting related to this curation. Useful for operations downstream of merge table\"\"\"\n    # expand the key\n    sorting_key = (cls &amp; key).fetch1(\"KEY\")\n    return Curation.get_curated_sorting(sorting_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.UnitInclusionParameters", "title": "<code>UnitInclusionParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>@schema\nclass UnitInclusionParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    unit_inclusion_param_name: varchar(80) # the name of the list of thresholds for unit inclusion\n    ---\n    inclusion_param_dict: blob # the dictionary of inclusion / exclusion parameters\n    \"\"\"\n\n    def insert1(self, key, **kwargs):\n        # check to see that the dictionary fits the specifications\n        # The inclusion parameter dict has the following form:\n        # param_dict['metric_name'] = (operator, value)\n        #    where operator is '&lt;', '&gt;', &lt;=', '&gt;=', or '==' and value is the comparison (float) value to be used ()\n        # param_dict['exclude_labels'] = [list of labels to exclude]\n        pdict = key[\"inclusion_param_dict\"]\n        metrics_list = CuratedSpikeSorting().metrics_fields()\n\n        for k in pdict:\n            if k not in metrics_list and k != \"exclude_labels\":\n                raise Exception(\n                    f\"key {k} is not a valid element of the inclusion_param_dict\"\n                )\n            if k in metrics_list:\n                if pdict[k][0] not in _comparison_to_function:\n                    raise Exception(\n                        f\"operator {pdict[k][0]} for metric {k} is not in the valid operators list: {_comparison_to_function.keys()}\"\n                    )\n            if k == \"exclude_labels\":\n                for label in pdict[k]:\n                    if label not in valid_labels:\n                        raise Exception(\n                            f\"exclude label {label} is not in the valid_labels list: {valid_labels}\"\n                        )\n        super().insert1(key, **kwargs)\n\n    def get_included_units(\n        self, curated_sorting_key, unit_inclusion_param_name\n    ):\n        \"\"\"Given a reference to a set of curated sorting units and the name of\n        a unit inclusion parameter list, returns unit key\n\n        Parameters\n        ----------\n        curated_sorting_key : dict\n            key to select a set of curated sorting\n        unit_inclusion_param_name : str\n            name of a unit inclusion parameter entry\n\n        Returns\n        -------\n        dict\n            key to select all of the included units\n        \"\"\"\n        curated_sortings = (CuratedSpikeSorting() &amp; curated_sorting_key).fetch()\n        inc_param_dict = (\n            UnitInclusionParameters\n            &amp; {\"unit_inclusion_param_name\": unit_inclusion_param_name}\n        ).fetch1(\"inclusion_param_dict\")\n        units = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch()\n        units_key = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch(\n            \"KEY\"\n        )\n        # get the list of labels to exclude if there is one\n        if \"exclude_labels\" in inc_param_dict:\n            exclude_labels = inc_param_dict[\"exclude_labels\"]\n            del inc_param_dict[\"exclude_labels\"]\n        else:\n            exclude_labels = []\n\n        # create a list of the units to kepp.\n        keep = np.asarray([True] * len(units))\n        for metric in inc_param_dict:\n            # for all units, go through each metric, compare it to the value\n            # specified, and update the list to be kept\n            keep = np.logical_and(\n                keep,\n                _comparison_to_function[inc_param_dict[metric][0]](\n                    units[metric], inc_param_dict[metric][1]\n                ),\n            )\n\n        # now exclude by label if it is specified\n        if len(exclude_labels):\n            for unit_ind in np.ravel(np.argwhere(keep)):\n                labels = units[unit_ind][\"label\"].split(\",\")\n                for label in labels:\n                    if label in exclude_labels:\n                        keep[unit_ind] = False\n                        break\n\n        # return units that passed all of the tests\n        # TODO: Make this more efficient\n        return {i: units_key[i] for i in np.ravel(np.argwhere(keep))}\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_curation/#src.spyglass.spikesorting.v0.spikesorting_curation.UnitInclusionParameters.get_included_units", "title": "<code>get_included_units(curated_sorting_key, unit_inclusion_param_name)</code>", "text": "<p>Given a reference to a set of curated sorting units and the name of a unit inclusion parameter list, returns unit key</p> <p>Parameters:</p> Name Type Description Default <code>curated_sorting_key</code> <code>dict</code> <p>key to select a set of curated sorting</p> required <code>unit_inclusion_param_name</code> <code>str</code> <p>name of a unit inclusion parameter entry</p> required <p>Returns:</p> Type Description <code>dict</code> <p>key to select all of the included units</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_curation.py</code> <pre><code>def get_included_units(\n    self, curated_sorting_key, unit_inclusion_param_name\n):\n    \"\"\"Given a reference to a set of curated sorting units and the name of\n    a unit inclusion parameter list, returns unit key\n\n    Parameters\n    ----------\n    curated_sorting_key : dict\n        key to select a set of curated sorting\n    unit_inclusion_param_name : str\n        name of a unit inclusion parameter entry\n\n    Returns\n    -------\n    dict\n        key to select all of the included units\n    \"\"\"\n    curated_sortings = (CuratedSpikeSorting() &amp; curated_sorting_key).fetch()\n    inc_param_dict = (\n        UnitInclusionParameters\n        &amp; {\"unit_inclusion_param_name\": unit_inclusion_param_name}\n    ).fetch1(\"inclusion_param_dict\")\n    units = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch()\n    units_key = (CuratedSpikeSorting().Unit() &amp; curated_sortings).fetch(\n        \"KEY\"\n    )\n    # get the list of labels to exclude if there is one\n    if \"exclude_labels\" in inc_param_dict:\n        exclude_labels = inc_param_dict[\"exclude_labels\"]\n        del inc_param_dict[\"exclude_labels\"]\n    else:\n        exclude_labels = []\n\n    # create a list of the units to kepp.\n    keep = np.asarray([True] * len(units))\n    for metric in inc_param_dict:\n        # for all units, go through each metric, compare it to the value\n        # specified, and update the list to be kept\n        keep = np.logical_and(\n            keep,\n            _comparison_to_function[inc_param_dict[metric][0]](\n                units[metric], inc_param_dict[metric][1]\n            ),\n        )\n\n    # now exclude by label if it is specified\n    if len(exclude_labels):\n        for unit_ind in np.ravel(np.argwhere(keep)):\n            labels = units[unit_ind][\"label\"].split(\",\")\n            for label in labels:\n                if label in exclude_labels:\n                    keep[unit_ind] = False\n                    break\n\n    # return units that passed all of the tests\n    # TODO: Make this more efficient\n    return {i: units_key[i] for i in np.ravel(np.argwhere(keep))}\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_populator/", "title": "spikesorting_populator.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_populator/#src.spyglass.spikesorting.v0.spikesorting_populator.spikesorting_pipeline_populator", "title": "<code>spikesorting_pipeline_populator(nwb_file_name, team_name, fig_url_repo=None, interval_list_name=None, sort_interval_name=None, pipeline_parameters_name=None, probe_restriction={}, artifact_parameters='ampl_2000_prop_75', preproc_params_name='franklab_tetrode_hippocampus', sorter='mountainsort4', sorter_params_name='franklab_tetrode_hippocampus_30KHz_tmp', waveform_params_name='default_whitened', metric_params_name='peak_offest_num_spikes_2', auto_curation_params_name='mike_noise_03_offset_2_isi_0025_mua')</code>", "text": "<p>Automatically populate the spike sorting pipeline for a given epoch</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>Session ID</p> required <code>team_name</code> <code>str</code> <p>Which team to assign the spike sorting to</p> required <code>fig_url_repo</code> <code>str</code> <p>Where to store the curation figurl json files (e.g., 'gh://LorenFrankLab/sorting-curations/main/user/'). Default None to skip figurl</p> <code>None</code> <code>interval_list_name</code> <code>(str)</code> <p>if sort_interval_name not provided, will create a sort interval for the given interval with the same name</p> <code>None</code> <code>sort_interval_name</code> <code>str</code> <p>if provided, will use the given sort interval, requires making this interval yourself</p> <code>None</code> <code>pipeline_parameters_name</code> <code>str</code> <p>If provided, will lookup pipeline parameters from the SpikeSortingPipelineParameters table, supersedes other values provided, by default None</p> <code>None</code> <code>probe_restriction</code> <code>dict</code> <p>Restricts analysis to sort groups with matching keys. Can use keys from the SortGroup and ElectrodeGroup Tables (e.g. electrode_group_name, probe_id, target_hemisphere), by default {}</p> <code>{}</code> <code>artifact_parameters</code> <code>str</code> <p>parameter set for artifact detection, by default \"ampl_2000_prop_75\"</p> <code>'ampl_2000_prop_75'</code> <code>preproc_params_name</code> <code>str</code> <p>parameter set for spikesorting recording, by default \"franklab_tetrode_hippocampus\"</p> <code>'franklab_tetrode_hippocampus'</code> <code>sorter</code> <code>str</code> <p>which spikesorting algorithm to use, by default \"mountainsort4\"</p> <code>'mountainsort4'</code> <code>sorter_params_name</code> <code>str</code> <p>parameters for the spike sorting algorithm, by default \"franklab_tetrode_hippocampus_30KHz_tmp\"</p> <code>'franklab_tetrode_hippocampus_30KHz_tmp'</code> <code>waveform_params_name</code> <code>str</code> <p>Parameters for spike waveform extraction. If empty string, will skip automatic curation steps, by default \"default_whitened\"</p> <code>'default_whitened'</code> <code>metric_params_name</code> <code>str</code> <p>Parameters defining which QualityMetrics to calculate and how. If empty string, will skip automatic curation steps, by default \"peak_offest_num_spikes_2\"</p> <code>'peak_offest_num_spikes_2'</code> <code>auto_curation_params_name</code> <code>str</code> <p>Thresholds applied to Quality metrics for automatic unit curation. If empty string, will skip automatic curation steps, by default \"mike_noise_03_offset_2_isi_0025_mua\"</p> <code>'mike_noise_03_offset_2_isi_0025_mua'</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_populator.py</code> <pre><code>def spikesorting_pipeline_populator(\n    nwb_file_name: str,\n    team_name: str,\n    fig_url_repo: str = None,\n    interval_list_name: str = None,\n    sort_interval_name: str = None,\n    pipeline_parameters_name: str = None,\n    probe_restriction: dict = {},\n    artifact_parameters: str = \"ampl_2000_prop_75\",\n    preproc_params_name: str = \"franklab_tetrode_hippocampus\",\n    sorter: str = \"mountainsort4\",\n    sorter_params_name: str = \"franklab_tetrode_hippocampus_30KHz_tmp\",\n    waveform_params_name: str = \"default_whitened\",\n    metric_params_name: str = \"peak_offest_num_spikes_2\",\n    auto_curation_params_name: str = \"mike_noise_03_offset_2_isi_0025_mua\",\n):\n    \"\"\"Automatically populate the spike sorting pipeline for a given epoch\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        Session ID\n    team_name : str\n        Which team to assign the spike sorting to\n    fig_url_repo : str, optional\n        Where to store the curation figurl json files (e.g.,\n        'gh://LorenFrankLab/sorting-curations/main/user/'). Default None to\n        skip figurl\n    interval_list_name : str,\n        if sort_interval_name not provided, will create a sort interval for the\n        given interval with the same name\n    sort_interval_name : str, default None\n        if provided, will use the given sort interval, requires making this\n        interval yourself\n    pipeline_parameters_name : str, optional\n        If provided, will lookup pipeline parameters from the\n        SpikeSortingPipelineParameters table, supersedes other values provided,\n        by default None\n    probe_restriction : dict, optional\n        Restricts analysis to sort groups with matching keys. Can use keys from\n        the SortGroup and ElectrodeGroup Tables (e.g. electrode_group_name,\n        probe_id, target_hemisphere), by default {}\n    artifact_parameters : str, optional\n        parameter set for artifact detection, by default \"ampl_2000_prop_75\"\n    preproc_params_name : str, optional\n        parameter set for spikesorting recording, by default\n        \"franklab_tetrode_hippocampus\"\n    sorter : str, optional\n        which spikesorting algorithm to use, by default \"mountainsort4\"\n    sorter_params_name : str, optional\n        parameters for the spike sorting algorithm, by default\n        \"franklab_tetrode_hippocampus_30KHz_tmp\"\n    waveform_params_name : str, optional\n        Parameters for spike waveform extraction. If empty string, will skip\n        automatic curation steps, by default \"default_whitened\"\n    metric_params_name : str, optional\n        Parameters defining which QualityMetrics to calculate and how. If empty\n        string, will skip automatic curation steps, by default\n        \"peak_offest_num_spikes_2\"\n    auto_curation_params_name : str, optional\n        Thresholds applied to Quality metrics for automatic unit curation. If\n        empty string, will skip automatic curation steps, by default\n        \"mike_noise_03_offset_2_isi_0025_mua\"\n    \"\"\"\n    nwbf_dict = dict(nwb_file_name=nwb_file_name)\n    # Define pipeline parameters\n    if pipeline_parameters_name is not None:\n        logger.info(f\"Using pipeline parameters {pipeline_parameters_name}\")\n        (\n            artifact_parameters,\n            preproc_params_name,\n            sorter,\n            sorter_params_name,\n            waveform_params_name,\n            metric_params_name,\n            auto_curation_params_name,\n        ) = (\n            SpikeSortingPipelineParameters\n            &amp; {\"pipeline_parameters_name\": pipeline_parameters_name}\n        ).fetch1(\n            \"artifact_parameters\",\n            \"preproc_params_name\",\n            \"sorter\",\n            \"sorter_params_name\",\n            \"waveform_params_name\",\n            \"metric_params_name\",\n            \"auto_curation_params_name\",\n        )\n\n    # make sort groups only if not currently available\n    # don't overwrite existing ones!\n    if not SortGroup() &amp; nwbf_dict:\n        logger.info(\"Generating sort groups\")\n        SortGroup().set_group_by_shank(nwb_file_name)\n\n    # Define sort interval\n    interval_dict = dict(**nwbf_dict, interval_list_name=interval_list_name)\n\n    if sort_interval_name is not None:\n        logger.info(f\"Using sort interval {sort_interval_name}\")\n        if not (\n            SortInterval\n            &amp; nwbf_dict\n            &amp; {\"sort_interval_name\": sort_interval_name}\n        ):\n            raise KeyError(f\"Sort interval {sort_interval_name} not found\")\n    else:\n        logger.info(f\"Generating sort interval from {interval_list_name}\")\n        interval_list = (IntervalList &amp; interval_dict).fetch1(\"valid_times\")[0]\n\n        sort_interval_name = interval_list_name\n        sort_interval = interval_list\n\n        SortInterval.insert1(\n            {\n                **nwbf_dict,\n                \"sort_interval_name\": sort_interval_name,\n                \"sort_interval\": sort_interval,\n            },\n            skip_duplicates=True,\n        )\n\n    sort_dict = dict(**nwbf_dict, sort_interval_name=sort_interval_name)\n\n    # find desired sort group(s) for these settings\n    sort_group_id_list = (\n        (SortGroup.SortGroupElectrode * ElectrodeGroup)\n        &amp; nwbf_dict\n        &amp; probe_restriction\n    ).fetch(\"sort_group_id\")\n\n    # make spike sorting recording\n    logger.info(\"Generating spike sorting recording\")\n    for sort_group_id in sort_group_id_list:\n        ssr_key = dict(\n            **sort_dict,\n            sort_group_id=sort_group_id,  # See SortGroup\n            preproc_params_name=preproc_params_name,  # See preproc_params\n            interval_list_name=interval_list_name,\n            team_name=team_name,\n        )\n        SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True)\n\n    SpikeSortingRecording.populate(interval_dict)\n\n    # Artifact detection\n    logger.info(\"Running artifact detection\")\n    artifact_keys = [\n        {**k, \"artifact_params_name\": artifact_parameters}\n        for k in (SpikeSortingRecordingSelection() &amp; interval_dict).fetch(\"KEY\")\n    ]\n    ArtifactDetectionSelection().insert(artifact_keys, skip_duplicates=True)\n    ArtifactDetection.populate(interval_dict)\n\n    # Spike sorting\n    logger.info(\"Running spike sorting\")\n    for artifact_key in artifact_keys:\n        ss_key = dict(\n            **(ArtifactDetection &amp; artifact_key).fetch1(\"KEY\"),\n            **(ArtifactRemovedIntervalList() &amp; artifact_key).fetch1(\"KEY\"),\n            sorter=sorter,\n            sorter_params_name=sorter_params_name,\n        )\n        ss_key.pop(\"artifact_params_name\")\n        SpikeSortingSelection.insert1(ss_key, skip_duplicates=True)\n    SpikeSorting.populate(sort_dict)\n\n    # initial curation\n    logger.info(\"Beginning curation\")\n    for sorting_key in (SpikeSorting() &amp; sort_dict).fetch(\"KEY\"):\n        if not (Curation() &amp; sorting_key):\n            Curation.insert_curation(sorting_key)\n\n    # Calculate quality metrics and perform automatic curation if specified\n    if (\n        len(waveform_params_name) &gt; 0\n        and len(metric_params_name) &gt; 0\n        and len(auto_curation_params_name) &gt; 0\n    ):\n        # Extract waveforms\n        logger.info(\"Extracting waveforms\")\n        curation_keys = [\n            {**k, \"waveform_params_name\": waveform_params_name}\n            for k in (Curation() &amp; sort_dict &amp; {\"curation_id\": 0}).fetch(\"KEY\")\n        ]\n        WaveformSelection.insert(curation_keys, skip_duplicates=True)\n        Waveforms.populate(sort_dict)\n\n        # Quality Metrics\n        logger.info(\"Calculating quality metrics\")\n        waveform_keys = [\n            {**k, \"metric_params_name\": metric_params_name}\n            for k in (Waveforms() &amp; sort_dict).fetch(\"KEY\")\n        ]\n        MetricSelection.insert(waveform_keys, skip_duplicates=True)\n        QualityMetrics().populate(sort_dict)\n\n        # Automatic Curation\n        logger.info(\"Creating automatic curation\")\n        metric_keys = [\n            {**k, \"auto_curation_params_name\": auto_curation_params_name}\n            for k in (QualityMetrics() &amp; sort_dict).fetch(\"KEY\")\n        ]\n        AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True)\n        AutomaticCuration().populate(sort_dict)\n\n        # Curated Spike Sorting\n        # get curation keys of the automatic curation to populate into curated\n        # spike sorting selection\n        logger.info(\"Creating curated spike sorting\")\n        auto_key_list = (AutomaticCuration() &amp; sort_dict).fetch(\n            \"auto_curation_key\"\n        )\n        for auto_key in auto_key_list:\n            curation_auto_key = (Curation() &amp; auto_key).fetch1(\"KEY\")\n            CuratedSpikeSortingSelection.insert1(\n                curation_auto_key, skip_duplicates=True\n            )\n\n    else:\n        # Perform no automatic curation, just populate curated spike sorting\n        # selection with the initial curation. Used in case of clusterless\n        # decoding\n        logger.info(\"Creating curated spike sorting\")\n        curation_keys = (Curation() &amp; sort_dict).fetch(\"KEY\")\n        for curation_key in curation_keys:\n            CuratedSpikeSortingSelection.insert1(\n                curation_key, skip_duplicates=True\n            )\n\n    # Populate curated spike sorting\n    CuratedSpikeSorting.populate(sort_dict)\n\n    if fig_url_repo:\n        # Curation Figurl\n        logger.info(\"Creating curation figurl\")\n        sort_interval_name = interval_list_name + \"_entire\"\n        gh_url = (\n            fig_url_repo\n            + str(nwb_file_name + \"_\" + sort_interval_name)  # session id\n            + \"/{}\"  # tetrode using auto_id['sort_group_id']\n            + \"/curation.json\"\n        )\n\n        for auto_id in (AutomaticCuration() &amp; sort_dict).fetch(\n            \"auto_curation_key\"\n        ):\n            auto_curation_out_key = dict(\n                **(Curation() &amp; auto_id).fetch1(\"KEY\"),\n                new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),\n            )\n            CurationFigurlSelection.insert1(\n                auto_curation_out_key, skip_duplicates=True\n            )\n            CurationFigurl.populate(auto_curation_out_key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/", "title": "spikesorting_recording.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup", "title": "<code>SortGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>@schema\nclass SortGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Set of electrodes that will be sorted together\n    -&gt; Session\n    sort_group_id: int  # identifier for a group of electrodes\n    ---\n    sort_reference_electrode_id = -1: int  # the electrode to use for reference. -1: no reference, -2: common median\n    \"\"\"\n\n    class SortGroupElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; SortGroup\n        -&gt; Electrode\n        \"\"\"\n\n    def set_group_by_shank(\n        self,\n        nwb_file_name: str,\n        references: dict = None,\n        omit_ref_electrode_group=False,\n        omit_unitrode=True,\n    ):\n        \"\"\"Divides electrodes into groups based on their shank position.\n\n        * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n          single group\n        * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n          placed in one group per shank\n        * Bad channels are omitted\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            the name of the NWB file whose electrodes should be put into\n            sorting groups\n        references : dict, optional\n            If passed, used to set references. Otherwise, references set using\n            original reference electrodes from config. Keys: electrode groups.\n            Values: reference electrode.\n        omit_ref_electrode_group : bool\n            Optional. If True, no sort group is defined for electrode group of\n            reference.\n        omit_unitrode : bool\n            Optional. If True, no sort groups are defined for unitrodes.\n        \"\"\"\n        # delete any current groups\n        (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n        e_groups.sort(key=int)  # sort electrode groups numerically\n        sort_group = 0\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        for e_group in e_groups:\n            # for each electrode group, get a list of the unique shank numbers\n            shank_list = np.unique(\n                electrodes[\"probe_shank\"][\n                    electrodes[\"electrode_group_name\"] == e_group\n                ]\n            )\n            sge_key[\"electrode_group_name\"] = e_group\n            # get the indices of all electrodes in this group / shank and set their sorting group\n            for shank in shank_list:\n                sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n                # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n                if not references:\n                    shank_elect_ref = electrodes[\n                        \"original_reference_electrode\"\n                    ][\n                        np.logical_and(\n                            electrodes[\"electrode_group_name\"] == e_group,\n                            electrodes[\"probe_shank\"] == shank,\n                        )\n                    ]\n                    if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                        sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                            0\n                        ]\n                    else:\n                        ValueError(\n                            f\"Error in electrode group {e_group}: reference \"\n                            + \"electrodes are not all the same\"\n                        )\n                else:\n                    if e_group not in references.keys():\n                        raise Exception(\n                            f\"electrode group {e_group} not a key in \"\n                            + \"references, so cannot set reference\"\n                        )\n                    else:\n                        sg_key[\"sort_reference_electrode_id\"] = references[\n                            e_group\n                        ]\n                # Insert sort group and sort group electrodes\n                reference_electrode_group = electrodes[\n                    electrodes[\"electrode_id\"]\n                    == sg_key[\"sort_reference_electrode_id\"]\n                ][\n                    \"electrode_group_name\"\n                ]  # reference for this electrode group\n                if (\n                    len(reference_electrode_group) == 1\n                ):  # unpack single reference\n                    reference_electrode_group = reference_electrode_group[0]\n                elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                    len(reference_electrode_group) != 1\n                ):\n                    raise Exception(\n                        \"Should have found exactly one electrode group for \"\n                        + \"reference electrode, but found \"\n                        + f\"{len(reference_electrode_group)}.\"\n                    )\n                if omit_ref_electrode_group and (\n                    str(e_group) == str(reference_electrode_group)\n                ):\n                    logger.warn(\n                        f\"Omitting electrode group {e_group} from sort groups \"\n                        + \"because contains reference.\"\n                    )\n                    continue\n                shank_elect = electrodes[\"electrode_id\"][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if (\n                    omit_unitrode and len(shank_elect) == 1\n                ):  # omit unitrodes if indicated\n                    logger.warn(\n                        f\"Omitting electrode group {e_group}, shank {shank} \"\n                        + \"from sort groups because unitrode.\"\n                    )\n                    continue\n                self.insert1(sg_key)\n                for elect in shank_elect:\n                    sge_key[\"electrode_id\"] = elect\n                    self.SortGroupElectrode().insert1(sge_key)\n                sort_group += 1\n\n    def set_group_by_electrode_group(self, nwb_file_name: str):\n        \"\"\"Assign groups to all non-bad channel electrodes based on their electrode group\n        and sets the reference for each group to the reference for the first channel of the group.\n\n        Parameters\n        ----------\n        nwb_file_name: str\n            the name of the nwb whose electrodes should be put into sorting groups\n        \"\"\"\n        # delete any current groups\n        (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = np.unique(electrodes[\"electrode_group_name\"])\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        sort_group = 0\n        for e_group in e_groups:\n            sge_key[\"electrode_group_name\"] = e_group\n            # sg_key['sort_group_id'] = sge_key['sort_group_id'] = sort_group\n            # TEST\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = int(e_group)\n            # get the list of references and make sure they are all the same\n            shank_elect_ref = electrodes[\"original_reference_electrode\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n            if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[0]\n            else:\n                ValueError(\n                    f\"Error in electrode group {e_group}: reference electrodes are not all the same\"\n                )\n            self.insert1(sg_key)\n\n            shank_elect = electrodes[\"electrode_id\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                self.SortGroupElectrode().insert1(sge_key)\n            sort_group += 1\n\n    def set_reference_from_list(self, nwb_file_name, sort_group_ref_list):\n        \"\"\"\n        Set the reference electrode from a list containing sort groups and reference electrodes\n        :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode]\n        :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated\n        :return: Null\n        \"\"\"\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        sort_group_list = (SortGroup() &amp; key).fetch1()\n        for sort_group in sort_group_list:\n            key[\"sort_group_id\"] = sort_group\n            self.insert(\n                dj_replace(\n                    sort_group_list,\n                    sort_group_ref_list,\n                    \"sort_group_id\",\n                    \"sort_reference_electrode_id\",\n                ),\n                replace=\"True\",\n            )\n\n    def get_geometry(self, sort_group_id, nwb_file_name):\n        \"\"\"\n        Returns a list with the x,y coordinates of the electrodes in the sort group\n        for use with the SpikeInterface package.\n\n        Converts z locations to y where appropriate.\n\n        Parameters\n        ----------\n        sort_group_id : int\n        nwb_file_name : str\n\n        Returns\n        -------\n        geometry : list\n            List of coordinate pairs, one per electrode\n        \"\"\"\n\n        # create the channel_groups dictiorary\n        channel_group = dict()\n        key = dict()\n        key[\"nwb_file_name\"] = nwb_file_name\n        electrodes = (Electrode() &amp; key).fetch()\n\n        key[\"sort_group_id\"] = sort_group_id\n        sort_group_electrodes = (SortGroup.SortGroupElectrode() &amp; key).fetch()\n        electrode_group_name = sort_group_electrodes[\"electrode_group_name\"][0]\n        probe_id = (\n            ElectrodeGroup\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"electrode_group_name\": electrode_group_name,\n            }\n        ).fetch1(\"probe_id\")\n        channel_group[sort_group_id] = dict()\n        channel_group[sort_group_id][\"channels\"] = sort_group_electrodes[\n            \"electrode_id\"\n        ].tolist()\n\n        n_chan = len(channel_group[sort_group_id][\"channels\"])\n\n        geometry = np.zeros((n_chan, 2), dtype=\"float\")\n        tmp_geom = np.zeros((n_chan, 3), dtype=\"float\")\n        for i, electrode_id in enumerate(\n            channel_group[sort_group_id][\"channels\"]\n        ):\n            # get the relative x and y locations of this channel from the probe table\n            probe_electrode = int(\n                electrodes[\"probe_electrode\"][\n                    electrodes[\"electrode_id\"] == electrode_id\n                ]\n            )\n            rel_x, rel_y, rel_z = (\n                Probe().Electrode()\n                &amp; {\"probe_id\": probe_id, \"probe_electrode\": probe_electrode}\n            ).fetch(\"rel_x\", \"rel_y\", \"rel_z\")\n            # TODO: Fix this HACK when we can use probeinterface:\n            rel_x = float(rel_x)\n            rel_y = float(rel_y)\n            rel_z = float(rel_z)\n            tmp_geom[i, :] = [rel_x, rel_y, rel_z]\n\n        # figure out which columns have coordinates\n        n_found = 0\n        for i in range(3):\n            if np.any(np.nonzero(tmp_geom[:, i])):\n                if n_found &lt; 2:\n                    geometry[:, n_found] = tmp_geom[:, i]\n                    n_found += 1\n                else:\n                    Warning(\n                        \"Relative electrode locations have three coordinates; only two are currently supported\"\n                    )\n        return np.ndarray.tolist(geometry)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_group_by_shank", "title": "<code>set_group_by_shank(nwb_file_name, references=None, omit_ref_electrode_group=False, omit_unitrode=True)</code>", "text": "<p>Divides electrodes into groups based on their shank position.</p> <ul> <li>Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a   single group</li> <li>Electrodes from probes with multiple shanks (e.g. polymer probes) are   placed in one group per shank</li> <li>Bad channels are omitted</li> </ul> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the NWB file whose electrodes should be put into sorting groups</p> required <code>references</code> <code>dict</code> <p>If passed, used to set references. Otherwise, references set using original reference electrodes from config. Keys: electrode groups. Values: reference electrode.</p> <code>None</code> <code>omit_ref_electrode_group</code> <code>bool</code> <p>Optional. If True, no sort group is defined for electrode group of reference.</p> <code>False</code> <code>omit_unitrode</code> <code>bool</code> <p>Optional. If True, no sort groups are defined for unitrodes.</p> <code>True</code> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_group_by_shank(\n    self,\n    nwb_file_name: str,\n    references: dict = None,\n    omit_ref_electrode_group=False,\n    omit_unitrode=True,\n):\n    \"\"\"Divides electrodes into groups based on their shank position.\n\n    * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n      single group\n    * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n      placed in one group per shank\n    * Bad channels are omitted\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        the name of the NWB file whose electrodes should be put into\n        sorting groups\n    references : dict, optional\n        If passed, used to set references. Otherwise, references set using\n        original reference electrodes from config. Keys: electrode groups.\n        Values: reference electrode.\n    omit_ref_electrode_group : bool\n        Optional. If True, no sort group is defined for electrode group of\n        reference.\n    omit_unitrode : bool\n        Optional. If True, no sort groups are defined for unitrodes.\n    \"\"\"\n    # delete any current groups\n    (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n    e_groups.sort(key=int)  # sort electrode groups numerically\n    sort_group = 0\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    for e_group in e_groups:\n        # for each electrode group, get a list of the unique shank numbers\n        shank_list = np.unique(\n            electrodes[\"probe_shank\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n        )\n        sge_key[\"electrode_group_name\"] = e_group\n        # get the indices of all electrodes in this group / shank and set their sorting group\n        for shank in shank_list:\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n            # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n            if not references:\n                shank_elect_ref = electrodes[\n                    \"original_reference_electrode\"\n                ][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                    sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                        0\n                    ]\n                else:\n                    ValueError(\n                        f\"Error in electrode group {e_group}: reference \"\n                        + \"electrodes are not all the same\"\n                    )\n            else:\n                if e_group not in references.keys():\n                    raise Exception(\n                        f\"electrode group {e_group} not a key in \"\n                        + \"references, so cannot set reference\"\n                    )\n                else:\n                    sg_key[\"sort_reference_electrode_id\"] = references[\n                        e_group\n                    ]\n            # Insert sort group and sort group electrodes\n            reference_electrode_group = electrodes[\n                electrodes[\"electrode_id\"]\n                == sg_key[\"sort_reference_electrode_id\"]\n            ][\n                \"electrode_group_name\"\n            ]  # reference for this electrode group\n            if (\n                len(reference_electrode_group) == 1\n            ):  # unpack single reference\n                reference_electrode_group = reference_electrode_group[0]\n            elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                len(reference_electrode_group) != 1\n            ):\n                raise Exception(\n                    \"Should have found exactly one electrode group for \"\n                    + \"reference electrode, but found \"\n                    + f\"{len(reference_electrode_group)}.\"\n                )\n            if omit_ref_electrode_group and (\n                str(e_group) == str(reference_electrode_group)\n            ):\n                logger.warn(\n                    f\"Omitting electrode group {e_group} from sort groups \"\n                    + \"because contains reference.\"\n                )\n                continue\n            shank_elect = electrodes[\"electrode_id\"][\n                np.logical_and(\n                    electrodes[\"electrode_group_name\"] == e_group,\n                    electrodes[\"probe_shank\"] == shank,\n                )\n            ]\n            if (\n                omit_unitrode and len(shank_elect) == 1\n            ):  # omit unitrodes if indicated\n                logger.warn(\n                    f\"Omitting electrode group {e_group}, shank {shank} \"\n                    + \"from sort groups because unitrode.\"\n                )\n                continue\n            self.insert1(sg_key)\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                self.SortGroupElectrode().insert1(sge_key)\n            sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_group_by_electrode_group", "title": "<code>set_group_by_electrode_group(nwb_file_name)</code>", "text": "<p>Assign groups to all non-bad channel electrodes based on their electrode group and sets the reference for each group to the reference for the first channel of the group.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the nwb whose electrodes should be put into sorting groups</p> required Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_group_by_electrode_group(self, nwb_file_name: str):\n    \"\"\"Assign groups to all non-bad channel electrodes based on their electrode group\n    and sets the reference for each group to the reference for the first channel of the group.\n\n    Parameters\n    ----------\n    nwb_file_name: str\n        the name of the nwb whose electrodes should be put into sorting groups\n    \"\"\"\n    # delete any current groups\n    (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = np.unique(electrodes[\"electrode_group_name\"])\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    sort_group = 0\n    for e_group in e_groups:\n        sge_key[\"electrode_group_name\"] = e_group\n        # sg_key['sort_group_id'] = sge_key['sort_group_id'] = sort_group\n        # TEST\n        sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = int(e_group)\n        # get the list of references and make sure they are all the same\n        shank_elect_ref = electrodes[\"original_reference_electrode\"][\n            electrodes[\"electrode_group_name\"] == e_group\n        ]\n        if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n            sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[0]\n        else:\n            ValueError(\n                f\"Error in electrode group {e_group}: reference electrodes are not all the same\"\n            )\n        self.insert1(sg_key)\n\n        shank_elect = electrodes[\"electrode_id\"][\n            electrodes[\"electrode_group_name\"] == e_group\n        ]\n        for elect in shank_elect:\n            sge_key[\"electrode_id\"] = elect\n            self.SortGroupElectrode().insert1(sge_key)\n        sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.set_reference_from_list", "title": "<code>set_reference_from_list(nwb_file_name, sort_group_ref_list)</code>", "text": "<p>Set the reference electrode from a list containing sort groups and reference electrodes :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode] :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated :return: Null</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def set_reference_from_list(self, nwb_file_name, sort_group_ref_list):\n    \"\"\"\n    Set the reference electrode from a list containing sort groups and reference electrodes\n    :param: sort_group_ref_list - 2D array or list where each row is [sort_group_id reference_electrode]\n    :param: nwb_file_name - The name of the NWB file whose electrodes' references should be updated\n    :return: Null\n    \"\"\"\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    sort_group_list = (SortGroup() &amp; key).fetch1()\n    for sort_group in sort_group_list:\n        key[\"sort_group_id\"] = sort_group\n        self.insert(\n            dj_replace(\n                sort_group_list,\n                sort_group_ref_list,\n                \"sort_group_id\",\n                \"sort_reference_electrode_id\",\n            ),\n            replace=\"True\",\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SortGroup.get_geometry", "title": "<code>get_geometry(sort_group_id, nwb_file_name)</code>", "text": "<p>Returns a list with the x,y coordinates of the electrodes in the sort group for use with the SpikeInterface package.</p> <p>Converts z locations to y where appropriate.</p> <p>Parameters:</p> Name Type Description Default <code>sort_group_id</code> <code>int</code> required <code>nwb_file_name</code> <code>str</code> required <p>Returns:</p> Name Type Description <code>geometry</code> <code>list</code> <p>List of coordinate pairs, one per electrode</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>def get_geometry(self, sort_group_id, nwb_file_name):\n    \"\"\"\n    Returns a list with the x,y coordinates of the electrodes in the sort group\n    for use with the SpikeInterface package.\n\n    Converts z locations to y where appropriate.\n\n    Parameters\n    ----------\n    sort_group_id : int\n    nwb_file_name : str\n\n    Returns\n    -------\n    geometry : list\n        List of coordinate pairs, one per electrode\n    \"\"\"\n\n    # create the channel_groups dictiorary\n    channel_group = dict()\n    key = dict()\n    key[\"nwb_file_name\"] = nwb_file_name\n    electrodes = (Electrode() &amp; key).fetch()\n\n    key[\"sort_group_id\"] = sort_group_id\n    sort_group_electrodes = (SortGroup.SortGroupElectrode() &amp; key).fetch()\n    electrode_group_name = sort_group_electrodes[\"electrode_group_name\"][0]\n    probe_id = (\n        ElectrodeGroup\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"electrode_group_name\": electrode_group_name,\n        }\n    ).fetch1(\"probe_id\")\n    channel_group[sort_group_id] = dict()\n    channel_group[sort_group_id][\"channels\"] = sort_group_electrodes[\n        \"electrode_id\"\n    ].tolist()\n\n    n_chan = len(channel_group[sort_group_id][\"channels\"])\n\n    geometry = np.zeros((n_chan, 2), dtype=\"float\")\n    tmp_geom = np.zeros((n_chan, 3), dtype=\"float\")\n    for i, electrode_id in enumerate(\n        channel_group[sort_group_id][\"channels\"]\n    ):\n        # get the relative x and y locations of this channel from the probe table\n        probe_electrode = int(\n            electrodes[\"probe_electrode\"][\n                electrodes[\"electrode_id\"] == electrode_id\n            ]\n        )\n        rel_x, rel_y, rel_z = (\n            Probe().Electrode()\n            &amp; {\"probe_id\": probe_id, \"probe_electrode\": probe_electrode}\n        ).fetch(\"rel_x\", \"rel_y\", \"rel_z\")\n        # TODO: Fix this HACK when we can use probeinterface:\n        rel_x = float(rel_x)\n        rel_y = float(rel_y)\n        rel_z = float(rel_z)\n        tmp_geom[i, :] = [rel_x, rel_y, rel_z]\n\n    # figure out which columns have coordinates\n    n_found = 0\n    for i in range(3):\n        if np.any(np.nonzero(tmp_geom[:, i])):\n            if n_found &lt; 2:\n                geometry[:, n_found] = tmp_geom[:, i]\n                n_found += 1\n            else:\n                Warning(\n                    \"Relative electrode locations have three coordinates; only two are currently supported\"\n                )\n    return np.ndarray.tolist(geometry)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_recording/#src.spyglass.spikesorting.v0.spikesorting_recording.SpikeSortingRecording", "title": "<code>SpikeSortingRecording</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_recording.py</code> <pre><code>@schema\nclass SpikeSortingRecording(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingRecordingSelection\n    ---\n    recording_path: varchar(1000)\n    -&gt; IntervalList.proj(sort_interval_list_name='interval_list_name')\n    \"\"\"\n\n    def make(self, key):\n        sort_interval_valid_times = self._get_sort_interval_valid_times(key)\n        recording = self._get_filtered_recording(key)\n        recording_name = self._get_recording_name(key)\n\n        # Path to files that will hold the recording extractors\n        recording_path = str(recording_dir / Path(recording_name))\n        if os.path.exists(recording_path):\n            shutil.rmtree(recording_path)\n\n        recording.save(\n            folder=recording_path, chunk_duration=\"10000ms\", n_jobs=8\n        )\n\n        IntervalList.insert1(\n            {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": recording_name,\n                \"valid_times\": sort_interval_valid_times,\n                \"pipeline\": \"spikesorting_recording_v0\",\n            },\n            replace=True,\n        )\n\n        self.insert1(\n            {\n                **key,\n                # store the list of valid times for the sort\n                \"sort_interval_list_name\": recording_name,\n                \"recording_path\": recording_path,\n            }\n        )\n\n    @staticmethod\n    def _get_recording_name(key):\n        return \"_\".join(\n            [\n                key[\"nwb_file_name\"],\n                key[\"sort_interval_name\"],\n                str(key[\"sort_group_id\"]),\n                key[\"preproc_params_name\"],\n            ]\n        )\n\n    @staticmethod\n    def _get_recording_timestamps(recording):\n        num_segments = recording.get_num_segments()\n\n        if num_segments &lt;= 1:\n            return recording.get_times()\n\n        frames_per_segment = [0] + [\n            recording.get_num_frames(segment_index=i)\n            for i in range(num_segments)\n        ]\n\n        cumsum_frames = np.cumsum(frames_per_segment)\n        total_frames = np.sum(frames_per_segment)\n\n        timestamps = np.zeros((total_frames,))\n        for i in range(num_segments):\n            start_index = cumsum_frames[i]\n            end_index = cumsum_frames[i + 1]\n            timestamps[start_index:end_index] = recording.get_times(\n                segment_index=i\n            )\n\n        return timestamps\n\n    def _get_sort_interval_valid_times(self, key):\n        \"\"\"Identifies the intersection between sort interval specified by the user\n        and the valid times (times for which neural data exist)\n\n        Parameters\n        ----------\n        key: dict\n            specifies a (partially filled) entry of SpikeSorting table\n\n        Returns\n        -------\n        sort_interval_valid_times: ndarray of tuples\n            (start, end) times for valid stretches of the sorting interval\n\n        \"\"\"\n        sort_interval = (\n            SortInterval\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_interval_name\": key[\"sort_interval_name\"],\n            }\n        ).fetch1(\"sort_interval\")\n\n        interval_list_name = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"interval_list_name\"\n        )\n\n        valid_interval_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"interval_list_name\": interval_list_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        valid_sort_times = interval_list_intersect(\n            sort_interval, valid_interval_times\n        )\n        # Exclude intervals shorter than specified length\n        params = (SpikeSortingPreprocessingParameters &amp; key).fetch1(\n            \"preproc_params\"\n        )\n        if \"min_segment_length\" in params:\n            valid_sort_times = intervals_by_length(\n                valid_sort_times, min_length=params[\"min_segment_length\"]\n            )\n        return valid_sort_times\n\n    def _get_filtered_recording(self, key: dict):\n        \"\"\"Filters and references a recording\n        * Loads the NWB file created during insertion as a spikeinterface Recording\n        * Slices recording in time (interval) and space (channels);\n          recording chunks from disjoint intervals are concatenated\n        * Applies referencing and bandpass filtering\n\n        Parameters\n        ----------\n        key: dict,\n            primary key of SpikeSortingRecording table\n\n        Returns\n        -------\n        recording: si.Recording\n        \"\"\"\n\n        nwb_file_abs_path = Nwbfile().get_abs_path(key[\"nwb_file_name\"])\n        recording = se.read_nwb_recording(\n            nwb_file_abs_path, load_time_vector=True\n        )\n\n        valid_sort_times = self._get_sort_interval_valid_times(key)\n        # shape is (N, 2)\n        valid_sort_times_indices = np.array(\n            [\n                np.searchsorted(recording.get_times(), interval)\n                for interval in valid_sort_times\n            ]\n        )\n        # join intervals of indices that are adjacent\n        valid_sort_times_indices = reduce(\n            union_adjacent_index, valid_sort_times_indices\n        )\n        if valid_sort_times_indices.ndim == 1:\n            valid_sort_times_indices = np.expand_dims(\n                valid_sort_times_indices, 0\n            )\n\n        # create an AppendRecording if there is more than one disjoint sort interval\n        if len(valid_sort_times_indices) &gt; 1:\n            recordings_list = []\n            for interval_indices in valid_sort_times_indices:\n                recording_single = recording.frame_slice(\n                    start_frame=interval_indices[0],\n                    end_frame=interval_indices[1],\n                )\n                recordings_list.append(recording_single)\n            recording = si.append_recordings(recordings_list)\n        else:\n            recording = recording.frame_slice(\n                start_frame=valid_sort_times_indices[0][0],\n                end_frame=valid_sort_times_indices[0][1],\n            )\n\n        channel_ids = (\n            SortGroup.SortGroupElectrode\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_group_id\": key[\"sort_group_id\"],\n            }\n        ).fetch(\"electrode_id\")\n        ref_channel_id = (\n            SortGroup\n            &amp; {\n                \"nwb_file_name\": key[\"nwb_file_name\"],\n                \"sort_group_id\": key[\"sort_group_id\"],\n            }\n        ).fetch1(\"sort_reference_electrode_id\")\n        channel_ids = np.setdiff1d(channel_ids, ref_channel_id)\n\n        # include ref channel in first slice, then exclude it in second slice\n        if ref_channel_id &gt;= 0:\n            channel_ids_ref = np.append(channel_ids, ref_channel_id)\n            recording = recording.channel_slice(channel_ids=channel_ids_ref)\n\n            recording = si.preprocessing.common_reference(\n                recording, reference=\"single\", ref_channel_ids=ref_channel_id\n            )\n            recording = recording.channel_slice(channel_ids=channel_ids)\n        elif ref_channel_id == -2:\n            recording = recording.channel_slice(channel_ids=channel_ids)\n            recording = si.preprocessing.common_reference(\n                recording, reference=\"global\", operator=\"median\"\n            )\n        else:\n            raise ValueError(\"Invalid reference channel ID\")\n        filter_params = (SpikeSortingPreprocessingParameters &amp; key).fetch1(\n            \"preproc_params\"\n        )\n        recording = si.preprocessing.bandpass_filter(\n            recording,\n            freq_min=filter_params[\"frequency_min\"],\n            freq_max=filter_params[\"frequency_max\"],\n        )\n\n        # if the sort group is a tetrode, change the channel location\n        # note that this is a workaround that would be deprecated when spikeinterface uses 3D probe locations\n        probe_type = []\n        electrode_group = []\n        for channel_id in channel_ids:\n            probe_type.append(\n                (\n                    Electrode * Probe\n                    &amp; {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"probe_type\")\n            )\n            electrode_group.append(\n                (\n                    Electrode\n                    &amp; {\n                        \"nwb_file_name\": key[\"nwb_file_name\"],\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"electrode_group_name\")\n            )\n        if (\n            all(p == \"tetrode_12.5\" for p in probe_type)\n            and len(probe_type) == 4\n            and all(eg == electrode_group[0] for eg in electrode_group)\n        ):\n            tetrode = pi.Probe(ndim=2)\n            position = [[0, 0], [0, 12.5], [12.5, 0], [12.5, 12.5]]\n            tetrode.set_contacts(\n                position, shapes=\"circle\", shape_params={\"radius\": 6.25}\n            )\n            tetrode.set_contact_ids(channel_ids)\n            tetrode.set_device_channel_indices(np.arange(4))\n            recording = recording.set_probe(tetrode, in_place=True)\n\n        return recording\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/", "title": "spikesorting_sorting.py", "text": ""}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorterParameters", "title": "<code>SpikeSorterParameters</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>@schema\nclass SpikeSorterParameters(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    sorter: varchar(32)\n    sorter_params_name: varchar(64)\n    ---\n    sorter_params: blob\n    \"\"\"\n\n    # NOTE: See #630, #664. Excessive key length.\n\n    def insert_default(self):\n        \"\"\"Default params from spike sorters available via spikeinterface\"\"\"\n        sorters = sis.available_sorters()\n        for sorter in sorters:\n            sorter_params = sis.get_default_sorter_params(sorter)\n            self.insert1(\n                [sorter, \"default\", sorter_params], skip_duplicates=True\n            )\n\n        # Insert Frank lab defaults\n        # Hippocampus tetrode default\n        sorter = \"mountainsort4\"\n        sorter_params_name = \"franklab_tetrode_hippocampus_30KHz\"\n        sorter_params = {\n            \"detect_sign\": -1,\n            \"adjacency_radius\": 100,\n            \"freq_min\": 600,\n            \"freq_max\": 6000,\n            \"filter\": False,\n            \"whiten\": True,\n            \"num_workers\": 1,\n            \"clip_size\": 40,\n            \"detect_threshold\": 3,\n            \"detect_interval\": 10,\n        }\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n\n        # Cortical probe default\n        sorter = \"mountainsort4\"\n        sorter_params_name = \"franklab_probe_ctx_30KHz\"\n        sorter_params = {\n            \"detect_sign\": -1,\n            \"adjacency_radius\": 100,\n            \"freq_min\": 300,\n            \"freq_max\": 6000,\n            \"filter\": False,\n            \"whiten\": True,\n            \"num_workers\": 1,\n            \"clip_size\": 40,\n            \"detect_threshold\": 3,\n            \"detect_interval\": 10,\n        }\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n\n        # clusterless defaults\n        sorter = \"clusterless_thresholder\"\n        sorter_params_name = \"default_clusterless\"\n        sorter_params = dict(\n            detect_threshold=100.0,  # uV\n            # Locally exclusive means one unit per spike detected\n            method=\"locally_exclusive\",\n            peak_sign=\"neg\",\n            exclude_sweep_ms=0.1,\n            local_radius_um=100,\n            # noise levels needs to be 1.0 so the units are in uV and not MAD\n            noise_levels=np.asarray([1.0]),\n            random_chunk_kwargs={},\n            # output needs to be set to sorting for the rest of the pipeline\n            outputs=\"sorting\",\n        )\n        self.insert1(\n            [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n        )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorterParameters.insert_default", "title": "<code>insert_default()</code>", "text": "<p>Default params from spike sorters available via spikeinterface</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def insert_default(self):\n    \"\"\"Default params from spike sorters available via spikeinterface\"\"\"\n    sorters = sis.available_sorters()\n    for sorter in sorters:\n        sorter_params = sis.get_default_sorter_params(sorter)\n        self.insert1(\n            [sorter, \"default\", sorter_params], skip_duplicates=True\n        )\n\n    # Insert Frank lab defaults\n    # Hippocampus tetrode default\n    sorter = \"mountainsort4\"\n    sorter_params_name = \"franklab_tetrode_hippocampus_30KHz\"\n    sorter_params = {\n        \"detect_sign\": -1,\n        \"adjacency_radius\": 100,\n        \"freq_min\": 600,\n        \"freq_max\": 6000,\n        \"filter\": False,\n        \"whiten\": True,\n        \"num_workers\": 1,\n        \"clip_size\": 40,\n        \"detect_threshold\": 3,\n        \"detect_interval\": 10,\n    }\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n\n    # Cortical probe default\n    sorter = \"mountainsort4\"\n    sorter_params_name = \"franklab_probe_ctx_30KHz\"\n    sorter_params = {\n        \"detect_sign\": -1,\n        \"adjacency_radius\": 100,\n        \"freq_min\": 300,\n        \"freq_max\": 6000,\n        \"filter\": False,\n        \"whiten\": True,\n        \"num_workers\": 1,\n        \"clip_size\": 40,\n        \"detect_threshold\": 3,\n        \"detect_interval\": 10,\n    }\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n\n    # clusterless defaults\n    sorter = \"clusterless_thresholder\"\n    sorter_params_name = \"default_clusterless\"\n    sorter_params = dict(\n        detect_threshold=100.0,  # uV\n        # Locally exclusive means one unit per spike detected\n        method=\"locally_exclusive\",\n        peak_sign=\"neg\",\n        exclude_sweep_ms=0.1,\n        local_radius_um=100,\n        # noise levels needs to be 1.0 so the units are in uV and not MAD\n        noise_levels=np.asarray([1.0]),\n        random_chunk_kwargs={},\n        # output needs to be set to sorting for the rest of the pipeline\n        outputs=\"sorting\",\n    )\n    self.insert1(\n        [sorter, sorter_params_name, sorter_params], skip_duplicates=True\n    )\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting", "title": "<code>SpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>@schema\nclass SpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingSelection\n    ---\n    sorting_path: varchar(1000)\n    time_of_sort: int   # in Unix time, to the nearest second\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Runs spike sorting on the data and parameters specified by the\n        SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n\n        Specifically,\n        1. Loads saved recording and runs the sort on it with spikeinterface\n        2. Saves the sorting with spikeinterface\n        3. Creates an analysis NWB file and saves the sorting there\n           (this is redundant with 2; will change in the future)\n\n        \"\"\"\n        # CBroz: does this not work w/o arg? as .populate() ?\n        recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n        recording = si.load_extractor(recording_path)\n\n        # first, get the timestamps\n        timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n        _ = recording.get_sampling_frequency()\n        # then concatenate the recordings\n        # Note: the timestamps are lost upon concatenation,\n        # i.e. concat_recording.get_times() doesn't return true timestamps anymore.\n        # but concat_recording.recoring_list[i].get_times() will return correct\n        # timestamps for ith recording.\n        if recording.get_num_segments() &gt; 1 and isinstance(\n            recording, si.AppendSegmentRecording\n        ):\n            recording = si.concatenate_recordings(recording.recording_list)\n        elif recording.get_num_segments() &gt; 1 and isinstance(\n            recording, si.BinaryRecordingExtractor\n        ):\n            recording = si.concatenate_recordings([recording])\n\n        # load artifact intervals\n        artifact_times = (\n            ArtifactRemovedIntervalList\n            &amp; {\n                \"artifact_removed_interval_list_name\": key[\n                    \"artifact_removed_interval_list_name\"\n                ]\n            }\n        ).fetch1(\"artifact_times\")\n        if len(artifact_times):\n            if artifact_times.ndim == 1:\n                artifact_times = np.expand_dims(artifact_times, 0)\n\n            # convert artifact intervals to indices\n            list_triggers = []\n            for interval in artifact_times:\n                list_triggers.append(\n                    np.arange(\n                        np.searchsorted(timestamps, interval[0]),\n                        np.searchsorted(timestamps, interval[1]),\n                    )\n                )\n            list_triggers = [list(np.concatenate(list_triggers))]\n            recording = sip.remove_artifacts(\n                recording=recording,\n                list_triggers=list_triggers,\n                ms_before=None,\n                ms_after=None,\n                mode=\"zeros\",\n            )\n\n        logger.info(f\"Running spike sorting on {key}...\")\n        sorter, sorter_params = (SpikeSorterParameters &amp; key).fetch1(\n            \"sorter\", \"sorter_params\"\n        )\n\n        sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n        # add tempdir option for mountainsort\n        sorter_params[\"tempdir\"] = sorter_temp_dir.name\n\n        if sorter == \"clusterless_thresholder\":\n            # need to remove tempdir and whiten from sorter_params\n            sorter_params.pop(\"tempdir\", None)\n            sorter_params.pop(\"whiten\", None)\n            sorter_params.pop(\"outputs\", None)\n\n            # Detect peaks for clusterless decoding\n            detected_spikes = detect_peaks(recording, **sorter_params)\n            sorting = si.NumpySorting.from_times_labels(\n                times_list=detected_spikes[\"sample_index\"],\n                labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n                sampling_frequency=recording.get_sampling_frequency(),\n            )\n        else:\n            if \"whiten\" in sorter_params.keys():\n                if sorter_params[\"whiten\"]:\n                    sorter_params[\"whiten\"] = False  # set whiten to False\n            # whiten recording separately; make sure dtype is float32\n            # to avoid downstream error with svd\n            recording = sip.whiten(recording, dtype=\"float32\")\n            sorting = sis.run_sorter(\n                sorter,\n                recording,\n                output_folder=sorter_temp_dir.name,\n                remove_existing_folder=True,\n                delete_output_folder=True,\n                **sorter_params,\n            )\n        key[\"time_of_sort\"] = int(time.time())\n\n        logger.info(\"Saving sorting results...\")\n\n        sorting_folder = Path(sorting_dir)\n\n        sorting_name = self._get_sorting_name(key)\n        key[\"sorting_path\"] = str(sorting_folder / Path(sorting_name))\n        if os.path.exists(key[\"sorting_path\"]):\n            shutil.rmtree(key[\"sorting_path\"])\n        sorting = sorting.save(folder=key[\"sorting_path\"])\n        self.insert1(key)\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        raise NotImplementedError\n        return None\n\n    def nightly_cleanup(self):\n        \"\"\"Clean up spike sorting directories that are not in the SpikeSorting table.\n        This should be run after AnalysisNwbFile().nightly_cleanup()\n        \"\"\"\n        # get a list of the files in the spike sorting storage directory\n        dir_names = next(os.walk(sorting_dir))[1]\n        # now retrieve a list of the currently used analysis nwb files\n        analysis_file_names = self.fetch(\"analysis_file_name\")\n        for dir in dir_names:\n            if dir not in analysis_file_names:\n                full_path = str(Path(sorting_dir) / dir)\n                logger.info(f\"removing {full_path}\")\n                shutil.rmtree(str(Path(sorting_dir) / dir))\n\n    @staticmethod\n    def _get_sorting_name(key):\n        recording_name = SpikeSortingRecording._get_recording_name(key)\n        sorting_name = (\n            recording_name + \"_\" + str(uuid.uuid4())[0:8] + \"_spikesorting\"\n        )\n        return sorting_name\n\n    # TODO: write a function to import sorting done outside of dj\n\n    def _import_sorting(self, key):\n        raise NotImplementedError\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting.make", "title": "<code>make(key)</code>", "text": "<p>Runs spike sorting on the data and parameters specified by the SpikeSortingSelection table and inserts a new entry to SpikeSorting table.</p> <p>Specifically, 1. Loads saved recording and runs the sort on it with spikeinterface 2. Saves the sorting with spikeinterface 3. Creates an analysis NWB file and saves the sorting there    (this is redundant with 2; will change in the future)</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Runs spike sorting on the data and parameters specified by the\n    SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n\n    Specifically,\n    1. Loads saved recording and runs the sort on it with spikeinterface\n    2. Saves the sorting with spikeinterface\n    3. Creates an analysis NWB file and saves the sorting there\n       (this is redundant with 2; will change in the future)\n\n    \"\"\"\n    # CBroz: does this not work w/o arg? as .populate() ?\n    recording_path = (SpikeSortingRecording &amp; key).fetch1(\"recording_path\")\n    recording = si.load_extractor(recording_path)\n\n    # first, get the timestamps\n    timestamps = SpikeSortingRecording._get_recording_timestamps(recording)\n    _ = recording.get_sampling_frequency()\n    # then concatenate the recordings\n    # Note: the timestamps are lost upon concatenation,\n    # i.e. concat_recording.get_times() doesn't return true timestamps anymore.\n    # but concat_recording.recoring_list[i].get_times() will return correct\n    # timestamps for ith recording.\n    if recording.get_num_segments() &gt; 1 and isinstance(\n        recording, si.AppendSegmentRecording\n    ):\n        recording = si.concatenate_recordings(recording.recording_list)\n    elif recording.get_num_segments() &gt; 1 and isinstance(\n        recording, si.BinaryRecordingExtractor\n    ):\n        recording = si.concatenate_recordings([recording])\n\n    # load artifact intervals\n    artifact_times = (\n        ArtifactRemovedIntervalList\n        &amp; {\n            \"artifact_removed_interval_list_name\": key[\n                \"artifact_removed_interval_list_name\"\n            ]\n        }\n    ).fetch1(\"artifact_times\")\n    if len(artifact_times):\n        if artifact_times.ndim == 1:\n            artifact_times = np.expand_dims(artifact_times, 0)\n\n        # convert artifact intervals to indices\n        list_triggers = []\n        for interval in artifact_times:\n            list_triggers.append(\n                np.arange(\n                    np.searchsorted(timestamps, interval[0]),\n                    np.searchsorted(timestamps, interval[1]),\n                )\n            )\n        list_triggers = [list(np.concatenate(list_triggers))]\n        recording = sip.remove_artifacts(\n            recording=recording,\n            list_triggers=list_triggers,\n            ms_before=None,\n            ms_after=None,\n            mode=\"zeros\",\n        )\n\n    logger.info(f\"Running spike sorting on {key}...\")\n    sorter, sorter_params = (SpikeSorterParameters &amp; key).fetch1(\n        \"sorter\", \"sorter_params\"\n    )\n\n    sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n    # add tempdir option for mountainsort\n    sorter_params[\"tempdir\"] = sorter_temp_dir.name\n\n    if sorter == \"clusterless_thresholder\":\n        # need to remove tempdir and whiten from sorter_params\n        sorter_params.pop(\"tempdir\", None)\n        sorter_params.pop(\"whiten\", None)\n        sorter_params.pop(\"outputs\", None)\n\n        # Detect peaks for clusterless decoding\n        detected_spikes = detect_peaks(recording, **sorter_params)\n        sorting = si.NumpySorting.from_times_labels(\n            times_list=detected_spikes[\"sample_index\"],\n            labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n    else:\n        if \"whiten\" in sorter_params.keys():\n            if sorter_params[\"whiten\"]:\n                sorter_params[\"whiten\"] = False  # set whiten to False\n        # whiten recording separately; make sure dtype is float32\n        # to avoid downstream error with svd\n        recording = sip.whiten(recording, dtype=\"float32\")\n        sorting = sis.run_sorter(\n            sorter,\n            recording,\n            output_folder=sorter_temp_dir.name,\n            remove_existing_folder=True,\n            delete_output_folder=True,\n            **sorter_params,\n        )\n    key[\"time_of_sort\"] = int(time.time())\n\n    logger.info(\"Saving sorting results...\")\n\n    sorting_folder = Path(sorting_dir)\n\n    sorting_name = self._get_sorting_name(key)\n    key[\"sorting_path\"] = str(sorting_folder / Path(sorting_name))\n    if os.path.exists(key[\"sorting_path\"]):\n        shutil.rmtree(key[\"sorting_path\"])\n    sorting = sorting.save(folder=key[\"sorting_path\"])\n    self.insert1(key)\n</code></pre>"}, {"location": "api/spikesorting/v0/spikesorting_sorting/#src.spyglass.spikesorting.v0.spikesorting_sorting.SpikeSorting.nightly_cleanup", "title": "<code>nightly_cleanup()</code>", "text": "<p>Clean up spike sorting directories that are not in the SpikeSorting table. This should be run after AnalysisNwbFile().nightly_cleanup()</p> Source code in <code>src/spyglass/spikesorting/v0/spikesorting_sorting.py</code> <pre><code>def nightly_cleanup(self):\n    \"\"\"Clean up spike sorting directories that are not in the SpikeSorting table.\n    This should be run after AnalysisNwbFile().nightly_cleanup()\n    \"\"\"\n    # get a list of the files in the spike sorting storage directory\n    dir_names = next(os.walk(sorting_dir))[1]\n    # now retrieve a list of the currently used analysis nwb files\n    analysis_file_names = self.fetch(\"analysis_file_name\")\n    for dir in dir_names:\n        if dir not in analysis_file_names:\n            full_path = str(Path(sorting_dir) / dir)\n            logger.info(f\"removing {full_path}\")\n            shutil.rmtree(str(Path(sorting_dir) / dir))\n</code></pre>"}, {"location": "api/spikesorting/v0/figurl_views/SpikeSortingRecordingView/", "title": "SpikeSortingRecordingView.py", "text": ""}, {"location": "api/spikesorting/v0/figurl_views/SpikeSortingView/", "title": "SpikeSortingView.py", "text": ""}, {"location": "api/spikesorting/v0/figurl_views/prepare_spikesortingview_data/", "title": "prepare_spikesortingview_data.py", "text": ""}, {"location": "api/spikesorting/v1/artifact/", "title": "artifact.py", "text": ""}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.ArtifactDetectionSelection", "title": "<code>ArtifactDetectionSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>@schema\nclass ArtifactDetectionSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Processed recording and artifact detection parameters. Use `insert_selection` method to insert new rows.\n    artifact_id: uuid\n    ---\n    -&gt; SpikeSortingRecording\n    -&gt; ArtifactDetectionParameters\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into ArtifactDetectionSelection with an\n        automatically generated unique artifact ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSortingRecording and ArtifactDetectionParameters\n\n        Returns\n        -------\n        artifact_id : str\n            the unique artifact ID serving as primary key for ArtifactDetectionSelection\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"artifact_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.ArtifactDetectionSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into ArtifactDetectionSelection with an automatically generated unique artifact ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSortingRecording and ArtifactDetectionParameters</p> required <p>Returns:</p> Name Type Description <code>artifact_id</code> <code>str</code> <p>the unique artifact ID serving as primary key for ArtifactDetectionSelection</p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into ArtifactDetectionSelection with an\n    automatically generated unique artifact ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSortingRecording and ArtifactDetectionParameters\n\n    Returns\n    -------\n    artifact_id : str\n        the unique artifact ID serving as primary key for ArtifactDetectionSelection\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.warn(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"artifact_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/artifact/#src.spyglass.spikesorting.v1.artifact.merge_intervals", "title": "<code>merge_intervals(intervals)</code>", "text": "<p>Takes a list of intervals each of which is [start_time, stop_time] and takes union over intervals that are intersecting</p> <p>Parameters:</p> Name Type Description Default <code>intervals</code> <code>_type_</code> <p>description</p> required <p>Returns:</p> Type Description <code>_type_</code> <p>description</p> Source code in <code>src/spyglass/spikesorting/v1/artifact.py</code> <pre><code>def merge_intervals(intervals):\n    \"\"\"Takes a list of intervals each of which is [start_time, stop_time]\n    and takes union over intervals that are intersecting\n\n    Parameters\n    ----------\n    intervals : _type_\n        _description_\n\n    Returns\n    -------\n    _type_\n        _description_\n    \"\"\"\n    if len(intervals) == 0:\n        return []\n\n    # Sort the intervals based on their start times\n    intervals.sort(key=lambda x: x[0])\n\n    merged = [intervals[0]]\n\n    for i in range(1, len(intervals)):\n        current_start, current_stop = intervals[i]\n        last_merged_start, last_merged_stop = merged[-1]\n\n        if current_start &lt;= last_merged_stop:\n            # Overlapping intervals, merge them\n            merged[-1] = [\n                last_merged_start,\n                max(last_merged_stop, current_stop),\n            ]\n        else:\n            # Non-overlapping intervals, add the current one to the list\n            merged.append([current_start, current_stop])\n\n    return np.asarray(merged)\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/", "title": "curation.py", "text": ""}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1", "title": "<code>CurationV1</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@schema\nclass CurationV1(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Curation of a SpikeSorting. Use `insert_curation` to insert rows.\n    -&gt; SpikeSorting\n    curation_id=0: int\n    ---\n    parent_curation_id=-1: int\n    -&gt; AnalysisNwbfile\n    object_id: varchar(72)\n    merges_applied: bool\n    description: varchar(100)\n    \"\"\"\n\n    @classmethod\n    def insert_curation(\n        cls,\n        sorting_id: str,\n        parent_curation_id: int = -1,\n        labels: Union[None, Dict[str, List[str]]] = None,\n        merge_groups: Union[None, List[List[str]]] = None,\n        apply_merge: bool = False,\n        metrics: Union[None, Dict[str, Dict[str, float]]] = None,\n        description: str = \"\",\n    ):\n        \"\"\"Insert a row into CurationV1.\n\n        Parameters\n        ----------\n        sorting_id : str\n            The key for the original SpikeSorting\n        parent_curation_id : int, optional\n            The curation id of the parent curation\n        labels : dict or None, optional\n            curation labels (e.g. good, noise, mua)\n        merge_groups : dict or None, optional\n            groups of unit IDs to be merged\n        metrics : dict or None, optional\n            Computed quality metrics, one for each neuron\n        description : str, optional\n            description of this curation or where it originates; e.g. FigURL\n\n        Note\n        ----\n        Example curation.json (output of figurl):\n        {\n         \"labelsByUnit\":\n            {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},\n         \"mergeGroups\":\n            [[11,12],[46,48],[51,54],[50,53]]\n        }\n\n        Returns\n        -------\n        curation_key : dict\n        \"\"\"\n        sort_query = cls &amp; {\"sorting_id\": sorting_id}\n        parent_curation_id = max(parent_curation_id, -1)\n        if parent_curation_id == -1:\n            parent_curation_id = -1\n            # check to see if this sorting with a parent of -1\n            # has already been inserted and if so, warn the user\n            query = sort_query &amp; {\"parent_curation_id\": -1}\n            if query:\n                Warning(\"Sorting has already been inserted.\")\n                return query.fetch(\"KEY\")\n\n        # generate curation ID\n        existing_curation_ids = sort_query.fetch(\"curation_id\")\n        curation_id = max(existing_curation_ids, default=-1) + 1\n\n        # write the curation labels, merge groups,\n        # and metrics as columns in the units table of NWB\n        analysis_file_name, object_id = _write_sorting_to_nwb_with_curation(\n            sorting_id=sorting_id,\n            labels=labels,\n            merge_groups=merge_groups,\n            metrics=metrics,\n            apply_merge=apply_merge,\n        )\n\n        # INSERT\n        AnalysisNwbfile().add(\n            (SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}).fetch1(\n                \"nwb_file_name\"\n            ),\n            analysis_file_name,\n        )\n\n        key = {\n            \"sorting_id\": sorting_id,\n            \"curation_id\": curation_id,\n            \"parent_curation_id\": parent_curation_id,\n            \"analysis_file_name\": analysis_file_name,\n            \"object_id\": object_id,\n            \"merges_applied\": apply_merge,\n            \"description\": description,\n        }\n        cls.insert1(\n            key,\n            skip_duplicates=True,\n        )\n\n        return key\n\n    @classmethod\n    def insert_metric_curation(cls, key: Dict, apply_merge=False):\n        \"\"\"Insert a row into CurationV1.\n\n        Parameters\n        ----------\n        key : Dict\n            primary key of MetricCuration\n\n        Returns\n        -------\n        curation_key : Dict\n        \"\"\"\n        from spyglass.spikesorting.v1.metric_curation import (\n            MetricCuration,\n            MetricCurationSelection,\n        )\n\n        sorting_id, parent_curation_id = (MetricCurationSelection &amp; key).fetch1(\n            \"sorting_id\", \"curation_id\"\n        )\n\n        curation_key = cls.insert_curation(\n            sorting_id=sorting_id,\n            parent_curation_id=parent_curation_id,\n            labels=MetricCuration.get_labels(key) or None,\n            merge_groups=MetricCuration.get_merge_groups(key) or None,\n            apply_merge=apply_merge,\n            description=(f\"metric_curation_id: {key['metric_curation_id']}\"),\n        )\n\n        return curation_key\n\n    @classmethod\n    def get_recording(cls, key: dict) -&gt; si.BaseRecording:\n        \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1 table\n        \"\"\"\n\n        analysis_file_name = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        recording = se.read_nwb_recording(\n            analysis_file_abs_path, load_time_vector=True\n        )\n        recording.annotate(is_filtered=True)\n\n        return recording\n\n    @classmethod\n    def get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1 table\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n        recording = cls.get_recording(key)\n        sampling_frequency = recording.get_sampling_frequency()\n        analysis_file_name = (CurationV1 &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.units.to_dataframe()\n        units_dict_list = [\n            {\n                unit_id: np.searchsorted(recording.get_times(), spike_times)\n                for unit_id, spike_times in zip(\n                    units.index, units[\"spike_times\"]\n                )\n            }\n        ]\n\n        sorting = si.NumpySorting.from_unit_dict(\n            units_dict_list, sampling_frequency=sampling_frequency\n        )\n\n        return sorting\n\n    @classmethod\n    def get_merged_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting with merges applied.\n\n        Parameters\n        ----------\n        key : dict\n            CurationV1 key\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n        recording = cls.get_recording(key)\n\n        curation_key = (cls &amp; key).fetch1()\n\n        sorting_analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            curation_key[\"analysis_file_name\"]\n        )\n        si_sorting = se.read_nwb_sorting(\n            sorting_analysis_file_abs_path,\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n\n        with pynwb.NWBHDF5IO(\n            sorting_analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            nwb_sorting = nwbfile.objects[curation_key[\"object_id\"]]\n            merge_groups = nwb_sorting[\"merge_groups\"][:]\n\n        if merge_groups:\n            units_to_merge = _merge_dict_to_list(merge_groups)\n            return sc.MergeUnitsSorting(\n                parent_sorting=si_sorting, units_to_merge=units_to_merge\n            )\n        else:\n            return si_sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_curation", "title": "<code>insert_curation(sorting_id, parent_curation_id=-1, labels=None, merge_groups=None, apply_merge=False, metrics=None, description='')</code>  <code>classmethod</code>", "text": "<p>Insert a row into CurationV1.</p> <p>Parameters:</p> Name Type Description Default <code>sorting_id</code> <code>str</code> <p>The key for the original SpikeSorting</p> required <code>parent_curation_id</code> <code>int</code> <p>The curation id of the parent curation</p> <code>-1</code> <code>labels</code> <code>dict or None</code> <p>curation labels (e.g. good, noise, mua)</p> <code>None</code> <code>merge_groups</code> <code>dict or None</code> <p>groups of unit IDs to be merged</p> <code>None</code> <code>metrics</code> <code>dict or None</code> <p>Computed quality metrics, one for each neuron</p> <code>None</code> <code>description</code> <code>str</code> <p>description of this curation or where it originates; e.g. FigURL</p> <code>''</code>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_curation--note", "title": "Note", "text": "<p>Example curation.json (output of figurl): {  \"labelsByUnit\":     {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},  \"mergeGroups\":     [[11,12],[46,48],[51,54],[50,53]] }</p> <p>Returns:</p> Name Type Description <code>curation_key</code> <code>dict</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef insert_curation(\n    cls,\n    sorting_id: str,\n    parent_curation_id: int = -1,\n    labels: Union[None, Dict[str, List[str]]] = None,\n    merge_groups: Union[None, List[List[str]]] = None,\n    apply_merge: bool = False,\n    metrics: Union[None, Dict[str, Dict[str, float]]] = None,\n    description: str = \"\",\n):\n    \"\"\"Insert a row into CurationV1.\n\n    Parameters\n    ----------\n    sorting_id : str\n        The key for the original SpikeSorting\n    parent_curation_id : int, optional\n        The curation id of the parent curation\n    labels : dict or None, optional\n        curation labels (e.g. good, noise, mua)\n    merge_groups : dict or None, optional\n        groups of unit IDs to be merged\n    metrics : dict or None, optional\n        Computed quality metrics, one for each neuron\n    description : str, optional\n        description of this curation or where it originates; e.g. FigURL\n\n    Note\n    ----\n    Example curation.json (output of figurl):\n    {\n     \"labelsByUnit\":\n        {\"1\":[\"noise\",\"reject\"],\"10\":[\"noise\",\"reject\"]},\n     \"mergeGroups\":\n        [[11,12],[46,48],[51,54],[50,53]]\n    }\n\n    Returns\n    -------\n    curation_key : dict\n    \"\"\"\n    sort_query = cls &amp; {\"sorting_id\": sorting_id}\n    parent_curation_id = max(parent_curation_id, -1)\n    if parent_curation_id == -1:\n        parent_curation_id = -1\n        # check to see if this sorting with a parent of -1\n        # has already been inserted and if so, warn the user\n        query = sort_query &amp; {\"parent_curation_id\": -1}\n        if query:\n            Warning(\"Sorting has already been inserted.\")\n            return query.fetch(\"KEY\")\n\n    # generate curation ID\n    existing_curation_ids = sort_query.fetch(\"curation_id\")\n    curation_id = max(existing_curation_ids, default=-1) + 1\n\n    # write the curation labels, merge groups,\n    # and metrics as columns in the units table of NWB\n    analysis_file_name, object_id = _write_sorting_to_nwb_with_curation(\n        sorting_id=sorting_id,\n        labels=labels,\n        merge_groups=merge_groups,\n        metrics=metrics,\n        apply_merge=apply_merge,\n    )\n\n    # INSERT\n    AnalysisNwbfile().add(\n        (SpikeSortingSelection &amp; {\"sorting_id\": sorting_id}).fetch1(\n            \"nwb_file_name\"\n        ),\n        analysis_file_name,\n    )\n\n    key = {\n        \"sorting_id\": sorting_id,\n        \"curation_id\": curation_id,\n        \"parent_curation_id\": parent_curation_id,\n        \"analysis_file_name\": analysis_file_name,\n        \"object_id\": object_id,\n        \"merges_applied\": apply_merge,\n        \"description\": description,\n    }\n    cls.insert1(\n        key,\n        skip_duplicates=True,\n    )\n\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.insert_metric_curation", "title": "<code>insert_metric_curation(key, apply_merge=False)</code>  <code>classmethod</code>", "text": "<p>Insert a row into CurationV1.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>Dict</code> <p>primary key of MetricCuration</p> required <p>Returns:</p> Name Type Description <code>curation_key</code> <code>Dict</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef insert_metric_curation(cls, key: Dict, apply_merge=False):\n    \"\"\"Insert a row into CurationV1.\n\n    Parameters\n    ----------\n    key : Dict\n        primary key of MetricCuration\n\n    Returns\n    -------\n    curation_key : Dict\n    \"\"\"\n    from spyglass.spikesorting.v1.metric_curation import (\n        MetricCuration,\n        MetricCurationSelection,\n    )\n\n    sorting_id, parent_curation_id = (MetricCurationSelection &amp; key).fetch1(\n        \"sorting_id\", \"curation_id\"\n    )\n\n    curation_key = cls.insert_curation(\n        sorting_id=sorting_id,\n        parent_curation_id=parent_curation_id,\n        labels=MetricCuration.get_labels(key) or None,\n        merge_groups=MetricCuration.get_merge_groups(key) or None,\n        apply_merge=apply_merge,\n        description=(f\"metric_curation_id: {key['metric_curation_id']}\"),\n    )\n\n    return curation_key\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Get recording related to this curation as spikeinterface BaseRecording</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1 table</p> required Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_recording(cls, key: dict) -&gt; si.BaseRecording:\n    \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1 table\n    \"\"\"\n\n    analysis_file_name = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    recording = se.read_nwb_recording(\n        analysis_file_abs_path, load_time_vector=True\n    )\n    recording.annotate(is_filtered=True)\n\n    return recording\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting in the analysis NWB file as spikeinterface BaseSorting</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1 table</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1 table\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n    recording = cls.get_recording(key)\n    sampling_frequency = recording.get_sampling_frequency()\n    analysis_file_name = (CurationV1 &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.units.to_dataframe()\n    units_dict_list = [\n        {\n            unit_id: np.searchsorted(recording.get_times(), spike_times)\n            for unit_id, spike_times in zip(\n                units.index, units[\"spike_times\"]\n            )\n        }\n    ]\n\n    sorting = si.NumpySorting.from_unit_dict(\n        units_dict_list, sampling_frequency=sampling_frequency\n    )\n\n    return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/curation/#src.spyglass.spikesorting.v1.curation.CurationV1.get_merged_sorting", "title": "<code>get_merged_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting with merges applied.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>CurationV1 key</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/curation.py</code> <pre><code>@classmethod\ndef get_merged_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting with merges applied.\n\n    Parameters\n    ----------\n    key : dict\n        CurationV1 key\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n    recording = cls.get_recording(key)\n\n    curation_key = (cls &amp; key).fetch1()\n\n    sorting_analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        curation_key[\"analysis_file_name\"]\n    )\n    si_sorting = se.read_nwb_sorting(\n        sorting_analysis_file_abs_path,\n        sampling_frequency=recording.get_sampling_frequency(),\n    )\n\n    with pynwb.NWBHDF5IO(\n        sorting_analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbfile = io.read()\n        nwb_sorting = nwbfile.objects[curation_key[\"object_id\"]]\n        merge_groups = nwb_sorting[\"merge_groups\"][:]\n\n    if merge_groups:\n        units_to_merge = _merge_dict_to_list(merge_groups)\n        return sc.MergeUnitsSorting(\n            parent_sorting=si_sorting, units_to_merge=units_to_merge\n        )\n    else:\n        return si_sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/", "title": "figurl_curation.py", "text": ""}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection", "title": "<code>FigURLCurationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@schema\nclass FigURLCurationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Use `insert_selection` method to insert a row. Use `generate_curation_uri` method to generate a curation uri.\n    figurl_curation_id: uuid\n    ---\n    -&gt; CurationV1\n    curation_uri: varchar(1000)     # GitHub-based URI to a file to which the manual curation will be saved\n    metrics_figurl: blob            # metrics to display in the figURL\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into FigURLCurationSelection.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of `CurationV1`, `curation_uri`, and `metrics_figurl`.\n            - If `curation_uri` is not provided, it will be generated from `generate_curation_uri` method.\n            - If `metrics_figurl` is not provided, it will be set to [].\n\n        Returns\n        -------\n        key : dict\n            primary key of `FigURLCurationSelection` table.\n        \"\"\"\n        if \"curation_uri\" not in key:\n            key[\"curation_uri\"] = cls.generate_curation_uri(key)\n        if \"metrics_figurl\" not in key:\n            key[\"metrics_figurl\"] = []\n        if \"figurl_curation_id\" in key:\n            query = cls &amp; {\"figurl_curation_id\": key[\"figurl_curation_id\"]}\n            if query:\n                logger.warn(\"Similar row(s) already inserted.\")\n                return query.fetch(as_dict=True)\n        key[\"figurl_curation_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n\n    @staticmethod\n    def generate_curation_uri(key: Dict) -&gt; str:\n        \"\"\"Generates a kachery-cloud URI from a row in CurationV1 table\n\n        Parameters\n        ----------\n        key : dict\n            primary key from CurationV1\n        \"\"\"\n        curation_key = (CurationV1 &amp; key).fetch1()\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            curation_key[\"analysis_file_name\"]\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbfile = io.read()\n            nwb_sorting = nwbfile.objects[\n                curation_key[\"object_id\"]\n            ].to_dataframe()\n            unit_ids = list(nwb_sorting.index)\n            labels = list(nwb_sorting[\"curation_label\"])\n            merge_groups = list(nwb_sorting[\"merge_groups\"])\n\n        unit_ids = [str(unit_id) for unit_id in unit_ids]\n\n        labels_dict = (\n            {unit_id: list(label) for unit_id, label in zip(unit_ids, labels)}\n            if labels\n            else {}\n        )\n\n        merge_groups_list = (\n            [\n                [str(unit_id) for unit_id in merge_group]\n                for merge_group in _merge_dict_to_list(\n                    dict(zip(unit_ids, merge_groups))\n                )\n            ]\n            if merge_groups\n            else []\n        )\n\n        return kcl.store_json(\n            {\n                \"labelsByUnit\": labels_dict,\n                \"mergeGroups\": merge_groups_list,\n            }\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into FigURLCurationSelection.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of <code>CurationV1</code>, <code>curation_uri</code>, and <code>metrics_figurl</code>. - If <code>curation_uri</code> is not provided, it will be generated from <code>generate_curation_uri</code> method. - If <code>metrics_figurl</code> is not provided, it will be set to [].</p> required <p>Returns:</p> Name Type Description <code>key</code> <code>dict</code> <p>primary key of <code>FigURLCurationSelection</code> table.</p> Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into FigURLCurationSelection.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of `CurationV1`, `curation_uri`, and `metrics_figurl`.\n        - If `curation_uri` is not provided, it will be generated from `generate_curation_uri` method.\n        - If `metrics_figurl` is not provided, it will be set to [].\n\n    Returns\n    -------\n    key : dict\n        primary key of `FigURLCurationSelection` table.\n    \"\"\"\n    if \"curation_uri\" not in key:\n        key[\"curation_uri\"] = cls.generate_curation_uri(key)\n    if \"metrics_figurl\" not in key:\n        key[\"metrics_figurl\"] = []\n    if \"figurl_curation_id\" in key:\n        query = cls &amp; {\"figurl_curation_id\": key[\"figurl_curation_id\"]}\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n    key[\"figurl_curation_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/figurl_curation/#src.spyglass.spikesorting.v1.figurl_curation.FigURLCurationSelection.generate_curation_uri", "title": "<code>generate_curation_uri(key)</code>  <code>staticmethod</code>", "text": "<p>Generates a kachery-cloud URI from a row in CurationV1 table</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key from CurationV1</p> required Source code in <code>src/spyglass/spikesorting/v1/figurl_curation.py</code> <pre><code>@staticmethod\ndef generate_curation_uri(key: Dict) -&gt; str:\n    \"\"\"Generates a kachery-cloud URI from a row in CurationV1 table\n\n    Parameters\n    ----------\n    key : dict\n        primary key from CurationV1\n    \"\"\"\n    curation_key = (CurationV1 &amp; key).fetch1()\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        curation_key[\"analysis_file_name\"]\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbfile = io.read()\n        nwb_sorting = nwbfile.objects[\n            curation_key[\"object_id\"]\n        ].to_dataframe()\n        unit_ids = list(nwb_sorting.index)\n        labels = list(nwb_sorting[\"curation_label\"])\n        merge_groups = list(nwb_sorting[\"merge_groups\"])\n\n    unit_ids = [str(unit_id) for unit_id in unit_ids]\n\n    labels_dict = (\n        {unit_id: list(label) for unit_id, label in zip(unit_ids, labels)}\n        if labels\n        else {}\n    )\n\n    merge_groups_list = (\n        [\n            [str(unit_id) for unit_id in merge_group]\n            for merge_group in _merge_dict_to_list(\n                dict(zip(unit_ids, merge_groups))\n            )\n        ]\n        if merge_groups\n        else []\n    )\n\n    return kcl.store_json(\n        {\n            \"labelsByUnit\": labels_dict,\n            \"mergeGroups\": merge_groups_list,\n        }\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/", "title": "metric_curation.py", "text": ""}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCurationSelection", "title": "<code>MetricCurationSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@schema\nclass MetricCurationSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Spike sorting and parameters for metric curation. Use `insert_selection` to insert a row into this table.\n    metric_curation_id: uuid\n    ---\n    -&gt; CurationV1\n    -&gt; WaveformParameters\n    -&gt; MetricParameters\n    -&gt; MetricCurationParameters\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into MetricCurationSelection with an\n        automatically generated unique metric curation ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters\n\n        Returns\n        -------\n        key : dict\n            key for the inserted row\n        \"\"\"\n        if cls &amp; key:\n            logger.warn(\"This row has already been inserted.\")\n            return (cls &amp; key).fetch1()\n        key[\"metric_curation_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCurationSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into MetricCurationSelection with an automatically generated unique metric curation ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters</p> required <p>Returns:</p> Name Type Description <code>key</code> <code>dict</code> <p>key for the inserted row</p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into MetricCurationSelection with an\n    automatically generated unique metric curation ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of CurationV1, WaveformParameters, MetricParameters MetricCurationParameters\n\n    Returns\n    -------\n    key : dict\n        key for the inserted row\n    \"\"\"\n    if cls &amp; key:\n        logger.warn(\"This row has already been inserted.\")\n        return (cls &amp; key).fetch1()\n    key[\"metric_curation_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration", "title": "<code>MetricCuration</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@schema\nclass MetricCuration(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    # Results of applying curation based on quality metrics. To do additional curation, insert another row in `CurationV1`\n    -&gt; MetricCurationSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # Object ID for the metrics in NWB file\n    \"\"\"\n\n    def make(self, key):\n        # FETCH\n        nwb_file_name = (\n            SpikeSortingSelection * MetricCurationSelection &amp; key\n        ).fetch1(\"nwb_file_name\")\n\n        waveform_params = (\n            WaveformParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"waveform_params\")\n        metric_params = (\n            MetricParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"metric_params\")\n        label_params, merge_params = (\n            MetricCurationParameters * MetricCurationSelection &amp; key\n        ).fetch1(\"label_params\", \"merge_params\")\n        sorting_id, curation_id = (MetricCurationSelection &amp; key).fetch1(\n            \"sorting_id\", \"curation_id\"\n        )\n        # DO\n        # load recording and sorting\n        recording = CurationV1.get_recording(\n            {\"sorting_id\": sorting_id, \"curation_id\": curation_id}\n        )\n        sorting = CurationV1.get_sorting(\n            {\"sorting_id\": sorting_id, \"curation_id\": curation_id}\n        )\n        # extract waveforms\n        if \"whiten\" in waveform_params:\n            if waveform_params.pop(\"whiten\"):\n                recording = sp.whiten(recording, dtype=np.float64)\n\n        waveforms_dir = temp_dir + \"/\" + str(key[\"metric_curation_id\"])\n        os.makedirs(waveforms_dir, exist_ok=True)\n\n        logger.info(\"Extracting waveforms...\")\n        waveforms = si.extract_waveforms(\n            recording=recording,\n            sorting=sorting,\n            folder=waveforms_dir,\n            overwrite=True,\n            **waveform_params,\n        )\n        # compute metrics\n        logger.info(\"Computing metrics...\")\n        metrics = {}\n        for metric_name, metric_param_dict in metric_params.items():\n            metrics[metric_name] = self._compute_metric(\n                waveforms, metric_name, **metric_param_dict\n            )\n        if metrics[\"nn_isolation\"]:\n            metrics[\"nn_isolation\"] = {\n                unit_id: value[0]\n                for unit_id, value in metrics[\"nn_isolation\"].items()\n            }\n\n        logger.info(\"Applying curation...\")\n        labels = self._compute_labels(metrics, label_params)\n        merge_groups = self._compute_merge_groups(metrics, merge_params)\n\n        logger.info(\"Saving to NWB...\")\n        (\n            key[\"analysis_file_name\"],\n            key[\"object_id\"],\n        ) = _write_metric_curation_to_nwb(\n            nwb_file_name, waveforms, metrics, labels, merge_groups\n        )\n\n        # INSERT\n        AnalysisNwbfile().add(\n            nwb_file_name,\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @classmethod\n    def get_waveforms(cls):\n        return NotImplementedError\n\n    @classmethod\n    def get_metrics(cls, key: dict):\n        \"\"\"Returns metrics identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id, metric_param_name, metric_params = (\n            cls * MetricCurationSelection * MetricParameters &amp; key\n        ).fetch1(\n            \"analysis_file_name\",\n            \"object_id\",\n            \"metric_param_name\",\n            \"metric_params\",\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        return {\n            name: dict(zip(units.index, units[name])) for name in metric_params\n        }\n\n    @classmethod\n    def get_labels(cls, key: dict):\n        \"\"\"Returns curation labels identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id = (cls &amp; key).fetch1(\n            \"analysis_file_name\", \"object_id\"\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        return dict(zip(units.index, units[\"curation_label\"]))\n\n    @classmethod\n    def get_merge_groups(cls, key: dict):\n        \"\"\"Returns merge groups identified by metric curation\n\n        Parameters\n        ----------\n        key : dict\n            primary key to MetricCuration\n        \"\"\"\n        analysis_file_name, object_id = (cls &amp; key).fetch1(\n            \"analysis_file_name\", \"object_id\"\n        )\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            path=analysis_file_abs_path,\n            mode=\"r\",\n            load_namespaces=True,\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.objects[object_id].to_dataframe()\n        merge_group_dict = dict(zip(units.index, units[\"merge_groups\"]))\n\n        return _merge_dict_to_list(merge_group_dict)\n\n    @staticmethod\n    def _compute_metric(waveform_extractor, metric_name, **metric_params):\n        metric_func = _metric_name_to_func[metric_name]\n\n        peak_sign_metrics = [\"snr\", \"peak_offset\", \"peak_channel\"]\n        if metric_name in peak_sign_metrics:\n            if \"peak_sign\" not in metric_params:\n                raise Exception(\n                    f\"{peak_sign_metrics} metrics require peak_sign\",\n                    \"to be defined in the metric parameters\",\n                )\n            return metric_func(\n                waveform_extractor,\n                peak_sign=metric_params.pop(\"peak_sign\"),\n                **metric_params,\n            )\n\n        return {\n            unit_id: metric_func(waveform_extractor, this_unit_id=unit_id)\n            for unit_id in waveform_extractor.sorting.get_unit_ids()\n        }\n\n    @staticmethod\n    def _compute_labels(\n        metrics: Dict[str, Dict[str, Union[float, List[float]]]],\n        label_params: Dict[str, List[Any]],\n    ) -&gt; Dict[str, List[str]]:\n        \"\"\"Computes the labels based on the metric and label parameters.\n\n        Parameters\n        ----------\n        quality_metrics : dict\n            Example: {\"snr\" : {\"1\" : 2, \"2\" : 0.1, \"3\" : 2.3}}\n            This indicates that the values of the \"snr\" quality metric\n            for the units \"1\", \"2\", \"3\" are 2, 0.1, and 2.3, respectively.\n\n        label_params : dict\n            Example: {\n                        \"snr\" : [(\"&gt;\", 1, [\"good\", \"mua\"]),\n                                 (\"&lt;\", 1, [\"noise\"])]\n                     }\n            This indicates that units with values of the \"snr\" quality metric\n            greater than 1 should be given the labels \"good\" and \"mua\" and values\n            less than 1 should be given the label \"noise\".\n\n        Returns\n        -------\n        labels : dict\n            Example: {\"1\" : [\"good\", \"mua\"], \"2\" : [\"noise\"], \"3\" : [\"good\", \"mua\"]}\n\n        \"\"\"\n        if not label_params:\n            return {}\n\n        unit_ids = [\n            unit_id for unit_id in metrics[list(metrics.keys())[0]].keys()\n        ]\n        labels = {unit_id: [] for unit_id in unit_ids}\n\n        for metric in label_params:\n            if metric not in metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n                continue\n\n            condition = label_params[metric]\n            if not len(condition) == 3:\n                raise ValueError(f\"Condition {condition} must be of length 3\")\n\n            compare = _comparison_to_function[condition[0]]\n            for unit_id in unit_ids:\n                if compare(\n                    metrics[metric][unit_id],\n                    condition[1],\n                ):\n                    labels[unit_id].extend(label_params[metric][2])\n        return labels\n\n    @staticmethod\n    def _compute_merge_groups(\n        metrics: Dict[str, Dict[str, Union[float, List[float]]]],\n        merge_params: Dict[str, List[Any]],\n    ) -&gt; Dict[str, List[str]]:\n        \"\"\"Identifies units to be merged based on the metrics and merge parameters.\n\n        Parameters\n        ---------\n        quality_metrics : dict\n            Example: {\"cosine_similarity\" : {\n                                             \"1\" : {\"1\" : 1.00, \"2\" : 0.10, \"3\": 0.95},\n                                             \"2\" : {\"1\" : 0.10, \"2\" : 1.00, \"3\": 0.70},\n                                             \"3\" : {\"1\" : 0.95, \"2\" : 0.70, \"3\": 1.00}\n                                            }}\n            This shows the pairwise values of the \"cosine_similarity\" quality metric\n            for the units \"1\", \"2\", \"3\" as a nested dict.\n\n        merge_params : dict\n            Example: {\"cosine_similarity\" : [\"&gt;\", 0.9]}\n            This indicates that units with values of the \"cosine_similarity\" quality metric\n            greater than 0.9 should be placed in the same merge group.\n\n\n        Returns\n        -------\n        merge_groups : dict\n            Example: {\"1\" : [\"3\"], \"2\" : [], \"3\" : [\"1\"]}\n\n        \"\"\"\n\n        if not merge_params:\n            return []\n\n        unit_ids = list(metrics[list(metrics.keys())[0]].keys())\n        merge_groups = {unit_id: [] for unit_id in unit_ids}\n        for metric in merge_params:\n            if metric not in metrics:\n                Warning(f\"{metric} not found in quality metrics; skipping\")\n                continue\n            compare = _comparison_to_function[merge_params[metric][0]]\n            for unit_id in unit_ids:\n                other_unit_ids = [\n                    other_unit_id\n                    for other_unit_id in unit_ids\n                    if other_unit_id != unit_id\n                ]\n                for other_unit_id in other_unit_ids:\n                    if compare(\n                        metrics[metric][unit_id][other_unit_id],\n                        merge_params[metric][1],\n                    ):\n                        merge_groups[unit_id].extend(other_unit_id)\n        return merge_groups\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_metrics", "title": "<code>get_metrics(key)</code>  <code>classmethod</code>", "text": "<p>Returns metrics identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_metrics(cls, key: dict):\n    \"\"\"Returns metrics identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id, metric_param_name, metric_params = (\n        cls * MetricCurationSelection * MetricParameters &amp; key\n    ).fetch1(\n        \"analysis_file_name\",\n        \"object_id\",\n        \"metric_param_name\",\n        \"metric_params\",\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    return {\n        name: dict(zip(units.index, units[name])) for name in metric_params\n    }\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_labels", "title": "<code>get_labels(key)</code>  <code>classmethod</code>", "text": "<p>Returns curation labels identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_labels(cls, key: dict):\n    \"\"\"Returns curation labels identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id = (cls &amp; key).fetch1(\n        \"analysis_file_name\", \"object_id\"\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    return dict(zip(units.index, units[\"curation_label\"]))\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_curation/#src.spyglass.spikesorting.v1.metric_curation.MetricCuration.get_merge_groups", "title": "<code>get_merge_groups(key)</code>  <code>classmethod</code>", "text": "<p>Returns merge groups identified by metric curation</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key to MetricCuration</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_curation.py</code> <pre><code>@classmethod\ndef get_merge_groups(cls, key: dict):\n    \"\"\"Returns merge groups identified by metric curation\n\n    Parameters\n    ----------\n    key : dict\n        primary key to MetricCuration\n    \"\"\"\n    analysis_file_name, object_id = (cls &amp; key).fetch1(\n        \"analysis_file_name\", \"object_id\"\n    )\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        path=analysis_file_abs_path,\n        mode=\"r\",\n        load_namespaces=True,\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.objects[object_id].to_dataframe()\n    merge_group_dict = dict(zip(units.index, units[\"merge_groups\"]))\n\n    return _merge_dict_to_list(merge_group_dict)\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/", "title": "metric_utils.py", "text": ""}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.compute_isi_violation_fractions", "title": "<code>compute_isi_violation_fractions(waveform_extractor, this_unit_id, isi_threshold_ms=2.0, min_isi_ms=0.0)</code>", "text": "<p>Computes the fraction of interspike interval violations.</p> <p>Parameters:</p> Name Type Description Default <code>waveform_extractor</code> <code>WaveformExtractor</code> <p>The extractor object for the recording.</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def compute_isi_violation_fractions(\n    waveform_extractor: si.WaveformExtractor,\n    this_unit_id: str,\n    isi_threshold_ms: float = 2.0,\n    min_isi_ms: float = 0.0,\n):\n    \"\"\"Computes the fraction of interspike interval violations.\n\n    Parameters\n    ----------\n    waveform_extractor: si.WaveformExtractor\n        The extractor object for the recording.\n\n    \"\"\"\n\n    # Extract the total number of spikes that violated the isi_threshold for each unit\n    _, isi_violation_counts = sq.compute_isi_violations(\n        waveform_extractor,\n        isi_threshold_ms=isi_threshold_ms,\n        min_isi_ms=min_isi_ms,\n    )\n    num_spikes = sq.compute_num_spikes(waveform_extractor)\n    return isi_violation_counts[this_unit_id] / (num_spikes[this_unit_id] - 1)\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_peak_offset", "title": "<code>get_peak_offset(waveform_extractor, peak_sign, **metric_params)</code>", "text": "<p>Computes the shift of the waveform peak from center of window.</p> <p>Parameters:</p> Name Type Description Default <code>waveform_extractor</code> <code>WaveformExtractor</code> <p>The extractor object for the recording.</p> required <code>peak_sign</code> <code>str</code> <p>The sign of the peak to compute. ('neg', 'pos', 'both')</p> required Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_peak_offset(\n    waveform_extractor: si.WaveformExtractor, peak_sign: str, **metric_params\n):\n    \"\"\"Computes the shift of the waveform peak from center of window.\n\n    Parameters\n    ----------\n    waveform_extractor: si.WaveformExtractor\n        The extractor object for the recording.\n    peak_sign: str\n        The sign of the peak to compute. ('neg', 'pos', 'both')\n    \"\"\"\n    if \"peak_sign\" in metric_params:\n        del metric_params[\"peak_sign\"]\n    peak_offset_inds = si.get_template_extremum_channel_peak_shift(\n        waveform_extractor=waveform_extractor,\n        peak_sign=peak_sign,\n        **metric_params,\n    )\n    return {key: int(abs(val)) for key, val in peak_offset_inds.items()}\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_peak_channel", "title": "<code>get_peak_channel(waveform_extractor, peak_sign, **metric_params)</code>", "text": "<p>Computes the electrode_id of the channel with the extremum peak for each unit.</p> Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_peak_channel(\n    waveform_extractor: si.WaveformExtractor, peak_sign: str, **metric_params\n):\n    \"\"\"Computes the electrode_id of the channel with the extremum peak for each unit.\"\"\"\n    if \"peak_sign\" in metric_params:\n        del metric_params[\"peak_sign\"]\n    peak_channel_dict = si.get_template_extremum_channel(\n        waveform_extractor=waveform_extractor,\n        peak_sign=peak_sign,\n        **metric_params,\n    )\n    return {key: int(val) for key, val in peak_channel_dict.items()}\n</code></pre>"}, {"location": "api/spikesorting/v1/metric_utils/#src.spyglass.spikesorting.v1.metric_utils.get_num_spikes", "title": "<code>get_num_spikes(waveform_extractor, this_unit_id)</code>", "text": "<p>Computes the number of spikes for each unit.</p> Source code in <code>src/spyglass/spikesorting/v1/metric_utils.py</code> <pre><code>def get_num_spikes(waveform_extractor: si.WaveformExtractor, this_unit_id: str):\n    \"\"\"Computes the number of spikes for each unit.\"\"\"\n    return sq.compute_num_spikes(waveform_extractor)[this_unit_id]\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/", "title": "recording.py", "text": ""}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SortGroup", "title": "<code>SortGroup</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SortGroup(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Set of electrodes to spike sort together\n    -&gt; Session\n    sort_group_id: int\n    ---\n    sort_reference_electrode_id = -1: int  # the electrode to use for referencing\n                                           # -1: no reference, -2: common median\n    \"\"\"\n\n    class SortGroupElectrode(SpyglassMixin, dj.Part):\n        definition = \"\"\"\n        -&gt; SortGroup\n        -&gt; Electrode\n        \"\"\"\n\n    @classmethod\n    def set_group_by_shank(\n        cls,\n        nwb_file_name: str,\n        references: dict = None,\n        omit_ref_electrode_group=False,\n        omit_unitrode=True,\n    ):\n        \"\"\"Divides electrodes into groups based on their shank position.\n\n        * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n          single group\n        * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n          placed in one group per shank\n        * Bad channels are omitted\n\n        Parameters\n        ----------\n        nwb_file_name : str\n            the name of the NWB file whose electrodes should be put into\n            sorting groups\n        references : dict, optional\n            If passed, used to set references. Otherwise, references set using\n            original reference electrodes from config. Keys: electrode groups.\n            Values: reference electrode.\n        omit_ref_electrode_group : bool\n            Optional. If True, no sort group is defined for electrode group of\n            reference.\n        omit_unitrode : bool\n            Optional. If True, no sort groups are defined for unitrodes.\n        \"\"\"\n        # delete any current groups\n        # (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n        # get the electrodes from this NWB file\n        electrodes = (\n            Electrode()\n            &amp; {\"nwb_file_name\": nwb_file_name}\n            &amp; {\"bad_channel\": \"False\"}\n        ).fetch()\n        e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n        e_groups.sort(key=int)  # sort electrode groups numerically\n        sort_group = 0\n        sg_key = dict()\n        sge_key = dict()\n        sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n        for e_group in e_groups:\n            # for each electrode group, get a list of the unique shank numbers\n            shank_list = np.unique(\n                electrodes[\"probe_shank\"][\n                    electrodes[\"electrode_group_name\"] == e_group\n                ]\n            )\n            sge_key[\"electrode_group_name\"] = e_group\n            # get the indices of all electrodes in this group / shank and set their sorting group\n            for shank in shank_list:\n                sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n                # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n                if not references:\n                    shank_elect_ref = electrodes[\n                        \"original_reference_electrode\"\n                    ][\n                        np.logical_and(\n                            electrodes[\"electrode_group_name\"] == e_group,\n                            electrodes[\"probe_shank\"] == shank,\n                        )\n                    ]\n                    if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                        sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                            0\n                        ]\n                    else:\n                        ValueError(\n                            f\"Error in electrode group {e_group}: reference \"\n                            + \"electrodes are not all the same\"\n                        )\n                else:\n                    if e_group not in references.keys():\n                        raise Exception(\n                            f\"electrode group {e_group} not a key in \"\n                            + \"references, so cannot set reference\"\n                        )\n                    else:\n                        sg_key[\"sort_reference_electrode_id\"] = references[\n                            e_group\n                        ]\n                # Insert sort group and sort group electrodes\n                reference_electrode_group = electrodes[\n                    electrodes[\"electrode_id\"]\n                    == sg_key[\"sort_reference_electrode_id\"]\n                ][\n                    \"electrode_group_name\"\n                ]  # reference for this electrode group\n                if (\n                    len(reference_electrode_group) == 1\n                ):  # unpack single reference\n                    reference_electrode_group = reference_electrode_group[0]\n                elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                    len(reference_electrode_group) != 1\n                ):\n                    raise Exception(\n                        \"Should have found exactly one electrode group for \"\n                        + \"reference electrode, but found \"\n                        + f\"{len(reference_electrode_group)}.\"\n                    )\n                if omit_ref_electrode_group and (\n                    str(e_group) == str(reference_electrode_group)\n                ):\n                    logger.warn(\n                        f\"Omitting electrode group {e_group} from sort groups \"\n                        + \"because contains reference.\"\n                    )\n                    continue\n                shank_elect = electrodes[\"electrode_id\"][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if (\n                    omit_unitrode and len(shank_elect) == 1\n                ):  # omit unitrodes if indicated\n                    logger.warn(\n                        f\"Omitting electrode group {e_group}, shank {shank} \"\n                        + \"from sort groups because unitrode.\"\n                    )\n                    continue\n                cls.insert1(sg_key, skip_duplicates=True)\n                for elect in shank_elect:\n                    sge_key[\"electrode_id\"] = elect\n                    cls.SortGroupElectrode().insert1(\n                        sge_key, skip_duplicates=True\n                    )\n                sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SortGroup.set_group_by_shank", "title": "<code>set_group_by_shank(nwb_file_name, references=None, omit_ref_electrode_group=False, omit_unitrode=True)</code>  <code>classmethod</code>", "text": "<p>Divides electrodes into groups based on their shank position.</p> <ul> <li>Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a   single group</li> <li>Electrodes from probes with multiple shanks (e.g. polymer probes) are   placed in one group per shank</li> <li>Bad channels are omitted</li> </ul> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>the name of the NWB file whose electrodes should be put into sorting groups</p> required <code>references</code> <code>dict</code> <p>If passed, used to set references. Otherwise, references set using original reference electrodes from config. Keys: electrode groups. Values: reference electrode.</p> <code>None</code> <code>omit_ref_electrode_group</code> <code>bool</code> <p>Optional. If True, no sort group is defined for electrode group of reference.</p> <code>False</code> <code>omit_unitrode</code> <code>bool</code> <p>Optional. If True, no sort groups are defined for unitrodes.</p> <code>True</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef set_group_by_shank(\n    cls,\n    nwb_file_name: str,\n    references: dict = None,\n    omit_ref_electrode_group=False,\n    omit_unitrode=True,\n):\n    \"\"\"Divides electrodes into groups based on their shank position.\n\n    * Electrodes from probes with 1 shank (e.g. tetrodes) are placed in a\n      single group\n    * Electrodes from probes with multiple shanks (e.g. polymer probes) are\n      placed in one group per shank\n    * Bad channels are omitted\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        the name of the NWB file whose electrodes should be put into\n        sorting groups\n    references : dict, optional\n        If passed, used to set references. Otherwise, references set using\n        original reference electrodes from config. Keys: electrode groups.\n        Values: reference electrode.\n    omit_ref_electrode_group : bool\n        Optional. If True, no sort group is defined for electrode group of\n        reference.\n    omit_unitrode : bool\n        Optional. If True, no sort groups are defined for unitrodes.\n    \"\"\"\n    # delete any current groups\n    # (SortGroup &amp; {\"nwb_file_name\": nwb_file_name}).delete()\n    # get the electrodes from this NWB file\n    electrodes = (\n        Electrode()\n        &amp; {\"nwb_file_name\": nwb_file_name}\n        &amp; {\"bad_channel\": \"False\"}\n    ).fetch()\n    e_groups = list(np.unique(electrodes[\"electrode_group_name\"]))\n    e_groups.sort(key=int)  # sort electrode groups numerically\n    sort_group = 0\n    sg_key = dict()\n    sge_key = dict()\n    sg_key[\"nwb_file_name\"] = sge_key[\"nwb_file_name\"] = nwb_file_name\n    for e_group in e_groups:\n        # for each electrode group, get a list of the unique shank numbers\n        shank_list = np.unique(\n            electrodes[\"probe_shank\"][\n                electrodes[\"electrode_group_name\"] == e_group\n            ]\n        )\n        sge_key[\"electrode_group_name\"] = e_group\n        # get the indices of all electrodes in this group / shank and set their sorting group\n        for shank in shank_list:\n            sg_key[\"sort_group_id\"] = sge_key[\"sort_group_id\"] = sort_group\n            # specify reference electrode. Use 'references' if passed, otherwise use reference from config\n            if not references:\n                shank_elect_ref = electrodes[\n                    \"original_reference_electrode\"\n                ][\n                    np.logical_and(\n                        electrodes[\"electrode_group_name\"] == e_group,\n                        electrodes[\"probe_shank\"] == shank,\n                    )\n                ]\n                if np.max(shank_elect_ref) == np.min(shank_elect_ref):\n                    sg_key[\"sort_reference_electrode_id\"] = shank_elect_ref[\n                        0\n                    ]\n                else:\n                    ValueError(\n                        f\"Error in electrode group {e_group}: reference \"\n                        + \"electrodes are not all the same\"\n                    )\n            else:\n                if e_group not in references.keys():\n                    raise Exception(\n                        f\"electrode group {e_group} not a key in \"\n                        + \"references, so cannot set reference\"\n                    )\n                else:\n                    sg_key[\"sort_reference_electrode_id\"] = references[\n                        e_group\n                    ]\n            # Insert sort group and sort group electrodes\n            reference_electrode_group = electrodes[\n                electrodes[\"electrode_id\"]\n                == sg_key[\"sort_reference_electrode_id\"]\n            ][\n                \"electrode_group_name\"\n            ]  # reference for this electrode group\n            if (\n                len(reference_electrode_group) == 1\n            ):  # unpack single reference\n                reference_electrode_group = reference_electrode_group[0]\n            elif (int(sg_key[\"sort_reference_electrode_id\"]) &gt; 0) and (\n                len(reference_electrode_group) != 1\n            ):\n                raise Exception(\n                    \"Should have found exactly one electrode group for \"\n                    + \"reference electrode, but found \"\n                    + f\"{len(reference_electrode_group)}.\"\n                )\n            if omit_ref_electrode_group and (\n                str(e_group) == str(reference_electrode_group)\n            ):\n                logger.warn(\n                    f\"Omitting electrode group {e_group} from sort groups \"\n                    + \"because contains reference.\"\n                )\n                continue\n            shank_elect = electrodes[\"electrode_id\"][\n                np.logical_and(\n                    electrodes[\"electrode_group_name\"] == e_group,\n                    electrodes[\"probe_shank\"] == shank,\n                )\n            ]\n            if (\n                omit_unitrode and len(shank_elect) == 1\n            ):  # omit unitrodes if indicated\n                logger.warn(\n                    f\"Omitting electrode group {e_group}, shank {shank} \"\n                    + \"from sort groups because unitrode.\"\n                )\n                continue\n            cls.insert1(sg_key, skip_duplicates=True)\n            for elect in shank_elect:\n                sge_key[\"electrode_id\"] = elect\n                cls.SortGroupElectrode().insert1(\n                    sge_key, skip_duplicates=True\n                )\n            sort_group += 1\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecordingSelection", "title": "<code>SpikeSortingRecordingSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SpikeSortingRecordingSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Raw voltage traces and parameters. Use `insert_selection` method to insert rows.\n    recording_id: uuid\n    ---\n    -&gt; Raw\n    -&gt; SortGroup\n    -&gt; IntervalList\n    -&gt; SpikeSortingPreprocessingParameters\n    -&gt; LabTeam\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into SpikeSortingRecordingSelection with an\n        automatically generated unique recording ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of Raw, SortGroup, IntervalList,\n            SpikeSortingPreprocessingParameters, LabTeam tables\n\n        Returns\n        -------\n            primary key of SpikeSortingRecordingSelection table\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.warn(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"recording_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecordingSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into SpikeSortingRecordingSelection with an automatically generated unique recording ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of Raw, SortGroup, IntervalList, SpikeSortingPreprocessingParameters, LabTeam tables</p> required <p>Returns:</p> Type Description <code>    primary key of SpikeSortingRecordingSelection table</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into SpikeSortingRecordingSelection with an\n    automatically generated unique recording ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of Raw, SortGroup, IntervalList,\n        SpikeSortingPreprocessingParameters, LabTeam tables\n\n    Returns\n    -------\n        primary key of SpikeSortingRecordingSelection table\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.warn(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"recording_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecording", "title": "<code>SpikeSortingRecording</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@schema\nclass SpikeSortingRecording(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    # Processed recording.\n    -&gt; SpikeSortingRecordingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40) # Object ID for the processed recording in NWB file\n    \"\"\"\n\n    def make(self, key):\n        # DO:\n        # - get valid times for sort interval\n        # - proprocess recording\n        # - write recording to NWB file\n        sort_interval_valid_times = self._get_sort_interval_valid_times(key)\n        recording, timestamps = self._get_preprocessed_recording(key)\n        recording_nwb_file_name, recording_object_id = _write_recording_to_nwb(\n            recording,\n            timestamps,\n            (SpikeSortingRecordingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        )\n        key[\"analysis_file_name\"] = recording_nwb_file_name\n        key[\"object_id\"] = recording_object_id\n\n        # INSERT:\n        # - valid times into IntervalList\n        # - analysis NWB file holding processed recording into AnalysisNwbfile\n        # - entry into SpikeSortingRecording\n        IntervalList.insert1(\n            {\n                \"nwb_file_name\": (SpikeSortingRecordingSelection &amp; key).fetch1(\n                    \"nwb_file_name\"\n                ),\n                \"interval_list_name\": key[\"recording_id\"],\n                \"valid_times\": sort_interval_valid_times,\n                \"pipeline\": \"spikesorting_recording_v1\",\n            }\n        )\n        AnalysisNwbfile().add(\n            (SpikeSortingRecordingSelection &amp; key).fetch1(\"nwb_file_name\"),\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key)\n\n    @classmethod\n    def get_recording(cls, key: dict) -&gt; si.BaseRecording:\n        \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSorting table\n        \"\"\"\n\n        analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        recording = se.read_nwb_recording(\n            analysis_file_abs_path, load_time_vector=True\n        )\n\n        return recording\n\n    @staticmethod\n    def _get_recording_timestamps(recording):\n        if recording.get_num_segments() &gt; 1:\n            frames_per_segment = [0]\n            for i in range(recording.get_num_segments()):\n                frames_per_segment.append(\n                    recording.get_num_frames(segment_index=i)\n                )\n\n            cumsum_frames = np.cumsum(frames_per_segment)\n            total_frames = np.sum(frames_per_segment)\n\n            timestamps = np.zeros((total_frames,))\n            for i in range(recording.get_num_segments()):\n                timestamps[cumsum_frames[i] : cumsum_frames[i + 1]] = (\n                    recording.get_times(segment_index=i)\n                )\n        else:\n            timestamps = recording.get_times()\n        return timestamps\n\n    def _get_sort_interval_valid_times(self, key: dict):\n        \"\"\"Identifies the intersection between sort interval specified by the user\n        and the valid times (times for which neural data exist, excluding e.g. dropped packets).\n\n        Parameters\n        ----------\n        key: dict\n            primary key of SpikeSortingRecordingSelection table\n\n        Returns\n        -------\n        sort_interval_valid_times: ndarray of tuples\n            (start, end) times for valid intervals in the sort interval\n\n        \"\"\"\n        # FETCH: - sort interval - valid times - preprocessing parameters\n        nwb_file_name, sort_interval_name, params = (\n            SpikeSortingPreprocessingParameters * SpikeSortingRecordingSelection\n            &amp; key\n        ).fetch1(\"nwb_file_name\", \"interval_list_name\", \"preproc_params\")\n\n        sort_interval = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": sort_interval_name,\n            }\n        ).fetch1(\"valid_times\")\n\n        valid_interval_times = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"interval_list_name\": \"raw data valid times\",\n            }\n        ).fetch1(\"valid_times\")\n\n        # DO: - take intersection between sort interval and valid times\n        return interval_list_intersect(\n            sort_interval,\n            valid_interval_times,\n            min_length=params[\"min_segment_length\"],\n        )\n\n    def _get_preprocessed_recording(self, key: dict):\n        \"\"\"Filters and references a recording.\n\n        - Loads the NWB file created during insertion as a spikeinterface Recording\n        - Slices recording in time (interval) and space (channels);\n          recording chunks from disjoint intervals are concatenated\n        - Applies referencing and bandpass filtering\n\n        Parameters\n        ----------\n        key: dict\n            primary key of SpikeSortingRecordingSelection table\n\n        Returns\n        -------\n        recording: si.Recording\n        \"\"\"\n        # FETCH:\n        # - full path to NWB file\n        # - channels to be included in the sort\n        # - the reference channel\n        # - probe type\n        # - filter parameters\n        nwb_file_name = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"nwb_file_name\"\n        )\n        sort_group_id = (SpikeSortingRecordingSelection &amp; key).fetch1(\n            \"sort_group_id\"\n        )\n        nwb_file_abs_path = Nwbfile().get_abs_path(nwb_file_name)\n        channel_ids = (\n            SortGroup.SortGroupElectrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"sort_group_id\": sort_group_id,\n            }\n        ).fetch(\"electrode_id\")\n        ref_channel_id = (\n            SortGroup\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n                \"sort_group_id\": sort_group_id,\n            }\n        ).fetch1(\"sort_reference_electrode_id\")\n        recording_channel_ids = np.setdiff1d(channel_ids, ref_channel_id)\n        all_channel_ids = np.unique(np.append(channel_ids, ref_channel_id))\n\n        probe_type_by_channel = []\n        electrode_group_by_channel = []\n        for channel_id in channel_ids:\n            probe_type_by_channel.append(\n                (\n                    Electrode * Probe\n                    &amp; {\n                        \"nwb_file_name\": nwb_file_name,\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"probe_type\")\n            )\n            electrode_group_by_channel.append(\n                (\n                    Electrode\n                    &amp; {\n                        \"nwb_file_name\": nwb_file_name,\n                        \"electrode_id\": channel_id,\n                    }\n                ).fetch1(\"electrode_group_name\")\n            )\n        probe_type = np.unique(probe_type_by_channel)\n        filter_params = (\n            SpikeSortingPreprocessingParameters * SpikeSortingRecordingSelection\n            &amp; key\n        ).fetch1(\"preproc_params\")\n\n        # DO:\n        # - load NWB file as a spikeinterface Recording\n        # - slice the recording object in time and channels\n        # - apply referencing depending on the option chosen by the user\n        # - apply bandpass filter\n        # - set probe to recording\n        recording = se.read_nwb_recording(\n            nwb_file_abs_path, load_time_vector=True\n        )\n        all_timestamps = recording.get_times()\n\n        # TODO: make sure the following works for recordings that don't have explicit timestamps\n        valid_sort_times = self._get_sort_interval_valid_times(key)\n        valid_sort_times_indices = _consolidate_intervals(\n            valid_sort_times, all_timestamps\n        )\n\n        # slice in time; concatenate disjoint sort intervals\n        if len(valid_sort_times_indices) &gt; 1:\n            recordings_list = []\n            timestamps = []\n            for interval_indices in valid_sort_times_indices:\n                recording_single = recording.frame_slice(\n                    start_frame=interval_indices[0],\n                    end_frame=interval_indices[1],\n                )\n                recordings_list.append(recording_single)\n                timestamps.extend(\n                    all_timestamps[interval_indices[0] : interval_indices[1]]\n                )\n            recording = si.concatenate_recordings(recordings_list)\n        else:\n            recording = recording.frame_slice(\n                start_frame=valid_sort_times_indices[0][0],\n                end_frame=valid_sort_times_indices[0][1],\n            )\n            timestamps = all_timestamps[\n                valid_sort_times_indices[0][0] : valid_sort_times_indices[0][1]\n            ]\n\n        # slice in channels; include ref channel in first slice, then exclude it in second slice\n        if ref_channel_id &gt;= 0:\n            recording = recording.channel_slice(channel_ids=all_channel_ids)\n            recording = si.preprocessing.common_reference(\n                recording,\n                reference=\"single\",\n                ref_channel_ids=ref_channel_id,\n                dtype=np.float64,\n            )\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n        elif ref_channel_id == -2:\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n            recording = si.preprocessing.common_reference(\n                recording,\n                reference=\"global\",\n                operator=\"median\",\n                dtype=np.float64,\n            )\n        elif ref_channel_id == -1:\n            recording = recording.channel_slice(\n                channel_ids=recording_channel_ids\n            )\n        else:\n            raise ValueError(\n                \"Invalid reference channel ID. Use -1 to skip referencing. Use \"\n                + \"-2 to reference via global median. Use positive integer to \"\n                + \"reference to a specific channel.\"\n            )\n\n        recording = si.preprocessing.bandpass_filter(\n            recording,\n            freq_min=filter_params[\"frequency_min\"],\n            freq_max=filter_params[\"frequency_max\"],\n            dtype=np.float64,\n        )\n\n        # if the sort group is a tetrode, change the channel location\n        # (necessary because the channel location for tetrodes are not set properly)\n        if (\n            len(probe_type) == 1\n            and probe_type[0] == \"tetrode_12.5\"\n            and len(recording_channel_ids) == 4\n            and len(np.unique(electrode_group_by_channel)) == 1\n        ):\n            tetrode = pi.Probe(ndim=2)\n            position = [[0, 0], [0, 12.5], [12.5, 0], [12.5, 12.5]]\n            tetrode.set_contacts(\n                position, shapes=\"circle\", shape_params={\"radius\": 6.25}\n            )\n            tetrode.set_contact_ids(channel_ids)\n            tetrode.set_device_channel_indices(np.arange(4))\n            recording = recording.set_probe(tetrode, in_place=True)\n\n        return recording, np.asarray(timestamps)\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeSortingRecording.get_recording", "title": "<code>get_recording(key)</code>  <code>classmethod</code>", "text": "<p>Get recording related to this curation as spikeinterface BaseRecording</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSorting table</p> required Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>@classmethod\ndef get_recording(cls, key: dict) -&gt; si.BaseRecording:\n    \"\"\"Get recording related to this curation as spikeinterface BaseRecording\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSorting table\n    \"\"\"\n\n    analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    recording = se.read_nwb_recording(\n        analysis_file_abs_path, load_time_vector=True\n    )\n\n    return recording\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeInterfaceRecordingDataChunkIterator", "title": "<code>SpikeInterfaceRecordingDataChunkIterator</code>", "text": "<p>             Bases: <code>GenericDataChunkIterator</code></p> <p>DataChunkIterator specifically for use on RecordingExtractor objects.</p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>class SpikeInterfaceRecordingDataChunkIterator(GenericDataChunkIterator):\n    \"\"\"DataChunkIterator specifically for use on RecordingExtractor objects.\"\"\"\n\n    def __init__(\n        self,\n        recording: si.BaseRecording,\n        segment_index: int = 0,\n        return_scaled: bool = False,\n        buffer_gb: Optional[float] = None,\n        buffer_shape: Optional[tuple] = None,\n        chunk_mb: Optional[float] = None,\n        chunk_shape: Optional[tuple] = None,\n        display_progress: bool = False,\n        progress_bar_options: Optional[dict] = None,\n    ):\n        \"\"\"\n        Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n        Parameters\n        ----------\n        recording : si.BaseRecording\n            The SpikeInterfaceRecording object which handles the data access.\n        segment_index : int, optional\n            The recording segment to iterate on.\n            Defaults to 0.\n        return_scaled : bool, optional\n            Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n            Defaults to False.\n        buffer_gb : float, optional\n            The upper bound on size in gigabytes (GB) of each selection from the iteration.\n            The buffer_shape will be set implicitly by this argument.\n            Cannot be set if `buffer_shape` is also specified.\n            The default is 1GB.\n        buffer_shape : tuple, optional\n            Manual specification of buffer shape to return on each iteration.\n            Must be a multiple of chunk_shape along each axis.\n            Cannot be set if `buffer_gb` is also specified.\n            The default is None.\n        chunk_mb : float, optional\n            The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n            The chunk_shape will be set implicitly by this argument.\n            Cannot be set if `chunk_shape` is also specified.\n            The default is 1MB, as recommended by the HDF5 group. For more details, see\n            https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n        chunk_shape : tuple, optional\n            Manual specification of the internal chunk shape for the HDF5 dataset.\n            Cannot be set if `chunk_mb` is also specified.\n            The default is None.\n        display_progress : bool, optional\n            Display a progress bar with iteration rate and estimated completion time.\n        progress_bar_options : dict, optional\n            Dictionary of keyword arguments to be passed directly to tqdm.\n            See https://github.com/tqdm/tqdm#parameters for options.\n        \"\"\"\n        self.recording = recording\n        self.segment_index = segment_index\n        self.return_scaled = return_scaled\n        self.channel_ids = recording.get_channel_ids()\n        super().__init__(\n            buffer_gb=buffer_gb,\n            buffer_shape=buffer_shape,\n            chunk_mb=chunk_mb,\n            chunk_shape=chunk_shape,\n            display_progress=display_progress,\n            progress_bar_options=progress_bar_options,\n        )\n\n    def _get_data(self, selection: Tuple[slice]) -&gt; Iterable:\n        return self.recording.get_traces(\n            segment_index=self.segment_index,\n            channel_ids=self.channel_ids[selection[1]],\n            start_frame=selection[0].start,\n            end_frame=selection[0].stop,\n            return_scaled=self.return_scaled,\n        )\n\n    def _get_dtype(self):\n        return self.recording.get_dtype()\n\n    def _get_maxshape(self):\n        return (\n            self.recording.get_num_samples(segment_index=self.segment_index),\n            self.recording.get_num_channels(),\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.SpikeInterfaceRecordingDataChunkIterator.__init__", "title": "<code>__init__(recording, segment_index=0, return_scaled=False, buffer_gb=None, buffer_shape=None, chunk_mb=None, chunk_shape=None, display_progress=False, progress_bar_options=None)</code>", "text": "<p>Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>BaseRecording</code> <p>The SpikeInterfaceRecording object which handles the data access.</p> required <code>segment_index</code> <code>int</code> <p>The recording segment to iterate on. Defaults to 0.</p> <code>0</code> <code>return_scaled</code> <code>bool</code> <p>Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False). Defaults to False.</p> <code>False</code> <code>buffer_gb</code> <code>float</code> <p>The upper bound on size in gigabytes (GB) of each selection from the iteration. The buffer_shape will be set implicitly by this argument. Cannot be set if <code>buffer_shape</code> is also specified. The default is 1GB.</p> <code>None</code> <code>buffer_shape</code> <code>tuple</code> <p>Manual specification of buffer shape to return on each iteration. Must be a multiple of chunk_shape along each axis. Cannot be set if <code>buffer_gb</code> is also specified. The default is None.</p> <code>None</code> <code>chunk_mb</code> <code>float</code> <p>The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset. The chunk_shape will be set implicitly by this argument. Cannot be set if <code>chunk_shape</code> is also specified. The default is 1MB, as recommended by the HDF5 group. For more details, see https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf</p> <code>None</code> <code>chunk_shape</code> <code>tuple</code> <p>Manual specification of the internal chunk shape for the HDF5 dataset. Cannot be set if <code>chunk_mb</code> is also specified. The default is None.</p> <code>None</code> <code>display_progress</code> <code>bool</code> <p>Display a progress bar with iteration rate and estimated completion time.</p> <code>False</code> <code>progress_bar_options</code> <code>dict</code> <p>Dictionary of keyword arguments to be passed directly to tqdm. See https://github.com/tqdm/tqdm#parameters for options.</p> <code>None</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>def __init__(\n    self,\n    recording: si.BaseRecording,\n    segment_index: int = 0,\n    return_scaled: bool = False,\n    buffer_gb: Optional[float] = None,\n    buffer_shape: Optional[tuple] = None,\n    chunk_mb: Optional[float] = None,\n    chunk_shape: Optional[tuple] = None,\n    display_progress: bool = False,\n    progress_bar_options: Optional[dict] = None,\n):\n    \"\"\"\n    Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n    Parameters\n    ----------\n    recording : si.BaseRecording\n        The SpikeInterfaceRecording object which handles the data access.\n    segment_index : int, optional\n        The recording segment to iterate on.\n        Defaults to 0.\n    return_scaled : bool, optional\n        Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n        Defaults to False.\n    buffer_gb : float, optional\n        The upper bound on size in gigabytes (GB) of each selection from the iteration.\n        The buffer_shape will be set implicitly by this argument.\n        Cannot be set if `buffer_shape` is also specified.\n        The default is 1GB.\n    buffer_shape : tuple, optional\n        Manual specification of buffer shape to return on each iteration.\n        Must be a multiple of chunk_shape along each axis.\n        Cannot be set if `buffer_gb` is also specified.\n        The default is None.\n    chunk_mb : float, optional\n        The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n        The chunk_shape will be set implicitly by this argument.\n        Cannot be set if `chunk_shape` is also specified.\n        The default is 1MB, as recommended by the HDF5 group. For more details, see\n        https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n    chunk_shape : tuple, optional\n        Manual specification of the internal chunk shape for the HDF5 dataset.\n        Cannot be set if `chunk_mb` is also specified.\n        The default is None.\n    display_progress : bool, optional\n        Display a progress bar with iteration rate and estimated completion time.\n    progress_bar_options : dict, optional\n        Dictionary of keyword arguments to be passed directly to tqdm.\n        See https://github.com/tqdm/tqdm#parameters for options.\n    \"\"\"\n    self.recording = recording\n    self.segment_index = segment_index\n    self.return_scaled = return_scaled\n    self.channel_ids = recording.get_channel_ids()\n    super().__init__(\n        buffer_gb=buffer_gb,\n        buffer_shape=buffer_shape,\n        chunk_mb=chunk_mb,\n        chunk_shape=chunk_shape,\n        display_progress=display_progress,\n        progress_bar_options=progress_bar_options,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.TimestampsDataChunkIterator", "title": "<code>TimestampsDataChunkIterator</code>", "text": "<p>             Bases: <code>GenericDataChunkIterator</code></p> <p>DataChunkIterator specifically for use on RecordingExtractor objects.</p> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>class TimestampsDataChunkIterator(GenericDataChunkIterator):\n    \"\"\"DataChunkIterator specifically for use on RecordingExtractor objects.\"\"\"\n\n    def __init__(\n        self,\n        recording: si.BaseRecording,\n        segment_index: int = 0,\n        return_scaled: bool = False,\n        buffer_gb: Optional[float] = None,\n        buffer_shape: Optional[tuple] = None,\n        chunk_mb: Optional[float] = None,\n        chunk_shape: Optional[tuple] = None,\n        display_progress: bool = False,\n        progress_bar_options: Optional[dict] = None,\n    ):\n        \"\"\"\n        Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n        Parameters\n        ----------\n        recording : SpikeInterfaceRecording\n            The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.\n        segment_index : int, optional\n            The recording segment to iterate on.\n            Defaults to 0.\n        return_scaled : bool, optional\n            Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n            Defaults to False.\n        buffer_gb : float, optional\n            The upper bound on size in gigabytes (GB) of each selection from the iteration.\n            The buffer_shape will be set implicitly by this argument.\n            Cannot be set if `buffer_shape` is also specified.\n            The default is 1GB.\n        buffer_shape : tuple, optional\n            Manual specification of buffer shape to return on each iteration.\n            Must be a multiple of chunk_shape along each axis.\n            Cannot be set if `buffer_gb` is also specified.\n            The default is None.\n        chunk_mb : float, optional\n            The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n            The chunk_shape will be set implicitly by this argument.\n            Cannot be set if `chunk_shape` is also specified.\n            The default is 1MB, as recommended by the HDF5 group. For more details, see\n            https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n        chunk_shape : tuple, optional\n            Manual specification of the internal chunk shape for the HDF5 dataset.\n            Cannot be set if `chunk_mb` is also specified.\n            The default is None.\n        display_progress : bool, optional\n            Display a progress bar with iteration rate and estimated completion time.\n        progress_bar_options : dict, optional\n            Dictionary of keyword arguments to be passed directly to tqdm.\n            See https://github.com/tqdm/tqdm#parameters for options.\n        \"\"\"\n        self.recording = recording\n        self.segment_index = segment_index\n        self.return_scaled = return_scaled\n        self.channel_ids = recording.get_channel_ids()\n        super().__init__(\n            buffer_gb=buffer_gb,\n            buffer_shape=buffer_shape,\n            chunk_mb=chunk_mb,\n            chunk_shape=chunk_shape,\n            display_progress=display_progress,\n            progress_bar_options=progress_bar_options,\n        )\n\n    # change channel id to always be first channel\n    def _get_data(self, selection: Tuple[slice]) -&gt; Iterable:\n        return self.recording.get_traces(\n            segment_index=self.segment_index,\n            channel_ids=[0],\n            start_frame=selection[0].start,\n            end_frame=selection[0].stop,\n            return_scaled=self.return_scaled,\n        )\n\n    def _get_dtype(self):\n        return self.recording.get_dtype()\n\n    # remove the last dim for the timestamps since it is always just a 1D vector\n    def _get_maxshape(self):\n        return (\n            self.recording.get_num_samples(segment_index=self.segment_index),\n        )\n</code></pre>"}, {"location": "api/spikesorting/v1/recording/#src.spyglass.spikesorting.v1.recording.TimestampsDataChunkIterator.__init__", "title": "<code>__init__(recording, segment_index=0, return_scaled=False, buffer_gb=None, buffer_shape=None, chunk_mb=None, chunk_shape=None, display_progress=False, progress_bar_options=None)</code>", "text": "<p>Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.</p> <p>Parameters:</p> Name Type Description Default <code>recording</code> <code>SpikeInterfaceRecording</code> <p>The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.</p> required <code>segment_index</code> <code>int</code> <p>The recording segment to iterate on. Defaults to 0.</p> <code>0</code> <code>return_scaled</code> <code>bool</code> <p>Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False). Defaults to False.</p> <code>False</code> <code>buffer_gb</code> <code>float</code> <p>The upper bound on size in gigabytes (GB) of each selection from the iteration. The buffer_shape will be set implicitly by this argument. Cannot be set if <code>buffer_shape</code> is also specified. The default is 1GB.</p> <code>None</code> <code>buffer_shape</code> <code>tuple</code> <p>Manual specification of buffer shape to return on each iteration. Must be a multiple of chunk_shape along each axis. Cannot be set if <code>buffer_gb</code> is also specified. The default is None.</p> <code>None</code> <code>chunk_mb</code> <code>float</code> <p>The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset. The chunk_shape will be set implicitly by this argument. Cannot be set if <code>chunk_shape</code> is also specified. The default is 1MB, as recommended by the HDF5 group. For more details, see https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf</p> <code>None</code> <code>chunk_shape</code> <code>tuple</code> <p>Manual specification of the internal chunk shape for the HDF5 dataset. Cannot be set if <code>chunk_mb</code> is also specified. The default is None.</p> <code>None</code> <code>display_progress</code> <code>bool</code> <p>Display a progress bar with iteration rate and estimated completion time.</p> <code>False</code> <code>progress_bar_options</code> <code>dict</code> <p>Dictionary of keyword arguments to be passed directly to tqdm. See https://github.com/tqdm/tqdm#parameters for options.</p> <code>None</code> Source code in <code>src/spyglass/spikesorting/v1/recording.py</code> <pre><code>def __init__(\n    self,\n    recording: si.BaseRecording,\n    segment_index: int = 0,\n    return_scaled: bool = False,\n    buffer_gb: Optional[float] = None,\n    buffer_shape: Optional[tuple] = None,\n    chunk_mb: Optional[float] = None,\n    chunk_shape: Optional[tuple] = None,\n    display_progress: bool = False,\n    progress_bar_options: Optional[dict] = None,\n):\n    \"\"\"\n    Initialize an Iterable object which returns DataChunks with data and their selections on each iteration.\n\n    Parameters\n    ----------\n    recording : SpikeInterfaceRecording\n        The SpikeInterfaceRecording object (RecordingExtractor or BaseRecording) which handles the data access.\n    segment_index : int, optional\n        The recording segment to iterate on.\n        Defaults to 0.\n    return_scaled : bool, optional\n        Whether to return the trace data in scaled units (uV, if True) or in the raw data type (if False).\n        Defaults to False.\n    buffer_gb : float, optional\n        The upper bound on size in gigabytes (GB) of each selection from the iteration.\n        The buffer_shape will be set implicitly by this argument.\n        Cannot be set if `buffer_shape` is also specified.\n        The default is 1GB.\n    buffer_shape : tuple, optional\n        Manual specification of buffer shape to return on each iteration.\n        Must be a multiple of chunk_shape along each axis.\n        Cannot be set if `buffer_gb` is also specified.\n        The default is None.\n    chunk_mb : float, optional\n        The upper bound on size in megabytes (MB) of the internal chunk for the HDF5 dataset.\n        The chunk_shape will be set implicitly by this argument.\n        Cannot be set if `chunk_shape` is also specified.\n        The default is 1MB, as recommended by the HDF5 group. For more details, see\n        https://support.hdfgroup.org/HDF5/doc/TechNotes/TechNote-HDF5-ImprovingIOPerformanceCompressedDatasets.pdf\n    chunk_shape : tuple, optional\n        Manual specification of the internal chunk shape for the HDF5 dataset.\n        Cannot be set if `chunk_mb` is also specified.\n        The default is None.\n    display_progress : bool, optional\n        Display a progress bar with iteration rate and estimated completion time.\n    progress_bar_options : dict, optional\n        Dictionary of keyword arguments to be passed directly to tqdm.\n        See https://github.com/tqdm/tqdm#parameters for options.\n    \"\"\"\n    self.recording = recording\n    self.segment_index = segment_index\n    self.return_scaled = return_scaled\n    self.channel_ids = recording.get_channel_ids()\n    super().__init__(\n        buffer_gb=buffer_gb,\n        buffer_shape=buffer_shape,\n        chunk_mb=chunk_mb,\n        chunk_shape=chunk_shape,\n        display_progress=display_progress,\n        progress_bar_options=progress_bar_options,\n    )\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/", "title": "sorting.py", "text": ""}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSortingSelection", "title": "<code>SpikeSortingSelection</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Manual</code></p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@schema\nclass SpikeSortingSelection(SpyglassMixin, dj.Manual):\n    definition = \"\"\"\n    # Processed recording and spike sorting parameters. Use `insert_selection` method to insert rows.\n    sorting_id: uuid\n    ---\n    -&gt; SpikeSortingRecording\n    -&gt; SpikeSorterParameters\n    -&gt; IntervalList\n    \"\"\"\n\n    @classmethod\n    def insert_selection(cls, key: dict):\n        \"\"\"Insert a row into SpikeSortingSelection with an\n        automatically generated unique sorting ID as the sole primary key.\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables\n\n        Returns\n        -------\n        sorting_id : uuid\n            the unique sorting ID serving as primary key for SpikeSorting\n        \"\"\"\n        query = cls &amp; key\n        if query:\n            logger.info(\"Similar row(s) already inserted.\")\n            return query.fetch(as_dict=True)\n        key[\"sorting_id\"] = uuid.uuid4()\n        cls.insert1(key, skip_duplicates=True)\n        return key\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSortingSelection.insert_selection", "title": "<code>insert_selection(key)</code>  <code>classmethod</code>", "text": "<p>Insert a row into SpikeSortingSelection with an automatically generated unique sorting ID as the sole primary key.</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables</p> required <p>Returns:</p> Name Type Description <code>sorting_id</code> <code>uuid</code> <p>the unique sorting ID serving as primary key for SpikeSorting</p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@classmethod\ndef insert_selection(cls, key: dict):\n    \"\"\"Insert a row into SpikeSortingSelection with an\n    automatically generated unique sorting ID as the sole primary key.\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSortingRecording, SpikeSorterParameters, IntervalList tables\n\n    Returns\n    -------\n    sorting_id : uuid\n        the unique sorting ID serving as primary key for SpikeSorting\n    \"\"\"\n    query = cls &amp; key\n    if query:\n        logger.info(\"Similar row(s) already inserted.\")\n        return query.fetch(as_dict=True)\n    key[\"sorting_id\"] = uuid.uuid4()\n    cls.insert1(key, skip_duplicates=True)\n    return key\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting", "title": "<code>SpikeSorting</code>", "text": "<p>             Bases: <code>SpyglassMixin</code>, <code>Computed</code></p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@schema\nclass SpikeSorting(SpyglassMixin, dj.Computed):\n    definition = \"\"\"\n    -&gt; SpikeSortingSelection\n    ---\n    -&gt; AnalysisNwbfile\n    object_id: varchar(40)          # Object ID for the sorting in NWB file\n    time_of_sort: int               # in Unix time, to the nearest second\n    \"\"\"\n\n    def make(self, key: dict):\n        \"\"\"Runs spike sorting on the data and parameters specified by the\n        SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n        \"\"\"\n        # FETCH:\n        # - information about the recording\n        # - artifact free intervals\n        # - spike sorter and sorter params\n\n        recording_key = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1()\n        artifact_removed_intervals = (\n            IntervalList\n            &amp; {\n                \"nwb_file_name\": (SpikeSortingSelection &amp; key).fetch1(\n                    \"nwb_file_name\"\n                ),\n                \"interval_list_name\": (SpikeSortingSelection &amp; key).fetch1(\n                    \"interval_list_name\"\n                ),\n            }\n        ).fetch1(\"valid_times\")\n        sorter, sorter_params = (\n            SpikeSorterParameters * SpikeSortingSelection &amp; key\n        ).fetch1(\"sorter\", \"sorter_params\")\n\n        # DO:\n        # - load recording\n        # - concatenate artifact removed intervals\n        # - run spike sorting\n        # - save output to NWB file\n        recording_analysis_nwb_file_abs_path = AnalysisNwbfile.get_abs_path(\n            recording_key[\"analysis_file_name\"]\n        )\n        recording = se.read_nwb_recording(\n            recording_analysis_nwb_file_abs_path, load_time_vector=True\n        )\n\n        timestamps = recording.get_times()\n\n        artifact_removed_intervals_ind = _consolidate_intervals(\n            artifact_removed_intervals, timestamps\n        )\n\n        # if the artifact removed intervals do not span the entire time range\n        if (\n            (len(artifact_removed_intervals_ind) &gt; 1)\n            or (artifact_removed_intervals_ind[0][0] &gt; 0)\n            or (artifact_removed_intervals_ind[-1][1] &lt; len(timestamps))\n        ):\n            # set the artifact intervals to zero\n            list_triggers = []\n            if artifact_removed_intervals_ind[0][0] &gt; 0:\n                list_triggers.append(\n                    np.array([0, artifact_removed_intervals_ind[0][0]])\n                )\n            for interval_ind in range(len(artifact_removed_intervals_ind) - 1):\n                list_triggers.append(\n                    np.arange(\n                        (artifact_removed_intervals_ind[interval_ind][1] + 1),\n                        artifact_removed_intervals_ind[interval_ind + 1][0],\n                    )\n                )\n            if artifact_removed_intervals_ind[-1][1] &lt; len(timestamps):\n                list_triggers.append(\n                    np.array(\n                        [\n                            artifact_removed_intervals_ind[-1][1],\n                            len(timestamps) - 1,\n                        ]\n                    )\n                )\n\n            list_triggers = [list(np.concatenate(list_triggers))]\n            recording = sip.remove_artifacts(\n                recording=recording,\n                list_triggers=list_triggers,\n                ms_before=None,\n                ms_after=None,\n                mode=\"zeros\",\n            )\n\n        if sorter == \"clusterless_thresholder\":\n            # need to remove tempdir and whiten from sorter_params\n            sorter_params.pop(\"tempdir\", None)\n            sorter_params.pop(\"whiten\", None)\n            sorter_params.pop(\"outputs\", None)\n\n            # Detect peaks for clusterless decoding\n            detected_spikes = detect_peaks(recording, **sorter_params)\n            sorting = si.NumpySorting.from_times_labels(\n                times_list=detected_spikes[\"sample_index\"],\n                labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n                sampling_frequency=recording.get_sampling_frequency(),\n            )\n        else:\n            # Specify tempdir (expected by some sorters like mountainsort4)\n            sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n            sorter_params[\"tempdir\"] = sorter_temp_dir.name\n            os.chmod(sorter_params[\"tempdir\"], 0o777)\n\n            if sorter == \"mountainsort5\":\n                _ = sorter_params.pop(\"tempdir\", None)\n\n            # if whitening is specified in sorter params, apply whitening separately\n            # prior to sorting and turn off \"sorter whitening\"\n            if sorter_params.get(\"whiten\", False):\n                recording = sip.whiten(recording, dtype=np.float64)\n                sorter_params[\"whiten\"] = False\n\n            common_sorter_items = {\n                \"sorter_name\": sorter,\n                \"recording\": recording,\n                \"output_folder\": sorter_temp_dir.name,\n                \"remove_existing_folder\": True,\n            }\n\n            if sorter.lower() in [\"kilosort2_5\", \"kilosort3\", \"ironclust\"]:\n                sorter_params = {\n                    k: v\n                    for k, v in sorter_params.items()\n                    if k\n                    not in [\"tempdir\", \"mp_context\", \"max_threads_per_process\"]\n                }\n                sorting = sis.run_sorter(\n                    **common_sorter_items,\n                    singularity_image=True,\n                    **sorter_params,\n                )\n            else:\n                sorting = sis.run_sorter(\n                    **common_sorter_items,\n                    **sorter_params,\n                )\n        key[\"time_of_sort\"] = int(time.time())\n        sorting = sic.remove_excess_spikes(sorting, recording)\n        key[\"analysis_file_name\"], key[\"object_id\"] = _write_sorting_to_nwb(\n            sorting,\n            timestamps,\n            artifact_removed_intervals,\n            (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        )\n\n        # INSERT\n        # - new entry to AnalysisNwbfile\n        # - new entry to SpikeSorting\n        AnalysisNwbfile().add(\n            (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n            key[\"analysis_file_name\"],\n        )\n        self.insert1(key, skip_duplicates=True)\n\n    @classmethod\n    def get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n        \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n        Parameters\n        ----------\n        key : dict\n            primary key of SpikeSorting\n\n        Returns\n        -------\n        sorting : si.BaseSorting\n\n        \"\"\"\n\n        recording_id = (\n            SpikeSortingRecording * SpikeSortingSelection &amp; key\n        ).fetch1(\"recording_id\")\n        recording = SpikeSortingRecording.get_recording(\n            {\"recording_id\": recording_id}\n        )\n        sampling_frequency = recording.get_sampling_frequency()\n        analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n        analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n            analysis_file_name\n        )\n        with pynwb.NWBHDF5IO(\n            analysis_file_abs_path, \"r\", load_namespaces=True\n        ) as io:\n            nwbf = io.read()\n            units = nwbf.units.to_dataframe()\n        units_dict_list = [\n            {\n                unit_id: np.searchsorted(recording.get_times(), spike_times)\n                for unit_id, spike_times in zip(\n                    units.index, units[\"spike_times\"]\n                )\n            }\n        ]\n\n        sorting = si.NumpySorting.from_unit_dict(\n            units_dict_list, sampling_frequency=sampling_frequency\n        )\n\n        return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting.make", "title": "<code>make(key)</code>", "text": "<p>Runs spike sorting on the data and parameters specified by the SpikeSortingSelection table and inserts a new entry to SpikeSorting table.</p> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>def make(self, key: dict):\n    \"\"\"Runs spike sorting on the data and parameters specified by the\n    SpikeSortingSelection table and inserts a new entry to SpikeSorting table.\n    \"\"\"\n    # FETCH:\n    # - information about the recording\n    # - artifact free intervals\n    # - spike sorter and sorter params\n\n    recording_key = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1()\n    artifact_removed_intervals = (\n        IntervalList\n        &amp; {\n            \"nwb_file_name\": (SpikeSortingSelection &amp; key).fetch1(\n                \"nwb_file_name\"\n            ),\n            \"interval_list_name\": (SpikeSortingSelection &amp; key).fetch1(\n                \"interval_list_name\"\n            ),\n        }\n    ).fetch1(\"valid_times\")\n    sorter, sorter_params = (\n        SpikeSorterParameters * SpikeSortingSelection &amp; key\n    ).fetch1(\"sorter\", \"sorter_params\")\n\n    # DO:\n    # - load recording\n    # - concatenate artifact removed intervals\n    # - run spike sorting\n    # - save output to NWB file\n    recording_analysis_nwb_file_abs_path = AnalysisNwbfile.get_abs_path(\n        recording_key[\"analysis_file_name\"]\n    )\n    recording = se.read_nwb_recording(\n        recording_analysis_nwb_file_abs_path, load_time_vector=True\n    )\n\n    timestamps = recording.get_times()\n\n    artifact_removed_intervals_ind = _consolidate_intervals(\n        artifact_removed_intervals, timestamps\n    )\n\n    # if the artifact removed intervals do not span the entire time range\n    if (\n        (len(artifact_removed_intervals_ind) &gt; 1)\n        or (artifact_removed_intervals_ind[0][0] &gt; 0)\n        or (artifact_removed_intervals_ind[-1][1] &lt; len(timestamps))\n    ):\n        # set the artifact intervals to zero\n        list_triggers = []\n        if artifact_removed_intervals_ind[0][0] &gt; 0:\n            list_triggers.append(\n                np.array([0, artifact_removed_intervals_ind[0][0]])\n            )\n        for interval_ind in range(len(artifact_removed_intervals_ind) - 1):\n            list_triggers.append(\n                np.arange(\n                    (artifact_removed_intervals_ind[interval_ind][1] + 1),\n                    artifact_removed_intervals_ind[interval_ind + 1][0],\n                )\n            )\n        if artifact_removed_intervals_ind[-1][1] &lt; len(timestamps):\n            list_triggers.append(\n                np.array(\n                    [\n                        artifact_removed_intervals_ind[-1][1],\n                        len(timestamps) - 1,\n                    ]\n                )\n            )\n\n        list_triggers = [list(np.concatenate(list_triggers))]\n        recording = sip.remove_artifacts(\n            recording=recording,\n            list_triggers=list_triggers,\n            ms_before=None,\n            ms_after=None,\n            mode=\"zeros\",\n        )\n\n    if sorter == \"clusterless_thresholder\":\n        # need to remove tempdir and whiten from sorter_params\n        sorter_params.pop(\"tempdir\", None)\n        sorter_params.pop(\"whiten\", None)\n        sorter_params.pop(\"outputs\", None)\n\n        # Detect peaks for clusterless decoding\n        detected_spikes = detect_peaks(recording, **sorter_params)\n        sorting = si.NumpySorting.from_times_labels(\n            times_list=detected_spikes[\"sample_index\"],\n            labels_list=np.zeros(len(detected_spikes), dtype=np.int),\n            sampling_frequency=recording.get_sampling_frequency(),\n        )\n    else:\n        # Specify tempdir (expected by some sorters like mountainsort4)\n        sorter_temp_dir = tempfile.TemporaryDirectory(dir=temp_dir)\n        sorter_params[\"tempdir\"] = sorter_temp_dir.name\n        os.chmod(sorter_params[\"tempdir\"], 0o777)\n\n        if sorter == \"mountainsort5\":\n            _ = sorter_params.pop(\"tempdir\", None)\n\n        # if whitening is specified in sorter params, apply whitening separately\n        # prior to sorting and turn off \"sorter whitening\"\n        if sorter_params.get(\"whiten\", False):\n            recording = sip.whiten(recording, dtype=np.float64)\n            sorter_params[\"whiten\"] = False\n\n        common_sorter_items = {\n            \"sorter_name\": sorter,\n            \"recording\": recording,\n            \"output_folder\": sorter_temp_dir.name,\n            \"remove_existing_folder\": True,\n        }\n\n        if sorter.lower() in [\"kilosort2_5\", \"kilosort3\", \"ironclust\"]:\n            sorter_params = {\n                k: v\n                for k, v in sorter_params.items()\n                if k\n                not in [\"tempdir\", \"mp_context\", \"max_threads_per_process\"]\n            }\n            sorting = sis.run_sorter(\n                **common_sorter_items,\n                singularity_image=True,\n                **sorter_params,\n            )\n        else:\n            sorting = sis.run_sorter(\n                **common_sorter_items,\n                **sorter_params,\n            )\n    key[\"time_of_sort\"] = int(time.time())\n    sorting = sic.remove_excess_spikes(sorting, recording)\n    key[\"analysis_file_name\"], key[\"object_id\"] = _write_sorting_to_nwb(\n        sorting,\n        timestamps,\n        artifact_removed_intervals,\n        (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n    )\n\n    # INSERT\n    # - new entry to AnalysisNwbfile\n    # - new entry to SpikeSorting\n    AnalysisNwbfile().add(\n        (SpikeSortingSelection &amp; key).fetch1(\"nwb_file_name\"),\n        key[\"analysis_file_name\"],\n    )\n    self.insert1(key, skip_duplicates=True)\n</code></pre>"}, {"location": "api/spikesorting/v1/sorting/#src.spyglass.spikesorting.v1.sorting.SpikeSorting.get_sorting", "title": "<code>get_sorting(key)</code>  <code>classmethod</code>", "text": "<p>Get sorting in the analysis NWB file as spikeinterface BaseSorting</p> <p>Parameters:</p> Name Type Description Default <code>key</code> <code>dict</code> <p>primary key of SpikeSorting</p> required <p>Returns:</p> Name Type Description <code>sorting</code> <code>BaseSorting</code> Source code in <code>src/spyglass/spikesorting/v1/sorting.py</code> <pre><code>@classmethod\ndef get_sorting(cls, key: dict) -&gt; si.BaseSorting:\n    \"\"\"Get sorting in the analysis NWB file as spikeinterface BaseSorting\n\n    Parameters\n    ----------\n    key : dict\n        primary key of SpikeSorting\n\n    Returns\n    -------\n    sorting : si.BaseSorting\n\n    \"\"\"\n\n    recording_id = (\n        SpikeSortingRecording * SpikeSortingSelection &amp; key\n    ).fetch1(\"recording_id\")\n    recording = SpikeSortingRecording.get_recording(\n        {\"recording_id\": recording_id}\n    )\n    sampling_frequency = recording.get_sampling_frequency()\n    analysis_file_name = (cls &amp; key).fetch1(\"analysis_file_name\")\n    analysis_file_abs_path = AnalysisNwbfile.get_abs_path(\n        analysis_file_name\n    )\n    with pynwb.NWBHDF5IO(\n        analysis_file_abs_path, \"r\", load_namespaces=True\n    ) as io:\n        nwbf = io.read()\n        units = nwbf.units.to_dataframe()\n    units_dict_list = [\n        {\n            unit_id: np.searchsorted(recording.get_times(), spike_times)\n            for unit_id, spike_times in zip(\n                units.index, units[\"spike_times\"]\n            )\n        }\n    ]\n\n    sorting = si.NumpySorting.from_unit_dict(\n        units_dict_list, sampling_frequency=sampling_frequency\n    )\n\n    return sorting\n</code></pre>"}, {"location": "api/spikesorting/v1/utils/", "title": "utils.py", "text": ""}, {"location": "api/spikesorting/v1/utils/#src.spyglass.spikesorting.v1.utils.generate_nwb_uuid", "title": "<code>generate_nwb_uuid(nwb_file_name, initial, len_uuid=6)</code>", "text": "<p>Generates a unique identifier related to an NWB file.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_name</code> <code>str</code> <p>description</p> required <code>initial</code> <code>str</code> <p>R if recording; A if artifact; S if sorting etc</p> required <code>len_uuid</code> <code>int</code> <p>how many digits of uuid4 to keep</p> <code>6</code> Source code in <code>src/spyglass/spikesorting/v1/utils.py</code> <pre><code>def generate_nwb_uuid(nwb_file_name: str, initial: str, len_uuid: int = 6):\n    \"\"\"Generates a unique identifier related to an NWB file.\n\n    Parameters\n    ----------\n    nwb_file_name : str\n        _description_\n    initial : str\n        R if recording; A if artifact; S if sorting etc\n    len_uuid : int\n        how many digits of uuid4 to keep\n    \"\"\"\n    uuid4 = str(uuid.uuid4())\n    nwb_uuid = nwb_file_name + \"_\" + initial + \"_\" + uuid4[:len_uuid]\n    return nwb_uuid\n</code></pre>"}, {"location": "api/spikesorting/v1/utils/#src.spyglass.spikesorting.v1.utils.get_spiking_sorting_v1_merge_ids", "title": "<code>get_spiking_sorting_v1_merge_ids(restriction)</code>", "text": "<p>Parses the SpikingSorting V1 pipeline to get a list of merge ids for a given restriction.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>dict</code> <p>A dictionary containing some or all of the following key-value pairs: nwb_file_name : str     name of the nwb file interval_list_name : str     name of the interval list sort_group_name : str     name of the sort group artifact_param_name : str     name of the artifact parameter curation_id : int, optional     id of the curation (if not specified, uses the latest curation)</p> required <p>Returns:</p> Name Type Description <code>merge_id_list</code> <code>list</code> <p>list of merge ids for the given restriction</p> Source code in <code>src/spyglass/spikesorting/v1/utils.py</code> <pre><code>def get_spiking_sorting_v1_merge_ids(restriction: dict):\n    \"\"\"\n    Parses the SpikingSorting V1 pipeline to get a list of merge ids for a given restriction.\n\n    Parameters\n    ----------\n    restriction : dict\n        A dictionary containing some or all of the following key-value pairs:\n        nwb_file_name : str\n            name of the nwb file\n        interval_list_name : str\n            name of the interval list\n        sort_group_name : str\n            name of the sort group\n        artifact_param_name : str\n            name of the artifact parameter\n        curation_id : int, optional\n            id of the curation (if not specified, uses the latest curation)\n    Returns\n    -------\n    merge_id_list : list\n        list of merge ids for the given restriction\n    \"\"\"\n    # list of recording ids\n    recording_id_list = (SpikeSortingRecordingSelection() &amp; restriction).fetch(\n        \"recording_id\"\n    )\n    # list of artifact ids for each recording\n    artifact_id_list = [\n        (\n            ArtifactDetectionSelection() &amp; restriction &amp; {\"recording_id\": id}\n        ).fetch1(\"artifact_id\")\n        for id in recording_id_list\n    ]\n    # list of sorting ids for each recording\n    sorting_restriction = restriction.copy()\n    del sorting_restriction[\"interval_list_name\"]\n    sorting_id_list = []\n    for r_id, a_id in zip(recording_id_list, artifact_id_list):\n        # if sorted with artifact detection\n        if (\n            SpikeSortingSelection()\n            &amp; sorting_restriction\n            &amp; {\"recording_id\": r_id, \"interval_list_name\": a_id}\n        ):\n            sorting_id_list.append(\n                (\n                    SpikeSortingSelection()\n                    &amp; sorting_restriction\n                    &amp; {\"recording_id\": r_id, \"interval_list_name\": a_id}\n                ).fetch1(\"sorting_id\")\n            )\n        # if sorted without artifact detection\n        else:\n            sorting_id_list.append(\n                (\n                    SpikeSortingSelection()\n                    &amp; sorting_restriction\n                    &amp; {\"recording_id\": r_id, \"interval_list_name\": r_id}\n                ).fetch1(\"sorting_id\")\n            )\n    # if curation_id is specified, use that id for each sorting_id\n    if \"curation_id\" in restriction:\n        curation_id = [restriction[\"curation_id\"] for _ in sorting_id_list]\n    # if curation_id is not specified, use the latest curation_id for each sorting_id\n    else:\n        curation_id = [\n            np.max((CurationV1 &amp; {\"sorting_id\": id}).fetch(\"curation_id\"))\n            for id in sorting_id_list\n        ]\n    # list of merge ids for the desired curation(s)\n    merge_id_list = [\n        (\n            SpikeSortingOutput.CurationV1()\n            &amp; {\"sorting_id\": id, \"curation_id\": c_id}\n        ).fetch1(\"merge_id\")\n        for id, c_id in zip(sorting_id_list, curation_id)\n    ]\n    return merge_id_list\n</code></pre>"}, {"location": "api/utils/database_settings/", "title": "database_settings.py", "text": ""}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings", "title": "<code>DatabaseSettings</code>", "text": "Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>class DatabaseSettings:\n    def __init__(\n        self,\n        user_name=None,\n        host_name=None,\n        debug=False,\n        target_database=None,\n        exec_user=None,\n        exec_pass=None,\n    ):\n        \"\"\"Class to manage common database settings\n\n        Roles:\n        - dj_guest:  select for all prefix\n        - dj_collab: select for all prefix, all for user prefix\n        - dj_user:   select for all prefix, all for user prefix, all for shared\n        - dj_admin:     all for all prefix\n\n        Note: To add dj_user role to all those with common access, run:\n            query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"\n            users = dj.conn().query(query).fetchall()\n            for user in users:\n                dj.conn().query(f\"GRANT dj_user TO '{user[0][0]}'@'%';\")\n\n        Parameters\n        ----------\n        user_name : str, optional\n            The name of the user to add to the database. Default from dj.config\n        host_name : str, optional\n            The name of the host to add to the database. Default from dj.config\n        debug : bool, optional\n            Default False. If True, pprint sql instead of running\n        target_database : str, optional\n            Default is mysql. Can also be docker container id\n        exec_user : str, optional\n            User for executing commands. If None, use dj.config\n        exec_pass : str, optional\n            Password for executing commands. If None, use dj.config\n        \"\"\"\n        self.shared_modules = [f\"{m}{ESC}\" for m in SHARED_MODULES]\n        self.user = user_name or dj.config[\"database.user\"]\n        self.host = (\n            host_name or dj.config[\"database.host\"] or \"lmf-db.cin.ucsf.edu\"\n        )\n        self.debug = debug\n        self.target_database = target_database or \"mysql\"\n        self.exec_user = exec_user or dj.config[\"database.user\"]\n        self.exec_pass = exec_pass or dj.config[\"database.password\"]\n\n    @property\n    def _create_roles_dict(self):\n        return dict(\n            guest=[\n                f\"{CREATE_ROLE}`dj_guest`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_guest`;\\n\",\n            ],\n            collab=[\n                f\"{CREATE_ROLE}`dj_collab`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_collab`;\\n\",\n            ],  # also gets own prefix below\n            user=[\n                f\"{CREATE_ROLE}`dj_user`;\\n\",\n                f\"{GRANT_SEL}`%`.* TO `dj_user`;\\n\",\n            ]\n            + [\n                f\"{GRANT_ALL}`{module}`.* TO `dj_user`;\\n\"\n                for module in self.shared_modules\n            ],  # also gets own prefix below\n            admin=[\n                f\"{CREATE_ROLE}`dj_admin`;\\n\",\n                f\"{GRANT_ALL}`%`.* TO `dj_admin`;\\n\",\n            ],\n        )\n\n    @property\n    def _create_roles_sql(self):\n        return sum(self._create_roles_dict.values(), [])\n\n    def _create_user_sql(self, role):\n        \"\"\"Create user and grant role\"\"\"\n        return [\n            f\"{CREATE_USR}'{self.user}'@'%'{TEMP_PASS}\\n\",  # create user\n            f\"GRANT {role} TO '{self.user}'@'%';\\n\",  # grant role\n        ]\n\n    @property\n    def _user_prefix_sql(self):\n        \"\"\"Grant user all permissions for user prefix\"\"\"\n        return [\n            f\"{GRANT_ALL}`{self.user}{ESC}`.* TO '{self.user}'@'%';\\n\",\n        ]\n\n    @property\n    def _add_guest_sql(self):\n        return self._create_user_sql(\"dj_guest\")\n\n    @property\n    def _add_collab_sql(self):\n        return self._create_user_sql(\"dj_collab\") + self._user_prefix_sql\n\n    @property\n    def _add_user_sql(self):\n        return self._create_user_sql(\"dj_user\") + self._user_prefix_sql\n\n    @property\n    def _add_admin_sql(self):\n        return self._create_user_sql(\"dj_admin\") + self._user_prefix_sql\n\n    def _add_module_sql(self, module_name):\n        return [f\"{GRANT_ALL}`{module_name}{ESC}`.* TO dj_user;\\n\"]\n\n    def add_guest(self, *args, **kwargs):\n        \"\"\"Add guest user with select permissions to shared modules\"\"\"\n        file = self.write_temp_file(self._add_guest_sql)\n        self.exec(file)\n\n    def add_collab(self, *args, **kwargs):\n        \"\"\"Add collaborator user with full permissions to shared modules\"\"\"\n        file = self.write_temp_file(self._add_collab_sql)\n        self.exec(file)\n\n    def add_user(self, check_exists=False, *args, **kwargs):\n        \"\"\"Add user to database with permissions to shared modules\"\"\"\n        if check_exists:\n            self.check_user_exists()\n        file = self.write_temp_file(self._add_user_sql)\n        self.exec(file)\n\n    def add_admin(self, *args, **kwargs):\n        \"\"\"Add admin user with full permissions to all modules\"\"\"\n        file = self.write_temp_file(self._add_admin_sql)\n        self.exec(file)\n\n    def add_module(self, module_name):\n        \"\"\"Add module to database. Grant permissions to all users in group\"\"\"\n        logger.info(f\"Granting everyone permissions to module {module_name}\")\n        file = self.write_temp_file(self._add_module_sql(module_name))\n        self.exec(file)\n\n    def check_user_exists(self):\n        \"\"\"Add user to database with permissions to shared modules\"\"\"\n        user_home = Path.home().parent / self.user\n        if user_home.exists():\n            logger.info(\"Creating database user \", self.user)\n        else:\n            sys.exit(\n                f\"Error: couldn't find {self.user} in home dir: {user_home}\"\n            )\n\n    def add_user_by_role(self, role, check_exists=False):\n        add_func = {\n            \"guest\": self.add_guest,\n            \"user\": self.add_user,\n            \"collab\": self.add_collab,\n            \"admin\": self.add_admin,\n        }\n        if role not in add_func:\n            raise ValueError(f\"Role {role} not recognized\")\n        add_func[role]()\n\n    def add_roles(self):\n        \"\"\"Add roles to database\"\"\"\n        file = self.write_temp_file(self._create_roles_sql)\n        self.exec(file)\n\n    def write_temp_file(self, content: list) -&gt; tempfile.NamedTemporaryFile:\n        \"\"\"Write content to a temporary file and return the file object\"\"\"\n        file = tempfile.NamedTemporaryFile(mode=\"w\")\n        for line in content:\n            file.write(line)\n        file.flush()\n\n        if self.debug:\n            from pprint import pprint  # noqa F401\n\n            pprint(file.name)\n            pprint(content)\n\n        return file\n\n    def exec(self, file):\n        \"\"\"Run commands saved to file in sql\"\"\"\n\n        if self.debug:\n            return\n\n        cmd = (\n            f\"mysql -p -h {self.host} &lt; {file.name}\"\n            if self.target_database == \"mysql\"\n            else f\"docker exec -i {self.target_database} mysql -u \"\n            + f\"{self.exec_user} --password={self.exec_pass} &lt; {file.name}\"\n        )\n        os.system(cmd)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.__init__", "title": "<code>__init__(user_name=None, host_name=None, debug=False, target_database=None, exec_user=None, exec_pass=None)</code>", "text": "<p>Class to manage common database settings</p> <p>Roles: - dj_guest:  select for all prefix - dj_collab: select for all prefix, all for user prefix - dj_user:   select for all prefix, all for user prefix, all for shared - dj_admin:     all for all prefix</p> <p>Note: To add dj_user role to all those with common access, run:     query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"     users = dj.conn().query(query).fetchall()     for user in users:         dj.conn().query(f\"GRANT dj_user TO '{user0}'@'%';\")</p> <p>Parameters:</p> Name Type Description Default <code>user_name</code> <code>str</code> <p>The name of the user to add to the database. Default from dj.config</p> <code>None</code> <code>host_name</code> <code>str</code> <p>The name of the host to add to the database. Default from dj.config</p> <code>None</code> <code>debug</code> <code>bool</code> <p>Default False. If True, pprint sql instead of running</p> <code>False</code> <code>target_database</code> <code>str</code> <p>Default is mysql. Can also be docker container id</p> <code>None</code> <code>exec_user</code> <code>str</code> <p>User for executing commands. If None, use dj.config</p> <code>None</code> <code>exec_pass</code> <code>str</code> <p>Password for executing commands. If None, use dj.config</p> <code>None</code> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def __init__(\n    self,\n    user_name=None,\n    host_name=None,\n    debug=False,\n    target_database=None,\n    exec_user=None,\n    exec_pass=None,\n):\n    \"\"\"Class to manage common database settings\n\n    Roles:\n    - dj_guest:  select for all prefix\n    - dj_collab: select for all prefix, all for user prefix\n    - dj_user:   select for all prefix, all for user prefix, all for shared\n    - dj_admin:     all for all prefix\n\n    Note: To add dj_user role to all those with common access, run:\n        query = \"SELECT user, host FROM mysql.db WHERE Db LIKE 'common%';\"\n        users = dj.conn().query(query).fetchall()\n        for user in users:\n            dj.conn().query(f\"GRANT dj_user TO '{user[0][0]}'@'%';\")\n\n    Parameters\n    ----------\n    user_name : str, optional\n        The name of the user to add to the database. Default from dj.config\n    host_name : str, optional\n        The name of the host to add to the database. Default from dj.config\n    debug : bool, optional\n        Default False. If True, pprint sql instead of running\n    target_database : str, optional\n        Default is mysql. Can also be docker container id\n    exec_user : str, optional\n        User for executing commands. If None, use dj.config\n    exec_pass : str, optional\n        Password for executing commands. If None, use dj.config\n    \"\"\"\n    self.shared_modules = [f\"{m}{ESC}\" for m in SHARED_MODULES]\n    self.user = user_name or dj.config[\"database.user\"]\n    self.host = (\n        host_name or dj.config[\"database.host\"] or \"lmf-db.cin.ucsf.edu\"\n    )\n    self.debug = debug\n    self.target_database = target_database or \"mysql\"\n    self.exec_user = exec_user or dj.config[\"database.user\"]\n    self.exec_pass = exec_pass or dj.config[\"database.password\"]\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_guest", "title": "<code>add_guest(*args, **kwargs)</code>", "text": "<p>Add guest user with select permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_guest(self, *args, **kwargs):\n    \"\"\"Add guest user with select permissions to shared modules\"\"\"\n    file = self.write_temp_file(self._add_guest_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_collab", "title": "<code>add_collab(*args, **kwargs)</code>", "text": "<p>Add collaborator user with full permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_collab(self, *args, **kwargs):\n    \"\"\"Add collaborator user with full permissions to shared modules\"\"\"\n    file = self.write_temp_file(self._add_collab_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_user", "title": "<code>add_user(check_exists=False, *args, **kwargs)</code>", "text": "<p>Add user to database with permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_user(self, check_exists=False, *args, **kwargs):\n    \"\"\"Add user to database with permissions to shared modules\"\"\"\n    if check_exists:\n        self.check_user_exists()\n    file = self.write_temp_file(self._add_user_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_admin", "title": "<code>add_admin(*args, **kwargs)</code>", "text": "<p>Add admin user with full permissions to all modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_admin(self, *args, **kwargs):\n    \"\"\"Add admin user with full permissions to all modules\"\"\"\n    file = self.write_temp_file(self._add_admin_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_module", "title": "<code>add_module(module_name)</code>", "text": "<p>Add module to database. Grant permissions to all users in group</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_module(self, module_name):\n    \"\"\"Add module to database. Grant permissions to all users in group\"\"\"\n    logger.info(f\"Granting everyone permissions to module {module_name}\")\n    file = self.write_temp_file(self._add_module_sql(module_name))\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.check_user_exists", "title": "<code>check_user_exists()</code>", "text": "<p>Add user to database with permissions to shared modules</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def check_user_exists(self):\n    \"\"\"Add user to database with permissions to shared modules\"\"\"\n    user_home = Path.home().parent / self.user\n    if user_home.exists():\n        logger.info(\"Creating database user \", self.user)\n    else:\n        sys.exit(\n            f\"Error: couldn't find {self.user} in home dir: {user_home}\"\n        )\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.add_roles", "title": "<code>add_roles()</code>", "text": "<p>Add roles to database</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def add_roles(self):\n    \"\"\"Add roles to database\"\"\"\n    file = self.write_temp_file(self._create_roles_sql)\n    self.exec(file)\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.write_temp_file", "title": "<code>write_temp_file(content)</code>", "text": "<p>Write content to a temporary file and return the file object</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def write_temp_file(self, content: list) -&gt; tempfile.NamedTemporaryFile:\n    \"\"\"Write content to a temporary file and return the file object\"\"\"\n    file = tempfile.NamedTemporaryFile(mode=\"w\")\n    for line in content:\n        file.write(line)\n    file.flush()\n\n    if self.debug:\n        from pprint import pprint  # noqa F401\n\n        pprint(file.name)\n        pprint(content)\n\n    return file\n</code></pre>"}, {"location": "api/utils/database_settings/#src.spyglass.utils.database_settings.DatabaseSettings.exec", "title": "<code>exec(file)</code>", "text": "<p>Run commands saved to file in sql</p> Source code in <code>src/spyglass/utils/database_settings.py</code> <pre><code>def exec(self, file):\n    \"\"\"Run commands saved to file in sql\"\"\"\n\n    if self.debug:\n        return\n\n    cmd = (\n        f\"mysql -p -h {self.host} &lt; {file.name}\"\n        if self.target_database == \"mysql\"\n        else f\"docker exec -i {self.target_database} mysql -u \"\n        + f\"{self.exec_user} --password={self.exec_pass} &lt; {file.name}\"\n    )\n    os.system(cmd)\n</code></pre>"}, {"location": "api/utils/dj_chains/", "title": "dj_chains.py", "text": ""}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains", "title": "<code>TableChains</code>", "text": "<p>Class for representing chains from parent to Merge table via parts.</p> <p>Functions as a plural version of TableChain, allowing a single <code>join</code> call across all chains from parent -&gt; Merge table.</p> <p>Attributes:</p> Name Type Description <code>parent</code> <code>Table</code> <p>Parent or origin of chains.</p> <code>child</code> <code>Table</code> <p>Merge table or destination of chains.</p> <code>connection</code> <code>(Connection, optional)</code> <p>Connection to database used to create FreeTable objects. Defaults to parent.connection.</p> <code>part_names</code> <code>List[str]</code> <p>List of full table names of child parts.</p> <code>chains</code> <code>List[TableChain]</code> <p>List of TableChain objects for each part in child.</p> <code>has_link</code> <code>bool</code> <p>Cached attribute to store whether parent is linked to child via any of child parts. False if (a) child is not in parent.descendants or (b) nx.NetworkXNoPath is raised by nx.shortest_path for all chains.</p> <p>Methods:</p> Name Description <code>__init__</code> <p>Initialize TableChains with parent and child tables.</p> <code>__repr__</code> <p>Return full representation of chains. Multiline parent -&gt; child for each chain.</p> <code>__len__</code> <p>Return number of chains with links.</p> <code>__getitem__</code> <p>Return TableChain object at index, or use substring of table name.</p> <code>join</code> <p>Return list of joins for each chain in self.chains.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>class TableChains:\n    \"\"\"Class for representing chains from parent to Merge table via parts.\n\n    Functions as a plural version of TableChain, allowing a single `join`\n    call across all chains from parent -&gt; Merge table.\n\n    Attributes\n    ----------\n    parent : Table\n        Parent or origin of chains.\n    child : Table\n        Merge table or destination of chains.\n    connection : datajoint.Connection, optional\n        Connection to database used to create FreeTable objects. Defaults to\n        parent.connection.\n    part_names : List[str]\n        List of full table names of child parts.\n    chains : List[TableChain]\n        List of TableChain objects for each part in child.\n    has_link : bool\n        Cached attribute to store whether parent is linked to child via any of\n        child parts. False if (a) child is not in parent.descendants or (b)\n        nx.NetworkXNoPath is raised by nx.shortest_path for all chains.\n\n    Methods\n    -------\n    __init__(parent, child, connection=None)\n        Initialize TableChains with parent and child tables.\n    __repr__()\n        Return full representation of chains.\n        Multiline parent -&gt; child for each chain.\n    __len__()\n        Return number of chains with links.\n    __getitem__(index: Union[int, str])\n        Return TableChain object at index, or use substring of table name.\n    join(restriction: str = None)\n        Return list of joins for each chain in self.chains.\n    \"\"\"\n\n    def __init__(self, parent, child, connection=None):\n        self.parent = parent\n        self.child = child\n        self.connection = connection or parent.connection\n        parts = child.parts(as_objects=True)\n        self.part_names = [part.full_table_name for part in parts]\n        self.chains = [TableChain(parent, part) for part in parts]\n        self.has_link = any([chain.has_link for chain in self.chains])\n\n    def __repr__(self):\n        return \"\\n\".join([str(chain) for chain in self.chains])\n\n    def __len__(self):\n        return len([c for c in self.chains if c.has_link])\n\n    @property\n    def max_len(self):\n        \"\"\"Return length of longest chain.\"\"\"\n        return max([len(chain) for chain in self.chains])\n\n    def __getitem__(self, index: Union[int, str]):\n        \"\"\"Return FreeTable object at index.\"\"\"\n        if isinstance(index, str):\n            for i, part in enumerate(self.part_names):\n                if index in part:\n                    return self.chains[i]\n        return self.chains[index]\n\n    def join(self, restriction=None) -&gt; List[QueryExpression]:\n        \"\"\"Return list of joins for each chain in self.chains.\"\"\"\n        restriction = restriction or self.parent.restriction or True\n        joins = []\n        for chain in self.chains:\n            if joined := chain.join(restriction):\n                joins.append(joined)\n        return joins\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.max_len", "title": "<code>max_len</code>  <code>property</code>", "text": "<p>Return length of longest chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.__getitem__", "title": "<code>__getitem__(index)</code>", "text": "<p>Return FreeTable object at index.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __getitem__(self, index: Union[int, str]):\n    \"\"\"Return FreeTable object at index.\"\"\"\n    if isinstance(index, str):\n        for i, part in enumerate(self.part_names):\n            if index in part:\n                return self.chains[i]\n    return self.chains[index]\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChains.join", "title": "<code>join(restriction=None)</code>", "text": "<p>Return list of joins for each chain in self.chains.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def join(self, restriction=None) -&gt; List[QueryExpression]:\n    \"\"\"Return list of joins for each chain in self.chains.\"\"\"\n    restriction = restriction or self.parent.restriction or True\n    joins = []\n    for chain in self.chains:\n        if joined := chain.join(restriction):\n            joins.append(joined)\n    return joins\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain", "title": "<code>TableChain</code>", "text": "<p>Class for representing a chain of tables.</p> <p>A chain is a sequence of tables from parent to child identified by networkx.shortest_path. Parent -&gt; Merge should use TableChains instead to handle multiple paths to the respective parts of the Merge table.</p> <p>Attributes:</p> Name Type Description <code>parent</code> <code>Table</code> <p>Parent or origin of chain.</p> <code>child</code> <code>Table</code> <p>Child or destination of chain.</p> <code>_connection</code> <code>(Connection, optional)</code> <p>Connection to database used to create FreeTable objects. Defaults to parent.connection.</p> <code>_link_symbol</code> <code>str</code> <p>Symbol used to represent the link between parent and child. Hardcoded to \" -&gt; \".</p> <code>has_link</code> <code>bool</code> <p>Cached attribute to store whether parent is linked to child. False if child is not in parent.descendants or nx.NetworkXNoPath is raised by nx.shortest_path.</p> <code>link_type</code> <code>str</code> <p>'directed' or 'undirected' based on whether path is found with directed or undirected graph. None if no path is found.</p> <code>graph</code> <code>DiGraph</code> <p>Directed graph of parent's dependencies from datajoint.connection.</p> <code>names</code> <code>List[str]</code> <p>List of full table names in chain.</p> <code>objects</code> <code>List[FreeTable]</code> <p>List of FreeTable objects for each table in chain.</p> <code>attr_maps</code> <code>List[dict]</code> <p>List of attribute maps for each link in chain.</p> <code>path</code> <code>OrderedDict[str, Dict[str, Union[FreeTable, dict]]]</code> <p>Dictionary of full table names in chain. Keys are self.names Values are a dict of free_table (self.objects) and attr_map (dict of new_name: old_name, self.attr_map).</p> <p>Methods:</p> Name Description <code>__str__</code> <p>Return string representation of chain: parent -&gt; child.</p> <code>__repr__</code> <p>Return full representation of chain: parent -&gt; {links} -&gt; child.</p> <code>__len__</code> <p>Return number of tables in chain.</p> <code>__getitem__</code> <p>Return FreeTable object at index, or use substring of table name.</p> <code>find_path</code> <p>Returns path OrderedDict of full table names in chain. If directed is True, uses directed graph. If False, uses undirected graph. Undirected excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc. to maintain valid joins.</p> <code>join</code> <p>Return join of tables in chain with restriction applied to parent.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>class TableChain:\n    \"\"\"Class for representing a chain of tables.\n\n    A chain is a sequence of tables from parent to child identified by\n    networkx.shortest_path. Parent -&gt; Merge should use TableChains instead to\n    handle multiple paths to the respective parts of the Merge table.\n\n    Attributes\n    ----------\n    parent : Table\n        Parent or origin of chain.\n    child : Table\n        Child or destination of chain.\n    _connection : datajoint.Connection, optional\n        Connection to database used to create FreeTable objects. Defaults to\n        parent.connection.\n    _link_symbol : str\n        Symbol used to represent the link between parent and child. Hardcoded\n        to \" -&gt; \".\n    has_link : bool\n        Cached attribute to store whether parent is linked to child. False if\n        child is not in parent.descendants or nx.NetworkXNoPath is raised by\n        nx.shortest_path.\n    link_type : str\n        'directed' or 'undirected' based on whether path is found with directed\n        or undirected graph. None if no path is found.\n    graph : nx.DiGraph\n        Directed graph of parent's dependencies from datajoint.connection.\n    names : List[str]\n        List of full table names in chain.\n    objects : List[dj.FreeTable]\n        List of FreeTable objects for each table in chain.\n    attr_maps : List[dict]\n        List of attribute maps for each link in chain.\n    path : OrderedDict[str, Dict[str, Union[dj.FreeTable,dict]]]\n        Dictionary of full table names in chain. Keys are self.names\n        Values are a dict of free_table (self.objects) and\n        attr_map (dict of new_name: old_name, self.attr_map).\n\n    Methods\n    -------\n    __str__()\n        Return string representation of chain: parent -&gt; child.\n    __repr__()\n        Return full representation of chain: parent -&gt; {links} -&gt; child.\n    __len__()\n        Return number of tables in chain.\n    __getitem__(index: Union[int, str])\n        Return FreeTable object at index, or use substring of table name.\n    find_path(directed=True)\n        Returns path OrderedDict of full table names in chain. If directed is\n        True, uses directed graph. If False, uses undirected graph. Undirected\n        excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc. to maintain\n        valid joins.\n    join(restriction: str = None)\n        Return join of tables in chain with restriction applied to parent.\n    \"\"\"\n\n    def __init__(self, parent: Table, child: Table, connection=None):\n        self._connection = connection or parent.connection\n        self.graph = self._connection.dependencies\n        self.graph.load()\n\n        if (  # if child is a merge table\n            get_master(child.full_table_name) == \"\"\n            and MERGE_PK in child.heading.names\n        ):\n            raise TypeError(\"Child is a merge table. Use TableChains instead.\")\n\n        self._link_symbol = \" -&gt; \"\n        self.parent = parent\n        self.child = child\n        self.link_type = None\n        self._searched = False\n\n        if child.full_table_name not in self.graph.nodes:\n            logger.warning(\n                \"Can't find item in graph. Try importing: \"\n                + f\"{child.full_table_name}\"\n            )\n            self._searched = True\n\n    def __str__(self):\n        \"\"\"Return string representation of chain: parent -&gt; child.\"\"\"\n        if not self.has_link:\n            return \"No link\"\n        return (\n            to_camel_case(self.parent.table_name)\n            + self._link_symbol\n            + to_camel_case(self.child.table_name)\n        )\n\n    def __repr__(self):\n        \"\"\"Return full representation of chain: parent -&gt; {links} -&gt; child.\"\"\"\n        if not self.has_link:\n            return \"No link\"\n        return \"Chain: \" + self._link_symbol.join(\n            [t.table_name for t in self.objects]\n        )\n\n    def __len__(self):\n        \"\"\"Return number of tables in chain.\"\"\"\n        if not self.has_link:\n            return 0\n        return len(self.names)\n\n    def __getitem__(self, index: Union[int, str]) -&gt; dj.FreeTable:\n        \"\"\"Return FreeTable object at index.\"\"\"\n        if not self.has_link:\n            return None\n        if isinstance(index, str):\n            for i, name in enumerate(self.names):\n                if index in name:\n                    return self.objects[i]\n        return self.objects[index]\n\n    @property\n    def has_link(self) -&gt; bool:\n        \"\"\"Return True if parent is linked to child.\n\n        If not searched, search for path. If searched and no link is found,\n        return False. If searched and link is found, return True.\n        \"\"\"\n        if not self._searched:\n            _ = self.path\n        return self.link_type is not None\n\n    def pk_link(self, src, trg, data) -&gt; float:\n        \"\"\"Return 1 if data[\"primary\"] else float(\"inf\").\n\n        Currently unused. Preserved for future debugging. shortest_path accepts\n        an option weight callable parameter.\n        nx.shortest_path(G, source, target,weight=pk_link)\n        \"\"\"\n        return 1 if data[\"primary\"] else float(\"inf\")\n\n    def find_path(self, directed=True) -&gt; OrderedDict:\n        \"\"\"Return list of full table names in chain.\n\n        Parameters\n        ----------\n        directed : bool, optional\n            If True, use directed graph. If False, use undirected graph.\n            Defaults to True. Undirected permits paths to traverse from merge\n            part-parent -&gt; merge part -&gt; merge table. Undirected excludes\n            PERIPHERAL_TABLES like interval_list, nwbfile, etc.\n\n        Returns\n        -------\n        OrderedDict\n            Dictionary of full table names in chain. Keys are full table names.\n            Values are free_table (dj.FreeTable representation) and attr_map\n            (dict of new_name: old_name). Attribute maps on the table upstream\n            of an alias node that can be used in .proj(). Returns None if no\n            path is found.\n\n        Ignores numeric table names in paths, which are\n        'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph\n        source code for comments on alias nodes.\n        \"\"\"\n        source, target = self.parent.full_table_name, self.child.full_table_name\n        if not directed:\n            self.graph = self.graph.to_undirected()\n            self.graph.remove_nodes_from(PERIPHERAL_TABLES)\n        try:\n            path = nx.shortest_path(self.graph, source, target)\n        except nx.NetworkXNoPath:\n            return None\n        except nx.NodeNotFound:\n            self._searched = True\n            return None\n\n        ret = OrderedDict()\n        prev_table = None\n        for i, table in enumerate(path):\n            if table.isnumeric():  # get proj() attribute map for alias node\n                if not prev_table:\n                    raise ValueError(\"Alias node found without prev table.\")\n                attr_map = self.graph[table][prev_table][\"attr_map\"]\n                ret[prev_table][\"attr_map\"] = attr_map\n            else:\n                free_table = dj.FreeTable(self._connection, table)\n                ret[table] = {\"free_table\": free_table, \"attr_map\": {}}\n                prev_table = table\n        return ret\n\n    @cached_property\n    def path(self) -&gt; OrderedDict:\n        \"\"\"Return list of full table names in chain.\"\"\"\n        if self._searched and not self.has_link:\n            return None\n\n        link = None\n        if link := self.find_path(directed=True):\n            self.link_type = \"directed\"\n        elif link := self.find_path(directed=False):\n            self.link_type = \"undirected\"\n        self._searched = True\n\n        return link\n\n    @cached_property\n    def names(self) -&gt; List[str]:\n        \"\"\"Return list of full table names in chain.\"\"\"\n        if not self.has_link:\n            return None\n        return list(self.path.keys())\n\n    @cached_property\n    def objects(self) -&gt; List[dj.FreeTable]:\n        \"\"\"Return list of FreeTable objects for each table in chain.\n\n        Unused. Preserved for future debugging.\n        \"\"\"\n        if not self.has_link:\n            return None\n        return [v[\"free_table\"] for v in self.path.values()]\n\n    @cached_property\n    def attr_maps(self) -&gt; List[dict]:\n        \"\"\"Return list of attribute maps for each table in chain.\n\n        Unused. Preserved for future debugging.\n        \"\"\"\n        if not self.has_link:\n            return None\n        return [v[\"attr_map\"] for v in self.path.values()]\n\n    def join(\n        self, restriction: str = None, reverse_order: bool = False\n    ) -&gt; dj.expression.QueryExpression:\n        \"\"\"Return join of tables in chain with restriction applied to parent.\n\n        Parameters\n        ----------\n        restriction : str, optional\n            Restriction to apply to first table in the order.\n            Defaults to self.parent.restriction.\n        reverse_order : bool, optional\n            If True, join tables in reverse order. Defaults to False.\n        \"\"\"\n        if not self.has_link:\n            return None\n\n        restriction = restriction or self.parent.restriction or True\n        path = (\n            OrderedDict(reversed(self.path.items()))\n            if reverse_order\n            else self.path\n        ).copy()\n\n        _, first_val = path.popitem(last=False)\n        join = first_val[\"free_table\"] &amp; restriction\n        for i, val in enumerate(path.values()):\n            attr_map, free_table = val[\"attr_map\"], val[\"free_table\"]\n            try:\n                join = (join.proj() * free_table).proj(**attr_map)\n            except dj.DataJointError as e:\n                attribute = str(e).split(\"attribute \")[-1]\n                logger.error(\n                    f\"{str(self)} at {free_table.table_name} with {attribute}\"\n                )\n                return None\n        return join\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__str__", "title": "<code>__str__()</code>", "text": "<p>Return string representation of chain: parent -&gt; child.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __str__(self):\n    \"\"\"Return string representation of chain: parent -&gt; child.\"\"\"\n    if not self.has_link:\n        return \"No link\"\n    return (\n        to_camel_case(self.parent.table_name)\n        + self._link_symbol\n        + to_camel_case(self.child.table_name)\n    )\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__repr__", "title": "<code>__repr__()</code>", "text": "<p>Return full representation of chain: parent -&gt; {links} -&gt; child.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __repr__(self):\n    \"\"\"Return full representation of chain: parent -&gt; {links} -&gt; child.\"\"\"\n    if not self.has_link:\n        return \"No link\"\n    return \"Chain: \" + self._link_symbol.join(\n        [t.table_name for t in self.objects]\n    )\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__len__", "title": "<code>__len__()</code>", "text": "<p>Return number of tables in chain.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __len__(self):\n    \"\"\"Return number of tables in chain.\"\"\"\n    if not self.has_link:\n        return 0\n    return len(self.names)\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.__getitem__", "title": "<code>__getitem__(index)</code>", "text": "<p>Return FreeTable object at index.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def __getitem__(self, index: Union[int, str]) -&gt; dj.FreeTable:\n    \"\"\"Return FreeTable object at index.\"\"\"\n    if not self.has_link:\n        return None\n    if isinstance(index, str):\n        for i, name in enumerate(self.names):\n            if index in name:\n                return self.objects[i]\n    return self.objects[index]\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.has_link", "title": "<code>has_link: bool</code>  <code>property</code>", "text": "<p>Return True if parent is linked to child.</p> <p>If not searched, search for path. If searched and no link is found, return False. If searched and link is found, return True.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.pk_link", "title": "<code>pk_link(src, trg, data)</code>", "text": "<p>Return 1 if data[\"primary\"] else float(\"inf\").</p> <p>Currently unused. Preserved for future debugging. shortest_path accepts an option weight callable parameter. nx.shortest_path(G, source, target,weight=pk_link)</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def pk_link(self, src, trg, data) -&gt; float:\n    \"\"\"Return 1 if data[\"primary\"] else float(\"inf\").\n\n    Currently unused. Preserved for future debugging. shortest_path accepts\n    an option weight callable parameter.\n    nx.shortest_path(G, source, target,weight=pk_link)\n    \"\"\"\n    return 1 if data[\"primary\"] else float(\"inf\")\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.find_path", "title": "<code>find_path(directed=True)</code>", "text": "<p>Return list of full table names in chain.</p> <p>Parameters:</p> Name Type Description Default <code>directed</code> <code>bool</code> <p>If True, use directed graph. If False, use undirected graph. Defaults to True. Undirected permits paths to traverse from merge part-parent -&gt; merge part -&gt; merge table. Undirected excludes PERIPHERAL_TABLES like interval_list, nwbfile, etc.</p> <code>True</code> <p>Returns:</p> Type Description <code>OrderedDict</code> <p>Dictionary of full table names in chain. Keys are full table names. Values are free_table (dj.FreeTable representation) and attr_map (dict of new_name: old_name). Attribute maps on the table upstream of an alias node that can be used in .proj(). Returns None if no path is found.</p> <p>Ignores numeric table names in paths, which are 'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph source code for comments on alias nodes.</p> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def find_path(self, directed=True) -&gt; OrderedDict:\n    \"\"\"Return list of full table names in chain.\n\n    Parameters\n    ----------\n    directed : bool, optional\n        If True, use directed graph. If False, use undirected graph.\n        Defaults to True. Undirected permits paths to traverse from merge\n        part-parent -&gt; merge part -&gt; merge table. Undirected excludes\n        PERIPHERAL_TABLES like interval_list, nwbfile, etc.\n\n    Returns\n    -------\n    OrderedDict\n        Dictionary of full table names in chain. Keys are full table names.\n        Values are free_table (dj.FreeTable representation) and attr_map\n        (dict of new_name: old_name). Attribute maps on the table upstream\n        of an alias node that can be used in .proj(). Returns None if no\n        path is found.\n\n    Ignores numeric table names in paths, which are\n    'gaps' or alias nodes in the graph. See datajoint.Diagram._make_graph\n    source code for comments on alias nodes.\n    \"\"\"\n    source, target = self.parent.full_table_name, self.child.full_table_name\n    if not directed:\n        self.graph = self.graph.to_undirected()\n        self.graph.remove_nodes_from(PERIPHERAL_TABLES)\n    try:\n        path = nx.shortest_path(self.graph, source, target)\n    except nx.NetworkXNoPath:\n        return None\n    except nx.NodeNotFound:\n        self._searched = True\n        return None\n\n    ret = OrderedDict()\n    prev_table = None\n    for i, table in enumerate(path):\n        if table.isnumeric():  # get proj() attribute map for alias node\n            if not prev_table:\n                raise ValueError(\"Alias node found without prev table.\")\n            attr_map = self.graph[table][prev_table][\"attr_map\"]\n            ret[prev_table][\"attr_map\"] = attr_map\n        else:\n            free_table = dj.FreeTable(self._connection, table)\n            ret[table] = {\"free_table\": free_table, \"attr_map\": {}}\n            prev_table = table\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.path", "title": "<code>path: OrderedDict</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of full table names in chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.names", "title": "<code>names: List[str]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of full table names in chain.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.objects", "title": "<code>objects: List[dj.FreeTable]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of FreeTable objects for each table in chain.</p> <p>Unused. Preserved for future debugging.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.attr_maps", "title": "<code>attr_maps: List[dict]</code>  <code>cached</code> <code>property</code>", "text": "<p>Return list of attribute maps for each table in chain.</p> <p>Unused. Preserved for future debugging.</p>"}, {"location": "api/utils/dj_chains/#src.spyglass.utils.dj_chains.TableChain.join", "title": "<code>join(restriction=None, reverse_order=False)</code>", "text": "<p>Return join of tables in chain with restriction applied to parent.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to first table in the order. Defaults to self.parent.restriction.</p> <code>None</code> <code>reverse_order</code> <code>bool</code> <p>If True, join tables in reverse order. Defaults to False.</p> <code>False</code> Source code in <code>src/spyglass/utils/dj_chains.py</code> <pre><code>def join(\n    self, restriction: str = None, reverse_order: bool = False\n) -&gt; dj.expression.QueryExpression:\n    \"\"\"Return join of tables in chain with restriction applied to parent.\n\n    Parameters\n    ----------\n    restriction : str, optional\n        Restriction to apply to first table in the order.\n        Defaults to self.parent.restriction.\n    reverse_order : bool, optional\n        If True, join tables in reverse order. Defaults to False.\n    \"\"\"\n    if not self.has_link:\n        return None\n\n    restriction = restriction or self.parent.restriction or True\n    path = (\n        OrderedDict(reversed(self.path.items()))\n        if reverse_order\n        else self.path\n    ).copy()\n\n    _, first_val = path.popitem(last=False)\n    join = first_val[\"free_table\"] &amp; restriction\n    for i, val in enumerate(path.values()):\n        attr_map, free_table = val[\"attr_map\"], val[\"free_table\"]\n        try:\n            join = (join.proj() * free_table).proj(**attr_map)\n        except dj.DataJointError as e:\n            attribute = str(e).split(\"attribute \")[-1]\n            logger.error(\n                f\"{str(self)} at {free_table.table_name} with {attribute}\"\n            )\n            return None\n    return join\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/", "title": "dj_helper_fn.py", "text": "<p>Helper functions for manipulating information from DataJoint fetch calls.</p>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.deprecated_factory", "title": "<code>deprecated_factory(classes, old_module='')</code>", "text": "<p>Creates a list of classes and logs a warning when instantiated</p> <p>Parameters:</p> Name Type Description Default <code>classes</code> <code>list</code> <p>list of tuples containing old_class, new_class</p> required <p>Returns:</p> Type Description <code>list</code> <p>list of classes that will log a warning when instantiated</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def deprecated_factory(classes: list, old_module: str = \"\") -&gt; list:\n    \"\"\"Creates a list of classes and logs a warning when instantiated\n\n    Parameters\n    ---------\n    classes : list\n        list of tuples containing old_class, new_class\n\n    Returns\n    ------\n    list\n        list of classes that will log a warning when instantiated\n    \"\"\"\n\n    if not isinstance(classes, list):\n        classes = [classes]\n\n    ret = [\n        _subclass_factory(old_name=c[0], new_class=c[1], old_module=old_module)\n        for c in classes\n    ]\n\n    return ret[0] if len(ret) == 1 else ret\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.dj_replace", "title": "<code>dj_replace(original_table, new_values, key_column, replace_column)</code>", "text": "<p>Given the output of a fetch() call from a schema and a 2D array made up of (key_value, replace_value) tuples, find each instance of key_value in the key_column of the original table and replace the specified replace_column with the associated replace_value. Key values must be unique.</p> <p>Parameters:</p> Name Type Description Default <code>original_table</code> <p>Result of a datajoint .fetch() call on a schema query.</p> required <code>new_values</code> <code>list</code> <p>List of tuples, each containing (key_value, replace_value).</p> required <code>replace_column</code> <code>str</code> <p>The name of the column where to-be-replaced values are located.</p> required <p>Returns:</p> Type Description <code>original_table</code> <p>Structured array of new table entries that can be inserted back into the schema</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def dj_replace(original_table, new_values, key_column, replace_column):\n    \"\"\"Given the output of a fetch() call from a schema and a 2D array made up\n    of (key_value, replace_value) tuples, find each instance of key_value in\n    the key_column of the original table and replace the specified\n    replace_column with the associated replace_value. Key values must be\n    unique.\n\n    Parameters\n    ----------\n    original_table\n        Result of a datajoint .fetch() call on a schema query.\n    new_values : list\n        List of tuples, each containing (key_value, replace_value).\n    replace_column : str\n        The name of the column where to-be-replaced values are located.\n\n    Returns\n    -------\n    original_table\n        Structured array of new table entries that can be inserted back into the schema\n    \"\"\"\n\n    # check to make sure the new_values are a list or array of tuples and fix if not\n    if isinstance(new_values, tuple):\n        tmp = list()\n        tmp.append(new_values)\n        new_values = tmp\n\n    new_val_array = np.asarray(new_values)\n    replace_ind = np.where(\n        np.isin(original_table[key_column], new_val_array[:, 0])\n    )\n    original_table[replace_column][replace_ind] = new_val_array[:, 1]\n    return original_table\n</code></pre>"}, {"location": "api/utils/dj_helper_fn/#src.spyglass.utils.dj_helper_fn.fetch_nwb", "title": "<code>fetch_nwb(query_expression, nwb_master, *attrs, **kwargs)</code>", "text": "<p>Get an NWB object from the given DataJoint query.</p> <p>Parameters:</p> Name Type Description Default <code>query_expression</code> <code>query</code> <p>A DataJoint query expression (e.g., join, restrict) or a table to call fetch on.</p> required <code>nwb_master</code> <code>tuple</code> <p>Tuple (table, attr) to get the NWB filepath from. i.e. absolute path to NWB file can be obtained by looking up attr column of table table is usually Nwbfile or AnalysisNwbfile; attr is usually 'nwb_file_abs_path' or 'analysis_file_abs_path'</p> required <code>*attrs</code> <code>list</code> <p>Attributes from normal DataJoint fetch call.</p> <code>()</code> <code>**kwargs</code> <code>dict</code> <p>Keyword arguments from normal DataJoint fetch call.</p> <code>{}</code> <p>Returns:</p> Name Type Description <code>nwb_objects</code> <code>list</code> <p>List of dicts containing fetch results and NWB objects.</p> Source code in <code>src/spyglass/utils/dj_helper_fn.py</code> <pre><code>def fetch_nwb(query_expression, nwb_master, *attrs, **kwargs):\n    \"\"\"Get an NWB object from the given DataJoint query.\n\n    Parameters\n    ----------\n    query_expression : query\n        A DataJoint query expression (e.g., join, restrict) or a table to call fetch on.\n    nwb_master : tuple\n        Tuple (table, attr) to get the NWB filepath from.\n        i.e. absolute path to NWB file can be obtained by looking up attr column of table\n        table is usually Nwbfile or AnalysisNwbfile;\n        attr is usually 'nwb_file_abs_path' or 'analysis_file_abs_path'\n    *attrs : list\n        Attributes from normal DataJoint fetch call.\n    **kwargs : dict\n        Keyword arguments from normal DataJoint fetch call.\n\n    Returns\n    -------\n    nwb_objects : list\n        List of dicts containing fetch results and NWB objects.\n    \"\"\"\n    kwargs[\"as_dict\"] = True  # force return as dictionary\n    tbl, attr_name = nwb_master\n\n    if not attrs:\n        attrs = query_expression.heading.names\n\n    # get the list of analysis or nwb files\n    file_name_str = (\n        \"analysis_file_name\" if \"analysis\" in nwb_master[1] else \"nwb_file_name\"\n    )\n    # TODO: avoid this import?\n    from ..common.common_nwbfile import AnalysisNwbfile, Nwbfile\n\n    file_path_fn = (\n        AnalysisNwbfile.get_abs_path\n        if \"analysis\" in nwb_master[1]\n        else Nwbfile.get_abs_path\n    )\n\n    # TODO: check that the query_expression restricts tbl - CBroz\n    nwb_files = (\n        query_expression * tbl.proj(nwb2load_filepath=attr_name)\n    ).fetch(file_name_str)\n    for file_name in nwb_files:\n        file_path = file_path_fn(file_name)\n        if not os.path.exists(file_path):\n            # retrieve the file from kachery. This also opens the file and stores the file object\n            get_nwb_file(file_path)\n\n    rec_dicts = (\n        query_expression * tbl.proj(nwb2load_filepath=attr_name)\n    ).fetch(*attrs, \"nwb2load_filepath\", **kwargs)\n\n    if not rec_dicts or not np.any(\n        [\"object_id\" in key for key in rec_dicts[0]]\n    ):\n        return rec_dicts\n\n    ret = []\n    for rec_dict in rec_dicts:\n        nwbf = get_nwb_file(rec_dict.pop(\"nwb2load_filepath\"))\n        # for each attr that contains substring 'object_id', store key-value: attr name to NWB object\n        # remove '_object_id' from attr name\n        nwb_objs = {\n            id_attr.replace(\"_object_id\", \"\"): _get_nwb_object(\n                nwbf.objects, rec_dict[id_attr]\n            )\n            for id_attr in attrs\n            if \"object_id\" in id_attr and rec_dict[id_attr] != \"\"\n        }\n        ret.append({**rec_dict, **nwb_objs})\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/", "title": "dj_merge_tables.py", "text": ""}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge", "title": "<code>Merge</code>", "text": "<p>             Bases: <code>Manual</code></p> <p>Adds funcs to support standard Merge table operations.</p> <p>Many methods have the @classmethod decorator to permit MergeTable.method() symtax. This makes access to instance attributes (e.g., (MergeTable &amp; \"example='restriction'\").restriction) harder, but these attributes have limited utility when the user wants to, for example, restrict the merged view rather than the master table itself.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>class Merge(dj.Manual):\n    \"\"\"Adds funcs to support standard Merge table operations.\n\n    Many methods have the @classmethod decorator to permit MergeTable.method()\n    symtax. This makes access to instance attributes (e.g., (MergeTable &amp;\n    \"example='restriction'\").restriction) harder, but these attributes have\n    limited utility when the user wants to, for example, restrict the merged\n    view rather than the master table itself.\n    \"\"\"\n\n    def __init__(self):\n        super().__init__()\n        self._reserved_pk = RESERVED_PRIMARY_KEY\n        self._reserved_sk = RESERVED_SECONDARY_KEY\n        merge_def = (\n            f\"\\n    {self._reserved_pk}: uuid\\n    ---\\n\"\n            + f\"    {self._reserved_sk}: varchar({RESERVED_SK_LENGTH})\\n    \"\n        )\n        if not self.is_declared:\n            # remove comments after # from each line of definition\n            if self._remove_comments(self.definition) != merge_def:\n                logger.warn(\n                    \"Merge table with non-default definition\\n\\t\"\n                    + f\"Expected: {merge_def.strip()}\\n\\t\"\n                    + f\"Actual  : {self.definition.strip()}\"\n                )\n            for part in self.parts(as_objects=True):\n                if part.primary_key != self.primary_key:\n                    logger.warn(\n                        f\"Unexpected primary key in {part.table_name}\"\n                        + f\"\\n\\tExpected: {self.primary_key}\"\n                        + f\"\\n\\tActual  : {part.primary_key}\"\n                    )\n        self._source_class_dict = {}\n\n    def _remove_comments(self, definition):\n        \"\"\"Use regular expressions to remove comments and blank lines\"\"\"\n        return re.sub(  # First remove comments, then blank lines\n            r\"\\n\\s*\\n\", \"\\n\", re.sub(r\"#.*\\n\", \"\\n\", definition)\n        )\n\n    @staticmethod\n    def _part_name(part=None):\n        \"\"\"Return the CamelCase name of a part table\"\"\"\n        if not isinstance(part, str):\n            part = part.table_name\n        return to_camel_case(part.split(\"__\")[-1].strip(\"`\"))\n\n    def get_source_from_key(self, key: dict) -&gt; str:\n        \"\"\"Return the source of a given key\"\"\"\n        return self._normalize_source(key)\n\n    def parts(self, camel_case=False, *args, **kwargs) -&gt; list:\n        \"\"\"Return a list of part tables, add option for CamelCase names.\n\n        See DataJoint `parts` for additional arguments. If camel_case is True,\n        forces return of strings rather than objects.\n        \"\"\"\n        self._ensure_dependencies_loaded()\n\n        if camel_case and kwargs.get(\"as_objects\"):\n            logger.warning(\n                \"Overriding as_objects=True to return CamelCase part names.\"\n            )\n            kwargs[\"as_objects\"] = False\n\n        parts = super().parts(*args, **kwargs)\n\n        if camel_case:\n            parts = [self._part_name(part) for part in parts]\n\n        return parts\n\n    @classmethod\n    def _merge_restrict_parts(\n        cls,\n        restriction: str = True,\n        as_objects: bool = True,\n        return_empties: bool = True,\n        add_invalid_restrict: bool = True,\n    ) -&gt; list:\n        \"\"\"Returns a list of parts with restrictions applied.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the parts. Default True, no restrictions.\n        as_objects: bool, optional\n            Default True. Return part tables as objects\n        return_empties: bool, optional\n            Default True. Return empty part tables\n        add_invalid_restrict: bool, optional\n            Default True. Include part for which the restriction is invalid.\n\n        Returns\n        ------\n        list\n            list of datajoint tables, parts of Merge Table\n        \"\"\"\n\n        cls._ensure_dependencies_loaded()\n\n        if not restriction:\n            restriction = True\n\n        # Normalize restriction to sql string\n        restr_str = make_condition(cls(), restriction, set())\n\n        parts_all = cls.parts(as_objects=True)\n        # If the restriction makes ref to a source, we only want that part\n        if (\n            not return_empties\n            and isinstance(restr_str, str)\n            and f\"`{cls()._reserved_sk}`\" in restr_str\n        ):\n            parts_all = [\n                part\n                for part in parts_all\n                if from_camel_case(\n                    restr_str.split(f'`{cls()._reserved_sk}`=\"')[-1].split('\"')[\n                        0\n                    ]\n                )  # Only look at source part table\n                in part.full_table_name\n            ]\n        if isinstance(restriction, dict):  # restr by source already done above\n            _ = restriction.pop(cls()._reserved_sk, None)  # won't work for str\n            # If a dict restriction has all invalid keys, it is treated as True\n            if not add_invalid_restrict:\n                parts_all = [  # so exclude tables w/ nonmatching attrs\n                    p\n                    for p in parts_all\n                    if all([k in p.heading.names for k in restriction.keys()])\n                ]\n\n        parts = []\n        for part in parts_all:\n            try:\n                parts.append(part.restrict(restriction))\n            except DataJointError:  # If restriction not valid on given part\n                if add_invalid_restrict:\n                    parts.append(part)\n\n        if not return_empties:\n            parts = [p for p in parts if len(p)]\n        if not as_objects:\n            parts = [p.full_table_name for p in parts]\n\n        return parts\n\n    @classmethod\n    def _merge_restrict_parents(\n        cls,\n        restriction: str = True,\n        parent_name: str = None,\n        as_objects: bool = True,\n        return_empties: bool = True,\n        add_invalid_restrict: bool = True,\n    ) -&gt; list:\n        \"\"\"Returns a list of part parents with restrictions applied.\n\n        Rather than part tables, we look at parents of those parts, the source\n        of the data.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the returned parent. Default True, no\n            restrictions.\n        parent_name: str, optional\n            CamelCase name of the parent.\n        as_objects: bool, optional\n            Default True. Return part tables as objects\n        return_empties: bool, optional\n            Default True. Return empty part tables\n        add_invalid_restrict: bool, optional\n            Default True. Include part for which the restriction is invalid.\n\n        Returns\n        ------\n        list\n            list of datajoint tables, parents of parts of Merge Table\n        \"\"\"\n        # .restrict(restriction) does not work on returned part FreeTable\n        # &amp; part.fetch below restricts parent to entries in merge table\n        part_parents = [\n            parent\n            &amp; part.fetch(*part.heading.secondary_attributes, as_dict=True)\n            for part in cls()._merge_restrict_parts(\n                restriction=restriction,\n                return_empties=return_empties,\n                add_invalid_restrict=add_invalid_restrict,\n            )\n            for parent in part.parents(as_objects=True)  # ID respective parents\n            if cls().table_name not in parent.full_table_name  # Not merge table\n        ]\n        if parent_name:\n            part_parents = [\n                p\n                for p in part_parents\n                if from_camel_case(parent_name) in p.full_table_name\n            ]\n        if not as_objects:\n            part_parents = [p.full_table_name for p in part_parents]\n\n        return part_parents\n\n    @classmethod\n    def _merge_repr(\n        cls, restriction: str = True, include_empties=False\n    ) -&gt; dj.expression.Union:\n        \"\"\"Merged view, including null entries for columns unique to one part.\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the merged view\n        include_empties: bool, optional\n            Default False. Add columns for empty parts.\n\n        Returns\n        ------\n        datajoint.expression.Union\n        \"\"\"\n\n        parts = [\n            cls() * p  # join with master to include sec key (i.e., 'source')\n            for p in cls._merge_restrict_parts(\n                restriction=restriction,\n                add_invalid_restrict=False,\n                return_empties=include_empties,\n            )\n        ]\n        if not parts:\n            logger.warning(\"No parts found. Try adjusting restriction.\")\n            return\n\n        attr_dict = {  # NULL for non-numeric, 0 for numeric\n            attr.name: \"0\" if attr.numeric else \"NULL\"\n            for attr in iter_chain.from_iterable(\n                part.heading.attributes.values() for part in parts\n            )\n        }\n\n        def _proj_part(part):\n            \"\"\"Project part, adding NULL/0 for missing attributes\"\"\"\n            return dj.U(*attr_dict.keys()) * part.proj(\n                ...,  # include all attributes from part\n                **{\n                    k: v\n                    for k, v in attr_dict.items()\n                    if k not in part.heading.names\n                },\n            )\n\n        query = _proj_part(parts[0])  # start with first part\n        for part in parts[1:]:  # add remaining parts\n            query += _proj_part(part)\n\n        return query\n\n    @classmethod\n    def _merge_insert(cls, rows: list, part_name: str = None, **kwargs) -&gt; None:\n        \"\"\"Insert rows into merge, ensuring data exists in part parent(s).\n\n        Parameters\n        ---------\n        rows: List[dict]\n            An iterable where an element is a dictionary.\n        part: str, optional\n            CamelCase name of the part table\n\n        Raises\n        ------\n        TypeError\n            If rows is not a list of dicts\n        ValueError\n            If data doesn't exist in part parents, integrity error\n        \"\"\"\n        cls._ensure_dependencies_loaded()\n\n        type_err_msg = \"Input `rows` must be a list of dictionaries\"\n        try:\n            for r in iter(rows):\n                if not isinstance(r, dict):\n                    raise TypeError(type_err_msg)\n        except TypeError:\n            raise TypeError(type_err_msg)\n\n        parts = cls._merge_restrict_parts(as_objects=True)\n        if part_name:\n            parts = [\n                p\n                for p in parts\n                if from_camel_case(part_name) in p.full_table_name\n            ]\n\n        master_entries = []\n        parts_entries = {p: [] for p in parts}\n        for row in rows:\n            keys = []  # empty to-be-inserted keys\n            for part in parts:  # check each part\n                part_name = cls._part_name(part)\n                part_parent = part.parents(as_objects=True)[-1]\n                if part_parent &amp; row:  # if row is in part parent\n                    keys = (part_parent &amp; row).fetch(\"KEY\")  # get pk\n                    if len(keys) &gt; 1:\n                        raise ValueError(\n                            \"Ambiguous entry. Data has mult rows in \"\n                            + f\"{part_name}:\\n\\tData:{row}\\n\\t{keys}\"\n                        )\n                    key = keys[0]\n                    master_sk = {cls()._reserved_sk: part_name}\n                    uuid = dj.hash.key_hash(key | master_sk)\n                    master_pk = {cls()._reserved_pk: uuid}\n\n                    master_entries.append({**master_pk, **master_sk})\n                    parts_entries[part].append({**master_pk, **key})\n\n            if not keys:\n                raise ValueError(\n                    \"Non-existing entry in any of the parent tables - Entry: \"\n                    + f\"{row}\"\n                )\n\n        with cls._safe_context():\n            super().insert(cls(), master_entries, **kwargs)\n            for part, part_entries in parts_entries.items():\n                part.insert(part_entries, **kwargs)\n\n    @classmethod\n    def _safe_context(cls):\n        \"\"\"Return transaction if not already in one.\"\"\"\n        return (\n            cls.connection.transaction\n            if not cls.connection.in_transaction\n            else nullcontext()\n        )\n\n    @classmethod\n    def _ensure_dependencies_loaded(cls) -&gt; None:\n        \"\"\"Ensure connection dependencies loaded.\n\n        Otherwise parts returns none\n        \"\"\"\n        if not dj.conn.connection.dependencies._loaded:\n            dj.conn.connection.dependencies.load()\n\n    def insert(self, rows: list, **kwargs):\n        \"\"\"Merges table specific insert, ensuring data exists in part parents.\n\n        Parameters\n        ---------\n        rows: List[dict]\n            An iterable where an element is a dictionary.\n\n        Raises\n        ------\n        TypeError\n            If rows is not a list of dicts\n        ValueError\n            If data doesn't exist in part parents, integrity error\n        \"\"\"\n        self._merge_insert(rows, **kwargs)\n\n    @classmethod\n    def merge_view(cls, restriction: str = True):\n        \"\"\"Prints merged view, including null entries for unique columns.\n\n        Note: To handle this Union as a table-like object, use `merge_resrict`\n\n        Parameters\n        ---------\n        restriction: str, optional\n            Restriction to apply to the merged view\n        \"\"\"\n\n        # If we overwrite `preview`, we then encounter issues with operators\n        # getting passed a `Union`, which doesn't have a method we can\n        # intercept to manage master/parts\n\n        return pprint(cls._merge_repr(restriction=restriction))\n\n    @classmethod\n    def merge_html(cls, restriction: str = True):\n        \"\"\"Displays HTML in notebooks.\"\"\"\n\n        return HTML(repr_html(cls._merge_repr(restriction=restriction)))\n\n    @classmethod\n    def merge_restrict(cls, restriction: str = True) -&gt; dj.U:\n        \"\"\"Given a restriction, return a merged view with restriction applied.\n\n        Example\n        -------\n            &gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n\n        Parameters\n        ----------\n        restriction: str\n            Restriction one would apply if `merge_view` was a real table.\n\n        Returns\n        -------\n        datajoint.Union\n            Merged view with restriction applied.\n        \"\"\"\n        return cls._merge_repr(restriction=restriction)\n\n    @classmethod\n    def merge_delete(cls, restriction: str = True, **kwargs):\n        \"\"\"Given a restriction string, delete corresponding entries.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before deletion from master/part\n            tables. If not provided, delete all entries.\n        kwargs: dict\n            Additional keyword arguments for DataJoint delete.\n\n        Example\n        -------\n            &gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n        \"\"\"\n        uuids = [\n            {k: v}\n            for entry in cls.merge_restrict(restriction).fetch(\"KEY\")\n            for k, v in entry.items()\n            if k == cls()._reserved_pk\n        ]\n        (cls() &amp; uuids).delete(**kwargs)\n\n    @classmethod\n    def merge_delete_parent(\n        cls, restriction: str = True, dry_run=True, **kwargs\n    ) -&gt; list:\n        \"\"\"Delete entries from merge master, part, and respective part parents\n\n        Note: Clears merge entries from their respective parents.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before deletion from parents. If not\n            provided, delete all entries present in Merge Table.\n        dry_run: bool\n            Default True. If true, return list of tables with entries that would\n            be deleted. Otherwise, table entries.\n        kwargs: dict\n            Additional keyword arguments for DataJoint delete.\n        \"\"\"\n        part_parents = cls._merge_restrict_parents(\n            restriction=restriction, as_objects=True, return_empties=False\n        )\n\n        if dry_run:\n            return part_parents\n\n        merge_ids = cls.merge_restrict(restriction).fetch(\n            RESERVED_PRIMARY_KEY, as_dict=True\n        )\n\n        # CB: Removed transaction protection here bc 'no' confirmation resp\n        # still resulted in deletes. If re-add, consider transaction=False\n        super().delete((cls &amp; merge_ids), **kwargs)\n\n        if cls &amp; merge_ids:  # If 'no' on del prompt from above, skip below\n            return  # User can still abort del below, but yes/no is unlikly\n\n        for part_parent in part_parents:\n            super().delete(part_parent, **kwargs)\n\n    def fetch_nwb(\n        self,\n        restriction: str = None,\n        multi_source=False,\n        disable_warning=False,\n        *attrs,\n        **kwargs,\n    ):\n        \"\"\"Return the (Analysis)Nwbfile file linked in the source.\n\n        Relies on SpyglassMixin._nwb_table_tuple to determine the table to\n        fetch from and the appropriate path attribute to return.\n\n        Parameters\n        ----------\n        restriction: str, optional\n            Restriction to apply to parents before running fetch. Default True.\n        multi_source: bool\n            Return from multiple parents. Default False.\n        \"\"\"\n        if isinstance(self, dict):\n            raise ValueError(\"Try replacing Merge.method with Merge().method\")\n        restriction = restriction or self.restriction or True\n\n        return self.merge_restrict_class(restriction).fetch_nwb()\n\n    @classmethod\n    def merge_get_part(\n        cls,\n        restriction: str = True,\n        join_master: bool = False,\n        restrict_part=True,\n        multi_source=False,\n        return_empties=False,\n    ) -&gt; dj.Table:\n        \"\"\"Retrieve part table from a restricted Merge table.\n\n        Note: unlike other Merge Table methods, returns the native table, not\n        a FreeTable\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining part to return.\n            Default True.\n        join_master: bool\n            Join part with Merge master to show source field. Default False.\n        restrict_part: bool\n            Apply restriction to part. Default True. If False, return the\n            native part table.\n        multi_source: bool\n            Return multiple parts. Default False.\n        return_empties: bool\n            Default False. Return empty part tables.\n\n        Returns\n        ------\n        Union[dj.Table, List[dj.Table]]\n            Native part table(s) of Merge. If `multi_source`, returns list.\n\n        Example\n        -------\n            &gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n            &gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n\n        Raises\n        ------\n        ValueError\n            If multiple sources are found, but not expected lists and suggests\n            restricting\n        \"\"\"\n        sources = [\n            cls._part_name(part)  # friendly part name\n            for part in cls._merge_restrict_parts(\n                restriction=restriction,\n                as_objects=False,\n                return_empties=return_empties,\n                add_invalid_restrict=False,\n            )\n        ]\n\n        if not multi_source and len(sources) != 1:\n            raise ValueError(\n                f\"Found {len(sources)} potential parts: {sources}\\n\\t\"\n                + \"Try adding a restriction before invoking `get_part`.\\n\\t\"\n                + \"Or permitting multiple sources with `multi_source=True`.\"\n            )\n        if len(sources) == 0:\n            return None\n\n        parts = [\n            (\n                getattr(cls, source)().restrict(restriction)\n                if restrict_part  # Re-apply restriction or don't\n                else getattr(cls, source)()\n            )\n            for source in sources\n        ]\n        if join_master:\n            parts = [cls * part for part in parts]\n\n        return parts if multi_source else parts[0]\n\n    @classmethod\n    def merge_get_parent(\n        cls,\n        restriction: str = True,\n        join_master: bool = False,\n        multi_source: bool = False,\n        return_empties: bool = False,\n        add_invalid_restrict: bool = True,\n    ) -&gt; dj.FreeTable:\n        \"\"\"Returns a list of part parents with restrictions applied.\n\n        Rather than part tables, we look at parents of those parts, the source\n        of the data, and only the rows that have keys inserted in the merge\n        table.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining parent to return.\n            Default True.\n        join_master: bool\n            Default False. Join part with Merge master to show uuid and source\n        multi_source: bool\n            Return multiple parents. Default False.\n        return_empties: bool\n            Default False. Return empty parent tables.\n        add_invalid_restrict: bool\n            Default True. Include parent for which the restriction is invalid.\n\n        Returns\n        ------\n        dj.FreeTable\n            Parent of parts of Merge Table as FreeTable.\n        \"\"\"\n\n        part_parents = cls._merge_restrict_parents(\n            restriction=restriction,\n            as_objects=True,\n            return_empties=return_empties,\n            add_invalid_restrict=add_invalid_restrict,\n        )\n\n        if not multi_source and len(part_parents) != 1:\n            raise ValueError(\n                f\"Found  {len(part_parents)} potential parents: {part_parents}\"\n                + \"\\n\\tTry adding a string restriction when invoking \"\n                + \"`get_parent`. Or permitting multiple sources with \"\n                + \"`multi_source=True`.\"\n            )\n\n        if join_master:\n            part_parents = [cls * part for part in part_parents]\n\n        return part_parents if multi_source else part_parents[0]\n\n    @property\n    def source_class_dict(self) -&gt; dict:\n        # NOTE: fails if table is aliased in dj.Part but not merge script\n        # i.e., must import aliased table as part name\n        if not self._source_class_dict:\n            module = getmodule(self)\n            self._source_class_dict = {\n                part_name: getattr(module, part_name)\n                for part_name in self.parts(camel_case=True)\n            }\n        return self._source_class_dict\n\n    def _normalize_source(\n        self, source: Union[str, dj.Table, dj.condition.AndList, dict]\n    ) -&gt; str:\n        fetched_source = None\n        if isinstance(source, (Merge, dj.condition.AndList)):\n            try:\n                fetched_source = (self &amp; source).fetch(self._reserved_sk)\n            except DataJointError:\n                raise ValueError(f\"Unable to find source for {source}\")\n            source = fetched_source[0]\n            if len(fetched_source) &gt; 1:\n                logger.warn(f\"Multiple sources. Selecting first: {source}.\")\n        if isinstance(source, dj.Table):\n            source = self._part_name(source)\n        if isinstance(source, dict):\n            source = self._part_name(self.merge_get_parent(source))\n\n        return source\n\n    def merge_get_parent_class(self, source: str) -&gt; dj.Table:\n        \"\"\"Return the class of the parent table for a given CamelCase source.\n\n        Parameters\n        ----------\n        source: Union[str, dict, dj.Table]\n            Accepts a CamelCase name of the source, or key as a dict, or a part\n            table.\n\n        Returns\n        -------\n        dj.Table\n            Class instance of the parent table, including class methods.\n        \"\"\"\n\n        ret = self.source_class_dict.get(self._normalize_source(source))\n        if not ret:\n            logger.error(\n                f\"No source class found for {source}: \\n\\t\"\n                + f\"{self.parts(camel_case=True)}\"\n            )\n        return ret\n\n    def merge_restrict_class(self, key: dict) -&gt; dj.Table:\n        \"\"\"Returns native parent class, restricted with key.\"\"\"\n        parent_key = self.merge_get_parent(key).fetch(\"KEY\", as_dict=True)\n\n        if len(parent_key) &gt; 1:\n            raise ValueError(\n                f\"Ambiguous entry. Data has mult rows in parent:\\n\\tData:{key}\"\n                + f\"\\n\\t{parent_key}\"\n            )\n\n        parent_class = self.merge_get_parent_class(key)\n        return parent_class &amp; parent_key\n\n    @classmethod\n    def merge_fetch(self, restriction: str = True, *attrs, **kwargs) -&gt; list:\n        \"\"\"Perform a fetch across all parts. If &gt;1 result, return as a list.\n\n        Parameters\n        ----------\n        restriction: str\n            Optional restriction to apply before determining parent to return.\n            Default True.\n        attrs, kwargs\n            arguments passed to DataJoint `fetch` call\n\n        Returns\n        -------\n        Union[ List[np.array], List[dict], List[pd.DataFrame] ]\n            Table contents, with type determined by kwargs\n        \"\"\"\n        results = []\n        parts = self()._merge_restrict_parts(\n            restriction=restriction,\n            as_objects=True,\n            return_empties=False,\n            add_invalid_restrict=False,\n        )\n\n        for part in parts:\n            try:\n                results.extend(part.fetch(*attrs, **kwargs))\n            except DataJointError as e:\n                logger.warn(\n                    f\"{e.args[0]} Skipping \"\n                    + to_camel_case(part.table_name.split(\"__\")[-1])\n                )\n\n        # Note: this could collapse results like merge_view, but user may call\n        # for recarray, pd.DataFrame, or dict, and fetched contents differ if\n        # attrs or \"KEY\" called. Intercept format, merge, and then transform?\n\n        if not results:\n            logger.info(\n                \"No merge_fetch results.\\n\\t\"\n                + \"If not restricting, try: `M.merge_fetch(True,'attr')\\n\\t\"\n                + \"If restricting by source, use dict: \"\n                + \"`M.merge_fetch({'source':'X'})\"\n            )\n        return results[0] if len(results) == 1 else results\n\n    def merge_populate(self, source: str, keys=None):\n        \"\"\"Populate the merge table with entries from the source table.\"\"\"\n        logger.warning(\"CBroz: Not fully tested. Use with caution.\")\n        parent_class = self.merge_get_parent_class(source)\n        if not keys:\n            keys = parent_class.key_source\n        parent_class.populate(keys)\n        successes = (parent_class &amp; keys).fetch(\"KEY\", as_dict=True)\n        self.insert(successes)\n\n    def delete(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n        for part in self.merge_get_part(\n            restriction=self.restriction,\n            multi_source=True,\n            return_empties=False,\n        ):\n            part.delete(force_permission=force_permission, *args, **kwargs)\n\n    def super_delete(self, *args, **kwargs):\n        \"\"\"Alias for datajoint.table.Table.delete.\n\n        Added to support MRO of SpyglassMixin\"\"\"\n        logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n\n        self._log_use(start=time(), super_delete=True)\n        super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.get_source_from_key", "title": "<code>get_source_from_key(key)</code>", "text": "<p>Return the source of a given key</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def get_source_from_key(self, key: dict) -&gt; str:\n    \"\"\"Return the source of a given key\"\"\"\n    return self._normalize_source(key)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.parts", "title": "<code>parts(camel_case=False, *args, **kwargs)</code>", "text": "<p>Return a list of part tables, add option for CamelCase names.</p> <p>See DataJoint <code>parts</code> for additional arguments. If camel_case is True, forces return of strings rather than objects.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def parts(self, camel_case=False, *args, **kwargs) -&gt; list:\n    \"\"\"Return a list of part tables, add option for CamelCase names.\n\n    See DataJoint `parts` for additional arguments. If camel_case is True,\n    forces return of strings rather than objects.\n    \"\"\"\n    self._ensure_dependencies_loaded()\n\n    if camel_case and kwargs.get(\"as_objects\"):\n        logger.warning(\n            \"Overriding as_objects=True to return CamelCase part names.\"\n        )\n        kwargs[\"as_objects\"] = False\n\n    parts = super().parts(*args, **kwargs)\n\n    if camel_case:\n        parts = [self._part_name(part) for part in parts]\n\n    return parts\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.insert", "title": "<code>insert(rows, **kwargs)</code>", "text": "<p>Merges table specific insert, ensuring data exists in part parents.</p> <p>Parameters:</p> Name Type Description Default <code>rows</code> <code>list</code> <p>An iterable where an element is a dictionary.</p> required <p>Raises:</p> Type Description <code>TypeError</code> <p>If rows is not a list of dicts</p> <code>ValueError</code> <p>If data doesn't exist in part parents, integrity error</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def insert(self, rows: list, **kwargs):\n    \"\"\"Merges table specific insert, ensuring data exists in part parents.\n\n    Parameters\n    ---------\n    rows: List[dict]\n        An iterable where an element is a dictionary.\n\n    Raises\n    ------\n    TypeError\n        If rows is not a list of dicts\n    ValueError\n        If data doesn't exist in part parents, integrity error\n    \"\"\"\n    self._merge_insert(rows, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_view", "title": "<code>merge_view(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Prints merged view, including null entries for unique columns.</p> <p>Note: To handle this Union as a table-like object, use <code>merge_resrict</code></p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to the merged view</p> <code>True</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_view(cls, restriction: str = True):\n    \"\"\"Prints merged view, including null entries for unique columns.\n\n    Note: To handle this Union as a table-like object, use `merge_resrict`\n\n    Parameters\n    ---------\n    restriction: str, optional\n        Restriction to apply to the merged view\n    \"\"\"\n\n    # If we overwrite `preview`, we then encounter issues with operators\n    # getting passed a `Union`, which doesn't have a method we can\n    # intercept to manage master/parts\n\n    return pprint(cls._merge_repr(restriction=restriction))\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_html", "title": "<code>merge_html(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Displays HTML in notebooks.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_html(cls, restriction: str = True):\n    \"\"\"Displays HTML in notebooks.\"\"\"\n\n    return HTML(repr_html(cls._merge_repr(restriction=restriction)))\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict", "title": "<code>merge_restrict(restriction=True)</code>  <code>classmethod</code>", "text": "<p>Given a restriction, return a merged view with restriction applied.</p>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n</code></pre> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction one would apply if <code>merge_view</code> was a real table.</p> <code>True</code> <p>Returns:</p> Type Description <code>Union</code> <p>Merged view with restriction applied.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_restrict(cls, restriction: str = True) -&gt; dj.U:\n    \"\"\"Given a restriction, return a merged view with restriction applied.\n\n    Example\n    -------\n        &gt;&gt;&gt; MergeTable.merge_restrict(\"field = 1\")\n\n    Parameters\n    ----------\n    restriction: str\n        Restriction one would apply if `merge_view` was a real table.\n\n    Returns\n    -------\n    datajoint.Union\n        Merged view with restriction applied.\n    \"\"\"\n    return cls._merge_repr(restriction=restriction)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete", "title": "<code>merge_delete(restriction=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Given a restriction string, delete corresponding entries.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from master/part tables. If not provided, delete all entries.</p> <code>True</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n</code></pre> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_delete(cls, restriction: str = True, **kwargs):\n    \"\"\"Given a restriction string, delete corresponding entries.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before deletion from master/part\n        tables. If not provided, delete all entries.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n\n    Example\n    -------\n        &gt;&gt;&gt; MergeTable.merge_delete(\"field = 1\")\n    \"\"\"\n    uuids = [\n        {k: v}\n        for entry in cls.merge_restrict(restriction).fetch(\"KEY\")\n        for k, v in entry.items()\n        if k == cls()._reserved_pk\n    ]\n    (cls() &amp; uuids).delete(**kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_delete_parent", "title": "<code>merge_delete_parent(restriction=True, dry_run=True, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Delete entries from merge master, part, and respective part parents</p> <p>Note: Clears merge entries from their respective parents.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from parents. If not provided, delete all entries present in Merge Table.</p> <code>True</code> <code>dry_run</code> <p>Default True. If true, return list of tables with entries that would be deleted. Otherwise, table entries.</p> <code>True</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_delete_parent(\n    cls, restriction: str = True, dry_run=True, **kwargs\n) -&gt; list:\n    \"\"\"Delete entries from merge master, part, and respective part parents\n\n    Note: Clears merge entries from their respective parents.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before deletion from parents. If not\n        provided, delete all entries present in Merge Table.\n    dry_run: bool\n        Default True. If true, return list of tables with entries that would\n        be deleted. Otherwise, table entries.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n    \"\"\"\n    part_parents = cls._merge_restrict_parents(\n        restriction=restriction, as_objects=True, return_empties=False\n    )\n\n    if dry_run:\n        return part_parents\n\n    merge_ids = cls.merge_restrict(restriction).fetch(\n        RESERVED_PRIMARY_KEY, as_dict=True\n    )\n\n    # CB: Removed transaction protection here bc 'no' confirmation resp\n    # still resulted in deletes. If re-add, consider transaction=False\n    super().delete((cls &amp; merge_ids), **kwargs)\n\n    if cls &amp; merge_ids:  # If 'no' on del prompt from above, skip below\n        return  # User can still abort del below, but yes/no is unlikly\n\n    for part_parent in part_parents:\n        super().delete(part_parent, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.fetch_nwb", "title": "<code>fetch_nwb(restriction=None, multi_source=False, disable_warning=False, *attrs, **kwargs)</code>", "text": "<p>Return the (Analysis)Nwbfile file linked in the source.</p> <p>Relies on SpyglassMixin._nwb_table_tuple to determine the table to fetch from and the appropriate path attribute to return.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to parents before running fetch. Default True.</p> <code>None</code> <code>multi_source</code> <p>Return from multiple parents. Default False.</p> <code>False</code> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def fetch_nwb(\n    self,\n    restriction: str = None,\n    multi_source=False,\n    disable_warning=False,\n    *attrs,\n    **kwargs,\n):\n    \"\"\"Return the (Analysis)Nwbfile file linked in the source.\n\n    Relies on SpyglassMixin._nwb_table_tuple to determine the table to\n    fetch from and the appropriate path attribute to return.\n\n    Parameters\n    ----------\n    restriction: str, optional\n        Restriction to apply to parents before running fetch. Default True.\n    multi_source: bool\n        Return from multiple parents. Default False.\n    \"\"\"\n    if isinstance(self, dict):\n        raise ValueError(\"Try replacing Merge.method with Merge().method\")\n    restriction = restriction or self.restriction or True\n\n    return self.merge_restrict_class(restriction).fetch_nwb()\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_part", "title": "<code>merge_get_part(restriction=True, join_master=False, restrict_part=True, multi_source=False, return_empties=False)</code>  <code>classmethod</code>", "text": "<p>Retrieve part table from a restricted Merge table.</p> <p>Note: unlike other Merge Table methods, returns the native table, not a FreeTable</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining part to return. Default True.</p> <code>True</code> <code>join_master</code> <code>bool</code> <p>Join part with Merge master to show source field. Default False.</p> <code>False</code> <code>restrict_part</code> <p>Apply restriction to part. Default True. If False, return the native part table.</p> <code>True</code> <code>multi_source</code> <p>Return multiple parts. Default False.</p> <code>False</code> <code>return_empties</code> <p>Default False. Return empty part tables.</p> <code>False</code> <p>Returns:</p> Type Description <code>Union[Table, List[Table]]</code> <p>Native part table(s) of Merge. If <code>multi_source</code>, returns list.</p>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_part--example", "title": "Example", "text": "<pre><code>&gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n&gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n</code></pre> <p>Raises:</p> Type Description <code>ValueError</code> <p>If multiple sources are found, but not expected lists and suggests restricting</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_get_part(\n    cls,\n    restriction: str = True,\n    join_master: bool = False,\n    restrict_part=True,\n    multi_source=False,\n    return_empties=False,\n) -&gt; dj.Table:\n    \"\"\"Retrieve part table from a restricted Merge table.\n\n    Note: unlike other Merge Table methods, returns the native table, not\n    a FreeTable\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining part to return.\n        Default True.\n    join_master: bool\n        Join part with Merge master to show source field. Default False.\n    restrict_part: bool\n        Apply restriction to part. Default True. If False, return the\n        native part table.\n    multi_source: bool\n        Return multiple parts. Default False.\n    return_empties: bool\n        Default False. Return empty part tables.\n\n    Returns\n    ------\n    Union[dj.Table, List[dj.Table]]\n        Native part table(s) of Merge. If `multi_source`, returns list.\n\n    Example\n    -------\n        &gt;&gt;&gt; (MergeTable &amp; restriction).get_part_table()\n        &gt;&gt;&gt; MergeTable().merge_get_part(restriction, join_master=True)\n\n    Raises\n    ------\n    ValueError\n        If multiple sources are found, but not expected lists and suggests\n        restricting\n    \"\"\"\n    sources = [\n        cls._part_name(part)  # friendly part name\n        for part in cls._merge_restrict_parts(\n            restriction=restriction,\n            as_objects=False,\n            return_empties=return_empties,\n            add_invalid_restrict=False,\n        )\n    ]\n\n    if not multi_source and len(sources) != 1:\n        raise ValueError(\n            f\"Found {len(sources)} potential parts: {sources}\\n\\t\"\n            + \"Try adding a restriction before invoking `get_part`.\\n\\t\"\n            + \"Or permitting multiple sources with `multi_source=True`.\"\n        )\n    if len(sources) == 0:\n        return None\n\n    parts = [\n        (\n            getattr(cls, source)().restrict(restriction)\n            if restrict_part  # Re-apply restriction or don't\n            else getattr(cls, source)()\n        )\n        for source in sources\n    ]\n    if join_master:\n        parts = [cls * part for part in parts]\n\n    return parts if multi_source else parts[0]\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_parent", "title": "<code>merge_get_parent(restriction=True, join_master=False, multi_source=False, return_empties=False, add_invalid_restrict=True)</code>  <code>classmethod</code>", "text": "<p>Returns a list of part parents with restrictions applied.</p> <p>Rather than part tables, we look at parents of those parts, the source of the data, and only the rows that have keys inserted in the merge table.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining parent to return. Default True.</p> <code>True</code> <code>join_master</code> <code>bool</code> <p>Default False. Join part with Merge master to show uuid and source</p> <code>False</code> <code>multi_source</code> <code>bool</code> <p>Return multiple parents. Default False.</p> <code>False</code> <code>return_empties</code> <code>bool</code> <p>Default False. Return empty parent tables.</p> <code>False</code> <code>add_invalid_restrict</code> <code>bool</code> <p>Default True. Include parent for which the restriction is invalid.</p> <code>True</code> <p>Returns:</p> Type Description <code>FreeTable</code> <p>Parent of parts of Merge Table as FreeTable.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_get_parent(\n    cls,\n    restriction: str = True,\n    join_master: bool = False,\n    multi_source: bool = False,\n    return_empties: bool = False,\n    add_invalid_restrict: bool = True,\n) -&gt; dj.FreeTable:\n    \"\"\"Returns a list of part parents with restrictions applied.\n\n    Rather than part tables, we look at parents of those parts, the source\n    of the data, and only the rows that have keys inserted in the merge\n    table.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining parent to return.\n        Default True.\n    join_master: bool\n        Default False. Join part with Merge master to show uuid and source\n    multi_source: bool\n        Return multiple parents. Default False.\n    return_empties: bool\n        Default False. Return empty parent tables.\n    add_invalid_restrict: bool\n        Default True. Include parent for which the restriction is invalid.\n\n    Returns\n    ------\n    dj.FreeTable\n        Parent of parts of Merge Table as FreeTable.\n    \"\"\"\n\n    part_parents = cls._merge_restrict_parents(\n        restriction=restriction,\n        as_objects=True,\n        return_empties=return_empties,\n        add_invalid_restrict=add_invalid_restrict,\n    )\n\n    if not multi_source and len(part_parents) != 1:\n        raise ValueError(\n            f\"Found  {len(part_parents)} potential parents: {part_parents}\"\n            + \"\\n\\tTry adding a string restriction when invoking \"\n            + \"`get_parent`. Or permitting multiple sources with \"\n            + \"`multi_source=True`.\"\n        )\n\n    if join_master:\n        part_parents = [cls * part for part in part_parents]\n\n    return part_parents if multi_source else part_parents[0]\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_get_parent_class", "title": "<code>merge_get_parent_class(source)</code>", "text": "<p>Return the class of the parent table for a given CamelCase source.</p> <p>Parameters:</p> Name Type Description Default <code>source</code> <code>str</code> <p>Accepts a CamelCase name of the source, or key as a dict, or a part table.</p> required <p>Returns:</p> Type Description <code>Table</code> <p>Class instance of the parent table, including class methods.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_get_parent_class(self, source: str) -&gt; dj.Table:\n    \"\"\"Return the class of the parent table for a given CamelCase source.\n\n    Parameters\n    ----------\n    source: Union[str, dict, dj.Table]\n        Accepts a CamelCase name of the source, or key as a dict, or a part\n        table.\n\n    Returns\n    -------\n    dj.Table\n        Class instance of the parent table, including class methods.\n    \"\"\"\n\n    ret = self.source_class_dict.get(self._normalize_source(source))\n    if not ret:\n        logger.error(\n            f\"No source class found for {source}: \\n\\t\"\n            + f\"{self.parts(camel_case=True)}\"\n        )\n    return ret\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_restrict_class", "title": "<code>merge_restrict_class(key)</code>", "text": "<p>Returns native parent class, restricted with key.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_restrict_class(self, key: dict) -&gt; dj.Table:\n    \"\"\"Returns native parent class, restricted with key.\"\"\"\n    parent_key = self.merge_get_parent(key).fetch(\"KEY\", as_dict=True)\n\n    if len(parent_key) &gt; 1:\n        raise ValueError(\n            f\"Ambiguous entry. Data has mult rows in parent:\\n\\tData:{key}\"\n            + f\"\\n\\t{parent_key}\"\n        )\n\n    parent_class = self.merge_get_parent_class(key)\n    return parent_class &amp; parent_key\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_fetch", "title": "<code>merge_fetch(restriction=True, *attrs, **kwargs)</code>  <code>classmethod</code>", "text": "<p>Perform a fetch across all parts. If &gt;1 result, return as a list.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Optional restriction to apply before determining parent to return. Default True.</p> <code>True</code> <code>attrs</code> <p>arguments passed to DataJoint <code>fetch</code> call</p> <code>()</code> <code>kwargs</code> <p>arguments passed to DataJoint <code>fetch</code> call</p> <code>()</code> <p>Returns:</p> Type Description <code>Union[List[array], List[dict], List[DataFrame]]</code> <p>Table contents, with type determined by kwargs</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>@classmethod\ndef merge_fetch(self, restriction: str = True, *attrs, **kwargs) -&gt; list:\n    \"\"\"Perform a fetch across all parts. If &gt;1 result, return as a list.\n\n    Parameters\n    ----------\n    restriction: str\n        Optional restriction to apply before determining parent to return.\n        Default True.\n    attrs, kwargs\n        arguments passed to DataJoint `fetch` call\n\n    Returns\n    -------\n    Union[ List[np.array], List[dict], List[pd.DataFrame] ]\n        Table contents, with type determined by kwargs\n    \"\"\"\n    results = []\n    parts = self()._merge_restrict_parts(\n        restriction=restriction,\n        as_objects=True,\n        return_empties=False,\n        add_invalid_restrict=False,\n    )\n\n    for part in parts:\n        try:\n            results.extend(part.fetch(*attrs, **kwargs))\n        except DataJointError as e:\n            logger.warn(\n                f\"{e.args[0]} Skipping \"\n                + to_camel_case(part.table_name.split(\"__\")[-1])\n            )\n\n    # Note: this could collapse results like merge_view, but user may call\n    # for recarray, pd.DataFrame, or dict, and fetched contents differ if\n    # attrs or \"KEY\" called. Intercept format, merge, and then transform?\n\n    if not results:\n        logger.info(\n            \"No merge_fetch results.\\n\\t\"\n            + \"If not restricting, try: `M.merge_fetch(True,'attr')\\n\\t\"\n            + \"If restricting by source, use dict: \"\n            + \"`M.merge_fetch({'source':'X'})\"\n        )\n    return results[0] if len(results) == 1 else results\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.merge_populate", "title": "<code>merge_populate(source, keys=None)</code>", "text": "<p>Populate the merge table with entries from the source table.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def merge_populate(self, source: str, keys=None):\n    \"\"\"Populate the merge table with entries from the source table.\"\"\"\n    logger.warning(\"CBroz: Not fully tested. Use with caution.\")\n    parent_class = self.merge_get_parent_class(source)\n    if not keys:\n        keys = parent_class.key_source\n    parent_class.populate(keys)\n    successes = (parent_class &amp; keys).fetch(\"KEY\", as_dict=True)\n    self.insert(successes)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.delete", "title": "<code>delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete, overwrites datajoint.table.Table.delete</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def delete(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n    for part in self.merge_get_part(\n        restriction=self.restriction,\n        multi_source=True,\n        return_empties=False,\n    ):\n        part.delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.Merge.super_delete", "title": "<code>super_delete(*args, **kwargs)</code>", "text": "<p>Alias for datajoint.table.Table.delete.</p> <p>Added to support MRO of SpyglassMixin</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def super_delete(self, *args, **kwargs):\n    \"\"\"Alias for datajoint.table.Table.delete.\n\n    Added to support MRO of SpyglassMixin\"\"\"\n    logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n\n    self._log_use(start=time(), super_delete=True)\n    super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_merge_tables/#src.spyglass.utils.dj_merge_tables.delete_downstream_merge", "title": "<code>delete_downstream_merge(table, restriction=None, dry_run=True, recurse_level=2, disable_warning=False, **kwargs)</code>", "text": "<p>Given a table/restriction, id or delete relevant downstream merge entries</p> <p>Parameters:</p> Name Type Description Default <code>table</code> <code>Table</code> <p>DataJoint table or restriction thereof</p> required <code>restriction</code> <code>str</code> <p>Optional restriction to apply before deletion from merge/part tables. If not provided, delete all downstream entries.</p> <code>None</code> <code>dry_run</code> <p>Default True. If true, return list of tuples, merge/part tables downstream of table input. Otherwise, delete merge/part table entries.</p> <code>True</code> <code>disable_warning</code> <p>Default False. If True, don't warn about restrictions on table object.</p> <code>False</code> <code>kwargs</code> <p>Additional keyword arguments for DataJoint delete.</p> <code>{}</code> <p>Returns:</p> Type Description <code>List[Tuple[Table, Table]]</code> <p>Entries in merge/part tables downstream of table input.</p> Source code in <code>src/spyglass/utils/dj_merge_tables.py</code> <pre><code>def delete_downstream_merge(\n    table: dj.Table,\n    restriction: str = None,\n    dry_run=True,\n    recurse_level=2,\n    disable_warning=False,\n    **kwargs,\n) -&gt; list:\n    \"\"\"Given a table/restriction, id or delete relevant downstream merge entries\n\n    Parameters\n    ----------\n    table: dj.Table\n        DataJoint table or restriction thereof\n    restriction: str\n        Optional restriction to apply before deletion from merge/part\n        tables. If not provided, delete all downstream entries.\n    dry_run: bool\n        Default True. If true, return list of tuples, merge/part tables\n        downstream of table input. Otherwise, delete merge/part table entries.\n    disable_warning: bool\n        Default False. If True, don't warn about restrictions on table object.\n    kwargs: dict\n        Additional keyword arguments for DataJoint delete.\n\n    Returns\n    -------\n    List[Tuple[dj.Table, dj.Table]]\n        Entries in merge/part tables downstream of table input.\n    \"\"\"\n    from spyglass.utils.dj_mixin import SpyglassMixin\n\n    if not isinstance(table, SpyglassMixin):\n        raise ValueError(\"Input must be a Spyglass Table.\")\n    table = table if isinstance(table, dj.Table) else table()\n\n    return table.delete_downstream_merge(\n        restriction=restriction, dry_run=dry_run, **kwargs\n    )\n</code></pre>"}, {"location": "api/utils/dj_mixin/", "title": "dj_mixin.py", "text": ""}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin", "title": "<code>SpyglassMixin</code>", "text": "<p>Mixin for Spyglass DataJoint tables.</p> <p>Provides methods for fetching NWBFile objects and checking user permission prior to deleting. As a mixin class, all Spyglass tables can inherit custom methods from a central location.</p> <p>Methods:</p> Name Description <code>fetch_nwb</code> <p>Fetch NWBFile object from relevant table. Uses either a foreign key to a NWBFile table (including AnalysisNwbfile) or a _nwb_table attribute to determine which table to use.</p> <code>delte_downstream_merge</code> <p>Delete downstream merge table entries associated with restriction. Requires caching of merge tables and links, which is slow on first call. <code>restriction</code> can be set to a string to restrict the delete. <code>dry_run</code> can be set to False to commit the delete. <code>reload_cache</code> can be set to True to reload the merge cache.</p> <code>ddm</code> <p>Alias for delete_downstream_merge.</p> <code>cautious_delete</code> <p>Check user permissions before deleting table rows. Permission is granted to users listed as admin in LabMember table or to users on a team with with the Session experimenter(s). If the table where the delete is executed cannot be linked to a Session, a warning is logged and the delete continues. If the Session has no experimenter, or if the user is not on a team with the Session experimenter(s), a PermissionError is raised. <code>force_permission</code> can be set to True to bypass permission check.</p> <code>cdel</code> <p>Alias for cautious_delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>class SpyglassMixin:\n    \"\"\"Mixin for Spyglass DataJoint tables.\n\n    Provides methods for fetching NWBFile objects and checking user permission\n    prior to deleting. As a mixin class, all Spyglass tables can inherit custom\n    methods from a central location.\n\n    Methods\n    -------\n    fetch_nwb(*attrs, **kwargs)\n        Fetch NWBFile object from relevant table. Uses either a foreign key to\n        a NWBFile table (including AnalysisNwbfile) or a _nwb_table attribute to\n        determine which table to use.\n    delte_downstream_merge(restriction=None, dry_run=True, reload_cache=False)\n        Delete downstream merge table entries associated with restriction.\n        Requires caching of merge tables and links, which is slow on first call.\n        `restriction` can be set to a string to restrict the delete. `dry_run`\n        can be set to False to commit the delete. `reload_cache` can be set to\n        True to reload the merge cache.\n    ddm(*args, **kwargs)\n        Alias for delete_downstream_merge.\n    cautious_delete(force_permission=False, *args, **kwargs)\n        Check user permissions before deleting table rows. Permission is granted\n        to users listed as admin in LabMember table or to users on a team with\n        with the Session experimenter(s). If the table where the delete is\n        executed cannot be linked to a Session, a warning is logged and the\n        delete continues. If the Session has no experimenter, or if the user is\n        not on a team with the Session experimenter(s), a PermissionError is\n        raised. `force_permission` can be set to True to bypass permission check.\n    cdel(*args, **kwargs)\n        Alias for cautious_delete.\n    \"\"\"\n\n    # _nwb_table = None # NWBFile table class, defined at the table level\n\n    # pks for delete permission check, assumed to be one field for each\n    _session_pk = None  # Session primary key. Mixin is ambivalent to Session pk\n    _member_pk = None  # LabMember primary key. Mixin ambivalent table structure\n\n    def __init__(self, *args, **kwargs):\n        \"\"\"Initialize SpyglassMixin.\n\n        Checks that schema prefix is in SHARED_MODULES.\n        \"\"\"\n        if (\n            self.database  # Connected to a database\n            and not self.is_declared  # New table\n            and self.database.split(\"_\")[0]  # Prefix\n            not in [\n                *SHARED_MODULES,  # Shared modules\n                dj.config[\"database.user\"],  # User schema\n                \"temp\",\n                \"test\",\n            ]\n        ):\n            logger.error(\n                f\"Schema prefix not in SHARED_MODULES: {self.database}\"\n            )\n\n    # ------------------------------- fetch_nwb -------------------------------\n\n    @cached_property\n    def _nwb_table_tuple(self) -&gt; tuple:\n        \"\"\"NWBFile table class.\n\n        Used to determine fetch_nwb behavior. Also used in Merge.fetch_nwb.\n        Implemented as a cached_property to avoid circular imports.\"\"\"\n        from spyglass.common.common_nwbfile import (\n            AnalysisNwbfile,\n            Nwbfile,\n        )  # noqa F401\n\n        table_dict = {\n            AnalysisNwbfile: \"analysis_file_abs_path\",\n            Nwbfile: \"nwb_file_abs_path\",\n        }\n\n        resolved = getattr(self, \"_nwb_table\", None) or (\n            AnalysisNwbfile\n            if \"-&gt; AnalysisNwbfile\" in self.definition\n            else Nwbfile if \"-&gt; Nwbfile\" in self.definition else None\n        )\n\n        if not resolved:\n            raise NotImplementedError(\n                f\"{self.__class__.__name__} does not have a \"\n                \"(Analysis)Nwbfile foreign key or _nwb_table attribute.\"\n            )\n\n        return (\n            resolved,\n            table_dict[resolved],\n        )\n\n    def fetch_nwb(self, *attrs, **kwargs):\n        \"\"\"Fetch NWBFile object from relevant table.\n\n        Implementing class must have a foreign key reference to Nwbfile or\n        AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition)\n        or a _nwb_table attribute. If both are present, the attribute takes\n        precedence.\n        \"\"\"\n        return fetch_nwb(self, self._nwb_table_tuple, *attrs, **kwargs)\n\n    # ------------------------ delete_downstream_merge ------------------------\n\n    @cached_property\n    def _merge_tables(self) -&gt; Dict[str, dj.FreeTable]:\n        \"\"\"Dict of merge tables downstream of self: {full_table_name: FreeTable}.\n\n        Cache of items in parents of self.descendants(as_objects=True). Both\n        descendant and parent must have the reserved primary key 'merge_id'.\n        \"\"\"\n        self.connection.dependencies.load()\n        merge_tables = {}\n\n        def search_descendants(parent):\n            for desc in parent.descendants(as_objects=True):\n                if (\n                    MERGE_PK not in desc.heading.names\n                    or not (master_name := get_master(desc.full_table_name))\n                    or master_name in merge_tables\n                ):\n                    continue\n                master = dj.FreeTable(self.connection, master_name)\n                if MERGE_PK in master.heading.names:\n                    merge_tables[master_name] = master\n                    search_descendants(master)\n\n        _ = search_descendants(self)\n\n        logger.info(\n            f\"Building merge cache for {self.table_name}.\\n\\t\"\n            + f\"Found {len(merge_tables)} downstream merge tables\"\n        )\n\n        return merge_tables\n\n    @cached_property\n    def _merge_chains(self) -&gt; OrderedDict[str, List[dj.FreeTable]]:\n        \"\"\"Dict of chains to merges downstream of self\n\n        Format: {full_table_name: TableChains}.\n\n        For each merge table found in _merge_tables, find the path from self to\n        merge via merge parts. If the path is valid, add it to the dict. Cache\n        prevents need to recompute whenever delete_downstream_merge is called\n        with a new restriction. To recompute, add `reload_cache=True` to\n        delete_downstream_merge call.\n        \"\"\"\n        merge_chains = {}\n        for name, merge_table in self._merge_tables.items():\n            chains = TableChains(self, merge_table, connection=self.connection)\n            if len(chains):\n                merge_chains[name] = chains\n\n        # This is ordered by max_len of chain from self to merge, which assumes\n        # that the merge table with the longest chain is the most downstream.\n        # A more sophisticated approach would order by length from self to\n        # each merge part independently, but this is a good first approximation.\n        return OrderedDict(\n            sorted(\n                merge_chains.items(), key=lambda x: x[1].max_len, reverse=True\n            )\n        )\n\n    def _get_chain(self, substring) -&gt; TableChains:\n        \"\"\"Return chain from self to merge table with substring in name.\"\"\"\n        for name, chain in self._merge_chains.items():\n            if substring.lower() in name:\n                return chain\n\n    def _commit_merge_deletes(\n        self, merge_join_dict: Dict[str, List[QueryExpression]], **kwargs\n    ) -&gt; None:\n        \"\"\"Commit merge deletes.\n\n        Parameters\n        ----------\n        merge_join_dict : Dict[str, List[QueryExpression]]\n            Dictionary of merge tables and their joins. Uses 'merge_id' primary\n            key to restrict delete.\n\n        Extracted for use in cautious_delete and delete_downstream_merge.\"\"\"\n        for table_name, part_restr in merge_join_dict.items():\n            table = self._merge_tables[table_name]\n            keys = [part.fetch(MERGE_PK, as_dict=True) for part in part_restr]\n            (table &amp; keys).delete(**kwargs)\n\n    def delete_downstream_merge(\n        self,\n        restriction: str = None,\n        dry_run: bool = True,\n        reload_cache: bool = False,\n        disable_warning: bool = False,\n        return_parts: bool = True,\n        **kwargs,\n    ) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n        \"\"\"Delete downstream merge table entries associated with restriction.\n\n        Requires caching of merge tables and links, which is slow on first call.\n\n        Parameters\n        ----------\n        restriction : str, optional\n            Restriction to apply to merge tables. Default None. Will attempt to\n            use table restriction if None.\n        dry_run : bool, optional\n            If True, return list of merge part entries to be deleted. Default\n            True.\n        reload_cache : bool, optional\n            If True, reload merge cache. Default False.\n        disable_warning : bool, optional\n            If True, do not warn if no merge tables found. Default False.\n        return_parts : bool, optional\n            If True, return list of merge part entries to be deleted. Default\n            True. If False, return dictionary of merge tables and their joins.\n        **kwargs : Any\n            Passed to datajoint.table.Table.delete.\n        \"\"\"\n        if reload_cache:\n            del self._merge_tables\n            del self._merge_chains\n\n        restriction = restriction or self.restriction or True\n\n        merge_join_dict = {}\n        for name, chain in self._merge_chains.items():\n            join = chain.join(restriction)\n            if join:\n                merge_join_dict[name] = join\n\n        if not merge_join_dict and not disable_warning:\n            logger.warning(\n                f\"No merge deletes found w/ {self.table_name} &amp; \"\n                + f\"{restriction}.\\n\\tIf this is unexpected, try importing \"\n                + \" Merge table(s) and running with `reload_cache`.\"\n            )\n\n        if dry_run:\n            return merge_join_dict.values() if return_parts else merge_join_dict\n\n        self._commit_merge_deletes(merge_join_dict, **kwargs)\n\n    def ddm(\n        self,\n        restriction: str = None,\n        dry_run: bool = True,\n        reload_cache: bool = False,\n        disable_warning: bool = False,\n        return_parts: bool = True,\n        *args,\n        **kwargs,\n    ) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n        \"\"\"Alias for delete_downstream_merge.\"\"\"\n        return self.delete_downstream_merge(\n            restriction=restriction,\n            dry_run=dry_run,\n            reload_cache=reload_cache,\n            disable_warning=disable_warning,\n            return_parts=return_parts,\n            *args,\n            **kwargs,\n        )\n\n    # ---------------------------- cautious_delete ----------------------------\n\n    @cached_property\n    def _delete_deps(self) -&gt; List[Table]:\n        \"\"\"List of tables required for delete permission check.\n\n        LabMember, LabTeam, and Session are required for delete permission.\n\n        Used to delay import of tables until needed, avoiding circular imports.\n        Each of these tables inheits SpyglassMixin.\n        \"\"\"\n        from spyglass.common import LabMember, LabTeam, Session  # noqa F401\n\n        self._session_pk = Session.primary_key[0]\n        self._member_pk = LabMember.primary_key[0]\n        return [LabMember, LabTeam, Session]\n\n    def _get_exp_summary(self):\n        \"\"\"Get summary of experimenters for session(s), including NULL.\n\n        Parameters\n        ----------\n        sess_link : datajoint.expression.QueryExpression\n            Join of table link with Session table.\n\n        Returns\n        -------\n        str\n            Summary of experimenters for session(s).\n        \"\"\"\n        Session = self._delete_deps[-1]\n        SesExp = Session.Experimenter\n        empty_pk = {self._member_pk: \"NULL\"}\n\n        format = dj.U(self._session_pk, self._member_pk)\n        sess_link = self._session_connection.join(\n            self.restriction, reverse_order=True\n        )\n\n        exp_missing = format &amp; (sess_link - SesExp).proj(**empty_pk)\n        exp_present = format &amp; (sess_link * SesExp - exp_missing).proj()\n\n        return exp_missing + exp_present\n\n    @cached_property\n    def _session_connection(self) -&gt; Union[TableChain, bool]:\n        \"\"\"Path from Session table to self. False if no connection found.\"\"\"\n        connection = TableChain(parent=self._delete_deps[-1], child=self)\n        return connection if connection.has_link else False\n\n    @cached_property\n    def _test_mode(self) -&gt; bool:\n        \"\"\"Return True if in test mode.\n\n        Avoids circular import. Prevents prompt on delete.\"\"\"\n        from spyglass.settings import test_mode\n\n        return test_mode\n\n    def _check_delete_permission(self) -&gt; None:\n        \"\"\"Check user name against lab team assoc. w/ self -&gt; Session.\n\n        Returns\n        -------\n        None\n            Permission granted.\n\n        Raises\n        ------\n        PermissionError\n            Permission denied because (a) Session has no experimenter, or (b)\n            user is not on a team with Session experimenter(s).\n        \"\"\"\n        LabMember, LabTeam, Session = self._delete_deps\n\n        dj_user = dj.config[\"database.user\"]\n        if dj_user in LabMember().admin:  # bypass permission check for admin\n            return\n\n        if not self._session_connection:\n            logger.warn(  # Permit delete if no session connection\n                \"Could not find lab team associated with \"\n                + f\"{self.__class__.__name__}.\"\n                + \"\\nBe careful not to delete others' data.\"\n            )\n            return\n\n        sess_summary = self._get_exp_summary()\n        experimenters = sess_summary.fetch(self._member_pk)\n        if None in experimenters:\n            # TODO: Check if allow delete of remainder?\n            raise PermissionError(\n                \"Please ensure all Sessions have an experimenter in \"\n                + f\"SessionExperimenter:\\n{sess_summary}\"\n            )\n\n        user_name = LabMember().get_djuser_name(dj_user)\n        for experimenter in set(experimenters):\n            # Check once with cache, if fails, reload and check again\n            # On eval as set, reload will only be called once\n            if user_name not in LabTeam().get_team_members(\n                experimenter\n            ) and user_name not in LabTeam().get_team_members(\n                experimenter, reload=True\n            ):\n                sess_w_exp = sess_summary &amp; {self._member_pk: experimenter}\n                raise PermissionError(\n                    f\"User '{user_name}' is not on a team with '{experimenter}'\"\n                    + \", an experimenter for session(s):\\n\"\n                    + f\"{sess_w_exp}\"\n                )\n        logger.info(f\"Queueing delete for session(s):\\n{sess_summary}\")\n\n    @cached_property\n    def _usage_table(self):\n        \"\"\"Temporary inclusion for usage tracking.\"\"\"\n        from spyglass.common.common_usage import CautiousDelete\n\n        return CautiousDelete()\n\n    def _log_use(self, start, merge_deletes=None, super_delete=False):\n        \"\"\"Log use of cautious_delete.\"\"\"\n        if isinstance(merge_deletes, QueryExpression):\n            merge_deletes = merge_deletes.fetch(as_dict=True)\n        safe_insert = dict(\n            duration=time() - start,\n            dj_user=dj.config[\"database.user\"],\n            origin=self.full_table_name,\n        )\n        restr_str = \"Super delete: \" if super_delete else \"\"\n        restr_str += \"\".join(self.restriction) if self.restriction else \"None\"\n        try:\n            self._usage_table.insert1(\n                dict(\n                    **safe_insert,\n                    restriction=restr_str[:255],\n                    merge_deletes=merge_deletes,\n                )\n            )\n        except (DataJointError, DataError):\n            self._usage_table.insert1(\n                dict(**safe_insert, restriction=\"Unknown\")\n            )\n\n    # TODO: Intercept datajoint delete confirmation prompt for merge deletes\n    def cautious_delete(self, force_permission: bool = False, *args, **kwargs):\n        \"\"\"Delete table rows after checking user permission.\n\n        Permission is granted to users listed as admin in LabMember table or to\n        users on a team with with the Session experimenter(s). If the table\n        cannot be linked to Session, a warning is logged and the delete\n        continues. If the Session has no experimenter, or if the user is not on\n        a team with the Session experimenter(s), a PermissionError is raised.\n\n        Parameters\n        ----------\n        force_permission : bool, optional\n            Bypass permission check. Default False.\n        *args, **kwargs : Any\n            Passed to datajoint.table.Table.delete.\n        \"\"\"\n        start = time()\n\n        if not force_permission:\n            self._check_delete_permission()\n\n        merge_deletes = self.delete_downstream_merge(\n            dry_run=True,\n            disable_warning=True,\n            return_parts=False,\n        )\n\n        safemode = (\n            dj.config.get(\"safemode\", True)\n            if kwargs.get(\"safemode\") is None\n            else kwargs[\"safemode\"]\n        )\n\n        if merge_deletes:\n            for table, content in merge_deletes.items():\n                count = sum([len(part) for part in content])\n                dj_logger.info(f\"Merge: Deleting {count} rows from {table}\")\n            if (\n                not self._test_mode\n                or not safemode\n                or user_choice(\"Commit deletes?\", default=\"no\") == \"yes\"\n            ):\n                self._commit_merge_deletes(merge_deletes, **kwargs)\n            else:\n                logger.info(\"Delete aborted.\")\n                self._log_use(start)\n                return\n\n        super().delete(*args, **kwargs)  # Additional confirm here\n\n        self._log_use(start=start, merge_deletes=merge_deletes)\n\n    def cdel(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete.\"\"\"\n        self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n\n    def delete(self, force_permission=False, *args, **kwargs):\n        \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n        self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n\n    def super_delete(self, *args, **kwargs):\n        \"\"\"Alias for datajoint.table.Table.delete.\"\"\"\n        logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n        self._log_use(start=time(), super_delete=True)\n        super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.__init__", "title": "<code>__init__(*args, **kwargs)</code>", "text": "<p>Initialize SpyglassMixin.</p> <p>Checks that schema prefix is in SHARED_MODULES.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def __init__(self, *args, **kwargs):\n    \"\"\"Initialize SpyglassMixin.\n\n    Checks that schema prefix is in SHARED_MODULES.\n    \"\"\"\n    if (\n        self.database  # Connected to a database\n        and not self.is_declared  # New table\n        and self.database.split(\"_\")[0]  # Prefix\n        not in [\n            *SHARED_MODULES,  # Shared modules\n            dj.config[\"database.user\"],  # User schema\n            \"temp\",\n            \"test\",\n        ]\n    ):\n        logger.error(\n            f\"Schema prefix not in SHARED_MODULES: {self.database}\"\n        )\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.fetch_nwb", "title": "<code>fetch_nwb(*attrs, **kwargs)</code>", "text": "<p>Fetch NWBFile object from relevant table.</p> <p>Implementing class must have a foreign key reference to Nwbfile or AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition) or a _nwb_table attribute. If both are present, the attribute takes precedence.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def fetch_nwb(self, *attrs, **kwargs):\n    \"\"\"Fetch NWBFile object from relevant table.\n\n    Implementing class must have a foreign key reference to Nwbfile or\n    AnalysisNwbfile (i.e., \"-&gt; (Analysis)Nwbfile\" in definition)\n    or a _nwb_table attribute. If both are present, the attribute takes\n    precedence.\n    \"\"\"\n    return fetch_nwb(self, self._nwb_table_tuple, *attrs, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.delete_downstream_merge", "title": "<code>delete_downstream_merge(restriction=None, dry_run=True, reload_cache=False, disable_warning=False, return_parts=True, **kwargs)</code>", "text": "<p>Delete downstream merge table entries associated with restriction.</p> <p>Requires caching of merge tables and links, which is slow on first call.</p> <p>Parameters:</p> Name Type Description Default <code>restriction</code> <code>str</code> <p>Restriction to apply to merge tables. Default None. Will attempt to use table restriction if None.</p> <code>None</code> <code>dry_run</code> <code>bool</code> <p>If True, return list of merge part entries to be deleted. Default True.</p> <code>True</code> <code>reload_cache</code> <code>bool</code> <p>If True, reload merge cache. Default False.</p> <code>False</code> <code>disable_warning</code> <code>bool</code> <p>If True, do not warn if no merge tables found. Default False.</p> <code>False</code> <code>return_parts</code> <code>bool</code> <p>If True, return list of merge part entries to be deleted. Default True. If False, return dictionary of merge tables and their joins.</p> <code>True</code> <code>**kwargs</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>{}</code> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def delete_downstream_merge(\n    self,\n    restriction: str = None,\n    dry_run: bool = True,\n    reload_cache: bool = False,\n    disable_warning: bool = False,\n    return_parts: bool = True,\n    **kwargs,\n) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n    \"\"\"Delete downstream merge table entries associated with restriction.\n\n    Requires caching of merge tables and links, which is slow on first call.\n\n    Parameters\n    ----------\n    restriction : str, optional\n        Restriction to apply to merge tables. Default None. Will attempt to\n        use table restriction if None.\n    dry_run : bool, optional\n        If True, return list of merge part entries to be deleted. Default\n        True.\n    reload_cache : bool, optional\n        If True, reload merge cache. Default False.\n    disable_warning : bool, optional\n        If True, do not warn if no merge tables found. Default False.\n    return_parts : bool, optional\n        If True, return list of merge part entries to be deleted. Default\n        True. If False, return dictionary of merge tables and their joins.\n    **kwargs : Any\n        Passed to datajoint.table.Table.delete.\n    \"\"\"\n    if reload_cache:\n        del self._merge_tables\n        del self._merge_chains\n\n    restriction = restriction or self.restriction or True\n\n    merge_join_dict = {}\n    for name, chain in self._merge_chains.items():\n        join = chain.join(restriction)\n        if join:\n            merge_join_dict[name] = join\n\n    if not merge_join_dict and not disable_warning:\n        logger.warning(\n            f\"No merge deletes found w/ {self.table_name} &amp; \"\n            + f\"{restriction}.\\n\\tIf this is unexpected, try importing \"\n            + \" Merge table(s) and running with `reload_cache`.\"\n        )\n\n    if dry_run:\n        return merge_join_dict.values() if return_parts else merge_join_dict\n\n    self._commit_merge_deletes(merge_join_dict, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.ddm", "title": "<code>ddm(restriction=None, dry_run=True, reload_cache=False, disable_warning=False, return_parts=True, *args, **kwargs)</code>", "text": "<p>Alias for delete_downstream_merge.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def ddm(\n    self,\n    restriction: str = None,\n    dry_run: bool = True,\n    reload_cache: bool = False,\n    disable_warning: bool = False,\n    return_parts: bool = True,\n    *args,\n    **kwargs,\n) -&gt; Union[List[QueryExpression], Dict[str, List[QueryExpression]]]:\n    \"\"\"Alias for delete_downstream_merge.\"\"\"\n    return self.delete_downstream_merge(\n        restriction=restriction,\n        dry_run=dry_run,\n        reload_cache=reload_cache,\n        disable_warning=disable_warning,\n        return_parts=return_parts,\n        *args,\n        **kwargs,\n    )\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.cautious_delete", "title": "<code>cautious_delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Delete table rows after checking user permission.</p> <p>Permission is granted to users listed as admin in LabMember table or to users on a team with with the Session experimenter(s). If the table cannot be linked to Session, a warning is logged and the delete continues. If the Session has no experimenter, or if the user is not on a team with the Session experimenter(s), a PermissionError is raised.</p> <p>Parameters:</p> Name Type Description Default <code>force_permission</code> <code>bool</code> <p>Bypass permission check. Default False.</p> <code>False</code> <code>*args</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>()</code> <code>**kwargs</code> <code>Any</code> <p>Passed to datajoint.table.Table.delete.</p> <code>()</code> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def cautious_delete(self, force_permission: bool = False, *args, **kwargs):\n    \"\"\"Delete table rows after checking user permission.\n\n    Permission is granted to users listed as admin in LabMember table or to\n    users on a team with with the Session experimenter(s). If the table\n    cannot be linked to Session, a warning is logged and the delete\n    continues. If the Session has no experimenter, or if the user is not on\n    a team with the Session experimenter(s), a PermissionError is raised.\n\n    Parameters\n    ----------\n    force_permission : bool, optional\n        Bypass permission check. Default False.\n    *args, **kwargs : Any\n        Passed to datajoint.table.Table.delete.\n    \"\"\"\n    start = time()\n\n    if not force_permission:\n        self._check_delete_permission()\n\n    merge_deletes = self.delete_downstream_merge(\n        dry_run=True,\n        disable_warning=True,\n        return_parts=False,\n    )\n\n    safemode = (\n        dj.config.get(\"safemode\", True)\n        if kwargs.get(\"safemode\") is None\n        else kwargs[\"safemode\"]\n    )\n\n    if merge_deletes:\n        for table, content in merge_deletes.items():\n            count = sum([len(part) for part in content])\n            dj_logger.info(f\"Merge: Deleting {count} rows from {table}\")\n        if (\n            not self._test_mode\n            or not safemode\n            or user_choice(\"Commit deletes?\", default=\"no\") == \"yes\"\n        ):\n            self._commit_merge_deletes(merge_deletes, **kwargs)\n        else:\n            logger.info(\"Delete aborted.\")\n            self._log_use(start)\n            return\n\n    super().delete(*args, **kwargs)  # Additional confirm here\n\n    self._log_use(start=start, merge_deletes=merge_deletes)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.cdel", "title": "<code>cdel(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def cdel(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete.\"\"\"\n    self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.delete", "title": "<code>delete(force_permission=False, *args, **kwargs)</code>", "text": "<p>Alias for cautious_delete, overwrites datajoint.table.Table.delete</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def delete(self, force_permission=False, *args, **kwargs):\n    \"\"\"Alias for cautious_delete, overwrites datajoint.table.Table.delete\"\"\"\n    self.cautious_delete(force_permission=force_permission, *args, **kwargs)\n</code></pre>"}, {"location": "api/utils/dj_mixin/#src.spyglass.utils.dj_mixin.SpyglassMixin.super_delete", "title": "<code>super_delete(*args, **kwargs)</code>", "text": "<p>Alias for datajoint.table.Table.delete.</p> Source code in <code>src/spyglass/utils/dj_mixin.py</code> <pre><code>def super_delete(self, *args, **kwargs):\n    \"\"\"Alias for datajoint.table.Table.delete.\"\"\"\n    logger.warning(\"!! Using super_delete. Bypassing cautious_delete !!\")\n    self._log_use(start=time(), super_delete=True)\n    super().delete(*args, **kwargs)\n</code></pre>"}, {"location": "api/utils/logging/", "title": "logging.py", "text": "<p>Logging configuration based on datajoint/logging.py</p>"}, {"location": "api/utils/nwb_helper_fn/", "title": "nwb_helper_fn.py", "text": "<p>NWB helper functions for finding processing modules and data interfaces.</p>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_nwb_file", "title": "<code>get_nwb_file(nwb_file_path)</code>", "text": "<p>Return an NWBFile object with the given file path in read mode.</p> <p>If the file is not found locally, this will check if it has been shared with kachery and if so, download it and open it.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_path</code> <code>str</code> <p>Path to the NWB file or NWB file name. If it does not start with a \"/\", get path with Nwbfile.get_abs_path</p> required <p>Returns:</p> Name Type Description <code>nwbfile</code> <code>NWBFile</code> <p>NWB file object for the given path opened in read mode.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_nwb_file(nwb_file_path):\n    \"\"\"Return an NWBFile object with the given file path in read mode.\n\n    If the file is not found locally, this will check if it has been shared\n    with kachery and if so, download it and open it.\n\n\n    Parameters\n    ----------\n    nwb_file_path : str\n        Path to the NWB file or NWB file name. If it does not start with a \"/\",\n        get path with Nwbfile.get_abs_path\n\n    Returns\n    -------\n    nwbfile : pynwb.NWBFile\n        NWB file object for the given path opened in read mode.\n    \"\"\"\n    if not nwb_file_path.startswith(\"/\"):\n        from ..common import Nwbfile\n\n        nwb_file_path = Nwbfile.get_abs_path(nwb_file_path)\n\n    _, nwbfile = __open_nwb_files.get(nwb_file_path, (None, None))\n\n    if nwbfile is None:\n        # check to see if the file exists\n        if not os.path.exists(nwb_file_path):\n            logger.info(\n                \"NWB file not found locally; checking kachery for \"\n                + f\"{nwb_file_path}\"\n            )\n            # first try the analysis files\n            from ..sharing.sharing_kachery import AnalysisNwbfileKachery\n\n            # the download functions assume just the filename, so we need to\n            # get that from the path\n            if not AnalysisNwbfileKachery.download_file(\n                os.path.basename(nwb_file_path)\n            ):\n                return None\n        # now open the file\n        io = pynwb.NWBHDF5IO(\n            path=nwb_file_path, mode=\"r\", load_namespaces=True\n        )  # keep file open\n        nwbfile = io.read()\n        __open_nwb_files[nwb_file_path] = (io, nwbfile)\n\n    return nwbfile\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_config", "title": "<code>get_config(nwb_file_path)</code>", "text": "<p>Return a dictionary of config settings for the given NWB file. If the file does not exist, return an empty dict.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_file_path</code> <code>str</code> <p>Absolute path to the NWB file.</p> required <p>Returns:</p> Type Description <code>dict</code> <p>Dictionary of configuration settings loaded from the corresponding YAML file</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_config(nwb_file_path):\n    \"\"\"Return a dictionary of config settings for the given NWB file.\n    If the file does not exist, return an empty dict.\n\n    Parameters\n    ----------\n    nwb_file_path : str\n        Absolute path to the NWB file.\n\n    Returns\n    -------\n    dict\n        Dictionary of configuration settings loaded from the corresponding YAML file\n    \"\"\"\n    if nwb_file_path in __configs:  # load from cache if exists\n        return __configs[nwb_file_path]\n\n    p = Path(nwb_file_path)\n    # NOTE use p.stem[:-1] to remove the underscore that was added to the file\n    config_path = p.parent / (p.stem[:-1] + \"_spyglass_config.yaml\")\n    if not os.path.exists(config_path):\n        logger.info(f\"No config found at file path {config_path}\")\n        return dict()\n    with open(config_path, \"r\") as stream:\n        d = yaml.safe_load(stream)\n\n    # TODO write a JSON schema for the yaml file and validate the yaml file\n    __configs[nwb_file_path] = d  # store in cache\n    return d\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_data_interface", "title": "<code>get_data_interface(nwbfile, data_interface_name, data_interface_class=None)</code>", "text": "<p>Search for NWBDataInterface or DynamicTable in processing modules of an NWB.</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object to search in.</p> required <code>data_interface_name</code> <code>str</code> <p>The name of the NWBDataInterface or DynamicTable to search for.</p> required <code>data_interface_class</code> <code>type</code> <p>The class (or superclass) to search for. This argument helps to prevent accessing an object with the same name but the incorrect type. Default: no restriction.</p> <code>None</code> <p>Warns:</p> Type Description <code>LoggerWarning</code> <p>If multiple NWBDataInterface and DynamicTable objects with the matching name are found.</p> <p>Returns:</p> Name Type Description <code>data_interface</code> <code>NWBDataInterface</code> <p>The data interface object with the given name, or None if not found.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_data_interface(nwbfile, data_interface_name, data_interface_class=None):\n    \"\"\"\n    Search for NWBDataInterface or DynamicTable in processing modules of an NWB.\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object to search in.\n    data_interface_name : str\n        The name of the NWBDataInterface or DynamicTable to search for.\n    data_interface_class : type, optional\n        The class (or superclass) to search for. This argument helps to prevent\n        accessing an object with the same name but the incorrect type. Default:\n        no restriction.\n\n    Warns\n    -----\n    LoggerWarning\n        If multiple NWBDataInterface and DynamicTable objects with the matching\n        name are found.\n\n    Returns\n    -------\n    data_interface : NWBDataInterface\n        The data interface object with the given name, or None if not found.\n    \"\"\"\n    ret = []\n    for module in nwbfile.processing.values():\n        match = module.data_interfaces.get(data_interface_name, None)\n        if match is not None:\n            if data_interface_class is not None and not isinstance(\n                match, data_interface_class\n            ):\n                continue\n            ret.append(match)\n    if len(ret) &gt; 1:\n        logger.warning(\n            f\"Multiple data interfaces with name '{data_interface_name}' \"\n            f\"found in NWBFile with identifier {nwbfile.identifier}. \"\n            + \"Using the first one found. \"\n            \"Use the data_interface_class argument to restrict the search.\"\n        )\n    if len(ret) &gt;= 1:\n        return ret[0]\n\n    return None\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_position_obj", "title": "<code>get_position_obj(nwbfile)</code>", "text": "<p>Return the Position object from the behavior processing module. Meant to find position spatial series that are not found by <code>get_data_interface(nwbfile, 'position', pynwb.behavior.Position)</code>. The code returns the first <code>pynwb.behavior.Position</code> object (technically there should only be one).</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object.</p> required <p>Returns:</p> Type Description <code>pynwb.behavior.Position object</code> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_position_obj(nwbfile):\n    \"\"\"Return the Position object from the behavior processing module.\n    Meant to find position spatial series that are not found by\n    `get_data_interface(nwbfile, 'position', pynwb.behavior.Position)`.\n    The code returns the first `pynwb.behavior.Position` object (technically\n    there should only be one).\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object.\n\n    Returns\n    -------\n    pynwb.behavior.Position object\n    \"\"\"\n    ret = []\n    for obj in nwbfile.processing[\"behavior\"].data_interfaces.values():\n        if isinstance(obj, pynwb.behavior.Position):\n            ret.append(obj)\n    if len(ret) &gt; 1:\n        raise ValueError(f\"Found more than one position object in {nwbfile}\")\n    return ret[0] if ret and len(ret) else None\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_raw_eseries", "title": "<code>get_raw_eseries(nwbfile)</code>", "text": "<p>Return all ElectricalSeries in the acquisition group of an NWB file.</p> <p>ElectricalSeries found within LFP objects in the acquisition will also be returned.</p> <p>Parameters:</p> Name Type Description Default <code>nwbfile</code> <code>NWBFile</code> <p>The NWB file object to search in.</p> required <p>Returns:</p> Name Type Description <code>ret</code> <code>list</code> <p>A list of all ElectricalSeries in the acquisition group of an NWB file</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_raw_eseries(nwbfile):\n    \"\"\"Return all ElectricalSeries in the acquisition group of an NWB file.\n\n    ElectricalSeries found within LFP objects in the acquisition will also be\n    returned.\n\n    Parameters\n    ----------\n    nwbfile : pynwb.NWBFile\n        The NWB file object to search in.\n\n    Returns\n    -------\n    ret : list\n        A list of all ElectricalSeries in the acquisition group of an NWB file\n    \"\"\"\n    ret = []\n    for nwb_object in nwbfile.acquisition.values():\n        if isinstance(nwb_object, pynwb.ecephys.ElectricalSeries):\n            ret.append(nwb_object)\n        elif isinstance(nwb_object, pynwb.ecephys.LFP):\n            ret.extend(nwb_object.electrical_series.values())\n    return ret\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.estimate_sampling_rate", "title": "<code>estimate_sampling_rate(timestamps, multiplier=1.75, verbose=False, filename='file')</code>", "text": "<p>Estimate the sampling rate given a list of timestamps.</p> <p>Assumes that the most common temporal differences between timestamps approximate the sampling rate. Note that this can fail for very high sampling rates and irregular timestamps.</p> <p>Parameters:</p> Name Type Description Default <code>timestamps</code> <code>ndarray</code> <p>1D numpy array of timestamp values.</p> required <code>multiplier</code> <code>float or int</code> <p>Deft</p> <code>1.75</code> <code>verbose</code> <code>bool</code> <p>Log sampling rate to stdout. Default, False</p> <code>False</code> <code>filename</code> <code>str</code> <p>Filename to reference when logging or err. Default, \"file\"</p> <code>'file'</code> <p>Returns:</p> Name Type Description <code>estimated_rate</code> <code>float</code> <p>The estimated sampling rate.</p> <p>Raises:</p> Type Description <code>ValueError</code> <p>If estimated rate is less than 0.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def estimate_sampling_rate(\n    timestamps, multiplier=1.75, verbose=False, filename=\"file\"\n):\n    \"\"\"Estimate the sampling rate given a list of timestamps.\n\n    Assumes that the most common temporal differences between timestamps\n    approximate the sampling rate. Note that this can fail for very high\n    sampling rates and irregular timestamps.\n\n    Parameters\n    ----------\n    timestamps : numpy.ndarray\n        1D numpy array of timestamp values.\n    multiplier : float or int, optional\n        Deft\n    verbose : bool, optional\n        Log sampling rate to stdout. Default, False\n    filename : str, optional\n        Filename to reference when logging or err. Default, \"file\"\n\n    Returns\n    -------\n    estimated_rate : float\n        The estimated sampling rate.\n\n    Raises\n    ------\n    ValueError\n        If estimated rate is less than 0.\n    \"\"\"\n\n    # approach:\n    # 1. use a box car smoother and a histogram to get the modal value\n    # 2. identify adjacent samples as those that have a\n    #    time difference &lt; the multiplier * the modal value\n    # 3. average the time differences between adjacent samples\n\n    sample_diff = np.diff(timestamps[~np.isnan(timestamps)])\n\n    if len(sample_diff) &lt; 10:\n        raise ValueError(\n            f\"Only {len(sample_diff)} timestamps are valid. Check the data.\"\n        )\n\n    smooth_diff = np.convolve(sample_diff, np.ones(10) / 10, mode=\"same\")\n\n    # we histogram with 100 bins out to 3 * mean, which should be fine for any\n    # reasonable number of samples\n    hist, bins = np.histogram(\n        smooth_diff, bins=100, range=[0, 3 * np.mean(smooth_diff)]\n    )\n    mode = bins[np.where(hist == np.max(hist))][0]\n\n    adjacent = sample_diff &lt; mode * multiplier\n\n    sampling_rate = np.round(1.0 / np.mean(sample_diff[adjacent]))\n\n    if sampling_rate &lt; 0:\n        raise ValueError(f\"Error calculating sampling rate. For {filename}\")\n    if verbose:\n        logger.info(\n            f\"Estimated sampling rate for {filename}: {sampling_rate} Hz\"\n        )\n\n    return sampling_rate\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_valid_intervals", "title": "<code>get_valid_intervals(timestamps, sampling_rate, gap_proportion=2.5, min_valid_len=0)</code>", "text": "<p>Finds the set of all valid intervals in a list of timestamps. Valid interval: (start time, stop time) during which there are no gaps (i.e. missing samples).</p> <p>Parameters:</p> Name Type Description Default <code>timestamps</code> <code>ndarray</code> <p>1D numpy array of timestamp values.</p> required <code>sampling_rate</code> <code>float</code> <p>Sampling rate of the data.</p> required <code>gap_proportion</code> <code>float</code> <p>Threshold for detecting a gap; i.e. if the difference (in samples) between consecutive timestamps exceeds gap_proportion, it is considered a gap. Must be &gt; 1. Default to 2.5</p> <code>2.5</code> <code>min_valid_len</code> <code>float</code> <p>Length of smallest valid interval. Default to 0. If greater than interval duration, log warning and use half the total time.</p> <code>0</code> <p>Returns:</p> Name Type Description <code>valid_times</code> <code>ndarray</code> <p>Array of start and stop times of shape (N, 2) for valid data.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_valid_intervals(\n    timestamps, sampling_rate, gap_proportion=2.5, min_valid_len=0\n):\n    \"\"\"Finds the set of all valid intervals in a list of timestamps.\n    Valid interval: (start time, stop time) during which there are\n    no gaps (i.e. missing samples).\n\n    Parameters\n    ----------\n    timestamps : numpy.ndarray\n        1D numpy array of timestamp values.\n    sampling_rate : float\n        Sampling rate of the data.\n    gap_proportion : float, optional\n        Threshold for detecting a gap; i.e. if the difference (in samples)\n        between consecutive timestamps exceeds gap_proportion, it is considered\n        a gap. Must be &gt; 1. Default to 2.5\n    min_valid_len : float, optional\n        Length of smallest valid interval. Default to 0. If greater\n        than interval duration, log warning and use half the total time.\n\n    Returns\n    -------\n    valid_times : np.ndarray\n        Array of start and stop times of shape (N, 2) for valid data.\n    \"\"\"\n\n    eps = 0.0000001\n\n    total_time = timestamps[-1] - timestamps[0]\n\n    if total_time &lt; min_valid_len:\n        half_total_time = total_time / 2\n        logger.warn(f\"Setting minimum valid interval to {half_total_time}\")\n        min_valid_len = half_total_time\n\n    # get rid of NaN elements\n    timestamps = timestamps[~np.isnan(timestamps)]\n    # find gaps\n    gap = np.diff(timestamps) &gt; 1.0 / sampling_rate * gap_proportion\n\n    # all true entries of gap represent gaps. Get the times bounding these intervals.\n    gapind = np.asarray(np.where(gap))\n    # The end of each valid interval are the indices of the gaps and the final value\n    valid_end = np.append(gapind, np.asarray(len(timestamps) - 1))\n\n    # the beginning of the gaps are the first element and gapind+1\n    valid_start = np.insert(gapind + 1, 0, 0)\n\n    valid_indices = np.vstack([valid_start, valid_end]).transpose()\n\n    valid_times = timestamps[valid_indices]\n    # adjust the times to deal with single valid samples\n    valid_times[:, 0] = valid_times[:, 0] - eps\n    valid_times[:, 1] = valid_times[:, 1] + eps\n\n    valid_intervals = (valid_times[:, 1] - valid_times[:, 0]) &gt; min_valid_len\n\n    return valid_times[valid_intervals, :]\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_electrode_indices", "title": "<code>get_electrode_indices(nwb_object, electrode_ids)</code>", "text": "<p>Return indices of the specified electrode_ids given an NWB file.</p> <p>Also accepts electrical series object. If an ElectricalSeries is given, then the indices returned are relative to the selected rows in ElectricalSeries.electrodes. For example, if electricalseries.electrodes = [5], and row index 5 of nwbfile.electrodes has ID 10, then calling get_electrode_indices(electricalseries, 10) will return 0, the index of the matching electrode in electricalseries.electrodes.</p> <p>Indices for electrode_ids that are not in the electrical series are returned as np.nan</p> <p>If an NWBFile is given, then the row indices with the matching IDs in the file's electrodes table are returned.</p> <p>Parameters:</p> Name Type Description Default <code>nwb_object</code> <code>NWBFile or ElectricalSeries</code> <p>The NWB file object or NWB electrical series object.</p> required <code>electrode_ids</code> <code>ndarray or list</code> <p>Array or list of electrode IDs.</p> required <p>Returns:</p> Name Type Description <code>electrode_indices</code> <code>list</code> <p>Array of indices of the specified electrode IDs.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_electrode_indices(nwb_object, electrode_ids):\n    \"\"\"Return indices of the specified electrode_ids given an NWB file.\n\n    Also accepts electrical series object. If an ElectricalSeries is given,\n    then the indices returned are relative to the selected rows in\n    ElectricalSeries.electrodes. For example, if electricalseries.electrodes =\n    [5], and row index 5 of nwbfile.electrodes has ID 10, then calling\n    get_electrode_indices(electricalseries, 10) will return 0, the index of the\n    matching electrode in electricalseries.electrodes.\n\n    Indices for electrode_ids that are not in the electrical series are\n    returned as np.nan\n\n    If an NWBFile is given, then the row indices with the matching IDs in the\n    file's electrodes table are returned.\n\n    Parameters\n    ----------\n    nwb_object : pynwb.NWBFile or pynwb.ecephys.ElectricalSeries\n        The NWB file object or NWB electrical series object.\n    electrode_ids : np.ndarray or list\n        Array or list of electrode IDs.\n\n    Returns\n    -------\n    electrode_indices : list\n        Array of indices of the specified electrode IDs.\n    \"\"\"\n    if isinstance(nwb_object, pynwb.ecephys.ElectricalSeries):\n        # electrodes is a DynamicTableRegion which may contain a subset of the\n        # rows in NWBFile.electrodes match against only the subset of\n        # electrodes referenced by this ElectricalSeries\n        electrode_table_indices = nwb_object.electrodes.data[:]\n        selected_elect_ids = [\n            nwb_object.electrodes.table.id[x] for x in electrode_table_indices\n        ]\n    elif isinstance(nwb_object, pynwb.NWBFile):\n        # electrodes is a DynamicTable that contains all electrodes\n        selected_elect_ids = list(nwb_object.electrodes.id[:])\n    else:\n        raise ValueError(\n            \"nwb_object must be of type ElectricalSeries or NWBFile\"\n        )\n\n    # for each electrode_id, find its index in selected_elect_ids and return\n    # that if it's there and invalid_electrode_index if not.\n\n    return [\n        (\n            selected_elect_ids.index(elect_id)\n            if elect_id in selected_elect_ids\n            else invalid_electrode_index\n        )\n        for elect_id in electrode_ids\n    ]\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_all_spatial_series", "title": "<code>get_all_spatial_series(nwbf, verbose=False, incl_times=True)</code>", "text": "<p>Given an NWB, get the spatial series and return a dictionary by epoch.</p> <p>If incl_times is True, then the valid intervals are included in the output.</p> <p>Parameters:</p> Name Type Description Default <code>nwbf</code> <code>NWBFile</code> <p>The source NWB file object.</p> required <code>verbose</code> <code>bool</code> <p>Flag representing whether to log the sampling rate.</p> <code>False</code> <code>incl_times</code> <code>bool</code> <p>Include valid times in the output. Default, True. Set to False for only spatial series object IDs.</p> <code>True</code> <p>Returns:</p> Name Type Description <code>pos_data_dict</code> <code>dict</code> <p>Dict mapping indices to a dict with keys 'valid_times' and 'raw_position_object_id'. Returns None if there is no position data in the file. The 'raw_position_object_id' is the object ID of the SpatialSeries object.</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_all_spatial_series(nwbf, verbose=False, incl_times=True) -&gt; dict:\n    \"\"\"\n    Given an NWB, get the spatial series and return a dictionary by epoch.\n\n    If incl_times is True, then the valid intervals are included in the output.\n\n    Parameters\n    ----------\n    nwbf : pynwb.NWBFile\n        The source NWB file object.\n    verbose : bool\n        Flag representing whether to log the sampling rate.\n    incl_times : bool\n        Include valid times in the output. Default, True. Set to False for only\n        spatial series object IDs.\n\n    Returns\n    -------\n    pos_data_dict : dict\n        Dict mapping indices to a dict with keys 'valid_times' and\n        'raw_position_object_id'. Returns None if there is no position data in\n        the file. The 'raw_position_object_id' is the object ID of the\n        SpatialSeries object.\n    \"\"\"\n    pos_interface = get_position_obj(nwbf)\n\n    if pos_interface is None:\n        return None\n\n    return _get_pos_dict(\n        position=pos_interface.spatial_series,\n        epoch_groups=_get_epoch_groups(pos_interface),\n        session_id=nwbf.session_id,\n        verbose=verbose,\n        incl_times=incl_times,\n    )\n</code></pre>"}, {"location": "api/utils/nwb_helper_fn/#src.spyglass.utils.nwb_helper_fn.get_nwb_copy_filename", "title": "<code>get_nwb_copy_filename(nwb_file_name)</code>", "text": "<p>Get file name of copy of nwb file without the electrophys data</p> Source code in <code>src/spyglass/utils/nwb_helper_fn.py</code> <pre><code>def get_nwb_copy_filename(nwb_file_name):\n    \"\"\"Get file name of copy of nwb file without the electrophys data\"\"\"\n\n    filename, file_extension = os.path.splitext(nwb_file_name)\n\n    if filename.endswith(\"_\"):\n        logger.warning(f\"File may already be a copy: {nwb_file_name}\")\n\n    return f\"{filename}_{file_extension}\"\n</code></pre>"}, {"location": "misc/", "title": "Misc Docs", "text": "<p>This folder contains miscellaneous supporting files documentation.</p> <ul> <li>Database Management</li> <li>figurl Views</li> <li>insert Data</li> <li>Merge Tables</li> <li>Session Groups</li> </ul>"}, {"location": "misc/database_management/", "title": "Database Management", "text": "<p>While Spyglass can help you organize your data, there are a number of things you'll need to do to manage users, database backups, and file cleanup.</p> <p>Some these tasks should be set to run regularly. Cron jobs can help with automation.</p>"}, {"location": "misc/database_management/#mysql-version", "title": "MySQL Version", "text": "<p>The Frank Lab's database is running MySQL 8.0 with a number of custom configurations set by our system admin to reflect UCSF's IT security requirements.</p> <p>DataJoint's default docker container for MySQL is version 5.7. As the Spyglass team has hit select compatibility issues, we've worked with the DataJoint team to update the open source package to support MySQL 8.0.</p> <p>While the Spyglass team won't be able to support earlier versions, if you run into any issues declaring Spyglass tables with an 8.0 instance, please let us know.</p>"}, {"location": "misc/database_management/#user-management", "title": "User Management", "text": "<p>The DatabaseSettings class provides a number of methods to help you manage users. By default, it will write out a temporary <code>.sql</code> file and execute it on the database.</p>"}, {"location": "misc/database_management/#privileges", "title": "Privileges", "text": "<p>DataJoint schemas correspond to MySQL databases. Privileges are managed by schema/database prefix.</p> <ul> <li><code>SELECT</code> privileges allow users to read, write, and delete data.</li> <li><code>ALL</code> privileges allow users to create, alter, or drop tables and schemas in     addition to operations above.</li> </ul> <p>In practice, DataJoint only permits alterations of secondary keys on existing tables, and more derstructive operations would require using DataJoint to execeute MySQL commands.</p> <p>Shared schema prefixes are those defined in the Spyglass package (e.g., <code>common</code>, <code>lfp</code>, etc.). A 'user schema' is any schema with the username as prefix. User types differ in the privileges they are granted on these prifixes. Declaring a table with the SpyglassMixin on a schema other than a shared module or the user's own prefix will raise a warning.</p>"}, {"location": "misc/database_management/#users-roles", "title": "Users roles", "text": "<p>When a database is first initialized, the team should run <code>add_roles</code> to create the following roles:</p> <ul> <li><code>dj_guest</code>: <code>SELECT</code> on all schemas.</li> <li><code>dj_collab</code>: <code>ALL</code> on user schema, <code>SELECT</code> on all other schemas.</li> <li><code>dj_user</code>: <code>ALL</code> on shared and user schema, <code>SELECT</code> on all other schemas.</li> <li><code>dj_admin</code>: <code>ALL</code> on all schemas.</li> </ul> <p>If new shared modules are introduced, the <code>add_module</code> method should be used to expand the privileges of the <code>dj_user</code> role.</p>"}, {"location": "misc/database_management/#setting-passwords", "title": "Setting Passwords", "text": "<p>New users are generated with the password <code>temppass</code>. In order to change this, we recommend downloading DataJoint <code>0.14.2</code> (currently pre-release).</p> <pre><code>git clone https://github.com/datajoint/datajoint-python/\npip install ./datajoint-python\n</code></pre> <p>Then, you the user can reset within Python:</p> <pre><code>import datajoint as dj\n\ndj.set_password()\n</code></pre>"}, {"location": "misc/database_management/#database-backups", "title": "Database Backups", "text": "<p>The following codeblockes are a series of files used to back up our database and migrate the contents to another server. Some conventions to note:</p> <ul> <li><code>.host</code>: files used in the host's context</li> <li><code>.container</code>: files used inside the database Docker container</li> <li><code>.env</code>: files used to set environment variables used by the scripts for     database name, backup name, and backup credentials</li> </ul> <p>This backup process uses a dedicated backup user, that an admin would need to criate with the relevant permissions.</p>"}, {"location": "misc/database_management/#mysqlenvhost", "title": "mysql.env.host", "text": "MySQL host environment variables  Values may be adjusted as needed for different building images.  <pre><code>ROOT_PATH=/usr/local/containers/mysql # path to this container's working area\n\n# variables for building image\nSRC=ubuntu\nVER=20.04\nDOCKERFILE=Dockerfile.base\n\n# variables for referencing image\nIMAGE=mysql8\nTAG=u20\n# variables for running the container\nCNAME=mysql-datajoint\nMACADDR=4e:b0:3d:42:e0:70\nRPORT=3306\n\n# variables for initializing/relaunching the container\n# - where the mysql data and backups will live - these values\n# are examples\nDB_PATH=/data/db\nDB_DATA=mysql\nDB_BACKUP=/data/mysql-backups\n\n# backup info\nBACK_USER=mysql-backup\nBACK_PW={password}\nBACK_DBNAME={database}\n# mysql root password - make sure to remove this AFTER the container\n# is initialized - and this file will be replicated inside the container\n# on initialization, so remove it from there: /opt/bin/mysql.env\n</code></pre>"}, {"location": "misc/database_management/#backup-databaseshhost", "title": "backup-database.sh.host", "text": "<p>This script runs the mysql-backup container script (exec inside the container) that dumps the database contents for each database as well as the entire database. Use cron to set this to run on your desired schedule.</p> MySQL host docker exec <pre><code>#!/bin/bash\n\nPRIOR_DIR=$(pwd)\ncd /usr/local/containers/mysql || exit\n. mysql.env\ncd \"$(dirname ${ROOT_PATH})\"\n#\ndocker exec ${CNAME} /opt/bin/mysql-backup.csh\n#\ncd \"$(dirname ${DB_BACKUP})\"\n#\ncd ${PRIOR_DIR}\n</code></pre>"}, {"location": "misc/database_management/#mysql-backup-xfercshhost", "title": "mysql-backup-xfer.csh.host", "text": "<p>This script transfers the backup to another server 'X' and is specific for us as it uses passwordless ssh keys to a local unprivileged user on X that has the mysql backup area on X as that user's home.</p> MySQL host transfer script <pre><code>#!/bin/csh\nset td=`date +\"%Y%m%d\"`\ncd /data/mysql-backups\nscp -P {port} -i ~/mysql-backup -r ${database}-${td} mysql-backup@${X}:~/\n/bin/rm -r lmf-db-${td}\n</code></pre>"}, {"location": "misc/database_management/#myenvcshcontainer", "title": "myenv.csh.container", "text": "Docker container environment variables <pre><code>set db_backup=mysql-backups\nset back_user=mysql-backup\nset back_pw={password}\nset back_dbname={database}\n</code></pre>"}, {"location": "misc/database_management/#mysql-backupcshcontainer", "title": "mysql-backup.csh.container", "text": "Generate backups from within container <pre><code>#!/bin/csh\nsource /opt/bin/myenv.csh\nset td=`date +\"%Y%m%d\"`\ncd /${db_backup}\nmkdir ${back_dbname}-${td}\n\nset list=`echo \"show databases;\" | mysql --user=${back_user} --password=${back_pw}`\nset cnt=0\n\nforeach db ($list)\n  if ($cnt == 0) then\n    echo \"dumping mysql databases on $td\"\n  else\n    echo \"dumping MySQL database : $db\"\n    # Per-schema backups\n    mysqldump $db --max_allowed_packet=512M --user=${back_user} --password=${back_pw} &gt; /${db_backup}/${back_dbname}-${td}/mysql.${db}.sql\n  endif\n@ cnt = $cnt + 1\nend\n# Full database backup\nmysqldump --all-databases --max_allowed_packet=512M --user=${back_user} --password=${back_pw} &gt; /${db_backup}/${back_dbname}-${td}/mysql-all.sql\n</code></pre>"}, {"location": "misc/database_management/#file-cleanup", "title": "File Cleanup", "text": "<p>Spyglass is designed to hold metadata for analyses that reference NWB files on disk. There are several tables that retain lists of files that have been generated during analyses. If someone deletes analysis entries, files will still be on disk.</p> <p>To remove orphaned files, we run the following commands in our cron jobs:</p> <pre><code>from spyglass.common import AnalysisNwbfile\nfrom spyglass.spikesorting import SpikeSorting\nfrom spyglass.common.common_nwbfile import schema as nwbfile_schema\nfrom spyglass.decoding.v1.sorted_spikes import schema as spikes_schema\nfrom spyglass.decoding.v1.clusterless import schema as clusterless_schema\n\n\ndef main():\n    AnalysisNwbfile().nightly_cleanup()\n    SpikeSorting().nightly_cleanup()\n    nwbfile_schema.external['analysis'].delete(delete_external_files=True))\n    nwbfile_schema.external['raw'].delete(delete_external_files=True))\n    spikes_schema.external['analysis'].delete(delete_external_files=True))\n    clusterless_schema.external['analysis'].delete(delete_external_files=True))\n</code></pre> <p>The <code>delete</code> calls above use DataJoint's <code>ExternalTable.delete</code> method, which will remove files from disk that are no longer referenced in the database.</p>"}, {"location": "misc/figurl_views/", "title": "Creating figurl views", "text": ""}, {"location": "misc/figurl_views/#spike-sorting-recording-view", "title": "Spike sorting recording view", "text": "<pre><code>import spyglass.common as ndc\nimport spyglass.figurl_views as ndf\n\nquery = ...\n\n# To replace:\n# (ndf.SpikeSortingRecordingView &amp; query).delete()\n\nndf.SpikeSortingRecordingView.populate([(ndc.SpikeSortingRecording &amp; query).proj()])\n</code></pre>"}, {"location": "misc/figurl_views/#spike-sorting-view", "title": "Spike sorting view", "text": "<pre><code>import spyglass.common as ndc\nimport spyglass.figurl_views as ndf\n\nquery = ...\n\n# To replace:\n# (ndf.SpikeSortingView &amp; query).delete()\n\nndf.SpikeSortingView.populate([(ndc.SpikeSorting &amp; query).proj()])\n</code></pre>"}, {"location": "misc/insert_data/", "title": "How to insert data into <code>spyglass</code>", "text": "<p>In <code>spyglass</code>, every table corresponds to an object. An experimental session is defined as a collection of such objects. When an NWB file is ingested into <code>spyglass</code>, the information about these objects is first read and inserted into tables in the <code>common</code> module (e.g. <code>Institution</code>, <code>Lab</code>, <code>Electrode</code>, etc). However, not every NWB file has all the information required by <code>spyglass</code>. For example, many NWB files do not contain any information about the <code>DataAcquisitionDevice</code> or <code>Probe</code> because NWB does not yet have an official standard for specifying them. In addition, one might find that the information contained in the NWB file is incorrect and would like to modify it before inserting it into <code>spyglass</code> without having to go through the time-consuming process of re-generating the NWB file. For these cases, we provide an alternative approach to inserting data to <code>spyglass</code>.</p> <p>This alternate approach consists of two steps. First, the user must identify entries that they would like to add to the <code>spyglass</code> database that exist independently of any particular NWB file. For example, information about a particular probe is stored in the <code>ProbeType</code> and <code>Probe</code> tables of <code>spyglass.common</code>. The user can either:</p> <ol> <li> <p>create these entries programmatically using DataJoint <code>insert</code> commands, for     example:</p> <pre><code>sgc.ProbeType.insert1(\n    {\n        \"probe_type\": \"128c-4s6mm6cm-15um-26um-sl\",\n        \"probe_description\": \"A Livermore flexible probe with 128 channels, 4 shanks, 6 mm shank length, 6 cm ribbon length. 15 um contact diameter, 26 um center-to-center distance (pitch), single-line configuration.\",\n        \"manufacturer\": \"Lawrence Livermore National Lab\",\n        \"num_shanks\": 4,\n    },\n    skip_duplicates=True,\n)\n</code></pre> </li> <li> <p>define these entries in a special YAML file called <code>entries.yaml</code> that is     processed when <code>spyglass</code> is imported. One can think of <code>entries.yaml</code> as a     place to define information that the database should come pre-equipped     prior to ingesting any NWB files. The <code>entries.yaml</code> file should be placed     in the <code>spyglass</code> base directory. An example can be found in     <code>examples/config_yaml/entries.yaml</code>. It has the following structure:</p> <pre><code>TableName:\n    - TableEntry1Field1: Value\n\nTableEntry1Field2:\n    - TableEntry2Field1: Value\n\nTableEntry2Field2: Value\n</code></pre> <p>For example,</p> <pre><code>ProbeType:\n    - probe_type: 128c-4s6mm6cm-15um-26um-sl\nprobe_description: A Livermore flexible probe with 128 channels, 4 shanks, 6 mm shank\n    length, 6 cm ribbon length. 15 um contact diameter, 26 um center-to-center distance\n    (pitch), single-line configuration.\nmanufacturer: Lawrence Livermore National Lab\nnum_shanks: 4\n</code></pre> </li> </ol> <p>Using a YAML file over programmatically creating these entries in a notebook or script has the advantages that the YAML file maintains a record of what entries have been added that is easy to access, and the file is portable and can be shared alongside an NWB file or set of NWB files from a given experiment.</p> <p>Next, the user must associate the NWB file with entries defined in the database. This is done by cresqating a configuration file, which must: be in the same directory as the NWB file that it configures be in YAML format have the following naming convention: <code>&lt;name_of_nwb_file&gt;_spyglass_config.yaml</code>.</p> <p>Users can programmatically generate this configuration file. It is then read by spyglass when calling <code>insert_session</code> on the associated NWB file.</p> <p>An example of this can be found at <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code>. This file is associated with the NWB file <code>sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb</code>.</p> <p>This is the general format for the config entry:</p> <pre><code>TableName:\n  - primary_key1: value1\n</code></pre> <p>For example:</p> <pre><code>DataAcquisitionDevice:\n  - data_acquisition_device_name: Neuropixels Recording Device\n</code></pre> <p>In this example, the NWB file that corresponds to this config YAML will become associated with the DataAcquisitionDevice with primary key data_acquisition_device_name: Neuropixels Recording Device. This entry must exist.</p>"}, {"location": "misc/merge_tables/", "title": "Merge Tables", "text": ""}, {"location": "misc/merge_tables/#why", "title": "Why", "text": "<p>A pipeline may diverge when we want to process the same data in different ways. Merge Tables allow us to join divergent pipelines together, and unify downstream processing steps. For a more in depth discussion, please refer to this notebook and related discussions here and here.</p> <p>Note: Deleting entries upstream of Merge Tables will throw errors related to deleting a part entry before the master. To circumvent this, you can add <code>force_parts=True</code> to the <code>delete</code> function call, but this will leave and orphaned primary key in the master. Instead, use <code>(YourTable &amp; restriction).delete_downstream_merge()</code> to delete master/part pairs. If errors persist, identify and import the offending part table and rerun <code>delete_downstream_merge</code> with <code>reload_cache=True</code>. This process will be faster for subsequent calls if you reassign the your table after importing.</p> <pre><code>from spyglass.common import Nwbfile\n\nnwbfile = Nwbfile()\n(nwbfile &amp; \"nwb_file_name LIKE 'Name%'\").delete_downstream_merge()\n</code></pre>"}, {"location": "misc/merge_tables/#what", "title": "What", "text": "<p>A Merge Table is fundamentally a master table with one part for each divergent pipeline. By convention...</p> <ol> <li> <p>The master table has one primary key, <code>merge_id</code>, a     UUID, and     one secondary attribute, <code>source</code>, which gives the part table name. Both     are managed with the custom <code>insert</code> function of this class.</p> </li> <li> <p>Each part table has inherits the final table in its respective pipeline, and     shares the same name as this table.</p> </li> </ol> <pre><code>from spyglass.utils.dj_merge_tables import _Merge\n\n\n@schema\nclass MergeOutput(_Merge):\n    definition = \"\"\"\n    merge_id: uuid\n    ---\n    source: varchar(32)\n    \"\"\"\n\n    class One(dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; One\n        \"\"\"\n\n    class Two(dj.Part):\n        definition = \"\"\"\n        -&gt; master\n        ---\n        -&gt; Two\n        \"\"\"\n</code></pre> <p></p> <p>By convention, Merge Tables have been named with the pipeline name plus <code>Output</code> (e.g., <code>LFPOutput</code>, <code>PositionOutput</code>). Using the underscore alias for this class allows us to circumvent a DataJoint protection that interprets the class as a table itself.</p>"}, {"location": "misc/merge_tables/#how", "title": "How", "text": ""}, {"location": "misc/merge_tables/#merging", "title": "Merging", "text": "<p>The Merge class in Spyglass's utils is a subclass of DataJoint's Manual Table and adds functions to make the awkwardness of part tables more manageable. These functions are described in the API section, under <code>utils.dj_merge_tables</code>.</p>"}, {"location": "misc/merge_tables/#restricting", "title": "Restricting", "text": "<p>In short: restrict Merge Tables with arguments, not the <code>&amp;</code> operator.</p> <ul> <li>Normally: <code>Table &amp; \"field='value'\"</code></li> <li>Instead: <code>MergeTable.merge_view(restriction=\"field='value'\"</code>).</li> </ul> <p>Caution. The <code>&amp;</code> operator may look like it's working when using <code>dict</code>, but this is because invalid keys will be ignored. <code>Master &amp; {'part_field':'value'}</code> is equivalent to <code>Master</code> alone (source).</p> <p>When provided as arguments, methods like <code>merge_get_part</code> and <code>merge_get_parent</code> will override the permissive treatment of mappings described above to only return relevant tables.</p>"}, {"location": "misc/merge_tables/#building-downstream", "title": "Building Downstream", "text": "<p>A downstream analysis will ideally be able to use all diverget pipelines interchangeably. If there are parameters that may be required for downstream processing, they should be included in the final table of the pipeline. In the example above, both <code>One</code> and <code>Two</code> might have a secondary key <code>params</code>. A downstream Computed table could do the following:</p> <pre><code>def make(self, key):\n    try:\n        params = MergeTable.merge_get_parent(restriction=key).fetch(\"params\")\n    except DataJointError:\n        params = default_params\n    processed_data = self.processing_func(key, params)\n</code></pre> <p>Note that the <code>try/except</code> above catches a possible error in the event <code>params</code> is not present in the parent.</p>"}, {"location": "misc/merge_tables/#example", "title": "Example", "text": "<p>For example usage, see our Merge Table notebook.</p>"}, {"location": "misc/session_groups/", "title": "Session groups", "text": "<p>A session group is a collection of sessions. Each group has a name (primary key) and a description.</p> <pre><code>from spyglass.common import SessionGroup\n\n# Create a new session group\nSessionGroup.add_group(\"test_group_1\", \"Description of test group 1\")\n\n# Get the table of session groups\nSessionGroup()\n\n# Add a session to the group\nSessionGroup.add_session_to_group(\"RN2_20191110_.nwb\", \"test_group_1\")\n\n# Remove a session from a group\n# SessionGroup.remove_session_from_group('RN2_20191110_.nwb', 'test_group_1')\n\n# Get all sessions in group\nSessionGroup.get_group_sessions(\"test_group_1\")\n\n# Update the description of a session group\nSessionGroup.update_session_group_description(\"test_group_1\", \"Test description\")\n</code></pre>"}, {"location": "notebooks/", "title": "Tutorial Notebooks", "text": "<p>There are several paths one can take to these notebooks. The notebooks have two-digits in their names, the first of which indicates it's 'batch', as described in the categories below.</p>"}, {"location": "notebooks/#0-intro", "title": "0. Intro", "text": "<p>Everyone should complete the Setup and Insert Data notebooks.</p> <p>Data Sync is an optional additional tool for collaborators that want to share analysis files.</p> <p>The Merge Tables notebook explains details on a new table tier unique to Spyglass that allows the user to use different versions of pipelines on the same data. This is important for understanding the later notebooks.</p>"}, {"location": "notebooks/#1-spike-sorting-pipeline", "title": "1. Spike Sorting Pipeline", "text": "<p>This series of notebooks covers the process of spike sorting, from automated spike sorting to optional manual curation of the output of the automated sorting.</p>"}, {"location": "notebooks/#2-position-pipeline", "title": "2. Position Pipeline", "text": "<p>This series of notebooks covers tracking the position(s) of the animal. The user can employ two different methods:</p> <ol> <li>the simple Trodes methods of tracking LEDs on the     animal's headstage</li> <li>DLC (DeepLabCut) which uses a neural network to track the     animal's body parts.</li> </ol> <p>Either case can be followed by the Linearization notebook if the user wants to linearize the position data for later use.</p>"}, {"location": "notebooks/#3-lfp-pipeline", "title": "3. LFP Pipeline", "text": "<p>This series of notebooks covers the process of LFP analysis. The LFP covers the extraction of the LFP in specific bands from the raw data. The Theta notebook shows specifically how to extract the theta band power and phase from the LFP data. Finally the Ripple Detection notebook shows how to detect ripples in the LFP data.</p>"}, {"location": "notebooks/#4-decoding-pipeline", "title": "4. Decoding Pipeline", "text": "<p>This series of notebooks covers the process of decoding the position of the animal from spiking data. It relies on the position data from the Position pipeline and the output of spike sorting from the Spike Sorting pipeline. Decoding can be from sorted or from unsorted data using spike waveform features (so-called clusterless decoding).</p> <p>The first notebook (Extracting Clusterless Waveform Features) in this series shows how to retrieve the spike waveform features used for clusterless decoding.</p> <p>The second notebook (Clusterless Decoding) shows a detailed example of how to decode the position of the animal from the spike waveform features. The third notebook (Decoding) shows how to decode the position of the animal from the sorted spikes.</p>"}, {"location": "notebooks/#developer-note", "title": "Developer note", "text": "<p>The <code>py_scripts</code> directory contains the same notebook data in <code>.py</code> form to facilitate GitHub PR reviews. To update them, run the following from the root Spyglass directory</p> <pre><code>pip install jupytext\njupytext --to py notebooks/*ipynb\nmv notebooks/*py notebooks/py_scripts\nblack .\n</code></pre> <p>Unfortunately, jupytext-generated py script are not black-compliant by default.</p> <p>You can ensure black compliace with the <code>pre-commit</code> hook by running</p> <pre><code>pip install pre-commit\n</code></pre> <p>This will run black whenever you commit changes to the repository.</p>"}, {"location": "notebooks/00_Setup/", "title": "Setup", "text": "<p>Welcome to Spyglass, a DataJoint pipeline maintained by the Frank Lab at UCSF.</p> <p>Spyglass will help you take an NWB file from raw data to analysis-ready preprocessed formats using DataJoint to (a) connect to a relational database (here, MySQL), and (b) automate processing steps. To use Spyglass, you'll need to ...</p> <ol> <li>Set up your local environment</li> <li>Connect to a database</li> </ol> <p>JupyterHub users can skip this step. Frank Lab members should first follow 'rec to nwb overview' steps on Google Drive to set up an ssh connection.</p> <p>For local use, download and install ...</p> <ol> <li>Python 3.9.</li> <li>mamba as a replacement for conda. Spyglass installation is significantly faster with mamba.<pre>wget \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\nbash Miniforge3-$(uname)-$(uname -m).sh\n</pre> </li> <li>VS Code with relevant python extensions, including Jupyter. Hold off on selecting your interpreter until after you make the environment with <code>mamba</code>.</li> <li>git for downloading the repository, including notebooks.</li> </ol> <p>In a terminal, ...</p> <ol> <li>navigate to your project directory.</li> <li>use <code>git</code> to download the Spyglass repository.</li> <li>navigate to the newly downloaded directory.</li> <li>create a <code>mamba</code> environment with either the standard <code>environment.yml</code> or the <code>environment_position.yml</code>, if you intend to use the full position pipeline. The latter will take longer to install.</li> <li>open this notebook with VSCode</li> </ol> <p>Commands for the steps above ...</p> <pre>cd /your/project/directory/ # 1\ngit clone https://github.com/LorenFrankLab/spyglass/ # 2\ncd spyglass # 3\nmamba env create -f environment.yml # 4\ncode notebooks/00_Setup.ipynb # 5\n</pre> <p>Note: Spyglass is also installable via pip and pypi with <code>pip install spyglass-neuro</code>, but downloading from GitHub will also download other files.</p> <p>Next, within VSCode, select the kernel that matches your spyglass environment created with <code>mamba</code>. To use other Python interfaces, be sure to activate the environment: <code>conda activate spyglass</code></p> <p>See this guide for additional details on each of these programs and the role they play in using the pipeline.</p> <p>You have a few options for databases.</p> <ol> <li>Connect to an existing database.</li> <li>Run your own database with Docker</li> <li>JupyterHub (coming soon...)</li> </ol> <p>Your choice above should result in a set of credentials, including host name, host port, user name, and password. Note these for the next step.</p> Note for MySQL 8 users, including Frank Lab members <p>Using a MySQL 8 server, like the server hosted by the Frank Lab, will require the pre-release version of DataJoint to change one's password.</p> <pre>cd /location/for/datajoint/source/files/\ngit clone https://github.com/datajoint/datajoint-python\npip install ./datajoint-python\n</pre> <p>Connecting to an existing database will require a user name and password. Please contact your database administrator for this information.</p> <p>Frank Lab members should contact Chris.</p> <ul> <li><p>First, install Docker.</p> </li> <li><p>Add yourself to the <code>docker</code> group so that you don't have to be sudo to run docker.</p> </li> <li><p>Download the docker image for <code>datajoint/mysql:8.0</code>.</p> <pre>docker pull datajoint/mysql:8.0\n</pre> </li> <li><p>When run, this is referred to as a 'Docker container'</p> </li> <li><p>Next start the container with a couple additional pieces of info...</p> <ul> <li>Root password. We use <code>tutorial</code>.</li> <li>Database name. Here, we use <code>spyglass-db</code>.</li> <li>Port mapping. Here, we map 3306 across the local machine and container.</li> </ul> <pre>docker run --name spyglass-db -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql:8.0\n</pre> </li> <li><p>For data to persist after terminating the container, attach a volume when running:</p> <pre>docker volume create dj-vol\ndocker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql\n</pre> </li> </ul> <p>Docker credentials are as follows:</p> <ul> <li>Host: localhost</li> <li>Password: tutorial</li> <li>User: root</li> <li>Port: 3306</li> </ul> <p>Spyglass can load settings from either a DataJoint config file (recommended) or environmental variables. The code below will generate a config file, but we first need to decide a 'base path'. This is generally the parent directory where the data will be stored, with subdirectories for <code>raw</code>, <code>analysis</code>, and other data folders. If they don't exist already, they will be created.</p> <p>The function below will create a config file (<code>~/.datajoint.config</code> if global, <code>./dj_local_conf.json</code> if local). Local is recommended for the notebooks, as each will start by loading this file. Custom json configs can be saved elsewhere, but will need to be loaded in startup with <code>dj.config.load('your-path')</code>.</p> <p>To point spyglass to a folder elsewhere (e.g., an external drive for waveform data), simply edit the json file. Note that the <code>raw</code> and <code>analysis</code> paths appear under both <code>stores</code> and <code>custom</code>.</p> In\u00a0[\u00a0]: Copied! <pre>import os\nfrom spyglass.settings import SpyglassConfig\n\n# change to the root directory of the project\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\n\nSpyglassConfig().save_dj_config(\n    save_method=\"local\",  # global or local\n    base_dir=\"/path/like/stelmo/nwb/\",\n    database_user=\"your username\",\n    database_password=\"your password\",  # remove this line for shared machines\n    database_host=\"localhost or lmf-db.cin.ucsf.edu\",\n    database_port=3306,\n    set_password=False,\n)\n</pre> import os from spyglass.settings import SpyglassConfig  # change to the root directory of the project if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\")  SpyglassConfig().save_dj_config(     save_method=\"local\",  # global or local     base_dir=\"/path/like/stelmo/nwb/\",     database_user=\"your username\",     database_password=\"your password\",  # remove this line for shared machines     database_host=\"localhost or lmf-db.cin.ucsf.edu\",     database_port=3306,     set_password=False, ) <p>If you used either a local or global save method, we can check the connection to the database with ...</p> In\u00a0[\u00a0]: Copied! <pre>import datajoint as dj\n\ndj.conn()  # test connection\ndj.config  # check config\n\nfrom spyglass.common import Nwbfile\n\nNwbfile()\n</pre> import datajoint as dj  dj.conn()  # test connection dj.config  # check config  from spyglass.common import Nwbfile  Nwbfile() <p>If you see an error saying <code>Could not find SPYGLASS_BASE_DIR</code>, try loading your config before importing Spyglass, try setting this as an environmental variable before importing Spyglass.</p> <pre>os.environ['SPYGLASS_BASE_DIR'] = '/your/base/path'\n\nimport spyglass\nfrom spyglass.settings import SpyglassConfig\nimport datajoint as dj\nprint(SpyglassConfig().config)\ndj.config.save_local() # or global\n</pre> <p>Next, we'll try inserting data</p>"}, {"location": "notebooks/00_Setup/#setup", "title": "Setup\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#local-environment", "title": "Local environment\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#database", "title": "Database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#existing-database", "title": "Existing Database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#running-your-own-database-with-docker", "title": "Running your own database with Docker\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#config-and-connecting-to-the-database", "title": "Config and Connecting to the database\u00b6", "text": ""}, {"location": "notebooks/00_Setup/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/", "title": "Insert Data", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> </ul> <p>Let's start by importing the <code>spyglass</code> package, along with a few others.</p> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\nwarnings.simplefilter(\"ignore\", category=UserWarning)\n\n# spyglass.common has the most frequently used tables\nimport spyglass.common as sgc\n\n# spyglass.data_import has tools for inserting NWB files into the database\nimport spyglass.data_import as sgi\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) warnings.simplefilter(\"ignore\", category=UserWarning)  # spyglass.common has the most frequently used tables import spyglass.common as sgc  # spyglass.data_import has tools for inserting NWB files into the database import spyglass.data_import as sgi <pre>[2024-01-29 16:24:30,933][INFO]: Connecting root@localhost:3309\n[2024-01-29 16:24:30,942][INFO]: Connected root@localhost:3309\n</pre> <p>Datajoint enables users to use Python to build and interact with a Relational Database. In a Relational Data Model, each table is an object that can reference information in other tables to avoid redundancy.</p> <p>DataJoint has built-in tools for generating/saving a Diagram of the relationships between tables. This page describes the notation used.</p> <p>Polygons are tables, colors reference table type:</p> <ul> <li>Green rectangle: tables whose entries are entered manually.</li> <li>Blue oval: tables whose entries are imported from external files (e.g. NWB file).</li> <li>Red circle: tables whose entries are computed from entries of other tables.</li> <li>No shape (only text): tables whose entries are part of the table upstream</li> </ul> <p>Lines are dependencies between tables. An upstream table is connected to a downstream table via inheritance of the primary key. This is the set of attributes (i.e., column names) used to uniquely define an entry (i.e., a row)</p> <ul> <li>Bold lines: the upstream primary key is the sole downstream primary key</li> <li>Solid lines: the upstream table as part of the downstream primary key</li> <li>Dashed lines: the primary key of upstream table as non-primary key</li> </ul> In\u00a0[2]: Copied! <pre># Draw tables that are two levels below and one level above Session\ndj.Diagram(sgc.Session) - 1 + 2\n</pre> # Draw tables that are two levels below and one level above Session dj.Diagram(sgc.Session) - 1 + 2 Out[2]: <p>By adding diagrams together, of adding and subtracting levels, we can visualize key parts of Spyglass.</p> <p>Note: Notice the Selection tables. This is a design pattern that selects a subset of upstream items for further processing. In some cases, these also pair the selected data with processing parameters.</p> <p>After exploring the pipeline's structure, we'll now grab some example data. Spyglass will assume that the data is a neural recording with relevant auxiliary in NWB.</p> <p>We offer a few examples:</p> <ul> <li><code>minirec20230622.nwb</code>, .3 GB: minimal recording, Link</li> <li><code>mediumnwb20230802.nwb</code>, 32 GB: full-featured dataset, Link</li> <li><code>montague20200802.nwb</code>, 8 GB: full experimental recording, Link</li> <li>For those in the UCSF network, these and many others on <code>/stelmo/nwb/raw</code></li> </ul> <p>If you are connected to the Frank lab database, please rename any downloaded files (e.g., <code>example20200101_yourname.nwb</code>) to avoid naming collisions, as the file name acts as the primary key across key tables.</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\n# Define the name of the file that you copied and renamed\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  # Define the name of the file that you copied and renamed nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) <p>Spyglass will create a copy with this name.</p> In\u00a0[4]: Copied! <pre>nwb_copy_file_name\n</pre> nwb_copy_file_name Out[4]: <pre>'minirec20230622_.nwb'</pre> <p>Let's start small by inserting personnel information.</p> <p>The <code>LabMember</code> table lists all lab members, with an additional part table for <code>LabMemberInfo</code>. This holds Google account and DataJoint username info for each member, for authentication purposes.</p> <p>We can insert lab member information using the NWB file <code>experimenter</code> field as follows...</p> In\u00a0[5]: Copied! <pre># take a look at the lab members\nsgc.LabMember.insert_from_nwbfile(nwb_file_name)\n</pre> # take a look at the lab members sgc.LabMember.insert_from_nwbfile(nwb_file_name) <pre>Please add the Google user ID for Firstname Lastname in the LabMember.LabMemberInfo table to help manage permissions.\nPlease add the Google user ID for Firstname2 Lastname2 in the LabMember.LabMemberInfo table to help manage permissions.\n</pre> <p>We can insert into <code>LabMemberInfo</code> directly with a list of lists that reflect the order of the fields present in the table. See this notebook for examples of inserting with <code>dicts</code>.</p> In\u00a0[6]: Copied! <pre>sgc.LabMember.LabMemberInfo.insert(\n    [  # Full name, Google email address, DataJoint username, admin\n        [\"Firstname Lastname\", \"example1@gmail.com\", \"example1\", 0],\n        [\"Firstname2 Lastname2\", \"example2@gmail.com\", \"example2\", 0],\n    ],\n    skip_duplicates=True,\n)\nsgc.LabMember.LabMemberInfo()\n</pre> sgc.LabMember.LabMemberInfo.insert(     [  # Full name, Google email address, DataJoint username, admin         [\"Firstname Lastname\", \"example1@gmail.com\", \"example1\", 0],         [\"Firstname2 Lastname2\", \"example2@gmail.com\", \"example2\", 0],     ],     skip_duplicates=True, ) sgc.LabMember.LabMemberInfo() Out[6]: Information about lab member in the context of Frank lab network <p>lab_member_name</p> <p>google_user_name</p> used for permission to curate <p>datajoint_user_name</p> used for permission to delete entries Firstname Lastname example1@gmail.com example1Firstname2 Lastname2 example2@gmail.com example2 <p>Total: 2</p> <p>A <code>LabTeam</code> is a set of lab members who own a set of NWB files and the associated information in the database. This is often a subgroup that collaborates on the same projects. Data is associated with a given team, granting members analysis (e.g., curation) and deletion (coming soon) privileges.</p> In\u00a0[7]: Copied! <pre>team_name = \"My Team\"\nsgc.LabTeam().create_new_team(\n    team_name=team_name,  # Should be unique\n    team_members=[\"Firstname Lastname\", \"Firstname2 Lastname2\"],\n    team_description=\"test\",  # Optional\n)\n</pre> team_name = \"My Team\" sgc.LabTeam().create_new_team(     team_name=team_name,  # Should be unique     team_members=[\"Firstname Lastname\", \"Firstname2 Lastname2\"],     team_description=\"test\",  # Optional ) <p>By default, each member is part of their own team. We can see all teams and members by looking at the part table <code>LabTeam.LabTeamMember</code>.</p> In\u00a0[8]: Copied! <pre>sgc.LabTeam.LabTeamMember()\n</pre> sgc.LabTeam.LabTeamMember() Out[8]: <p>team_name</p> <p>lab_member_name</p> Firstname Lastname Firstname LastnameMy Team Firstname LastnameFirstname2 Lastname2 Firstname2 Lastname2My Team Firstname2 Lastname2 <p>Total: 4</p> <p><code>spyglass.data_import.insert_sessions</code> helps take the many fields of data present in an NWB file and insert them into various tables across Spyglass. If the NWB file is properly composed, this includes...</p> <ul> <li>the experimenter (replicating part of the process above)</li> <li>animal behavior (e.g. video recording of position)</li> <li>neural activity (extracellular recording of multiple brain areas)</li> <li>etc.</li> </ul> <p>Note: this may take time as Spyglass creates the copy. You may see a prompt about inserting device information.</p> In\u00a0[3]: Copied! <pre>sgi.insert_sessions(nwb_file_name)\n</pre> sgi.insert_sessions(nwb_file_name) <pre>Creating a copy of NWB file minirec20230622.nwb with link to raw ephys data: minirec20230622_.nwb\nPopulate Session...\nNo config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\nInstitution...\nLab...\nLabMember...\nPlease add the Google user ID for Firstname2 Lastname2 in the LabMember.LabMemberInfo table to help manage permissions.\nSubject...\nPopulate CameraDevice...\nInserted camera devices ['test camera 1']\n\nPopulate Probe...\nProbe ID '128c-4s6mm6cm-15um-26um-sl' already exists in the database. Spyglass will use that and not create a new Probe, Shanks, or Electrodes.\nInserted probes {'128c-4s6mm6cm-15um-26um-sl'}\n\nSkipping Apparatus for now...\nIntervalList...\nLabMember with name lastname, firstname does not exist. Cannot link Session with LabMember in Session.Experimenter.\nLabMember with name lastname2, firstname2 does not exist. Cannot link Session with LabMember in Session.Experimenter.\nPopulate ElectrodeGroup...\nPopulate Electrode...\nNo config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\nPopulate Raw...\nEstimating sampling rate...\nEstimated sampling rate for file: 30000.0 Hz\nImporting raw data: Sampling rate:\t30000.0 Hz\nNumber of valid intervals:\t2\nPopulate SampleCount...\nPopulate DIOEvents...\nPopulate TaskEpochs\nPopulate StateScriptFile\nPopulate VideoFile\nNo video found corresponding to file minirec20230622_.nwb, epoch 01_s1\nNo video found corresponding to file minirec20230622_.nwb, epoch 02_s2\nRawPosition...\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.1912336349487305\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.1912336349487305\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.339195609092712\nEstimated sampling rate for 12345: 30.0 Hz\nWARNING: Setting minimum valid interval to 5.339195609092712\nPopulated PosIntervalMap for minirec20230622_.nwb, 01_s1\nPopulated PosIntervalMap for minirec20230622_.nwb, 02_s2\n</pre> <p>To look at data, we can query a table with <code>Table()</code> syntax.</p> In\u00a0[11]: Copied! <pre>sgc.Lab()\n</pre> sgc.Lab() Out[11]: <p>lab_name</p> Loren Frank Lab <p>Total: 1</p> <p>The <code>Session</code> table has considerably more fields</p> In\u00a0[12]: Copied! <pre>sgc.Session.heading.names\n</pre> sgc.Session.heading.names Out[12]: <pre>['nwb_file_name',\n 'subject_id',\n 'institution_name',\n 'lab_name',\n 'session_id',\n 'session_description',\n 'session_start_time',\n 'timestamps_reference_time',\n 'experiment_description']</pre> <p>But a short primary key</p> In\u00a0[13]: Copied! <pre>sgc.Session.heading.primary_key\n</pre> sgc.Session.heading.primary_key Out[13]: <pre>['nwb_file_name']</pre> <p>The primary key is shown in bold in the html</p> In\u00a0[14]: Copied! <pre>sgc.Session()\n</pre> sgc.Session() Out[14]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p>Text only interfaces designate the primary key fields with <code>*</code></p> In\u00a0[15]: Copied! <pre>print(sgc.Session())\n</pre> print(sgc.Session()) <pre>*nwb_file_name subject_id     institution_na lab_name       session_id     session_descri session_start_ timestamps_ref experiment_des\n+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\nminirec2023062 54321          UCSF           Loren Frank La 12345          test yaml inse 2023-06-22 15: 1970-01-01 00: Test Conversio\n (Total: 1)\n\n</pre> <p>To see a the table definition, including data types, use <code>describe</code>.</p> <ul> <li><code>---</code> separates the primary key</li> <li><code>:</code> are used to separate field name from data type</li> <li><code>#</code> can be used to add comments to a field</li> </ul> In\u00a0[16]: Copied! <pre>from pprint import pprint  # adds line breaks\n\npprint(sgc.Session.describe())\n</pre> from pprint import pprint  # adds line breaks  pprint(sgc.Session.describe()) <pre>('# Table for holding experimental sessions.\\n'\n '-&gt; sgc.Nwbfile\\n'\n '---\\n'\n '-&gt; [nullable] sgc.Subject\\n'\n '-&gt; [nullable] sgc.Institution\\n'\n '-&gt; [nullable] sgc.Lab\\n'\n 'session_id=null      : varchar(200)                 \\n'\n 'session_description  : varchar(2000)                \\n'\n 'session_start_time   : datetime                     \\n'\n 'timestamps_reference_time : datetime                     \\n'\n 'experiment_description=null : varchar(2000)                \\n')\n</pre> <p>To look at specific entries in a table, we can use the <code>&amp;</code> operator. Below, we restrict based on a <code>dict</code>, but you can also use a string.</p> In\u00a0[17]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[17]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p><code>Raw</code> is connected to <code>Session</code> with a bold line, so it has the same primary key.</p> In\u00a0[18]: Copied! <pre>dj.Diagram(sgc.Session) + dj.Diagram(sgc.Raw)\n</pre> dj.Diagram(sgc.Session) + dj.Diagram(sgc.Raw) Out[18]: In\u00a0[19]: Copied! <pre>sgc.Raw &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Raw &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[19]: Raw voltage timeseries data, ElectricalSeries in NWB. <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>raw_object_id</p> the NWB object ID for loading this object from the file <p>sampling_rate</p> Sampling rate calculated from data, in Hz <p>comments</p> <p>description</p> minirec20230622_.nwb raw data valid times 4e756642-9203-4f00-b9d0-0e9747c14493 30000.0 No comment Recording of extracellular voltage <p>Total: 1</p> <p><code>IntervalList</code> is connected to <code>Session</code> with a solid line because it has additional primary key attributes. Here, you need to know both <code>nwb_file_name</code> and <code>interval_list_name</code> to uniquely identify an entry.</p> In\u00a0[20]: Copied! <pre># join/split condenses long spaces before field comments\npprint(\"\".join(sgc.IntervalList.describe().split(\"  \")))\n</pre> # join/split condenses long spaces before field comments pprint(\"\".join(sgc.IntervalList.describe().split(\"  \"))) <pre>('# Time intervals used for analysis\\n'\n '-&gt; sgc.Session\\n'\n 'interval_list_name : varchar(200) # descriptive name of this interval list\\n'\n '---\\n'\n 'valid_times: longblob # numpy array with start and end times for each '\n 'interval\\n')\n</pre> In\u00a0[21]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[21]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start and end times for each interval minirec20230622_.nwb 01_s1 =BLOB=minirec20230622_.nwb 02_s2 =BLOB=minirec20230622_.nwb pos 0 valid times =BLOB=minirec20230622_.nwb pos 1 valid times =BLOB=minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 5</p> <p>Raw data types like <code>valid_times</code> are shown as <code>=BLOB=</code>. We can inspect these with <code>fetch</code></p> <p>Note: like <code>insert</code>/<code>insert1</code>, <code>fetch</code> can be uses as <code>fetch1</code> to raise an error when many (or no) entries are retrieved. To limit to one entry when there may be many, use <code>query.fetch(limit=1)[0]</code></p> In\u00a0[22]: Copied! <pre>(\n    sgc.IntervalList\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": \"pos 1 valid times\",\n    }\n).fetch1(\"valid_times\")\n</pre> (     sgc.IntervalList     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": \"pos 1 valid times\",     } ).fetch1(\"valid_times\") Out[22]: <pre>array([[1.68747483e+09, 1.68747484e+09]])</pre> <p>In DataJoint operators, <code>&amp;</code> selects by a condition and <code>-</code> removes a condition.</p> In\u00a0[23]: Copied! <pre>(\n    (\n        (sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name})\n        - {\"interval_list_name\": \"pos 1 valid times\"}\n    )\n    - {\"interval_list_name\": \"pos 2 valid times\"}\n).fetch(\"interval_list_name\")\n</pre> (     (         (sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name})         - {\"interval_list_name\": \"pos 1 valid times\"}     )     - {\"interval_list_name\": \"pos 2 valid times\"} ).fetch(\"interval_list_name\") Out[23]: <pre>array(['01_s1', '02_s2', 'pos 0 valid times', 'raw data valid times'],\n      dtype=object)</pre> <p><code>IntervalList</code> has an additional secondary key <code>pipeline</code> which can describe the origin of the data. Because it is a secondary key, it is not required to uniquely identify an entry. Current values for this key from spyglass pipelines are:</p> pipeline Source position sg.common.PositionSource lfp_v0 sg.common.LFP lfp_v1 sg.lfp.v1.LFPV1 lfp_band sg.common.LFPBand, sg.lfp.analysis.v1.LFPBandV1 lfp_artifact sg.lfp.v1.LFPArtifactDetection spikesorting_artifact_v0 sg.spikesorting.ArtifactDetection spikesorting_artifact_v1 sg.spikesorting.v1.ArtifactDetection spikesorting_recording_v0 sg.spikesorting.SpikeSortingRecording spikesorting_recording_v1 sg.spikesorting.v1.SpikeSortingRecording <p>Another neat feature of DataJoint is that it automatically maintains data integrity with cascading deletes. For example, if we delete our <code>Session</code> entry, all associated downstream entries are also deleted (e.g. <code>Raw</code>, <code>IntervalList</code>).</p> <p>Note: The deletion process can be complicated by Merge Tables when the entry is referenced by a part table. To demo deletion in these cases, run the hidden code below.</p> Quick Merge Insert <pre>import spyglass.lfp as lfp\n\nsgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"lfp_electrode_group_name\": \"test\",\n    \"target_interval_list_name\": \"01_s1\",\n    \"filter_name\": \"LFP 0-400 Hz\",\n    \"filter_sampling_rate\": 30_000,\n}\nlfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True)\nlfp.v1.LFPV1().populate(lfp_key)\n</pre> Deleting Merge Entries <pre>nwbfile = sgc.Nwbfile()\n\n(nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete_downstream_merge(\n    dry_run=False, # True will show Merge Table entries that would be deleted\n)\n</pre> <p>Please see the next notebook for a more detailed explanation.</p> In\u00a0[24]: Copied! <pre>session_entry = sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\nsession_entry\n</pre> session_entry = sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} session_entry Out[24]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p><code>Session.Experimenter</code> is used for permissions checks when deleting. The session will need to have an experimenter in order avoid an error being raised during this check.</p> In\u00a0[\u00a0]: Copied! <pre>sess_key = (sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(\n    \"KEY\", as_dict=True\n)[0]\nexp_key = (sgc.LabMember).fetch(\"KEY\", as_dict=True)[0]\nsgc.Session.Experimenter.insert1(\n    dict(**sess_key, **exp_key), skip_duplicates=True\n)\n</pre> sess_key = (sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(     \"KEY\", as_dict=True )[0] exp_key = (sgc.LabMember).fetch(\"KEY\", as_dict=True)[0] sgc.Session.Experimenter.insert1(     dict(**sess_key, **exp_key), skip_duplicates=True ) <p>Even with the experimenter specified, there are still delete protections in place. To see an example, uncomment the cell below.</p> In\u00a0[\u00a0]: Copied! <pre># session_entry.delete()\n</pre> # session_entry.delete() <p>To delete, you'll need to share a team with the session experimenter.</p> In\u00a0[\u00a0]: Copied! <pre>your_name = \"YourFirst YourLast\"\nparts = your_name.split(\" \")\nsgc.LabMember.insert1([your_name, parts[0], parts[1]])\nsgc.LabMember.LabMemberInfo.insert1(\n    [your_name, \"your_gmail\", dj.config[\"database.user\"], 0]\n)\nsgc.LabTeam.LabTeamMember.insert1([team_name, your_name])\n</pre> your_name = \"YourFirst YourLast\" parts = your_name.split(\" \") sgc.LabMember.insert1([your_name, parts[0], parts[1]]) sgc.LabMember.LabMemberInfo.insert1(     [your_name, \"your_gmail\", dj.config[\"database.user\"], 0] ) sgc.LabTeam.LabTeamMember.insert1([team_name, your_name]) <p>By default, DataJoint is cautious about deletes and will prompt before deleting. To delete, uncomment the cell below and respond <code>yes</code> in the prompt.</p> In\u00a0[26]: Copied! <pre># session_entry.delete()\n</pre> # session_entry.delete() <pre>[2023-09-28 08:29:15,814][INFO]: Deleting 4 rows from `common_behav`.`_raw_position__pos_object`\nINFO:datajoint:Deleting 4 rows from `common_behav`.`_raw_position__pos_object`\n[2023-09-28 08:29:15,822][INFO]: Deleting 2 rows from `common_behav`.`_raw_position`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`_raw_position`\n[2023-09-28 08:29:15,834][INFO]: Deleting 4 rows from `common_behav`.`position_source__spatial_series`\nINFO:datajoint:Deleting 4 rows from `common_behav`.`position_source__spatial_series`\n[2023-09-28 08:29:15,841][INFO]: Deleting 2 rows from `common_behav`.`position_source`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`position_source`\n[2023-09-28 08:29:15,851][INFO]: Deleting 7 rows from `common_dio`.`_d_i_o_events`\nINFO:datajoint:Deleting 7 rows from `common_dio`.`_d_i_o_events`\n[2023-09-28 08:29:15,871][INFO]: Deleting 128 rows from `common_ephys`.`_electrode`\nINFO:datajoint:Deleting 128 rows from `common_ephys`.`_electrode`\n[2023-09-28 08:29:15,879][INFO]: Deleting 1 rows from `common_ephys`.`_electrode_group`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_electrode_group`\n[2023-09-28 08:29:15,887][INFO]: Deleting 1 rows from `common_ephys`.`_raw`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_raw`\n[2023-09-28 08:29:15,896][INFO]: Deleting 1 rows from `common_ephys`.`_sample_count`\nINFO:datajoint:Deleting 1 rows from `common_ephys`.`_sample_count`\n[2023-09-28 08:29:15,908][INFO]: Deleting 2 rows from `common_behav`.`__position_interval_map`\nINFO:datajoint:Deleting 2 rows from `common_behav`.`__position_interval_map`\n[2023-09-28 08:29:15,918][INFO]: Deleting 2 rows from `common_task`.`_task_epoch`\nINFO:datajoint:Deleting 2 rows from `common_task`.`_task_epoch`\n[2023-09-28 08:29:15,924][INFO]: Deleting 5 rows from `common_interval`.`interval_list`\nINFO:datajoint:Deleting 5 rows from `common_interval`.`interval_list`\n[2023-09-28 08:29:15,931][INFO]: Deleting 1 rows from `common_session`.`_session`\nINFO:datajoint:Deleting 1 rows from `common_session`.`_session`\n[2023-09-28 08:29:18,765][INFO]: Deletes committed.\nINFO:datajoint:Deletes committed.\n</pre> Out[26]: <pre>1</pre> <p>We can check that delete worked, both for <code>Session</code> and <code>IntervalList</code></p> In\u00a0[27]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[27]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> <p>Total: 0</p> In\u00a0[28]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[28]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start and end times for each interval <p>Total: 0</p> <p><code>delete</code> is useful for re-running something. Editing entries is possible, but discouraged because it can lead to integrity issues. Instead, re-enter and let the automation handle the rest.</p> <p>Spyglass falls short, however, in that deleting from <code>Session</code> doesn't also delete the associated entry in <code>Nwbfile</code>, which has to be removed separately (for now). This table offers a <code>cleanup</code> method to remove the added files (with the <code>delete_files</code> argument as <code>True</code>).</p> <p>Note: this also applies to deleting files from <code>AnalysisNwbfile</code> table.</p> In\u00a0[53]: Copied! <pre># Uncomment to delete\n# (sgc.Nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete()\n</pre> # Uncomment to delete # (sgc.Nwbfile &amp; {\"nwb_file_name\": nwb_copy_file_name}).delete() <pre>[2023-07-18 19:01:15,343][INFO]: Deleting 1 rows from `common_nwbfile`.`nwbfile`\nINFO:datajoint:Deleting 1 rows from `common_nwbfile`.`nwbfile`\n[2023-07-18 19:01:17,130][INFO]: Deletes committed.\nINFO:datajoint:Deletes committed.\n</pre> Out[53]: <pre>1</pre> <p>Note that the file (ends with <code>_.nwb</code>) has not been deleted, even if the entry was deleted above.</p> In\u00a0[\u00a0]: Copied! <pre>!ls $SPYGLASS_BASE_DIR/raw\n</pre> !ls $SPYGLASS_BASE_DIR/raw <pre>minirec20230622.nwb\nminirec20230622_.nwb\nmontague20200802_tutorial.nwb\nmontague20200802_tutorial_.nwb\nmontague20200802_tutorial__.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim_.nwb\ntonks20211103_.nwb\n</pre> <p>We can clean these files with the <code>cleanup</code> method</p> In\u00a0[55]: Copied! <pre>sgc.Nwbfile().cleanup(delete_files=True)\n</pre> sgc.Nwbfile().cleanup(delete_files=True) <pre>100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1/1 [00:00&lt;00:00, 304.24it/s]\n</pre> In\u00a0[56]: Copied! <pre>!ls $SPYGLASS_BASE_DIR/raw\n</pre> !ls $SPYGLASS_BASE_DIR/raw <pre>minirec20230622.nwb\nmontague20200802_tutorial.nwb\nmontague20200802_tutorial_.nwb\nmontague20200802_tutorial__.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim.nwb\nsub-despereaux_ses-despereaux-08_behavior+ecephys_trim_.nwb\ntonks20211103_.nwb\n</pre> <p>In the next notebook, we'll explore tools for syncing.</p>"}, {"location": "notebooks/01_Insert_Data/#insert-data", "title": "Insert Data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#visualizing-the-database", "title": "Visualizing the database\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#example-data", "title": "Example data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#basic-inserts-lab-team", "title": "Basic Inserts: Lab Team\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#inserting-from-nwb", "title": "Inserting from NWB\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#inspecting-the-data", "title": "Inspecting the data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#deleting-data", "title": "Deleting data\u00b6", "text": ""}, {"location": "notebooks/01_Insert_Data/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/", "title": "Data Sync", "text": "<p>This notebook will cover ...</p> <ol> <li>General Kachery information</li> <li>Setting up Kachery as a host. If you'll use an existing host, skip this.</li> <li>Setting up Kachery in your database. If you're using an existing database, skip this.</li> <li>Adding Kachery data.</li> </ol> <p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>To fully demonstrate syncing features, we'll need to run some basic analyses. This can either be done with code in this notebook or by running another notebook (e.g., LFP)</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> </ul> <p>Let's start by importing the <code>spyglass</code> package and testing that your environment is properly configured for kachery sharing</p> <p>If you haven't already done so, be sure to set up your Spyglass base directory and Kachery sharing directory with Setup</p> In\u00a0[\u00a0]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection\n\nimport spyglass.common as sgc\nimport spyglass.sharing as sgs\nfrom spyglass.settings import config\n\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection  import spyglass.common as sgc import spyglass.sharing as sgs from spyglass.settings import config  import warnings  warnings.filterwarnings(\"ignore\") <pre>[2023-12-22 08:22:32,189][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2023-12-22 08:22:32,244][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n</pre> <p>For example analysis files, run the code hidden below.</p> Quick Analysis <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nimport spyglass.data_import as sgi\nimport spyglass.lfp as lfp\n\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n\nsgi.insert_sessions(nwb_file_name)\nsgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp.v1.LFPSelection.insert1(\n    {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"lfp_electrode_group_name\": \"test\",\n        \"target_interval_list_name\": \"01_s1\",\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,\n    },\n    skip_duplicates=True,\n)\nlfp.v1.LFPV1().populate()\n</pre> <p>This notebook contains instructions for setting up data sharing/syncing through Kachery Cloud, which makes it possible to share analysis results, stored in NWB files. When a user tries to access a file, Spyglass does the following:</p> <ol> <li>Try to load from the local file system/store.</li> <li>If unavailable, check if it is in the relevant sharing table (i.e., <code>NwbKachery</code> or <code>AnalysisNWBKachery</code>).</li> <li>If present, attempt to download from the associated Kachery Resource to the user's spyglass analysis directory.</li> </ol> <p>Note: large file downloads may take a long time, so downloading raw data is not supported. We suggest direct transfer with globus or a similar service.</p> <p>A Kachery Zone is a cloud storage host. The Frank laboratory has three separate Kachery zones:</p> <ol> <li><code>franklab.default</code>: Internal file sharing, including figurls</li> <li><code>franklab.collaborator</code>: File sharing with collaborating labs.</li> <li><code>franklab.public</code>: Public file sharing (not yet active)</li> </ol> <p>Setting your zone can either be done as as an environment variable or an item in a DataJoint config. Spyglass will automatically handle setting the appropriate zone when downloading database files through kachery</p> <ul> <li><p>Environment variable:</p> <pre>export KACHERY_ZONE=franklab.default\nexport KACHERY_CLOUD_DIR=/stelmo/nwb/.kachery-cloud\n</pre> </li> <li><p>DataJoint Config:</p> <pre>\"custom\": {\n   \"kachery_zone\": \"franklab.default\",\n   \"kachery_dirs\": {\n      \"cloud\": \"/your/base/path/.kachery-cloud\"\n   }\n}\n</pre> </li> </ul> <p>See instructions for setting up new Kachery Zones, including creating a cloud bucket and registering it with the Kachery team.</p> <p>Notes:</p> <ul> <li>Bucket names cannot include periods, so we substitute a dash, as in <code>franklab-default</code>.</li> <li>You only need to create an API token for your first zone.</li> </ul> <p>See instructions for setting up zone resources. This allows for sharing files on demand. We suggest using the same name for the zone and resource.</p> <p>Note: For each zone, you need to run the local daemon that listens for requests from that zone and uploads data to the bucket for client download when requested. An example of the bash script we use is</p> <pre>export KACHERY_ZONE=franklab.collaborators\n    export KACHERY_CLOUD_DIR=/stelmo/nwb/.kachery-cloud\n    cd /stelmo/nwb/franklab_collaborators_resource\n    npx kachery-resource@latest share\n</pre> <p>For convenience, we recommend saving this code as a bash script which can be executed by the local daemon. For franklab member, these scripts can be found in the directory <code>/home/loren/bin/</code>:</p> <ul> <li>run_restart_kachery_collab.sh</li> <li>run_restart_kachery_default.sh</li> </ul> <p>Once you have a hosted zone running, we need to add its information to the Spyglass database. This will allow spyglass to manage linking files from our analysis tables to kachery. First, we'll check existing Zones.</p> In\u00a0[2]: Copied! <pre>sgs.KacheryZone()\n</pre> sgs.KacheryZone() Out[2]: <p>kachery_zone_name</p> the name of the kachery zone. Note that this is the same as the name of the kachery resource. <p>description</p> description of this zone <p>kachery_cloud_dir</p> kachery cloud directory on local machine where files are linked <p>kachery_proxy</p> kachery sharing proxy <p>lab_name</p> franklab.collaborators franklab collaborator zone /stelmo/nwb/.kachery-cloud https://kachery-resource-proxy.herokuapp.com Loren Frankfranklab.default internal franklab kachery zone /stelmo/nwb/.kachery-cloud https://kachery-resource-proxy.herokuapp.com Loren Frank <p>Total: 2</p> <p>To add a new hosted Zone, we need to prepare an entry for the <code>KacheryZone</code> table. Note that the <code>kacherycloud_dir</code> key should be the path for the server daemon hosting the zone, and is not required to be present on the client machine:</p> In\u00a0[38]: Copied! <pre>zone_name = config.get(\"KACHERY_ZONE\")\ncloud_dir = config.get(\"KACHERY_CLOUD_DIR\")\n\nzone_key = {\n    \"kachery_zone_name\": zone_name,\n    \"description\": \" \".join(zone_name.split(\".\")) + \" zone\",\n    \"kachery_cloud_dir\": cloud_dir,\n    \"kachery_proxy\": \"https://kachery-resource-proxy.herokuapp.com\",\n    \"lab_name\": sgc.Lab.fetch(\"lab_name\", limit=1)[0],\n}\n</pre> zone_name = config.get(\"KACHERY_ZONE\") cloud_dir = config.get(\"KACHERY_CLOUD_DIR\")  zone_key = {     \"kachery_zone_name\": zone_name,     \"description\": \" \".join(zone_name.split(\".\")) + \" zone\",     \"kachery_cloud_dir\": cloud_dir,     \"kachery_proxy\": \"https://kachery-resource-proxy.herokuapp.com\",     \"lab_name\": sgc.Lab.fetch(\"lab_name\", limit=1)[0], } <p>Use caution when inserting into an active database, as it could interfere with ongoing work.</p> In\u00a0[39]: Copied! <pre>sgs.KacheryZone().insert1(zone_key, skip_duplicates=True)\n</pre> sgs.KacheryZone().insert1(zone_key, skip_duplicates=True) <p>Once the zone exists, we can add <code>AnalysisNWB</code> files we want to share with members of the zone.</p> <p>The <code>AnalysisNwbFileKachery</code> table links analysis files made within other spyglass tables with a <code>uri</code> used by kachery. We can view files already made available through kachery here:</p> In\u00a0[8]: Copied! <pre>sgs.AnalysisNwbfileKachery()\n</pre> sgs.AnalysisNwbfileKachery() Out[8]: <p>kachery_zone_name</p> the name of the kachery zone. Note that this is the same as the name of the kachery resource. <p>analysis_file_name</p> name of the file <p>analysis_file_uri</p> the uri of the file franklab.collaborators Banner20220224_18NJSA2B42.nwb sha1://562b488936e5288eb89e7c480ae5c10b31c9cf2ffranklab.collaborators Frodo20230810_0F936W4B9Z.nwb sha1://b38d2b0fc1e9cde91cc239e1a0b50e3211b976fcfranklab.collaborators Frodo20230810_2MJ374GSJX.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_4L35OWMGHQ.nwb sha1://a8452cf8cf6e596b44569eb9189612d2dcd4c7d6franklab.collaborators Frodo20230810_63PWL1N0VS.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_7LYW2MK0C9.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_998JNA1VBF.nwb sha1://aa0e06028d52f5195cf24d61922ace233d8da783franklab.collaborators Frodo20230810_CFKWZTGXX0.nwb sha1://ca9c238b83fd8539658a5100a9770a459a539771franklab.collaborators Frodo20230810_GMCOCDSJ54.nwb sha1://2889b68d7aa2b30561e62be519c19759facad2d3franklab.collaborators Frodo20230810_I25NQSZQ5O.nwb sha1://973ea71d97aef91e050117bf860ea2ed83950b10franklab.collaborators Frodo20230810_JS06HC1RLC.nwb sha1://088a345c5eadfa3adea021de3f158aa86a527d4efranklab.collaborators Frodo20230810_KEEEEBDUNE.nwb sha1://4aa3199011b1405e745bbe96b62b825cd93bdacd <p>...</p> <p>Total: 298</p> <p>We can share additional results by populating new entries in this table.</p> <p>To do so we first add these entries to the <code>AnalysisNwbfileKacherySelection</code> table.</p> <p>Note: This step depends on having previously run an analysis on the example file.</p> In\u00a0[40]: Copied! <pre>nwb_copy_filename = \"minirec20230622_.nwb\"\n\nanalysis_file_list = (  # Grab all analysis files for this nwb file\n    sgc.AnalysisNwbfile() &amp; {\"nwb_file_name\": nwb_copy_filename}\n).fetch(\"analysis_file_name\")\n\nkachery_selection_key = {\"kachery_zone_name\": zone_name}\n\nfor file in analysis_file_list:  # Add all analysis to shared list\n    kachery_selection_key[\"analysis_file_name\"] = file\n    sgs.AnalysisNwbfileKacherySelection.insert1(\n        kachery_selection_key, skip_duplicates=True\n    )\n</pre> nwb_copy_filename = \"minirec20230622_.nwb\"  analysis_file_list = (  # Grab all analysis files for this nwb file     sgc.AnalysisNwbfile() &amp; {\"nwb_file_name\": nwb_copy_filename} ).fetch(\"analysis_file_name\")  kachery_selection_key = {\"kachery_zone_name\": zone_name}  for file in analysis_file_list:  # Add all analysis to shared list     kachery_selection_key[\"analysis_file_name\"] = file     sgs.AnalysisNwbfileKacherySelection.insert1(         kachery_selection_key, skip_duplicates=True     ) <p>With those files in the selection table, we can add them as links to the zone by populating the <code>AnalysisNwbfileKachery</code> table:</p> In\u00a0[\u00a0]: Copied! <pre>sgs.AnalysisNwbfileKachery.populate()\n</pre> sgs.AnalysisNwbfileKachery.populate() <p>Alternatively, we can share data based on its source table in the database using the helper function <code>share_data_to_kachery()</code></p> <p>This will take a list of tables and add all associated analysis files for entries corresponding with a passed restriction. Here, we are sharing LFP and position data for the Session \"minirec20230622_.nwb\"</p> In\u00a0[4]: Copied! <pre>from spyglass.sharing import share_data_to_kachery\nfrom spyglass.lfp.v1 import LFPV1\nfrom spyglass.position.v1 import TrodesPosV1\n\ntables = [LFPV1, TrodesPosV1]\nrestriction = {\"nwb_file_name\": \"minirec20230622_.nwb\"}\nshare_data_to_kachery(\n    table_list=tables,\n    restriction=restriction,\n    zone_name=zone_name,\n)\n</pre> from spyglass.sharing import share_data_to_kachery from spyglass.lfp.v1 import LFPV1 from spyglass.position.v1 import TrodesPosV1  tables = [LFPV1, TrodesPosV1] restriction = {\"nwb_file_name\": \"minirec20230622_.nwb\"} share_data_to_kachery(     table_list=tables,     restriction=restriction,     zone_name=zone_name, ) <p>If all of that worked,</p> <ol> <li>Go to https://kachery-gateway.figurl.org/admin?zone=your_zone (changing your_zone to the name of your zone)</li> <li>Go to the Admin/Authorization Settings tab</li> <li>Add the GitHub login names and permissions for the users you want to share with.</li> </ol> <p>If those users can connect to your database, they should now be able to use the <code>.fetch_nwb()</code> method to download any <code>AnalysisNwbfiles</code> that have been shared through Kachery.</p> <p>For example:</p> <pre>from spyglass.spikesorting import CuratedSpikeSorting\n\ntest_sort = (\n    CuratedSpikeSorting &amp; {\"nwb_file_name\": \"minirec20230622_.nwb\"}\n).fetch()[0]\nsort = (CuratedSpikeSorting &amp; test_sort).fetch_nwb()\n</pre> <p>If you are a collaborator accessing datasets, you first need to be given access to the zone by a collaborator admin (see above).</p> <p>If you know the uri for the dataset you are accessing you can test this process below (example is for members of <code>franklab.collaborators</code>)</p> In\u00a0[\u00a0]: Copied! <pre>import kachery_cloud as kcl\n\npath = \"/path/to/save/file/to/test\"\nzone_name = \"franklab.collaborators\"\nuri = \"sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4\"\n\nos.environ[\"KACHERY_ZONE\"] = zone_name\nkcl.load_file(uri=uri, dest=path, verbose=True)\nassert os.path.exists(path), f\"File not downloaded to {path}\"\n</pre> import kachery_cloud as kcl  path = \"/path/to/save/file/to/test\" zone_name = \"franklab.collaborators\" uri = \"sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4\"  os.environ[\"KACHERY_ZONE\"] = zone_name kcl.load_file(uri=uri, dest=path, verbose=True) assert os.path.exists(path), f\"File not downloaded to {path}\" <p>In normal use, spyglass will manage setting the zone and uri when accessing files. In general, the easiest way to access data valueswill be through the <code>fetch1_dataframe()</code> function part of many of the spyglass tables. In brief this will check for the appropriate nwb analysis file in your local directory, and if not found, attempt to download it from the appropriate kachery zone. It will then parse the relevant information from that nwb file into a pandas dataframe.</p> <p>We will look at an example with data from the <code>LFPV1</code> table:</p> In\u00a0[9]: Copied! <pre>from spyglass.lfp.v1 import LFPV1\n\n# Here is the data we are going to access\nLFPV1 &amp; {\n    \"nwb_file_name\": \"Winnie20220713_.nwb\",\n    \"target_interval_list_name\": \"pos 0 valid times\",\n}\n</pre> from spyglass.lfp.v1 import LFPV1  # Here is the data we are going to access LFPV1 &amp; {     \"nwb_file_name\": \"Winnie20220713_.nwb\",     \"target_interval_list_name\": \"pos 0 valid times\", } Out[9]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ Winnie20220713_.nwb tetrode_sample_Winnie pos 0 valid times LFP 0-400 Hz 30000 Winnie20220713_C52XDICU6D.nwb lfp_tetrode_sample_Winnie_pos 0 valid times_valid times a89c590f-290b-4f9c-a568-b9ae67eee96d 1000.0 <p>Total: 1</p> <p>We can access the data using <code>fetch1_dataframe()</code></p> In\u00a0[10]: Copied! <pre>(\n    LFPV1\n    &amp; {\n        \"nwb_file_name\": \"Winnie20220713_.nwb\",\n        \"target_interval_list_name\": \"pos 0 valid times\",\n    }\n).fetch1_dataframe()\n</pre> (     LFPV1     &amp; {         \"nwb_file_name\": \"Winnie20220713_.nwb\",         \"target_interval_list_name\": \"pos 0 valid times\",     } ).fetch1_dataframe() Out[10]: 0 1 2 3 4 5 6 7 8 9 ... 18 19 20 21 22 23 24 25 26 27 time 1.657741e+09 -90 -65 -104 -89 -31 -68 -27 -26 -32 -92 ... -91 -99 -87 -117 -123 -85 -73 -74 -62 13 1.657741e+09 -202 -145 -227 -220 -57 -130 -84 -68 -30 -191 ... -168 -199 -176 -250 -238 -172 -158 -140 -127 54 1.657741e+09 -218 -150 -224 -216 -84 -154 -84 -93 -29 -206 ... -125 -153 -158 -219 -206 -137 -132 -129 -120 69 1.657741e+09 -226 -151 -240 -230 -97 -144 -71 -95 -38 -236 ... -105 -136 -149 -183 -210 -111 -83 -129 -92 116 1.657741e+09 -235 -154 -250 -231 -54 -91 -81 -89 -30 -247 ... -85 -107 -116 -140 -190 -68 -28 -114 -36 193 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1.657742e+09 -3 -27 -6 29 -227 -442 -1 67 25 -15 ... -83 -217 -61 -248 -196 -63 -111 -211 -52 166 1.657742e+09 44 19 44 82 -175 -407 13 95 62 38 ... 3 -112 32 -177 -123 22 -5 -147 54 285 1.657742e+09 94 63 92 129 -121 -341 61 132 88 88 ... 62 -28 104 -99 -53 82 61 -62 125 347 1.657742e+09 142 107 135 179 -106 -370 88 178 120 148 ... 113 48 199 -44 7 145 108 -13 213 453 1.657742e+09 108 84 95 130 -82 -281 52 134 73 105 ... 97 46 169 -16 22 118 94 -3 175 348 <p>901529 rows \u00d7 28 columns</p> <p>In the next notebook, we'll explore the details of a table tier unique to Spyglass, Merge Tables.</p>"}, {"location": "notebooks/02_Data_Sync/#sync-data", "title": "Sync Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#kachery", "title": "Kachery\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#cloud", "title": "Cloud\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#zone", "title": "Zone\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#host-setup", "title": "Host Setup\u00b6", "text": "<ul> <li><p>If you are a member of a team with a pre-existing database and zone who will be sharing data, please skip to <code>Sharing Data</code></p> </li> <li><p>If you are a collaborator outside your team's network and need to access files shared with you, please skip to <code>Accessing Shared Data</code></p> </li> </ul>"}, {"location": "notebooks/02_Data_Sync/#zones", "title": "Zones\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#resources", "title": "Resources\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#database-setup", "title": "Database Setup\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#sharing-data", "title": "Sharing Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#managing-access", "title": "Managing access\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#accessing-shared-data", "title": "Accessing Shared Data\u00b6", "text": ""}, {"location": "notebooks/02_Data_Sync/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/", "title": "Merge Tables", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>To insert data, see the Insert Data notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see these additional tutorials</li> <li>For information on why we use merge tables, and how to make one, see our documentation</li> </ul> <p>In short, merge tables represent the end processing point of a given way of processing the data in our pipelines. Merge Tables allow us to build new processing pipeline, or a new version of an existing pipeline, without having to drop or migrate the old tables. They allow data to be processed in different ways, but with a unified end result that downstream pipelines can all access.</p> <p>Let's start by importing the <code>spyglass</code> package, along with a few others.</p> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\nwarnings.simplefilter(\"ignore\", category=UserWarning)\n\nimport spyglass.common as sgc\nimport spyglass.lfp as lfp\nfrom spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nfrom spyglass.utils.dj_merge_tables import delete_downstream_merge, Merge\nfrom spyglass.common.common_ephys import LFP as CommonLFP  # Upstream 1\nfrom spyglass.lfp.lfp_merge import LFPOutput  # Merge Table\nfrom spyglass.lfp.v1.lfp import LFPV1  # Upstream 2\n</pre> import os import datajoint as dj  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) warnings.simplefilter(\"ignore\", category=UserWarning)  import spyglass.common as sgc import spyglass.lfp as lfp from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename from spyglass.utils.dj_merge_tables import delete_downstream_merge, Merge from spyglass.common.common_ephys import LFP as CommonLFP  # Upstream 1 from spyglass.lfp.lfp_merge import LFPOutput  # Merge Table from spyglass.lfp.v1.lfp import LFPV1  # Upstream 2 <pre>[2024-01-29 16:15:00,903][INFO]: Connecting root@localhost:3309\n[2024-01-29 16:15:00,912][INFO]: Connected root@localhost:3309\n</pre> <p>Check to make sure the data inserted in the previour notebook is still there.</p> In\u00a0[2]: Copied! <pre>nwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nnwb_file_dict = {\"nwb_file_name\": nwb_copy_file_name}\nsgc.Session &amp; nwb_file_dict\n</pre> nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) nwb_file_dict = {\"nwb_file_name\": nwb_copy_file_name} sgc.Session &amp; nwb_file_dict Out[2]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> minirec20230622_.nwb 54321 UCSF Loren Frank Lab 12345 test yaml insertion 2023-06-22 15:59:58 1970-01-01 00:00:00 Test Conversion <p>Total: 1</p> <p>If you haven't already done so, insert data into a Merge Table.</p> <p>Note: Some existing parents of Merge Tables perform the Merge Table insert as part of the populate methods. This practice will be revised in the future.</p> In\u00a0[3]: Copied! <pre>sgc.FirFilterParameters().create_standard_filters()\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test\",\n    electrode_list=[0],\n)\nlfp_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"lfp_electrode_group_name\": \"test\",\n    \"target_interval_list_name\": \"01_s1\",\n    \"filter_name\": \"LFP 0-400 Hz\",\n    \"filter_sampling_rate\": 30_000,\n}\nlfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True)\nlfp.v1.LFPV1().populate(lfp_key)\nLFPOutput.insert([lfp_key], skip_duplicates=True)\n</pre> sgc.FirFilterParameters().create_standard_filters() lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test\",     electrode_list=[0], ) lfp_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"lfp_electrode_group_name\": \"test\",     \"target_interval_list_name\": \"01_s1\",     \"filter_name\": \"LFP 0-400 Hz\",     \"filter_sampling_rate\": 30_000, } lfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True) lfp.v1.LFPV1().populate(lfp_key) LFPOutput.insert([lfp_key], skip_duplicates=True) <p>Merge Tables have multiple custom methods that begin with <code>merge</code>.</p> <p><code>help</code> can show us the docstring of each</p> In\u00a0[4]: Copied! <pre>merge_methods = [d for d in dir(Merge) if d.startswith(\"merge\")]\nprint(merge_methods)\n</pre> merge_methods = [d for d in dir(Merge) if d.startswith(\"merge\")] print(merge_methods) <pre>['merge_delete', 'merge_delete_parent', 'merge_fetch', 'merge_get_parent', 'merge_get_parent_class', 'merge_get_part', 'merge_html', 'merge_populate', 'merge_restrict', 'merge_restrict_class', 'merge_view']\n</pre> In\u00a0[5]: Copied! <pre>help(getattr(Merge, merge_methods[-1]))\n</pre> help(getattr(Merge, merge_methods[-1])) <pre>Help on method merge_view in module spyglass.utils.dj_merge_tables:\n\nmerge_view(restriction: str = True) method of datajoint.user_tables.TableMeta instance\n    Prints merged view, including null entries for unique columns.\n    \n    Note: To handle this Union as a table-like object, use `merge_resrict`\n    \n    Parameters\n    ---------\n    restriction: str, optional\n        Restriction to apply to the merged view\n\n</pre> <p><code>merge_view</code> shows a union of the master and all part tables.</p> <p>Note: Restrict Merge Tables with arguments, not the <code>&amp;</code> operator.</p> <ul> <li>Normally: <code>Table &amp; \"field='value'\"</code></li> <li>Instead: <code>MergeTable.merge_view(restriction=\"field='value'\"</code>).</li> </ul> In\u00a0[6]: Copied! <pre>LFPOutput.merge_view()\n</pre> LFPOutput.merge_view() <pre>*merge_id      *source    *nwb_file_name *lfp_electrode *target_interv *filter_name   *filter_sampli\n+------------+ +--------+ +------------+ +------------+ +------------+ +------------+ +------------+\nc34f98c5-7de7- LFPV1      minirec2023062 test           01_s1          LFP 0-400 Hz   30000         \n (Total: 1)\n\n</pre> <p>UUIDs help retain unique entries across all part tables. We can fetch NWB file by referencing this or other features.</p> In\u00a0[\u00a0]: Copied! <pre>uuid_key = (LFPOutput &amp; nwb_file_dict).fetch(limit=1, as_dict=True)[-1]\nrestrict = LFPOutput &amp; uuid_key\nrestrict\n</pre> uuid_key = (LFPOutput &amp; nwb_file_dict).fetch(limit=1, as_dict=True)[-1] restrict = LFPOutput &amp; uuid_key restrict Out[\u00a0]: <p>merge_id</p> <p>source</p> c34f98c5-7de7-1daf-6eaf-1e15981def44 LFPV1 <p>Total: 1</p> In\u00a0[8]: Copied! <pre>result1 = restrict.fetch_nwb(restrict.fetch1(\"KEY\"))\nresult1\n</pre> result1 = restrict.fetch_nwb(restrict.fetch1(\"KEY\")) result1 Out[8]: <pre>[{'nwb_file_name': 'minirec20230622_.nwb',\n  'lfp_electrode_group_name': 'test',\n  'target_interval_list_name': '01_s1',\n  'filter_name': 'LFP 0-400 Hz',\n  'filter_sampling_rate': 30000,\n  'analysis_file_name': 'minirec20230622_R5DWQ6S53S.nwb',\n  'interval_list_name': 'lfp_test_01_s1_valid times',\n  'lfp_object_id': 'ffb893d1-a31e-41d3-aec7-8dc8936c8898',\n  'lfp_sampling_rate': 1000.0,\n  'lfp': filtered data pynwb.ecephys.ElectricalSeries at 0x129602752674544\n  Fields:\n    comments: no comments\n    conversion: 1.0\n    data: &lt;HDF5 dataset \"data\": shape (10476, 1), type \"&lt;i2\"&gt;\n    description: filtered data\n    electrodes: electrodes &lt;class 'hdmf.common.table.DynamicTableRegion'&gt;\n    interval: 1\n    offset: 0.0\n    resolution: -1.0\n    timestamps: &lt;HDF5 dataset \"timestamps\": shape (10476,), type \"&lt;f8\"&gt;\n    timestamps_unit: seconds\n    unit: volts}]</pre> In\u00a0[9]: Copied! <pre>nwb_key = LFPOutput.merge_restrict(nwb_file_dict).fetch(as_dict=True)[0]\nnwb_key\n</pre> nwb_key = LFPOutput.merge_restrict(nwb_file_dict).fetch(as_dict=True)[0] nwb_key Out[9]: <pre>{'merge_id': UUID('c34f98c5-7de7-1daf-6eaf-1e15981def44'),\n 'source': 'LFPV1',\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'lfp_electrode_group_name': 'test',\n 'target_interval_list_name': '01_s1',\n 'filter_name': 'LFP 0-400 Hz',\n 'filter_sampling_rate': 30000}</pre> In\u00a0[10]: Copied! <pre>result2 = LFPOutput().fetch_nwb(nwb_key)\nresult2 == result1\n</pre> result2 = LFPOutput().fetch_nwb(nwb_key) result2 == result1 Out[10]: <pre>True</pre> <p>There are also functions for retrieving part/parent table(s) and fetching data.</p> <p>These <code>get</code> functions will either return the part table of the Merge table or the parent table with the source information for that part.</p> In\u00a0[11]: Copied! <pre>result4 = LFPOutput.merge_get_part(restriction=nwb_file_dict, join_master=True)\nresult4\n</pre> result4 = LFPOutput.merge_get_part(restriction=nwb_file_dict, join_master=True) result4 Out[11]: <p>merge_id</p> <p>source</p> <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter c34f98c5-7de7-1daf-6eaf-1e15981def44 LFPV1 minirec20230622_.nwb test 01_s1 LFP 0-400 Hz 30000 <p>Total: 1</p> In\u00a0[12]: Copied! <pre>result5 = LFPOutput.merge_get_parent(restriction='nwb_file_name LIKE \"mini%\"')\nresult5\n</pre> result5 = LFPOutput.merge_get_parent(restriction='nwb_file_name LIKE \"mini%\"') result5 Out[12]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ minirec20230622_.nwb test 01_s1 LFP 0-400 Hz 30000 minirec20230622_R5DWQ6S53S.nwb lfp_test_01_s1_valid times ffb893d1-a31e-41d3-aec7-8dc8936c8898 1000.0 <p>Total: 1</p> <p><code>fetch</code> will collect all relevant entries and return them as a list in the format specified by keyword arguments and one's DataJoint config.</p> In\u00a0[13]: Copied! <pre>result6 = result5.fetch(\"lfp_sampling_rate\")  # Sample rate for all mini* files\nresult6\n</pre> result6 = result5.fetch(\"lfp_sampling_rate\")  # Sample rate for all mini* files result6 Out[13]: <pre>array([1000.])</pre> <p><code>merge_fetch</code> requires a restriction as the first argument. For no restriction, use <code>True</code>.</p> In\u00a0[14]: Copied! <pre>result7 = LFPOutput.merge_fetch(True, \"filter_name\", \"nwb_file_name\")\nresult7\n</pre> result7 = LFPOutput.merge_fetch(True, \"filter_name\", \"nwb_file_name\") result7 Out[14]: <pre>[array(['LFP 0-400 Hz'], dtype=object),\n array(['minirec20230622_.nwb'], dtype=object)]</pre> In\u00a0[15]: Copied! <pre>result8 = LFPOutput.merge_fetch(as_dict=True)\nresult8\n</pre> result8 = LFPOutput.merge_fetch(as_dict=True) result8 Out[15]: <pre>{'merge_id': UUID('c34f98c5-7de7-1daf-6eaf-1e15981def44'),\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'lfp_electrode_group_name': 'test',\n 'target_interval_list_name': '01_s1',\n 'filter_name': 'LFP 0-400 Hz',\n 'filter_sampling_rate': 30000}</pre> <p>When deleting from Merge Tables, we can either...</p> <ol> <li><p>delete from the Merge Table itself with <code>merge_delete</code>, deleting both the master and part.</p> </li> <li><p>use <code>merge_delete_parent</code> to delete from the parent sources, getting rid of the entries in the source table they came from.</p> </li> <li><p>use <code>delete_downstream_merge</code> to find Merge Tables downstream of any other table and get rid full entries, avoiding orphaned master table entries.</p> </li> </ol> <p>The two latter cases can be destructive, so we include an extra layer of protection with <code>dry_run</code>. When true (by default), these functions return a list of tables with the entries that would otherwise be deleted.</p> In\u00a0[16]: Copied! <pre>LFPOutput.merge_delete(nwb_file_dict)  # Delete from merge table\n</pre> LFPOutput.merge_delete(nwb_file_dict)  # Delete from merge table <pre>[2024-01-29 16:15:23,054][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output__l_f_p_v1`\n[2024-01-29 16:15:23,058][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output`\n</pre> <pre>[2024-01-29 16:15:24,953][WARNING]: Deletes cancelled\n</pre> In\u00a0[17]: Copied! <pre>LFPOutput.merge_delete_parent(restriction=nwb_file_dict, dry_run=True)\n</pre> LFPOutput.merge_delete_parent(restriction=nwb_file_dict, dry_run=True) Out[17]: <pre>[FreeTable(`lfp_v1`.`__l_f_p_v1`)\n *nwb_file_name *lfp_electrode *target_interv *filter_name   *filter_sampli analysis_file_ interval_list_ lfp_object_id  lfp_sampling_r\n +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\n minirec2023062 test           01_s1          LFP 0-400 Hz   30000          minirec2023062 lfp_test_01_s1 ffb893d1-a31e- 1000.0        \n  (Total: 1)]</pre> <p><code>delete_downstream_merge</code> is available from any other table in the pipeline, but it does take some time to find the links downstream. If you're using this, you can save time by reassigning your table to a variable, which will preserve a copy of the previous search.</p> <p>Because the copy is stored, this function may not see additional merge tables you've imported. To refresh this copy, set <code>reload_cache=True</code></p> In\u00a0[18]: Copied! <pre>nwbfile = sgc.Nwbfile()\n\n(nwbfile &amp; nwb_file_dict).delete_downstream_merge(\n    dry_run=True,\n    reload_cache=False,  # if still encountering errors, try setting this to True\n)\n</pre> nwbfile = sgc.Nwbfile()  (nwbfile &amp; nwb_file_dict).delete_downstream_merge(     dry_run=True,     reload_cache=False,  # if still encountering errors, try setting this to True ) <pre>[16:15:37][INFO] Spyglass: Building merge cache for nwbfile.\n\tFound 3 downstream merge tables\n</pre> Out[18]: <pre>dict_values([[*nwb_file_name *analysis_file *lfp_electrode *target_interv *filter_name   *filter_sampli *merge_id      nwb_file_a analysis_f analysis_file_ analysis_p interval_list_ lfp_object_id  lfp_sampling_r\n+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +--------+ +--------+ +------------+ +--------+ +------------+ +------------+ +------------+\nminirec2023062 minirec2023062 test           01_s1          LFP 0-400 Hz   30000          c34f98c5-7de7- =BLOB=     =BLOB=                    =BLOB=     lfp_test_01_s1 ffb893d1-a31e- 1000.0        \n (Total: 1)\n]])</pre> <p>This function is run automatically whin you use <code>cautious_delete</code>, which checks team permissions before deleting.</p> In\u00a0[\u00a0]: Copied! <pre>(nwbfile &amp; nwb_file_dict).cautious_delete()\n</pre> (nwbfile &amp; nwb_file_dict).cautious_delete() <p>In the next notebook, we'll start working with ephys data with spike sorting.</p>"}, {"location": "notebooks/03_Merge_Tables/#merge-tables", "title": "Merge Tables\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#intro", "title": "Intro\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#example-data", "title": "Example data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#helper-functions", "title": "Helper functions\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#showing-data", "title": "Showing data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#selecting-data", "title": "Selecting data\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#deletion-from-merge-tables", "title": "Deletion from Merge Tables\u00b6", "text": ""}, {"location": "notebooks/03_Merge_Tables/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/04_PopulateConfigFile/", "title": "Config Populate", "text": "In\u00a0[\u00a0]: Copied! <pre>nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys_rly.nwb\"\n</pre> nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys_rly.nwb\" In\u00a0[\u00a0]: Copied! <pre># this configuration yaml file should be placed next to the downloaded NWB file\nyaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_rly_spyglass_config.yaml\"\nwith open(yaml_config_path, \"w\") as config_file:\n    lines = [\n        \"DataAcquisitionDevice\",\n        \"- data_acquisition_device_name: data_acq_device0\",\n    ]\n    config_file.writelines(line + \"\\n\" for line in lines)\n</pre> # this configuration yaml file should be placed next to the downloaded NWB file yaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_rly_spyglass_config.yaml\" with open(yaml_config_path, \"w\") as config_file:     lines = [         \"DataAcquisitionDevice\",         \"- data_acquisition_device_name: data_acq_device0\",     ]     config_file.writelines(line + \"\\n\" for line in lines) <p>Then call <code>insert_sessions</code> as usual.</p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.data_import as sgi\n\nsgi.insert_sessions(nwb_file_name)\n</pre> import spyglass.data_import as sgi  sgi.insert_sessions(nwb_file_name) <p>Confirm the session was inserted with the correct <code>DataAcquisitionDevice</code></p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.common as sgc\nfrom spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n\nsgc.Session.DataAcquisitionDevice &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> import spyglass.common as sgc from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)  sgc.Session.DataAcquisitionDevice &amp; {\"nwb_file_name\": nwb_copy_file_name}"}, {"location": "notebooks/04_PopulateConfigFile/#customizing-data-insertion-into-spyglass", "title": "Customizing Data Insertion into Spyglass\u00b6", "text": "<p>If you would like to insert data into Spyglass that does not follow the naming or organizational format expected by Spyglass,  or you would like to override what values are ingested into Spyglass from  your NWB files, including missing values, follow this guide.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#general-approach", "title": "General Approach\u00b6", "text": "<p>When an NWB file is ingested into Spyglass, metadata about the session is first read from the NWB file and inserted into tables in the <code>common</code> module (e.g. <code>Institution</code>, <code>Lab</code>, <code>Electrode</code>, etc). However, not every NWB file has all the information required by Spyglass or the information in the NWB file is not in a format that Spyglass expects. For example, many NWB files do not contain information about the <code>DataAcquisitionDevice</code> or <code>Probe</code> because the NWB data standard does not yet have an official standard for specifying them. For these cases, we provide a way to customize how data is ingested into Spyglass.</p> <p>Let's say that you want to ingest an NWB file into Spyglass where the lab name in the file is written as \"Loren Frank Lab\" or it is not specified, but you  know the data comes from the Loren Frank Lab. Let's say that in Spyglass, the lab name that is associated with sessions from the Loren Frank Lab is  \"Frank Lab\" and you would like to use the same name in order to facilitate data search in Spyglass. To change the lab name when you insert your new data  to Spyglass, you could either 1) edit the NWB file directly and then  insert it into Spyglass, or 2) define an override value \"Frank Lab\" to be  used instead of the value specified in the NWB file (or lack thereof).</p> <p>Note that if this is your own data and you want to make changes to information about how the data is interpreted, e.g., the units of measurement are incorrect, we recommend that you edit the NWB file directly because the  file or derivatives of it might eventually be shared outside of Spyglass and  they will not reflect any modifications that you have made to  the data only in Spyglass.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#define-a-configuration-yaml-file", "title": "Define a Configuration YAML File\u00b6", "text": "<p>To override values in the NWB file during insertion into Spyglass,  you will need to create a configuration  YAML file that lives in the same directory as your NWB file, named:  <code>&lt;name_of_nwb_file&gt;_spyglass_config.yaml</code></p> <p>An example configuration YAML file can be found at <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code>. This file is associated with the NWB file <code>sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb</code>.</p> <p>This is the general format for entries in this configuration file:</p> <pre>TableName:\n- primary_key1: value1\n</pre> <p>For example:</p> <pre>Lab:\n- lab_name: Frank Lab\n</pre> <p>In this example, the NWB file that corresponds to this config YAML will become associated with the entry in the <code>Lab</code> table with the value <code>Frank Lab</code> for  the primary key <code>lab_name</code>. This entry must already exist. More specifically, when the NWB file is ingested into Spyglass,  a new <code>Session</code> entry will be created for the NWB file that has a foreign key to the <code>Lab</code> entry with <code>lab_name</code> = \"Frank Lab\", ignoring whatever lab value is  in the NWB file, even if one does not exist.</p> <p>TODO implement this for <code>Lab</code>.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#create-entries-to-reference-in-the-configuration-yaml", "title": "Create Entries to Reference in the Configuration YAML\u00b6", "text": "<p>As mentioned earlier, the table entry that you want to associate with your NWB file must already exist in the database. This entry would typically be a value that is independent of any particular NWB file, such as <code>DataAcquisitionDevice</code>, <code>Lab</code>, <code>Probe</code>, and <code>BrainRegion</code>.</p> <p>If the entry does not already exist, you can either:</p> <ol> <li><p>create the entry programmatically using DataJoint <code>insert</code> commands, or</p> </li> <li><p>define the entry in a YAML file called <code>entries.yaml</code> that is automatically processed when Spyglass is imported. You can think of <code>entries.yaml</code> as a place to define information that the database should come pre-equipped prior to ingesting your NWB files. The <code>entries.yaml</code> file should be placed in the <code>spyglass</code> base directory (next to <code>README.md</code>). An example can be found in <code>examples/config_yaml/entries.yaml</code>. This file should have the following structure:</p> <pre>TableName:\n- TableEntry1Field1: Value\n  TableEntry1Field2: Value\n- TableEntry2Field1: Value\n  TableEntry2Field2: Value\n</pre> <p>For example,</p> <pre>DataAcquisitionDeviceSystem:\n  data_acquisition_device_system: SpikeGLX\nDataAcquisitionDevice:\n- data_acquisition_device_name: Neuropixels_SpikeGLX\n  data_acquisition_device_system: SpikeGLX\n  data_acquisition_device_amplifier: Intan\n</pre> <p>Only <code>dj.Manual</code>, <code>dj.Lookup</code>, and <code>dj.Part</code> tables can be populated  using this approach.</p> </li> </ol> <p>Once the entry that you want to associate with your NWB file exists in the database, you can write the configuration YAML file and then ingest your NWB file. As an another example, let's say that you want to associate your NWB file with the <code>DataAcquisitionDevice</code> entry with <code>data_acquisition_device_name</code> = \"Neuropixels_SpikeGLX\" that was defined above. You would write the following configuration YAML file:</p> <pre>DataAcquisitionDevice:\n- data_acquisition_device_name: Neuropixels_SpikeGLX\n</pre> <p>The example in <code>examples/config_yaml/\u200b\u200bsub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml</code> includes additional examples.</p>"}, {"location": "notebooks/04_PopulateConfigFile/#example-ingestion-with-real-data", "title": "Example Ingestion with Real Data\u00b6", "text": "<p>For this example, you will need to download the 5 GB NWB file  <code>sub-JDS-NFN-AM2_behavior+ecephys.nwb</code> from dandiset 000447 here:  https://dandiarchive.org/dandiset/000447/0.230316.2133/files?location=sub-JDS-NFN-AM2&amp;page=1</p> <p>Click the download arrow button to download the file to your computer. Add it to the folder containing your raw NWB data to be ingested into Spyglass.</p> <p>This file does not specify a data acquisition device. Let's say that the data was collected from a SpikeGadgets system with an Intan amplifier. This matches an existing entry in the <code>DataAcquisitionDevice</code> table with name \"data_acq_device0\". We will create a configuration YAML file to associate this entry with the NWB file.</p> <p>If you are connected to the Frank lab database, please rename any downloaded files (e.g., <code>example20200101_yourname.nwb</code>) to avoid naming collisions, as the file name acts as the primary key across key tables.</p>"}, {"location": "notebooks/10_Spike_SortingV0/", "title": "Spike Sorting V0", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> Extract the recording <ol> <li>Specifying your NWB file.</li> <li>Specifying which electrodes involved in the recording to sort data from. - <code>SortGroup</code></li> <li>Specifying the time segment of the recording we want to sort. - <code>IntervalList</code>, <code>SortInterval</code></li> <li>Specifying the parameters to use for filtering the recording. - <code>SpikeSortingPreprocessingParameters</code></li> <li>Combining these parameters. - <code>SpikeSortingRecordingSelection</code></li> <li>Extracting the recording. - <code>SpikeSortingRecording</code></li> <li>Specifying the parameters to apply for artifact detection/removal. -<code>ArtifactDetectionParameters</code></li> </ol> Spike sorting the recording <ol> <li>Specify the spike sorter and parameters to use. - <code>SpikeSorterParameters</code></li> <li>Combine these parameters. - <code>SpikeSortingSelection</code></li> <li>Spike sort the extracted recording according to chose parameter set. - <code>SpikeSorting</code></li> </ol> <p> </p> In\u00a0[10]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.spikesorting.v0 as sgs\nfrom spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.spikesorting.v0 as sgs from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <p>If you haven't already done so, add yourself to <code>LabTeam</code></p> In\u00a0[6]: Copied! <pre># Full name, Google email address, DataJoint username, admin\nname, email, dj_user, admin = (\n    \"Firstname Lastname\",\n    \"example@gmail.com\",\n    \"user\",\n    0,\n)\nsgc.LabMember.insert_from_name(name)\nsgc.LabMember.LabMemberInfo.insert1(\n    [\n        name,\n        email,\n        dj_user,\n        admin,\n    ],\n    skip_duplicates=True,\n)\nsgc.LabMember.LabMemberInfo() &amp; {\n    \"team_name\": \"My Team\",\n    \"lab_member_name\": \"Firstname Lastname\",\n}\n</pre> # Full name, Google email address, DataJoint username, admin name, email, dj_user, admin = (     \"Firstname Lastname\",     \"example@gmail.com\",     \"user\",     0, ) sgc.LabMember.insert_from_name(name) sgc.LabMember.LabMemberInfo.insert1(     [         name,         email,         dj_user,         admin,     ],     skip_duplicates=True, ) sgc.LabMember.LabMemberInfo() &amp; {     \"team_name\": \"My Team\",     \"lab_member_name\": \"Firstname Lastname\", } Out[6]: Information about lab member in the context of Frank lab network <p>lab_member_name</p> <p>google_user_name</p> For permission to curate <p>datajoint_user_name</p> For permission to delete <p>admin</p> Ignore permission checks FirstName LastName gmail@gmail.com user 0 <p>Total: 1</p> <p>We can try <code>fetch</code> to confirm.</p> <p>Exercise: Try to write a fer lines to generate a dictionary with team names as keys and lists of members as values. It may be helpful to add more data with the code above and use <code>fetch(as_dict=True)</code>.</p> In\u00a0[7]: Copied! <pre>my_team_members = (\n    (sgc.LabTeam.LabTeamMember &amp; {\"team_name\": \"My Team\"})\n    .fetch(\"lab_member_name\")\n    .tolist()\n)\nif name in my_team_members:\n    print(\"You made it in!\")\n</pre> my_team_members = (     (sgc.LabTeam.LabTeamMember &amp; {\"team_name\": \"My Team\"})     .fetch(\"lab_member_name\")     .tolist() ) if name in my_team_members:     print(\"You made it in!\") <pre>You made it in!\n</pre> Code hidden here <pre>members = sgc.LabTeam.LabTeamMember.fetch(as_dict=True)\nteams_dict = {member[\"team_name\"]: [] for member in members}\nfor member in members:\n    teams_dict[member[\"team_name\"]].append(member[\"lab_member_name\"])\nprint(teams_dict)\n</pre> <p>If you haven't already, load an NWB file. For more details on downloading and importing data, see this notebook.</p> In\u00a0[8]: Copied! <pre>import spyglass.data_import as sdi\n\nsdi.insert_sessions(\"minirec20230622.nwb\")\nnwb_file_name = \"minirec20230622_.nwb\"\n</pre> import spyglass.data_import as sdi  sdi.insert_sessions(\"minirec20230622.nwb\") nwb_file_name = \"minirec20230622_.nwb\" <pre>/home/sambray/Documents/spyglass/src/spyglass/data_import/insert_sessions.py:58: UserWarning: Cannot insert data from minirec20230622.nwb: minirec20230622_.nwb is already in Nwbfile table.\n  warnings.warn(\n</pre> <p>Each NWB file will have multiple electrodes we can use for spike sorting. We commonly use multiple electrodes in a <code>SortGroup</code> selected by what tetrode or shank of a probe they were on.</p> <p>Note: This will delete any existing entries. Answer 'yes' when prompted, or skip running this cell to leave data in place.</p> In\u00a0[12]: Copied! <pre>sgs.SortGroup().set_group_by_shank(nwb_file_name)\n</pre> sgs.SortGroup().set_group_by_shank(nwb_file_name) <p>Each electrode has an <code>electrode_id</code> and is associated with an <code>electrode_group_name</code>, which corresponds with a <code>sort_group_id</code>.</p> <p>For example, data recorded from a 32 tetrode (128 channel) drive results in 128 unique <code>electrode_id</code>. This could result in 32 unique <code>electrode_group_name</code> and 32 unique <code>sort_group_id</code>.</p> In\u00a0[13]: Copied! <pre>sgs.SortGroup.SortGroupElectrode &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgs.SortGroup.SortGroupElectrode &amp; {\"nwb_file_name\": nwb_file_name} Out[13]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode minirec20230622_.nwb 0 0 0minirec20230622_.nwb 0 0 1minirec20230622_.nwb 0 0 2minirec20230622_.nwb 0 0 3minirec20230622_.nwb 1 1 4minirec20230622_.nwb 1 1 5minirec20230622_.nwb 1 1 6minirec20230622_.nwb 1 1 7minirec20230622_.nwb 10 10 40minirec20230622_.nwb 10 10 41minirec20230622_.nwb 10 10 42minirec20230622_.nwb 10 10 43 <p>...</p> <p>Total: 128</p> In\u00a0[14]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name} Out[14]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1 =BLOB= minirec20230622_.nwb 01_s1_first9 =BLOB= minirec20230622_.nwb 01_s1_first9 lfp band 100Hz =BLOB= lfp bandminirec20230622_.nwb 02_s2 =BLOB= minirec20230622_.nwb lfp_test_01_s1_first9_valid times =BLOB= lfp_v1minirec20230622_.nwb lfp_test_01_s1_valid times =BLOB= lfp_v1minirec20230622_.nwb minirec20230622_.nwb_01_s1_first9_0_default_hippocampus =BLOB= spikesorting_recording_v0minirec20230622_.nwb minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times =BLOB= spikesorting_artifact_v0minirec20230622_.nwb pos 0 valid times =BLOB= minirec20230622_.nwb pos 1 valid times =BLOB= minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 11</p> <p>Let's start with the first run interval (<code>01_s1</code>) and fetch corresponding <code>valid_times</code>. For the <code>minirec</code> example, this is relatively short.</p> In\u00a0[15]: Copied! <pre>interval_list_name = \"01_s1\"\ninterval_list = (\n    sgc.IntervalList\n    &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": interval_list_name}\n).fetch1(\"valid_times\")[0]\n\n\ndef print_interval_duration(interval_list: np.ndarray):\n    duration = np.round((interval_list[1] - interval_list[0]))\n    print(f\"This interval list is {duration:g} seconds long\")\n\n\nprint_interval_duration(interval_list)\n</pre> interval_list_name = \"01_s1\" interval_list = (     sgc.IntervalList     &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": interval_list_name} ).fetch1(\"valid_times\")[0]   def print_interval_duration(interval_list: np.ndarray):     duration = np.round((interval_list[1] - interval_list[0]))     print(f\"This interval list is {duration:g} seconds long\")   print_interval_duration(interval_list) <pre>This interval list is 10 seconds long\n</pre> In\u00a0[16]: Copied! <pre>n = 9\nsort_interval_name = interval_list_name + f\"_first{n}\"\nsort_interval = np.array([interval_list[0], interval_list[0] + n])\n</pre> n = 9 sort_interval_name = interval_list_name + f\"_first{n}\" sort_interval = np.array([interval_list[0], interval_list[0] + n]) <p>With the above, we can insert into <code>SortInterval</code></p> In\u00a0[17]: Copied! <pre>sgs.SortInterval.insert1(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"sort_interval_name\": sort_interval_name,\n        \"sort_interval\": sort_interval,\n    },\n    skip_duplicates=True,\n)\n</pre> sgs.SortInterval.insert1(     {         \"nwb_file_name\": nwb_file_name,         \"sort_interval_name\": sort_interval_name,         \"sort_interval\": sort_interval,     },     skip_duplicates=True, ) <p>And verify the entry</p> In\u00a0[18]: Copied! <pre>print_interval_duration(\n    (\n        sgs.SortInterval\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"sort_interval_name\": sort_interval_name,\n        }\n    ).fetch1(\"sort_interval\")\n)\n</pre> print_interval_duration(     (         sgs.SortInterval         &amp; {             \"nwb_file_name\": nwb_file_name,             \"sort_interval_name\": sort_interval_name,         }     ).fetch1(\"sort_interval\") ) <pre>This interval list is 9 seconds long\n</pre> <p><code>SpikeSortingPreprocessingParameters</code> contains the parameters used to filter the recorded data in the spike band prior to sorting.</p> In\u00a0[19]: Copied! <pre>sgs.SpikeSortingPreprocessingParameters()\n</pre> sgs.SpikeSortingPreprocessingParameters() Out[19]: <p>preproc_params_name</p> <p>preproc_params</p> default =BLOB=default_hippocampus =BLOB=default_min_seg =BLOB=franklab_default_hippocampus =BLOB=franklab_default_hippocampus_min_segment_length =BLOB=franklab_tetrode_hippocampus =BLOB=franklab_tetrode_hippocampus_min_seg =BLOB=lf_test =BLOB= <p>Total: 8</p> <p>Here, we insert the default parameters and then fetch them.</p> In\u00a0[20]: Copied! <pre>sgs.SpikeSortingPreprocessingParameters().insert_default()\npreproc_params = (\n    sgs.SpikeSortingPreprocessingParameters()\n    &amp; {\"preproc_params_name\": \"default\"}\n).fetch1(\"preproc_params\")\nprint(preproc_params)\n</pre> sgs.SpikeSortingPreprocessingParameters().insert_default() preproc_params = (     sgs.SpikeSortingPreprocessingParameters()     &amp; {\"preproc_params_name\": \"default\"} ).fetch1(\"preproc_params\") print(preproc_params) <pre>{'frequency_min': 300, 'frequency_max': 6000, 'margin_ms': 5, 'seed': 0}\n</pre> <p>Let's adjust the <code>frequency_min</code> to 600, the preference for hippocampal data, and insert that into the table as a new set of parameters for hippocampal data.</p> In\u00a0[21]: Copied! <pre>preproc_params[\"frequency_min\"] = 600\nsgs.SpikeSortingPreprocessingParameters().insert1(\n    {\n        \"preproc_params_name\": \"default_hippocampus\",\n        \"preproc_params\": preproc_params,\n    },\n    skip_duplicates=True,\n)\n</pre> preproc_params[\"frequency_min\"] = 600 sgs.SpikeSortingPreprocessingParameters().insert1(     {         \"preproc_params_name\": \"default_hippocampus\",         \"preproc_params\": preproc_params,     },     skip_duplicates=True, ) In\u00a0[22]: Copied! <pre>interval_list_name\n</pre> interval_list_name Out[22]: <pre>'01_s1'</pre> In\u00a0[23]: Copied! <pre>ssr_key = dict(\n    nwb_file_name=nwb_file_name,\n    sort_group_id=0,  # See SortGroup\n    sort_interval_name=sort_interval_name,  # First N seconds above\n    preproc_params_name=\"default_hippocampus\",  # See preproc_params\n    interval_list_name=interval_list_name,\n    team_name=\"My Team\",\n)\n</pre> ssr_key = dict(     nwb_file_name=nwb_file_name,     sort_group_id=0,  # See SortGroup     sort_interval_name=sort_interval_name,  # First N seconds above     preproc_params_name=\"default_hippocampus\",  # See preproc_params     interval_list_name=interval_list_name,     team_name=\"My Team\", ) In\u00a0[24]: Copied! <pre>sgs.SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True)\nsgs.SpikeSortingRecordingSelection() &amp; ssr_key\n</pre> sgs.SpikeSortingRecordingSelection.insert1(ssr_key, skip_duplicates=True) sgs.SpikeSortingRecordingSelection() &amp; ssr_key Out[24]: Defines recordings to be sorted <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team 01_s1 <p>Total: 1</p> In\u00a0[25]: Copied! <pre>ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; ssr_key).proj()\nsgs.SpikeSortingRecording.populate([ssr_pk])\n</pre> ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; ssr_key).proj() sgs.SpikeSortingRecording.populate([ssr_pk]) <p>Now we can see our recording in the table. E x c i t i n g !</p> In\u00a0[26]: Copied! <pre>sgs.SpikeSortingRecording() &amp; ssr_key\n</pre> sgs.SpikeSortingRecording() &amp; ssr_key Out[26]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>recording_path</p> <p>sort_interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team /stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus minirec20230622_.nwb_01_s1_first9_0_default_hippocampus <p>Total: 1</p> In\u00a0[27]: Copied! <pre>sgs.ArtifactDetectionParameters().insert_default()\nartifact_key = (sgs.SpikeSortingRecording() &amp; ssr_key).fetch1(\"KEY\")\nartifact_key[\"artifact_params_name\"] = \"none\"\n</pre> sgs.ArtifactDetectionParameters().insert_default() artifact_key = (sgs.SpikeSortingRecording() &amp; ssr_key).fetch1(\"KEY\") artifact_key[\"artifact_params_name\"] = \"none\" <p>We then pair artifact detection parameters in <code>ArtifactParameters</code> with a recording extracted through population of <code>SpikeSortingRecording</code> and insert into <code>ArtifactDetectionSelection</code>.</p> In\u00a0[29]: Copied! <pre>sgs.ArtifactDetectionSelection().insert1(artifact_key, skip_duplicates=True)\nsgs.ArtifactDetectionSelection() &amp; artifact_key\n</pre> sgs.ArtifactDetectionSelection().insert1(artifact_key, skip_duplicates=True) sgs.ArtifactDetectionSelection() &amp; artifact_key Out[29]: Specifies artifact detection parameters to apply to a sort group's recording. <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>artifact_params_name</p> <p>custom_artifact_detection</p> minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team none 0 <p>Total: 1</p> <p>Then, we can populate <code>ArtifactDetection</code>, which will find periods where there are artifacts, as specified by the parameters.</p> In\u00a0[30]: Copied! <pre>sgs.ArtifactDetection.populate(artifact_key)\n</pre> sgs.ArtifactDetection.populate(artifact_key) <p>Populating <code>ArtifactDetection</code> also inserts an entry into <code>ArtifactRemovedIntervalList</code>, which stores the interval without detected artifacts.</p> In\u00a0[31]: Copied! <pre>sgs.ArtifactRemovedIntervalList() &amp; artifact_key\n</pre> sgs.ArtifactRemovedIntervalList() &amp; artifact_key Out[31]: Stores intervals without detected artifacts. <p>artifact_removed_interval_list_name</p> <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>artifact_params_name</p> <p>artifact_removed_valid_times</p> <p>artifact_times</p> np array of artifact intervals minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team none =BLOB= =BLOB= <p>Total: 1</p> In\u00a0[32]: Copied! <pre>sgs.SpikeSorterParameters().insert_default()\n\n# Let's look at the default params\nsorter_name = \"mountainsort4\"\nms4_default_params = (\n    sgs.SpikeSorterParameters\n    &amp; {\"sorter\": sorter_name, \"sorter_params_name\": \"default\"}\n).fetch1()\nprint(ms4_default_params)\n</pre> sgs.SpikeSorterParameters().insert_default()  # Let's look at the default params sorter_name = \"mountainsort4\" ms4_default_params = (     sgs.SpikeSorterParameters     &amp; {\"sorter\": sorter_name, \"sorter_params_name\": \"default\"} ).fetch1() print(ms4_default_params) <pre>{'sorter': 'mountainsort4', 'sorter_params_name': 'default', 'sorter_params': {'detect_sign': -1, 'adjacency_radius': -1, 'freq_min': 300, 'freq_max': 6000, 'filter': True, 'whiten': True, 'num_workers': 1, 'clip_size': 50, 'detect_threshold': 3, 'detect_interval': 10}}\n</pre> <p>Now we can change these default parameters to line up more closely with our preferences.</p> In\u00a0[33]: Copied! <pre>sorter_params = {\n    **ms4_default_params[\"sorter_params\"],  # start with defaults\n    \"detect_sign\": -1,  # downward going spikes (1 for upward, 0 for both)\n    \"adjacency_radius\": 100,  # Sort electrodes together within 100 microns\n    \"filter\": False,  # No filter, since we filter prior to starting sort\n    \"freq_min\": 0,\n    \"freq_max\": 0,\n    \"whiten\": False,  # Turn whiten, since we whiten it prior to starting sort\n    \"num_workers\": 4,  #  same number as number of electrodes\n    \"verbose\": True,\n    \"clip_size\": np.int64(\n        1.33e-3  # same as # of samples for 1.33 ms based on the sampling rate\n        * (sgc.Raw &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\"sampling_rate\")\n    ),\n}\nfrom pprint import pprint\n\npprint(sorter_params)\n</pre> sorter_params = {     **ms4_default_params[\"sorter_params\"],  # start with defaults     \"detect_sign\": -1,  # downward going spikes (1 for upward, 0 for both)     \"adjacency_radius\": 100,  # Sort electrodes together within 100 microns     \"filter\": False,  # No filter, since we filter prior to starting sort     \"freq_min\": 0,     \"freq_max\": 0,     \"whiten\": False,  # Turn whiten, since we whiten it prior to starting sort     \"num_workers\": 4,  #  same number as number of electrodes     \"verbose\": True,     \"clip_size\": np.int64(         1.33e-3  # same as # of samples for 1.33 ms based on the sampling rate         * (sgc.Raw &amp; {\"nwb_file_name\": nwb_file_name}).fetch1(\"sampling_rate\")     ), } from pprint import pprint  pprint(sorter_params) <pre>{'adjacency_radius': 100,\n 'clip_size': 39,\n 'detect_interval': 10,\n 'detect_sign': -1,\n 'detect_threshold': 3,\n 'filter': False,\n 'freq_max': 0,\n 'freq_min': 0,\n 'num_workers': 4,\n 'verbose': True,\n 'whiten': False}\n</pre> <p>We can give these <code>sorter_params</code> a <code>sorter_params_name</code> and insert into <code>SpikeSorterParameters</code>.</p> In\u00a0[34]: Copied! <pre>sorter_params_name = \"hippocampus_tutorial\"\nsgs.SpikeSorterParameters.insert1(\n    {\n        \"sorter\": sorter_name,\n        \"sorter_params_name\": sorter_params_name,\n        \"sorter_params\": sorter_params,\n    },\n    skip_duplicates=True,\n)\n(\n    sgs.SpikeSorterParameters\n    &amp; {\"sorter\": sorter_name, \"sorter_params_name\": sorter_params_name}\n).fetch1()\n</pre> sorter_params_name = \"hippocampus_tutorial\" sgs.SpikeSorterParameters.insert1(     {         \"sorter\": sorter_name,         \"sorter_params_name\": sorter_params_name,         \"sorter_params\": sorter_params,     },     skip_duplicates=True, ) (     sgs.SpikeSorterParameters     &amp; {\"sorter\": sorter_name, \"sorter_params_name\": sorter_params_name} ).fetch1() Out[34]: <pre>{'sorter': 'mountainsort4',\n 'sorter_params_name': 'hippocampus_tutorial',\n 'sorter_params': {'detect_sign': -1,\n  'adjacency_radius': 100,\n  'freq_min': 0,\n  'freq_max': 0,\n  'filter': False,\n  'whiten': False,\n  'num_workers': 4,\n  'clip_size': 39,\n  'detect_threshold': 3,\n  'detect_interval': 10,\n  'verbose': True}}</pre> In\u00a0[35]: Copied! <pre>ss_key = dict(\n    **(sgs.ArtifactDetection &amp; ssr_key).fetch1(\"KEY\"),\n    **(sgs.ArtifactRemovedIntervalList() &amp; ssr_key).fetch1(\"KEY\"),\n    sorter=sorter_name,\n    sorter_params_name=sorter_params_name,\n)\nss_key.pop(\"artifact_params_name\")\nss_key\n</pre> ss_key = dict(     **(sgs.ArtifactDetection &amp; ssr_key).fetch1(\"KEY\"),     **(sgs.ArtifactRemovedIntervalList() &amp; ssr_key).fetch1(\"KEY\"),     sorter=sorter_name,     sorter_params_name=sorter_params_name, ) ss_key.pop(\"artifact_params_name\") ss_key Out[35]: <pre>{'nwb_file_name': 'minirec20230622_.nwb',\n 'sort_group_id': 0,\n 'sort_interval_name': '01_s1_first9',\n 'preproc_params_name': 'default_hippocampus',\n 'team_name': 'My Team',\n 'artifact_removed_interval_list_name': 'minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times',\n 'sorter': 'mountainsort4',\n 'sorter_params_name': 'hippocampus_tutorial'}</pre> In\u00a0[36]: Copied! <pre>sgs.SpikeSortingSelection.insert1(ss_key, skip_duplicates=True)\n(sgs.SpikeSortingSelection &amp; ss_key)\n</pre> sgs.SpikeSortingSelection.insert1(ss_key, skip_duplicates=True) (sgs.SpikeSortingSelection &amp; ss_key) Out[36]: Table for holding selection of recording and parameters for each spike sorting run <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>import_path</p> optional path to previous curated sorting output minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times <p>Total: 1</p> In\u00a0[39]: Copied! <pre># [(sgs.SpikeSortingSelection &amp; ss_key).proj()]\nsgs.SpikeSorting.populate(ss_key)\n</pre> # [(sgs.SpikeSortingSelection &amp; ss_key).proj()] sgs.SpikeSorting.populate(ss_key) <pre>[11:33:09][INFO] Spyglass: Running spike sorting on {'nwb_file_name': 'minirec20230622_.nwb', 'sort_group_id': 0, 'sort_interval_name': '01_s1_first9', 'preproc_params_name': 'default_hippocampus', 'team_name': 'My Team', 'sorter': 'mountainsort4', 'sorter_params_name': 'hippocampus_tutorial', 'artifact_removed_interval_list_name': 'minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times'}...\n</pre> <pre>Mountainsort4 use the OLD spikeextractors mapped with NewToOldRecording\nUsing temporary directory /stelmo/nwb/tmp/tmpo38gkza9\nUsing 4 workers.\nUsing tempdir: /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06\nNum. workers = 4\nPreparing /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06/timeseries.hdf5...\nPreparing neighborhood sorters (M=3, N=269997)...\nNeighboorhood of channel 1 has 3 channels.\nDetecting events on channel 2 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.035859\nNum events detected on channel 2 (phase1): 334\nComputing PCA features for channel 2 (phase1)...\nClustering for channel 2 (phase1)...\nFound 1 clusters for channel 2 (phase1)...\nComputing templates for channel 2 (phase1)...\nRe-assigning events for channel 2 (phase1)...\nNeighboorhood of channel 2 has 3 channels.\nDetecting events on channel 3 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.036157\nNum events detected on channel 3 (phase1): 463\nComputing PCA features for channel 3 (phase1)...\nClustering for channel 3 (phase1)...\nFound 1 clusters for channel 3 (phase1)...\nComputing templates for channel 3 (phase1)...\nRe-assigning events for channel 3 (phase1)...\nNeighboorhood of channel 0 has 3 channels.\nDetecting events on channel 1 (phase1)...\nElapsed time for detect on neighborhood: 0:00:00.033028\nNum events detected on channel 1 (phase1): 341\nComputing PCA features for channel 1 (phase1)...\nClustering for channel 1 (phase1)...\nFound 2 clusters for channel 1 (phase1)...\nComputing templates for channel 1 (phase1)...\nRe-assigning events for channel 1 (phase1)...\nNeighboorhood of channel 1 has 3 channels.\nComputing PCA features for channel 2 (phase2)...\nNo duplicate events found for channel 1 in phase2\nClustering for channel 2 (phase2)...\nFound 1 clusters for channel 2 (phase2)...\nNeighboorhood of channel 2 has 3 channels.\nComputing PCA features for channel 3 (phase2)...\nNo duplicate events found for channel 2 in phase2\nClustering for channel 3 (phase2)...\nFound 1 clusters for channel 3 (phase2)...\nNeighboorhood of channel 0 has 3 channels.\nComputing PCA features for channel 1 (phase2)...\nNo duplicate events found for channel 0 in phase2\nClustering for channel 1 (phase2)...\nFound 1 clusters for channel 1 (phase2)...\nPreparing output...\nDone with ms4alg.\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py:234: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n[11:33:24][INFO] Spyglass: Saving sorting results...\n</pre> <pre>Cleaning tempdir::::: /stelmo/nwb/tmp/tmpo38gkza9/tmp05afo_06\nmountainsort4 run time 12.62s\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/basesorting.py:212: UserWarning: The registered recording will not be persistent on disk, but only available in memory\n  warnings.warn(\"The registered recording will not be persistent on disk, but only available in memory\")\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up &lt;TemporaryDirectory '/stelmo/nwb/tmp/tmpo38gkza9'&gt;\n  _warnings.warn(warn_message, ResourceWarning)\n</pre> In\u00a0[40]: Copied! <pre>sgs.SpikeSorting() &amp; ss_key\n</pre> sgs.SpikeSorting() &amp; ss_key Out[40]: <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>sorting_path</p> <p>time_of_sort</p> in Unix time, to the nearest second minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times /stelmo/nwb/spikesorting/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_d318c3f1_spikesorting 1710873204 <p>Total: 1</p> In\u00a0[42]: Copied! <pre>for sorting_key in (sgs.SpikeSorting() &amp; ss_key).fetch(\"KEY\"):\n    # insert_curation will make an entry with a new curation_id regardless of whether it already exists\n    # to avoid this, we check if the curation already exists\n    if not (sgs.Curation() &amp; sorting_key):\n        sgs.Curation.insert_curation(sorting_key)\n\nsgs.Curation() &amp; ss_key\n</pre> for sorting_key in (sgs.SpikeSorting() &amp; ss_key).fetch(\"KEY\"):     # insert_curation will make an entry with a new curation_id regardless of whether it already exists     # to avoid this, we check if the curation already exists     if not (sgs.Curation() &amp; sorting_key):         sgs.Curation.insert_curation(sorting_key)  sgs.Curation() &amp; ss_key Out[42]: Stores each spike sorting; similar to IntervalList <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>parent_curation_id</p> <p>curation_labels</p> a dictionary of labels for the units <p>merge_groups</p> a list of merge groups for the units <p>quality_metrics</p> a list of quality metrics for the units (if available) <p>description</p> optional description for this curated sort <p>time_of_creation</p> in Unix time, to the nearest second 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times -1 =BLOB= =BLOB= =BLOB= 1710873795 <p>Total: 1</p> In\u00a0[45]: Copied! <pre># Parameters used for waveform extraction from the recording\nwaveform_params_name = \"default_whitened\"\n(\n    sgs.WaveformParameters() &amp; {\"waveform_params_name\": waveform_params_name}\n).fetch(as_dict=True)[0]\n</pre> # Parameters used for waveform extraction from the recording waveform_params_name = \"default_whitened\" (     sgs.WaveformParameters() &amp; {\"waveform_params_name\": waveform_params_name} ).fetch(as_dict=True)[0] Out[45]: <pre>{'waveform_params_name': 'default_whitened',\n 'waveform_params': {'ms_before': 0.5,\n  'ms_after': 0.5,\n  'max_spikes_per_unit': 5000,\n  'n_jobs': 5,\n  'total_memory': '5G',\n  'whiten': True}}</pre> In\u00a0[46]: Copied! <pre># extract waveforms\ncuration_keys = [\n    {**k, \"waveform_params_name\": waveform_params_name}\n    for k in (sgs.Curation() &amp; ss_key &amp; {\"curation_id\": 0}).fetch(\"KEY\")\n]\nsgs.WaveformSelection.insert(curation_keys, skip_duplicates=True)\nsgs.Waveforms.populate(ss_key)\n</pre> # extract waveforms curation_keys = [     {**k, \"waveform_params_name\": waveform_params_name}     for k in (sgs.Curation() &amp; ss_key &amp; {\"curation_id\": 0}).fetch(\"KEY\") ] sgs.WaveformSelection.insert(curation_keys, skip_duplicates=True) sgs.Waveforms.populate(ss_key) <pre>[11:48:56][INFO] Spyglass: Extracting waveforms...\n</pre> <pre>extract waveforms memmap:   0%|          | 0/1 [00:00&lt;?, ?it/s]</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f76c8677d00&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f752b912d00&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/multiprocessing/popen_fork.py:66: ResourceWarning: Unclosed socket &lt;zmq.Socket(zmq.PUSH) at 0x7f76c8677760&gt;\n  self.pid = os.fork()\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n[11:48:57][INFO] Spyglass: Writing new NWB file minirec20230622_4ZZBN5G9DY.nwb\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/hdmf/build/objectmapper.py:260: DtypeConversionWarning: Spec 'Units/spike_times': Value with data type int64 is being converted to data type float64 as specified.\n  warnings.warn(full_warning_msg, DtypeConversionWarning)\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_4ZZBN5G9DY.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:276: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size &gt; 0` to check that an array is not empty.\n  if check_hash:\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> In\u00a0[58]: Copied! <pre># parameters which define what quality metrics are calculated and how\nmetric_params_name = \"franklab_default3\"\n(sgs.MetricParameters() &amp; {\"metric_params_name\": metric_params_name}).fetch(\n    \"metric_params\"\n)[0]\n</pre> # parameters which define what quality metrics are calculated and how metric_params_name = \"franklab_default3\" (sgs.MetricParameters() &amp; {\"metric_params_name\": metric_params_name}).fetch(     \"metric_params\" )[0] Out[58]: <pre>{'snr': {'peak_sign': 'neg',\n  'random_chunk_kwargs_dict': {'num_chunks_per_segment': 20,\n   'chunk_size': 10000,\n   'seed': 0}},\n 'isi_violation': {'isi_threshold_ms': 1.5, 'min_isi_ms': 0.0},\n 'nn_isolation': {'max_spikes': 1000,\n  'min_spikes': 10,\n  'n_neighbors': 5,\n  'n_components': 7,\n  'radius_um': 100,\n  'seed': 0},\n 'nn_noise_overlap': {'max_spikes': 1000,\n  'min_spikes': 10,\n  'n_neighbors': 5,\n  'n_components': 7,\n  'radius_um': 100,\n  'seed': 0},\n 'peak_channel': {'peak_sign': 'neg'},\n 'num_spikes': {}}</pre> In\u00a0[59]: Copied! <pre>waveform_keys = [\n    {**k, \"metric_params_name\": metric_params_name}\n    for k in (sgs.Waveforms() &amp; ss_key).fetch(\"KEY\")\n]\nsgs.MetricSelection.insert(waveform_keys, skip_duplicates=True)\nsgs.QualityMetrics().populate(ss_key)\nsgs.QualityMetrics() &amp; ss_key\n</pre> waveform_keys = [     {**k, \"metric_params_name\": metric_params_name}     for k in (sgs.Waveforms() &amp; ss_key).fetch(\"KEY\") ] sgs.MetricSelection.insert(waveform_keys, skip_duplicates=True) sgs.QualityMetrics().populate(ss_key) sgs.QualityMetrics() &amp; ss_key <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/postprocessing/template_tools.py:23: DeprecationWarning: The spikeinterface.postprocessing.template_tools is submodule is deprecated.Use spikeinterface.core.template_tools instead\n  _warn()\n[12:17:37][INFO] Spyglass: Computed all metrics: {'snr': {1: 3.627503, 2: 3.598743, 3: 3.6419973}, 'isi_violation': {'1': 0.03896103896103896, '2': 0.036065573770491806, '3': 0.03488372093023256}, 'nn_isolation': {'1': 0.9591503267973855, '2': 0.9594771241830065, '3': 0.9872549019607844}, 'nn_noise_overlap': {'1': 0.49642857142857144, '2': 0.44738562091503264, '3': 0.4}, 'peak_channel': {1: 1, 2: 2, 3: 3}, 'num_spikes': {'1': 309, '2': 306, '3': 431}}\n[12:17:37][INFO] Spyglass: Writing new NWB file minirec20230622_L3O536PHYB.nwb\n[12:17:38][INFO] Spyglass: Adding metric snr : {1: 3.627503, 2: 3.598743, 3: 3.6419973}\n[12:17:38][INFO] Spyglass: Adding metric isi_violation : {'1': 0.03896103896103896, '2': 0.036065573770491806, '3': 0.03488372093023256}\n[12:17:38][INFO] Spyglass: Adding metric nn_isolation : {'1': 0.9591503267973855, '2': 0.9594771241830065, '3': 0.9872549019607844}\n[12:17:38][INFO] Spyglass: Adding metric nn_noise_overlap : {'1': 0.49642857142857144, '2': 0.44738562091503264, '3': 0.4}\n[12:17:38][INFO] Spyglass: Adding metric peak_channel : {1: 1, 2: 2, 3: 3}\n[12:17:38][INFO] Spyglass: Adding metric num_spikes : {'1': 309, '2': 306, '3': 431}\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_L3O536PHYB.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:276: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size &gt; 0` to check that an array is not empty.\n  if check_hash:\n/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> Out[59]: <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>waveform_params_name</p> name of waveform extraction parameters <p>metric_params_name</p> <p>quality_metrics_path</p> <p>analysis_file_name</p> name of the file <p>object_id</p> Object ID for the metrics in NWB file 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times default_whitened franklab_default3 /stelmo/nwb/waveforms/minirec20230622_.nwb_0105557c_0_default_whitened_waveforms_qm.json minirec20230622_L3O536PHYB.nwb 4b1512bc-861f-4710-8fff-55aad7fbb6ba <p>Total: 1</p> In\u00a0[64]: Copied! <pre># Look at the quality metrics for the first curation\n(sgs.QualityMetrics() &amp; ss_key).fetch_nwb()[0][\"object_id\"]\n</pre> # Look at the quality metrics for the first curation (sgs.QualityMetrics() &amp; ss_key).fetch_nwb()[0][\"object_id\"] <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file &lt;_io.BufferedReader name='/stelmo/nwb/analysis/minirec20230622/minirec20230622_L3O536PHYB.nwb'&gt;\n  return uuid_from_stream(Path(filepath).open(\"rb\"), init_string=init_string)\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> Out[64]: snr isi_violation nn_isolation nn_noise_overlap peak_channel num_spikes id 1 3.627503 0.038961 0.959150 0.496429 1 309 2 3.598743 0.036066 0.959477 0.447386 2 306 3 3.641997 0.034884 0.987255 0.400000 3 431 In\u00a0[67]: Copied! <pre># We can select our criteria for unit labeling here\nauto_curation_params_name = \"default\"\n(\n    sgs.AutomaticCurationParameters()\n    &amp; {\"auto_curation_params_name\": auto_curation_params_name}\n).fetch1()\n</pre> # We can select our criteria for unit labeling here auto_curation_params_name = \"default\" (     sgs.AutomaticCurationParameters()     &amp; {\"auto_curation_params_name\": auto_curation_params_name} ).fetch1() Out[67]: <pre>{'auto_curation_params_name': 'default',\n 'merge_params': {},\n 'label_params': {'nn_noise_overlap': ['&gt;', 0.1, ['noise', 'reject']]}}</pre> In\u00a0[72]: Copied! <pre># We can now apply the automatic curation criteria to the quality metrics\nmetric_keys = [\n    {**k, \"auto_curation_params_name\": auto_curation_params_name}\n    for k in (sgs.QualityMetrics() &amp; ss_key).fetch(\"KEY\")\n]\nsgs.AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True)\n# populating this table will make a new entry in the curation table\nsgs.AutomaticCuration().populate(ss_key)\nsgs.Curation() &amp; ss_key\n</pre> # We can now apply the automatic curation criteria to the quality metrics metric_keys = [     {**k, \"auto_curation_params_name\": auto_curation_params_name}     for k in (sgs.QualityMetrics() &amp; ss_key).fetch(\"KEY\") ] sgs.AutomaticCurationSelection.insert(metric_keys, skip_duplicates=True) # populating this table will make a new entry in the curation table sgs.AutomaticCuration().populate(ss_key) sgs.Curation() &amp; ss_key Out[72]: Stores each spike sorting; similar to IntervalList <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> <p>parent_curation_id</p> <p>curation_labels</p> a dictionary of labels for the units <p>merge_groups</p> a list of merge groups for the units <p>quality_metrics</p> a list of quality metrics for the units (if available) <p>description</p> optional description for this curated sort <p>time_of_creation</p> in Unix time, to the nearest second 0 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times -1 =BLOB= =BLOB= =BLOB= 17108737951 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times 0 =BLOB= =BLOB= =BLOB= auto curated 1710876397 <p>Total: 2</p> In\u00a0[82]: Copied! <pre># get the curation keys corresponding to the automatic curation\nauto_curation_key_list = (sgs.AutomaticCuration() &amp; ss_key).fetch(\n    \"auto_curation_key\"\n)\n\n# insert into CuratedSpikeSorting\nfor auto_key in auto_curation_key_list:\n    # get the full key information needed\n    curation_auto_key = (sgs.Curation() &amp; auto_key).fetch1(\"KEY\")\n    sgs.CuratedSpikeSortingSelection.insert1(\n        curation_auto_key, skip_duplicates=True\n    )\nsgs.CuratedSpikeSorting.populate(ss_key)\n\n# Add the curated spike sorting to the SpikeSortingOutput merge table\nkeys_for_merge_tables = (\n    sgs.CuratedSpikeSorting &amp; auto_curation_key_list\n).fetch(\"KEY\")\nSpikeSortingOutput.insert(\n    keys_for_merge_tables,\n    skip_duplicates=True,\n    part_name=\"CuratedSpikeSorting\",\n)\n# Here's our result!\nSpikeSortingOutput.CuratedSpikeSorting() &amp; ss_key\n</pre> # get the curation keys corresponding to the automatic curation auto_curation_key_list = (sgs.AutomaticCuration() &amp; ss_key).fetch(     \"auto_curation_key\" )  # insert into CuratedSpikeSorting for auto_key in auto_curation_key_list:     # get the full key information needed     curation_auto_key = (sgs.Curation() &amp; auto_key).fetch1(\"KEY\")     sgs.CuratedSpikeSortingSelection.insert1(         curation_auto_key, skip_duplicates=True     ) sgs.CuratedSpikeSorting.populate(ss_key)  # Add the curated spike sorting to the SpikeSortingOutput merge table keys_for_merge_tables = (     sgs.CuratedSpikeSorting &amp; auto_curation_key_list ).fetch(\"KEY\") SpikeSortingOutput.insert(     keys_for_merge_tables,     skip_duplicates=True,     part_name=\"CuratedSpikeSorting\", ) # Here's our result! SpikeSortingOutput.CuratedSpikeSorting() &amp; ss_key Out[82]: <p>merge_id</p> <p>curation_id</p> a number correponding to the index of this curation <p>nwb_file_name</p> name of the NWB file <p>sort_group_id</p> identifier for a group of electrodes <p>sort_interval_name</p> name for this interval <p>preproc_params_name</p> <p>team_name</p> <p>sorter</p> <p>sorter_params_name</p> <p>artifact_removed_interval_list_name</p> 662f3e35-c81e-546c-69c3-b3a2f5ed2776 1 minirec20230622_.nwb 0 01_s1_first9 default_hippocampus My Team mountainsort4 hippocampus_tutorial minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_none_artifact_removed_valid_times <p>Total: 1</p> <p>As of June 2021, members of the Frank Lab can use the <code>sortingview</code> web app for manual curation. To make use of this, we need to populate the <code>CurationFigurl</code> table.</p> <p>We begin by selecting a starting point from the curation entries. In this case we will use the AutomaticCuration populated above as a starting point for manual curation, though you could also  start from the opriginal curation entry by selecting the proper key from the <code>Curation</code> table</p> <p>Note: This step requires setting up your kachery sharing through the sharing notebook</p> In\u00a0[107]: Copied! <pre>starting_curations = (sgs.AutomaticCuration() &amp; ss_key).fetch(\n    \"auto_curation_key\"\n)  # you could also select any key from the sgs.Curation table here\n\nusername = \"username\"\nfig_url_repo = f\"gh://LorenFrankLab/sorting-curations/main/{username}/\"  # settings for franklab members\n\nsort_interval_name = interval_list_name\ngh_url = (\n    fig_url_repo\n    + str(nwb_file_name + \"_\" + sort_interval_name)  # session id\n    + \"/{}\"  # tetrode using auto_id['sort_group_id']\n    + \"/curation.json\"\n)  # url where the curation is stored\n\nfor auto_id in starting_curations:\n    auto_curation_out_key = dict(\n        **(sgs.Curation() &amp; auto_id).fetch1(\"KEY\"),\n        new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),\n    )\n    sgs.CurationFigurlSelection.insert1(\n        auto_curation_out_key, skip_duplicates=True\n    )\n    sgs.CurationFigurl.populate(auto_curation_out_key)\n</pre> starting_curations = (sgs.AutomaticCuration() &amp; ss_key).fetch(     \"auto_curation_key\" )  # you could also select any key from the sgs.Curation table here  username = \"username\" fig_url_repo = f\"gh://LorenFrankLab/sorting-curations/main/{username}/\"  # settings for franklab members  sort_interval_name = interval_list_name gh_url = (     fig_url_repo     + str(nwb_file_name + \"_\" + sort_interval_name)  # session id     + \"/{}\"  # tetrode using auto_id['sort_group_id']     + \"/curation.json\" )  # url where the curation is stored  for auto_id in starting_curations:     auto_curation_out_key = dict(         **(sgs.Curation() &amp; auto_id).fetch1(\"KEY\"),         new_curation_uri=gh_url.format(str(auto_id[\"sort_group_id\"])),     )     sgs.CurationFigurlSelection.insert1(         auto_curation_out_key, skip_duplicates=True     )     sgs.CurationFigurl.populate(auto_curation_out_key) <pre>[16:04:49][INFO] Spyglass: Preparing spikesortingview data\n</pre> <pre>Initial pass: segment 0\nSegment 0 of 1\n/stelmo/nwb/kachery-cloud/sha1/de/41/7a/de417af585eda5dc274c1389ad1b28ef1a0580ab\n</pre> <pre>/home/sambray/mambaforge-pypy3/envs/spyglass/lib/python3.9/site-packages/datajoint/autopopulate.py:292: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/stelmo/nwb/recording/minirec20230622_.nwb_01_s1_first9_0_default_hippocampus/traces_cached_seg0.raw' mode='r' encoding='UTF-8'&gt;\n  make(dict(key), **(make_kwargs or {}))\nResourceWarning: Enable tracemalloc to get the object allocation traceback\n</pre> <p>We can then access the url for the curation figurl like so:</p> In\u00a0[108]: Copied! <pre>print((sgs.CurationFigurl &amp; ss_key).fetch(\"url\")[0])\n</pre> print((sgs.CurationFigurl &amp; ss_key).fetch(\"url\")[0]) <pre>https://figurl.org/f?v=npm://@fi-sci/figurl-sortingview@12/dist&amp;d=sha1://b0d9355ba302bcbcb7005822796fc850c06b6d3d&amp;s={\"initialSortingCuration\":\"sha1://2800ea072728fd141d8e5bc88525ac0c6c137d04\",\"sortingCuration\":\"gh://LorenFrankLab/sorting-curations/main/sambray/minirec20230622_.nwb_01_s1/0/curation.json\"}&amp;label=minirec20230622_.nwb_01_s1_first9_0_default_hippocampus%20minirec20230622_.nwb_01_s1_first9_0_default_hippocampus_42be1215_spikesorting&amp;zone=franklab.collaborators\n</pre> <p>This will take you to a workspace on the <code>sortingview</code> app. The workspace, which you can think of as a list of recording and associated sorting objects, was created at the end of spike sorting. On the workspace view, you will see a set of recordings that have been added to the workspace.</p> <p></p> <p>Clicking on a recording then takes you to a page that gives you information about the recording as well as the associated sorting objects.</p> <p></p> <p>Click on a sorting to see the curation view. Try exploring the many visualization widgets.</p> <p></p> <p>The most important is the <code>Units Table</code> and the <code>Curation</code> menu, which allows you to give labels to the units. The curation labels will persist even if you suddenly lose connection to the app; this is because the curation actions are appended to the workspace as soon as they are created. Note that if you are not logged in with your Google account, <code>Curation</code> menu may not be visible. Log in and refresh the page to access this feature.</p> <p></p>"}, {"location": "notebooks/10_Spike_SortingV0/#spike-sorting", "title": "Spike Sorting\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#imports", "title": "Imports\u00b6", "text": "<p>Let's start by importing tables from Spyglass and quieting warnings caused by some dependencies.</p> <p>Note: It the imports below throw a <code>FileNotFoundError</code>, make a cell with <code>!env | grep X</code> where X is part of the problematic directory. This will show the variable causing issues. Make another cell that sets this variable elsewhere with <code>%env VAR=\"/your/path/\"</code></p>"}, {"location": "notebooks/10_Spike_SortingV0/#fetch-exercise", "title": "Fetch Exercise\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#adding-an-nwb-file", "title": "Adding an NWB file\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#import-data", "title": "Import Data\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#extracting-the-recording", "title": "Extracting the recording\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#sortgroup", "title": "<code>SortGroup</code>\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#intervallist", "title": "<code>IntervalList</code>\u00b6", "text": "<p>Next, we make a decision about the time interval for our spike sorting using <code>IntervalList</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#sortinterval", "title": "<code>SortInterval</code>\u00b6", "text": "<p>For longer recordings, Spyglass subsets this interval with <code>SortInterval</code>. Below, we select the first <code>n</code> seconds of this interval.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#preprocessing-parameters", "title": "Preprocessing Parameters\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#processing-a-key", "title": "Processing a key\u00b6", "text": "<p>key is often used to describe an entry we want to move through the pipeline, and keys are often managed as dictionaries. Here, we'll manage the spike sort recording key, <code>ssr_key</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#recording-selection", "title": "Recording Selection\u00b6", "text": "<p>We now insert this key <code>SpikeSortingRecordingSelection</code> table to specify what time/tetrode/etc. of the recording we want to extract.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesortingrecording", "title": "<code>SpikeSortingRecording</code>\u00b6", "text": "<p>And now we're ready to extract the recording! The <code>populate</code> command will automatically process data in Computed or Imported table tiers.</p> <p>If we only want to process certain entries, we can grab their primary key with the <code>.proj()</code> command and use a list of primary keys when calling <code>populate</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#artifact-detection", "title": "Artifact Detection\u00b6", "text": "<p><code>ArtifactDetectionParameters</code> establishes the parameters for removing artifacts from the data. We may want to target artifact signal that is within the frequency band of our filter (600Hz-6KHz), and thus will not get removed by filtering.</p> <p>For this demo, we'll use a parameter set to skip this step.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spike-sorting", "title": "Spike sorting\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#spikesorterparameters", "title": "<code>SpikeSorterParameters</code>\u00b6", "text": "<p>For our example, we will be using <code>mountainsort4</code>. There are already some default parameters in the <code>SpikeSorterParameters</code> table we'll <code>fetch</code>.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesortingselection", "title": "<code>SpikeSortingSelection</code>\u00b6", "text": "<p>Gearing up to Spike Sort!</p> <p>We now collect our various keys to insert into <code>SpikeSortingSelection</code>, which is specific to this recording and eventual sorting segment.</p> <p>Note: the spike sorter parameters defined above are specific to <code>mountainsort4</code> and may not work for other sorters.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#spikesorting", "title": "<code>SpikeSorting</code>\u00b6", "text": "<p>After adding to <code>SpikeSortingSelection</code>, we can simply populate <code>SpikeSorting</code>.</p> <p>Note: This may take time with longer data sets. Be sure to <code>pip install mountainsort4</code> if this is your first time spike sorting.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#check-to-make-sure-the-table-populated", "title": "Check to make sure the table populated\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV0/#automatic-curation", "title": "Automatic Curation\u00b6", "text": "<p>Spikesorting algorithms can sometimes identify noise or other undesired features as spiking units. Spyglass provides a curation pipeline to detect and label such features to exclude them from downstream analysis.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#initial-curation", "title": "Initial Curation\u00b6", "text": "<p>The <code>Curation</code> table keeps track of rounds of spikesorting curations in the spikesorting v0 pipeline. Before we begin, we first insert an initial curation entry with the spiking results.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#waveform-extraction", "title": "Waveform Extraction\u00b6", "text": "<p>Some metrics used for curating units are dependent on features of the spike waveform. We extract these for each unit's initial curation here</p>"}, {"location": "notebooks/10_Spike_SortingV0/#quality-metrics", "title": "Quality Metrics\u00b6", "text": "<p>With these waveforms, we can calculate the metrics used to determine the quality of each unit.</p>"}, {"location": "notebooks/10_Spike_SortingV0/#automatic-curation-labeling", "title": "Automatic Curation Labeling\u00b6", "text": "<p>With these metrics, we can assign labels to the sorted units using the <code>AutomaticCuration</code> table</p>"}, {"location": "notebooks/10_Spike_SortingV0/#insert-desired-curation-into-downstream-and-merge-tables-for-future-analysis", "title": "Insert desired curation into downstream and merge tables for future analysis\u00b6", "text": "<p>Now that we've performed auto-curation, we can insert the results of our chosen curation into  <code>CuratedSpikeSorting</code> (the final table of this pipeline), and the merge table <code>SpikeSortingOutput</code>. Downstream analyses such as decoding will access the spiking data from there</p>"}, {"location": "notebooks/10_Spike_SortingV0/#manual-curation-with-figurl", "title": "Manual Curation with figurl\u00b6", "text": ""}, {"location": "notebooks/10_Spike_SortingV1/", "title": "Spike Sorting V1", "text": "<p>This is a tutorial for Spyglass spike sorting pipeline version 1 (V1). This pipeline coexists with version 0 but differs in that:</p> <ul> <li>it stores more of the intermediate results (e.g. filtered and referenced recording) in the NWB format</li> <li>it has more streamlined curation pipelines</li> <li>it uses UUIDs as the primary key for important tables (e.g. <code>SpikeSorting</code>) to reduce the number of keys that make up the composite primary key</li> </ul> <p>The output of both versions of the pipeline are saved in a merge table called <code>SpikeSortingOutput</code>.</p> <p>To start, connect to the database. See instructions in Setup.</p> In\u00a0[\u00a0]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config[\"enable_python_native_blobs\"] = True\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n%load_ext autoreload\n%autoreload 2\n</pre> import os import datajoint as dj import numpy as np  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config[\"enable_python_native_blobs\"] = True dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  %load_ext autoreload %autoreload 2 <p>First, import the pipeline and other necessary modules.</p> In\u00a0[\u00a0]: Copied! <pre>import spyglass.common as sgc\nimport spyglass.spikesorting.v1 as sgs\nimport spyglass.data_import as sgi\n</pre> import spyglass.common as sgc import spyglass.spikesorting.v1 as sgs import spyglass.data_import as sgi <p>We will be using <code>mediumnwb20230802.nwb</code> as our example. As usual, first insert the NWB file into <code>Session</code> (can skip if you have already done so).</p> In\u00a0[\u00a0]: Copied! <pre>nwb_file_name = \"mediumnwb20230802.nwb\"\nnwb_file_name2 = \"mediumnwb20230802_.nwb\"\n</pre> nwb_file_name = \"mediumnwb20230802.nwb\" nwb_file_name2 = \"mediumnwb20230802_.nwb\" In\u00a0[\u00a0]: Copied! <pre>sgi.insert_sessions(nwb_file_name)\n</pre> sgi.insert_sessions(nwb_file_name) In\u00a0[\u00a0]: Copied! <pre>sgc.Session()\n</pre> sgc.Session() <p>Next, we will define the groups of electrodes to sort together. These are stored in <code>SortGroup</code> table.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_file_name2)\n</pre> sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_file_name2) <p>The next step is to filter and reference the recording so that we isolate the spike band data. This is done by combining the data with the parameters in <code>SpikeSortingRecordingSelection</code>. For inserting into this table, use <code>insert_selection</code> method. This automatically generates a UUID for a recording.</p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"nwb_file_name\": nwb_file_name2,\n    \"sort_group_id\": 0,\n    \"preproc_param_name\": \"default\",\n}\n</pre> key = {     \"nwb_file_name\": nwb_file_name2,     \"sort_group_id\": 0,     \"preproc_param_name\": \"default\", } In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecordingSelection.insert_selection(key)\n</pre> sgs.SpikeSortingRecordingSelection.insert_selection(key) <p>Next we will call <code>populate</code> method of <code>SpikeSortingRecording</code>.</p> In\u00a0[\u00a0]: Copied! <pre># Assuming 'key' is a dictionary with fields that you want to include in 'ssr_key'\nssr_key = {\n    \"recording_id\": (sgs.SpikeSortingRecordingSelection() &amp; key).fetch1('recording_id'),\n} | key\n</pre> # Assuming 'key' is a dictionary with fields that you want to include in 'ssr_key' ssr_key = {     \"recording_id\": (sgs.SpikeSortingRecordingSelection() &amp; key).fetch1('recording_id'), } | key In\u00a0[\u00a0]: Copied! <pre>#sgs.SpikeSortingRecording.populate()\nssr_pk = (sgs.SpikeSortingRecordingSelection &amp; key).proj()\n\n\nsgs.SpikeSortingRecording.populate(ssr_pk)\n</pre> #sgs.SpikeSortingRecording.populate() ssr_pk = (sgs.SpikeSortingRecordingSelection &amp; key).proj()   sgs.SpikeSortingRecording.populate(ssr_pk) In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecording() &amp; ssr_key\n</pre> sgs.SpikeSortingRecording() &amp; ssr_key In\u00a0[\u00a0]: Copied! <pre>key = (\n    sgs.SpikeSortingRecordingSelection &amp; key\n).fetch1()\n</pre> key = (     sgs.SpikeSortingRecordingSelection &amp; key ).fetch1() <p>Sometimes the recording may contain artifacts that can confound spike sorting. For example, we often have artifacts when the animal licks the reward well for milk during behavior. These appear as sharp transients across all channels, and sometimes they are not adequately removed by filtering and referencing. We will identify the periods during which this type of artifact appears and set them to zero so that they won't interfere with spike sorting.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetectionSelection.insert_selection(\n    {\"recording_id\": key[\"recording_id\"], \"artifact_param_name\": \"default\"}\n)\n</pre> sgs.ArtifactDetectionSelection.insert_selection(     {\"recording_id\": key[\"recording_id\"], \"artifact_param_name\": \"default\"} ) In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetection.populate()\n</pre> sgs.ArtifactDetection.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.ArtifactDetection()\n</pre> sgs.ArtifactDetection() <p>The output of <code>ArtifactDetection</code> is actually stored in <code>IntervalList</code> because it is another type of interval. The UUID however can be found in both.</p> <p>Now that we have prepared the recording, we will pair this with a spike sorting algorithm and associated parameters. This will be inserted to <code>SpikeSortingSelection</code>, again via <code>insert_selection</code> method.</p> <p>The spike sorting pipeline is powered by <code>spikeinterface</code>, a community-developed Python package that enables one to easily apply multiple spike sorters to a single recording. Some spike sorters have special requirements, such as GPU. Others need to be installed separately from spyglass. In the Frank lab, we have been using <code>mountainsort4</code>, though the pipeline have been tested with <code>mountainsort5</code>, <code>kilosort2_5</code>, <code>kilosort3</code>, and <code>ironclust</code> as well.</p> <p>When using <code>mountainsort5</code>, make sure to run <code>pip install mountainsort5</code>. <code>kilosort2_5</code>, <code>kilosort3</code>, and <code>ironclust</code> are MATLAB-based, but we can run these without having to install MATLAB thanks to <code>spikeinterface</code>. It does require downloading additional files (as singularity containers) so make sure to do <code>pip install spython</code>. These sorters also require GPU access, so also do <code>pip install cuda-python</code> (and make sure your computer does have a GPU).</p> In\u00a0[\u00a0]: Copied! <pre>sorter = 'mountainsort4'\n\ncommon_key = {\n    \"recording_id\": key[\"recording_id\"],\n    \"sorter\": sorter,\n    \"nwb_file_name\": nwb_file_name2,\n    \"interval_list_name\": str(\n        (\n            sgs.ArtifactDetectionSelection\n            &amp; {\"recording_id\": key[\"recording_id\"]}\n        ).fetch1(\"artifact_id\")\n    ),\n}\n\nif sorter == 'mountainsort4':\n    key = {\n        **common_key,\n        \"sorter_param_name\": \"franklab_tetrode_hippocampus_30KHz\",\n    }\nelse:\n    key = {\n        **common_key,\n        \"sorter_param_name\": \"default\",\n    }\n</pre> sorter = 'mountainsort4'  common_key = {     \"recording_id\": key[\"recording_id\"],     \"sorter\": sorter,     \"nwb_file_name\": nwb_file_name2,     \"interval_list_name\": str(         (             sgs.ArtifactDetectionSelection             &amp; {\"recording_id\": key[\"recording_id\"]}         ).fetch1(\"artifact_id\")     ), }  if sorter == 'mountainsort4':     key = {         **common_key,         \"sorter_param_name\": \"franklab_tetrode_hippocampus_30KHz\",     } else:     key = {         **common_key,         \"sorter_param_name\": \"default\",     } In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingSelection.insert_selection(key)\n</pre> sgs.SpikeSortingSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingSelection() &amp; key\n</pre> sgs.SpikeSortingSelection() &amp; key <p>Once <code>SpikeSortingSelection</code> is populated, let's run <code>SpikeSorting.populate</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sss_pk = (sgs.SpikeSortingSelection &amp; key).proj()\n\nsgs.SpikeSorting.populate(sss_pk)\n</pre> sss_pk = (sgs.SpikeSortingSelection &amp; key).proj()  sgs.SpikeSorting.populate(sss_pk) <p>The spike sorting results (spike times of detected units) are saved in an NWB file. We can access this in two ways. First, we can access it via the <code>fetch_nwb</code> method, which allows us to directly access the spike times saved in the <code>units</code> table of the NWB file. Second, we can access it as a <code>spikeinterface.NWBSorting</code> object. Ths allows us to take advantage of the rich APIs of <code>spikeinterface</code> to further analyze the sorting.</p> In\u00a0[\u00a0]: Copied! <pre>sorting_nwb = (sgs.SpikeSorting &amp; key).fetch_nwb()\nsorting_si = sgs.SpikeSorting.get_sorting(key)\n</pre> sorting_nwb = (sgs.SpikeSorting &amp; key).fetch_nwb() sorting_si = sgs.SpikeSorting.get_sorting(key) <p>Note that the spike times of <code>fetch_nwb</code> is in units of seconds aligned with the timestamps of the recording. The spike times of the <code>spikeinterface.NWBSorting</code> object is in units of samples (as is generally true for sorting objects in <code>spikeinterface</code>).</p> <p>Next step is to curate the results of spike sorting. This is often necessary because spike sorting algorithms are not perfect; they often return clusters that are clearly not biological in origin, and sometimes oversplit clusters that should have been merged. We have two main ways of curating spike sorting: by computing quality metrics followed by thresholding, and manually applying curation labels. To do either, we first insert the spike sorting to <code>CurationV1</code> using <code>insert_curation</code> method.</p> In\u00a0[\u00a0]: Copied! <pre>sgs.SpikeSortingRecording &amp; key\n</pre> sgs.SpikeSortingRecording &amp; key In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=(\n        sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}\n    ).fetch1(\"sorting_id\"),\n    description=\"testing sort\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=(         sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}     ).fetch1(\"sorting_id\"),     description=\"testing sort\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>We will first do an automatic curation based on quality metrics. Under the hood, this part again makes use of <code>spikeinterface</code>. Some of the quality metrics that we often compute are the nearest neighbor isolation and noise overlap metrics, as well as SNR and ISI violation rate. For computing some of these metrics, the waveforms must be extracted and projected onto a feature space. Thus here we set the parameters for waveform extraction as well as how to curate the units based on these metrics (e.g. if <code>nn_noise_overlap</code> is greater than 0.1, mark as <code>noise</code>).</p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"sorting_id\": (\n        sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}\n    ).fetch1(\"sorting_id\"),\n    \"curation_id\": 0,\n    \"waveform_param_name\": \"default_not_whitened\",\n    \"metric_param_name\": \"franklab_default\",\n    \"metric_curation_param_name\": \"default\",\n}\n</pre> key = {     \"sorting_id\": (         sgs.SpikeSortingSelection &amp; {\"recording_id\": key[\"recording_id\"]}     ).fetch1(\"sorting_id\"),     \"curation_id\": 0,     \"waveform_param_name\": \"default_not_whitened\",     \"metric_param_name\": \"franklab_default\",     \"metric_curation_param_name\": \"default\", } In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCurationSelection.insert_selection(key)\n</pre> sgs.MetricCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCurationSelection()\n</pre> sgs.MetricCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCuration.populate()\n</pre> sgs.MetricCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.MetricCuration()\n</pre> sgs.MetricCuration() <p>to do another round of curation, fetch the relevant info and insert back into CurationV1 using <code>insert_curation</code></p> In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"metric_curation_id\": (\n        sgs.MetricCurationSelection &amp; {\"sorting_id\": key[\"sorting_id\"]}\n    ).fetch1(\"metric_curation_id\")\n}\n</pre> key = {     \"metric_curation_id\": (         sgs.MetricCurationSelection &amp; {\"sorting_id\": key[\"sorting_id\"]}     ).fetch1(\"metric_curation_id\") } In\u00a0[\u00a0]: Copied! <pre>labels = sgs.MetricCuration.get_labels(key)\n</pre> labels = sgs.MetricCuration.get_labels(key) In\u00a0[\u00a0]: Copied! <pre>merge_groups = sgs.MetricCuration.get_merge_groups(key)\n</pre> merge_groups = sgs.MetricCuration.get_merge_groups(key) In\u00a0[\u00a0]: Copied! <pre>metrics = sgs.MetricCuration.get_metrics(key)\n</pre> metrics = sgs.MetricCuration.get_metrics(key) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=(\n        sgs.MetricCurationSelection\n        &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n    ).fetch1(\"sorting_id\"),\n    parent_curation_id=0,\n    labels=labels,\n    merge_groups=merge_groups,\n    metrics=metrics,\n    description=\"after metric curation\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=(         sgs.MetricCurationSelection         &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}     ).fetch1(\"sorting_id\"),     parent_curation_id=0,     labels=labels,     merge_groups=merge_groups,     metrics=metrics,     description=\"after metric curation\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>next we will do manual curation. this is done with figurl. to incorporate info from other stages of processing (e.g. metrics) we have to store that with kachery cloud and get curation uri referring to it. it can be done with <code>generate_curation_uri</code>.</p> In\u00a0[\u00a0]: Copied! <pre>curation_uri = sgs.FigURLCurationSelection.generate_curation_uri(\n    {\n        \"sorting_id\": (\n            sgs.MetricCurationSelection\n            &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n        ).fetch1(\"sorting_id\"),\n        \"curation_id\": 1,\n    }\n)\n</pre> curation_uri = sgs.FigURLCurationSelection.generate_curation_uri(     {         \"sorting_id\": (             sgs.MetricCurationSelection             &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}         ).fetch1(\"sorting_id\"),         \"curation_id\": 1,     } ) In\u00a0[\u00a0]: Copied! <pre>key = {\n    \"sorting_id\": (\n        sgs.MetricCurationSelection\n        &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}\n    ).fetch1(\"sorting_id\"),\n    \"curation_id\": 1,\n    \"curation_uri\": curation_uri,\n    \"metrics_figurl\": list(metrics.keys()),\n}\n</pre> key = {     \"sorting_id\": (         sgs.MetricCurationSelection         &amp; {\"metric_curation_id\": key[\"metric_curation_id\"]}     ).fetch1(\"sorting_id\"),     \"curation_id\": 1,     \"curation_uri\": curation_uri,     \"metrics_figurl\": list(metrics.keys()), } In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection()\n</pre> sgs.FigURLCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection.insert_selection(key)\n</pre> sgs.FigURLCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection()\n</pre> sgs.FigURLCurationSelection() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration.populate()\n</pre> sgs.FigURLCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration()\n</pre> sgs.FigURLCuration() <p>or you can manually specify it if you already have a <code>curation.json</code></p> In\u00a0[\u00a0]: Copied! <pre>gh_curation_uri = (\n    \"gh://LorenFrankLab/sorting-curations/main/khl02007/test/curation.json\"\n)\n\nkey = {\n    \"sorting_id\": key[\"sorting_id\"],\n    \"curation_id\": 1,\n    \"curation_uri\": gh_curation_uri,\n    \"metrics_figurl\": [],\n}\n</pre> gh_curation_uri = (     \"gh://LorenFrankLab/sorting-curations/main/khl02007/test/curation.json\" )  key = {     \"sorting_id\": key[\"sorting_id\"],     \"curation_id\": 1,     \"curation_uri\": gh_curation_uri,     \"metrics_figurl\": [], } In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCurationSelection.insert_selection(key)\n</pre> sgs.FigURLCurationSelection.insert_selection(key) In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration.populate()\n</pre> sgs.FigURLCuration.populate() In\u00a0[\u00a0]: Copied! <pre>sgs.FigURLCuration()\n</pre> sgs.FigURLCuration() <p>once you apply manual curation (curation labels and merge groups) you can store them as nwb by inserting another row in CurationV1. And then you can do more rounds of curation if you want.</p> In\u00a0[\u00a0]: Copied! <pre>labels = sgs.FigURLCuration.get_labels(gh_curation_uri)\n</pre> labels = sgs.FigURLCuration.get_labels(gh_curation_uri) In\u00a0[\u00a0]: Copied! <pre>merge_groups = sgs.FigURLCuration.get_merge_groups(gh_curation_uri)\n</pre> merge_groups = sgs.FigURLCuration.get_merge_groups(gh_curation_uri) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1.insert_curation(\n    sorting_id=key[\"sorting_id\"],\n    parent_curation_id=1,\n    labels=labels,\n    merge_groups=merge_groups,\n    metrics=metrics,\n    description=\"after figurl curation\",\n)\n</pre> sgs.CurationV1.insert_curation(     sorting_id=key[\"sorting_id\"],     parent_curation_id=1,     labels=labels,     merge_groups=merge_groups,     metrics=metrics,     description=\"after figurl curation\", ) In\u00a0[\u00a0]: Copied! <pre>sgs.CurationV1()\n</pre> sgs.CurationV1() <p>We now insert the curated spike sorting to a <code>Merge</code> table for feeding into downstream processing pipelines.</p> In\u00a0[\u00a0]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput()\n</pre> SpikeSortingOutput() In\u00a0[\u00a0]: Copied! <pre>key\n</pre> key In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.insert([key], part_name=\"CurationV1\")\n</pre> SpikeSortingOutput.insert([key], part_name=\"CurationV1\") In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.merge_view()\n</pre> SpikeSortingOutput.merge_view() In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.CurationV1()\n</pre> SpikeSortingOutput.CurationV1() In\u00a0[\u00a0]: Copied! <pre>SpikeSortingOutput.CuratedSpikeSorting()\n</pre> SpikeSortingOutput.CuratedSpikeSorting() In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/10_Spike_SortingV1/#spike-sorting-pipeline-version-1", "title": "Spike Sorting: pipeline version 1\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/", "title": "Position Trodes", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>In this tutorial, we'll process position data extracted with Trodes Tracking by</p> <ul> <li>Defining parameters</li> <li>Processing raw position</li> <li>Extracting centroid and orientation</li> <li>Insert the results into the <code>TrodesPosV1</code> table</li> <li>Plotting the head position/direction results for quality assurance</li> </ul> <p>The pipeline takes the 2D video pixel data of green/red LEDs, and computes:</p> <ul> <li>head position (in cm)</li> <li>head orientation (in radians)</li> <li>head velocity (in cm/s)</li> <li>head speed (in cm/s)</li> </ul> In\u00a0[1]: Copied! <pre>import os\nimport datajoint as dj\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position as sgp\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position as sgp  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-12 13:47:50,578][INFO]: Connecting root@localhost:3306\n[2024-01-12 13:47:50,652][INFO]: Connected root@localhost:3306\n</pre> <p>First, we'll grab let us make sure that the session we want to analyze is inserted into the <code>RawPosition</code> table</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\n# Define the name of the file that you copied and renamed\nnwb_file_name = \"minirec20230622.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  # Define the name of the file that you copied and renamed nwb_file_name = \"minirec20230622.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) In\u00a0[3]: Copied! <pre>sgc.common_behav.RawPosition() &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.common_behav.RawPosition() &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[3]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list minirec20230622_.nwb pos 0 valid timesminirec20230622_.nwb pos 1 valid times <p>Total: 2</p> <p>Parameters are set by the <code>TrodesPosParams</code> table, with a <code>default</code> set available. To adjust the default, insert a new set into this table. The parameters are...</p> <ul> <li><code>max_separation</code>, default 9 cm: maximum acceptable distance between red and green LEDs.<ul> <li>If exceeded, the times are marked as NaNs and inferred by interpolation.</li> <li>Useful when the inferred LED position tracks a reflection instead of the true position.</li> </ul> </li> <li><code>max_speed</code>, default 300.0 cm/s: maximum speed the animal can move.<ul> <li>If exceeded, times are marked as NaNs and inferred by interpolation.</li> <li>Useful to prevent big jumps in position.</li> </ul> </li> <li><code>position_smoothing_duration</code>, default 0.100 s: LED position smoothing before computing average position to get head position.</li> <li><code>speed_smoothing_std_dev</code>, default 0.100 s: standard deviation of the Gaussian kernel used to smooth the head speed.</li> <li><code>front_led1</code>, default 1 (True), use <code>xloc</code>/<code>yloc</code>: Which LED is the front LED for calculating the head direction.<ul> <li>1: LED corresponding to <code>xloc</code>, <code>yloc</code> in the <code>RawPosition</code> table is the front, <code>xloc2</code>, <code>yloc2</code> as the back.</li> <li>0: LED corresponding to <code>xloc2</code>, <code>yloc2</code> in the <code>RawPosition</code> table is the front, <code>xloc</code>, <code>yloc</code> as the back.</li> </ul> </li> </ul> <p>We can see these defaults with <code>TrodesPosParams().default_params</code>.</p> In\u00a0[4]: Copied! <pre>from pprint import pprint\n\nparameters = sgp.v1.TrodesPosParams().default_params\npprint(parameters)\n</pre> from pprint import pprint  parameters = sgp.v1.TrodesPosParams().default_params pprint(parameters) <pre>{'is_upsampled': 0,\n 'led1_is_front': 1,\n 'max_LED_separation': 9.0,\n 'max_plausible_speed': 300.0,\n 'orient_smoothing_std_dev': 0.001,\n 'position_smoothing_duration': 0.125,\n 'speed_smoothing_std_dev': 0.1,\n 'upsampling_interpolation_method': 'linear',\n 'upsampling_sampling_rate': None}\n</pre> <p>For the <code>minirec</code> demo file, only one LED is moving. The following paramset will allow us to process this data.</p> In\u00a0[5]: Copied! <pre>trodes_params_name = \"single_led\"\ntrodes_params = {\n    \"max_separation\": 10000.0,\n    \"max_speed\": 300.0,\n    \"position_smoothing_duration\": 0.125,\n    \"speed_smoothing_std_dev\": 0.1,\n    \"orient_smoothing_std_dev\": 0.001,\n    \"led1_is_front\": 1,\n    \"is_upsampled\": 0,\n    \"upsampling_sampling_rate\": None,\n    \"upsampling_interpolation_method\": \"linear\",\n}\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": trodes_params_name,\n        \"params\": trodes_params,\n    },\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosParams()\n</pre> trodes_params_name = \"single_led\" trodes_params = {     \"max_separation\": 10000.0,     \"max_speed\": 300.0,     \"position_smoothing_duration\": 0.125,     \"speed_smoothing_std_dev\": 0.1,     \"orient_smoothing_std_dev\": 0.001,     \"led1_is_front\": 1,     \"is_upsampled\": 0,     \"upsampling_sampling_rate\": None,     \"upsampling_interpolation_method\": \"linear\", } sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": trodes_params_name,         \"params\": trodes_params,     },     skip_duplicates=True, ) sgp.v1.TrodesPosParams() Out[5]: <p>trodes_pos_params_name</p> name for this set of parameters <p>params</p> default =BLOB=single_led =BLOB=single_led_upsampled =BLOB= <p>Total: 3</p> <p>Later, we'll pair the above parameters with an interval from our NWB file and insert into <code>TrodesPosSelection</code>.</p> <p>First, let's select an interval from the <code>IntervalList</code> table.</p> In\u00a0[6]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[6]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval minirec20230622_.nwb 01_s1 =BLOB=minirec20230622_.nwb 02_s2 =BLOB=minirec20230622_.nwb pos 0 valid times =BLOB=minirec20230622_.nwb pos 1 valid times =BLOB=minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 5</p> <p>The raw position in pixels is in the <code>RawPosition</code> table is extracted from the video data by the algorithm in Trodes. We have timepoints available for the duration when position tracking was turned on and off, which may be a subset of the video itself.</p> <p><code>fetch1_dataframe</code> returns the position of the LEDs as a pandas dataframe where time is the index.</p> In\u00a0[7]: Copied! <pre>interval_list_name = \"pos 0 valid times\"  # pos # is epoch # minus 1\nraw_position_df = (\n    sgc.RawPosition()\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": interval_list_name,\n    }\n).fetch1_dataframe()\nraw_position_df\n</pre> interval_list_name = \"pos 0 valid times\"  # pos # is epoch # minus 1 raw_position_df = (     sgc.RawPosition()     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": interval_list_name,     } ).fetch1_dataframe() raw_position_df <pre>[2024-01-12 13:47:54,251][WARNING]: Skipped checksum for file with hash: cce88743-51b1-5ad9-836a-260c938383dd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/minirec20230622_.nwb\n[2024-01-12 13:47:54,254][WARNING]: Skipped checksum for file with hash: cce88743-51b1-5ad9-836a-260c938383dd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/minirec20230622_.nwb\n</pre> Out[7]: xloc1 yloc1 xloc2 yloc2 time 1.687475e+09 445 567 0 0 1.687475e+09 445 567 0 0 1.687475e+09 445 567 0 0 1.687475e+09 444 568 0 0 1.687475e+09 444 568 0 0 ... ... ... ... ... 1.687475e+09 479 536 0 0 1.687475e+09 480 534 0 0 1.687475e+09 480 533 0 0 1.687475e+09 481 530 0 0 1.687475e+09 481 530 0 0 <p>267 rows \u00d7 4 columns</p> <p>Let's just quickly plot the two LEDs to get a sense of the inputs to the pipeline:</p> In\u00a0[8]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 10))\nax.plot(raw_position_df.xloc1, raw_position_df.yloc1, color=\"green\")\n# Uncomment for multiple LEDs\n# ax.plot(raw_position_df.xloc2, raw_position_df.yloc2, color=\"red\")\nax.set_xlabel(\"x-position [pixels]\", fontsize=18)\nax.set_ylabel(\"y-position [pixels]\", fontsize=18)\nax.set_title(\"Raw Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 10)) ax.plot(raw_position_df.xloc1, raw_position_df.yloc1, color=\"green\") # Uncomment for multiple LEDs # ax.plot(raw_position_df.xloc2, raw_position_df.yloc2, color=\"red\") ax.set_xlabel(\"x-position [pixels]\", fontsize=18) ax.set_ylabel(\"y-position [pixels]\", fontsize=18) ax.set_title(\"Raw Position\", fontsize=28) Out[8]: <pre>Text(0.5, 1.0, 'Raw Position')</pre> <p>To associate a set of parameters with a given interval, insert them into the <code>TrodesPosSelection</code> table.</p> In\u00a0[9]: Copied! <pre>trodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": trodes_params_name,\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\n</pre> trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": trodes_params_name, } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) <p>Now let's check to see if we've inserted the parameters correctly:</p> In\u00a0[10]: Copied! <pre>trodes_key = (sgp.v1.TrodesPosSelection() &amp; trodes_s_key).fetch1(\"KEY\")\n</pre> trodes_key = (sgp.v1.TrodesPosSelection() &amp; trodes_s_key).fetch1(\"KEY\") <p>We can run the pipeline for our chosen interval/parameters by using the <code>TrodesPosV1.populate</code>.</p> In\u00a0[11]: Copied! <pre>sgp.v1.TrodesPosV1.populate(trodes_key)\n</pre> sgp.v1.TrodesPosV1.populate(trodes_key) <p>Each NWB file, interval, and parameter set is now associated with a new analysis file and object ID.</p> In\u00a0[12]: Copied! <pre>sgp.v1.TrodesPosV1 &amp; trodes_key\n</pre> sgp.v1.TrodesPosV1 &amp; trodes_key Out[12]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>position_object_id</p> <p>orientation_object_id</p> <p>velocity_object_id</p> minirec20230622_.nwb pos 0 valid times single_led minirec20230622_AQQP7U6Y24.nwb f519d1e4-0919-492a-85a4-6730cce26c10 6ae01b40-f5d9-4dd1-9203-76881d7bd339 002ce1f0-50a6-40b5-8b97-4b5a80140193 <p>Total: 1</p> <p>When we populatethe <code>TrodesPosV1</code> table, we automatically create an entry in the <code>PositionOutput</code> merge table. Since this table supports position information from multiple methods, it's best practive to access data through here.</p> <p>We can view the entry in this table:</p> In\u00a0[13]: Copied! <pre>from spyglass.position import PositionOutput\n\nPositionOutput.TrodesPosV1 &amp; trodes_key\n</pre> from spyglass.position import PositionOutput  PositionOutput.TrodesPosV1 &amp; trodes_key Out[13]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters 15055247-04ba-ab0f-3b67-522f0ecc17e3 minirec20230622_.nwb pos 0 valid times single_led <p>Total: 1</p> <p>To retrieve the results as a pandas DataFrame with time as the index, we use <code>PositionOutput.fetch1_dataframe</code>. When doing so, we need to restric the merge table by the</p> <p>This dataframe has the following columns:</p> <ul> <li><code>position_{x,y}</code>: X or Y position of the head in cm.</li> <li><code>orientation</code>: Direction of the head relative to the bottom left corner in radians</li> <li><code>velocity_{x,y}</code>: Directional change in head position over time in cm/s</li> <li><code>speed</code>: the magnitude of the change in head position over time in cm/s</li> </ul> In\u00a0[14]: Copied! <pre># get the merge id corresponding to our inserted trodes_key\nmerge_key = (PositionOutput.merge_get_part(trodes_key)).fetch1(\"KEY\")\n# use this to restrict PositionOutput and fetch the data\nposition_info = (PositionOutput &amp; merge_key).fetch1_dataframe()\nposition_info\n</pre> # get the merge id corresponding to our inserted trodes_key merge_key = (PositionOutput.merge_get_part(trodes_key)).fetch1(\"KEY\") # use this to restrict PositionOutput and fetch the data position_info = (PositionOutput &amp; merge_key).fetch1_dataframe() position_info <pre>[2024-01-12 13:47:55,378][WARNING]: Skipped checksum for file with hash: c87c4027-855f-0181-d477-cf78242a7c20, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/minirec20230622/minirec20230622_AQQP7U6Y24.nwb\n</pre> Out[14]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.687475e+09 0 22.250000 28.350000 0.905373 -0.148627 0.116866 0.189071 1.687475e+09 1 22.250000 28.350000 0.905373 -0.219948 0.162667 0.273565 1.687475e+09 2 22.250000 28.350000 0.905373 -0.298575 0.205042 0.362201 1.687475e+09 3 22.233333 28.366667 0.906022 -0.371837 0.233439 0.439041 1.687475e+09 4 22.216667 28.383333 0.906671 -0.424639 0.238911 0.487234 ... ... ... ... ... ... ... ... 1.687475e+09 262 23.766667 27.066667 0.850225 2.738364 -3.998619 4.846400 1.687475e+09 263 23.933333 26.800000 0.841843 2.347307 -3.708857 4.389245 1.687475e+09 264 23.983333 26.716667 0.839258 1.907429 -3.237540 3.757652 1.687475e+09 265 24.016667 26.616667 0.836703 1.457185 -2.647347 3.021893 1.687475e+09 266 24.033333 26.550000 0.835110 1.040384 -2.021305 2.273340 <p>267 rows \u00d7 7 columns</p> <p><code>.index</code> on the pandas dataframe gives us timestamps.</p> In\u00a0[15]: Copied! <pre>position_info.index\n</pre> position_info.index Out[15]: <pre>Index([1687474800.1833298,  1687474800.216676,  1687474800.250001,\n        1687474800.283326,  1687474800.316672, 1687474801.8658931,\n       1687474801.8992178, 1687474801.9325643,  1687474801.965889,\n        1687474801.999214,\n       ...\n        1687474810.265809,  1687474810.299134, 1687474810.3324802,\n       1687474810.3658051,   1687474810.39913,  1687474810.432476,\n        1687474810.465801,  1687474810.499126, 1687474810.5324721,\n        1687474810.565797],\n      dtype='float64', name='time', length=267)</pre> <p>We should always spot check our results to verify that the pipeline worked correctly.</p> <p>Let's plot some of the variables first:</p> In\u00a0[16]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(8, 8))\nax.plot(position_info.position_x, position_info.position_y)\nax.set_xlabel(\"x-position [cm]\", fontsize=16)\nax.set_ylabel(\"y-position [cm]\", fontsize=16)\nax.set_title(\"Position\", fontsize=20)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(8, 8)) ax.plot(position_info.position_x, position_info.position_y) ax.set_xlabel(\"x-position [cm]\", fontsize=16) ax.set_ylabel(\"y-position [cm]\", fontsize=16) ax.set_title(\"Position\", fontsize=20) Out[16]: <pre>Text(0.5, 1.0, 'Position')</pre> In\u00a0[17]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(8, 8))\nax.plot(position_info.velocity_x, position_info.velocity_y)\nax.set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\nax.set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\nax.set_title(\"Velocity\", fontsize=20)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(8, 8)) ax.plot(position_info.velocity_x, position_info.velocity_y) ax.set_xlabel(\"x-velocity [cm/s]\", fontsize=16) ax.set_ylabel(\"y-velocity [cm/s]\", fontsize=16) ax.set_title(\"Velocity\", fontsize=20) Out[17]: <pre>Text(0.5, 1.0, 'Velocity')</pre> In\u00a0[18]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(16, 3))\nax.plot(position_info.index, position_info.speed)\nax.set_xlabel(\"Time\", fontsize=16)\nax.set_ylabel(\"Speed [cm/s]\", fontsize=16)\nax.set_title(\"Head Speed\", fontsize=20)\nax.set_xlim((position_info.index.min(), position_info.index.max()))\n</pre> fig, ax = plt.subplots(1, 1, figsize=(16, 3)) ax.plot(position_info.index, position_info.speed) ax.set_xlabel(\"Time\", fontsize=16) ax.set_ylabel(\"Speed [cm/s]\", fontsize=16) ax.set_title(\"Head Speed\", fontsize=20) ax.set_xlim((position_info.index.min(), position_info.index.max())) Out[18]: <pre>(1687474800.1833298, 1687474810.565797)</pre> In\u00a0[19]: Copied! <pre># sgp.v1.TrodesPosVideo().populate(\n#     {\n#         \"nwb_file_name\": nwb_copy_file_name,\n#         \"interval_list_name\": interval_list_name,\n#         \"position_info_param_name\": trodes_params_name,\n#     }\n# )\n</pre> # sgp.v1.TrodesPosVideo().populate( #     { #         \"nwb_file_name\": nwb_copy_file_name, #         \"interval_list_name\": interval_list_name, #         \"position_info_param_name\": trodes_params_name, #     } # ) In\u00a0[20]: Copied! <pre># sgp.v1.TrodesPosVideo()\n</pre> # sgp.v1.TrodesPosVideo() In\u00a0[21]: Copied! <pre>trodes_params_up_name = trodes_params_name + \"_upsampled\"\ntrodes_params_up = {\n    **trodes_params,\n    \"is_upsampled\": 1,\n    \"upsampling_sampling_rate\": 500,\n}\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": trodes_params_up_name,\n        \"params\": trodes_params_up,\n    },\n    skip_duplicates=True,\n)\n\nsgp.v1.TrodesPosParams()\n</pre> trodes_params_up_name = trodes_params_name + \"_upsampled\" trodes_params_up = {     **trodes_params,     \"is_upsampled\": 1,     \"upsampling_sampling_rate\": 500, } sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": trodes_params_up_name,         \"params\": trodes_params_up,     },     skip_duplicates=True, )  sgp.v1.TrodesPosParams() Out[21]: <p>trodes_pos_params_name</p> name for this set of parameters <p>params</p> default =BLOB=single_led =BLOB=single_led_upsampled =BLOB= <p>Total: 3</p> In\u00a0[22]: Copied! <pre>trodes_s_up_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": trodes_params_up_name,\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_up_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_up_key)\n</pre> trodes_s_up_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": trodes_params_up_name, } sgp.v1.TrodesPosSelection.insert1(     trodes_s_up_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_up_key) In\u00a0[23]: Copied! <pre>merge_key = (PositionOutput.merge_get_part(trodes_s_up_key)).fetch1(\"KEY\")\nupsampled_position_info = (PositionOutput &amp; merge_key).fetch1_dataframe()\nupsampled_position_info\n</pre> merge_key = (PositionOutput.merge_get_part(trodes_s_up_key)).fetch1(\"KEY\") upsampled_position_info = (PositionOutput &amp; merge_key).fetch1_dataframe() upsampled_position_info <pre>[13:47:56][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[2024-01-12 13:47:56,476][WARNING]: Skipped checksum for file with hash: 119a4889-1117-30a9-c774-3c7db7048f02, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/minirec20230622/minirec20230622_PBPM9HN98Y.nwb\n</pre> Out[23]: position_x position_y orientation velocity_x velocity_y speed time 1.687475e+09 22.250000 28.350000 0.808756 -0.083265 0.082262 0.117047 1.687475e+09 22.250000 28.350000 0.905134 -0.084928 0.083888 0.119373 1.687475e+09 22.250000 28.350000 0.905373 -0.086591 0.085514 0.121699 1.687475e+09 22.250000 28.350000 0.905373 -0.088254 0.087138 0.124023 1.687475e+09 22.250000 28.350000 0.905373 -0.089915 0.088760 0.126345 ... ... ... ... ... ... ... 1.687475e+09 24.029412 26.565686 0.835485 1.095035 -2.036613 2.312335 1.687475e+09 24.030392 26.561765 0.835391 1.070802 -1.998650 2.267425 1.687475e+09 24.031373 26.557843 0.835298 1.046820 -1.960860 2.222792 1.687475e+09 24.032353 26.553922 0.834983 1.023096 -1.923260 2.178452 1.687475e+09 24.033333 26.550000 0.746002 0.999636 -1.885865 2.134422 <p>5193 rows \u00d7 6 columns</p> In\u00a0[24]: Copied! <pre>fig, axes = plt.subplots(\n    1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.position_x, position_info.position_y)\naxes[0].set_xlabel(\"x-position [cm]\", fontsize=16)\naxes[0].set_ylabel(\"y-position [cm]\", fontsize=16)\naxes[0].set_title(\"Position\", fontsize=20)\n\naxes[1].plot(\n    upsampled_position_info.position_x,\n    upsampled_position_info.position_y,\n)\naxes[1].set_xlabel(\"x-position [cm]\", fontsize=16)\naxes[1].set_ylabel(\"y-position [cm]\", fontsize=16)\naxes[1].set_title(\"Upsampled Position\", fontsize=20)\n</pre> fig, axes = plt.subplots(     1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.position_x, position_info.position_y) axes[0].set_xlabel(\"x-position [cm]\", fontsize=16) axes[0].set_ylabel(\"y-position [cm]\", fontsize=16) axes[0].set_title(\"Position\", fontsize=20)  axes[1].plot(     upsampled_position_info.position_x,     upsampled_position_info.position_y, ) axes[1].set_xlabel(\"x-position [cm]\", fontsize=16) axes[1].set_ylabel(\"y-position [cm]\", fontsize=16) axes[1].set_title(\"Upsampled Position\", fontsize=20) Out[24]: <pre>Text(0.5, 1.0, 'Upsampled Position')</pre> In\u00a0[25]: Copied! <pre>fig, axes = plt.subplots(\n    2, 1, figsize=(16, 6), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.index, position_info.speed)\naxes[0].set_xlabel(\"Time\", fontsize=16)\naxes[0].set_ylabel(\"Speed [cm/s]\", fontsize=16)\naxes[0].set_title(\"Speed\", fontsize=20)\naxes[0].set_xlim((position_info.index.min(), position_info.index.max()))\n\naxes[1].plot(upsampled_position_info.index, upsampled_position_info.speed)\naxes[1].set_xlabel(\"Time\", fontsize=16)\naxes[1].set_ylabel(\"Speed [cm/s]\", fontsize=16)\naxes[1].set_title(\"Upsampled Speed\", fontsize=20)\n</pre> fig, axes = plt.subplots(     2, 1, figsize=(16, 6), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.index, position_info.speed) axes[0].set_xlabel(\"Time\", fontsize=16) axes[0].set_ylabel(\"Speed [cm/s]\", fontsize=16) axes[0].set_title(\"Speed\", fontsize=20) axes[0].set_xlim((position_info.index.min(), position_info.index.max()))  axes[1].plot(upsampled_position_info.index, upsampled_position_info.speed) axes[1].set_xlabel(\"Time\", fontsize=16) axes[1].set_ylabel(\"Speed [cm/s]\", fontsize=16) axes[1].set_title(\"Upsampled Speed\", fontsize=20) Out[25]: <pre>Text(0.5, 1.0, 'Upsampled Speed')</pre> In\u00a0[26]: Copied! <pre>fig, axes = plt.subplots(\n    1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True\n)\naxes[0].plot(position_info.velocity_x, position_info.velocity_y)\naxes[0].set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\naxes[0].set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\naxes[0].set_title(\"Velocity\", fontsize=20)\n\naxes[1].plot(\n    upsampled_position_info.velocity_x,\n    upsampled_position_info.velocity_y,\n)\naxes[1].set_xlabel(\"x-velocity [cm/s]\", fontsize=16)\naxes[1].set_ylabel(\"y-velocity [cm/s]\", fontsize=16)\naxes[1].set_title(\"Upsampled Velocity\", fontsize=20)\n</pre> fig, axes = plt.subplots(     1, 2, figsize=(16, 8), sharex=True, sharey=True, constrained_layout=True ) axes[0].plot(position_info.velocity_x, position_info.velocity_y) axes[0].set_xlabel(\"x-velocity [cm/s]\", fontsize=16) axes[0].set_ylabel(\"y-velocity [cm/s]\", fontsize=16) axes[0].set_title(\"Velocity\", fontsize=20)  axes[1].plot(     upsampled_position_info.velocity_x,     upsampled_position_info.velocity_y, ) axes[1].set_xlabel(\"x-velocity [cm/s]\", fontsize=16) axes[1].set_ylabel(\"y-velocity [cm/s]\", fontsize=16) axes[1].set_title(\"Upsampled Velocity\", fontsize=20) Out[26]: <pre>Text(0.5, 1.0, 'Upsampled Velocity')</pre> <p>In the next notebook, we'll explore using DeepLabCut to generate position data from video.</p>"}, {"location": "notebooks/20_Position_Trodes/#trodes-position", "title": "Trodes Position\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#loading-the-data", "title": "Loading the data\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#setting-parameters", "title": "Setting parameters\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#select-interval", "title": "Select interval\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#pairing-interval-and-parameters", "title": "Pairing interval and parameters\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#running-the-pipeline", "title": "Running the pipeline\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#examine-results", "title": "Examine results\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#plots", "title": "Plots\u00b6", "text": ""}, {"location": "notebooks/20_Position_Trodes/#video", "title": "Video\u00b6", "text": "<p>To keep <code>minirec</code> small, the download link does not include videos by default.</p> <p>If it is available, you can uncomment the code, populate the  <code>TrodesPosVideo</code> table, and plot the results on the video using the <code>make_video</code> function, which will appear in the current working directory.</p>"}, {"location": "notebooks/20_Position_Trodes/#upsampling-position", "title": "Upsampling position\u00b6", "text": "<p>To get position data in smaller in time bins, we can upsample using the  following parameters</p> <ul> <li><code>is_upsampled</code>, default 0 (False): If 1, perform upsampling.</li> <li><code>upsampling_sampling_rate</code>, default None: the rate to upsample to (e.g., 33 Hz video might be upsampled to 500 Hz).</li> <li><code>upsampling_interpolation_method</code>, default linear: interpolation method. See pandas.DataFrame.interpolate for alternate methods.</li> </ul>"}, {"location": "notebooks/20_Position_Trodes/#up-next", "title": "Up Next\u00b6", "text": ""}, {"location": "notebooks/21_DLC/", "title": "DLC Models", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This tutorial will extract position via DeepLabCut (DLC). It will walk through...</p> <ul> <li>creating a DLC project</li> <li>extracting and labeling frames</li> <li>training your model</li> <li>executing pose estimation on a novel behavioral video</li> <li>processing the pose estimation output to extract a centroid and orientation</li> <li>inserting the resulting information into the <code>PositionOutput</code> table</li> </ul> <p>Note 2: Make sure you are running this within the spyglass-position Conda environment (instructions for install are in the environment_position.yml)</p> <p>Here is a schematic showing the tables used in this pipeline.</p> <p></p> <p>You can click on any header to return to the Table of Contents</p> In\u00a0[12]: Copied! <pre>%load_ext autoreload\n%autoreload 2\n</pre> %load_ext autoreload %autoreload 2 In\u00a0[13]: Copied! <pre>import os\nimport datajoint as dj\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\n\nimport numpy as np\nimport pandas as pd\nimport pynwb\nfrom spyglass.position import PositionOutput\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj  import spyglass.common as sgc import spyglass.position.v1 as sgp  import numpy as np import pandas as pd import pynwb from spyglass.position import PositionOutput  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) Notes:<ul> <li>         The cells within this <code>DLCProject</code> step need to be performed          in a local Jupyter notebook to allow for use of the frame labeling GUI.     </li> <li>         Please do not add to the <code>BodyPart</code> table in the production          database unless necessary.     </li> </ul> <p>We'll begin by looking at the <code>BodyPart</code> table, which stores standard names of body parts used in DLC models throughout the lab with a concise description.</p> In\u00a0[14]: Copied! <pre>sgp.BodyPart()\n</pre> sgp.BodyPart() Out[14]: <p>bodypart</p> <p>bodypart_description</p> back middle of the rat's backdriveBack back of drivedriveFront front of driveearL left ear of the ratearR right ear of the ratforelimbL left forelimb of the ratforelimbR right forelimb of the ratgreenLED greenLEDhindlimbL left hindlimb of the rathindlimbR right hindlimb of the ratnose tip of the nose of the ratredLED_C redLED_C <p>...</p> <p>Total: 23</p> <p>If the bodyparts you plan to use in your model are not yet in the table, here is code to add bodyparts:</p> <pre>sgp.BodyPart.insert(\n    [\n        {\"bodypart\": \"bp_1\", \"bodypart_description\": \"concise descrip\"},\n        {\"bodypart\": \"bp_2\", \"bodypart_description\": \"concise descrip\"},\n    ],\n    skip_duplicates=True,\n)\n</pre> <p>To train a model, we'll need to extract frames, which we can label as training data. We can construct a list of videos from which we'll extract frames.</p> <p>The list can either contain dictionaries identifying behavioral videos for NWB files that have already been added to Spyglass, or absolute file paths to the videos you want to use.</p> <p>For this tutorial, we'll use two videos for which we already have frames labeled.</p> <p>Defining camera name is optional: it should be done in cases where there are multiple cameras streaming per epoch, but not necessary otherwise.  example: <code>camera_name = \"HomeBox_camera\"</code></p> <p>NOTE: The official release of Spyglass does not yet support multicamera projects. You can monitor progress on the effort to add this feature by checking this PR or use this experimental branch, which takes the keys nwb_file_name and epoch, and camera_name in the video_list variable.</p> In\u00a0[15]: Copied! <pre>video_list = [\n    {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},\n    {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4},\n]\n</pre> video_list = [     {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},     {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4}, ] In\u00a0[16]: Copied! <pre>from spyglass.settings import config\n\nconfig\n</pre> from spyglass.settings import config  config Out[16]: <pre>{'debug_mode': False,\n 'prepopulate': True,\n 'SPYGLASS_BASE_DIR': '/stelmo/nwb',\n 'SPYGLASS_RAW_DIR': '/stelmo/nwb/raw',\n 'SPYGLASS_ANALYSIS_DIR': '/stelmo/nwb/analysis',\n 'SPYGLASS_RECORDING_DIR': '/stelmo/nwb/recording',\n 'SPYGLASS_SORTING_DIR': '/stelmo/nwb/sorting',\n 'SPYGLASS_WAVEFORMS_DIR': '/stelmo/nwb/waveforms',\n 'SPYGLASS_TEMP_DIR': '/stelmo/nwb/tmp/spyglass',\n 'SPYGLASS_VIDEO_DIR': '/stelmo/nwb/video',\n 'KACHERY_CLOUD_DIR': '/stelmo/nwb/.kachery-cloud',\n 'KACHERY_STORAGE_DIR': '/stelmo/nwb/kachery_storage',\n 'KACHERY_TEMP_DIR': '/stelmo/nwb/tmp',\n 'DLC_PROJECT_DIR': '/nimbus/deeplabcut/projects',\n 'DLC_VIDEO_DIR': '/nimbus/deeplabcut/video',\n 'DLC_OUTPUT_DIR': '/nimbus/deeplabcut/output',\n 'KACHERY_ZONE': 'franklab.default',\n 'FIGURL_CHANNEL': 'franklab2',\n 'DJ_SUPPORT_FILEPATH_MANAGEMENT': 'TRUE',\n 'KACHERY_CLOUD_EPHEMERAL': 'TRUE',\n 'HD5_USE_FILE_LOCKING': 'FALSE'}</pre> <p>Before creating our project, we need to define a few variables.</p> <ul> <li>A team name, as shown in <code>LabTeam</code> for setting permissions. Here, we'll use \"LorenLab\".</li> <li>A <code>project_name</code>, as a unique identifier for this DLC project. Here, we'll use \"tutorial_scratch_yourinitials\"</li> <li><code>bodyparts</code> is a list of body parts for which we want to extract position. The pre-labeled frames we're using include the bodyparts listed below.</li> <li>Number of frames to extract/label as <code>frames_per_video</code>. Note that the DLC creators recommend having 200 frames as the minimum total number for each project.</li> </ul> In\u00a0[17]: Copied! <pre>team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\"\nproject_name = \"tutorial_scratch_DG\"\nframes_per_video = 100\nbodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"]\nproject_key = sgp.DLCProject.insert_new_project(\n    project_name=project_name,\n    bodyparts=bodyparts,\n    lab_team=team_name,\n    frames_per_video=frames_per_video,\n    video_list=video_list,\n    skip_duplicates=True,\n)\n</pre> team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\" project_name = \"tutorial_scratch_DG\" frames_per_video = 100 bodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"] project_key = sgp.DLCProject.insert_new_project(     project_name=project_name,     bodyparts=bodyparts,     lab_team=team_name,     frames_per_video=frames_per_video,     video_list=video_list,     skip_duplicates=True, ) <pre>project name: tutorial_scratch_DG is already in use.\n</pre> <p>Now that we've initialized our project we'll need to extract frames which we will then label.</p> In\u00a0[\u00a0]: Copied! <pre># comment this line out after you finish frame extraction for each project\nsgp.DLCProject().run_extract_frames(project_key)\n</pre> # comment this line out after you finish frame extraction for each project sgp.DLCProject().run_extract_frames(project_key) <p>This is the line used to label the frames you extracted, if you wish to use the DLC GUI on the computer you are currently using.</p> <pre><code>#comment\nsgp.DLCProject().run_label_frames(project_key)\n</code></pre> <p>Otherwise, it is best/easiest practice to label the frames on your local computer (like a MacBook) that can run DeepLabCut's GUI well. Instructions: </p> <ol> <li>Install DLC on your local (preferably into a 'Src' folder): https://deeplabcut.github.io/DeepLabCut/docs/installation.html</li> <li>Upload frames extracted and saved in nimbus (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/labeled-data</code>) AND the project's associated config file (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/config.yaml</code>) to Box (we get free with UCSF)</li> <li>Download labeled-data and config files on your local from Box</li> <li>Create a 'projects' folder where you installed DeepLabCut; create a new folder with your complete project name there; save the downloaded files there.</li> <li>Edit the config.yaml file: line 9 defining <code>project_path</code> needs to be the file path where it is saved on your local (ex: <code>/Users/lorenlab/Src/DeepLabCut/projects/tutorial_sratch_DG-LorenLab-2023-08-16</code>)</li> <li>Open the DLC GUI through terminal (ex: <code>conda activate miniconda/envs/DEEPLABCUT_M1</code> <code>pythonw -m deeplabcut</code>)</li> <li>Load an existing project; choose the config.yaml file</li> <li>Label frames; labeling tutorial: https://www.youtube.com/watch?v=hsA9IB5r73E.</li> <li>Once all frames are labeled, you should re-upload labeled-data folder back to Box and overwrite it in the original nimbus location so that your completed frames are ready to be used in the model.</li> </ol> <p>Now we can check the <code>DLCProject.File</code> part table and see all of our training files and videos there!</p> In\u00a0[18]: Copied! <pre>sgp.DLCProject.File &amp; project_key\n</pre> sgp.DLCProject.File &amp; project_key Out[18]: Paths of training files (e.g., labeled pngs, CSV or video) <p>project_name</p> name of DLC project <p>file_name</p> Concise name to describe file <p>file_ext</p> extension of file <p>file_path</p> tutorial_scratch_DG 20201103_peanut_04_r2 mp4 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20201103_peanut_04_r2.mp4tutorial_scratch_DG 20201103_peanut_04_r2_labeled_data h5 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20201103_peanut_04_r2/CollectedData_LorenLab.h5tutorial_scratch_DG 20210529_J16_02_r1 mp4 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20210529_J16_02_r1.mp4tutorial_scratch_DG 20210529_J16_02_r1_labeled_data h5 /nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20210529_J16_02_r1/CollectedData_LorenLab.h5 <p>Total: 4</p>      This step and beyond should be run on a GPU-enabled machine.  In\u00a0[19]: Copied! <pre>sgp.dlc_utils.get_gpu_memory()\n</pre> sgp.dlc_utils.get_gpu_memory() Out[19]: <pre>{0: 305}</pre> <p>Set GPU core:</p> In\u00a0[20]: Copied! <pre>gputouse = 1  # 1-9\n</pre> gputouse = 1  # 1-9 <p>Now we'll define the rest of our parameters and insert the entry.</p> <p>To see all possible parameters, try:</p> <pre>sgp.DLCModelTrainingParams.get_accepted_params()\n</pre> In\u00a0[21]: Copied! <pre>training_params_name = \"tutorial\"\nsgp.DLCModelTrainingParams.insert_new_params(\n    paramset_name=training_params_name,\n    params={\n        \"trainingsetindex\": 0,\n        \"shuffle\": 1,\n        \"gputouse\": gputouse,\n        \"net_type\": \"resnet_50\",\n        \"augmenter_type\": \"imgaug\",\n    },\n    skip_duplicates=True,\n)\n</pre> training_params_name = \"tutorial\" sgp.DLCModelTrainingParams.insert_new_params(     paramset_name=training_params_name,     params={         \"trainingsetindex\": 0,         \"shuffle\": 1,         \"gputouse\": gputouse,         \"net_type\": \"resnet_50\",         \"augmenter_type\": \"imgaug\",     },     skip_duplicates=True, ) <pre>New param set not added\nA param set with name: tutorial already exists\n</pre> <p>Next we'll modify the <code>project_key</code> from above to include the necessary entries for <code>DLCModelTraining</code></p> In\u00a0[22]: Copied! <pre># project_key['project_path'] = os.path.dirname(project_key['config_path'])\nif \"config_path\" in project_key:\n    del project_key[\"config_path\"]\n</pre> # project_key['project_path'] = os.path.dirname(project_key['config_path']) if \"config_path\" in project_key:     del project_key[\"config_path\"] <p>We can insert an entry into <code>DLCModelTrainingSelection</code> and populate <code>DLCModelTraining</code>.</p> <p>Note: You can stop training at any point using <code>I + I</code> or interrupt the Kernel.</p> <p>The maximum total number of training iterations is 1030000; you can end training before this amount if the loss rate (lr) and total loss plateau and are very close to 0.</p> In\u00a0[23]: Copied! <pre>sgp.DLCModelTrainingSelection.heading\n</pre> sgp.DLCModelTrainingSelection.heading Out[23]: <pre>project_name         : varchar(100)                 # name of DLC project\ndlc_training_params_name : varchar(50)                  # descriptive name of parameter set\ntraining_id          : int                          # unique integer,\n---\nmodel_prefix=\"\"      : varchar(32)                  # </pre> In\u00a0[24]: Copied! <pre>sgp.DLCModelTrainingSelection().insert1(\n    {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n        \"training_id\": 0,\n        \"model_prefix\": \"\",\n    }\n)\nmodel_training_key = (\n    sgp.DLCModelTrainingSelection\n    &amp; {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n    }\n).fetch1(\"KEY\")\nsgp.DLCModelTraining.populate(model_training_key)\n</pre> sgp.DLCModelTrainingSelection().insert1(     {         **project_key,         \"dlc_training_params_name\": training_params_name,         \"training_id\": 0,         \"model_prefix\": \"\",     } ) model_training_key = (     sgp.DLCModelTrainingSelection     &amp; {         **project_key,         \"dlc_training_params_name\": training_params_name,     } ).fetch1(\"KEY\") sgp.DLCModelTraining.populate(model_training_key) <pre>2024-01-18 10:23:30.406102: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n</pre> <pre>Loading DLC 2.2.3...\nOpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py\n</pre> <pre>\n---------------------------------------------------------------------------\nPermissionError                           Traceback (most recent call last)\nCell In[24], line 16\n      1 sgp.DLCModelTrainingSelection().insert1(\n      2     {\n      3         **project_key,\n   (...)\n      7     }\n      8 )\n      9 model_training_key = (\n     10     sgp.DLCModelTrainingSelection\n     11     &amp; {\n   (...)\n     14     }\n     15 ).fetch1(\"KEY\")\n---&gt; 16 sgp.DLCModelTraining.populate(model_training_key)\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:241, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)\n    237 if processes == 1:\n    238     for key in (\n    239         tqdm(keys, desc=self.__class__.__name__) if display_progress else keys\n    240     ):\n--&gt; 241         error = self._populate1(key, jobs, **populate_kwargs)\n    242         if error is not None:\n    243             error_list.append(error)\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:292, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)\n    290 self.__class__._allow_insert = True\n    291 try:\n--&gt; 292     make(dict(key), **(make_kwargs or {}))\n    293 except (KeyboardInterrupt, SystemExit, Exception) as error:\n    294     try:\n\nFile ~/Src/spyglass/src/spyglass/position/v1/position_dlc_training.py:150, in DLCModelTraining.make(self, key)\n    144     from deeplabcut.utils.auxiliaryfunctions import (\n    145         GetModelFolder as get_model_folder,\n    146     )\n    147 config_path, project_name = (DLCProject() &amp; key).fetch1(\n    148     \"config_path\", \"project_name\"\n    149 )\n--&gt; 150 with OutputLogger(\n    151     name=\"DLC_project_{project_name}_training\",\n    152     path=f\"{os.path.dirname(config_path)}/log.log\",\n    153     print_console=True,\n    154 ) as logger:\n    155     dlc_config = read_config(config_path)\n    156     project_path = dlc_config[\"project_path\"]\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:192, in OutputLogger.__init__(self, name, path, level, **kwargs)\n    191 def __init__(self, name, path, level=\"INFO\", **kwargs):\n--&gt; 192     self.logger = self.setup_logger(name, path, **kwargs)\n    193     self.name = self.logger.name\n    194     self.level = getattr(logging, level)\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:244, in OutputLogger.setup_logger(self, name_logfile, path_logfile, print_console)\n    241         logger.addHandler(self._get_stream_handler())\n    243 else:\n--&gt; 244     file_handler = self._get_file_handler(path_logfile)\n    245     logger.addHandler(file_handler)\n    246     if print_console:\n\nFile ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:255, in OutputLogger._get_file_handler(self, path)\n    253 if not os.path.exists(output_dir):\n    254     output_dir.mkdir(parents=True, exist_ok=True)\n--&gt; 255 file_handler = logging.FileHandler(path, mode=\"a\")\n    256 file_handler.setFormatter(self._get_formatter())\n    257 return file_handler\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1146, in FileHandler.__init__(self, filename, mode, encoding, delay, errors)\n   1144     self.stream = None\n   1145 else:\n-&gt; 1146     StreamHandler.__init__(self, self._open())\n\nFile ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1175, in FileHandler._open(self)\n   1170 def _open(self):\n   1171     \"\"\"\n   1172     Open the current base file with the (original) mode and encoding.\n   1173     Return the resulting stream.\n   1174     \"\"\"\n-&gt; 1175     return open(self.baseFilename, self.mode, encoding=self.encoding,\n   1176                 errors=self.errors)\n\nPermissionError: [Errno 13] Permission denied: '/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/log.log'</pre> <p>Here we'll make sure that the entry made it into the table properly!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTraining() &amp; model_training_key\n</pre> sgp.DLCModelTraining() &amp; model_training_key <p>Populating <code>DLCModelTraining</code> automatically inserts the entry into <code>DLCModelSource</code>, which is used to select between models trained using Spyglass vs. other tools.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource() &amp; model_training_key\n</pre> sgp.DLCModelSource() &amp; model_training_key <p>The <code>source</code> field will only accept \"FromImport\" or \"FromUpstream\" as entries. Let's checkout the <code>FromUpstream</code> part table attached to <code>DLCModelSource</code> below.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource.FromUpstream() &amp; model_training_key\n</pre> sgp.DLCModelSource.FromUpstream() &amp; model_training_key In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelParams.get_default()\n</pre> sgp.DLCModelParams.get_default() <p>Here is the syntax to add your own parameter set:</p> <pre>dlc_model_params_name = \"make_this_yours\"\nparams = {\n    \"params\": {},\n    \"shuffle\": 1,\n    \"trainingsetindex\": 0,\n    \"model_prefix\": \"\",\n}\nsgp.DLCModelParams.insert1(\n    {\"dlc_model_params_name\": dlc_model_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We can insert sets of parameters into <code>DLCModelSelection</code> and populate <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\")\n</pre> temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\") In\u00a0[\u00a0]: Copied! <pre># comment these lines out after successfully inserting, for each project\nsgp.DLCModelSelection().insert1(\n    {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True\n)\n</pre> # comment these lines out after successfully inserting, for each project sgp.DLCModelSelection().insert1(     {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True ) In\u00a0[\u00a0]: Copied! <pre>model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\")\nsgp.DLCModel.populate(model_key)\n</pre> model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\") sgp.DLCModel.populate(model_key) <p>Again, let's make sure that everything looks correct in <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModel() &amp; model_key\n</pre> sgp.DLCModel() &amp; model_key In\u00a0[3]: Copied! <pre>nwb_file_name = \"J1620210604_.nwb\"\nsgc.VideoFile() &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> nwb_file_name = \"J1620210604_.nwb\" sgc.VideoFile() &amp; {\"nwb_file_name\": nwb_file_name} Out[3]: <p>nwb_file_name</p> name of the NWB file <p>epoch</p> the session epoch for this task and apparatus(1 based) <p>video_file_num</p> <p>camera_name</p> <p>video_file_object_id</p> the object id of the file object J1620210604_.nwb 1 0 178f5746-30e3-4957-891e-8024e23522dcJ1620210604_.nwb 2 0 d64ec979-326b-429f-b3fe-1bbfbf806293J1620210604_.nwb 3 0 cf14bcd2-c0a9-457b-8791-42f3f28dd912J1620210604_.nwb 4 0 183c9910-36fd-46c1-a24c-8d1c306d7248J1620210604_.nwb 5 0 4677c7cd-8cd8-4801-8f6e-5b7bb14a6d6bJ1620210604_.nwb 6 0 0e46532b-483f-43af-ba6e-ba75ccf340eaJ1620210604_.nwb 7 0 c6d1d037-44ec-4d91-99d1-172d371bf82aJ1620210604_.nwb 8 0 4d7e070c-6220-47de-8173-993f013fafa8J1620210604_.nwb 9 0 b50108ec-f587-46df-b1c8-3ca23091bde0J1620210604_.nwb 10 0 b9b5da20-da39-4274-9be2-55610cfd1b5bJ1620210604_.nwb 11 0 6c827b8d-513c-4dba-ae75-0b36dcf4811fJ1620210604_.nwb 12 0 41bd2344-1b41-4737-8dfb-7c860d089155 <p>...</p> <p>Total: 20</p> In\u00a0[5]: Copied! <pre>epoch = 14  # change based on VideoFile entry\nvideo_file_num = 0  # change based on VideoFile entry\n</pre> epoch = 14  # change based on VideoFile entry video_file_num = 0  # change based on VideoFile entry <p>Using <code>insert_estimation_task</code> will convert out video to be in .mp4 format (DLC struggles with .h264) and determine the directory in which we'll store the pose estimation results.</p> <ul> <li><code>task_mode</code> (trigger or load) determines whether or not populating <code>DLCPoseEstimation</code> triggers a new pose estimation, or loads an existing.</li> <li><code>video_file_num</code> will be 0 in almost all cases.</li> <li><code>gputouse</code> was already set during training. It may be a good idea to make sure that core is still free before moving forward.</li> </ul> <p>The <code>DLCPoseEstimationSelection</code> insertion step will convert your .h264 video to an .mp4 first and save it in <code>/nimbus/deeplabcut/video</code>. If this video already exists here, the insertion will never complete.</p> <p>We first delete any .mp4 that exists for this video from the nimbus folder. Remove the <code>#</code> to run this line. The <code>!</code> tells the notebook that this is a system command to be run with a shell script instead of python. Be sure to change the string based on date and rat with which you are training the model</p> In\u00a0[\u00a0]: Copied! <pre>#! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete\n</pre> #! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete In\u00a0[6]: Copied! <pre>pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    },\n    task_mode=\"trigger\",  # trigger or load\n    params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n)\n</pre> pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(     {         \"nwb_file_name\": nwb_file_name,         \"epoch\": epoch,         \"video_file_num\": video_file_num,         **model_key,     },     task_mode=\"trigger\",  # trigger or load     params={\"gputouse\": gputouse, \"videotype\": \"mp4\"}, ) <pre>\n---------------------------------------------------------------------------\nNameError                                 Traceback (most recent call last)\nCell In[6], line 6\n      1 pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n      2     {\n      3         \"nwb_file_name\": nwb_file_name,\n      4         \"epoch\": epoch,\n      5         \"video_file_num\": video_file_num,\n----&gt; 6         **model_key,\n      7     },\n      8     task_mode=\"trigger\", #trigger or load\n      9     params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n     10 )\n\nNameError: name 'model_key' is not defined</pre> <p>If the above insertion step fails in either trigger or load mode for an epoch, run the following lines:</p> <pre><code>(pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    }).delete()\n</code></pre> <p>And now we populate <code>DLCPoseEstimation</code>! This might take some time for full datasets.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPoseEstimation().populate(pose_estimation_key)\n</pre> sgp.DLCPoseEstimation().populate(pose_estimation_key) <p>Let's visualize the output from Pose Estimation</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPoseEstimation() &amp; pose_estimation_key).fetch_dataframe()\n</pre> (sgp.DLCPoseEstimation() &amp; pose_estimation_key).fetch_dataframe() <p>Now that we've completed pose estimation, it's time to identify NaNs and optionally interpolate over low likelihood periods and smooth the resulting positions.First we need to define some parameters for smoothing and interpolation. We can see the default parameter set below.Note: it is recommended to use the <code>just_nan</code> parameters here and save interpolation and smoothing for the centroid step as this provides for a better end result.</p> In\u00a0[\u00a0]: Copied! <pre># The default parameter set to interpolate and smooth over each LED individually\nprint(sgp.DLCSmoothInterpParams.get_default())\n</pre> # The default parameter set to interpolate and smooth over each LED individually print(sgp.DLCSmoothInterpParams.get_default()) In\u00a0[\u00a0]: Copied! <pre># The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step\nprint(sgp.DLCSmoothInterpParams.get_nan_params())\nsi_params_name = \"just_nan\"  # could also use \"default\"\n</pre> # The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step print(sgp.DLCSmoothInterpParams.get_nan_params()) si_params_name = \"just_nan\"  # could also use \"default\" <p>To change any of these parameters, one would do the following:</p> <pre>si_params_name = \"your_unique_param_name\"\nparams = {\n    \"smoothing_params\": {\n        \"smoothing_duration\": 0.00,\n        \"smooth_method\": \"moving_avg\",\n    },\n    \"interp_params\": {\"likelihood_thresh\": 0.00},\n    \"max_plausible_speed\": 0,\n    \"speed_smoothing_std_dev\": 0.000,\n}\nsgp.DLCSmoothInterpParams().insert1(\n    {\"dlc_si_params_name\": si_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We'll create a dictionary with the correct set of keys for the <code>DLCSmoothInterpSelection</code> table</p> In\u00a0[\u00a0]: Copied! <pre>si_key = pose_estimation_key.copy()\nfields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())\nsi_key = {key: val for key, val in si_key.items() if key in fields}\nsi_key\n</pre> si_key = pose_estimation_key.copy() fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys()) si_key = {key: val for key, val in si_key.items() if key in fields} si_key <p>We can insert all of the bodyparts we want to process into <code>DLCSmoothInterpSelection</code> First lets visualize the bodyparts we have available to us.</p> In\u00a0[\u00a0]: Copied! <pre>print((sgp.DLCPoseEstimation.BodyPart &amp; pose_estimation_key).fetch(\"bodypart\"))\n</pre> print((sgp.DLCPoseEstimation.BodyPart &amp; pose_estimation_key).fetch(\"bodypart\")) <p>We can use <code>insert1</code> to insert a single bodypart, but would suggest using <code>insert</code> to insert a list of keys with different bodyparts.</p> <p>To insert a single bodypart, one would do the following:</p> <pre>sgp.DLCSmoothInterpSelection.insert1(\n    {\n        **si_key,\n        'bodypart': 'greenLED',\n        'dlc_si_params_name': si_params_name,\n    },\n    skip_duplicates=True)\n</pre> <p>We'll see a list of bodyparts and then insert them into <code>DLCSmoothInterpSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>bodyparts = [\"greenLED\", \"redLED_C\"]\nsgp.DLCSmoothInterpSelection.insert(\n    [\n        {\n            **si_key,\n            \"bodypart\": bodypart,\n            \"dlc_si_params_name\": si_params_name,\n        }\n        for bodypart in bodyparts\n    ],\n    skip_duplicates=True,\n)\n</pre> bodyparts = [\"greenLED\", \"redLED_C\"] sgp.DLCSmoothInterpSelection.insert(     [         {             **si_key,             \"bodypart\": bodypart,             \"dlc_si_params_name\": si_params_name,         }         for bodypart in bodyparts     ],     skip_duplicates=True, ) <p>And verify the entry:</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpSelection() &amp; si_key\n</pre> sgp.DLCSmoothInterpSelection() &amp; si_key <p>Now, we populate <code>DLCSmoothInterp</code>, which will perform smoothing and interpolation on all of the bodyparts specified.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterp().populate(si_key)\n</pre> sgp.DLCSmoothInterp().populate(si_key) <p>And let's visualize the resulting position data using a scatter plot</p> In\u00a0[\u00a0]: Copied! <pre>(\n    sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}\n).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))\n</pre> (     sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]} ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5)) <p>After smoothing/interpolation, we need to select bodyparts from which we want to derive a centroid and orientation, which is performed by the <code>DLCSmoothInterpCohort</code> table.</p> <p>First, let's make a key that represents the 'cohort', using <code>dlc_si_cohort_selection_name</code>. We'll need a bodypart dictionary using bodypart keys and smoothing/interpolation parameters used as value.</p> In\u00a0[\u00a0]: Copied! <pre>cohort_key = si_key.copy()\nif \"bodypart\" in cohort_key:\n    del cohort_key[\"bodypart\"]\nif \"dlc_si_params_name\" in cohort_key:\n    del cohort_key[\"dlc_si_params_name\"]\ncohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"\ncohort_key[\"bodyparts_params_dict\"] = {\n    \"greenLED\": si_params_name,\n    \"redLED_C\": si_params_name,\n}\nprint(cohort_key)\n</pre> cohort_key = si_key.copy() if \"bodypart\" in cohort_key:     del cohort_key[\"bodypart\"] if \"dlc_si_params_name\" in cohort_key:     del cohort_key[\"dlc_si_params_name\"] cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\" cohort_key[\"bodyparts_params_dict\"] = {     \"greenLED\": si_params_name,     \"redLED_C\": si_params_name, } print(cohort_key) <p>We'll insert the cohort into <code>DLCSmoothInterpCohortSelection</code> and populate <code>DLCSmoothInterpCohort</code>, which collates the separately smoothed and interpolated bodyparts into a single entry.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True)\nsgp.DLCSmoothInterpCohort.populate(cohort_key)\n</pre> sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True) sgp.DLCSmoothInterpCohort.populate(cohort_key) <p>And verify the entry:</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCSmoothInterpCohort.BodyPart() &amp; cohort_key\n</pre> sgp.DLCSmoothInterpCohort.BodyPart() &amp; cohort_key <p>With this cohort, we can determine a centroid using another set of parameters.</p> In\u00a0[\u00a0]: Copied! <pre># Here is the default set\nprint(sgp.DLCCentroidParams.get_default())\ncentroid_params_name = \"default\"\n</pre> # Here is the default set print(sgp.DLCCentroidParams.get_default()) centroid_params_name = \"default\" <p>Here is the syntax to add your own parameters:</p> <pre>centroid_params = {\n    \"centroid_method\": \"two_pt_centroid\",\n    \"points\": {\n        \"greenLED\": \"greenLED\",\n        \"redLED_C\": \"redLED_C\",\n    },\n    \"speed_smoothing_std_dev\": 0.100,\n}\ncentroid_params_name = \"your_unique_param_name\"\nsgp.DLCCentroidParams.insert1(\n    {\n        \"dlc_centroid_params_name\": centroid_params_name,\n        \"params\": centroid_params,\n    },\n    skip_duplicates=True,\n)\n</pre> <p>We'll make a key to insert into <code>DLCCentroidSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>centroid_key = cohort_key.copy()\nfields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())\ncentroid_key = {key: val for key, val in centroid_key.items() if key in fields}\ncentroid_key[\"dlc_centroid_params_name\"] = centroid_params_name\nprint(centroid_key)\n</pre> centroid_key = cohort_key.copy() fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys()) centroid_key = {key: val for key, val in centroid_key.items() if key in fields} centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name print(centroid_key) <p>After inserting into the selection table, we can populate <code>DLCCentroid</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)\nsgp.DLCCentroid.populate(centroid_key)\n</pre> sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True) sgp.DLCCentroid.populate(centroid_key) <p>Here we can visualize the resulting centroid position</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(\n    x=\"position_x\",\n    y=\"position_y\",\n    c=\"speed\",\n    colormap=\"viridis\",\n    alpha=0.5,\n    s=0.5,\n    figsize=(10, 10),\n)\n</pre> (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(     x=\"position_x\",     y=\"position_y\",     c=\"speed\",     colormap=\"viridis\",     alpha=0.5,     s=0.5,     figsize=(10, 10), ) <p>We'll now go through a similar process to identify the orientation.</p> In\u00a0[\u00a0]: Copied! <pre>print(sgp.DLCOrientationParams.get_default())\ndlc_orientation_params_name = \"default\"\n</pre> print(sgp.DLCOrientationParams.get_default()) dlc_orientation_params_name = \"default\" <p>We'll prune the <code>cohort_key</code> we used above and add our <code>dlc_orientation_params_name</code> to make it suitable for <code>DLCOrientationSelection</code>.</p> In\u00a0[\u00a0]: Copied! <pre>fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())\norient_key = {key: val for key, val in cohort_key.items() if key in fields}\norient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name\nprint(orient_key)\n</pre> fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys()) orient_key = {key: val for key, val in cohort_key.items() if key in fields} orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name print(orient_key) <p>We'll insert into <code>DLCOrientationSelection</code> and populate <code>DLCOrientation</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)\nsgp.DLCOrientation().populate(orient_key)\n</pre> sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True) sgp.DLCOrientation().populate(orient_key) <p>We can fetch the orientation as a dataframe as quality assurance.</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCOrientation() &amp; orient_key).fetch1_dataframe()\n</pre> (sgp.DLCOrientation() &amp; orient_key).fetch1_dataframe() <p>After processing the position data, we have to do a few table manipulations to standardize various outputs.</p> <p>To summarize, we brought in a pretrained DLC project, used that model to run pose estimation on a new behavioral video, smoothed and interpolated the result, formed a cohort of bodyparts, and determined the centroid and orientation of this cohort.</p> <p>Now we'll populate <code>DLCPos</code> with our centroid/orientation entries above.</p> In\u00a0[\u00a0]: Copied! <pre>fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())\ndlc_key = {key: val for key, val in centroid_key.items() if key in fields}\ndlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\"dlc_si_cohort_selection_name\"]\ndlc_key[\"dlc_si_cohort_orientation\"] = orient_key[\n    \"dlc_si_cohort_selection_name\"\n]\ndlc_key[\"dlc_orientation_params_name\"] = orient_key[\n    \"dlc_orientation_params_name\"\n]\nprint(dlc_key)\n</pre> fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys()) dlc_key = {key: val for key, val in centroid_key.items() if key in fields} dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\"dlc_si_cohort_selection_name\"] dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[     \"dlc_si_cohort_selection_name\" ] dlc_key[\"dlc_orientation_params_name\"] = orient_key[     \"dlc_orientation_params_name\" ] print(dlc_key) <p>Now we can insert into <code>DLCPosSelection</code> and populate <code>DLCPos</code> with our <code>dlc_key</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)\nsgp.DLCPosV1().populate(dlc_key)\n</pre> sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True) sgp.DLCPosV1().populate(dlc_key) <p>We can also make sure that all of our data made it through by fetching the dataframe attached to this entry.We should expect 8 columns:</p> <p>timevideo_frame_indposition_xposition_yorientationvelocity_xvelocity_yspeed</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPosV1() &amp; dlc_key).fetch1_dataframe()\n</pre> (sgp.DLCPosV1() &amp; dlc_key).fetch1_dataframe() <p>And even more, we can fetch the <code>pose_eval_result</code> that is calculated during this step. This field contains the percentage of frames that each bodypart was below the likelihood threshold of 0.95 as a means of assessing the quality of the pose estimation.</p> In\u00a0[\u00a0]: Copied! <pre>(sgp.DLCPosV1() &amp; dlc_key).fetch1(\"pose_eval_result\")\n</pre> (sgp.DLCPosV1() &amp; dlc_key).fetch1(\"pose_eval_result\") <p>We can create a video with the centroid and orientation overlaid on the original video. This will also plot the likelihood of each bodypart used in the cohort. This is optional, but a good quality assurance step.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideoParams.insert_default()\n</pre> sgp.DLCPosVideoParams.insert_default() In\u00a0[\u00a0]: Copied! <pre>params = {\n    \"percent_frames\": 0.05,\n    \"incl_likelihood\": True,\n}\nsgp.DLCPosVideoParams.insert1(\n    {\"dlc_pos_video_params_name\": \"five_percent\", \"params\": params},\n    skip_duplicates=True,\n)\n</pre> params = {     \"percent_frames\": 0.05,     \"incl_likelihood\": True, } sgp.DLCPosVideoParams.insert1(     {\"dlc_pos_video_params_name\": \"five_percent\", \"params\": params},     skip_duplicates=True, ) In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideoSelection.insert1(\n    {**dlc_key, \"dlc_pos_video_params_name\": \"five_percent\"},\n    skip_duplicates=True,\n)\n</pre> sgp.DLCPosVideoSelection.insert1(     {**dlc_key, \"dlc_pos_video_params_name\": \"five_percent\"},     skip_duplicates=True, ) In\u00a0[\u00a0]: Copied! <pre>sgp.DLCPosVideo().populate(dlc_key)\n</pre> sgp.DLCPosVideo().populate(dlc_key) <p><code>PositionOutput</code> is the final table of the pipeline and is automatically populated when we populate <code>DLCPosV1</code></p> In\u00a0[\u00a0]: Copied! <pre>sgp.PositionOutput.merge_get_part(dlc_key)\n</pre> sgp.PositionOutput.merge_get_part(dlc_key) <p><code>PositionOutput</code> also has a part table, similar to the <code>DLCModelSource</code> table above. Let's check that out as well.</p> In\u00a0[\u00a0]: Copied! <pre>PositionOutput.DLCPosV1() &amp; dlc_key\n</pre> PositionOutput.DLCPosV1() &amp; dlc_key In\u00a0[\u00a0]: Copied! <pre>(PositionOutput.DLCPosV1() &amp; dlc_key).fetch1_dataframe()\n</pre> (PositionOutput.DLCPosV1() &amp; dlc_key).fetch1_dataframe() <p>We can use the <code>PositionVideo</code> table to create a video that overlays just the centroid and orientation on the video. This table uses the parameter <code>plot</code> to determine whether to plot the entry deriving from the DLC arm or from the Trodes arm of the position pipeline. This parameter also accepts 'all', which will plot both (if they exist) in order to compare results.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.PositionVideoSelection().insert1(\n    {\n        \"nwb_file_name\": \"J1620210604_.nwb\",\n        \"interval_list_name\": \"pos 13 valid times\",\n        \"trodes_position_id\": 0,\n        \"dlc_position_id\": 1,\n        \"plot\": \"DLC\",\n        \"output_dir\": \"/home/dgramling/Src/\",\n    }\n)\n</pre> sgp.PositionVideoSelection().insert1(     {         \"nwb_file_name\": \"J1620210604_.nwb\",         \"interval_list_name\": \"pos 13 valid times\",         \"trodes_position_id\": 0,         \"dlc_position_id\": 1,         \"plot\": \"DLC\",         \"output_dir\": \"/home/dgramling/Src/\",     } ) In\u00a0[\u00a0]: Copied! <pre>sgp.PositionVideo.populate({\"plot\": \"DLC\"})\n</pre> sgp.PositionVideo.populate({\"plot\": \"DLC\"}) Return To Table of Contents"}, {"location": "notebooks/21_DLC/#position-deeplabcut-from-scratch", "title": "Position- DeepLabCut from Scratch\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#table-of-contents", "title": "Table of Contents\u00b6", "text": "<p><code>DLCProject</code> <code>DLCModelTraining</code> <code>DLCModel</code> <code>DLCPoseEstimation</code> <code>DLCSmoothInterp</code> <code>DLCCentroid</code> <code>DLCOrientation</code> <code>DLCPosV1</code> <code>DLCPosVideo</code> <code>PositionOutput</code></p>"}, {"location": "notebooks/21_DLC/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcproject", "title": "DLCProject \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#body-parts", "title": "Body Parts\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#define-videos-and-camera-name-optional-for-training-set", "title": "Define videos and camera name (optional) for training set\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#path-variables", "title": "Path variables\u00b6", "text": "<p>The position pipeline also keeps track of paths for project, video, and output. Just like we saw in Setup, you can manage these either with environmental variables...</p> <pre>export DLC_PROJECT_DIR=\"/nimbus/deeplabcut/projects\"\nexport DLC_VIDEO_DIR=\"/nimbus/deeplabcut/video\"\nexport DLC_OUTPUT_DIR=\"/nimbus/deeplabcut/output\"\n</pre> <p>Or these can be set in your datajoint config:</p> <pre>{\n  \"custom\": {\n    \"dlc_dirs\": {\n      \"base\": \"/nimbus/deeplabcut/\",\n      \"project\": \"/nimbus/deeplabcut/projects\",\n      \"video\": \"/nimbus/deeplabcut/video\",\n      \"output\": \"/nimbus/deeplabcut/output\"\n    }\n  }\n}\n</pre> <p>NOTE: If only <code>base</code> is specified as shown above, spyglass will assume the relative directories shown.</p> <p>You can check the result of this setup process with...</p>"}, {"location": "notebooks/21_DLC/#dlcmodeltraining", "title": "DLCModelTraining\u00b6", "text": "<p>Please make sure you're running this notebook on a GPU-enabled machine.</p> <p>Now that we've imported existing frames, we can get ready to train our model.</p> <p>First, we'll need to define a set of parameters for <code>DLCModelTrainingParams</code>, which will get used by DeepLabCut during training. Let's start with <code>gputouse</code>, which determines which GPU core to use.</p> <p>The cell below determines which core has space and set the <code>gputouse</code> variable accordingly.</p>"}, {"location": "notebooks/21_DLC/#dlcmodel", "title": "DLCModel \u00b6", "text": "<p>Next we'll populate the <code>DLCModel</code> table, which holds all the relevant information for all trained models.</p> <p>First, we'll need to determine a set of parameters for our model to select the correct model file. Here is the default:</p>"}, {"location": "notebooks/21_DLC/#dlcposeestimation", "title": "DLCPoseEstimation \u00b6", "text": "<p>Alright, now that we've trained model and populated the <code>DLCModel</code> table, we're ready to set-up Pose Estimation on a behavioral video of your choice.For this tutorial, you can choose to use an epoch of your choice, we can also use the one specified below. If you'd like to use your own video, just specify the <code>nwb_file_name</code> and <code>epoch</code> number and make sure it's in the <code>VideoFile</code> table!</p>"}, {"location": "notebooks/21_DLC/#dlcsmoothinterp", "title": "DLCSmoothInterp \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcsmoothinterpcohort", "title": "DLCSmoothInterpCohort \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlccentroid", "title": "DLCCentroid \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcorientation", "title": "DLCOrientation \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcposv1", "title": "DLCPosV1 \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#dlcposvideo", "title": "DLCPosVideo \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#positionoutput", "title": "PositionOutput \u00b6", "text": ""}, {"location": "notebooks/21_DLC/#positionvideo", "title": "PositionVideo\u00b6", "text": ""}, {"location": "notebooks/21_DLC/#congratulations", "title": "CONGRATULATIONS!!\u00b6", "text": "<p>Please treat yourself to a nice tea break :-)</p>"}, {"location": "notebooks/22_DLC_Loop/", "title": "Looping DLC", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This tutorial will extract position via DeepLabCut (DLC). It will walk through...</p> <ul> <li>creating a DLC project</li> <li>extracting and labeling frames</li> <li>training your model</li> <li>executing pose estimation on a novel behavioral video</li> <li>processing the pose estimation output to extract a centroid and orientation</li> <li>inserting the resulting information into the <code>PositionOutput</code> table</li> </ul> <p>Note 2: Make sure you are running this within the spyglass-position Conda environment (instructions for install are in the environment_position.yml)</p> <p>Here is a schematic showing the tables used in this pipeline.</p> <p></p> <p>You can click on any header to return to the Table of Contents</p> In\u00a0[1]: Copied! <pre>%load_ext autoreload\n%autoreload 2\n</pre> %load_ext autoreload %autoreload 2 In\u00a0[2]: Copied! <pre>import os\nimport datajoint as dj\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\n\nimport numpy as np\nimport pandas as pd\nimport pynwb\nfrom spyglass.position import PositionOutput\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj  import spyglass.common as sgc import spyglass.position.v1 as sgp  import numpy as np import pandas as pd import pynwb from spyglass.position import PositionOutput  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-18 10:12:13,219][INFO]: Connecting ebroyles@lmf-db.cin.ucsf.edu:3306\n[2024-01-18 10:12:13,255][INFO]: Connected ebroyles@lmf-db.cin.ucsf.edu:3306\nOMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n</pre> Notes:<ul> <li>         The cells within this <code>DLCProject</code> step need to be performed          in a local Jupyter notebook to allow for use of the frame labeling GUI.     </li> <li>         Please do not add to the <code>BodyPart</code> table in the production          database unless necessary.     </li> </ul> <p>We'll begin by looking at the <code>BodyPart</code> table, which stores standard names of body parts used in DLC models throughout the lab with a concise description.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.BodyPart()\n</pre> sgp.BodyPart() <p>If the bodyparts you plan to use in your model are not yet in the table, here is code to add bodyparts:</p> <pre>sgp.BodyPart.insert(\n    [\n        {\"bodypart\": \"bp_1\", \"bodypart_description\": \"concise descrip\"},\n        {\"bodypart\": \"bp_2\", \"bodypart_description\": \"concise descrip\"},\n    ],\n    skip_duplicates=True,\n)\n</pre> <p>To train a model, we'll need to extract frames, which we can label as training data. We can construct a list of videos from which we'll extract frames.</p> <p>The list can either contain dictionaries identifying behavioral videos for NWB files that have already been added to Spyglass, or absolute file paths to the videos you want to use.</p> <p>For this tutorial, we'll use two videos for which we already have frames labeled.</p> <p>Defining camera name is optional: it should be done in cases where there are multiple cameras streaming per epoch, but not necessary otherwise.  example: <code>camera_name = \"HomeBox_camera\"</code></p> <p>NOTE: The official release of Spyglass does not yet support multicamera projects. You can monitor progress on the effort to add this feature by checking this PR or use this experimental branch, which takes the keys nwb_file_name and epoch, and camera_name in the video_list variable.</p> In\u00a0[\u00a0]: Copied! <pre>video_list = [\n    {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},\n    {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4},\n]\n</pre> video_list = [     {\"nwb_file_name\": \"J1620210529_.nwb\", \"epoch\": 2},     {\"nwb_file_name\": \"peanut20201103_.nwb\", \"epoch\": 4}, ] In\u00a0[\u00a0]: Copied! <pre>from spyglass.settings import config\n\nconfig\n</pre> from spyglass.settings import config  config <p>Before creating our project, we need to define a few variables.</p> <ul> <li>A team name, as shown in <code>LabTeam</code> for setting permissions. Here, we'll use \"LorenLab\".</li> <li>A <code>project_name</code>, as a unique identifier for this DLC project. Here, we'll use \"tutorial_scratch_yourinitials\"</li> <li><code>bodyparts</code> is a list of body parts for which we want to extract position. The pre-labeled frames we're using include the bodyparts listed below.</li> <li>Number of frames to extract/label as <code>frames_per_video</code>. Note that the DLC creators recommend having 200 frames as the minimum total number for each project.</li> </ul> In\u00a0[\u00a0]: Copied! <pre>team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\"\nproject_name = \"tutorial_scratch_DG\"\nframes_per_video = 100\nbodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"]\nproject_key = sgp.DLCProject.insert_new_project(\n    project_name=project_name,\n    bodyparts=bodyparts,\n    lab_team=team_name,\n    frames_per_video=frames_per_video,\n    video_list=video_list,\n    skip_duplicates=True,\n)\n</pre> team_name = sgc.LabTeam.fetch(\"team_name\")[0]  # If on lab DB, \"LorenLab\" project_name = \"tutorial_scratch_DG\" frames_per_video = 100 bodyparts = [\"redLED_C\", \"greenLED\", \"redLED_L\", \"redLED_R\", \"tailBase\"] project_key = sgp.DLCProject.insert_new_project(     project_name=project_name,     bodyparts=bodyparts,     lab_team=team_name,     frames_per_video=frames_per_video,     video_list=video_list,     skip_duplicates=True, ) <p>Now that we've initialized our project we'll need to extract frames which we will then label.</p> In\u00a0[\u00a0]: Copied! <pre># comment this line out after you finish frame extraction for each project\nsgp.DLCProject().run_extract_frames(project_key)\n</pre> # comment this line out after you finish frame extraction for each project sgp.DLCProject().run_extract_frames(project_key) <p>This is the line used to label the frames you extracted, if you wish to use the DLC GUI on the computer you are currently using.</p> <pre><code>#comment\nsgp.DLCProject().run_label_frames(project_key)\n</code></pre> <p>Otherwise, it is best/easiest practice to label the frames on your local computer (like a MacBook) that can run DeepLabCut's GUI well. Instructions: </p> <ol> <li>Install DLC on your local (preferably into a 'Src' folder): https://deeplabcut.github.io/DeepLabCut/docs/installation.html</li> <li>Upload frames extracted and saved in nimbus (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/labeled-data</code>) AND the project's associated config file (should be <code>/nimbus/deeplabcut/&lt;YOUR_PROJECT_NAME&gt;/config.yaml</code>) to Box (we get free with UCSF)</li> <li>Download labeled-data and config files on your local from Box</li> <li>Create a 'projects' folder where you installed DeepLabCut; create a new folder with your complete project name there; save the downloaded files there.</li> <li>Edit the config.yaml file: line 9 defining <code>project_path</code> needs to be the file path where it is saved on your local (ex: <code>/Users/lorenlab/Src/DeepLabCut/projects/tutorial_sratch_DG-LorenLab-2023-08-16</code>)</li> <li>Open the DLC GUI through terminal (ex: <code>conda activate miniconda/envs/DEEPLABCUT_M1</code> <code>pythonw -m deeplabcut</code>)</li> <li>Load an existing project; choose the config.yaml file</li> <li>Label frames; labeling tutorial: https://www.youtube.com/watch?v=hsA9IB5r73E.</li> <li>Once all frames are labeled, you should re-upload labeled-data folder back to Box and overwrite it in the original nimbus location so that your completed frames are ready to be used in the model.</li> </ol> <p>Now we can check the <code>DLCProject.File</code> part table and see all of our training files and videos there!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCProject.File &amp; project_key\n</pre> sgp.DLCProject.File &amp; project_key      This step and beyond should be run on a GPU-enabled machine.  In\u00a0[\u00a0]: Copied! <pre>sgp.dlc_utils.get_gpu_memory()\n</pre> sgp.dlc_utils.get_gpu_memory() <p>Set GPU core:</p> In\u00a0[\u00a0]: Copied! <pre>gputouse = 1  # 1-9\n</pre> gputouse = 1  # 1-9 <p>Now we'll define the rest of our parameters and insert the entry.</p> <p>To see all possible parameters, try:</p> <pre>sgp.DLCModelTrainingParams.get_accepted_params()\n</pre> In\u00a0[\u00a0]: Copied! <pre>training_params_name = \"tutorial\"\nsgp.DLCModelTrainingParams.insert_new_params(\n    paramset_name=training_params_name,\n    params={\n        \"trainingsetindex\": 0,\n        \"shuffle\": 1,\n        \"gputouse\": gputouse,\n        \"net_type\": \"resnet_50\",\n        \"augmenter_type\": \"imgaug\",\n    },\n    skip_duplicates=True,\n)\n</pre> training_params_name = \"tutorial\" sgp.DLCModelTrainingParams.insert_new_params(     paramset_name=training_params_name,     params={         \"trainingsetindex\": 0,         \"shuffle\": 1,         \"gputouse\": gputouse,         \"net_type\": \"resnet_50\",         \"augmenter_type\": \"imgaug\",     },     skip_duplicates=True, ) <p>Next we'll modify the <code>project_key</code> from above to include the necessary entries for <code>DLCModelTraining</code></p> In\u00a0[\u00a0]: Copied! <pre># project_key['project_path'] = os.path.dirname(project_key['config_path'])\nif \"config_path\" in project_key:\n    del project_key[\"config_path\"]\n</pre> # project_key['project_path'] = os.path.dirname(project_key['config_path']) if \"config_path\" in project_key:     del project_key[\"config_path\"] <p>We can insert an entry into <code>DLCModelTrainingSelection</code> and populate <code>DLCModelTraining</code>.</p> <p>Note: You can stop training at any point using <code>I + I</code> or interrupt the Kernel.</p> <p>The maximum total number of training iterations is 1030000; you can end training before this amount if the loss rate (lr) and total loss plateau and are very close to 0.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTrainingSelection.heading\n</pre> sgp.DLCModelTrainingSelection.heading In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTrainingSelection().insert1(\n    {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n        \"training_id\": 0,\n        \"model_prefix\": \"\",\n    }\n)\nmodel_training_key = (\n    sgp.DLCModelTrainingSelection\n    &amp; {\n        **project_key,\n        \"dlc_training_params_name\": training_params_name,\n    }\n).fetch1(\"KEY\")\nsgp.DLCModelTraining.populate(model_training_key)\n</pre> sgp.DLCModelTrainingSelection().insert1(     {         **project_key,         \"dlc_training_params_name\": training_params_name,         \"training_id\": 0,         \"model_prefix\": \"\",     } ) model_training_key = (     sgp.DLCModelTrainingSelection     &amp; {         **project_key,         \"dlc_training_params_name\": training_params_name,     } ).fetch1(\"KEY\") sgp.DLCModelTraining.populate(model_training_key) <p>Here we'll make sure that the entry made it into the table properly!</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelTraining() &amp; model_training_key\n</pre> sgp.DLCModelTraining() &amp; model_training_key <p>Populating <code>DLCModelTraining</code> automatically inserts the entry into <code>DLCModelSource</code>, which is used to select between models trained using Spyglass vs. other tools.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource() &amp; model_training_key\n</pre> sgp.DLCModelSource() &amp; model_training_key <p>The <code>source</code> field will only accept \"FromImport\" or \"FromUpstream\" as entries. Let's checkout the <code>FromUpstream</code> part table attached to <code>DLCModelSource</code> below.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelSource.FromUpstream() &amp; model_training_key\n</pre> sgp.DLCModelSource.FromUpstream() &amp; model_training_key In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModelParams.get_default()\n</pre> sgp.DLCModelParams.get_default() <p>Here is the syntax to add your own parameter set:</p> <pre>dlc_model_params_name = \"make_this_yours\"\nparams = {\n    \"params\": {},\n    \"shuffle\": 1,\n    \"trainingsetindex\": 0,\n    \"model_prefix\": \"\",\n}\nsgp.DLCModelParams.insert1(\n    {\"dlc_model_params_name\": dlc_model_params_name, \"params\": params},\n    skip_duplicates=True,\n)\n</pre> <p>We can insert sets of parameters into <code>DLCModelSelection</code> and populate <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\")\n</pre> temp_model_key = (sgp.DLCModelSource &amp; model_training_key).fetch1(\"KEY\") In\u00a0[\u00a0]: Copied! <pre># comment these lines out after successfully inserting, for each project\nsgp.DLCModelSelection().insert1(\n    {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True\n)\n</pre> # comment these lines out after successfully inserting, for each project sgp.DLCModelSelection().insert1(     {**temp_model_key, \"dlc_model_params_name\": \"default\"}, skip_duplicates=True ) In\u00a0[\u00a0]: Copied! <pre>model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\")\nsgp.DLCModel.populate(model_key)\n</pre> model_key = (sgp.DLCModelSelection &amp; temp_model_key).fetch1(\"KEY\") sgp.DLCModel.populate(model_key) <p>Again, let's make sure that everything looks correct in <code>DLCModel</code>.</p> In\u00a0[\u00a0]: Copied! <pre>sgp.DLCModel() &amp; model_key\n</pre> sgp.DLCModel() &amp; model_key <p>We can view all <code>VideoFile</code> entries with the specidied <code>camera_ name</code> for this project to ensure the rat whose position you wish to model is in this table <code>matching_rows</code></p> In\u00a0[\u00a0]: Copied! <pre>camera_name = \"SleepBox_camera\"\nmatching_rows = sgc.VideoFile() &amp; {\"camera_name\": camera_name}\nmatching_rows\n</pre> camera_name = \"SleepBox_camera\" matching_rows = sgc.VideoFile() &amp; {\"camera_name\": camera_name} matching_rows <p>The <code>DLCPoseEstimationSelection</code> insertion step will convert your .h264 video to an .mp4 first and save it in <code>/nimbus/deeplabcut/video</code>. If this video already exists here, the insertion will never complete.</p> <p>We first delete any .mp4 that exists for this video from the nimbus folder:</p> In\u00a0[\u00a0]: Copied! <pre>! find /nimbus/deeplabcut/video -type f -name '*20230606_SC38*' -delete # change based on date and rat with which you are training the model\n</pre> ! find /nimbus/deeplabcut/video -type f -name '*20230606_SC38*' -delete # change based on date and rat with which you are training the model <p>If the first insertion step (for pose estimation task) fails in either trigger or load mode for an epoch, run the following lines:</p> <pre><code>(pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"epoch\": epoch,\n        \"video_file_num\": video_file_num,\n        **model_key,\n    }).delete()\n</code></pre> <p>This loop will generate posiiton data for all epochs associated with the pre-defined camera in one day, for one rat (based on the NWB file; see ***) The output should print Pose Estimation and Centroid plots for each epoch.</p> <ul> <li>It defines <code>col1val</code> as each <code>nwb_file_name</code> entry in the table, one at a time.</li> <li>Next, it sees if the trial on which you are testing this model is in the string for the current <code>col1val</code>; if not, it re-defines <code>col1val</code> as the next <code>nwb_file_name</code> entry and re-tries this step.</li> <li>If the previous step works, it then saves <code>col2val</code> and <code>col3val</code> as the <code>epoch</code> and the <code>video_file_num</code>, respectively, based on the nwb_file_name. From there, it iterates through the insertion and population steps required to extract position data, which we see laid out in notebook 05_DLC.ipynb.</li> </ul> In\u00a0[\u00a0]: Copied! <pre>for row in matching_rows:\n    col1val = row[\"nwb_file_name\"]\n    if \"SC3820230606\" in col1val:  # *** change depending on rat/day!!!\n        col2val = row[\"epoch\"]\n        col3val = row[\"video_file_num\"]\n\n        ##insert pose estimation task\n        pose_estimation_key = (\n            sgp.DLCPoseEstimationSelection.insert_estimation_task(\n                {\n                    \"nwb_file_name\": col1val,\n                    \"epoch\": col2val,\n                    \"video_file_num\": col3val,\n                    **model_key,\n                },\n                task_mode=\"trigger\",  # load or trigger\n                params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},\n            )\n        )\n\n        ##populate DLC Pose Estimation\n        sgp.DLCPoseEstimation().populate(pose_estimation_key)\n\n        ##start smooth interpolation\n        si_params_name = \"just_nan\"\n        si_key = pose_estimation_key.copy()\n        fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())\n        si_key = {key: val for key, val in si_key.items() if key in fields}\n        bodyparts = [\"greenLED\", \"redLED_C\"]\n        sgp.DLCSmoothInterpSelection.insert(\n            [\n                {\n                    **si_key,\n                    \"bodypart\": bodypart,\n                    \"dlc_si_params_name\": si_params_name,\n                }\n                for bodypart in bodyparts\n            ],\n            skip_duplicates=True,\n        )\n        sgp.DLCSmoothInterp().populate(si_key)\n        (\n            sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}\n        ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))\n\n        ##smoothinterpcohort\n        cohort_key = si_key.copy()\n        if \"bodypart\" in cohort_key:\n            del cohort_key[\"bodypart\"]\n        if \"dlc_si_params_name\" in cohort_key:\n            del cohort_key[\"dlc_si_params_name\"]\n        cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"\n        cohort_key[\"bodyparts_params_dict\"] = {\n            \"greenLED\": si_params_name,\n            \"redLED_C\": si_params_name,\n        }\n        sgp.DLCSmoothInterpCohortSelection().insert1(\n            cohort_key, skip_duplicates=True\n        )\n        sgp.DLCSmoothInterpCohort.populate(cohort_key)\n\n        ##DLC Centroid\n        centroid_params_name = \"default\"\n        centroid_key = cohort_key.copy()\n        fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())\n        centroid_key = {\n            key: val for key, val in centroid_key.items() if key in fields\n        }\n        centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name\n        sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)\n        sgp.DLCCentroid.populate(centroid_key)\n        (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(\n            x=\"position_x\",\n            y=\"position_y\",\n            c=\"speed\",\n            colormap=\"viridis\",\n            alpha=0.5,\n            s=0.5,\n            figsize=(10, 10),\n        )\n\n        ##DLC Orientation\n        dlc_orientation_params_name = \"default\"\n        fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())\n        orient_key = {\n            key: val for key, val in cohort_key.items() if key in fields\n        }\n        orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name\n        sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)\n        sgp.DLCOrientation().populate(orient_key)\n\n        ##DLCPosV1\n        fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())\n        dlc_key = {\n            key: val for key, val in centroid_key.items() if key in fields\n        }\n        dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[\n            \"dlc_si_cohort_selection_name\"\n        ]\n        dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[\n            \"dlc_si_cohort_selection_name\"\n        ]\n        dlc_key[\"dlc_orientation_params_name\"] = orient_key[\n            \"dlc_orientation_params_name\"\n        ]\n        sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)\n        sgp.DLCPosV1().populate(dlc_key)\n\n    else:\n        continue\n</pre> for row in matching_rows:     col1val = row[\"nwb_file_name\"]     if \"SC3820230606\" in col1val:  # *** change depending on rat/day!!!         col2val = row[\"epoch\"]         col3val = row[\"video_file_num\"]          ##insert pose estimation task         pose_estimation_key = (             sgp.DLCPoseEstimationSelection.insert_estimation_task(                 {                     \"nwb_file_name\": col1val,                     \"epoch\": col2val,                     \"video_file_num\": col3val,                     **model_key,                 },                 task_mode=\"trigger\",  # load or trigger                 params={\"gputouse\": gputouse, \"videotype\": \"mp4\"},             )         )          ##populate DLC Pose Estimation         sgp.DLCPoseEstimation().populate(pose_estimation_key)          ##start smooth interpolation         si_params_name = \"just_nan\"         si_key = pose_estimation_key.copy()         fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())         si_key = {key: val for key, val in si_key.items() if key in fields}         bodyparts = [\"greenLED\", \"redLED_C\"]         sgp.DLCSmoothInterpSelection.insert(             [                 {                     **si_key,                     \"bodypart\": bodypart,                     \"dlc_si_params_name\": si_params_name,                 }                 for bodypart in bodyparts             ],             skip_duplicates=True,         )         sgp.DLCSmoothInterp().populate(si_key)         (             sgp.DLCSmoothInterp() &amp; {**si_key, \"bodypart\": bodyparts[0]}         ).fetch1_dataframe().plot.scatter(x=\"x\", y=\"y\", s=1, figsize=(5, 5))          ##smoothinterpcohort         cohort_key = si_key.copy()         if \"bodypart\" in cohort_key:             del cohort_key[\"bodypart\"]         if \"dlc_si_params_name\" in cohort_key:             del cohort_key[\"dlc_si_params_name\"]         cohort_key[\"dlc_si_cohort_selection_name\"] = \"green_red_led\"         cohort_key[\"bodyparts_params_dict\"] = {             \"greenLED\": si_params_name,             \"redLED_C\": si_params_name,         }         sgp.DLCSmoothInterpCohortSelection().insert1(             cohort_key, skip_duplicates=True         )         sgp.DLCSmoothInterpCohort.populate(cohort_key)          ##DLC Centroid         centroid_params_name = \"default\"         centroid_key = cohort_key.copy()         fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())         centroid_key = {             key: val for key, val in centroid_key.items() if key in fields         }         centroid_key[\"dlc_centroid_params_name\"] = centroid_params_name         sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)         sgp.DLCCentroid.populate(centroid_key)         (sgp.DLCCentroid() &amp; centroid_key).fetch1_dataframe().plot.scatter(             x=\"position_x\",             y=\"position_y\",             c=\"speed\",             colormap=\"viridis\",             alpha=0.5,             s=0.5,             figsize=(10, 10),         )          ##DLC Orientation         dlc_orientation_params_name = \"default\"         fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())         orient_key = {             key: val for key, val in cohort_key.items() if key in fields         }         orient_key[\"dlc_orientation_params_name\"] = dlc_orientation_params_name         sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)         sgp.DLCOrientation().populate(orient_key)          ##DLCPosV1         fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())         dlc_key = {             key: val for key, val in centroid_key.items() if key in fields         }         dlc_key[\"dlc_si_cohort_centroid\"] = centroid_key[             \"dlc_si_cohort_selection_name\"         ]         dlc_key[\"dlc_si_cohort_orientation\"] = orient_key[             \"dlc_si_cohort_selection_name\"         ]         dlc_key[\"dlc_orientation_params_name\"] = orient_key[             \"dlc_orientation_params_name\"         ]         sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)         sgp.DLCPosV1().populate(dlc_key)      else:         continue Return To Table of Contents"}, {"location": "notebooks/22_DLC_Loop/#position-deeplabcut-from-scratch", "title": "Position- DeepLabCut from Scratch\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#table-of-contents", "title": "Table of Contents\u00b6", "text": "<p><code>DLCProject</code> <code>DLCModelTraining</code> <code>DLCModel</code> <code>DLCPoseEstimation</code> <code>DLCSmoothInterp</code> <code>DLCCentroid</code> <code>DLCOrientation</code> <code>DLCPosV1</code> <code>DLCPosVideo</code> <code>PositionOutput</code></p>"}, {"location": "notebooks/22_DLC_Loop/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#dlcproject", "title": "DLCProject \u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#body-parts", "title": "Body Parts\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#define-videos-and-camera-name-optional-for-training-set", "title": "Define videos and camera name (optional) for training set\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#path-variables", "title": "Path variables\u00b6", "text": "<p>The position pipeline also keeps track of paths for project, video, and output. Just like we saw in Setup, you can manage these either with environmental variables...</p> <pre>export DLC_PROJECT_DIR=\"/nimbus/deeplabcut/projects\"\nexport DLC_VIDEO_DIR=\"/nimbus/deeplabcut/video\"\nexport DLC_OUTPUT_DIR=\"/nimbus/deeplabcut/output\"\n</pre> <p>Or these can be set in your datajoint config:</p> <pre>{\n  \"custom\": {\n    \"dlc_dirs\": {\n      \"base\": \"/nimbus/deeplabcut/\",\n      \"project\": \"/nimbus/deeplabcut/projects\",\n      \"video\": \"/nimbus/deeplabcut/video\",\n      \"output\": \"/nimbus/deeplabcut/output\"\n    }\n  }\n}\n</pre> <p>NOTE: If only <code>base</code> is specified as shown above, spyglass will assume the relative directories shown.</p> <p>You can check the result of this setup process with...</p>"}, {"location": "notebooks/22_DLC_Loop/#dlcmodeltraining", "title": "DLCModelTraining\u00b6", "text": "<p>Please make sure you're running this notebook on a GPU-enabled machine.</p> <p>Now that we've imported existing frames, we can get ready to train our model.</p> <p>First, we'll need to define a set of parameters for <code>DLCModelTrainingParams</code>, which will get used by DeepLabCut during training. Let's start with <code>gputouse</code>, which determines which GPU core to use.</p> <p>The cell below determines which core has space and set the <code>gputouse</code> variable accordingly.</p>"}, {"location": "notebooks/22_DLC_Loop/#dlcmodel", "title": "DLCModel \u00b6", "text": "<p>Next we'll populate the <code>DLCModel</code> table, which holds all the relevant information for all trained models.</p> <p>First, we'll need to determine a set of parameters for our model to select the correct model file. Here is the default:</p>"}, {"location": "notebooks/22_DLC_Loop/#loop-begins", "title": "Loop Begins\u00b6", "text": ""}, {"location": "notebooks/22_DLC_Loop/#congratulations", "title": "CONGRATULATIONS!!\u00b6", "text": "<p>Please treat yourself to a nice tea break :-)</p>"}, {"location": "notebooks/24_Linearization/", "title": "Linearization", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>This pipeline takes 2D position data from the <code>PositionOutput</code> table and \"linearizes\" it to 1D position. If you haven't already done so, please generate input data with either the Trodes or DLC notebooks (1, 2, 3).</p> In\u00a0[1]: Copied! <pre>%reload_ext autoreload\n%autoreload 2\n</pre> %reload_ext autoreload %autoreload 2 In\u00a0[2]: Copied! <pre>import os\nimport pynwb\nimport numpy as np\nimport datajoint as dj\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\nimport spyglass.linearization.v1 as sgpl\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import pynwb import numpy as np import datajoint as dj import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position.v1 as sgp import spyglass.linearization.v1 as sgpl  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-24 09:19:28,736][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-01-24 09:19:28,767][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n[09:19:30][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> <p>To retrieve 2D position data, we'll specify an nwb file, a position time interval, and the set of parameters used to compute the position info.</p> In\u00a0[3]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n\nnwb_file_name = \"minirec20230622.nwb\"  # detailed data: chimi20200216_new.nwb\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nnwb_copy_file_name\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename  nwb_file_name = \"minirec20230622.nwb\"  # detailed data: chimi20200216_new.nwb nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) nwb_copy_file_name Out[3]: <pre>'chimi20200216_new_.nwb'</pre> <p>We will fetch the pandas dataframe from the <code>PositionOutput</code> table.</p> In\u00a0[4]: Copied! <pre>from spyglass.position import PositionOutput\nimport pandas as pd\n\npos_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",  # For chimi, \"pos 1 valid times\"\n    \"trodes_pos_params_name\": \"single_led_upsampled\",  # For chimi, \"default\"\n}\n\n# Note: You'll have to change the part table to the one where your data came from\nmerge_id = (PositionOutput.TrodesPosV1() &amp; pos_key).fetch1(\"merge_id\")\nposition_info = (PositionOutput &amp; {\"merge_id\": merge_id}).fetch1_dataframe()\nposition_info\n</pre> from spyglass.position import PositionOutput import pandas as pd  pos_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",  # For chimi, \"pos 1 valid times\"     \"trodes_pos_params_name\": \"single_led_upsampled\",  # For chimi, \"default\" }  # Note: You'll have to change the part table to the one where your data came from merge_id = (PositionOutput.TrodesPosV1() &amp; pos_key).fetch1(\"merge_id\") position_info = (PositionOutput &amp; {\"merge_id\": merge_id}).fetch1_dataframe() position_info Out[4]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.581887e+09 0 91.051650 211.127050 2.999696 1.387074 2.848838 3.168573 1.581887e+09 1 90.844337 211.417287 3.078386 3.123201 3.411111 4.624939 1.581887e+09 2 90.637025 211.707525 -3.114572 5.431643 4.089597 6.799085 1.581887e+09 3 90.802875 211.596958 -3.033109 8.097753 4.979262 9.506138 1.581887e+09 4 91.288579 211.482443 -3.062550 10.840482 6.071373 12.424880 ... ... ... ... ... ... ... ... 1.581888e+09 39335 182.158583 201.452467 -0.986926 0.348276 0.218575 0.411182 1.581888e+09 39336 182.158583 201.397183 -0.978610 0.279135 -0.058413 0.285182 1.581888e+09 39337 182.213867 201.341900 -0.957589 0.193798 -0.283200 0.343162 1.581888e+09 39338 182.158583 201.341900 -0.970083 0.110838 -0.417380 0.431846 1.581888e+09 39339 182.158583 201.286617 -0.936414 0.045190 -0.453966 0.456209 <p>39340 rows \u00d7 7 columns</p> <p>Before linearizing, plotting the head position will help us understand the data.</p> In\u00a0[5]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(5, 5))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\nax.set_title(\"Head Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(5, 5)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\", ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) ax.set_title(\"Head Position\", fontsize=28) Out[5]: <pre>Text(0.5, 1.0, 'Head Position')</pre> <p>To linearize, we need a graph of nodes and edges to represent track geometry in the <code>TrackGraph</code> table with four variables:</p> <ul> <li><code>node_positions</code> (cm): the 2D positions of the graph</li> <li><code>edges</code>: how the nodes are connected, as pairs of node indices, labeled by their respective index in <code>node_positions</code>.</li> <li><code>linear_edge_order</code>: layout of edges in linear space in order, as tuples.</li> <li><code>linear_edge_spacing</code>: spacing between each edge, as either a single number for all gaps or an array with a number for each gap. Gaps may be important for edges not connected in 2D space.</li> </ul> <p>For example, (79.910, 216.720) is the 2D position of node 0 and (183.784, 45.375) is the 2D position of node 8. Edge (0, 8) means there is an edge between node 0 and node 8. Nodes order controls order in 1D space. Edge (0, 1) connects from node 0 to 1. Edge (1, 0) would connect from node 1 to 0, reversing the linear positions for that edge.</p> <p>For more examples, see this notebook.</p> In\u00a0[6]: Copied! <pre>node_positions = np.array(\n    [\n        (79.910, 216.720),  # top left well 0\n        (132.031, 187.806),  # top middle intersection 1\n        (183.718, 217.713),  # top right well 2\n        (132.544, 132.158),  # middle intersection 3\n        (87.202, 101.397),  # bottom left intersection 4\n        (31.340, 126.110),  # middle left well 5\n        (180.337, 104.799),  # middle right intersection 6\n        (92.693, 42.345),  # bottom left well 7\n        (183.784, 45.375),  # bottom right well 8\n        (231.338, 136.281),  # middle right well 9\n    ]\n)\n\nedges = np.array(\n    [\n        (0, 1),\n        (1, 2),\n        (1, 3),\n        (3, 4),\n        (4, 5),\n        (3, 6),\n        (6, 9),\n        (4, 7),\n        (6, 8),\n    ]\n)\n\nlinear_edge_order = [\n    (3, 6),\n    (6, 8),\n    (6, 9),\n    (3, 1),\n    (1, 2),\n    (1, 0),\n    (3, 4),\n    (4, 5),\n    (4, 7),\n]\nlinear_edge_spacing = 15\n</pre> node_positions = np.array(     [         (79.910, 216.720),  # top left well 0         (132.031, 187.806),  # top middle intersection 1         (183.718, 217.713),  # top right well 2         (132.544, 132.158),  # middle intersection 3         (87.202, 101.397),  # bottom left intersection 4         (31.340, 126.110),  # middle left well 5         (180.337, 104.799),  # middle right intersection 6         (92.693, 42.345),  # bottom left well 7         (183.784, 45.375),  # bottom right well 8         (231.338, 136.281),  # middle right well 9     ] )  edges = np.array(     [         (0, 1),         (1, 2),         (1, 3),         (3, 4),         (4, 5),         (3, 6),         (6, 9),         (4, 7),         (6, 8),     ] )  linear_edge_order = [     (3, 6),     (6, 8),     (6, 9),     (3, 1),     (1, 2),     (1, 0),     (3, 4),     (4, 5),     (4, 7), ] linear_edge_spacing = 15 <p>With these variables, we then add a <code>track_graph_name</code> and the corresponding <code>environment</code>.</p> In\u00a0[7]: Copied! <pre>sgpl.TrackGraph.insert1(\n    {\n        \"track_graph_name\": \"6 arm\",\n        \"environment\": \"6 arm\",\n        \"node_positions\": node_positions,\n        \"edges\": edges,\n        \"linear_edge_order\": linear_edge_order,\n        \"linear_edge_spacing\": linear_edge_spacing,\n    },\n    skip_duplicates=True,\n)\n\ngraph = sgpl.TrackGraph &amp; {\"track_graph_name\": \"6 arm\"}\ngraph\n</pre> sgpl.TrackGraph.insert1(     {         \"track_graph_name\": \"6 arm\",         \"environment\": \"6 arm\",         \"node_positions\": node_positions,         \"edges\": edges,         \"linear_edge_order\": linear_edge_order,         \"linear_edge_spacing\": linear_edge_spacing,     },     skip_duplicates=True, )  graph = sgpl.TrackGraph &amp; {\"track_graph_name\": \"6 arm\"} graph Out[7]: <p>track_graph_name</p> <p>environment</p> Type of Environment <p>node_positions</p> 2D position of track_graph nodes, shape (n_nodes, 2) <p>edges</p> shape (n_edges, 2) <p>linear_edge_order</p> order of track graph edges in the linear space, shape (n_edges, 2) <p>linear_edge_spacing</p> amount of space between edges in the linear space, shape (n_edges,) 6 arm 6 arm =BLOB= =BLOB= =BLOB= =BLOB= <p>Total: 1</p> <p><code>TrackGraph</code> has several methods for visualizing in 2D and 1D space. <code>plot_track_graph</code> plots in 2D to make sure our layout makes sense.</p> In\u00a0[8]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(5, 5))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n    alpha=0.7,\n    zorder=0,\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\ngraph.plot_track_graph(ax=ax)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(5, 5)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\",     alpha=0.7,     zorder=0, ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) graph.plot_track_graph(ax=ax) <p><code>plot_track_graph_as_1D</code> shows what this looks like in 1D.</p> In\u00a0[9]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 1))\ngraph.plot_track_graph_as_1D(ax=ax)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 1)) graph.plot_track_graph_as_1D(ax=ax) <p>By default, linearization assigns each 2D position to its nearest point on the track graph. This is then translated into 1D space.</p> <p>If <code>use_hmm</code> is set to <code>true</code>, a Hidden Markov model is used to assign points. The HMM takes into account the prior position and edge, and can keep the position from suddenly jumping to another. Position jumping like this may occur at intersections or the head position swings closer to a non-target reward well while on another edge.</p> In\u00a0[10]: Copied! <pre>sgpl.LinearizationParameters.insert1(\n    {\"linearization_param_name\": \"default\"}, skip_duplicates=True\n)\nsgpl.LinearizationParameters()\n</pre> sgpl.LinearizationParameters.insert1(     {\"linearization_param_name\": \"default\"}, skip_duplicates=True ) sgpl.LinearizationParameters() Out[10]: <p>linearization_param_name</p> name for this set of parameters <p>use_hmm</p> use HMM to determine linearization <p>route_euclidean_distance_scaling</p> <p>sensor_std_dev</p> Uncertainty of position sensor (in cm). <p>diagonal_bias</p> default 0 1.0 5.0 0.5 <p>Total: 1</p> <p>With linearization parameters, we specify the position interval we wish to linearize from the <code>PositionOutput</code> table and create an entry in <code>LinearizationSelection</code></p> In\u00a0[11]: Copied! <pre>sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> sgc.Session &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[11]: Table for holding experimental sessions. <p>nwb_file_name</p> name of the NWB file <p>subject_id</p> <p>institution_name</p> <p>lab_name</p> <p>session_id</p> <p>session_description</p> <p>session_start_time</p> <p>timestamps_reference_time</p> <p>experiment_description</p> chimi20200216_new_.nwb chimi University of California, San Francisco Loren Frank chimi_20200216 Spatial bandit task (regular) 2020-02-16 12:15:09 1970-01-01 00:00:00 Memory and value guided decision making <p>Total: 1</p> In\u00a0[12]: Copied! <pre>sgpl.LinearizationSelection.insert1(\n    {\n        \"pos_merge_id\": merge_id,\n        \"track_graph_name\": \"6 arm\",\n        \"linearization_param_name\": \"default\",\n    },\n    skip_duplicates=True,\n)\n\nsgpl.LinearizationSelection()\n</pre> sgpl.LinearizationSelection.insert1(     {         \"pos_merge_id\": merge_id,         \"track_graph_name\": \"6 arm\",         \"linearization_param_name\": \"default\",     },     skip_duplicates=True, )  sgpl.LinearizationSelection() Out[12]: <p>pos_merge_id</p> <p>track_graph_name</p> <p>linearization_param_name</p> name for this set of parameters af2e637c-47d9-8e27-11f1-3ac12431f417 6 arm default001b5af0-489e-5437-9e34-f2130727ebc5 ms_lineartrack default002476a5-c344-0236-d86c-89b995e22227 ms_lineartrack default00551a35-a28e-c884-0c33-8f3817cf59c2 ms_lineartrack default006bfebf-d986-e309-06ed-d7ea7c871c46 ms_lineartrack default00a9088c-6746-7892-d2ca-2e86e9047a60 ms_lineartrack default00afca31-a7b3-7d44-4752-531de37263a3 ms_lineartrack default00c9698e-146e-3bdc-61e4-c19ad4c8840d ms_lineartrack default00eaf8e9-168b-9c46-1621-e11b3bee558d ms_lineartrack default01717e02-3d2c-14ad-0a74-d1bd7f8417f0 ms_lineartrack default022c034d-da79-ca8c-a758-6c2e78b65fcf ms_lineartrack default025149ec-2d7e-4e52-3748-2efdb7f100a7 ms_lineartrack default <p>...</p> <p>Total: 1970</p> <p>And then run linearization by populating <code>LinearizedPositionV1</code>.</p> In\u00a0[13]: Copied! <pre>sgpl.LinearizedPositionV1().populate()\nsgpl.LinearizedPositionV1()\n</pre> sgpl.LinearizedPositionV1().populate() sgpl.LinearizedPositionV1() Out[13]: <p>pos_merge_id</p> <p>track_graph_name</p> <p>linearization_param_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>linearized_position_object_id</p> 001b5af0-489e-5437-9e34-f2130727ebc5 ms_lineartrack default Winnie20220714_8FOZKCPEXW.nwb 9b7b03d8-1df8-4a04-9566-ada9ad7d935e002476a5-c344-0236-d86c-89b995e22227 ms_lineartrack default Totoro20220614_BSOZ7EAWQU.nwb ddb91b90-8d36-4908-aa7c-2d117461d7a9002476a5-c344-0236-d86c-89b995e22227 ms_wtrack default Totoro20220614_GPQ1PDBI6S.nwb ee0a93ba-1233-4b1a-bffc-c810a8ba46b800551a35-a28e-c884-0c33-8f3817cf59c2 ms_lineartrack default Olive20220707_FIDPJQ458I.nwb 71ab61e9-cf9b-436c-b489-9f09b6f989da006bfebf-d986-e309-06ed-d7ea7c871c46 ms_lineartrack default Totoro20220530_X9K189ZS6H.nwb dff301c8-eb66-4b11-bca0-d5e05fd922c100a9088c-6746-7892-d2ca-2e86e9047a60 ms_lineartrack default Olive20220711_6QS37PFOW3.nwb 70a78fcd-7e17-43ba-8cb5-7f918047d68400a9088c-6746-7892-d2ca-2e86e9047a60 ms_wtrack default Olive20220711_8NMXZVVD5L.nwb 27b1097a-d612-4967-ac42-3699cc1de5b200afca31-a7b3-7d44-4752-531de37263a3 ms_lineartrack default Wallie20220922_EP1QE0SBJR.nwb b2797c39-606f-43cb-8d8f-652870b9db6700c9698e-146e-3bdc-61e4-c19ad4c8840d ms_lineartrack default Olive20220708_XXXT7RYLRX.nwb 8bbd3fb5-6cab-491a-8d74-c0681867824800eaf8e9-168b-9c46-1621-e11b3bee558d ms_lineartrack default Winnie20220717_2BIU1TYFMB.nwb 2b590d50-445a-4a5a-965e-1d64f3d8eafb01717e02-3d2c-14ad-0a74-d1bd7f8417f0 ms_lineartrack default Olive20220707_4WHPR9WFOK.nwb 8a31926a-132f-4957-b2ce-d5a2647f480f022c034d-da79-ca8c-a758-6c2e78b65fcf ms_lineartrack default Winnie20220720_F3YE2F28QX.nwb 2fc84409-b55f-44b0-9d72-740fdf5269c4 <p>...</p> <p>Total: 1970</p> <p>Populating <code>LinearizedPositionV1</code> also creates a corresponding entry in the <code>LinearizedPositionOutput</code> merge table. For downstream compatibility with alternate versions of the Linearization pipeline, we should fetch our data from here</p> <p>Running <code>fetch1_dataframe</code> will retrieve the linear position data, including...</p> <ul> <li><code>time</code>: dataframe index</li> <li><code>linear_position</code>: 1D linearized position</li> <li><code>track_segment_id</code>: index number of the edges given to track graph</li> <li><code>projected_{x,y}_position</code>: 2D position projected to the track graph</li> </ul> In\u00a0[18]: Copied! <pre>linear_key = {\n    \"pos_merge_id\": merge_id,\n    \"track_graph_name\": \"6 arm\",\n    \"linearization_param_name\": \"default\",\n}\n\nfrom spyglass.linearization.merge import LinearizedPositionOutput\n\nlinear_merge_key = LinearizedPositionOutput.merge_restrict(linear_key).fetch1(\n    \"KEY\"\n)\nlinear_position_df = (\n    LinearizedPositionOutput &amp; linear_merge_key\n).fetch1_dataframe()\nlinear_position_df\n</pre> linear_key = {     \"pos_merge_id\": merge_id,     \"track_graph_name\": \"6 arm\",     \"linearization_param_name\": \"default\", }  from spyglass.linearization.merge import LinearizedPositionOutput  linear_merge_key = LinearizedPositionOutput.merge_restrict(linear_key).fetch1(     \"KEY\" ) linear_position_df = (     LinearizedPositionOutput &amp; linear_merge_key ).fetch1_dataframe() linear_position_df Out[18]: linear_position track_segment_id projected_x_position projected_y_position time 1.581887e+09 412.042773 0 90.802281 210.677533 1.581887e+09 412.364853 0 90.520636 210.833775 1.581887e+09 412.686934 0 90.238990 210.990018 1.581887e+09 412.488270 0 90.412714 210.893645 1.581887e+09 412.007991 0 90.832697 210.660660 ... ... ... ... ... 1.581888e+09 340.401589 1 175.500994 212.958497 1.581888e+09 340.373902 1 175.477029 212.944630 1.581888e+09 340.394065 1 175.494481 212.954729 1.581888e+09 340.346214 1 175.453064 212.930764 1.581888e+09 340.318527 1 175.429100 212.916898 <p>39340 rows \u00d7 4 columns</p> <p>We'll plot the 1D position over time, colored by edge, and use the 1D track graph layout on the y-axis.</p> In\u00a0[19]: Copied! <pre>fig, ax = plt.subplots(figsize=(20, 13))\nax.scatter(\n    linear_position_df.index,\n    linear_position_df.linear_position,\n    c=linear_position_df.track_segment_id,\n    s=1,\n)\ngraph.plot_track_graph_as_1D(\n    ax=ax, axis=\"y\", other_axis_start=linear_position_df.index[-1] + 10\n)\n\nax.set_xlabel(\"Time [s]\", fontsize=18)\nax.set_ylabel(\"Linear Position [cm]\", fontsize=18)\nax.set_title(\"Linear Position\", fontsize=28)\n</pre> fig, ax = plt.subplots(figsize=(20, 13)) ax.scatter(     linear_position_df.index,     linear_position_df.linear_position,     c=linear_position_df.track_segment_id,     s=1, ) graph.plot_track_graph_as_1D(     ax=ax, axis=\"y\", other_axis_start=linear_position_df.index[-1] + 10 )  ax.set_xlabel(\"Time [s]\", fontsize=18) ax.set_ylabel(\"Linear Position [cm]\", fontsize=18) ax.set_title(\"Linear Position\", fontsize=28) Out[19]: <pre>Text(0.5, 1.0, 'Linear Position')</pre> <p>We can also plot the 2D position projected to the track graph</p> In\u00a0[20]: Copied! <pre>fig, ax = plt.subplots(1, 1, figsize=(10, 10))\nax.plot(\n    position_info.position_x,\n    position_info.position_y,\n    color=\"lightgrey\",\n    alpha=0.7,\n    zorder=0,\n)\nax.set_xlabel(\"x-position [cm]\", fontsize=18)\nax.set_ylabel(\"y-position [cm]\", fontsize=18)\nax.plot(\n    linear_position_df.projected_x_position,\n    linear_position_df.projected_y_position,\n)\n</pre> fig, ax = plt.subplots(1, 1, figsize=(10, 10)) ax.plot(     position_info.position_x,     position_info.position_y,     color=\"lightgrey\",     alpha=0.7,     zorder=0, ) ax.set_xlabel(\"x-position [cm]\", fontsize=18) ax.set_ylabel(\"y-position [cm]\", fontsize=18) ax.plot(     linear_position_df.projected_x_position,     linear_position_df.projected_y_position, ) Out[20]: <pre>[&lt;matplotlib.lines.Line2D at 0x7fc2ffb63040&gt;]</pre>"}, {"location": "notebooks/24_Linearization/#position-linearization", "title": "Position - Linearization\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#retrieve-2d-position", "title": "Retrieve 2D position\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#specifying-the-track", "title": "Specifying the track\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#parameters", "title": "Parameters\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#linearization", "title": "Linearization\u00b6", "text": ""}, {"location": "notebooks/24_Linearization/#examine-data", "title": "Examine data\u00b6", "text": ""}, {"location": "notebooks/30_LFP/", "title": "LFP", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> In\u00a0[2]: Copied! <pre>import os\nimport copy\nimport datajoint as dj\nimport numpy as np\nimport pandas as pd\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.lfp as lfp\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import copy import datajoint as dj import numpy as np import pandas as pd  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.lfp as lfp  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <pre>[2024-01-12 08:39:21,871][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-01-12 08:39:21,915][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\nWARNING:root:Populate: Entry in DataAcquisitionDeviceSystem with primary keys {'data_acquisition_device_system': 'SpikeGadgets'} already exists.\nWARNING:root:Populate: Entry in DataAcquisitionDeviceAmplifier with primary keys {'data_acquisition_device_amplifier': 'Intan'} already exists.\n</pre> <p>First, we select the NWB file, which corresponds to the dataset we want to extract LFP from.</p> In\u00a0[3]: Copied! <pre>nwb_file_name = \"minirec20230622_.nwb\"\n</pre> nwb_file_name = \"minirec20230622_.nwb\" <p>Next, we create the standard LFP Filters. This only needs to be done once.</p> In\u00a0[4]: Copied! <pre>sgc.FirFilterParameters().create_standard_filters()\nsgc.FirFilterParameters()\n</pre> sgc.FirFilterParameters().create_standard_filters() sgc.FirFilterParameters() Out[4]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Delta 0.5-4 Hz 1000 lowpass 0.25 0.5 4.0 5.0 delta filter for 1 KHz data =BLOB= =BLOB=Delta 0.5-4 Hz pass, 0.25-4.5 Hz stop 1000 lowpass 0.25 0.5 4.0 4.5 revised delta filter for 1 KHz data =BLOB= =BLOB=Epilepsy project 0.2-40 Hz 2000 bandpass 0.1 0.2 40.0 41.0 Sharp wave filter =BLOB= =BLOB=Epilepsy project 140-800 Hz 2000 bandpass 135.0 140.0 800.0 825.0 Extended ripple filter =BLOB= =BLOB=Epilepsy project 140-800 Hz ripple 2000 bandpass 135.0 140.0 800.0 825.0 Extended ripple filter =BLOB= =BLOB=Epilepsy project 40-600 Hz 2000 bandpass 39.0 40.0 600.0 625.0 HFE filter =BLOB= =BLOB=Fast Gamma 65-100 Hz 1000 bandpass 55.0 65.0 100.0 110.0 slow gamma filter for 1 Khz data =BLOB= =BLOB=LFP 0-400 Hz 1000 lowpass 0.0 0.0 400.0 425.0 LFP filter for referencing =BLOB= =BLOB=LFP 0-400 Hz 20000 lowpass 0.0 0.0 400.0 425.0 standard LFP filter for 20 KHz data =BLOB= =BLOB=LFP 0-400 Hz 29995 lowpass 0.0 0.0 400.0 425.0 adapted LFP filter for Banner20220105 =BLOB= =BLOB=LFP 0-400 Hz 30000 lowpass 0.0 0.0 400.0 425.0 standard LFP filter for 20 KHz data =BLOB= =BLOB=LFP 0-800 Hz 30000 lowpass 0.0 0.0 800.0 825.0 LFP filter 0-800 Hz =BLOB= =BLOB= <p>...</p> <p>Total: 31</p> <p>Now, we create an LFP electrode group, or the set of electrodes we want to filter for LFP data. We can grab all electrodes and brain regions as a data frame.</p> In\u00a0[5]: Copied! <pre>electrodes_df = (\n    pd.DataFrame(\n        (sgc.Electrode &amp; {\"nwb_file_name\": nwb_file_name, \"probe_electrode\": 0})\n        * sgc.BrainRegion\n    )\n    .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]\n    .sort_values(by=\"electrode_id\")\n)\nelectrodes_df\n</pre> electrodes_df = (     pd.DataFrame(         (sgc.Electrode &amp; {\"nwb_file_name\": nwb_file_name, \"probe_electrode\": 0})         * sgc.BrainRegion     )     .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]     .sort_values(by=\"electrode_id\") ) electrodes_df Out[5]: nwb_file_name electrode_id region_name 0 minirec20230622_.nwb 0 corpus callosum and associated subcortical whi... 1 minirec20230622_.nwb 4 corpus callosum and associated subcortical whi... 12 minirec20230622_.nwb 8 corpus callosum and associated subcortical whi... 23 minirec20230622_.nwb 12 corpus callosum and associated subcortical whi... 26 minirec20230622_.nwb 16 corpus callosum and associated subcortical whi... 27 minirec20230622_.nwb 20 corpus callosum and associated subcortical whi... 28 minirec20230622_.nwb 24 corpus callosum and associated subcortical whi... 29 minirec20230622_.nwb 28 corpus callosum and associated subcortical whi... 30 minirec20230622_.nwb 32 corpus callosum and associated subcortical whi... 31 minirec20230622_.nwb 36 corpus callosum and associated subcortical whi... 2 minirec20230622_.nwb 40 corpus callosum and associated subcortical whi... 3 minirec20230622_.nwb 44 corpus callosum and associated subcortical whi... 4 minirec20230622_.nwb 48 corpus callosum and associated subcortical whi... 5 minirec20230622_.nwb 52 corpus callosum and associated subcortical whi... 6 minirec20230622_.nwb 56 corpus callosum and associated subcortical whi... 7 minirec20230622_.nwb 60 corpus callosum and associated subcortical whi... 8 minirec20230622_.nwb 64 corpus callosum and associated subcortical whi... 9 minirec20230622_.nwb 68 corpus callosum and associated subcortical whi... 10 minirec20230622_.nwb 72 corpus callosum and associated subcortical whi... 11 minirec20230622_.nwb 76 corpus callosum and associated subcortical whi... 13 minirec20230622_.nwb 80 corpus callosum and associated subcortical whi... 14 minirec20230622_.nwb 84 corpus callosum and associated subcortical whi... 15 minirec20230622_.nwb 88 corpus callosum and associated subcortical whi... 16 minirec20230622_.nwb 92 corpus callosum and associated subcortical whi... 17 minirec20230622_.nwb 96 corpus callosum and associated subcortical whi... 18 minirec20230622_.nwb 100 corpus callosum and associated subcortical whi... 19 minirec20230622_.nwb 104 corpus callosum and associated subcortical whi... 20 minirec20230622_.nwb 108 corpus callosum and associated subcortical whi... 21 minirec20230622_.nwb 112 corpus callosum and associated subcortical whi... 22 minirec20230622_.nwb 116 corpus callosum and associated subcortical whi... 24 minirec20230622_.nwb 120 corpus callosum and associated subcortical whi... 25 minirec20230622_.nwb 124 corpus callosum and associated subcortical whi... <p>For a larger dataset, we might want to filter by region, but our example data only has one electrode.</p> <pre>lfp_electrode_ids = electrodes_df.loc[\n    electrodes_df.region_name == \"ca1\"\n].electrode_id\n</pre> In\u00a0[6]: Copied! <pre>lfp_electrode_ids = [0]\nlfp_electrode_group_name = \"test\"\nlfp_eg_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"lfp_electrode_group_name\": lfp_electrode_group_name,\n}\n\nlfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_file_name,\n    group_name=lfp_electrode_group_name,\n    electrode_list=lfp_electrode_ids,\n)\n</pre> lfp_electrode_ids = [0] lfp_electrode_group_name = \"test\" lfp_eg_key = {     \"nwb_file_name\": nwb_file_name,     \"lfp_electrode_group_name\": lfp_electrode_group_name, }  lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_file_name,     group_name=lfp_electrode_group_name,     electrode_list=lfp_electrode_ids, ) <p>We can verify the electrode list as follows</p> In\u00a0[7]: Copied! <pre>lfp.lfp_electrode.LFPElectrodeGroup.LFPElectrode() &amp; {\n    \"nwb_file_name\": nwb_file_name\n}\n</pre> lfp.lfp_electrode.LFPElectrodeGroup.LFPElectrode() &amp; {     \"nwb_file_name\": nwb_file_name } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode minirec20230622_.nwb test 0 0 <p>Total: 1</p> <p>Recall from the previous notebook that <code>IntervalList</code> selects time frames from the experiment. We can select the interval and subset to the first <code>n</code> seconds...</p> In\u00a0[8]: Copied! <pre>sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name}\n</pre> sgc.IntervalList &amp; {\"nwb_file_name\": nwb_file_name} Out[8]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1 =BLOB= minirec20230622_.nwb 01_s1_first9 =BLOB= minirec20230622_.nwb 02_s2 =BLOB= minirec20230622_.nwb pos 0 valid times =BLOB= minirec20230622_.nwb pos 1 valid times =BLOB= minirec20230622_.nwb raw data valid times =BLOB= <p>Total: 6</p> In\u00a0[9]: Copied! <pre>n = 9\norig_interval_list_name = \"01_s1\"\ninterval_list_name = orig_interval_list_name + f\"_first{n}\"\n\nvalid_times = (\n    sgc.IntervalList\n    &amp; {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_list_name\": orig_interval_list_name,\n    }\n).fetch1(\"valid_times\")\n\ninterval_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"valid_times\": np.asarray([[valid_times[0, 0], valid_times[0, 0] + n]]),\n}\n\nsgc.IntervalList.insert1(\n    interval_key,\n    skip_duplicates=True,\n)\n</pre> n = 9 orig_interval_list_name = \"01_s1\" interval_list_name = orig_interval_list_name + f\"_first{n}\"  valid_times = (     sgc.IntervalList     &amp; {         \"nwb_file_name\": nwb_file_name,         \"interval_list_name\": orig_interval_list_name,     } ).fetch1(\"valid_times\")  interval_key = {     \"nwb_file_name\": nwb_file_name,     \"interval_list_name\": interval_list_name,     \"valid_times\": np.asarray([[valid_times[0, 0], valid_times[0, 0] + n]]), }  sgc.IntervalList.insert1(     interval_key,     skip_duplicates=True, ) In\u00a0[10]: Copied! <pre>sgc.IntervalList() &amp; {\n    \"nwb_file_name\": nwb_file_name,\n    \"interval_list_name\": interval_list_name,\n}\n</pre> sgc.IntervalList() &amp; {     \"nwb_file_name\": nwb_file_name,     \"interval_list_name\": interval_list_name, } Out[10]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') minirec20230622_.nwb 01_s1_first9 =BLOB= <p>Total: 1</p> <p>LFPSelection combines the data, interval and filter</p> In\u00a0[23]: Copied! <pre>lfp_s_key = copy.deepcopy(lfp_eg_key)\n\nlfp_s_key.update(\n    {\n        \"target_interval_list_name\": interval_list_name,\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)\n        \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)\n    }\n)\n\nlfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)\n</pre> lfp_s_key = copy.deepcopy(lfp_eg_key)  lfp_s_key.update(     {         \"target_interval_list_name\": interval_list_name,         \"filter_name\": \"LFP 0-400 Hz\",         \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)         \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)     } )  lfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True) In\u00a0[24]: Copied! <pre>lfp.v1.LFPSelection() &amp; lfp_s_key\n</pre> lfp.v1.LFPSelection() &amp; lfp_s_key Out[24]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>target_sampling_rate</p> the desired output sampling rate, in HZ minirec20230622_.nwb test 01_s1_first9 LFP 0-400 Hz 30000 1000.0 <p>Total: 1</p> <p><code>LFPV1</code> has a similar <code>populate</code> command as we've seen before.</p> <p>Notes:</p> <ul> <li>For full recordings, this takes ~2h when done locally, for all electrodes</li> <li>This <code>populate</code> also inserts into the LFP Merge Table, <code>LFPOutput</code>. For more on Merge Tables, see our documentation. In short, this collects different LFP processing streams into one table.</li> </ul> In\u00a0[25]: Copied! <pre>lfp.v1.LFPV1().populate(lfp_s_key)\n</pre> lfp.v1.LFPV1().populate(lfp_s_key) <p>We can now look at the LFP table to see the data we've extracted</p> In\u00a0[26]: Copied! <pre>lfp.LFPOutput.LFPV1() &amp; lfp_s_key\n</pre> lfp.LFPOutput.LFPV1() &amp; lfp_s_key Out[26]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 minirec20230622_.nwb test 01_s1_first9 LFP 0-400 Hz 30000 <p>Total: 1</p> In\u00a0[27]: Copied! <pre>lfp_key = {\"merge_id\": (lfp.LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")}\nlfp_key\n</pre> lfp_key = {\"merge_id\": (lfp.LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")} lfp_key Out[27]: <pre>{'merge_id': UUID('1e7fbe35-034b-ed8e-7965-a0467ae5c0a4')}</pre> In\u00a0[28]: Copied! <pre>lfp.LFPOutput &amp; lfp_key\n</pre> lfp.LFPOutput &amp; lfp_key Out[28]: <p>merge_id</p> <p>source</p> 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 LFPV1 <p>Total: 1</p> <p>From the Merge Table, we can get the keys for the LFP data we want to see</p> In\u00a0[31]: Copied! <pre>lfp_df = (lfp.LFPOutput &amp; lfp_key).fetch1_dataframe()\nlfp_df\n</pre> lfp_df = (lfp.LFPOutput &amp; lfp_key).fetch1_dataframe() lfp_df Out[31]: 0 time 1.687475e+09 21 1.687475e+09 10 1.687475e+09 37 1.687475e+09 81 1.687475e+09 61 ... ... 1.687475e+09 -125 1.687475e+09 -126 1.687475e+09 -123 1.687475e+09 -129 1.687475e+09 -122 <p>9000 rows \u00d7 1 columns</p> In\u00a0[32]: Copied! <pre>lfp_sampling_rate = lfp.LFPOutput.merge_get_parent(lfp_key).fetch1(\n    \"lfp_sampling_rate\"\n)\n\nfilter_name = \"Theta 5-11 Hz\"\n\nsgc.common_filter.FirFilterParameters().add_filter(\n    filter_name,\n    lfp_sampling_rate,\n    \"bandpass\",\n    [4, 5, 11, 12],\n    \"theta filter for 1 Khz data\",\n)\n\nsgc.common_filter.FirFilterParameters() &amp; {\n    \"filter_name\": filter_name,\n    \"filter_sampling_rate\": lfp_sampling_rate,\n}\n</pre> lfp_sampling_rate = lfp.LFPOutput.merge_get_parent(lfp_key).fetch1(     \"lfp_sampling_rate\" )  filter_name = \"Theta 5-11 Hz\"  sgc.common_filter.FirFilterParameters().add_filter(     filter_name,     lfp_sampling_rate,     \"bandpass\",     [4, 5, 11, 12],     \"theta filter for 1 Khz data\", )  sgc.common_filter.FirFilterParameters() &amp; {     \"filter_name\": filter_name,     \"filter_sampling_rate\": lfp_sampling_rate, } Out[32]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Theta 5-11 Hz 1000 lowpass 4.0 5.0 11.0 12.0 theta filter for 1 KHz data =BLOB= =BLOB= <p>Total: 1</p> In\u00a0[33]: Copied! <pre>sgc.IntervalList()\n</pre> sgc.IntervalList() Out[33]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') aj8020230628_.nwb 02_lineartrack =BLOB= aj8020230628_.nwb 04_lineartrack =BLOB= aj8020230628_.nwb 06_lineartrack =BLOB= aj8020230628_.nwb pos 0 valid times =BLOB= aj8020230628_.nwb pos 1 valid times =BLOB= aj8020230628_.nwb pos 2 valid times =BLOB= aj8020230628_.nwb raw data valid times =BLOB= aj8420230703_.nwb 01_lineartrack =BLOB= aj8420230703_.nwb pos 0 valid times =BLOB= aj8420230703_.nwb raw data valid times =BLOB= aj8620230628_.nwb 02_lineartrack =BLOB= aj8620230628_.nwb 04_lineartrack =BLOB= <p>...</p> <p>Total: 130046</p> <p>We can specify electrodes of interest, and desired sampling rate.</p> In\u00a0[34]: Copied! <pre>from spyglass.lfp.analysis.v1 import lfp_band\n\nlfp_band_electrode_ids = [0]  # assumes we've filtered these electrodes\nlfp_band_sampling_rate = 100  # desired sampling rate\n\nlfp_band.LFPBandSelection().set_lfp_band_electrodes(\n    nwb_file_name=nwb_file_name,\n    lfp_merge_id=lfp_key[\"merge_id\"],\n    electrode_list=lfp_band_electrode_ids,\n    filter_name=filter_name,  # defined above\n    interval_list_name=interval_list_name,  # Defined in IntervalList above\n    reference_electrode_list=[-1],  # -1 means no ref electrode for all channels\n    lfp_band_sampling_rate=lfp_band_sampling_rate,\n)\n\nlfp_band.LFPBandSelection()\n</pre> from spyglass.lfp.analysis.v1 import lfp_band  lfp_band_electrode_ids = [0]  # assumes we've filtered these electrodes lfp_band_sampling_rate = 100  # desired sampling rate  lfp_band.LFPBandSelection().set_lfp_band_electrodes(     nwb_file_name=nwb_file_name,     lfp_merge_id=lfp_key[\"merge_id\"],     electrode_list=lfp_band_electrode_ids,     filter_name=filter_name,  # defined above     interval_list_name=interval_list_name,  # Defined in IntervalList above     reference_electrode_list=[-1],  # -1 means no ref electrode for all channels     lfp_band_sampling_rate=lfp_band_sampling_rate, )  lfp_band.LFPBandSelection() Out[34]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>min_interval_len</p> the minimum length of a valid interval to filter 001d08a1-5365-3c20-716b-f431ab136a55 Theta 5-11 Hz 1000 Totoro20220602_.nwb pos 4 valid times 100 1.000213375-550b-ae22-394b-82da3cdfddb3 Theta 5-11 Hz 1000 Yoshi20220518_.nwb pos 15 valid times 100 1.0004560d5-417b-ff99-5a13-99a37a3eb998 ripple_150_250 1000 j1620210715_.nwb 09_s5 1000 1.0004560d5-417b-ff99-5a13-99a37a3eb998 theta_5_11 1000 j1620210715_.nwb 09_s5 1000 1.0006b9516-88d1-dd58-75e6-e519fb7dfea8 ripple_150_250 1000 senor20201103_.nwb 15_s8 1000 1.0006b9516-88d1-dd58-75e6-e519fb7dfea8 theta_5_11 1000 senor20201103_.nwb 15_s8 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Fast Gamma 65-100 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b ms_stim_125ms_period 1000 Winnie20220714_.nwb pos 9 valid times 200 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Slow Gamma 25-55 Hz 1000 Winnie20220714_.nwb pos 9 valid times 1000 1.00087e094-8238-32b8-9e8d-ecb7d9352b3b Theta 5-11 Hz 1000 Winnie20220714_.nwb pos 9 valid times 100 1.00093a2c5-d720-d0c4-6cc4-35f1f34a0a01 ripple_150_250 1000 senor20201118_.nwb 08_r4 noPrePostTrialTimes 1000 1.0 <p>...</p> <p>Total: 8457</p> <p>Next we add an entry for the LFP Band and the electrodes we want to filter</p> In\u00a0[39]: Copied! <pre>lfp_band_key = (\n    lfp_band.LFPBandSelection\n    &amp; {\n        \"lfp_merge_id\": lfp_key[\"merge_id\"],\n        \"filter_name\": filter_name,\n        \"lfp_band_sampling_rate\": lfp_band_sampling_rate,\n    }\n).fetch1(\"KEY\")\nlfp_band_key\n</pre> lfp_band_key = (     lfp_band.LFPBandSelection     &amp; {         \"lfp_merge_id\": lfp_key[\"merge_id\"],         \"filter_name\": filter_name,         \"lfp_band_sampling_rate\": lfp_band_sampling_rate,     } ).fetch1(\"KEY\") lfp_band_key Out[39]: <pre>{'lfp_merge_id': UUID('1e7fbe35-034b-ed8e-7965-a0467ae5c0a4'),\n 'filter_name': 'Theta 5-11 Hz',\n 'filter_sampling_rate': 1000,\n 'nwb_file_name': 'minirec20230622_.nwb',\n 'target_interval_list_name': '01_s1_first9',\n 'lfp_band_sampling_rate': 100}</pre> <p>Check to make sure it worked</p> In\u00a0[40]: Copied! <pre>lfp_band.LFPBandSelection() &amp; lfp_band_key\n</pre> lfp_band.LFPBandSelection() &amp; lfp_band_key Out[40]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>min_interval_len</p> the minimum length of a valid interval to filter 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 1.0 <p>Total: 1</p> In\u00a0[42]: Copied! <pre>lfp_band.LFPBandV1().populate(lfp_band.LFPBandSelection() &amp; lfp_band_key)\nlfp_band.LFPBandV1() &amp; lfp_band_key\n</pre> lfp_band.LFPBandV1().populate(lfp_band.LFPBandSelection() &amp; lfp_band_key) lfp_band.LFPBandV1() &amp; lfp_band_key Out[42]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_DCP0O105KG.nwb 01_s1_first9 lfp band 100Hz 277bdc87-98b9-4eff-a33e-00b64b4c3bb2 <p>Total: 1</p> <p>Now we can plot the original signal, the LFP filtered trace, and the theta filtered trace together. Get the three electrical series objects and the indices of the electrodes we band pass filtered</p> <p>Note: Much of the code below could be replaced by a function calls that would return the data from each electrical series.</p> <p>Note: If you see an error <code>Qt: Session Management Error</code>, try running the following unix command: <code>export -n SESSION_MANAGER</code>. See also</p> In\u00a0[43]: Copied! <pre>orig_eseries = (sgc.Raw() &amp; {\"nwb_file_name\": nwb_file_name}).fetch_nwb()[0][\n    \"raw\"\n]\norig_elect_indices = sgc.get_electrode_indices(\n    orig_eseries, lfp_band_electrode_ids\n)\norig_timestamps = np.asarray(orig_eseries.timestamps)\n</pre> orig_eseries = (sgc.Raw() &amp; {\"nwb_file_name\": nwb_file_name}).fetch_nwb()[0][     \"raw\" ] orig_elect_indices = sgc.get_electrode_indices(     orig_eseries, lfp_band_electrode_ids ) orig_timestamps = np.asarray(orig_eseries.timestamps) In\u00a0[44]: Copied! <pre>lfp_eseries = lfp.LFPOutput().fetch_nwb(lfp_key)[0][\"lfp\"]\nlfp_elect_indices = sgc.get_electrode_indices(\n    lfp_eseries, lfp_band_electrode_ids\n)\nlfp_timestamps = np.asarray(lfp_eseries.timestamps)\n</pre> lfp_eseries = lfp.LFPOutput().fetch_nwb(lfp_key)[0][\"lfp\"] lfp_elect_indices = sgc.get_electrode_indices(     lfp_eseries, lfp_band_electrode_ids ) lfp_timestamps = np.asarray(lfp_eseries.timestamps) In\u00a0[45]: Copied! <pre>lfp_band_eseries = (lfp_band.LFPBandV1 &amp; lfp_band_key).fetch_nwb()[0][\n    \"lfp_band\"\n]\nlfp_band_elect_indices = sgc.get_electrode_indices(\n    lfp_band_eseries, lfp_band_electrode_ids\n)\nlfp_band_timestamps = np.asarray(lfp_band_eseries.timestamps)\n</pre> lfp_band_eseries = (lfp_band.LFPBandV1 &amp; lfp_band_key).fetch_nwb()[0][     \"lfp_band\" ] lfp_band_elect_indices = sgc.get_electrode_indices(     lfp_band_eseries, lfp_band_electrode_ids ) lfp_band_timestamps = np.asarray(lfp_band_eseries.timestamps) <p>Get a list of times for the first run epoch and then select a 2 second interval 100 seconds from the beginning</p> In\u00a0[46]: Copied! <pre>plottimes = [valid_times[0][0] + 1, valid_times[0][0] + 8]\n</pre> plottimes = [valid_times[0][0] + 1, valid_times[0][0] + 8] In\u00a0[47]: Copied! <pre># get the time indices for each dataset\norig_time_ind = np.where(\n    np.logical_and(\n        orig_timestamps &gt; plottimes[0], orig_timestamps &lt; plottimes[1]\n    )\n)[0]\n\nlfp_time_ind = np.where(\n    np.logical_and(lfp_timestamps &gt; plottimes[0], lfp_timestamps &lt; plottimes[1])\n)[0]\nlfp_band_time_ind = np.where(\n    np.logical_and(\n        lfp_band_timestamps &gt; plottimes[0],\n        lfp_band_timestamps &lt; plottimes[1],\n    )\n)[0]\n</pre> # get the time indices for each dataset orig_time_ind = np.where(     np.logical_and(         orig_timestamps &gt; plottimes[0], orig_timestamps &lt; plottimes[1]     ) )[0]  lfp_time_ind = np.where(     np.logical_and(lfp_timestamps &gt; plottimes[0], lfp_timestamps &lt; plottimes[1]) )[0] lfp_band_time_ind = np.where(     np.logical_and(         lfp_band_timestamps &gt; plottimes[0],         lfp_band_timestamps &lt; plottimes[1],     ) )[0] In\u00a0[48]: Copied! <pre>import matplotlib.pyplot as plt\n\nplt.plot(\n    orig_eseries.timestamps[orig_time_ind],\n    orig_eseries.data[orig_time_ind, orig_elect_indices[0]],\n    \"k-\",\n)\nplt.plot(\n    lfp_eseries.timestamps[lfp_time_ind],\n    lfp_eseries.data[lfp_time_ind, lfp_elect_indices[0]],\n    \"b-\",\n)\nplt.plot(\n    lfp_band_eseries.timestamps[lfp_band_time_ind],\n    lfp_band_eseries.data[lfp_band_time_ind, lfp_band_elect_indices[0]],\n    \"r-\",\n)\nplt.xlabel(\"Time (sec)\")\nplt.ylabel(\"Amplitude (AD units)\")\n\n# Uncomment to see plot\n# plt.show()\n</pre> import matplotlib.pyplot as plt  plt.plot(     orig_eseries.timestamps[orig_time_ind],     orig_eseries.data[orig_time_ind, orig_elect_indices[0]],     \"k-\", ) plt.plot(     lfp_eseries.timestamps[lfp_time_ind],     lfp_eseries.data[lfp_time_ind, lfp_elect_indices[0]],     \"b-\", ) plt.plot(     lfp_band_eseries.timestamps[lfp_band_time_ind],     lfp_band_eseries.data[lfp_band_time_ind, lfp_band_elect_indices[0]],     \"r-\", ) plt.xlabel(\"Time (sec)\") plt.ylabel(\"Amplitude (AD units)\")  # Uncomment to see plot # plt.show() Out[48]: <pre>Text(0, 0.5, 'Amplitude (AD units)')</pre>"}, {"location": "notebooks/30_LFP/#lfp-extraction", "title": "LFP Extraction\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#select-data", "title": "Select data\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#create-filters", "title": "Create Filters\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#electrode-group", "title": "Electrode Group\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#intervallist", "title": "<code>IntervalList</code>\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#lfpselection", "title": "<code>LFPSelection</code>\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#populate-lfp", "title": "Populate LFP\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#lfp-band", "title": "LFP Band\u00b6", "text": "<p>Now that we've created the LFP object we can perform a second level of filtering for a band of interest, in this case the theta band. We first need to create the filter.</p>"}, {"location": "notebooks/30_LFP/#plotting", "title": "Plotting\u00b6", "text": ""}, {"location": "notebooks/30_LFP/#next-steps", "title": "Next Steps\u00b6", "text": "<p>Next, we'll use look at Theta bands within LFP data.</p>"}, {"location": "notebooks/31_Theta/", "title": "Theta", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> <li>To run this notebook, you should have already completed the  LFP notebook and populated the <code>LFPBand</code> table.</li> </ul> <p>In this tutorial, we demonstrate how to generate analytic signals from the LFP data, as well as how to compute theta phases and power.</p> In\u00a0[4]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.lfp.analysis.v1.lfp_band as lfp_band\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np import matplotlib.pyplot as plt  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.lfp.analysis.v1.lfp_band as lfp_band  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) In\u00a0[5]: Copied! <pre>lfp_band.LFPBandV1()\n</pre> lfp_band.LFPBandV1() Out[5]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_H89T89A0CK.nwb 01_s1_first9 lfp band 100Hz 71bbb1a8-409e-48eb-a336-8c70fd5d6b74 <p>Total: 1</p> <p>Now, we create a keys to reference the theta band data.</p> In\u00a0[6]: Copied! <pre>nwb_file_name = \"minirec20230622_.nwb\"\nlfp_key = dict(\n    nwb_file_name=nwb_file_name,\n    filter_name=\"Theta 5-11 Hz\",\n)\nlfp_band.LFPBandV1() &amp; lfp_key\n</pre> nwb_file_name = \"minirec20230622_.nwb\" lfp_key = dict(     nwb_file_name=nwb_file_name,     filter_name=\"Theta 5-11 Hz\", ) lfp_band.LFPBandV1() &amp; lfp_key Out[6]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file 1e7fbe35-034b-ed8e-7965-a0467ae5c0a4 Theta 5-11 Hz 1000 minirec20230622_.nwb 01_s1_first9 100 minirec20230622_H89T89A0CK.nwb 01_s1_first9 lfp band 100Hz 71bbb1a8-409e-48eb-a336-8c70fd5d6b74 <p>Total: 1</p> <p>We do not need all electrodes for theta phase/power, so we define a list for analyses. When working with full data, this list might limit to hippocampal reference electrodes.</p> <p>Make sure that the chosen electrodes already exist in the LFPBand data; if not, go to the LFP tutorial to generate them.</p> In\u00a0[7]: Copied! <pre>electrode_list = [0]\n\nall_electrodes = (  # All available electrode ids\n    (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"]\n).electrodes.data[:]\n\nnp.isin(electrode_list, all_electrodes)  # Check if our list is in 'all'\n</pre> electrode_list = [0]  all_electrodes = (  # All available electrode ids     (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"] ).electrodes.data[:]  np.isin(electrode_list, all_electrodes)  # Check if our list is in 'all' Out[7]: <pre>array([ True])</pre> <p>Next, we'll compute the theta analytic signal.</p> In\u00a0[9]: Copied! <pre>theta_analytic_signal = (\n    lfp_band.LFPBandV1() &amp; lfp_key\n).compute_analytic_signal(electrode_list=electrode_list)\n\ntheta_analytic_signal\n</pre> theta_analytic_signal = (     lfp_band.LFPBandV1() &amp; lfp_key ).compute_analytic_signal(electrode_list=electrode_list)  theta_analytic_signal Out[9]: electrode 0 time 1.687475e+09 -124.000000-133.973800j 1.687475e+09 -30.000000-308.834746j 1.687475e+09 56.000000-267.135234j 1.687475e+09 139.000000-293.141114j 1.687475e+09 227.000000-236.059129j ... ... 1.687475e+09 -548.000000+29.032951j 1.687475e+09 -464.000000-225.303955j 1.687475e+09 -291.000000-329.028626j 1.687475e+09 -86.000000-404.908506j 1.687475e+09 92.000000-178.143737j <p>900 rows \u00d7 1 columns</p> <p>In the dataframe above, the index is the timestamps, and the columns are the analytic signals of theta band (complex numbers) for each electrode.</p> <p>Using a similar method, we can compute theta phase and power from the LFPBand table.</p> In\u00a0[10]: Copied! <pre>theta_phase = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_phase(\n    electrode_list=electrode_list\n)\ntheta_power = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_power(\n    electrode_list=electrode_list\n)\n</pre> theta_phase = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_phase(     electrode_list=electrode_list ) theta_power = (lfp_band.LFPBandV1() &amp; lfp_key).compute_signal_power(     electrode_list=electrode_list ) <p>We can get the theta band data to plot with the theta phase results.</p> In\u00a0[11]: Copied! <pre>theta_band = (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"]\nelectrode_index = np.isin(theta_band.electrodes.data[:], electrode_list)\ntheta_band_selected = theta_band.data[:, electrode_index]\n</pre> theta_band = (lfp_band.LFPBandV1() &amp; lfp_key).fetch_nwb()[0][\"lfp_band\"] electrode_index = np.isin(theta_band.electrodes.data[:], electrode_list) theta_band_selected = theta_band.data[:, electrode_index] In\u00a0[13]: Copied! <pre>electrode_id = electrode_list[0]  # the electrode for which we want to plot\nplot_start, plot_end = 0, 5000  # start/end time of plotting\n\nfig, ax1 = plt.subplots(figsize=(20, 6))\nax1.set_xlabel(\"Time (sec)\", fontsize=15)\nax1.set_ylabel(\"Amplitude (AD units)\", fontsize=15)\nax1.plot(\n    theta_phase.index[plot_start:plot_end],\n    theta_band_selected[\n        plot_start:plot_end, np.where(np.array(electrode_list) == 0)[0][0]\n    ],\n    \"k-\",\n    linewidth=1,\n    alpha=0.9,\n)\nax1.tick_params(axis=\"y\", labelcolor=\"k\")\n\nax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis\nax2.set_ylabel(\"Phase(deg)\", color=\"b\", fontsize=15)\nax2.plot(\n    theta_phase.index[plot_start:plot_end],\n    theta_phase[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],\n    \"b\",\n)\nax2.tick_params(axis=\"y\", labelcolor=\"b\")\nax2.axhline(y=0, color=\"r\", linestyle=\"-\")\n\nfig.tight_layout()\nax1.set_title(\n    f\"Theta band amplitude and phase, electrode {electrode_id}\",\n    fontsize=20,\n)\n</pre> electrode_id = electrode_list[0]  # the electrode for which we want to plot plot_start, plot_end = 0, 5000  # start/end time of plotting  fig, ax1 = plt.subplots(figsize=(20, 6)) ax1.set_xlabel(\"Time (sec)\", fontsize=15) ax1.set_ylabel(\"Amplitude (AD units)\", fontsize=15) ax1.plot(     theta_phase.index[plot_start:plot_end],     theta_band_selected[         plot_start:plot_end, np.where(np.array(electrode_list) == 0)[0][0]     ],     \"k-\",     linewidth=1,     alpha=0.9, ) ax1.tick_params(axis=\"y\", labelcolor=\"k\")  ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis ax2.set_ylabel(\"Phase(deg)\", color=\"b\", fontsize=15) ax2.plot(     theta_phase.index[plot_start:plot_end],     theta_phase[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],     \"b\", ) ax2.tick_params(axis=\"y\", labelcolor=\"b\") ax2.axhline(y=0, color=\"r\", linestyle=\"-\")  fig.tight_layout() ax1.set_title(     f\"Theta band amplitude and phase, electrode {electrode_id}\",     fontsize=20, ) <p>We can also plot the theta power.</p> In\u00a0[14]: Copied! <pre>electrode_id = electrode_list[0]  # the electrode for which we want to plot\nplot_start, plot_end = 0, 5000  # start/end time of plotting\n\nfig, ax = plt.subplots(figsize=(20, 6))\nax.set_xlabel(\"Time (sec)\", fontsize=15)\nax.set_ylabel(\"Theta power\", fontsize=15)\nax.plot(\n    theta_power.index[plot_start:plot_end],\n    theta_power[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],\n    \"k-\",\n    linewidth=1,\n)\nax.tick_params(axis=\"y\", labelcolor=\"k\")\nax.set_title(\n    f\"Theta band power, electrode {electrode_id}\",\n    fontsize=20,\n)\n</pre> electrode_id = electrode_list[0]  # the electrode for which we want to plot plot_start, plot_end = 0, 5000  # start/end time of plotting  fig, ax = plt.subplots(figsize=(20, 6)) ax.set_xlabel(\"Time (sec)\", fontsize=15) ax.set_ylabel(\"Theta power\", fontsize=15) ax.plot(     theta_power.index[plot_start:plot_end],     theta_power[f\"electrode {electrode_id}\"].iloc[plot_start:plot_end],     \"k-\",     linewidth=1, ) ax.tick_params(axis=\"y\", labelcolor=\"k\") ax.set_title(     f\"Theta band power, electrode {electrode_id}\",     fontsize=20, )"}, {"location": "notebooks/31_Theta/#theta-phase-and-power", "title": "Theta phase and power\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#acquire-signal", "title": "Acquire Signal\u00b6", "text": "<p>First, we'll acquire the theta band analytic signal from the <code>LFPBand</code> data for electrodes of interest. We can grab keys from this table.</p>"}, {"location": "notebooks/31_Theta/#compute-phase-and-power", "title": "Compute phase and power\u00b6", "text": ""}, {"location": "notebooks/31_Theta/#plot-results", "title": "Plot results\u00b6", "text": "<p>We can overlay theta and detected phase for each electrode.</p> <p>Note: The red horizontal line indicates phase 0, corresponding to the through of theta.</p>"}, {"location": "notebooks/31_Theta/#up-next", "title": "Up Next\u00b6", "text": "<p>Next, we'll turn our attention to position data.</p>"}, {"location": "notebooks/32_Ripple_Detection/", "title": "Ripple Detection", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> </ul> <p>Ripple detection depends on a set of LFPs, the parameters used for detection and the speed of the animal. You will need <code>RippleLFPSelection</code>, <code>RippleParameters</code>, and <code>PositionOutput</code> to be populated accordingly.</p> In\u00a0[32]: Copied! <pre>import os\nimport datajoint as dj\nimport numpy as np\nimport pandas as pd\n\n# change to the upper level folder to detect dj_local_conf.json\nif os.path.basename(os.getcwd()) == \"notebooks\":\n    os.chdir(\"..\")\ndj.config.load(\"dj_local_conf.json\")  # load config for database connection info\n\nimport spyglass.common as sgc\nimport spyglass.position.v1 as sgp\nimport spyglass.lfp.analysis.v1 as lfp_analysis\nfrom spyglass.lfp import LFPOutput\nimport spyglass.lfp as sglfp\nfrom spyglass.position import PositionOutput\nimport spyglass.ripple.v1 as sgrip\nimport spyglass.ripple.v1 as sgr\n\n# ignore datajoint+jupyter async warnings\nimport warnings\n\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)\nwarnings.simplefilter(\"ignore\", category=ResourceWarning)\n</pre> import os import datajoint as dj import numpy as np import pandas as pd  # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == \"notebooks\":     os.chdir(\"..\") dj.config.load(\"dj_local_conf.json\")  # load config for database connection info  import spyglass.common as sgc import spyglass.position.v1 as sgp import spyglass.lfp.analysis.v1 as lfp_analysis from spyglass.lfp import LFPOutput import spyglass.lfp as sglfp from spyglass.position import PositionOutput import spyglass.ripple.v1 as sgrip import spyglass.ripple.v1 as sgr  # ignore datajoint+jupyter async warnings import warnings  warnings.simplefilter(\"ignore\", category=DeprecationWarning) warnings.simplefilter(\"ignore\", category=ResourceWarning) <p>First, we need to generate a filter band from the LFP data at the ripple frequency.  This process is analogous to that in 30_LFP.ipynb.</p> <p>If you have already populated the LFP table for your data you may skip this step. Here, we will begin by creating a lfp group of just electrodes in the hippocampus region, and populate the lfp on a subsetted interval:</p> In\u00a0[9]: Copied! <pre>nwb_file_name = \"mediumnwb20230802_.nwb\"\nlfp_electrode_group_name = \"test_hippocampus\"\ninterval_list_name = \"02_r1_ripple_demo\"\n\n# select hippocampus electrodes\nelectrodes_df = (\n    pd.DataFrame(\n        (\n            sgc.Electrode\n            &amp; {\n                \"nwb_file_name\": nwb_file_name,\n            }\n        )\n        * (sgc.BrainRegion &amp; {\"region_name\": \"hippocampus\"})\n    )\n    .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]\n    .sort_values(by=\"electrode_id\")\n)\n# create lfp_electrode_group\nlfp_eg_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"lfp_electrode_group_name\": lfp_electrode_group_name,\n}\nsglfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n    nwb_file_name=nwb_file_name,\n    group_name=lfp_electrode_group_name,\n    electrode_list=electrodes_df.electrode_id.tolist(),\n)\n\n# make a shorter interval to run this demo on\ninterval_start = (\n    sgc.IntervalList\n    &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": \"02_r1\"}\n).fetch1(\"valid_times\")[0][0]\ntruncated_interval = np.array(\n    [[interval_start, interval_start + 120]]\n)  # first 2 minutes of epoch\nsgc.IntervalList.insert1(\n    {\n        \"nwb_file_name\": nwb_file_name,\n        \"interval_list_name\": \"02_r1_ripple_demo\",\n        \"valid_times\": truncated_interval,\n    },\n    skip_duplicates=True,\n)\n\n# make the lfp selection\nlfp_s_key = lfp_eg_key.copy()\nlfp_s_key.update(\n    {\n        \"target_interval_list_name\": interval_list_name,\n        \"filter_name\": \"LFP 0-400 Hz\",\n        \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)\n        \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)\n    }\n)\nsglfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)\n\n# populate the lfp\nsglfp.v1.LFPV1.populate(lfp_s_key, display_progress=True)\nsglfp.v1.LFPV1 &amp; lfp_s_key\n</pre> nwb_file_name = \"mediumnwb20230802_.nwb\" lfp_electrode_group_name = \"test_hippocampus\" interval_list_name = \"02_r1_ripple_demo\"  # select hippocampus electrodes electrodes_df = (     pd.DataFrame(         (             sgc.Electrode             &amp; {                 \"nwb_file_name\": nwb_file_name,             }         )         * (sgc.BrainRegion &amp; {\"region_name\": \"hippocampus\"})     )     .loc[:, [\"nwb_file_name\", \"electrode_id\", \"region_name\"]]     .sort_values(by=\"electrode_id\") ) # create lfp_electrode_group lfp_eg_key = {     \"nwb_file_name\": nwb_file_name,     \"lfp_electrode_group_name\": lfp_electrode_group_name, } sglfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(     nwb_file_name=nwb_file_name,     group_name=lfp_electrode_group_name,     electrode_list=electrodes_df.electrode_id.tolist(), )  # make a shorter interval to run this demo on interval_start = (     sgc.IntervalList     &amp; {\"nwb_file_name\": nwb_file_name, \"interval_list_name\": \"02_r1\"} ).fetch1(\"valid_times\")[0][0] truncated_interval = np.array(     [[interval_start, interval_start + 120]] )  # first 2 minutes of epoch sgc.IntervalList.insert1(     {         \"nwb_file_name\": nwb_file_name,         \"interval_list_name\": \"02_r1_ripple_demo\",         \"valid_times\": truncated_interval,     },     skip_duplicates=True, )  # make the lfp selection lfp_s_key = lfp_eg_key.copy() lfp_s_key.update(     {         \"target_interval_list_name\": interval_list_name,         \"filter_name\": \"LFP 0-400 Hz\",         \"filter_sampling_rate\": 30_000,  # sampling rate of the data (Hz)         \"target_sampling_rate\": 1_000,  # smpling rate of the lfp output (Hz)     } ) sglfp.v1.LFPSelection.insert1(lfp_s_key, skip_duplicates=True)  # populate the lfp sglfp.v1.LFPV1.populate(lfp_s_key, display_progress=True) sglfp.v1.LFPV1 &amp; lfp_s_key Out[9]: <p>nwb_file_name</p> name of the NWB file <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>target_interval_list_name</p> descriptive name of this interval list <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_object_id</p> the NWB object ID for loading this object from the file <p>lfp_sampling_rate</p> the sampling rate, in HZ mediumnwb20230802_.nwb test_hippocampus 02_r1_ripple_demo LFP 0-400 Hz 30000 mediumnwb20230802_7625J294O4.nwb lfp_test_hippocampus_02_r1_ripple_demo_valid times 95ac8100-eca8-4dff-a504-b1c139a2a3af 1000.0 <p>Total: 1</p> In\u00a0[10]: Copied! <pre>sgc.FirFilterParameters().add_filter(\n    filter_name=\"Ripple 150-250 Hz\",\n    fs=1000.0,\n    filter_type=\"bandpass\",\n    band_edges=[140, 150, 250, 260],\n    comments=\"ripple band filter for 1 kHz data\",\n)\n\nsgc.FirFilterParameters() &amp; \"filter_name='Ripple 150-250 Hz'\"\n</pre> sgc.FirFilterParameters().add_filter(     filter_name=\"Ripple 150-250 Hz\",     fs=1000.0,     filter_type=\"bandpass\",     band_edges=[140, 150, 250, 260],     comments=\"ripple band filter for 1 kHz data\", )  sgc.FirFilterParameters() &amp; \"filter_name='Ripple 150-250 Hz'\" Out[10]: <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>filter_type</p> <p>filter_low_stop</p> lowest frequency for stop band for low frequency side of filter <p>filter_low_pass</p> lowest frequency for pass band of low frequency side of filter <p>filter_high_pass</p> highest frequency for pass band for high frequency side of filter <p>filter_high_stop</p> highest frequency for stop band of high frequency side of filter <p>filter_comments</p> comments about the filter <p>filter_band_edges</p> numpy array containing the filter bands (redundant with individual parameters) <p>filter_coeff</p> numpy array containing the filter coefficients Ripple 150-250 Hz 1000 lowpass 140.0 150.0 250.0 260.0 ripple band filter for 1 kHz data =BLOB= =BLOB= <p>Total: 1</p> <p>We can then populate the ripple band</p> In\u00a0[18]: Copied! <pre>from spyglass.lfp.analysis.v1 import lfp_band\n\nfilter_name = \"Ripple 150-250 Hz\"\nlfp_band_electrode_ids = (\n    electrodes_df.electrode_id.tolist()\n)  # assumes we've filtered these electrodes\nlfp_band_sampling_rate = 1000  # desired sampling rate\n\nlfp_merge_id = (LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\")\nlfp_band.LFPBandSelection().set_lfp_band_electrodes(\n    nwb_file_name=nwb_file_name,\n    lfp_merge_id=lfp_merge_id,\n    electrode_list=lfp_band_electrode_ids,\n    filter_name=filter_name,\n    interval_list_name=interval_list_name,\n    reference_electrode_list=[-1],  # -1 means no ref electrode for all channels\n    lfp_band_sampling_rate=lfp_band_sampling_rate,\n)\n\nlfp_band.LFPBandV1.populate(\n    {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name},\n    display_progress=True,\n)\nlfp_band.LFPBandV1 &amp; {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name}\n</pre> from spyglass.lfp.analysis.v1 import lfp_band  filter_name = \"Ripple 150-250 Hz\" lfp_band_electrode_ids = (     electrodes_df.electrode_id.tolist() )  # assumes we've filtered these electrodes lfp_band_sampling_rate = 1000  # desired sampling rate  lfp_merge_id = (LFPOutput.LFPV1() &amp; lfp_s_key).fetch1(\"merge_id\") lfp_band.LFPBandSelection().set_lfp_band_electrodes(     nwb_file_name=nwb_file_name,     lfp_merge_id=lfp_merge_id,     electrode_list=lfp_band_electrode_ids,     filter_name=filter_name,     interval_list_name=interval_list_name,     reference_electrode_list=[-1],  # -1 means no ref electrode for all channels     lfp_band_sampling_rate=lfp_band_sampling_rate, )  lfp_band.LFPBandV1.populate(     {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name},     display_progress=True, ) lfp_band.LFPBandV1 &amp; {\"lfp_merge_id\": lfp_merge_id, \"filter_name\": filter_name} Out[18]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>analysis_file_name</p> name of the file <p>interval_list_name</p> descriptive name of this interval list <p>lfp_band_object_id</p> the NWB object ID for loading this object from the file e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 mediumnwb20230802_.nwb 02_r1_ripple_demo 1000 mediumnwb20230802_XVQHZZVY3B.nwb 02_r1_ripple_demo lfp band 1000Hz eca68871-9a8d-4fa4-b265-95dafdfdda29 <p>Total: 1</p> <p>Next, we'll pick the electrodes on which we'll run ripple detection on, using <code>RippleLFPSelection.set_lfp_electrodes</code></p> In\u00a0[19]: Copied! <pre>?sgr.RippleLFPSelection.set_lfp_electrodes\n</pre> ?sgr.RippleLFPSelection.set_lfp_electrodes <pre>Signature:\nsgr.RippleLFPSelection.set_lfp_electrodes(\n    key,\n    electrode_list=None,\n    group_name='CA1',\n    **kwargs,\n)\nDocstring:\nRemoves all electrodes for the specified nwb file and then\nadds back the electrodes in the list\n\nParameters\n----------\nkey : dict\n    dictionary corresponding to the LFPBand entry to use for\n    ripple detection\nelectrode_list : list\n    list of electrodes from LFPBandSelection.LFPBandElectrode\n    to be used as the ripple LFP during detection\ngroup_name : str, optional\n    description of the electrode group, by default \"CA1\"\nFile:      ~/Documents/spyglass/src/spyglass/ripple/v1/ripple.py\nType:      function</pre> <p>We'll need the <code>nwb_file_name</code>, an <code>electrode_list</code>, and to a <code>group_name</code>.</p> <ul> <li>By default, <code>group_name</code> is set to CA1 for ripple detection, but we could alternatively use PFC.</li> <li>We use <code>nwb_file_name</code> to explore which electrodes are available for the <code>electrode_list</code>.</li> </ul> <p>Now we can look at <code>electrode_id</code> in the <code>Electrode</code> table:</p> In\u00a0[20]: Copied! <pre>electrodes = (\n    (sgc.Electrode() &amp; {\"nwb_file_name\": nwb_file_name})\n    * (\n        lfp_analysis.LFPBandSelection.LFPBandElectrode()\n        &amp; {\n            \"nwb_file_name\": nwb_file_name,\n            \"filter_name\": filter_name,\n            \"target_interval_list_name\": interval_list_name,\n        }\n    )\n    * sgc.BrainRegion\n).fetch(format=\"frame\")\nelectrodes\n</pre> electrodes = (     (sgc.Electrode() &amp; {\"nwb_file_name\": nwb_file_name})     * (         lfp_analysis.LFPBandSelection.LFPBandElectrode()         &amp; {             \"nwb_file_name\": nwb_file_name,             \"filter_name\": filter_name,             \"target_interval_list_name\": interval_list_name,         }     )     * sgc.BrainRegion ).fetch(format=\"frame\") electrodes Out[20]: probe_id probe_shank probe_electrode name original_reference_electrode x y z filtering impedance bad_channel x_warped y_warped z_warped contacts region_name subregion_name subsubregion_name nwb_file_name electrode_group_name electrode_id lfp_merge_id filter_name filter_sampling_rate target_interval_list_name lfp_band_sampling_rate lfp_electrode_group_name reference_elect_id region_id mediumnwb20230802_.nwb 0 0 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 0 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 1 1 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 2 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 2 2 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 3 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 3 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 4 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 4 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 8 35 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 35 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 9 36 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 36 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 37 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 1 37 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 38 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 2 38 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 39 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 3 39 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None <p>88 rows \u00d7 18 columns</p> <p>For ripple detection, we want only tetrodes, and only the first good wire on each tetrode. We will assume that is the first wire on each tetrode. I will do this using pandas syntax but you could use datajoint to filter this table as well. Here is the filtered table.</p> In\u00a0[21]: Copied! <pre>hpc_names = [\"ca1\", \"hippocampus\", \"CA1\", \"Hippocampus\"]\nelectrodes.loc[\n    (electrodes.region_name.isin(hpc_names)) &amp; (electrodes.probe_electrode == 0)\n]\n</pre> hpc_names = [\"ca1\", \"hippocampus\", \"CA1\", \"Hippocampus\"] electrodes.loc[     (electrodes.region_name.isin(hpc_names)) &amp; (electrodes.probe_electrode == 0) ] Out[21]: probe_id probe_shank probe_electrode name original_reference_electrode x y z filtering impedance bad_channel x_warped y_warped z_warped contacts region_name subregion_name subsubregion_name nwb_file_name electrode_group_name electrode_id lfp_merge_id filter_name filter_sampling_rate target_interval_list_name lfp_band_sampling_rate lfp_electrode_group_name reference_elect_id region_id mediumnwb20230802_.nwb 0 0 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 0 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 1 4 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 4 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 10 40 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 40 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 11 44 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 44 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 13 52 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 52 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 14 56 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 56 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 15 60 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 60 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 16 64 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 64 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 17 68 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 68 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 18 72 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 72 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 19 76 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 76 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 20 80 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 80 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 21 84 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 84 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 22 88 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 88 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 23 92 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 92 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 3 12 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 12 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 4 16 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 16 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 5 20 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 20 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 6 24 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 24 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 7 28 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 28 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 8 32 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 32 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None 9 36 e5c8a41b-5bb1-c12c-d306-e80e5491d6dd Ripple 150-250 Hz 1000 02_r1_ripple_demo 1000 test_hippocampus -1 16 tetrode_12.5 0 0 36 8 0.0 0.0 0.0 None 0.0 False 0.0 0.0 0.0 hippocampus None None <p>We only want the electrode_id to put in the <code>electrode_list</code>:</p> In\u00a0[22]: Copied! <pre>electrode_list = np.unique(\n    (\n        electrodes.loc[\n            (electrodes.region_name.isin(hpc_names))\n            &amp; (electrodes.probe_electrode == 0)\n        ]\n        .reset_index()\n        .electrode_id\n    ).tolist()\n)\n\nelectrode_list.sort()\n</pre> electrode_list = np.unique(     (         electrodes.loc[             (electrodes.region_name.isin(hpc_names))             &amp; (electrodes.probe_electrode == 0)         ]         .reset_index()         .electrode_id     ).tolist() )  electrode_list.sort() <p>By default, <code>set_lfp_electrodes</code> will use all the available electrodes from <code>LFPBandV1</code>.</p> <p>We can insert into <code>RippleLFPSelection</code> and the <code>RippleLFPElectrode</code> part table, passing the key for the entry from <code>LFPBandV1</code>, our <code>electrode_list</code>, and the <code>group_name</code> into <code>set_lfp_electrodes</code></p> In\u00a0[24]: Copied! <pre>group_name = \"CA1_test\"\n\nlfp_band_key = (\n    lfp_analysis.LFPBandV1()\n    &amp; {\n        \"filter_name\": filter_name,\n        \"nwb_file_name\": nwb_file_name,\n        \"lfp_band_sampling_rate\": 1000,\n    }\n).fetch1(\"KEY\")\n\nsgr.RippleLFPSelection.set_lfp_electrodes(\n    lfp_band_key,\n    electrode_list=electrode_list,\n    group_name=group_name,\n)\n</pre> group_name = \"CA1_test\"  lfp_band_key = (     lfp_analysis.LFPBandV1()     &amp; {         \"filter_name\": filter_name,         \"nwb_file_name\": nwb_file_name,         \"lfp_band_sampling_rate\": 1000,     } ).fetch1(\"KEY\")  sgr.RippleLFPSelection.set_lfp_electrodes(     lfp_band_key,     electrode_list=electrode_list,     group_name=group_name, ) In\u00a0[25]: Copied! <pre>sgr.RippleLFPSelection.RippleLFPElectrode()\n</pre> sgr.RippleLFPSelection.RippleLFPElectrode() Out[25]: <p>lfp_merge_id</p> <p>filter_name</p> descriptive name of this filter <p>filter_sampling_rate</p> sampling rate for this filter <p>nwb_file_name</p> name of the NWB file <p>target_interval_list_name</p> descriptive name of this interval list <p>lfp_band_sampling_rate</p> the sampling rate for this band <p>group_name</p> <p>lfp_electrode_group_name</p> the name of this group of electrodes <p>electrode_group_name</p> electrode group name from NWBFile <p>electrode_id</p> the unique number for this electrode <p>reference_elect_id</p> the reference electrode to use; -1 for no reference 0087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 0 0 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 1 4 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 11 44 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 12 49 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 13 52 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 14 56 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 16 64 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 17 68 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 18 72 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 19 76 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 2 8 -10087e094-8238-32b8-9e8d-ecb7d9352b3b Ripple 150-250 Hz 1000 Winnie20220714_.nwb Winnie20220714_.nwb_pos 9 valid times_LFP_default_difference 1000 CA1 tetrode_sample_Winnie 20 80 -1 <p>...</p> <p>Total: 60159</p> <p>Here's the ripple selection key we'll use downstream</p> In\u00a0[26]: Copied! <pre>rip_sel_key = (sgrip.RippleLFPSelection &amp; lfp_band_key).fetch1(\"KEY\")\n</pre> rip_sel_key = (sgrip.RippleLFPSelection &amp; lfp_band_key).fetch1(\"KEY\") In\u00a0[27]: Copied! <pre>sgr.RippleParameters().insert_default\n</pre> sgr.RippleParameters().insert_default Out[27]: <p>ripple_param_name</p> a name for this set of parameters <p>ripple_param_dict</p> dictionary of parameters default =BLOB=default_ms =BLOB=default_sharon =BLOB= <p>Total: 3</p> <p>Here are the default ripple parameters:</p> In\u00a0[49]: Copied! <pre>(sgrip.RippleParameters() &amp; {\"ripple_param_name\": \"default_trodes\"}).fetch1()\n</pre> (sgrip.RippleParameters() &amp; {\"ripple_param_name\": \"default_trodes\"}).fetch1() Out[49]: <pre>{'ripple_param_name': 'default_trodes',\n 'ripple_param_dict': {'speed_name': 'speed',\n  'ripple_detection_algorithm': 'Kay_ripple_detector',\n  'ripple_detection_params': {'speed_threshold': 4.0,\n   'minimum_duration': 0.015,\n   'zscore_threshold': 2.0,\n   'smoothing_sigma': 0.004,\n   'close_ripple_threshold': 0.0}}}</pre> <ul> <li><code>filter_name</code>: which bandpass filter is used</li> <li><code>speed_name</code>: the name of the speed parameters in <code>IntervalPositionInfo</code></li> </ul> <p>For the <code>Kay_ripple_detector</code> (options are currently Kay and Karlsson, see <code>ripple_detection</code> package for specifics) the parameters are:</p> <ul> <li><code>speed_threshold</code> (cm/s): maximum speed the animal can move</li> <li><code>minimum_duration</code> (s): minimum time above threshold</li> <li><code>zscore_threshold</code> (std): minimum value to be considered a ripple, in standard deviations from mean</li> <li><code>smoothing_sigma</code> (s): how much to smooth the signal in time</li> <li><code>close_ripple_threshold</code> (s): exclude ripples closer than this amount</li> </ul> In\u00a0[41]: Copied! <pre>pos_key = {\n    \"nwb_file_name\": nwb_file_name,\n    \"trodes_pos_params_name\": \"single_led\",\n    \"interval_list_name\": \"pos 0 valid times\",\n}\nsgp.TrodesPosSelection().insert1(pos_key, skip_duplicates=True)\nsgp.TrodesPosV1.populate(pos_key, display_progress=True)\nsgp.TrodesPosV1 &amp; pos_key\n</pre> pos_key = {     \"nwb_file_name\": nwb_file_name,     \"trodes_pos_params_name\": \"single_led\",     \"interval_list_name\": \"pos 0 valid times\", } sgp.TrodesPosSelection().insert1(pos_key, skip_duplicates=True) sgp.TrodesPosV1.populate(pos_key, display_progress=True) sgp.TrodesPosV1 &amp; pos_key Out[41]: <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters <p>analysis_file_name</p> name of the file <p>position_object_id</p> <p>orientation_object_id</p> <p>velocity_object_id</p> mediumnwb20230802_.nwb pos 0 valid times single_led mediumnwb20230802_9GTXMUKTK1.nwb 6d725947-3ba0-4cbe-9483-e77b897ba1ab 848f5b77-cf49-41c2-906d-06b58a731086 41cf08d3-f114-4201-b7d8-d6e541015b42 <p>Total: 1</p> In\u00a0[46]: Copied! <pre>from spyglass.position import PositionOutput\n\npos_key = PositionOutput.merge_get_part(pos_key).fetch1(\"KEY\")\n(PositionOutput &amp; pos_key).fetch1_dataframe()\n</pre> from spyglass.position import PositionOutput  pos_key = PositionOutput.merge_get_part(pos_key).fetch1(\"KEY\") (PositionOutput &amp; pos_key).fetch1_dataframe() Out[46]: video_frame_ind position_x position_y orientation velocity_x velocity_y speed time 1.625936e+09 0 180.251195 162.885335 1.584011 -1.062553 1.052022 1.495249 1.625936e+09 1 180.090400 163.287322 1.579705 -1.162356 0.674062 1.343663 1.625936e+09 2 180.143998 163.421318 1.581228 -1.218606 0.243100 1.242618 1.625936e+09 3 179.983203 162.938933 1.576679 -1.243190 -0.085662 1.246138 1.625936e+09 4 180.197597 162.670942 1.582475 -1.230759 -0.224637 1.251091 ... ... ... ... ... ... ... ... 1.625937e+09 44186 38.483603 113.574868 -1.402199 1.008364 -0.865117 1.328618 1.625937e+09 44187 38.483603 113.521270 -1.398606 0.603148 -0.534938 0.806192 1.625937e+09 44188 38.430005 113.574868 -1.416478 0.256839 -0.219871 0.338096 1.625937e+09 44189 38.376407 113.574868 -1.432157 0.017772 0.025349 0.030958 1.625937e+09 44190 38.376407 113.628467 -1.435269 -0.107231 0.174792 0.205063 <p>44191 rows \u00d7 7 columns</p> <p>We'll use the <code>speed</code> above as part of <code>RippleParameters</code>. Ensure your selected ripple parameters value for <code>speed_name</code> matches for your data.</p> <p>Now we can put everything together.</p> In\u00a0[52]: Copied! <pre>key = {\n    \"ripple_param_name\": \"default_trodes\",\n    **rip_sel_key,\n    \"pos_merge_id\": pos_key[\"merge_id\"],\n}\nsgrip.RippleTimesV1().populate(key)\n</pre> key = {     \"ripple_param_name\": \"default_trodes\",     **rip_sel_key,     \"pos_merge_id\": pos_key[\"merge_id\"], } sgrip.RippleTimesV1().populate(key) <p>And then <code>fetch1_dataframe</code> for ripple times</p> In\u00a0[53]: Copied! <pre>ripple_times = (sgrip.RippleTimesV1() &amp; key).fetch1_dataframe()\nripple_times\n</pre> ripple_times = (sgrip.RippleTimesV1() &amp; key).fetch1_dataframe() ripple_times Out[53]: start_time end_time id 0 1.625936e+09 1.625936e+09 1 1.625936e+09 1.625936e+09 2 1.625936e+09 1.625936e+09 3 1.625936e+09 1.625936e+09 4 1.625936e+09 1.625936e+09 5 1.625936e+09 1.625936e+09 6 1.625936e+09 1.625936e+09 7 1.625936e+09 1.625936e+09 8 1.625936e+09 1.625936e+09 9 1.625936e+09 1.625936e+09 10 1.625936e+09 1.625936e+09 11 1.625936e+09 1.625936e+09 12 1.625936e+09 1.625936e+09 13 1.625936e+09 1.625936e+09 14 1.625936e+09 1.625936e+09 15 1.625936e+09 1.625936e+09 16 1.625936e+09 1.625936e+09 17 1.625936e+09 1.625936e+09 18 1.625936e+09 1.625936e+09 19 1.625936e+09 1.625936e+09 20 1.625936e+09 1.625936e+09 21 1.625936e+09 1.625936e+09 22 1.625936e+09 1.625936e+09 23 1.625936e+09 1.625936e+09 24 1.625936e+09 1.625936e+09 25 1.625936e+09 1.625936e+09 26 1.625936e+09 1.625936e+09 27 1.625936e+09 1.625936e+09 28 1.625936e+09 1.625936e+09 29 1.625936e+09 1.625936e+09 30 1.625936e+09 1.625936e+09 31 1.625936e+09 1.625936e+09 32 1.625936e+09 1.625936e+09 33 1.625936e+09 1.625936e+09 34 1.625936e+09 1.625936e+09 35 1.625936e+09 1.625936e+09 36 1.625936e+09 1.625936e+09 37 1.625936e+09 1.625936e+09 38 1.625936e+09 1.625936e+09 <p>We can also inspect the lfp trace at these ripple times.</p> <ul> <li>Note: The ripple detection algorithm depends on estimates of the standard deviation of power in the ripple band. Running analysis on longer intervals will lead to better estimates of this value, and thereby better segmentation of ripple events</li> </ul> In\u00a0[103]: Copied! <pre>import matplotlib.pyplot as plt\n\nripple_band_df = (lfp_band.LFPBandV1() &amp; lfp_band_key).fetch1_dataframe()\n\nwindow = 0.1\ni = -1\nripple_start = ripple_times.iloc[i].start_time\nripple_end = ripple_times.iloc[i].end_time\nplt.plot(\n    ripple_band_df.loc[ripple_start - window : ripple_end + window].index,\n    ripple_band_df.loc[ripple_start - window : ripple_end + window].iloc[\n        :, ::15\n    ],\n)\nplt.axvline(ripple_start, color=\"r\")\nplt.axvline(ripple_end, color=\"r\")\n\nplt.xlabel(\"Time (s)\")\nplt.ylabel(\"Voltage (uV)\")\n</pre> import matplotlib.pyplot as plt  ripple_band_df = (lfp_band.LFPBandV1() &amp; lfp_band_key).fetch1_dataframe()  window = 0.1 i = -1 ripple_start = ripple_times.iloc[i].start_time ripple_end = ripple_times.iloc[i].end_time plt.plot(     ripple_band_df.loc[ripple_start - window : ripple_end + window].index,     ripple_band_df.loc[ripple_start - window : ripple_end + window].iloc[         :, ::15     ], ) plt.axvline(ripple_start, color=\"r\") plt.axvline(ripple_end, color=\"r\")  plt.xlabel(\"Time (s)\") plt.ylabel(\"Voltage (uV)\") Out[103]: <pre>Text(0, 0.5, 'Voltage (uV)')</pre>"}, {"location": "notebooks/32_Ripple_Detection/#ripple-detection", "title": "Ripple Detection\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#overview", "title": "Overview\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#imports", "title": "Imports\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#generate-lfp-ripple-band", "title": "Generate LFP Ripple Band\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#make-lfp", "title": "Make LFP\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#populate-ripple-band", "title": "Populate Ripple Band\u00b6", "text": "<p>We now create a filter for this frequency band</p>"}, {"location": "notebooks/32_Ripple_Detection/#selecting-ripple-analysis-electrodes", "title": "Selecting Ripple Analysis Electrodes\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#setting-ripple-parameters", "title": "Setting Ripple Parameters\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#check-interval-speed", "title": "Check interval speed\u00b6", "text": "<p>The speed for this interval should exist under the default position parameter set and for a given interval. We can quickly populate this here</p>"}, {"location": "notebooks/32_Ripple_Detection/#run-ripple-detection", "title": "Run Ripple Detection\u00b6", "text": ""}, {"location": "notebooks/32_Ripple_Detection/#up-next", "title": "Up Next\u00b6", "text": "<p>Next, we'll extract mark indicator.</p>"}, {"location": "notebooks/40_Extracting_Clusterless_Waveform_Features/", "title": "Extracting Waveforms", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on clusterless decoding in Spyglass</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook</li> <li>Prior to running, please familiarize yourself with the spike sorting pipeline and generate input position data with either the Trodes or DLC notebooks (1, 2, 3).</li> </ul> <p>The goal of this notebook is to populate the <code>UnitWaveformFeatures</code> table, which depends <code>SpikeSortingOutput</code>. This table contains the features of the waveforms of each unit.</p> <p>While clusterless decoding avoids actual spike sorting, we need to pass through these tables to maintain (relative) pipeline simplicity. Pass-through tables keep spike sorting and clusterless waveform extraction as similar as possible, by using shared steps. Here, \"spike sorting\" involves simple thresholding (sorter: clusterless_thresholder).</p> In\u00a0[1]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info <p>First, if you haven't inserted the the <code>mediumnwb20230802.wnb</code> file into the database, you should do so now. This is the file that we will use for the decoding tutorials.</p> <p>It is a truncated version of the full NWB file, so it will run faster, but bigger than the minirec file we used in the previous tutorials so that decoding makes sense.</p> In\u00a0[2]: Copied! <pre>from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\nimport spyglass.data_import as sgi\nimport spyglass.position as sgp\n\n# Insert the nwb file\nnwb_file_name = \"mediumnwb20230802.nwb\"\nnwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\nsgi.insert_sessions(nwb_file_name)\n\n# Position\nsgp.v1.TrodesPosParams.insert_default()\n\ninterval_list_name = \"pos 0 valid times\"\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": interval_list_name,\n    \"trodes_pos_params_name\": \"default\",\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_key)\n</pre> from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename import spyglass.data_import as sgi import spyglass.position as sgp  # Insert the nwb file nwb_file_name = \"mediumnwb20230802.nwb\" nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name) sgi.insert_sessions(nwb_file_name)  # Position sgp.v1.TrodesPosParams.insert_default()  interval_list_name = \"pos 0 valid times\"  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": interval_list_name,     \"trodes_pos_params_name\": \"default\", } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_key) <pre>[2024-01-17 22:14:51,194][INFO]: Connecting root@localhost:3306\n[2024-01-17 22:14:51,274][INFO]: Connected root@localhost:3306\n/Users/edeno/Documents/GitHub/spyglass/src/spyglass/data_import/insert_sessions.py:58: UserWarning: Cannot insert data from mediumnwb20230802.nwb: mediumnwb20230802_.nwb is already in Nwbfile table.\n  warnings.warn(\n</pre> <p>These next steps are the same as in the Spike Sorting notebook, but we'll repeat them here for clarity. These are pre-processing steps that are shared between spike sorting and clusterless decoding.</p> <p>We first set the <code>SortGroup</code> to define which contacts are sorted together.</p> <p>We then setup for spike sorting by bandpass filtering and whitening the data via the <code>SpikeSortingRecording</code> table.</p> In\u00a0[3]: Copied! <pre>import spyglass.spikesorting.v1 as sgs\n\nsgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_copy_file_name)\n\nsort_group_ids = (sgs.SortGroup &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(\n    \"sort_group_id\"\n)\n\ngroup_keys = []\nfor sort_group_id in sort_group_ids:\n    key = {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"sort_group_id\": sort_group_id,\n        \"interval_list_name\": interval_list_name,\n        \"preproc_param_name\": \"default\",\n        \"team_name\": \"Alison Comrie\",\n    }\n    group_keys.append(key)\n    sgs.SpikeSortingRecordingSelection.insert_selection(key)\n\nsgs.SpikeSortingRecording.populate(group_keys)\n</pre> import spyglass.spikesorting.v1 as sgs  sgs.SortGroup.set_group_by_shank(nwb_file_name=nwb_copy_file_name)  sort_group_ids = (sgs.SortGroup &amp; {\"nwb_file_name\": nwb_copy_file_name}).fetch(     \"sort_group_id\" )  group_keys = [] for sort_group_id in sort_group_ids:     key = {         \"nwb_file_name\": nwb_copy_file_name,         \"sort_group_id\": sort_group_id,         \"interval_list_name\": interval_list_name,         \"preproc_param_name\": \"default\",         \"team_name\": \"Alison Comrie\",     }     group_keys.append(key)     sgs.SpikeSortingRecordingSelection.insert_selection(key)  sgs.SpikeSortingRecording.populate(group_keys) <pre>[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:55][WARNING] Spyglass: Similar row(s) already inserted.\n</pre> <p>Next we do artifact detection. Here we skip it by setting the <code>artifact_param_name</code> to <code>None</code>, but in practice you should detect artifacts as it will affect the decoding.</p> In\u00a0[4]: Copied! <pre>recording_ids = (\n    sgs.SpikeSortingRecordingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name}\n).fetch(\"recording_id\")\n\ngroup_keys = []\nfor recording_id in recording_ids:\n    key = {\n        \"recording_id\": recording_id,\n        \"artifact_param_name\": \"none\",\n    }\n    group_keys.append(key)\n    sgs.ArtifactDetectionSelection.insert_selection(key)\n\nsgs.ArtifactDetection.populate(group_keys)\n</pre> recording_ids = (     sgs.SpikeSortingRecordingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name} ).fetch(\"recording_id\")  group_keys = [] for recording_id in recording_ids:     key = {         \"recording_id\": recording_id,         \"artifact_param_name\": \"none\",     }     group_keys.append(key)     sgs.ArtifactDetectionSelection.insert_selection(key)  sgs.ArtifactDetection.populate(group_keys) <pre>[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n[22:14:56][WARNING] Spyglass: Similar row(s) already inserted.\n</pre> <p>Now we run the \"spike sorting\", which in our case is simply thresholding the signal to find spikes. We use the <code>SpikeSorting</code> table to store the results. Note that <code>sorter_param_name</code> defines the parameters for thresholding the signal.</p> In\u00a0[5]: Copied! <pre>group_keys = []\nfor recording_id in recording_ids:\n    key = {\n        \"recording_id\": recording_id,\n        \"sorter\": \"clusterless_thresholder\",\n        \"sorter_param_name\": \"default_clusterless\",\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": str(\n            (\n                sgs.ArtifactDetectionSelection &amp; {\"recording_id\": recording_id}\n            ).fetch1(\"artifact_id\")\n        ),\n    }\n    group_keys.append(key)\n    sgs.SpikeSortingSelection.insert_selection(key)\n\nsgs.SpikeSorting.populate(group_keys)\n</pre> group_keys = [] for recording_id in recording_ids:     key = {         \"recording_id\": recording_id,         \"sorter\": \"clusterless_thresholder\",         \"sorter_param_name\": \"default_clusterless\",         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": str(             (                 sgs.ArtifactDetectionSelection &amp; {\"recording_id\": recording_id}             ).fetch1(\"artifact_id\")         ),     }     group_keys.append(key)     sgs.SpikeSortingSelection.insert_selection(key)  sgs.SpikeSorting.populate(group_keys) <pre>[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n[22:14:56][INFO] Spyglass: Similar row(s) already inserted.\n</pre> <p>For clusterless decoding we do not need any manual curation, but for the sake of the pipeline, we need to store the output of the thresholding in the <code>CurationV1</code> table and insert this into the <code>SpikeSortingOutput</code> table.</p> In\u00a0[6]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\nsorting_ids = (\n    sgs.SpikeSortingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name}\n).fetch(\"sorting_id\")\n\nfor sorting_id in sorting_ids:\n    try:\n        sgs.CurationV1.insert_curation(sorting_id=sorting_id)\n    except KeyError as e:\n        pass\n\nSpikeSortingOutput.insert(\n    sgs.CurationV1().fetch(\"KEY\"),\n    part_name=\"CurationV1\",\n    skip_duplicates=True,\n)\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  sorting_ids = (     sgs.SpikeSortingSelection &amp; {\"nwb_file_name\": nwb_copy_file_name} ).fetch(\"sorting_id\")  for sorting_id in sorting_ids:     try:         sgs.CurationV1.insert_curation(sorting_id=sorting_id)     except KeyError as e:         pass  SpikeSortingOutput.insert(     sgs.CurationV1().fetch(\"KEY\"),     part_name=\"CurationV1\",     skip_duplicates=True, ) <p>Finally, we extract the waveform features of each SortGroup. This is done by the <code>UnitWaveformFeatures</code> table.</p> <p>To set this up, we use the <code>WaveformFeaturesParams</code> to define the time around the spike that we want to use for feature extraction, and which features to extract. Here is an example of the parameters used for extraction the amplitude of the negative peak of the waveform:</p> <pre>waveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    }\n}\n</pre> <p>We see that we want 0.5 ms of time before and after the peak of the negative spike. We also see that we want to extract the amplitude of the negative peak, and that we do not want to estimate the peak time (since we know it is at 0 ms).</p> <p>You can define other features to extract such as spatial location of the spike:</p> <pre>waveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    },\n    \"spike location\": {}\n}\n</pre> <p>Note: Members of the Frank Lab can use \"ampl_10_jobs_v2\" instead of \"amplitude\" for significant speed improvements.</p> In\u00a0[7]: Copied! <pre>from spyglass.decoding.v1.waveform_features import WaveformFeaturesParams\n\nwaveform_extraction_params = {\n    \"ms_before\": 0.5,\n    \"ms_after\": 0.5,\n    \"max_spikes_per_unit\": None,\n    \"n_jobs\": 5,\n    \"total_memory\": \"5G\",\n}\nwaveform_feature_params = {\n    \"amplitude\": {\n        \"peak_sign\": \"neg\",\n        \"estimate_peak_time\": False,\n    }\n}\n\nWaveformFeaturesParams.insert1(\n    {\n        \"features_param_name\": \"amplitude\",\n        \"params\": {\n            \"waveform_extraction_params\": waveform_extraction_params,\n            \"waveform_feature_params\": waveform_feature_params,\n        },\n    },\n    skip_duplicates=True,\n)\n\nWaveformFeaturesParams()\n</pre> from spyglass.decoding.v1.waveform_features import WaveformFeaturesParams  waveform_extraction_params = {     \"ms_before\": 0.5,     \"ms_after\": 0.5,     \"max_spikes_per_unit\": None,     \"n_jobs\": 5,     \"total_memory\": \"5G\", } waveform_feature_params = {     \"amplitude\": {         \"peak_sign\": \"neg\",         \"estimate_peak_time\": False,     } }  WaveformFeaturesParams.insert1(     {         \"features_param_name\": \"amplitude\",         \"params\": {             \"waveform_extraction_params\": waveform_extraction_params,             \"waveform_feature_params\": waveform_feature_params,         },     },     skip_duplicates=True, )  WaveformFeaturesParams() Out[7]: <p>features_param_name</p> a name for this set of parameters <p>params</p> the parameters for the waveform features amplitude =BLOB=amplitude, spike_location =BLOB= <p>Total: 2</p> <p>Now that we've inserted the waveform features parameters, we need to define which parameters to use for each SortGroup. This is done by the <code>UnitWaveformFeaturesSelection</code> table. We need to link the primary key <code>merge_id</code> from the <code>SpikeSortingOutput</code> table to a features parameter set.</p> In\u00a0[8]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\nUnitWaveformFeaturesSelection()\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection  UnitWaveformFeaturesSelection() Out[8]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>First we find the units we need:</p> In\u00a0[9]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\n\nmerge_ids = (\n    (SpikeSortingOutput.CurationV1 * sgs.SpikeSortingSelection)\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"sorter\": \"clusterless_thresholder\",\n        \"sorter_param_name\": \"default_clusterless\",\n    }\n).fetch(\"merge_id\")\nmerge_ids\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput  merge_ids = (     (SpikeSortingOutput.CurationV1 * sgs.SpikeSortingSelection)     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"sorter\": \"clusterless_thresholder\",         \"sorter_param_name\": \"default_clusterless\",     } ).fetch(\"merge_id\") merge_ids Out[9]: <pre>array([UUID('485a4ddf-332d-35b5-3ad4-0561736c1844'),\n       UUID('6acb99b8-6a0c-eb83-1141-5f603c5895e0'),\n       UUID('f7237e18-4e73-4aee-805b-90735e9147de'),\n       UUID('7e3fa66e-727e-1541-819a-b01309bb30ae'),\n       UUID('6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5'),\n       UUID('e0e9133a-7a4e-1321-a43a-e8afcb2f25da'),\n       UUID('9959b614-2318-f597-6651-a3a82124d28a'),\n       UUID('c0eb6455-fc41-c200-b62e-e3ca81b9a3f7'),\n       UUID('912e250e-56d8-ee33-4525-c844d810971b'),\n       UUID('d7d2c97a-0e6e-d1b8-735c-d55dc66a30e1'),\n       UUID('abb92dce-4410-8f17-a501-a4104bda0dcf'),\n       UUID('74e10781-1228-4075-0870-af224024ffdc'),\n       UUID('8bbddc0f-d6ae-6260-9400-f884a6e25ae8'),\n       UUID('614d796c-0b95-6364-aaa0-b6cb1e7bbb83'),\n       UUID('b332482b-e430-169d-8ac0-0a73ce968ed7'),\n       UUID('86897349-ff68-ac72-02eb-739dd88936e6'),\n       UUID('4a712103-c223-864f-82e0-6c23de79cc14'),\n       UUID('cf858380-e8a3-49de-c2a9-1a277e307a68'),\n       UUID('cc4ee561-f974-f8e5-0ea4-83185263ac67'),\n       UUID('4a72c253-b3ca-8c13-e615-736a7ebff35c'),\n       UUID('b92a94d8-ee1e-2097-a81f-5c1e1556ed24'),\n       UUID('5c53bd33-d57c-fbba-e0fb-55e0bcb85d03'),\n       UUID('0751a1e1-a406-7f87-ae6f-ce4ffc60621c')], dtype=object)</pre> <p>Then we link them with the features parameters:</p> In\u00a0[10]: Copied! <pre>selection_keys = [\n    {\n        \"spikesorting_merge_id\": merge_id,\n        \"features_param_name\": \"amplitude\",\n    }\n    for merge_id in merge_ids\n]\nUnitWaveformFeaturesSelection.insert(selection_keys, skip_duplicates=True)\n\nUnitWaveformFeaturesSelection &amp; selection_keys\n</pre> selection_keys = [     {         \"spikesorting_merge_id\": merge_id,         \"features_param_name\": \"amplitude\",     }     for merge_id in merge_ids ] UnitWaveformFeaturesSelection.insert(selection_keys, skip_duplicates=True)  UnitWaveformFeaturesSelection &amp; selection_keys Out[10]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>Finally, we extract the waveform features, by populating the <code>UnitWaveformFeatures</code> table:</p> In\u00a0[11]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeatures\n\nUnitWaveformFeatures.populate(selection_keys)\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeatures  UnitWaveformFeatures.populate(selection_keys) <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:08,494][WARNING]: Skipped checksum for file with hash: 6629fd95-636a-4ad4-c9af-cee507de2130, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_AMBBKQ9RIY.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:08][INFO] Spyglass: Writing new NWB file mediumnwb20230802_NQEPSMKPK0.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:19,450][WARNING]: Skipped checksum for file with hash: 6d04cbdb-e1e4-f44f-7274-0e1ab0356d75, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_W1MLF0Q86S.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:19][INFO] Spyglass: Writing new NWB file mediumnwb20230802_F02UG5Z5FR.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:30,787][WARNING]: Skipped checksum for file with hash: 8993754e-7dbe-94a1-403d-8c55aa9c6c42, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_JN4A4GSLZB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:31][INFO] Spyglass: Writing new NWB file mediumnwb20230802_OTV91MLKDT.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:41,633][WARNING]: Skipped checksum for file with hash: 9e24661c-b021-6ad4-f224-89e331334f18, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T2DBO3EMZ8.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:41][INFO] Spyglass: Writing new NWB file mediumnwb20230802_TSPNTCGNN1.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:15:52,561][WARNING]: Skipped checksum for file with hash: f64f34ee-e72d-e566-a048-65f2ea31708a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_USMRXAAV8I.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:15:52][INFO] Spyglass: Writing new NWB file mediumnwb20230802_QSK70WFDJH.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:03,559][WARNING]: Skipped checksum for file with hash: 6d13e338-41bd-b011-beb5-4de53d9d467b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_JA2OA12RPN.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:03][INFO] Spyglass: Writing new NWB file mediumnwb20230802_DO45HKXYTB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:14,288][WARNING]: Skipped checksum for file with hash: d740eb7d-ce29-e140-06a2-c56655e0842a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_L92EE1VRPB.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:14][INFO] Spyglass: Writing new NWB file mediumnwb20230802_KFIYRJ4HFO.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:24,130][WARNING]: Skipped checksum for file with hash: 1f386cd3-89da-0233-03ff-76ba94e91a3a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TX2ZX3DAP4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:24][INFO] Spyglass: Writing new NWB file mediumnwb20230802_0YIM5K3H47.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:35,048][WARNING]: Skipped checksum for file with hash: fa76d419-77a4-697a-325d-5c2ddbe517f9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0R6AWXMC6G.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:35][INFO] Spyglass: Writing new NWB file mediumnwb20230802_CTLEGE2TWZ.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:46,009][WARNING]: Skipped checksum for file with hash: ce4cb0c3-3dd0-70fd-8ea0-98a8b84592d9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7UIA2ILMG6.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:46][INFO] Spyglass: Writing new NWB file mediumnwb20230802_7EN0N1U4U1.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:16:56,814][WARNING]: Skipped checksum for file with hash: e43f95ff-9779-b980-00a3-99e104864462, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_AKOI7OTASI.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:16:57][INFO] Spyglass: Writing new NWB file mediumnwb20230802_DHKWBWWAMC.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:05,013][WARNING]: Skipped checksum for file with hash: ff81d274-17f7-702d-a2b4-92ac43c29316, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Y2YF504C5D.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:05][INFO] Spyglass: Writing new NWB file mediumnwb20230802_PEN0D79Q0B.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:15,903][WARNING]: Skipped checksum for file with hash: e282a8e5-844b-20f6-345c-cded12e761a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DUNM1TZUGR.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:16][INFO] Spyglass: Writing new NWB file mediumnwb20230802_WP7SIXDJ2A.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:26,609][WARNING]: Skipped checksum for file with hash: 7d05460d-7366-27c9-2ba7-de2ad5d402f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_4JXWFJ3JRI.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:26][INFO] Spyglass: Writing new NWB file mediumnwb20230802_B82OS6W1QA.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:37,652][WARNING]: Skipped checksum for file with hash: c202eb9e-ca43-0a72-4086-57a5bb6eb937, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_5TY04H3B5T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:37][INFO] Spyglass: Writing new NWB file mediumnwb20230802_XO17FQLN6T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:47,269][WARNING]: Skipped checksum for file with hash: 4357905c-c6b9-3990-4d62-740a54cfc667, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_X84BYVM2B0.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:47][INFO] Spyglass: Writing new NWB file mediumnwb20230802_OCFI0GFLZ9.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:17:58,240][WARNING]: Skipped checksum for file with hash: 4c1103ac-eaca-b282-e5ff-aa2194e65a43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_2R6VQ8EDL4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:17:58][INFO] Spyglass: Writing new NWB file mediumnwb20230802_60M9VSZX0W.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:09,119][WARNING]: Skipped checksum for file with hash: 023c874f-8114-3ef6-7fcf-813844787d5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_L7HDY9IDHO.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:09][INFO] Spyglass: Writing new NWB file mediumnwb20230802_Z5HJ68LHYW.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:20,605][WARNING]: Skipped checksum for file with hash: fde8b240-6adc-86f0-6391-f3f6fad72ee9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_HWU3E4EKP4.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:20][INFO] Spyglass: Writing new NWB file mediumnwb20230802_U5U5JVGY4F.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:31,780][WARNING]: Skipped checksum for file with hash: c592e63b-4db1-40be-632e-0180e6fa02d7, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_SGAU9PX7US.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:32][INFO] Spyglass: Writing new NWB file mediumnwb20230802_0D5Z0NSIP8.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:42,644][WARNING]: Skipped checksum for file with hash: 148d9058-e6dc-e959-4c4d-75db9aa0b6e4, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EF6N6XI3AH.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:42][INFO] Spyglass: Writing new NWB file mediumnwb20230802_EYV2NARUKU.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:18:54,570][WARNING]: Skipped checksum for file with hash: b4b6404f-aaf8-c4cc-9abe-ceea56e103f3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_O7ZZ0F1XN7.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:18:54][INFO] Spyglass: Writing new NWB file mediumnwb20230802_T4XBCIW44T.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/spikeinterface/core/waveform_extractor.py:275: UserWarning: Sorting object is not dumpable, which might result in downstream errors for parallel processing. To make the sorting dumpable, use the `sorting.save()` function.\n  warn(\n</pre> <pre>extract waveforms memmap:   0%|          | 0/2 [00:00&lt;?, ?it/s]</pre> <pre>[2024-01-17 22:19:05,568][WARNING]: Skipped checksum for file with hash: 26f7bdc7-da8d-6ad5-3f4a-554ceb48755e, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0TKF5589B7.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[22:19:05][INFO] Spyglass: Writing new NWB file mediumnwb20230802_UD55CR8LZK.nwb\n</pre> In\u00a0[12]: Copied! <pre>UnitWaveformFeatures &amp; selection_keys\n</pre> UnitWaveformFeatures &amp; selection_keys Out[12]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters <p>analysis_file_name</p> name of the file <p>object_id</p> the NWB object that stores the waveforms 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude mediumnwb20230802_NQEPSMKPK0.nwb 8607d6a6-213c-431d-ab99-70196b6cf0bf485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude mediumnwb20230802_F02UG5Z5FR.nwb 9f693a74-a203-4628-b3ec-50a32b3549d84a712103-c223-864f-82e0-6c23de79cc14 amplitude mediumnwb20230802_OTV91MLKDT.nwb 648953e8-1891-4c90-9756-d6b7cc2b7c3d4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude mediumnwb20230802_TSPNTCGNN1.nwb 6d0af664-f811-4781-9a23-cac437fb2d155c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude mediumnwb20230802_QSK70WFDJH.nwb a67ed5bb-3edd-465e-8737-ee08f3e7d7d5614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude mediumnwb20230802_DO45HKXYTB.nwb 13218b00-bf34-455c-9c38-c3b3174d40096acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude mediumnwb20230802_KFIYRJ4HFO.nwb d892bb47-94fc-4c29-acab-d5b3d9565c976d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude mediumnwb20230802_0YIM5K3H47.nwb 60f4d280-a42a-4a77-9c35-9bd4d2c7699174e10781-1228-4075-0870-af224024ffdc amplitude mediumnwb20230802_CTLEGE2TWZ.nwb 99f51e3d-54b5-41d7-a61e-7013b22fb0667e3fa66e-727e-1541-819a-b01309bb30ae amplitude mediumnwb20230802_7EN0N1U4U1.nwb 535b28d1-c9b5-4d7d-a4f5-4d80508542b486897349-ff68-ac72-02eb-739dd88936e6 amplitude mediumnwb20230802_DHKWBWWAMC.nwb 67ee1547-c570-4746-b886-748d96075b548bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude mediumnwb20230802_PEN0D79Q0B.nwb 5dd7b87f-4cf0-4a91-a281-15c6f6c86d61 <p>...</p> <p>Total: 23</p> <p>Now that we've extracted the data, we can inspect the results. Let's fetch the data:</p> In\u00a0[13]: Copied! <pre>spike_times, spike_waveform_features = (\n    UnitWaveformFeatures &amp; selection_keys\n).fetch_data()\n</pre> spike_times, spike_waveform_features = (     UnitWaveformFeatures &amp; selection_keys ).fetch_data() <pre>[2024-01-17 22:19:07,354][WARNING]: Skipped checksum for file with hash: a7c9b1d9-d1a2-7f40-9127-206e83a87006, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_NQEPSMKPK0.nwb\n[2024-01-17 22:19:07,359][WARNING]: Skipped checksum for file with hash: ec7faa5b-3847-6649-1a93-74ebd50dcfb9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_F02UG5Z5FR.nwb\n[2024-01-17 22:19:07,369][WARNING]: Skipped checksum for file with hash: 8e964932-96ab-e1c9-2133-edce8eacab5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OTV91MLKDT.nwb\n[2024-01-17 22:19:07,379][WARNING]: Skipped checksum for file with hash: 895bac7b-bfd6-b4f2-b2ad-460362aaafa8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TSPNTCGNN1.nwb\n[2024-01-17 22:19:07,382][WARNING]: Skipped checksum for file with hash: 58713583-cf49-4527-7707-105f9c9ee477, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_QSK70WFDJH.nwb\n[2024-01-17 22:19:07,385][WARNING]: Skipped checksum for file with hash: a64829f8-ab12-fecc-eda9-a22b90b20d43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DO45HKXYTB.nwb\n[2024-01-17 22:19:07,391][WARNING]: Skipped checksum for file with hash: 3a580271-9126-8e57-048e-a7bbb3f917b9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_KFIYRJ4HFO.nwb\n[2024-01-17 22:19:07,395][WARNING]: Skipped checksum for file with hash: 13cf8ad9-023c-c9b7-05c3-eaa3330304f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0YIM5K3H47.nwb\n[2024-01-17 22:19:07,397][WARNING]: Skipped checksum for file with hash: 7ce8a640-0a25-4866-6d5a-aa2c65f0aca5, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_CTLEGE2TWZ.nwb\n[2024-01-17 22:19:07,399][WARNING]: Skipped checksum for file with hash: aa657f4f-f409-d444-8b32-31d37abe0797, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7EN0N1U4U1.nwb\n[2024-01-17 22:19:07,401][WARNING]: Skipped checksum for file with hash: f3b4bd22-1439-e6d2-4e15-aa3650143fdf, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DHKWBWWAMC.nwb\n[2024-01-17 22:19:07,404][WARNING]: Skipped checksum for file with hash: 68eac0b2-e5be-e0c5-9eae-cd8dbe6676a8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_PEN0D79Q0B.nwb\n[2024-01-17 22:19:07,407][WARNING]: Skipped checksum for file with hash: c8b95099-2cb3-df0b-5ab1-7a5e120a8e2f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_WP7SIXDJ2A.nwb\n[2024-01-17 22:19:07,409][WARNING]: Skipped checksum for file with hash: 8fae8089-f683-5f0a-4e59-c71d6ee14f38, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_B82OS6W1QA.nwb\n[2024-01-17 22:19:07,412][WARNING]: Skipped checksum for file with hash: dd9d0f51-6445-b368-32bd-b1f142bf6ed3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_XO17FQLN6T.nwb\n[2024-01-17 22:19:07,414][WARNING]: Skipped checksum for file with hash: 4e2cf5f5-ff7c-1a2b-db85-2d1c4f036fbd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OCFI0GFLZ9.nwb\n[2024-01-17 22:19:07,416][WARNING]: Skipped checksum for file with hash: 8691c252-0bd1-122b-8cf3-b89c4d0fdee0, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_60M9VSZX0W.nwb\n[2024-01-17 22:19:07,419][WARNING]: Skipped checksum for file with hash: 57b89835-8edb-e91d-0798-09d22fb4fbc9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Z5HJ68LHYW.nwb\n[2024-01-17 22:19:07,422][WARNING]: Skipped checksum for file with hash: 54401121-4426-86c9-72f7-e056bc16e99d, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_U5U5JVGY4F.nwb\n[2024-01-17 22:19:07,424][WARNING]: Skipped checksum for file with hash: 0ff21e84-2214-6911-2575-a9c92a541407, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0D5Z0NSIP8.nwb\n[2024-01-17 22:19:07,426][WARNING]: Skipped checksum for file with hash: 0949b006-5309-93c8-fd8b-1308e8130869, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EYV2NARUKU.nwb\n[2024-01-17 22:19:07,428][WARNING]: Skipped checksum for file with hash: b4b31e50-dfa2-0d02-514a-525782a81255, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T4XBCIW44T.nwb\n[2024-01-17 22:19:07,430][WARNING]: Skipped checksum for file with hash: c18a9ac4-06bc-4249-2bad-439d4f618421, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_UD55CR8LZK.nwb\n</pre> <p>Let's look at the features shape. This is a list corresponding to tetrodes, with each element being a numpy array of shape (n_spikes, n_features). The features in this case are the amplitude of each tetrode wire at the negative peak of the waveform.</p> In\u00a0[14]: Copied! <pre>for features in spike_waveform_features:\n    print(features.shape)\n</pre> for features in spike_waveform_features:     print(features.shape) <pre>(49808, 4)\n(21675, 4)\n(21024, 4)\n(51330, 4)\n(43804, 4)\n(6348, 4)\n(12188, 4)\n(2654, 4)\n(99400, 4)\n(8952, 4)\n(39886, 4)\n(18, 4)\n(44284, 4)\n(8283, 4)\n(36687, 4)\n(803, 4)\n(76353, 4)\n(11367, 4)\n(41622, 4)\n(106549, 4)\n(57394, 4)\n(30772, 4)\n(4198, 4)\n</pre> <p>We can plot the amplitudes to see if there is anything that looks neural and to look for outliers:</p> In\u00a0[15]: Copied! <pre>import matplotlib.pyplot as plt\n\ntetrode_ind = 1\nplt.scatter(\n    spike_waveform_features[tetrode_ind][:, 0],\n    spike_waveform_features[tetrode_ind][:, 1],\n    s=1,\n)\n</pre> import matplotlib.pyplot as plt  tetrode_ind = 1 plt.scatter(     spike_waveform_features[tetrode_ind][:, 0],     spike_waveform_features[tetrode_ind][:, 1],     s=1, ) Out[15]: <pre>&lt;matplotlib.collections.PathCollection at 0x1c61bf940&gt;</pre> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/41_Decoding_Clusterless/", "title": "Decoding Clusterless", "text": "In\u00a0[1]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info In\u00a0[2]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\nimport spyglass.spikesorting.v1 as sgs\nfrom spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\nsorter_keys = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorter\": \"clusterless_thresholder\",\n    \"sorter_param_name\": \"default_clusterless\",\n}\n\nfeature_key = {\"features_param_name\": \"amplitude\"}\n\n(sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 * (\n    UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")\n    &amp; feature_key\n)\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput import spyglass.spikesorting.v1 as sgs from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection   nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  sorter_keys = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorter\": \"clusterless_thresholder\",     \"sorter_param_name\": \"default_clusterless\", }  feature_key = {\"features_param_name\": \"amplitude\"}  (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 * (     UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")     &amp; feature_key ) <pre>[2024-01-29 10:24:06,266][INFO]: Connecting root@localhost:3306\n[2024-01-29 10:24:06,337][INFO]: Connected root@localhost:3306\n[10:24:07][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> Out[2]: <p>sorting_id</p> <p>merge_id</p> <p>features_param_name</p> a name for this set of parameters <p>recording_id</p> <p>sorter</p> <p>sorter_param_name</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>curation_id</p> 08a302b6-5505-40fa-b4d5-62162f8eef58 485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude 449b64e3-db0b-437e-a1b9-0d29928aa2dd clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 45f6b9a1-eef3-46eb-866d-d0999afebda6 00ca508ee-af4c-4a89-8181-d48bd209bfd4 6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude 328da21c-1d9c-41e2-9800-76b3484b707b clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 686d9951-1c0f-4d5e-9f5c-09e6fd8bdd4c 0209dc048-6fae-4315-b293-c06fff29f947 f7237e18-4e73-4aee-805b-90735e9147de amplitude aff78f2f-2ba0-412a-95cc-447c3a2f4683 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 719e8a86-fcf1-4ffc-8c1f-ea912f67ad5d 021a9a593-f6f3-4b82-99d7-8fc46556eff3 7e3fa66e-727e-1541-819a-b01309bb30ae amplitude 2402805a-04f9-4a88-9ccf-071376c8de19 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb d581b117-160e-4311-b096-7781a4de4394 0406a20e3-5a9f-4fec-b046-a6561f72461e 6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude f1427e00-2974-4301-b2ac-b4dc29277c51 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 0e848c38-9105-4ea4-b6ba-dbdd5b46a088 04131c51b-c56d-41fa-b046-46635fc17fd9 e0e9133a-7a4e-1321-a43a-e8afcb2f25da amplitude 9e332d82-1daf-4e92-bb50-12e4f9430875 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 9ed11db5-c42e-491a-8caf-7d9a37a65f13 04c5a629a-71d9-481d-ab11-a4cb0fc16087 9959b614-2318-f597-6651-a3a82124d28a amplitude 3a2c3eed-413a-452a-83c8-0e4648141bde clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 2b9fbf14-74a0-4294-a805-26702340aac9 04d629c07-1931-4e1f-a3a8-cbf1b72161e3 c0eb6455-fc41-c200-b62e-e3ca81b9a3f7 amplitude f07bc0b0-de6b-4424-8ef9-766213aaca26 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 5c68f0f0-f577-4905-8a09-e4d171d0a22d 0554a9a3c-0461-48be-8435-123eed59c228 912e250e-56d8-ee33-4525-c844d810971b amplitude 7f128981-6868-4976-ba20-248655dcac21 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb f4b9301f-bc91-455b-9474-c801093f3856 07bb007f2-26d3-463f-b7dc-7bd4d271725e d7d2c97a-0e6e-d1b8-735c-d55dc66a30e1 amplitude a9b7cec0-1256-49cf-abf0-8c45fd155379 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 74270cba-36ee-4afb-ab50-2a6cc948e68c 080e1f37f-48a7-4087-bd37-7a37b6a2c160 abb92dce-4410-8f17-a501-a4104bda0dcf amplitude 3c40ebdc-0b61-4105-9971-e1348bd49bc7 clusterless_thresholder default_clusterless mediumnwb20230802_.nwb 0f91197e-bebb-4dc6-ad41-5bf89c3eed28 08848c4a8-a2f2-4f3d-82cd-51b13b8bae3c 74e10781-1228-4075-0870-af224024ffdc amplitude 257c077b-8f3b-4abb-a631-6b8084d6a1ea clusterless_thresholder default_clusterless mediumnwb20230802_.nwb e289e03d-32ad-461a-a1cc-c88537343149 0 <p>...</p> <p>Total: 23</p> In\u00a0[3]: Copied! <pre>from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection\n\nspikesorting_merge_id = (\n    (sgs.SpikeSortingSelection &amp; sorter_keys)\n    * SpikeSortingOutput.CurationV1\n    * (\n        UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")\n        &amp; feature_key\n    )\n).fetch(\"merge_id\")\n\nwaveform_selection_keys = [\n    {\"spikesorting_merge_id\": merge_id, \"features_param_name\": \"amplitude\"}\n    for merge_id in spikesorting_merge_id\n]\n\nUnitWaveformFeaturesSelection &amp; waveform_selection_keys\n</pre> from spyglass.decoding.v1.waveform_features import UnitWaveformFeaturesSelection  spikesorting_merge_id = (     (sgs.SpikeSortingSelection &amp; sorter_keys)     * SpikeSortingOutput.CurationV1     * (         UnitWaveformFeaturesSelection.proj(merge_id=\"spikesorting_merge_id\")         &amp; feature_key     ) ).fetch(\"merge_id\")  waveform_selection_keys = [     {\"spikesorting_merge_id\": merge_id, \"features_param_name\": \"amplitude\"}     for merge_id in spikesorting_merge_id ]  UnitWaveformFeaturesSelection &amp; waveform_selection_keys Out[3]: <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitude485a4ddf-332d-35b5-3ad4-0561736c1844 amplitude4a712103-c223-864f-82e0-6c23de79cc14 amplitude4a72c253-b3ca-8c13-e615-736a7ebff35c amplitude5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitude614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitude6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitude6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitude74e10781-1228-4075-0870-af224024ffdc amplitude7e3fa66e-727e-1541-819a-b01309bb30ae amplitude86897349-ff68-ac72-02eb-739dd88936e6 amplitude8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> <p>We will create a group called <code>test_group</code> that contains all of the tetrodes that we want to decode from. We will use the <code>create_group</code> function to create this group. This function takes two arguments: the name of the group, and the keys of the tables that we want to include in the group.</p> In\u00a0[4]: Copied! <pre>from spyglass.decoding.v1.clusterless import UnitWaveformFeaturesGroup\n\nUnitWaveformFeaturesGroup().create_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test_group\",\n    keys=waveform_selection_keys,\n)\nUnitWaveformFeaturesGroup &amp; {\"waveform_features_group_name\": \"test_group\"}\n</pre> from spyglass.decoding.v1.clusterless import UnitWaveformFeaturesGroup  UnitWaveformFeaturesGroup().create_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test_group\",     keys=waveform_selection_keys, ) UnitWaveformFeaturesGroup &amp; {\"waveform_features_group_name\": \"test_group\"} Out[4]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> mediumnwb20230802_.nwb test_group <p>Total: 1</p> <p>We can see that we successfully associated \"test_group\" with the tetrodes that we want to decode from by using the <code>get_group</code> function.</p> In\u00a0[5]: Copied! <pre>UnitWaveformFeaturesGroup.UnitFeatures &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"waveform_features_group_name\": \"test_group\",\n}\n</pre> UnitWaveformFeaturesGroup.UnitFeatures &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"waveform_features_group_name\": \"test_group\", } Out[5]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>spikesorting_merge_id</p> <p>features_param_name</p> a name for this set of parameters mediumnwb20230802_.nwb test_group 0751a1e1-a406-7f87-ae6f-ce4ffc60621c amplitudemediumnwb20230802_.nwb test_group 485a4ddf-332d-35b5-3ad4-0561736c1844 amplitudemediumnwb20230802_.nwb test_group 4a712103-c223-864f-82e0-6c23de79cc14 amplitudemediumnwb20230802_.nwb test_group 4a72c253-b3ca-8c13-e615-736a7ebff35c amplitudemediumnwb20230802_.nwb test_group 5c53bd33-d57c-fbba-e0fb-55e0bcb85d03 amplitudemediumnwb20230802_.nwb test_group 614d796c-0b95-6364-aaa0-b6cb1e7bbb83 amplitudemediumnwb20230802_.nwb test_group 6acb99b8-6a0c-eb83-1141-5f603c5895e0 amplitudemediumnwb20230802_.nwb test_group 6d039a63-17ad-0b78-4b1e-f02d5f3dbbc5 amplitudemediumnwb20230802_.nwb test_group 74e10781-1228-4075-0870-af224024ffdc amplitudemediumnwb20230802_.nwb test_group 7e3fa66e-727e-1541-819a-b01309bb30ae amplitudemediumnwb20230802_.nwb test_group 86897349-ff68-ac72-02eb-739dd88936e6 amplitudemediumnwb20230802_.nwb test_group 8bbddc0f-d6ae-6260-9400-f884a6e25ae8 amplitude <p>...</p> <p>Total: 23</p> In\u00a0[6]: Copied! <pre>from spyglass.position import PositionOutput\nimport spyglass.position as sgp\n\n\nsgp.v1.TrodesPosParams.insert1(\n    {\n        \"trodes_pos_params_name\": \"default_decoding\",\n        \"params\": {\n            \"max_LED_separation\": 9.0,\n            \"max_plausible_speed\": 300.0,\n            \"position_smoothing_duration\": 0.125,\n            \"speed_smoothing_std_dev\": 0.100,\n            \"orient_smoothing_std_dev\": 0.001,\n            \"led1_is_front\": 1,\n            \"is_upsampled\": 1,\n            \"upsampling_sampling_rate\": 250,\n            \"upsampling_interpolation_method\": \"linear\",\n        },\n    },\n    skip_duplicates=True,\n)\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",\n    \"trodes_pos_params_name\": \"default_decoding\",\n}\nsgp.v1.TrodesPosSelection.insert1(\n    trodes_s_key,\n    skip_duplicates=True,\n)\nsgp.v1.TrodesPosV1.populate(trodes_s_key)\n\nPositionOutput.TrodesPosV1 &amp; trodes_s_key\n</pre> from spyglass.position import PositionOutput import spyglass.position as sgp   sgp.v1.TrodesPosParams.insert1(     {         \"trodes_pos_params_name\": \"default_decoding\",         \"params\": {             \"max_LED_separation\": 9.0,             \"max_plausible_speed\": 300.0,             \"position_smoothing_duration\": 0.125,             \"speed_smoothing_std_dev\": 0.100,             \"orient_smoothing_std_dev\": 0.001,             \"led1_is_front\": 1,             \"is_upsampled\": 1,             \"upsampling_sampling_rate\": 250,             \"upsampling_interpolation_method\": \"linear\",         },     },     skip_duplicates=True, )  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",     \"trodes_pos_params_name\": \"default_decoding\", } sgp.v1.TrodesPosSelection.insert1(     trodes_s_key,     skip_duplicates=True, ) sgp.v1.TrodesPosV1.populate(trodes_s_key)  PositionOutput.TrodesPosV1 &amp; trodes_s_key <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[10:24:13][INFO] Spyglass: Writing new NWB file mediumnwb20230802_FUSH604NQA.nwb\n</pre> <pre>Computing position for: {'nwb_file_name': 'mediumnwb20230802_.nwb', 'interval_list_name': 'pos 0 valid times', 'trodes_pos_params_name': 'default_decoding'}\n</pre> <pre>[2024-01-29 10:24:13,819][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[2024-01-29 10:24:13,821][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/ecephys.py:90: UserWarning: ElectricalSeries 'e-series': The second dimension of data does not match the length of electrodes. Your data may be transposed.\n  warnings.warn(\"%s '%s': The second dimension of data does not match the length of electrodes. \"\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/pynwb/base.py:193: UserWarning: TimeSeries 'analog': Length of data does not match length of timestamps. Your data may be transposed. Time should be on the 0th dimension\n  warn(\"%s '%s': Length of data does not match length of timestamps. Your data may be transposed. \"\n[2024-01-29 10:24:13,996][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[2024-01-29 10:24:13,998][WARNING]: Skipped checksum for file with hash: f05fc782-7d7e-7835-5aef-bc4f5837358b, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/raw/mediumnwb20230802_.nwb\n[10:24:14][INFO] Spyglass: No video frame index found. Assuming all camera frames are present.\n</pre> Out[6]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>trodes_pos_params_name</p> name for this set of parameters 6dfae23d-6034-e483-06e7-28ab4c29282f mediumnwb20230802_.nwb pos 0 valid times default_decoding <p>Total: 1</p> In\u00a0[7]: Copied! <pre>from spyglass.decoding.v1.core import PositionGroup\n\nposition_merge_ids = (\n    PositionOutput.TrodesPosV1\n    &amp; {\n        \"nwb_file_name\": nwb_copy_file_name,\n        \"interval_list_name\": \"pos 0 valid times\",\n        \"trodes_pos_params_name\": \"default_decoding\",\n    }\n).fetch(\"merge_id\")\n\nPositionGroup().create_group(\n    nwb_file_name=nwb_copy_file_name,\n    group_name=\"test_group\",\n    keys=[{\"pos_merge_id\": merge_id} for merge_id in position_merge_ids],\n)\n\nPositionGroup &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"position_group_name\": \"test_group\",\n}\n</pre> from spyglass.decoding.v1.core import PositionGroup  position_merge_ids = (     PositionOutput.TrodesPosV1     &amp; {         \"nwb_file_name\": nwb_copy_file_name,         \"interval_list_name\": \"pos 0 valid times\",         \"trodes_pos_params_name\": \"default_decoding\",     } ).fetch(\"merge_id\")  PositionGroup().create_group(     nwb_file_name=nwb_copy_file_name,     group_name=\"test_group\",     keys=[{\"pos_merge_id\": merge_id} for merge_id in position_merge_ids], )  PositionGroup &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"position_group_name\": \"test_group\", } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>position_group_name</p> <p>position_variables</p> list of position variables to decode mediumnwb20230802_.nwb test_group =BLOB= <p>Total: 1</p> In\u00a0[8]: Copied! <pre>(\n    PositionGroup\n    &amp; {\"nwb_file_name\": nwb_copy_file_name, \"position_group_name\": \"test_group\"}\n).fetch1(\"position_variables\")\n</pre> (     PositionGroup     &amp; {\"nwb_file_name\": nwb_copy_file_name, \"position_group_name\": \"test_group\"} ).fetch1(\"position_variables\") Out[8]: <pre>['position_x', 'position_y']</pre> In\u00a0[9]: Copied! <pre>PositionGroup.Position &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"position_group_name\": \"test_group\",\n}\n</pre> PositionGroup.Position &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"position_group_name\": \"test_group\", } Out[9]: <p>nwb_file_name</p> name of the NWB file <p>position_group_name</p> <p>pos_merge_id</p> mediumnwb20230802_.nwb test_group 6dfae23d-6034-e483-06e7-28ab4c29282f <p>Total: 1</p> In\u00a0[10]: Copied! <pre>from non_local_detector.models import ContFragClusterlessClassifier\n\nContFragClusterlessClassifier()\n</pre> from non_local_detector.models import ContFragClusterlessClassifier  ContFragClusterlessClassifier() Out[10]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> <p>You can change these parameters like so:</p> In\u00a0[11]: Copied! <pre>from non_local_detector.models import ContFragClusterlessClassifier\n\nContFragClusterlessClassifier(\n    clusterless_algorithm_params={\n        \"block_size\": 10000,\n        \"position_std\": 12.0,\n        \"waveform_std\": 24.0,\n    },\n)\n</pre> from non_local_detector.models import ContFragClusterlessClassifier  ContFragClusterlessClassifier(     clusterless_algorithm_params={         \"block_size\": 10000,         \"position_std\": 12.0,         \"waveform_std\": 24.0,     }, ) Out[11]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 12.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, us...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 12.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, us...\n                              environments=(Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0),),\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> <p>This is how to insert the model parameters into the database:</p> In\u00a0[12]: Copied! <pre>from spyglass.decoding.v1.core import DecodingParameters\n\n\nDecodingParameters.insert1(\n    {\n        \"decoding_param_name\": \"contfrag_clusterless\",\n        \"decoding_params\": ContFragClusterlessClassifier(),\n        \"decoding_kwargs\": dict(),\n    },\n    skip_duplicates=True,\n)\n\nDecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"}\n</pre> from spyglass.decoding.v1.core import DecodingParameters   DecodingParameters.insert1(     {         \"decoding_param_name\": \"contfrag_clusterless\",         \"decoding_params\": ContFragClusterlessClassifier(),         \"decoding_kwargs\": dict(),     },     skip_duplicates=True, )  DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"} Out[12]: <p>decoding_param_name</p> a name for this set of parameters <p>decoding_params</p> initialization parameters for model <p>decoding_kwargs</p> additional keyword arguments contfrag_clusterless =BLOB= =BLOB= <p>Total: 1</p> <p>We can retrieve these parameters and rebuild the model like so:</p> In\u00a0[13]: Copied! <pre>model_params = (\n    DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"}\n).fetch1()\n\nContFragClusterlessClassifier(**model_params[\"decoding_params\"])\n</pre> model_params = (     DecodingParameters &amp; {\"decoding_param_name\": \"contfrag_clusterless\"} ).fetch1()  ContFragClusterlessClassifier(**model_params[\"decoding_params\"]) Out[13]: <pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=[Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0)],\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.ContFragClusterlessClassifier<pre>ContFragClusterlessClassifier(clusterless_algorithm='clusterless_kde',\n                              clusterless_algorithm_params={'block_size': 10000,\n                                                            'position_std': 6.0,\n                                                            'waveform_std': 24.0},\n                              continuous_initial_conditions_types=[UniformInitialConditions(),\n                                                                   UniformInitialConditions()],\n                              continuous_transition_types=[[RandomWalk(environment_name='', movement_var=6.0, movement_mean=0.0, use...\n                              environments=[Environment(environment_name='', place_bin_size=2.0, track_graph=None, edge_order=None, edge_spacing=None, is_track_interior=None, position_range=None, infer_track_interior=True, fill_holes=False, dilate=False, bin_count_threshold=0)],\n                              infer_track_interior=True, no_spike_rate=1e-10,\n                              observation_models=[ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False),\n                                                  ObservationModel(environment_name='', encoding_group=0, is_local=False, is_no_spike=False)],\n                              sampling_frequency=500.0,\n                              state_names=['Continuous', 'Fragmented'])</pre> In\u00a0[14]: Copied! <pre>from non_local_detector.environment import Environment\n\n?Environment\n</pre> from non_local_detector.environment import Environment  ?Environment <pre>Init signature:\nEnvironment(\n    environment_name: str = '',\n    place_bin_size: Union[float, Tuple[float]] = 2.0,\n    track_graph: Optional[networkx.classes.graph.Graph] = None,\n    edge_order: Optional[tuple] = None,\n    edge_spacing: Optional[tuple] = None,\n    is_track_interior: Optional[numpy.ndarray] = None,\n    position_range: Optional[numpy.ndarray] = None,\n    infer_track_interior: bool = True,\n    fill_holes: bool = False,\n    dilate: bool = False,\n    bin_count_threshold: int = 0,\n) -&gt; None\nDocstring:     \nRepresent the spatial environment with a discrete grid.\n\nParameters\n----------\nenvironment_name : str, optional\nplace_bin_size : float, optional\n    Approximate size of the position bins.\ntrack_graph : networkx.Graph, optional\n    Graph representing the 1D spatial topology\nedge_order : tuple of 2-tuples, optional\n    The order of the edges in 1D space\nedge_spacing : None or int or tuples of len n_edges-1, optional\n    Any gapes between the edges in 1D space\nis_track_interior : np.ndarray or None, optional\n    If given, this will be used to define the valid areas of the track.\n    Must be of type boolean.\nposition_range : sequence, optional\n    A sequence of `n_position_dims`, each an optional (lower, upper)\n    tuple giving the outer bin edges for position.\n    An entry of None in the sequence results in the minimum and maximum\n    values being used for the corresponding dimension.\n    The default, None, is equivalent to passing a tuple of\n    `n_position_dims` None values.\ninfer_track_interior : bool, optional\n    If True, then use the given positions to figure out the valid track\n    areas.\nfill_holes : bool, optional\n    Fill holes when inferring the track\ndilate : bool, optional\n    Inflate the available track area with binary dilation\nbin_count_threshold : int, optional\n    Greater than this number of samples should be in the bin for it to\n    be considered on the track.\nFile:           ~/miniconda3/envs/spyglass/lib/python3.9/site-packages/non_local_detector/environment.py\nType:           type\nSubclasses:     </pre> In\u00a0[15]: Copied! <pre>from spyglass.decoding.v1.clusterless import ClusterlessDecodingSelection\n\nClusterlessDecodingSelection()\n</pre> from spyglass.decoding.v1.clusterless import ClusterlessDecodingSelection  ClusterlessDecodingSelection() Out[15]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters <p>Total: 0</p> In\u00a0[16]: Copied! <pre>from spyglass.common import IntervalList\n\nIntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name}\n</pre> from spyglass.common import IntervalList  IntervalList &amp; {\"nwb_file_name\": nwb_copy_file_name} Out[16]: Time intervals used for analysis <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>valid_times</p> numpy array with start/end times for each interval <p>pipeline</p> type of interval list (e.g. 'position', 'spikesorting_recording_v1') mediumnwb20230802_.nwb 02_r1 =BLOB= mediumnwb20230802_.nwb 04f3ecb4-a18c-4ffb-85d8-2f5f62d4d6d4 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 0e848c38-9105-4ea4-b6ba-dbdd5b46a088 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 0f91197e-bebb-4dc6-ad41-5bf89c3eed28 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 15c8a3e8-5ce9-4654-891e-6ee4109d6f1a =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 1d2b5966-415a-4c65-955a-0e422d8b5b00 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 1e3f3707-613e-4a44-93f1-c7e5484112cd =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 2402805a-04f9-4a88-9ccf-071376c8de19 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 24107d8c-ce26-4c77-8f6a-bf6955d8a3c7 =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 257c077b-8f3b-4abb-a631-6b8084d6a1ea =BLOB= spikesorting_recording_v1mediumnwb20230802_.nwb 2b93bcd0-7b05-457c-8aab-c41ef543ecf2 =BLOB= spikesorting_artifact_v1mediumnwb20230802_.nwb 2b9fbf14-74a0-4294-a805-26702340aac9 =BLOB= spikesorting_artifact_v1 <p>...</p> <p>Total: 52</p> In\u00a0[17]: Copied! <pre>decoding_interval_valid_times = [\n    [1625935714.6359036, 1625935714.6359036 + 15.0]\n]\n\nIntervalList.insert1(\n    {\n        \"nwb_file_name\": \"mediumnwb20230802_.nwb\",\n        \"interval_list_name\": \"test decoding interval\",\n        \"valid_times\": decoding_interval_valid_times,\n    },\n    skip_duplicates=True,\n)\n</pre> decoding_interval_valid_times = [     [1625935714.6359036, 1625935714.6359036 + 15.0] ]  IntervalList.insert1(     {         \"nwb_file_name\": \"mediumnwb20230802_.nwb\",         \"interval_list_name\": \"test decoding interval\",         \"valid_times\": decoding_interval_valid_times,     },     skip_duplicates=True, ) <p>Once we have figured out the keys that we need, we can insert the <code>ClusterlessDecodingSelection</code> into the database.</p> In\u00a0[18]: Copied! <pre>selection_key = {\n    \"waveform_features_group_name\": \"test_group\",\n    \"position_group_name\": \"test_group\",\n    \"decoding_param_name\": \"contfrag_clusterless\",\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"encoding_interval\": \"pos 0 valid times\",\n    \"decoding_interval\": \"test decoding interval\",\n    \"estimate_decoding_params\": False,\n}\n\nClusterlessDecodingSelection.insert1(\n    selection_key,\n    skip_duplicates=True,\n)\n\nClusterlessDecodingSelection &amp; selection_key\n</pre> selection_key = {     \"waveform_features_group_name\": \"test_group\",     \"position_group_name\": \"test_group\",     \"decoding_param_name\": \"contfrag_clusterless\",     \"nwb_file_name\": nwb_copy_file_name,     \"encoding_interval\": \"pos 0 valid times\",     \"decoding_interval\": \"test decoding interval\",     \"estimate_decoding_params\": False, }  ClusterlessDecodingSelection.insert1(     selection_key,     skip_duplicates=True, )  ClusterlessDecodingSelection &amp; selection_key Out[18]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> In\u00a0[19]: Copied! <pre>ClusterlessDecodingSelection()\n</pre> ClusterlessDecodingSelection() Out[19]: <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>To run decoding, we simply populate the <code>ClusterlessDecodingOutput</code> table. This will run the decoding and insert the results into the database. We can then retrieve the results from the database.</p> In\u00a0[20]: Copied! <pre>from spyglass.decoding.v1.clusterless import ClusterlessDecodingV1\n\nClusterlessDecodingV1.populate(selection_key)\n</pre> from spyglass.decoding.v1.clusterless import ClusterlessDecodingV1  ClusterlessDecodingV1.populate(selection_key) <pre>[10:24:17][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[2024-01-29 10:24:17,234][WARNING]: Skipped checksum for file with hash: 0cd40383-03e0-44ec-5dac-36c66063796a, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_FUSH604NQA.nwb\n[2024-01-29 10:24:17,409][WARNING]: Skipped checksum for file with hash: a7c9b1d9-d1a2-7f40-9127-206e83a87006, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_NQEPSMKPK0.nwb\n[2024-01-29 10:24:17,411][WARNING]: Skipped checksum for file with hash: ec7faa5b-3847-6649-1a93-74ebd50dcfb9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_F02UG5Z5FR.nwb\n[2024-01-29 10:24:17,413][WARNING]: Skipped checksum for file with hash: 8e964932-96ab-e1c9-2133-edce8eacab5f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OTV91MLKDT.nwb\n[2024-01-29 10:24:17,415][WARNING]: Skipped checksum for file with hash: 895bac7b-bfd6-b4f2-b2ad-460362aaafa8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_TSPNTCGNN1.nwb\n[2024-01-29 10:24:17,417][WARNING]: Skipped checksum for file with hash: 58713583-cf49-4527-7707-105f9c9ee477, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_QSK70WFDJH.nwb\n[2024-01-29 10:24:17,419][WARNING]: Skipped checksum for file with hash: a64829f8-ab12-fecc-eda9-a22b90b20d43, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DO45HKXYTB.nwb\n[2024-01-29 10:24:17,420][WARNING]: Skipped checksum for file with hash: 3a580271-9126-8e57-048e-a7bbb3f917b9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_KFIYRJ4HFO.nwb\n[2024-01-29 10:24:17,423][WARNING]: Skipped checksum for file with hash: 13cf8ad9-023c-c9b7-05c3-eaa3330304f2, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0YIM5K3H47.nwb\n[2024-01-29 10:24:17,425][WARNING]: Skipped checksum for file with hash: 7ce8a640-0a25-4866-6d5a-aa2c65f0aca5, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_CTLEGE2TWZ.nwb\n[2024-01-29 10:24:17,427][WARNING]: Skipped checksum for file with hash: aa657f4f-f409-d444-8b32-31d37abe0797, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_7EN0N1U4U1.nwb\n[2024-01-29 10:24:17,429][WARNING]: Skipped checksum for file with hash: f3b4bd22-1439-e6d2-4e15-aa3650143fdf, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_DHKWBWWAMC.nwb\n[2024-01-29 10:24:17,430][WARNING]: Skipped checksum for file with hash: 68eac0b2-e5be-e0c5-9eae-cd8dbe6676a8, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_PEN0D79Q0B.nwb\n[2024-01-29 10:24:17,432][WARNING]: Skipped checksum for file with hash: c8b95099-2cb3-df0b-5ab1-7a5e120a8e2f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_WP7SIXDJ2A.nwb\n[2024-01-29 10:24:17,434][WARNING]: Skipped checksum for file with hash: 8fae8089-f683-5f0a-4e59-c71d6ee14f38, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_B82OS6W1QA.nwb\n[2024-01-29 10:24:17,437][WARNING]: Skipped checksum for file with hash: dd9d0f51-6445-b368-32bd-b1f142bf6ed3, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_XO17FQLN6T.nwb\n[2024-01-29 10:24:17,439][WARNING]: Skipped checksum for file with hash: 4e2cf5f5-ff7c-1a2b-db85-2d1c4f036fbd, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_OCFI0GFLZ9.nwb\n[2024-01-29 10:24:17,441][WARNING]: Skipped checksum for file with hash: 8691c252-0bd1-122b-8cf3-b89c4d0fdee0, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_60M9VSZX0W.nwb\n[2024-01-29 10:24:17,443][WARNING]: Skipped checksum for file with hash: 57b89835-8edb-e91d-0798-09d22fb4fbc9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_Z5HJ68LHYW.nwb\n[2024-01-29 10:24:17,445][WARNING]: Skipped checksum for file with hash: 54401121-4426-86c9-72f7-e056bc16e99d, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_U5U5JVGY4F.nwb\n[2024-01-29 10:24:17,447][WARNING]: Skipped checksum for file with hash: 0ff21e84-2214-6911-2575-a9c92a541407, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_0D5Z0NSIP8.nwb\n[2024-01-29 10:24:17,449][WARNING]: Skipped checksum for file with hash: 0949b006-5309-93c8-fd8b-1308e8130869, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_EYV2NARUKU.nwb\n[2024-01-29 10:24:17,451][WARNING]: Skipped checksum for file with hash: b4b31e50-dfa2-0d02-514a-525782a81255, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_T4XBCIW44T.nwb\n[2024-01-29 10:24:17,453][WARNING]: Skipped checksum for file with hash: c18a9ac4-06bc-4249-2bad-439d4f618421, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_UD55CR8LZK.nwb\n</pre> <pre>Encoding models:   0%|          | 0/23 [00:00&lt;?, ?electrode/s]</pre> <pre>Non-Local Likelihood:   0%|          | 0/23 [00:00&lt;?, ?electrode/s]</pre> <pre>/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/non_local_detector/models/base.py:780: FutureWarning: the `pandas.MultiIndex` object(s) passed as 'state_bins' coordinate(s) or data variable(s) will no longer be implicitly promoted and wrapped into multiple indexed coordinates in the future (i.e., one coordinate for each multi-index level + one dimension coordinate). If you want to keep this behavior, you need to first wrap it explicitly using `mindex_coords = xarray.Coordinates.from_pandas_multiindex(mindex_obj, 'dim')` and pass it as coordinates, e.g., `xarray.Dataset(coords=mindex_coords)`, `dataset.assign_coords(mindex_coords)` or `dataarray.assign_coords(mindex_coords)`.\n  results = xr.Dataset(\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n</pre> <p>We can now see it as an entry in the <code>DecodingOutput</code> table.</p> In\u00a0[21]: Copied! <pre>from spyglass.decoding.decoding_merge import DecodingOutput\n\nDecodingOutput.ClusterlessDecodingV1 &amp; selection_key\n</pre> from spyglass.decoding.decoding_merge import DecodingOutput  DecodingOutput.ClusterlessDecodingV1 &amp; selection_key Out[21]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>waveform_features_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters b63395dd-402e-270a-a8d1-7aabaf83d452 mediumnwb20230802_.nwb test_group test_group contfrag_clusterless pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>We can load the results of the decoding:</p> In\u00a0[22]: Copied! <pre>decoding_results = (ClusterlessDecodingV1 &amp; selection_key).fetch_results()\ndecoding_results\n</pre> decoding_results = (ClusterlessDecodingV1 &amp; selection_key).fetch_results() decoding_results <pre>[2024-01-29 10:26:49,467][WARNING]: Skipped checksum for file with hash: 10c77056-5508-ace0-bd84-5a4d7497f7a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.nc\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n/Users/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/xarray/namedarray/core.py:487: UserWarning: Duplicate dimension names present: dimensions {'states'} appear more than once in dims=('states', 'states'). We do not yet support duplicate dimension names, but we do allow initial construction of the object. We recommend you rename the dims immediately to become distinct, as most xarray functionality is likely to fail silently if you do not. To rename the dimensions you will need to set the ``.dims`` attribute of each variable, ``e.g. var.dims=('x0', 'x1')``.\n  warnings.warn(\n</pre> Out[22]: <pre>&lt;xarray.Dataset&gt;\nDimensions:                      (state_ind: 26668, dim_0: 26668, time: 3750,\n                                  states: 2, intervals: 1, state_bins: 26668)\nCoordinates:\n  * state_ind                    (state_ind) int32 0 0 0 0 0 0 0 ... 1 1 1 1 1 1\n  * time                         (time) float64 1.626e+09 ... 1.626e+09\n  * states                       (states) object 'Continuous' 'Fragmented'\n    environments                 (states) object ...\n    encoding_groups              (states) int32 ...\n  * state_bins                   (state_bins) object MultiIndex\n  * state                        (state_bins) object 'Continuous' ... 'Fragme...\n  * x_position                   (state_bins) float64 29.02 29.02 ... 262.7\n  * y_position                   (state_bins) float64 0.5211 2.516 ... 224.0\nDimensions without coordinates: dim_0, intervals\nData variables:\n    initial_conditions           (dim_0) float64 ...\n    discrete_state_transitions   (states, states) float64 ...\n    acausal_posterior            (intervals, time, state_bins) float32 ...\n    acausal_state_probabilities  (intervals, time, states) float64 ...\nAttributes:\n    marginal_log_likelihoods:  -159596.89</pre>xarray.Dataset<ul><li>Dimensions:<ul><li>state_ind: 26668</li><li>dim_0: 26668</li><li>time: 3750</li><li>states: 2</li><li>intervals: 1</li><li>state_bins: 26668</li></ul></li><li>Coordinates: (9)<ul><li>state_ind(state_ind)int320 0 0 0 0 0 0 0 ... 1 1 1 1 1 1 1 1<pre>array([0, 0, 0, ..., 1, 1, 1], dtype=int32)</pre></li><li>time(time)float641.626e+09 1.626e+09 ... 1.626e+09<pre>array([1.625936e+09, 1.625936e+09, 1.625936e+09, ..., 1.625936e+09,\n       1.625936e+09, 1.625936e+09])</pre></li><li>states(states)object'Continuous' 'Fragmented'<pre>array(['Continuous', 'Fragmented'], dtype=object)</pre></li><li>environments(states)object...<pre>[2 values with dtype=object]</pre></li><li>encoding_groups(states)int32...<pre>[2 values with dtype=int32]</pre></li><li>state_bins(state_bins)objectMultiIndex<pre>array([('Continuous', 29.022380454030234, 0.5210949074118973),\n       ('Continuous', 29.022380454030234, 2.5161439814859774),\n       ('Continuous', 29.022380454030234, 4.5111930555600575), ...,\n       ('Fragmented', 262.70451087811205, 219.9764930555607),\n       ('Fragmented', 262.70451087811205, 221.97154212963477),\n       ('Fragmented', 262.70451087811205, 223.96659120370884)], dtype=object)</pre></li><li>state(state_bins)object'Continuous' ... 'Fragmented'<pre>array(['Continuous', 'Continuous', 'Continuous', ..., 'Fragmented',\n       'Fragmented', 'Fragmented'], dtype=object)</pre></li><li>x_position(state_bins)float6429.02 29.02 29.02 ... 262.7 262.7<pre>array([ 29.02238 ,  29.02238 ,  29.02238 , ..., 262.704511, 262.704511,\n       262.704511])</pre></li><li>y_position(state_bins)float640.5211 2.516 4.511 ... 222.0 224.0<pre>array([  0.521095,   2.516144,   4.511193, ..., 219.976493, 221.971542,\n       223.966591])</pre></li></ul></li><li>Data variables: (4)<ul><li>initial_conditions(dim_0)float64...<pre>[26668 values with dtype=float64]</pre></li><li>discrete_state_transitions(states, states)float64...<pre>[4 values with dtype=float64]</pre></li><li>acausal_posterior(intervals, time, state_bins)float32...<pre>[100005000 values with dtype=float32]</pre></li><li>acausal_state_probabilities(intervals, time, states)float64...<pre>[7500 values with dtype=float64]</pre></li></ul></li><li>Indexes: (4)<ul><li>state_indPandasIndex<pre>PandasIndex(Index([0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n       ...\n       1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n      dtype='int32', name='state_ind', length=26668))</pre></li><li>timePandasIndex<pre>PandasIndex(Index([1625935714.6373355, 1625935714.6413357, 1625935714.6453357,\n       1625935714.6493356, 1625935714.6533356, 1625935714.6573355,\n       1625935714.6613357, 1625935714.6653357, 1625935714.6693356,\n       1625935714.6733356,\n       ...\n       1625935729.5973077, 1625935729.6013076, 1625935729.6053078,\n       1625935729.6093078, 1625935729.6133077, 1625935729.6173077,\n       1625935729.6213076, 1625935729.6253078, 1625935729.6293077,\n       1625935729.6333077],\n      dtype='float64', name='time', length=3750))</pre></li><li>statesPandasIndex<pre>PandasIndex(Index(['Continuous', 'Fragmented'], dtype='object', name='states'))</pre></li><li>state_binsstatex_positiony_positionPandasMultiIndex<pre>PandasIndex(MultiIndex([('Continuous', 29.022380454030234, 0.5210949074118973),\n            ('Continuous', 29.022380454030234, 2.5161439814859774),\n            ('Continuous', 29.022380454030234, 4.5111930555600575),\n            ('Continuous', 29.022380454030234,  6.506242129634137),\n            ('Continuous', 29.022380454030234,  8.501291203708217),\n            ('Continuous', 29.022380454030234, 10.496340277782297),\n            ('Continuous', 29.022380454030234, 12.491389351856377),\n            ('Continuous', 29.022380454030234, 14.486438425930455),\n            ('Continuous', 29.022380454030234, 16.481487500004537),\n            ('Continuous', 29.022380454030234, 18.476536574078615),\n            ...\n            ('Fragmented', 262.70451087811205, 206.01114953704212),\n            ('Fragmented', 262.70451087811205, 208.00619861111622),\n            ('Fragmented', 262.70451087811205,  210.0012476851903),\n            ('Fragmented', 262.70451087811205, 211.99629675926437),\n            ('Fragmented', 262.70451087811205, 213.99134583333844),\n            ('Fragmented', 262.70451087811205, 215.98639490741252),\n            ('Fragmented', 262.70451087811205, 217.98144398148662),\n            ('Fragmented', 262.70451087811205,  219.9764930555607),\n            ('Fragmented', 262.70451087811205, 221.97154212963477),\n            ('Fragmented', 262.70451087811205, 223.96659120370884)],\n           name='state_bins', length=26668))</pre></li></ul></li><li>Attributes: (1)marginal_log_likelihoods :-159596.89</li></ul> <p>Finally, if we deleted the results, we can use the <code>cleanup</code> function to delete the results from the file system:</p> In\u00a0[23]: Copied! <pre>DecodingOutput().cleanup()\n</pre> DecodingOutput().cleanup() <pre>[10:26:49][INFO] Spyglass: Cleaning up decoding outputs\n[2024-01-29 10:26:49,699][WARNING]: Skipped checksum for file with hash: 10c77056-5508-ace0-bd84-5a4d7497f7a9, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.nc\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_259e8498-1eb6-4e84-a0f6-7575c4ab9b87.nc\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_8404dc49-081c-48c7-b448-34767512e8ed.nc\n[2024-01-29 10:26:49,774][WARNING]: Skipped checksum for file with hash: c4a577dc-6f11-bd71-cc5f-e131c6eaa39f, and path: /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_c17fbf1c-67bd-4d5e-b179-83f501713b9c.pkl\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_259e8498-1eb6-4e84-a0f6-7575c4ab9b87.pkl\n[10:26:49][INFO] Spyglass: Removing /Users/edeno/Documents/GitHub/spyglass/DATA/analysis/mediumnwb20230802/mediumnwb20230802_8404dc49-081c-48c7-b448-34767512e8ed.pkl\n</pre> In\u00a0[24]: Copied! <pre># from non_local_detector.visualization import (\n#     create_interactive_2D_decoding_figurl,\n# )\n\n# (\n#     position_info,\n#     position_variable_names,\n# ) = ClusterlessDecodingV1.fetch_position_info(selection_key)\n# results_time = decoding_results.acausal_posterior.isel(intervals=0).time.values\n# position_info = position_info.loc[results_time[0] : results_time[-1]]\n\n# env = ClusterlessDecodingV1.fetch_environments(selection_key)[0]\n# spike_times, _ = ClusterlessDecodingV1.fetch_spike_data(selection_key)\n\n\n# create_interactive_2D_decoding_figurl(\n#     position_time=position_info.index.to_numpy(),\n#     position=position_info[position_variable_names],\n#     env=env,\n#     results=decoding_results,\n#     posterior=decoding_results.acausal_posterior.isel(intervals=0)\n#     .unstack(\"state_bins\")\n#     .sum(\"state\"),\n#     spike_times=spike_times,\n#     head_dir=position_info[\"orientation\"],\n#     speed=position_info[\"speed\"],\n# )\n</pre> # from non_local_detector.visualization import ( #     create_interactive_2D_decoding_figurl, # )  # ( #     position_info, #     position_variable_names, # ) = ClusterlessDecodingV1.fetch_position_info(selection_key) # results_time = decoding_results.acausal_posterior.isel(intervals=0).time.values # position_info = position_info.loc[results_time[0] : results_time[-1]]  # env = ClusterlessDecodingV1.fetch_environments(selection_key)[0] # spike_times, _ = ClusterlessDecodingV1.fetch_spike_data(selection_key)   # create_interactive_2D_decoding_figurl( #     position_time=position_info.index.to_numpy(), #     position=position_info[position_variable_names], #     env=env, #     results=decoding_results, #     posterior=decoding_results.acausal_posterior.isel(intervals=0) #     .unstack(\"state_bins\") #     .sum(\"state\"), #     spike_times=spike_times, #     head_dir=position_info[\"orientation\"], #     speed=position_info[\"speed\"], # ) In\u00a0[25]: Copied! <pre>import jax\n\njax.devices()\n</pre> import jax  jax.devices() Out[25]: <pre>[CpuDevice(id=0)]</pre> In\u00a0[26]: Copied! <pre># device_id = 2\n# device = jax.devices()[device_id]\n# jax.config.update(\"jax_default_device\", device)\n# device\n</pre> # device_id = 2 # device = jax.devices()[device_id] # jax.config.update(\"jax_default_device\", device) # device"}, {"location": "notebooks/41_Decoding_Clusterless/#clusterless-decoding", "title": "Clusterless Decoding\u00b6", "text": ""}, {"location": "notebooks/41_Decoding_Clusterless/#overview", "title": "Overview\u00b6", "text": "<p>Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the <code>gitignore</code> prefix <code>temp</code> to avoid future conflicts.</p> <p>This is one notebook in a multi-part series on Spyglass.</p> <ul> <li>To set up your Spyglass environment and database, see the Setup notebook</li> <li>This tutorial assumes you've already  extracted waveforms, as well as loaded  position data. If 1D decoding, this data should also be linearized.</li> </ul> <p>Clusterless decoding can be performed on either 1D or 2D data. We will start with 2D data.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#elements-of-clusterless-decoding", "title": "Elements of Clusterless Decoding\u00b6", "text": "<ul> <li>Position Data: This is the data that we want to decode. It can be 1D or 2D.</li> <li>Spike Waveform Features: These are the features that we will use to decode the position data.</li> <li>Decoding Model Parameters: This is how we define the model that we will use to decode the position data.</li> </ul>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-data", "title": "Grouping Data\u00b6", "text": "<p>An important concept will be groups. Groups are tables that allow use to specify collections of data. We will use groups in two situations here:</p> <ol> <li>Because we want to decode from more than one tetrode (or probe), so we will create a group that contains all of the tetrodes that we want to decode from.</li> <li>Similarly, we will create a group for the position data that we want to decode, so that we can decode from position data from multiple sessions.</li> </ol>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-waveform-features", "title": "Grouping Waveform Features\u00b6", "text": "<p>Let's start with grouping the Waveform Features. We will first inspect the waveform features that we have extracted to figure out the primary keys of the data that we want to decode from. We need to use the tables <code>SpikeSortingSelection</code> and <code>SpikeSortingOutput</code> to figure out the <code>merge_id</code> associated with <code>nwb_file_name</code> to get the waveform features associated with the NWB file of interest.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#grouping-position-data", "title": "Grouping Position Data\u00b6", "text": "<p>We will now create a group called <code>02_r1</code> that contains all of the position data that we want to decode from. As before, we will use the <code>create_group</code> function to create this group. This function takes two arguments: the name of the group, and the keys of the tables that we want to include in the group.</p> <p>We use the the <code>PositionOutput</code> table to figure out the <code>merge_id</code> associated with <code>nwb_file_name</code> to get the position data associated with the NWB file of interest. In this case, we only have one position to insert, but we could insert multiple positions if we wanted to decode from multiple sessions.</p> <p>Note that the position data sampling frequency is what determines the time step of the decoding. In this case, the position data sampling frequency is 30 Hz, so the time step of the decoding will be 1/30 seconds. In practice, you will want to use a smaller time step such as 500 Hz. This will allow you to decode at a finer time scale. To do this, you will want to interpolate the position data to a higher sampling frequency as shown in the position trodes notebook.</p> <p>You will also want to specify the name of the position variables if they are different from the default names. The default names are <code>position_x</code> and <code>position_y</code>.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding-model-parameters", "title": "Decoding Model Parameters\u00b6", "text": "<p>We will use the <code>non_local_detector</code> package to decode the data. This package is highly flexible and allows several different types of models to be used. In this case, we will use the <code>ContFragClusterlessClassifier</code> to decode the data. This has two discrete states: Continuous and Fragmented, which correspond to different types of movement models. To read more about this model, see:</p> <p>Denovellis, E.L., Gillespie, A.K., Coulter, M.E., Sosa, M., Chung, J.E., Eden, U.T., and Frank, L.M. (2021). Hippocampal replay of experience at real-world speeds. eLife 10, e64505. 10.7554/eLife.64505.</p> <p>Let's first look at the model and the default parameters:</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#1d-decoding", "title": "1D Decoding\u00b6", "text": "<p>If you want to do 1D decoding, you will need to specify the <code>track_graph</code>, <code>edge_order</code>, and <code>edge_spacing</code> in the <code>environments</code> parameter. You can read more about these parameters in the linearization notebook. You can retrieve these parameters from the <code>TrackGraph</code> table if you have stored them there. These will then go into the <code>environments</code> parameter of the <code>ContFragClusterlessClassifier</code> model.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding", "title": "Decoding\u00b6", "text": "<p>Now that we have grouped the data and defined the model parameters, we have finally set up the elements in tables that we need to decode the data. We now need to use the <code>ClusterlessDecodingSelection</code> to fully specify all the parameters and data that we want.</p> <p>This has:</p> <ul> <li><code>waveform_features_group_name</code>: the name of the group that contains the waveform features that we want to decode from</li> <li><code>position_group_name</code>: the name of the group that contains the position data that we want to decode from</li> <li><code>decoding_param_name</code>: the name of the decoding parameters that we want to use</li> <li><code>nwb_file_name</code>: the name of the NWB file that we want to decode from</li> <li><code>encoding_interval</code>: the interval of time that we want to train the initial model on</li> <li><code>decoding_interval</code>: the interval of time that we want to decode from</li> <li><code>estimate_decoding_params</code>: whether or not we want to estimate the decoding parameters</li> </ul> <p>The first three parameters should be familiar to you.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#decoding-and-encoding-intervals", "title": "Decoding and Encoding Intervals\u00b6", "text": "<p>The <code>encoding_interval</code> is the interval of time that we want to train the initial model on. The <code>decoding_interval</code> is the interval of time that we want to decode from. These two intervals can be the same, but they do not have to be. For example, we may want to train the model on a long interval of time, but only decode from a short interval of time. This is useful if we want to decode from a short interval of time that is not representative of the entire session. In this case, we will train the model on a longer interval of time that is representative of the entire session.</p> <p>These keys come from the <code>IntervalList</code> table. We can see that the <code>IntervalList</code> table contains the <code>nwb_file_name</code> and <code>interval_name</code> that we need to specify the <code>encoding_interval</code> and <code>decoding_interval</code>. We will specify a short decoding interval called <code>test decoding interval</code> and use that to decode from.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#estimating-decoding-parameters", "title": "Estimating Decoding Parameters\u00b6", "text": "<p>The last parameter is <code>estimate_decoding_params</code>. This is a boolean that specifies whether or not we want to estimate the decoding parameters. If this is <code>True</code>, then we will estimate the initial conditions and discrete transition matrix from the data.</p> <p>NOTE: If estimating parameters, then we need to treat times outside decoding interval as missing. this means that times outside the decoding interval will not use the spiking data and only the state transition matrix and previous time step will be used. This may or may not be desired depending on the length of this missing interval.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#visualization-of-decoding-output", "title": "Visualization of decoding output.\u00b6", "text": "<p>The output of decoding can be challenging to visualize with static graphs, especially if the decoding is performed on 2D data.</p> <p>We can interactively visualize the output of decoding using the figurl package. This package allows to create a visualization of the decoding output that can be viewed in a web browser. This is useful for exploring the decoding output over time and sharing the results with others.</p> <p>NOTE: You will need a kachery cloud instance to use this feature. If you are a member of the Frank lab, you should have access to the Frank lab kachery cloud instance. If you are not a member of the Frank lab, you can create your own kachery cloud instance by following the instructions here.</p> <p>For each user, you will need to run <code>kachery-cloud-init</code> in the terminal and follow the instructions to associate your computer with your GitHub user on the kachery-cloud network.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#gpus", "title": "GPUs\u00b6", "text": "<p>We can use GPUs for decoding which will result in a significant speedup. This is achieved using the jax package.</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#ensuring-jax-can-find-a-gpu", "title": "Ensuring jax can find a GPU\u00b6", "text": "<p>Assuming you've set up a GPU, we can use <code>jax.devices()</code> to make sure the decoding code can see the GPU. If a GPU is available, it will be listed.</p> <p>In the following instance, we do not have a GPU:</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#selecting-a-gpu", "title": "Selecting a GPU\u00b6", "text": "<p>If you do have multiple GPUs, you can use the <code>jax</code> package to set the device (GPU) that you want to use. For example, if you want to use the second GPU, you can use the following code (uncomment first):</p>"}, {"location": "notebooks/41_Decoding_Clusterless/#monitoring-gpu-usage", "title": "Monitoring GPU Usage\u00b6", "text": "<p>You can see which GPUs are occupied (if you have multiple GPUs) by running the command <code>nvidia-smi</code> in a terminal (or <code>!nvidia-smi</code> in a notebook). Pick a GPU with low memory usage.</p> <p>We can monitor GPU use with the terminal command <code>watch -n 0.1 nvidia-smi</code>, will update <code>nvidia-smi</code> every 100 ms. This won't work in a notebook, as it won't display the updates.</p> <p>Other ways to monitor GPU usage are:</p> <ul> <li>A  jupyter widget by nvidia to monitor GPU usage in the notebook</li> <li>A terminal program like nvidia-smi with more information about  which GPUs are being utilized and by whom.</li> </ul>"}, {"location": "notebooks/42_Decoding_SortedSpikes/", "title": "Decoding Sorted Spikes", "text": "In\u00a0[9]: Copied! <pre>from pathlib import Path\nimport datajoint as dj\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n</pre> from pathlib import Path import datajoint as dj  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info In\u00a0[1]: Copied! <pre>from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams\n\nUnitSelectionParams().insert_default()\n\n# look at the filter set we'll use here\nunit_filter_params_name = \"default_exclusion\"\nprint(\n    (\n        UnitSelectionParams()\n        &amp; {\"unit_filter_params_name\": unit_filter_params_name}\n    ).fetch1()\n)\n# look at full table\nUnitSelectionParams()\n</pre> from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams  UnitSelectionParams().insert_default()  # look at the filter set we'll use here unit_filter_params_name = \"default_exclusion\" print(     (         UnitSelectionParams()         &amp; {\"unit_filter_params_name\": unit_filter_params_name}     ).fetch1() ) # look at full table UnitSelectionParams() <pre>[2024-02-02 12:06:04,725][INFO]: Connecting sambray@lmf-db.cin.ucsf.edu:3306\n[2024-02-02 12:06:04,762][INFO]: Connected sambray@lmf-db.cin.ucsf.edu:3306\n[12:06:05][WARNING] Spyglass: Please update position_tools to &gt;= 0.1.0\n</pre> <pre>{'unit_filter_params_name': 'default_exclusion', 'include_labels': [], 'exclude_labels': ['noise', 'mua']}\n</pre> Out[1]: <p>unit_filter_params_name</p> <p>include_labels</p> <p>exclude_labels</p> all_units =BLOB= =BLOB=default_exclusion =BLOB= =BLOB=exclude_noise =BLOB= =BLOB= <p>Total: 3</p> <p>Now we can make our sorted spikes group with this unit selection parameter</p> In\u00a0[2]: Copied! <pre>from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput\nimport spyglass.spikesorting.v1 as sgs\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\nsorter_keys = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorter\": \"mountainsort4\",\n    \"curation_id\": 1,\n}\n# check the set of sorting we'll use\n(sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1\n</pre> from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput import spyglass.spikesorting.v1 as sgs  nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  sorter_keys = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorter\": \"mountainsort4\",     \"curation_id\": 1, } # check the set of sorting we'll use (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 Out[2]: <p>sorting_id</p> <p>merge_id</p> <p>recording_id</p> <p>sorter</p> <p>sorter_param_name</p> <p>nwb_file_name</p> name of the NWB file <p>interval_list_name</p> descriptive name of this interval list <p>curation_id</p> 642242ff-5f0e-45a2-bcc1-ca681f37b4a3 75286bf3-f876-4550-f235-321f2a7badef 01c5b8e9-933d-4f1e-9a5d-c494276edb3a mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 0a6611b3-c593-4900-a715-66bb1396940e 1a4b5a94d-ba41-4634-92d0-1d31c9daa913 143dff79-3779-c0d2-46fe-7c5040404219 a8a1d29d-ffdf-4370-8b3d-909fef57f9d4 mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 3d782852-a56b-4a9d-89ca-be9e1a15c957 1874775be-df0f-4850-8f88-59ba1bbead89 a900c1c8-909d-e583-c377-e98c4f0deebf 747f4eea-6df3-422b-941e-b5aaad7ec607 mountainsort4 franklab_tetrode_hippocampus_30KHz mediumnwb20230802_.nwb 9cf9e3cd-7115-4b59-a718-3633725d4738 1 <p>Total: 3</p> In\u00a0[6]: Copied! <pre>from spyglass.decoding.v1.sorted_spikes import SortedSpikesGroup\n\nSortedSpikesGroup()\n</pre> from spyglass.decoding.v1.sorted_spikes import SortedSpikesGroup  SortedSpikesGroup() Out[6]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb all_units test_group <p>Total: 1</p> In\u00a0[7]: Copied! <pre># get the merge_ids for the selected sorting\nspikesorting_merge_ids = (\n    (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1\n).fetch(\"merge_id\")\n\n# create a new sorted spikes group\nunit_filter_params_name = \"default_exclusion\"\nSortedSpikesGroup().create_group(\n    group_name=\"test_group\",\n    nwb_file_name=nwb_copy_file_name,\n    keys=[\n        {\"spikesorting_merge_id\": merge_id}\n        for merge_id in spikesorting_merge_ids\n    ],\n    unit_filter_params_name=unit_filter_params_name,\n)\n# check the new group\nSortedSpikesGroup &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n}\n</pre> # get the merge_ids for the selected sorting spikesorting_merge_ids = (     (sgs.SpikeSortingSelection &amp; sorter_keys) * SpikeSortingOutput.CurationV1 ).fetch(\"merge_id\")  # create a new sorted spikes group unit_filter_params_name = \"default_exclusion\" SortedSpikesGroup().create_group(     group_name=\"test_group\",     nwb_file_name=nwb_copy_file_name,     keys=[         {\"spikesorting_merge_id\": merge_id}         for merge_id in spikesorting_merge_ids     ],     unit_filter_params_name=unit_filter_params_name, ) # check the new group SortedSpikesGroup &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\", } Out[7]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb all_units test_groupmediumnwb20230802_.nwb default_exclusion test_group <p>Total: 2</p> In\u00a0[8]: Copied! <pre># look at the sorting within the group we just made\nSortedSpikesGroup.Units &amp; {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": unit_filter_params_name,\n}\n</pre> # look at the sorting within the group we just made SortedSpikesGroup.Units &amp; {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": unit_filter_params_name, } Out[8]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>spikesorting_merge_id</p> mediumnwb20230802_.nwb default_exclusion test_group 143dff79-3779-c0d2-46fe-7c5040404219mediumnwb20230802_.nwb default_exclusion test_group 75286bf3-f876-4550-f235-321f2a7badefmediumnwb20230802_.nwb default_exclusion test_group a900c1c8-909d-e583-c377-e98c4f0deebf <p>Total: 3</p> In\u00a0[9]: Copied! <pre>from spyglass.decoding.v1.core import DecodingParameters\nfrom non_local_detector.models import ContFragSortedSpikesClassifier\n\n\nDecodingParameters.insert1(\n    {\n        \"decoding_param_name\": \"contfrag_sorted\",\n        \"decoding_params\": ContFragSortedSpikesClassifier(),\n        \"decoding_kwargs\": dict(),\n    },\n    skip_duplicates=True,\n)\n\nDecodingParameters()\n</pre> from spyglass.decoding.v1.core import DecodingParameters from non_local_detector.models import ContFragSortedSpikesClassifier   DecodingParameters.insert1(     {         \"decoding_param_name\": \"contfrag_sorted\",         \"decoding_params\": ContFragSortedSpikesClassifier(),         \"decoding_kwargs\": dict(),     },     skip_duplicates=True, )  DecodingParameters() Out[9]: <p>decoding_param_name</p> a name for this set of parameters <p>decoding_params</p> initialization parameters for model <p>decoding_kwargs</p> additional keyword arguments contfrag_clusterless =BLOB= =BLOB=contfrag_clusterless_0.5.13 =BLOB= =BLOB=contfrag_clusterless_6track =BLOB= =BLOB=contfrag_sorted =BLOB= =BLOB=contfrag_sorted_0.5.13 =BLOB= =BLOB=j1620210710_contfrag_clusterless_1D =BLOB= =BLOB=j1620210710_test_contfrag_clusterless =BLOB= =BLOB=MS2220180629_contfrag_sorted =BLOB= =BLOB=ms_lineartrack_2023_contfrag_sorted =BLOB= =BLOB=ms_lineartrack_contfrag_clusterless =BLOB= =BLOB=ms_lineartrack_contfrag_sorted =BLOB= =BLOB=ms_wtrack_2023_contfrag_sorted =BLOB= =BLOB= <p>...</p> <p>Total: 15</p> In\u00a0[2]: Copied! <pre>selection_key = {\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": \"default_exclusion\",\n    \"position_group_name\": \"test_group\",\n    \"decoding_param_name\": \"contfrag_sorted\",\n    \"nwb_file_name\": \"mediumnwb20230802_.nwb\",\n    \"encoding_interval\": \"pos 0 valid times\",\n    \"decoding_interval\": \"test decoding interval\",\n    \"estimate_decoding_params\": False,\n}\n\nfrom spyglass.decoding import SortedSpikesDecodingSelection\n\nSortedSpikesDecodingSelection.insert1(\n    selection_key,\n    skip_duplicates=True,\n)\n</pre> selection_key = {     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": \"default_exclusion\",     \"position_group_name\": \"test_group\",     \"decoding_param_name\": \"contfrag_sorted\",     \"nwb_file_name\": \"mediumnwb20230802_.nwb\",     \"encoding_interval\": \"pos 0 valid times\",     \"decoding_interval\": \"test decoding interval\",     \"estimate_decoding_params\": False, }  from spyglass.decoding import SortedSpikesDecodingSelection  SortedSpikesDecodingSelection.insert1(     selection_key,     skip_duplicates=True, ) In\u00a0[14]: Copied! <pre>from spyglass.decoding.v1.sorted_spikes import SortedSpikesDecodingV1\n\nSortedSpikesDecodingV1.populate(selection_key)\n</pre> from spyglass.decoding.v1.sorted_spikes import SortedSpikesDecodingV1  SortedSpikesDecodingV1.populate(selection_key) <pre>[12:19:30][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\nNo GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n</pre> <pre>Encoding models:   0%|          | 0/54 [00:00&lt;?, ?cell/s]</pre> <pre>Non-Local Likelihood:   0%|          | 0/54 [00:00&lt;?, ?cell/s]</pre> <p>We verify that the results have been inserted into the <code>DecodingOutput</code> merge table.</p> In\u00a0[3]: Copied! <pre>from spyglass.decoding.decoding_merge import DecodingOutput\n\nDecodingOutput.SortedSpikesDecodingV1 &amp; selection_key\n</pre> from spyglass.decoding.decoding_merge import DecodingOutput  DecodingOutput.SortedSpikesDecodingV1 &amp; selection_key Out[3]: <p>merge_id</p> <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>position_group_name</p> <p>decoding_param_name</p> a name for this set of parameters <p>encoding_interval</p> descriptive name of this interval list <p>decoding_interval</p> descriptive name of this interval list <p>estimate_decoding_params</p> whether to estimate the decoding parameters 42e9e7f9-a6f2-9242-63ce-94228bc72743 mediumnwb20230802_.nwb default_exclusion test_group test_group contfrag_sorted pos 0 valid times test decoding interval 0 <p>Total: 1</p> <p>We can load the results as before:</p> In\u00a0[6]: Copied! <pre>results = (SortedSpikesDecodingV1 &amp; selection_key).fetch_results()\nresults\n</pre> results = (SortedSpikesDecodingV1 &amp; selection_key).fetch_results() results Out[6]: <pre>&lt;xarray.Dataset&gt;\nDimensions:                      (state_ind: 25752, dim_0: 25752, time: 5001,\n                                  states: 2, intervals: 1, state_bins: 25752)\nCoordinates:\n  * state_ind                    (state_ind) int32 0 0 0 0 0 0 0 ... 1 1 1 1 1 1\n  * time                         (time) float64 1.626e+09 ... 1.626e+09\n  * states                       (states) object 'Continuous' 'Fragmented'\n    environments                 (states) object ...\n    encoding_groups              (states) int32 ...\n  * state_bins                   (state_bins) object MultiIndex\n  * state                        (state_bins) object 'Continuous' ... 'Fragme...\n  * x_position                   (state_bins) float64 29.02 29.02 ... 258.8\n  * y_position                   (state_bins) float64 5.828 7.811 ... 224.0\nDimensions without coordinates: dim_0, intervals\nData variables:\n    initial_conditions           (dim_0) float64 ...\n    discrete_state_transitions   (states, states) float64 ...\n    acausal_posterior            (intervals, time, state_bins) float32 ...\n    acausal_state_probabilities  (intervals, time, states) float64 ...\nAttributes:\n    marginal_log_likelihoods:  -39514.59</pre>xarray.Dataset<ul><li>Dimensions:<ul><li>state_ind: 25752</li><li>dim_0: 25752</li><li>time: 5001</li><li>states: 2</li><li>intervals: 1</li><li>state_bins: 25752</li></ul></li><li>Coordinates: (9)<ul><li>state_ind(state_ind)int320 0 0 0 0 0 0 0 ... 1 1 1 1 1 1 1 1<pre>array([0, 0, 0, ..., 1, 1, 1], dtype=int32)</pre></li><li>time(time)float641.626e+09 1.626e+09 ... 1.626e+09<pre>array([1.625936e+09, 1.625936e+09, 1.625936e+09, ..., 1.625936e+09,\n       1.625936e+09, 1.625936e+09])</pre></li><li>states(states)object'Continuous' 'Fragmented'<pre>array(['Continuous', 'Fragmented'], dtype=object)</pre></li><li>environments(states)object...<pre>[2 values with dtype=object]</pre></li><li>encoding_groups(states)int32...<pre>[2 values with dtype=int32]</pre></li><li>state_bins(state_bins)objectMultiIndex<pre>array([('Continuous', 29.019020985146526, 5.828088389548259),\n       ('Continuous', 29.019020985146526, 7.811111022216012),\n       ('Continuous', 29.019020985146526, 9.794133654883765), ...,\n       ('Fragmented', 258.79439532749024, 219.99453271766555),\n       ('Fragmented', 258.79439532749024, 221.9775553503333),\n       ('Fragmented', 258.79439532749024, 223.96057798300103)], dtype=object)</pre></li><li>state(state_bins)object'Continuous' ... 'Fragmented'<pre>array(['Continuous', 'Continuous', 'Continuous', ..., 'Fragmented',\n       'Fragmented', 'Fragmented'], dtype=object)</pre></li><li>x_position(state_bins)float6429.02 29.02 29.02 ... 258.8 258.8<pre>array([ 29.019021,  29.019021,  29.019021, ..., 258.794395, 258.794395,\n       258.794395])</pre></li><li>y_position(state_bins)float645.828 7.811 9.794 ... 222.0 224.0<pre>array([  5.828088,   7.811111,   9.794134, ..., 219.994533, 221.977555,\n       223.960578])</pre></li></ul></li><li>Data variables: (4)<ul><li>initial_conditions(dim_0)float64...<pre>[25752 values with dtype=float64]</pre></li><li>discrete_state_transitions(states, states)float64...<pre>[4 values with dtype=float64]</pre></li><li>acausal_posterior(intervals, time, state_bins)float32...<pre>[128785752 values with dtype=float32]</pre></li><li>acausal_state_probabilities(intervals, time, states)float64...<pre>[10002 values with dtype=float64]</pre></li></ul></li><li>Indexes: (4)<ul><li>state_indPandasIndex<pre>PandasIndex(Index([0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n       ...\n       1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n      dtype='int32', name='state_ind', length=25752))</pre></li><li>timePandasIndex<pre>PandasIndex(Index([1625935703.3693566, 1625935703.3713567, 1625935703.3733566,\n       1625935703.3753567, 1625935703.3773565, 1625935703.3793566,\n       1625935703.3813567, 1625935703.3833566, 1625935703.3853567,\n       1625935703.3873565,\n       ...\n       1625935713.3513515, 1625935713.3533516, 1625935713.3553517,\n       1625935713.3573515, 1625935713.3593516, 1625935713.3613515,\n       1625935713.3633516, 1625935713.3653517, 1625935713.3673515,\n       1625935713.3693516],\n      dtype='float64', name='time', length=5001))</pre></li><li>statesPandasIndex<pre>PandasIndex(Index(['Continuous', 'Fragmented'], dtype='object', name='states'))</pre></li><li>state_binsstatex_positiony_positionPandasMultiIndex<pre>PandasIndex(MultiIndex([('Continuous', 29.019020985146526,  5.828088389548259),\n            ('Continuous', 29.019020985146526,  7.811111022216012),\n            ('Continuous', 29.019020985146526,  9.794133654883765),\n            ('Continuous', 29.019020985146526, 11.777156287551517),\n            ('Continuous', 29.019020985146526,  13.76017892021927),\n            ('Continuous', 29.019020985146526, 15.743201552887022),\n            ('Continuous', 29.019020985146526, 17.726224185554777),\n            ('Continuous', 29.019020985146526, 19.709246818222525),\n            ('Continuous', 29.019020985146526,  21.69226945089028),\n            ('Continuous', 29.019020985146526, 23.675292083558034),\n            ...\n            ('Fragmented', 258.79439532749024, 206.11337428899128),\n            ('Fragmented', 258.79439532749024, 208.09639692165902),\n            ('Fragmented', 258.79439532749024, 210.07941955432676),\n            ('Fragmented', 258.79439532749024, 212.06244218699453),\n            ('Fragmented', 258.79439532749024,  214.0454648196623),\n            ('Fragmented', 258.79439532749024, 216.02848745233004),\n            ('Fragmented', 258.79439532749024, 218.01151008499778),\n            ('Fragmented', 258.79439532749024, 219.99453271766555),\n            ('Fragmented', 258.79439532749024,  221.9775553503333),\n            ('Fragmented', 258.79439532749024, 223.96057798300103)],\n           name='state_bins', length=25752))</pre></li></ul></li><li>Attributes: (1)marginal_log_likelihoods :-39514.59</li></ul> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#sorted-spikes-decoding", "title": "Sorted Spikes Decoding\u00b6", "text": "<p>The mechanics of decoding with sorted spikes are largely similar to those of decoding with unsorted spikes. You should familiarize yourself with the clusterless decoding tutorial before proceeding with this one.</p> <p>The elements we will need to decode with sorted spikes are:</p> <ul> <li><code>PositionGroup</code></li> <li><code>SortedSpikesGroup</code></li> <li><code>DecodingParameters</code></li> <li><code>encoding_interval</code></li> <li><code>decoding_interval</code></li> </ul> <p>This time, instead of extracting waveform features, we can proceed directly from the SpikeSortingOutput table to specify which units we want to decode. The rest of the decoding process is the same as before.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#sortedspikesgroup", "title": "SortedSpikesGroup\u00b6", "text": "<p><code>SortedSpikesGroup</code> is a child table of <code>SpikeSortingOutput</code> in the spikesorting pipeline. It allows us to group the spikesorting results from multiple  sources (e.g. multiple terode groups or intervals) into a single entry. Here we will group together the spiking of multiple tetrode groups to use for decoding.</p> <p>This table allows us filter units by their annotation labels from curation (e.g only include units labeled \"good\", exclude units labeled \"noise\") by defining parameters from <code>UnitSelectionParams</code>. When accessing data through <code>SortedSpikesGroup</code> the table will include only units with at least one label in <code>include_labels</code> and no labels in <code>exclude_labels</code>. We can look at those here:</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#model-parameters", "title": "Model parameters\u00b6", "text": "<p>As before we can specify the model parameters. The only difference is that we will use the <code>ContFragSortedSpikesClassifier</code> instead of the <code>ContFragClusterlessClassifier</code>.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#1d-decoding", "title": "1D Decoding\u00b6", "text": "<p>As in the clusterless notebook, we can decode 1D position if we specify the <code>track_graph</code>, <code>edge_order</code>, and <code>edge_spacing</code> parameters in the <code>Environment</code> class constructor. See the clusterless decoding tutorial for more details.</p>"}, {"location": "notebooks/42_Decoding_SortedSpikes/#decoding", "title": "Decoding\u00b6", "text": "<p>Now we can decode the position using the sorted spikes using the <code>SortedSpikesDecodingSelection</code> table. Here we assume that <code>PositionGroup</code> has been specified as in the clusterless decoding tutorial.</p>"}, {"location": "notebooks/50_MUA_Detection/", "title": "MUA Detection", "text": "In\u00a0[1]: Copied! <pre>import datajoint as dj\nfrom pathlib import Path\n\ndj.config.load(\n    Path(\"../dj_local_conf.json\").absolute()\n)  # load config for database connection info\n\nfrom spyglass.mua.v1.mua import MuaEventsV1, MuaEventsParameters\n</pre> import datajoint as dj from pathlib import Path  dj.config.load(     Path(\"../dj_local_conf.json\").absolute() )  # load config for database connection info  from spyglass.mua.v1.mua import MuaEventsV1, MuaEventsParameters <pre>[2024-02-07 14:44:16,324][INFO]: Connecting edeno@lmf-db.cin.ucsf.edu:3306\n[2024-02-07 14:44:16,357][INFO]: Connected edeno@lmf-db.cin.ucsf.edu:3306\n</pre> In\u00a0[2]: Copied! <pre>MuaEventsParameters()\n</pre> MuaEventsParameters() Out[2]: <p>mua_param_name</p> a name for this set of parameters <p>mua_param_dict</p> dictionary of parameters default =BLOB= <p>Total: 1</p> In\u00a0[3]: Copied! <pre>MuaEventsV1()\n</pre> MuaEventsV1() Out[3]: <p>mua_param_name</p> a name for this set of parameters <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>pos_merge_id</p> <p>detection_interval</p> descriptive name of this interval list <p>analysis_file_name</p> name of the file <p>mua_times_object_id</p> default mediumnwb20230802_.nwb default_exclusion test_group 4eb59a18-045a-5768-d12e-b6473415ae1c pos 0 valid times mediumnwb20230802_235KSEV39O.nwb a93532eb-2947-4552-8877-19ea0d2dcc4f <p>Total: 1</p> In\u00a0[4]: Copied! <pre>from spyglass.position import PositionOutput\n\nnwb_copy_file_name = \"mediumnwb20230802_.nwb\"\n\ntrodes_s_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"interval_list_name\": \"pos 0 valid times\",\n    \"trodes_pos_params_name\": \"single_led_upsampled\",\n}\n\npos_merge_id = (PositionOutput.TrodesPosV1 &amp; trodes_s_key).fetch1(\"merge_id\")\npos_merge_id\n</pre> from spyglass.position import PositionOutput  nwb_copy_file_name = \"mediumnwb20230802_.nwb\"  trodes_s_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"interval_list_name\": \"pos 0 valid times\",     \"trodes_pos_params_name\": \"single_led_upsampled\", }  pos_merge_id = (PositionOutput.TrodesPosV1 &amp; trodes_s_key).fetch1(\"merge_id\") pos_merge_id Out[4]: <pre>UUID('4eb59a18-045a-5768-d12e-b6473415ae1c')</pre> In\u00a0[5]: Copied! <pre>from spyglass.spikesorting.analysis.v1.group import (\n    SortedSpikesGroup,\n)\n\nsorted_spikes_group_key = {\n    \"nwb_file_name\": nwb_copy_file_name,\n    \"sorted_spikes_group_name\": \"test_group\",\n    \"unit_filter_params_name\": \"default_exclusion\",\n}\n\nSortedSpikesGroup &amp; sorted_spikes_group_key\n</pre> from spyglass.spikesorting.analysis.v1.group import (     SortedSpikesGroup, )  sorted_spikes_group_key = {     \"nwb_file_name\": nwb_copy_file_name,     \"sorted_spikes_group_name\": \"test_group\",     \"unit_filter_params_name\": \"default_exclusion\", }  SortedSpikesGroup &amp; sorted_spikes_group_key Out[5]: <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> mediumnwb20230802_.nwb default_exclusion test_group <p>Total: 1</p> In\u00a0[6]: Copied! <pre>mua_key = {\n    \"mua_param_name\": \"default\",\n    **sorted_spikes_group_key,\n    \"pos_merge_id\": pos_merge_id,\n    \"detection_interval\": \"pos 0 valid times\",\n}\n\nMuaEventsV1().populate(mua_key)\nMuaEventsV1 &amp; mua_key\n</pre> mua_key = {     \"mua_param_name\": \"default\",     **sorted_spikes_group_key,     \"pos_merge_id\": pos_merge_id,     \"detection_interval\": \"pos 0 valid times\", }  MuaEventsV1().populate(mua_key) MuaEventsV1 &amp; mua_key Out[6]: <p>mua_param_name</p> a name for this set of parameters <p>nwb_file_name</p> name of the NWB file <p>unit_filter_params_name</p> <p>sorted_spikes_group_name</p> <p>pos_merge_id</p> <p>detection_interval</p> descriptive name of this interval list <p>analysis_file_name</p> name of the file <p>mua_times_object_id</p> default mediumnwb20230802_.nwb default_exclusion test_group 4eb59a18-045a-5768-d12e-b6473415ae1c pos 0 valid times mediumnwb20230802_235KSEV39O.nwb a93532eb-2947-4552-8877-19ea0d2dcc4f <p>Total: 1</p> In\u00a0[7]: Copied! <pre>mua_times = (MuaEventsV1 &amp; mua_key).fetch1_dataframe()\nmua_times\n</pre> mua_times = (MuaEventsV1 &amp; mua_key).fetch1_dataframe() mua_times Out[7]: start_time end_time duration mean_zscore median_zscore max_zscore min_zscore speed_at_start speed_at_end max_speed min_speed median_speed mean_speed id 0 1.625936e+09 1.625936e+09 0.238 1.204139 1.220925 2.258331 0.028349 2.760825 1.517296 3.576624 1.427160 2.851154 2.616020 1 1.625936e+09 1.625936e+09 0.120 1.364368 1.349665 2.412096 0.015503 1.803533 0.497099 1.803533 0.464203 0.553494 0.722099 2 1.625936e+09 1.625936e+09 0.082 1.350428 1.490812 2.229001 0.016666 0.639882 0.830598 0.838116 0.639882 0.791716 0.772154 3 1.625936e+09 1.625936e+09 0.074 1.609420 1.803123 2.578417 0.013637 2.690052 0.684351 2.690052 0.684351 1.327693 1.451886 4 1.625936e+09 1.625936e+09 0.208 1.459232 1.441996 2.610289 0.040791 0.208621 1.136773 1.298535 0.162133 1.088990 0.935486 ... ... ... ... ... ... ... ... ... ... ... ... ... ... 160 1.625937e+09 1.625937e+09 0.212 3.099559 3.296528 5.312823 0.013042 1.183369 0.500407 1.183369 0.386452 0.827476 0.791367 161 1.625937e+09 1.625937e+09 0.406 2.234536 2.555549 4.133360 0.013230 0.483370 0.415434 0.967244 0.170981 0.615845 0.603776 162 1.625937e+09 1.625937e+09 0.106 1.272785 1.090846 2.567647 0.029594 1.173970 1.531232 1.586072 1.173970 1.533011 1.472277 163 1.625937e+09 1.625937e+09 0.130 2.206783 2.288842 3.916084 0.000061 1.843658 2.197606 2.263417 1.843658 2.212691 2.156476 164 1.625937e+09 1.625937e+09 0.088 1.682973 1.689252 3.048358 0.021588 1.288769 1.058330 1.334759 1.058330 1.296841 1.260912 <p>165 rows \u00d7 13 columns</p> In\u00a0[8]: Copied! <pre>import matplotlib.pyplot as plt\nimport numpy as np\n\nfig, axes = plt.subplots(2, 1, sharex=True, figsize=(15, 4))\nspeed = MuaEventsV1.get_speed(mua_key).to_numpy()\ntime = speed.index.to_numpy()\nmultiunit_firing_rate = MuaEventsV1.get_firing_rate(mua_key, time)\n\ntime_slice = slice(\n    np.searchsorted(time, mua_times.loc[10].start_time) - 1_000,\n    np.searchsorted(time, mua_times.loc[10].start_time) + 5_000,\n)\n\naxes[0].plot(\n    time[time_slice],\n    multiunit_firing_rate[time_slice],\n    color=\"black\",\n)\naxes[0].set_ylabel(\"firing rate (Hz)\")\naxes[0].set_title(\"multiunit\")\naxes[1].fill_between(time[time_slice], speed[time_slice], color=\"lightgrey\")\naxes[1].set_ylabel(\"speed (cm/s)\")\naxes[1].set_xlabel(\"time (s)\")\n\nfor id, mua_time in mua_times.loc[\n    np.logical_and(\n        mua_times[\"start_time\"] &gt; time[time_slice].min(),\n        mua_times[\"end_time\"] &lt; time[time_slice].max(),\n    )\n].iterrows():\n    axes[0].axvspan(\n        mua_time[\"start_time\"], mua_time[\"end_time\"], color=\"red\", alpha=0.5\n    )\n</pre> import matplotlib.pyplot as plt import numpy as np  fig, axes = plt.subplots(2, 1, sharex=True, figsize=(15, 4)) speed = MuaEventsV1.get_speed(mua_key).to_numpy() time = speed.index.to_numpy() multiunit_firing_rate = MuaEventsV1.get_firing_rate(mua_key, time)  time_slice = slice(     np.searchsorted(time, mua_times.loc[10].start_time) - 1_000,     np.searchsorted(time, mua_times.loc[10].start_time) + 5_000, )  axes[0].plot(     time[time_slice],     multiunit_firing_rate[time_slice],     color=\"black\", ) axes[0].set_ylabel(\"firing rate (Hz)\") axes[0].set_title(\"multiunit\") axes[1].fill_between(time[time_slice], speed[time_slice], color=\"lightgrey\") axes[1].set_ylabel(\"speed (cm/s)\") axes[1].set_xlabel(\"time (s)\")  for id, mua_time in mua_times.loc[     np.logical_and(         mua_times[\"start_time\"] &gt; time[time_slice].min(),         mua_times[\"end_time\"] &lt; time[time_slice].max(),     ) ].iterrows():     axes[0].axvspan(         mua_time[\"start_time\"], mua_time[\"end_time\"], color=\"red\", alpha=0.5     ) <pre>[14:44:19][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.6.0 because version 1.5.1 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n/home/edeno/miniconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.3.0 because version 0.2.0 is already loaded.\n  warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n[14:44:24][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n</pre> In\u00a0[9]: Copied! <pre>(MuaEventsV1 &amp; mua_key).create_figurl(\n    zscore_mua=True,\n)\n</pre> (MuaEventsV1 &amp; mua_key).create_figurl(     zscore_mua=True, ) <pre>[14:44:25][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n[14:44:29][WARNING] Spyglass: Upsampled position data, frame indices are invalid. Setting add_frame_ind=False\n</pre> <pre>WARNING: TimeseriesGraph::_add_series y argument is not 1D array. Using squeeze.\n</pre> Out[9]: <pre>'https://figurl.org/f?v=gs://figurl/spikesortingview-10&amp;d=sha1://08d7c35434e8469686fca8aff3ec9319bccb44f9&amp;label=Multiunit%20Detection&amp;zone=franklab.default'</pre> In\u00a0[\u00a0]: Copied! <pre>\n</pre>"}]}
\ No newline at end of file
diff --git a/0.5/sitemap.xml.gz b/0.5/sitemap.xml.gz
index e53b67514cbe6b210a26359df6067e36ea5f7c57..34701dd7e2f8ae144bf1fcd99149596a9cfde0ff 100644
GIT binary patch
delta 15
WcmbQvHJyu1zMF&NLGMO3K~?}GXaqd~

delta 15
WcmbQvHJyu1zMF&NQO`y;K~?}GZv;L7