Filtering units for decoding (LorenFrankLab#807)

* Remove old decoding notebook * Save initial conditions and discrete transitions * Apply suggestions from code review Co-authored-by: Chris Brozdowski <[email protected]> * Be more specific with import error * Remove unneeded comments * Remove incorrect dimension name * Project merge_id from SpikeSortingOutput for clarity * Update src/spyglass/decoding/v0/clusterless.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/decoding/v0/clusterless.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/decoding/v0/clusterless.py Co-authored-by: Chris Brozdowski <[email protected]> * Fix linting * Update notebooks * Ignore .pem * Add session as a primary key for Groups * Add some helper methods * Update notebooks * Update README.md * Update pyscripts * Update 42_Decoding_Clusterless.ipynb * Update CHANGELOG.md * Add fetch and insert * Simplify class conversion * Do the dictionary conversion of class for the user * Update CHANGELOG.md * Update .gitignore * Use methods in populate * Avoid fetching interval range if not needed * Generalize finding class from modules * Use args/kwargs * Simplify tuple unpacking * Make decoding kwargs nullable * Add function for get_recording and get_sorting to the spikesorting merge table * make decoding waveform features agnostic to spikesorting source * Fix spelling * Use fetch1_dataframe for position * Use self instead of class * Update src/spyglass/decoding/v1/sorted_spikes.py Co-authored-by: Samuel Bray <[email protected]> * Be more careful about populating select keys * Make more readable/remove unused imports * Save classifier * Clean up saved model paths * add function load_linear_position_info * Update src/spyglass/decoding/v1/sorted_spikes.py Co-authored-by: Samuel Bray <[email protected]> * Update 41_Extracting_Clusterless_Waveform_Features.py * Update docstring * Apply suggestions from code review Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/decoding/v1/clusterless.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/decoding/v1/clusterless.py Co-authored-by: Chris Brozdowski <[email protected]> * Fix linting * Fix syntax * Rename variable to avoid confusion * Restrict UnitWaveformFeaturesGroup and SortedSpikesGroup * Concatenate linear position and position dataframes * Static methods don't require instantiating class * Avoid merge restrict * Add version to defaults * Remove unused import * Fix classifier path * Add dry run * Remove non-default * Handle permissions and file not found * Keep position info within encoding/decoding interval * Add methods to get the spike_times, spike_indicators, firing rate * Fix docstring to match default * Implement function rather than import * Remove unused broken imports * Add decoding cleanup * Fix import * Put old vis code back * Fix import * Add draft helper functions * Limit options on input * Fix logic * Fix where the key is passed * Update notebooks * Host main visualizations in non_local_detector repo * Update notebooks/py_scripts/41_Extracting_Clusterless_Waveform_Features.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/spikesorting/merge.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/decoding/decoding_merge.py Co-authored-by: Chris Brozdowski <[email protected]> * Revert "Limit options on input" This reverts commit 386714c. * Use f-string for version * Add useful imports to the top level This would have to change a bit if there were multiple versions of the pipeline. * Make source class a hidden attribute * Update CHANGELOG.md * Centralize get_class logic in Merge (LorenFrankLab#749) * get_class logic -> dj_merge * blackify --------- Co-authored-by: Eric Denovellis <[email protected]> * Add _nwb_table for fetch_nwb * Method is static method * Add merge insert * string split is brittle, use defaults if it didn't work * WIP: Mixin resolves _nwb_table attr for Merge (LorenFrankLab#783) * Change import * Handle single position 3D case * Fix getting source from key * Use merge_restrict_class for fetch_nwb on merge tables * Move this back * Remove for now * Temp patch for tests * Revert "Temp patch for tests" This reverts commit 281bf36. * Temp patch for tests * Handle None decoding kwargs * fetch_nwb is a method not a class method now * Fix _merge_repr for numeric data types (LorenFrankLab#786) * Easily calculate firing rate * Add sorting spike times by place field and ahead behind distance * Account for differently named position variables * Handle orientation name and fix linear position fetch * Fix 2D ahead/behind * Add `UnitSelection` table (LorenFrankLab#788) * Add UnitSelection * Rename table * Addressing LorenFrankLab#789, failing tests (LorenFrankLab#795) * Addressing LorenFrankLab#789. See details. - Edit settings.py to permit fail to load config on startup - Simplify instructions to use func to generate config, negating need for config.py script - Edit notebook 00 and installation doc to point to save-config func - Expand example config to demonstrate all possible values - Lint dj_merge_table.py to remove unused import and `var is True` * LorenFrankLab#794 * Fix failing tests * Update Changelog * Typo: chared -> shared * Apply suggestions from code review Co-authored-by: Chris Brozdowski <[email protected]> * Fix formatting * Update src/spyglass/common/common_behav.py Co-authored-by: Chris Brozdowski <[email protected]> * Update src/spyglass/settings.py Co-authored-by: Chris Brozdowski <[email protected]> * Fix syntax * Update merge.py * Revert "Update merge.py" This reverts commit 4e5e5dc. * Change load_results to fetch_results for consistency * More load -> fetch * load -> fetch * Use merge table methods * load -> fetch * Rename SortedSpikesGroup.SortGroup to SortedSpikesGroup.Units * Move old spike sorting pipeline to v0 for organizational purposes * Specify full import path * Rename merge module for consistency with other pipelines * Fix import * Fix notebook v0/v1 imports * Fix import * Fix import * Fix import * Fix names * Formatting imports * Remove visualization folder * Move figurl views * Fix formatting for black==24.*.* * Use full import path * Fix method name * Add UnitInclusion merge table * Update 43_Decoding_SortedSpikes.py * Update pyscripts * Remove unneeded sections * Move import inside function * Fix versioning * Fix versioning * Be more uniform about explicit imports * Update CHANGELOG.md * Fix import * Fix LorenFrankLab#752 * Make imports more explicit * Only import cv2 when needed because it sometimes causes issues * Remove position tools as dependency The functions are relatively simple so copying them over doesn't cause much burden * Fix import name * Rename to linearization_merge for consistency with other modules * Reduce cv2 dependency * Remove unused import * Remove defaults channel to avoid issues * Remove position tools dependency * Add opencv dependency position tools had this as a dependency but since this is removed * Revert "Remove position tools as dependency" This reverts commit 35cd5ed. * Revert "Remove position tools dependency" This reverts commit 451e058. * Revert "Add opencv dependency" This reverts commit 996be9b. * Update dependencies to match environment.yml * Update environment_position.yml * Add figure 1 image and pypi version badge * Update citation * Update to add all authors * Add features to readme * Create CODE_OF_CONDUCT.md * Create .mailmap * Update .mailmap * Fix linting * Remove unused tables * Fix imports * Don't make time relative * Move sorted spikes group to spikesorting * Add MUA detection * Update CHANGELOG.md * Handle discrete states and squeeze to avoid intervals if possible * Fix table name * Add intervallist to exclude artifacts * Remove .mailmap * Revert "Add intervallist to exclude artifacts" This reverts commit 205874a. * Apply suggestions from code review Co-authored-by: Chris Brozdowski <[email protected]> * Revert "Revert "Add intervallist to exclude artifacts"" This reverts commit e102646. * remove interval list * Fix linting * Fix linting * Add missing authors * Update CHANGELOG.md * Fix schema name * Remove unused imports * Revert back * Use quotes * Fix name * Move mua because of dependency on position * Fix prefix * Fix schema prefix * Fix name * Add mua * Update 51_MUA_Detection.ipynb * Update src/spyglass/spikesorting/v1/unit_inclusion.py Co-authored-by: Chris Brozdowski <[email protected]> * Imported spikes annotation * Fix name of annotated keys and add documentation * Enable SortedSpikesGroup to filter directly on the SpikeSortingOutput nwb file * improve filter_units execution * Apply suggestions from code review Co-authored-by: Chris Brozdowski <[email protected]> * style cleanup * remove UnitInclusion tables * fix fetch_spike_data for case where no units in merge id * Update src/spyglass/decoding/v1/sorted_spikes.py Co-authored-by: Samuel Bray <[email protected]> * rename UnitSelectionParams * updated sorted decoding notebook * fix linting * fix linting * Fix spelling * Fix import order * Squeeze posterior * Make spike indicator 2D * insert class contents --------- Co-authored-by: Chris Brozdowski <[email protected]> Co-authored-by: Sam Bray <[email protected]> Co-authored-by: Kyu Hyun Lee <[email protected]>
CBroz1 · Feb 3, 2024 · e8ae471 · e8ae471
1 parent 42f1a1a
commit e8ae471
Show file tree

Hide file tree

Showing 88 changed files with 4,906 additions and 1,099 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -18,6 +18,15 @@
 - Increase pytest coverage for `common`, `lfp`, and `utils`. #743
 - Update docs to reflect new notebooks. #776
 - Add overview of Spyglass to docs. #779
+- Update linting for Black 24. #808
+
+### Pipelines
+
+- Spike sorting:
+    - Add SpikeSorting V1 pipeline. #651
+    - Move modules into spikesorting.v0 #807
+    - Add MUA analysis to spike sorting pipeline
+- LFP: Minor fixes to LFPBandV1 populator and `make`. #706, #795
 
 ### Pipelines
 
@@ -32,13 +41,18 @@
 - Position:
     - Refactor input validation in DLC pipeline. #688
     - DLC path handling from config, and normalize naming convention. #722
+    - Fix in place column bug #752
 - Decoding:
     - Add `decoding` pipeline V1. #731, #769
     - Add a table to store the decoding results #731
     - Use the new `non_local_detector` package for decoding #731
-    - Allow multiple spike waveform features for clusterelss decoding #731
+    - Allow multiple spike waveform features for clusterless decoding #731
     - Reorder notebooks #731
     - Add fetch class functionality to `Merge` table. #783, #786
+    - Add ability to filter sorted units in decoding #807
+    - Rename SortedSpikesGroup.SortGroup to SortedSpikesGroup.Units #807
+    - Change methods with load_... to fetch_... for consistency #807
+    - Use merge table methods to access part methods #807
 
 ## [0.4.3] (November 7, 2023)
 

diff --git a/CITATION.cff b/CITATION.cff
@@ -1,15 +1,150 @@
+# This CITATION.cff file was generated with cffinit.
+# Visit https://bit.ly/cffinit to generate yours today!
+
 cff-version: 1.2.0
-message: "If you use this software, please cite it as below."
+title: spyglass
+message: 'If you use this software, please cite it as below.'
+type: software
 authors:
-- family-names: "Lee"
-  given-names: "Kyu Hyun"
-- family-names: "Denovellis"
-  given-names: "Eric"
-- family-names: "Ly"
-  given-names: "Ryan"
-- family-names: "Frank"
-  given-names: "Loren"
-title: "spyglass"
+  - given-names: Kyu Hyun
+    family-names: Lee
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0001-6483-9444'
+  - given-names: Eric L.
+    family-names: Denovellis
+    email: [email protected]
+    affiliation: Howard Hughes Medical Institute
+    orcid: 'https://orcid.org/0000-0003-4606-087X'
+  - given-names: Ryan
+    family-names: Ly
+    email: [email protected]
+    affiliation: Lawrence Berkeley National Laboratory
+    orcid: 'https://orcid.org/0000-0001-9238-0642'
+  - given-names: Jeremy
+    family-names: Magland
+    email: [email protected]
+    affiliation: Flatiron Institute
+    orcid: 'https://orcid.org/0000-0002-5286-4375'
+  - given-names: Jeff
+    family-names: Soules
+    email: [email protected]
+    affiliation: Flatiron Institute
+  - given-names: Alison E.
+    family-names: Comrie
+    orcid: 'https://orcid.org/0000-0002-2251-4892'
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+  - given-names: Daniel P.
+    family-names: Gramling
+    orcid: 'https://orcid.org/0009-0004-6136-3769'
+    affiliation: University of Tübingen
+  - given-names: Jennifer A.
+    family-names: Guidera
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0003-4013-214X'
+  - given-names: Rhino
+    family-names: Nevers
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+  - given-names: Philip
+    family-names: Adenekan
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0002-4999-7847'
+  - given-names: Chris
+    family-names: Brozdowski
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0002-3055-6718'
+  - given-names: Samuel
+    family-names: Bray
+    email: [email protected]
+    orcid: 'https://orcid.org/0000-0002-9696-6920'
+    affiliation: 'University of California, San Francisco'
+  - given-names: Emily
+    family-names: Monroe
+    affiliation: University of California, Davis
+  - given-names: Ji Hyun
+    family-names: Bak
+    orcid: 'https://orcid.org/0000-0002-5700-7823'
+  - given-names: Michael E.
+    family-names: Coulter
+    email: [email protected]
+    orcid: 'https://orcid.org/0000-0003-0357-351X'
+    affiliation: 'University of California, San Francisco'
+  - given-names: Xulu
+    family-names: Sun
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0003-3076-0994'
+  - given-names: Andrew
+    family-names: Tritt
+    email: [email protected]
+    affiliation: Lawrence Berkeley National Laboratory
+    orcid: 'https://orcid.org/0000-0002-1617-449X'
+  - given-names: Oliver
+    family-names: Rübel
+    email: [email protected]
+    affiliation: Lawrence Berkeley National Laboratory
+    orcid: 'https://orcid.org/0000-0001-9902-1984'
+  - given-names: Thinh
+    family-names: Nguyen
+    affiliation: Datajoint
+    email: [email protected]
+  - given-names: Dimitri
+    family-names: Yatsenko
+    email: [email protected]
+    affiliation: Datajoint
+    orcid: 'https://orcid.org/0000-0002-1844-641X'
+  - given-names: Joshua
+    family-names: Chu
+    email: [email protected]
+    affiliation: Rice University
+  - given-names: Caleb
+    family-names: Kemere
+    email: [email protected]
+    affiliation: Rice University
+    orcid: 'https://orcid.org/0000-0003-2054-0234'
+  - given-names: Samuel
+    family-names: Garcia
+    email: [email protected]
+    affiliation: 'Centre de Recherche en Neurosciences de Lyon'
+    orcid: 'https://orcid.org/0000-0001-6389-9779'
+  - given-names: Alessio
+    family-names: Buccino
+    email: [email protected]
+    affiliation: Allen Institute
+    orcid: 'https://orcid.org/0000-0003-3661-527X'
+  - given-names: Loren M.
+    family-names: Frank
+    email: [email protected]
+    affiliation: 'University of California, San Francisco'
+    orcid: 'https://orcid.org/0000-0002-1752-5677'
+identifiers:
+  - type: doi
+    value: 10.1101/2024.01.25.577295
+    description: bioRxiv preprint
+repository-code: 'https://github.com/LorenFrankLab/spyglass'
+url: 'https://lorenfranklab.github.io/spyglass/'
+abstract: >-
+  Spyglass is a data analysis framework that facilitates the
+  storage, analysis, visualization, and sharing of
+  neuroscience data to support reproducible research. It is
+  designed to be interoperable with the NWB format and
+  integrates open-source tools into a coherent framework.
+keywords:
+  - neuroscience
+  - open science
+  - electrophysiology
+  - figurl
+  - interactive data visualization
+  - Neurodata Without Borders
+  - Spike interface
+  - Datajoint
+  - spike sorting
+  - kachery
+license: MIT
 version: 0.4.3
-date-released: 2023-11-07
-url: "https://github.com/LorenFrankLab/spyglass"
+date-released: '2023-11-07'
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
@@ -0,0 +1,133 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[email protected].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations