WIP: Fetch upstream

CHANGELOG.md

@@ -10,6 +10,14 @@
 
 - Create class `SpyglassGroupPart` to aid delete propagations #899
 - Fix bug report template #955
+- Add rollback option to `populate_all_common` #957, #971
+- Add long-distance restrictions via `<<` and `>>` operators. #943, #969
+- Fix relative pathing for `mkdocstring-python=>1.9.1`. #967, #968
+
+### Pipelines
+
+- DLC
+   - Allow dlc without pre-existing tracking data #973, #975
 
 ## [0.5.2] (April 22, 2024)
 

docs/build-docs.sh

@@ -7,7 +7,7 @@
 cp ./CHANGELOG.md ./docs/src/
 cp ./LICENSE ./docs/src/LICENSE.md
 mkdir -p ./docs/src/notebooks
-rm -r ./docs/src/notebooks/*
+rm -fr ./docs/src/notebooks/*
 cp ./notebooks/*ipynb ./docs/src/notebooks/
 cp ./notebooks/*md ./docs/src/notebooks/
 mv ./docs/src/notebooks/README.md ./docs/src/notebooks/index.md

docs/mkdocs.yml

@@ -74,6 +74,7 @@ nav:
       - FigURL: misc/figurl_views.md
       - Session Groups: misc/session_groups.md
       - Insert Data: misc/insert_data.md
+      - Mixin: misc/mixin.md
       - Merge Tables: misc/merge_tables.md
       - Database Management: misc/database_management.md
       - Export: misc/export.md
@@ -105,6 +106,7 @@ plugins:
             group_by_category: false
             line_length: 80
             docstring_style: numpy
+          paths: [../src]
   - literate-nav:
       nav_file: navigation.md
   - exclude-search:

docs/src/api/make_pages.py

@@ -16,8 +16,11 @@
     if path.stem in ignored_stems or "cython" in path.stem:
         continue
     rel_path = path.relative_to("src/spyglass")
+
+    # parts[0] is the src directory, ignore as of mkdocstrings-python 1.9.1
+    module_path = ".".join([p for p in path.with_suffix("").parts[1:]])
+
     with mkdocs_gen_files.open(f"api/{rel_path.with_suffix('')}.md", "w") as f:
-        module_path = ".".join([p for p in path.with_suffix("").parts])
         print(f"::: {module_path}", file=f)
     nav[rel_path.parts] = f"{rel_path.with_suffix('')}.md"
 

docs/src/misc/index.md

@@ -3,7 +3,9 @@
 This folder contains miscellaneous supporting files documentation.
 
 - [Database Management](./database_management.md)
+- [Export](./export.md)
 - [figurl Views](./figurl_views.md)
-- [insert Data](./insert_data.md)
+- [Insert Data](./insert_data.md)
 - [Merge Tables](./merge_tables.md)
+- [Mixin Class](./mixin.md)
 - [Session Groups](./session_groups.md)

docs/src/misc/mixin.md

@@ -4,23 +4,22 @@ The Spyglass Mixin provides a way to centralize all Spyglass-specific
 functionalities that have been added to DataJoint tables. This includes...
 
 - Fetching NWB files
+- Long-distance restrictions.
 - Delete functionality, including permission checks and part/master pairs
 - Export logging. See [export doc](export.md) for more information.
 
 To add this functionality to your own tables, simply inherit from the mixin:
 
 ```python
 import datajoint as dj
+
 from spyglass.utils import SpyglassMixin
 
-schema = dj.schema('my_schema')
+schema = dj.schema("my_schema")
 
-@schema
-class MyOldTable(dj.Manual):
-    pass
 
 @schema
-class MyNewTable(SpyglassMixin, dj.Manual):)
+class MyOldTable(dj.Manual):
     pass
 ```
 
@@ -44,6 +43,60 @@ should be fetched from `Nwbfile` or an analysis file should be fetched from
 `AnalysisNwbfile`. If neither is foreign-key-referenced, the function will refer
 to a `_nwb_table` attribute.
 
+## Long-Distance Restrictions
+
+In complicated pipelines like Spyglass, there are often tables that 'bury' their
+foreign keys as secondary keys. This is done to avoid having to pass a long list
+of foreign keys through the pipeline, potentially hitting SQL limits (see also
+[Merge Tables](./merge_tables.md)). This burrying makes it difficult to restrict
+a given table by familiar attributes.
+
+Spyglass provides a function, `restrict_by`, to handle this. The function takes
+your restriction and checks parents/children until the restriction can be
+applied. Spyglass introduces `<<` as a shorthand for `restrict_by` an upstream
+key and `>>` as a shorthand for `restrict_by` a downstream key.
+
+```python
+from spyglass.example import AnyTable
+
+AnyTable() << 'upstream_attribute="value"'
+AnyTable() >> 'downsteam_attribute="value"'
+
+# Equivalent to
+AnyTable().restrict_by('downsteam_attribute="value"', direction="down")
+AnyTable().restrict_by('upstream_attribute="value"', direction="up")
+```
+
+Some caveats to this function:
+
+1. 'Peripheral' tables, like `IntervalList` and `AnalysisNwbfile` make it hard
+    to determine the correct parent/child relationship and have been removed
+    from this search by default.
+2. This function will raise an error if it attempts to check a table that has
+    not been imported into the current namespace. It is best used for exploring
+    and debugging, not for production code.
+3. It's hard to determine the attributes in a mixed dictionary/string
+    restriction. If you are having trouble, try using a pure string
+    restriction.
+4. The most direct path to your restriction may not be the path your data took,
+    especially when using Merge Tables. When the result is empty see the
+    warning about the path used. Then, ban tables from the search to force a
+    different path.
+
+```python
+my_table = MyTable()  # must be instanced
+my_table.ban_search_table(UnwantedTable1)
+my_table.ban_search_table([UnwantedTable2, UnwantedTable3])
+my_table.unban_search_table(UnwantedTable3)
+my_table.see_banned_tables()
+
+my_table << my_restriction
+my_table << upstream_restriction >> downstream_restriction
+```
+
+When providing a restriction of the parent, use 'up' direction. When providing a
+restriction of the child, use 'down' direction.
+
 ## Delete Functionality
 
 The mixin overrides the default `delete` function to provide two additional

notebooks/01_Insert_Data.ipynb

@@ -1082,8 +1082,22 @@
     "- neural activity (extracellular recording of multiple brain areas)\n",
     "- etc.\n",
     "\n",
-    "_Note:_ this may take time as Spyglass creates the copy. You may see a prompt\n",
-    "about inserting device information.\n"
+    "_Notes:_ this may take time as Spyglass creates the copy. You may see a prompt\n",
+    "about inserting device information.\n",
+    "\n",
+    "By default, the session insert process is error permissive. It will log an\n",
+    "error and continue attempts across various tables. You have two options you can\n",
+    "toggle to adjust this.\n",
+    "\n",
+    "- `rollback_on_fail`: Default False. If True, errors will still be logged for\n",
+    "  all tables and, if any are registered, the `Nwbfile` entry will be deleted.\n",
+    "  This is helpful for knowing why your file failed, and making it easy to retry.\n",
+    "- `raise_err`: Default False. If True, errors will not be logged and will\n",
+    "  instead be raised. This is useful for debugging and exploring the error stack.\n",
+    "  The end result may be that some tables may still have entries from this file\n",
+    "  that will need to be manually deleted after a failed attempt. 'transactions'\n",
+    "  are used where possible to rollback sibling tables, but child table errors\n",
+    "  will still leave entries from parent tables.\n"
    ]
   },
   {
@@ -1146,7 +1160,7 @@
     }
    ],
    "source": [
-    "sgi.insert_sessions(nwb_file_name)"
+    "sgi.insert_sessions(nwb_file_name, rollback_on_fail=False, raise_error=False)"
    ]
   },
   {

notebooks/py_scripts/01_Insert_Data.py

@@ -198,11 +198,25 @@
 # - neural activity (extracellular recording of multiple brain areas)
 # - etc.
 #
-# _Note:_ this may take time as Spyglass creates the copy. You may see a prompt
+# _Notes:_ this may take time as Spyglass creates the copy. You may see a prompt
 # about inserting device information.
 #
+# By default, the session insert process is error permissive. It will log an
+# error and continue attempts across various tables. You have two options you can
+# toggle to adjust this.
+#
+# - `rollback_on_fail`: Default False. If True, errors will still be logged for
+#   all tables and, if any are registered, the `Nwbfile` entry will be deleted.
+#   This is helpful for knowing why your file failed, and making it easy to retry.
+# - `raise_err`: Default False. If True, errors will not be logged and will
+#   instead be raised. This is useful for debugging and exploring the error stack.
+#   The end result may be that some tables may still have entries from this file
+#   that will need to be manually deleted after a failed attempt. 'transactions'
+#   are used where possible to rollback sibling tables, but child table errors
+#   will still leave entries from parent tables.
+#
 
-sgi.insert_sessions(nwb_file_name)
+sgi.insert_sessions(nwb_file_name, rollback_on_fail=False, raise_error=False)
 
 # ## Inspecting the data
 #

notebooks/py_scripts/50_MUA_Detection.py

@@ -0,0 +1,111 @@
+# ---
+# jupyter:
+#   jupytext:
+#     text_representation:
+#       extension: .py
+#       format_name: light
+#       format_version: '1.5'
+#       jupytext_version: 1.16.0
+#   kernelspec:
+#     display_name: spyglass
+#     language: python
+#     name: python3
+# ---
+
+# +
+import datajoint as dj
+from pathlib import Path
+
+dj.config.load(
+    Path("../dj_local_conf.json").absolute()
+)  # load config for database connection info
+
+from spyglass.mua.v1.mua import MuaEventsV1, MuaEventsParameters
+
+# -
+
+MuaEventsParameters()
+
+MuaEventsV1()
+
+# +
+from spyglass.position import PositionOutput
+
+nwb_copy_file_name = "mediumnwb20230802_.nwb"
+
+trodes_s_key = {
+    "nwb_file_name": nwb_copy_file_name,
+    "interval_list_name": "pos 0 valid times",
+    "trodes_pos_params_name": "single_led_upsampled",
+}
+
+pos_merge_id = (PositionOutput.TrodesPosV1 & trodes_s_key).fetch1("merge_id")
+pos_merge_id
+
+# +
+from spyglass.spikesorting.analysis.v1.group import (
+    SortedSpikesGroup,
+)
+
+sorted_spikes_group_key = {
+    "nwb_file_name": nwb_copy_file_name,
+    "sorted_spikes_group_name": "test_group",
+    "unit_filter_params_name": "default_exclusion",
+}
+
+SortedSpikesGroup & sorted_spikes_group_key
+
+# +
+mua_key = {
+    "mua_param_name": "default",
+    **sorted_spikes_group_key,
+    "pos_merge_id": pos_merge_id,
+    "detection_interval": "pos 0 valid times",
+}
+
+MuaEventsV1().populate(mua_key)
+MuaEventsV1 & mua_key
+# -
+
+mua_times = (MuaEventsV1 & mua_key).fetch1_dataframe()
+mua_times
+
+# +
+import matplotlib.pyplot as plt
+import numpy as np
+
+fig, axes = plt.subplots(2, 1, sharex=True, figsize=(15, 4))
+speed = MuaEventsV1.get_speed(mua_key).to_numpy()
+time = speed.index.to_numpy()
+multiunit_firing_rate = MuaEventsV1.get_firing_rate(mua_key, time)
+
+time_slice = slice(
+    np.searchsorted(time, mua_times.loc[10].start_time) - 1_000,
+    np.searchsorted(time, mua_times.loc[10].start_time) + 5_000,
+)
+
+axes[0].plot(
+    time[time_slice],
+    multiunit_firing_rate[time_slice],
+    color="black",
+)
+axes[0].set_ylabel("firing rate (Hz)")
+axes[0].set_title("multiunit")
+axes[1].fill_between(time[time_slice], speed[time_slice], color="lightgrey")
+axes[1].set_ylabel("speed (cm/s)")
+axes[1].set_xlabel("time (s)")
+
+for id, mua_time in mua_times.loc[
+    np.logical_and(
+        mua_times["start_time"] > time[time_slice].min(),
+        mua_times["end_time"] < time[time_slice].max(),
+    )
+].iterrows():
+    axes[0].axvspan(
+        mua_time["start_time"], mua_time["end_time"], color="red", alpha=0.5
+    )
+# -
+
+(MuaEventsV1 & mua_key).create_figurl(
+    zscore_mua=True,
+)