More robust delete_downstream_merge (LorenFrankLab#806)

* WIP: fix for LorenFrankLab#791 * WIP: LorenFrankLab#791, pt 2 * WIP: LorenFrankLab#791, needs testing * Faster tree search with networkx * Blackify * Blackify 2 * Update changelog/docs * Update notebooks * Overwrite . Mixin add cached_property decorator * Cleanup docstrings and type annotations
CBroz1 · Jan 31, 2024 · b42432f · b42432f
1 parent 5d8f7ad
commit b42432f
Show file tree

Hide file tree

Showing 13 changed files with 681 additions and 412 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,19 +8,23 @@
 - Clean up following pre-commit checks. #688
 - Add Mixin class to centralize `fetch_nwb` functionality. #692, #734
 - Refactor restriction use in `delete_downstream_merge` #703
-- Add `cautious_delete` to Mixin class, initial implementation. #711, #762
+- Add `cautious_delete` to Mixin class
+    - Initial implementation. #711, #762
+    - More robust caching of join to downstream tables. #806
+    - Overwrite datajoint `delete` method to use `cautious_delete`. #806
 - Add `deprecation_factory` to facilitate table migration. #717
 - Add Spyglass logger. #730
 - IntervalList: Add secondary key `pipeline` #742
 - Increase pytest coverage for `common`, `lfp`, and `utils`. #743
 - Update docs to reflect new notebooks. #776
 - Add overview of Spyglass to docs. #779
-- LFPV1: Fix error for multiple lfp settings on same data #775
 
 ### Pipelines
 
 - Spike sorting: Add SpikeSorting V1 pipeline. #651
-- LFP: Minor fixes to LFPBandV1 populator and `make`. #706, #795
+- LFP:
+    - Minor fixes to LFPBandV1 populator and `make`. #706, #795
+    - LFPV1: Fix error for multiple lfp settings on same data #775
 - Linearization:
     - Minor fixes to LinearizedPositionV1 pipeline #695
     - Rename `position_linearization` -> `linearization`. #717

diff --git a/docs/src/misc/merge_tables.md b/docs/src/misc/merge_tables.md
@@ -15,8 +15,17 @@ deleting a part entry before the master. To circumvent this, you can add
 `force_parts=True` to the
 [`delete` function](https://datajoint.com/docs/core/datajoint-python/0.14/api/datajoint/__init__/#datajoint.table.Table.delete)
 call, but this will leave and orphaned primary key in the master. Instead, use
-`spyglass.utils.dj_merge_tables.delete_downstream_merge` to delete master/part
-pairs.
+`(YourTable & restriction).delete_downstream_merge()` to delete master/part
+pairs. If errors persist, identify and import the offending part table and rerun
+`delete_downstream_merge` with `reload_cache=True`. This process will be faster
+for subsequent calls if you reassign the your table after importing.
+
+```python
+from spyglass.common import Nwbfile
+
+nwbfile = Nwbfile()
+(nwbfile & "nwb_file_name LIKE 'Name%'").delete_downstream_merge()
+```
 
 ## What
 

diff --git a/notebooks/01_Insert_Data.ipynb b/notebooks/01_Insert_Data.ipynb
@@ -45,8 +45,8 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "[2023-10-05 11:48:12,292][INFO]: Connecting root@localhost:3306\n",
-      "[2023-10-05 11:48:12,302][INFO]: Connected root@localhost:3306\n"
+      "[2024-01-29 16:24:30,933][INFO]: Connecting root@localhost:3309\n",
+      "[2024-01-29 16:24:30,942][INFO]: Connected root@localhost:3309\n"
      ]
     }
    ],
@@ -719,9 +719,9 @@
     "\n",
     "- `minirec20230622.nwb`, .3 GB: minimal recording,\n",
     "  [Link](https://ucsf.box.com/s/k3sgql6z475oia848q1rgms4zdh4rkjn)\n",
-    "- `mediumnwb20230802.nwb`, 32 GB: full-featured dataset, \n",
-    "  [Link](https://ucsf.box.com/s/2qbhxghzpttfam4b7q7j8eg0qkut0opa) \n",
-    "- `montague20200802.nwb`, 8 GB: full experimental recording, \n",
+    "- `mediumnwb20230802.nwb`, 32 GB: full-featured dataset,\n",
+    "  [Link](https://ucsf.box.com/s/2qbhxghzpttfam4b7q7j8eg0qkut0opa)\n",
+    "- `montague20200802.nwb`, 8 GB: full experimental recording,\n",
     "  [Link](https://ucsf.box.com/s/26je2eytjpqepyznwpm92020ztjuaomb)\n",
     "- For those in the UCSF network, these and many others on `/stelmo/nwb/raw`\n",
     "\n",
@@ -747,7 +747,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Spyglass will create a copy with this name."
+    "Spyglass will create a copy with this name.\n"
    ]
   },
   {
@@ -1072,7 +1072,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "\n",
     "`spyglass.data_import.insert_sessions` helps take the many fields of data\n",
     "present in an NWB file and insert them into various tables across Spyglass. If\n",
     "the NWB file is properly composed, this includes...\n",
@@ -1082,8 +1081,8 @@
     "- neural activity (extracellular recording of multiple brain areas)\n",
     "- etc.\n",
     "\n",
-    "_Note:_ this may take time as Spyglass creates the copy. You may see a prompt \n",
-    "about inserting device information."
+    "_Note:_ this may take time as Spyglass creates the copy. You may see a prompt\n",
+    "about inserting device information.\n"
    ]
   },
   {
@@ -2053,21 +2052,20 @@
    "metadata": {},
    "source": [
     "`IntervalList` has an additional secondary key `pipeline` which can describe the origin of the data.\n",
-    "Because it is a _secondary_ key, it is not required to uniquely identify an entry. \n",
+    "Because it is a _secondary_ key, it is not required to uniquely identify an entry.\n",
     "Current values for this key from spyglass pipelines are:\n",
     "\n",
-    "| pipeline | Source|\n",
-    "| ---   | --- |\n",
-    "| position | sg.common.PositionSource |\n",
-    "| lfp_v0 | sg.common.LFP |\n",
-    "| lfp_v1 | sg.lfp.v1.LFPV1 |\n",
-    "| lfp_band | sg.common.LFPBand,<br>  sg.lfp.analysis.v1.LFPBandV1 |\n",
-    "| lfp_artifact | sg.lfp.v1.LFPArtifactDetection |\n",
-    "| spikesorting_artifact_v0 | sg.spikesorting.ArtifactDetection |\n",
-    "| spikesorting_artifact_v1 | sg.spikesorting.v1.ArtifactDetection |\n",
-    "| spikesorting_recording_v0 | sg.spikesorting.SpikeSortingRecording |\n",
-    "| spikesorting_recording_v1 | sg.spikesorting.v1.SpikeSortingRecording |\n",
-    "\n"
+    "| pipeline                  | Source                                              |\n",
+    "| ------------------------- | --------------------------------------------------- |\n",
+    "| position                  | sg.common.PositionSource                            |\n",
+    "| lfp_v0                    | sg.common.LFP                                       |\n",
+    "| lfp_v1                    | sg.lfp.v1.LFPV1                                     |\n",
+    "| lfp_band                  | sg.common.LFPBand,<br> sg.lfp.analysis.v1.LFPBandV1 |\n",
+    "| lfp_artifact              | sg.lfp.v1.LFPArtifactDetection                      |\n",
+    "| spikesorting_artifact_v0  | sg.spikesorting.ArtifactDetection                   |\n",
+    "| spikesorting_artifact_v1  | sg.spikesorting.v1.ArtifactDetection                |\n",
+    "| spikesorting_recording_v0 | sg.spikesorting.SpikeSortingRecording               |\n",
+    "| spikesorting_recording_v1 | sg.spikesorting.v1.SpikeSortingRecording            |\n"
    ]
   },
   {
@@ -2086,9 +2084,9 @@
     "with _cascading deletes_. For example, if we delete our `Session` entry, all\n",
     "associated downstream entries are also deleted (e.g. `Raw`, `IntervalList`).\n",
     "\n",
-    "_Note_: The deletion process can be complicated by \n",
+    "_Note_: The deletion process can be complicated by\n",
     "[Merge Tables](https://lorenfranklab.github.io/spyglass/0.4/misc/merge_tables/)\n",
-    "when the entry is referenced by a part table. To demo deletion in these cases, \n",
+    "when the entry is referenced by a part table. To demo deletion in these cases,\n",
     "run the hidden code below.\n",
     "\n",
     "<details>\n",
@@ -2113,20 +2111,23 @@
     "lfp.v1.LFPSelection.insert1(lfp_key, skip_duplicates=True)\n",
     "lfp.v1.LFPV1().populate(lfp_key)\n",
     "```\n",
+    "\n",
     "</details>\n",
     "<details>\n",
     "<summary>Deleting Merge Entries</summary>\n",
     "\n",
     "```python\n",
-    "from spyglass.utils.dj_merge_tables import delete_downstream_merge\n",
+    "nwbfile = sgc.Nwbfile()\n",
     "\n",
-    "delete_downstream_merge(\n",
-    "    sgc.Nwbfile(),\n",
-    "    restriction={\"nwb_file_name\": nwb_copy_file_name},\n",
+    "(nwbfile & {\"nwb_file_name\": nwb_copy_file_name}).delete_downstream_merge(\n",
     "    dry_run=False, # True will show Merge Table entries that would be deleted\n",
-    ") \n",
+    ")\n",
     "```\n",
-    "</details>"
+    "\n",
+    "Please see the [next notebook](./03_Merge_Tables.ipynb) for a more detailed\n",
+    "explanation.\n",
+    "\n",
+    "</details>\n"
    ]
   },
   {
@@ -2659,7 +2660,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Up Next"
+    "## Up Next\n"
    ]
   },
   {