Update changelog and notebooks

CBroz1 · Oct 6, 2023 · 737b9f2 · 737b9f2
1 parent c69dff7
commit 737b9f2
Show file tree

Hide file tree

Showing 8 changed files with 1,331 additions and 72 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,30 @@
 # Change Log
 
+## [0.4.2] (October 10, 2023)
+
+### Infrastructure / Support
+
+- Bumped Python version to 3.9. #583
+- Updated user management helper scripts for MySQL 8. #650
+- Centralized config/path handling to permit setting via datajoint config. #593
+- Fixed Merge Table deletes: error specificity and transaction context. #617
+
+### Pipelines
+
+- Common:
+  - Added support multiple cameras per epoch. #557
+  - Removed `common_backup` schema. #631
+  - Added support for multiple position objects per NWB in `common_behav` via
+    PositionSource.SpatialSeries and RawPosition.PosObject #628, #616.
+    _Note:_ Existing functions have been made compatible, but column labels for
+    `RawPosition.fetch1_dataframe` may change.
+- Spike sorting:
+  - Added pipeline populator. #637, #646, #647
+  - Fixed curation functionality for `nn_isolation`. #597, #598
+- Position: Added position interval/epoch mapping via PositionIntervalMap. #620,
+  #621, #627
+- LFP: Refactored pipeline. #594, #588, #605, #606, #607, #608, #615, #629
+
 ## [0.4.1] (June 30, 2023)
 
 - Add mkdocs automated deployment. #527, #537, #549, #551
@@ -62,6 +87,7 @@
 - Allow creation and linkage of device metadata from YAML #400
 - Move helper functions to utils directory #386
 
+[0.4.2]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.2
 [0.4.1]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.1
 [0.4.0]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.0
 [0.3.4]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.3.4

diff --git a/notebooks/01_Insert_Data.ipynb b/notebooks/01_Insert_Data.ipynb
@@ -45,8 +45,8 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "[2023-09-28 08:20:36,230][INFO]: Connecting root@localhost:3307\n",
-      "[2023-09-28 08:20:36,302][INFO]: Connected root@localhost:3307\n"
+      "[2023-10-05 11:48:12,292][INFO]: Connecting root@localhost:3306\n",
+      "[2023-10-05 11:48:12,302][INFO]: Connected root@localhost:3306\n"
      ]
     }
    ],
@@ -732,7 +732,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1088,11 +1088,63 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 3,
    "metadata": {
     "tags": []
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Creating a copy of NWB file minirec20230622.nwb with link to raw ephys data: minirec20230622_.nwb\n",
+      "Populate Session...\n",
+      "No config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\n",
+      "Institution...\n",
+      "Lab...\n",
+      "LabMember...\n",
+      "Please add the Google user ID for Firstname2 Lastname2 in the LabMember.LabMemberInfo table to help manage permissions.\n",
+      "Subject...\n",
+      "Populate CameraDevice...\n",
+      "Inserted camera devices ['test camera 1']\n",
+      "\n",
+      "Populate Probe...\n",
+      "Probe ID '128c-4s6mm6cm-15um-26um-sl' already exists in the database. Spyglass will use that and not create a new Probe, Shanks, or Electrodes.\n",
+      "Inserted probes {'128c-4s6mm6cm-15um-26um-sl'}\n",
+      "\n",
+      "Skipping Apparatus for now...\n",
+      "IntervalList...\n",
+      "LabMember with name lastname, firstname does not exist. Cannot link Session with LabMember in Session.Experimenter.\n",
+      "LabMember with name lastname2, firstname2 does not exist. Cannot link Session with LabMember in Session.Experimenter.\n",
+      "Populate ElectrodeGroup...\n",
+      "Populate Electrode...\n",
+      "No config found at file path /home/cb/wrk/data/raw/minirec20230622_spyglass_config.yaml\n",
+      "Populate Raw...\n",
+      "Estimating sampling rate...\n",
+      "Estimated sampling rate for file: 30000.0 Hz\n",
+      "Importing raw data: Sampling rate:\t30000.0 Hz\n",
+      "Number of valid intervals:\t2\n",
+      "Populate SampleCount...\n",
+      "Populate DIOEvents...\n",
+      "Populate TaskEpochs\n",
+      "Populate StateScriptFile\n",
+      "Populate VideoFile\n",
+      "No video found corresponding to file minirec20230622_.nwb, epoch 01_s1\n",
+      "No video found corresponding to file minirec20230622_.nwb, epoch 02_s2\n",
+      "RawPosition...\n",
+      "Estimated sampling rate for 12345: 30.0 Hz\n",
+      "WARNING: Setting minimum valid interval to 5.1912336349487305\n",
+      "Estimated sampling rate for 12345: 30.0 Hz\n",
+      "WARNING: Setting minimum valid interval to 5.1912336349487305\n",
+      "Estimated sampling rate for 12345: 30.0 Hz\n",
+      "WARNING: Setting minimum valid interval to 5.339195609092712\n",
+      "Estimated sampling rate for 12345: 30.0 Hz\n",
+      "WARNING: Setting minimum valid interval to 5.339195609092712\n",
+      "Populated PosIntervalMap for minirec20230622_.nwb, 01_s1\n",
+      "Populated PosIntervalMap for minirec20230622_.nwb, 02_s2\n"
+     ]
+    }
+   ],
    "source": [
     "sgi.insert_sessions(nwb_file_name)"
    ]
@@ -2010,7 +2062,51 @@
     "Another neat feature of DataJoint is that it automatically maintains\n",
     "[data integrity](https://datajoint.com/docs/core/datajoint-python/0.14/design/integrity/)\n",
     "with _cascading deletes_. For example, if we delete our `Session` entry, all\n",
-    "associated downstream entries are also deleted (e.g. `Raw`, `IntervalList`).\n"
+    "associated downstream entries are also deleted (e.g. `Raw`, `IntervalList`).\n",
+    "\n",
+    "_Note_: The deletion process can be complicated by \n",
+    "[Merge Tables](https://lorenfranklab.github.io/spyglass/0.4/misc/merge_tables/)\n",
+    "when the entry is referenced by a part table. To demo deletion in these cases, \n",
+    "run the hidden code below.\n",
+    "\n",
+    "<details>\n",
+    "<summary>Quick Merge Insert</summary>\n",
+    "\n",
+    "```python\n",
+    "import spyglass.lfp as lfp\n",
+    "\n",
+    "sgc.FirFilterParameters().create_standard_filters()\n",
+    "lfp.lfp_electrode.LFPElectrodeGroup.create_lfp_electrode_group(\n",
+    "    nwb_file_name=nwb_copy_file_name,\n",
+    "    group_name=\"test\",\n",
+    "    electrode_list=[0],\n",
+    ")\n",
+    "lfp.v1.LFPSelection.insert1(\n",
+    "    {\n",
+    "        \"nwb_file_name\": nwb_copy_file_name,\n",
+    "        \"lfp_electrode_group_name\": \"test\",\n",
+    "        \"target_interval_list_name\": \"01_s1\",\n",
+    "        \"filter_name\": \"LFP 0-400 Hz\",\n",
+    "        \"filter_sampling_rate\": 30_000,\n",
+    "    },\n",
+    "    skip_duplicates=True,\n",
+    ")\n",
+    "lfp.v1.LFPV1().populate()\n",
+    "```\n",
+    "</details>\n",
+    "<details>\n",
+    "<summary>Deleting Merge Entries</summary>\n",
+    "\n",
+    "```python\n",
+    "from spyglass.utils.dj_merge_tables import delete_downstream_merge\n",
+    "\n",
+    "delete_downstream_merge(\n",
+    "    sgc.Nwbfile(),\n",
+    "    restriction={\"nwb_file_name\": nwb_copy_file_name},\n",
+    "    dry_run=False, # True will show Merge Table entries that would be deleted\n",
+    ") \n",
+    "```\n",
+    "</details>"
    ]
   },
   {