v0.5.1 - Light maintenance fixes

Fixes to keep it sort of running and catch up on dependency drift....

Still haven't gotten pyside6 to work yet though which is a PROBLEM

• #211
  • Fix: #209 - update location of dtoverlays
  • Convert travis-based tests to github actions so they actually run
  • Don't raise NotImplementedError in datajoint module on import - what the hell was i thinking lmao
  • don't access deprecated alias np.float, just use float
  • fix a bunch of docs stuff to make it build
  • update dep versions and remove some upper bounds

v0.5.0a0 - The Data Modeling Edition

v0.5.0a0 - ~ The Data Modeling Edition ~

A lot. Namely a whole rewriting of the {mod}autopilot.data module.

This is being released as an alpha version while we finish working out a few of the kinks in the data modeling
system because Jonny is finishing their dissertation and has a deadline determined by the viscera of
institutional science rather than software development!

Upgrading From <v0.5.0

• The subject class will attempt to update any of your existing
  .h5 files to the new format -- it should be nondestructive (making a backup of the whole file first
  and then making backups up any tables that it can't create a new version of that fully preserves
  the data from the previous version), but don't be alarmed when your subject files start updating!
  We advise, as we always do, making a full backup of your data directory before making this update.
• Change references in launch_autopilot.sh scripts from autopilot.core.terminal or autopilot.core.pilot
  to autopilot.agents.terminal or autopilot.agents.pilot, respectively
• Update any TrialData declarations in plugin tasks to use the new {class}.Trial_Data model. See {attr}.Nafc.TrialData for an example.
  The old pytables IsDescriptor style descriptions will still work, but you will be given a warning every time.
• We advise updating pigpio to at least 3c23715 to
  avoid warnings and make use of some new functionality.

Package Structure

• The autopilot.core module was removed entirely, and was split into
  • {mod}autopilot.agent: which contains the {class}.Terminal and {class}.Pilot classes and a stub metaclass
  • {mod}autopilot.utils.loggers: formerly autopilot.core.loggers
  • {mod}autopilot.gui (described in module documentation) now has all the GUI modules, split out by type rather than all in one file.
  • {mod}autopilot.data was created from autopilot.core.subject, see below.
• {mod}autopilot.root contains root classes:
  • {class}.Autopilot_Type - types and data models for handling and saving live data
  • {class}.Autopilot_Object - General root metaclass, beneath which will be top-level metaclasses for each of the major
    class hierarchies that are currently separate.
  • {class}.Autopilot_Pref - Stub for later converting prefs and environmental configuration parameters from their scattered state.
• {mod}autopilot.exceptions will be used for more explicit exception handling.
• Code files were given 755 permissions to allow execution (as opposed to 644)
• The unused git submodules for pigpio and an old temperature sensor were removed.

Major updates

• {mod}autopilot.data is the major change in this release! See the module-level doc page for more details
  • autopilot.core.subject was moved to {mod}autopilot.data.subject and remains the main interface to autopilot data.
    It was effectively rewritten to accomodate the use of data models, and many of its public methods were made private to
    make its use more obvious.
    • The subject structure that determines which things are located where has been made into its own class {class}.Subject_Structure
    • Making new subjects is now done with the {meth}.Subject.new class method rather than from the __init__ method
    • The open_hdf method has been replaced with the _h5f private method that is a more explicit context manager allowing
      locking read/write access and nonlocking read access
    • The subject file now explicitly handles updates between different versions of the Subject class.
    • Subject will check if the protocol .json file has been updated from when it was assigned and automatically update to the new version,
      logging the change.
    • All trial data is now saved with a session_uuid unique identifier column that is an always-unique key in case of any
      overlapping session IDs from reassignments, etc.
    • Many attributes were replaced by properties that return data models:
      • {attr}.Subject.bio and {attr}.Subject.info -> {class}.Biography
      • {attr}.Subject.protocol -> {class}.Protocol_Status, which manages the current trial, session, step, etc. rather than being treated separately
  • {mod}autopilot.data.interfaces contains classes to relate abstract data models to different representations and formats
    • {mod}.interfaces.datajoint allows creating datajoint schema definitions from autopilot models using the
      companion datajoint-babel project
    • {mod}.interfaces.tables translates pydantic models to HDF5 files
  • {mod}autopilot.data.modeling has basic types for use in data models
  • {mod}autopilot.data.models has the models themselves
  • {mod}autopilot.data.units stub module for using explicit units instead of anonymous floats
• A new set of general model filling widgets - {mod}.widgets.input and {mod}.widgets.model - were created that will
  eventually replace much of the existing GUI which suffers from code duplication problems from minor variations between
  representations of parameters, etc.
• {class}~.loggers.Log, {class}.Log_Format, {class}.LogEntry, {class}.ParseError,
  {data}.MESSAGE_FORMATS, {data}.LOG_FORMATS were added to allow programmatic loading of logfiles, and {mod}.utils.log_parsers
  was added to allow recovery of structured data (eg. from logged trial data). Loggers now use the rich logging handler
  for much more readable logs in stderr.
• {class}.Jack_Sound classes now put all frames at once into the processing queue, rather than one at a time for much less
  variability in sound onset jitter. The {class}.JackClient class was updated to reflect that as well by adding a play_q_size
  argument that controls the size of the deque used to store frames to play (rather than pulling each from a Queue as was done before). {class}.JackClient
  also has an optional disable_gc (False by default) parameter that allows the garbage collector to be disabled
  in the process for further (unquantified) jitter reduction.

Minor updates

• The bandwidth test ({class}.Bandwidth_Test widget and {meth}.Pilot.l_bandwidth listen) was updated to allow selecting
  compression, use of a random array, or preserialized messages
• The {mod}.Terminal now closes cleanly (still with a warning of leaked semaphors) without hanging.
• External processes {func}.external.start_pigpiod and {func}.external.start_jackd first check if the process is already started
• {meth}.Digital_Out.set now has a result argument for faster setting that doesn't confirm the result when False
• {class}.Message s now provide clearer support for compression in serialization, and automatically detect and handle
  a compressed message when deserializing. This is also carried through to the {meth}.Net_Node.send method.
• {meth}.Station.l_kill was moved up to the top level {class}.Station class, rather than being something the Terminal station owns.
• The {mod}~.setup.run_script functions no longer uses the totally unreadable white on green ANSI code combo that I have no idea why i ever used in the first place.
• A picamera_legacy script was added for enabling the picamera on bullseye.
• The {class}.sounds.Gammatone sound now accepts a filter_kwargs dictionary that is passed on to the {class}.timeseries.Gammatone filter.
• The {class}.Task and {class}.Nafc classes use the new {class}.Trial_Data style data declarations.
• Two utility functions {func}.common.walk_dicts and {func}.common.flatten_dicts were added to make handling nested dictionaries a bit easier.

Bugfixes

• The Subject class would incorrectly overwrite data given a mismatch in system times between the Terminal and Pilot.
  The subject class should now ideally no longer overwrite anything ever.
• The subject class would drop trial data silently if it was not in the TrialData description. En route to making the table
  automatically expand to accomodate unexpected data, dropped data is now logged as an exception.
• The prefs manager handles being launched from within ipython and other processes better, but is still a bit buggy.
  Now it uses the check used internally in the multiprocessing module to see if a manager can be launched,
  and falls back to using a standard dictionary if not.
• jackd_source script uses correct https:// rather than git:// protocol.
• A _TASK_LIST was added to {mod}.utils.registry to support deprecated task specifications.

Regressions

• With additional checking for monotonic increases in trial_num and checks that prevent data overwriting,
  individual writes of trial data are now a bit slower, which should be optimized for when we complete the transition to
  uniform data models throughout the library.

Prefs

• PIGPIOD - bool - if True, start pigpiod on pilot start
• Stubs were created for converting the prefs to using data models, but they have not been filled yet.
• A AUTOPILOT_NO_PREFS_MANAGER environment variable now controls the use of a multiprocessing manager explicit. Documentation
  for environmental variables is forthcoming.
• A AUTOPILOT_WARN_DEFAULTS environment variable controls whether warnings should be printed for when a default pref value
  is retrieved, because that warning is a good one but can be really annoying.

Packaging & Dependencies

• Autopilot is now packaged with Poetry! This allows for fully deterministic installation with the poetry.lock file and
  updates from the old setuptools style of dependency specification.
• The source repository has moved from https://github.com/wehr-lab/autopilot to https://github.com...

v0.4.4 - Sound and Timing

Several parts to this update!

• See #146 for details about improvements to jackd sound timing! In short:
• Changed the way that continuous sounds work. Rather than cycling through an array, which was easy to drop, now pass a sound object that can generate its own samples on the fly using the hydration module.
• More accurate timing of sound ending callbacks. Before, the event would be called immediately on buffering the sounds into the jack ports, but that was systematically too early. Instead, use jack timing methods to account for delay from blocksize and n_periods to wait_until a certain delay to set() the event. See _wait_for_end

Other stuff:

New

• hydration module for creating and storing autopilot objects between processes and computers!
• @Introspect made and added to sound classes. Will be moved to root class. Allows storing the parameters given on instantiation.
• requires module for more explicit declarations of by-object dependencies to resolve lots of the little fragile checks throughout the package, as well as make it easier for plugins :)
• minor - added exceptions module, just stubs for now
• Made dummy sound class to just use sounds without needing a running sound server
• New transformations! The Prasad line fitting algorithm as Linefit_Prasad and ordering points in a line from, eg. edge detection in Order_Points

Improvements

• Only warn once for returning a default pref value, and make its own warning class so that it can be filtered.
• Cleaning up the base sound classes and moved them to their own module because sounds was very cumbersome and hard to reason about. Now use get_sound_class instead of declaring within the module.
• Made optional install packages as extras_require so now can install with pip install auto-pi-lot -E pilot rather than autodetecting based on architecture. Further improvements (moving to poetry) will be in v0.5.0

Bugfixes

• Correctly identify filenames in logging, before the last module name was treated as a suffix on the path and removed, and so only the most recent logger created would actually log to disk. Logging now works across threads and processes.
• Fall back to a non-multiprocessing-based prefs if for some reason we can't use a mp.Manager in the given context (eg. ipython) - Still need to figure out a way to not print the exception because it is thrown asynchronously.
• as much as i love it, the splash screen being absent for whatever reason shouldn't crash the program.
• Raise an exception when instantiating a picamera without having picamera installed, re: #142
• Raise ImportError when ffmpeg is not present and trying to use a videowriter class
• Use a deque rather than an infinitely growing list to store GPIO events.

Docs

• Documenting the scripts module a bit better.
• Lots more docs on jack_server

v0.4.3 - Gammatones and a few bugfixes

New Features

• timeseries.Gammatone filter and sounds.Gammatone filtered noise classes!
  Thank you scipy team for making this simple!

Minor Improvements

• 579ef1a - En route to implementing universal calibrations, load and save them in a specified place for each hardware object instead of the horrific olde way which was built into prefs for some reason
• prefs attempts to make directories if they don't exist
• plenty of new debugging flags!

Bugfixes

• a775723 - Sleep before graduating tasks, lateral fix until we rework the task initiation ritual
• 360062d - pad sounds with silence or continuous sounds if they aren't a full period length
• 6614c80 - Revert to old way of making chunks to make it work with both padded and unpadded sounds
• Import sounds module directly instead of referring from the package root in tests
• Terminal node pings pilots instead of an erroneous reference to a nonexistent Terminal.send method
• 47dd4c2 - Fix pinging by passing pilot id, and handle pressing start/stop button when subject not selected
• Fixed some GUI exceptions from trying to make blank lines in reassign window, improperly handling the Subject class.

v0.4.2

Bugfixes! primarily in importing the stim module! see the changelog:
https://docs.auto-pi-lot.com/en/main/changelog/v0.4.0.html

v0.4.1

• The :class:autopilot.setup.forms.HARDWARE_FORM would incorrectly use the class object itself
  rather than the class name in a few places which caused hardware names to incorrectly display and
  be impossible to add!
• Correctly handle module name in loggers when running interactively
• Use accelerometer calibration when computing :meth:~autopilot.hardware.i2c.I2C_9DOF.rotation
• Use :func:autopilot.get in :func:autopilot.transform.make_transform

v0.4.0 - Become Multifarious

See the changelog! https://docs.auto-pi-lot.com/en/latest/changelog/v0.4.0.html

v0.4.0 - Become Multifarious (August 3rd, 2021)

This release is primarily to introduce the new plugin system, the autopilot wiki, and their integration as a way of
starting the transformation of Autopilot into a tool with decentralized development and governance (as well as
make using the tool a whole lot easier and more powerful).

With humble thanks to Lucas Ott, Tillie Morris, Chris Rodgers <https://github.com/cxrodgers/>,
Arne Meyer <https://github.com/arnefmeyer> , Mikkel Roald-Arbøl <https://github.com/roaldarbol>_ ,
David Robbe <https://github.com/neurodavidus>_ ,
and an anonymous discussion board poster for being part of this release.

New Features

• Registries & Plugins <https://github.com/wehr-lab/autopilot/pull/109>_ - Autopilot now supports users writing their code
  outside of the library as plugins! To support this, a registry system was implemented throughout the program. Plugin objects
  can be developed as objects that inherit from the Autopilot object tree -- eg. implementing a GPIO object by subclassing
  :class:.hardware.gpio.GPIO , or a new task by subclassing :class:~.tasks.task.Task . This system is flexible enough
  to allow any lineage of objects to be included as a plugin -- stimuli, tasks, and so on -- and we will be working to
  expand registries to every object in Autopilot, including the ability for plugins to replace core modules to make
  Autopilot's flexibility verge on ludicrous. The basic syntax of the registry system is simple and doesn't require any
  additional logic beyond inheritance to be implemented on plugin objects -- autopilot.get('object_type', 'object_name')
  is the basic method, with a few aliases for specific object types like autopilot.get_hardware(). Also thanks to
  Arne Meyer <https://github.com/arnefmeyer>_ for submitting an early draft of the registry system and
  Mikkel Roald-Arbøl <https://github.com/roaldarbol>_ for raising the issue.
• At long last, the Autopilot Wiki is alive!!!! - https://wiki.auto-pi-lot.com/ - The wiki is the place for communal
  preservation of technical knowledge about using Autopilot, like hardware designs, build guides, parameter sets,
  and beyond! This isn't any ordinary wiki, though, we got ourselves a semantic wiki which augments traditional wikis
  with a rich system of human and computer-readable linked attributes: a particular type of page will have some set of attributes,
  like a page about a 3D printed part will have an associated .stl file, but rather than having these be in plaintext
  they are specified in a format that is queryable, extensible, and infinitely mutable. The vision for the wiki is much
  grander (but not speculative! very concrete!) than just a place to take notes, but is intended to blend the use of
  Autopilot as an experimental tool with body of knowledge that supports it. Autopilot can query the wiki with the wiki module
  like wiki.ask('[[Category:3D_CAD]]', 'Has STL') to get links to all .stl files for all 3D parts on the wiki. The integration
  between the two makes using and submitting information trivial, but also makes designing whole new types of community interfaces
  completely trivial. As a first pass, the Wiki will be the place to index plugins, the system for submitting them, querying them,
  and downloading them only took a few hours and few dozen lines of code to implement. The wiki is infinitely malleable -- that's the point --
  and I am very excited to see how people use it.
• Tests & Continuous Integration with Travis! We are on the board with having nonzero tests! The travis page is here: https://travis-ci.com/github/wehr-lab/autopilot
  and the coveralls page is here: https://coveralls.io/github/wehr-lab/autopilot . At the moment we have a whopping 27% coverage,
  but as we build out our testing suite we hope that it will become much easier for people to contribute to Autopilot and be
  confident that it works!
• New Hardware Objects
  • :class:.cameras.PiCamera - A fast interface to the PiCamera, wrapping the picamera library, and using tips from its developer to juice every bit of speed i could!
  • The I2C_9DOF object was massively improved to take better advantage of its onboard DSP and expose more of its i2c commands.
• New Transforms
  • :class:.timeseries.Kalman - adapted a Kalman filter from the wonderful filterpy package! it's in the new timeseries transform module
  • :class:.geometry.IMU_Orientation - IMU_Orientation performs a sensor fusion algorithm with the Kalman Filter class to combine gyroscope and accelerometer measurements into a better estimate of earth-centric roll and pitch.
    This is used by the IMU class, but is made independent so it can be used without an Autopilot hardware object/post-facto/etc.
  • :class:.timeseries.Filter_IIR - Filter_IIR implements scipy's IIR filter as a transform object.
  • :class:.timeseries.Integrate - Integrate adds successive numbers together (scaled by dt if requested). not much by itself, but when used with a kalman filter very useful :)
  • :class:.geometry.Rotate - use scipy to rotate a vector by some angle in x, y, and/or z
  • :class:.geometry.Spheroid - fit and transform 3d coordinates according to some spheroid - used in the IMU's accelerometer calibration method: given some target spheroid, and some deformed spheroid (eg. a miscalibrated accelerometer might have the x, y, or z axis scaled or offset) either explicitly set or estimated from a series of point measurements, transform future input given that transformation to correct for the deformed source spheroid.
• New Prefs
  • 'AUTOPLUGIN' - Attempt to import the contents of the plugin directory,
  • 'PLUGIN_DB' - filename to use for the .json plugin_db that keeps track of installed plugins',
  • 'PING_INTERVAL' - How many seconds should pilots wait in between pinging the Terminal?',
  • 'TERMINAL_SETTINGS_FN' - filename to store QSettings file for Terminal',
  • 'TERMINAL_WINSIZE_BEHAVIOR' - Strategy for resizing terminal window on opening',
  • 'TERMINAL_CUSTOM_SIZE' - Custom size for window, specified as [px from left, px from top, width, height]',

Major Improvements

• Stereo Sound (Thank you Chris Rodgers! <https://github.com/cxrodgers/>_) - #102
• Multihop messages & direct messaging - #99 - it is now possible to
  send multihop messages through multiple Station objects, as well as easier to send messages directly
  between net nodes. See the examples in the network tests section of the docs.
• Multiple Children (Thank you Chris Rodgers! <https://github.com/cxrodgers/>_) - #103 -
  the CHILDID field now accepts a list, allowing a Pilot to initialize child tasks on multiple children. (this syntax and
  the hierarchical nature of pilots and children will be deprecated as we refactor the networking modules into a general mesh system,
  but this is lovely to have for now :)
• Programmatic Setup - #33 - noninteractive setup of prefs and
  scripts by using autopilot.setup -f prefs.json -p PREFNAME=VALUE -s scriptname1 -s scriptname2
• Widget to stream video, en route to more widgets for direct GUI control of hardware objects connected to pilots
• Support python 3.8 and 3.9 essentially by not insisting that the spinnaker SDK be installable by all users (which at the time
  was only available for 3.7)

Minor Improvements

• Terminal can be opened maximized, or have its size and position set explicitly, preserve between launches (Thank you Chris Rodgers! <https://github.com/cxrodgers/>_) - #70
• Pilots will periodically ping the Terminal again, Terminal can manually ping Pilots that may have gone silent - #91
• Pilots share their prefs with the Terminal in their initial handshake - #91
• Reintroduce router ports for net-nodes to allow them to bind a port to receive messages - 35be5d6
• Listen methods are now optional for net_nodes
• Allowed the creation of dataless tasks - 628e1fb
• Allowed the creation of plotless tasks - 08d99d5
• The I2C_9DOF clas uses memoryviews rather than buffers for a small performance boost - 890f2c5
• Phasing out using Queue s in favor of collections.deque for applications that only need thread and not process safety because they
  are way faster and what we wanted in the first place anyway.
• New Scripts - i2c, picamera, env_terminal
• utils.NumpyEncoder and decoder to allow numpy arrays to be json serialized
• calibrations are now loaded by hardware objects themselves instead of the extraordinarily convoluted system in prefs -- though
  some zombie code still remains there.
• Net nodes know their ip now, but this is a lateral improvement pending a reworking of the networking modules.
• performance script now sets swappiness = 10 to discourage the use of swapfiles - see https://www.raspberrypi.org/forums/viewtopic.php?t=198765
• Setting a string in the deprecation field of a pref in _DEFAULTS prints it as a warning to start actually deprecating responsibly.
• Logging in more places like Subject creation, manipulation, protocol as...

v0.3.5 - Fixing pilot db access

#79

A very minor bugfix in response to #72 (reply in thread)

• Very minor one, fixes to the way :class:.Terminal accesses the pilot_db.json file to use :attr:.Terminal.pilots
  property that makes a new pilot_db.json file if one doesn't exist, but otherwise loads the one that is found in
  prefs.get('PILOT_DB')
• Reorganized :class:.Terminal source to group properties together & minor additions of type hinting
• Fixed some bad fallback behavior looking for files in old hardcoded default directories, eg. in the ye olde
  :func:.utils.get_pilotdb
• bump version to 0.3.5

Unifying prefs, logging, setup, and scripts

Improvements:

• Unify the creation of loggers!!!! See the docs ;) https://docs.auto-pi-lot.com/en/latest/autopilot.core.loggers.html : d55638f #38
• Unify prefs, including sensible defaults, refactoring of scripts into a reasonable format, multiprocess-safety, and just generally a big weight off my mind. Note that this is a breaking change to the way prefs are accessed. Previously one would do prefs.PREF_NAME, but that made it very difficult to provide default values or handle missing prefs. the new syntax is prefs.get('PREF_NAME') which returns defaults with a warning and None if the pref is not set: c40a212 #38
• completely clean up scripts, and together that opened the path to clean up setup as well. so all things configuration got a major promotion
• We're on the board with CI and automated testing with a positively massive 3% code coverage!!! 743bb8f
• new scripts to eg. create autopilot alias: 211919b

Bugfixes:

• cleanup scripts on object deletion: e821830 #41
• don't drop 'floats' from gui when we say we can use them...: 743bb8f
• pigpio scripts dont like floats: 9f939cd

Docs:

• Clarification of supported systems: ce0ddf7
• Solved an ancient sphinx riddle of how to get data objects/constants to pretty-print: ec6d5a7
• Clarify hardware prefs f3a7609
• what numbering system do we use: 6426724

Logging:

• catch pigpio script init exception: 3743f8a
• more of it idk b682d08

v0.3.3 - More independent hardware objects and bugfixes!

Bugfixes

• Fix layout in batch reassign from python 3 float division
• Cleaner close by catching KeyboardInterrupt in networking modules
• Fixing audioserver boot options -- if 'AUDIOSERVER' is set even if 'AUDIO' isn't set in prefs, should still start server. Not full fixed, need to make single plugin handler, single point of enabling/disabling optional services like audio server
• Fix conflict between polarity and pull in initializing pulls in pilot
• Catch tables.HDF5ExtError if local .h5 file corrupt in pilot
• For some reason 'fs' wasn't being replaced in the jackd string, reinstated.
• Fix comparison in LED_RGB that caused '0' to turn on full becuse 'value' was being checked for its truth value (0 is false) rather than checking if value is None.
• obj.next() to next(obj) in jackdserver

Improvements

• Better internal handling of pigpiod -- you're now able to import and use hardware modules without needing to explicitly start pigpiod!!
• Hopefully better killing of processes on exit, though still should work into unified process manager so don't need to reimplement everything (eg. as is done with launching pigpiod and jackd)
• Environment scripts have been split out into setup/scripts.py and you can now run them with python -m autopilot.setup.run_script (use --help to see how!)
• Informative error when setup is run with too narrow terminal: #23
• More loggers, but increased need to unify logger creation!!!

Cleanup

• remove unused imports in main __init__.py that made cyclical imports happen more frequently than necessary
• single-sourcing version number from __init__.py
• more cleanup of unnecessary meta and header stuff left from early days
• more debugging flags
• filter NaturalNameWarning from pytables
• quieter cleanups for hardware objects