2018.11.09

2018.11.09

Contributors: Andrew Champion, Chris Barnes, Tom Kazimiers, William Patton, Eric Trautman

Notes

• Python 3 is now required for the back-end. We recommend the use of Python 3.6.

• CATMAID's version information is now presented in a different form. It follows
  the pattern <base-version>[-dev]-<commit>. The <base-version> is baked
  into the source code on a release. The -dev part will only be present if
  CATMAID's dev branch is used for deployment. It won't be present for
  master branch based setups. The <commit> part is the 10 digit version of
  the Git commit ID. This version representation is now also consistent with
  what is display in Docker images. In the rare event that no commit information
  can be found, <commit> will fallback to "unknown". This version will now
  also logged during start-up of the back-end.

Features and enhancements

Connectivity matrix:

• The new checkbox labeled "Fractions" in the "Main" tab makes it now possible
  to display connector fraction instead of an absolute link number in each cell.
  The number of connections from one source row to a target column is divided by
  the number of total posynaptic connections that are made to the target
  skeleton (column). This makes columns better comparable to each other.

• The auto-connectivity matrix of a large set of skeletons can now be exported
  as CSV without displaying it using the "Auto-connectivity CSV" button. This
  makes it possible to export larger connectivity matrices in a usable format.

• The aggregation method for the connectivity count in groups can now be
  selected. Available are: sum (default), min, max and average.

3D viewer:

• Treenodes that are linked to connector nodes can now be scaled independently
  from other node handles using the "Link node scaling" option.

• Volume picking is now optional and disabled by default, i.e Shift + Click will
  go through volumes. To enable volume surface location selection, the
  "Pickable" checkbox needs to be checked.

• If Reconstruction Sampler domain shading or interval shading is used, a list
  of valid domains and intervals can now be specified in the "Shadings
  parameter" tab.

• The X/Y/Z rainbow coloring modes are now also available normalized to each
  individual skeleton.

• Add custom connector coloring. The pre and post colors can be adjusted in the
  "Shading parameters" tab.

• The X/Y/Z/ axes can now be displayed in the lower right corner using the
  "Axes" checkbox in the "View settings" tab.

Reconstruction sampler:

• Improve performance of interval length computation

• Sampled skeletons can now be split after the users confirm this is their
  intention. The split-off part of the skeleton will not contain any sampler
  information anymore. Intervals on the split-off part are removed, split-point
  crossing intervals are shortened and domain end points will be removed and
  recreated as needed.

• Sampled skeletons can now be merged into. All samplers that reference the
  merged-in fragment are deleted. If the merged fragment is merged outside
  of a domain, nothing special is happening---it is a regular merge. If the
  merge treenode is in a sampler domain, there are currently three options,
  "Branch", "Domain end" and "New domain": 1. Branch: add the new fragment to the
  skeleton without changing domain end nodes or intervals. This is only allowed
  if the merge target is not the start or end of an interval. 2. Domain end: add
  a new domain end node right where the merged in fragment starts. This keeps
  the new fragment isolated from the sampled domain. 3. New domain: a new
  domain is created for the merged in fragment. This also adds the domain end
  node from (2).

• Domain completion is now shown in percent along with interval coverage of the
  domain in "Interval" step.

• Merge decisions can now be limited when the 'merge' or 'merge-or-create' leaf
  handling mode is selected. This means a percentage can be specified which
  defined below which ratio of extra cable versus interval length the extra
  cable should be merged into the previous interval (if possible) rather than a
  new interval is created.

• Ignored lead segments can now be inspected in more detail using the
  "Uncovered domain parts" button the "Interval" step. This will open a dialog
  window with a histogram on all ignored leaf segments in the current domain.
  Clicking on individual bins will open a treenode table containing the
  respective start nodes of the ignored leaf segments. From this dialog, it is
  also possible to list all downstream/upstream partners linked to nodes in
  ignored segments of a domain.

• For samplers using the 'ignore' leaf handing mode, it is now possible to
  update this to 'short-interval' mode including the generation of missing
  intervals for existing domains. A visual confirmation dialog is shown. To use
  this, press the "Set short-interval leaf mode" button in the Domain tab.

• The Synapse tab now also shows all leaf nodes of an interval. This makes it
  easier to find the places where an interval needs to be continued.

Tracing layer:

• Tracign layer: cycle open end in reverse using Shift + Alt + R.

• Alt + Click now opens consistently the link type context menu, regardless of
  whether a treenode or connector node is currently selected.

• The layer options now allow to select a user who's tracing data won't be
  fetched from the server. The main motivation is to hide data imported by a
  dedicated import user by default and not even fetch it from the server.

• Similar to image data mirrors, it is now possible to configure read-only
  tracing data mirror servers from which the tracing data will load all data
  except for the active node, which will be read from the main server. This is
  particularly useful if connecting to the main server from a remote location.
  To make this work reliably, it is expected that physical replication is setup
  on the database level that mirrors the main server constantly. A separate
  CATMAID instance needs to be setup on the mirror server as well. To configure
  this, the "Read-only CATMAID mirrors" option in the settings widget can be
  used together with the "Read-only mirror index".

Neuron similarity:

• The new Neuron Similarity Widget makes it possible to compare neurons to each
  other, to neurons transformed based on landmarks as well as to arbitrary point
  clouds. Point clouds can for instance be created from light microscopy data.
  It creates a similarity ranking based on NBLAST. To open the widget, use Ctrl

  • Space or the Open Widget button and then search for "Neuron similarity".

• To compare two different objects, NBLAST will compare a query object pairwise
  with potential target objects. It iterates over each point of the query
  object, find the closest point in a target object and computes a score based
  on the distance of these points and their orientation to each other.

• This scoring is done based on a scoring matrix, which needs to be created
  before any comparisons can be made. Scoring matrices are typically reused and
  don't need to be recomputed every time. The "Configurations" tab allows to
  create new similarity matrices and lists existing ones. For a new scoring
  matrix, probabilities for distance and orientation are computed for both a set
  of of similar neurons and a representative sample of random neurons. Both are
  combined into a single matrix in which a value of zero makes a particular a
  pair of points equally likely to be random or to be a match. Values above zero
  make a match more likely. Computed similarity matrices can be visualized by
  clicking the "View" link in the Scoring column of the respective similarity
  configuration.

• With a similarity matrix computed, similarity queries ca be performed from the
  "Neuron similarity" tab. In its most basic form, this compares neurons to
  other neurons. It is also possible to select transformed neurons and point
  clouds as query type or target type in a search. This however requires
  additional setup (see below). Query and target skeletons can be selected by
  selecting a skeleton source for each. A similarity matrix has to be selected
  as well, but all other options have reasonable defaults. A click on "Compute
  similarity" queues a new similarity request, which is computed asynchronously.
  Once the task is complete its table entry will switch its status to
  "complete".

• Once completed, the similarity query results can be viewed by clicking on
  "View" in the "Scoring" column. This will open a new result window (or
  dialog, if selected in the "View" option), which shows the similarity ranking.

• To query with or against transformed skeletons, a landamark based "display
  transformation" has to be created. To do so, open the Landmark Widget, and
  create a transformation in its "Display" tab. Transformations created this way
  are selectable from the Similarity Widget, if "transformed skeletons" is
  selected for either query or target. Depending on available landmark groups,
  this could be for instance a skeleton transformation to its contralateral
  location.

• The Point cloud tab allows to import individual point clouds, along with an
  optional transformation and representative images. It also provides a list of
  all point clouds that are visible to the current user. A group visibility
  option during import allows to restrict visibility of imported point clouds to
  selected groups (which need to be added from the admin interface).

• The "Point cloud import" tab allows to import many point clouds at the same
  time, optionally transformed and with linked representative images.

Volumes:

• The general widget controls are now distributed across tabs.

• Annotations can now be used on volumes too. The "Annotate" button in the
  Volume Manager can be used to annotate all selected volumes.

• The "Add from file" button in the Volume Manager can now be used to import
  volumes from STL files.

• The "Skeleton innervations" tab allows to search for volumes that intersect
  with a set of query skeletons. A volume annotation can be specified to look
  only at volumes having this annotation. Optionally, exact result computation
  can be disabled to only compute skeleton/bounding-box intersections. This is
  slightly faster, but leads to false positives.

Docker:

• More CATMAID configuration options are now accessible through Docker
  environment cariables: CM_DEBUG, CM_FORCE_CONFIG_UPDATE, CM_WRITABLE_PATH,
  CM_NODE_LIMIT, CM_NODE_PROVIDERS, CM_SUBDIRECTORY, CM_CSRF_TRUSTED_ORIGINS.

• The Git commit from which a Docker image was built is now preserved and
  included in the CATMAID's version information.

CLI importer:

• If no user information except for IDs is present in the imported data, the
  importer will by default ask for a username and create new inactive users for
  those IDs (and update the referenced IDs). Alternatively, the --map-user-ids
  parameter can be specified, which will make the importer map referenced IDs to
  existing users. If an existing user with the respective ID is not available,
  the user is asked for a username and a new user will be created.

• With the help of a few additional progress bars, import progress can be better
  monitored.

• Database statistics are now automatically recomputed after an import, i.e.
  ANALYZE is run.

Miscellaneous:

• Each Tracing Layer is now listed as a skeleton source. All skeletons visible
  in their fields of view are made available to other widgets that way.

• Tracing tool: add icon button to toggle a node coloring mode in which each
  node is colored according to the length of their respective skeleton. The
  colors and cable length values when to use it can be set for three colors in
  the Settings Widget in the "Skeleton length coloring" section.

• The SWC exporter can now optionally mark a node as soma if it either is tagged
  as "soma", if it has a radius larger than a defined value or if it is the root
  node. All three conditions can be selected, and they will be applied in the
  order they are listed above. If selected, a soma tagged node will always take
  precedence.

• There is small copy-to-clipboard button left to the "URL to the view" link in
  the upper right corner of the user interface.

• The Ctrl modifier can now be used with - and + to animate zooming.

• New widget: stack info, which displays properties for stacks related to the
  active project.

• The node cache update management command catmaid_update_cache_tables can now
  update all caches configured in the NODE_PROVIDERS settings variable
  automatically when the --from-config option is provided.

• Admin: a projects/export JSON export of the visible project/stack structure
  can now be used directly in the Project/stack Importer by selection "JSON
  representation" as source and pasting the data into the text field.

• Neuron search: regular expressions are now optional for the neuron name.
  Unless the search string starts with '/', no regular expressions are used,
  but a regular case insensitive text search.

• Two new connector types are available: "tight junction" and "desmosome". Both
  can be created through the Alt + Click menu, are reciprocal and two links at
  one connector are allowed for each type.

• Connectivity widget: all available link types can no be selected to be
  displayed. Both the gapjunction and attachment checkboxes have been removed in
  favor of a more generic list select element.

• Tracing layer: the displayed tracing data can now be constrained to show only
  the N most recently edited skeletons. This settings adds to the skeletons
  selected by the N largest skeletons filter.

• WebGL layers are now preferred by default.

• Project statistics: a top ten of the largest neurons is now displayed.

• Admin: a user import view is now available to import users from other CATMAID
  instances. It requires superuser permissions on the remote instance.

• DB integrity check management command: volumes are now checked to make sure
  all faces are triangles. The --tracing [true|false] and --volumes [true|false]
  command line parameters now allow to explicitly test only some parts of the
  database. By default all is tested.

Bug fixes

• 3D viewer: loading a single node skeleton with smoothing enabled no longer
  causes an error.

• 3D viewer: nodes taged with 'uncertain' can be loaded again.

• 3D viewer: various rendering bugs for Reconstruction Sampler domains and
  intervals have been fixed.

• 3D viewer: stored node scaling settings are now properly restored.

• 3D viewer: TODO tag coloring doesn't override custom label colors anymore.

• 3D viewer: all synaptic site spheres are created and colored again.

• 3D viewer: camera won't flip anymore if it is upside down during animations.

• Tracing overlay: the border of the tracing window is now properly rendered.

• Tracing overlay: child node edition times are now correctly updated if the
  parent is deleted, which fixes occasional state matching errors.

• Tracing overlay: Shift + Click now works also with attachment connectors,
  should they be selected as default connector type in the settings.

• Reconstruction sampler: deleting samplers while other samplers on the same
  neuron refer to the same created boundary nodes no longer causes an error.

• Reconstruction sampler: cable length columns are now sorted numercially.

• Reconstruction sampler: all settings are now correctly reset when a sampler is
  selected or "New session" is pressed.

• Reconstruction sampler: a few corner cases for binary interval coloring have
  been fixed. Colors should now alternate in most cases.

• Connectivity matrix: synapse count based ordering works again.

• Connectivity widget: annotations on neurons (used for filtering) are now
  properly updated when they are changed in another part of CATMAID.

• Graph widget: edge color updates now trigger a redraw operation again.

• CLI importer: the ID sequence for the auth_user table is now properly reset
  after an import.

• CLI importer: missing treenode-connector links are now imported between
  connectors and placeholer nodes.

• CLI impoerter: the ID of reused objects is now proplery updated in imported
  data when --preserve-ids is used.

• CLI importer: skeleton summaries and edge tables aew correctly created again.

• CLI importer: unmapped imported users are now correctly saved.

• The error dialog prints now linen breaks and spaces correctly again, which
  improves its formatting.

API changes

Additions

• POST /{project_id}/skeletons/cable-length:
  The POST version of the already existing GET endpoint. It allows passing in
  more skeleton IDs for which to get the cable length.

• POST /{project_id}/skeletons/connectivity-counts:
  Allows to get the number connector links per relation type for each passed in
  skeleton. Also accepts GET parameters.

• POST /{project_id}/skeletons/connectivity_matrix/csv:
  Return a CSV file containing the connectivity matrix for the passed in rows
  and columns.

• POST /{project_id}/skeletons/sampler-count:
  Return the number of samplers linked to each skeleton in the passed in
  skeleton_ids list.

• GET /{project_id}/skeletons/{skeleton_id}/sampler-count:
  Return the number of samplers linked to this skeleton.

• GET /{project_id}/labels/detail:
  Returns a list of of label objects, each with a name field and an ID field.

• POST /{project_id}/volumes/import:
  Import volumes as STL files.

• GET /{project_id}/volumes/{volume_id}/export.{extension}:
  Export a particular volume. Currentl only exports AS STL are supported.

• GET /{projec_id}/stats/cable-length:
  Get the top N largest skeletons. N can be specified using the 'n_skeletons'
  parameter.

• POST /{project_id}/volumes/skeleton-innervations:
  Get a list of all volume/skeleton combinations that intersect. Skeletons are
  provided with a 'skeleton_ids' parameter. This can be constrained by volume
  annotation, cable length and node count.

• POST /{project_id}/volumes/:
  Previously only the GET method would be supported. With the introduction of an
  optional volume_ids parameter, POST can be used as well. This parameter can
  be used to constrain the returned set of volumes.

Modifications

• POST /{project_id}/skeletons/node-label:
  The new label_names parameter accepts a list of strings that an be used
  instead of or together with label IDs to get skeletons with nodes that have
  particular labels.

• GET /{project_id}/stacks/{stack_id}/info:
  Now includes the comment field for each stack.

• GET /{project_id}/connectors/:
  The partner data returned when the with_partners is set, includes now
  additionally the creator of the link and the creation and edition time (in
  epoch format).

• GET /user-list:
  Accepts a new boolean parameter with_passwords to export encrypted
  passwords, which in turn can be imported into other CATMAID instances. This
  requires the requesting user to be a superuser.

• GET /{project_id}/skeleton/{skeleton_id}/swc:
  A new optional parameter soma_markers can be supplied. It is a list of
  'tag:soma', 'radius:' and 'root'. These are conditions for when a node is
  marked as soma in the SWC export. The first matching condition in this order
  wins.

• GET /{project_id}/volumes/:
  The return format changed. Instead of a list of volume objects an object with
  a 'columns' field and a 'data' field are returned. The data fields contains a
  list of lists, with each inner list being a volume. The entries are described
  by the 'columns' field. Along with the already returned fields, annotations
  are now retuned as well.

• GET /{project_id}/volumes/:
  An optional volume_ids parameter can now be used and the POST method is
  supported as well (to transfer larger volume ID lists). This parameter can be
  used to constrain the returned set of volumes.

• POST /{project_id}/skeletons/connectivity:
  A new optional option 'link_types' allows to specify which connectivity types
  should be returned. By default 'incoming' and 'outgoing' is returned.
  Additionally, the following options are allowed: gapjunction, abutting,
  attachment, close_object. The return field names have also been adapted to
  these parameters and match them. Instead of 'gapjunctions' and 'attachments',
  the fields are now named 'gapjunction' and 'attachment'.

• GET /{project_id}/annotations/:
  Supports now an optional boolean parameter named simple, which returns only
  names and IDs and is therefore much faster.

Deprecations

None.

Removals

None.