Skip to content

Releases: catmaid/CATMAID

2021.12.21

21 Dec 13:01
2021.12.21
b74f51f
Compare
Choose a tag to compare

2021.12.21

Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan Gerhard, Sanja Jasek, Tom Kazimiers

Notes

  • The version requires PostgreSQL 12. If you also want to upgrade PostGIS,
    update PostGIS first and run ALTER EXTENSION postgis UPDATE; in every
    existing database in the cluster that should be upgraded. For docker-compose
    setups this database update is performed automatically if DB_UPDATE=true is
    set for the db container (watch the Docker output). CATMAID's documentation
    Docker has more information. If a replication setup is in use, the database
    configuration changes for Postgres 12. CATMAID's replication documentation
    explains what needs to be done. The next CATMAID release will require Postgres
    13, which is also supported by this version.

    In Postgres 12, JIT compilation is available to the planner by default. We
    found that being more conservative with the use of it helped a few common
    queries (like the tracing data field of view). We set the required minimum
    cost of JIT use to 1,000,000 in postgresql.conf::

    jit_above_cost = 1000000

  • If R extensions are used, make sure to use R 3.6. On Ubuntu this can be made
    available by first installing the official R PPA repository:

    sudo gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-key E084DAB9
    sudo gpg -a --export E084DAB9 | sudo apt-key add -
    echo "deb https://cloud.r-project.org/bin/linux/ubuntu xenial-cran35/" | sudo tee -a /etc/apt/sources.list

    And second update the package index and update the local R setup:

    sudo apt-get update
    sudo apt-get install r-base r-base-dev mesa-common-dev libglu1-mesa-dev
    libssl-dev libssh2-1-dev libcurl4-openssl-dev cmtk

    This also requires updating all installed R packages. In all likelihood this
    requires executing "manage.py catmaid_setup_nblast_environment".

  • The default downsample factors for a stack without explicit factors defined,
    changed slightly: if you relied on the automatic creation of downsample
    factors for image stacks, make sure the changed behavior still works for you
    or enforce the previous configuration with an explicit specification of
    downsample factors. Previously as many zoom levels as needed were created to
    map the image size in a quadratic fashion per level onto a 1024x1024px tile.
    Instead of hard-coding the 1024px, the minimum tile size defined in the mirror
    set is used. Without any mirrors, no extra zoom levels will be assumed.

  • The recommended python version is now 3.8, although support for 3.6 is maintained.
    The next CATMAID release will require at least Python 3.8 and will drop support for
    Python 3.6.

  • A virtualenv update is required.

Features and enhancements

Notes widget:

  • This new and simple widget can be used to take notes either only for the
    current project, shared across all projects or globally visible for all users.
    The last option is only available to admin users for writing. Use the "Open
    Widget" dialog to open it using the "Notes" keyword.

Publication widget:

  • This new widget makes management of publication related annotations easier. It
    lists all annotations that annotated with one of the publication annotations
    defined in the Settings Widget in the Publication section. Each of them
    represents a publication for the widget. The export options for each one of
    them can now be configured through the table.

Project management widget:

  • This new widget allows superusers and users with can_administer role in a
    project to edit user and group permissions for this project. This interface is
    easier to find and manage then the admin view and allows for more granular
    admin permissions. It is also useful to share own spaces (own copies of
    projects) with other users.

  • The widget allows to define for both user and groups the following
    permissions: can browse (read), can annotate (write), can administer, can
    import, can queue, compute taks, can write through API, can fork

  • Clicking any checkbox will store the value right away.

  • If a user has "can_administer" permissions in a project, they can edit the
    project title and project description (comment) in the second tab.

  • If a user has "can_administer" and "delete_project" permissions in a project,
    the project management widget allows to delete the current project. This is
    mainly useful for personal spaces, in which users typically have
    delete_project permissions.

  • The "User data permissions" tab allows to configure user level edit
    permissions (who is allowed to edit who's data).

  • The "Project token" tab allows project admins to view existing project tokens
    ("invitation codes") available for a project. For each generated token, the
    table allows to copy both the token itself as well as an invitation link using
    this token to the clipboard. If a user opens an invitation link, they will
    automatically apply the token. Anonymous users are asked to log in first.

  • As admin user in a project, it is also possible from to manually change
    skeleton IDs from the "ID updates" tab. This is only allowed if the new ID
    isn't used already. Also, if the new ID is a new maximum ID number, all new
    IDs will be automatically be larger than this ID. This functionality is also
    available as the back-end managment command catmaid_change_skeleton_id.

Skeleton ID change management command:

  • Just like the Project Management functionality explained in the last
    paragraph, the management command catmaid_change_skeleton_id can be used to
    change skeleton IDs (and optionally neuron IDs) from the command line.

Link widget and deep links:

  • This new widget allows users to create persistent deep links into the
    dataset using an alias, allowing for friendlier URLs of the form
    <catmaid-url>/<project-id>/links/<alias>. The stored parameters are the same
    as for regular deep links, with most of them being optional.

  • Like with fully parameterized URLs, features like layout, widget settings and
    loaded skeletons can be used.

  • Links can be marked private, making them only accessible (and listable) by the
    creator of the link.

  • An optional message can be stored with the link and is shown to the user when
    the links is opened.

  • The context menu for creating different types of links moved from entries
    in the Layouts menu to its own menu that is shown when hovering the mouse
    cursor over the "URL to this view" link. The first menu entry is new and opens
    a link creation dialog, similar to the Link Widget's 'Add link' dialog.

Neuron renaming:

  • There is now a new neuron renaming dialog, which allows a search/replace
    renaming workflow for many neurons at the same time. This can be launched
    through the "Rename" button in both the Neuron Search widget and the neuron
    lists of the Neuron Navigator (e.g. neurons with a particular annotation).

  • The dialog supports simple find/replace, but also regular expressions
    including capture groups and presents live previews.

Neuron search:

  • Neuron search: add option to show meta data columns (cable length, number of
    nodes, creation time and last edition time) for neurons.

  • Removal tools for annotations are now opt-in, i.e. not shown by default. To
    enable them during annotation display, click the "Show removal tools"
    checkbox.

  • The optional annotation column has now the option to only show annotations
    with a particular meta-annotation. This meta-annotation can be configured in
    the input field in the table column header.

  • The CSV export comes now with a header row to make clear the first column is a
    neuron ID.

  • The CSV export includes now the skeleton ID as well and will export metadata
    like cable length, if metadata display is enabled in the UI.

  • Expanded elements (e.g. annotations) are now sorted correctly within their
    expansion group. This also fixes the out-of-order expansion display.

  • Escape underscore and percent sign properly. They were treated as wildcards
    before.

Connectivity Matrix:

  • The new buttons "Save JSON" and "Open JSON" can be used to save the row and
    column skeletons, including groups and widget settings.

  • The loading speed has been improved significantly.

  • With the new "Link type" selector in the Main tab it is now possible to show
    the connectivity matrix for other link types (like desmosomes or gap
    junctions). The selected link type is also respected by the CSV export.

Layouts:

  • New layout menu item: "Copy URL to view with layout". This will create a URL
    to the current view like the link button on the right hand side of the second
    toolbar, but with a layout spec included. It will open all widgets in the new
    CATMAID like in the current one. This includes skeletons, their colors and
    widget configurations by default. A second menu entry with the "(no
    skeletons)" suffix allows to create link with only the layout and no
    skeletons. And a third menu entry with the suffix ("no widget settings)"
    allows to copy only the layout plus skeletons, without any widget
    configuration.

  • New layout menu item: "Copy current layout spec". This will copy the layout
    specification of the current view to the clipboard. This can be useful when
    constructing URLs to specific views.

  • Widgets in layouts can now optionally specify a list of skeletons to load.
    This is done by adding the skeletons parameter:

    "skeletons": [16035815, {"id": 16035701, "color": "rgb(0.5,1,0.2)"}]

    The list of skeletons can consist of skeleton IDs and/or optionally objects
    that can also specify a color in terms of common CSS definitions.

  • The Selection Table can now receive an options parameter from the layout
    initialization. If it contains a list of skeleton IDs like ...

Read more

2020.02.15

16 Feb 05:31
2020.02.15
4e39b4a
Compare
Choose a tag to compare

2020.02.15

Contributors: Chris Barnes, Andrew Champion, Stephan Gerhard, Pat Gunn, Tom Kazimiers

Notes

  • Python 3.5 is not supported anymore. Use Python 3.6, 3.7 or 3.8.

  • Postgres 11 and PostGIS 2.5 is required, Postgres 12 and PostGIS 3 is
    recommended. If Postgres needs to be updated, update directly to Postgres 12.
    If both needs to be updated, update PostGIS first and run
    ALTER EXTENSION postgis UPDATE; in every database. For docker-compose
    setups this database update is performed automatically. If a replication setup is in
    use, the database configuration changes for Postgres 12. CATMAID's replication
    documentation explains what needs to be done.

  • The next version of CATMAID will require Postgres 12 and PostGIS 3.

  • A virtualenv update is required. Before you start it, please remove some
    packages that are not needed anymore first:

    pip uninstall asgi-ipc asgi-rabbitmq

  • If ASGI was set up before, make sure to install channels_rabbitmq or
    channels_redis (depending on what yous use). The older asgi_rabbitmq and
    asgi_redis packages aren't supported anymore. This also requires an update of
    the CHANNELS_LAYERS in settings.py. The channels_rabbitmq documentation for an
    example: https://github.com/CJWorkbench/channels_rabbitmq/. This variable
    isn't defined by default anymore. Therefore you likely have to replace any
    CHANNELS_LAYERS[…] = … with something like CHANNELS_LAYERS = { … }. The
    new format is (use custom credentials on any production system!):

    CHANNEL_LAYERS = {
      "default": {
          "BACKEND": "channels_rabbitmq.core.RabbitmqChannelLayer",
          "CONFIG": {
              "host": "amqp://guest:[email protected]/asgi",
          },
      },
    }
    

    Also, if supervisord is in use to manage CATMAID process groups, the main
    Daphne process needs an adjustment: instead of calling daphne with the
    mysite.asgi:channel_layer parameter, use mysite.asgi:application. A
    complete supervisord entry would then look something like this:

    [program:catmaid-daphne]
    directory = /home/catmaid/catmaid/django/projects/
    command = /home/catmaid/catmaid/django/env/bin/daphne --unix-socket=/var/run/daphne/catmaid.sock mysite.asgi:application
    user = www-data
    stdout_logfile = /var/log/daphne/catmaid-server.log
    redirect_stderr = true
    

    As last step, the supervisor entry for the daphne worker process has to be
    removed. New types of workers can be added, but are not needed in most cases.
    The channels documentation has more information on this.

  • GDAL v2 or newer is now needed. If your Ubuntu version doesn't support this
    yet, there is an official PPA:

    sudo add-apt-repository ppa:ubuntugis/ppa`
    sudo apt-get update`
    sudo apt-get install gdal-bin
    
  • The management command gdal-bincatmaid_populate_summary_tables is now known as
    catmaid_refresh_node_statistics.

  • When enabling and disabling both history tracking and spatial update events on
    startup, advisory locks are now used. This should make the startup of parallel
    CATMAID workers more robust.

  • The application of migrations 88-91 and 98-99 might take a while to complete,
    because they rewrite potentially big database table (treenode_edge, treenode,
    class_instance, and more). Therefore, make also sure that there is enough
    space available at the database storage location (25% of database data
    directory should be plenty). If no replication is used, setting the following
    Postgres options can speed up the process: wal_level = minimal,
    archive_mode = off and max_wal_senders = 0.

    Due to this database update data consistency and correctness was improved,
    because additional foreign key relationships have been added that were missing
    before.

  • Both configuration.py.example and create_configuration.py support now the
    option catmaid_default_enabled_tools, which defines the list of front-end
    tools that are enabled by default for new users. The following options are
    supported: cropping, tagging, textlabel, tracing, ontology and roi. By default
    the tracing tool is now enabled for new users, because it seems in many setups
    users want to have this enabled by default.

  • Back-end errors result now in actual HTTP error status codes. Third-party
    clients need possibly some adjustments to handle API errors. In case of an
    error, status 400 is returned if an input data or parameter problem, 401 for
    permission problems and 500 otherwise.

  • On startup, CATMAID will now test if the output directory is accessible as
    well as if its expected folder layout is present. If expected subfolders are
    missing, they will now be created.

  • Python code is now linted by flake8. The config is fairly relaxed
    (see .travis.flake8), but contributions which do not conform with the selected
    rules will fail CI checks.

Features and enhancements

Permissions:

  • If data is imported by a user, e.g. from a remote CATMAID instance, this user
    has all permissions over the imported data. If data was imported or created by
    someone else, the regular rules apply: permissions are granted if the user is
    creator, superuser or has permission over the data's creator. If user A has
    permissions over user B's data, user A will also have permission to edit user
    B's imported skeletons. This is done in order to be able to maintain the
    original creator and editor data of the imported data, should it be available.

Node filters:

  • Binary split: a new toggle labeled "inclusive" allows to decide whether to
    include the split not in an upstream sub-arbor (default: true).

CLI importer:

  • Performance improvement: only the skeleton summary entries of the imported
    skeletons are now created.

  • Only update treenode edges and skeleton summary if treenodes were actually
    imported (rather than e.g. only volumes).

  • Add --auto-name-unknown-users option, to automatically generate names for
    users that are not available from an import, but referenced in it (by ID).

CLI exporter:

  • The new --annotation-annotation option allows to specify a meta-annotation
    that all exported annotations need to share transitively. If for instance all
    all top-level annotations that should be exported are annotated with
    "exportable" and "--annotation-annotation exportable" is used, only
    annotations from those annotation hierarchies marked as "exportable" will be
    exported. Without this option, all annotations are exported.

  • If users should be exported, also export users of volumes.

Project statistics:

  • Requests to the back-end our now performed in parallel, resulting in faster
    loading times.

  • It is now possible to configure how many largest neurons should be displayed
    as well as an optional name pattern.

  • The user/time action table will now also list imported nodes and imported
    cable length. These values are subtracted from the corresponding regular
    counters. If "Include imports" is unchecked, the same information that was
    displayed before this commit is shown, i.e. the regular node count and cable
    length field include the imported data.

Tracing tool:

  • The tool bar button to refresh caches has been replaced with a button to open
    a context menu titled "More tools". The context menu has an entry to refresh
    cashes and a list of tracing data in other projects.

  • Tracing data from other CATMAID instances (or other local projects) can now be
    added to the active stack viewer by selecting the respective remote project
    from the "More tools" menu item "Remote data".

  • To open a remote CATMAID instance at the same location as the current view,
    find the remote project in the "Remote data" section of the "More tools" menu,
    and click on "Open remote view".

  • The marker type used fore remote tracing data can be configured through the
    Settings Widget in the Tracing Overlay section, option "Remote node marker". A
    ring is used by default.

  • Merges will now by default respect the special annotation "stable". If a
    neuron is marked as such it will always win a merge and if both merge partners
    are marked "stable", the merge is canceled. This can be disabled by setting
    the project or instance setting "Stable join annotation" in the Tracing
    section of the Settings Widget to an empty string. The merge dialog will show
    a warning if stable merge annotations have any bearing on the merge direction.

  • Fast merge mode can now optionally be applied only to neurons/fragments with
    a particular name pattern (regular expression). This can be configured in the
    Fast Merge section of the Tracing section in the Settings Widget.

  • Alt + G will now select the closest node in the active skeleton. If no
    skeleton is selected, the globally closest node is select. Regular G works as
    before in the current section.

  • Skeleton splitting: if a skeleton sampler was in use, a split could so far
    only be successfully executed if the split node is within a sampler domain.
    This is fixed now and sampled skeletons can now be split upstream or
    downstream of a domain as well. If split upstream, the domain will be deleted,
    because it is part of the split-off fragment.

  • The default interaction mode (skeleton tracing, synapse dropping, select,
    move, import) can now be configured in the Tracing section of the Settings
    Widget. This interaction mode will be enabled when opening the Tracing tool.

Tracing overlay:

  • The remote mirror CATMAID instance configuration is now taken from the central
    configuration, which can be edited in the "Other CATMAID instances" section in
    the Settings Widget. The "Read-only mirror index" settings is kept but refers
    to list elements in the central configuration list.

-...

Read more

2019.06.20

21 Jun 04:12
2019.06.20
abe05d2
Compare
Choose a tag to compare

2019.06.20

Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan Gerhard, Pat Gunn, Tom Kazimiers, William Patton

Notes

  • A virtualenv update is required.

  • All *.pyc files in the folders django/applications/catmaid/migrations/, django/applications/performancetests/migrations/ and django/applications/pgcompat/migrations/ need to be deleted.

  • Python 3.7 is now supported.

  • Heads-up: The next CATMAID version will require Postgres 11, PostGIS 2.5 and Python 3.6 or 3.7.

  • Should migration 0057 fail due a permission error, the Postgres extension "pg_trgm" has to be installed manually into the CATMAID database using a Postgres superuser: sudo -u postgres psql -d <catmaid-db> -c 'CREATE EXTENSION pg_trgm;'

  • CATMAID's version information changes back to a plain git describe format. This results overall in a simpler setup and makes live easier for some third-party front-ends, because the commit counter is included again. The version display is now the same git describe format for both regular setups and Docker containers.

  • Tile loading is clamped to (0,0) again, i.e. there are no negative tile coordinates anymore by default. If you need them, set the respective stack's metadata field to {"clamp": false}.

  • To write to CATMAID through its API using an API token, users need to have now dedicated "API write" permission, called "Can annotate project using API token" in the admin UI. To restore the previous behavior (regular annotate permission allows API write access) the settings.py variable REQUIRE_EXTRA_TOKEN_PERMISSIONS can be set to False. This is done as a safety measure to prevent accidental changes through automation.

  • If R based NBLAST is used, make sure to execute to update all dependencies: manage.py catmaid_setup_nblast_environment.

  • The main documentation on catmaid.org has now a place for widget specific documentation as well. Only a few widgets have been updated yet, but more will follow.

Features and enhancements

Tracing tool:

  • Add a new Tracing Tool icon button to compute the distance between two nodes on a skeleton. Respects virtual nodes.

  • The globally nearest node can now be selected and brought into view using Alt + G, as opposed to selecting the nearest node in the current section using the G key without modifier.

  • Using the H shortcut now without an active skeleton, the most recently edited node of the current user (Alt: anyone) in the last active skeleton will be selected. Using Shift will look in all skeletons.

  • Using Shift + Alt + Click with a connector selected, will will now create a presynaptic node. This is consistent with the already existing Shift + Alt + Click behavior with a treenode selected, which creates a presynaptic connector.

Graph widget:

  • GraphML files can now be imported, positions and colors are respected. This is useful if layouting is done in e.g. Gephi and coloring should be done in CATMAID. The help page explains a possible workflow for this.

  • The button "Group equally colored" in the "Main" tab will group all skeletons with the same color into a single group. The user is asked for group names for each color.

3D viewer:

  • A new synapse coloring mode has been added: polyadicity. The number of partner nodes for each connector is color coded (for synaptic connectors, this is the number of postsynapses). The colors and ranges can be configured through the "Polyadicity colors" button in the "Shading parameters" tab. This is basically a configurable version of an absolute "N with partner" coloring.

  • Branches with leaf nodes tagged "not a branch" can now be collapsed using the 'Collapse "not a branch"' checkbox in the Skeleton Filters tab.

  • The visibility of radius information can now be controlled using the "Show radius" checkbox in the "Views settings" tab.

  • History animations can now be exported in full length without requiring to guess the number of frames for the export. The animation export dialog will show an additional checkbox ("Complete history") if a history animation should be exported. If complete history is enabled, CATMAID will export the complete history of the exported skeletons.

  • Animations can now be exported as stream directly to a file, which allows for much larger exports (32GB maximum at the moment).

  • Fractional rotations are now allowed in the animation export.

  • The default time per rotation in the animation export is set to 15 seconds now, slowing down the default by a factor of 3, which makes it easier to look at.

  • Stack Z slices can now be animated. Configurable are the change frequency and the change step in terms of sections. This is available for animation exports as well.

  • Stack Z slices can now be thresholded to replace a background color with another color. If enabled and the sum of all channels is in a configurable range [a,b] it will be replaced with another color.

  • The rotation time for animations can now specified in seconds rather than angular distance.

  • The background color is now fully adjustable through the "backgorund" button in the "View settings" tab.

  • Basic support for Virtual Reality is now available on Windows platforms using a Mixed Reality / SteamVR.3D setup and Firefox >= v65. To enable, check the VR checkbox in the "View" tab and click the "Enter" button right next to it.

Skeleton history widget:

  • A basic view of the change of a set of skeleton IDs over time based on all nodes that are part of a given skeleton ID or that have been in the past.

  • Skeleton history can also be used with past skeleton IDs to see into what skeleton they changed (if any).

  • All past and present treenodes with a passed in skeleton ID are tracked through the complete history and their path of skeleton ID changes is recorded along with the number of treenodes following a given skeleton path.

  • The widget shows a graph from origin skeletons to the final skeleton IDs in every available path, summing the treenode counts for each contributing path.

  • Existing skeletons are colored in yellow, past skeletons are colored in cyan. Selected skeletons are colored green.

  • Ctrl+Click on skeleton will select it and go to the closest location in it. Shift+Click allows selecting multiple skeletons. All selected skeletons are available through the Skeleton Source interface.

Node and skeleton filters:

  • Filter rules support now an "invert" option during creation, which allows to create filters that include everything but whatever is matched by a particular filter strategy. This can be useful e.g. during neuron review to only look at segments that have been created by people other than oneself or connectivity everywhere excluding a particular compartment.

Measurement table:

  • Node filters are now supported. Like in other widgets, the respective panel can be opened through the funnel icon in the widget title bar. Measurements are displayed for all independent fragments of a skeleton after a set of node filter rules has been applied.

  • With the help of the "Sum fragments" toggle all fragments that result from a node filter application can be aggregated into one row by summing individual values.

Remote CATMAID instances

  • Some tools in CATMAID gained support to communicate with other CATMAID instances, e.g. the Landmark Widget (see below). To enable this functionality, remote CATMAID instances can now be added to the user settings.

  • The Settings Widget contains now a section labeled "Other CATMAID instances". It allows to manage a list of other CATMAID servers based on their URL, an API key and optional HTTP authentication. This information is stored under an alias.

Landmarks:

  • Clicking on the name of a landmark group in the Landmarks tab will now open the group member edit dialog that was previously accessible by clicking any landmark name. Clicking a landmark name will now show a new dialog, which allows to specify of which landmark groups the clicked landmark is a member of.

  • The color of each active display transformation can be established separately in the respective table row. Color changes are now also visible in the 3D Viewer.

  • It is now possible to load skeletons and landmarks from other CATMAID instances. Point matches are done on the basis of matching landmark names. To enable the UI for this, check the "Source other projects" checkbox in the Display tab.

  • With the "other projects" UI enabled and a remote CATMAID instance configured in the Setting Widget (see above), it is now possible to select a remote CATMAID instance from the "Source remote" dropdown menu. Alternatively, the local instance can be selected if another project from the same instance should be used.

  • Next the source project in the selected instance needs to be specified. This list is updated when a different remote instance is selected.

  • Skeletons from a remote instance are collected through annotations. The respective annotation has to be entered in the "Source skeleton annotation". With the help of the "Preview" button it is possible to load the matching skeletons from their remote CATMAID to inspect if the correct ones are selected.

  • As a last step for the remote data configuration, the source landmark group has to be defined. This list is updated if the source project changes. Landmarks from this group are mapped to the selected target group. The matching is done by name, i.e. no landmarks can have the same names in a group.

  • Adding such a transformation adds it to the list at the bottom of the widget, just like with regular transformations and they can be used in the same way. The can be shown in 3D Viewers, superimposed on the Tracing Layer and used in NBLAST queries from the Neuron Similarity Widget (see below).

  • The new checkbox 'Multiple mappings' displays additional user interface elements...

Read more

2018.11.09

09 Nov 07:30
2018.11.09
0e2b5fe
Compare
Choose a tag to compare

2018.11.09

Contributors: Andrew Champion, Chris Barnes, Tom Kazimiers, William Patton, Eric Trautman

Notes

  • Python 3 is now required for the back-end. We recommend the use of Python 3.6.

  • CATMAID's version information is now presented in a different form. It follows
    the pattern <base-version>[-dev]-<commit>. The <base-version> is baked
    into the source code on a release. The -dev part will only be present if
    CATMAID's dev branch is used for deployment. It won't be present for
    master branch based setups. The <commit> part is the 10 digit version of
    the Git commit ID. This version representation is now also consistent with
    what is display in Docker images. In the rare event that no commit information
    can be found, <commit> will fallback to "unknown". This version will now
    also logged during start-up of the back-end.

Features and enhancements

Connectivity matrix:

  • The new checkbox labeled "Fractions" in the "Main" tab makes it now possible
    to display connector fraction instead of an absolute link number in each cell.
    The number of connections from one source row to a target column is divided by
    the number of total posynaptic connections that are made to the target
    skeleton (column). This makes columns better comparable to each other.

  • The auto-connectivity matrix of a large set of skeletons can now be exported
    as CSV without displaying it using the "Auto-connectivity CSV" button. This
    makes it possible to export larger connectivity matrices in a usable format.

  • The aggregation method for the connectivity count in groups can now be
    selected. Available are: sum (default), min, max and average.

3D viewer:

  • Treenodes that are linked to connector nodes can now be scaled independently
    from other node handles using the "Link node scaling" option.

  • Volume picking is now optional and disabled by default, i.e Shift + Click will
    go through volumes. To enable volume surface location selection, the
    "Pickable" checkbox needs to be checked.

  • If Reconstruction Sampler domain shading or interval shading is used, a list
    of valid domains and intervals can now be specified in the "Shadings
    parameter" tab.

  • The X/Y/Z rainbow coloring modes are now also available normalized to each
    individual skeleton.

  • Add custom connector coloring. The pre and post colors can be adjusted in the
    "Shading parameters" tab.

  • The X/Y/Z/ axes can now be displayed in the lower right corner using the
    "Axes" checkbox in the "View settings" tab.

Reconstruction sampler:

  • Improve performance of interval length computation

  • Sampled skeletons can now be split after the users confirm this is their
    intention. The split-off part of the skeleton will not contain any sampler
    information anymore. Intervals on the split-off part are removed, split-point
    crossing intervals are shortened and domain end points will be removed and
    recreated as needed.

  • Sampled skeletons can now be merged into. All samplers that reference the
    merged-in fragment are deleted. If the merged fragment is merged outside
    of a domain, nothing special is happening---it is a regular merge. If the
    merge treenode is in a sampler domain, there are currently three options,
    "Branch", "Domain end" and "New domain": 1. Branch: add the new fragment to the
    skeleton without changing domain end nodes or intervals. This is only allowed
    if the merge target is not the start or end of an interval. 2. Domain end: add
    a new domain end node right where the merged in fragment starts. This keeps
    the new fragment isolated from the sampled domain. 3. New domain: a new
    domain is created for the merged in fragment. This also adds the domain end
    node from (2).

  • Domain completion is now shown in percent along with interval coverage of the
    domain in "Interval" step.

  • Merge decisions can now be limited when the 'merge' or 'merge-or-create' leaf
    handling mode is selected. This means a percentage can be specified which
    defined below which ratio of extra cable versus interval length the extra
    cable should be merged into the previous interval (if possible) rather than a
    new interval is created.

  • Ignored lead segments can now be inspected in more detail using the
    "Uncovered domain parts" button the "Interval" step. This will open a dialog
    window with a histogram on all ignored leaf segments in the current domain.
    Clicking on individual bins will open a treenode table containing the
    respective start nodes of the ignored leaf segments. From this dialog, it is
    also possible to list all downstream/upstream partners linked to nodes in
    ignored segments of a domain.

  • For samplers using the 'ignore' leaf handing mode, it is now possible to
    update this to 'short-interval' mode including the generation of missing
    intervals for existing domains. A visual confirmation dialog is shown. To use
    this, press the "Set short-interval leaf mode" button in the Domain tab.

  • The Synapse tab now also shows all leaf nodes of an interval. This makes it
    easier to find the places where an interval needs to be continued.

Tracing layer:

  • Tracign layer: cycle open end in reverse using Shift + Alt + R.

  • Alt + Click now opens consistently the link type context menu, regardless of
    whether a treenode or connector node is currently selected.

  • The layer options now allow to select a user who's tracing data won't be
    fetched from the server. The main motivation is to hide data imported by a
    dedicated import user by default and not even fetch it from the server.

  • Similar to image data mirrors, it is now possible to configure read-only
    tracing data mirror servers from which the tracing data will load all data
    except for the active node, which will be read from the main server. This is
    particularly useful if connecting to the main server from a remote location.
    To make this work reliably, it is expected that physical replication is setup
    on the database level that mirrors the main server constantly. A separate
    CATMAID instance needs to be setup on the mirror server as well. To configure
    this, the "Read-only CATMAID mirrors" option in the settings widget can be
    used together with the "Read-only mirror index".

Neuron similarity:

  • The new Neuron Similarity Widget makes it possible to compare neurons to each
    other, to neurons transformed based on landmarks as well as to arbitrary point
    clouds. Point clouds can for instance be created from light microscopy data.
    It creates a similarity ranking based on NBLAST. To open the widget, use Ctrl

    • Space or the Open Widget button and then search for "Neuron similarity".
  • To compare two different objects, NBLAST will compare a query object pairwise
    with potential target objects. It iterates over each point of the query
    object, find the closest point in a target object and computes a score based
    on the distance of these points and their orientation to each other.

  • This scoring is done based on a scoring matrix, which needs to be created
    before any comparisons can be made. Scoring matrices are typically reused and
    don't need to be recomputed every time. The "Configurations" tab allows to
    create new similarity matrices and lists existing ones. For a new scoring
    matrix, probabilities for distance and orientation are computed for both a set
    of of similar neurons and a representative sample of random neurons. Both are
    combined into a single matrix in which a value of zero makes a particular a
    pair of points equally likely to be random or to be a match. Values above zero
    make a match more likely. Computed similarity matrices can be visualized by
    clicking the "View" link in the Scoring column of the respective similarity
    configuration.

  • With a similarity matrix computed, similarity queries ca be performed from the
    "Neuron similarity" tab. In its most basic form, this compares neurons to
    other neurons. It is also possible to select transformed neurons and point
    clouds as query type or target type in a search. This however requires
    additional setup (see below). Query and target skeletons can be selected by
    selecting a skeleton source for each. A similarity matrix has to be selected
    as well, but all other options have reasonable defaults. A click on "Compute
    similarity" queues a new similarity request, which is computed asynchronously.
    Once the task is complete its table entry will switch its status to
    "complete".

  • Once completed, the similarity query results can be viewed by clicking on
    "View" in the "Scoring" column. This will open a new result window (or
    dialog, if selected in the "View" option), which shows the similarity ranking.

  • To query with or against transformed skeletons, a landamark based "display
    transformation" has to be created. To do so, open the Landmark Widget, and
    create a transformation in its "Display" tab. Transformations created this way
    are selectable from the Similarity Widget, if "transformed skeletons" is
    selected for either query or target. Depending on available landmark groups,
    this could be for instance a skeleton transformation to its contralateral
    location.

  • The Point cloud tab allows to import individual point clouds, along with an
    optional transformation and representative images. It also provides a list of
    all point clouds that are visible to the current user. A group visibility
    option during import allows to restrict visibility of imported point clouds to
    selected groups (which need to be added from the admin interface).

  • The "Point cloud import" tab allows to import many point clouds at the same
    time, optionally transformed and with linked representative images.

Volumes:

  • The general widget controls are now distributed across tabs.
    ...
Read more

2018.07.19

19 Jul 04:39
2018.07.19
16afda4
Compare
Choose a tag to compare

2018.07.19

Contributors: Albert Cardona, Andrew Champion, Pat Gunn, Tom Kazimiers, Will Patton, Eric Trautman

Notes

  • Both the standalone Docker image and the Docker-compose setup can now be updated after a Postgres version change. This makes it possible again to use CATMAID versions after 2018.02.16 with Docker. The documentation has more information.

  • This is the last CATMAID version with support for Python 2.7. Starting from next version, only Python 3 will be supported.

Features and enhancements

Volume widget:

  • Add a "List connectors" link to each volume, to show all connectors in a volume bounding box.

  • The new "Min skeleton nodes" and "Min skeleton length" options allow to further constrain volume based skeleton selections ("List skeletons").

  • The Connector List widget that is shown when clicking on "List connectors", now supports connector filtering. The volume of the link's row is now automatically set as filter in the new Connector List widget. This means connector links in this table are now shown only if they intersect exactly with the volume (and not only with the bounding box like before).

  • State saving is now supported.

3D Viewer:

  • The line width of skeletons can be adjusted again on platforms other than Linux. This Requires "Volumetric lines" in the "View settings" tab to be enabled (it is by default). Unchecking this option brings back the previous line rendering behavior.

  • The new "Focus skeleton" button in the "View" tab will look at the active skeleton's center of mass from the current camea location.

  • Volumes can now be smoothed by subdivision. The volume option panel available from the View Settings tab now contains a "Subdivide" checkbox.

  • A volume's bounding box can now be displayed using the "BB" checkbox that is available for visible volumes in the volume option panel.

  • New coloring option: X/Y/Z rainbow lookup table coloring for active stack dimensions.

  • The active node respects now a node radius by default and is scaled to 1.5x its size. This behavior can be disabled using the "Radius adaptive active node" checkbox in the "View settings" tab.

  • Catalog export: use global neuron name for sorting and display by default.

  • Catalog export: support for multiple neurons per panel has been added (separate from pinned neurons). The export dialog contains now a "Skeletons per panel" input field. Essentially, the displayed skeletons can bow be iterated in batches.

  • Catalog export: individual skeleton panels can now also be exported as PNG instead of SVG, which reduces the file size, export time, parsing time. Plus it allows for an exact copy of what is shown in the 3D Viewer.

  • Catalog export: in orthographic mode it is now possible to export a scale bar on either none, the first or all exported panels.

  • If the estimated size of the tiles to load for a Z plane exceeds 100 MB, users are asked for confirmation.

Connectivity matrix:

  • Connector node filters can now be applied using the funnel icon in the widget title bar.

  • The new "Groups" tab allows to group rows, columns or both by their displayed name.

  • State saving is now supported.

Reconstruction sampler:

  • Different leaf handling strategies are now available to be selected for a sampler. The behavior so far (and current default) is to just ignore leaf segments that are shorter than the interval length minus the error margin. Alternatively, it is now possible to merge the leaf segment into the last interval, to create new shorter intervals for the leaf segments or, combining both, it is possible to try to merge it into the last interval and if that's not possible (e.g. on a small twig with no previous interval on the same segment), then create a new short interval. This option is available in the Sampler tab.

  • Both the domain table and the interval table now show the cable length of each domain and interval in nanometers, respectively. Additionally, the interval tab also show the aggregated cable length of all completed intervals.

Tracing layer:

  • The new option "Update tracing data while panning" allows to configure weather the tracing data on the layer will be updated when the view is panned around.

  • A set of new options allows now to configure a "tracing window", which will restrict tracing data loading by allowing it only in view centered rectangle. Width and height can be configured independently. This is useful for remote review and tracing.

  • A new option to show only the N largest skeletons in a field of view is now available for the layer settings (and the API).

Miscellaneous:

  • The behavior of the Ctrl modifier on section navigation with , and . can now be inverted using the "Animate section change by default" option in the Settings Widget.

  • Connectivity widget: annotations can now be used for additional filtering per partner table.

  • Split/merge dialog: the node count of the individual parts is now shown when hovering over their cable length information.

  • Review widget: the user who created the last node of each segment is now displayed in the review table. This allows to focus review on segments not created by oneself.

  • A skeleton cable length limit can be set so that a warning is displayed if a change to the skeleton morphology results in a cable length larger than the limit. This is available in the Warnings section of the Settings Widget.

  • Node filters: if a neuron name is provided for a rule, the rule is now valid for all neurons with neuron names that include the provided name and not only exact matches.

  • Neuron name display: neighboring duplicate name components are now removed by the default. This setting can be adjusted from the Annotation sections in the Settings Widget.

  • Boolean parameters for API endpoints are no case-insensitive, allowing the use of regular boolean values in requests from Python.

  • CLI exporter: the new --excluded-annotation parameter can be used to exclude neurons from the export based on annotations.

  • CLI exporter: placeholder nodes are now exported as completely new skeletons that are not linked to their original skeleton, unless it is part of the set of exported skeletons or the --original-placeholder-context flag is provided.

Bug fixes

  • Measurements table: column headers in CSV export are now quoted.

  • Export management command: the default output filename can be used again.

  • Export management command: class instances and links of skeletons and neuronsa are now exported alongside treenodes.

  • Neuron name service: missing naming components don't lead to removal of all whitespace between neighbors anymore.

  • Initial skeleton coloring of merge dialog when merging from smaller into larger skeletons is fixed.

  • 3D viewer: refreshing the active skeleton does not refresh all skeletons anymore.

  • 3D viewer: connector restrictions like "show only shared connectors" now respect the pre/post visibility toggles in the Selection Table.

  • 3D viewer: the initial text scaling for label text is now correctly set again.

  • Reconstruction sampler: during interval creation preview, only intervals from the currently active domain are now shown.

  • Graph Widget: when subscribed to other widgets, their skeletons are not removed anymore from the Graph Widget when the other widget is closed.

  • Neuron Search: annotation data range can be used again.

  • The Strahler number computation no correctly increases the Strahler number when two more children have the local maximum number rather than requiring all children to share the same number.

  • Radius editing: using undo (Ctrl + Z) after editing the radius of a node works message.

API changes

Additions

  • POST /{project_id}/treenodes/compact-detail:
    Retrieve treenode information for multiple nodes in the format of regular node
    queries, accepts a parameter "treenode_ids".

  • GET /{project_id}/treenodes/{treenode_id}/compact-detail:
    Retrieve treenode information for a single node.

  • DELETE /{project_id}/landmarks/{landmark_id}/groups/{group_id}/:
    Delete landmark location links to the same locations from a landmark and
    group.

  • PUT /{project_id}/landmarks/groups/links/:
    Add new links between groups, e.g. "adjacent_to".

  • DELETE /{project_id}/landmarks/groups/links/{link_id}/:
    Delete a single landmark group link.

  • GET /{project_id}/landmarks/groups/{landmarkgroup_id}/transitively-linked:
    Get a list of landmark groups that are linked to the referenced group using a
    passed in relation, respects reciprocal relations.

  • POST /{project_id}/landmarks/groups/materialize:
    Create pairs of landmark groups along with their landmarks based on a simple
    description.

  • GET /{project_id}/skeletons/cable-length:
    Get the cable length for multiple skeletons using the skeleton_ids parameter.

  • GET /{project_id}/skeletons/{skeleton_id}/cable-length:
    Get the cable length for a single skeleton.

  • GET /{project_id}/skeletons/in-bounding-box:
    Get IDs of all skeltons that intersect with the passed in bounding box.

Modifications

  • POST /{project_id}/annotations/query-targets:
    Accepts now a "annotation_reference" parameter which can either be 'id'
    (default) or 'name'. If it is set to 'name', all annotation references in
    annotated_with, not_annotated_with and sub_annotated_with are interpreted as
    annotation names instead of IDs.

  • POST /{project_id}/skeleton/split:
    Returns now also the split location as fields x, y and z.

  • POST|GET /{project_id}/node/list offers a new optional parameter
    "with_relation_map", whi...

Read more

2018.04.15

16 Apr 04:39
2018.04.15
477b88f
Compare
Choose a tag to compare

2018.04.15

Contributors: Albert Cardona, Andrew Champion, Chris Barnes, Rob Court, Tom Kazimiers

A recording of a presentation discussing the changes in this release can be found here.

Notes

  • Requires a virtualenv update.

  • A new management command "catmaid_find_node_provider_config" is available,
    which can be used to compare different node providers on existing data, which
    is useful to configure the NODE_PROVIDERS setting.

  • The Docker images now support the options CM_HOST and CM_PORT to configure
    where uwsgi is listening.

  • When using the Javascript console, node positions returned from tracing
    overlays are now in project space coordinates (physical nm), not stack space
    voxel coordinates. Transformation to project space is no longer necessary.

  • Postgres 10+ is now required.

Features and enhancements

Layouts:

  • A new "Layouts" menu is shown in the top bar when a project is opened. It
    allows to save the current window layout under a name, it provides an option
    to close all widgets and will show all available layouts as menu entries.

  • Layouts store window arrangement, window sizes, tabs and subscriptions.

  • Saved layouts can also be manually configured from the "Custom layouts"
    setting in the Settings Widget.

  • Tabbed windows are now supported in layout specs by using "t([a, b, c])" where
    a, b, c or any other number of elements can be children of the tabbed window.

Landmarks:

  • Support transformation of nodes that cross space between landmark groups and
    even reach into target groups. This allows to e.g. transform skeletons that
    cross the midline. Doing this is enabled by default, but can be disabled
    through the "Interpolate between groups" option.

  • Virtual transformed skeletons are now also shown on a separate layer in all
    open Stack Viewers. Nodes of those skeletons can currently not selected. To
    disable the Landmark Layer, uncheck the "Show landmark layers" checkbox in the
    Display tab's button panel.

  • The new "Edit landmark" tab provides a simpler interface to add new landmark
    locations to landmark groups. If the option "Update existing landmark
    locations" is selected, new landmark locations will replace existing ones
    shared between the provided landmark and the selected group. The lower
    section of this tab allows to edit links between landmark groups, which can
    be used for rule based display transformations.

  • As an alternative to selecting a target landmark group explicitly to create a
    display transformation, it is now possible to instead select a target relation
    in the Display tab. Doing so will automatically create all display
    transformations from the source group to all landmark groups transitively
    linked to the source group using the selected relation. The reciprocity of
    relations is respected.

  • The new "Create groups" tab provides an option to create landmark groups along
    with required landmarks from the bounding boxes of two volumes. This allow to
    quickly create simple landmark group mappings.

  • The color and extra scaling for nodes on Landmark Layers can be adjusted from
    the widget.

  • All 3D Viewers are now enabled by default as transformation display target.

  • State saving is now supported.

3D Viewer:

  • A scale bar can now be displayed when in orthographic mode. The scale bar
    can disabled via a checkbox in the View Settings tab.

  • PNG and SVG exports offer now a filename input field.

  • Loaded volumes are now stored along with their styling in the widget state.

  • Landmark groups are now show with landmark name labels. This can be disabled
    from the landmark menu.

  • Text scaling can be adjusted from the View tab.

  • The width and height of animation exports are now restricted to even numbers.
    This is required by the H264 codec we refer to in our documentation.

Graph widget:

  • The old "Graph" tab was split into two: "Nodes" and "Edges", each with the
    corresponding functionality.

  • New feature: color edges with the same color as the source node, the target
    node, or the general color specified in the "Properties" menu (from the "Main"
    tab). See the "Edges" tab.

  • New feature: change the arrow shape to a circle, diamond, tee, etc. The new
    "Set" button in the "Edges" tab applies the change to selected nodes.

  • New feature: the new "Selections" tab can record sets of nodes, stored as a
    named selection. Then these can be selected or deselected. The "Select all"
    button selects all nodes from all created selections.

  • Basic state saving is now supported.

Reconstruction Sampler:

  • Intervals are now displayed with only two colors by default, it makes
    distinguishing many intervals easier. The previous multi-color mode can be
    reactivated from the preview window.

Tracing general:

  • The settings widget allows now to configure a "fast split mode" and a "fast
    merge mode" to allow particular groups of skeletons to be split and merged
    without confirmation. Similarly to Visibility Groups, these Fast Split/Merge
    Groups can be defined in terms of a universal match (all skeletons), a
    required meta-annotation or a creator ID. In fast split mode, all annotations
    from the split skeletons are copied over to the split off part. In fast merge
    mode, all annotations are taken over from a skeleton merged in without
    confirmation.

  • The P shortcut (peek) will now show the closest skeleton to the cursor in
    all open 3D Viewers. To show the active skeleton use Shift + P.

  • A move/navigation mode can now be used using the new (third) button in the
    tracing tool bar. If enabled, no mouse based node actions will be performed
    anymore. Left mouse button clicks/movements are handled like right mouse
    button clicks/movements.

  • The Split Neuron Dialog has now "select all" checkboxes for annotations.

Statistics widget:

  • Statistics widget: The new option "All" in the time unit selection control
    allow to aggregate user data for the whole time range.

  • The number of newly created treenodes is now displayed alongside the cable
    length in the contribution table.

  • Each user can now be included in an aggregate statistics row at the end of the
    table by checking the checkbox in front of the username.

  • Add extra Refresh button to top bar.

Volume widget:

  • The new link "List skeletons" in each volume table row allows to open a new
    Selection Table containing all skeletons the bounding box of the respective
    volume intersects. Due to large numbers of skeletons in bigger volumes, this
    is currently mainly useful for smaller volumes.

  • When creating box volumes, the new button "Define cube at
    current location" allows to conveniently create a cube with a configurable
    edge length at the current location.

Miscellaneous:

  • Basic Search: allow search for treenode IDs and connector IDs.

  • Project administration: selected projects can now be exported as JSON or YAML
    file using the respective action from the drop-down menu.

  • In addition to the '/apis' endpoint, the API documentation is now also
    available as part of the general documentation.

  • Connectivity Widget: partner header indexes can now optionally be replaced
    with the neuron name and rotated by 90 degrees.

  • The Keyboard/Mouse Help Widget has now a text filter, which allows to show
    only items containing a particular text.

  • The Tracing Overlay has a new setting "Allow lazy node updates", available in
    the Settings Widget. If enabled (default), stack viewers can skip node updates
    if a change didn't occur in the Viewer's field of view. This is useful when
    e.g. the Connector Viewer is open while tracing, because not all viewers get
    update due to the addition of a node.

  • Selection Table: Skeletons can now be imported from CSV files.

  • The status bar shows now both stack space and project space coordinates of the
    mouse cursor.

  • The skeleton projection layer works now with orthogonal views.

  • Detailed review colors are now enabled by default. To get the old behavior
    back, adjust your settings (admins can do this for the whole project or
    server).

  • Stacks can now be created whose planar axes have anisotropic resolution.
    The stack viewer will display these stacks correctly by scaling tiles
    separately along each axis. The tracing overlay is also compatible with these
    stacks in all orthoviews.

  • Added H2N5 tile source type.

  • Due to the new skeleton summary tables, some APIs are much faster now:

    1. Obtaining review info skeletons with /{project_id}/skeletons/review-status,
    2. API and connectivity information through /{project_id}/skeletons/connectivity,
    3. Listing skeletons with a minimum node count with /{project_ids}/skeletons/
    4. Getting skeleton node count with /{project_id}/skeleton/{skeleton_id}/node_count.

    In consequence the following widgets became faster too: Connectivity Widget,
    Selecting skeleton counts in the Statistics Widget, Review count based
    coloring (e.g. in Graph Widget). Opening the Merge Dialog is also faster due
    to this change.

Bug fixes

  • Reconstruction Sampler: 3D visualizations for interval preview and color mode
    were sometimes wrong and showed additional intervals. This is fixed now.
    Actual interval boundaries were not affected and are correct.

  • Reconstruction Sampler: Individual intervals are not silently deleted anymore
    if referenced start or end node is deleted. As consequence interval start and
    end nodes can't be deleted anymore.

  • Measurement table: no error is shown anymore after merging two listed
    skeletons.

  • Radio button drop-downs and checkbox drop-downs now hide on a mouse click
    outside of the contr...

Read more

2018.02.16

16 Feb 23:54
Compare
Choose a tag to compare

2018.02.16

Contributors: Albert Cardona, Andrew Champion, Chris Barnes, Tom Kazimiers

Notes

  • Three new OS package dependencies have been added (due to a Django framework
    upgrade), make sure they are installed:

    sudo apt-get install binutils libproj-dev gdal-bin

  • Python 3.6 is now supported. Make sure to update your settings.py by replacing
    the line

    COOKIE_SUFFIX = hashlib.md5(CATMAID_URL).hexdigest()

    with the following line:

    COOKIE_SUFFIX = hashlib.md5(CATMAID_URL.encode('utf-8')).hexdigest()

  • A virtualenv upgrade is required. To correctly install one updated dependency,
    the django-rest-swagger Python package has to be removed first from from the
    virtualenv, before the virtualenv is updated:

    pip uninstall django-rest-swagger
    pip install -r requirements.txt

  • requirements files now inherit from each other: -test.txt includes
    the production requirements, -dev.txt includes test (and therefore
    production) and doc.

  • CATMAID extensions no longer require users to manually edit their
    INSTALLED_APPS in settings.py. Remove if they are already in use.

  • The NODE_PROVIDER settings variable (settings.py) is replaced with the
    NODE_PROVIDERS variable. The new variable takes a list of node provider names,
    which are iterated as long as no result nodes are found. Replace the former
    single string value with a list with this name as single element, e.g. if
    the current setting reads NODE_PROVIDER = 'postgis2d', replace it with
    NODE_PROVIDERS = ['postgis2d'].

  • Three new types of cache backed node providers have been added: cached_json,
    cached_json_text, cached_msgpack. Tests suggest that cached_msgpack is the
    fastest.

Features and enhancements

Reconstruction Sampler:

  • 3D previews now allow to use a white background and toggle meta spheres.

  • To match a given interval length, the Reconstruction Sampler will now create
    new nodes during interval creation. This allows better sampling on long
    straight lines without intermediate nodes. To disable this behavior uncheck
    the "Create bounding nodes" setting. Newly created nodes are automatically
    labeled with the "sampler-created" tag.

  • By adjusting the new interval "max error" setting, it is possible to only
    create bounding nodes (if enabled) if the closest existing node is farther
    away from the ideal location than the max error value. This allows better
    reuse of existing nodes.

  • Created interval boundary nodes are also automatically deleted on sampler
    removal, if possible. If a created node has not been altered in any way during
    its life time and it still forms a straight line with its parent and single
    child, it can be removed. Otherwise the node is kept.

  • A new "Partner" step has been added. After a connector is selected at random,
    users are now asked to add all partner nodes to the synapse and let the widget
    choose a partner site for them.

  • Sections that should be interpolated (e.g. because of large tracing data
    shifts due to image data shifts) are now respected and not over-sampled
    anymore.

  • Completed intervals can now be optionally excluded from random selection.

Node Search:

  • Tagged connector nodes are now shown in search results.

  • Annotation search results allow now to open the Neuron Navigator page for the
    respective annotation.

Graph Widget:

  • It is now possible to show links of all connector types. By default, only
    synaptic connectors are available, but this can be changed using the "Link
    types" drop down control in the Graph tab. There also the edge color for
    particular link types can be set.

  • The edge color button in the option dialog has been replaced with a link type
    drop down which allows to set color and visibility per connection type.

3D Viewer:

  • Stored interpolatable sections are now loaded by default. Additional project
    space sections can be configured from the Skeleton filters tab.

  • Custom tags can now also apply to predefined labels like TODO or
    uncertain_end.

Miscellaneous:

  • The maximum number of frames per second rendered by a stack viewer window can
    now be adjusted through the Settings Widget > Stack view > Max frames per
    second.

  • The Open Widget dialog is now only opened with Ctrl + Space when no other keys
    are pressed. This allows to use key combinations like Ctrl + Shift + Space + >
    to browse smoothly through the image stack while hiding the tracing layer.

  • The active node can now be moved in Z by holding the Alt key and
    using ,/.. The stack viewer follows the node.

  • Filters: a Reconstruction Sampler Domain node filer is now available. The
    required node ID can be optained from the Reconstruction Sampler Widget.

  • Neuron History: a lower and upper bound for the time window in which changes
    are respected and counted can now be set.

  • On CATMAID front-pages (project overview) the keys 0-9 can now be used to open
    the n-th stackgroup or stack, stackgroups have precedence.

  • Sections in which nodes should be interpolated for various displays can now be
    persisted in the database using the "interpolatable sections" data model from
    the admin interface.

Bug fixes

  • Reconstruction sampler: Strahler shading color updates work now reliably.

  • Reconstruction sampler: connectors in neighboring intervals are now not
    included anymore in connector listing.

  • 3D Viewer: SVG export styles are correctly applied again, the exported data
    isn't transparent anymore.

  • 3D Viewer: custom label matches are now always given precedence when multiple
    labels are matched on a node.

2017.12.07

07 Dec 06:01
Compare
Choose a tag to compare

2017.12.07

Contributors: Albert Cardona, Andrew Champion, Chris Barnes, Tom Kazimiers

Notes

  • PostgreSQL 9.6 and Postgis 2.4 are now required.

  • A virtualenv update is required.

  • New settings field: CROPPING_VERIFY_CERTIFICATES. This controls whether SSL
    certificates should be verified during cropping and defaults to True.

  • The NODE_LIST_MAXIMUM_COUNT setting can now be set to None, which disables
    node limiting. If node count limiting isn't really needed in most cases, this
    can slighly improve query performance.

  • The file UPDATE.md will now list all administration related changes for each
    release, including required manual tasks. These changes also continue to be
    part of this change log.

  • Requires running of: manage.py catmaid_update_project_configuration

  • Tracing data is now by default transmitted in a binary form, please make
    therefore sure your web-server applies GZIP not only to the "application/json"
    content type, but also to "application/octet-stream". For Nginx this would be
    the gzip_types setting.

Features and enhancements

Node filters:

  • A new node filter called "In skeleton source" has been added. This allows to
    add a filter which only allows nodes of a particular set of skeletons,
    accessed through a skeleton source. This can be useful to look at connectivity
    of a particular set of neurons. It can of course be combined with other
    filters like volume intersection.

  • New filter: Pruned arbor, which keeps the arbor proximal to the (tagged) cut
    points, discarding distal subarbors.

Connector table:

  • Node filters are now supported.

  • Attachment connectors are now supported.

Landmarks:

  • The new "Landmark Widget" allows to create landmarks, form groups of them and
    use these groups to virtually transform skeletons from one landmark group into
    another one. This can be used e.g. to find homologues neurons. Virtual
    skeletons can currently displayed in the 3D Viewer.

  • Defining landmarks and their groups is done in two stages: 1. define abstract
    landmarks and group those abstract landmarks. 2. link locations to individual
    landmarks and their presence in a group.

  • Abstract landmarks represent e.g. a feature that can be found at multiple
    places in the data set, like the entry point of a particular neuron on the
    left and right side of a Drosophila brain. The left and right side of the
    brain would then be represented as groups, each one having expected abstract
    landmarks as members.

  • Creating new abstract landmarks and groups can be done through the respective
    "Name" fields and "Add" buttons in the Landmarks tab. To link an abstract
    landmark to a group, click on either an existing landmark group member in the
    landmark groups table or on the "(none)" placeholder. The newly opened dialog
    will allow to associate landmarks with groups. Landmarks can be member of
    multiple groups.

  • Linking locations to landmarks and the groups they are member of is done by
    using the right click context menu on either an existing location in the ...

Read more

2017.10.02

03 Oct 03:39
Compare
Choose a tag to compare

2017.10.02

Contributors: Chris Barnes, Dylan Simon, Albert Cardona, Andrew Champion, Tom Kazimiers

Notes

  • A virtualenv update is required.

  • This release adds optional statics summary tables, which can increase the
    performance of project/user statistics significantly, but statistics will also
    be correct without them. The additional table keeps aggregated information
    about various user actions. To initialize this table, the following
    manangement command has to be run after the migration:

    ./manage.py catmaid_populate_summary_tables

    To maintain good performance, this command has to be run regularly, e.g.
    through a cron job or Celery every night. Because summary updates are
    incremental by default, they don't take much time to update.

  • CATMAID's Docker images changed: The existing catmaid/catmaid image is now
    only a base image that is used for a simple standalone demo image, available
    as catmaid/catmaid-standalone. Additionally, the base image is used in a new
    docker-compose setup, which can be used if persistent data is required. The
    documentation has been updared with all the details.

  • The Docker default port is now 8000 for both regular setups and docker-compose
    setups.

  • The data view "legacy project list" has been renamed to "simple project list".
    It now supports separate project title and stack title filters, which can be
    preconfigured using the data view config options "projectFilterTerm" and
    "stackFilterTerm". For both simple terms and regular expressions can be used.
    The filter input boxes can optionally be hidden by setting the "filter" config
    property to false.

  • Running periodic tasks is now easier and a default setup for cleaning up
    cropping data at 23:30 and update tracing project statistics at 23:45 every
    night is now available. All it needs is to run a Celery worker and a Celery
    beat scheduler. The documentation has more details.

  • The cropping output file file name prefix and file extension can now be
    specified from settings.py. The defaults are:

    CROPPING_OUTPUT_FILE_EXTENSION = "tiff"
    CROPPING_OUTPUT_FILE_PREFIX = "crop_"

  • CATMAID can now make use of a ASGI server to utilize WebSocket connections.
    Such a server (e.g. Daphne) needs to handle URLs that start with /channels/.
    Currently only messages can be updated this way, which already removes many
    requests for setups with many users. This allows for instance an immediate
    feedback after a cropping a sub-stack finished without requiring the client to
    check for new messages every minute.

Features and enhancements

Project/user statistics:

  • The widget should now be much faster.

  • Import actions are not counted anymore by default. The "Include imports"
    checkbox can be used to change this.

  • State saving is now supported.

Neuron history:

  • Events by different users are now collected in separate bouts to attribute
    parallel user activity. To restore the previous behavior (users are ignored)
    the "Merge parallel events" checkbox can be checked.

  • State saving is now supported

  • Individual neurons can be removed with the help of an "X" icon in the firs
    column.

  • A "total time" column has been added, which aggregates time across all active
    bouts formed by both tracing and review events. Since events are binnned in
    bouts, the total time is not just the sum of both tracing time and review
    time.

Node filters:

  • A Reconstruction Sampler interval can now be used as a node filter. This
    allows e.g. reviewing only an interval or look only at the connectivity of
    the interval.

Graph Widget:

  • Edge labels can now take different forms and can be configured in the
    properties dialog. There are two new label options available: "Fractions of
    outbound connections" and "Fractions of inbound connections". Instead of an
    absolute number they display the relative fraction. This works for both
    regular connections and connections involving groups.

Reconstruction sampler:

  • Intervals for a particular interval length can now be previewed before
    creating a new sampler. To do so, use the "Preview intervals" button in the
    Sampler tab.

  • Interval reviews can now be initiated directly from the interval step table.
    It will open a new review widget with pre-set interval filter and added
    skeleton. The same is possible from the synapse workflow step using the
    "Review interval" button.

Layouts:

  • The Stack View section of the Settings Widget allows now the configuration of
    a list of default layouts that can be applied to newly opened stacks and stack
    groups. Layouts are useful for having a reasonable default configuration of a
    newly opened CATMAID workspace including stacks and stack groups.

  • Layouts mimic nested function calls and are constructed from v(a,b) and h(a,b)
    elements for vertical and horizontal splits, o(a) for optional windows, where
    a and b can each be other v() or h() nodes, one of [XY, XZ, ZY, F1, X3D] or any
    quoted widget handle (e.g. "neuron-search", see Ctrl + Space). At the moment,
    in o(a), "a" can't be XY, XZ or ZY. To reference the 3D Viewer use X3D.

  • By default only one layout is available: organize XY, XZ and ZY in four
    quadrants using one additional window (if non available the help page). This
    is its specification: h(v(XY, XZ), v(ZY, o(F1))). With h() and v(), horizontal
    and vertical splits are declared, respectively. With o(F1) a help window will
    be opened as fourth window if not already another window exists.

  • Useful for organizing orthogonal views in a custom way and to create default
    workspaces. For instance, to always open a Neuron Search widget right to the
    a single XY view stack, the layout can be used: h(XY, "neuron-search").

CATMAID extensions:

  • As well as supporting standalone 3rd party static files, CATMAID now supports
    fully-featured extensions, which can include database models, API endpoints,
    and tests as well as static files.

  • CATMAID extensions are Django apps which follow a particular layout, and can
    be installed with pip

  • The goal is for CATMAID extensions to be reusable and interoperable
    between versions and installations of CATMAID, reducing the need to fork it
    and keep the fork updated in parallel.

  • More details can be found in the docs.

3D viewer:

  • Volumes can now be filtered in the volume drop-down menu with the help of a
    built-in text box.

Miscellaneous:

  • The Open Widget dialog now displays a table that includes more information on
    widgets. Instead of the previously used auto-completion this table is now
    filtered.

  • If multiple tile layers are used, the stack viewer layer settings (blue white
    box in lower left corner) allow to set which stacks should be respected if
    broken sections are skipped. This can be done through the "Skip broken
    sections of stacks" drop-down menu. Whether layers added after the first layer
    should be respected by default when checking for broken sections can be set in
    the Settings Widget in the "Stack view" section.

  • The dialog to add a new annotation displays now the existing annotations as
    well.

  • Tabbed windows: changing window aliases are now reflected in tab headers.

  • The performance of node creation and deletion operations has been improved by
    preventing full node updates after these operations.

  • Widgets with state saving support now also support removing previously saved
    states through a button in the widget controls available through the window
    icon in the widget title bar.

  • The Selection Tool (the first icon in the top bar) has been removed, because
    it didn't provide any functionality. It is replaced by an icon to show the
    "Open Widget" dialog, which can otherwise be shown using Ctrl + Space.

  • When splitting a skeleton on a virtual node, the virtual node will now only be
    instantiated if the user presses OK in the dialog, canceling won't cause a
    change of the virtual node.

  • The default for hiding tile layers if the nearest section is broken (instead
    of showing the next available) can now be configured from the Settings Widget
    in its Stack View section.

  • plotly.js, a d3-based plotting library, is now available within CATMAID,
    making it much easier to generate common plots.

Bug fixes

  • Review widget: moving in reverse direction from a virtual node doesn't show
    error anymore.

  • Review widget: the Shift + Q key combination to select the next unreviewed
    node in downstream direction is respected again.

  • Review widget: Shift + W and Shift + Q work now correctly if the first
    unreviewed node is a virtual node.

  • Key combinations involving the Alt key were not respected on Mac OS. This is
    fixed now.

  • Reconstruction sampler: the list of connectors in the selected interval of the
    synapse workflow step is now complete when refreshed.

  • 3D viewer: the error shown when changing skeleton properties with an active
    connector restriction is fixed.

  • The neuron history widget now calculates both tracing time and review time
    correctly.

  • 'Peek skeleton' (P key) works again.

  • The Split Skeleton Dialog updates the embedded 3D view again after all
    skeletons are loaded.

  • Former partner nodes of deleted connector nodes can now be deleted without an
    additional tracing layer update.

  • The Skeleton Projection Layer can now use the skeleton source colors and other
    colors source again.

  • The simple front end data view's project filter works again

  • Skeleton source subscriptions: opting in and out of applying color changes in
    source skeletons to existing target skeletons works again.

  • Dragging nodes in stacks with smaller resolutions (larger nm/px) doesn't
    require large drag distances anymore, ...

Read more

2017.07.28

28 Jul 04:52
Compare
Choose a tag to compare

2017.07.28

Contributors: Chris Barnes, Albert Cardona, Tom Kazimiers, Daniel Witvliet

Notes

  • Prepared statements can now also be used together with connection pooling.

  • A virtualenv update is required.

  • The following lines have to be removed from settings.py,

    import djcelery
    djcelery.setup_loader()
    INSTALLED_APPs += ("kombu.transport.django")
    BROKER_URL = 'django://'
    
  • Two new treenode and connector node providers have been added: postgis2dblurry
    and postgis3dblurry. They works like the regular postgis2d and postgis3d node
    providers except that they allow more false positives, because edges are only
    tested for bounding box intersection with the query bounding box. Depending on
    the dataset, this can help performance but might require a larger node limit.

Features and enhancements

Synapse Fractions:

  • New button "Append as group". Multiple neurons will be shown in a single column.

  • New UI functions to set the synapse confidence.

  • Default to Upstream (fractions for input neurons).

  • Shift-click to toggle selected state of partner neurons or groups, and then
    push 'J' to create a new partner group.

  • X axis labels can now be rotated, from the Options tab.

  • Node filters can now be applied to filter valid connector links.

3D Viewer:

  • New skeleton shading modes:

    • "Axon and dendrite": like the coloring mode "Axon and dendrite", but using
      shading like in the "Split at active node" mode.
    • "Single Strahler number": color branches of a specific Strahler number
      in full color, and darken everything else. The Strahler number is specified
      in the tab named "Shading parameters".
    • "Strahler threshold": color branches of a Strahler number equal or higher
      than the number specifed in the Stahler number field ("Shading paramters"
      tab) in full color, darken all others. When inverted, show in full color
      branches with a Strahler number strictly lower than the specified one.
  • The Z plane display now supports stack viewer layers. If multiple layers are
    shown in a stack viewer, its Z plane will render all visible layers on top of
    each other.

Review widget:

  • Large neurons are handled better, rendering tables with many segments is
    now faster.

  • The set of displayed columns can now be constrained with the help of the new
    "Visible reviewers" setting in the widget. Available options are "All", "Team"
    and "Self".

  • A new option to automatically scroll the active segment into view has been
    added ("Scroll to active segment"). With this enabled (default), when a node
    of a segment of the reviewed skeleton is selected in the tracing layer, the
    review widget will scroll the segment into view.

  • State saving is now supported.

Node filters:

  • A Strahler number filter has been added to only show nodes with a Strahler
    value below/same as/above a user defined number.

Neuron Search:

  • It is now possible to use the "not" operator with neuron name and annotation
    search criteria.

  • Both neuron name and type columns can now be sorted.

  • Filters are now supported. Only neurons will be shown of which at least one
    node is in the node filter result set.

Neuron History:

  • This new widget can be opened using Ctrl+Space together with the key
    "neuron-history" or through a button in the Neuron Navigator.

  • For all neurons added to this widget, time related information is presented.
    Currently, the following is calculated: Tracing time, review time, cable
    length before review, cable length after review, connectors before review and
    connectors after review.

  • The components that contribute to the tracing time can be adjusted with the
    "Tracing time" drop down menu.

Reconstruction Sampler:

  • This new widget can be opened using Ctrl + Space and the keyword
    "reconstruction-sampler". It allows to target reconstruction effort based on
    the spatial sampling of a skeleton. This is mainly useful for large neurons
    that can't be quickly traced to completion. The widget is organized as a
    workflow that prevents skipping steps.

  • To keep track of reconstruction progress, a so called sampler is created
    for a skeleton of interest. This skeleton is typically the backbone of a
    larger neuron. The sampler keeps track of some global properties for the
    sampling. Once created, a sampler can be "opened" either by clicking "Open" or
    by double clicking the respective table row. Also note that with a sampler
    attached to a skeleton, the skeleton can not be deleted. If a skeleton should
    be deleted, delete its samplers first.

  • Once opened, a sampler allows creating so called sampler domains, which are
    regions on the skeletons which should in principle be considered for sampling.
    Topological and tag based definitions of sampler domains are possible. Created
    samplers can be opened through a click on "Open" or a double click on the
    table row.

  • Once opened, sampler domains can be further divided in so called sampler
    intervals. To do so the "Create intervals" buttons has to be pressed. These
    have initially all the same length (defined in sampler) and no branches.
    Intervals are meant to picked at random through the respective button.

  • With an interval selected, the goal is now to reconstruct it to completion
    with all branches and connectors. CATMAID will show a warning when moved out
    of the interval. The workflow page will show both input and output connectors.

  • As soon as the interval is reconstructed completely, a synapse can be picked
    at random from which the next backbone can be reconstructed. Once the backbone
    is found and reconstructed, the sampling can start over. Alternatively,
    another interval in the original skeleton can be selected at random.

Miscellaneous:

  • Split/Merge dialog: the node count for both respectively remaining/new and
    remaining/old are now on top of the annotation lists.

  • The General Settings section of the Settings Widget now provides an option to
    disable asking for confirmation when the active project is closed.

  • Tile layers can now be configured to not automatically switch to the next
    accessible mirror if the present one is inaccessible. This is useful for some
    custom mirror use cases and can be set through the "Change mirror on
    inaccessible data" checkbox in the layer settings.

  • Empty neuron name pattern components are now by default trimmed automatically.
    Empty components at the pattern endings won't leave any white space behind.
    Empty components between other components are allowed one space maximum (if
    there were spaces before). The difference is visible e.g. in the Graph Widget.

  • URL-to-view links now allow selected connector nodes.

  • Adding a custom mirror will now by default disable automatic mirror switching
    of the respective tile layer. Custom mirrors often come only with a subset of
    the data, which currently triggers a mirror switch by default if some tile is
    not accessible.

  • The tracing layer is now faster with creating skeleton nodes, connectors and
    partner nodes by avoiding unneeded node updates.

  • Export widget: exporting neurons as NRRD files is now supported. The NAT R
    package is used for this. Check documentation for setup.

  • The tracing tool has now a button to refresh CATMAID's caches for neuron names
    and annotations. This can be used to update neuron names with components that
    were changed by other users. Additionally, such a cache update is performed
    automatically once every hour.

  • URLs to a particular view work now also with a location and a skeleton ID only
    rather than requiring a node ID always.

  • Using "Shift + x" and "Ctrl + x" will now activate a checkbox selection mode
    and the cursor turns into a crosshair. In this mode one can draw a rectangle
    everywhere on the screen and all checkboxes that are behind the rectangle will
    be toggled using "Ctrl + x" (turned on if off and vise versa) or checked with
    "Shift + x". Either releasing the drawing mouse click or a second "Shift + x"
    or "Ctrl + x" will deactivate the tool again.

  • All skeleton source widgets (typically those with a skeleton source drop-down
    menu) now support copy and paste of skeleton models. Pressing "Alt + Ctrl + c"
    in an active widget will copy its skeletons (along with colors) into a
    clipboard. Pressing "Alt + Ctrl + v" in another widget will then paste those
    models into the now active widget.

  • When using the Z key to create new nodes, existing nodes are not as easily
    selected from a distance anymore. Before, the radius around a click event was
    too large and the closest node around the click was selected. Other nodes will
    now only be selected if the mouse cursor is close to their circle graphic on
    screen.

Bug fixes

  • Graph widget: re-layout works again.

  • The Docker image can be used again without requiring a manual restart of the
    uWSGI server.

  • Zooming in using the "=" key and using Shift + Z to join and create nodes both
    work again.

  • Links inside the simple Search Widget can be clicked again.

  • Back-end calls with large parameter lists can be used again (e.g. review
    status in the Connectivity Widget).

  • Highlighted tags in the Neuron Dendrogram widget are now correctly displayed
    next to the nodes they belong to. Before, each tagged node had all selected
    tags shown next to it, regardless of whether it would be tagged with them.

  • 3D viewer: the textured Z section rendering in the 3D Viewer now respects the
    mirror setting from the tile layer.

  • 3D viewer: a memory leak in displaying Z sections with images has been fixed.
    This allows long continued Z section browsing without crashing.

  • 3D viewer: the accurateness of object picking on neurons and location...

Read more