Merge branch 'master' into jpeg2000

.codespellrc

@@ -1,4 +1,4 @@
 [codespell]
-skip = .git,.venv,venvs,*.svg,_build
+skip = .git,.venv,venvs,*.svg,_build,build
 # te -- TE as codespell is case insensitive
 ignore-words-list = bu,nd,te

.github/workflows/lint.yml

@@ -14,9 +14,9 @@ jobs:
           fetch-depth: 0
 
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
-          python-version: '3.7'
+          python-version: '3.8'
 
       - name: Install dependencies
         run: |

.github/workflows/release.yml

@@ -19,24 +19,28 @@ jobs:
       - name: Download auto
         run: |
           #curl -vL -o - "$(curl -fsSL https://api.github.com/repos/intuit/auto/releases/latest | jq -r '.assets[] | select(.name == "auto-linux.gz") | .browser_download_url')" | gunzip > ~/auto
-          # Pin to 10.16.1 so we don't break if & when
+          # Pin so we don't break if & when
           # <https://github.com/intuit/auto/issues/1778> is fixed.
-          wget -O- https://github.com/intuit/auto/releases/download/v10.16.1/auto-linux.gz | gunzip > ~/auto
+          # 11.0.5 is needed for <https://github.com/intuit/auto/issues/2432>
+          wget -O- https://github.com/intuit/auto/releases/download/v11.0.5/auto-linux.gz | gunzip > ~/auto
           chmod a+x ~/auto
 
-      - name: Check whether a release is due
+      - name: Query 'auto' on type of the release
         id: auto-version
         run: |
-          version="$(~/auto version)"
+          # to be able to debug if something goes wrong
+          set -o pipefail
+          ~/auto version -vv | tee /tmp/auto-version
+          version="$(sed -ne '/Calculated SEMVER bump:/s,.*: *,,p' /tmp/auto-version)"
           echo "version=$version" >> "$GITHUB_OUTPUT"
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Set up Python
         if: steps.auto-version.outputs.version != ''
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
-          python-version: '^3.7'
+          python-version: '^3.8'
 
       - name: Install Python dependencies
         if: steps.auto-version.outputs.version != ''

.github/workflows/test.yml

@@ -14,7 +14,6 @@ jobs:
       fail-fast: false
       matrix:
         python-version:
-          - '3.7'
           - '3.8'
           - '3.9'
           - '3.10'
@@ -26,7 +25,7 @@ jobs:
           fetch-depth: 0
 
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -54,8 +53,9 @@ jobs:
         run: coverage run `which pytest` -s -v heudiconv
 
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4
         with:
           fail_ci_if_error: false
+          token: ${{ secrets.CODECOV_TOKEN }}
 
 # vim:set et sts=2:

.github/workflows/typing.yml

@@ -14,9 +14,9 @@ jobs:
           fetch-depth: 0
 
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
-          python-version: '3.7'
+          python-version: '3.8'
 
       - name: Install dependencies
         run: |

CHANGELOG.md

@@ -1,3 +1,83 @@
+# v1.1.0 (Wed Feb 28 2024)
+
+#### 🚀 Enhancement
+
+- Add support for a  custom seqinfo to extract from DICOMs any additional metadata desired for a heuristic [#581](https://github.com/nipy/heudiconv/pull/581) ([@yarikoptic](https://github.com/yarikoptic) [@bpinsard](https://github.com/bpinsard))
+- codespell: ignore "build" folder which might be on the system [#581](https://github.com/nipy/heudiconv/pull/581) ([@yarikoptic](https://github.com/yarikoptic))
+
+#### Authors: 2
+
+- Basile ([@bpinsard](https://github.com/bpinsard))
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
+# v1.0.2 (Mon Feb 26 2024)
+
+#### 🐛 Bug Fix
+
+- properly remove GE multiecho bvals/bvecs [#728](https://github.com/nipy/heudiconv/pull/728) ([@bpinsard](https://github.com/bpinsard))
+- datalad sensitive marking fixes [#739](https://github.com/nipy/heudiconv/pull/739) ([@bpinsard](https://github.com/bpinsard))
+- Reject "Missing images" in sensor-dicoms [#735](https://github.com/nipy/heudiconv/pull/735) ([@chaselgrove](https://github.com/chaselgrove))
+
+#### ⚠️ Pushed to `master`
+
+- Adding workflow figure ([@TheChymera](https://github.com/TheChymera))
+- Added figures to master branch ([@TheChymera](https://github.com/TheChymera))
+
+#### 🏠 Internal
+
+- auto 11.0.5 is needed to avoid hitting some "Error: fatal: ... not an integer" bug [#746](https://github.com/nipy/heudiconv/pull/746) ([@yarikoptic](https://github.com/yarikoptic))
+- Fix - auto is in ~/, not in the PATH [#745](https://github.com/nipy/heudiconv/pull/745) ([@yarikoptic](https://github.com/yarikoptic))
+- Make it possible to review auto version -v output during release + adjust that workflow step description [#743](https://github.com/nipy/heudiconv/pull/743) ([@yarikoptic](https://github.com/yarikoptic))
+- [gh-actions](deps): Bump codecov/codecov-action from 3 to 4 [#736](https://github.com/nipy/heudiconv/pull/736) ([@dependabot[bot]](https://github.com/dependabot[bot]) [@yarikoptic](https://github.com/yarikoptic))
+- [gh-actions](deps): Bump actions/setup-python from 4 to 5 [#723](https://github.com/nipy/heudiconv/pull/723) ([@dependabot[bot]](https://github.com/dependabot[bot]))
+
+#### 📝 Documentation
+
+- Adjust wording on heuristics page -- do not claim creating some skeleton [#741](https://github.com/nipy/heudiconv/pull/741) ([@yarikoptic](https://github.com/yarikoptic))
+- Document how to release and add changelog entries [#737](https://github.com/nipy/heudiconv/pull/737) ([@asmacdo](https://github.com/asmacdo) [@yarikoptic](https://github.com/yarikoptic))
+- Add dianne tutorials [#734](https://github.com/nipy/heudiconv/pull/734) ([@asmacdo](https://github.com/asmacdo) [@yarikoptic](https://github.com/yarikoptic))
+- Add documentation building instructions [#730](https://github.com/nipy/heudiconv/pull/730) ([@asmacdo](https://github.com/asmacdo))
+- Allowing RTD to access images under the same path as README [#734](https://github.com/nipy/heudiconv/pull/734) ([@TheChymera](https://github.com/TheChymera))
+- Using environment figure in about section [#730](https://github.com/nipy/heudiconv/pull/730) ([@TheChymera](https://github.com/TheChymera))
+- Make README more concrete [#724](https://github.com/nipy/heudiconv/pull/724) ([@asmacdo](https://github.com/asmacdo))
+
+#### Authors: 6
+
+- [@chaselgrove](https://github.com/chaselgrove)
+- [@dependabot[bot]](https://github.com/dependabot[bot])
+- Austin Macdonald ([@asmacdo](https://github.com/asmacdo))
+- Basile ([@bpinsard](https://github.com/bpinsard))
+- Horea Christian ([@TheChymera](https://github.com/TheChymera))
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
+# v1.0.1 (Fri Dec 08 2023)
+
+#### 🐛 Bug Fix
+
+- Drop Python 3.7 support [#722](https://github.com/nipy/heudiconv/pull/722) ([@yarikoptic](https://github.com/yarikoptic))
+- ReproIn: give an informative assertion message when multiple values are found [#718](https://github.com/nipy/heudiconv/pull/718) ([@yarikoptic](https://github.com/yarikoptic))
+- Convert assertion into a warning that we would not use dicom dir template option [#709](https://github.com/nipy/heudiconv/pull/709) ([@yarikoptic](https://github.com/yarikoptic))
+- Do not demand --files for all commands, even those which do not care about it (like populate-intended-for) [#708](https://github.com/nipy/heudiconv/pull/708) ([@yarikoptic](https://github.com/yarikoptic))
+
+#### ⚠️ Pushed to `master`
+
+- Add script to sensor dicoms -- for the error where dcm2niix might or might not fail but issues an Error ([@yarikoptic](https://github.com/yarikoptic))
+
+#### 🏠 Internal
+
+- Ran pre-commit on everything, black decided to adjust some formatting [#721](https://github.com/nipy/heudiconv/pull/721) ([@yarikoptic](https://github.com/yarikoptic))
+- Make sensor-dicoms use gnu-getopt if present (on OSX) [#721](https://github.com/nipy/heudiconv/pull/721) ([@yarikoptic](https://github.com/yarikoptic))
+
+#### Authors: 1
+
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
 # v1.0.0 (Wed Sep 20 2023)
 
 #### 💥 Breaking Change

CONTRIBUTING.rst

@@ -51,11 +51,33 @@ If you are unsure what that means, here is a set-up workflow you may wish to fol
 
      git push -u origin topic_of_your_contribution
 
-
 (If any of the above seems overwhelming, you can look up the `Git documentation
 <http://git-scm.com/documentation>`_ on the web.)
 
 
+Releases and Changelog
+----------------------
+
+HeuDiConv uses the `auto <https://intuit.github.io/auto/>`_ tool to generate the changelog and automatically release the project.
+
+`auto` is used in the HeuDiConv GitHub actions, which monitors the labels on the pull request.
+HeuDiConv automation can add entries to the changelog, cut releases, and
+push new images to `dockerhub <https://hub.docker.com/r/nipy/heudiconv>`_.
+
+The following pull request labels are respected:
+
+    * major: Increment the major version when merged
+    * minot: Increment the minot version when merged
+    * patch: Increment the patch version when merged
+    * skip-release: Preserve the current version when merged
+    * release: Create a release when this pr is merged
+    * internal: Changes only affect the internal API
+    * documentation: Changes only affect the documentation
+    * tests: Add or improve existing tests
+    * dependencies: Update one or more dependencies version
+    * performance: Improve performance of an existing feature
+
+
 Development environment
 -----------------------
 
@@ -78,6 +100,18 @@ This is best accomplished via::
   pip install -e .[all]
 
 
+Documentation
+-------------
+
+To contribute to the documentation, we recommend building the docs
+locally prior to submitting a patch.
+
+To build the docs locally:
+
+ 1. From the root of the heudiconv repository, `pip install -r docs/requirements.txt`
+ 2. From the `docs/` directory, run `make html`
+
+
 Additional Hints
 ----------------
 

LICENSE

@@ -1,4 +1,4 @@
-Copyright [2014-2019] [Heudiconv developers]
+Copyright [2014-2024] [HeuDiConv developers]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
@@ -11,3 +11,10 @@ Copyright [2014-2019] [Heudiconv developers]
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+
+Some parts of the codebase/documentation are borrowed from other sources:
+
+- HeuDiConv tutorial from https://bitbucket.org/dpat/neuroimaging_core_docs/src
+
+  Copyright 2023 Dianne Patterson

README.rst

@@ -51,6 +51,10 @@ into structured directory layouts.
 - It integrates with `DataLad <https://www.datalad.org/>`_ to place converted and original data under git/git-annex
   version control while automatically annotating files with sensitive information (e.g., non-defaced anatomicals, etc).
 
+Heudiconv can be inserted into your workflow to provide automatic conversion as part of a data acquisition pipeline, as seen in the figure below:
+
+.. image:: figs/environment.png
+
 Installation
 ------------
 
@@ -60,13 +64,18 @@ on heudiconv.readthedocs.io .
 HOWTO 101
 ---------
 
-In a nutshell -- ``heudiconv`` operates using a heuristic which, given metadata from DICOMs, would decide how to name
-resultant (from conversion using `dcm2niix`_) files. Heuristic `convertall <https://github
-.com/nipy/heudiconv/blob/master/heudiconv/heuristics/convertall.py>`_ could actually be used with no real
-heuristic and by simply establish your own conversion mapping through editing produced mapping files.
-In most use-cases of retrospective study data conversion, you would need to create your custom heuristic following
-`existing heuristics as examples <https://github.com/nipy/heudiconv/tree/master/heudiconv/heuristics>`_ and/or
-referring to `"Heuristic" section <https://heudiconv.readthedocs.io/en/latest/heuristics.html>`_ in the documentation.
+In a nutshell -- ``heudiconv`` is given a file tree of DICOMs, and it produces a restructured file tree of NifTI files (conversion handled by `dcm2niix`_) with accompanying metadata files.
+The input and output structure is as flexible as your data, which is accomplished by using a Python file called a ``heuristic`` that knows how to read your input structure and decides how to name the resultant files.
+You can run your conversion automatically (which will produce a ``.heudiconv`` directory storing the used parameters), or generate the default parameters, edit them to customize file naming, and continue conversion via an additional invocation of `heudiconv`:
+
+.. image:: figs/workflow.png
+
+
+``heudiconv`` comes with `existing heuristics <https://github.com/nipy/heudiconv/tree/master/heudiconv/heuristics>`_ which can be used as is, or as examples.
+For instance, the Heuristic `convertall <https://github.com/nipy/heudiconv/blob/master/heudiconv/heuristics/convertall.py>`_ extracts standard metadata from all matching DICOMs.
+``heudiconv`` creates mapping files, ``<something>.edit.text`` which lets researchers simply establish their own conversion mapping.
+
+In most use-cases of retrospective study data conversion, you would need to create your custom heuristic following the examples and the `"Heuristic" section <https://heudiconv.readthedocs.io/en/latest/heuristics.html>`_ in the documentation.
 **Note** that `ReproIn heuristic <https://github.com/nipy/heudiconv/blob/master/heudiconv/heuristics/reproin.py>`_ is
 generic and powerful enough to be adopted virtually for *any* study: For prospective studies, you would just need
 to name your sequences following the `ReproIn convention <https://github.com/nipy/heudiconv/blob/master/heudiconv/heuristics/reproin.py#L26>`_, and for
@@ -106,3 +115,17 @@ Docker image preparation being found in ``.github/workflows/release.yml``.
 ---------------------
 
 - https://github.com/courtois-neuromod/ds_prep/blob/main/mri/convert/heuristics_unf.py
+
+
+Support
+-------
+
+All bugs, concerns and enhancement requests for this software can be submitted here:
+https://github.com/nipy/heudiconv/issues.
+
+If you have a problem or would like to ask a question about how to use ``heudiconv``,
+please submit a question to `NeuroStars.org <http://neurostars.org/tags/heudiconv>`_ with a ``heudiconv`` tag.
+NeuroStars.org is a platform similar to StackOverflow but dedicated to neuroinformatics.
+
+All previous ``heudiconv`` questions are available here:
+http://neurostars.org/tags/heudiconv/

docs/commandline.rst

@@ -0,0 +1,12 @@
+=============
+CLI Reference
+=============
+
+``heudiconv`` processes DICOM files and converts the output into user defined
+paths.
+
+.. argparse::
+   :ref: heudiconv.cli.run.get_parser
+   :prog: heudiconv
+   :nodefault:
+   :nodefaultconst:

docs/container.rst

@@ -0,0 +1,49 @@
+==============================
+Using heudiconv in a Container
+==============================
+
+If heudiconv is :ref:`installed via a Docker container <install_container>`, you
+can run the commands in the following format::
+   
+    docker run nipy/heudiconv:latest [heudiconv options]
+
+So a user running via container would check the version with this command::
+
+    docker run nipy/heudiconv:latest --version
+
+Which is equivalent to the locally installed command::
+
+    heudiconv --version
+
+Bind mount
+----------
+
+Typically, users of heudiconv will be operating on data that is on their local machine. We can give heudiconv access to that data via a ``bind mount``, which is the ``-v`` syntax.
+
+Once common pattern is to share the working directory with ``-v $PWD:$PWD``, so heudiconv will behave as though it is installed on your system. However, you should be aware of how permissions work depending on your container toolset.
+
+
+Docker Permissions
+******************
+
+When you run a container with docker without specifying a user, it will be run as root.
+This isn't ideal if you are operating on data owned by your local user, so for ``Docker`` it is recommended to specify that the container will run as your user.::
+
+    docker run --user=$(id -u):$(id -g) -e "UID=$(id -u)" -e "GID=$(id -g)" --rm -t -v $PWD:$PWD nipy/heudiconv:latest --version
+
+Podman Permissions
+******************
+
+When running Podman without specifying a user, the container is run as root inside the container, but your user outside of the container.
+This default behavior usually works for heudiconv users::
+
+    docker run -v $PWD:PWD nipy/heudiconv:latest --version
+
+Other Common Options
+--------------------
+
+We typically recommend users make use of the following flags to Docker and Podman
+
+* ``-it`` Interactive terminal
+* ``--rm`` Remove the changes to the container when it completes
+