-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can path to .bidscoin be configured? #226
Comments
It's stated when you run: $ bidscoin
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST]
[--tracking {yes,no,show}] [-v]
BIDScoin is a toolkit to convert and organize raw data-sets according to the Brain Imaging Data Structure (BIDS)
The basic workflow is to run these two tools:
$ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI
$ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap
Default settings and template bidsmaps are stored in the ".bidscoin" folder in your home directory (you can modify
them to your needs with any plain text editor) It's also stated in the docs (at least in the latest), but that can always be improved of course. The path to the config is now defined like this (in # Create a BIDScoin user configuration directory if needed and load the BIDScoin user settings
configfile = Path.home()/'.bidscoin'/__version__/'config.toml'
templatefolder = configfile.parent/'templates' I don't see an easy way to configure the path to the config :-p. Where would you like it? Note that the user specific templates are copied from the central install, so if you change it there then all your users will have that version (and they can modify it individually). Or what is your use case? |
I guess I can add a few lines to check if home exists, otherwise use the central location |
My use case is a fully containerized conversion pipeline. In Nextflow, the home is not mounted by default and that causes the problem. I can mount it, so you don't have to change anything. I simply wasn't aware of these changes and need to document that myself. Wasn't BIDScoin writing to |
It certainly writes to the |
Also note that bidscoiner can now run in parallel. I don't know if you already did so in nextflow, but I guess that would lead to concurrency issues (e.g. multiple workers writing to the same |
I don't see ant tsv file. Which version introduce it? I'm using 4.2.1. |
No, it will be in the upcoming release. Since I plan to release it today or tomorrow, you have to be quick if you want to put something in :-) |
Oh. Thanks a lot for this kind offer :-) With config files, I usually see some priority lists, like config in your home, an env variable set to point to your config, config in But I don't want to push this one. I simply didn't want to mount the entire home in the container. |
That's a good example, I can add a check for environment variable that contains the path to the configuration folder (but the folder needs to be writable) |
That was easy :-), you can now use |
I've seen other files in
|
I think so, but I haven't tested it :-) |
It seems the configuration consists of more files. (venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin -t assets/bidscoin_map.yaml
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
... |
I don't like it but I could bind any host directory to |
Yes, but they are all automatically stored in the same folder as the config file. I just tested it and fixed a minor bug (so it should now work) |
$ export BIDSCOIN_CONFIG=~/tmp/config.toml
$ bidsmapper raw bids
$ tree ~/tmp
/home/marzwi/tmp
├── config.toml
├── templates
│ ├── bidsmap_bids2bids.yaml
│ ├── bidsmap_dccn.yaml
│ ├── bidsmap_sst.yaml
│ └── schema.json
└── usage
├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.bak
├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dat
└── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dir |
I installed the newest version with (venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -r ~/.bidscoin/
rm: cannot remove '/home/mpawlik/.bidscoin/': No such file or directory
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin --version
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
BIDS-version: 1.9.0
BIDScoin-version: 4.3.0, NB: Your BIDScoin version is NOT up-to-date: 4.3.0 -> 4.2.1
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ tree assets/
assets/
├── a003_bidscoin_map.yaml
├── bidscoin_config.toml
└── bidscoin_map.yaml |
Despite the flag, that doesn't upgrade when installing from github (if the version nr doesn't change), try: pip install --force-reinstall git+https://github.com/Donders-Institute/bidscoin |
Well, this errors out now
uninstall and install also didn't help. |
Ah, that's due to a bug (not mine) when building manpages (see here) |
A fresh venv didn't help and gave the same error. I did: (venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ deactivate
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -rf venv/
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ python3 -m venv venv
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ source venv/bin/activate
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ pip install git+https://github.com/Donders-Institute/bidscoin I can't figure out where are the man pages. This is what I should delete, right? Any of these?
|
Yeah, that's not my expertise either, I think it also depends on your distro. In my case they where build in the cwd (CentOS) or somewhere in my venv (OpenSuse). What if you do: $ find -name bidscoin.1 |
I don't find it anywhere in my home :-( Neither |
I tried previous commits and |
I don't think it's the ma pages issue here. Here's why installing the newest commits is failing:
4f13041 (newest)
|
And this is because my config file pointed by |
Yes, I saw and fixed that
That |
Ah, you set an empty path, yes, that probably fails. Try |
Just to make sure your expectations are in line, using the environment variable is not permanent, bidscoin only uses that config file when the variable is non-empty, otherwise it will use the default |
Perhaps that's a bug, let me check |
These file are only created when bidscoin is imported (it is all done in |
Do you get the same issue if you don't use apptainer? |
But it does 🤔 Bidscoin in Apptainer certainly doesn't remember where it saved the files. It keeps recreating Relevant Apptainer issue: apptainer/apptainer#388 |
Is #226 (comment) an answer? |
Ah, I think building the manpages imports bidscoin, that's when it saves the config file |
Nevertheless, it shouldn't matter, you can have multiple locations where config files are stored. As long as there is nothing in the environment, then the home directory is used, otherwise the location from the environment variable is used. Perhaps there is a weird interaction somewhere that I don't see, but otherwise I don't see how this doesn't work. I'll test a bit more. |
It seems that my problems were triggered mainly by Apptainer and the hard-coded path to store I'll try to summarize a bit. Outside of the ApptainerIt seems to work as intended now. Template files are copied to the parent of Inside of the containerversion 4.2.1I fix it with current master version (cd409e1)Adding If In generalI should not need to set |
Ok, that's not good, I will make a fallback-fix for this (in
So you pass a config file, that should work, but pointing it to
If it's not set, then bidscoin will create it in home, but I'll make that fallback-fix that should cover this
Sorry, I'm not that used to working with containers, but I don't see why you have to bind that root path? Isn't passing
I agree that the config folder shouldn't be a user concern and I'm surely open for suggestions to improve things here..
Well, that sounds great but on the other hand, it is also very common for applications to simply keep their settings in a hidden file or directory in home (e.g. dcm2niix).
I don't see how bidscoin can remember that without storing it somewhere on disk. Which is what the config file is
Currently I don't provide a tool or CLI options for editing/overruling the config file, the user needs to do that him/herself using a text editor (which is very easy and works very well). The behaviour should consistently be: Use the default config (in home or /tmp) or use the config specified in the environment variable. Do you perhaps know what |
Never mind, I found the answer. The problem I have is that tempfile.gettempdir() doesn't necessarily return a usable result... |
If your container has a tempdir (doesn't it?) then I hope my last commit works as expected... |
I just tested it in my own apptainer and there things now work (latest version from github) without setting any environment variable. So I guess that settles it, or? |
apptainer run --no-home bidscoin.sifINFO: fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /tmp/.bidscoin/4.3.0/config.toml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /tmp/.bidscoin/4.3.0/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]
BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)
The basic workflow is to run these two tools:
$ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI
$ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap
Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)
Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)
For more documentation see: https://bidscoin.readthedocs.io
options:
-h, --help show this help message and exit
-l, --list List all executables (i.e. the apps, bidsapps and utilities)
-p, --plugins List all installed plugins and template bidsmaps
-i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to install
-u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to uninstall
-d DOWNLOAD, --download DOWNLOAD
Download tutorial MRI data to the DOWNLOAD folder
-t [TEST], --test [TEST]
Test the bidscoin installation and template bidsmap
-b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
-c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
--tracking {yes,no,show}
Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
-v, --version Show the installed version and check for updates
examples:
bidscoin -l
bidscoin -d data/bidscoin_tutorial
bidscoin -t
bidscoin -t my_template_bidsmap
bidscoin -b my_study_bidsmap
bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
bidscoin -c myproject/bids
bidscoin -c myproject/bids format bibtex
bidscoin --tracking show apptainer run bidscoin.sifINFO: fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /home/marzwi/.bidscoin/4.3.0/config.toml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]
BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)
The basic workflow is to run these two tools:
$ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI
$ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap
Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)
Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)
For more documentation see: https://bidscoin.readthedocs.io
options:
-h, --help show this help message and exit
-l, --list List all executables (i.e. the apps, bidsapps and utilities)
-p, --plugins List all installed plugins and template bidsmaps
-i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to install
-u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to uninstall
-d DOWNLOAD, --download DOWNLOAD
Download tutorial MRI data to the DOWNLOAD folder
-t [TEST], --test [TEST]
Test the bidscoin installation and template bidsmap
-b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
-c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
--tracking {yes,no,show}
Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
-v, --version Show the installed version and check for updates
examples:
bidscoin -l
bidscoin -d data/bidscoin_tutorial
bidscoin -t
bidscoin -t my_template_bidsmap
bidscoin -b my_study_bidsmap
bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
bidscoin -c myproject/bids
bidscoin -c myproject/bids format bibtex
bidscoin --tracking show apptainer run --env BIDSCOIN_CONFIG=~/somefolder/config.toml bidscoin.sifINFO: fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /home/marzwi/somefolder/config.toml
-> /home/marzwi/somefolder/templates/bidsmap_bids2bids.yaml
-> /home/marzwi/somefolder/templates/bidsmap_dccn.yaml
-> /home/marzwi/somefolder/templates/bidsmap_sst.yaml
-> /home/marzwi/somefolder/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]
BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)
The basic workflow is to run these two tools:
$ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI
$ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap
Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)
Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)
For more documentation see: https://bidscoin.readthedocs.io
options:
-h, --help show this help message and exit
-l, --list List all executables (i.e. the apps, bidsapps and utilities)
-p, --plugins List all installed plugins and template bidsmaps
-i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to install
-u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to uninstall
-d DOWNLOAD, --download DOWNLOAD
Download tutorial MRI data to the DOWNLOAD folder
-t [TEST], --test [TEST]
Test the bidscoin installation and template bidsmap
-b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
-c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
--tracking {yes,no,show}
Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
-v, --version Show the installed version and check for updates
examples:
bidscoin -l
bidscoin -d data/bidscoin_tutorial
bidscoin -t
bidscoin -t my_template_bidsmap
bidscoin -b my_study_bidsmap
bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
bidscoin -c myproject/bids
bidscoin -c myproject/bids format bibtex
bidscoin --tracking show |
Yes, I know. It points to host |
It does. But depending on the execution parameters, it may be mounted from the host or not. |
As long as |
Thanks for reporting this, I will try to release a new version today (I think the best and most robust version so far) |
I like it.
When you build an Apptainer image (especially from a Docker image), there is only root user, and therefore there are no user homes. During building of the container,
Ok. This seems to be on Apptainer and
In this example (second sentence of my comment), you still have to mount
It may be solved with your newest changes. I'll test it in a bit.
Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container,
Good point. Maybe the problem is really only this Apptainer issue.
I like the tmp solution. In my particular case, however, it won't work. I mount a subdirectory of my home in the container (absolute path). It means that
|
Many thanks for dealing with that, although it wasn't on you. I learned a lot. |
No, because then it is not writable
I didn't know that, I'll add an even more thorough test then
You mean in your case
No, I really appreciate it, it makes bidscoin more robust |
Done! |
Right 🙈
I meant that it won't work for me if you only check for existence of
:-) |
FYI, I changed |
That is actually great :-) I agree. Thanks. |
** General summary **
I've just realized that the newer BIDScoin hard-codes the path to
.bidscoin
directory to reside in home. What if I don't want it there?** Additional detail **
I came across that when executing BIDScoin (dicomsort tool) in a container without home mounted.
** Next steps **
It's not up to me. Maybe at least an option to configure that? If not, make it more visible in the documentation?
The text was updated successfully, but these errors were encountered: