Define scope of project

[tsalo]

The primary goal of this project is to create a lightweight library for querying BIDS datasets, using the BIDS schema. While this may ultimately be integrated into pybids as a sort of lightweight-layout module, at the moment we feel it would be best to develop independently.

In order to make this tool as light as possible, we plan to rely on schema-generated regular expressions to find and parse files, without using SQLAlchemy or indexing the full dataset.

Our current plan is discussed in this HackMD.

[tsalo]

@erdalkaraca drafted a proposal here: https://docs.google.com/presentation/d/12x3cQGRD9-T1bkpK--t0e_OdgMb3UuO4jn5fGpj0a4w

[tsalo]

To do list from today's call:

1. Create new repository for new tool.
   • Work on pybids-compatible interface.
2. Compile list of functions/methods we need to support with new tool in issue in new repository.
   • Metadata search methods.
   • Include deep parameter for Dataset/BIDSLayout?
3. Comment on bids-validator repository about using Erdal's approach to validation based on the schema.
4. Comment on pybids repo about (1) splitting up repository and (2) using new schema-based approach in BIDSLayout. Do this after benchmarking new tool against old one.
5. Weigh in on schema structure and content.
   • Especially [SCHEMA] Separate schema object definitions from rules bids-specification#834

Does that sound like everything?

EDIT: Tagging @erdalkaraca and Jochem Rieger (IDK your username)

[erdalkaraca]

Thanks, Taylor!
I have created a new repo at https://github.com/ANCPLabOldenburg/ancp-bids
Will file issues against the new repo for the topics you listed above.

[yarikoptic]

The primary goal of this project is to create a lightweight library for querying BIDS datasets, using the BIDS schema. While this may ultimately be integrated into pybids as a sort of lightweight-layout module, at the moment we feel it would be best to develop independently.

echoing my https://github.com/physiopy/phys2bids/pull/374/files#r809277094 and ongoing nipy/heudiconv#544 I wonder if it would be worthwhile working toward a library not for "querying" per se, but for "manipulation" of bids things, such as

• easy parsing and changing (adding/removing entities etc) of BIDS filenames so they remain bids compliant (RF+ENH: provide bids schema driven support for manipulation of bids filenames nipy/heudiconv#544)?
• basic CLI/Python interfaces for manipulating files (eh, found elderly Should this repo get 2nd life as bids-cmdline or smth like that? bidsutils#6 by myself, more recent discussion: Rename tasks pybids#774)
  • mv and rm to rename/move and delete a file so it takes care possibly about
    • renaming/deleting also sidecar .json etc file
    • ensuring metadata associated with the file remains the same (which might become not true due to inheritance principle so not only immediate sidecar needs to be considered)
    • adjusting _scans.tsv file records

I am not yet 100% sure it would be possible to avoid growing another "pybids", but the goal is to keep it lean/quick to install and use. OR should we really go back to pybids, and see how we could partition it (e.g. pybids-core + pybids; may be just at pypi level) so we could keep improving pybids while being able to gain the "lightweight" version with only needed helpers without full "understand the BIDS dataset universe" bloat?

[poldrack]

I think it would be really unfortunate if another pybids-like package were to be developed, as opposed to working together to improve/extend pybids.

…

On Thu, Feb 17, 2022 at 9:23 AM Yaroslav Halchenko ***@***.***> wrote: The primary goal of this project is to create a *lightweight* library for querying BIDS datasets, using the BIDS schema. While this may ultimately be integrated into pybids as a sort of lightweight-layout module, at the moment we feel it would be best to develop independently. echoing my https://github.com/physiopy/phys2bids/pull/374/files#r809277094 and ongoing nipy/heudiconv#544 <nipy/heudiconv#544> I wonder if it would be worthwhile working toward a library not for "querying" per se, but for "manipulation" of bids things, such as - easy parsing and changing (adding/removing entities etc) of BIDS filenames so they remain bids compliant (nipy/heudiconv#544 <nipy/heudiconv#544>)? - basic CLI/Python interfaces for manipulating files (eh, found elderly bids-standard/bidsutils#6 <bids-standard/bidsutils#6> by myself, more recent discussion: bids-standard/pybids#774 <bids-standard/pybids#774>) - mv and rm to rename/move and delete a file so it takes care possibly about - renaming/deleting also sidecar .json etc file - ensuring metadata associated with the file remains the same (which might become not true due to inheritance principle so not only immediate sidecar needs to be considered) - adjusting _scans.tsv file records I am not yet 100% sure it would be possible to avoid growing another "pybids", but the goal is to keep it lean/quick to install and use. OR should we really go back to pybids, and see how we could partition it (e.g. pybids-core + pybids; may be just at pypi level) so we could keep improving pybids while being able to gain the "lightweight" version with only needed helpers without full "understand the BIDS dataset universe" bloat? — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGUVED76ZA7ZDISKKZ4PQDU3UVH3ANCNFSM45YPBYBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Russell A. Poldrack Albert Ray Lang Professor of Psychology Associate Director, Stanford Data Science Director, SDS Center for Open and Reproducible Science Building 420 Stanford University Stanford, CA 94305 ***@***.*** ***@***.***> http://www.poldracklab.org/

[tsalo]

We've started moving the pybids.reports module into a namespace package (https://github.com/bids-standard/pybids-reports). Perhaps something similar could be done with this pybids-light idea. I.e., it could be implemented as a namespace package (e.g., pybids.ext.light) maybe?

[yarikoptic]

but isn't it the other way around kinda -- that pybids-report extends pybids, i.e. have (already heavy) pybids as its install_requires? would make not sense to have a "light" version of pybids which requires heavy pybids... or it doesn't really have to require pybids, and just provide pybids.ext.light while pybids could install_requires it if it would start to use its functionality?

[yarikoptic]

I think it would be really unfortunate if another pybids-like package were to be developed, as opposed to working together to improve/extend pybids.

Totally agree! Starting yet another ball rolling hopefully to avoid that via bids-standard/pybids#818 ;) But echoing @tsalo's comment on pybids-reports, figuring out modularization of pybids seems to be something in dire need .

[erdalkaraca]

With ancpBIDS we have a modular and extendable (using a plugins mechanism) implementation of a "pybids lite" without any "heavyweight dependencies". The library also provides a subset of PyBIDS' BIDSLayout interface. Documentation is work in progress, but you can already have a look at it:

https://ancpbids.readthedocs.io/en/latest/usage.html#query-using-the-pybids-api

We proposed two topics for GSoC 2022 as part of INCF's participation as a mentoring organization:

• interactive graph visualization of the BIDS Schema
• validation engine to handle rules derivable from the BIDS schema and the BIDS Specification

[poldrack]

given that pybids is probably up for a full refactor at some point, it would be worth it for the pybids team to have a closer look at the ancpBIDS project and see whether it could be a suitable base for pybids-core 2.0. my only goal here is to avoid fragmentation and the inevitable duplication of efforts that comes with it.

…

On Fri, Feb 18, 2022 at 9:14 AM Erdal Karaca ***@***.***> wrote: With ancpBIDS we have a modular and extendable (using a plugins mechanism) implementation of a "pybids lite" without any "heavyweight dependencies". The library also provides a subset of PyBIDS' BIDSLayout interface. Documentation is work in progress, but you can already have a look at it: https://ancpbids.readthedocs.io/en/latest/usage.html#query-using-the-pybids-api We proposed two topics for GSoC 2022 as part of INCF's participation as a mentoring organization: - interactive graph visualization of the BIDS Schema - validation engine to handle rules derivable from the BIDS schema and the BIDS Specification — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGUVECU6NJMZUVH4J6EE4DU3Z47VANCNFSM45YPBYBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- Russell A. Poldrack Albert Ray Lang Professor of Psychology Associate Director, Stanford Data Science Director, SDS Center for Open and Reproducible Science Building 420 Stanford University Stanford, CA 94305 ***@***.*** ***@***.***> http://www.poldracklab.org/

[erdalkaraca]

At least at a conceptual level, ancpBIDS has some interesting approaches to consider:

• Python code generation from the BIDS schema (you get content assist in professional IDEs)
• in-memory representation of a BIDS dataset (as a graph)
• own (low-cost) query language to operate on the in-memory graph
• plugins mechanism to prevent monolithic code and handle future complexity

given that pybids is probably up for a full refactor at some point, it would be worth it for the pybids team to have a closer look at the ancpBIDS project and see whether it could be a suitable base for pybids-core 2.0. my only goal here is to avoid fragmentation and the inevitable duplication of efforts that comes with it.
…
On Fri, Feb 18, 2022 at 9:14 AM Erdal Karaca @.> wrote: With ancpBIDS we have a modular and extendable (using a plugins mechanism) implementation of a "pybids lite" without any "heavyweight dependencies". The library also provides a subset of PyBIDS' BIDSLayout interface. Documentation is work in progress, but you can already have a look at it: https://ancpbids.readthedocs.io/en/latest/usage.html#query-using-the-pybids-api We proposed two topics for GSoC 2022 as part of INCF's participation as a mentoring organization: - interactive graph visualization of the BIDS Schema - validation engine to handle rules derivable from the BIDS schema and the BIDS Specification — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGUVECU6NJMZUVH4J6EE4DU3Z47VANCNFSM45YPBYBQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>
-- Russell A. Poldrack Albert Ray Lang Professor of Psychology Associate Director, Stanford Data Science Director, SDS Center for Open and Reproducible Science Building 420 Stanford University Stanford, CA 94305 @.*** @.***> http://www.poldracklab.org/

[ancpJR]

In close communication with @tsalo back in 2021 we decided to call the library ancpBIDS to prevent any overlap with PyBIDS and get started quickly to explore technical ideas and approaches. Yet, we agreed that a migration path for PyBIDS users should be considered by providing the established BIDSLayout interface as an entry point. We would be happy to discuss about the current implementation and whether it could be a suitable base for pybids-core 2. How should we proceed?

given that pybids is probably up for a full refactor at some point, it would be worth it for the pybids team to have a closer look at the ancpBIDS project and see whether it could be a suitable base for pybids-core 2.0. my only goal here is to avoid fragmentation and the inevitable duplication of efforts that comes with it.
…
On Fri, Feb 18, 2022 at 9:14 AM Erdal Karaca @.> wrote: With ancpBIDS we have a modular and extendable (using a plugins mechanism) implementation of a "pybids lite" without any "heavyweight dependencies". The library also provides a subset of PyBIDS' BIDSLayout interface. Documentation is work in progress, but you can already have a look at it: https://ancpbids.readthedocs.io/en/latest/usage.html#query-using-the-pybids-api We proposed two topics for GSoC 2022 as part of INCF's participation as a mentoring organization: - interactive graph visualization of the BIDS Schema - validation engine to handle rules derivable from the BIDS schema and the BIDS Specification — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGUVECU6NJMZUVH4J6EE4DU3Z47VANCNFSM45YPBYBQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>
-- Russell A. Poldrack Albert Ray Lang Professor of Psychology Associate Director, Stanford Data Science Director, SDS Center for Open and Reproducible Science Building 420 Stanford University Stanford, CA 94305 @.*** @.***> http://www.poldracklab.org/