Add some Zarr-based datatypes

[davelopez]

Requires #17614 and #19077

Includes the following datatypes:

General Zarr datatypes

• CompressedZarrZipArchive (zarr.zip): represents a Zarr ZipStore. It seems to have some limitations (i.e. it doesn't work with zips containing the zarr store in a subfolder).
• ZarrDirectory (zarr): represents a Zarr DirectoryStore. Contains the zarr structure in the extra_files_path of the dataset. I wonder if there is a way to make the "main" dataset to really point to the actual root folder in extra_files like a symlink or something like this, but I don't know if this really makes sense. Right now, to access your zarr store root you will need to use the input like this '$zarrinput.extra_files_path/$zarrinput.metadata.store_root' (see Handling different input types section below)

OME-Zarr datatypes

• CompressedOMEZarrZipArchive (ome_zarr.zip): Similar to CompressedZarrZipArchive but expects to find an OME/METADATA.ome.xml file in the store root so it can be easily converted/extracted to an OMEZarr directory.
• OMEZarr (ome_zarr): Similar to ZarrDirectory but identify this datatype as an OME Zarr image.

Handling different input types

If your zarr input is defined like this:

<inputs>
    <param name="zarrinput" type="data" format="zarr,zarr.zip" allow_uri_if_protocol="https" label="Zarr input"/>
</inputs>

You need to access the zarr store in a "slightly" different way depending on how the store is provided as input.

Input as zarr.zip

UploadOMEZarr.zip.mp4

You can upload a zip file containing the zarr store. In order to open it in your tool you need to do the following:

if '$zarrinput.extension' == 'zarr.zip':
    compression = int('$zarrinput.metadata.compression' or 0)
    zip_store = zarr.ZipStore('$zarrinput', mode='r', compression=compression)
    input_zarr = zarr.open(zip_store, mode='r')`

Please note that the root store must be in the root of the zip (i.e. no subdirectory containing the store)

Input as zarr (Directory)

You cannot directly upload a directory, but you can upload a zarr.zip and then extract it to a "zarr directory dataset".

ConvertZipToOMEZarr.mp4

To open this kind of zarr in your tool you must do the following:

if '$zarrinput.extension' == 'zarr' and '$zarrinput.state' != 'deferred':
        input_store = '$zarrinput.extra_files_path/$zarrinput.metadata.store_root'
        input_zarr = zarr.open(input_store, mode='r')`

Note that you need to reference the extra_files_path and append the $zarrinput.metadata.store_root to it. This will point to the right internal directory where the zarr store has been extracted and will work even if the uploaded zip had a parent subfolder containing the zarr store.

Input as a deferred zarr (URI to a remote zarr)

DeferredZarr.mp4

In your tool, you can just directly use '$zarrinput', it will point to the remote URI. But you need to check this is actually a deferred dataset like this:

if '$zarrinput.extension' == 'zarr' and '$zarrinput.state' == 'deferred':
        input_zarr = zarr.open('$zarrinput', mode='r')`

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  • For testing the "directory" variant datatype:
    • Upload any zarr directory in a zip file. You can download them from any public repository or generate one using https://zarr.readthedocs.io/en/stable/tutorial.html
    • Go to Edit Attributes then select the Datatypes section and convert it to zarr.
  • For testing the "deferred" variant (remote URI):
    • Paste the following URI as content: https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr. You can find more examples here.
    • Make sure you set the dataset options to Defer dataset resolution
  • You can use this "simple" tool wrapper to test or make your own:

<tool id="zarr_test" name="Zarr Test" version="0.1.0+galaxy0" profile="23.0">
    <description>test wrapper for zarr format</description>
    <requirements>
        <requirement type="package" version="2.18.3">zarr</requirement>
        <requirement type="package" version="2024.9.0">s3fs</requirement>
    </requirements>
    <command detect_errors="exit_code"><![CDATA[
        python $script
    ]]></command>
    <configfiles>
        <configfile name="script"><![CDATA[
import zarr

input_store = None
if '$zarrinput.extension' == 'zarr':
    if '$zarrinput.is_deferred':
        print('Using remote S3 store as input')
        # Deferred datasets will return directly the URI in combination with allow_uri_if_protocol
        input_store = '$zarrinput'
    else:
        print('Using local directory store as input')
        # TODO: Is there a way for the dataset file to directly reference the extra_files_path?
        # We need to append the store_root to the extra_files_path to get the full path to the zarr store
        # in case it was extracted to a subdirectory
        input_store = '$zarrinput.extra_files_path/$zarrinput.metadata.store_root'
elif '$zarrinput.extension' == 'zarr.zip':
    compression = int('$zarrinput.metadata.compression' or 0)
    print(f'Using zipped store as input with compression {compression}')
    # Currently only zip files containing the zarr store at the root (i.e. no subdirectories) are supported
    input_store = zarr.ZipStore('$zarrinput', mode='r', compression=compression)
else:
    raise ValueError('Unsupported input format')

if input_store is None:
    raise ValueError('Unable to determine input store. Your input dataset may be an unsupported format')

# Inspect the input zarr
input_zarr = zarr.open(input_store, mode='r')
print("Input zarr info:")
print(f"  {input_zarr.info_items()}")

# Create the output store where the new zarr will be written
output_store = '$zarroutput.extra_files_path'
output_zarr = zarr.open(output_store, mode='w')
foo = output_zarr.create_group('foo')
bar = foo.create_group('bar')
baz = bar.zeros('baz', shape=(10000, 10000), chunks=(1000, 1000), dtype='i4')
        ]]></configfile>
    </configfiles>
    <inputs>
        <param name="zarrinput" type="data" format="zarr,zarr.zip" allow_uri_if_protocol="https" label="Zarr input"/>
    </inputs>
    <outputs>
        <data name="zarroutput" format="zarr" label="${tool.name} on ${on_string}: New ZARR"/>
    </outputs>
    <help><![CDATA[

simple wrapper to test zarr datatype

    ]]></help>
</tool>

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jmchilton]

I think the URI datatype is a little wonky also. I don't want to encode remote URIs this way if we can avoid it. Lets focus on a syntax in tool data parameters that allows accepting URIs instead of doing this. The other datatypes and enhancements all look really great to me. I'll create an issue for the data parameter handling.

[davelopez]

I think the URI datatype is a little wonky also. I don't want to encode remote URIs this way if we can avoid it. Lets focus on a syntax in tool data parameters that allows accepting URIs instead of doing this. The other datatypes and enhancements all look really great to me. I'll create an issue for the data parameter handling.

Thanks again for the great feedback! I've dropped the "wonky" URI datatypes and implemented #19077 as discussed. Once that is merged, I'll rebase here and should be ready to go :)

[bgruening]

This one needs now a final rebase :)
Thanks @davelopez!

[bgruening]

Can we merge with this or do we need to investigate this more?

[mvdbeek]

I would try to get rid of this ? Otherwise I'd like to see a tool actually use this metadata element.

[davelopez]

Let me explain the issue in more detail to see if we can eliminate the need for store_root metadata. This store_root isn't true metadata—it's more of a workaround that indicates the folder containing the actual root of the Zarr directory.

Both tools and visualizations require a path to this root directory to access the correct contents.

When we upload a zip file containing a Zarr directory, it’s common for the zip to include the parent folder of the Zarr store. Many Zarr zips I’ve encountered are structured this way. Ideally, the Zarr store would be zipped without this extra parent folder, but even if it isn’t, when we extract it using the converter, it creates a new folder (like dataset_{uuid}) within extra_files_path, resulting in an additional layer.

Currently, to access the Zarr directory correctly, any tool needs to reference it as follows:

input_zarr = zarr.open('$zarrinput.extra_files_path/$zarrinput.metadata.store_root', mode='r')

This approach, however, is not fully reliable—what if the Zarr store is nested deeper within subdirectories? A better solution might be to use a dedicated converter (rather than archive_to_directory.xml) that finds and extracts the root store directly to extra_files_path, without any parent folders, would this be better?

Another drawback is that tool developers must remember to reference the $zarrinput.extra_files_path, and even add /$zarrinput.metadata.store_root to reach the actual Zarr store.

Any ideas on how to make this process more elegant and eliminate the store_root?