"TypeError: reject_bytes is on.." ujson.dumps(kerchunk.zarr.single_zarr(uri))

[raybellwaves]

I have many individual zarr stores. I'm hoping to use kerchunk to avoid the overhead of doing xr.open_mfdataset on them. I'm trying to create and store reference files for the zarr stores.

When doing:

import ujson
import kerchunk.zarr

uri = "s3://BUCKET/FILE.zarr"
zarrchunks = kerchunk.zarr.single_zarr(uri)
ujson.dumps(zarrchunks)

I get:

TypeError                               
      1 uri = "s3://BUCKET/FILE.zarr"
      2 zarrchunks = kerchunk.zarr.single_zarr(uri)
----> 3 ujson.dumps(zarrchunks)

TypeError: reject_bytes is on and '{
    "Conventions": "CF-1.7",
    "GRIB_centre": "eums",
    "GRIB_centreDescription": "EUMETSAT Operation Centre",
    "GRIB_edition": 2,
    "GRIB_subCentre": 65535,
    "history": "2024-03-20T08:28 GRIB to CDM+CF via cfgrib-0.9.10.4/ecCodes-2.29.0 with {\"source\": \"../tmp/tmpno1cdzeb/d8f3c360f092a8de85100e59909bd0ad7b220ec80d62745d80d53b98edffa4f7\", \"filter_by_keys\": {}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}",
    "institution": "EUMETSAT Operation Centre"
}' is bytes

These are the attributes of the Dataset.

I may be missing something in how to create a reference file from a zarr store.

I get the same error on a public zarr file:

import kerchunk.zarr
import ujson

uri = "s3://hrrrzarr/sfc/20240321/20240321_00z_anl.zarr/2m_above_ground/TMP/2m_above_ground"
storage_options = {"anon": True}
ujson.dumps(kerchunk.zarr.single_zarr(uri, storage_options=storage_options))

gives

TypeError 
      1 uri = "s3://hrrrzarr/sfc/20240321/20240321_00z_anl.zarr/2m_above_ground/TMP/2m_above_ground"
      2 storage_options = {"anon": True}
----> 3 ujson.dumps(kerchunk.zarr.single_zarr(uri, storage_options=storage_options))

TypeError: reject_bytes is on and '{
    "zarr_format": 2
}' is bytes

[martindurant]

kerchunk.utils.consolidate does this kind of conversion. Probably kerchunk.zarr.single_zarr should do this automatically and support writing directly to JSON as other scanner classes do.

[raybellwaves]

Thanks. I'll take a look