idr0026-weigelin-immunotherapy S-BIAD860 #648
Comments
|
Conversion time: 9min |
|
Trying to estimate how much space is needed for this conversion. First image is Images vary in size for the study, but about 111 .pattern images (see IDR/idr-utils#56) Maybe 300 GB or more needed (maybe up to 500 GB)? |
|
Looks like all the pattern files we need to convert are under: Corresponds to image count from IDR/idr-utils#56 |
|
|
Checking on s3... This image has only a single
cc @sbesson |
|
However, the OME.xml looks OK, e.g. https://uk1s3.embassy.ebi.ac.uk/idr0026/zarr/3.50.6-3.140922_11-36-07.00.ome.zarr/OME/METADATA.ome.xml |
|
On pilot-idr0125... These look good in OMERO, compared to existing IDR
|
|
@will-moore not 100% sure of what went wrong on your conversion but using the converter library shipping with the current IDR version of Bio-Formats, I get and the $ cat 3.49.6-3.140922_11-33-57.00.zarr/0/.zattrs
{
"multiscales" : [ {
"metadata" : {
"method" : "loci.common.image.SimpleImageScaler",
"version" : "Bio-Formats 0.6.10"
},
"axes" : [ {
"name" : "t",
"type" : "time"
}, {
"name" : "c",
"type" : "channel"
}, {
"name" : "z",
"type" : "space"
}, {
"name" : "y",
"type" : "space"
}, {
"name" : "x",
"type" : "space"
} ],
"name" : "11-33-57_PMT - PMT [BD2_GREEN] [00]_Time Time0000.tif",
"datasets" : [ {
"path" : "0",
"coordinateTransformations" : [ {
"scale" : [ 1.0, 1.0, 1.0, 1.0, 1.0 ],
"type" : "scale"
} ]
}, {
"path" : "1",
"coordinateTransformations" : [ {
"scale" : [ 1.0, 1.0, 1.0, 2.0, 2.0 ],
"type" : "scale"
} ]
} ],
"version" : "0.4"
} ],
"omero" : {
"channels" : [ {
"color" : "FF0000",
"coefficient" : 1,
"active" : true,
"label" : "BD2_GREEN",
"window" : {
"min" : 372.0,
"max" : 15788.0,
"start" : 372.0,
"end" : 15788.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "00FF00",
"coefficient" : 1,
"active" : true,
"label" : "BD8_DEEPR",
"window" : {
"min" : 373.0,
"max" : 16188.0,
"start" : 373.0,
"end" : 16188.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "0000FF",
"coefficient" : 1,
"active" : true,
"label" : "BD7_RED",
"window" : {
"min" : 759.0,
"max" : 8978.0,
"start" : 759.0,
"end" : 8978.0
},
"family" : "linear",
"inverted" : false
}, {
"color" : "FF0000",
"coefficient" : 1,
"active" : false,
"label" : "FD6_FDRED",
"window" : {
"min" : 237.0,
"max" : 12339.0,
"start" : 237.0,
"end" : 12339.0
},
"family" : "linear",
"inverted" : false
} ],
"rdefs" : {
"defaultT" : 0,
"model" : "color",
"defaultZ" : 13
}
}
}Sounds like the best way forward would be to redo the whole conversion? |
|
@sbesson Looking at #648 (comment), it looks like I used the same version: |
|
Testing... (and failing!)... |
|
Even using the same lib as @sbesson gives me same result?! |
|
I suspect there's something wrong with your environment and particularly the |
|
That didn't work either! I wanted to try on a different machine completely... Do I need to clone all of https://github.com/IDR/idr-metadata here? |
|
Just confirming... |
For the sake of testing, you might just want to copy the single
The inconsistency in the output is very concerning. Have you tried after fully deactivating Conda, not just your environment? |
No. How do you do that? |
|
|
|
I already did that. How's that different from deactivating your environment? I tried on a different machine... Then tried... WAT!? |
|
@will-moore I think I found the source of the issue. Can you try one more test with your last configuration, running |
|
👍 - yes that worked - 4 channels |
|
So the difference is really whether the directory containing the pattern file is writeable or not, meaning that a memo file can be saved or not. I suspect something incorrect is happening in the case where a memo file cannot be written and the wrong reader is used. It might be specific to either Either way for the scope of this testing, I think the solution is to re-run the conversion using the same user that owns the |
|
OK I've I see that in the original conversion I used |
|
Meanwhile, deleted invalid data previously uploaded... |
|
The above removal of contents of the bucket ran very slowly and has only resulted in the removal of a handful of zarr filesets out of the 111 originally there. Since we want to delete ALL the filesets uploaded, probably quicker to delete the bucket and recreate.. ran This seemed to hang/time-out and doesn't seem to have had any affect: Reverted to running the rm again in a screen |
|
@sbesson - Seems that the memo issue is something it would be good to fix (or at least warn) to prevent others suffering the pain above! I can create an issue somewhere, but where? |
|
From my side, the immediate candidates are:
Possibly the outstanding action would be to retest a similar scenario using bioformats2raw 0.7.0, a multi-channel pattern dataset and identify whether it's IDR specific. /cc @melissalinkert |
|
Ah - apologies @sbesson: I just realised you meant that there is probably just 1 issue (not 4) but it needs testing to determine where the issue lies! |
|
Retested with a simpler version of the pattern file with 2 timepoints compatible with upstream Bio-Formats Placed a copy of this pattern file under The Based on the above, I am leaning towards options 1 and 2 i.e. it's an IDR/bioformats specific issue which probably will be classified as |
|
Started creating zips in a Screen |
|
With all the previous Filesets deleted from s3, uploaded just a couple of different new ones to test... Ooops - got an extra directory in there, but the images look good: |
|
Uploading zips to BioStudies... |
|
francesw
changed the title
|
Currently we have 20 out of 111 Filesets "viewable" at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD860.html... |
|
All good (missing thumbnails in screenshot are for images not included in the 20 updated by mkngff above:
|
|
Testing on Updated to today's OMEZarrReader.jar (only on omeroreadwrite server - not proxies). Use all 111 Images in idr0026.csv - see IDR/idr-utils@003b3a3 Started |
|
mkngff just done (nearly 12:00). 25869ms is 26 secs for setId |
|
This issue has been mentioned on Image.sc Forum. There might be relevant details there: https://forum.image.sc/t/file-format-to-store-images-using-ngff-coverter/98320/10 |
idr0026-weigelin-immunotherapy
The text was updated successfully, but these errors were encountered: