Add depth option to importer and include ZarrReader #310

dominikl · 2022-07-08T15:00:52Z

This PR adds an additional "depth" option to the importer (with default value 50) and adds the ZarrReader (cherry-picked from #309 , thanks @jburel !).

Test: Import an OME-NGFF Zarr file.

dominikl · 2022-07-08T15:20:47Z

Hm, maybe the depth option isn't actually required. I can set it to 1 and it still works, weird.

dominikl · 2022-07-13T14:34:34Z

I think some zarr files might not actually need the option, respectively even a value of 1 will be enough for them. But some don't, so you have to test with a zarr file with a deeper directory structure.

dominikl · 2022-07-14T07:34:39Z

The import progress display is a bit weird with zarr files. The importer seems to count the files correctly because it switches to the "multi file import display", but then it only shows 1 as file count. I'll try to fix that in the PR too.

dominikl · 2022-07-15T13:15:32Z

The issue was that Insight was counting files and not images in order to decide if the detailed or "simple" progress display should be used. Hence for zarr (many files / image) that won't work. Should be fixed with the last commit.

snoopycrimecop · 2022-07-22T01:06:08Z

Conflicting PR. Removed from build OMERO-insight-push#1206. See the console output for more details.
Possible conflicts:

PR Move gateway calls to OMEROGateway #305 dominikl 'Move gateway calls to OMEROGateway'
- src/main/java/org/openmicroscopy/shoola/env/data/OMEROGateway.java
- src/main/java/org/openmicroscopy/shoola/env/data/OmeroImageServiceImpl.java

~~--conflicts~~ Conflict resolved in build OMERO-insight-push#1231. See the console output for more details.

jburel · 2022-08-18T08:30:27Z

@dominikl Was this tested via UI?

dominikl · 2022-08-18T09:42:15Z

Pretty sure I did. But I'll give it another test.

jburel · 2022-08-18T09:43:56Z

@pwalczysko is going to test it. It will be good to have a release with that extension following 5.6.5 OMERO.server

pwalczysko · 2022-08-18T10:41:02Z

See below from the insight log.
I think the error appears when an ome.zarr is put into the RHP (into the Queue). This action also elicits a transient non-responsiveness of insight (for example cannot add new files to queue, or cannot navigate to other files in local machine using the LHP of importer)

2022-08-18 11:36:45,222 INFO  [   ome.formats.importer.ImportCandidates] (entQueue-0) 22845 file(s) parsed into 1 group(s) with 1 call(s) to setId in 19002ms. (30103ms total) [0 unknowns] 
2022-08-18 11:36:45,222 INFO  [       ome.formats.importer.ImportConfig] (entQueue-0) OMERO.blitz Version: 5.5.13-SNAPSHOT 
2022-08-18 11:36:45,223 INFO  [       ome.formats.importer.ImportConfig] (entQueue-0) Bioformats version: 6.10.2-SNAPSHOT revision: 4b387420d90924cd391239c2ea9bf321a239aa45 date: 18 August 2022 
2022-08-18 11:36:45,246 INFO  [   ome.formats.importer.ImportCandidates] (entQueue-0) Depth: 50 Metadata Level: MINIMUM 
2022-08-18 11:36:45,246 ERROR [   ome.formats.importer.ImportCandidates] (entQueue-0) Error on null with ome.formats.importer.ImportCandidates$SCANNING@2a570dd2 
java.lang.NullPointerException: null
	at ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
	at ome.formats.importer.ImportCandidates.scanWithCancel(ImportCandidates.java:523)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:388)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:219)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterUI.imageCount(ImporterUI.java:579)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterUI.addImporterElement(ImporterUI.java:549)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterComponent.importData(ImporterComponent.java:415)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterControl.handlePropertyChangedEvent(ImporterControl.java:399)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterControl.propertyChange(ImporterControl.java:376)
	at java.beans.PropertyChangeSupport.fire(PropertyChangeSupport.java:335)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:327)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:263)
	at java.awt.Component.firePropertyChange(Component.java:8448)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.ImportDialog.importFiles(ImportDialog.java:1305)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterUI.startImport(ImporterUI.java:851)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterControl.handlePropertyChangedEvent(ImporterControl.java:395)
	at org.openmicroscopy.shoola.agents.fsimporter.view.ImporterControl.propertyChange(ImporterControl.java:376)
	at java.beans.PropertyChangeSupport.fire(PropertyChangeSupport.java:335)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:327)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:263)
	at java.awt.Component.firePropertyChange(Component.java:8448)
	at org.openmicroscopy.shoola.agents.fsimporter.chooser.ImportDialog.actionPerformed(ImportDialog.java:1843)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:262)
	at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
	at java.awt.Component.processMouseEvent(Component.java:6539)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3318)
	at java.awt.Component.processEvent(Component.java:6304)
	at java.awt.Container.processEvent(Container.java:2239)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Container.dispatchEventImpl(Container.java:2297)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
	at java.awt.Container.dispatchEventImpl(Container.java:2283)
	at java.awt.Window.dispatchEventImpl(Window.java:2746)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
	at java.awt.EventQueue$4.run(EventQueue.java:733)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
2022-08-18 11:36:45,633 INFO  [                loci.formats.ImageReader] (entQueue-0) OMETiffReader initializing /Users/pwalczysko/ngff-testing/BBBC/NIRHTa-001.ome.tiff

Insight is frozen now. The workflow was:

Open insight, open importer
Connect to merge-ci as user-3
Open in the LHP ngff-testing folder, and from there, add to the queue in RHP 6 items (3 ome.zarrs and 3 ome.tiffs)
Observe that each addition of ome.zarr caused an intermittent freeze of the UI
Click "Import" button, observe that insight freezes completely

See
https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/binaryonly/
https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/tubhiswt-3D/
https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/BBBC/
are the basic sources of the 3 ome.zarr&ome.tiff doubles which you see on the screenshot
from the tubsw… it is the “zero” one

dominikl · 2022-08-19T10:55:35Z

There's definitely something odd, but it looks like with the ome.tiff files. They take ages for the "Scanning" phase. And as with this PR the scanning also happens when an image is placed into the queue (in order to work out how many files/images to deal with) and not just on import, the whole UI hangs at that point.

dominikl · 2022-08-19T12:10:21Z

It depends on the file, for example if you try https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/sub-resolutions/Fluorescence/ both works, ome.tiff and zarr. Should test the other ones which cause problems with older version of Insight. Wonder if it's actually a bioformats issue.

pwalczysko · 2022-08-19T12:57:47Z

It depends on the file, for example if you try https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/sub-resolutions/Fluorescence/ both works, ome.tiff and zarr. Should test the other ones which cause problems with older version of Insight. Wonder if it's actually a bioformats issue.

I have a simplified test. It is enough to concentrate on https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/BBBC/ . The plate there is called NIHRTa-001.ome.tiff.
This plate is capable of freezing the insight from this PR at the moment when the Import button is clicked.
This plate has a long scanning time (12 mins) in the release insight, but eventually, it is handled and the import goes on, see screenshot.
I would argue that even if BF has problems, insight should catch such eventuality and not freeze at any cost ?

Release insight handles the file successfully (after ca 12 minutes of scanning phase, which is not captured in screenshot, but read it from logs)

joshmoore · 2022-09-12T15:24:50Z

As a side note, making use of ome/ZarrReader#41 during OMERO import (in order to not create bazillions of chunk files in the DB) will likely require detecting a .zarr fileset as something "special" and adding custom behavior. I mention in case it is of use to also bump up the depth setting in this special case.

snoopycrimecop · 2022-10-27T01:07:31Z

Conflicting PR. Removed from build OMERO-insight-push#1297. See the console output for more details.
Possible conflicts:

PR Move gateway calls to OMEROGateway #305 dominikl 'Move gateway calls to OMEROGateway'
- src/main/java/org/openmicroscopy/shoola/env/data/OMEROGateway.java
- src/main/java/org/openmicroscopy/shoola/env/data/OmeroImageServiceImpl.java

--conflicts

dominikl · 2022-11-09T13:41:23Z

Sorry, I can't replicate the issue with NIHRTa-001.ome.tiff, same with and without PR, the image is uploaded but then the processing step never stops for whatever (non insight related) reason. But a potential issue is still that scanning is triggered already when a file is added to the queue. It will at least need some kind of indication that something's happening, not just freeze the UI.

Edit: Sorry, I just realised that this PR actually wasn't in the build I tested due to the merge conflict... Will test again, after rebase.

snoopycrimecop · 2022-11-10T01:07:35Z

Conflicting PR. Removed from build OMERO-insight-push#1307. See the console output for more details.
Possible conflicts:

PR Move gateway calls to OMEROGateway #305 dominikl 'Move gateway calls to OMEROGateway'
- src/main/java/org/openmicroscopy/shoola/env/data/OmeroImageServiceImpl.java
- src/main/java/org/openmicroscopy/shoola/env/data/OMEROGateway.java

~~--conflicts~~ Conflict resolved in build OMERO-insight-push#1308. See the console output for more details.

brau-frederic · 2022-11-10T10:13:57Z

Hi, many thanks to this work. Just to mention, if it helps, that I tried an Omero.insight importation on our Omero server (last version I think) of a 327 images z stack after NGFF conversion with NGFF converter from Glencoe and the zar folder was imported. In omero.web 5.13.0 the name of the file was changed to METADATA.ome.xml the image properties seem to be kept but the image is dark, and also in omero.viewer whatever the contrast adjustments. I could try things for you if needed.

joshmoore · 2022-11-10T14:08:38Z

Thanks for the feedback, @brau-frederic! The darkness is likely an issue of there not being a rendering model in OME-XML and therefore no way for Bio-Formats to hand that to OMERO.

cc: @dgault

jburel · 2022-12-08T16:35:06Z

@dominikl @pwalczysko What is the status of this PR? I will prepare a release of omero.insight after the openmicroscopy release

jburel · 2022-12-08T16:36:40Z

src/main/java/org/openmicroscopy/shoola/agents/fsimporter/chooser/ImportDialog.java

@@ -671,6 +677,13 @@ public void actionPerformed(ActionEvent e) {
 		tagsMap = new LinkedHashMap<JButton, TagAnnotationData>();
 		mapAnnotation=new LinkedHashMap<String,List<MapAnnotationData>>();

+		depth = new NumericalTextField();
+		depth.setMinimum(1);
+		depth.setText(System.getProperty("omero.import.depth", "50"));


That property should be put in the config or retrieve from the server if available.

jburel · 2022-12-08T16:37:51Z

src/main/java/org/openmicroscopy/shoola/env/data/model/ImportableObject.java

@@ -225,6 +222,9 @@ public static boolean isArbitraryFile(File f)
 	/** Map of skip options mapping to ImportConfig functions */
 	private Map<String, Object> skipChoices;

+	/** The depth Bioformats uses to find the importable files */
+	private int depth = Integer.valueOf(System.getProperty("omero.import.depth", "4"));


dominikl · 2022-12-08T16:40:14Z

I couldn't replicate the difference with/without PR with this specific NIHRTa-001.ome.tiff image. But the delay when adding files to the import queue could be an issue. I have to try again importing a plate or some image which takes a long time for bioformats to scan.

jburel · 2022-12-08T16:56:21Z

Thanks @dominikl. I don't think we can consider this PR for next release.
We can do a release with that PR when ready

jburel · 2023-01-18T14:48:57Z

--exclude

jburel mentioned this pull request Jul 11, 2022

add Zarr Reader #309

Closed

dominikl force-pushed the depth branch from 57f5eec to 73b8436 Compare July 15, 2022 13:12

dominikl force-pushed the depth branch from 73b8436 to 4c05c4e Compare July 21, 2022 14:50

dominikl and others added 3 commits November 10, 2022 10:58

Add depth option to import dialog

fa43992

add Zarr Reader

0e4e3e4

Count images not files for import progress display

5035ba4

dominikl force-pushed the depth branch from 4c05c4e to 5035ba4 Compare November 10, 2022 10:59

jburel closed this Nov 24, 2022

jburel reopened this Nov 24, 2022

jburel reviewed Dec 8, 2022

View reviewed changes

snoopycrimecop mentioned this pull request Jan 18, 2023

Declare logback-classic as explicit dependency #337

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add depth option to importer and include ZarrReader #310

Add depth option to importer and include ZarrReader #310

dominikl commented Jul 8, 2022 •

edited

Loading

dominikl commented Jul 8, 2022

dominikl commented Jul 13, 2022

dominikl commented Jul 14, 2022

dominikl commented Jul 15, 2022

snoopycrimecop commented Jul 22, 2022 •

edited

Loading

jburel commented Aug 18, 2022

dominikl commented Aug 18, 2022

jburel commented Aug 18, 2022

pwalczysko commented Aug 18, 2022 •

edited

Loading

dominikl commented Aug 19, 2022

dominikl commented Aug 19, 2022

pwalczysko commented Aug 19, 2022 •

edited

Loading

joshmoore commented Sep 12, 2022

snoopycrimecop commented Oct 27, 2022

dominikl commented Nov 9, 2022 •

edited

Loading

snoopycrimecop commented Nov 10, 2022 •

edited

Loading

brau-frederic commented Nov 10, 2022

joshmoore commented Nov 10, 2022

jburel commented Dec 8, 2022

jburel Dec 8, 2022

jburel Dec 8, 2022

dominikl commented Dec 8, 2022

jburel commented Dec 8, 2022

jburel commented Jan 18, 2023

Add depth option to importer and include ZarrReader #310

Are you sure you want to change the base?

Add depth option to importer and include ZarrReader #310

Conversation

dominikl commented Jul 8, 2022 • edited Loading

dominikl commented Jul 8, 2022

dominikl commented Jul 13, 2022

dominikl commented Jul 14, 2022

dominikl commented Jul 15, 2022

snoopycrimecop commented Jul 22, 2022 • edited Loading

jburel commented Aug 18, 2022

dominikl commented Aug 18, 2022

jburel commented Aug 18, 2022

pwalczysko commented Aug 18, 2022 • edited Loading

dominikl commented Aug 19, 2022

dominikl commented Aug 19, 2022

pwalczysko commented Aug 19, 2022 • edited Loading

joshmoore commented Sep 12, 2022

snoopycrimecop commented Oct 27, 2022

dominikl commented Nov 9, 2022 • edited Loading

snoopycrimecop commented Nov 10, 2022 • edited Loading

brau-frederic commented Nov 10, 2022

joshmoore commented Nov 10, 2022

jburel commented Dec 8, 2022

jburel Dec 8, 2022

Choose a reason for hiding this comment

jburel Dec 8, 2022

Choose a reason for hiding this comment

dominikl commented Dec 8, 2022

jburel commented Dec 8, 2022

jburel commented Jan 18, 2023

dominikl commented Jul 8, 2022 •

edited

Loading

snoopycrimecop commented Jul 22, 2022 •

edited

Loading

pwalczysko commented Aug 18, 2022 •

edited

Loading

pwalczysko commented Aug 19, 2022 •

edited

Loading

dominikl commented Nov 9, 2022 •

edited

Loading

snoopycrimecop commented Nov 10, 2022 •

edited

Loading