Problems when deploying web service (NLprot) #44

martavillegas · 2018-06-20T14:00:46Z

We have deployed a WS as wrapper for a command line tool (NLProt) that reads PDF and returns a JSON (we understood that XMI was recommended but not compulsory).

you can run it:
curl -F [email protected] inb.bsc.es/service/openminted/process

input-test.pdf

When running the app in test.openminted we get the following message:

Failed
Unable to locate named workflow

gkirtzou · 2018-06-20T14:24:58Z

@martavillegas
the error you have mentioned is a know issue in the test.openminted.eu for the software that is registered as application (and is reported to the responsible tech team in the redmine, see issue 840)

Independent of that, we need first to insure that the software you provide is OMTD compliant to the specifications and guidelines for web services. The OMTD specifications and guidelines for a web services require that both the input and output are corpora of UIMA CAS XMI files. Furthermore, in order to deserialize correct the output of the web service it requires a typesystem, which is available in the maven central repository.

Given the metadata you have provided us, in order for your software to run within OMTD platform, you would need to make it OMTD compliant. In other words,

you need to generate UIMA CAS XMI files as output, not JSON
you need to define what typesystem the XMI CAS file use so as to be deserialized correctly and
make that typesystem available via the maven central repository.

For more details see here:

Guidelines : https://guidelines.openminted.eu/guidelines_for_providers_of_sw_resources/sharing-web-services.html
Specifications : https://openminted.github.io/releases/processing-web-services/1.0.0/specification

martavillegas · 2018-06-20T14:35:03Z

Thanks! just two questions:

we understood that apps differ from components in that they did not to consume/return XMI. Is this right? (that's why we changed to app)
can we consume PDF files directly?

gkirtzou · 2018-06-21T09:54:39Z

@martavillegas To answer your first question what is the difference between a component and an application and what an application or a component can consume/produce, the OMTD interoperability specification say that :

components that support (i.e. produce & consume) data in the XML Metadata Interchange (XMI) format; in the case of components that support other formats, appropriate readers and writers for converting XMI messages from and to the components’ format must also be provided

applications that support as input file formats that are used for publications (e.g. PDF, PubMed XML, plain text etc.) or XMI.

In general, both components and applications can consume/produce whatever they want. For the components it is advisable to offer other components converters (readers/writers from/to XMI) for interoperability reasons. ie to allow a TDM expert to combine components from different providers/architectures.

Note that an application can be create by two different ways:

Via the Workflow editor, where a TDM expert combines interoperable components or
By registering a single component as an application.

In this second case, where a single component is registered as application, the way the OMTD workflow service handles them makes the assumption that the application consumes XMI, and thus always executes before the component-application the following two components:

the omtd importer so as to fetch the data from the OMTD store to OMTD workflow executor
and the PDF reader component to transform PDF files to XMI.

As for the second question, since you have decided to offer your software as a web service, you need to make your component compliant not to the generic specifications, but to the web service ones. Which means that you need to:

ensure that it uses UIMA CAS XMI messaging system as specified at https://openminted.github.io/releases/processing-web-services/1.0.0/specification and
upload the typesystem that the web service deploys in Maven.

So to answer your second question, you cannot read PDF files directly if you are offering your software as web service. If you want to read PDF files directly, you could register your software as component of docker image and create an application using the workflow editor so that you can execute your software. The instructions how you can build a workflow can be found here:
https://openminted.github.io/releases/workflow-editor/1.0.0/workflow

martavillegas · 2018-07-06T14:48:58Z

Hi Katerina, we are back....

Our NLProt service reads and delivers MXI with DKPro Core type system. in this case do we need to uoload the typesystem (we understand it is already uploaded).

Thanks!

gkirtzou · 2018-07-09T08:15:03Z

@martavillegas

Our NLProt service reads and delivers MXI with DKPro Core type system. in this case do we need to uoload the typesystem (we understand it is already uploaded).

That's great! In order to be compliant you need to update the metadata of the application/component by adding the typesystem details, as the guidelines mention:

Please, note that in the metadata record, you must include
(1) the URL where the web service can be executed in the element distributionLocation, e.g.
<distributionLocation>http://nactem.ac.uk/api/openminted/chebi</distributionLocation>
(2) the typesystem details in the the module componentDependencies/typesystem and more
specifically in the elements resourceName and resourceIdentifier (Maven coordinates), e.g.
<resourceName>Chebi Curation Type System</resourceName>
and
<resourceIdentifier resourceIdentifierSchemeName="maven">mvn:uk.ac.nactem.uima:ChebiCurationTypeSystem:0.1</resourceIdentifier>

Then you could try again to call you web service and see whether it works as expected.

martavillegas · 2018-07-11T11:00:25Z

Sorry Katerina, but in case we use default type system, namely:

xmlns:type="http:///de/tudarmstadt/ukp/dkpro/core/api/metadata/type.ecore"
xmlns:type2="http:///de/tudarmstadt/ukp/dkpro/core/api/segmentation/type.ecore"

do we need to upload it in Maven?

reckart · 2018-07-11T11:18:07Z

The DKPro Core type system is split over various "API" artifacts on Maven: http://search.maven.org/#search%7Cga%7C1%7Cde.tudarmstadt.ukp.dkpro.core.api

galanisd · 2018-07-11T11:28:37Z

@reckart just said the one thing that I wanted to say

The DKPro Core type system is split over various "API" artifacts on Maven: http://search.maven.org/#search%7Cga%7C1%7Cde.tudarmstadt.ukp.dkpro.core.api

The typesystem that you use is already uploaded to maven central. So you have just specify in the metadata its coordinates. E.g. like

a. chebi web service

https://services.openminted.eu/landingPage/application/11dc2e5e-1cd8-40bf-a46f-df17664b3c77
->
mvn:uk.ac.nactem.uima:ChebiCurationTypeSystem:0.1

https://mvnrepository.com/artifact/uk.ac.nactem.uima/ChebiCurationTypeSystem/0.1

b. the OGER web service
https://services.openminted.eu/landingPage/application/71345d18-297f-4ac5-b4de-38ef3cacbe75
->
mvn:de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.api.ner-asl:1.9.1

https://mvnrepository.com/artifact/de.tudarmstadt.ukp.dkpro.core/de.tudarmstadt.ukp.dkpro.core.api.ner-asl/1.9.1

martavillegas · 2018-07-11T14:01:36Z

Thanks guys with the types!!

I updated the metadata with types (two coordinated from maven) but now when I try to run the application I get the following error message:

System error executing service (Server responded: #: 2 schema violations found)

gkirtzou · 2018-07-12T06:50:38Z

@martavillegas Would be possible to send the xml file of the metadata with us, to check them?

martavillegas · 2018-07-12T08:34:48Z

Sure!
2c448b7c-6fe2-451f-b9d3-1426330251ef (2).zip

galanisd · 2018-07-12T10:34:43Z

We have checked the workflow engine (Galaxy); it seems that the Galaxy wrapper that is required has been generated as it should. The wrapper contains the command that will be executed from the generic web service executor that we have designed. The command seems correct and the executor was designed to support more that one typesystems. I remember also that I have tested it in such a case and it worked.

Also we checked Galaxy (with @gkirtzou) and it seems that nothing has been executed the last few days. So I assume (not sure 100%) that the error in the Registry side, in particular in the point where the workflow execution is initiated. I think that we should ask @antleb.

Please could you test the same app also in services.openminted.eu? test.openminted usually is not as stable as services.

Dimitris

martavillegas · 2018-07-12T11:25:41Z

I registered the tool in services. When I run it I get:

There was a problem running the application. Try again in a while. (Unable to locate named workflow)

gkirtzou · 2018-07-12T13:38:08Z

@martavillegas This is an know problem that used to be in test platform, maybe has been moved in the services as well (see here : http://redmine.openminted.eu/issues/840)

You can avoid this, if you register your software as a component and create a simple workflow using the workflow editor by adding the omtd_importer and the pdf reader (with parameter pattern="**/*.pdf"). In other words, by creating the parts that the platform add by itself if you register something as application.
For more details how you can create a workflow please check here:
https://openminted.github.io/releases/workflow-editor/1.0.0/workflow

martavillegas · 2018-07-13T10:22:59Z

Ok, I registered the service as a component. When I go to galaxy editor I have one problem:
When I search for NLProt I get one result ("NLProt by kat") but if I browse I find three NLProts, the 'one by kat' and two identical NLProts (mine I guess)

martavillegas · 2018-07-13T10:55:39Z

I created a workflow, with PDFreader and NLProt.
When I go to apps to run it I find two NLProt applications....

I tried to run the first one and got the message:
There was a problem running the application. Try again in a while. (Unable to locate named workflow)
I received an email message with a link. When I follow the link I get a white screen with this message:
{"url":"http://services.openminted.eu/omtd-registry/mySpace/operations","error":"No handler found for GET /omtd-registry/mySpace/operations"}

then I tried to run the second one and it is still running (no error messages but running for a long time. I used the 2 pdf files corpus used for testing with Chebi web service

gkirtzou · 2018-07-13T11:06:04Z

@martavillegas I have delete my private NLprot component ("NLProt by kat") so that you don't get confused.

I tried to run the first one and got the message:
There was a problem running the application. Try again in a while. (Unable to locate named workflow)
I received an email message with a link. When I follow the link I get a white screen with this message:
{"url":"http://services.openminted.eu/omtd-registry/mySpace/operations","error":"No handler found for GET /omtd-registry/mySpace/operations"}

I don't know what is causing this error. I would suggest to report it to the redmine, so that someone could have a look at it

then I tried to run the second one and it is still running (no error messages but running for a long time. I used the 2 pdf files corpus used for testing with Chebi web service

I found a workflow with nlprot in the webservice engine, but it hasn't load any documents. Are you sure you followed the guidelines I sent you? Meaning is the first step of the workflow omtdImporter, the second step is Pdf Reader with parameter patterns="**/.pdf" and then NLprot????

martavillegas · 2018-07-13T12:53:31Z

Hi,

sorry I'm afraid the initial workflow was wrong... I edited it again (following your instructions) and I get this message, thanks:

There was a problem running the application. Try again in a while. (a workflow state is in error: b445e1e784215eb6/a4128fc2307c4024/f44bd52ac458d92a/5c4779f75d735313)

gkirtzou · 2018-07-13T13:04:05Z

@martavillegas did you initialize the parameter patterns with **/*.pdf in the PDFreader? Because I found the workflow, but the PDFReader produced zero xmi files.

martavillegas · 2018-07-13T13:33:41Z

I get the same error.....

gkirtzou · 2018-07-13T13:44:54Z

The log from pdf reader shows that the pattern is patterns="**/.pdf", not **/*.pdf. Thus zero files matched the pattern and the workflow ended up with an error. Maybe you added it, but sometimes, galaxy editor/registry don't save it correctly. Could you verify by opening the workflow that the pattern is set correctly? And try again?

martavillegas · 2018-07-13T13:54:17Z

I cannot edit the parameters. When I try to edit the workflow, galaxy does not display the parameters on the right panel. What I do is to delete the pdfreader and add another one, then I can edit. The screenshot shows the empty pannel....

I'm gona delete this workflow and create a new one ...

martavillegas · 2018-07-13T14:04:46Z

I sent you the wrong schreenshot... the thing is that when I try to edit an existing workflow I get nothing on the right panel....

martavillegas · 2018-07-13T14:06:26Z

Sorry, it failed again with the newly create workflow. I got this message:

There was a problem running the application. Try again in a while. (a workflow state is in error: 5b590cfdc83940a2/c5e4e2683b6e9022/d6f6ae8e19773cf9/349ddf9556a98b88)

gkirtzou · 2018-07-13T14:17:34Z

PDF reader worked this time, so it produced XMI files from the PDFs. Below you would find the log from the omtd-webservice client. It seems that there is a problem with the webservice URI. It is correct?

Executing..
========================
className:null
inputDir:tmp
outputDir:/srv/galaxy/database/jobs_directory/006/6080/working/out/
**parameters
wsurl:http://inb.bsc.es/service/openminted
typesystems:de.tudarmstadt.ukp.dkpro.core_de.tudarmstadt.ukp.dkpro.core.api.metadata-asl_1.9.2
========================
========================






2018-07-13 14:04:34.927  INFO 53 --- [           main] d.t.ukp.dkpro.core.io.xmi.XmiReader      : Scanning [file:/opt/omtd-component-executor/scripts/working/tmp/]
2018-07-13 14:04:34.932  INFO 53 --- [           main] d.t.ukp.dkpro.core.io.xmi.XmiReader      : Found [2] resources to be read
2018-07-13 14:04:36.142 ERROR 53 --- [           main] o.a.u.a.i.PrimitiveAnalysisEngine_impl   : Exception occurred

org.apache.uima.analysis_engine.AnalysisEngineProcessException: Annotator processing failed.    
	at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:416) [uimaj-core-2.10.2.jar!/:2.10.2]
	at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.processAndOutputNewCASes(PrimitiveAnalysisEngine_impl.java:318) [uimaj-core-2.10.2.jar!/:2.10.2]
	at org.apache.uima.analysis_engine.impl.AnalysisEngineImplBase.process(AnalysisEngineImplBase.java:269) [uimaj-core-2.10.2.jar!/:2.10.2]
	at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:342) [uimafit-core-2.4.0.jar!/:2.4.0]
	at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:226) [uimafit-core-2.4.0.jar!/:2.4.0]
	at eu.openminted.component.webservice.WebServiceClient.run(WebServiceClient.java:51) [classes!/:0.0.1-SNAPSHOT]
	at eu.openminted.component.webservice.executor.WSExecutor.run(WSExecutor.java:45) [classes!/:0.0.1-SNAPSHOT]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
	at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
	at eu.openminted.component.webservice.executor.WSExecutor.main(WSExecutor.java:30) [classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:521) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: URI is not absolute
	at java.net.URI.toURL(URI.java:1088) ~[na:1.8.0_161]
	at org.springframework.http.client.SimpleClientHttpRequestFactory.createRequest(SimpleClientHttpRequestFactory.java:137) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
	at org.springframework.http.client.support.HttpAccessor.createRequest(HttpAccessor.java:77) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:615) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
	at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:287) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
	at eu.openminted.remoteexecution.client.RemoteComponent.waitForResult(RemoteComponent.java:112) ~[omtd-remote-execution-client-0.0.1-SNAPSHOT.jar!/:na]
	at eu.openminted.remoteexecution.client.RemoteComponent.process(RemoteComponent.java:66) ~[omtd-remote-execution-client-0.0.1-SNAPSHOT.jar!/:na]
	at org.apache.uima.analysis_component.JCasAnnotator_ImplBase.process(JCasAnnotator_ImplBase.java:48) ~[uimaj-core-2.10.2.jar!/:2.10.2]
	at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:401) [uimaj-core-2.10.2.jar!/:2.10.2]
	... 19 common frames omitted

martavillegas · 2018-07-13T14:24:53Z

will check with Miguel.... thanks!

openminted/Open-Call-Discussions#44 (comment)

martavillegas · 2018-07-17T12:57:24Z

Hi, I'm here again ....
Miguel, fixed the "not absolute URIs" problem but it seems something is wrong. I just run the application and I get an error:

There was a problem running the application. Try again in a while. (a workflow state is in error: 5b590cfdc83940a2/358acff37bc4d34d/6ef404cf6a7da0be/8b6129c80e356c68)

martavillegas · 2018-07-19T13:15:39Z

@gkirtzou could you check our previous post? thanks!

gkirtzou · 2018-07-20T07:06:37Z

@martavillegas Sorry I didn't see your comment on time. In the workflow service engine, I found that you tried to run your workflow 2 days again and the workflow seems to have stacked at the NLProt component. Unfortunately, I couldn't find any log in the workflow engine slaves, that the docker (OMTD web service client) actually ran. Could you please give it another try to today and let me know, so I could check?

Also is the declared URL in the metadata the correct one? Or does it need to update?

martavillegas · 2018-07-20T14:05:17Z

Hi @gkirtzou thanks. I changed the URL and deleted the workflow just in case. When the workflow editor is back I'll build a new workflow and try again n(. Let you know... Thanks

martavillegas · 2018-07-24T15:04:18Z

Hi @gkirtzou ,still getting the "Sorry, our editor is down at the moment, please try again later." message when trying to build a workflow. Did I miss anything?

gkirtzou · 2018-07-25T07:03:01Z

@martavillegas I confirm that the workflow editor is down. I have no idea why it is not working. Please contact @antleb who is responsible for setting up the production site.

martavillegas · 2018-07-25T07:34:32Z

@antleb it seems the workflow editor is down (has been down since last Friday). Thanks

antleb · 2018-07-25T07:35:42Z

We are looking into it. I’ll let you know when it’s back online.

…

On 25 Jul 2018, at 10:34, martavillegas ***@***.***> wrote: @antleb <https://github.com/antleb> it seems the workflow editor is down (has been down since last Friday). Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALr2wgP3LCzCsbLqfKpbdUwQix04oGWks5uKB-JgaJpZM4UvVSV>.

galanisd · 2018-07-25T07:41:50Z

The Galaxy editor instance is up.
I think it has to do with the communication between Registry & Editor instance.

martavillegas · 2018-07-25T07:54:15Z

@galanisd @antleb I still get the same error message (at 09:53)

Sorry, our editor is down at the moment, please try again later.

greenwoodma · 2018-07-25T08:05:04Z

Could it be that someone has started implementing the proxy rules to block access to the rest of the Galaxy instance and the editor has been accidentally included? This would explain why it's up but not embedded in the UI.

martavillegas · 2018-07-31T07:21:01Z

Hi, I still get the same error message when trying to use the galaxy editor: "Sorry, our editor is down at the moment, please try again later.".

galanisd assigned antleb and courado Jun 20, 2018

mimame added a commit to inab/openminted-service that referenced this issue Jul 13, 2018

openminted-service: Fix not absolute URIs

8f2cf68

openminted/Open-Call-Discussions#44 (comment)

Problems when deploying web service (NLprot) #44

Problems when deploying web service (NLprot) #44

Comments

martavillegas commented Jun 20, 2018

gkirtzou commented Jun 20, 2018

martavillegas commented Jun 20, 2018

gkirtzou commented Jun 21, 2018

martavillegas commented Jul 6, 2018

gkirtzou commented Jul 9, 2018

martavillegas commented Jul 11, 2018

reckart commented Jul 11, 2018

galanisd commented Jul 11, 2018

martavillegas commented Jul 11, 2018

gkirtzou commented Jul 12, 2018

martavillegas commented Jul 12, 2018

galanisd commented Jul 12, 2018 • edited Loading

martavillegas commented Jul 12, 2018

gkirtzou commented Jul 12, 2018

martavillegas commented Jul 13, 2018

martavillegas commented Jul 13, 2018

gkirtzou commented Jul 13, 2018

martavillegas commented Jul 13, 2018

gkirtzou commented Jul 13, 2018

martavillegas commented Jul 13, 2018

gkirtzou commented Jul 13, 2018

martavillegas commented Jul 13, 2018

martavillegas commented Jul 13, 2018

martavillegas commented Jul 13, 2018

gkirtzou commented Jul 13, 2018

martavillegas commented Jul 13, 2018

martavillegas commented Jul 17, 2018

martavillegas commented Jul 19, 2018

gkirtzou commented Jul 20, 2018

martavillegas commented Jul 20, 2018

martavillegas commented Jul 24, 2018

gkirtzou commented Jul 25, 2018

martavillegas commented Jul 25, 2018

antleb commented Jul 25, 2018 via email

galanisd commented Jul 25, 2018

martavillegas commented Jul 25, 2018

greenwoodma commented Jul 25, 2018

martavillegas commented Jul 31, 2018

galanisd commented Jul 12, 2018 •

edited

Loading