CBI Aggregator waits forever because P2 Provisioning Agent is not stopping

[ChrisHolter]

Hi,
I have currently the issue with the latest nightly (but also with older versions, even with the last official release) that the CBI Aggregator is stuck waiting for the P2 Provisioning Agent to stop. So I'd propose to call ProvisioningAgent.stop in BackgroundProvisioningAgent.conditionalStop with a reasonable timeout that not the whole execution is blocked.

"main" #1 prio=5 os_prio=0 cpu=13706.96ms elapsed=413.03s tid=0x00007f0220014180 nid=0x9124 in Object.wait() [0x00007f022677b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on
at org.eclipse.core.internal.jobs.JobManager.join(JobManager.java:1042)
- locked <0x00000006241ad7c8> (a java.util.Collections$SynchronizedSet)
at org.eclipse.equinox.internal.p2.engine.SimpleProfileRegistry.stop(SimpleProfileRegistry.java:981)
at org.eclipse.equinox.internal.p2.core.ProvisioningAgent.stop(ProvisioningAgent.java:138)
at org.eclipse.cbi.p2repo.p2.util.BackgroundProvisioningAgent.conditionalStop(BackgroundProvisioningAgent.java:38)
at org.eclipse.cbi.p2repo.p2.util.BackgroundProvisioningAgent.stop(BackgroundProvisioningAgent.java:60)
- locked <0x000000060bc0bba0> (a org.eclipse.cbi.p2repo.p2.util.BackgroundProvisioningAgent)
at org.eclipse.cbi.p2repo.p2.util.P2Utils.stopProvisioningAgent(P2Utils.java:165)
at org.eclipse.cbi.p2repo.aggregator.engine.Builder.run(Builder.java:1749)
at org.eclipse.cbi.p2repo.aggregator.engine.Builder.run(Builder.java:1764)
at org.eclipse.cbi.p2repo.cli.AbstractCommand.run(AbstractCommand.java:143)
at org.eclipse.cbi.p2repo.cli.Headless.run(Headless.java:166)
at org.eclipse.cbi.p2repo.cli.Headless.start(Headless.java:201)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:568)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
at org.eclipse.equinox.launcher.Main.main(Main.java:1440)

I'm running on a Ubuntu 22.04 system and the behavior is stable reproducable for me.

[merks]

This is used regularly here without a problem:

https://ci.eclipse.org/simrel/job/simrel.runaggregator.pipeline/

Also here:

https://ci.eclipse.org/releng/view/Publish%20to%20Maven/job/CBIaggregator/

The version of the job manager looks to be not the latest one:

And there has been quite a bit of changes to the job manager in recent months.

I kicked off a build that should build with the latest JobManager

https://ci.eclipse.org/cbi/view/p2RepoRelated/job/cbi.p2repo.aggregator-build/49/

So please try the latest nightly build.

[ChrisHolter]

Thx (also for the quick reaction!), I'll give it a try

[ChrisHolter]

I tried, but the result is the same. But I think I've got an idea where the problem is coming from. As this is running in an internal ci environment the machine as no direct access to the internet but only via proxy. I see in the jstack a lot of canceled p2 jobs that are in timed_waiting as the are canceled due to the missing internet connection (just saw in the logs, they try to download an xml from the official p2 repo, but we use only a local mirror).

Is there a way to give cbi the proxy settings (in a headless env)?

[merks]

Under the p2 covers, ECF is used which is currently using using httpclient5 so there are likely ECF or httpclient system properties you can set. I found this old thing:

https://wiki.eclipse.org/ECF_Filetransfer_Support_for_NTLMv2_Proxies

I've used -Dorg.eclipse.ecf.core.util.traceAll=true to get more information about the network traffic. I'd have to Google to find more information like this:

https://stackoverflow.com/questions/18147722/how-to-set-the-proxy-configuration-for-a-headless-eclipse-application-on-windows

[ChrisHolter]

Hm, unfortunately setting the proxy does not work.
Nevertheless I found a (ugly and dirty) way to work around that by killing the aggregator as soon as the p2.index is created...

[merks]

Sorry I couldn't be more helpful. 😞