Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Build] Run Eclipse-SDK aggregator build with multiple threads #2146

Merged

Conversation

HannesWell
Copy link
Member

This hopefully reduces the runtime of the build. Since no tests are run in these builds, I hope there are no interference problems.

@HannesWell HannesWell force-pushed the build-aggregator-parallel branch 10 times, most recently from f954f36 to 29ec169 Compare June 25, 2024 16:03
@HannesWell HannesWell changed the title [Build] Run eclipse sdk aggregator builds in parallel [Build] Run Eclipse-SDK aggregator build with multiple threads Jun 25, 2024
@HannesWell
Copy link
Member Author

/request-license-review

Copy link
Contributor

/request-license-review

License review requests:

After all reviews have concluded, re-run the license-vetting check from the Github Actions web-interface to update its status.

Workflow run (with attached summary files):
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/actions/runs/9665940221

@merks
Copy link
Contributor

merks commented Jun 25, 2024

I think the problem is that some dependencies aren't resolving at all not that there are actual license dependencies with 3rd party libraries:

[INFO] --------------------------------[ pom ]---------------------------------
Warning:  The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
Warning:  o org.eclipse.jdt.ui:org.eclipse.jdt.ui.junit.sampleproject:jar:dist:1.0.0-SNAPSHOT (compile)
Warning:  Try running the build up to the lifecycle phase "package"

@HannesWell
Copy link
Member Author

HannesWell commented Jun 25, 2024

In the verification builds of this PR using -T 1C reduced the runtime of the Maven build to almost 50% of the current time.
Lets try this and see what reduction will be achieved in I-builds and if there are any issues.

I assume the additional Maven-dependency of eclipse.platform.repository to eclipse-junit-tests is completely internal and therefore not a problem.

I think the problem is that some dependencies aren't resolving at all not that there are actual license dependencies with 3rd party libraries:

[INFO] --------------------------------[ pom ]---------------------------------
Warning:  The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
Warning:  o org.eclipse.jdt.ui:org.eclipse.jdt.ui.junit.sampleproject:jar:dist:1.0.0-SNAPSHOT (compile)
Warning:  Try running the build up to the lifecycle phase "package"

This warning is also present for the current master build, therefore I assume it is unrelated.
I think it is because the extra maven dependency was added. I simply hope that @waynebeaton can resolve https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/15495. IIRC such artifacts were added to a special list in the past or the dash license tool was improved to find-out why a reactor project was considered as third-party dependency.

But since we probably cannot anything here, do you mind if I submit this now and we await the resolution of the vetting issue, wherever it happens?

@@ -1,2 +1,2 @@
-XX:+PrintFlagsFinal
-Xmx2G
-Xmx3G
Copy link
Member Author

@HannesWell HannesWell Jun 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest verification build ran out of heap space. Looks like a build with multiple threads needs more RAM (not really a surprise).
Since we are running on the large 8GB executors since #2148, I think 3GB should not be a problem.

@merks
Copy link
Contributor

merks commented Jun 25, 2024

@HannesWell

I don't think we should let glitches like this hold stuff up so I suggest we merge this when the build is good.

@HannesWell
Copy link
Member Author

@HannesWell

I don't think we should let glitches like this hold stuff up so I suggest we merge this when the build is good.

Then lets try this.
From the last runtime of builds in other PRs I'm not so confident anymore that there is really the runtime improvement mentioned above. It might have been heavily influenced by other builds but at least when I checked it my 'test' builds where running alone.

Anyway I think it's worth trying it out and if it eventually does not give sufficient speed-ups or causes too much trouble we can still revert it.

@HannesWell HannesWell merged commit 466dfde into eclipse-platform:master Jun 25, 2024
2 checks passed
@HannesWell HannesWell deleted the build-aggregator-parallel branch June 25, 2024 17:10
@HannesWell
Copy link
Member Author

In the I-build that contained this the Maven build stage took only 24min instead of ~40min. Five days ago, before the signing problem was addressed it took often more than two hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants