[WIP] Faster rpm build #436

jrafanie · 2024-01-09T17:52:28Z

Here are some very WIP small and larger changes to try to speed up the rpm build.

My plan is to extract things we like and get them in.

disable pythondeps - wastes 45 minutes locally... not sure if we need it, and definitely shouldn't be changing all that often. We can possibly run this once per major change.
final commit is just debug logging output

EDIT:
Extracted to #437

.build-id artifacts - To extract, pretty minor... tens of files are eliminated from the tarballs
use built-in vcs to catch .git, .gitignore and friends instead of hardcoded list - seems safe to extract and it's super minor
faster brp-strip... was 33 minutes... now around 2. Uses new version of the file.

This was externalized in subsequent versions of rpmbuild: rpm-software-management/rpm#1607 I believe we can manually run this if needed to build any python provides, requires, etc. whenever we pull in new python stuff.

Fryguy · 2024-01-09T17:59:36Z

Another option might be to just install rpmbuild 4.19 directly to avoid the curl and os macro changes. Not sure how hard that is or not

Fryguy · 2024-01-09T17:56:14Z

lib/manageiq/rpm_build/generate_tar_files.rb

@@ -22,14 +33,14 @@ def create_gemset_tarball
        plugin_index.write(plugin_index.read.gsub(BUILD_DIR.join("manageiq").to_s, '/var/www/miq/vmdb'))

        name = "gemset"
-        shell_cmd("tar -C #{BUILD_DIR} -zcf #{tar_full_path(name)} -X #{exclude_file(name)} #{tar_basename(name)}")
+        shell_cmd("tar -C #{BUILD_DIR} -zcf #{tar_full_path(name)} --exclude-vcs -X #{exclude_file(name)} #{tar_basename(name)}")


This could be a problem if it removes more than the .git dir from git based gems within bundler (well have to check if that matters or not, and it's possible we already had those removed)

Good point. I really want to see if I can run rpmdiff before and after this PR. I will not extract that until I can do that.

Fryguy · 2024-01-09T17:57:42Z

rpm_spec/brp-strip-with-ruby-js-filtering

@@ -0,0 +1,50 @@
+#!/bin/sh
+# From https://github.com/rpm-software-management/rpm/blob/rpm-4.19.1-release/scripts/brp-strip


As discussed, since this is a GPL file let's try curl and patch instead

yes, I think it's similar to pulling down the source for repmgr in https://github.com/ManageIQ/manageiq-rpm_build/blob/master/packages/repmgr13/repmgr.spec#L24. Then, we just have patch file with the minor change to add rb/js filtering and we can apply that on top.

jrafanie · 2024-01-09T18:01:09Z

Another option might be to just install rpmbuild 4.19 directly to avoid the curl and os macro changes. Not sure how hard that is or not

My assumption is we'd need to add a new dnf repo to install it as it will likely pull in lots of other dependencies. We're using the latest rpm/rpmbuild available in our ubi8 repo.

Fryguy · 2024-01-09T18:16:42Z

We also discussed a possible rm -rf of all python test directories. While that might not change much since disabling pythondeps is the bigger change, it might have other benefits, such as

Smaller RPM files
Even less to traverse as part of brp-strip
Removing files that are subject to license auditing (I actually get these frequently because some test suites have copies of GPL files within them)

Fryguy · 2024-01-09T20:42:30Z

Dockerfile

+RUN curl -o brp-strip https://raw.githubusercontent.com/rpm-software-management/rpm/rpm-4.19.1-release/scripts/brp-strip
+
+RUN chmod +x brp-strip
+
+RUN patch -p2 < Add-js-rb-filtering-on-top-of-4.19.1.patch && \
+  ls -l /usr/lib/rpm/brp-strip && \
+  mv brp-strip /usr/lib/rpm/brp-strip && \
+  ls -l /usr/lib/rpm/brp-strip && \
+  rm -f Add-js-rb-filtering-on-top-of-4.19.1.patch


Nice! A few comments here:

The patch file should probably live in the container-assets directory, as opposed to the root. That way you can reuse the COPY . /build_scripts command which already copies the container-assets directory.

Prefer curling the file directly into /usr/lib/rpm/brp-strip, overwriting it, and then patching /usr/lib/rpm/brp-strip directly.

No need to remove the patch file (it doesn't really get deleted from the container anyway)

Smash all of the RUN commands into a single line.

From: https://github.com/rpm-software-management/rpm/blob/rpm-4.19.1-release/scripts/brp-strip Added js, rb files to the go filtering in upstream version Curl brp-strip 4.19.1 and patch it at image build time

miq-bot · 2024-01-09T20:55:26Z

Some comments on commits jrafanie/manageiq-rpm_build@215db4c~...79931aa

lib/manageiq/rpm_build/generate_tar_files.rb

⚠️ - 11 - Detected puts. Remove all debugging statements.
⚠️ - 13 - Detected puts. Remove all debugging statements.
⚠️ - 15 - Detected puts. Remove all debugging statements.
⚠️ - 17 - Detected puts. Remove all debugging statements.
⚠️ - 19 - Detected puts. Remove all debugging statements.
⚠️ - 21 - Detected puts. Remove all debugging statements.
⚠️ - 23 - Detected puts. Remove all debugging statements.
⚠️ - 25 - Detected puts. Remove all debugging statements.

miq-bot · 2024-01-09T20:55:29Z

Checked commits jrafanie/manageiq-rpm_build@215db4c~...79931aa with ruby 2.7.8, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
3 files checked, 9 offenses detected

lib/manageiq/rpm_build/generate_tar_files.rb

❗ - Line 11, Col 1 - Layout/IndentationWidth - Use 2 (not -6) spaces for indentation.
❗ - Line 11, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 13, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 15, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 17, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 19, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 21, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 23, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
❗ - Line 25, Col 1 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.

Fryguy · 2024-01-09T21:06:08Z

Dockerfile

+RUN curl -o /usr/lib/rpm/brp-strip https://raw.githubusercontent.com/rpm-software-management/rpm/rpm-4.19.1-release/scripts/brp-strip
+
+RUN chmod +x /usr/lib/rpm/brp-strip
+
 COPY . /build_scripts

+RUN cd /usr/lib/rpm/ && \
+  patch -p2 < /build_scripts/container-assets/Add-js-rb-filtering-on-top-of-4.19.1.patch
+


You can still smash all these together further

Suggested change

RUN curl -o /usr/lib/rpm/brp-strip https://raw.githubusercontent.com/rpm-software-management/rpm/rpm-4.19.1-release/scripts/brp-strip

RUN chmod +x /usr/lib/rpm/brp-strip

COPY . /build_scripts

RUN cd /usr/lib/rpm/ && \

patch -p2 < /build_scripts/container-assets/Add-js-rb-filtering-on-top-of-4.19.1.patch

COPY . /build_scripts

RUN curl -o /usr/lib/rpm/brp-strip https://raw.githubusercontent.com/rpm-software-management/rpm/rpm-4.19.1-release/scripts/brp-strip && \

chmod +x /usr/lib/rpm/brp-strip && \

cd /usr/lib/rpm/ && \

patch -p2 < /build_scripts/container-assets/Add-js-rb-filtering-on-top-of-4.19.1.patch

miq-bot · 2024-01-18T19:40:25Z

This pull request is not mergeable. Please rebase and repush.

Fryguy · 2024-01-18T22:48:27Z

Merged #437, so this needs a rebase.

miq-bot · 2024-04-22T00:00:13Z

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s).

miq-bot · 2024-07-29T00:00:25Z

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s).

jrafanie added 3 commits January 9, 2024 12:40

Don't create .build-id directory artifacts in the rpm

215db4c

Use built-in vcs for excluding version control files

7ef750e

Turn off python dependency checking

6426786

This was externalized in subsequent versions of rpmbuild: rpm-software-management/rpm#1607 I believe we can manually run this if needed to build any python provides, requires, etc. whenever we pull in new python stuff.

miq-bot added the wip label Jan 9, 2024

Fryguy reviewed Jan 9, 2024

View reviewed changes

jrafanie force-pushed the faster_rpm_build branch from d8e8854 to 54175d1 Compare January 9, 2024 20:38

Fryguy reviewed Jan 9, 2024

View reviewed changes

jrafanie added 2 commits January 9, 2024 15:53

Use 4.19.1 brp-strip with xargs parallel processing and filtering

4f3ca65

From: https://github.com/rpm-software-management/rpm/blob/rpm-4.19.1-release/scripts/brp-strip Added js, rb files to the go filtering in upstream version Curl brp-strip 4.19.1 and patch it at image build time

WIP Add debug logging and enforce x86_64 platform

79931aa

jrafanie force-pushed the faster_rpm_build branch from 54175d1 to 79931aa Compare January 9, 2024 20:54

Fryguy reviewed Jan 9, 2024

View reviewed changes

jrafanie mentioned this pull request Jan 10, 2024

Faster rpm build step 1 #437

Merged

miq-bot added the unmergeable label Jan 18, 2024

miq-bot added the stale label Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Faster rpm build #436

[WIP] Faster rpm build #436

jrafanie commented Jan 9, 2024 •

edited

Loading

Fryguy commented Jan 9, 2024

Fryguy Jan 9, 2024

jrafanie Jan 9, 2024

Fryguy Jan 9, 2024

jrafanie Jan 9, 2024

jrafanie commented Jan 9, 2024

Fryguy commented Jan 9, 2024 •

edited

Loading

Fryguy Jan 9, 2024 •

edited

Loading

miq-bot commented Jan 9, 2024

miq-bot commented Jan 9, 2024

Fryguy Jan 9, 2024

miq-bot commented Jan 18, 2024

Fryguy commented Jan 18, 2024

miq-bot commented Apr 22, 2024

miq-bot commented Jul 29, 2024

		@@ -0,0 +1,50 @@
		#!/bin/sh
		# From https://github.com/rpm-software-management/rpm/blob/rpm-4.19.1-release/scripts/brp-strip

[WIP] Faster rpm build #436

Are you sure you want to change the base?

[WIP] Faster rpm build #436

Conversation

jrafanie commented Jan 9, 2024 • edited Loading

Fryguy commented Jan 9, 2024

Fryguy Jan 9, 2024

Choose a reason for hiding this comment

jrafanie Jan 9, 2024

Choose a reason for hiding this comment

Fryguy Jan 9, 2024

Choose a reason for hiding this comment

jrafanie Jan 9, 2024

Choose a reason for hiding this comment

jrafanie commented Jan 9, 2024

Fryguy commented Jan 9, 2024 • edited Loading

Fryguy Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

miq-bot commented Jan 9, 2024

miq-bot commented Jan 9, 2024

Fryguy Jan 9, 2024

Choose a reason for hiding this comment

miq-bot commented Jan 18, 2024

Fryguy commented Jan 18, 2024

miq-bot commented Apr 22, 2024

miq-bot commented Jul 29, 2024

jrafanie commented Jan 9, 2024 •

edited

Loading

Fryguy commented Jan 9, 2024 •

edited

Loading

Fryguy Jan 9, 2024 •

edited

Loading