Add tools to assemble DEB packages (#96)

* Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script
wazuh · Jan 4, 2024 · c466c7c · c466c7c
1 parent 53e33e9
commit c466c7c
Show file tree

Hide file tree

Showing 9 changed files with 439 additions and 186 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -50,7 +50,7 @@ jobs:
         exclude:
           # skip arm64 until we have arm runners
           - architecture: arm64
-          - distribution: [tar, deb] # Exclude deb assembly until it's implemented
+          - distribution: tar
 
     uses: ./.github/workflows/r_assemble.yml
     with:

diff --git a/.github/workflows/r_assemble.yml b/.github/workflows/r_assemble.yml
@@ -39,6 +39,11 @@ jobs:
           name: ${{ inputs.min }}
           path: artifacts/dist
 
+      - name: Provision
+        if: ${{ inputs.distribution == 'deb' }}
+        run: |
+          sudo bash scripts/provision.sh
+
       - name: Run `assemble.sh`
         run: |
           bash scripts/assemble.sh -v ${{ vars.OPENSEARCH_VERSION }} -p linux -a ${{ inputs.architecture }} -d ${{ inputs.distribution }}
@@ -55,4 +60,3 @@ jobs:
           name: ${{ inputs.name }}
           path: artifacts/dist/${{ steps.get_name.outputs.name }}
           if-no-files-found: error
-
diff --git a/...azuh-indexer-performance-analyzer.service → ...azuh-indexer-performance-analyzer.service b/...azuh-indexer-performance-analyzer.service → ...azuh-indexer-performance-analyzer.service
diff --git a/distribution/packages/src/deb/Makefile b/distribution/packages/src/deb/Makefile
@@ -0,0 +1,19 @@
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# deb opensearch Makefile
+
+all: install
+
+install:
+	./debmake_install.sh $(CURDIR)
+
+clean: ;
+
+distclean: clean
+
+.PHONY: all clean distclean install
diff --git a/distribution/packages/src/deb/debmake_install.sh b/distribution/packages/src/deb/debmake_install.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# debmake opensearch install script
+
+set -ex
+
+if [ -z "$1" ]; then
+    echo "Missing curdir path"
+    exit 1
+fi
+
+curdir=$1
+product_dir=/usr/share/wazuh-indexer
+# config_dir=/etc/wazuh-indexer
+data_dir=/var/lib/wazuh-indexer
+log_dir=/var/log/wazuh-indexer
+pid_dir=/var/run/wazuh-indexer
+buildroot=${curdir}/debian/wazuh-indexer
+
+# Create necessary directories
+mkdir -p "${buildroot}"
+mkdir -p "${buildroot}${pid_dir}"
+mkdir -p "${buildroot}${product_dir}/plugins"
+
+# Install directories/files
+cp -a "${curdir}"/etc "${curdir}"/usr "${curdir}"/var "${buildroot}"/
+chmod -c 0755 "${buildroot}${product_dir}"/bin/*
+if [ -d "${buildroot}${product_dir}"/plugins/opensearch-security ]; then
+    chmod -c 0755 "${buildroot}${product_dir}"/plugins/opensearch-security/tools/*
+fi
+
+# Symlinks (do not symlink config dir as security demo installer has dependency, if no presense it will switch to rpm/deb mode)
+ln -s ${data_dir} "${buildroot}${product_dir}/data"
+ln -s ${log_dir} "${buildroot}${product_dir}/logs"
+
+# Change Permissions
+chmod -Rf a+rX,u+w,g-w,o-w "${buildroot}"/*
+
+exit 0
diff --git a/docker/README.md b/docker/README.md
@@ -1,4 +1,4 @@
-# Frontend development environments
+# Indexer development environments
 
 Install [Docker Desktop][docker-desktop] as per its instructions, available for Windows, Mac
 and Linux (Ubuntu, Debian & Fedora).

diff --git a/scripts/README.md b/scripts/README.md
@@ -2,9 +2,9 @@
 
 The packages' generation process consists on 2 steps:
 
-* **Build**: compiles the Java application and bundles it into a package.
-* **Assembly**: uses the package from the previous step and inflates it with plugins and 
-configuration files, ready for production deployment.
+- **Build**: compiles the Java application and bundles it into a package.
+- **Assembly**: uses the package from the previous step and inflates it with plugins and
+  configuration files, ready for production deployment.
 
 We usually generate the packages using GitHub Actions, however, the process is designed to
 be independent enough for maximum portability. GitHub Actions provides infrastructure, while
@@ -27,7 +27,6 @@ act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts
 [Build slim packages/build] 🏁  Job succeeded
 ```
 
-
 #### Running in Docker
 
 Within the [Docker environment](../docker):
@@ -38,60 +37,162 @@ bash scripts/build.sh -v 2.11.0 -s false -p linux -a {x64|arm64} -d {rpm|deb|tar
 
 The generated package is sent to `artifacts/`
 
-
 ## Assemble
 
-<!-- 
+<!--
 ### TAR
-### DEB
 -->
 
+### DEB
+
+The script will:
+
+- Extract the deb package using `ar` and `tar` tools.
+
+  > By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`. The script takes care of creating the required folder structure, copying also the min package and the Makefile.
+
+  Current folder loadout at this stage:
+
+  ```
+  artifacts/
+  |-- dist
+  |   |-- wazuh-indexer-min_4.9.0_amd64.deb
+  `-- tmp
+      `-- deb
+          |-- Makefile
+          |-- data.tar.gz
+          |-- debmake_install.sh
+          |-- etc
+          |-- usr
+          |-- var
+          `-- wazuh-indexer-min_4.9.0_amd64.deb
+  ```
+
+  `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`.
+  `Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`.
+  The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
+
+- Install the plugins using the `opensearch-plugin` CLI tool.
+- Set up configuration files.
+
+  > Included in `min-package`. Default files are overwritten.
+
+- Bundle a DEB file with `debmake` and the `Makefile`.
+
+  > `debmake` and other dependencies can be installed using the provision.sh script. The
+  > script is invoked by the GitHub Workflow.
+
+  Current folder loadout at this stage:
+
+  ```
+  artifacts/
+  |-- artifact_name.txt
+  |-- dist
+  |   |-- wazuh-indexer-min_4.9.0_amd64.deb
+  |   `-- wazuh-indexer_4.9.0_amd64.deb
+  `-- tmp
+      `-- deb
+          |-- Makefile
+          |-- data.tar.gz
+          |-- debmake_install.sh
+          |-- etc
+          |-- usr
+          |-- var
+          `-- wazuh-indexer-min_4.9.0_amd64.deb
+  ```
+
+### Running in Act
+
+```console
+act -j assemble -W .github/workflows/build.yml --artifact-server-path ./artifacts --matrix distribution:deb --matrix architecture:x64 --var OPENSEARCH_VERSION=2.11.0
+
+[Build slim packages/build] 🏁  Job succeeded
+```
+
+#### Running in Docker
+
+Pre-requisites:
+
+- Current directory: `wazuh-indexer/`
+- Existing deb package in `wazuh-indexer/artifacts/dist/deb`, as a result of the _Build_ stage.
+
+```console
+MIN_PKG_PATH="./artifacts"
+docker run --rm \
+    -v ./scripts/:/home/wazuh-indexer/scripts \
+    -v $MIN_PKG_PATH:/home/wazuh-indexer/artifacts \
+    -v ./distribution/packages/src:/home/wazuh-indexer/distribution/packages/src \
+    -w /home/wazuh-indexer \
+    -it ubuntu:jammy /bin/bash
+
+# https://github.com/opensearch-project/opensearch-build/blob/2.11.1/docker/ci/dockerfiles/current/build.ubuntu2004.opensearch.x64.arm64.dockerfile
+
+# Install necessary packages
+apt-get update -y && apt-get upgrade -y && apt-get install -y curl build-essential curl &&
+    apt-get install -y debmake debhelper-compat &&
+    apt-get install -y libxrender1 libxtst6 libasound2 libxi6 libgconf-2-4 &&
+    apt-get install -y libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libatspi2.0-dev libxcomposite-dev libxdamage1 libxfixes3 libxfixes-dev libxrandr2 libgbm-dev libxkbcommon-x11-0 libpangocairo-1.0-0 libcairo2 libcairo2-dev libnss3 libnspr4 libnspr4-dev freeglut3 &&
+    apt-get clean -y
+
+# Install aptly and required changes to debmake
+# Remove lintian for now due to it takes nearly 20 minutes for OpenSearch as well as nearly an hour for OpenSearch-Dashboards during debmake
+curl -o- https://www.aptly.info/pubkey.txt | apt-key add - &&
+    echo "deb http://repo.aptly.info/ squeeze main" | tee -a /etc/apt/sources.list.d/aptly.list &&
+    apt-get update -y && apt-get install -y aptly && apt-get clean -y &&
+    dpkg -r lintian
+
+bash scripts/assemble.sh -v 2.11.0 -p linux -a x64 -d deb
+```
+
 ### RPM
 
-The `assemble.sh` script will use the output from the `build.sh` script and use it as a 
-base to bundle together a final package containing the plugins, the production configuration 
+The `assemble.sh` script will use the output from the `build.sh` script and use it as a
+base to bundle together a final package containing the plugins, the production configuration
 and the service files.
 
 The script will:
 
 - Extract the rpm package using `rpm2cpio` and `cpio` tools.
-
-    > By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.
-
-    Current folder loadout at this stage:
-    ```
-    /rpm/$ARCH
-        /etc
-        /usr
-        /var
-        wazuh-indexer-min-*.rpm
-        wazuh-indexer.rpm.spec
-    ```
-
-    `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
-    `wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`. 
-    The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
+
+  > By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file.
+
+  Current folder loadout at this stage:
+
+  ```
+  /rpm/$ARCH
+      /etc
+      /usr
+      /var
+      wazuh-indexer-min-*.rpm
+      wazuh-indexer.rpm.spec
+  ```
+
+  `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`.
+  `wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`.
+  The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file.
 
 - Install the plugins using the `opensearch-plugin` CLI tool.
 - Set up configuration files.
 
-    > Included in `min-package`. Default files are overwritten.
-    
+  > Included in `min-package`. Default files are overwritten.
+
 - Bundle an RPM file with `rpmbuild` and the SPEC file `wazuh-indexer.rpm.spec`.
-    - `rpmbuild` is part of the `rpm` OS package.
 
-    > `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.
+  - `rpmbuild` is part of the `rpm` OS package.
 
-    Current folder loadout at this stage:
-    ```
-    /rpm/$ARCH
-        /{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
-        /etc
-        /usr
-        /var
-        wazuh-indexer-min-*.rpm
-        wazuh-indexer.rpm.spec
-    ```
+  > `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents.
+
+  Current folder loadout at this stage:
+
+  ```
+  /rpm/$ARCH
+      /{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
+      /etc
+      /usr
+      /var
+      wazuh-indexer-min-*.rpm
+      wazuh-indexer.rpm.spec
+  ```
 
 ### Running in Act
 
@@ -105,8 +206,8 @@ act -j assemble -W .github/workflows/build.yml --artifact-server-path ./artifact
 
 Pre-requisites:
 
-* Current directory: `wazuh-indexer/`
-* Existing rpm package in `wazuh-indexer/artifacts/dist/rpm`, as a result of the _Build_ stage.
+- Current directory: `wazuh-indexer/`
+- Existing rpm package in `wazuh-indexer/artifacts/dist/rpm`, as a result of the _Build_ stage.
 
 ```console
 MIN_PKG_PATH="./artifacts"
@@ -121,4 +222,3 @@ apt-get update
 apt-get install -y rpm2cpio rpm cpio
 bash scripts/assemble.sh -v 2.11.0 -p linux -a x64 -d rpm
 ```
-