This project helps create validate hadoop release candidates
It has an Apache Ant build.xml
file to help with preparing the release,
validating gpg signatures, creating release messages and other things.
There is a maven pom.xml
file. This is used to validate the dependencies
from staging repositories as well as run some basic tests to validate
the classpath.
It has an Apache Ant build.xml
file to help with preparing the release,
validating gpg signatures, creating release messages and other things.
There is a maven pom.xml
file. This is used to validate the dependencies
from staging repositories as well as run some basic tests to validate
the classpath.
This is an optional property file which contains all user-specific customizations and options to assist in the release process.
This file is not SCM-managed.
It is read before all other property files are read/ant properties set, so can override any subsequent declarations.
This is a single-entry property file which provides a relative path to the latest release being worked on in this branch.
- It is SCM-managed.
- It is read after
build.properties
release.info.file=src/releases/release-info-3.4.0.properties
Definition files of base properties for the active RC.
- SCM-managed
- Defines properties which are common to everyone building/validating an RC.
As an example, here is the value src/releases/release-info-3.4.0.properties
for the RC2
release candidate
hadoop.version=3.4.0
rc=RC2
previous.version=3.3.6
release.branch=3.4
git.commit.id=88fbe62f27e
jira.id=HADOOP-19018
jira.title=Release 3.4.0
amd.src.dir=https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2
arm.src.dir=${amd.src.dir}
http.source=${amd.src.dir}
asf.staging.url=https://repository.apache.org/content/repositories/orgapachehadoop-1402
cloudstore.profile=sdk2
Build the RC using the docker process on whichever host is set to do it.
Create a new release properties file, using an existing one as a template. Update as a appropriate.
Update the value of release.info.file
in /release.properties
to
point to the newly created file.
release.info.file=src/releases/release-info-X.Y.Z.properties
# info for copying down the RC from the build host
scp.hostname=stevel-ubuntu
scp.user=stevel
scp.hadoop.dir=hadoop
# SVN managed staging dir
staging.dir=/Users/stevel/hadoop/release/staging
# where various modules live for build and test
spark.dir=/Users/stevel/Projects/sparkwork/spark
cloud-examples.dir=/Users/stevel/Projects/sparkwork/cloud-integration/cloud-examples
cloud.test.configuration.file=/Users/stevel/Projects/config/cloud-test-configs/s3a.xml
bigdata-interop.dir=/Users/stevel/Projects/gcs/bigdata-interop
hboss.dir=/Users/stevel/Projects/hbasework/hbase-filesystem
cloudstore.dir=/Users/stevel/Projects/cloudstore
fs-api-shim.dir=/Users/stevel/Projects/Formats/fs-api-shim/
ant clean
And then purge all artifacts of that release from maven. This is critical when validating downstream project builds.
ant purge-from-maven
This will take a while! look in target/incoming for progress
ant scp-artifacts
Copies the files from downloads/incoming/artifacts
to downloads/hadoop-$version-$rc
'
ant copy-scp-artifacts release.dir.check
The release.dir.check
target just lists the directory.
If arm64 binaries are being created then they must be
built on an arm docker image.
Do not use the --asfrelease
option as this stages the JARs.
Instead use the explicit --deploy --native --sign
options.
The arm process is one of
- Create the full set of artifacts on an arm machine (macbook, cloud vm, ...)
- Use the ant build to copy and rename the
.tar.gz
with the native binaries only - Create a new
.asc
file. - Generate new sha512 checksum file containing the new name.
- Move these files into the
downloads/release/$RC
dir
To perform these stages, you need a clean directory of the same hadoop commit ID as for the x86 release.
In build.properties
declare its location
arm.hadoop.dir=/Users/stevel/hadoop/release/hadoop
In that dir, create the relese.
time dev-support/bin/create-release --docker --dockercache --mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false" --deploy --native --sign
Important make sure there is no duplicate staged hadoop repo in nexus. If there is: drop and restart the x86 release process to make sure it is the one published
# create the release.
# Broken until someone fixes HADOOP-18664. you can't launch create-release --docker from a build file
#ant arm.create.release
# copy the artifacts to this project's target/ dir, renaming
ant arm.copy.artifacts
# sign artifacts then move to the shared RC dir alongside the x86 artifacts
ant arm.release release.dir.check
ant gpg.keys gpg.verify
When committed to subversion it will be uploaded and accessible via a https://svn.apache.org URL.
This makes it visible to others via the apache svn site, but it is not mirrored yet.
When the RC is released, an svn move
operation can promote it
directly.
do this after preparing the arm64 binaries
Final review of the release files
ant release.dir.check
Now stage the files, first by copying the dir of release artifacts into the svn-mananaged location
ant stage
This can take a while...exit any VPN for extra speed.
ant stage-to-svn
Manual
cd $stagingdir
svn update
svn add <RC directory name>
svn commit
This isn't automated as it needs to be done in the source tree.
The ant print-tag-command
prints the command needed to create and sign
a tag.
ant print-tag-command
- Go to https://repository.apache.org/#stagingRepositories
- Find the hadoop repo for the RC
- "close" it and wait for that to go through
Review/update template message in src/text/vote.txt
.
All ant properties referenced will be expanded if set.
ant vote-message
The message is printed and saved to the file target/vote.txt
do not send it until you have validated the URLs resolve
Now wait for the votes to come in. This is a good time to repeat all the testing of downstream projects, this time validating the staged artifacts, rather than any build locally.
This relies on the release-info file pointing to the source directory
http.source=https://dist.apache.org/repos/dist/dev/hadoop/hadoop-${hadoop.version}-RC${rc}/
target | action |
---|---|
release.fetch.http |
fetch artifacts |
release.dir.check |
verify release dir exists |
release.src.untar |
untar retrieved artifacts |
release.src.build |
build the source |
release.src.test |
build and test the source |
gpg.keys |
import the hadoop KEYS |
gpg.verify |
verify the D/L'd artifacts |
set check.native.binaries
to false to skip native binary checks on platforms without them
Downloads under downloads/incoming
ant release.fetch.http
This puts the built artifacts into the local maven repo so
do not do this while building/testing downstream projects
and call ant purge-from-maven
after
ant release.src.untar release.src.build
ant release.site.untar
Untars the (already downloaded) binary tar to target/bin-untar
ant release.bin.untar
once expanded, the binary commands can be tested
ant release.bin.commands
This will fail on a platform where the native binaries don't load, unless the checknative command has been disabled.
check.native.binaries=false
ant release.bin.commands -Dcheck.native.binaries=false
A lot of the targets build maven projects from the staged maven artifacts.
For this to work
- Check out the relevant projects in your local system.
- Set their location in the
build.properties
file - Make sure that the branch checked out is the one you want to build. This matters for anyone who works on those other projects on their own branches.
- Some projects need java11 or later.
First, purge your maven repository of all hadoop- JAR files of the pending release version
ant purge-from-maven
Download the artifacts from maven staging repositories and compile/test a minimal application
ant mvn-test
ant cloudstore.build
- This is java 11+ only.
- currently only builds against AWS v1 SDK.
Ideally, you should run the tests, or even better, run them before the RC is up for review.
Building the libraries. Do this only if you aren't running the tests.
ant gcs.build
Validates hadoop client artifacts; the cloud tests cover hadoop cloud storage clients.
ant spark.build
And to to run the hadoop-cloud
tests
ant spark.test.hadoop-cloud
A full spark test run takes so long that CI infrastructure should be used.
Then followup cloud integration tests if you are set up to build. Spark itself does not include any integration tests of the object store connectors. This independent module tests the s3a, gcs and abfs connectors, and associated committers, through the spark RDD and SQL APIs.
ant cloud-examples.build
ant cloud-examples.test
The test run is fairly tricky to get running; don't try and do this while
- MUST be java 11+
- Must have
cloud.test.configuration.file
set to an XML conf file declaring the auth credentials and stores to use for the target object stores (s3a, abfs, gcs)
Adds zookeeper-based locking on those filesystem API calls for which atomic access is required.
Integration tests will go through S3A connector.
ant hboss.build
Edit src/text/announcement.txt
to have an up-to-date
description of the release.
The release.site.announcement
target will generate these
annoucements. Execute the target and then review
the generated files in target/
ant release.site.announcement
The announcement must be geneated before the next stage,
so make sure the common body of the site and email
annoucement is up to date: src/text/core-announcement.txt
Set hadoop.site.dir
to be the path of the
local clone of the ASF site repository
https://gitbox.apache.org/repos/asf/hadoop-site.git
hadoop.site.dir=/Users/stevel/hadoop/release/hadoop-site
Prepare the site; this also demand-generates the release announcement
The site .tar.gz distributable is used for the site; this must already
have been downloaded. It must be untarred and copied under the
SCM-managed ${hadoop.site.dir}
repository, linked up
and then committed.
ant release.site.untar
ant release.site.docs
In the hadoop site dir content/docs subdir
# update
git pull
# review current status
ls -l
# symlink current
rm current3
ln -s r.3.3.5 current3
# symlink stable
rm stable3
ln -s r3.3.5 stable3
# review new status
ls -l
Finally, commit
git add .
git status
git commit -S -m "HADOOP-18470. Release Hadoop 3.3.5"
git push
# check that the source and dest URLs are good
ant staging-init
# do the promotion
ant stage-move-to-production
https://dist.apache.org/repos/dist/release/hadoop/common
change the
Check that release URL in your browser.
do this at https://repository.apache.org/#stagingRepositories
to verify this is visible search for hadoop-common -verify the latest version is in the production repository.
The ant print-tag-command
prints the command needed to create and sign
a tag.
ant print-tag-command
Use the "tagging the final release" commands printed
For safety, purge your maven repo of all versions of the release, so as to guarantee that everything comes from the production store.
ant purge-from-maven
There are a lot of files, and if your shell has a prompt which shoes the git repo state, scanning can take a long time. Disable it, such as for fish:
set -e __fish_git_prompt_showdirtystate
Many projects have a profile to use a staging repository, especially the ASF one.
Not all do -these builds are likely to fail.
Here is a profile, asf-staging
which can be used to enable this.
The paths to the repository can be changed too, if desired.
Some of the maven builds invoked rely on this profile (e.g. avro). For some unknown reason the parquet build doesn't seem to cope.
<profile>
<id>asf-staging</id>
<properties>
<!-- override point for ASF staging/snapshot repos -->
<asf.staging>https://repository.apache.org/content/groups/staging/</asf.staging>
<asf.snapshots>https://repository.apache.org/content/repositories/snapshots/</asf.snapshots>
</properties>
<pluginRepositories>
<pluginRepository>
<id>ASF Staging</id>
<url>${asf.staging}</url>
</pluginRepository>
<pluginRepository>
<id>ASF Snapshots</id>
<url>${asf.snapshots}</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>ASF Staging</id>
<url>${asf.staging}</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>ASF Snapshots</id>
<url>${asf.snapshots}</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>
</profile>
Drop the staged artifacts from nexus https://repository.apache.org/#stagingRepositories
Delete the tag. Print out the delete command and then copy/paste it into a terminal in the hadoop repo
ant print-tag-command
Remove downloaded files and maven artifactgs
ant clean purge-from-maven
- Go to the svn staging dir
svn rm
the RC subdirsvn commit -m "rollback RC"
ant stage-svn-rollback
# and get the log
ant stage-svn-log