diff --git a/development.md b/development.md index bb93be94..a1cbb340 100644 --- a/development.md +++ b/development.md @@ -9,57 +9,64 @@ brand new UDF image, and run it on a pipeline to verify the image continues work In order to do this you can follow these steps: 1. Make the changes. -2. First increase the version in the root `pom.xml` file (increasing the PATCH version by 1 is suggested, example: X.X.0 -> X.X.1). -This can be achieved by running the following at the root directory: - ```shell - ./update_examples -u - ``` -3. You can now build the test UDF image and push it to a remote repository. You can do this by simply running: - ```shell - ./update_examples -bpe -t - ``` - the -bpe flag builds and pushes the given example to the remote registry, must be - the id tag used within the execution element of the desired example. It is recommended to specify a tag, - like `test`, as the default is `stable` which is used by the Github Actions CI. For example, in order to push - the example for the reducer SDK, specifically the sum function, we would run: - ```shell - ./update_examples -bpe reduce-sum -t test - ``` - If you would like to build and push all the examples at once, you can run: - ```shell - ./update_examples -bp -t test - ``` -4. Once the version is updated and the desired image is built, tagged, and pushed, you can run a pipeline -using the image, i.e. we can use Numaflow e2e test cases to verify the changes. - - Checkout the latest Numaflow repository - - Find the test case and update the image and path in the test pipeline yaml file - - Run `make Test*` +2. Update your local repository using maven, to reflect your latest changes. After this, ensure that the `pom.xml` file +in the examples directory is also using these changes. +3. Build your test images, and test them in a pipeline. -### Deploying +There exists a script `update_examples.sh`, which you can also use while testing, for your convenience. + +If you would like to build and push a specific example, you can run: + +```shell +./update_examples -bpe -t + ``` +The -bpe flag builds and pushes to the remote registry. must be +the id tag used within the execution element of the desired example. It is recommended to specify a tag, +like `test`, as the default is `stable`, which is used by the Github Actions CI. For example, in order to push +the example, for the reducer SDK, specifically the sum function, you would run: +```shell +./update_examples -bpe reduce-sum -t test +``` +If you would like to build and push all the examples at once, you can run: +```shell +./update_examples -bp -t test +``` +Both `bpe` and `bp` first build a local image with the naming convention +`numaflow-java-examples/:`, which then gets pushed as +`quay.io/numaio/numaflow-java/:`. If while testing, you would like +to build your image locally, without pushing to quay.io, you can run the following in the +examples directory: -After confirming that your changes pass local testing, you should roll back the version updates before -publishing the PR. The real version updates can be raised in a separate PR, which is discussed -in the [After Release](#after-release) section. +```shell +mvn jib:dockerBuild@ -Ddocker.tag= +``` -1. Revert both the `pom.xml` changes by re-running the update script with the version flag: +If you would like to build all example images without pushing, in the examples directory you can also run: ```shell -./update_examples -u +mvn clean install -Ddocker.tag= ``` -2. Clean up testing artifacts, i.e. delete any images that are no longer used on the remote registry repositories, -so that they are not flooded with testing images. -3. Create a PR. Once your PR has been merged a Github Actions workflow will be triggered to build, tag, and push + +### Deploying + +After confirming that your changes pass local testing: + +1. Clean up testing artifacts. +2. Create a PR. Once your PR has been merged a Github Actions workflow will be triggered to build, tag (with `stable`), and push all example images, so that they are using the most up-to-date version of the SDK, i.e. the one including your changes. -### After Release +### Before Release -Once a new release has been made, and its corresponding version tag exists on the remote repo, we want to update the dependency -management files to reflect this new version. Locally run: +Before releasing a new SDK version, make sure to update all references from the old version to the new one. +For example, the version in the `pom.xml` in the root and example directories. For [reference +](https://github.com/numaproj/numaflow-java/pull/89/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8). -```shell -./update_examples.sh -u - ``` +### Adding a New Example + +If you add a new example, there are a few steps to follow in order for it to be used by the update script and the Docker +Publish workflow: -Similar to the deployment step, create a PR for the changes created (which should just be the version bump -in the `pom.xml` files in the root and example directories), and a Github Actions workflow will be triggered to -build, tag, and push all example images. +1. Add the example to the `pom.xml` file in the examples directory, within an execution element. Note that the +id tag you specify must be exactly the same as the quay.io repository name for the example. +2. Add the id tag you specified in step 1 to the executionIDs array in `update_examples.sh`. +3. Add the id tag you specified in step 1 to the execution_ids matrix in `build-push.yaml`. diff --git a/examples/pom.xml b/examples/pom.xml index cd62e24e..51d3c6c1 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -10,6 +10,7 @@ UTF-8 + stable @@ -46,7 +47,7 @@ - numaflow-java-examples/mapt-event-time-filter-function + numaflow-java-examples/mapt-event-time-filter-function:${docker.tag} @@ -63,7 +64,7 @@ - numaflow-java-examples/flat-map-stream + numaflow-java-examples/flat-map-stream:${docker.tag} @@ -80,7 +81,7 @@ - numaflow-java-examples/map-flatmap + numaflow-java-examples/map-flatmap:${docker.tag} @@ -97,7 +98,7 @@ - numaflow-java-examples/even-odd + numaflow-java-examples/even-odd:${docker.tag} @@ -113,7 +114,7 @@ - numaflow-java-examples/simple-sink + numaflow-java-examples/simple-sink:${docker.tag} @@ -130,7 +131,7 @@ - numaflow-java-examples/reduce-sum + numaflow-java-examples/reduce-sum:${docker.tag} @@ -147,7 +148,7 @@ - numaflow-java-examples/reduce-stream-sum + numaflow-java-examples/reduce-stream-sum:${docker.tag} @@ -164,7 +165,7 @@ - numaflow-java-examples/map-forward-message + numaflow-java-examples/map-forward-message:${docker.tag} @@ -181,7 +182,7 @@ - numaflow-java-examples/reduce-counter + numaflow-java-examples/reduce-counter:${docker.tag} @@ -198,7 +199,7 @@ - numaflow-java-examples/sideinput-example + numaflow-java-examples/sideinput-example:${docker.tag} @@ -215,7 +216,7 @@ - numaflow-java-examples/udf-sideinput-example + numaflow-java-examples/udf-sideinput-example:${docker.tag} @@ -232,7 +233,7 @@ - numaflow-java-examples/source-simple-source + numaflow-java-examples/source-simple-source:${docker.tag} @@ -249,7 +250,7 @@ - numaflow-java-examples/session-reduce-count + numaflow-java-examples/session-reduce-count:${docker.tag} diff --git a/update_examples.sh b/update_examples.sh index d5577baa..b4c04cd1 100755 --- a/update_examples.sh +++ b/update_examples.sh @@ -1,12 +1,11 @@ #!/bin/bash function show_help () { - echo "Usage: $0 [-h|--help | -t|--tag ] (-bp|--build-push | -bpe|--build-push-example | -u|--update )" + echo "Usage: $0 [-h|--help | -t|--tag ] (-bp|--build-push | -bpe|--build-push-example )" echo " -h, --help Display help message and exit" echo " -bp, --build-push Build all the examples and push them to the quay.io registry" echo " -bpe, --build-push-example Build the given example id (found in examples/pom.xml), and push it to the quay.io registry" echo " -t, --tag To be optionally used with -bpe or -bp. Specify the tag to build with. Default tag: stable" - echo " -u, --update Update the pom.xml files in the root and example directories to the specified version" } if [ $# -eq 0 ]; then @@ -18,9 +17,7 @@ fi usingHelp=0 usingBuildPush=0 usingBuildPushExample=0 -usingVersion=0 usingTag=0 -version="" executionID="" tag="stable" @@ -55,17 +52,6 @@ function handle_options () { tag=$2 shift ;; - -u | --update) - if [ -z "$2" ]; then - echo "Version not specified." >&2 - show_help - exit 1 - fi - - usingVersion=1 - version=$2 - shift - ;; *) echo "Invalid option: $1" >&2 show_help @@ -78,13 +64,13 @@ function handle_options () { handle_options "$@" -if (( usingBuildPush + usingBuildPushExample + usingHelp + usingVersion > 1 )); then - echo "Only one of '-h', '-bp', '-bpe', or '-u' is allowed at a time" >&2 +if (( usingBuildPush + usingBuildPushExample + usingHelp > 1 )); then + echo "Only one of '-h', '-bp', or '-bpe' is allowed at a time" >&2 show_help exit 1 fi -if (( (usingTag + usingHelp + usingVersion > 1) || (usingTag && usingBuildPush + usingBuildPushExample == 0) )); then +if (( (usingTag + usingHelp > 1) || (usingTag && usingBuildPush + usingBuildPushExample == 0) )); then echo "Can only use -t with -bp or -bpe" >&2 show_help exit 1 @@ -98,7 +84,7 @@ if [ -n "$executionID" ]; then echo "Updating example: $executionID" fi -if [ -n "$tag" ] && (( ! usingHelp )) && (( ! usingVersion )); then +if [ -n "$tag" ] && (( ! usingHelp )); then echo "Using tag: $tag" fi @@ -110,12 +96,12 @@ executionIDs=("mapt-event-time-filter-function" "flat-map-stream" "map-flatmap" function dockerPublish () { echo "Docker publish for example: $1" - if ! docker tag numaflow-java-examples/"$1":latest quay.io/numaio/numaflow-java/"$1":"$2"; then - echo "Error: failed to tag example $1 with tag $2" >&2 + if ! docker tag numaflow-java-examples/"$1":"$tag" quay.io/numaio/numaflow-java/"$1":"$tag"; then + echo "Error: failed to tag example $1 with tag $tag" >&2 exit 1 fi - if ! docker push quay.io/numaio/numaflow-java/reduce-sum:"$2"; then - echo "Error: failed to push example $1 with tag $2" >&2 + if ! docker push quay.io/numaio/numaflow-java/"$1":"$tag"; then + echo "Error: failed to push example $1 with tag $tag" >&2 exit 1 fi } @@ -126,13 +112,13 @@ if (( usingBuildPush )); then exit 1 fi cd examples || exit - if ! mvn clean install; then + if ! mvn clean install -Ddocker.tag="$tag"; then echo "Error: failed to build images in examples directory" >&2 exit 1 fi for id in "${executionIDs[@]}" do - dockerPublish "$id" "$tag" + dockerPublish "$id" done elif (( usingBuildPushExample )); then if ! mvn clean install; then @@ -140,22 +126,11 @@ elif (( usingBuildPushExample )); then exit 1 fi cd examples || exit - if ! mvn jib:dockerBuild@"$executionID"; then + if ! mvn jib:dockerBuild@"$executionID" -Ddocker.tag="$tag"; then echo "Error: failed to build example image $executionID" >&2 exit 1 fi - dockerPublish "$executionID" "$tag" -elif (( usingVersion )); then - if ! mvn versions:set -DnewVersion="$version"; then - echo "Error: failed to update version in pom.xml file in root directory" >&2 - exit 1 - fi - - cd examples || exit - if ! mvn versions:use-dep-version -Dincludes=io.numaproj.numaflow -DdepVersion="$version" -DforceVersion=true; then - echo "Error: failed to update version in pom.xml file in examples directory" >&2 - exit 1 - fi + dockerPublish "$executionID" elif (( usingHelp )); then show_help fi