Skip to content

Commit

Permalink
refactor
Browse files Browse the repository at this point in the history 
Signed-off-by: a3hadi <[email protected]>
  • Loading branch information
@ayildirim21
ayildirim21 committed Mar 22, 2024
1 parent 5012716 commit 4315cf4
Show file tree
Hide file tree
Showing 3 changed files with 77 additions and 94 deletions.
93 changes: 50 additions & 43 deletions development.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,57 +9,64 @@ brand new UDF image, and run it on a pipeline to verify the image continues work
In order to do this you can follow these steps:

1. Make the changes.
2. First increase the version in the root `pom.xml` file (increasing the PATCH version by 1 is suggested, example: X.X.0 -> X.X.1).
This can be achieved by running the following at the root directory:
```shell
./update_examples -u <version>
```
3. You can now build the test UDF image and push it to a remote repository. You can do this by simply running:
```shell
./update_examples -bpe <example-image> -t <tag>
```
the -bpe flag builds and pushes the given example to the remote registry, <example-image> must be
the id tag used within the execution element of the desired example. It is recommended to specify a tag,
like `test`, as the default is `stable` which is used by the Github Actions CI. For example, in order to push
the example for the reducer SDK, specifically the sum function, we would run:
```shell
./update_examples -bpe reduce-sum -t test
```
If you would like to build and push all the examples at once, you can run:
```shell
./update_examples -bp -t test
```
4. Once the version is updated and the desired image is built, tagged, and pushed, you can run a pipeline
using the image, i.e. we can use Numaflow e2e test cases to verify the changes.
- Checkout the latest Numaflow repository
- Find the test case and update the image and path in the test pipeline yaml file
- Run `make Test*`
2. Update your local repository using maven, to reflect your latest changes. After this, ensure that the `pom.xml` file
in the examples directory is also using these changes.
3. Build your test images, and test them in a pipeline.

### Deploying
There exists a script `update_examples.sh`, which you can also use while testing, for your convenience.

If you would like to build and push a specific example, you can run:

```shell
./update_examples -bpe <example-image> -t <tag>
```
The -bpe flag builds and pushes to the remote registry. <example-image> must be
the id tag used within the execution element of the desired example. It is recommended to specify a tag,
like `test`, as the default is `stable`, which is used by the Github Actions CI. For example, in order to push
the example, for the reducer SDK, specifically the sum function, you would run:
```shell
./update_examples -bpe reduce-sum -t test
```
If you would like to build and push all the examples at once, you can run:
```shell
./update_examples -bp -t test
```
Both `bpe` and `bp` first build a local image with the naming convention
`numaflow-java-examples/<example_id>:<tag>`, which then gets pushed as
`quay.io/numaio/numaflow-java/<example_id>:<tag>`. If while testing, you would like
to build your image locally, without pushing to quay.io, you can run the following in the
examples directory:

After confirming that your changes pass local testing, you should roll back the version updates before
publishing the PR. The real version updates can be raised in a separate PR, which is discussed
in the [After Release](#after-release) section.
```shell
mvn jib:dockerBuild@<example_id> -Ddocker.tag=<tag>
```

1. Revert both the `pom.xml` changes by re-running the update script with the version flag:
If you would like to build all example images without pushing, in the examples directory you can also run:
```shell
./update_examples -u <version-before-testing-update>
mvn clean install -Ddocker.tag=<tag>
```
2. Clean up testing artifacts, i.e. delete any images that are no longer used on the remote registry repositories,
so that they are not flooded with testing images.
3. Create a PR. Once your PR has been merged a Github Actions workflow will be triggered to build, tag, and push

### Deploying

After confirming that your changes pass local testing:

1. Clean up testing artifacts.
2. Create a PR. Once your PR has been merged a Github Actions workflow will be triggered to build, tag (with `stable`), and push
all example images, so that they are using the most up-to-date version of the SDK, i.e. the one including your
changes.

### After Release
### Before Release

Once a new release has been made, and its corresponding version tag exists on the remote repo, we want to update the dependency
management files to reflect this new version. Locally run:
Before releasing a new SDK version, make sure to update all references from the old version to the new one.
For example, the version in the `pom.xml` in the root and example directories. For [reference
](https://github.com/numaproj/numaflow-java/pull/89/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8).

```shell
./update_examples.sh -u <new-released-version>
```
### Adding a New Example

If you add a new example, there are a few steps to follow in order for it to be used by the update script and the Docker
Publish workflow:

Similar to the deployment step, create a PR for the changes created (which should just be the version bump
in the `pom.xml` files in the root and example directories), and a Github Actions workflow will be triggered to
build, tag, and push all example images.
1. Add the example to the `pom.xml` file in the examples directory, within an execution element. Note that the
id tag you specify must be exactly the same as the quay.io repository name for the example.
2. Add the id tag you specified in step 1 to the executionIDs array in `update_examples.sh`.
3. Add the id tag you specified in step 1 to the execution_ids matrix in `build-push.yaml`.
27 changes: 14 additions & 13 deletions examples/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<docker.tag>stable</docker.tag>
</properties>

<dependencies>
Expand Down Expand Up @@ -46,7 +47,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/mapt-event-time-filter-function</image>
<image>numaflow-java-examples/mapt-event-time-filter-function:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -63,7 +64,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/flat-map-stream</image>
<image>numaflow-java-examples/flat-map-stream:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -80,7 +81,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/map-flatmap</image>
<image>numaflow-java-examples/map-flatmap:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -97,7 +98,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/even-odd</image>
<image>numaflow-java-examples/even-odd:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -113,7 +114,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/simple-sink</image>
<image>numaflow-java-examples/simple-sink:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -130,7 +131,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/reduce-sum</image>
<image>numaflow-java-examples/reduce-sum:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -147,7 +148,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/reduce-stream-sum</image>
<image>numaflow-java-examples/reduce-stream-sum:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -164,7 +165,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/map-forward-message</image>
<image>numaflow-java-examples/map-forward-message:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -181,7 +182,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/reduce-counter</image>
<image>numaflow-java-examples/reduce-counter:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -198,7 +199,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/sideinput-example</image>
<image>numaflow-java-examples/sideinput-example:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -215,7 +216,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/udf-sideinput-example</image>
<image>numaflow-java-examples/udf-sideinput-example:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -232,7 +233,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/source-simple-source</image>
<image>numaflow-java-examples/source-simple-source:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand All @@ -249,7 +250,7 @@
</mainClass>
</container>
<to>
<image>numaflow-java-examples/session-reduce-count</image>
<image>numaflow-java-examples/session-reduce-count:${docker.tag}</image>
</to>
</configuration>
</execution>
Expand Down
51 changes: 13 additions & 38 deletions update_examples.sh
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
#!/bin/bash

function show_help () {
echo "Usage: $0 [-h|--help | -t|--tag <tag>] (-bp|--build-push | -bpe|--build-push-example <execution-id> | -u|--update <SDK-version>)"
echo "Usage: $0 [-h|--help | -t|--tag <tag>] (-bp|--build-push | -bpe|--build-push-example <execution-id>)"
echo " -h, --help Display help message and exit"
echo " -bp, --build-push Build all the examples and push them to the quay.io registry"
echo " -bpe, --build-push-example Build the given example id (found in examples/pom.xml), and push it to the quay.io registry"
echo " -t, --tag To be optionally used with -bpe or -bp. Specify the tag to build with. Default tag: stable"
echo " -u, --update Update the pom.xml files in the root and example directories to the specified version"
}

if [ $# -eq 0 ]; then
Expand All @@ -18,9 +17,7 @@ fi
usingHelp=0
usingBuildPush=0
usingBuildPushExample=0
usingVersion=0
usingTag=0
version=""
executionID=""
tag="stable"

Expand Down Expand Up @@ -55,17 +52,6 @@ function handle_options () {
tag=$2
shift
;;
-u | --update)
if [ -z "$2" ]; then
echo "Version not specified." >&2
show_help
exit 1
fi

usingVersion=1
version=$2
shift
;;
*)
echo "Invalid option: $1" >&2
show_help
Expand All @@ -78,13 +64,13 @@ function handle_options () {

handle_options "$@"

if (( usingBuildPush + usingBuildPushExample + usingHelp + usingVersion > 1 )); then
echo "Only one of '-h', '-bp', '-bpe', or '-u' is allowed at a time" >&2
if (( usingBuildPush + usingBuildPushExample + usingHelp > 1 )); then
echo "Only one of '-h', '-bp', or '-bpe' is allowed at a time" >&2
show_help
exit 1
fi

if (( (usingTag + usingHelp + usingVersion > 1) || (usingTag && usingBuildPush + usingBuildPushExample == 0) )); then
if (( (usingTag + usingHelp > 1) || (usingTag && usingBuildPush + usingBuildPushExample == 0) )); then
echo "Can only use -t with -bp or -bpe" >&2
show_help
exit 1
Expand All @@ -98,7 +84,7 @@ if [ -n "$executionID" ]; then
echo "Updating example: $executionID"
fi

if [ -n "$tag" ] && (( ! usingHelp )) && (( ! usingVersion )); then
if [ -n "$tag" ] && (( ! usingHelp )); then
echo "Using tag: $tag"
fi

Expand All @@ -110,12 +96,12 @@ executionIDs=("mapt-event-time-filter-function" "flat-map-stream" "map-flatmap"

function dockerPublish () {
echo "Docker publish for example: $1"
if ! docker tag numaflow-java-examples/"$1":latest quay.io/numaio/numaflow-java/"$1":"$2"; then
echo "Error: failed to tag example $1 with tag $2" >&2
if ! docker tag numaflow-java-examples/"$1":"$tag" quay.io/numaio/numaflow-java/"$1":"$tag"; then
echo "Error: failed to tag example $1 with tag $tag" >&2
exit 1
fi
if ! docker push quay.io/numaio/numaflow-java/reduce-sum:"$2"; then
echo "Error: failed to push example $1 with tag $2" >&2
if ! docker push quay.io/numaio/numaflow-java/"$1":"$tag"; then
echo "Error: failed to push example $1 with tag $tag" >&2
exit 1
fi
}
Expand All @@ -126,36 +112,25 @@ if (( usingBuildPush )); then
exit 1
fi
cd examples || exit
if ! mvn clean install; then
if ! mvn clean install -Ddocker.tag="$tag"; then
echo "Error: failed to build images in examples directory" >&2
exit 1
fi
for id in "${executionIDs[@]}"
do
dockerPublish "$id" "$tag"
dockerPublish "$id"
done
elif (( usingBuildPushExample )); then
if ! mvn clean install; then
echo "Error: failed to mvn clean install in root directory" >&2
exit 1
fi
cd examples || exit
if ! mvn jib:dockerBuild@"$executionID"; then
if ! mvn jib:dockerBuild@"$executionID" -Ddocker.tag="$tag"; then
echo "Error: failed to build example image $executionID" >&2
exit 1
fi
dockerPublish "$executionID" "$tag"
elif (( usingVersion )); then
if ! mvn versions:set -DnewVersion="$version"; then
echo "Error: failed to update version in pom.xml file in root directory" >&2
exit 1
fi

cd examples || exit
if ! mvn versions:use-dep-version -Dincludes=io.numaproj.numaflow -DdepVersion="$version" -DforceVersion=true; then
echo "Error: failed to update version in pom.xml file in examples directory" >&2
exit 1
fi
dockerPublish "$executionID"
elif (( usingHelp )); then
show_help
fi

0 comments on commit 4315cf4

Please sign in to comment.