Skip to content

Override Kafka Image Dynamically

Jump to bottom
Luke Chen edited this page Jan 19, 2023 · 11 revisions

During development process, you might want to deploy your own Kafka image for testing. This wiki is a workable method to do this job.

Prerequisites

  1. An OSD is deployed using kas-installer. This OSD can be listed with ocm command:
> ocm list cluster
ID                                NAME                          API URL                                                     OPENSHIFT_VERSION   PRODUCT ID      CLOUD_PROVIDER  REGION ID       STATE
1s94hivl6n6kuki40rh4o3huni11k9s3  lukchen-osd                   https://api.lukchen-osd.qlny.s1.devshift.org:6443           4.10.12             osd             aws             us-east-1       ready
  1. Kafka is deployed onto the OSD.
> kubectl get pods -n kafka-ca2a7m2f61gurg1o3h00
NAME                                          READY   STATUS    RESTARTS        AGE
lukchen-osd-admin-server-64574ff4c4-h24b5     1/1     Running   1               4d23h
lukchen-osd-canary-6ffc9fd4fb-j4q72           1/1     Running   2               4d23h
lukchen-osd-kafka-0                           1/1     Running   4 (5m49s ago)   3d20h
lukchen-osd-kafka-1                           1/1     Running   4 (5m24s ago)   3d20h
lukchen-osd-kafka-2                           1/1     Running   4 (5m43s ago)   3d20h
lukchen-osd-kafka-exporter-5774987b6f-4962p   1/1     Running   3 (6m33s ago)   4d23h
lukchen-osd-zookeeper-0                       1/1     Running   2 (6m36s ago)   3d20h
lukchen-osd-zookeeper-1                       1/1     Running   2 (6m36s ago)   3d20h
lukchen-osd-zookeeper-2                       1/1     Running   1               3d20h

  1. install podman or docker (for M1 chip MAC users, docker might be a better choice because podman needs some workaround to run command in x86 container. See here)

  2. The targeting jar file is ready. ex: kafka-quotas-plugin.jar, group-authorizer.jar, ...

Creating a new Kafka image

In this example we will override the kas-broker-plugins jars in the RHOSAK Kafka image

  1. Determine the current rhosak kafka image version If you have an oc cluster running:
oc get pod my-kafka-kafka-0 -n ${your kafka namespace here} -ojson | jq '.spec.containers[] | select(.name == "kafka") | .image' -r
  1. create a Dockerfile in the kas-broker-plugins repository that you have cloned locally
FROM ${image-from-step-1}
USER root
RUN rm /opt/kafka/libs/kas-broker-plugins*
USER strimzi
COPY target/kas-broker-plugins.jar /opt/kafka/libs/
  1. build the docker image
podman build . -t quay.io/lukchen/kafka:rhosak
  1. push the image (you might need to login to quay.io first)
podman login quay.io
podman push quay.io/lukchen/kafka:rhosak
IMAGE_HASH_IN_QUAY=$(podman inspect --format='{{index .RepoDigests 0}}' quay.io/lukchen/kafka:rhosak)

Overriding the kafka image

Override the kafka image in subscription yaml

> oc edit  subscription -n  redhat-kas-fleetshard-operator kas-fleetshard-operator-sub
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  ...
spec:
  config:
    env:
    # add a IMAGE_KAFKA env key and value point to the image postion created above
    - name: IMAGE_KAFKA
      value: quay.io/lukchen/kafka@sha256:IMAGE_HASH_IN_QUAY

if it's got reverted, try editing CSV

oc edit csv -n redhat-kas-fleetshard-operator-qe  kas-fleetshard-operator.v1.0.14

The zk and kafka pods should have rolling update now

Note: If you found the ZK and kafka pods don't have rolling update after the subscription editing, try to delete the strimzi operation and edit again. ex:

kubectl delete pod strimzi-cluster-operator.v0.26.0-9-65889d76f4-m85bl -n redhat-managed-kafka-operator
Clone this wiki locally