This repo includes a set of resources to demonstrate how Apache MirrorMaker2 could help in a migration from a source Apache Kafka cluster to another Apache Kafka cluster 🚀.
For more context, please review our blog post.
The scenario covered in this repo is to have an Active-Passive deployment of Apache Kafka clusters deployed in different OpenShift clusters.
This repo was tested ✨ in:
-
Source Environment:
- Red Hat OpenShift Container Platform 4.5.18
- Strimzi Operators 0.22.1 (Apache Kafka 2.5)
-
Target Environment:
- Red Hat OpenShift Container Platform 4.9.0
- Strimzi Operators 0.26.0 (Apache Kafka 2.8)
NOTE: To follow this demo you should have two different OpenShift clusters available following the versions described above.
🚨 WARN: This repo is not defined to be a production-ready implementation but it could be used as a base line to design and develop your specific use case. Use carefully and by own your responsibility.
This migration process covers the scenario when you have an Apache Kafka cluster deployed in an OpenShift platform (source) and you need to migrate the data and your applications to a new OpenShift platform (target). This process could be done using other tools or processes however in this repo we focused on using one of the tools provided by Apache Kafka ecosystem: Mirror Maker 2.
The migration process could be summarized as:
- An Apache Kafka cluster up and running in the source OpenShift platform. This platform already has some producer and consumer applications running successfully.
- Expose the source Apache Kafka cluster to external clients of OpenShift.
- Extract from the source Apache Kafka cluster the certificates and credentials needed to allow authenticated and authorized connections from outside of OpenShift.
- Deploy a new Apache Kafka cluster in the target OpenShift platform. This new Apache Kafka cluster could be a higher version of the original one (as we did in this repo) but it is needed to have a similar deployment topology as the source one.
- Deploy and create the secrets with the credentials coming from the source Kafka cluster needed to be used by the applications. Create the users definitions in the target Kafka cluster.
- Set up MirrorMaker2 with the source and target Kafka clusters (connection string, users, ...)
- Deploy MirrorMaker2 in the target OpenShift platform.
- Verify the topics are created in the target Kafka. This process could also sync the consumer groups offsets from the source Kafka cluster.
- Stop consumers in the source OpenShift platform.
- Deploy and start consumers in the target OpenShift platform.
- Stop producers in the source OpenShift platform.
- Deploy and start producers in the target OpenShift platform.
This process started with an Active(source)-Passive(target) deployment, but when it is completed the target platform will be the new active one and we could stop and remove the source platform.
NOTE: MirrorMaker2 helps to cover other kinds of topologies to migrate or replicate data between different Apache Kafka clusters (multi-cloud, different data-flows, back-ups, ...), all of them out of the scope of this repo.
Both Apache Kafka deployments are managed and operated by the Strimzi operators.
As a normal user (non cluster-admin) in your source OpenShift cluster, create the following namespace:
❯ oc login -u user
❯ oc new-project strimzi-migrationFollow the instructions
Follow the instructions
Follow the instructions
Follow the instructions
Follow the instructions
NOTE: This part of the demo must done in the target environment. Check that you are connected into the target OpenShift cluster.
As a normal user (non cluster-admin) in your OpenShift cluster, create the following namespaces:
❯ oc login -u user
❯ oc new-project strimzi-migrationFollow the instructions
Follow the instructions
Follow the instructions
Follow the instructions
Follow the instructions
If everything was fine you will have the original topics created from the source cluster in your target cluster and with the same data:
❯ oc get kt
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY
apps.samples.greetings event-bus 3 3 True
apps.samples.greetings.reversed event-bus 3 3 True
heartbeats event-bus 1 3 True
mirrormaker2-cluster-configs event-bus 1 3 True
mirrormaker2-cluster-offsets event-bus 25 3 True
mirrormaker2-cluster-status event-bus 5 3 True
monitor.ocp.logs event-bus 10 3 True
monitor.ocp.metrics event-bus 10 3 True
my-source-cluster.checkpoints.internal event-bus 1 3 True