This repository contains the files required to run a CDC pipeline locally in Docker, replicating data from a MySQL container to a PostgreSQL container using Kafka Connect.
- Startup docker and the appropriate containers. Run:
bash startup.sh- Note: There is a pseudo
Dockerfilethat could be a real project, but is just there for show. - The docker compose file is referenced here in startup; all the required containers and configurations are setup, using the resources listed below.
- Note: There is a pseudo
- Setup a kafka connector. Run:
bash create_connectors.sh- Note: You may have to wait a minute after running the startup to successfully create the connections.
- To view connectors:
curl -H "Accept:application/json" localhost:8083/connectors/
- Test the connector is setup properly:
- Make changes in the MySQL db to pass some events to the connector.
- To get to the mysql command client, on the local MySQL container:
- Exec onto the container. Run:
docker exec -it kafka-cdc_mysql_1 bash - Run commandline mysql for the
inventorydatabase. Run:mysql -u mysqluser -p inventory- It will prompt for password. Enter:
mysqlpw
- It will prompt for password. Enter:
- To show tables:
show tables;
- Exec onto the container. Run:
- Now, make some chages. Example insert:
INSERT INTO customers VALUES (default, "Sarah", "Thompson", "[email protected]")
- To get to the mysql command client, on the local MySQL container:
- View the above changes in the connector by logging the container. Run:
docker logs --tail 1000 -f kafka-cdc_connect_1- Note: the logs above will show scanning of all tables in the inventory connector.
- Checkout the data on the PostgreSQL db locally to validate the changes
- To get the pgsql client:
- Exec onto the container. Run:
docker exec -it kafka-cdc_pgsql_1 bash - Get to the psql command line client:
psql testdb - To view relations:
\dt - To view the customers table, linked to the connector:
select * from customers;. - Now, see any changes you make in mysql!
- Exec onto the container. Run:
- To get the pgsql client:
- Make changes in the MySQL db to pass some events to the connector.
- Stop all docker containers. Run:
bash down.sh - Remove containers and rebuild:
docker-compose -f docker-compose-debezium-local.yml stopdocker-compose -f docker-compose-debezium-local.yml rmdocker-compose -f docker-compose-debezium-local.yml up -d
https://debezium.io/documentation/reference/1.1/tutorial.html
https://github.com/debezium/debezium-examples/blob/master/unwrap-smt/docker-compose.yaml
https://info.crunchydata.com/blog/postgresql-change-data-capture-with-debezium
https://github.com/debezium/debezium-examples/tree/master/end-to-end-demo