README.md

🐳 kafka-docker-playground

Fully automated Apache Kafka® and Confluent Docker based examples // Easily build reproduction models

🔥 Check out kafka-docker-playground.io to learn how to use the playground, how it works and how to build your own reproduction models !

👾 Content

🔗 Connectors

• ActiveMQ Sink (also with 🔑 mTLS)  
• ActiveMQ Source (also with 🔑 mTLS)  
• Amazon CloudWatch Logs Source  
• Amazon CloudWatch Metrics Sink  
• Amazon DynamoDB Sink (also with 🌐 proxy)  
• Amazon Kinesis Source (also with 🌐 proxy)  
• Amazon Redshift Sink        
• Amazon Redshift Source (using JDBC)        
• Amazon S3 Sink (also with 🌐 proxy)  
• Amazon S3 Source  
• Amazon SQS Source (also with 🌐 proxy)  
• AMPS Source  
• Apache Kudu Source  
• Apache Kudu Sink  
• AWS Lambda Sink  
• Azure Blob Storage Sink (also with 🌐 proxy)  
• Azure Blob Storage Source (also with 🌐 proxy)  
• Azure Cosmos DB Sink  
• Azure Cosmos DB Source  
• Azure Data Lake Storage Gen1 Sink  
• Azure Data Lake Storage Gen2 Sink  
• Azure Event Hubs Source  
• Azure Functions Sink  
• Azure Cognitive Search Sink (also with 🌐 proxy)  
• Azure Service Bus Source  
• Azure SQL Data Warehouse Sink  
• Cassandra Sink  
• Couchbase Sink  
• Couchbase Source  
• Databricks Delta Lake table Sink  
• Datagen Source  
• Debezium CDC Microsoft SQL Server Source (also with 🔑 SSL)  
• Debezium CDC MySQL Source  
• Debezium CDC PostgreSQL Source (also with 🔑 SSL and mTLS)  
• Debezium CDC MongoDB Source  
• Data Diode Sink  
• Data Diode Source  
• Datadog Metrics Sink  
• ElasticSearch Sink  
• ElasticSearch Sink with Elastic Cloud
• FilePulse Source  
• FileStream Source
• FileStream Sink
• FTPS Sink  
• FTPS Source  
• Gemfire Sink  
• Github Source  
• Google BigQuery Sink  
• Google Cloud BigTable Sink (also with 🌐 proxy)  
• Google Cloud Functions Sink  
• Google Cloud Pub/Sub Source (also with 🌐 proxy)  
• Google Cloud Spanner Sink (also with 🌐 proxy)  
• Google Cloud Storage Sink  
• Google Cloud Storage Source  
• Google Firebase Realtime Database Sink  
• Google Firebase Realtime Database Source  
• HBase Sink   []() []() []() []() []()
• HDFS 2 Source  
• HDFS 3 Source  
• HDFS 2 Sink  
• HDFS 3 Sink  
• HTTP Sink  
• IBM MQ Sink (also with 🔑 SSL and mTLS)  
• IBM MQ Source (also with 🔑 SSL and mTLS)  
• InfluxDB Sink  
• InfluxDB Source  
• JDBC CockroachDB Source  
• JDBC IBM DB2 Sink  
• JDBC IBM DB2 Source  
• JDBC MySQL Sink (also with 🔑 SSL and mTLS)  
• JDBC Oracle 11 Sink  
• JDBC Oracle 12 Sink (also with 🔑 SSL and mTLS)  
• JDBC Oracle 19c Sink (also with 🔑 SSL and mTLS)  
• JDBC PostGreSQL Sink (also with 🔑 SSL and mTLS)  
• JDBC Microsoft SQL Server Sink (also with 🔑 SSL)  
• JDBC MySQL Source (also with 🔑 SSL and mTLS)  
• JDBC Oracle 11 Source  
• JDBC Oracle 12 Source (also with 🔑 SSL and mTLS)  
• JDBC Oracle 19c Source (also with 🔑 SSL and mTLS)  
• JDBC PostGreSQL Source (also with 🔑 SSL and mTLS)  
• JDBC Singlestore Source  
• JDBC Microsoft SQL Server Source (also with 🔑 SSL)  
• JDBC Snowflake Sink  
• JDBC Snowflake Source  
• JDBC Vertica Sink  
• JIRA Source  
• JMS ActiveMQ Sink  
• JMS Solace Sink  
• JMS Solace Source  
• JMS TIBCO EMS Sink  
• JMS TIBCO EMS Source  
• JMS Oracle Weblogic Sink  
• JMS Oracle Weblogic Source  
• Mapr Sink  
• Marketo Source (also with 🌐 proxy)  
• Minio Sink  
• MongoDB Sink  
• MongoDB Source  
• MQTT Sink  
• MQTT Source  
• Neo4j Sink  
• OmniSci Sink  
• Oracle 11 CDC Source  
• Oracle 12 CDC Source (also with 🔑 SSL and mTLS)  
• Oracle 19c CDC Source (also with 🔑 SSL and mTLS)  
• Oracle Weblogic Source  
• PagerDuty Sink (also with 🌐 proxy)  
• Prometheus Sink  
• RabbitMQ Sink  
• RabbitMQ Source  
• Redis Sink  
• SalesForce Bulk API Sink (also with 🌐 proxy)  
• SalesForce Bulk API Source (also with 🌐 proxy)  
• SalesForce CDC Source (also with 🌐 proxy)  
• SalesForce Platform Events Sink (also with 🌐 proxy)  
• SalesForce Platform Events Source (also with 🌐 proxy)  
• SalesForce PushTopics Source (also with 🌐 proxy)  
• SalesForce SObject Sink (also with 🌐 proxy)  
• ServiceNow Sink (also with 🌐 proxy)  
• ServiceNow Source (also with 🌐 proxy)  
• SFTP Sink (also with 🔑 Kerberos)  
• SFTP Source (also with 🔑 Kerberos)  
• Singlestore Sink  
• SNMP Trap Source  
• Snowflake Sink (also with 🌐 proxy)  
• Solace Sink  
• Solace Source  
• Splunk Sink  
• Splunk Source  
• Splunk S2S Source  
• Spool Dir Source  
• Syslog Source  
• TIBCO EMS Sink  
• TIBCO EMS Source  
• Vertica Sink  
• Zendesk Source  

* You can change default connector version by setting CONNECTOR_TAG environment variable before starting a test, get more details here

☁️ Confluent Cloud

🤖 Fully-Managed Connectors

• Amazon S3 Sink
• Amazon S3 Source
• Azure Event Hubs Source
• Databricks Delta Lake table Sink
• Google BigQuery Sink
• IBM MQ Source
• Oracle 19c CDC Source
• Snowflake Sink

🔗 Self-Managed Connectors

• AWS Kinesis source  
• Debezium CDC MongoDB source  
• Debezium CDC MySQL source  
• Firebase sink  
• ServiceNow source  
• ServiceNow sink  
• MQTT Source  
• Oracle 19c CDC Source

🚀 And much more...

• ⛈️ Using cp-ansible with Confluent Cloud
• 🌤 Using cp-helm-charts with Confluent Cloud
• 🌦️ Using Confluent Operator with Confluent Cloud
• 🌨 Using Confluent for Kubenetes with Confluent Cloud
• 📈 Demo using dabz/ccloudexporter
• 💠 .NET client (producer/consumer)  
• 🗯 Go client (producer/consumer)  
• 🐚 KafkaJS client (producer/consumer)  
• 🐍 Python client (producer/consumer)  
• 🧑‍🏭 Managing topics and ACLs using matt-mangia/kafka-admin  
• ♻️ Confluent Replicator OnPrem to cloud and Cloud to Cloud examples  
• 🗺️ Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
• 🔑 Confluent REST Proxy Security Plugin with Principal Propagation  
• 🗝️ Confluent Schema Registry Security Plugin  
• 📦 Migrate Schemas to Confluent Cloud using Confluent Replicator
• 🔰 Confluent Cloud Networking using HAProxy
• 🌎️ Apache Mirror Maker 2  
• ♻️ Cluster Linking Quick Start with service account only
• 🧬 Confluent Cloud example of connector getting data from Audit Log cluster  

🌍 Multi-Data-Center deployments

Using Multi-Data-Center setup with 🇺🇸 and 🇪🇺 clusters

• 🔗 Confluent Replicator as connector  
  • With PLAINTEXT
  • With SASL_PLAIN
  • With Kerberos
• 🖇️ Confluent Replicator as executable  
  • With PLAINTEXT
  • With SASL_PLAIN
  • With Kerberos
• 🌎️ Apache Mirror Maker 2  
  • With PLAINTEXT
• ♻️ Cluster Linking  

🔐 Environments

Using single cluster:

• PLAINTEXT: no security  
• SASL/PLAIN: no SSL encryption, SASL/PLAIN authentication  
• SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication  
• SASL/SSL: SSL encryption, SASL/PLAIN authentication  
• 2WAY/SSL: SSL encryption, SSL authentication  
• KERBEROS: no SSL encryption, Kerberos GSSAPI authentication  
• SSL/KERBEROS SSL encryption, Kerberos GSSAPI authentication  
• LDAP Authentication with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication using LDAP  
• LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization  
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication  

Using KRaft (KIP-500, i.e without zookeeper) cluster:

• KRAFT PLAINTEXT: no security  

Using Multi-Data-Center setup with 🇺🇸 and 🇪🇺 clusters

• PLAINTEXT: no security  
• SASL/PLAIN: no SSL encryption, SASL/PLAIN authentication  
• KERBEROS: no SSL encryption, Kerberos GSSAPI authentication  

💸 Confluent Commercial

• 💻 Control Center
  • Control Center in "Read-Only" mode
  • Configuring Control Center with LDAP authentication
• 💾 Tiered Storage
  • Tiered storage with AWS S3  
• ⚖ Confluent Rebalancer
• 👴 JMS Client
• 🗃️ Audit Logs  
• 🗝️ Confluent Schema Registry Security Plugin  
• 🔒️ Confluent REST Proxy Security Plugin with SASL/SSL and 2WAY/SSL Principal Propagation  
• ♻️ Cluster Linking  
• 📒 Testing RBAC with Azure AD
• 🛂 Schema Validation on Confluent Server Schema Validation on Confluent Server  
• 🙊 Secrets Management with Connect  
• ⛓ Connect Secret Registry  

🪄 CP-Ansible Playground

 

Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository

There is also a Confluent Cloud version available here

👾 Other Playgrounds

• 👬 Separate principals when using connectors: example with SFTP source  
• 📃 How to write logs to files when using docker-compose  
• 📢 Publish logs to kafka with Elastic Filebeat  
• 💠 .NET client (producer/consumer)  
• 🐚 KafkaJS client (producer/consumer)  
• 🐶 Monitor Confluent Platform with Datadog
• 👻 Testing KIP-108 Create Topic Policy
• 📉 Monitoring Demo
• 🕵️‍♀️ Kafka Connect Sink Monitoring Demo Showcase different Kafka Connect Sink troubleshooting scenarios
• 🎫 Multiple Event Types in the Same Topic  
• 👊 Integrate syslogs to detect SSH failure connections using Syslog source connector, LogStash and ksqlDB  
• 📶 How to ensure high availability of LDAP using DNS SRV Records  
• 🕹 AVRO examples including a JAVA producer  
• 🧩 Protobuf examples including a JAVA producer  
• 🎱 JSON Schema examples including a JAVA producer  
• 🚏 How to use kafka-avro-console-producer and kafka-avro-console-consumer when Schema Registry is behind a proxy