Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Confluent Notebook #40

Merged
merged 9 commits into from
Jan 23, 2024
Merged

Update Confluent Notebook #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2a4baea
Update meta.toml
adityamish Jan 23, 2024
1b9f3ea
Update notebook.ipynb
adityamish Jan 23, 2024
a2ea6f3
Update notebook.ipynb
adityamish Jan 23, 2024
6da1730
Update notebook.ipynb
adityamish Jan 23, 2024
7f65ea4
Update notebook.ipynb
adityamish Jan 23, 2024
579bc01
Update notebook.ipynb
adityamish Jan 23, 2024
0b4dd8a
Merge pull request #1 from oyeliseiev-ua/aditya/updates
pmishchenko-ua Jan 23, 2024
14b22e3
Use confluent logo
Jan 23, 2024
e977fa9
Merge pull request #2 from oyeliseiev-ua/logo
pmishchenko-ua Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions notebooks/confluent-cloud-integration/meta.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[meta]
title = "Ingest data from Confluent Cloud (Kafka)"
description = """
Provides step-by-step guidance on seamlessly ingesting data from Confluent Cloud (Kafka) to SingleStore Database using a robust pipeline. """
icon = "database"
A step-by-step guide on seamlessly ingesting data from Confluent Cloud (Kafka) into your SingleStoreDB database using a robust pipeline. """
icon = "confluent-logo"
tags = ["confluent", "training", "kafka", "pipeline", "realtime"]
destinations = ["spaces"]
59 changes: 31 additions & 28 deletions notebooks/confluent-cloud-integration/notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
"id": "8c76da18-b057-494b-bb1a-d13a608f7b04",
"metadata": {},
"source": [
"<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(235, 249, 245, 0.25); padding: 5px;\">\n",
"<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(124, 195, 235, 0.25); padding: 5px;\">\n",
" <div id=\"icon-image\" style=\"width: 90px; height: 90px;\">\n",
" <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/database.png\" />\n",
" <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/confluent-logo.png\" />\n",
" </div>\n",
" <div id=\"text\" style=\"padding: 5px; margin-left: 10px;\">\n",
" <div id=\"badge\" style=\"display: inline-block; background-color: rgba(0, 0, 0, 0.15); border-radius: 4px; padding: 4px 8px; align-items: center; margin-top: 6px; margin-bottom: -2px; font-size: 80%\">SingleStore Notebooks</div>\n",
Expand All @@ -22,7 +22,7 @@
"id": "d89587bf-6790-4a03-8d33-c5a1cb0fa94d",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-kafka-integration.png width=\"100%\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-kafka-integration.png width=\"100%\" />"
]
},
{
Expand All @@ -31,11 +31,13 @@
"id": "17ab18b5-176d-4b36-b653-30155855230d",
"metadata": {},
"source": [
"### Confluent Cluster (Kafka) set up\n",
"### Set Up a Kafka Cluster on Confluent Cloud\n",
"\n",
"Prior to initiating the integration process, it is essential to configure a Confluent Kafka cluster. Please refer to the provided <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html\">link</a> for a quick start guide.\n",
"Before initiating the integration process, it is essential to configure a Kafka cluster on Confluent Cloud. Refer to the <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html\">Quick Start for Confluent Cloud</a> guide for related information.\n",
"\n",
"- Once the cluster is created, proceed to create a topic named for example <b>'s2-topic'</b> and configure the value AVRO schema, as example choose proposed default:"
"Once the cluster is created, perform the following tasks:\n",
"\n",
"- Create a topic, for example <b>'s2-topic'</b>. On the topic overview page, select <b>Schema &gt; Set a schema &gt; Avro</b>, and add a new Avro schema. In this guide, the default schema is used."
]
},
{
Expand All @@ -44,7 +46,7 @@
"id": "35bee169-fec8-47eb-89b2-712bde11b745",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/kafka-value-schema.png width=\"1000\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/kafka-value-schema.png width=\"100%\" />"
]
},
{
Expand All @@ -53,7 +55,7 @@
"id": "679133dc-5c6f-4ea3-97f9-3317d4c98fb6",
"metadata": {},
"source": [
"- Create API Keys and save for later usage:"
"- Create API keys. The API key is displayed <b>only once</b>. Be sure to copy and securely store the API key."
]
},
{
Expand All @@ -62,7 +64,7 @@
"id": "4fba3d16-437c-40d0-903f-91560bc71019",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-api-key.png width=\"1000\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-api-key.png width=\"100%\" />"
]
},
{
Expand All @@ -71,8 +73,8 @@
"id": "50713fc6-a965-4a70-b732-4c33ae0eaa5f",
"metadata": {},
"source": [
"- Go to 'Connectors' and create a sample producer <b>'datagen'</b> with 'Use an existing API key' option for the established Kafka topic <b>'s2-topic'</b>(<a href=\"https://docs.confluent.io/cloud/current/get-started/index.html#step-3-create-a-sample-producer\">Step 3</a> from quick guide). Configure the producer to utilize the same schema as the created topic.\n",
"- Launch <b>'datagen'</b> producer and check <b>'s2-topic'</b> has new messages."
"- On the left navigation pane, select <b>Connectors</b> and create a sample producer named <b>'datagen'</b> using the <b>Datagen Source</b> connector. In the <b>Topic selection</b> pane, select the <b>'s2-topic'</b> created earlier. In the <b>Kafka credentials</b> pane, select the <b>Use an existing API key</b> option. Configure the producer to use the same schema as the one in the created topic. Refer to <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html#step-3-create-a-sample-producer\">Step 3: Create a sample producer</a> for more information.\n",
"- Launch the <b>'datagen'</b> producer and verify that the <b>'s2-topic'</b> has new messages."
]
},
{
Expand All @@ -81,20 +83,21 @@
"id": "2a05848e-fad9-47c7-8a4b-7c6fd627ac44",
"metadata": {},
"source": [
"### Set up variables\n",
"### Set Up Variables\n",
"\n",
"Choose <b>S2_DATABASE_NAME</b>, <b>S2_TABLE_NAME</b> and <b>S2_PIPELINE_NAME</b> to use for integration\n",
"Use the <b>S2_DATABASE_NAME</b>, <b>S2_TABLE_NAME</b>, and <b>S2_PIPELINE_NAME</b> variables for integration.\n",
"\n",
"### Copy data from Confluent Cloud\n",
"- Set up <b>CONFLUENT_KAFKA_TOPIC_NAME</b> - put created topic name ('s2-topic')\n",
"- Set up <b>CONFLUENT_API_KEY</b> and <b>CONFLUENT_API_SECRET</b> - put API key and secret\n",
"### Copy Data from Confluent Cloud\n",
"- Assign the topic name <b>'s2-topic'</b> to the <b>CONFLUENT_KAFKA_TOPIC_NAME</b> variable.\n",
"- Specify the API key and secret using the <b>CONFLUENT_API_KEY</b> and <b>CONFLUENT_API_SECRET</b> variables, respectively.\n",
"\n",
"Go to 'Clients' -> choose language (for example java) and set up following variables:\n",
"On the left navigation pane, select <b>Clients</b>. Select a language (for example Java), and configure the following variables:\n",
"- <b>CONFLUENT_CLUSTER_BOOTSTRAP_SERVER</b> from <b>bootstrap.servers</b>\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_URL</b> from <b>schema.registry.url</b>\n",
"\n",
"Click 'Create Schema Registry API key' to create schema api key and set up:\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_KEY</b> and <b>CONFLUENT_SCHEMA_REGISTRY_SECRET</b>"
"Select <b>Create Schema Registry API key</b> to create a schema API key and configure the following variables:\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_KEY</b>\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_SECRET</b>"
]
},
{
Expand Down Expand Up @@ -123,7 +126,7 @@
"id": "db9054b4-aec8-4aae-8069-efa042f759d5",
"metadata": {},
"source": [
"### Create database"
"### Create a Database"
]
},
{
Expand All @@ -145,7 +148,7 @@
"id": "279607d8-35c2-4a84-a4fa-50f5156269f1",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-warning\"> <b class=\"fa fa-solid fa-exclamation-circle\"></b> <div> <p><b>Action Required</b></p> <p>Make sure to select the <tt>{{S2_DATABASE_NAME}}</tt> database from the drop-down menu at the top of this notebook. It updates the <tt>connection_url</tt> which is used by the <tt>%%sql</tt> magic command and SQLAlchemy to make connections to the selected database.</p> </div></div>"
"<div class=\"alert alert-block alert-warning\"> <b class=\"fa fa-solid fa-exclamation-circle\"></b> <div> <p><b>Action Required</b></p> <p>Be sure to select the <tt>{{S2_DATABASE_NAME}}</tt> database from the drop-down list at the top of this notebook. It updates the <tt>connection_url</tt> which is used by the <tt>%%sql</tt> magic command and SQLAlchemy to connect to the selected database.</p> </div></div>"
]
},
{
Expand All @@ -154,7 +157,7 @@
"id": "4d4df1f2-9075-4558-8633-b80a578fec5b",
"metadata": {},
"source": [
"### Create a table according to kafka avro schema"
"### Create a Table Based on the Kafka Avro Schema"
]
},
{
Expand All @@ -181,9 +184,9 @@
"id": "569ba3ca-93e9-41ce-a9d8-ae802ae79485",
"metadata": {},
"source": [
"### Create kafka pipeline\n",
"### Create a Kafka Pipeline\n",
"\n",
"(Update schema registry mapping section according to your schema registry in format like: <i>'table column name'</i> <- <i>'schema registry field name'</i>)"
"<b>Note</b>: Update the schema registry mapping section according to your schema registry in the <i>'table column name'</i> <- <i>'schema registry field name'</i> format."
]
},
{
Expand Down Expand Up @@ -218,7 +221,7 @@
"id": "f80798c7-6160-4337-91b0-5bfabcf9e1bf",
"metadata": {},
"source": [
"### Test created pipeline"
"### Test the Created Pipeline"
]
},
{
Expand All @@ -238,7 +241,7 @@
"id": "b26eacfa-9d48-4f4f-9622-b9d80eaee868",
"metadata": {},
"source": [
"### Start pipeline"
"### Start the Pipeline"
]
},
{
Expand All @@ -259,7 +262,7 @@
"id": "bb7de29c-161d-48a8-ad62-97cf9d42278e",
"metadata": {},
"source": [
"### Stop pipeline"
"### Stop the Pipeline"
]
},
{
Expand All @@ -280,7 +283,7 @@
"id": "52388e5a-9f8d-43e1-90d6-d2f3dede536b",
"metadata": {},
"source": [
"### Select consumed events"
"### View Consumed Events"
]
},
{
Expand Down
1 change: 1 addition & 0 deletions resources/nb-check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
'chart-network': 'rgba(210, 255, 153, 0.25)',
'chart-scatter': 'rgba(124, 195, 235, 0.25)',
'clouds': 'rgba(124, 195, 235, 0.25)',
'confluent-logo': 'rgba(124, 195, 235, 0.25)',
'crystal-ball': 'rgba(255, 167, 103, 0.25)',
'database': 'rgba(235, 249, 245, 0.25)',
'dollar-circle': 'rgba(255, 167, 103, 0.25)',
Expand Down