Skip to content

Commit

Permalink
Merge pull request #1 from oyeliseiev-ua/aditya/updates
Browse files Browse the repository at this point in the history
Content Updates for the Notebook
  • Loading branch information
pmishchenko-ua authored Jan 23, 2024
2 parents 3085265 + 579bc01 commit 0b4dd8a
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 30 deletions.
2 changes: 1 addition & 1 deletion notebooks/confluent-cloud-integration/meta.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[meta]
title = "Ingest data from Confluent Cloud (Kafka)"
description = """
Provides step-by-step guidance on seamlessly ingesting data from Confluent Cloud (Kafka) to SingleStore Database using a robust pipeline. """
A step-by-step guide on seamlessly ingesting data from Confluent Cloud (Kafka) into your SingleStoreDB database using a robust pipeline. """
icon = "database"
tags = ["confluent", "training", "kafka", "pipeline", "realtime"]
destinations = ["spaces"]
61 changes: 32 additions & 29 deletions notebooks/confluent-cloud-integration/notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
"source": [
"<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(235, 249, 245, 0.25); padding: 5px;\">\n",
" <div id=\"icon-image\" style=\"width: 90px; height: 90px;\">\n",
" <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/database.png\" />\n",
" <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/confluent-logo.png\" />\n",
" </div>\n",
" <div id=\"text\" style=\"padding: 5px; margin-left: 10px;\">\n",
" <div id=\"badge\" style=\"display: inline-block; background-color: rgba(0, 0, 0, 0.15); border-radius: 4px; padding: 4px 8px; align-items: center; margin-top: 6px; margin-bottom: -2px; font-size: 80%\">SingleStore Notebooks</div>\n",
" <h1 style=\"font-weight: 500; margin: 8px 0 0 4px;\">Ingest data from Confluent Cloud (Kafka)</h1>\n",
" <h1 style=\"font-weight: 500; margin: 8px 0 0 4px;\">Ingest Data from Confluent Cloud (Kafka)</h1>\n",
" </div>\n",
"</div>"
]
Expand All @@ -22,7 +22,7 @@
"id": "d89587bf-6790-4a03-8d33-c5a1cb0fa94d",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-kafka-integration.png width=\"100%\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-kafka-integration.png width=\"100%\" />"
]
},
{
Expand All @@ -31,11 +31,13 @@
"id": "17ab18b5-176d-4b36-b653-30155855230d",
"metadata": {},
"source": [
"### Confluent Cluster (Kafka) set up\n",
"### Set Up a Kafka Cluster on Confluent Cloud\n",
"\n",
"Prior to initiating the integration process, it is essential to configure a Confluent Kafka cluster. Please refer to the provided <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html\">link</a> for a quick start guide.\n",
"Before initiating the integration process, it is essential to configure a Kafka cluster on Confluent Cloud. Refer to the <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html\">Quick Start for Confluent Cloud</a> guide for related information.\n",
"\n",
"- Once the cluster is created, proceed to create a topic named for example <b>'s2-topic'</b> and configure the value AVRO schema, as example choose proposed default:"
"Once the cluster is created, perform the following tasks:\n",
"\n",
"- Create a topic, for example <b>'s2-topic'</b>. On the topic overview page, select <b>Schema &gt; Set a schema &gt; Avro</b>, and add a new Avro schema. In this guide, the default schema is used."
]
},
{
Expand All @@ -44,7 +46,7 @@
"id": "35bee169-fec8-47eb-89b2-712bde11b745",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/kafka-value-schema.png width=\"1000\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/kafka-value-schema.png width=\"100%\" />"
]
},
{
Expand All @@ -53,7 +55,7 @@
"id": "679133dc-5c6f-4ea3-97f9-3317d4c98fb6",
"metadata": {},
"source": [
"- Create API Keys and save for later usage:"
"- Create API keys. The API key is displayed <b>only once</b>. Be sure to copy and securely store the API key."
]
},
{
Expand All @@ -62,7 +64,7 @@
"id": "4fba3d16-437c-40d0-903f-91560bc71019",
"metadata": {},
"source": [
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-api-key.png width=\"1000\">"
"<img src=https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/notebooks/confluent-cloud-integration/images/confluent-api-key.png width=\"100%\" />"
]
},
{
Expand All @@ -71,8 +73,8 @@
"id": "50713fc6-a965-4a70-b732-4c33ae0eaa5f",
"metadata": {},
"source": [
"- Go to 'Connectors' and create a sample producer <b>'datagen'</b> with 'Use an existing API key' option for the established Kafka topic <b>'s2-topic'</b>(<a href=\"https://docs.confluent.io/cloud/current/get-started/index.html#step-3-create-a-sample-producer\">Step 3</a> from quick guide). Configure the producer to utilize the same schema as the created topic.\n",
"- Launch <b>'datagen'</b> producer and check <b>'s2-topic'</b> has new messages."
"- On the left navigation pane, select <b>Connectors</b> and create a sample producer named <b>'datagen'</b> using the <b>Datagen Source</b> connector. In the <b>Topic selection</b> pane, select the <b>'s2-topic'</b> created earlier. In the <b>Kafka credentials</b> pane, select the <b>Use an existing API key</b> option. Configure the producer to use the same schema as the one in the created topic. Refer to <a href=\"https://docs.confluent.io/cloud/current/get-started/index.html#step-3-create-a-sample-producer\">Step 3: Create a sample producer</a> for more information. \n",
"- Launch the <b>'datagen'</b> producer and verify that the <b>'s2-topic'</b> has new messages."
]
},
{
Expand All @@ -81,20 +83,21 @@
"id": "2a05848e-fad9-47c7-8a4b-7c6fd627ac44",
"metadata": {},
"source": [
"### Set up variables\n",
"### Set Up Variables\n",
"\n",
"Choose <b>S2_DATABASE_NAME</b>, <b>S2_TABLE_NAME</b> and <b>S2_PIPELINE_NAME</b> to use for integration\n",
"Use the <b>S2_DATABASE_NAME</b>, <b>S2_TABLE_NAME</b>, and <b>S2_PIPELINE_NAME</b> variables for integration.\n",
"\n",
"### Copy data from Confluent Cloud\n",
"- Set up <b>CONFLUENT_KAFKA_TOPIC_NAME</b> - put created topic name ('s2-topic')\n",
"- Set up <b>CONFLUENT_API_KEY</b> and <b>CONFLUENT_API_SECRET</b> - put API key and secret\n",
"### Copy Data from Confluent Cloud\n",
"- Assign the topic name <b>'s2-topic'</b> to the <b>CONFLUENT_KAFKA_TOPIC_NAME</b> variable.\n",
"- Specify the API key and secret using the <b>CONFLUENT_API_KEY</b> and <b>CONFLUENT_API_SECRET</b> variables, respectively.\n",
"\n",
"Go to 'Clients' -> choose language (for example java) and set up following variables:\n",
"On the left navigation pane, select <b>Clients</b>. Select a language (for example Java), and configure the following variables:\n",
"- <b>CONFLUENT_CLUSTER_BOOTSTRAP_SERVER</b> from <b>bootstrap.servers</b>\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_URL</b> from <b>schema.registry.url</b>\n",
"\n",
"Click 'Create Schema Registry API key' to create schema api key and set up:\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_KEY</b> and <b>CONFLUENT_SCHEMA_REGISTRY_SECRET</b>"
"Select <b>Create Schema Registry API key</b> to create a schema API key and configure the following variables:\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_KEY</b>\n",
"- <b>CONFLUENT_SCHEMA_REGISTRY_SECRET</b>"
]
},
{
Expand Down Expand Up @@ -123,7 +126,7 @@
"id": "db9054b4-aec8-4aae-8069-efa042f759d5",
"metadata": {},
"source": [
"### Create database"
"### Create a Database"
]
},
{
Expand All @@ -145,7 +148,7 @@
"id": "279607d8-35c2-4a84-a4fa-50f5156269f1",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-warning\"> <b class=\"fa fa-solid fa-exclamation-circle\"></b> <div> <p><b>Action Required</b></p> <p>Make sure to select the <tt>{{S2_DATABASE_NAME}}</tt> database from the drop-down menu at the top of this notebook. It updates the <tt>connection_url</tt> which is used by the <tt>%%sql</tt> magic command and SQLAlchemy to make connections to the selected database.</p> </div></div>"
"<div class=\"alert alert-block alert-warning\"> <b class=\"fa fa-solid fa-exclamation-circle\"></b> <div> <p><b>Action Required</b></p> <p>Be sure to select the <tt>{{S2_DATABASE_NAME}}</tt> database from the drop-down list at the top of this notebook. It updates the <tt>connection_url</tt> which is used by the <tt>%%sql</tt> magic command and SQLAlchemy to connect to the selected database.</p> </div></div>"
]
},
{
Expand All @@ -154,7 +157,7 @@
"id": "4d4df1f2-9075-4558-8633-b80a578fec5b",
"metadata": {},
"source": [
"### Create a table according to kafka avro schema"
"### Create a Table Based on the Kafka Avro Schema"
]
},
{
Expand All @@ -181,9 +184,9 @@
"id": "569ba3ca-93e9-41ce-a9d8-ae802ae79485",
"metadata": {},
"source": [
"### Create kafka pipeline\n",
"### Create a Kafka Pipeline\n",
"\n",
"(Update schema registry mapping section according to your schema registry in format like: <i>'table column name'</i> <- <i>'schema registry field name'</i>)"
"<b>Note</b>: Update the schema registry mapping section according to your schema registry in the <i>'table column name'</i> <- <i>'schema registry field name'</i> format."
]
},
{
Expand Down Expand Up @@ -218,7 +221,7 @@
"id": "f80798c7-6160-4337-91b0-5bfabcf9e1bf",
"metadata": {},
"source": [
"### Test created pipeline"
"### Test the Created Pipeline"
]
},
{
Expand All @@ -238,7 +241,7 @@
"id": "b26eacfa-9d48-4f4f-9622-b9d80eaee868",
"metadata": {},
"source": [
"### Start pipeline"
"### Start the Pipeline"
]
},
{
Expand All @@ -259,7 +262,7 @@
"id": "bb7de29c-161d-48a8-ad62-97cf9d42278e",
"metadata": {},
"source": [
"### Stop pipeline"
"### Stop the Pipeline"
]
},
{
Expand All @@ -280,7 +283,7 @@
"id": "52388e5a-9f8d-43e1-90d6-d2f3dede536b",
"metadata": {},
"source": [
"### Select consumed events"
"### View Consumed Events"
]
},
{
Expand All @@ -301,7 +304,7 @@
"metadata": {},
"source": [
"<div id=\"singlestore-footer\" style=\"background-color: rgba(194, 193, 199, 0.25); height:2px; margin-bottom:10px\"></div>\n",
"<div><img src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/singlestore-logo-grey.png\" style=\"padding: 0px; margin: 0px; height: 24px\"/></div>"
"<div><img src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/singlestore-logo-grey.png\" style=\"padding: 0px; margin: 0px; height: 24px\" /></div>"
]
}
],
Expand Down

0 comments on commit 0b4dd8a

Please sign in to comment.