From ade6de8d215a0967c33b32aa1f39c13fc12db3a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C4=B0hsan=20Baran=20S=C3=96NMEZ?= Date: Wed, 8 Nov 2023 22:21:01 +0300 Subject: [PATCH 01/16] Add the step of installing dependencies to the multi-app run steps in the quickstart tutorial document. Add note for python3 problem in Windows environments. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: İhsan Baran SÖNMEZ --- .../quickstarts/pubsub-quickstart.md | 2 ++ .../quickstarts/serviceinvocation-quickstart.md | 1 + .../quickstarts/statemanagement-quickstart.md | 17 ++++++++++++----- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md index 06f6402cf2d..63ba887755d 100644 --- a/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md @@ -60,6 +60,8 @@ With the following command, simultaneously run the following services alongside ```bash dapr run -f . ``` +> **Note**: Since Python3.exe is not defined in Windows, you may need to change `python3` to `python` in the [`dapr.yaml`]({{< ref "#dapryaml-multi-app-run-template-file" >}}) file before running `dapr run -f .` + **Expected output** diff --git a/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md index c33c529f952..d75e1a7a6e3 100644 --- a/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md @@ -57,6 +57,7 @@ With the following command, simultaneously run the following services alongside ```bash dapr run -f . ``` +> **Note**: Since Python3.exe is not defined in Windows, you may need to change `python3` to `python` in the [`dapr.yaml`]({{< ref "#dapryaml-multi-app-run-template-file" >}}) file before running `dapr run -f .` **Expected output** diff --git a/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md index 4d1224c4eb7..26d63ae9826 100644 --- a/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md @@ -51,8 +51,9 @@ cd state_management/python/sdk/order-processor Run the `order-processor` service alongside a Dapr sidecar using [Multi-App Run]({{< ref multi-app-dapr-run >}}). ```bash -dapr run -f +dapr run -f . ``` +> **Note**: Since Python3.exe is not defined in Windows, you may need to change `python3` to `python` in the [`dapr.yaml`]({{< ref "#dapryaml-multi-app-run-template-file" >}}) file before running `dapr run -f .` The `order-processor` service writes, reads, and deletes an `orderId` key/value pair to the `statestore` instance [defined in the `statestore.yaml` component]({{< ref "#statestoreyaml-component-file" >}}). As soon as the service starts, it performs a loop. @@ -173,7 +174,7 @@ cd state_management/javascript/sdk/order-processor Run the `order-processor` service alongside a Dapr sidecar. ```bash -dapr run -f +dapr run -f . ``` The `order-processor` service writes, reads, and deletes an `orderId` key/value pair to the `statestore` instance [defined in the `statestore.yaml` component]({{< ref "#statestoreyaml-component-file" >}}). As soon as the service starts, it performs a loop. @@ -299,7 +300,7 @@ cd state_management/csharp/sdk/order-processor Run the `order-processor` service alongside a Dapr sidecar. ```bash -dapr run -f +dapr run -f . ``` The `order-processor` service writes, reads, and deletes an `orderId` key/value pair to the `statestore` instance [defined in the `statestore.yaml` component]({{< ref "#statestoreyaml-component-file" >}}). As soon as the service starts, it performs a loop. @@ -423,10 +424,16 @@ In a terminal window, navigate to the `order-processor` directory. cd state_management/java/sdk/order-processor ``` +Install the dependencies: + +```bash +mvn clean install +``` + Run the `order-processor` service alongside a Dapr sidecar. ```bash -dapr run -f +dapr run -f . ``` The `order-processor` service writes, reads, and deletes an `orderId` key/value pair to the `statestore` instance [defined in the `statestore.yaml` component]({{< ref "#statestoreyaml-component-file" >}}). As soon as the service starts, it performs a loop. @@ -553,7 +560,7 @@ cd state_management/go/sdk/order-processor Run the `order-processor` service alongside a Dapr sidecar. ```bash -dapr run -f +dapr run -f . ``` The `order-processor` service writes, reads, and deletes an `orderId` key/value pair to the `statestore` instance [defined in the `statestore.yaml` component]({{< ref "#statestoreyaml-component-file" >}}). As soon as the service starts, it performs a loop. From 3815827b13bd359c26a060d98bb68d8fc161b877 Mon Sep 17 00:00:00 2001 From: Oliver Tomlinson Date: Wed, 22 Nov 2023 01:43:15 +0000 Subject: [PATCH 02/16] include instructions on how to debug a local app that is part of a docker compose session (#3881) --- .../debugging/debugging-docker-compose.md | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 daprdocs/content/en/developing-applications/debugging/debugging-docker-compose.md diff --git a/daprdocs/content/en/developing-applications/debugging/debugging-docker-compose.md b/daprdocs/content/en/developing-applications/debugging/debugging-docker-compose.md new file mode 100644 index 00000000000..ad2af8cab5d --- /dev/null +++ b/daprdocs/content/en/developing-applications/debugging/debugging-docker-compose.md @@ -0,0 +1,81 @@ +--- +type: docs +title: "Debugging Dapr Apps running in Docker Compose" +linkTitle: "Debugging Docker Compose" +weight: 300 +description: "Debug Dapr apps locally which are part of a Docker Compose deployment" +--- + +The goal of this article is to demonstrate a way to debug one or more daprised applications (via your IDE, locally) while remaining integrated with the other applications that have deployed in the docker compose environment. + +Let's take the minimal example of a docker compose file which contains just two services : +- `nodeapp` - your app +- `nodeapp-dapr` - the dapr sidecar process to your `nodeapp` service + +#### compose.yml +```yaml +services: + nodeapp: + build: ./node + ports: + - "50001:50001" + networks: + - hello-dapr + nodeapp-dapr: + image: "daprio/daprd:edge" + command: [ + "./daprd", + "--app-id", "nodeapp", + "--app-port", "3000", + "--resources-path", "./components" + ] + volumes: + - "./components/:/components" + depends_on: + - nodeapp + network_mode: "service:nodeapp" +networks: + hello-dapr +``` + +When you run this docker file with `docker compose -f compose.yml up` this will deploy to Docker and run as normal. + +But how do we debug the `nodeapp` while still integrated to the running dapr sidecar process, and anything else that you may have deployed via the Docker compose file? + +Lets start by introducing a *second* docker compose file called `compose.debug.yml`. This second compose file will augment with the first compose file when the `up` command is ran. + +#### compose.debug.yml +```yaml +services: + nodeapp: # Isolate the nodeapp by removing its ports and taking it off the network + ports: !reset [] + networks: !reset + - "" + nodeapp-dapr: + command: ["./daprd", + "--app-id", "nodeapp", + "--app-port", "8080", # This must match the port that your app is exposed on when debugging in the IDE + "--resources-path", "./components", + "--app-channel-address", "host.docker.internal"] # Make the sidecar look on the host for the App Channel + network_mode: !reset "" # Reset the network_mode... + networks: # ... so that the sidecar can go into the normal network + - hello-dapr + ports: + - "3500:3500" # Expose the HTTP port to the host + - "50001:50001" # Expose the GRPC port to the host (Dapr Worfklows depends upon the GRPC channel) + +``` + +Next, ensure that your `nodeapp` is running/debugging in your IDE of choice, and is exposed on the same port that you specifed above in the `compose.debug.yml` - In the example above this is set to port `8080`. + +Next, stop any existing compose sessions you may have started, and run the following command to run both docker compose files combined together : + +`docker compose -f compose.yml -f compose.debug.yml up` + +You should now find that the dapr sidecar and your debugging app will have bi-directional communication with each other as if they were running together as normal in the Docker compose environment. + +**Note** : It's important to highlight that the `nodeapp` service in the docker compose environment is actually still running, however it has been removed from the docker network so it is effectively orphaned as nothing can communicate to it. + +**Demo** : Watch this video on how to debug local Dapr apps with Docker Compose + + \ No newline at end of file From 646016926332cd205aa08784de835d6bfc858fa6 Mon Sep 17 00:00:00 2001 From: Guillermo Caracuel <633810+gcaracuel@users.noreply.github.com> Date: Sat, 2 Dec 2023 06:53:04 +0100 Subject: [PATCH 03/16] Fix Subscription docs to match Kubernetes CRDs (#3888) Signed-off-by: gcaracuel <633810+gcaracuel@users.noreply.github.com> --- .../resource-specs/subscription-schema.md | 24 +++++++++---------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/daprdocs/content/en/reference/resource-specs/subscription-schema.md b/daprdocs/content/en/reference/resource-specs/subscription-schema.md index 55b8bc76f5f..bd5fc8263a8 100644 --- a/daprdocs/content/en/reference/resource-specs/subscription-schema.md +++ b/daprdocs/content/en/reference/resource-specs/subscription-schema.md @@ -21,18 +21,17 @@ kind: Subscription metadata: name: spec: - version: v2alpha1 topic: # Required routes: # Required - rules: - - match: + - match: path: pubsubname: # Required - deadlettertopic: # Optional - bulksubscribe: # Optional - - enabled: - - maxmessages: - - maxawaitduration: + deadLetterTopic: # Optional + bulkSubscribe: # Optional + - enabled: + - maxMessagesCount: + - maxAwaitDurationMs: scopes: - ``` @@ -42,10 +41,10 @@ scopes: | Field | Required | Details | Example | |--------------------|:--------:|---------|---------| | topic | Y | The name of the topic to which your component subscribes. | `orders` | -| routes | Y | The routes configuration for this topic, including specifying the condition for sending a message to a specific path. Includes the following fields:
  • match: _Optional._ The CEL expression used to match the event. If not specified, the route is considered the default.
  • path: The path for events that match this rule.
The endpoint to which all topic messages are sent. | `match: event.type == "widget"`
`path: /widgets` | +| routes | Y | The routes configuration for this topic, including specifying the condition for sending a message to a specific path. Includes the following fields:
  • match: The CEL expression used to match the event. If not specified, the route is considered the default.
  • path: The path for events that match this rule.
The endpoint to which all topic messages are sent. | `match: event.type == "widget"`
`path: /widgets` | | pubsubname | N | The name of your pub/sub component. | `pubsub` | -| deadlettertopic | N | The name of the dead letter topic that forwards undeliverable messages. | `poisonMessages` | -| bulksubscribe | N | Enable bulk subscribe properties. | `true`, `false` | +| deadLetterTopic | N | The name of the dead letter topic that forwards undeliverable messages. | `poisonMessages` | +| bulkSubscribe | N | Enable bulk subscribe properties. | `true`, `false` | ## `v1alpha1` format @@ -58,15 +57,14 @@ kind: Subscription metadata: name: spec: - version: v1alpha1 topic: # Required route: # Required pubsubname: # Required deadLetterTopic: # Optional bulkSubscribe: # Optional - enabled: - - maxmessages: - - maxawaitduration: + - maxMessagesCount: + - maxAwaitDurationMs: scopes: - ``` From 2bbff4b3ea9e9434e332d1ec1c36e9544d9ccc38 Mon Sep 17 00:00:00 2001 From: Stuart Leeks Date: Thu, 7 Dec 2023 09:05:14 +0000 Subject: [PATCH 04/16] Fix method name `startServer` method doesn't exist on JavaScript `DaprServer` (method name is `start` - see https://docs.dapr.io/developing-applications/sdks/js/js-server/#bindings-api) Signed-off-by: Stuart Leeks --- .../building-blocks/bindings/howto-triggers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md index 56a24b0aece..70320818a3e 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md +++ b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md @@ -253,7 +253,7 @@ async function start() { } }); await server.binding.receive('checkout', async (orderId) => console.log(`Received Message: ${JSON.stringify(orderId)}`)); - await server.startServer(); + await server.start(); } ``` @@ -292,4 +292,4 @@ Event delivery guarantees are controlled by the binding implementation. Dependin - [Bindings building block]({{< ref bindings >}}) - [Bindings API]({{< ref bindings_api.md >}}) - [Components concept]({{< ref components-concept.md >}}) -- [Supported bindings]({{< ref supported-bindings >}}) \ No newline at end of file +- [Supported bindings]({{< ref supported-bindings >}}) From 3dd8edee51a7d9f0202182a214126e8b9f4117d8 Mon Sep 17 00:00:00 2001 From: joshvanl Date: Tue, 19 Dec 2023 16:40:55 +0000 Subject: [PATCH 05/16] Component Hot Reloading Adds documentation for Component Hot Reloading Part of https://github.com/dapr/dapr/issues/1172 Signed-off-by: joshvanl --- daprdocs/content/en/concepts/components-concept.md | 8 ++++++++ .../en/operations/support/support-preview-features.md | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index b7248d65684..98668207690 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -52,6 +52,14 @@ For example: For more information read [Pluggable components overview]({{< ref "pluggable-components-overview" >}}) +## Hot Reloading + +With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), Components are able to be "hot reloaded" at runtime. +This means that you can update component configuration without restarting the Dapr runtime. +Component reloading happens when a Component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +When a component is updated, the Component is first closed, and then reinitialized using the new configuration. +There will be a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. + ## Available component types The following are the component types provided by Dapr: diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index 9c9fff69010..b62ea901e40 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | - +| ** Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self Hosted mode.| N/A | [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From 149325b0d42f922c40bbb6781ed2eb4ca01ba51e Mon Sep 17 00:00:00 2001 From: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Date: Wed, 20 Dec 2023 11:03:29 -0500 Subject: [PATCH 06/16] Update `dapr init` CLI reference with Docker network examples (#3901) * update dapr init doc Signed-off-by: Hannah Hunter * Update daprdocs/content/en/reference/cli/dapr-init.md Signed-off-by: Mark Fussell --------- Signed-off-by: Hannah Hunter Signed-off-by: Mark Fussell Co-authored-by: Mark Fussell --- .../content/en/reference/cli/dapr-init.md | 74 +++++++++++++++---- 1 file changed, 60 insertions(+), 14 deletions(-) diff --git a/daprdocs/content/en/reference/cli/dapr-init.md b/daprdocs/content/en/reference/cli/dapr-init.md index bdb93bf5d15..7cc2d86ad7d 100644 --- a/daprdocs/content/en/reference/cli/dapr-init.md +++ b/daprdocs/content/en/reference/cli/dapr-init.md @@ -49,32 +49,44 @@ dapr init [flags] ### Examples -#### Self hosted environment +{{< tabs "Self-hosted" "Kubernetes" >}} -Install Dapr by pulling container images for Placement, Redis and Zipkin. By default these images are pulled from Docker Hub. To switch to Dapr Github container registry as the default registry, set the `DAPR_DEFAULT_IMAGE_REGISTRY` environment variable value to be `GHCR`. To switch back to Docker Hub as default registry, unset this environment variable. +{{% codetab %}} + +**Install** + +Install Dapr by pulling container images for Placement, Redis, and Zipkin. By default, these images are pulled from Docker Hub. ```bash dapr init ``` -You can also specify a specific runtime version. Be default, the latest version is used. +Dapr can also run [Slim self-hosted mode]({{< ref self-hosted-no-docker.md >}}), without Docker. ```bash -dapr init --runtime-version 1.4.0 +dapr init -s ``` -You can also install Dapr with a particular image variant, for example: [mariner]({{< ref "kubernetes-deploy.md#using-mariner-based-images" >}}). +> To switch to Dapr Github container registry as the default registry, set the `DAPR_DEFAULT_IMAGE_REGISTRY` environment variable value to be `GHCR`. To switch back to Docker Hub as default registry, unset this environment variable. + +**Specify a runtime version** + +You can also specify a specific runtime version. Be default, the latest version is used. ```bash -dapr init --image-variant mariner +dapr init --runtime-version 1.13.0 ``` -Dapr can also run [Slim self-hosted mode]({{< ref self-hosted-no-docker.md >}}) without Docker. +**Install with image variant** + +You can also install Dapr with a particular image variant, for example: [mariner]({{< ref "kubernetes-deploy.md#using-mariner-based-images" >}}). ```bash -dapr init -s +dapr init --image-variant mariner ``` +**Use Dapr Installer Bundle** + In an offline or airgap environment, you can [download a Dapr Installer Bundle](https://github.com/dapr/installer-bundle/releases) and use this to install Dapr instead of pulling images from the network. ```bash @@ -87,17 +99,17 @@ Dapr can also run in slim self-hosted mode without Docker in an airgap environme dapr init -s --from-dir ``` -You can also specify a private registry to pull container images from. These images need to be published to private registries as shown below to enable Dapr CLI to pull them successfully via the `dapr init` command - +**Specify private registry** + +You can also specify a private registry to pull container images from. These images need to be published to private registries as shown below to enable Dapr CLI to pull them successfully via the `dapr init` command: 1. Dapr runtime container image(dapr) (Used to run Placement) - dapr/dapr: 2. Redis container image(rejson) - dapr/3rdparty/rejson 3. Zipkin container image(zipkin) - dapr/3rdparty/zipkin -> All the required images used by Dapr needs to be under the`dapr` path. +All the required images used by Dapr needs to be under the `dapr` path. The 3rd party images have to be published under `dapr/3rdparty` path. -> The 3rd party images have to be published under `dapr/3rdparty` path. - -> image-registry uri follows this format - `docker.io/` +`image-registry` uri follows the `docker.io/` format. ```bash dapr init --image-registry docker.io/username @@ -114,7 +126,37 @@ You can specify a different container runtime while setting up Dapr. If you omit dapr init --container-runtime podman ``` -#### Kubernetes environment +**Use Docker network** + +You can deploy local containers into Docker networks, which is useful for deploying into separate networks or when using Docker Compose for local development to deploy applications. + +Create the Docker network. + +```bash +docker network create mynet +``` + +Initialize Dapr and specify the created Docker network. + +```bash +dapr init --network mynet +``` + +Verify all containers are running in the specified network. + +```bash +docker ps +``` + +Uninstall Dapr from that Docker network. + +```bash +dapr uninstall --all --network mynet +``` + +{{% /codetab %}} + +{{% codetab %}} ```bash dapr init -k @@ -149,3 +191,7 @@ Scenario 2 : dapr image hosted under a new/different directory in private regist ```bash dapr init -k --image-registry docker.io/username/ ``` + +{{% /codetab %}} + +{{< /tabs >}} From 00593a517ca4b6e3df38743da485c726f3aa59eb Mon Sep 17 00:00:00 2001 From: Hannah Hunter Date: Thu, 21 Dec 2023 12:33:47 -0500 Subject: [PATCH 07/16] cross link for debugging and viewing logs Signed-off-by: Hannah Hunter --- .../debugging/debug-k8s/debug-daprd.md | 8 +++++++- .../en/operations/troubleshooting/logs-troubleshooting.md | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md b/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md index 3b13efc5aeb..283f8f3c75c 100644 --- a/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md +++ b/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md @@ -11,7 +11,7 @@ description: "How to debug the Dapr sidecar (daprd) on your Kubernetes cluster" Sometimes it is necessary to understand what's going on in the Dapr sidecar (daprd), which runs as a sidecar next to your application, especially when you diagnose your Dapr application and wonder if there's something wrong in Dapr itself. Additionally, you may be developing a new feature for Dapr on Kubernetes and want to debug your code. -his guide will cover how to use built-in Dapr debugging to debug the Dapr sidecar in your Kubernetes pods. +This guide covers how to use built-in Dapr debugging to debug the Dapr sidecar in your Kubernetes pods. To learn how to view logs and troubleshoot Dapr in Kubernetes, see the [Configure and view Dapr logs guide]({{< ref "logs-troubleshooting.md#logs-in-kubernetes-mode" >}}) ## Pre-requisites @@ -87,6 +87,12 @@ Forwarding from [::1]:40000 -> 40000 All done. Now you can point to port 40000 and start a remote debug session to daprd from your favorite IDE. +## Watch the demo + +See the presentation on troubleshooting Dapr on Kubernetes in the [Dapr Community Call #36](https://youtu.be/pniLPRbuLD8?si=bGid7oYSp9cThtiI&t=838). + + + ## Related links - [Overview of Dapr on Kubernetes]({{< ref kubernetes-overview >}}) diff --git a/daprdocs/content/en/operations/troubleshooting/logs-troubleshooting.md b/daprdocs/content/en/operations/troubleshooting/logs-troubleshooting.md index d2316a1fd84..14eb822c194 100644 --- a/daprdocs/content/en/operations/troubleshooting/logs-troubleshooting.md +++ b/daprdocs/content/en/operations/troubleshooting/logs-troubleshooting.md @@ -73,6 +73,8 @@ dapr run node myapp.js ## Logs in Kubernetes mode +> [Learn how to debug `daprd` on Kubernetes.]({{< ref "debug-daprd.md" >}}) + You can set the log level individually for every sidecar by providing the following annotation in your pod spec template: ```yml From 3cac886e6b90f6f9be24e573bf54d231b06eb96d Mon Sep 17 00:00:00 2001 From: Hannah Hunter Date: Thu, 21 Dec 2023 14:53:45 -0500 Subject: [PATCH 08/16] add debug commands Signed-off-by: Hannah Hunter --- .../debugging/debug-k8s/debug-daprd.md | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md b/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md index 283f8f3c75c..24aca6c85a4 100644 --- a/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md +++ b/daprdocs/content/en/developing-applications/debugging/debug-k8s/debug-daprd.md @@ -87,6 +87,56 @@ Forwarding from [::1]:40000 -> 40000 All done. Now you can point to port 40000 and start a remote debug session to daprd from your favorite IDE. +## Commonly used `kubectl` commands + +Use the following common `kubectl` commands when debugging daprd and applications running on Kubernetes. + +Get all pods, events, and services: + +```bash +kubectl get all +kubectl get all --n +kubectl get all --all-namespaces +``` + +Get each specifically: + +```bash +kubectl get pods +``` + +```bash +kubectl get events --n +kubectl get events --sort-by=.metadata.creationTimestamp --n +``` + +```bash +kubectl get services +``` + +Check logs: + +```bash +kubectl logs daprd +kubectl logs +kuebctl logs daprd +kubectl logs +``` + +```bash +kubectl describe pod +kubectl describe deploy +kubectl describe replicaset +``` + +Restart a pod by running the following command: + +```bash +kubectl delete pod +``` + +This causes the `replicaset` controller to restart the pod after the delete. + ## Watch the demo See the presentation on troubleshooting Dapr on Kubernetes in the [Dapr Community Call #36](https://youtu.be/pniLPRbuLD8?si=bGid7oYSp9cThtiI&t=838). From 4822030d6a6fa4f37390a67ecc1d9afceac56fb3 Mon Sep 17 00:00:00 2001 From: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Date: Thu, 4 Jan 2024 15:44:51 -0500 Subject: [PATCH 09/16] Submodules into 1.13 (#3927) * javascript submodule for 1.13 Signed-off-by: Hannah Hunter * go submodules into 1.13 Signed-off-by: Hannah Hunter * java submodules into 1.13 Signed-off-by: Hannah Hunter * dotnet submodules for 1.13 Signed-off-by: Hannah Hunter * python submodule Signed-off-by: Hannah Hunter --------- Signed-off-by: Hannah Hunter Co-authored-by: Mark Fussell --- sdkdocs/dotnet | 2 +- sdkdocs/go | 2 +- sdkdocs/java | 2 +- sdkdocs/js | 2 +- sdkdocs/python | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/sdkdocs/dotnet b/sdkdocs/dotnet index 99d874a2b13..10ef81873b3 160000 --- a/sdkdocs/dotnet +++ b/sdkdocs/dotnet @@ -1 +1 @@ -Subproject commit 99d874a2b138af020df099a0fc0a09a7d0597fae +Subproject commit 10ef81873b3448fb136c73ad26a9fd2768954c2f diff --git a/sdkdocs/go b/sdkdocs/go index e16e0350a52..04f7b595b6d 160000 --- a/sdkdocs/go +++ b/sdkdocs/go @@ -1 +1 @@ -Subproject commit e16e0350a52349b5a05138edc0b58e3be78ee753 +Subproject commit 04f7b595b6d19bbf1c42a3364992016c3ae3e40e diff --git a/sdkdocs/java b/sdkdocs/java index 5e45aa86b81..6759f19f837 160000 --- a/sdkdocs/java +++ b/sdkdocs/java @@ -1 +1 @@ -Subproject commit 5e45aa86b81748bf1e6efdbf7f52c20645a12435 +Subproject commit 6759f19f8374c7c550c709b1fe8118ce738280a8 diff --git a/sdkdocs/js b/sdkdocs/js index df7eff281a5..6e89215f5ca 160000 --- a/sdkdocs/js +++ b/sdkdocs/js @@ -1 +1 @@ -Subproject commit df7eff281a5a1395a7967c658a5707e8dfb2b99e +Subproject commit 6e89215f5ca26f8f4d109424e2cad7792b9d8a28 diff --git a/sdkdocs/python b/sdkdocs/python index 6171b67db60..c08e71494a6 160000 --- a/sdkdocs/python +++ b/sdkdocs/python @@ -1 +1 @@ -Subproject commit 6171b67db60d51704ed8425ae71dda9226bf1255 +Subproject commit c08e71494a644f9ff875941c669c6a1e1f3a3340 From 1a24364b3484e150ee7e4e47df9d8c4c78c3a920 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Tue, 9 Jan 2024 13:37:19 +0000 Subject: [PATCH 10/16] Apply suggestions from code review Co-authored-by: Mark Fussell Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index 98668207690..48365186d26 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -54,11 +54,11 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com ## Hot Reloading -With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), Components are able to be "hot reloaded" at runtime. +With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. -Component reloading happens when a Component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. When a component is updated, the Component is first closed, and then reinitialized using the new configuration. -There will be a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. +There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. ## Available component types From 7271911f32d0e83101f55f2e2fb9e4426f83a66f Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Tue, 9 Jan 2024 13:37:50 +0000 Subject: [PATCH 11/16] Update daprdocs/content/en/concepts/components-concept.md Co-authored-by: Mark Fussell Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index 48365186d26..db5122e0a93 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -57,7 +57,7 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. -When a component is updated, the Component is first closed, and then reinitialized using the new configuration. +When a component is updated, the component is first closed, and then reinitialized using the new configuration. There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. ## Available component types From d00938488cc1d624dcbacd50c7a655cb703f39b2 Mon Sep 17 00:00:00 2001 From: joshvanl Date: Tue, 9 Jan 2024 14:13:55 +0000 Subject: [PATCH 12/16] Update compoents-updates.md with hot reloading feature Signed-off-by: joshvanl --- .../en/operations/components/component-updates.md | 15 ++++++++++++--- .../support/support-preview-features.md | 2 +- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index 583eee28e21..eb5c51ef413 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -6,18 +6,27 @@ weight: 300 description: "Updating deployed components used by applications" --- -When making an update to an existing deployed component used by an application, Dapr does not update the component automatically. The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. How this done depends on the hosting environment. +When making an update to an existing deployed component used by an application, Dapr does not update the component automatically unless the `HotReload` feature gate is enabled. +The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. +How this done depends on the hosting environment. + +{{% alert title="Note" color="primary" %}} +Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. +This is enabled by via the [`HotReload` feature gate]({{< ref "support-preview-features.md" >}}). +All component types are supported for hot reloading. +This feature is currently in preview. +{{% /alert %}} ## Kubernetes When running in Kubernetes, the process of updating a component involves two steps: 1. Applying the new component YAML to the desired namespace -2. Performing a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component +2. (Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}})), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component ## Self Hosted -When running in Self Hosted mode, the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. +Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. ## Further reading - [Components concept]({{< ref components-concept.md >}}) diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index b62ea901e40..c960c2e5f19 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | -| ** Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self Hosted mode.| N/A | [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | +| **Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From afec1f1d58780f9ecc2a996b716a52849d86ecb5 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Thu, 11 Jan 2024 11:52:36 +0000 Subject: [PATCH 13/16] Apply suggestions from code review Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/concepts/components-concept.md | 4 ++-- .../content/en/operations/components/component-updates.md | 8 ++++---- .../en/operations/support/support-preview-features.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index db5122e0a93..cb2f9e7ec51 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -56,9 +56,9 @@ For more information read [Pluggable components overview]({{< ref "pluggable-com With the [`HotReload` feature enabled]({{< ref "support-preview-features.md" >}}), components are able to be "hot reloaded" at runtime. This means that you can update component configuration without restarting the Dapr runtime. -Component reloading happens when a component resource is created, updated, or deleted, either in the Kubernetes API or in Self Hosted mode when a file is changed in the `resources` directory. +Component reloading occurs when a component resource is created, updated, or deleted, either in the Kubernetes API or in self-hosted mode when a file is changed in the `resources` directory. When a component is updated, the component is first closed, and then reinitialized using the new configuration. -There is a short period of time where the component is unavailable during the duration of the reload until the component is reinitialized. +The component is unavailable for a short period of time during reload and reinitialization. ## Available component types diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index eb5c51ef413..7f6dd50780e 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -8,7 +8,7 @@ description: "Updating deployed components used by applications" When making an update to an existing deployed component used by an application, Dapr does not update the component automatically unless the `HotReload` feature gate is enabled. The Dapr sidecar needs to be restarted in order to pick up the latest version of the component. -How this done depends on the hosting environment. +How this is done depends on the hosting environment. {{% alert title="Note" color="primary" %}} Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. @@ -21,12 +21,12 @@ This feature is currently in preview. When running in Kubernetes, the process of updating a component involves two steps: -1. Applying the new component YAML to the desired namespace -2. (Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}})), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component +1. Apply the new component YAML to the desired namespace +1. Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), perform a [rollout restart operation](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources) on your deployments to pick up the latest component ## Self Hosted -Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping the `daprd` process and starting it again to pick up the latest component. +Unless the [`HotReload` feature gate is enabled]({{< ref "support-preview-features.md" >}}), the process of updating a component involves a single step of stopping and restarting the `daprd` process to pick up the latest component. ## Further reading - [Components concept]({{< ref components-concept.md >}}) diff --git a/daprdocs/content/en/operations/support/support-preview-features.md b/daprdocs/content/en/operations/support/support-preview-features.md index c960c2e5f19..cd7fa499181 100644 --- a/daprdocs/content/en/operations/support/support-preview-features.md +++ b/daprdocs/content/en/operations/support/support-preview-features.md @@ -22,4 +22,4 @@ For CLI there is no explicit opt-in, just the version that this was first made a | **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 | | **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 | | **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 | -| **Component Hot Reloading** | Allows for dapr loaded Components to be "hot reloaded". A Component spec will be reloaded when it is created/updated/deleted in Kubernetes or on file when running in Self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | +| **Component Hot Reloading** | Allows for Dapr-loaded components to be "hot reloaded". A component spec is reloaded when it is created/updated/deleted in Kubernetes or on file when running in self-hosted mode.| `HotReload`| [Hot Reloading]({{< ref components-concept.md >}}) | v1.13 | From be597e9b356713dcbbf7f26d668411bbf383fb20 Mon Sep 17 00:00:00 2001 From: Josh van Leeuwen Date: Thu, 11 Jan 2024 11:52:51 +0000 Subject: [PATCH 14/16] Update daprdocs/content/en/operations/components/component-updates.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Josh van Leeuwen --- daprdocs/content/en/operations/components/component-updates.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/operations/components/component-updates.md b/daprdocs/content/en/operations/components/component-updates.md index 7f6dd50780e..fb8a9313ce6 100644 --- a/daprdocs/content/en/operations/components/component-updates.md +++ b/daprdocs/content/en/operations/components/component-updates.md @@ -11,7 +11,7 @@ The Dapr sidecar needs to be restarted in order to pick up the latest version of How this is done depends on the hosting environment. {{% alert title="Note" color="primary" %}} -Dapr can be made to "hot reload" components where updates will be picked up automatically without needing a restart. +Dapr can be made to "hot reload" components, where updates are picked up automatically without needing a restart. This is enabled by via the [`HotReload` feature gate]({{< ref "support-preview-features.md" >}}). All component types are supported for hot reloading. This feature is currently in preview. From ae4e13754b46c86f4aff55419f3fab06b2be2eea Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Fri, 12 Jan 2024 11:25:17 -0800 Subject: [PATCH 15/16] Feature/add-kafka-pubsub-schema-registry (#3946) * Added doc for kafka pubsub Avro schema registry support Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Patrick Assuied * Update setup-apache-kafka.md Signed-off-by: Patrick Assuied * Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md Co-authored-by: Mark Fussell Signed-off-by: Patrick Assuied --------- Signed-off-by: Patrick Assuied Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Co-authored-by: Mark Fussell --- .../supported-bindings/kafka.md | 15 +++ .../supported-pubsub/setup-apache-kafka.md | 112 ++++++++++++++++++ 2 files changed, 127 insertions(+) diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md b/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md index 43f030799ce..510e73c2da3 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md @@ -49,6 +49,16 @@ spec: value: "2.0.0" - name: direction value: "input, output" + - name: schemaRegistryURL # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry URL. + value: http://localhost:8081 + - name: schemaRegistryAPIKey # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry API Key. + value: XYAXXAZ + - name: schemaRegistryAPISecret # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. + value: "ABCDEFGMEADFF" + - name: schemaCachingEnabled # Optional. When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. + value: true + - name: schemaLatestVersionCacheTTL # Optional. When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. + value: 5m ``` ## Spec metadata fields @@ -75,6 +85,11 @@ spec: | `version` | N | Input/Output | Kafka cluster version. Defaults to 2.0.0. Please note that this needs to be mandatorily set to `1.0.0` for EventHubs with Kafka. | `"1.0.0"` | | `direction` | N | Input/Output | The direction of the binding. | `"input"`, `"output"`, `"input, output"` | | `oidcExtensions` | N | Input/Output | String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token | `{"cluster":"kafka","poolid":"kafkapool"}` | +| `schemaRegistryURL` | N | Required when using Schema Registry Avro serialization/deserialization. The Schema Registry URL. | `http://localhost:8081` | +| `schemaRegistryAPIKey` | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Key. | `XYAXXAZ` | +| `schemaRegistryAPISecret` | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. | `ABCDEFGMEADFF` | +| `schemaCachingEnabled` | N | When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. Default is `true` | `true` | +| `schemaLatestVersionCacheTTL` | N | When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. Default is 5 min | `5m` | #### Note The metadata `version` must be set to `1.0.0` when using Azure EventHubs with Kafka. diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 05ff835208e..7c39f01afe5 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -49,6 +49,17 @@ spec: value: 2.0.0 - name: disableTls # Optional. Disable TLS. This is not safe for production!! You should read the `Mutual TLS` section for how to use TLS. value: "true" + - name: schemaRegistryURL # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry URL. + value: http://localhost:8081 + - name: schemaRegistryAPIKey # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry API Key. + value: XYAXXAZ + - name: schemaRegistryAPISecret # Optional. When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. + value: "ABCDEFGMEADFF" + - name: schemaCachingEnabled # Optional. When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. + value: true + - name: schemaLatestVersionCacheTTL # Optional. When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. + value: 5m + ``` > For details on using `secretKeyRef`, see the guide on [how to reference secrets in components]({{< ref component-secrets.md >}}). @@ -81,6 +92,11 @@ spec: | oidcClientSecret | N | The OAuth2 client secret that has been provisioned in the identity provider: Required when `authType` is set to `oidc` | `"KeFg23!"` | | oidcScopes | N | Comma-delimited list of OAuth2/OIDC scopes to request with the access token. Recommended when `authType` is set to `oidc`. Defaults to `"openid"` | `"openid,kafka-prod"` | | oidcExtensions | N | Input/Output | String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token | `{"cluster":"kafka","poolid":"kafkapool"}` | +| schemaRegistryURL | N | Required when using Schema Registry Avro serialization/deserialization. The Schema Registry URL. | `http://localhost:8081` | +| schemaRegistryAPIKey | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Key. | `XYAXXAZ` | +| schemaRegistryAPISecret | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. | `ABCDEFGMEADFF` | +| schemaCachingEnabled | N | When using Schema Registry Avro serialization/deserialization. Enables caching for schemas. Default is `true` | `true` | +| schemaLatestVersionCacheTTL | N | When using Schema Registry Avro serialization/deserialization. The TTL for schema caching when publishing a message with latest schema available. Default is 5 min | `5m` | The `secretKeyRef` above is referencing a [kubernetes secrets store]({{< ref kubernetes-secret-store.md >}}) to access the tls information. Visit [here]({{< ref setup-secret-store.md >}}) to learn more about how to configure a secret store component. @@ -348,6 +364,102 @@ curl -X POST http://localhost:3500/v1.0/publish/myKafka/myTopic?metadata.correla }' ``` +## Avro Schema Registry serialization/deserialization +You can configure pub/sub to publish or consume data encoded using [Avro binary serialization](https://avro.apache.org/docs/), leveraging an [Apache Schema Registry](https://developer.confluent.io/courses/apache-kafka/schema-registry/) (for example, [Confluent Schema Registry](https://developer.confluent.io/courses/apache-kafka/schema-registry/), [Apicurio](https://www.apicur.io/registry/)). + +### Configuration + +{{% alert title="Important" color="warning" %}} +Currently, only message value serialization/deserialization is supported. Since cloud events are not supported, the `rawPayload=true` metadata must be passed. +{{% /alert %}} + +When configuring the Kafka pub/sub component metadata, you must define: +- The schema registry URL +- The API key/secret, if applicable + +Schema subjects are automatically derived from topic names, using the standard naming convention. For example, for a topic named `my-topic`, the schema subject will be `my-topic-value`. +When interacting with the message payload within the service, it is in JSON format. The payload is transparently serialized/deserialized within the Dapr component. +Date/Datetime fields must be passed as their [Epoch Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) equivalent (rather than typical Iso8601). For example: +- `2024-01-10T04:36:05.986Z` should be passed as `1704861365986` (the number of milliseconds since Jan 1st, 1970) +- `2024-01-10` should be passed as `19732` (the number of days since Jan 1st, 1970) + +### Publishing Avro messages +In order to indicate to the Kafka pub/sub component that the message should be using Avro serialization, the `valueSchemaType` metadata must be set to `Avro`. + +{{< tabs curl "Python SDK">}} + +{{% codetab %}} +```bash +curl -X "POST" http://localhost:3500/v1.0/publish/pubsub/my-topic?metadata.rawPayload=true&metadata.valueSchemaType=Avro -H "Content-Type: application/json" -d '{"order_number": "345", "created_date": 1704861365986}' +``` +{{% /codetab %}} + +{{% codetab %}} +```python +from dapr.clients import DaprClient + +with DaprClient() as d: + req_data = { + 'order_number': '345', + 'created_date': 1704861365986 + } + # Create a typed message with content type and body + resp = d.publish_event( + pubsub_name='pubsub', + topic_name='my-topic', + data=json.dumps(req_data), + publish_metadata={'rawPayload': 'true', 'valueSchemaType': 'Avro'} + ) + # Print the request + print(req_data, flush=True) +``` +{{% /codetab %}} + +{{< /tabs >}} + + +### Subscribing to Avro topics +In order to indicate to the Kafka pub/sub component that the message should be deserialized using Avro, the `valueSchemaType` metadata must be set to `Avro` in the subscription metadata. + +{{< tabs "Python (FastAPI)" >}} + +{{% codetab %}} + +```python +from fastapi import APIRouter, Body, Response, status +import json +import sys + +app = FastAPI() + +router = APIRouter() + + +@router.get('/dapr/subscribe') +def subscribe(): + subscriptions = [{'pubsubname': 'pubsub', + 'topic': 'my-topic', + 'route': 'my_topic_subscriber', + 'metadata': { + 'rawPayload': 'true', + 'valueSchemaType': 'Avro', + } }] + return subscriptions + +@router.post('/my_topic_subscriber') +def my_topic_subscriber(event_data=Body()): + print(event_data, flush=True) + return Response(status_code=status.HTTP_200_OK) +``` + +app.include_router(router) + +{{% /codetab %}} + +{{< /tabs >}} + + + ## Create a Kafka instance {{< tabs "Self-Hosted" "Kubernetes">}} From b0dafe0897e4ae9332475f781de9f25d54f840be Mon Sep 17 00:00:00 2001 From: Patrick Assuied Date: Fri, 12 Jan 2024 13:03:21 -0800 Subject: [PATCH 16/16] fix formatting (#3950) Signed-off-by: Patrick Assuied --- .../supported-pubsub/setup-apache-kafka.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 7c39f01afe5..ad1fa1e3b1b 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -450,10 +450,11 @@ def subscribe(): def my_topic_subscriber(event_data=Body()): print(event_data, flush=True) return Response(status_code=status.HTTP_200_OK) -``` app.include_router(router) +``` + {{% /codetab %}} {{< /tabs >}}