Skip to content

Commit

Permalink
Added note about kafka not being master, and FAQ entry about schema b…
Browse files Browse the repository at this point in the history
…ackup (#537)

* Added note about kafka not being master, and FAQ entry about schema backup

* Don't mention on-prem kafka at all
  • Loading branch information
mortenlj authored Oct 16, 2023
1 parent 7326731 commit efe0f25
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 6 deletions.
17 changes: 14 additions & 3 deletions docs/persistence/kafka/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,24 @@
# Kafka

NAV uses Aiven hosted Kafka, the on-premise version of Kafka has been deprecated since 2021. Start using Kafka by creating a `Topic` resource in one of our Kubernetes clusters.
NAV uses Aiven hosted Kafka.
Start using Kafka by creating a `Topic` resource in one of our Kubernetes clusters.

Upon defining a Topic, [Kafkarator](https://github.com/nais/kafkarator) will create the Topic in one of the Kafka _pools_. A pool is a highly available, replicated Kafka cluster running at Aiven. After the topic is created, Kafkarator will add relevant users to the topic's access control list \(ACL\).
Upon defining a Topic, [Kafkarator](https://github.com/nais/kafkarator) will create the Topic in one of the Kafka _pools_.
A pool is a highly available, replicated Kafka cluster running at Aiven.
After the topic is created, Kafkarator will add relevant users to the topic's access control list \(ACL\).

When an application that uses Kafka is deployed, [Aivenator](https://github.com/nais/aivenator) will orchestrate generation of users and credentials. These credentials are made available to applications through a _Secret_ in the relevant team namespace. This secret is automatically mounted by Naiserator into application pods as environment variables and files.
When an application that uses Kafka is deployed, [Aivenator](https://github.com/nais/aivenator) will orchestrate generation of users and credentials.
These credentials are made available to applications through a _Secret_ in the relevant team namespace.
This secret is automatically mounted by Naiserator into application pods as environment variables and files.

To get started, see [accessing topics from an application](manage_topics.md#accessing-topics-from-an-application).

!!! note "Backups and recovery"
Kafka as a system is highly durable, and is designed to be able to keep your data safe in the event of a failure.
This requires a properly configured replication factor for your topic, and that your clients use the appropriate strategy when sending messages and committing offsets.
Even so, our recommendation is that Kafka should not be the master of your data, and you should have the ability to restore your data from some other system.


## Status and roadmap

* Release status: Generally Available
Expand Down
8 changes: 5 additions & 3 deletions docs/persistence/kafka/faq.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@

# FAQ/Troubleshooting

!!! warning
This feature applies only to _Aiven hosted Kafka_. On-premises Kafka is deprecated, and creating new topics on-premises was disabled summer 2021. For on-premises Kafka, see [on-premises Kafka documentation](https://confluence.adeo.no/display/AURA/Kafka).

## Why do I have to specify a pool name if there is only `nav-dev` and `nav-prod`?

Custom pools might be added in the future, so this is done to avoid changing that part of the API.
Expand All @@ -19,3 +16,8 @@ Make sure you added the application to `.spec.acl` in your `topic.yaml`.
## I get the error _MountVolume.SetUp failed for volume "kafka-credentials" : secret ... not found_

Check the status of the `AivenApplication` resource created by Naiserator to look for errors.

## Are Schemas backed up?

Aiven makes backups of configuration and schemas every 3 hours, but no topic data is backed up by default.
See the [Aiven documentation](https://docs.aiven.io/docs/products/kafka/concepts/configuration-backup) for more details.

0 comments on commit efe0f25

Please sign in to comment.