Multiple Schema Registry Support

[kjkondratuk]

I currently have a use case where it seems like I could benefit from support for schema registries as a separate entity from kafka clusters. While it's true that in many cases its admirable to strive for a single schema registry to centrally manage schemas, I find myself at a bit of an impasse where I have to support a Buf schema registry (with compatibility for the Confluent Schema Registry API) as well as a self-hosted Confluent Schema Registry.

As far as I'm aware, it is not currently possible to define a kafka cluster which can resolve schemas against multiple registries--you need to define a separate cluster for each registry with the same broker configuration. Duplicating the cluster is necessary to have a version of the cluster which points to each registry. This is necessary in my circumstance because Buf doesn't support hosting Avro schemas (it's primarily a protobuf tool) which we use in some legacy cases, and CSR doesn't support some of the advanced features of Buf.

Currently, what the config looks like is:

kafka:
  clusters:
    - name: cluster-a
      bootstrapServers: cluster-a:<port>
      schemaRegistry: http://registry-a

    - name: cluster-a-proto
      bootstrapServers: cluster-a:<port>
      schemaRegistry: http://registry-b

I'd think something like the following would accomplish this quite nicely:

kafka:
  clusters:
    - name: cluster-a
      bootstrapServers: cluster-a:<port>
      schemaRegistries:
        - proto-registry
        - non-proto-registry

  schemaRegistries:
    proto-registry: http://registry-a
    non-proto-registry: http://registry-b

Where the schema registry connection reference can be re-used, and multiple schema registries specified per cluster. There are some UI controls that would have to be changed, but I'd expect that the schema registry could be selected (from amongst the configured schema registries, and possibly a default configured) when browsing messages in a topic.

It looks like there are a couple similar issues (or with overlap) out there already:

• Add separate section for Schema Registry and Kafka Connect in both config and UI #4120
• Feature: Make it possible to use UI exclusively for SR and KC #1076

I am likely going to go with the simple repetition for the time being, but would be willing to contribute even though my Java is about 3 years out of date, if the change is a welcome one :)

[kjkondratuk]

Closing this. Moved to kafbat/kafka-ui.