GH-3291 - Possible inconsistency in DLT topic naming convention

[Watlas]

Standardize DLT Topic Naming Convention

Issue Reference

Description:

This pull request addresses the inconsistency in the naming convention of DLT topics when using different retry mechanisms. By standardizing the suffix to "-dlt", we aim to ensure compatibility and avoid conflicts during migration between retry solutions.

Expected Behavior:

DLT topics should be created consistently, using a standardized suffix, preferably with a dash, for example, all ending with "-dlt".

Current Behavior:

By default, DLT topics are created with ".dlt" at the end. However, when using the retry non-blocking solution, topics are created with "-dlt". This causes inconsistency in the naming of the topics.

Context:

This inconsistency can cause issues when migrating from one solution to another. For example, if a user wants to migrate from a solution that uses blocking retry (which creates topics with ".dlt") to a non-blocking retry solution (which creates topics with "-dlt"), the old ".dlt" topics will become obsolete. The new system will no longer communicate with these topics due to the difference in naming.

Additionally, Kafka recommends not mixing dashes and dots in topic names to avoid conflicts. This mix can lead to issues in identifying and managing topics, further complicating system maintenance and monitoring.

Solution:

Standardize the creation of DLT topics to use a consistent naming convention, preferably using a dash suffix, such as "-dlt". This will ensure compatibility and avoid conflicts when transitioning between different retry solutions.

Implementation Details:

• Updated the naming convention for DLT topics to use the "-dlt" suffix consistently.
• Adjusted relevant configurations and documentation to reflect this change.

Please review the changes and provide feedback. If there are any suggestions or concerns regarding this implementation, I am open to further discussion.

[sobychacko]

While I concur with you that there is an inconsistency in the DLT topic creation and the need to fix it, I wonder if breaking the current .dlt for the regular case (blocking) may introduce any regression for existing users. For example, think about a 3.2.0 user (or prior versions) who relies on the .dlt convention and, in a future release, see the topic created with -dlt. It may generate some unwanted user experience. We can make this change in 3.3.0, but there needs to be a warning added in the docs for any users who rely on .dlt (maybe in the whats-new section) to indicate that they want the old behavior by setting the dlt name property. Currently, they don't need to do this since .dlt is the default for the blocking scenario.

[Watlas]

@sobychacko I agree that this change may cause problems for users of previous versions. However, these problems could be resolved quickly. The spring-kafka framework may automatically create the new topic or may throw an error indicating that the topic does not exist. It also makes sense to add this to the “What’s New” section. I also believe that this should be considered a MINOR update (3.3.0), as it affects the functionality of the system. And thanks for the feedback.

[sobychacko]

Could you add the note as part of the what-new section? In it, indicate that if any users want the .dlt behavior, they need to opt-in for that. Feel free to update the PR. Then we can do further review. Thanks!

[Watlas]

@sobychacko Done.