docs(refactor): updates the cruise control rebalance concepts

[PaulRMellor]

Documentation

Refactor and refresh of Cruise Control concepts

• Adds new optimization proposal process flow description and diagram, including description of partition reassignment commands
• Less verbose and more direct intro and concepts
• Removes three overview files to create a single "components and features" file for goals and proposals concepts
• Consolidates related conceptual information (goals, proposals) into single sections
• Retitled sections to provide more direction to readers from ToC

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[fvaleri]

Hi @PaulRMellor, thanks. I must say that our CC doc is quite good :)

I left some comments for your consideration.

[PaulRMellor]

Thanks for the reviews @fvaleri and @kyguy
I've addressed all the comments except the suggestion to change "user-defined goals" to something else.
I agree it's not clear, but I'm not sure changing to "KafkaRebalance goals" is the way:
#10810 (comment)

[fvaleri]

LGTM. Thanks.

[ppatierno]

maybe it's just my luck of knowledge here, can we really change the priority order? @kyguy ?

[kyguy]

To my understanding, this is true for the Kafka resource default.goal and KafkaRebalance goals lists but not for Kafka goals list

[ppatierno]

Ok. So I think we should clarify here (and everywhere goals priority is mentioned) that the priority is based on the order. Not sure if "priority order" is used for this purpose but it's not clear to me.

[fvaleri]

We have to admit that CC naming doesn't help much here.

IMO, this is the default inter-broker goals priority list:

https://github.com/linkedin/cruise-control/blob/main/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/config/constants/AnalyzerConfig.java#L312-L327

To change it, the user can configure Kafka default.goals or KafkaRebalance goals with a different order. Kafka goals priority is irrelevant, but only goals listed there can be used in the previous configurations (DEFAULT_DEFAULT_GOALS is a subset of DEFAULT_GOALS).

• Do you agree?
• Are we exposing Kafka goals because we intend to eventually support custom goals?

[PaulRMellor]

Thanks for the latest comments @kyguy and @ppatierno
I've updated the diagram and addressed all comments apart from this #10810 (comment)
Looking for guidance

[PaulRMellor]

Thanks for the latest @kyguy and @ppatierno
I've changed "main" to "supported" goals. (Much better)
Diagram now shows "Rebalance" as flow towards CC

[ppatierno]

LGTM.

[PaulRMellor]

Thanks for latest comments @kyguy
I've addressed them all as suggested.

Regarding the summary of Cruise Control components. Previously, we mentioned some but not all them without any context. I thought it would be useful at least to mention them all in summary and show how they operate within a Strimzi context, how requests are made and proposals generated etc. I think the diagram is useful (and good for SEO and user experience).

[kyguy]

Suggested change

	* *Proposal-specific goals* are a subset of default goals configured for specific proposals.
	* *Proposal-specific goals* are a subset of supported goals configured for specific proposals.

[kyguy]

I would move this sentence/information to the Proposal-specific goals section, saying that if proposal-specific goals are not set in the KafkaRebalance resource then default goals are used

[kyguy]

Suggested change

	A partition reassignment command consists of either of the following types of operations:
	A partition reassignment consists of either of the following types of operations:

[kyguy]

Suggested change

	Cruise Control issues partition reassignment commands to the Kafka cluster in batches.
	Cruise Control issues partition reassignments to the Kafka cluster in batches.

[kyguy]

Suggested change

	The performance of the cluster during the rebalance is affected by the number of each type of movement contained in each batch.
	The performance of the cluster during the rebalance is affected by the number and magnitude of each type of movement contained in each batch.

[kyguy]

Suggested change

	The `After` score is based on the generated optimization proposal.
	The `After` score is based on the predicted workload model after applying the generated optimization proposal.

[kyguy]

Suggested change

	When a Kafka rebalance proposal reaches the `ProposalReady` state, Cruise Control generates a `ConfigMap` (named after the `KafkaRebalance` custom resource) containing a JSON string of broker metrics.
	When a Kafka rebalance proposal reaches the `ProposalReady` state, Strimzi creates a `ConfigMap` (named after the `KafkaRebalance` custom resource) containing a JSON string of broker metrics generated from Cruise Control.

[kyguy]

Movements or reassignments might be better here

Suggested change

	By default, Cruise Control uses the `BaseReplicaMovementStrategy`, which applies the commands in the order they were generated.
	By default, Cruise Control uses the `BaseReplicaMovementStrategy`, which applies the reassignments in the order they were generated.

[kyguy]

Suggested change

	However, if large partition reassignments are handled early, this strategy may delay other reassignments.
	However, this strategy could lead to the delay of other partition reassignments if some large partition reassignments are generated then ordered first.

[kyguy]

Suggested change

	* Set individual rebalances in `KafkaRebalance.spec` in the `KafkaRebalance` resource.
	* Set proposal-specific configurations in `KafkaRebalance.spec` in the `KafkaRebalance` resource.