-
Notifications
You must be signed in to change notification settings - Fork 56
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
SearchBackPressure Service Node/Cluster RCA (#437)
* Remove log files and add DCO (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Remove extra files (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Remove styling difference (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Remove unnecessary file changes (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add RCA_Decider (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Extract Heap Usage from SQlitedb (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Extract required searchbp metrics for deciders (signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add SearchBackPressureRCA Metric (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Use SearchBackPressureRCAMetrics to aggregate metrics (signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add the conf file extracted part for SearchBackPressureRcaConfig.java (signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add MinMaxSlidingWindow in OldGenRca (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Rename SearchBackPressureClusterRCA and add it to AnalysisGraph (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add basic UTs for SearchBackPressureRCA cluster/node level (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add unhealthy/healthy stats UTs for SearchBackPressureRCA cluster/node level (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add healthy resource unit UT (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add UT s both shard/task level (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add a new SearchBp Resource Unit (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add UTs to test shard/task level resource include-ness (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Remove styling changes for Version.java (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Add metadata to resourceSummary (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Update to more general framework (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Refactor the MinMaxSlidingWindow and bug fix (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Refactor Heap Stats Metrics Getter(Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Refactor HeapUsed and HeapMax Getters (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Refactor operate() (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Refactor operate() and remove dead comments (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Merged Main (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Merged Main (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * remove trailing space in build.gradle (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * nit javadoc update (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * nit javadoc updates (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * nit javadoc updates (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Remove dead comments (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * update javadoc (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * LOG Level Change (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> * Change from static class to enum (Signed-off-by: Jeffrey Liu [email protected]) Signed-off-by: CoderJeffrey <[email protected]> --------- Signed-off-by: CoderJeffrey <[email protected]> (cherry picked from commit 2b970e3)
- Loading branch information
1 parent
d29ac81
commit dc86c3a
Showing
12 changed files
with
1,374 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
148 changes: 148 additions & 0 deletions
148
...main/java/org/opensearch/performanceanalyzer/rca/configs/SearchBackPressureRcaConfig.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package org.opensearch.performanceanalyzer.rca.configs; | ||
|
|
||
|
|
||
| import org.opensearch.performanceanalyzer.rca.framework.core.RcaConf; | ||
|
|
||
| public class SearchBackPressureRcaConfig { | ||
| public static final String CONFIG_NAME = "search-back-pressure-rca-policy"; | ||
|
|
||
| /* Metadata fields for thresholds */ | ||
| public static final String INCREASE_THRESHOLD_BY_JVM_STR = "increase_jvm"; | ||
| public static final String DECREASE_THRESHOLD_BY_JVM_STR = "decrease_jvm"; | ||
|
|
||
| public static final int SLIDING_WINDOW_SIZE_IN_MINS = 1; | ||
|
|
||
| // Interval period in seconds | ||
| public static final long DEFAULT_EVALUATION_INTERVAL_IN_S = 60; | ||
|
|
||
| /* interval period to call operate() */ | ||
| public static final long EVAL_INTERVAL_IN_S = 5; | ||
|
|
||
| /* Increase Threshold */ | ||
| // node max heap usage in last 60 secs is less than 70% | ||
| public static final int DEFAULT_MAX_HEAP_INCREASE_THRESHOLD = 70; | ||
| private Integer maxHeapIncreasePercentageThreshold; | ||
|
|
||
| // cancellationCount due to heap is more than 50% of all task cancellations in shard level | ||
| public static final int DEFAULT_SHARD_MAX_HEAP_CANCELLATION_THRESHOLD = 50; | ||
| private Integer maxShardHeapCancellationPercentageThreshold; | ||
|
|
||
| // cancellationCount due to heap is more than 50% of all task cancellations in task level | ||
| public static final int DEFAULT_TASK_MAX_HEAP_CANCELLATION_THRESHOLD = 50; | ||
| private Integer maxTaskHeapCancellationPercentageThreshold; | ||
|
|
||
| /* Decrease Threshold */ | ||
| // node min heap usage in last 60 secs is more than 80% | ||
| public static final int DEFAULT_MIN_HEAP_DECREASE_THRESHOLD = 80; | ||
| private Integer minHeapDecreasePercentageThreshold; | ||
|
|
||
| // cancellationCount due to heap is less than 30% of all task cancellations in shard level | ||
| public static final int DEFAULT_SHARD_MIN_HEAP_CANCELLATION_THRESHOLD = 30; | ||
| private Integer minShardHeapCancellationPercentageThreshold; | ||
|
|
||
| // cancellationCount due to heap is less than 30% of all task cancellations in task level | ||
| public static final int DEFAULT_TASK_MIN_HEAP_CANCELLATION_THRESHOLD = 30; | ||
| private Integer minTaskHeapCancellationPercentageThreshold; | ||
|
|
||
| public SearchBackPressureRcaConfig(final RcaConf conf) { | ||
| // (s) -> s > 0 is the validator, if validated, fields from conf file will be returned, | ||
| // else, default value gets returned | ||
| maxHeapIncreasePercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MAX_HEAP_USAGE_INCREASE_FIELD.toString(), | ||
| DEFAULT_MAX_HEAP_INCREASE_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| maxShardHeapCancellationPercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MAX_SHARD_HEAP_CANCELLATION_PERCENTAGE_FIELD | ||
| .toString(), | ||
| DEFAULT_SHARD_MAX_HEAP_CANCELLATION_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| maxTaskHeapCancellationPercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MAX_TASK_HEAP_CANCELLATION_PERCENTAGE_FIELD | ||
| .toString(), | ||
| DEFAULT_TASK_MAX_HEAP_CANCELLATION_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| minHeapDecreasePercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MAX_HEAP_USAGE_DECREASE_FIELD.toString(), | ||
| DEFAULT_MIN_HEAP_DECREASE_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| minShardHeapCancellationPercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MIN_SHARD_HEAP_CANCELLATION_PERCENTAGE_FIELD | ||
| .toString(), | ||
| DEFAULT_SHARD_MIN_HEAP_CANCELLATION_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| minTaskHeapCancellationPercentageThreshold = | ||
| conf.readRcaConfig( | ||
| CONFIG_NAME, | ||
| SearchBackPressureRcaConfigKeys.MIN_TASK_HEAP_CANCELLATION_PERCENTAGE_FIELD | ||
| .toString(), | ||
| DEFAULT_TASK_MIN_HEAP_CANCELLATION_THRESHOLD, | ||
| (s) -> s >= 0 && s <= 100, | ||
| Integer.class); | ||
| } | ||
|
|
||
| // Getters for private field | ||
| public Integer getMaxHeapIncreasePercentageThreshold() { | ||
| return maxHeapIncreasePercentageThreshold; | ||
| } | ||
|
|
||
| public Integer getMaxShardHeapCancellationPercentageThreshold() { | ||
| return maxShardHeapCancellationPercentageThreshold; | ||
| } | ||
|
|
||
| public Integer getMaxTaskHeapCancellationPercentageThreshold() { | ||
| return maxTaskHeapCancellationPercentageThreshold; | ||
| } | ||
|
|
||
| public Integer getMinHeapDecreasePercentageThreshold() { | ||
| return minHeapDecreasePercentageThreshold; | ||
| } | ||
|
|
||
| public Integer getMinShardHeapCancellationPercentageThreshold() { | ||
| return minShardHeapCancellationPercentageThreshold; | ||
| } | ||
|
|
||
| public Integer getMinTaskHeapCancellationPercentageThreshold() { | ||
| return minTaskHeapCancellationPercentageThreshold; | ||
| } | ||
|
|
||
| // name for the configuration field | ||
| public enum SearchBackPressureRcaConfigKeys { | ||
| MAX_HEAP_USAGE_INCREASE_FIELD("max-heap-usage-increase"), | ||
| MAX_SHARD_HEAP_CANCELLATION_PERCENTAGE_FIELD("max-shard-heap-cancellation-percentage"), | ||
| MAX_TASK_HEAP_CANCELLATION_PERCENTAGE_FIELD("max-task-heap-cancellation-percentage"), | ||
| MAX_HEAP_USAGE_DECREASE_FIELD("max-heap-usage-decrease"), | ||
| MIN_SHARD_HEAP_CANCELLATION_PERCENTAGE_FIELD("min-shard-heap-cancellation-percentage"), | ||
| MIN_TASK_HEAP_CANCELLATION_PERCENTAGE_FIELD("min-task-heap-cancellation-percentage"); | ||
|
|
||
| private final String value; | ||
|
|
||
| SearchBackPressureRcaConfigKeys(final String value) { | ||
| this.value = value; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return this.value; | ||
| } | ||
| } | ||
| } |
18 changes: 18 additions & 0 deletions
18
...ain/java/org/opensearch/performanceanalyzer/rca/framework/api/metrics/Searchbp_Stats.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package org.opensearch.performanceanalyzer.rca.framework.api.metrics; | ||
|
|
||
|
|
||
| import org.opensearch.performanceanalyzer.commons.metrics.AllMetrics; | ||
| import org.opensearch.performanceanalyzer.rca.framework.api.Metric; | ||
|
|
||
| public class Searchbp_Stats extends Metric { | ||
| public Searchbp_Stats(long evaluationIntervalSeconds) { | ||
| super( | ||
| AllMetrics.SearchBackPressureStatsValue.SEARCHBP_TABLE_NAME.toString(), | ||
| evaluationIntervalSeconds); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
...ch/performanceanalyzer/rca/store/rca/searchbackpressure/SearchBackPressureClusterRCA.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| package org.opensearch.performanceanalyzer.rca.store.rca.searchbackpressure; | ||
|
|
||
|
|
||
| import org.apache.logging.log4j.LogManager; | ||
| import org.apache.logging.log4j.Logger; | ||
| import org.opensearch.performanceanalyzer.rca.framework.api.Rca; | ||
| import org.opensearch.performanceanalyzer.rca.framework.api.flow_units.ResourceFlowUnit; | ||
| import org.opensearch.performanceanalyzer.rca.framework.api.summaries.HotNodeSummary; | ||
| import org.opensearch.performanceanalyzer.rca.store.rca.cluster.BaseClusterRca; | ||
|
|
||
| public class SearchBackPressureClusterRCA extends BaseClusterRca { | ||
|
|
||
| public static final String RCA_TABLE_NAME = SearchBackPressureClusterRCA.class.getSimpleName(); | ||
| private static final Logger LOG = LogManager.getLogger(SearchBackPressureClusterRCA.class); | ||
|
|
||
| public <R extends Rca<ResourceFlowUnit<HotNodeSummary>>> SearchBackPressureClusterRCA( | ||
| final int rcaPeriod, final R SearchBackPressureRCA) { | ||
| super(rcaPeriod, SearchBackPressureRCA); | ||
| } | ||
| } |
Oops, something went wrong.