[BUG] SearchBackPressureRCA causing Gauntlet test failure

[khushbr]

What is the bug?
SearchBackPressureRCA causing Gauntlet test failure. refer: https://github.com/opensearch-project/performance-analyzer-rca/actions/runs/6385777195/job/17331223490

 The Runner found the following errors in log: [
  00:20:20.398 [ELECTED_CLUSTER_MANAGER-task-1-] ERROR SearchBackPressureRCA:getHeapStats()::line:403 - Failed to parse metric in FlowUnit from org.opensearch.performanceanalyzer.rca.framework.api.metrics.Heap_Used
  00:20:20.480 [DATA_0-task-1-] ERROR SearchBackPressureRCA:getHeapStats()::line:403 - Failed to parse metric in FlowUnit from org.opensearch.performanceanalyzer.rca.framework.api.metrics.Heap_Used
  00:20:25.402 [ELECTED_CLUSTER_MANAGER-task-1-] ERROR SearchBackPressureRCA:getHeapStats()::line:403 - Failed to parse metric in FlowUnit from org.opensearch.performanceanalyzer.rca.framework.api.metrics.Heap_Used
  00:20:25.488 [DATA_0-task-0-] ERROR SearchBackPressureRCA:getHeapStats()::line:403 - Failed to parse metric in FlowUnit from org.opensearch.performanceanalyzer.rca.framework.api.metrics.Heap_Used

The search backpressure metrics on PA end are:

1696544030000:{"searchbp_shard_stats_cancellationCount":0,"searchbp_shard_stats_limitReachedCount":0,"searchbp_shard_stats_resource_heap_usage_cancellationCount":0,"searchbp_shard_stats_resource_heap_usage_currentMax":0,"searchbp_shard_stats_resource_heap_usage_rollingAvg":0,"searchbp_shard_stats_resource_cpu_usage_cancellationCount":0,"searchbp_shard_stats_resource_cpu_usage_currentMax":0,"searchbp_shard_stats_resource_cpu_usage_currentAvg":0,"searchbp_shard_stats_resource_elaspedtime_usage_cancellationCount":0,"searchbp_shard_stats_resource_elaspedtime_usage_currentMax":0,"searchbp_shard_stats_resource_elaspedtime_usage_currentAvg":0,"searchbp_task_stats_cancellationCount":0,"searchbp_task_stats_limitReachedCount":0,"searchbp_task_stats_resource_heap_usage_cancellationCount":0,"searchbp_task_stats_resource_heap_usage_currentMax":0,"searchbp_task_stats_resource_heap_usage_rollingAvg":0,"searchbp_task_stats_resource_cpu_usage_cancellationCount":0,"searchbp_task_stats_resource_cpu_usage_currentMax":0,"searchbp_task_stats_resource_cpu_usage_currentAvg":0,"searchbp_task_stats_resource_elaspedtime_usage_cancellationCount":0,"searchbp_task_stats_resource_elaspedtime_usage_currentMax":0,"searchbp_task_stats_resource_elaspedtime_usage_currentAvg":0,"searchbp_mode":"MONITOR_ONLY","searchbp_nodeid":"lGFg--rtQ1CRMVK_x8SWmA"}$

Do you have any additional context?
Add any other context about the problem.

[khushbr]

ThegetHeapStats method is failing to parse the metrics:

performance-analyzer-rca/src/main/java/org/opensearch/performanceanalyzer/rca/store/rca/searchbackpressure/SearchBackPressureRCA.java

Line 399 in 7391dae

AllMetrics.HeapDimension.MEM_TYPE.getField(),

[khushbr]

The SBP changes have been reverted and the gauntlet tests are now passing. This is blocked by availability of opensearch-project/OpenSearch#10028