[Backport 2.2] Handling OpenSearchRejectExecuteException Exception (#1004)

[sricharanvuppu]

• Handling OpenSearchRejectExecuteException Exception
• introduced writersPerShard setting.

(cherry picked from commit 448e7a7)

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #1018 (fdb8117) into 2.2 (ce598ff) will increase coverage by 1.12%.
Report is 1 commits behind head on 2.2.
The diff coverage is 69.44%.

❗ Current head fdb8117 differs from pull request most recent head 411c0e0. Consider uploading reports for the commit 411c0e0 to get more accurate results

@@             Coverage Diff              @@
##                2.2    #1018      +/-   ##
============================================
+ Coverage     74.79%   75.91%   +1.12%     
- Complexity     1026     1041      +15     
============================================
  Files           141      141              
  Lines          4685     4721      +36     
  Branches        521      527       +6     
============================================
+ Hits           3504     3584      +80     
+ Misses          859      798      -61     
- Partials        322      339      +17     

Files	Coverage Δ
...in/org/opensearch/replication/ReplicationPlugin.kt	91.78% <100.00%> (+0.17%)	⬆️
.../org/opensearch/replication/ReplicationSettings.kt	100.00% <100.00%> (ø)
...on/resume/TransportResumeIndexReplicationAction.kt	86.95% <100.00%> (-0.37%)	⬇️
...ication/seqno/RemoteClusterRetentionLeaseHelper.kt	72.44% <100.00%> (ø)
...rch/replication/task/shard/ShardReplicationTask.kt	75.18% <100.00%> (ø)
...org/opensearch/replication/ReplicationException.kt	11.11% <20.00%> (-3.18%)	⬇️
...tlin/org/opensearch/replication/util/Extensions.kt	68.13% <58.33%> (-2.11%)	⬇️
...search/replication/task/shard/TranslogSequencer.kt	78.46% <68.29%> (-6.91%)	⬇️

... and 9 files with indirect coverage changes