Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Extensions] Increasing AD Extension Request timeout to 20 and fixes exception handling for Delete/Profile #916

Merged
merged 4 commits into from
Jun 1, 2023

Conversation

joshpalis
Copy link
Member

@joshpalis joshpalis commented May 31, 2023

Description

Removes all hardcoded timeout values for SDKClient/OpenSearchAsyncClinet requests with AnomalyDetectorSettings.REQUEST_TIMEOUT value, increases timeout from 10 seconds to 20.

This PR is an effort to unblock REST integration tests, as all REST IT tests are currently failing due to the following timeout error :

org.opensearch.ad.rest.AnomalyDetectorRestApiIT > testCreateAnomalyDetector FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:46289], URI [/_extensions/_ad-extension/detectors], status line [HTTP/1.1 408 Request Timeout]
    No response from extension to request.
        at __randomizedtesting.SeedInfo.seed([B4358DC928E0BE79:E58B5BBE98C1E8F1]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:354)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:329)
        at app//org.opensearch.sdk.SDKClient$SDKRestClient.performRequest(SDKClient.java:557)
        at app//org.opensearch.ad.TestHelpers.makeRequest(TestHelpers.java:221)
        at app//org.opensearch.ad.TestHelpers.makeRequest(TestHelpers.java:194)
        at app//org.opensearch.ad.rest.AnomalyDetectorRestApiIT.testCreateAnomalyDetector(AnomalyDetectorRestApiIT.java:168)

Additionally, fixes exception handling for DeleteDetector/ProfileDetector to check exception message rather than type to determine if the exception is due to a missing index

Issues Resolved

Part of opensearch-project/opensearch-sdk-java#724, fixes timeout issues seen in integration testing PR (#913)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ST_TIMEOUT setting, increased request timeout setting to 20, fixed affected tests

Signed-off-by: Joshua Palis <[email protected]>
@@ -57,7 +57,7 @@ private LegacyOpenDistroAnomalyDetectorSettings() {}
public static final Setting<TimeValue> REQUEST_TIMEOUT = Setting
.positiveTimeSetting(
"opendistro.anomaly_detection.request_timeout",
TimeValue.timeValueSeconds(10),
TimeValue.timeValueSeconds(20),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a TODO here to revert this back once we have support for cluster state? That's one of major reason for the timeout

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

@codecov
Copy link

codecov bot commented May 31, 2023

Codecov Report

Merging #916 (aeedfbe) into feature/extensions (63553d7) will increase coverage by 1.47%.
The diff coverage is 9.67%.

Impacted file tree graph

@@                   Coverage Diff                    @@
##             feature/extensions     #916      +/-   ##
========================================================
+ Coverage                 41.88%   43.36%   +1.47%     
- Complexity                 2335     2435     +100     
========================================================
  Files                       301      301              
  Lines                     18026    18065      +39     
  Branches                   1871     1872       +1     
========================================================
+ Hits                       7550     7833     +283     
+ Misses                     9918     9651     -267     
- Partials                    558      581      +23     
Flag Coverage Δ
plugin 43.36% <9.67%> (+1.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...va/org/opensearch/ad/AnomalyDetectorExtension.java 0.00% <0.00%> (ø)
...va/org/opensearch/ad/AnomalyDetectorJobRunner.java 0.00% <0.00%> (ø)
...ensearch/ad/rest/RestAnomalyDetectorJobAction.java 0.00% <0.00%> (ø)
...ain/java/org/opensearch/ad/task/ADTaskManager.java 0.07% <0.00%> (-0.01%) ⬇️
...ransport/DeleteAnomalyDetectorTransportAction.java 0.00% <0.00%> (ø)
...ain/java/org/opensearch/ad/util/ExceptionUtil.java 48.83% <0.00%> (ø)
...c/main/java/org/opensearch/ad/util/IndexUtils.java 2.12% <0.00%> (-0.26%) ⬇️
...opensearch/ad/indices/AnomalyDetectionIndices.java 12.25% <25.00%> (+0.12%) ⬆️
.../main/java/org/opensearch/ad/ml/CheckpointDao.java 61.82% <57.14%> (+0.73%) ⬆️
...tings/LegacyOpenDistroAnomalyDetectorSettings.java 100.00% <100.00%> (ø)

... and 12 files with indirect coverage changes

@joshpalis joshpalis changed the title [Extensions] Increasing AD Extension Request timeout to 20 [Extensions] Increasing AD Extension Request timeout to 20 and fixes exception handling for Delete/Profile May 31, 2023
Copy link
Member

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but with a strong preference to change a few things... please consider that but go ahead and merge if it's urgent.

@@ -163,6 +167,7 @@ public class CheckpointDao {
public CheckpointDao(
SDKRestClient client,
OpenSearchAsyncClient sdkJavaAsyncClient,
Settings settings,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm really iffy on adding a new parameter to an already-long list, but if we do so it should probably be at the end of the list, not in the middle. Also we could add an overload that delegates from the previous version to the new version with Settings.EMPTY to avoid having to change every single existing call.

sdkClusterService,
indexNameExpressionResolver,
javaAsyncClient(),
environmentSettings
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment here; we do have the settings at the end but can we just add an overloaded constructor that assumes empty settings if not passed so we don't have to rewrite existing constructor calls?

…ntDao to assume an empty settings object if not passed directly

Signed-off-by: Joshua Palis <[email protected]>
@joshpalis joshpalis merged commit c20e72d into opensearch-project:feature/extensions Jun 1, 2023
@joshpalis joshpalis deleted the timeout branch June 1, 2023 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants