Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a search AD results tool #52

Merged
merged 7 commits into from
Dec 27, 2023

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Dec 22, 2023

Description

This PR adds a search AD results tool. It exposes several different parameters to search and sort the desired results. Note these are subject to change as more integration testing is done with agents configured with the tool.

Testing:

  • manually tested the generated queries against a test cluster to ensure they work as expected against different scenarios and different types of anomaly results
  • adds limited UT consistent with existing coverage of tools for now. Again this may be improved when the inputs/outputs/parameters become more standardized and we can have more robust coverage and detailed test cases. We can see how the performance is when we benchmark the tools in integration tests with an equipped agent configured with an LLM

Issues Resolved

opensearch-project/ml-commons#1548

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tyler Ohlsen <[email protected]>
Copy link

codecov bot commented Dec 22, 2023

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (cd1cfc4) 44.41% compared to head (bdfdc28) 81.57%.
Report is 6 commits behind head on main.

Files Patch % Lines
...ensearch/agent/tools/SearchAnomalyResultsTool.java 80.72% 7 Missing and 9 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##               main      #52       +/-   ##
=============================================
+ Coverage     44.41%   81.57%   +37.16%     
- Complexity       30       78       +48     
=============================================
  Files             6        7        +1     
  Lines           358      456       +98     
  Branches         42       63       +21     
=============================================
+ Hits            159      372      +213     
+ Misses          178       45      -133     
- Partials         21       39       +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

final String detectorId = parameters.getOrDefault("detectorId", null);
final Boolean realTime = parameters.containsKey("realTime") ? Boolean.parseBoolean(parameters.get("realTime")) : null;
final Double anomalyGradeThreshold = parameters.containsKey("anomalyGradeThreshold")
? Double.parseDouble(parameters.get("anomalyGradeThreshold"))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add sufficient test for this method?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we add more info in the test like anomalyGradeThreshold, dataStartTime, dataEndTime etc?

Copy link
Member Author

@ohltyler ohltyler Dec 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added for other valid cases with the other params. But there won't be any other invalid cases since the other params are tested with isNumeric, which if it fails, will default to null values and not throw exceptions.

I don't use isNumeric specifically because of #52 (comment)

final String detectorId = parameters.getOrDefault("detectorId", null);
final Boolean realTime = parameters.containsKey("realTime") ? Boolean.parseBoolean(parameters.get("realTime")) : null;
final Double anomalyGradeThreshold = parameters.containsKey("anomalyGradeThreshold")
? Double.parseDouble(parameters.get("anomalyGradeThreshold"))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check if the parameters.get("anomalyGradeThreshold") is a number before parsing to Double?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isNumeric requires each character to be a digit, but users may want to pass -1 for example. Prefer to leave as-is, as -1 may be a common value (within LLM prompt potentially) to indicate they want to fetch all anomaly results, regardless if they are actual anomalies with grades > 0.

Signed-off-by: Tyler Ohlsen <[email protected]>
@zane-neo zane-neo merged commit 18445e6 into opensearch-project:main Dec 27, 2023
13 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Dec 27, 2023
* Set up results tool boilerplate

Signed-off-by: Tyler Ohlsen <[email protected]>

* Finish AD result tool implementation

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add AD result tool

Signed-off-by: Tyler Ohlsen <[email protected]>

* remove TODO

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add UT for invalid anomaly grade

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add more UT for params

Signed-off-by: Tyler Ohlsen <[email protected]>

* Fix typo

Signed-off-by: Tyler Ohlsen <[email protected]>

---------

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 18445e6)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@ohltyler ohltyler deleted the ad-results-tool branch December 27, 2023 00:41
ohltyler pushed a commit that referenced this pull request Dec 27, 2023

(cherry picked from commit 18445e6)

Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
yuye-aws pushed a commit to yuye-aws/skills that referenced this pull request Apr 26, 2024
…ect#67)

(cherry picked from commit 18445e6)

Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: yuye-aws <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants