Adding a state index #110

amitgalitz · 2023-10-20T00:42:51Z

Description

Adding a state index along with new methods to add documents to the new state index and update the state index.

Logs when we run the create API:

023-10-30T13:28:27,856][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.plugins-ai-global-context/iGdhXze_RUK6NXz18zZwBw]
[2023-10-30T13:28:27,995][INFO ][o.o.c.m.MetadataCreateIndexService] [integTest-0] [.plugins-ai-global-context] creating index, cause [api], templates [], shards [1]/[1]
[2023-10-30T13:28:28,004][INFO ][o.o.c.r.a.AllocationService] [integTest-0] updating number_of_replicas to [0] for indices [.plugins-ai-global-context]
[2023-10-30T13:28:28,034][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [3] with uuid [8vAwFmrmTDKscToof_pwCg], diff size [703]
[2023-10-30T13:28:28,136][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.plugins-ai-global-context/iGdhXze_RUK6NXz18zZwBw]
[2023-10-30T13:28:28,223][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=3}
[2023-10-30T13:28:28,473][INFO ][o.o.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ai-global-context][0]]]).
[2023-10-30T13:28:28,477][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [4] with uuid [3_aC0P0OTDyICTMjlY-mIw], diff size [385]
[2023-10-30T13:28:28,632][INFO ][o.o.f.i.FlowFrameworkIndicesHandler] [integTest-0] create index:.plugins-ai-global-context
[2023-10-30T13:28:28,694][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=4}
[2023-10-30T13:28:28,833][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.plugins-workflow-state/a_NUpbQfRwqDFtt0Xq-nfQ]
[2023-10-30T13:28:28,862][INFO ][o.o.c.m.MetadataCreateIndexService] [integTest-0] [.plugins-workflow-state] creating index, cause [api], templates [], shards [1]/[1]
[2023-10-30T13:28:28,863][INFO ][o.o.c.r.a.AllocationService] [integTest-0] updating number_of_replicas to [0] for indices [.plugins-workflow-state]
[2023-10-30T13:28:28,865][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [5] with uuid [e-XHuc0mR-uLi0zixhfnaQ], diff size [683]
[2023-10-30T13:28:28,960][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[.plugins-workflow-state/a_NUpbQfRwqDFtt0Xq-nfQ]
[2023-10-30T13:28:29,047][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=5}
[2023-10-30T13:28:29,351][INFO ][o.o.c.r.a.AllocationService] [integTest-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-workflow-state][0]]]).
[2023-10-30T13:28:29,353][DEBUG][o.o.c.c.PublicationTransportHandler] [integTest-0] received diff cluster state version [6] with uuid [WeQ7h7_2RsGLv3yvlycb2w], diff size [384]
[2023-10-30T13:28:29,429][INFO ][o.o.f.i.FlowFrameworkIndicesHandler] [integTest-0] create index:.plugins-workflow-state
[2023-10-30T13:28:29,450][DEBUG][o.o.c.c.C.CoordinatorPublication] [integTest-0] publication ended successfully: Publication{term=1, version=6}
[2023-10-30T13:28:29,532][INFO ][o.o.f.t.CreateWorkflowTransportAction] [integTest-0] create state workflow doc

Issues Resolved

#74

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

joshpalis

Initial pass, looks good so far. I had a few comments regarding exception handling, but I'll just rebase PR #106 after this is merged in and handle it there.

src/main/java/org/opensearch/flowframework/model/WorkflowState.java

dbwiddis

Initially LGTM with some nits.

I can't get this to build locally, there's a version mismatch in log4j. Would like this rebased so I can compile before approving.

Dependency resolution failed because of conflict on the following module:
   - org.apache.logging.log4j:log4j-api between versions 2.21.0 and 2.20.0

org.apache.logging.log4j:log4j-api:2.21.0
  Variant compile:
    | Attribute Name                 | Provided | Requested    |
    |--------------------------------|----------|--------------|
    | org.gradle.status              | release  |              |
    | org.gradle.category            | library  | library      |
    | org.gradle.libraryelements     | jar      | classes      |
    | org.gradle.usage               | java-api | java-api     |
    | org.gradle.dependency.bundling |          | external     |
    | org.gradle.jvm.environment     |          | standard-jvm |
    | org.gradle.jvm.version         |          | 11           |
   Selection reasons:
      - By conflict resolution: between versions 2.21.0 and 2.20.0

org.apache.logging.log4j:log4j-api:2.21.0
\--- compileClasspath

org.apache.logging.log4j:log4j-api:2.20.0 -> 2.21.0
+--- org.opensearch:opensearch:3.0.0-SNAPSHOT
|    \--- compileClasspath
\--- org.opensearch:opensearch-core:3.0.0-SNAPSHOT
     +--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)
     +--- org.opensearch:opensearch-compress:3.0.0-SNAPSHOT
     |    \--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)
     \--- org.opensearch:opensearch-x-content:3.0.0-SNAPSHOT
          \--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)

src/main/java/org/opensearch/flowframework/util/ParseUtils.java

src/main/java/org/opensearch/flowframework/workflow/CreateIndexStep.java

src/test/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandlerTests.java

amitgalitz · 2023-10-27T17:40:46Z

Initially LGTM with some nits.

I can't get this to build locally, there's a version mismatch in log4j. Would like this rebased so I can compile before approving.

Dependency resolution failed because of conflict on the following module:
   - org.apache.logging.log4j:log4j-api between versions 2.21.0 and 2.20.0

org.apache.logging.log4j:log4j-api:2.21.0
  Variant compile:
    | Attribute Name                 | Provided | Requested    |
    |--------------------------------|----------|--------------|
    | org.gradle.status              | release  |              |
    | org.gradle.category            | library  | library      |
    | org.gradle.libraryelements     | jar      | classes      |
    | org.gradle.usage               | java-api | java-api     |
    | org.gradle.dependency.bundling |          | external     |
    | org.gradle.jvm.environment     |          | standard-jvm |
    | org.gradle.jvm.version         |          | 11           |
   Selection reasons:
      - By conflict resolution: between versions 2.21.0 and 2.20.0

org.apache.logging.log4j:log4j-api:2.21.0
\--- compileClasspath

org.apache.logging.log4j:log4j-api:2.20.0 -> 2.21.0
+--- org.opensearch:opensearch:3.0.0-SNAPSHOT
|    \--- compileClasspath
\--- org.opensearch:opensearch-core:3.0.0-SNAPSHOT
     +--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)
     +--- org.opensearch:opensearch-compress:3.0.0-SNAPSHOT
     |    \--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)
     \--- org.opensearch:opensearch-x-content:3.0.0-SNAPSHOT
          \--- org.opensearch:opensearch:3.0.0-SNAPSHOT (*)

yes I am still fixing some commit conflicts from latest PR that was merged in.

owaiskazi19

Great progress!

src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java

src/main/java/org/opensearch/flowframework/transport/ProvisionWorkflowTransportAction.java

src/main/java/org/opensearch/flowframework/util/ParseUtils.java

src/main/java/org/opensearch/flowframework/workflow/CreateIndexStep.java

src/test/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandlerTests.java

joshpalis

Approved with a small comment, thanks @amitgalitz for implementing this. One ask that I have is if you could post the WorkflowState document created when invoking the CreateWorkflow rest action and when invoking the ProvisionWorkflow rest action.

src/main/java/org/opensearch/flowframework/model/State.java

Signed-off-by: Amit Galitzky <[email protected]>

codecov · 2023-10-30T19:25:12Z

Codecov Report

Merging #110 (36bcd2a) into main (6ee3d53) will decrease coverage by 14.20%.
Report is 1 commits behind head on main.
The diff coverage is 33.63%.

@@              Coverage Diff              @@
##               main     #110       +/-   ##
=============================================
- Coverage     80.24%   66.05%   -14.20%     
- Complexity      250      266       +16     
=============================================
  Files            30       34        +4     
  Lines           992     1299      +307     
  Branches         95      123       +28     
=============================================
+ Hits            796      858       +62     
- Misses          160      399      +239     
- Partials         36       42        +6

Files	Coverage Δ
.../opensearch/flowframework/FlowFrameworkPlugin.java	`100.00% <100.00%> (ø)`
...g/opensearch/flowframework/common/CommonValue.java	`100.00% <100.00%> (ø)`
...arch/flowframework/indices/FlowFrameworkIndex.java	`100.00% <100.00%> (ø)`
...nsearch/flowframework/model/PipelineProcessor.java	`100.00% <ø> (ø)`
...arch/flowframework/model/ProvisioningProgress.java	`100.00% <100.00%> (ø)`
...java/org/opensearch/flowframework/model/State.java	`100.00% <100.00%> (ø)`
...g/opensearch/flowframework/model/WorkflowNode.java	`90.54% <ø> (ø)`
...ch/flowframework/workflow/CreateConnectorStep.java	`73.84% <100.00%> (ø)`
...search/flowframework/workflow/CreateIndexStep.java	`85.29% <100.00%> (+26.57%)`	⬆️
...opensearch/flowframework/workflow/ProcessNode.java	`93.47% <ø> (ø)`
... and 9 more

src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java

owaiskazi19

Almost there. With few open comments. Let's answer/resolve them and get this in

Signed-off-by: Amit Galitzky <[email protected]>

owaiskazi19

Thanks for addressing the comments! Approved. Let's address the remaining comments/tests in a follow up PR.

* adding state index initial Signed-off-by: Amit Galitzky <[email protected]> * addressed comments and added more fields to state index Signed-off-by: Amit Galitzky <[email protected]> * addressed comments and fixed some unit tests Signed-off-by: Amit Galitzky <[email protected]> * moved variables to common value and adressed other comments Signed-off-by: Amit Galitzky <[email protected]> --------- Signed-off-by: Amit Galitzky <[email protected]> (cherry picked from commit 4106cba) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Adding a state index (#110) * adding state index initial * addressed comments and added more fields to state index * addressed comments and fixed some unit tests * moved variables to common value and adressed other comments --------- (cherry picked from commit 4106cba) Signed-off-by: Amit Galitzky <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

github-actions bot added the backport 2.x backport PRs to 2.x branch label Oct 20, 2023

amitgalitz force-pushed the state-index branch 2 times, most recently from 9469bf1 to cd9ddb9 Compare October 20, 2023 00:47

joshpalis reviewed Oct 20, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/model/WorkflowState.java Show resolved Hide resolved

amitgalitz marked this pull request as ready for review October 23, 2023 16:45

amitgalitz requested review from dbwiddis, owaiskazi19, ohltyler and jackiehanyang as code owners October 23, 2023 16:45

dbwiddis reviewed Oct 26, 2023

View reviewed changes

amitgalitz force-pushed the state-index branch from cd9ddb9 to 818405e Compare October 27, 2023 17:53

owaiskazi19 reviewed Oct 27, 2023

View reviewed changes

joshpalis approved these changes Oct 30, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/model/State.java Outdated Show resolved Hide resolved

amitgalitz added 3 commits October 30, 2023 12:14

adding state index initial

68464ed

Signed-off-by: Amit Galitzky <[email protected]>

addressed comments and added more fields to state index

c70a18e

Signed-off-by: Amit Galitzky <[email protected]>

addressed comments and fixed some unit tests

730d61e

Signed-off-by: Amit Galitzky <[email protected]>

amitgalitz force-pushed the state-index branch from 2f8efed to 730d61e Compare October 30, 2023 19:21

owaiskazi19 reviewed Oct 30, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java Show resolved Hide resolved

owaiskazi19 reviewed Oct 30, 2023

View reviewed changes

owaiskazi19 mentioned this pull request Oct 30, 2023

[FEATURE] Enhance update workflow API to allow update on provisioned workflow #122

Closed

moved variables to common value and adressed other comments

36bcd2a

Signed-off-by: Amit Galitzky <[email protected]>

owaiskazi19 approved these changes Oct 30, 2023

View reviewed changes

amitgalitz merged commit 4106cba into opensearch-project:main Oct 30, 2023
19 of 21 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 30, 2023

[Backport 2.x] Adding a state index #125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a state index #110

Adding a state index #110

amitgalitz commented Oct 20, 2023 •

edited

Loading

joshpalis left a comment

dbwiddis left a comment

amitgalitz commented Oct 27, 2023

owaiskazi19 left a comment

joshpalis left a comment

codecov bot commented Oct 30, 2023 •

edited

Loading

owaiskazi19 left a comment

owaiskazi19 left a comment

Adding a state index #110

Adding a state index #110

Conversation

amitgalitz commented Oct 20, 2023 • edited Loading

Description

Issues Resolved

joshpalis left a comment

Choose a reason for hiding this comment

dbwiddis left a comment

Choose a reason for hiding this comment

amitgalitz commented Oct 27, 2023

owaiskazi19 left a comment

Choose a reason for hiding this comment

joshpalis left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 30, 2023 • edited Loading

Codecov Report

owaiskazi19 left a comment

Choose a reason for hiding this comment

owaiskazi19 left a comment

Choose a reason for hiding this comment

amitgalitz commented Oct 20, 2023 •

edited

Loading

codecov bot commented Oct 30, 2023 •

edited

Loading