Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration with REPL Spark job #2327

Merged
merged 24 commits into from
Oct 20, 2023

Conversation

penghuo
Copy link
Collaborator

@penghuo penghuo commented Oct 18, 2023

Description

  1. create query_execution_request when create session, query_execution_request is bind to datasource, index naming conversion is .query_execution_request_{{datasourceName}}
  2. stash query_execution_request context when access query_execution_request indices.
  3. add REPL spark parameters.
  4. add IT in transport layer.

Major changes

  • add datasourceName for statement.
  • encode datasourceName into sessionId.

Issues Resolved

#2089

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
@codecov
Copy link

codecov bot commented Oct 18, 2023

Codecov Report

Merging #2327 (edff270) into main (8f5e01d) will decrease coverage by 0.20%.
Report is 1 commits behind head on main.
The diff coverage is 88.12%.

@@             Coverage Diff              @@
##               main    #2327      +/-   ##
============================================
- Coverage     96.69%   96.49%   -0.20%     
- Complexity     4839     4884      +45     
============================================
  Files           454      458       +4     
  Lines         13203    13398     +195     
  Branches        896      909      +13     
============================================
+ Hits          12767    12929     +162     
- Misses          422      452      +30     
- Partials         14       17       +3     
Flag Coverage Δ
sql-engine 96.49% <88.12%> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
.../spark/asyncquery/model/SparkSubmitParameters.java 100.00% <100.00%> (ø)
...earch/sql/spark/data/constants/SparkConstants.java 0.00% <ø> (ø)
...rch/sql/spark/dispatcher/SparkQueryDispatcher.java 100.00% <100.00%> (ø)
.../spark/execution/session/CreateSessionRequest.java 100.00% <100.00%> (ø)
...ql/spark/execution/session/InteractiveSession.java 100.00% <100.00%> (ø)
...nsearch/sql/spark/execution/session/SessionId.java 100.00% <100.00%> (ø)
...ch/sql/spark/execution/session/SessionManager.java 100.00% <100.00%> (ø)
...arch/sql/spark/execution/session/SessionState.java 100.00% <100.00%> (ø)
...earch/sql/spark/execution/session/SessionType.java 100.00% <100.00%> (ø)
...earch/sql/spark/execution/statement/Statement.java 100.00% <100.00%> (ø)
... and 4 more

... and 7 files with indirect coverage changes

@penghuo penghuo self-assigned this Oct 18, 2023
@penghuo penghuo marked this pull request as ready for review October 18, 2023 23:07
dai-chen
dai-chen previously approved these changes Oct 20, 2023
Signed-off-by: Peng Huo <[email protected]>
@penghuo penghuo merged commit 7b4156e into opensearch-project:main Oct 20, 2023
19 of 21 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 20, 2023
* add InteractiveSession and SessionManager

Signed-off-by: Peng Huo <[email protected]>

* add statement

Signed-off-by: Peng Huo <[email protected]>

* add statement

Signed-off-by: Peng Huo <[email protected]>

* fix format

Signed-off-by: Peng Huo <[email protected]>

* snapshot

Signed-off-by: Peng Huo <[email protected]>

* address comments

Signed-off-by: Peng Huo <[email protected]>

* update

Signed-off-by: Peng Huo <[email protected]>

* Update REST and Transport interface

Signed-off-by: Peng Huo <[email protected]>

* Revert on transport layer

Signed-off-by: Peng Huo <[email protected]>

* format code

Signed-off-by: Peng Huo <[email protected]>

* add API doc

Signed-off-by: Peng Huo <[email protected]>

* modify api

Signed-off-by: Peng Huo <[email protected]>

* create query_execution_request index on demand

Signed-off-by: Peng Huo <[email protected]>

* add REPL spark parameters

Signed-off-by: Peng Huo <[email protected]>

* Add IT

Signed-off-by: Peng Huo <[email protected]>

* format code

Signed-off-by: Peng Huo <[email protected]>

* bind request index to datasource

Signed-off-by: Peng Huo <[email protected]>

* fix bug when fetch query result

Signed-off-by: Peng Huo <[email protected]>

* revert entrypoint class

Signed-off-by: Peng Huo <[email protected]>

* update mapping

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit 7b4156e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 20, 2023
* add InteractiveSession and SessionManager

Signed-off-by: Peng Huo <[email protected]>

* add statement

Signed-off-by: Peng Huo <[email protected]>

* add statement

Signed-off-by: Peng Huo <[email protected]>

* fix format

Signed-off-by: Peng Huo <[email protected]>

* snapshot

Signed-off-by: Peng Huo <[email protected]>

* address comments

Signed-off-by: Peng Huo <[email protected]>

* update

Signed-off-by: Peng Huo <[email protected]>

* Update REST and Transport interface

Signed-off-by: Peng Huo <[email protected]>

* Revert on transport layer

Signed-off-by: Peng Huo <[email protected]>

* format code

Signed-off-by: Peng Huo <[email protected]>

* add API doc

Signed-off-by: Peng Huo <[email protected]>

* modify api

Signed-off-by: Peng Huo <[email protected]>

* create query_execution_request index on demand

Signed-off-by: Peng Huo <[email protected]>

* add REPL spark parameters

Signed-off-by: Peng Huo <[email protected]>

* Add IT

Signed-off-by: Peng Huo <[email protected]>

* format code

Signed-off-by: Peng Huo <[email protected]>

* bind request index to datasource

Signed-off-by: Peng Huo <[email protected]>

* fix bug when fetch query result

Signed-off-by: Peng Huo <[email protected]>

* revert entrypoint class

Signed-off-by: Peng Huo <[email protected]>

* update mapping

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit 7b4156e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo pushed a commit that referenced this pull request Oct 20, 2023
* add InteractiveSession and SessionManager



* add statement



* add statement



* fix format



* snapshot



* address comments



* update



* Update REST and Transport interface



* Revert on transport layer



* format code



* add API doc



* modify api



* create query_execution_request index on demand



* add REPL spark parameters



* Add IT



* format code



* bind request index to datasource



* fix bug when fetch query result



* revert entrypoint class



* update mapping



---------


(cherry picked from commit 7b4156e)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo pushed a commit that referenced this pull request Oct 21, 2023
* add InteractiveSession and SessionManager



* add statement



* add statement



* fix format



* snapshot



* address comments



* update



* Update REST and Transport interface



* Revert on transport layer



* format code



* add API doc



* modify api



* create query_execution_request index on demand



* add REPL spark parameters



* Add IT



* format code



* bind request index to datasource



* fix bug when fetch query result



* revert entrypoint class



* update mapping



---------


(cherry picked from commit 7b4156e)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
vamsi-amazon added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023
vamsi-amazon added a commit that referenced this pull request Nov 13, 2023
* Revert "Add more metrics and handle emr exception message (#2422) (#2426)"

This reverts commit b57f7cc.

* Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)"

This reverts commit 3024737.

* Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)"

This reverts commit 6e17ae6.

* Revert "Redefine Drop Index as logical delete (#2386) (#2397)"

This reverts commit e939bb6.

* Revert "add concurrent limit on datasource and sessions (#2390) (#2395)"

This reverts commit deb3ccf.

* Revert "Add Flint Index Purging Logic (#2372) (#2389)"

This reverts commit dd48b9b.

* Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)"

This reverts commit 37e010f.

* Revert "Enable session by default (#2373) (#2375)"

This reverts commit 7d95e4c.

* Revert "Create new session if client provided session is invalid (#2368) (#2371)"

This reverts commit 5ab7858.

* Revert "Add where clause support in create statement (#2366) (#2370)"

This reverts commit b620a56.

* Revert "create new session if current session not ready (#2363) (#2365)"

This reverts commit 5d07281.

* Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)"

This reverts commit 16e2f30.

* Revert "Add Session limitation (#2354) (#2359)"

This reverts commit 0f334f8.

* Revert "Bug Fix, support cancel query in running state (#2351) (#2353)"

This reverts commit 9a40591.

* Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)"

This reverts commit e4827a5.

* Revert "Add missing tags and MV support (#2336) (#2346)"

This reverts commit 8791bb0.

* Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)"

This reverts commit bea432c.

* Revert "Integration with REPL Spark job (#2327) (#2338)"

This reverts commit 58a5ae5.

* Revert "Implement patch API for datasources (#2273) (#2329)"

This reverts commit 4c151fe.

* Revert "Add sessionId parameters for create async query API (#2312) (#2324)"

This reverts commit 3d1a376.

* Revert "Add Statement (#2294) (#2318) (#2319)"

This reverts commit b3c2e94.

* Revert "Upgrade json (#2307) (#2314)"

This reverts commit 6c65bb4.

* Revert "Minor Refactoring (#2308) (#2317)"

This reverts commit 051cc4f.

* Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)"

This reverts commit 6ac197b.

---------

Co-authored-by: Vamsi Manohar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants