Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: http handler support temp table. #16375

Merged
merged 6 commits into from
Sep 3, 2024
Merged

Conversation

youngsofun
Copy link
Member

@youngsofun youngsofun commented Sep 3, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  1. Implement a mechanism to clear data on S3 in case of a crash. Both MySQL and HTTP sessions need to write an ID to metadata with a TTL (time-to-live) and refresh it at regular intervals. The MySQL handler uses a daemon task for each connection, starting when the first query is made. The HTTP handler does it when renewing token.
  2. Both handlers need to clear data when a session ends to free up memory.
  3. The HTTP handler needs to clone the temp_tbl_mgr into the global ClientSessionManager and inject it into the session object when the next query arrives.
  4. Consequently, the HTTP handler needs to release it (and clean up data) when a session times out. only keep clones that are not empty, so we will not have to scan a lot of items when checking expiration

note for HTTP handler, will return error if want to create temp table but token is not used (we need logout to free resouces). waiting for datafuselabs/bendsql#479 and modify to other clients

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Sep 3, 2024
@youngsofun youngsofun marked this pull request as draft September 3, 2024 06:36
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Sep 3, 2024
@dosubot dosubot bot added the A-query Area: databend query label Sep 3, 2024
@youngsofun youngsofun marked this pull request as ready for review September 3, 2024 07:52
@dosubot dosubot bot added the C-feature Category: feature label Sep 3, 2024
@SkyFan2002
Copy link
Member

Let's track progress of http handler and clients in #16310

@youngsofun youngsofun mentioned this pull request Sep 3, 2024
11 tasks
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 3, 2024
@youngsofun youngsofun added this pull request to the merge queue Sep 3, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Sep 3, 2024
@youngsofun youngsofun added this pull request to the merge queue Sep 3, 2024
@BohuTANG BohuTANG removed this pull request from the merge queue due to a manual request Sep 3, 2024
@BohuTANG BohuTANG merged commit af391e7 into datafuselabs:main Sep 3, 2024
71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-query Area: databend query C-feature Category: feature lgtm This PR has been approved by a maintainer pr-feature this PR introduces a new feature to the codebase size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants