Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Cache all the id-name mapping of metalakes and catalogs to reduces IO operations #307

Open
yuqi1129 opened this issue Sep 1, 2023 · 0 comments
Assignees
Labels
improvement Improvements on everything

Comments

@yuqi1129
Copy link
Contributor

yuqi1129 commented Sep 1, 2023

What would you like to be improved?

Currently, in order to load a table entity, there will be at least four IO accesses (three for id-name mappings and one for real load), which is very time-consuming and costly.

Considering there will be only a few metalake and catalog entities and those entities are almost unchanged, So we propose we can cache the id-name mapping of those entities to reduce the access time.

How should we improve?

We can use Guava cache or Caffeine cache to keep the ids that map to metalakes and catalogs in memory.

Note: We should be careful about the consistency issue between the backend storage and cache

@yuqi1129 yuqi1129 added the improvement Improvements on everything label Sep 1, 2023
@yuqi1129 yuqi1129 self-assigned this Sep 1, 2023
@jerryshao jerryshao added this to the Graviton 0.3.0 milestone Oct 18, 2023
@jerryshao jerryshao removed this from the Gravitino 0.3.0 milestone Nov 30, 2023
@jerryshao jerryshao added this to the Gravitino 0.4.0 milestone Dec 14, 2023
@jerryshao jerryshao removed this from the Gravitino 0.4.0 milestone Jan 23, 2024
@jerryshao jerryshao added the 0.6.0 Release v0.6.0 label Aug 1, 2024
@jerryshao jerryshao added this to the Gravitino 0.7.0 milestone Aug 19, 2024
@jerryshao jerryshao removed the 0.6.0 Release v0.6.0 label Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvements on everything
Projects
None yet
Development

No branches or pull requests

2 participants