Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POC for ssd cache #3723

Closed
wants to merge 25 commits into from
Closed

POC for ssd cache #3723

wants to merge 25 commits into from

Commits on Aug 18, 2023

  1. SSD Cache prototype

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    f28c81c View commit details
    Browse the repository at this point in the history

  2. Fix test_cache_storage_factory and cleanup

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1d4dad2 View commit details
    Browse the repository at this point in the history

  3. Clean up format

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    44379e3 View commit details
    Browse the repository at this point in the history

  4. Remove unnecessary dependencies

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    ab5dd9b View commit details
    Browse the repository at this point in the history

  5. Start cache storage service refactoring

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    4406858 View commit details
    Browse the repository at this point in the history

  6. Start cache storage controller cleanup

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    d66f6c8 View commit details
    Browse the repository at this point in the history

  7. Update quickwit/quickwit-control-plane/src/cache_storage_controller.rs 

    Co-authored-by: Paul Masurel <[email protected]>
    @imotov @fulmicoton
    imotov and fulmicoton committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    14be853 View commit details
    Browse the repository at this point in the history

  8. Update quickwit/quickwit-control-plane/src/cache_storage_controller.rs 

    Co-authored-by: Paul Masurel <[email protected]>
    @imotov @fulmicoton
    imotov and fulmicoton committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    028154b View commit details
    Browse the repository at this point in the history

  9. Update quickwit/quickwit-control-plane/src/cache_storage_controller.rs 

    Co-authored-by: Paul Masurel <[email protected]>
    @imotov @fulmicoton
    imotov and fulmicoton committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    859adb9 View commit details
    Browse the repository at this point in the history

  10. Refactor cache storage service configuration

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    686d73d View commit details
    Browse the repository at this point in the history

  11. Start refactoring cache storage

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    46cc9eb View commit details
    Browse the repository at this point in the history

  12. Fix format

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    3ee114c View commit details
    Browse the repository at this point in the history

  13. Fix build after main branch merge

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    d324652 View commit details
    Browse the repository at this point in the history

  14. Add proper loading and deletion logic to cache storage 

    WIP, still needs tests, throttling and handling of node restart.
    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    3a5198c View commit details
    Browse the repository at this point in the history

  15. Add cache storage communication flowchart.

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    090d6bb View commit details
    Browse the repository at this point in the history

  16. Enable cache storage stats 

    That should help with adding tests.
    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1636889 View commit details
    Browse the repository at this point in the history

  17. Add cache cleanup when an index is deleted

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    3422581 View commit details
    Browse the repository at this point in the history

  18. Fix build after rebase

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    13a350f View commit details
    Browse the repository at this point in the history

  19. Fix MetastoreError reference

    @imotov
    imotov committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    5475b8e View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2023

  1. Fix cache storage grpc server 

    Also adds an integration test to ensure that it works end-to-end.
    @imotov
    imotov committed Aug 19, 2023
    Configuration menu
    Copy the full SHA
    84db9e2 View commit details
    Browse the repository at this point in the history

  2. Fix format

    @imotov
    imotov committed Aug 19, 2023
    Configuration menu
    Copy the full SHA
    93acedb View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2023

  1. Minor code cleanups. 

    Cache storage is resolved only onced and cached.
Cache uri is a Uri.
Counters are not copied all over the place. The Factory one was actually disconnected from the rest.
Removed redundant `Arc` indirection
    @fulmicoton
    fulmicoton committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    bd28308 View commit details
    Browse the repository at this point in the history

  2. Fix accesing cache storage that got broken during cleanup

    @imotov
    imotov committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d0f1818 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. Fix cached_splits_registry's test_basic_workflow test

    @imotov
    imotov committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    03b4328 View commit details
    Browse the repository at this point in the history

  2. First step in refactoring of storage configuration 

    In this step we just change the way CacheStorageService is getting
access to CachedSplitRegistry.
    @imotov
    imotov committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    664e339 View commit details
    Browse the repository at this point in the history