-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DNM] memdb: replace the current implementation with ART(adaptive radix tree) #1400
Draft
you06
wants to merge
18
commits into
tikv:master
Choose a base branch
from
you06:staging-art
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+4,163
−56
Draft
Changes from 3 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
86b9e7d
Add an art(from plar/go-adaptive-radix-tree) with staging support as …
you06 98cc66b
clean code
you06 4c6c5f7
skip slow race test
you06 3fe84bd
opt minimum
you06 8b06ac0
make longestCommonPrefix a common func
you06 b5b791f
remove unnecessary check & use unsafe.Sizeof
you06 f8eecf2
remove unnecessary check in minimum
you06 15affcc
remove recursive func usage for minimum
you06 9bcaf42
remove unused present for n4 and n16 & enlarge in-node prefix
you06 6d1aa8b
Merge branch 'master' into staging-art
you06 90e7e39
fast search for n256
you06 be7f324
address comment
you06 30dd323
use uint32 address & wip design doc
you06 ce55f0c
add node section
you06 9f5d150
fix lint
you06 5584048
manually inline critical path
you06 f2e8c08
refine code
you06 bded7db
remove art lib & add tests
you06 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| package unionstore | ||
|
|
||
| import ( | ||
| "context" | ||
|
|
||
| tikverr "github.com/tikv/client-go/v2/error" | ||
| art "github.com/tikv/client-go/v2/internal/unionstore/art" | ||
| ) | ||
|
|
||
| var _ MemBuffer = &ArenaArt{} | ||
|
|
||
| type ArenaArt struct { | ||
| *art.Art | ||
| } | ||
|
|
||
| func NewArenaArt() *ArenaArt { | ||
| return &ArenaArt{ | ||
| Art: art.New(), | ||
| } | ||
| } | ||
|
|
||
| func (a *ArenaArt) setSkipMutex(skip bool) { | ||
| a.Art.SetSkipMutex(skip) | ||
| } | ||
|
|
||
| func (a *ArenaArt) Get(_ context.Context, k []byte) ([]byte, error) { | ||
| return a.Art.Get(k) | ||
| } | ||
|
|
||
| // GetLocal gets the value from the buffer in local memory. | ||
| // It makes nonsense for MemDB, but makes a difference for pipelined DML. | ||
| func (a *ArenaArt) GetLocal(_ context.Context, key []byte) ([]byte, error) { | ||
| return a.Art.Get(key) | ||
| } | ||
|
|
||
| // BatchGet gets the values for given keys from the MemBuffer and cache the result if there are remote buffer. | ||
| func (a *ArenaArt) BatchGet(_ context.Context, keys [][]byte) (map[string][]byte, error) { | ||
| if !a.Art.Dirty() { | ||
| return map[string][]byte{}, nil | ||
| } | ||
| m := make(map[string][]byte, len(keys)) | ||
| for _, k := range keys { | ||
| v, err := a.Art.Get(k) | ||
| if err != nil { | ||
| if tikverr.IsErrNotFound(err) { | ||
| continue | ||
| } | ||
| return nil, err | ||
| } | ||
| m[string(k)] = v | ||
| } | ||
| return m, nil | ||
| } | ||
|
|
||
| func (a *ArenaArt) RemoveFromBuffer(key []byte) {} | ||
|
|
||
| // Iter implements the Retriever interface. | ||
| func (a *ArenaArt) Iter(start []byte, end []byte) (Iterator, error) { | ||
| it, err := a.Art.Iter(start, end) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return it, err | ||
| } | ||
|
|
||
| // IterReverse implements the Retriever interface. | ||
| func (a *ArenaArt) IterReverse(end, start []byte) (Iterator, error) { | ||
| it, err := a.Art.IterReverse(end, start) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return it, err | ||
| } | ||
|
|
||
| // SnapshotIter returns an Iterator for a snapshot of MemBuffer. | ||
| func (a *ArenaArt) SnapshotIter(lower, upper []byte) Iterator { | ||
| return a.Art.SnapshotIter(lower, upper) | ||
| } | ||
|
|
||
| // SnapshotIterReverse returns a reversed Iterator for a snapshot of MemBuffer. | ||
| func (a *ArenaArt) SnapshotIterReverse(upper, lower []byte) Iterator { | ||
| return a.Art.SnapshotIterReverse(upper, lower) | ||
| } | ||
|
|
||
| func (a *ArenaArt) SnapshotGetter() Getter { | ||
| return a.Art.SnapshotGetter() | ||
| } | ||
|
|
||
| func (a *ArenaArt) Dirty() bool { | ||
| return a.Art.Dirty() | ||
| } | ||
|
|
||
| // Checkpoint returns the checkpoint of the MemBuffer. | ||
| func (a *ArenaArt) Checkpoint() *MemDBCheckpoint { | ||
| blockSize, blocks, offsetInBlock := a.Art.Checkpoint() | ||
| return &MemDBCheckpoint{ | ||
| blockSize: blockSize, | ||
| blocks: blocks, | ||
| offsetInBlock: offsetInBlock, | ||
| } | ||
| } | ||
|
|
||
| // RevertToCheckpoint reverts the MemBuffer to the specified checkpoint. | ||
| func (a *ArenaArt) RevertToCheckpoint(cp *MemDBCheckpoint) { | ||
| a.Art.RevertToCheckpoint(cp.blockSize, cp.blocks, cp.offsetInBlock) | ||
| } | ||
|
|
||
| // GetMemDB returns the MemDB binding to this MemBuffer. | ||
| // This method can also be used for bypassing the wrapper of MemDB. | ||
| func (a *ArenaArt) GetMemDB() *MemDB { | ||
| return nil | ||
| } | ||
|
|
||
| // Flush flushes the pipelined memdb when the keys or sizes reach the threshold. | ||
| // If force is true, it will flush the memdb without size limitation. | ||
| // it returns true when the memdb is flushed, and returns error when there are any failures. | ||
| func (a *ArenaArt) Flush(force bool) (bool, error) { | ||
| return false, nil | ||
| } | ||
|
|
||
| // FlushWait waits for the flushing task done and return error. | ||
| func (a *ArenaArt) FlushWait() error { | ||
| return nil | ||
| } | ||
|
|
||
| // GetFlushMetrics returns the metrics related to flushing | ||
| func (a *ArenaArt) GetFlushMetrics() FlushMetrics { | ||
| return FlushMetrics{} | ||
| } | ||
|
|
||
| func (a *ArenaArt) stages() []art.ARTCheckpoint { | ||
| return a.Art.Stages() | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we also need to review this? Shall we consider merge the changes to upstream?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to review this, I import it only for benchmark test. My changes in
0278e8bfcd2bis only for the special usage in p-dml. As the mem arena implementation, I don't think the auther would like to accept it, since the impact of this change is too large(all the pointers are replaced by our self-defined address), and it actually sacrifice the read performance by introducing vlog.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe the memory arena without vlog is useful for them, but I don't have any performance data for it, what about discussing with them to see if they would like to adopt our improvement, which is not related to our implementation in client-go(I think client-go's requirements are quite unique).