Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement leveldb storage for dataposter #1736

Merged
merged 29 commits into from
Jul 12, 2023
Merged

Conversation

anodar
Copy link
Contributor

@anodar anodar commented Jul 5, 2023

No description provided.

@cla-bot cla-bot bot added the s Automatically added by the CLA bot if the creator of a PR is registered as having signed the CLA. label Jul 5, 2023
@anodar anodar changed the title draft Implement leveldb storage Jul 6, 2023
@anodar anodar marked this pull request as ready for review July 6, 2023 15:53
@anodar anodar changed the title Implement leveldb storage Implement leveldb storage for dataposter Jul 6, 2023
@anodar anodar requested a review from PlasmaPower July 6, 2023 15:53
arbnode/schema.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/leveldb/leveldb.go Outdated Show resolved Hide resolved
arbnode/dataposter/storage_test.go Show resolved Hide resolved
arbnode/dataposter/storage_test.go Show resolved Hide resolved
@anodar anodar requested a review from PlasmaPower July 11, 2023 21:09
Copy link
Collaborator

@PlasmaPower PlasmaPower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI it looks like tests are failing due to cmp, and I think the linter's complaining about "printf: (*testing.common).Errorf call needs 2 args but has 3 args (govet)" for storage_test.go line 182

func (s *Storage[Item]) GetLast(ctx context.Context) (*Item, error) {
lastItemIdx, err := s.lastItemIdx(ctx)
if err != nil {
return nil, fmt.Errorf("getting last item index: %w", err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should return nil, nil if the last item index is missing (i.e. there are no items). We should also think about what happens when all items are pruned. In that case, we probably want to have pruning remove the last item index from the db.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Checking length, if it's 0 then returning nil, nil. So no need to remove last item idx from db when pruning (for all public methods behavior will be exactly the same).

}
if err := b.Put(countKey, []byte(strconv.Itoa(cnt+1))); err != nil {
return fmt.Errorf("updating length counter: %w", err)
if bytes.Compare(key, lastItemIdx) > 0 {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love assuming the return value of lastItemIdx is empty if leveldb.ErrNotFound is returned. In practice I'm sure it'll be empty, but I don't think it's good Go practice to read the value returned in an error is present.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -71,6 +71,8 @@ func (s *Storage[Item]) Put(_ context.Context, index uint64, prevItem *Item, new
if queueIdx > len(s.queue) {
return fmt.Errorf("attempted to set out-of-bounds index %v in queue starting at %v of length %v", index, s.firstNonce, len(s.queue))
}
// TODO: come up with a better way of comparing prev/new values,
// we shouldn't use cmp.Diff outside tests.
if diff := cmp.Diff(prevItem, s.queue[queueIdx]); diff != "" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI the original code did intentionally use pointer comparisons, because that worked with the batch poster's design. But I agree it isn't ideal. We should either RLP encode both and compare them here, or we could use reflect.DeepEqual.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@anodar anodar requested a review from PlasmaPower July 11, 2023 21:50
@anodar anodar dismissed PlasmaPower’s stale review July 11, 2023 21:53

The merge-base changed after approval.

PlasmaPower
PlasmaPower previously approved these changes Jul 11, 2023
Copy link
Collaborator

@PlasmaPower PlasmaPower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@anodar anodar requested a review from PlasmaPower July 11, 2023 22:26
Copy link
Collaborator

@PlasmaPower PlasmaPower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@PlasmaPower PlasmaPower merged commit f8a9421 into master Jul 12, 2023
7 checks passed
@PlasmaPower PlasmaPower deleted the leveldb-based-dataposter branch July 12, 2023 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
s Automatically added by the CLA bot if the creator of a PR is registered as having signed the CLA.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants