Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize sequential fetching #614

Open
jbearer opened this issue Jun 3, 2024 · 0 comments
Open

Optimize sequential fetching #614

jbearer opened this issue Jun 3, 2024 · 0 comments

Comments

@jbearer
Copy link
Member

jbearer commented Jun 3, 2024

When a DA node is syncing, via the proactive fetching task, we go through each block sequentially, and the fetcher automatically spawns off requests for blocks that are missing. This would perform much better if we fetched batches at a time instead of firing a separate request for each block. Unfortunately, this is a bit tricky to implement, because the streaming logic is very decoupled from the fetching logic (intentionally, this separation allows us to write very simple code without worrying about how/when objects are being fetched).

There's a couple of approaches we could take:

  • couple the streaming logic more tightly with the fetching logic (unfortunate breaking down of the abstraction)
  • at the fetching level, buffer fetch requests and then combining anything in the buffer that is from a consecutive range
  • optimize for streaming workflow by prefetching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant