Add support for the async iterators #21
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current implementation of the iterators is not flexible. There are
each
,range
,keys
andvalues
.range
,keys
andvalues
- all accumulate the results and return it like that. Given the use cases in the bcoin and hsd - this is never a good idea. Some entries in db if they can't be range filtered (gte/lte) properly and used w/o limits can result in huge memory usage and also double processing in most of them time (stored in array -> filter/modify -> etc.).each
allows to work on each while they are received from the database, but it's mostly still accumulated as you would need to setup callback chain in order to use it properly.This implementation of the iterators instead uses
for await ()
which allows for the chaining database results to the codebase. Instead of consuming the data from db at once and constructing array to use, we can chain asyng generators if we want to have transformations which can reduce the amount of information we store in memory.Iterators, DB and bucket now have additional methods for the async generators:
values: true
in the iterator configs.Examples:
Best translation of this concept is using each:
Which can be now written as:
In case where we want to chain multiple calls it becomes more complicated than just using
array
s, but as I mentioned above - accumulating arrays first comes with huge cost for the big queries.Example of chaining these calls: