Add support for deals using a data segment index #154
Conversation
A deal may, rather than a single car, be a series of concatinated car files with a segment index at the end containing an inclusion proof. When such an index is found, we will allow reading multiple cars and taking valid cids found in all of them as the contents of the overall deal.
|
filing filecoin-project/go-data-segment#9 to see if we can get a fixture to use for validation |
|
This PR brings Filecoin implementation detail to Dagstore and multiple new dependencies. Is there a way to implement this using Dagstore's API on the layer above, rather then polluting its internals? |
|
That's a good point, though carv2 indexes are in a way an equally opinionated existing equivalent. I'll see what it would look like for the 'get an index for this shard' to be fully handled by the caller |
|
I think there's an implicit decision being taken here, which is to model segments as indexed entries over a single shard (= storage deal in the Filecoin context), instead of as dedicated shards. Establishing a shard for every segment was not possible in the past since there was no standard way of delineating logical units within a Filecoin deal. With PoDSI, this is now possible. Making shard == segment could let us keep the dagstore Filecoin-agnostic, but I suspect is not viable without additional complexity. @willscott could you walk us through the tradeoffs here? |
shard == segment
shard == deal
|
|
Thanks! I think the case for shard == segment isn't strong today, but it may strengthen over time as selective unsealing becomes available, PoDSI gains traction, and segment-based access patterns become a bottleneck and the next frontier for optimization. Agree with the implicit decision made here, but +1 to @Wondertan's request to invert the control of the index population, so that the API caller can feed the segments. |
|
cc @Kubuxu |
…gment indices when present. This change will result in lotus parsing deals for local indexing according to filecoin-project/FIPs#512 This is a continuation of filecoin-project/dagstore#154 - This piece of the logic was seen as filecoin-specific rather than part of common dagstore functionality. A parallel to this logic is expected to be ported also to boost, but there remain confguration in the use of the dagstore where this is the common locations where this indexing functionality can be registered.
…gment indices when present. This change will result in lotus parsing deals for local indexing according to filecoin-project/FIPs#512 This is a continuation of filecoin-project/dagstore#154 - This piece of the logic was seen as filecoin-specific rather than part of common dagstore functionality. A parallel to this logic is expected to be ported also to boost, but there remain confguration in the use of the dagstore where this is the common locations where this indexing functionality can be registered.
…gment indices when present. This change will result in lotus parsing deals for local indexing according to filecoin-project/FIPs#512 This is a continuation of filecoin-project/dagstore#154 - This piece of the logic was seen as filecoin-specific rather than part of common dagstore functionality. A parallel to this logic is expected to be ported also to boost, but there remain confguration in the use of the dagstore where this is the common locations where this indexing functionality can be registered.
A deal may, rather than a single car, be a series of concatinated car files with a segment index at the end containing an inclusion proof.
When such an index is found, we will allow reading multiple cars and taking valid cids found in all of them as the contents of the overall deal.
ref: filecoin-project/boost#1258