Block extension data available via SHiP but not in ABI

[n8d]

It seems we have a new block_extension (type 3) in the SHiP stream, but I'm unable to deserialize it because it's not a type that is included in the ABI received on connecting to SHiP.

I'm actually not sure if type 2 is in the ABI as I've not tried to deserialize that one, but in general it seems if data is there we should be able to deserialize it.

[spoonincode]

As far as I can tell this isn't anything new -- other extensions aren't explicitly communicated in the ABI either. For example it seems for block header extension 0 one would need to manually know that when seeing extension.type: 0 then extension.bytes would need to be deserialized as checksum256[] -- this isn't communicated via the ABI.

I wonder if maybe we could change the various extensions in the ABI to appropriate variants. This would make accessing those extensions more seamless, but of course this would be a very breaking change that would require some sort of opt-in mechanism from the client. The main problem is that the block extension IDs are sparse which makes using a variant not a perfect fit.

Alternatively we can simply include the new extension types in the ABI and leave it up to the client to manually know which types goes with which ID.

[n8d]

Yeah it's not been explicit for others, but it's been possible, at least for the extensions I've needed. For example header extension 1 can be deserialized with producer_authority_schedule. Typically I will brute force it by trying to deserialize it with every type in the ABI until I find one that looks right, but I'm not having any luck with this one.

That said, I'm not actually sure if block extension 3 has what I need (because I haven't been able to view it). I'm trying to find where the new producer schedules are coming across in savanna blocks (or headers).

[heifner]

3 is a quorum_certificate_extension

spring/libraries/chain/include/eosio/chain/block.hpp

Line 64 in 5abe003

struct quorum_certificate_extension : fc::reflect_init {

spring/libraries/chain/include/eosio/chain/finality/qc.hpp

Line 77 in 5abe003

struct qc_t {

[n8d]

Ok, so not seeing a producer schedule there. If they are not in the original new_producers, nor header extension 1, where are they now?

[greg7mdp]

In the finality_extension, block header extension id 2. See finality_extension.hpp. Member new_proposer_policy_diff.

[heifner]

They are in the block header extension with id 2, finality_extension, when they are proposed.

spring/libraries/chain/include/eosio/chain/finality/finality_extension.hpp

Line 9 in 9e578e1

struct finality_extension : fc::reflect_init {

Note it is a diff of the new proposer policy to the old proposer policy. The diff is created by:

spring/libraries/libfc/include/fc/container/ordered_diff.hpp

Line 12 in 9e578e1

* @class ordered_diff

[n8d]

Thanks, but now it seems we are back to the original title of the GitHub issue with a different extension. 😆

I tested every type in the ABI and I'm unable to deserialize block header extension 2.

[arhag]

We will add extra types in the SHiP ABI to allow deserializing the block header extension (this is added by #783). But we decided to not add support for deserializing the block extension because of the problems introduced by the serialization of the dynamic bitset.