Add block indexing to Indexer API

[aaronbuchwald]

This PR adds block indexing to the indexer API.

The new API serves both the block JSON and the ExecutedBlock bytes to support both #1513 and decoding the response in the golang indexer client.

We can't unmarshal the block's JSON as is because it requires unmarshalling into the Action and Auth interfaces within the transaction. This PR includes both the JSON and binary in the response to support both cases.

[aaronbuchwald]

Alternative here is to include the requested encoding in the request

[containerman17]

It might create a mess. There aren't that many methods. It's fine to duplicate unless we plan to increase the number of methods.

[containerman17]

1. GetBlock improvements

{
    "jsonrpc": "2.0",
    "result": {
        "block": {
            "blockID": "2NmFKR118wRQc88qgmgeLqjz125UWxRb7jsv9nFNj6ygfedKyL",
            "block": {
                "parent": "qsFoFWSJkFgXgtWmh8MtyVgdTtzv77qNyjJPB7Fc43UXeUUKj",
                "timestamp": 1727846356939,
                "height": 3420,
                "txs": [
                    {
                        // 1.4 Transaction ID  
                        "base": {
                            "timestamp": 1727846413000,
                            "chainId": "2gdnF9MkpFFBsmFQHSpDrnXT5N3DGKXPmBJfradUyE3kzHc4QE",
                            "maxFee": 10000000
                        },
                        "actions": [
                            {// 1.2 Action names or IDs
                                "to": "0x0000000000000000000000000000000000000000000000000000000000deadc0de",
                                "value": 123456789,
                                "memo": "THVpZ2k="
                            }
                        ],
                        "auth": {
                            "signer": [ // 1.3 Auth.* should be hex
                                70,
                                220,
                                217,
                                //... more bytes here
                            ],
                            "signature": [ // 1.3 Auth.* should be hex
                                103,
                                151,
                                155,
                                //... more bytes here
                            ]
                        }
                    }
                ],
                "stateRoot": "28TpB5nFDvBYZWfEZesFpDA8zLic4MnXYwniEFDKvpkWkiBKYD"
            },
            "results": [
                { // 1.6 Result fields should be lowercase
                    "Success": true,
                    "Error": "",// 1.5 Transaction error should be a Unicode string
                    "Outputs": [
                        "aW52YWxpZCBiYWxhbmNlOiBjb...."
                    ],
                    // 1.1 Units and unit prices should be an array or object
                    "Units": "(Bandwidth=197, Compute=7, Storage(Read)=14, Storage(Allocate)=50, Storage(Write)=26)", 
                    "Fee": 29400
                }
            ],
            // 1.1 Units and unit prices should be an array or object
            "unitPrices": "(Bandwidth=100, Compute=100, Storage(Read)=100, Storage(Allocate)=100, Storage(Write)=100)"
        },
        "blockBytes": "000001196ef5112686d9fcfc4b3a3d2f2d439e1" //shortened
    },
    "id": 2652924454900838
}

1.1 Units and unit prices format

Should be an array or object. Binary format (see fees.UnpackDimensions) would work too. If needed, I can write a parser for the current string format too.

1.2 Action names or IDs

There is action payload, but no action name or ID. The ideal solution is to provide action binary, including the first TypeID byte, as discussed.

1.3 Auth.* as hex

Currently an array of numbers. It should be in hex format (note that in Golang, []byte serializes as base64 by default).

1.4 Transaction ID

Transaction ID is missing.

1.5 Transaction error should be a Unicode string

I can convert from base64 to bytes, then into Unicode, but a regular string format would look much better in the API

1.6 Result fields should be lowercase

For consistency across APIs.

[containerman17]

2. GetTX improvements

{
    "jsonrpc": "2.0",
    "result": {
        // 2.1 Error field 
        "timestamp": 1727848107595,
        "success": false,
        "units": "(Bandwidth=197, Compute=7, Storage(Read)=14, Storage(Allocate)=50, Storage(Write)=26)", // 2.2 Units format
        "fee": 29400,
        "result": []
        // 2.3 Sender address
    },
    "id": 9669870267519104
}

2.1 Error field

Although this transaction failed, there is no error field in the JSON. Unicode would be better, than base64 too.

2.2 Units and unit prices format

Same as 1.1 above, it should be formatted as an array or object.

2.3 Sender address

In GetBlock, I infer the sender's address by hashing their public key from the auth section. Since there's no auth here, adding a sender address field would be the best solution.

[aaronbuchwald]

Ok here are the changes I'm planning to make as a follow on to this PR:

Tx JSON Marshalling

• include txID in JSON
• marshal actions and auth as bytes and encode them as hex via codec.Bytes
• Add TransactionJSON intermediate type that supports MarshalJSON and UnmarshalJSON and stores the action/auth as bytes
• add MarshalJSON implementation to chain.Transaction
• add UnmarshalJSON that takes chain.Parser in as a parameter so that it can parse the actions/auth

Results

• Add JSON annotations to lowercase the fields of chain.Result
• Add ResultJSON intermediate representation that supports json marshalling and add MarshalJSON and UnmarshalJSON to chain.Result that uses the intermediate type (will use codec.Bytes for the error result)
• Already added JSON marshalling for fees.Dimensions here: Add JSON marshalling for fee dimensions #1622

Side note (out of scope for this PR): we currently use an error string for the result. There's a weird edge case here where if we or a developer changes an error string that is actually part of chain processing. This should be clear, but I've found that it's easy to forget this and think we could make it more explicit. We could, for example, include errors as possible return values in the ABI.

Added here #1627

Block Requests

• Add encoding parameter to block request types
• Use a single block response type w/ omitempty so that it sends JSON with either the block bytes or JSON of the block

Note: golang clients can use binary representation, so that it can decode with chain.Parser argument and other languages can handle the JSON or binary if they choose

[containerman17]

Why do we limit it? If someone wants to run an indexer that stores all transactions and they have 20TB of NVMe, why should we stop them?

[aaronbuchwald]

We don't maintain a blockID -> height mapping on disk to avoid using hashes as keys. This means we iterate over all the data on startup, so we set a reasonable guardrail.

If you want to store all transactions, you need to scale horizontally rather than writing to a single pebbledb instance. This won't work at scale, so I'd prefer to set a reasonable guardrail than allow users to try and scale something that wasn't meant to.

[containerman17]

What prevents a Block from being unmarshalled from JSON? It's the Auth and Action interfaces, right?

[aaronbuchwald]

Yup

[joshua-kim]

Why even have this separate type? If we're only going to use the block bytes from the response since we can't unmarshal the actual block type why don't we just drop the block field from the server-side response struct entirely and get rid of this second struct?

[aaronbuchwald]

This is to support clients (JS) that prefer to handle JSON than the binary representation. For golang, we actually cannot handle the JSON using the current requester because it will simply unmarshal the json into the struct and we can't unmarshal JSON directly into the block type due to the Action and Auth interfaces.

Have a note here #1606 (comment) to add an encoding parameter that we could block this PR on if you prefer.

[containerman17]

I would either store the entire transaction or just the result. There's no need for a custom marshaler in this case.

[aaronbuchwald]

I agree and think storing the entire tx + result makes sense here #1626

This is not changed in this PR though, so don't think it needs to be in scope

[joshua-kim]

This test feels like it's testing a lot of different cases which is making it complex to read/debug/add to. I feel like testing more targeted code edge-cases instead of a single god-test would make sense here.

[aaronbuchwald]

Moved a helper to generate a sequence of empty executed blocks to chaintest and separated out the restart portion of the test.