API 2.0

[avive]

Additional Functionality

Below are API changes and addition required due to support the new 0.3 features such as self healing and Spacemesh Apps.

MeshService

Layer Status

Update layer statuses due to self healing related changes. (see #144)

enum LayerStatus {
     LAYER_STATUS_UNSPECIFIED = 0; // unknown
     LAYER_STATUS_APPROVED = 1; // approved by hare
     LAYER_STATUS_CONFIRMED = 2; // confirmed by tortoise and state applied for txs in the layer
     LAYER_STATUS_PENDING = 3; // not yet approved or confirmed
     LAYER_STATUS_VERIFIED = 4; // approved by tortoise
 }

Compatibility: this new enum is syntactically compatible with api 1.0 however the meaning of the first 3 values changes.
This is the v0.1.1 syntax:

enum LayerStatus {
 LAYER_STATUS_UNSPECIFIED = 0; // not yet approved or confirmed
 LAYER_STATUS_APPROVED = 1; // approved by hare
 LAYER_STATUS_CONFIRMED = 2; // confirmed by tortoise
}

Activation Layer

Disassociate activation from a specific layer and associate it with a specific epoch. (see #132, #133)

Compatibility: this can be added in a backward-compatible way if we keep the Activation.layer field but set it to const 0 in new api responses.

---

GlobalStateService

GetStateRoot

Expects:

• layer (layer_num) - This parameter optional and defaults to the most recent layer of the Node.

The API returns:

• layer_num - The layer_num used to process the request
• state - the State (the Global State root hash) associated with the layer_num.

Namespaces

Supports state namespaces - A namespace and Templates namespace.

Get Template

The API expects a Template Address and it will return back the following:

1. Template (`Address) - The value supplied to the request
2. Name (String)
3. Author (Address)
4. DeployedAt Layer (layer_num)
   The layer_num at which the Template has been deployed at
5. Tx Id (An array of 32 bytes - needs to review again the number of bytes)
   The id of the transaction that deployed the template (a transaction of type deploy template)
6. Code (Binary Wasm)
   This is a blob of data that can be translated to Wasm textual representation of inspection (at the client-side)
7. Schema
   Right now we only support fixed-length variables for an App's storage.
   So for now, the schema will be an Array of fixed-length variables.

Each variable returned will specify:

• id - an integer serving as a unique identifier within the App's variables (fits into 2 bytes)
• name - String
• type
  A type can be a primitive: i8 / u8 / i16 / i16/ u32 / i32 / u64 / i64 / bool / Amount / Address
  or it can be an Array of primitives for example: Amount of length = 3

GetAllTemplates

In Genesis, we only deploy the Vaults templates (the exact number of templates will be within 1 to 3 probably)
We'll be able to return the whole Templates in a single call.
Since we want to be future compatible when there will be more Template (created not as part of Genesis),
we'd like the API to support pagination from the start.

The API will have pagination, so each request will contain:

• offset - an u32 integer.
• count - how many Templates to return
• returnCode - optional. we want to return a code for each returned Template (defaults to false)

The API will return:

• offset (u32) - The value supplied to the request
• count (u32) - The value supplied to the request
• hasMore (bool) - Whether there're more templates
• total (u32) - The total number of templates
• [Template] - an Array where each item is a Template of the same format as Get Template above.
• Code - This is a blob of data that can be translated to Wasm textual representation of inspection (at the client-side).
  This field is returned when `returnCode = true in the request

App

Get App Variable

The API expects:

• App (Address)
• Layer (layer_num) - This parameter will be optional. It defaults to the most recent layer.
• variable id - an integer serving as a unique identifier within the App's variables.

The response will be:

• App - The Address supplied to the request
• Layer - The layer of the App used for processing the request
• id - The variable id supplied to the request
• Name - String
• Spawner - The Address of the Account that spawned the App
• Template (Address)
  The Address of the Template that the App has been spawned from
• State - The State associated with processed layer
• SpawnedAt Layer (layer_num)
  The layer_num at which the App has been spawned
• Tx Id (An array of 32 bytes - needs to review again the number of bytes)
• type - see above the options for type
• value
  • i8 - i8 number
  • u8 - u8 number
  • i16 - i16 number
  • u16 - u16 number
  • u32 - u32 number
  • i32 - i32 number
  • i64 - i64 number
  • u64 - u64 number
  • bool - true / false
  • Amount - u64 number
  • Address - 20 bytes

Get App

The API expects:

• App (Address)
• Layer (layer_num) - This parameter will be optional. It defaults to the most recent layer.
  The value of the layer will affect the values of the returned variables.

The API will return:

• App - The Address supplied to the request
• Layer - The layer of the App used for processing the request
• Name (String)
• Spawner - The Address of the Account that spawned the App
• Template (Address)
  The Address of the Template that the App has been spawned from
• State - The State associated with processed layer
• SpawnedAt Layer (layer_num)
  The layer_num at which the App has been spawned
• Tx Id (An array of 32 bytes - needs to review again the number of bytes)
  The id of the transaction that spawned the app (a transaction of type spawn app)
• variables (an Array of variable and their values)
  Each variable will be returned as specified above under Get App Variable

Compatibility notes: these changes can be backward compatible by adding new methods and depreciating older methods (such as AccountDataQuery) while still supporting them.

Receipts

SVM executes 3 types of transactions: deploy template / spawn app / execute app.
Each kind of transaction has a corresponding Receipt type.

Following are the high-level specifications of the Receipts.

Deploy Template Receipt

The Deploy Template receipt should be created after executing a deploy template transaction.
Each Receipt will contain the following:

1. version (u16)
2. success (bool) - Whether the deploy template transaction has succeeded
3. error - optional. When success=false it will contain the error that occurred (see: Receipt Error later)
4. gas_used (u64) - The gas units used for executing the transaction.
5. logs - see Receipt Logs

Spawn App Receipt

1. version (u16)
2. success (bool) - Whether the spawn app transaction has succeeded
3. error - optional. When success=false it will contain the error that occurred (see: Receipt Error later)
4. app (Address) - The newly spawned app Address. Relevant only when success=true
5. init_state (State) - The newly spawned app initial State. Relevant only when success=true
6. return_data (bytes) - The return data is the spawn app ctor execution output.
   The encoding of the returned data is determined by the code running. The best practice is to use the same ABI used
   when encoding input CallData of the ctor input.

@avive in order to decode at the client the return data the client needs to write ad-hoc code to the ABI used.
Right now, the Vaults can use the svm_codec.wasm for achieving that easily. This file is generated after each SVM successful CI run (can be taken from the CI artifacts).

1. gas_used (u64) - The gas units used for executing the transaction.
2. logs - see Receipt Logs

The Spawn App receipt should be created after executing a spawn template transaction.
Each Receipt will contain the following:

Execute App Receipt

1. version (u16)
2. success (bool) - Whether the exec app transaction has succeeded
3. error - optional. When success=false it will contain the error that occurred (see: Receipt Error later)
4. new_state (State) - The newly executed app State. Relevant only when success=true
5. return_data (bytes) - The return data is the executed function output.
   The encoding of the returned data is determined by the code running. The best practice is to use the same ABI used
   when encoding input CallData of the ctor input.

@avive in order to decode at the client the return data the client needs to write ad-hoc code to the ABI used.
Right now, the Vaults can use the svm_codec.wasm for achieving that easily. This file is generated after each SVM successful CI run (can be taken from the CI artifacts).

1. gas_used (u64) - The gas units used for executing the transaction.
2. logs - see Receipt Logs

Receipt Logs

Each Receipt will be accompanied with zero or more logs.
It's very useful for debugging and extracting events that have happened during the running of the transaction.
It's important to stress, that logs are allowed to be returned when transaction fails.
So if a program ran and accumulate 5 logs before panicking - these 5 log entries will be attached to the Receipt as well.

Each log entry contains:

• msg (bytes) - usually it'll be a valid UTF-8 string, but it depends on the code executed
• code (u8) - serving as the logging level / application tag

Receipt Error

Each transaction might fail. In that case, the success field of the matching Receipt will be set to false
and an error will be returned.

• OOG (short for Out-Of-Gas)
• TemplateNotFound(TemplateAddress) -
  When trying to spawn an app of non-existing template. In general, such case should never happen unless there is a bug
  in the validation phase (normally, such transactions should have been discarded from the mempool earlier)
• AppNotFound(Address)
  When trying to execute an app transaction but app is not found.
  As explained above - proper validation should avoid that from happening from the first place
• CompilationFailed
  • app (Address) - The address
  • template (TemplateAddress`)
  • message (String)

@avive we need to discuss whether deploy template should compile the code or suffice with basic validation of the Wasm code
as it's done right now. (using the wasmparser crate). In that case the app field will become optional.

• InstantiationFailed - Didn't manage to create an Instance to run
  • app_addr (Address)
  • template_addr (TemplateAddress`)
  • message (String)
    There is one easy way I can think of right now and it's calling a function with wrong parameters. Violation of the function signature is something I think we can avoid since we always pass data to a function via the CallData mechanism.

Nonetheless, we still need to support that error in our API.

• FuncNotFound - Invalid input, function has not been found

  • app_addr (Address)
  • template_addr (TemplateAddress`)
  • func (String) - The name of the function not found

• FuncFailed - function has failed while running (e.g panicking)

  • app_addr (Address)
  • template_addr (TemplateAddress`)
  • func (String) - The name of the failing function
  • msg (String) - An error message that we managed to grab when failing

API Versioning

If we can add the new features and make the changes in a fully backward compatible way then we can have these changes in v0.1.2. Otherwise, these changes will be in API v0.2.0.

---

TBD / WIP

We need to consider additional API capabilities (which might be non-backward compatible):

1. Multisig transactions support.
2. Fully support account abstraction.

[avive]

Modeling transaction receipt based on provided meta-data

[avive]

@avive we need to discuss whether deploy template should compile the code or suffice with basic validation of the Wasm code as it's done right now. (using the wasmparser crate). In that case the app field will become optional.

I think it must accept wasm compiled code and perform the validation. Compilation and local validation pre deploy should be part of the dev workflow done off-chain. There are many issues with having a transaction run a compiler and I don't think it is needed. @lrettig - thoughts on this?

[avive]

@lrettig @YaronWittenstein please have a look at TransactionReceipt - I'm trying to add the svm related features in a backward compatible way and I think it is feasible. I haven't added all the data Yaron listed above - we should go over this together.

[lrettig]

We also need to add status to blocks

I think it must accept wasm compiled code and perform the validation. Compilation and local validation pre deploy should be part of the dev workflow done off-chain. There are many issues with having a transaction run a compiler and I don't think it is needed. @lrettig - thoughts on this?

It's a good question but I don't think this is the right place to discuss it as it's not directly relevant to the API design. Can we move it to the research forum?

[lrettig]

GetStateRoot
...
state - the State (the Global State root hash) associated with the layer_num

Please call this stateRoot to avoid confusion. It's also important to note in the design that no state root should be available for a given layer until that layer has been confirmed by Tortoise, and that it's subject to change later due to self-healing.

Namespaces

This design doesn't seem to be fleshed out. What are the endpoints here, what do they accept and what do they return?

Get Template
Name (String)

Does a template have a name? Must it? Feels unnecessary. The address it's deployed to should be its only, unique identifier. @YaronWittenstein ?

Author (Address)

Let's call this deployer

DeployedAt Layer (layer_num)
The layer_num at which the Template has been deployed at

Why/how is this useful? Feels relatively arbitrary. As of a given point in time, the template either exists, or it doesn't. Not sure it matters when it was deployed.

Schema

Is there a more complete spec for what this will look like? What information does it contain, exactly?

GetAllTemplates
returnCode - optional. we want to return a code for each returned Template (defaults to false)

What is this and how is it used?

App

I'm worried that this name will confuse people. We can use the term "App" internally if it makes sense to us, but I'd prefer to use a more familiar term in the API, such as "contract", "smartContract", "script", etc.

Layer (layer_num) - This parameter will be optional. It defaults to the most recent layer.

Note that only archive nodes will be able to accept a layer param. Non-archive nodes only contain current state, not past state for previous layers.

Name - String

As above - do we really need this? Feels unnecessary.

Spawner - The Address of the Account that spawned the App

I prefer deployer

Template (Address)
The Address of the Template that the App has been spawned from

Add: "...if any." (not all smart contracts are deployed from a template, right?)

State - The State associated with processed layer

Is this a state root? Is this necessary?

SpawnedAt Layer (layer_num)

As above, is this necessary? How/why is it used?

value

What is this and how is it used?

Receipts

SVM executes 3 types of transactions: deploy template / spawn app / execute app.

I thought SVM will also be responsible for simple coin tx, no?

Deploy Template Receipt

Do we allow these transactions to run arbitrary logic/make arbitrary state changes, or not?

This should probably also include the destination address of a deployed template (if any), and txid, payor/source

Spawn App Receipt

Can we use a more generic name here, too? E.g., "deploy contract"

Shouldn't this also include the address of the template (if any) and the address of the newly-deployed contract (if any)? Also, txid, payor/source

return_data (bytes) - The return data is the spawn app ctor execution output

I'm not sure we will allow the constructor to have any return data, @YaronWittenstein ?

Each log entry contains:
msg (bytes) - usually it'll be a valid UTF-8 string, but it depends on the code executed
code (u8) - serving as the logging level / application tag

Ethereum logs include a "name" field, in addition to the msg. Not sure if this is what you meant by "application tag", but this is very helpful.

[avive]

@lrettig - I think it will be helpful if you look at the proposed grpc changes - they are a bit more coherent - here what we have is mostly an initial survey of everything we need to consider, but the api proposed changes are refactored differently mainly to support backward compatibility and reduce the number of types that an api user needs to comprehend to use it.

[avive]

We also need to add status to blocks

I think it must accept wasm compiled code and perform the validation. Compilation and local validation pre deploy should be part of the dev workflow done off-chain. There are many issues with having a transaction run a compiler and I don't think it is needed. @lrettig - thoughts on this?

It's a good question but I don't think this is the right place to discuss it as it's not directly relevant to the API design. Can we move it to the research forum?

Sure but for now we want to progress with the safe option of needing accepting wasm compiled code.

[avive]

Note that only archive nodes will be able to accept a layer param. Non-archive nodes only contain current state, not past state for previous layers.

That's not the design pattern we talked about for state in the various smipps. We have discussed that full nodes keep state for n layers in the past and archive nodes since genesis. So the api should support this and node will return data if it has it which is based on the layer number, the current layer and the node type.

[avive]

Please call this stateRoot to avoid confusion. It's also important to note in the design that no state root should be available for a given layer until that layer has been confirmed by Tortoise, and that it's subject to change later due to self-healing.

Good feedback. Make sense. Check recent commit for implementation of this.

[avive]

Added name to log entries. So I think we are ready to do a proper design review on the proposed api changes in the 0.3 branch. Incorporated most feedback that required changes. Some of the feedback issues are already addressed in the current api design.

[avive]

We have a draft proposal for api changes that implement most requirements here in this branch: https://github.com/spacemeshos/api/tree/sm0.3-update/proto/spacemesh/v1
Next step is to design review it together.

[avive]

2 PRs to resolve this issue are waiting for review: #156 #149
@moshababo @lrettig