This document walks through the basics of programming in Oak.
- Writing an Oak Node
- Running an Oak Application
- Using an Oak Application from a client
- gRPC Request Processing Path
- Nodes, Channels and Handles
- Persistent Storage
- Using External gRPC Services
- Testing
Oak Applications are built from a collection of inter-connected Oak Nodes, so the first step is to understand how to build a single Oak Node.
Writing software for the Oak system involves a certain amount of boilerplate, so
this section will cover the hows – and whys – of this, using code
taken from the various examples. If you're impatient, you could
start by copying the hello_world
example in particular, and just try modifying
things from there.
An Oak Node needs to provide a single main entrypoint, which is the point at which Node execution begins. However, Node authors don't have to implement this function themselves; for a Node which receives messages (a combination of bytes and handles) that can be decoded into a Rust type, there are helper functions in the Oak SDK that make this easier.
To use these helpers, an Oak Node should be a struct
of some kind to represent
the internal state of the Node itself (which may be empty), implement the
oak::Node
trait
for it, then define an
entrypoint
so the Oak SDK knows how to instantiate it:
oak::entrypoint!(oak_main => {
oak::logger::init_default();
Node {
training_set_size: 1000,
test_set_size: 1000,
config: None,
model: NaiveBayes::new(),
}
});
If the Node needs per-instance state, this Node struct
is an ideal place to
store it. For example, the running average example has a Node
struct
with a
running sum and count of samples:
struct Node {
sum: u64,
count: u64,
}
Under the covers the
entrypoint!
macro implements an external function identified by
the name of the entrypoint for you, with the following default behaviour:
- Take the channel handle passed to the entrypoint and use it for gRPC input.
- Take the newly constructed instance of the Node
struct
returned from the argument to the macro. - Pass this Node
struct
and the channel handle to therun_event_loop()
function.
For Nodes that act as gRPC servers (the normal "front door" for an Oak
Application), the easiest way to use a gRPC service implementation is to wrap it
with the automatically generated Dispatcher
, as described in the next section.
oak::entrypoint!(oak_main => {
oak::logger::init_default();
FormatServiceDispatcher::new(Node)
}
Alternatively a Node can implement the
oak::grpc::ServerNode
trait (which provides an automatic implementation of the
Node
). The
machine learning example
demonstrates this.
Any Rust panic originating in an Oak Node must be caught before going through
the Wasm FFI boundary. If you use the entrypoint!
macro, this is done for you,
but a manually implemented Node should use the
catch_unwind
method from the Rust standard library:
#[no_mangle]
pub extern "C" fn frontend_oak_main(in_handle: u64) {
let _ = std::panic::catch_unwind(|| {
oak::set_panic_hook();
let node = FrontendNode::new();
let dispatcher = OakAbiTestServiceDispatcher::new(node);
oak::run_event_loop(dispatcher, in_handle);
});
}
The Oak SDK provides oak_utils::compile_protos
to autogenerate Rust code from
a gRPC service definition. Adding a
build.rs
file to the Node that uses this function results in a generated file
<service>_grpc.rs
appearing under the crate's build
OUT_DIR
(by default).
fn main() {
oak_utils::compile_protos(
&["../../proto/hello_world.proto"],
&["../../proto", "../../../../third_party"],
);
}
The autogenerated code includes three parts, described in more detail below:
- A server-side trait definition.
- A
Dispatcher
struct
that handles routing of gRPC method invocations. - A client-side
struct
to allow easy use of the gRPC service from an Oak Node.
The first part is a trait definition that includes a method for each of the methods in the gRPC service, taking the relevant (auto-generated) request and response types. The Oak Node implements the gRPC service by implementing this trait.
pub trait FormatService {
fn format(&mut self, req: super::rustfmt::FormatRequest) -> grpc::Result<super::rustfmt::FormatResponse>;
}
The second part of the autogenerated code includes a Dispatcher
struct which
maps a request (as a method name and encoded request) to an invocation of the
relevant method on the service trait. This Dispatcher
struct can then form the
entire implementation of the ServerNode::invoke()
method described in the
previous section.
Taken altogether, these two parts cover all of the boilerplate needed to have a Node act as a gRPC server:
- The main
oak_main
entrypoint is auto-generated, and invokesoak::grpc::event_loop
with aDispatcher
. - This
Dispatcher
is created by wrapping a Nodestruct
that implements the gRPC generated service trait. - The
Dispatcher
implementsoak::grpc::ServerNode
so theevent_loop()
method can call into the relevant per-service method of the Node.
Finally, the third part of the autogenerated code includes a stub implementation of the client side for the gRPC service. If a Node offers a gRPC service to other Nodes in the same Application, they can use this client stub code to get simple access to the service.
In order to run the Oak Application, each of the Nodes that comprise the Application must first be compiled into one or more WebAssembly modules, and these compiled WebAssembly modules are then assembled into an overall Application Configuration File.
The Application Configuration also includes the port that the Oak Server should use for its gRPC service to appear on; the resulting (host:port) service endpoint is connected to by any clients of the Application.
Each of these steps is described in the following sections.
In order to load an Oak Application into the Oak Server its configuration must be serialized into a binary file. The Application first needs to specify a template configuration file:
node_configs {
name: "app"
wasm_config {
module_bytes: "<bytes>"
}
}
node_configs {
name: "log"
log_config {}
}
grpc_port: 8080
initial_node_config_name: "app"
initial_entrypoint_name: "oak_main"
The module_bytes: "<bytes>"
means that this value will be filled with
WebAssembly module bytes after serialization using the
Application Configuration Serializer,
as follows:
./bazel-bin/oak/common/app_config_serializer \
--textproto=examples/hello_world/config/config.textproto \
--modules=app://target/wasm32-unknown-unknown/release/hello_world.wasm \
--output_file=config.bin"
Here:
- The
--textproto
option gives the location of the template configuration file described above. - The
--modules
option gives a comma-separated list of name:filename pairs, which provides the WebAssembly module bytes for all of the namedwasm_config
entries in the configuration. - The
--output_file
gives the output location of the assembled Application configuration.
All these steps are implemented as a part of the
./scripts/build_example -e hello_world
script.
The Oak Application is then loaded using the Oak Runner:
./scripts/run_server -a "${PWD}/config.bin"
The Oak Runner will launch an Oak Runtime, and this Runtime will check the provided Wasm module(s) and application configuration. Assuming everything is correct (e.g. the Nodes all have a main entrypoint and only expect to link to the Oak host functions), the Oak Runtime opens up the gRPC port specified by the Application Configuration. This port is then used by clients to connect to the Oak Application.
A client that is outside of the Oak ecosystem can use an Oak Application by interacting with it as a gRPC service, using the endpoint (host:port) from the previous section (which would typically be published by the ISV providing the Oak Application).
The client connects to the gRPC service, and sends (Application-specific) gRPC requests to it, over a channel that has end-to-end encryption into the Runtime instance:
// Connect to the Oak Application.
auto stub = HelloWorld::NewStub(oak::ApplicationClient::CreateTlsChannel(address, ca_cert));
Because the Oak Application is available as a gRPC service, clients written in any language that supports gRPC can use the service. For example in Go:
// Connect to the Oak Application.
creds, err := credentials.NewClientTLSFromFile(*caCert, "")
if err != nil {
glog.Exitf("Failed to set up TLS client credentials from %q: %v", *caCert, err)
}
conn, err := grpc.Dial(*address, grpc.WithTransportCredentials(creds))
if err != nil {
glog.Exitf("Failed to dial Oak Application at %v: %v", *address, err)
}
defer conn.Close()
client := translator_pb.NewTranslatorClient(conn)
At this point, the client code can interact with the Node code via gRPC. A typical sequence for this (using the various helpers described in previous sections) would be as follows:
- The Node code (Wasm code running in a Wasm interpreter, running in the Oak
Runtime) is blocked inside a call to the
oak.wait_on_channels()
host function from theoak::grpc::event_loop
helper function.event_loop()
was invoked directly from the auto-generatedoak_main()
exported function.
- The client C++ code builds a gRPC request and sends it to the Oak Runtime.
- This connection is end-to-end encrypted using TLS.
- The Oak Runtime receives the message and encapsulates it in a
GrpcRequest
wrapper message. - The Oak Runtime serializes the
GrpcRequest
and writes it to the gRPC-in channel for the Node. It also creates a new channel for any responses, and passes a handle for this response channel alongside the request. - This unblocks the Node code, and
oak::grpc::event_loop
reads and deserializes the incoming gRPC request. It then calls theDispatcher
'sinvoke()
method with the method name and (serialized) gRPC request. - The auto-generated
Dispatcher
invokes the relevant method on theNode
. - The (user-written) code in this method does its work, and returns a response.
- The auto-generated
Dispatcher
struct encapsulates the response into aGrpcResponse
wrapper message, and serializes into the response channel. - The Oak Runtime reads this message from the response channel, deserializes it and sends the inner response back to the client.
- The client C++ code receives the response.
So far, we've only discussed writing a single Node for an Oak Application. This
Node communicates with the outside world via a single channel, and the handle
for the read half of this channel is acquired at start-of-day as the parameter
to the oak_main()
entrypoint. The other half of this single channel is a gRPC
pseudo-Node, which passes on requests from external clients (and which is
automatically created by the Oak Runtime at Application start-of-day).
More sophisticated Applications are normally built from multiple interacting Nodes, for several reasons:
- Dividing software into well-defined interacting components is a normal way to reduce the overall complexity of software design.
- Software that handles sensitive data, or which has additional privileges, often separates out the parts that deal with this (the "principle of least privilege"), to reduce the blast radius if something goes wrong.
- Information flow analysis can be more precise and fine-grained if components are smaller and the interactions between them are constrained.
The first step in building a multi-Node Application is to write the code for all
of the Nodes; the ApplicationConfiguration
needs to include the configuration
and code for any Node that might get run as part of the Application. New Node
types cannot be added after the application starts; any Node that the
Application might need has to be included in the original configuration.
As before, each Node must include a main entrypoint with signature
fn(u64) -> ()
, but for an internal Node it's entirely up to the ISV as to what
channel handle gets passed to this entrypoint, and as to what messages are sent
down that channel. The application may choose to use protobuf-encoded messages
(as gRPC does) for its internal communications, or something else entirely (e.g.
the serde crate).
Regardless of how the Application communicates with the new Node, the typical pattern for the existing Node is to:
- Create a new channel with the
channel_create
host function, receiving local handles for both halves of the channel. - Create a new Node instance with the
node_create
host function, passing in the handle for the read half of the new channel. - Afterwards, close the local handle for the read half, as it is no longer needed, and use the local handle for the write half to send messages to the new Node instance.
For example, the example Chat application creates a Node for each chat room and saves off the write handle that will be used to send messages to the room:
let (wh, rh) = oak::channel_create().unwrap();
oak::node_create("app", "backend_oak_main", rh).expect("could not create node");
oak::channel_close(rh.handle).expect("could not close channel");
Room {
sender: oak::io::Sender::new(wh),
admin_token,
}
The same code (identified by "room-config"
) will be run for each per-room
Node, but each instance will have its own Web Assembly linear memory (≈heap) and
stack.
The node_create()
call triggers the Oak Runtime to invoke the main entrypoint
for the new Node (as specified in the Application configuration), passing in the
handle value for the channel read half that was provided as a parameter to
node_create()
. Note that the actual handle value passed into the main
entrypoint will (almost certainly) be different; internally, the Runtime
translates the creator Node's handle value to a reference to the underlying
channel object, then assigns a new numeric value for the created Node to use to
refer to the underlying channel.
Once a new Node has started, the existing Node can communicate with the new Node
by sending messages over the channel via channel_write
. Of course, the new
Node only has a handle to the read half of a channel, and so only has a way of
receiving.
To cope with this, it's normal for the inbound messages to be accompanied by a handle for the write half of a different channel, which is then used for responses – so the new Node has a way of sending externally, as well as receiving.
TODO: describe use of storage
TODO: describe use of gRPC client pseudo-Node to connect to external gRPC services.
"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth
Regardless of how the code for an Oak Application is produced, it's always a good idea to write tests. The oak_tests crate allows Node gRPC service methods to be tested with the Oak SDK framework via the Oak Runtime:
// Test invoking the SayHello Node service method via the Oak runtime.
#[test]
fn test_say_hello() {
simple_logger::init_by_env();
let (runtime, entry_handle) = oak_tests::run_single_module_default(MODULE_CONFIG_NAME)
.expect("Unable to configure runtime with test wasm!");
let req = HelloRequest {
greeting: "world".into(),
};
let result: grpc::Result<HelloResponse> = oak_tests::grpc_request(
&runtime,
entry_handle,
"/oak.examples.hello_world.HelloWorld/SayHello",
&req,
);
assert_matches!(result, Ok(_));
assert_eq!("HELLO world!", result.unwrap().reply);
runtime.stop_runtime();
}
This has a little bit of boilerplate to explain:
- The
oak_tests
crate provides arun_single_module_default
method that is designed for use with single-Node Applications. It assumes that the Node has a main entrypoint calledoak_main
, which is runs in a separate thread. (This entrypoint would typically be set up by theoak::entrypoint!
macro described above). - The injection of the gRPC request has to specify the method name (in the call
to
oak_tests::grpc_request()
). - The per-Node thread needs to be stopped at the end of the test
(
oak_runtime::stop
).
However, this extra complication does allow the Node to be tested in a way that is closer to real execution, and (more importantly) allows testing of a Node that makes use of the functionality of the Oak Runtime.
It's also possible to test an Oak Application that's built from multiple Nodes,
using oak_runtime::application_configuration
to create an application
configuration and then oak_runtime::Runtime::configure_and_run(configuration)
to configure and run the Runtime.
let configuration = oak_runtime::application_configuration(
build_wasm().expect("failed to build wasm modules"),
LOG_CONFIG_NAME,
FRONTEND_CONFIG_NAME,
FRONTEND_ENTRYPOINT_NAME,
);
let (runtime, entry_channel) =
oak_runtime::configure_and_run(configuration, oak_runtime::RuntimeConfiguration::default())