Skip to content
forked from RubixML/Server

A standalone inference server for trained Rubix ML estimators.

License

Notifications You must be signed in to change notification settings

Boorinio/Server

 
 

Repository files navigation

Rubix Server

Rubix Server is a library for bringing your trained Rubix ML models into production. Inference servers are stand-alone services that run on your private or public network and wrap your trained estimator in an API that can be queried locally or over the network in real-time using standard protocols. In addition, the library provides async-compatible client implementations for making queries to the server from your PHP applications.

  • Optimized for low latency predictions
  • Scalable horizontally by adding more instances
  • Monitoring with real-time analytics dashboard
  • Robust to common threats and failure modes

Installation

Install Rubix Server using Composer:

$ composer require rubix/server

Docker Image

A Docker Image is available for a quick start or deployment.

Requirements

  • PHP 7.4 or above

Documentation

The latest documentation can be found below.

Table of Contents


Servers

Rubix model servers are stand-alone processes that wrap an estimator in an API that can be queried over a network connection. Since servers implement their own networking stack, they can be run directly from the PHP command line interface (CLI) without the need for an intermediary server such as Nginx or Apache.

To boot up a server, pass a trained estimator instance to the serve() method:

public function serve(Estimator $estimator) : void
use Rubix\Server\HTTPServer;
use Rubix\ML\Classifiers\KNearestNeighbors;

$server = new HTTPServer('127.0.0.1', 8000);

$estimator = new KNearestNeighbors(5);

// Import a dataset

$estimator->train($dataset);

$server->serve($estimator);

Or, you can load a previously trained estimator from storage and serve it like in the example below.

use Rubix\ML\PersistentModel;
use Rubix\ML\Persisters\Filesystem;

$estimator = PersistentModel::load(new Filesystem('example.model'));

$server->serve($estimator);

Note: The server will stay running until the process is terminated. It is a good practice to use a process monitor such as Supervisor to start and autorestart the server in case of a failure.

Shutting Down The Server

To gracefully shut down the server, send a quit signal (SIGQUIT) to the process. To shut down immediately, without waiting for current connections to close, you can either send a terminate (SIGTERM) or interrupt (SIGINT) signal.

Note: Signal handling does not work in Windows environments.

For example, to shut down gracefully, first identify the server's process ID (PID) and then send the QUIT signal to it.

$ kill -s QUIT 1234

Verbose Interface

Servers that implement the Verbose interface accept any PSR-3 compatible logger instance and begin logging critical information such as errors and start/stop events. To set a logger pass the PSR-3 logger instance to the setLogger() method on the server instance.

use Rubix\Server\Loggers\File;

$server->setLogger(new File('example.log'));

HTTP Server

A JSON over HTTP server exposing Representational State Transfer (REST) and GraphQL APIs. The HTTP Server operates using ubiquitous standards making it compatible with a wide range of systems. In addition, it provides its own web-based user interface for real-time server monitoring.

Interfaces: Server, Verbose

Parameters

# Param Default Type Description
1 host '127.0.0.1' string The host address to bind the server to. Use '0.0.0.0' to bind to all interfaces.
2 port 8000 int The network port to run the HTTP services on.
3 cert null string The path to the certificate used to authenticate and encrypt the HTTP channel.
4 middlewares [] array The stack of server middleware to run on each request/response.
5 max concurrent requests 10 int The maximum number of requests that can be handled concurrently.
6 static assets cache InMemoryCache Cache The cache used to serve static asset requests.
7 sse reconnect buffer 50 int The maximum number of events to store in the server-sent events (SSE) reconnect buffer.

PHP INI Configuration

Name Default Description
memory_limit 128M The maximum amount of memory the server is allowed to consume.
post_max_size 8M The maximum size of a request body the server can buffer.

Example

use Rubix\Server\HTTPServer;
use Rubix\Server\HTTP\Middleware\Server\AccessLogGenerator;
use Rubix\Server\Loggers\File;
use Rubix\Server\HTTP\Middleware\Server\BasicAuthenticator;
use Rubix\Server\Services\Caches\InMemoryCache;

$server = new HTTPServer('127.0.0.1', 443, '/cert.pem', [
	new AccessLogGenerator(new File('access.log')),
	new BasicAuthenticator([
		'morgan' => 'secret',
		'taylor' => 'secret',
	]),
], 50, new InMemoryCache(86400), 100);

Routes

The HTTP server exposes the following resources and their methods.

Method URI Description
GET /ui The web user interface.
GET /ui/dashboard The server dashboard interface.
GET /model Return the properties of the model.
POST /model/predictions Make a set of predictions on a dataset.
POST /model/probabilities Return the joint probabilities of each sample in a dataset.
POST /model/anomaly-scores Return the anomaly scores of each sample in a dataset.
GET /server Return the properties of the server.
GET /dashboard/events Subscribe to the dashboard events stream.
POST /graphql Query the server using GraphQL.

Server Analytics

The HTTP server provides its own high-level user interface (UI) to the GraphQL API it exposes under the hood offering features such as server monitoring and traffic visualization. To access the web interface, navigate to http://hostname:port/ui (or https://hostname:port/ui if using a secure socket connection) using your favorite modern web browser.

The example below is a screen capture of the server dashboard in dark mode.

Server Web UI Screenshot

References

  • R. Fielding et al. (2014). Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content.

Server Middleware

HTTP middleware are processors of the incoming HTTP requests and outgoing responses produced by the request handler (or Controller). They allow the user to hook into the HTTP request/response cycle by inserting additional logic into the pipeline.

Access Log Generator

Generates an HTTP access log using a format similar to the Apache log format.

Parameters

# Param Default Type Description
1 logger LoggerInterface A PSR-3 logger instance.

Example

use Rubix\Server\HTTP\Middleware\Server\AccessLog;
use Rubix\Server\Loggers\File;

$middleware = new AccessLog(new File('access.log'));
[2020-11-04 23:10:57] INFO: 127.0.0.1 "POST /predictions HTTP/1.1" 200 140 - "Rubix ML REST Client/0.2.3"
[2020-11-04 23:11:54] INFO: 127.0.0.1 "POST /predictions/sample HTTP/1.1" 200 96 - "Rubix ML REST Client/0.2.3"

Basic Authenticator

An implementation of HTTP Basic Auth as described in RFC7617.

Note: This authorization strategy is only secure to man-in-the-middle attacks over HTTPS.

Parameters

# Param Default Type Description
1 passwords array An associative map from usernames to their passwords.
2 realm 'auth' string The unique name given to the scope of permissions required for this server.

Example

use Rubix\Server\HTTP\Middleware\Server\BasicAuthenticator;

$middleware = new BasicAuthenticator([
	'morgan' => 'secret',
	'taylor' => 'secret',
], 'ml models');

Shared Token Authenticator

Authenticates incoming requests using a shared key that is kept secret between the client and server. It uses the Authorization header with the Bearer prefix to indicate the shared key.

Note: This authorization strategy is only secure to man-in-the-middle attacks over HTTPS.

Parameters

# Param Default Type Description
1 tokens array The shared secret keys (bearer tokens) used to authorize requests.
2 realm 'auth' string The unique name given to the scope of permissions required for this server.

Example

use Rubix\Server\HTTP\Middleware\Server\SharedTokenAuthenticator;

$middleware = new SharedTokenAuthenticator([
	'secret', 'another-secret',
], 'ml models');

Trusted Clients

A whitelist of clients that can access the server - all other connections will be dropped.

Parameters

# Param Default Type Description
1 ips ['127.0.0.1'] array An array of trusted client ip addresses.

Example

use Rubix\Server\HTTP\Middleware\Server\TrustedClients;

$middleware = new TrustedClients([
	'127.0.0.1', '192.168.4.1', '45.63.67.15',
]);

Clients

Clients allow you to communicate directly with a model server using a friendly object-oriented interface inside your PHP applications. Under the hood, clients handle all the networking communication and content negotiation for you so you can write programs as if the model was directly accessible in your applications.

Return the predictions from the model:

public predict(Dataset $dataset) : array
use Rubix\Server\RESTClient;

$client = new RESTClient('127.0.0.1', 8080);

// Import a dataset

$predictions = $client->predict($dataset);

Calculate the joint probabilities of each sample in a dataset:

public proba(Dataset $dataset) : array

Calculate the anomaly scores of each sample in a dataset:

public score(Dataset $dataset) : array

Async Clients

Clients that implement the Async Client interface have asynchronous versions of all the standard client methods. All asynchronous methods return a Promises/A+ object that resolves to the return value of the response. Promises allow you to perform other work while the request is processing or to execute multiple requests in parallel. Calling the wait() method on the promise will block until the promise is resolved and return the value.

public predictAsync(Dataset $dataset) : Promise
$promise = $client->predictAsync($dataset);

// Do something else

$predictions = $promise->wait();

Return a promise for the probabilities predicted by the model:

public probaAsync(Dataset $dataset) : Promise

Return a promise for the anomaly scores predicted by the model:

public scoreAsync(Dataset $dataset) : Promise

REST Client

The REST Client communicates with the HTTP Server through the JSON REST API it exposes.

Interfaces: Client, AsyncClient

Parameters

# Param Default Type Description
1 host '127.0.0.1' string The IP address or hostname of the server.
2 port 8000 int The network port that the HTTP server is running on.
3 secure false bool Should we use an encrypted HTTP channel (HTTPS)?
4 middlewares array The stack of client middleware to run on each request/response.
5 timeout float The number of seconds to wait before giving up on the request.
6 verify certificate true bool Should we try to verify the server's TLS certificate?

Example

use Rubix\Server\RESTClient;
use Rubix\Server\HTTP\Middleware\Client\BasicAuthenticator;
use Rubix\Server\HTTP\Middleware\Client\CompressRequestBody;
use Rubix\Server\HTTP\Middleware\Client\BackoffAndRetry;
use Rubix\Server\HTTP\Encoders\Gzip;

$client = new RESTClient('127.0.0.1', 443, true, [
	new BasicAuthenticator('user', 'password'),
	new CompressRequestBody(new Gzip(1)),
	new BackoffAndRetry(),
], 0.0, true);

Client Middleware

Similarly to Server middleware, client middlewares are functions that hook into the request/response cycle but from the client end. Some of the server middlewares have accompanying client middleware such as Basic Authenticator and Shared Token Authenticator.

Backoff and Retry

The Backoff and Retry middleware handles Too Many Requests (429) and Service Unavailable (503) responses by retrying the request after waiting for a period of time to avoid overloading the server even further. An acceptable backoff period is gradually achieved by multiplicatively increasing the delay between retries.

Parameters

# Param Default Type Description
1 max retries 3 int The maximum number of times to retry the request before giving up.
2 initial delay 0.5 float The number of seconds to delay between retries before exponential backoff is applied.

Example

use Rubix\Server\HTTP\Middleware\Client\BackoffAndRetry;

$middleware = new BackoffAndRetry(5, 0.5);

Basic Authenticator (Client Side)

Adds the necessary authorization headers to the request using the Basic scheme.

Parameters

# Param Default Type Description
1 username string The user's name.
2 password string The user's password.

Example

use Rubix\Server\HTTP\Middleware\Client\BasicAuthenticator;

$middleware = new BasicAuthenticator('morgan', 'secret');

Compress Request Body

Apply the Gzip compression algorithm to the request body.

Parameters

# Param Default Type Description
1 level 1 int The compression level between 0 and 9 with 0 meaning no compression.
2 threshold 65535 int The minimum size of the request body in bytes in order to be compressed.

Example

use Rubix\Server\HTTP\Middleware\Client\CompressRequestBody;

$middleware = new CompressRequestBody(5, 65535);

Shared Token Authenticator (Client Side)

Adds the necessary authorization headers to the request using the Bearer scheme.

Parameters

# Param Default Type Description
1 token string The shared token to authenticate the request.

Example

use Rubix\Server\HTTP\Middleware\Client\SharedtokenAuthenticator;

$middleware = new SharedTokenAuthenticator('secret');

Loggers

PSR-3 compatible loggers for capturing important server events.

File

A simple append-only file logger.

Parameters

# Name Default Type Description
1 path string The path to the append-only log file. A new file will be created if it doesn't exist yet.
2 channel '' string The channel name that appears on each line.
3 timestampFormat 'Y-m-d H:i:s' string The format of the timestamp.

Example

use Rubix\Server\Loggers\File;

$logger = new File('server.log', 'example', 'Y-m-d H:i:s');

FAQs

Here you will find answers to the most frequently asked questions.

How do I run the server?

All model servers are designed to be run from the PHP command line interface (CLI). Model servers are long-running asynchronous processes that handle concurrent requests and implement their own networking stack avoiding the need for a third-party web server such as Nginx or Apache.

To run the server, you can execute your script containing the server code by entering the following on the command line.

$ php server.php

Can I run the model server on the same host as a regular web server?

Yes, model server are designed to coexist with other web servers (including other model servers) seamlessly. Just make sure that each server runs on its own unique port.

How do I scale inference throughput?

Since model servers are inference-only (i.e. they only support queries), they scale horizontally by adding more instances behind a load balancer such as Nginx.

Do servers support compression?

Yes, the HTTP Server supports both Gzip and Deflate compression schemes applied to the request bodies and to the response bodies of requests for static assets.

License

The code is licensed MIT and the documentation is licensed CC BY-NC 4.0.

About

A standalone inference server for trained Rubix ML estimators.

Resources

License

Security policy

Stars

Watchers

Forks

Packages

No packages published

Languages

  • PHP 75.6%
  • Vue 18.1%
  • JavaScript 2.4%
  • TypeScript 2.3%
  • Other 1.6%