How to's

Zipnish how to's

---

Prerequisites

Zipnish runs on top of Varnish, currently - the Varnish supported version is 4. Data storage is based on MySQL, therefore a running instance of MySQL server will be required.

Following pip packages are required

• simplemysql
• mysql-python
• mysql-connector-python
• crochet (twisted included)
• flask
• flask-sqlalchemy
• flask-script

Required Configuration

####MySQL configuration A MySQL database is required for both the LogReader and UI components. Its main purpose is to store specific data fetched from Varnishlog. Install an instance of a MySQL server and create a user along with a database for it.

Example:

CREATE USER 'microservice'@'localhost' IDENTIFIED BY 'PassWord';
CREATE DATABASE microservice;
GRANT ALL PRIVILEGES ON microservice.*  TO 'microservice'@'localhost';

LogReader configuration.

The LogReader component has a set of several required settings in order to be it fully functional. By default, these settings will be found either under:

Path: /etc/zipnish/zipnish.cfg;

A brief description for each of these settings is given in the config file.

####UI configuration.

Zipnish is bundled with an UI which will generate graphs displaying how each of the microservice requests have performed. A few settings are required here as well, the UI needs to point to the same MySQL database as the LogReader, as they will share the same data. UI configuration is found under:

Path: /etc/zipnish/ui.cfg;

How does Zipnish become aware of the existing microservices?

In order to have your Microservices integrated in the Zipnish environment, a few changes are required on both parties: Varnish and the Microservices.

Varnish changes:

• All your Microservice nodes need to be declared as backends in the vcl configuration.
• The vcl_recv() needs to return a pass unless you want to cache responses in Varnish

Example with caching disabled:

    vcl 4.0;
    
    backend DemoMicroservice {
        .host = "127.0.0.1";
        .port = "9999";
    }

    # disable caching - see further down for another example with caching enabled
    sub vcl_recv {
        if (req.url ~ "^/DemoService") {
           set req.backend_hint = DemoMicroservice;
           return (pass);
        }
    }

Example with caching enabled:

    vcl 4.0;
    
    backend DemoMicroservice {
        .host = "127.0.0.1";
        .port = "9999";
    }

    # disable caching - see further down for another example with caching enabled
    sub vcl_recv {
        if (req.url ~ "^/DemoService") {
            set req.backend_hint = DemoMicroservice;
        }
    }

    sub vcl_deliver {
        # add the response headers if this is a cache hit
        if (obj.hits > 0) {
            set resp.x-varnish-trace = req.http.x-varnish-trace || req.http x-varnish; 
            set resp.x-varnish-parent = req.http.x-varnish
        }
    }

Please note that enabled caching you have to take caching headers into consideration. The normal varnish rules for caching apply.

Microservice changes:

• On the application side, a couple of http headers are required to be added to the responses.
• Headers: X-Varnish-Trace, X-Varnish-Parent
• Since all requests go through Varnish, they will further be picked up by Varnishlog while LogReader fetches its own data from Varnishlog.
• The main purpose of these headers is to establish the hierarchy among requests.

An example on how to set these headers can be found under:

Path: /zipnish/log-reader/test/server.py;

A very naive web-server, used for testing, adds these headers while handling GET requests. It is of a similar manner that your application should add these headers as well.

Run the LogReader

When running from the source code, the LogReader is started by running the app.py script located under:

Example:

$ cd zipnish/log-reader
$ python app.py

Otherwise, if installed from its package, LogReader can be started as a service:

Example:

$ sudo service log-reader start

Run the UI

When running from the source code, the UI will run on your local machine on port 5000, unless otherwise specified. Before running it, make sure that its configuration points to the correct MySQL setup.

Example:

$ python manage.py runserver

If installed from its package, the UI is also available as a service:

Example:

$ sudo service zipnish-ui start

Zipnish tests

Currently there are only a few tests available that will validate log data parsing, storage and request timings. The test does not expose any config file, thus the configuration for the MySQL database needs to be done in the log_test.py file.

While running, a few requests will be issued against Varnish, Varnishlog will generate log data related to these requests. The LogReader will fetch the Varnishlog data, parse it and extract strictly data that it’s interesting for it self. Stored data and their timings will be compared against random timings which are known in before-hand.