samlare

Samlare (it means “collector/gatherer” in swedish) is a simple tool for collecting data from expvar endpoints and submitting it to graphite.

I also wrote a short blog post introducing samlare.

Installation

It’s available for Linux as a plain binary for linux or OS X (let me know if you need it for other platforms). Download it from the releases and put it in your path (and probably rename it, set permissions, etc).

It is also available as a docker image (thraxil/samlare on the docker hub):

$ docker pull thraxil/samlare
$ docker run -v /path/to/my/config.toml:/etc/samlare/config.toml thraxil/samlare

Configuration

The only commandline argument it takes is a path to a config file (it will default to looking for /etc/samlare/config.toml if you don’t specify a path):

$ samlare -config=/path/to/my/config.toml

That’s a TOML file with a fairly simple format.

A simple example is included (see sample-config.toml) and looks something like:

CarbonHost = "graphite.example.com"
CarbonPort = 2003
CheckInterval = 60000
Timeout = 3000

[[endpoint]]
URL = "http://localhost:14001/debug/vars"
Prefix = "apps.app1"

[[endpoint]]
URL = "http://localhost:14002/debug/vars"
Prefix = "apps.app2"
FailureMetric = "apps.app1.failure"

[[rename]]
From = "BuckHashSys"
To = "buck_hash_sys"

[[rename]]
From = "Sys"
To = "sys"

The top level fields that need to be set are:

• CarbonHost – hostname/ip of your carbon server
• CarbonPort – port for your carbon server
• CheckInterval – how many milliseconds between checks
• Timeout – max milliseconds to wait for a response from each endpoint

Then you have a stanza for each endpoint that you want to watch.

Each requires

• URL – the full URL of the endpoint
• Prefix – the graphite metric prefix to use

Optionall, it also supports:

• FailureMetric – if this is specified, it will submit a 1 to this metric if the fetch failed and a 0 otherwise. In other words, this can be a simple availability check. If this isn’t set, it will simply ignore failed fetches.
• IgnoreMetrics - a list of metric names (corresponding to keys in the expvar JSON) that will just be ignored and not passed along to graphite. Handy if there are metrics that you just don’t care about and would rather that they don’t take up space on your graphite server.

CheckInterval and Timeout can also be individually overridden for each endpoint as well.

Finally, you can tell it to rename metrics. Eg, expvar generally outputs camelCase variables, but perhaps you have standardized on all lower case and underscores for your graphite metrics. Just add as many [[rename]] blocks as you need (yeah, you have to explicitly do each. If you want to add support for general renaming policies, I’d take a patch). That can also be overridden on a per-endpoint basis.

Additional Features

Samlare will respond to a SIGHUP signal by reloading its config file. This is particularly handy if you use something like consul-template to generate your config file automatically.

Samlare logs to STDOUT in a nice JSON format.

Samlare adds a random 10% jitter to the CheckInterval for each metric. Ie, if it’s set to 60 seconds, the actual interval will be some random value between 60-66 seconds. This prevents thundering herd problems.