Probe It

Execute system commands or probe HTTP URLs and AWS S3 buckets to produce metrics in JSON or for Prometheus. Can be used on the command line or as a server.

Inspired by Prometheus Blackbox Exporter.

Usage
- Installation
Metrics
- Prometheus format
- Generic metrics
  - duration
Command probe
- Function commands
- System commands
  - System command metrics
  - commandExitCode
HTTP probe
S3 probe
- S3 probe metrics
- S3 probe parameters
Versioning policy

Usage

Visit http://probeit.herokuapp.com/?target=http://google.com&pretty=true for a sample probe of an HTTP endpoint.

The target query parameter indicates what to probe:

An HTTP(S) endpoint, e.g. https://google.com, to use the HTTP probe
An S3 bucket URL, e.g. s3://bucket_name, to use the S3 probe

The response will be a JSON object with:

A success boolean property indicating whether the probe was successful.
A metrics array of objects, describing various metrics collected from the probed target, such as duration of the probe, HTTP status code, number of S3 objects, etc.
A failures array which may indicate possible causes of the probe's failure.

{
  "failures": [],
  "metrics": [
    {
      "description": "How long the probe took to complete in seconds",
      "name": "duration",
      "tags": {},
      "type": "seconds",
      "value": 0.12
    },
    ...
  ],
  "success": true
}

Installation

Run it with Docker

docker run -p 3000:3000 alphahydrae/probeit

Or, run it with npx

npx probeit

Or, install and run it manually

npm install -g probeit
probeit

Then, try it

Visit http://localhost:3000?target=http://google.com&pretty=true

Metrics

The server's probes provide various metrics about their target, mostly numeric values such as counts, bytesizes, last modification dates, etc.

Each metric is a JSON object with the following format:

{
  "description": "What I am",
  "name": "myName",
  "tags": {
    "meta": "data"
  },
  "type": "seconds",
  "value": 1.2
}

The following metric types exist at this time:

boolean - true or false.
bytes - Integer greater than or equal to zero.
datetime - Date in ISO-8601 format.
number - Number with no associated unit of measurement.
quantity - Integer greater than or equal to zero, representing an amount of something.
seconds - Number greater than or equal to zero.

The value of a metric may be null if it cannot be determined (e.g. the httpCertificateExpiry metric can be null if the HTTP request is not made over TLS, or the s3LastObjectModificationDate metric can be null if the S3 bucket contains no objects).

Prometheus format

To get the metrics in Prometheus's text format, use the /metrics path:

http://probeit.herokuapp.com/metrics?target=http://google.com

Metric names are converted from the JSON's camel-case format to underscored format and the probe_ prefix is prepended (e.g. httpStatusCode becomes probe_http_status_code). Additionally, the metric's type is added as a suffix if it's bytes or seconds (e.g. duration becomes probe_duration_seconds).

Metric tags are added as labels.

All metrics will be provided as gauges by applying the following conversions by type:

boolean - true becomes 1 and false becomes 0.
bytes - No conversion, except for null which becomes -1.
datetime - Dates are provided in Unix time, i.e. the number of seconds that have elapsed since 00:00:00 UTC on Thursday, 1 January 1970. null becomes -1.
number - No conversion, except for null which becomes NaN.
quantity - No conversion, except for null which becomes -1.
seconds - No conversion, except for null which becomes -1.

Generic metrics

The following sub-headings document metrics that are provided by all probes.

`duration`

Type: seconds

How long the probe took to complete in seconds.

{
  "description": "...",
  "name": "duration",
  "tags": {},
  "type": "seconds",
  "value": 0.12
}

Command probe

The command probe is used when the target is an URI that starts with command:, e.g. command:foo. This will execute the command named foo and return its metrics.

Named commands must be pre-defined with the commands property of Probe It's configuration file (config.yml by default, or the file indicated by the -c, --config <FILE> command line option or the $PROBE_CONFIG environment variable). The property is an object in which keys are command names and values are the definitions of the commands to run:

commands:
  lsRoot:
    type: system
    command: ls
    args: [ -la, / ]
  unameAll
    type: system
    command: uname
    args: [ -a ]

The following command types are supported:

Arbitrary JavaScript functions to be executed by Probe It.
System commands to spawn in a new process.

Function commands

If your configuration file is a JavaScript file instead of a JSON or YAML file, you can also define a command to probe as an arbitrary JavaScript function:

exports.commands = {
  doStuff: {
    type: 'function',
    command: async function() {

      try {
        // Read metrics from a file
        const metrics = await fs.Promises.readFile('/metrics.json', 'utf8');
        return {
          metrics,
          failures: [],
          success: true
        };
      } catch (err) {
        // Describe possible failure
        return {
          failures: [
            cause: "invalidMetricsFile",
            description: "Could not read metrics file"
          ],
          metrics: [],
          success: false
        };
      }
    }
  }
};

The function may be synchronous or asynchronous, and must return an object with the following properties:

metrics - An array of objects describin the metrics produced by running the command. Each metrics object must be in the correct format.
failures - An array of objects describing reasons why the probe failed (it may be empty). Each failure object must have:
- A cause property which is a string code identifying the failure.
- A human-readable description property.
- An optional expected property indicating the expected value.
- An optional actual property indicating the actual value which differs from the expected one.
- Optional extra properties describing the failure.
success - true or false to indicate whether the probe succeeded.

System commands

System commands are commands that will be spawned in a new process on the machine on which Probe It is running. They can be defined with all configuration file formats (JavaScript, JSON or YAML), for example in JSON:

{
  "commands": {
    "lsRoot": {
      "type": "system",
      "command": "ls",
      "args": [ "-la", "/" ],
      "cwd": "/"
    }
  }
}

The following options describe a system command:

command - The executable to run.
args - An optional array of arguments to pass to the executable.
cwd - An optional working directory to run the executable in.

System command metrics

The following sub-headings document the metrics provided by a system command probe.

`commandExitCode`

Type: number

The exit code of the executed command. 0 indicates successful execution, while and non-zero code indicates some kind of failure.

{
  "description": "...",
  "name": "commandExitCode",
  "tags": {},
  "type": "number",
  "value": 0
}

HTTP probe

The HTTP probe is used when the target is an URL that starts with http:// or https://. By default, it will make a GET request to that URL, following any redirects, and provide various metrics about the HTTP response.

HTTP probe metrics

The following sub-headings document the metrics provided by the HTTP probe.

`httpCertificateExpiry`

Type: datetime

Expiration date of the SSL certificate (when the target starts with https://).

{
  "description": "...",
  "name": "httpCertificateExpiry",
  "tags": {},
  "type": "datetime",
  "value": "2018-05-01T00:00:00Z"
}

`httpContentLength`

Type: bytes

Length of the HTTP response in bytes.

{
  "description": "...",
  "name": "httpContentLength",
  "tags": {},
  "type": "bytes",
  "value": 2801239
}

`httpDuration`

Type: seconds, Repeated

Duration of the HTTP request(s) by phase, summed over all redirects, in seconds. Several instances of this metric will be provided, one for each request phase:

dnsLookup - How long it took to perform the DNS lookup.
tcpConnection - How long it took to open the TCP connection after the DNS lookup.
tlsHandshake - How long it took to perform the TLS handshake for an HTTPS probe, after the TCP connection was established. (This metric will be absent for URLs starting with http://.)
firstByte - How long it took to receive the first byte of the response after the TLS handshake (or TCP connection).
contentTransfer - How long it took to transfer the entire response entity once the first byte was received.

{
  "description": "...",
  "name": "httpDuration",
  "tags": {
    "phase": "tlsHandshake"
  },
  "type": "seconds",
  "value": 0.02
}

`httpRedirects`

Type: quantity

Number of times HTTP 301 or 302 redirects were followed.

{
  "description": "...",
  "name": "httpRedirects",
  "tags": {},
  "type": "quantity",
  "value": 2
}

`httpSecure`

Type: boolean

Indicates whether SSL/TLS was used for the final redirect.

{
  "description": "...",
  "name": "httpSecure",
  "tags": {},
  "type": "boolean",
  "value": true
}

`httpStatusCode`

Type: number

HTTP status code of the final response.

{
  "description": "...",
  "name": "httpStatusCode",
  "tags": {},
  "type": "number",
  "value": 404
}

`httpVersion`

Type: number

HTTP version of the final response.

{
  "description": "...",
  "name": "httpVersion",
  "tags": {},
  "type": "number",
  "value": 1.1
}

HTTP probe parameters

The following sub-headings document URL query parameters that can be provided to customize the behavior of the HTTP probe.

`allowUnauthorized`

Type: boolean, Default: false

Whether to consider an HTTP response with an invalid SSL certificate as a success.

?allowUnauthorized=true

`followRedirects`

Type: boolean, Default: true

Whether the probe will follow redirects (e.g. HTTP 301 Moved Permanently or HTTP 302 Found) to provide metrics about the final response, or whether it will simply provide metrics about the first response sent by the server.

?followRedirects=false

`header`

Type: key=value pair, Repeatable

HTTP header to add to the probe's request(s). This parameter can be repeated to set multiple headers.

// The value is "Authorization=Basic YWRtaW46Y2hhbmdlbWUh", URL-encoded
?header=Authorization%3DBasic%20YWRtaW46Y2hhbmdlbWUh

`method`

Type: string (GET, POST, PUT, etc.), Default: GET

The HTTP method to use for the request on the target URL. GET by default.

?method=POST

HTTP probe expectations

The following sub-headings document URL query parameters that can be provided to customize how the HTTP probe will determine if it was successful.

By default, it only expects that the final HTTP response will have a code in the 2xx or 3xx range.

`expectHttpRedirects`

Type: boolean or integer

For the probe to be considered successful with this parameter:

If true, at least 1 redirect must have been followed.
If false, no redirect must have been followed.
If an integer, exactly that number of redirects must have been followed.

?expectHttpRedirects=true
?expectHttpRedirects=2

Possible failures:

invalidHttpRedirectCount
missingHttpRedirect
unexpectedHttpRedirect

`expectHttpRedirectTo`

Type: URL

For the probe to be considered successful with this parameter:

At least 1 redirect must have been followed.
The protocol, host, port and path of the final HTTP request's URL must correspond to the specified URL. (Other parts of the URL, such as authentication, query string or hash, are ignored for the comparison.)

// The value is "http://example.com/path", URL-encoded
?expectHttpRedirectTo=http%3A%2F%2Fexample.com%2Fpath

Possible failures:

invalidHttpRedirectLocation

`expectHttpResponseBodyMatch`

Type: regular expression, Repeatable

For the probe to be considered successful with this parameter, the HTTP response body must match the regular expression.

This parameter can be repeated to check the presence of multiple patterns.

// The value is "Catch \d{2}", URL-encoded
?expectHttpResponseBodyMatch=Catch%20%5Cd%7B2%7D

Possible failures:

httpResponseBodyMismatch

`expectHttpResponseBodyMismatch`

Type: regular expression, Repeatable

For the probe to be considered successful with this parameter, the HTTP response body must not match the regular expression.

This parameter can be repeated to check the absence of multiple patterns.

// The value is "connection lost", URL-encoded
?expectHttpResponseBodyMismatch=connection+lost

Possible failures:

unexpectedHttpResponseBodyMatch

`expectHttpSecure`

Type: boolean

For the probe to be considered successful with this parameter:

If true, the final HTTP response must have been over TLS (e.g. https://).
If false, the final HTTP response must not have been over TLS (e.g. http://).

By default, either is considered successful.

Note that this does not affect the probe's behavior of failing if an SSL certificate is invalid. Use the allowUnauthorized parameter for that.

?expectHttpSecure=true

Possible failures:

insecureHttp
unexpectedlySecureHttp

`expectHttpStatusCode`

Type: number or HTTP status code class (e.g. 2xx), Repeatable, Default: [ "2xx", "3xx" ]

For the probe to be considered successful with this parameter, the final HTTP response's status code must be one of the expected codes, or fall within one of the expected classes (e.g. 204 falls within the 2xx class). Both individual codes and code classes may be provided.

?expectHttpStatusCode=204
?expectHttpStatusCode=200&expectHttpStatusCode=3xx

Possible failures:

invalidHttpStatusCode

`expectHttpVersion`

Type: number

For the probe to be considered successful with this parameter, the HTTP version of the final response must match the expected version.

?expectHttpVersion=1.1

Possible failures:

invalidHttpVersion

HTTP probe failures

The following sub-headings document the possible causes of failure that may be included in the HTTP probe's result.

`httpResponseBodyMismatch`

The body of the final HTTP response did not match some of the regular expressions provided with the expectHttpResponseBodyMatch parameter.

This failure will be repeated for each regular expression that did not match.

{
  "cause": "httpResponseBodyMismatch",
  "description": "...",
  "expected": "[a-z0-9]+"
}

`insecureHttp`

The final HTTP request was expected to be over TLS due to the expectHttpSecure parameter being set to true, but it was not.

{
  "cause": "insecureHttp",
  "description": "..."
}

`invalidHttpRedirectCount`

An integer was provided as the expected number of redirects with the expectHttpRedirects parameter, and the actual number of followed redirects until the final HTTP response did not match that expectation.

{
  "actual": 2,
  "cause": "invalidHttpRedirectCount",
  "description": "...",
  "expected": 1
}

`invalidHttpRedirectLocation`

The final redirection was expected to be made to the URL specified with the expectHttpRedirectTo parameter, but a different URL was provided by the server.

{
  "actual": "http://example.com/foo",
  "cause": "invalidHttpRedirectLocation",
  "description": "...",
  "expected": "http://example.com/bar"
}

`invalidHttpStatusCode`

The HTTP status code of the final response did not match any of the expected codes or code classes provided with the expectHttpStatusCode parameter (or set by default).

{
  "actual": 404,
  "cause": "invalidHttpStatusCode",
  "description": "...",
  "expected": [ "200", "3xx" ]
}

`invalidHttpVersion`

The HTTP version of the final response was not the expected version provided with the expectHttpVersion parameter.

{
  "actual": 2.0,
  "cause": "invalidHttpStatusCode",
  "description": "...",
  "expected": 1.1
}

`missingHttpRedirect`

The expectHttpRedirects parameter was set to true, but no redirect was issued by the server.

{
  "cause": "missingHttpRedirect",
  "description": "..."
}

`unexpectedHttpRedirect`

The expectHttpRedirects parameter was set to false, but the server issued one or more redirects.

{
  "actual": 3,
  "cause": "missingHttpRedirect",
  "description": "...",
  "expected": 0
}

`unexpectedHttpResponseBodyMatch`

The body of the final HTTP response matched some of the regular expressions provided with the expectHttpResponseBodyMismatch parameter.

This failure will be repeated for each regular expression that matched.

The failure object's actual property will contain the full regular expression match.

{
  "actual": "error: connection lost",
  "cause": "unexpectedHttpResponseBodyMatch",
  "description": "...",
  "expected": "error: .+"
}

`unexpectedlySecureHttp`

The final HTTP request was expected not to be over TLS due to the expectHttpSecure parameter being set to false, but it was.

{
  "cause": "unexpectedlySecureHttp",
  "description": "..."
}

S3 probe

The S3 probe is used when the target is an URL that starts with s3://. The format is as follows:

s3://[access_key_id:secret_access_key@]bucket_name[/prefix]

For example:

s3://my_bucket
s3://A28sdf8A:oa83ufozsr8b@secure_backups/daily

By default, the probe will list all the objects in the bucket and provide various metrics about these objects' sizes and last modification dates. It can also be configured to list object versions with the s3Versions parameter.

S3 probe metrics

The following sub-headings document the metrics provided by the S3 probe.

`s3FirstObjectModificationDate`

Type: datetime

The modification date of the earliest modified object.

{
  "description": "...",
  "name": "s3FirstObjectModificationDate",
  "tags": {},
  "type": "datetime",
  "value": "2018-05-01T00:00:00Z"
}

`s3FirstObjectSize`

Type: bytes

The size of the earliest modified object in bytes.

{
  "description": "...",
  "name": "s3FirstObjectSize",
  "tags": {},
  "type": "bytes",
  "value": 1850912
}

`s3FirstObjectVersionModificationDate`

Type: datetime

The modification date of the earliest modified object version.

Note: this metric will only be provided if the [s3Versions parameter] is set to true.

{
  "description": "...",
  "name": "s3FirstObjectVersionModificationDate",
  "tags": {},
  "type": "datetime",
  "value": "2018-05-01T00:00:00Z"
}

`s3FirstObjectVersionSize`

Type: bytes

The size of the earliest modified object version in bytes.

Note: this metric will only be provided if the [s3Versions parameter] is set to true.

{
  "description": "...",
  "name": "s3FirstObjectVersionSize",
  "tags": {},
  "type": "bytes",
  "value": 1850912
}

`s3LargestObjectSize`

Type: bytes

The size of the largest object in bytes.

{
  "description": "...",
  "name": "s3LargestObjectSize",
  "tags": {},
  "type": "bytes",
  "value": 18392047
}

`s3LastObjectModificationDate`

Type: datetime

The modification date of the most recently modified object.

{
  "description": "...",
  "name": "s3LastObjectModificationDate",
  "tags": {},
  "type": "datetime",
  "value": "2018-05-01T00:00:00Z"
}

`s3LastObjectSize`

Type: bytes

The size of the most recently modified object in bytes.

{
  "description": "...",
  "name": "s3LastObjectSize",
  "tags": {},
  "type": "bytes",
  "value": 1850912
}

`s3LastObjectVersionModificationDate`

Type: datetime

The modification date of the most recently modified object version.

Note: this metric will only be provided if the [s3Versions parameter] is set to true.

{
  "description": "...",
  "name": "s3LastObjectVersionModificationDate",
  "tags": {},
  "type": "datetime",
  "value": "2018-05-01T00:00:00Z"
}

`s3LastObjectVersionSize`

Type: bytes

The size of the most recently modified object version in bytes.

Note: this metric will only be provided if the [s3Versions parameter] is set to true.

{
  "description": "...",
  "name": "s3LastObjectVersionSize",
  "tags": {},
  "type": "bytes",
  "value": 1850912
}

`s3ObjectsCount`

Type: quantity

Number of objects.

{
  "description": "...",
  "name": "s3ObjectsCount",
  "tags": {},
  "type": "quantity",
  "value": 23
}

`s3ObjectsTotalSize`

Type: bytes

Total size of objects.

{
  "description": "...",
  "name": "s3ObjectsTotalSize",
  "tags": {},
  "type": "bytes",
  "value": 4027329041
}

`s3ObjectVersionsCount`

Type: quantity

Number of object versions.

Note: this metric will only be provided if the [s3Versions parameter] is set to true.

{
  "description": "...",
  "name": "s3ObjectVersionsCount",
  "tags": {},
  "type": "quantity",
  "value": 32
}

`s3SmallestObjectSize`

Type: bytes

The size of the smallest object in bytes.

{
  "description": "...",
  "name": "s3SmallestObjectSize",
  "tags": {},
  "type": "bytes",
  "value": 351
}

S3 probe parameters

The following sub-headings document URL query parameters that can be provided to customize the behavior of the S3 probe.

`s3ByPrefix`

Type: string, Repeatable

When provided N times, this parameter will cause each metric to be provided N + 1 times in the response:

Once for each prefix (N)
Once for no prefix (+ 1)

?s3ByPrefix=daily&s3ByPrefix=monthly

For example, if the parameter is provided twice with the values daily and monthly like in the example above, the s3ObjectsTotalSize metric will be present twice (as will all other metrics of the S3 probe). Those separate metric objects can be differentiated by their tags property:

{
  "description": "...",
  "name": "s3ObjectsTotalSize",
  "tags": {
    "prefix": ""
  },
  "type": "bytes",
  "value": 4027329041
},
{
  "description": "...",
  "name": "s3ObjectsTotalSize",
  "tags": {
    "prefix": "daily"
  },
  "type": "bytes",
  "value": 230918
},
{
  "description": "...",
  "name": "s3ObjectsTotalSize",
  "tags": {
    "prefix": "monthly"
  },
  "type": "bytes",
  "value": 2507812
}

`s3ByPrefixOnly`

When using the previous s3ByPrefix option, all S3 objects are analyzed by default, including those that do not match the provided prefix or prefixes. Enabling this option limits the analysis to only those objects that match the prefixes.

?s3ByPrefixOnly=true

`s3Versions`

Type: boolean, Default: false

Whether to list objects versions in the bucket and generate metrics for them.

?s3Versions=true

Versioning policy

This project follows the rules of semantic versioning v2.0.0. Check the CHANGELOG for breaking changes between major versions.

Beware: changes to the description property of metrics or failures will not be considered breaking.

Please file a bug If you notice an actual breaking change without a corresponding major version number change.

Name		Name	Last commit message	Last commit date
Latest commit History 67 Commits
bin		bin
src		src
.ackrc		.ackrc
.dockerignore		.dockerignore
.eslintrc.js		.eslintrc.js
.gitignore		.gitignore
.tool-versions		.tool-versions
CHANGELOG.md		CHANGELOG.md
Dockerfile		Dockerfile
LICENSE.txt		LICENSE.txt
README.md		README.md
nodemon.json		nodemon.json
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
webpack.js		webpack.js

License

AlphaHydrae/probeit

Folders and files

Latest commit

History

Repository files navigation

Probe It

Usage

Installation

Metrics

Prometheus format

Generic metrics

duration

Command probe

Function commands

System commands

System command metrics

commandExitCode

HTTP probe

HTTP probe metrics

httpCertificateExpiry

httpContentLength

httpDuration

httpRedirects

httpSecure

httpStatusCode

httpVersion

HTTP probe parameters

allowUnauthorized

followRedirects

header

method

HTTP probe expectations

expectHttpRedirects

expectHttpRedirectTo

expectHttpResponseBodyMatch

expectHttpResponseBodyMismatch

expectHttpSecure

expectHttpStatusCode

expectHttpVersion

HTTP probe failures

httpResponseBodyMismatch

insecureHttp

invalidHttpRedirectCount

invalidHttpRedirectLocation

invalidHttpStatusCode

invalidHttpVersion

missingHttpRedirect

unexpectedHttpRedirect

unexpectedHttpResponseBodyMatch

unexpectedlySecureHttp

S3 probe

S3 probe metrics

s3FirstObjectModificationDate

s3FirstObjectSize

s3FirstObjectVersionModificationDate

s3FirstObjectVersionSize

s3LargestObjectSize

s3LastObjectModificationDate

s3LastObjectSize

s3LastObjectVersionModificationDate

s3LastObjectVersionSize

s3ObjectsCount

s3ObjectsTotalSize

s3ObjectVersionsCount

s3SmallestObjectSize

S3 probe parameters

s3ByPrefix

s3ByPrefixOnly

s3Versions

Versioning policy

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

`duration`

`commandExitCode`

`httpCertificateExpiry`

`httpContentLength`

`httpDuration`

`httpRedirects`

`httpSecure`

`httpStatusCode`

`httpVersion`

`allowUnauthorized`

`followRedirects`

`header`

`method`

`expectHttpRedirects`

`expectHttpRedirectTo`

`expectHttpResponseBodyMatch`

`expectHttpResponseBodyMismatch`

`expectHttpSecure`

`expectHttpStatusCode`

`expectHttpVersion`

`httpResponseBodyMismatch`

`insecureHttp`

`invalidHttpRedirectCount`

`invalidHttpRedirectLocation`

`invalidHttpStatusCode`

`invalidHttpVersion`

`missingHttpRedirect`

`unexpectedHttpRedirect`

`unexpectedHttpResponseBodyMatch`

`unexpectedlySecureHttp`

`s3FirstObjectModificationDate`

`s3FirstObjectSize`

`s3FirstObjectVersionModificationDate`

`s3FirstObjectVersionSize`

`s3LargestObjectSize`

`s3LastObjectModificationDate`

`s3LastObjectSize`

`s3LastObjectVersionModificationDate`

`s3LastObjectVersionSize`

`s3ObjectsCount`

`s3ObjectsTotalSize`

`s3ObjectVersionsCount`

`s3SmallestObjectSize`

`s3ByPrefix`

`s3ByPrefixOnly`

`s3Versions`

Packages