Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add microbenchmarks to perf benchmark doc #3696

Closed
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 32 additions & 10 deletions specification/performance-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,35 @@
This document describes common performance benchmark guidelines on how to
measure and report the performance of OpenTelemetry SDKs.

## Microbenchmarks

Microbenchmarks are defined as tests that measure performance of small units of
code. SDKs may implement any number of microbechmark tests as needed.
SDKs can use the following tests as a guideline to get started:

### Create Spans

Number of spans which could be created and ended in 1 second per logical core.
martinkuba marked this conversation as resolved.
Show resolved Hide resolved
The configuration of the span should be as follows:

- No parent `Span` or parent `SpanContext`.
- Default Span [Kind](./trace/api.md#spankind) and
[Status](./trace/api.md#set-status).
- 10 [attributes](./common/README.md#attribute) with two 20-character strings -
one as attribute name and the other as value.
- The `AlwaysOn` sampler should be enabled.
- No `SpanProcessor`s are enabled.
- Each `Span` is created and immediately ended.

## End-to-end benchmark

The goal of this benchmark is to provide a tool to get the basic performance
overhead of the OpenTelemetry SDK for given events throughput on the target
platform.

## Benchmark Configuration
### Benchmark Configuration

### Span Configuration
#### Span Configuration

- No parent `Span` or parent `SpanContext`.
- Default Span [Kind](./trace/api.md#spankind) and
Expand All @@ -24,24 +46,24 @@ platform.
- The `AlwaysOn` sampler should be enabled.
- Each `Span` is created and immediately ended.

### Measurement Configuration
#### Measurement Configuration

For the languages with bootstrap cost like JIT compilation, a warm-up phase is
recommended to take place before the measurement, which runs under the same
`Span` [configuration](#span-configuration).

## Throughput Measurement
### Throughput Measurement

### Create Spans
#### Create Spans

Number of spans which could be created and exported via OTLP exporter in 1
second per logical core and average number over all logical cores, with each
span containing 10 attributes, and each attribute containing two 20 characters
strings, one as attribute name the other as value.

## Instrumentation Cost
### Instrumentation Cost

### CPU Usage Measurement
#### CPU Usage Measurement

With given number of span throughput specified by user, or 10,000 spans per
second as default if user does not input the number, measure and report the CPU
Expand All @@ -57,14 +79,14 @@ of SDK's CPU consumption.
The total running time for one test iteration is suggested to be at least 15
seconds. The average and peak CPU usage should be reported.

### Memory Usage Measurement
#### Memory Usage Measurement

Measure dynamic memory consumption, e.g. heap, for the same scenario as above
CPU Usage section with 15 seconds duration.

## Report
### Report

### Report Format
#### Report Format

All the numbers above should be measured multiple times (suggest 10 times at
least) and reported.
Loading