Capability to enable/disable loggerSampler in the exporterHelper

[ajimenez1503]

Issue:

In one of the use cases which we are working on, we have identified that sampledLogger mechanism used in the Opentelemetry exporters (otlpexporter and otlphttpexporter), could become a memory bottleneck, when the number of exports in the same pipeline is high (we are talking about 100s of instances of exporters running on the same machine).

The sampledLogger is created inside the exporterhelper module and the sampling is always activated unless the collector is in debug mode:

• https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/queued_retry.go#L88

func newQueuedRetrySender(id component.ID, signal component.DataType, qCfg QueueSettings, rCfg RetrySettings, reqUnmarshaler internal.RequestUnmarshaler, nextSender requestSender, logger *zap.Logger) *queuedRetrySender {
	retryStopCh := make(chan struct{})
	sampledLogger := createSampledLogger(logger)

• https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/queued_retry.go#L269

func createSampledLogger(logger *zap.Logger) *zap.Logger {
	if logger.Core().Enabled(zapcore.DebugLevel) {
		// Debugging is enabled. Don't do any sampling.
		return logger
	}

	// Create a logger that samples all messages to 1 per 10 seconds initially,
	// and 1/100 of messages after that.
	opts := zap.WrapCore(func(core zapcore.Core) zapcore.Core {
		return zapcore.NewSamplerWithOptions(
			core,
			10*time.Second,
			1,
			100,
		)
	})
	return logger.WithOptions(opts)
}

Investigation:

The high memory usage has been discovered using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/pprofextension

• In the first scenario, default configuration with the sampledLogger active, based on the profiled data we could observe that sampling mechanism was responsible for allocating aprox. 75% of the memory of the running collector:
  

• In the second scenario, with the same number of running exporters, but with the sampledLogger mechanism disabled (using debug level configuration check configuration below), we were able to observe a much smaller memory footprint:

    service:
      telemetry:
        logs:
          level: debug

Given that the logging sampling mechanism could become a bottleneck in certain scenarios (our case), we would like to propose a contribution, which potentially can improve the memory footprint of the exporters in certain use cases (potentially making it more scalable)

Proposal:

Add a new configuration SampledLoggerSetting to the otlpexporter and otlphttpexporter

// SampledLoggerSettings. The SampledLogger samples logging messages, which
// caps the CPU and I/O load of logging while attempting to preserve a
// representative subset of your logs.
type SampledLoggerSettings struct {
	// Enable the sampledLogger
	Enabled bool `mapstructure:"enabled"`
}

Default configuration:

sampled_logger:
 enabled: true

Documentation:

Update Readme of otlpexporter and otlphttpexporter with the new configuration field SampledLoggerSettings

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: softwarejimenez / name: Antonio Jimenez (c067b92, b022f34, 50e3447, 5a7cf15, 9801d53, 23829f5, b7995a7, 70cbd78)

[ajimenez1503]

/review @pedro-te @sboychen-te @vutnguye-te @sfreitas-te @xvilaca-te @vkumov-te

[ajimenez1503]

``