Rethink how instrumentation packages should handle dependencies

[owais]

Instrumentation packages patch popular libraries or frameworks at runtime to make them observable. However, it is not practical for instrumentation packages to support all versions of any given library. As a result, instrumentations need a way to specify the supported version ranges for the libraries. Today, we solve this by specifying the libraries as dependencies for instrumentations. For example, if Flask instrumentation works with Flask 1.0, the instrumentation specifies flask ~= 1.0 as a dependency. This ensures that users installing the instrumentation with an unsupported library version will get a dependency conflict and as a result not let the user install the instrumentation package.

This solves the problem but it also forces users to install the libraries when installing the instrumentation packages. This may not be ideal in all situations.

For example, this complicates getting started with OpenTelemetry and requires additional tooling to make things easier (opentelemetry-bootstrap command) as users must only install the exact instrumentation packages that are a perfect fit for their environment.

Proposal

I propose we stop specifying the target libraries as dependencies and use an alternate mechanism to specify supported version ranges.

Pros

• Allow us to ship all instrumentations as a single bundle or meta-package, e.g pip install opentelemetry-instrumentations-all.
• Solves issues like opentelemetry-bootstrap -a install shouldn't upgrade libraries #1563
• Get rid of the opentelemetry-bootstrap command and simplify instrumenting Python services.
  • No need to integrate the bootstrap command in your CI/CD system or run it on servers.
  • Or no need to run it locally to find matching instrumentations.
  • Would make installing instrumentation just like installing any other Python package eliminating surprises and reducing friction.
• Dramatically simplify installing instrumentations in serverless environments like AWS Lambda. People can use any of dozens of available tools to write, deploy and manage their serverless applications. A lot of these tools handle dependencies differently and it is not obvious how to integrate the bootstrap command with such tools. Making this change would allow distributions to ship all instrumentatoins in a single (meta)package and simplify the build/deploy process by relying on standard practices for Python.
• Bring it more in sync with how other Otel projects such as Node and Java handle instrumentations. Java includes all instrumentation in the same agent. Node offers npm packages like @opentelemetry/plugins-node-core-and-contrib. In both cases installing additional instrumentations is harmless as they don't pull in additional libraries and are too small to inflate overall size.

Alternatives

Runtime version checks

Instrumentations attempt to import the target libraries and monkey patches them. We can simply update the instrumentations to check if the installed target library is a supported version and only patch it in that case. When an unsupported library is found, instrumentation can log a warning and return early without applying any patches.

Identifying versions

1. Use information provided by libraries

Most libraries include their version number in code and make it importable. This should be used by instrumentations to identify the target library version when possible. For example,

In [1]: from requests import __version__

In [2]: __version__
Out[2]: '2.24.0'

2. Use pkg_resources

If a library does not include it's version number in code, pkg_resources can be used to load metadata about installed Python packages. This can be used to identify installed versions. For example,

In [8]: import pkg_resources

In [9]: requests_pkg = pkg_resources.get_distribution("requests")

In [10]: requests_pkg.version
Out[10]: '2.24.0'

• Needs investigation: How reliable is pkg_resources and is it available on all Python versions? Can it reliably report version information for all any kind of package?

3. Identify by structure/traits

If for some reason pkg_resources cannot be used, the instrumentation should try to identify the version by looking for well known traits such as presence of certain attributes, classes, modules etc.

[Oberon00]

Relatedly, instead of eagerly loading all the libraries available on the Python path, it would be cool if import hooks (e.g. wrapt's @when_imported https://github.com/GrahamDumpleton/wrapt/blob/develop/src/wrapt/importer.py#L223-L226) could be used instead to load only on demand, in case someone has a library lying around that is not actually used, or maybe even needs to be configured somehow before first import.

[owais]

Nice. This is similar to Node's "require in the middle". I didn't know about this. Thanks

[srikanthccv]

Thanks for starting this. I agree with most of what has been said and still consolidating the scattered thoughts. Here is some related issue #1563.

[srikanthccv]

Have a question about coordination of opentelemetry-instrumentations-all and packages, don't we find ourselves in a place where users come and ask how could they make single bundle work for them? We don't have to support every possible scenario but should be able to cover most use cases.

[aabmass]

I think "3. Try to instrument anyway" might lead to unexpected results, like if the patched function's signature changes in a subtle way and we accidentally mix up span attributes. Runtime version checks sound like a good idea.

One concern is it becomes difficult for users to tell the actual supported versions of instrumented libraries. It would be great if users could opt-in to the dependency conflict checks. Maybe with extras_require entries in instrumentation packages so they could write requirements.txt like

...
flask~=1.0
psycopg2~=2.0

# try out everything
opentelemetry-instrumentations-all

# strictly assert the version of these
opentelemetry-instrumentation-flask[strict]~=1.0
opentelemetry-instrumentation-psycopg2[strict]~=1.0

[owais]

@aabmass Agree. I didn't mean to suggest to instrument an unsupported version but to figure out if the version is supported or not by trying to instrument. For example by looking for an attribute or other identifying behavior and bail out when such an identifier is not detected either through some checks or exceptions.

I like the idea of using extra_require to allow old behavior optionally. This can even be used in CI to not actually install packages but just check for conflicts. +1

Is there any other way for a Python packaging to specify compatible packages without actually having them as dependencies? Something like NPM's peer dependcies I guess.

[owais]

Updated to clarify

[aabmass]

For example by looking for an attribute or other identifying behavior and bail out when such an identifier is not detected either through some checks or exceptions.

Even then it seems tricky. If the function you plan to wrap is still there but e.g. parameter order changed, you could end up with weird results. Not sure if wrapt has any helpers for this, but you could use inspect.signature() on the wrapped function to check the signature is compatible

Is there any other way for a Python packaging to specify compatible packages without actually having them as dependencies? Something like NPM's peer dependcies I guess.

I think extras_require is the closest you can get, but not sure on that

[owais]

We don't need to come up with any heuristics to generalize the approach. Each instrumentation author would consciously identify such versions and write the instrumentation accordingly with appropriate safe guards in place. It'd be helpful to take a look at existing instrumenations and see how many such libraries do we support today that can't be solved with 1st and 2nd proposed approaches. If there aren't any, we probably don't even need this.

[owais]

Turns out we can use the 2nd option and implement this it centrally in the instrument command instead of changing every single instrumentation. @aabmass suggested we specify the dependencies as extras instead of completely removing them so that users can still enforce compatibility at install time. This also allows the instrument command to inspect instrumentation packages and only load/apply instrumentations whose extra dependencies are installed.

This allows us to not specify the instrumented packages as dependencies for instrumentations while allowing users to retain the previous behavior if they want. Also, this does not require any code or interface changes to instrumentations at all.

Link to POC: #1741