Add upgrading and version management documentation (#3695)

Long ago, we defined upgrading procedures for OpenTelemetry, in [OTEP
147](https://github.com/open-telemetry/oteps/blob/main/text/0147-upgrade-procedures.md).
However, we never added this language to the specification.

This pull request adds the OTEP to the specification, verbatim, except
for an addition at the top which clearly spells out dependency
management for application developers, library maintainers, and SDK
implementors.

It has been two years, so I am open to changes to this document. Both
for clarity, and to conform to how we currently practice upgrading and
dependency management.

CHANGELOG.md

@@ -34,6 +34,9 @@ release.
 
 ### Common
 
+- Add upgrading and version management documentation
+  ([#3695](https://github.com/open-telemetry/opentelemetry-specification/pull/3695))
+
 ### Supplementary Guidelines
 
 ## v1.25.0 (2023-09-13)

specification/upgrading.md

@@ -0,0 +1,164 @@
+# The OpenTelemetry approach to upgrading
+
+Managing widely distributed software at scale requires careful design related to
+backwards compatibility, versioning, and upgrading. The OpenTelemetry approach is
+described below. If you are planning on using OpenTelemetry, it can be helpful to
+understand how we approach this problem.
+
+## Component Overview
+
+To facilitate smooth upgrading and long term support, OpenTelemetry clients are
+factored into several components. We use the following terms in the rest of this
+document.
+
+**Packages** is a generic term for units of code which reference each other via
+some form of dependency management. Note that every programming language has a
+different approach to dependency management, and may use a different term, such
+as module or library, to represent this concept.
+
+**The API** refers to the set of software packages that contain all of the interfaces
+and constants needed to write OpenTelemetry instrumentation. An implementation of
+the API may be registered during application startup. If no other implementation
+is registered, the API registers a no-op implementation by default.
+
+**The SDK** refers to a framework which implements the API, provided by the OpenTelemetry
+project. While alternative API implementations may be written to handle special
+cases, we expect most users to install the SDK when running OpenTelemetry.
+
+**Plugin Interfaces** refer to extension points provided by the SDK. These include
+interfaces for controlling sampling, exporting data, and various other lifecycle
+hooks. Note that these interfaces are not part of the API. They are part of the SDK.
+
+**Instrumentation** refers to any code which calls the API. This includes the instrumentation
+provided by the OpenTelemetry project, third party instrumentation, plus application
+code and libraries which instrument themselves natively.
+
+**Plugins** refer to any package which implements an SDK Plugin Interface. This
+includes the plugins provided by the OpenTelemetry project, plus third party plugins.
+
+There is an important distinction between Plugins and Instrumentation. Plugins implement
+the Plugin Interfaces. Instrumentation calls the API. This difference is relevant
+to OpenTelemetry’s approach to upgrading.
+
+## Dependency Management
+
+It is very important that OpenTelemetry users and implementors stay up to date with
+the latest version of the OpenTelemetry API. OpenTelemetry follows strict rules in
+regards to backwards compatibility. It is always safe to upgrade to the latest minor
+version of an OpenTelemetry package.
+
+### Application Developers
+
+Developers who plan to install the OpenTelemetry SDK in their application are advised
+to are advised to accept all minor version upgrades. This will ensure that applications
+will always be built with the latest optimizations and security patches.
+
+Likewise, application developers are encouraged to depend upon all future versions
+of the current major version of the OpenTelemetry API. This will ensure that version
+conflicts will not occur when instrumentation is upgraded to make use of new features
+in future minor versions.
+
+### Library Maintainers
+
+Developers who maintain OSS libraries are encouraged to use the OpenTelemetry API
+to add instrumentation directly into their library code. We refer to this approach
+as "native instrumentation."
+
+When adding the OpenTelemetry API as a dependency to your library, it is very important
+to depend upon to ***all future versions*** of the current major version.
+
+Depending on a specific minor version ***can lead to version conflicts*** with other
+libraries that make use of API features added in later minor versions. Minor version
+bumps of the API will never cause a compatibility issue with existing instrumentation.
+It is completely safe to depend on future versions of the API.
+
+### SDK Implementors
+
+Maintainers of the official OpenTelemetry SDKs as well as maintainers of alternative
+SDKs MUST keep their implementations up to date and compatible with the latest version
+of the OpenTelemetry API.
+
+When new minor versions of the OpenTelemetry API are released, they will contain
+new features. This includes new interfaces, as well as new methods on current interfaces.
+SDK maintainers are expected to release new versions which implement these features
+in a timely manner.
+
+If you do not believe you will be able to continue to maintain an implementation
+in this manner, we strongly recommend that you do not implement an alternative SDK.
+It is inevitable that OpenTelemetry will release new features, and that application
+developers and library maintainers will make use of these features.
+
+## The OpenTelemetry upgrade path
+
+Before we get into the design requirements, here’s how upgrading actually works.
+Note that all OpenTelemetry components (API, SDK, Plugins, Instrumentation) have
+separate version numbers.
+
+### API changes
+
+When new functionality is added to the OpenTelemetry API, a new minor version of
+the API is released. These API changes are always additive and backwards compatible
+from the perspective of existing Instrumentation packages which import and call
+prior versions. Instrumentation written against all prior minor versions of the
+API continues to work, and may be composed together into the same application without
+creating a dependency conflict.
+
+API implementations are expected to always target the latest version of the API.
+When a new version of the API is released, a version of the SDK which supports the
+API is released in tandem. Old versions of the SDK are not expected to support newer
+versions of the API.
+
+### SDK changes
+
+Bugs fixes, security patches, and performance improvements are released as patch
+versions of the SDK. Support for new versions of the API are released as minor versions.
+New Plugin Interfaces are released as a minor version bump to the SDK.
+
+Breaking changes to Plugin Interfaces are handled through deprecation. Instead of
+breaking a Plugin Interface, a new interface is created and the existing interface
+is marked as deprecated. Plugins which target the deprecated interface continue
+to work, and the SDK provides default implementations of any missing functionality.
+After one year, the deprecated Plugin Interface may be removed in a major version
+release of the SDK.
+
+## Design requirements and explanations
+
+This approach to upgrading solves two critical design requirements, while minimizing
+the maintenance overhead associated with legacy code.
+
+* Callers of the API are never broken.
+* Users of the SDK can easily upgrade to the latest version.
+
+Indefinite support for existing Instrumentation is a critical feature of OpenTelemetry.
+Millions of lines of code are expected to be written against the API. This includes
+shared libraries which ship with integrated OpenTelemetry instrumentation. These
+libraries must be able to compose together into applications without OpenTelemetry
+creating a dependency conflict. While some Instrumentation will be updated to the
+latest version of the API, such as that provided by the OpenTelemetry project, other
+Instrumentation will never be updated.
+
+Consuming new Instrumentation may require users to upgrade to the latest version
+of the SDK. If it was not easy to perform this upgrade, the OpenTelemetry project
+would be forced to support older versions of the SDK, as well as older versions
+of the entire Instrumentation ecosystem.
+
+This would be an enormous maintenance effort at best. But, as the OpenTelemetry
+project only controls a portion of that ecosystem, it is also unfeasible. OpenTelemetry
+cannot require that libraries with native instrumentation support multiple versions
+of the API. Ensuring that application owners and operators can upgrade to the latest
+version of the SDK resolves this issue.
+
+The primary blocker to upgrading the SDK is out of date Plugins. If a new version
+of the SDK were to break existing Plugin Interfaces, no user would be able to upgrade
+their SDK until the Plugins they depend on have been upgraded. Users could be caught
+between instrumentation they depend on requiring a version of the API which is not
+compatible with the version of the SDK which supports their Plugins.
+
+By following a deprecation pattern with Plugin Interfaces, we create a one year
+window in which the Plugin ecosystem can upgrade after the release of a new SDK.
+We believe this is sufficient time for any Plugin which is actively maintained
+to make an upgrade, and for defunct Plugins to be identified and replaced.
+
+By ensuring that the SDK can be easily upgraded, we also provide a path for application
+owners and operators to rapidly consume critical bug fixes and security patches,
+without the need to backport these patches across a large number of prior SDK versions.