Document that clients are compatible across at least 2 major versions

[dblock]

What kind of business use case are you trying to solve? What are your requirements?

Coming from #12, clients are being upgraded to support OpenSearch 2.0 while breaking compatibility with 1.0. For example, opensearch-java 1.0 works against OpenSearch 1.x, opensearch-java 2.0 works against OpenSearch 2.x, but there's no version of client that works against both OpenSearch 1.x and 2.x. Thus, in order to upgrade from OpenSearch 1.x to 2.x one needs to either run a cluster in mixed mode for a long time, or take downtime.

With rolling upgrades:

1. Upgrade some nodes to 2.0, cluster is now in mixed state.
2. Upgrade all clients to 2.0 and direct trafic from these clients to 2.0 nodes.
3. Finish upgrading the cluster to 2.0.

What is the problem? What is preventing you from meeting the requirements?

Users have to upgrade both clients and server at the same time and/or run in mixed cluster mode for a very long time. This is a lot of work, hard to coordinate, risky, etc. Most users will choose not to upgrade across major versions.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?
Ensure that there's at least 1 version of a client that works against two major versions. For example, opensearch-java 2.0 would work against 1.x and 2.0 and deprecate all 1.x methods, and remove them in 3.0.

Describe alternatives you've considered
REST API versioning so users can specify their desired API version to ensure client compatibility is independent of server upgrades.

Other

This is tangentially related to opensearch-project/OpenSearch#3023.

[dlvenable]

Ensure that there's at least 1 version of a client that works against two major versions. For example, opensearch-java 2.0 would work against 1.x and 2.0 and deprecate all 1.x methods, and remove them in 3.0.

I think the goal should be that at least 1 version of a client in major version N works in major version N+1.

It seems it could be a problem to require that OpenSearch 2.0 client work against OpenSearch 1.0. Some breaking changes may make this infeasible.

[dlvenable]

As part of this proposal, I'd also like to include that clients be sure to correctly deprecate APIs, fields, or methods which are deprecated. This will make it easier for a user to remove all the methods which are slated for removal in the upcoming version. For Java, for example, these APIs should have the @Deprecated annotation.

[VachaShah]

The removals happening for 2.0 had already been deprecated and made optional in the 1.x versions in the client (example: https://github.com/opensearch-project/opensearch-java/blob/1.x/java-client/src/main/java/org/opensearch/client/opensearch/core/GetSourceRequest.java#L225).

[nknize]

👎
Wire compatiblity is the last Minor release of Major - 1; this is to avoid long running deprecations in the transport layer.
In core, Index compatibility is the first major release of Major - 1; this is to support rolling upgrades across the entire previous version.

Separately, the REST index API should be versioned to ensure the REST interface is compatible across all running versions. We just don't have that yet; so I'd put the effort into developing API versioning.

[reta]

As per my understanding, the REST Java client for 2.0 should have no issues working with 1.x (for existing APIs) - in theory. The major change here is, of cause, removal of _type and its usage across all HTTP endpoints. We know that 1.x supports dual with/without _type endpoints, but 2.0 - only without _type, and this is what REST Java client for 2.0 should use.

[andrross]

As per my understanding, the REST Java client for 2.0 should have no issues working with 1.x (for existing APIs)

To generalize this a bit, I would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0. Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

[reta]

To generalize this a bit, I would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0

👍

Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

👍 (but the _type removal affects majority of the HTTP API landscape)

[dblock]

For the Java client I opened opensearch-project/opensearch-java#152 to re-add compatibility with 1.x in the terms described above.

[dblock]

@nknize Do you not agree that one would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0?

[nknize]

Do you not agree that one would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0?

Not w/ the current implementation, no. This is an argument for REST API versioning.

Update: I caution that this proposal is an intermediate hack that will force OpenSearch to carry server side tech debt and deprecated features for an extra major version making it harder to remove stale code until REST API Versioning becomes available.

[nknize]

Hello community. If you were brought here from the RFC Tweet be sure to read the RFC Steps to make OpenSearch Extensible [RFC] REST API Versioning to get full context on a REST API Versioning. This will enable users to specify their desired API version to ensure client compatibility is independent of server upgrades while providing full compatibility across clients without requiring to carry deprecated features or tech debt across more than one major server version.

We will update the description to this accordingly

[andrross]

Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

👍 (but the _type removal affects majority of the HTTP API landscape)

@reta All the non _type variants of the APIs existed in 1.x, so was it not possible to use the 1.x client in a way that didn't hit any deprecated API? If we execute properly on the policy of always deprecating things one major version before removal on the server side, then it seems like it should be possible to use the previous client in a way that is guaranteed to be forward compatible with the next major version. Where did we go wrong that we ended up in a state where "there's no version of client that works against both OpenSearch 1.x and 2.x"?

[reta]

Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

+1 (but the _type removal affects majority of the HTTP API landscape)

@reta All the non _type variants of the APIs existed in 1.x, so was it not possible to use the 1.x client in a way that didn't hit any deprecated API?

You are actually correct, if no deprecated APIs are used, 1.x should also work.

Where did we go wrong that we ended up in a state where "there's no version of client that works against both OpenSearch 1.x and 2.x"?

That statement seems to be not entirely correct (2.x should work with 1.x and 2.x), the usual migration path is:

• migrate client to next major
• ensure it compiles (also ensures that no deprecated APIs are used)
• migrate servers and use new APIs

But again, as you mentioned, if no deprecated APIs are used in 1.x, it should also work with 2.x (it could be difficult to audit).

[andrross]

I basically rephrased what @dlvenable said here #12 (comment)

I believe OpenSearch's goal should be: Users update to the latest client version in a major version (e.g. 1.3). They should see deprecations clearly (e.g. Java's @deprecated, READMEs). And if they are diligent to remove these deprecated APIs, then this client should be compatible with the the next major version (e.g. 2.0). They won't be able to use new features, but they should not be broken. Then after updating their cluster to OpenSearch 2.0, they can update their client libraries.

This does require client maintainers and users to be very diligent and stay up-to-date with all deprecations, but this should be possible today with the server-side policy of using semver and deprecating features one major version before removal. Users should be able to audit this with client-side deprecation annotations as well as server-side deprecation logs. The REST API Versioning proposal is a way to relax this constraint and could make life easier for users and clients, but I don't believe it is required to avoid the situation where no client works with both 1.x and 2.x. I also don't believe it should be necessary for the server side to carry additional tech debt to solve this, which @nknize expressed concern about.

My question is: assuming a user doesn't explicitly invoke a type-related API endpoint or parameter, then why does the 1.x client break against a 2.0 server?

[nateynateynate]

Do you not agree that one would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0?

Not w/ the current implementation, no. This is an argument for REST API versioning.

Update: I caution that this proposal is an intermediate hack that will force OpenSearch to carry server side tech debt and deprecated features for an extra major version making it harder to remove stale code until REST API Versioning becomes available.

I'm inclined to agree with @nknize here. Sorry I don't have much street cred around here yet, but at face value "ensuring compatibility" feels like duct tape (for lack of a better term). It also seems like something that will have to be repeated awkwardly at each major. Having clients report the API version they're expecting to use, despite the difficulty / risk, seems like a permanent and structured solution to allow users to upgrade either the client or the server without worrying about anything breaking.

Doesn't one way have the potential to break entire workflows for people, while API versioning provides a nice little safe space for clients not quite all the way caught up to latest?

It also makes me a little nervous as these sharp cutoffs with the potential for a bit of chaos when moving to the next major introduce the risk of regression, which IMO would be even worse than technical debt.

Let's also not forget that we can take on documentation debt as well as technical debt - we'll have to find a way to communicate deprecations and changes clearly and completely. With API versioning, that documentation debt is minimized because older parts won't have to be constantly revisited. At some point in the future we can still decide to deprecate endpoints for older API versions.

[reta]

My question is: assuming a user doesn't explicitly invoke a type-related API endpoint or parameter, then why does the 1.x client break against a 2.0 server?

@andrross among the few examples I can think of is search (see please opensearch-project/OpenSearch#2989), the 2.0 removed _type field semantics (correct) but 1.x could still rely on using exists query for _type fe. (it won't fail but return no results). It still falls under implicit deprecation though to not rely on _type field.

With respect of type-related API endpoint or parameter - I don't see gaps here, it should NOT break

[nknize]

if no deprecated APIs are used in 1.x, it should also work with 2.x (it could be difficult to audit).

correct. What opensearch-project/OpenSearch#3035 proposes is adding version semantics to the HLRC so that it's baked in. If we had this mechanism already, 1.x clients would implicitly pass the 1.x version string in the HTTP header. HLRC would parse the version string from the header and offer the compatibility needed. What this might look like for _type removal (for example) is providing the removed type endpoints & responses for bwc but removing and migrating the server logic appropriately.

[dblock]

I am a bit concerned that we're discussing how HLRC can be improved (opensearch-project/OpenSearch#3035) as opposed to how users upgrade 1.x to 2.x without downtime. So, @nknize what do you suggest users do for 1.x -> 2.x wrt Java client if we don't do anything about this proposal?

[dlvenable]

Java has two high-level clients - rest-high-level from OpenSearch core, and opensearch-java. Regarding Java, is this current proposal for only opensearch-java? Or both?

[dblock]

@dlvenable I didn't intend to apply this to https://github.com/opensearch-project/OpenSearch/tree/main/client/rest-high-level - @nknize do you have any opinions of what we should do about that one?

[dlvenable]

@dlvenable I didn't intend to apply this to https://github.com/opensearch-project/OpenSearch/tree/main/client/rest-high-level

I believe that is reasonable. I wanted to align on this so that we know the best approach to take going forward.

[wbeckler]

I think there's a distinction when we say "compatible" that impacts deprecation and semver. So far, in the discussion above, a "client" is a library that calls out to a server API. But the client has its own API in the form of the available methods accessed by the code that uses that library. So in fact there are three codebases (application, client, server) and two APIs connecting them all together. Let's call the library's interface the "client library API" and the server's interface the "server API":

[Application] <-client library API-> [language client] <- server API -> [OpenSearch server]

So far in this discussion, the overriding concept under debate has been the application of semver to the server API. But we might fall into the trap of thinking that the client library API must follow the same deprecations at the same pace. This is how these principles have been applied in the meta issue and child issues for removing/deprecating client library APIs.

Instead, we should consider having a separate pacing for versioning/deprecating the client library API. If you disable a method in the client library API when you disable it in the server API, you limit the options of a team managing a variety of aging applications that share client libraries or hit the same database, even if the server API is designed for backward compatibility.

For example, say a company has two java applications running in their company, and let's call them "Edsel" and "Tesla." They are in a monorepo with a single client library, v1.0. The Edsel application hasn't been touched in years and is hard to change, having poor test coverage. The Tesla application is easier to upgrade and needs to access features of OpenSearch v2.x. If the server is upgraded to 2.x, it could work with client library v1.x if we ensure that kind of backward compatibility guaranteed by semver. However the features Tesla needs are only accessible in client v2.x. Edsel uses a deprecated method foo() and sees warnings in library v1.x about foo()'s impending disappearance, but Edsel hasn't cared until now. When the company tries to upgrade the client library to 2.x, they find that Edsel breaks. Would there exist a client library that they both can use that supports method foo() and that can talk to OpenSearch server v2.x? Even if a 2.x server can talk to a 1.x client library, it doesn't help the Tesla application. The only way this company can avoid lots of changes to Edsel is if we allow a client library with 2.x methods to talk to an application that uses the v1.x client library API. This is impossible if we intentionally break old methods in new versions of the client library.

What I'm getting at is that evolution of the client library API has consequences for application maintainers that can't be solved by ensuring backward compatibility of the server API.

The solution for the above example would be to allow the 1.x version of the client to access new methods in the 2.x version of the server. If there are ambiguities or incompatibilities in the server API, we can have the client library API specify which server API version to use, and the one client library would support multiple server API versions. We would only increment the client version if this becomes unwieldy or breaks, and there is a need to force the Edsels to change the way they access the client library API.

Therefore, we might want to reconsider the original idea of intentionally removing/deprecating client library APIs as per #12.

When we do something like opensearch-project/opensearch-java#152, we should consider the case where an application shouldn't have to upgrade its usage of the client library. And opensearch-project/opensearch-java#152 would trigger a release of opensearch-java v1.4 or 1.5 or whatever is the next increment in the 1.x series.

To get this right, I think we need to use the phrase "client library API" and shine a light on the work that goes into maintaining applications that have evolving library APIs.

Thoughts?

[dblock]

The tl;dr of client versioning is that clients also follow semver. When a bug is fixed, it's a patch version increment. When a feature is added, minor version is incremented. When a backwards incompatible change is made, major version is incremented. "Compatibility with OpenSearch X.Y" is a feature.

The tl;dr of my initial proposal is that we cannot remove support for OpenSearch 1.0 and add support for OpenSearch 2.0 in one (same) client release. If we do, users can't upgrade without downtime.

This may be hard to do, e.g. opensearch-project/opensearch-net#51. Proposals such as opensearch-project/OpenSearch#3035 make that easier, but that's not the only way. For example, the majority of the client could be generated, and the client would carry both the generated code for version 1.x of OpenSearch, and version 2.x. Then at runtime it would switch between the two versions (all the bugs are found at runtime). This is not an uncommon approach for script-based clients (Ruby/Python) which need significant integration testing anyway.

[dblock]

We've been following this in all clients. Plan is to add some words about this in this repo .md and resolve.