Maven profile for test to try Protobuf 4.x runtime

[suztomo]

I'm exploring how we can confirm whether the libraries in this repository are compatible with Protobuf 4.x runtime. This pull request is an option to specify a special test runtime dependencies through surefire plugin's <additionalClasspathDependencies> element. This setup generates the NoSuchMethodError for the method in the Protobuf class.

How to run a problematic test:

~/hadoop-connectors$ mvn -pl gcsio install -Pprotobuf4-test -Dtest=com.google.cloud.hadoop.gcsio.BatchHelperTest
...

[ERROR] Tests run: 7, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.416 s <<< FAILURE! -- in com.google.cloud.hadoop.gcsio.BatchHelperTest
[ERROR] com.google.cloud.hadoop.gcsio.BatchHelperTest.allRequestsAreSentInSingleBatch_withOneMaxThreads -- Time elapsed: 0.206 s <<< ERROR!
java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$ServiceOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)'
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.builder(GoogleCloudStorageOptions.java:58)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.<clinit>(GoogleCloudStorageOptions.java:44)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageTest.<clinit>(GoogleCloudStorageTest.java:134)
	at com.google.cloud.hadoop.gcsio.BatchHelperTest.allRequestsAreSentInSingleBatch_withOneMaxThreads(BatchHelperTest.java:171)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

Analysis

@blakeli0 analyzed the binary incompatibility:

For this specific issue, we found that the root cause is changes to ServiceOptions.getExtension() is not binary compatible and hadoop-connectors need to recompile and release with the new protobuf 4.28.2.
In 3.x, ServiceOptions extends from GeneratedMessageV3.ExtendableMessage which contains a method getExtention(GeneratedExtension extension).
In 4.x, ServiceOptions extends from GeneratedMessage.ExtendableMessage which contains a method getExtention(ExtensionLite extension). ExtensionLite is a super class of GeneratedExtension, hence it is source compatible but not binary compatible.

Solution?

The simplest solution is just compile the libraries with Protobuf 4.x. However doing so might drop the compatibility with Protobuf 3 library, requiring us to wait for a new major version release of this repository. How might we provide a good compatibility with both Protobuf 4 and 3 libraries?

[suztomo]

@arunkumarchacko In this (normal) Maven setup, the JUnit tests run before packaging (shading). If you explore the shading Protobuf dependencies, the tests may have to run as integration tests.