Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Okio present in beam-sdks-java-extensions-sql-jdbc #27700

Open
1 of 15 tasks
carl-mastrangelo opened this issue Jul 26, 2023 · 5 comments
Open
1 of 15 tasks

[Bug]: Okio present in beam-sdks-java-extensions-sql-jdbc #27700

carl-mastrangelo opened this issue Jul 26, 2023 · 5 comments
Labels
awaiting triage bug java P3

Comments

@carl-mastrangelo
Copy link

What happened?

I am trying to make some HTTP calls from my Java Beam (2.48.0) job. I am using OkHTTP for my HTTP dependency (4.11.0) which has a dependency on Okio (3.2.0). The Jar beam-sdks-java-extensions-sql-jdbc includes a pretty old version of Okio (1.6.0), which happens to conflict with the version needed by OkHTTP.

As a result, the later stages of my Beam job fail with class loader errors since the beam jar is earlier on the classpath (error included below).

org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class okhttp3.internal.concurrent.TaskRunner

at org.apache.beam.sdk.util.UserCodeException.wrap ( org/apache.beam.sdk.util/UserCodeException.java:39 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Filter$1.processElement ( org/apache.beam.sdk.transforms/Filter.java:211 )
at org.apache.beam.sdk.transforms.Filter$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn.process ( org/apache.beam.sdk.io/FileIO.java:872 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.MapElements$2.processElement ( org/apache.beam.sdk.transforms/MapElements.java:151 )
at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2523 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1.processElement ( org/apache.beam.sdk.transforms/Reify.java:206 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1.process ( org/apache.beam.sdk.transforms/ReifyTimestamps.java:70 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Reshuffle$1.processElement ( org/apache.beam.sdk.transforms/Reshuffle.java:101 )
at org.apache.beam.sdk.transforms.Reshuffle$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer ( org/apache.beam.fn.harness/BeamFnDataReadRunner.java:213 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements ( org/apache.beam.sdk.fn.data/BeamFnDataInboundObserver.java:158 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion ( org/apache.beam.sdk.fn.data/BeamFnDataInboundObserver.java:123 )
at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle ( org/apache.beam.fn.harness.control/ProcessBundleHandler.java:550 )
at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType ( org/apache.beam.fn.harness.control/BeamFnControlClient.java:151 )
at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0 ( org/apache.beam.fn.harness.control/BeamFnControlClient.java:116 )
at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:539 )
at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 )
at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run ( org/apache.beam.sdk.util/UnboundedScheduledExecutorService.java:163 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )
Caused by: java.lang.NoClassDefFoundError

at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:41 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:47 )
at okhttp3.OkHttpClient$Builder.<init> ( OkHttpClient.kt:471 )
Caused by: java.lang.ExceptionInInitializerError

at okhttp3.internal.Util.<clinit> ( Util.kt:71 )
at okhttp3.internal.concurrent.TaskRunner.<clinit> ( TaskRunner.kt:309 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:41 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:47 )
at okhttp3.OkHttpClient$Builder.<init> ( OkHttpClient.kt:471 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Filter$1.processElement ( Filter.java:211 )
at org.apache.beam.sdk.transforms.Filter$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn.process ( FileIO.java:872 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.MapElements$2.processElement ( MapElements.java:151 )
at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp ( FnApiDoFnRunner.java:2523 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1.processElement ( Reify.java:206 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1.process ( ReifyTimestamps.java:70 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Reshuffle$1.processElement ( Reshuffle.java:101 )
at org.apache.beam.sdk.transforms.Reshuffle$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer ( BeamFnDataReadRunner.java:213 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements ( BeamFnDataInboundObserver.java:158 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion ( BeamFnDataInboundObserver.java:123 )
at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle ( ProcessBundleHandler.java:550 )
at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType ( BeamFnControlClient.java:151 )
at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0 ( BeamFnControlClient.java:116 )
at java.util.concurrent.Executors$RunnableAdapter.call ( Executors.java:539 )
at java.util.concurrent.FutureTask.run ( FutureTask.java:264 )
at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run ( UnboundedScheduledExecutorService.java:163 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( Thread.java:833 )

Issue Priority

Priority: 3 (minor)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@Abacn
Copy link
Contributor

Abacn commented Jul 27, 2023 •
edited
Loading

Beam does not directly depend on okhttp. running ./gradlew :sdks:java:extensions:sql:jdbc:dependencies

Dependency tree:

+--- project :sdks:java:extensions:sql
|    +--- project :sdks:java:io:parquet
|    |    +--- project :sdks:java:io:hadoop-common
|    |    |    +--- org.apache.hadoop:hadoop-client:2.10.2
|    |    |    |    +--- org.apache.hadoop:hadoop-hdfs-client:2.10.2
|    |    |    |    |    \--- com.squareup.okhttp:okhttp:2.7.5

If you're not using hadoop, probably it is safe to overwrite okhttp to 3 in your project.

@carl-mastrangelo
Copy link
Author

carl-mastrangelo commented Jul 27, 2023 •
edited
Loading

Unfortunately, it still pollutes the class path:

$ unzip -l beam-sdks-java-extensions-sql-jdbc-2.48.0.jar | grep \ ok
        0  02-01-1980 00:00   okio/
     2046  02-01-1980 00:00   okio/AsyncTimeout$1.class
     1823  02-01-1980 00:00   okio/AsyncTimeout$2.class
      750  02-01-1980 00:00   okio/AsyncTimeout$Watchdog.class
     3546  02-01-1980 00:00   okio/AsyncTimeout.class
     3385  02-01-1980 00:00   okio/Base64.class
     1213  02-01-1980 00:00   okio/Buffer$1.class
     1310  02-01-1980 00:00   okio/Buffer$2.class
    28779  02-01-1980 00:00   okio/Buffer.class
     1314  02-01-1980 00:00   okio/BufferedSink.class
     1598  02-01-1980 00:00   okio/BufferedSource.class
     9275  02-01-1980 00:00   okio/ByteString.class
     3600  02-01-1980 00:00   okio/DeflaterSink.class
     1482  02-01-1980 00:00   okio/ForwardingSink.class
     1391  02-01-1980 00:00   okio/ForwardingSource.class
     1680  02-01-1980 00:00   okio/ForwardingTimeout.class
     3213  02-01-1980 00:00   okio/GzipSink.class
     4637  02-01-1980 00:00   okio/GzipSource.class
     3359  02-01-1980 00:00   okio/InflaterSource.class
     2011  02-01-1980 00:00   okio/Okio$1.class
     1950  02-01-1980 00:00   okio/Okio$2.class
     2017  02-01-1980 00:00   okio/Okio$3.class
     4352  02-01-1980 00:00   okio/Okio.class
     1738  02-01-1980 00:00   okio/RealBufferedSink$1.class
     6950  02-01-1980 00:00   okio/RealBufferedSink.class
     2054  02-01-1980 00:00   okio/RealBufferedSource$1.class
     9757  02-01-1980 00:00   okio/RealBufferedSource.class
     2246  02-01-1980 00:00   okio/Segment.class
     5960  02-01-1980 00:00   okio/SegmentedByteString.class
     1105  02-01-1980 00:00   okio/SegmentPool.class
      323  02-01-1980 00:00   okio/Sink.class
      275  02-01-1980 00:00   okio/Source.class
      728  02-01-1980 00:00   okio/Timeout$1.class
     2296  02-01-1980 00:00   okio/Timeout.class
     2098  02-01-1980 00:00   okio/Util.class

Also to be clear, It's Okio, not OkHttp directly.

@Abacn
Copy link
Contributor

Abacn commented Jul 27, 2023

Okio is from OkHttp:

|    |    |    |    +--- org.apache.hadoop:hadoop-hdfs-client:2.10.2
|    |    |    |    |    \--- com.squareup.okhttp:okhttp:2.7.5
|    |    |    |    |         \--- com.squareup.okio:okio:1.6.0

I see, its because beam-sdks-java-extensions-sql-jdbc is a fat" jar (having all dependency shaded) so one cannot overwrite its dependencies. What functionality are you use for beam-sdks-java-extensions-sql-jdbc ? If it's not used then don't include it in classpath; if it is used probably need a custom build. This module (sql-extensions-jdbc) is not actively maintained

@carl-mastrangelo
Copy link
Author

What functionality are you use for beam-sdks-java-extensions-sql-jdbc ?

It may be the incorrect jar to use, but I need to write the result of my HTTP calls to a database. In effect, GET a resource and then save it to a DB.

I see, its because beam-sdks-java-extensions-sql-jdbc is a fat" jar (having all dependency shaded)

It is a fat jar, but I don't believe it is shaded. I think that would help though. (If it were shaded, the packages would look more like org.apache.beam.vendored.okio. BufferedSink). However, any one who was accidentally depending on this jar would also fail at runtime with a NoClassDefError if the class was shaded. Personally, I think shading would be the reasonable choice.

@Abacn
Copy link
Contributor

Abacn commented Jul 28, 2023 •
edited
Loading

Are you using Beam's JdbcIO or Beam SQL to write data to database? If just using JdbcIO, then beam-sdks-jaca-io-jdbc is the correct dependency to use. The extension-sql-jdbc is used to run a self contained service for beam SQL (it has a main function even) so usually not meant to be used as dependency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting triage bug java P3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Abacn @carl-mastrangelo