[CDAP-21070] Introduce error details provider to get more information about exceptions from plugins

[itsankit-google]

Tested in CDAP Sandbox:

GCS Source

Caused by: io.cdap.cdap.api.exception.WrappedStageException: Stage 'NYT Best Sellers Raw Data' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Splitting'. Error message: xxxxx.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
	at io.cdap.cdap.etl.common.ErrorDetails.handleException(ExceptionUtils.java:77)
	at io.cdap.cdap.etl.spark.io.StageTrackingInputFormat.getSplits(StageTrackingInputFormat.java:55)
	at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:136)
	..................................
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Splitting'. Error message: xxxxxx.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
	at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:186)
	at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:161)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getProgramFailureException(GCPErrorDetailsProvider.java:80)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getExceptionDetails(GCPErrorDetailsProvider.java:52)
	at io.cdap.cdap.etl.common.ErrorDetails.handleException(ExceptionUtils.java:75)
	... 66 common frames omitted
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/cdf_example/o?delimiter=/&fields=items(bucket,name,size,updated),prefixes,nextPageToken&includeTrailingDelimiter=true&maxResults=5000
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "xxxxxx.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).",
    "reason" : "forbidden"
  } ],
  "message" : "xxxxx.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist)."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)

GCS Sink

Caused by: io.cdap.cdap.api.exception.WrappedStageException: Stage 'GCS2' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Validating Output Specs'. Error message: xxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
	at io.cdap.cdap.etl.common.ExceptionUtils.handleException(ExceptionUtils.java:77)
	at io.cdap.cdap.etl.spark.io.StageTrackingOutputFormat.checkOutputSpecs(StageTrackingOutputFormat.java:67)
	at io.cdap.cdap.etl.common.output.MultiOutputFormat.checkOutputSpecs(MultiOutputFormat.java:114)
	.......................................
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Validating Output Specs'. Error message: xxxxxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
	at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:186)
	at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:161)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getProgramFailureException(GCPErrorDetailsProvider.java:80)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getExceptionDetails(GCPErrorDetailsProvider.java:52)
	at io.cdap.cdap.etl.common.ErrorDetails.handleException(ExceptionUtils.java:75)
	... 47 common frames omitted
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/cdf_example/o/2024-10-15-08-53?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "xxxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).",
    "reason" : "forbidden"
  } ],
  "message" : "xxxxxxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist)."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)

GCS Multi Sink

Caused by: io.cdap.cdap.api.exception.WrappedStageException: Stage 'GCS Multi File' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Committing'. Error message: xxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
	at io.cdap.cdap.etl.common.ExceptionUtils.handleException(ExceptionUtils.java:77)
	at io.cdap.cdap.etl.spark.io.StageTrackingOutputCommitter.setupJob(StageTrackingOutputCommitter.java:57)
	at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:188)
	...............
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Committing'. Error message: xxxxxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
	at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:186)
	at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:161)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getProgramFailureException(GCPErrorDetailsProvider.java:80)
	at io.cdap.plugin.gcp.common.GCPErrorDetailsProvider.getExceptionDetails(GCPErrorDetailsProvider.java:52)
	at io.cdap.cdap.etl.common.ErrorDetails.handleException(ExceptionUtils.java:75)
	... 47 common frames omitted
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: null
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageExceptions.createJsonResponseException(GoogleCloudStorageExceptions.java:89)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$9.onFailure(GoogleCloudStorageImpl.java:1944)
	at com.google.cloud.hadoop.gcsio.BatchHelper.execute(BatchHelper.java:199)
	at com.google.cloud.hadoop.gcsio.BatchHelper.lambda$queue$0(BatchHelper.java:178)
	

[sonarcloud]

Quality Gate failed

Failed conditions
47.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud