Retrieving run error after executing a job using WorkspaceClient

[ahmadnazeri]

We are using databricks-sdk-go to execute a Python Task Job. And when it fails, we want to retrieve the error.

We used the following code in order to retrieve the output.

// client is a WorkspaceClient
output, err := client.GetRunOutput(ctx, request)

Unfortunately, when we try to retrieve the error message, we get the following error.

"message": "Retrieving the output of runs with multiple tasks is not supported. Please retrieve the output of each individual task run instead."

Currently, the work around that we have in placed is using the Databricks client with API v2.0.

var runOutput jobs.RunOutput
path := "/api/2.0/jobs/runs/get-output"
err = db.errorMessageClient.Do(ctx, http.MethodGet, path, request, &runOutput)
if err != nil {
	return nil, fmt.Errorf("could not get run output: %v", err)
}

Wondering if there is a way to retrieve run output using the WorkspaceClient. If not, would there be support in the future?

[nfx]

@ahmadnazeri could you list runs of subtasks for that specific job run?

sorry for not noticing this issue earlier.

[KoningJasper]

@ahmadnazeri

When you create and execute a job it consists of a job-run and 1 to 100 task-runs. The job-run itself doesn't have any output it's simply a holder status for the collection of all your task-runs. Therefore, API 2.1 will return this error message when you try to retrieve the output of the job-run, since it's only a holder. For API 2.0 we added a compatibility layer, if your job-run has only a single task-run we will retrieve it's output.
In-order to retrieve the output using API 2.1 you have to call it with the run-id of the task-run (NOT the job-run). You can retrieve this run-id by calling get-run. That will show the tasks array containing all the task-runs.

[marwansalem]

same as @KoningJasper
here's the code to this

run, err := client.Jobs.GetRun(d.Context, jobs.GetRunRequest{IncludeHistory: true, RunId: runID})
if err != nil {
    log.Printf("Failed to fetch job run %v", err)
    return
}
task := run.Tasks[0]
// use task.RunId to get output for the task
output, err := d.DatabricksClient.Jobs.GetRunOutputByRunId(context, task.RunId)

From the job run in the UI, you can see there is the Job ID, Job run ID, and Task run ID.
Use the Task run ID

[ahmadnazeri]

Thanks @KoningJasper for the explanation and thanks @marwansalem for the code snippet!