Output Data Prefix

[SandraGH5]

Can someone share what is:

--output-location-prefix ${OUTPUT_DATA_PREFIX}

and whether this is required for some tasks? Like map tasks?

@migueltol22

[SandraGH5]

This is the prefix where offloaded types will upload their raw data.

For example, if you return a FlyteFile (or a Blob) representing a local file you generated in a task, Flyte will upload the contents of that file to the output location prefix (under a new sub-prefix) and then return a reference to that location as the output of the task.

So it is needed for certain types (as opposed to certain tasks), namely Schema (DataFrames... etc.) and Blobs (Files/Directory... etc.)

@EngHabu