Epic: Passing data between steps

[kminehart]

When writing Shipwright pipelines, you should be able to pass data between steps.

Currently

Currently, you can have a Step that requests a known argument.

In the step's definition you must define a list of Arguments

action := func(opts pipeline.ActionOpts) error {
    // do something...
    return nil
}

step := pipeline.NewStep(action).WithArguments(pipeline.ArgumentWorkingDir, pipeline.ArgumentCommitRef)

This allows Client implementations to read the requested arguments, like ArgumentWorkingDir, and provide them in a way that makes sense for that implementation.

For example, the Docker and CLI implementations can check to see if the argument was provided via a flag, like -arg=working-dir=$PWD, and that value can be passed into the docker run command.

For the Drone client, however, the client knows that ArgumentWorkingDir might be satisfied by an environment variable or might just be a known value like /src.

Summary

Steps are able to request / define values that are known before runtime, but can not easily load data from other steps.

Plan

The data in the Step is descriptive, while the Action function itself is what is executed. Steps must describe what arguments they require.

This can be done by using a key/value system that extends the Arguments system that's there now. We are missing the functions that allow a Step to define what arguments in provides and the mechanism for sharing those values.

In this simple example, step0 which provides "some-unique-key" runs before step1, which requires "some-unique-key".

Simple example pseudocode:

sw := shipwright.New()
defer sw.Done()

action := func(opts pipeline.ActionOpts) error {
    // At this moment, if the function is running, it is safe to assume that the argument is there and that it is of the correct type.
    value := sw.Argument(pipeline.NewStringArgument("some-unique-key")).String()
    return nil
}

step0 := pipeline.NewStep(action).Provides(pipeline.NewStringArgument("some-unique-key"))
step1 := pipeline.NewStep(action).WithArguments(pipeline.NewStringArgument("some-unique-key"))

sw.Run(step0, step1)

Docker and Drone mode

Docker and Drone modes are interesting because they do not run the entire pipeline in one pipeline execution. Each step is delegated to a separate container or "step" in Drone, and is executed individually. Each step execution does not necessarily happen after the previous one has executed. This means that the state that is populated by previous steps is not actually populated.

To solve this, we must be able to provide the pipeline with a state file that it can safely read from and modify. We will need a new pipeline argument, -state={PATH}. If not provided, then we can use os.TempDir(). In order to be able to modify this file, we may need to consider an exclusive file lock, though I think it may be a safe assumption that no other processes will be writing to this file simultaneously.

Known Arguments

Currently, there is a concept of "Known Arugments". For example, if I want my step to be provided with the current git SHA, in Docker mode I know that I can get the current SHA by most likely using git rev-parse HEAD, and in Drone I know I can read the environment variable $DRONE_COMMIT_SHA (or something like that).

Previous steps do not necessarily have to provide these values. They are defined in the pipeline package, and each Client is expected to properly satisfy each known argument. Example: CLI Client.

These should be used very sparingly as they can create assumptions about the running environment that are not necessarily true. For example, I highly doubt we could ever use the "docker socket" in a GitHub Action.

Requirements

• If a step provides an argument is ran in parallel with the step that requests it, then an error should be returned.
• If a step requests an argument that is not provided by a previous step, then an error should be returned.
• No warning should be displayed if a step provides an argument that is not used.
• Steps provided in the shipwright package should be pre-defined with provided arguments.
• Different types of arguments should expect different data. Some examples:
  • ArgumentTypeFS should provide a fs.FS type. This could easily be used in Docker scenarios with volumes, and in cloud scenarios with tarballs.
  • ArgumentTypeString should provide a string value.

Tasks

• Add function that allows Steps to define the arguments they "Provide".