RFC: Pipeline config

[6543]

Lets collect ideas for the pipeline config format. 🚀

We should have a critical look at the current config format and propose all possible changes to the format with the following aspects in mind:

• allowing new users to easily understand configs and write own one (=> exp. one way to solve a problem)
• be as generic and extendable as possible
• stick to standards to allow the usage of (parsing) libraries with minimal additional code required

General feature requests for the pipeline (can be implemented in the current version already if possible)

• pipeline config changes
  • remove support for root.pipeline.[step].commands string values in favor of array lists -> single string is array with one item
  • move root.platform, root.branches, root.labels, root.runs_on, root.depends_on to root.when.xxx to match step filters Use when keyword for pipeline conditions #283
  • make root.when and root.pipeline.[step].when an array of the current with items using the current settings to allow ORs Allow multiple when filters #686 (Add support for pipeline root.when conditions #770,...)
    -> non breaking
  • remove root.Cache and functionality in favor of plugins? -> need more discussion
  • allow multiple when filters Allow multiple when filters #686
  • replace group with needs if no needs is set step will start directly
  • use needs instead of group drop step.group in favour of depends_on to create a DAG #1860

from #393

Current idea:

Version could be used to run pipeline parsing with some kind of sub-program for that specific version.

Backend data format / type should not depend on docker types. For example we should not have networks as a property in steps.

version: 2

depends_on:
  - test
  - web

when:
  - branch: main
    event: push
    path: docs/
  - event: pull_request
    path: docs/

steps:
  - name: Test
    plugin: test-java:next
  - name: Build
    image: node:16-alpine
    commands:
      - npm run build
    depends_on:
      - Test

[LamaAni]

My recommendation:

when:
  - ...

images:
  runner: ubuntu:20
  maker: golang:latest

# All types show echo the yaml structure, and no "special"
# Handling. This allows for expansion without "custom" knowlage.
steps:
  - name: run
    runs_on: runner
    commands:
      - echo ok
  - name: make
    run_on: maker
    environment:
      A: b
      C: d
    secrets:
      A: b
      C: d
    commands:
      - make

Following the principles:

1. All types should echo the yaml structure, and no/minimal "special" handling.
2. Least amount of custom knowhow. WYSIWYG. (For example when is list. Always).
3. Dictionaries (like envs) should either be a Name: Value, of list of values, but always following yaml standards. e.g.

    environment:
      - name: a
        value: a
    # or
    environment:
      A: b
      C: d
    # But never
    environment:
      - A=b
      - C=d

   This would allow for use of anchors properly in values.
4. We should stick to kubernetes container definition, since everyone knows it and it proved easy over the years. You can find it in the pod help here

[6543]

we can but it would break all existing configs, what's the benefit?

you can add it to #829 as comment, but for now I don't see a benefit - if it would technically improve a ting or make something possible that was not before it should be considered - but just for the sake of taste we should not break things - call it backwards compatibility

[anbraten]

All types should echo the yaml structure, and no/minimal "special" handling.
(For example when is list. Always).

👍🏾 I really like that idea of having just one way of doing things. Having the option to use arrays or just a single string confuses be quite often and makes things like json-schema a hell to write.

[LamaAni]

It would. When we are following the yaml config we can do the following things:

1. Add default definition to a server, E.G. Allow a "default pipeline" that will be merged.
2. Easier for new people to understand.
3. Allow anchors to be used in the yaml config anywhere -> leads to more configurable yaml.

In general, the idea of version is good since it allows one to slowly migrate to a more "readable" yaml. Following standards, IMO, is the best way to expand use.

Finally, Once proper yaml exists, we can in general start considering dynamic pipelines.

[LamaAni]

Oh another thing, one can see an example of over complication by not following yaml standards in the Containers unmarshel,

	for i, n := range value.Content {
		if i%2 == 1 {
			container := Container{}
			err := n.Decode(&container)
			if err != nil {
				return err
			}

			if container.Name == "" {
				container.Name = fmt.Sprintf("%v", value.Content[i-1].Value)
			}
			c.Containers = append(c.Containers, &container)
		}
	}

Instead of

containers := []*Container{}
err := value.Decode(&containers)

Which complicates the code and causes errors. Makes things harder to manage or write.

[anbraten]

To be able to have multiple parser (for different versions) in Woodpecker I would like to do some changes to the parsing (exp. creating a clean interface what needs to be inserted and whats the output). I already started that a bit in #802.

[silverwind]

Two points from me:

1. Allow to define steps in array format (as in OP), to avoid undefined dict key iteration problems with YAML parsers. yaml pipeline vs. 2 #771 (comment).
2. Support parallelism inside a pipeline via a DAG. This requires a way to define step dependencies which can then allow the runner to build said DAG. Edit: I see group provides something similar for static fan-out, but certainly not as flexible as defining dependencies directly.

[6543]

1. is already resolved

[silverwind]

I should actually read OP, needs sounds like what I want for 2, e.g. a way to define step dependencies directly.

[gapodo]

Things that to mind would be:

• an and/or/not wrapper around conditions (recursive) to allow for more complex conditionals.
• an ignoremessage (best case array) in any item / by itself (or am I overlooking something that would allow for this currently)
• a forceMessage allowing to completely disable the when: or parts of the when: (could be a not.ignoremessage with some and: or: usage)

Use-cases would be highly conditional pipelines (possibly controlled via Commit message), which would allow for easy sub-builds (i.e. when developing woodpecker, only build OCI-images for the agent and server with the default image, so no binaries, no docks, no alpine,...)

when:
  or:
    branch:
      includes: [ release/*, main]
    tag:
      includes: [ v0.* ]
  and:
    path:
      includes: [ **.md ]
    not:
      ignoremessage: [ "no docs" ]

Also since v2 is a "breaking" version I would like to try and get as much of the major refractoring / syntax changing done as possible to not break the schema soon to add new features.

[remram44]

You can just have the job decide what is appropriate for it to do. The job runs actual code, recreating so much of it in the config is probably not necessary.

[gapodo]

maybe also relativ, like /org/repo/dir/pipeline.yaml for easier move to a new git hosting platform.

I'd tend to not use the relative path as a "git instance" path, if I'd use it at all I'd go with repo relative (allowing reuse within the repo, which may get useful when running pipelines for multiple arches, or having standard prepare steps, that are used in PR pipelines as well as test and release...)

Regarding templating, I'm fairly sure it would be the way to go, as you could import, and use the step at multiple places, reconfiguring it as you need it (changing names, just the command, the environment,...). Gitlab-CI which @genofire used as an example handles variables completely differently and in their context you can really achieve most things that way (though also not all).

[6543]

If we gona introduce imports and templates ...
two questions:

• these features could be combined into one?
• would RFC: pipeline compile step #1400 also solve this?

[6543]

To minimize the risk, it would be nice to set an version (git commit id or an git Tag). In the end, you should just import own developed or verified by an code review ... (like on all Software components you use).

it we really wana make import secure, we would have to come up with something similar to how go-modules, rust crates etc... are handled :/
-> a trusted version/hash database which woodpecker do have to check against ... and some lock mechanism

[6543]

two questions: should templates at max contain a whole pipeline, only a workflow or only steps
because as soon as it should cover more than steps we have tradeoffs:

• workflows -> we have to overwrite all other things of a workflow to get a predicted behavior
• pipelines -> well in that case I would revere to RFC: pipeline compile step #1400 totaly

[gapodo]

Cross-referencing #1504 as another proposal for the Pipeline config... (setting up envs in one step for use in another one, that may also be a plugin)

[6543]

what a pure base config not become -> scriptable (turing-complete)

that's what isolated environments are for ...

[gapodo]

two questions: should templates at max contain a whole pipeline, only a workflow or only steps because as soon as it should cover more than steps we have tradeoffs:

* workflows -> we have to overwrite all other things of a workflow to get a predicted behavior

* pipelines -> well in that case I would revere to [RFC: pipeline compile step #1400](https://github.com/woodpecker-ci/woodpecker/issues/1400) totaly

With imports I'm a bit unsure, to me I'd tend to go with the step as unit of inclusion (and potential templating, which BTW is a native yaml feature), I can see use-cases, where including more than a single step (potentially even a complete pipeline) could be useful though (think standard build pipeline, that just gets a few values set, e.g. debug-runs and release runs)

[6543]

well if its so generic, it can be covered by a plugin ... (and/or) that geneates the pipeline ... and so #1400 would cover it without adding complexity to woodpecker itselve

[gapodo]

what a pure base config not become -> scriptable (turing-complete)

that's what isolated environments are for ...

Is this a reference to #1504 or the templating / imports discussion?

If it's re #1504, I'd not call it scriptable, it's more of a way to prepare / setup an environment, which would either need to be repeated often within the pipeline, or is used by a step, that is not capable of setting up the environment as needed (think plugins)

[gapodo]

For the import step, I'd agree that it is possible to do this in a compile step (if that gets implemented) as for yaml templating it's a native yaml feature, that would "just" need a slight refractor of the pipeline config to allow for it (it's mostly, that we name the step by naming the root node instead of having step as the root for a step definition and a step.name inside it)

[gapodo]

And I need to add a correction the templating via merge keys (which I was referring to when talking about templating) is implemented in many yaml-implementations, but isn't in the yaml1.2 spec... (due to a "conflict" with one of the yaml core folks) sorry if I caused confusion...

It's documentation is here... https://yaml.org/type/merge.html

Also the go-library used by woodpecker (gopkg.in/yaml.v3) supports it. It's implemented in the decode.go (tested in decode_test.go) and is part of resolve.go.

[6543]

ah ... that's already on the todo list -> #1192

[gapodo]

ah ... that's already on the todo list -> #1192

ah sorry didn't notice that one... (it would play nicely with templating though)