RFC: pipeline compile workflow/(step)

[6543]

We want to allow admins of a repo to use custom compilers.

It should be possible to:

• use a customer compiler per repo
• the custom compiler should be able to use non-yaml config files (maybe allow to not add any woodpecker specific config files at all)
• the compiler should be executed / started and stopped by woodpecker without the need to host it externally etc (exp no externally hosted http service)

Open questions

• How to execute the compiler (exp running as backend step)? How can this support all backends?
• Where should the compiler be configured?
  • admin settings of the repo
  • in the repo yaml config => would require to add a config)
• How can we pass and get data from the compiler?
  • env vars
  • shell input and shell output
  • files
  • http
• What data should be passed to the compiler?
  • woodpecker yaml configs (would not allow to run woodpecker in a repo without woodpecker config exp read-only forks)?
  • netrc (clone credentials)?
  • woodpecker api token?
• Should the compiler output woodpecker yaml configs or the internal pipeline backend config directly?

[6543]

Add a own section called compile into pipeline config.

which uses normal workflow logic (default or custom clone) ... but it's output is no log but an pipeline config.

tldr: with way we could support starlark, jsonnet (#1396), drone >= 1.0, ... without needing to add it to woodpecker core

details

we either could make a special compile pipeline, and show it as own entry, witch trigger the actual pipeline with its config
(if we do so, we need to somehow link them UI/UX wise ...)
or we just use our multi workflow approach ... where one workflow (eventually with a different icon) is the compile and all generated workflows from it are also i it ... (as a compile could create one or more workflows)

we need to make sure we dont have a recursion in it -> see permutate example

example:

compile:
  image: wp/jsonnet
  settings:
    file: woodpecker.jsonnet

compile:
  image: wp/replace-image # permutate normal wp pipeline config

pipeline:
  normal-steps:
...

workaround:

use a Configuration Service

be aware as we would support with multible workflows multible compiles ... that this get's complicated quite fast

[anbraten]

@6543 I've updated the description to search contain the needs and open questions and moved your description to this comment so we can discuss your proposal and add other proposals and vote and develop a final solution.

[anbraten]

#3349 implements an option to specify a config service allowing to compile a pipeline using the requested features.

[6543]

we also might add a server flag to ignore this feature

[gapodo]

I like the idea, this would also allow one to "emulate" some of the features of GitHub workflows by basically overriding a step declared with just a name and e.g. "fake" image with a step with all settings prepared (basically a plugin + the necessary config / commands)... which would increase code reusability and also ease updating a step in all repos (e.g. switching to a newer buildx plugin or changing defaults like proxy settings to match the environment)

[lafriks]

would be nice to be able to specify global (or org wide) permutate image that would by applied than to all respective pipelines

[wiktor-k]

This feature looks really great! Even if it was super-simple: generating entire woodpecker.yml using a docker image this would already provide enough tooling to write virtually any integration one can imagine. Thanks for thinking ahead and for your work 👍

[bkleiner]

What would be the input/output scheme of this compile step?
stdin/stdout seems like a obvious choice, however would prevent the compile step outputting logs to the user and would make it rather hard to take advantage of multi-workflows.

maybe an approach to how github workflows handles output could be taken, eg:

the env-var $COMPILE_INPUT holds the regular pipeline for mutation.
$COMPILE_OUTPUT holds the path to a folder where the compile-step can write files to, which then are handled the same way as the .woodpecker folder normally would, basically overwriting what is delivered by the csv.

adding this feature would make woodpicker a viable alternative for us, as we rely on jsonnet for some of our pipelines.

[ambroisie]

stdin/stdout seems like a obvious choice, however would prevent the compile step outputting logs to the user and would make it rather hard to take advantage of multi-workflows.

The standard is to use stderr for user-oriented output in that case.

adding this feature would make woodpicker a viable alternative for us, as we rely on jsonnet for some of our pipelines.

I had a good experience using the matrix builds in tandem with the when.evaluate option to replace jsonnet in my use-case. But that doesn't get you very far.

[bkleiner]

The standard is to use stderr for user-oriented output in that case.

well, yeah, in theory, tho i think you gonna have a very hard time to make all programs a user would potentially want to use comply with that.

[pinpox]

Isn't that something that can be solved with redirection for non-complying programs?

[pinpox]

I would propose to implement this as a generalization of the External configuration API. In fact, it might be able to replace that API, as issuing a POST to receive a pipeline configuration (as the API currenty does) would just be a special case of a new pipeline compilation step.

The current workaround mentioned above does not work for private repositories or setups where the configurtion service can't pull the repo if that is needed to generate the pipeline configuration. I'm specifically running into this issue with the configuration service I build for nix flakes and private repositories.

The API could also be extended to pass credentials in the POST, but a more generalized compile step like proposed here, would be a better and cleaner approach in my opinion.

I don't know however, how this would look in the UI. It would make sense to have some dedicated step shown, with logging and status and so on to allow for debugging.

[anbraten]

@pinpox I just updated the description with our needs and open questions. Would be cool if we could collect some proposals which try to solve the description and select the best one to implement. Please let me know if I miss sth in the description.

[pinpox]

• Should the compile step run before, after or alongside the clone step?
• I'd generalize the "should we pass the yaml" to: What data from the repository (e.g. yaml, but also other files) should be passed to the compile step? We might want to read files of the repository when creating the pipeline

[anbraten]

My current proposal to solve this

In the repo-settings an admin can define docker image names for the config, secrets and repositories service (similar to #915). Those service will be automatically started when needed. The config service for example when a pipeline should be created / compiled from some config im the repo.

The service is started by an agent similar to a step / pipeline service and receives a http request from woodpecker via an internal network. The service replies to woodpecker with its config data and the agent stops the service again.

[anbraten]

Would the only difference to the current external configuration service be, that it is started and stopped by the agent?

Kind of yes.

Would it be possible to use non-docker services, i.e. run a local command?

That gets tricky TBH. Kubernetes will work, local and ssh would need to interpret the config image (maybe a better / more generic term than image should be choosen) and use a locally installed tool / script which will locally open a port / unix-file-socket and do the same as the docker service. It could be a good idea to add a function to the backend which will do the config service starting, and config fetching and can do whatever that backend is capable of internally.

How do we pass the repo's contents to the service via HTTP?

I would pass netrc as propose by you in #2268, but no files. Getting the config files would be up to the config service.

[pinpox]

What about the following idea:

Instead of the agent starting/stopping a service or docker container, just do the following:

• Start listening on a specific port/endpoint
• Run a specially named step of from the .woodpecker.yml file. The repository would still include this file and have a reserved name that identifies this "special" step. Pass the listening endpoint that is still open via an environment var to that step.
• The step is supposed to issue a POST to the endpoint with the configuration.
• When a POST is received, close the endpoint again and use the received configuration.

My reasoning behind this: The agent does not need to start/stop services specifically, we already have a way of starting things - pipeline steps. Also, this allow to place the special step after other steps, e.g. after the clone. Temporarily opening and then closing a listening endpoint seems easier than to start and stop services.

A very minimal config that would just POST an existing config defined in a file called pipeline.json would look like this:

 steps:
   woodpecker-compile-config:
     image: debian
     commands:
       - curl -X POST -H 'Content-Type: application/json' -d @./pipeline.json $CI_PIPELINE_ENDPOINT

Where $CI_PIPELINE_ENDPOINT would be the endpoint temporarily opened by the agent

Just a first thought, maybe I'm missing something

[6543]

well as for config I would add a optional extra section same as with clone

[anbraten]

Not sure if this would help. For example if I want to use starlark configs I would need a yaml config to somehow parse my yaml config first to then get a new pipeline compiled out of it, so it would create two pipelines or some kind of intermediate one which sounds pretty complicated.

[pinpox]

Hm, maybe you are right. I just thought it might be cleaner than starting and stopping stuff.

I that case, I'll have to think about what the benefits over the existing external config service solution + adding the netrc to it actually would be.

If we are not passing anything to this new config task, I might as well render a starlark config e.g. in the current external services. Am I missing something or would it just do the same thing but Start and stop every time?

[mcm]

Not sure if this is still being decided, but... what if instead of an explicit "compile" step, there was a mechanism in place for any step to generate additional steps to be run? Because of steps running in parallel, its probably not reasonable to allow a step to modify the whole pipeline, but being able to just add additional steps might solve this problem.

[pinpox]

This would be enough for all my use cases. If it is easier to implement, I'd say go for it.

[6543]

it's not easyer to implement ... as the server "creates" the steps ... and we explizite want to use only the agents to execute arbitrary code.

[jagerman]

I feel like the perfect is the enemy of the good here.

It's been more than a year since #1396 was opened (and closed), which was a fairly small PR, and yet would have allowed people with jsonnet setups to move away from drone-ci to woodpecker. Meanwhile there has been a year of sporadic discussion with no visible traction to a solution on the ideal solution (doing this generically). So yeah, a generic implementation would certainly be great, but until that comes along can we not have something like #1396 to unblock those of us who really want to move from drone-ci to woodpecker but can't because of this blocker?

[mcm]

That would be incredibly helpful!

[6543]

if you prever microservice architecture there also is https://woodpecker-ci.org/docs/next/administration/external-configuration-api right now

[6543]

And I rather want to have it in 90% perfect, as to have code we cant get rid of later anymore (... KISS pricipe pay out in the long run - and i think we are here for the long run not the short one ;) )

[6543]

PS: the priority was just elswhere until now ... you just have to check the huge changelogs between versions

if you want to speedup the process - just draft a pull request ;)

[qwerty287]

Here you go for the addon way: https://codeberg.org/qwerty287/woodpecker-jsonnet-addon

You need to compile yourself:

1. Clone source
2. If you're not using 2.1.1: Install correct Woodpecker version with go get go.woodpecker-ci.org/woodpecker/v2@<version>
3. Compile with go build -buildmode plugin
4. Use the output as addon.

I just wrote that to give you an example, it's completely untested from my side.

(I might improve this in future.)

This addon does not support private repos yet.

[6543]

ok i now got a exec plan:

1. a workflow can contain compile key ...
2. ... if a pipeline is multi workflow, only one can contain a workflow with compile
3. ... a compile workflow respekts the skip_clone
4. the build in environment variables are cleanly split into 3 scopes: those available at compile time; parse time and run time (an env var can be present in one or multible)
5. a compile step within the compile workflow gets the config via CI_PIPELINE_CONFIG (json)
6. the matrix feature will be turned into a compile plugin and for backwards compatibilety auto added if detected
7. where we previous had the code parst where we applyed the matrix permutation, we hook in the code for compile config
8. a compile workflow can output multible workflows.
9. all workflows will be run afterwards as usual, merged with the generated one (from compile workflow)
10. if a compile step outputs a workflow with the same name it will overwrite the old workflow with same name

to start implementing it:

1. we first make the matrix-workflow-mutation-func async and behave like it would be external
2. create a matrix mutation compile plugin
3. add new infrastructure for compile workflow stuff and
4. make the matrix compile use the new infrastructure

[6543]

as for example config, it would look like:

compile:
  image: woodpeckerci/compile-matrix

steps:
    test:
      image: golang:${GO_VERSION}
      commands:
        - go test ./...
    build:
      image: golang:${GO_VERSION}
      commands:
        - go build -v -o myapp

matrix:
  GO_VERSION:
    - 1.19
    - 1.20

when:
  branch: main

[lafriks]

imho matrix should go into compile step as it's an direct option for compiler, this way we can declare compiler yaml to have map[string]any param so that both linter and extendability for compile would not hurt

[6543]

exactly my point