How to create PersistantVolume (ReadWriteOnce) on the node that runs the workflow? (Kubernetes backend)

[itytophile]

Hi. I am running Woodpecker on a little k3s cluster. The workflows run correctly, however they always run on the same node (I have two nodes).

I've set the nodeSelector option in woodpecker.yml to force the workflow to run on the node I want. The newly created pod wants to run on the good node but the PersistantVolume is created on the other node. The PersistentVolume uses the ReadWriteOnce access mode, so the workflow pod can't access it and stays "pending" (they are not on the same node).

Pending pod description:

Node-Selectors:              kubernetes.io/arch=arm64
                             merde=merde
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  19m                  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  9m35s (x2 over 14m)  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..

From the PersistantVolume:

Node Affinity:     
  Required Terms:  
    Term 0:        kubernetes.io/hostname in [not-the-good-node]

Good node description:

Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    has-space=true
                    kubernetes.io/arch=arm64
                    kubernetes.io/hostname=the-good-node
                    kubernetes.io/os=linux
                    merde=merde
                    node.kubernetes.io/instance-type=k3s

My woodpecker.yml:

steps:
  coucou:
    image: busybox:musl
    commands:
      - echo coucou
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
        nodeSelector:
          merde: merde

[zc-devs]

1. Do you use Rancher's Local Path Provisioner?
2. How many Woodpecker Agents do you use?
   2.1 If one, which node does it run on?
3. How many Workflows per agent is set?
4. Make sure that Cancel previous pipelines is not set.
5. Make sure you are running a few pipelines simultaneously.

[itytophile]

1. Yes I do
2. Two, they both run on the same node (the good node)
3. One
4. Done
5. Done

[zc-devs]

2. It is strange. They should spawn across nodes. Can you test the case when Agents run on different nodes? Can you attach also full definition of Pods, PV, PVC as files? Especially I want to look at volumeBindingMode, but other info might helpful also.

[itytophile]

That's on me. There is a nodeSelector (I was testing things) on the agent pods that's why they are on the same node. I'll change that and retry running a workflow.

[itytophile]

Here are the files. With one agent on each node. I didn't edit the nodes name this time: the "good" node is "renegade-one" and the other one is "nanopi.lan". The secret has been edited.
workflow-pod.txt
server.txt
pvc.txt
pv.txt
agent-2.txt
agent-1.txt

[zc-devs]

K3s v1.29.0+k3s1, 3 nodes
Woodpecker 2.2.2, 1 server, 1 agent with max workflows 4

I ran pipeline four times

and workflows were spawned across three nodes

At start of pipeline workspace volume is created. Then repository cloned into it. Then it mounts into each step also.
Volume is created on node, where first pod runs (binding). Rest of the pods are scheduled on the same node due to affinity, that LocalPathProvisioner sets.
So, seems works as intended.

I've set the nodeSelector option in woodpecker.yml to force the workflow to run on the node I want. The newly created pod wants to run on the good node but the PersistantVolume is created on the other node.

That also seems logical. Local volume was created on one node. You created pod on another. Pod cannot mount local volume from first node.

they always run on the same node (I have two nodes)

And you have two agents. Looks like an issue with Server->Agent scheduling.

• Is there some documentation on how tasks are scheduled to agents? What's the algorithm? Can I assign some priorities on agents other than select by labels or exclude agent from taking tasks completely?
• Well priorizing is an open issue 😅 The algo is fifo

Maybe related: Kubernetes: smart(er) agent affinity

You supposed to be able to disable Agents, but agent option "stop agent from taking new tasks" not working. I just delete unnecessary Agent, run pipeline via the second and then restart the first.

In your case, it is, probably, simpler to use one agent and give away scheduling to Kubernetes.