Agent Cannot Connect To Server

[TheDarkula]

Component

server, agent

Describe the bug

The server seems to refuse connections on port 9000, even though it says it is listening.

System Info

2.1.1

This is running in kubernetes via a custom helm chart.

I have the server available through a traefik ingress.

The gRPC service is available to the agent as a clusterIP.

I did also try this with the `latest` image tag for both the server and the agent.

Additional context

Agent logs:

{"level":"info","time":"2024-01-20T15:16:24Z","caller":"/src/cmd/common/logger.go:100","message":"LogLevel = trace"}
{"level":"info","time":"2024-01-20T15:16:24Z","caller":"/src/cmd/agent/config.go:45","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
rpc error: code = DeadlineExceeded desc = context deadline exceeded

Given that there is no shell (scratch image) on either the server or the agent, I could not exec in and do any diagnostics.

I did run a kubectl exec ... /bin/woodpecker-server:

3:26PM INF LogLevel = info
3:26PM FTL failed to listen on grpc-addr error="listen tcp :9000: bind: address already in use"

I have also written an initContainer for the agent:

while ! nc -z woodpecker.woodpecker.svc.cluster.local 9000; do
  echo "Waiting for port 9000 to be available..."
  sleep 2
done

Unfortunately, this hangs indefinitely.

Also, your Contributing Guidelines link does not exist.

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
• Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

[zc-devs]

there is no shell

There are Alpine images:

docker run -it --entrypoint=/bin/sh woodpeckerci/woodpecker-server:v2.1.1-alpine

1. Both Server and Agent work. In Kubernetes too. Should be question in Discussions.
2. You have not provided configuration. And I mean not Values of your custom Heml, but raw yamls deployed in cluster.
3. Network policies?

[TheDarkula]

Thank you so much for the alpine image suggestion :)
I was able to sort it all out.
It was a service misconfiguration in helm.
Thank you so much!

[TheDarkula]

@zc-devs I have run into a subsequent issue.

In a gitea repository, I created .woodpecker/.build.yml:

steps:
  build:
    image: debian:stable-slim
    commands:
      - echo building
      - sleep 5

I triggered this pipeline manually, but I was greeted with this message:

check for git clone capabilities failed: exec: "git": executable file not found in $PATH

I thought it a bit odd that the runner was missing the git binary, but I shelled in, and added it with apk add git.
I ended up needing to add the git-lfs package as well, but it still failed:

exec: "debian:stable-slim": executable file not found in $PATH

Am I missing something here?

[qwerty287]

It's using the local back end, you must use docker or kubernetes

[TheDarkula]

@qwerty287 Ah, thank you!
It just straight fails now, but I did check the agent logs, and it shows this:

{"level":"error","Repo":"username/Test","Pipeline":"6","ID":"6","error":"pods \"wp-01hmm2153htb8ea1v78hq4vnnx-0-clone\" is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot delete resource \"pods\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:27:20Z","message":"could not destroy engine"}
{"level":"error","error":"rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:27:20Z","message":"grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\""}
{"level":"warn","repo":"username/Test","pipeline":"6","id":"6","error":"rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:27:20Z","message":"cancel signal received"}
{"level":"error","ID":"7","Repo":"username/Test","Pipeline":"7","error":"pods \"wp-01hmm22j9gqcnr2kemj7trh5xz-0-clone\" is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot delete resource \"pods\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:28:06Z","message":"could not destroy engine"}
{"level":"error","error":"rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:28:06Z","message":"grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\""}
{"level":"warn","repo":"username/Test","pipeline":"7","id":"7","error":"rpc error: code = Unknown desc = Step finished with exit code 1, persistentvolumeclaims is forbidden: User \"system:serviceaccount:woodpecker:default\" cannot create resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"woodpecker\"","time":"2024-01-20T18:28:06Z","message":"cancel signal received"}

[zc-devs]

ServiceAccount, Role, RoleBinding