Agent keeps dying in docker-compose (podman)

[Fogapod]

Component

agent

Describe the bug

I have similar issue to this one: #2575 except their fix does not work. I took example docker compose from documentation and only set forgejo env vars.

I am running docker compose using podman: podman compose --env-file=.env up.
Podman has socket via podman.socket user service at /run/user/1234/podman/podman.sock which i mount as docker socket as suggested in #85.

Config:

version: '3'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:${WOODPECKER_TAG}
    ports:
      - 8002:8000
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=${WOODPECKER_HOST}
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=${WOODPECKER_GITEA_URL}
      - WOODPECKER_GITEA_CLIENT=${WOODPECKER_CLIENT}
      - WOODPECKER_GITEA_SECRET=${WOODPEKCER_SECRET}
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:${WOODPECKER_TAG}
    command: agent
    restart: always
    depends_on:
      - woodpecker-server
    volumes:
      - woodpecker-agent-config:/etc/woodpecker
      - ${XDG_RUNTIME_DIR}/podman/podman.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}


volumes:
  woodpecker-server-data:
  woodpecker-agent-config:

WOODPECKER_TAG=v2.3.0
WOODPECKER_HOST=https://woodpecker.example.org
WOODPECKER_GITEA_URL=https://git.example.org
WOODPECKER_CLIENT=forgejo_client
WOODPEKCER_SECRET=forgejo_secret
WOODPECKER_AGENT_SECRET=supersecret

I tried adding WOODPECKER_BACKEND=docker env to server and agent, setting WOODPECKER_GRPC_ADDR=:9000 in server, tried next tag, none of these worked after container, network and volumes purges.

Not sure if related to agent failures but i cannot finish oauth via forgejo either. Authorization page shows up, after allowing access to account, this error happens:

woodpecker-server-1  | {"level":"error","error":"Post \"https://git.example.org/login/oauth/access_token\": dial tcp: lookup git.example.org on 10.89.0.1:53: read udp 10.89.0.27:36500->10.89.0.1:53: read: connection refused","time":"2024-02-05T15:39:49Z","message":"cannot authenticate user"}

Forgejo shows successful oauth in account that can be revoked after this but woodpecker is stuck on login page with Error while authenticating against OAuth provider

If it matters, my nginx config is:

    upstream woodpecker {
        server 127.0.0.1:8002;
    }

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        server_name woodpecker.example.org;

        location / {
            proxy_pass http://woodpecker;
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_buffering off;
            chunked_transfer_encoding off;

            proxy_set_header host               $host;
            proxy_set_header x-real-ip          $remote_addr;
            proxy_set_header x-forwarded-for    $proxy_add_x_forwarded_for;
            proxy_set_header x-forwarded-proto  $scheme;
        }

        ssl_certificate /etc/letsencrypt/live/woodpecker.example.org/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/woodpecker.example.org/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }

This is podman network info:

$ podman network inspect woodpecker_default
[
     {
          "name": "woodpecker_default",
          "id": "07531aeab2deda39dafa389c3e664138801a2a720fe048bb261c2f67379b9ddb",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-02-05T16:25:46.510521179+01:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "com.docker.compose.network": "default",
               "com.docker.compose.project": "woodpecker",
               "com.docker.compose.version": "2.24.5"
          },
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

System Info

`2.3.0`

Additional context

woodpecker-server-1  | {"level":"info","time":"2024-02-05T15:34:17Z","message":"log level: info"}
woodpecker-agent-1   | {"level":"info","time":"2024-02-05T15:34:17Z","message":"log level: info"}
woodpecker-agent-1   | {"level":"info","time":"2024-02-05T15:34:17Z","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
woodpecker-server-1  | {"level":"info","time":"2024-02-05T15:34:17Z","message":"starting Woodpecker server with version '2.3.0'"}
woodpecker-agent-1   | {"level":"fatal","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2024-02-05T15:34:22Z","message":"error running agent"}
woodpecker-agent-1 exited with code 0
woodpecker-agent-1   | {"level":"fatal","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2024-02-05T15:34:27Z","message":"error running agent"}
woodpecker-agent-1 exited with code 0
woodpecker-agent-1   | {"level":"fatal","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2024-02-05T15:34:33Z","message":"error running agent"}
woodpecker-agent-1 exited with code 0

If i put something stupid into WOODPECKER_SERVER, error changes:

woodpecker-agent-1   | {"level":"warn","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.89.0.1:9000: connect: connection refused\"","time":"2024-02-05T16:06:14Z","message":"cannot connect to server, retrying in 2s"}

Validations

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]

[pat-s]

The agent is dying because it can't connect to the server at port 9000.

Try the alpine image, start with a sleep and see if there is a connectivity to the server from inside the agent container.

You likely don't want use a 10.x IP but either a hostname (if you have a network) or your docker/podman gateway IP to talk to a port exposed to the host.

[qwerty287]

closing as no response from opener within a month, please comment if you still have issues