Running booth arbitrator in kubernetes environment

[jfriesse]

Right now it is not possible to run booth arbitrator in docker/podman environment because of how network configuration in these environments (NAT) works. Idea is to allow such functionality.

The main problem is hidden in the fact, that:

1. Config file contains external IP, internal in the docker/podman differs
2. Changing external IP to internal IP on both site and arbitrator doesn't work because sites cannot reach internal IP of arbitrator
3. Changing external IP to internal only on arbitrator site mostly works, because sites can reach arbitrator and arbitrator can find itself, sadly it will generate different site_id and message sent from arbitrator will be ignored by sites.

As a possible solution we (probably) need to enhance file so it will contain two addresses of arbitrator (one internal and one external) and arbitrator will use external as an site_id. I think it might be handy to allow specify internal IP as an ANY so user don't need to find out internal IP (not super easy in docker environment).

So proposed solution is to have sites like:

authfile = /etc/booth/booth.key
site = site_ip
site = site_ip
arbitrator = arbitrator_external_ip
ticket = "apacheticket"

and arbitrator as:

authfile = /etc/booth/booth.key
site = site_ip
site = site_ip
arbitrator = arbitrator_external_ip|ANY
ticket = "apacheticket"

or some flag like force_arbitrator_mode_bind_in_any or maybe different (better) solution.

Example how to test in docker (copy&paste from original report):

Arbitrator running inside docker container tries to send UDP packet to
booth site - but this UDP packet gets dropped after getting out of docker
bridge (on host machine), whereas non-arbitrator UDP packets reach
destination booth sites perfectly. Issue is observed only with arbitrator
UDP packets.

Steps to reproduce:

   - Extract the zip file and cd into dockerfile directory
   - docker build -t arbitrator .
   - docker run -d --privileged arbitrator
   - docker ps (check CONTAINER ID for arbitrator container)
   - docker exec -it <container-id> bash
   - Once in docker container fire below commands
   - /bin/supervisord
   - pcs cluster auth <booth-ip>
   - pcs booth pull <booth-ip>
   - replace arbitrator ip (from /etc/booth/booth.conf) with eth0 ip (to
   check eth0 ip fire "ip address show" command)
   - supervisorctl start booth

Note: We are running centos7 in docker container, so supervisord is used
instead of systemd/systemctl (as systemd does`t work inside container/k8s
pod)

dockerfile.zip