kind not working in codepaces

[pieterv-icloud-com]

Trying to create a kind cluster in codespaces

What happened:

Running kind create cluster --config ./scripts/kind-config.yaml --retain

Gives the following error:

Creating cluster "kind" ...
ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-5d5f5e9ee3aa -j MASQUERADE: ip6tables v1.8.10 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

Running kind export logs produces not logs:

ERROR: unknown cluster "kind"

What you expected to happen:

Expecting a kind cluster as when running this locally,

How to reproduce it (as minimally and precisely as possible):

We have the following features in devcontainer.json:

		"ghcr.io/devcontainers/features/docker-in-docker:2": {
			"version": "latest",
			// https://github.com/devcontainers/features/issues/743
			"dockerDashComposeVersion": "v2"
		},
		"ghcr.io/devcontainers/features/kubectl-helm-minikube:1": {
			"version": "latest",
			"minikube": "none"
		},
		"ghcr.io/mpriscella/features/kind:1": {
			"version": "latest"
		},

Dockerfile:

# Note: You can use any Debian/Ubuntu based image you want. 
ARG VARIANT="buster"
FROM mcr.microsoft.com/devcontainers/base:dev-${VARIANT}

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends curl python3-dev python3-pip cmake coreutils openvpn jp2a jq dnsutils iptables build-essential

RUN mkdir /run/openvpn 

# Switch User to the user we will be running the devcontainer as
ARG USERNAME=vscode
USER $USERNAME
RUN sudo chown -R $USERNAME:$USERNAME /home/$USERNAME \
    && sudo mkdir -p /home/$USERNAME/bin \
    && sudo mkdir -p /home/$USERNAME/.local/state \
    && sudo mkdir -p /home/$USERNAME/.local/share \
    && sudo chown -R $USERNAME:$USERNAME /home/$USERNAME/.local/share \
    && sudo chown -R $USERNAME:$USERNAME /home/$USERNAME/.local/state

# mkdocs
# Workaround: https://www.jeffgeerling.com/blog/2023/how-solve-error-externally-managed-environment-when-installing-pip3
COPY requirements.txt .
RUN sudo rm /usr/lib/python*/EXTERNALLY-MANAGED \
    && sudo pip install --no-cache-dir --no-input -r requirements.txt

# bash history
# ARG USERNAME=vscode
# RUN SNIPPET="export PROMPT_COMMAND='history -a' && export HISTFILE=/commandhistory/.bash_history" \
#     && echo $SNIPPET >> "/home/$USERNAME/.bashrc"   

# oh-my-posh
ARG USERNAME=vscode
ARG TARGETPLATFORM
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE=amd64; elif [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then ARCHITECTURE=arm; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE=arm64; else ARCHITECTURE=amd64; fi \
    && sudo wget https://github.com/JanDeDobbeleer/oh-my-posh/releases/latest/download/posh-linux-${ARCHITECTURE} -O /home/${USERNAME}/bin/oh-my-posh \
    && sudo chmod +x /home/${USERNAME}/bin/oh-my-posh /    

# eksctl
ARG TARGETPLATFORM
ARG USERNAME=vscode
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE="amd64"; elif [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then ARCHITECTURE="arm"; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE="arm64"; else ARCHITECTURE="amd64"; fi \
    && sudo curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_$ARCHITECTURE.tar.gz" \
    && sudo tar -xzf eksctl_Linux_$ARCHITECTURE.tar.gz -C /tmp  \
    && sudo rm eksctl_Linux_$ARCHITECTURE.tar.gz \
    && sudo chmod +x /tmp/eksctl \
    && sudo mv /tmp/eksctl /home/${USERNAME}/bin

# ORAS cli
ARG TARGETPLATFORM
ARG ORAS_VERSION="1.0.0"
ARG USERNAME=vscode
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE="amd64"; elif [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then ARCHITECTURE="armv7"; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE="arm64"; else ARCHITECTURE="amd64"; fi \
    && sudo curl -LO "https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_${ORAS_VERSION}_linux_${ARCHITECTURE}.tar.gz" \
    && sudo mkdir -p oras-install \
    && sudo tar -zxf oras_${ORAS_VERSION}_*.tar.gz -C oras-install \
    && sudo mv oras-install/oras /home/${USERNAME}/bin \
    && sudo rm -rf oras_${ORAS_VERSION}_*.tar.gz oras-install 

# gitversion
ARG TARGETPLATFORM
ARG GIT_VERSION="5.12.0"
ARG USERNAME=vscode
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE="x64"; elif [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then ARCHITECTURE="armv7"; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE="arm64"; else ARCHITECTURE="x64"; fi \
    && sudo curl -LO "https://github.com/GitTools/GitVersion/releases/download/${GIT_VERSION}/gitversion-linux-${ARCHITECTURE}-${GIT_VERSION}.tar.gz" \
    && sudo mkdir -p gitversion-install \
    && sudo tar -zxf gitversion-linux-${ARCHITECTURE}-${GIT_VERSION}.tar.gz -C gitversion-install \
    && sudo mv gitversion-install/gitversion /home/${USERNAME}/bin \
    && sudo chmod +x /home/${USERNAME}/bin/gitversion \
    && sudo rm -rf gitversion_${GIT_VERSION}_*.tar.gz gitversion-install 

# clusterctl
# curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.6.1/clusterctl-linux-amd64 -o clusterctl
ARG TARGETPLATFORM
ARG CLUSTERCTL_VERSION="1.7.4"
ARG USERNAME=vscode
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE="amd64"; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE="arm64"; else ARCHITECTURE="amd64"; fi \
    && sudo curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v${CLUSTERCTL_VERSION}/clusterctl-linux-${ARCHITECTURE} -o clusterctl \
    && sudo mv clusterctl /home/${USERNAME}/bin \
    && sudo chmod +x /home/${USERNAME}/bin/clusterctl

# goodhosts
ARG TARGETPLATFORM
ARG GOODHOSTS_VERSION="1.1.2"
ARG USERNAME=vscode
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then ARCHITECTURE="amd64"; elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then ARCHITECTURE="arm64"; else ARCHITECTURE="amd64"; fi \
    && sudo curl -L https://github.com/goodhosts/cli/releases/download/v${GOODHOSTS_VERSION}/goodhosts-${GOODHOSTS_VERSION}-linux-${ARCHITECTURE}.tar.gz -o goodhosts.tar.gz \
    && sudo tar -zxf goodhosts.tar.gz \
    && sudo mv goodhosts /home/${USERNAME}/bin/goodhosts \
    && sudo chmod +x /home/${USERNAME}/bin/goodhosts \
    && sudo rm -rf goodhosts*

Kind configuration:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
  ipFamily: ipv4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
    labels:
      tier: application
  - role: worker
    labels:
      tier: database

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.23.0 go1.21.10 linux/amd64
• Runtime info: (use docker info, podman info or nerdctl info):

Client:
 Version:    27.0.3-1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 27.0.3-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e9e2c7707933f32aa891dda794a1df36a6ec7aee
 runc version: 58aa9203c123022138b22cf96540c284876a7910
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1022-azure
 Operating System: Ubuntu 24.04 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB
 Name: codespaces-555c1e
 ID: f5158680-804d-474a-bbad-72d47b438f67
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: codespacesdev
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release): Ubuntu 24.04 LTS
• Kubernetes version: (use kubectl version): v1.30.3
• Any proxies or other special environment settings?: No

[aojea]

you are missing one kernel module that is required ip6tables

Unable to enable NAT rule: (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-5d5f5e9ee3aa -j MASQUERADE: ip6tables v1.8.10 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

Operating System: Ubuntu 24.04 LTS (containerized)

[pieterv-icloud-com]

Could you maybe tell me how I would do this?

[aojea]

I;m not familiar with this environment, but based on tailscale/tailscale#3002 it seems it depends on the project to enable them, so you may need to ask for this feature to enable this kernel module https://github.com/features/codespaces

[pieterv-icloud-com]

Thx, opened a support ticket with GitHub

[BenTheElder]

Note that docker v27 is enabling IPv6 by default #3677 so it's probably best for github to enable these modules even if the hosts are themselves ipv4 only.

https://docs.docker.com/engine/release-notes/27.0/#ipv6

[pieterv-icloud-com]

I created a debian bookworm vm in gcp, used vscode tunnel to connect to it, rebuild the devcontainer and tried to create kind cluster.

Got the same error:

ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-ebe302d9acd5 -j MASQUERADE: ip6tables v1.8.9 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

I then disabled the ip6tables in the docker daemon.json file:

{
  "ip6tables": false
}

Did not work, also tried --ip6tables=false but docker told me it's an unknown parameter.

[stmcginnis]

Make sure you have Docker engine v27. But even if you could disable it for docker, that doesn't really get you far. You will still need the ipv6 kernel module for running kubernetes.

[pieterv-icloud-com]

Only to fix it was with

sudo modprobe ip6_tables

When the vm starts

[pavelpi]

@pieterv-icloud-com
Could you please post here link to Github ticket?