FileType xfs Mount failed: exit status 32 wrong fs type, bad option, bad superblock on

[tanguofu]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]

I1227 08:59:54.054947       1 mount_linux.go:366] Disk "/dev/datavg/pvc-0b6c8388-c194-463d-85f8-188e63697023" appears to be unformatted, attempting to format as type: "xfs" with options: [/dev/datavg/pvc-0b6c8388-c194-463d-85f8-188e63697023]
I1227 08:59:54.347269       1 mount_linux.go:376] Disk successfully formatted (mkfs): xfs - /dev/datavg/pvc-0b6c8388-c194-463d-85f8-188e63697023 /var/lib/kubelet/pods/5ef3f1b3-4b0c-4a47-aeea-1b1badaf038f/volumes/kubernetes.io~csi/pvc-0b6c8388-c194-463d-85f8-188e63697023/mount
E1227 08:59:54.370507       1 mount_linux.go:150] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o defaults /dev/datavg/pvc-0b6c8388-c194-463d-85f8-188e63697023 /var/lib/kubelet/pods/5ef3f1b3-4b0c-4a47-aeea-1b1badaf038f/volumes/kubernetes.io~csi/pvc-0b6c8388-c194-463d-85f8-188e63697023/mount
Output: mount: /var/lib/kubelet/pods/5ef3f1b3-4b0c-4a47-aeea-1b1badaf038f/volumes/kubernetes.io~csi/pvc-0b6c8388-c194-463d-85f8-188e63697023/mount: wrong fs type, bad option, bad superblock on /dev/mapper/datavg-pvc--0b6c8388--c194--463d--85f8--188e63697023, missing codepage or helper program, or other error.

What did you expect to happen:

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

• kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin
• kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-plugin
• kubectl get pods -n kube-system
• kubectl get lvmvol -A -o yaml
  [root@node1 releases]# kubectl get lvmvol -A -o yaml

apiVersion: v1
items:
- apiVersion: local.openebs.io/v1alpha1
  kind: LVMVolume
  metadata:
    creationTimestamp: "2021-12-27T08:59:51Z"
    finalizers:
    - lvm.openebs.io/finalizer
    generation: 3
    labels:
      kubernetes.io/nodename: node1
    name: pvc-0b6c8388-c194-463d-85f8-188e63697023
    namespace: kube-system
    resourceVersion: "1664"
    uid: 1be50599-b130-45bc-bc00-d29e94336979
  spec:
    capacity: "21474836480"
    ownerNodeID: node1
    shared: "no"
    thinProvision: "no"
    vgPattern: ^datavg$
    volGroup: datavg
  status:
    state: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• LVM Driver version: master , deploy with helm chart
• Kubernetes version (use kubectl version): 1.21.5
• Kubernetes installer & version: 1.21.5
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): centos 7.9

[pawanpraka1]

@tanguofu we support ubuntu as of now. It is in our roadmap to support other distros. see this issue #75.

[dsharma-dc]

The linked issue mentioned above was rather caused by xfs util behaviour mismatch. That looks like the case here as well. Please try to upgrade the kernel version on nodes to >=5.10 if this is still an issue. The linked issue also mentions that after manually intervening using the utility on host, things worked ok on RHEL.

The project homepage mentions:

Supported System

Name	Version
K8S	1.23+
Distro	Alpine, Arch, CentOS, Debian, Fedora, NixOS, SUSE, Talos, RHEL, Ubuntu
Kernel	oldest supported kernel is 2.6
LVM2	2.03.21

[avishnu]

Need investigation, maybe use approach similar to ZFS.

[avishnu]

@datacore-tilangovan can you try this scenario with RHEL or CentOS 7.9?