[BUG]: mkfsFormatOption not working for powerflex

[AkshaySainiDell]

Bug Description

Pods are stuck into container creating state when mkfsFormatOption is set in storage class

Pod events:

Storage Class:

Logs

time="2024-07-03T08:29:58Z" level=info msg="/csi.v1.Node/NodePublishVolume: REQ 0007: VolumeId=be84bfe40e796c0f-e2e477e400000031, TargetPath=/var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount, VolumeCapability=mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_MULTI_WRITER > , Readonly=false, VolumeContext=map[CreationTime:2024-07-03 08:23:37 +0000 UTC InstallationID:773e768231a8dad6 Name:k8s-13da7884a7 StoragePoolID:71e5091c00000000 StoragePoolName:SP1 StorageSystem:be84bfe40e796c0f csi.storage.k8s.io/ephemeral:false csi.storage.k8s.io/pod.name:pflex-mkfs-pod-1 csi.storage.k8s.io/pod.namespace:default csi.storage.k8s.io/pod.uid:5e831828-7b23-420f-a46c-03610ce34332 csi.storage.k8s.io/serviceAccount.name:default fsType:ext4 mkfsFormatOption:-L MyVolume -m 1 -T largefile4 storage.kubernetes.io/csiProvisionerIdentity:1719994497871-894-csi-vxflexos.dellemc.com], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2024-07-03T08:29:58Z" level=info msg="VolumeContext:"
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" StorageSystem=be84bfe40e796c0f
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" mkfsFormatOption="-L MyVolume -m 1 -T largefile4"
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" csi.storage.k8s.io/ephemeral=false
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" Name=k8s-13da7884a7
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" StoragePoolName=SP1
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" InstallationID=773e768231a8dad6
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" StoragePoolID=71e5091c00000000
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" csi.storage.k8s.io/serviceAccount.name=default
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" csi.storage.k8s.io/pod.namespace=default
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" csi.storage.k8s.io/pod.uid=5e831828-7b23-420f-a46c-03610ce34332
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" CreationTime="2024-07-03 08:23:37 +0000 UTC"
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" csi.storage.k8s.io/pod.name=pflex-mkfs-pod-1
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" storage.kubernetes.io/csiProvisionerIdentity=1719994497871-894-csi-vxflexos.dellemc.com
time="2024-07-03T08:29:58Z" level=info msg="found in VolumeContext" fsType=ext4
time="2024-07-03T08:29:58Z" level=info msg="[NodePublishVolume] csiVolID: be84bfe40e796c0f-e2e477e400000031"
time="2024-07-03T08:29:58Z" level=info msg="[NodePublishVolume] volumeID: e2e477e400000031"
time="2024-07-03T08:29:58Z" level=info msg="[NodePublishVolume] systemID: be84bfe40e796c0f harvested from csiVolID: be84bfe40e796c0f-e2e477e400000031"
time="2024-07-03T08:29:58Z" level=info msg="[NodePublishVolume] We are about to probe the system with systemID be84bfe40e796c0f"
time="2024-07-03T08:29:58Z" level=info msg="Volume ID: be84bfe40e796c0f-e2e477e400000031 contains system ID: be84bfe40e796c0f. checkVolumesMap passed"
time="2024-07-03T08:29:58Z" level=info msg="Found matching SDC mapped volume &{be84bfe40e796c0f e2e477e400000031 /dev/scinia}"
time="2024-07-03T08:29:58Z" level=warning msg="Unable to check stat of file: /var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount with error: stat /var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount: no such file or directory"
time="2024-07-03T08:29:58Z" level=debug msg="created directory" path="/var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount"
time="2024-07-03T08:29:58Z" level=debug msg=fields CSIRequestID=7 device=/dev/scinia id=be84bfe40e796c0f-e2e477e400000031 privateMount=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031 target="/var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount" volumePath=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="attempting mount to private area" CSIRequestID=7 device=/dev/scinia id=be84bfe40e796c0f-e2e477e400000031 privateMount=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031 target="/var/lib/kubelet/pods/5e831828-7b23-420f-a46c-03610ce34332/volumes/kubernetes.io~csi/k8s-13da7884a7/mount" volumePath=/dev/scinia
time="2024-07-03T08:29:58Z" level=warning msg="Unable to check stat of file: /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031 with error: stat /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031: no such file or directory"
time="2024-07-03T08:29:58Z" level=debug msg="created directory" path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031
time="2024-07-03T08:29:58Z" level=info msg="attempting to mount disk" fsType=ext4 options="[defaults]" reqID=7 source=/dev/scinia target=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031
time="2024-07-03T08:29:58Z" level=info msg="mount command" args="-t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031" cmd=mount
time="2024-07-03T08:29:58Z" level=error msg="mount Failed" args="-t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031" cmd=mount error="exit status 32" output="mount: /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031: wrong fs type, bad option, bad superblock on /dev/scinia, missing codepage or helper program, or other error.\n"
time="2024-07-03T08:29:58Z" level=info msg="Mount attempt failed" mountErr="mount failed: exit status 32\nmounting arguments: -t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031\noutput: mount: /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031: wrong fs type, bad option, bad superblock on /dev/scinia, missing codepage or helper program, or other error.\n"
time="2024-07-03T08:29:58Z" level=info msg="checking if disk is formatted using lsblk" args="[-n -o FSTYPE /dev/scinia]" disk=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="getDiskFormat returned after initial mount failed" existingFormat= reqID=7 source=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="disk is unformatted" existingFormat= reqID=7 source=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="disk appears unformatted, attempting format" existingFormat= fsType=ext4 reqID=7 source=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="mkfs args: [:-L MyVolume -m 1 -T largefile4 -E nodiscard /dev/scinia]"
time="2024-07-03T08:29:58Z" level=error msg="format of disk failed" error="exit status 1" existingFormat= fsType=ext4 reqID=7 source=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="re-attempting disk mount" existingFormat= fsType=ext4 reqID=7 source=/dev/scinia
time="2024-07-03T08:29:58Z" level=info msg="mount command" args="-t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031" cmd=mount
time="2024-07-03T08:29:58Z" level=error msg="mount Failed" args="-t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031" cmd=mount error="exit status 32" output="mount: /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031: wrong fs type, bad option, bad superblock on /dev/scinia, missing codepage or helper program, or other error.\n"
time="2024-07-03T08:29:58Z" level=info msg="Cleaning up private target" CSIRequestID=7 privTgt=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031
time="2024-07-03T08:29:58Z" level=info msg="/csi.v1.Node/NodePublishVolume: REP 0007: rpc error: code = Internal desc = error performing private mount: mount failed: exit status 32"
time="2024-07-03T08:29:58Z" level=info msg="mounting arguments: -t ext4 -o defaults /dev/scinia /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031"
time="2024-07-03T08:29:58Z" level=info msg="output: mount: /var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks/be84bfe40e796c0f-e2e477e400000031: wrong fs type, bad option, bad superblock on /dev/scinia, missing codepage or helper program, or other error."

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

Set mkfsFormatOption in storage class -> create PVC and POD

Expected Behavior

Pod should go into running state

CSM Driver(s)

CSI-PowerFlex

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

Kubernetes v1.29.3

Operating System

openSUSE Leap 15.5

[csmbot]

@AkshaySainiDell: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at container.storage.modules@dell.com.

[AkshaySainiDell]

link: 25962