Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[K8S][EKS] 'MountVolume.SetUp failed for volume "volumes-dir" : hostPath type check failed: /mnt/persistent-volumes is not a directory' error installing local csi driver #7113

Closed
2 tasks
vponomaryov opened this issue Jan 17, 2024 · 1 comment
Assignees

Comments

@vponomaryov
Copy link
Contributor

Issue description

  • This issue is a regression.
  • It is unknown if this issue is a regression.

Local CSI driver failed with this:

'MountVolume.SetUp failed for volume "volumes-dir" : hostPath type check failed:
  /mnt/persistent-volumes is not a directory'

The /mnt/persistent-volumes must have been created by the NodeConfig.
It has following config:

    localDiskSetup:
      filesystems:
      - device: /dev/md/nvmes
        type: xfs
      mounts:
      - device: /dev/md/nvmes
        mountPoint: /mnt/persistent-volumes
        unsupportedOptions:
        - prjquota
      raids:
      - RAID0:
          devices:
            modelRegex: Amazon EC2 NVMe Instance Storage
            nameRegex: /dev/nvme\d+n\d+$
        name: nvmes
        type: RAID0

List of disks on one of the K8S Scylla node:

+ lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1       259:0    0    30G  0 disk 
|-nvme0n1p1   259:2    0    30G  0 part /mnt/hostfs
`-nvme0n1p128 259:3    0     1M  0 part 
nvme1n1       259:1    0 872.6G  0 disk 

So, probably we should update the regex for the Scylla disks the way it doesn't touch OS-related disk if matches to it.

Impact

Scylla-operator cannot be tested on EKS backend using SCT.

How frequently does it reproduce?

100%. All our EKS deployments fail this way.

Installation details

Cluster size: 4 nodes (i4i.4xlarge)

Scylla Nodes used in this run:
No resources left at the end of the run

OS / Image: `` (k8s-eks: eu-north-1, `eu-west-1`)

Test: longevity-scylla-operator-multidc-12h-eks
Test id: 51927961-58a8-4e79-bace-60475111bb39
Test name: scylla-operator/operator-master/eks/longevity-scylla-operator-multidc-12h-eks
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 51927961-58a8-4e79-bace-60475111bb39
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 51927961-58a8-4e79-bace-60475111bb39

Logs:

Jenkins job URL
Argus

@vponomaryov
Copy link
Contributor Author

The real root cause is here: scylladb/scylla-operator#1690

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant