Mayastor⚓︎

Links to external documentation

OpenEBS Mayastor is a persistent volume provider for Kubernetes. prokube uses Mayastor as a distributed and redundant storage provider on highly-available setups (usually 3 nodes or more).

FAQ⚓︎

Some PVCs are pending and are not getting bound⚓︎

Possibly the cluster does not have enough storage available.

You can use the kubectl mayastor plugin (should be pre-installed on all mayastor enabled nodes) to check how much disk capacity is available, e.g.:

ansible@master-focal-still-goberian:~$ kubectl mayastor get pools
 ID                                     DISKS           MANAGED  NODE                           STATUS  CAPACITY  ALLOCATED  AVAILABLE  COMMITTED
 pool-on-worker-1-focal-still-goberian  aio:///dev/sdb  true     worker-1-focal-still-goberian  Online  99.9GiB   90GiB      9.9GiB     90GiB
 pool-on-master-focal-still-goberian    aio:///dev/sdb  true     master-focal-still-goberian    Online  99.9GiB   90GiB      9.9GiB     90GiB
 pool-on-worker-0-focal-still-goberian  aio:///dev/sdb  true     worker-0-focal-still-goberian  Online  99.9GiB   90GiB      9.9GiB     90GiB

CRs (custom resources) also report capacity and usage, so mayastor plugin is not strictly needed:

kubectl get diskpool -n mayastor

I'm having problems with Mayastor backed PVCs⚓︎

Mayastor needs the nvme_tcp kernel module, which on Ubuntu lives in linux-modules-extra-<kernel version>. If a node boots into a kernel for which that package is missing, Mayastor cannot access disks via nvme. In turn, PVCs cannot be accessed, which leads to error messages such as this:

kubelet  MountVolume.MountDevice failed for volume "pvc-15a0fe75-9961-469e-a3b8-e61b0d4cda6a" : rpc error: code = Internal desc = Failed to stage volume 15a0fe75-9961-469e-a3b8-e61b0d4cda6a: attach failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2)

or

 Warning  FailedAttachVolume  112s (x10 over 17m)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-e1ce0497-88a8-4dbe-b311-f7ade8a9838a" : rpc error: code = Internal desc = Operation failed: PreconditionFailed("error in response: status code '412 Precondition Failed', content: 'RestJsonError { details: \"\", message: \"SvcError :: NoOnlineReplicas: No online replicas are available for Volume 'e1ce0497-88a8-4dbe-b311-f7ade8a9838a'\", kind: FailedPrecondition }'")

or

ansible@worker-1-focal-capital-pekingese:~$ kubectl-mayastor get volumes
 ID                                    REPLICAS  TARGET-NODE                       ACCESSIBILITY  STATUS  SIZE   THIN-PROVISIONED  ALLOCATED
 24a14e2f-c881-4a82-9615-2938529e9315  3         master-focal-capital-pekingese    nvmf           Online  10GiB  false             10GiB
 2d04b9fd-7124-4ba1-84ed-febc42a104a7  3         master-focal-capital-pekingese    nvmf           Online  10GiB  false             10GiB
 46387e0f-4cf9-4724-b2ab-da90273b976a  3         master-focal-capital-pekingese    nvmf           Online  50GiB  false             50GiB
 654ce71f-64d8-49d5-b067-18a532751f6d  3         worker-0-focal-capital-pekingese  nvmf           Online  10GiB  false             10GiB
 80f28fe3-599c-4db9-adc1-c9d22982c7ee  3         master-focal-capital-pekingese    nvmf           Online  10GiB  false             10GiB
 8a82833e-212a-4e5d-b333-7774c2e7ad82  3         worker-0-focal-capital-pekingese  nvmf           Online  20GiB  false             20GiB
 c4b34d32-5527-4248-8945-3ef86164cac0  3         master-focal-capital-pekingese    nvmf           Online  10GiB  false             10GiB
 e1ce0497-88a8-4dbe-b311-f7ade8a9838a  3         <none>                            <none>         Online  5GiB   false             0 B

Check with lsmod | grep nvme_tcp on the affected nodes. Fix by installing the matching package and loading the module:

sudo apt-get install -y "linux-modules-extra-$(uname -r)"
sudo modprobe nvme_tcp

Then delete Mayastor io-engine / csi pods that are still stuck.

To prevent this, the prepare_vms ansible role installs the kernel flavour metapackage (linux-modules-extra-{gcp,aws,azure} or linux-image-generic) so linux-modules-extra follows kernel upgrades, and excludes kernel packages from unattended-upgrades unless unattended_kernel_upgrades: true is set. After a manual kernel upgrade, make sure linux-modules-extra-<new version> is installed before rebooting.

Reclaiming Mayastor drives⚓︎

A Mayastor diskpool pool on a Mayastor formatted drive survives a kubernetes teardown and can be recreated on the node using the same unique name as before. If you want to change the name, or use the drive for other purposes, you need to wipe the partition table and overwrite the drive with zeros.

Wipe partition table⚓︎

First list them by:

sudo wipefs /dev/sdb
DEVICE OFFSET       TYPE UUID LABEL
sdb    0x200        gpt
sdb    0xf9fffffe00 gpt
sdb    0x1fe        PMBR

and wipe them, e.g. by type (WARNING YOU WILL LOSE ALL DATA!):

sudo wipefs -a -t gpt -f /dev/<replace>

Overwrite the drive with zeros⚓︎

Mayastor uses low level access and if a drive is reused it doesn't delete old data. In order to release the whole drive you have to overwrite it with zeros (WARNING YOU WILL LOSE ALL DATA!):

sudo dd if=/dev/zero of=/dev/<replace> bs=4096 status=progress