MinIO FAQ⚓︎

Increasing MinIO tenant volume⚓︎

MinIO uses a default storage class to save data. This is openebs-hostpath and mayastor-three-replicas for single node and high-availability, respectively. Mayastor hosted volumes can't be resized, so we have to delete the old MinIO tenant and set up a new one with bigger volume. You can backup data with mc.

# expose minio via port forwarding
kubectl port-forward -n minio defaulttenant-ss-0-0 59090:9000

# new terminal
# connect mc 
mc alias set minio http://localhost:59090 <key> <secret>
# copy over data to local disk
mc cp --recursive minio/ .
# delete old tenant and release pvc
kubectl delete tenant defaulttenant -n minio
kubectl delete pvc -n minio 0-defaulttenant-ss-0-0
kubectl delete pvc -n minio defaulttenant-prometheus-defaulttenant-prometheus-0

# create new tenant
kubectl apply -k paas/data_storage/minio/tenants

# remove old mc alias and connect new tenant (you will likely have to restart port forwarding)
mc alias remove minio
mc alias set minio http://localhost:59090 <key> <secret>
# create all buckets and copy over data
mc mb minio/mlpipeline
mc cp --recursive mlpipeline minio/
# ...

Transferring MinIO tenant to a new StorageClass⚓︎

Use case: a MinIO tenant already has a PV with data which uses an undesired StorageClass (e.g. OpenEBS Hostpath on managed kubernetes). This section is similar to the previous one, but provides alternative instructions on how to use intermediate PV to transfer the data (without copying data locally). Steps:

Downscale the MinIO tenant StatefulSet to 0 (required because the underlying PVC uses ReadWriteOnce access mode):
```
kubectl scale statefulset defaulttenant-ss-0 -n minio --replicas=0
```

Create an intermediate PV and a migrator pod:

# Set this to your target StorageClass
STORAGE_CLASS="your-desired-storageclass"

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tenant-migrate-pvc
  namespace: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: ${STORAGE_CLASS}
---
apiVersion: v1
kind: Pod
metadata:
  name: data-migrator
  namespace: minio
spec:
  containers:
  - name: migrator
    image: alpine
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: old
      mountPath: /mnt/old
    - name: new
      mountPath: /mnt/new
  volumes:
  - name: old
    persistentVolumeClaim:
      claimName: 0-defaulttenant-ss-0-0
  - name: new
    persistentVolumeClaim:
      claimName: tenant-migrate-pvc
EOF

Open pod shell and copy the content of the old tenant's PV:

# exec into pod first
kubectl exec -it data-migrator -n minio -- /bin/sh

In the pod shell:

cp -a /mnt/old/. /mnt/new/  # inside pod

If copy fails

If the cp command fails (e.g., connection to pod is lost), you may have incomplete files in the destination. Before retrying, run the following (inside the pod!) to remove any incomplete files:

find /mnt/old -type f -print0 | while IFS= read -r -d '' f; do
  newf="/mnt/new/${f#/mnt/old/}"
  if [ -f "$newf" ]; then
    if [ "$(stat -c%s "$f")" -ne "$(stat -c%s "$newf")" ]; then
      echo "Removing incomplete $newf"
      rm "$newf"
    fi
  fi
done

Then resume copying:

cp -au /mnt/old/. /mnt/new/

Verify the contents are copied (inside pod):

# Compare file counts
find /mnt/old -type f | wc -l
find /mnt/new -type f | wc -l

# Compare directory sizes (note: sizes may differ slightly due to filesystem overhead)
du -sh /mnt/old
du -sh /mnt/new

# OPTIONAL: Verify no files are missing by comparing checksums (for critical data, can be very slow)
cd /mnt/old && find . -type f -exec md5sum {} \; | sort > /tmp/old.md5
cd /mnt/new && find . -type f -exec md5sum {} \; | sort > /tmp/new.md5
diff /tmp/old.md5 /tmp/new.md5

Delete the migrator pod and the old PVC:

kubectl delete po data-migrator -n minio
kubectl delete pvc 0-defaulttenant-ss-0-0 -n minio  # this should also delete the corresponding PV

Recreate the tenant:

Using non-default StorageClass

If you need to use a specific StorageClass (other than the cluster default), you must edit the tenant CR manifest to add or update spec.pools[0].volumeClaimTemplate.spec.storageClassName before applying. For example:

spec:
  pools:
  - volumeClaimTemplate:
      spec:
        storageClassName: your-desired-storageclass
        # ... other specs

If using GitOps with ArgoCD (prokube default), the general instructions are:

# Step 1: Disable MinIO tenant autosync:
kubectl -n argocd patch applicationset storage-apps \
  --type='json' \
  -p='[
    {"op":"remove","path":"/spec/template/spec/syncPolicy/automated"}
  ]'

# Step 2: update the Tenant manifest (if needed, see note above) and push to your GitOps branch
...

# Step 3: delete the existing tenant
kubectl delete tenant defaulttenant -n minio

# Step 4: enable autosync again - this should trigger tenant recreation
kubectl -n argocd patch applicationset storage-apps \
  --type='merge' \
  -p='{"spec":{"template":{"spec":{"syncPolicy":{"automated":{}}}}}}'

If GitOps is disabled, you can just recreate the tenant manually:

kubectl delete tenant defaulttenant -n minio
# command below will use current cluster DEFAULT StorageClass for tenant's new PV
# if tenant was not edited
kubectl apply -k paas/data_storage/minio/tenants

Downscale the tenant StatefulSet again (required for ReadWriteOnce PVC access):
```
kubectl scale statefulset defaulttenant-ss-0 -n minio --replicas=0
```

Migrate the copied content to the newly created PV:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: data-migrator-2
  namespace: minio
spec:
  containers:
  - name: migrator
    image: alpine
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: old
      mountPath: /mnt/old
    - name: new
      mountPath: /mnt/new
  volumes:
  - name: old
    persistentVolumeClaim:
      claimName: tenant-migrate-pvc
  - name: new
    persistentVolumeClaim:
      claimName: 0-defaulttenant-ss-0-0
EOF

Open shell in the pod:

kubectl exec -it data-migrator-2 -n minio -- /bin/sh

Copy the data in the pod shell:

cp -a /mnt/old/. /mnt/new/  # inside pod

If it fails, see the note in step 4. On successful completion, you might want to run the commands from step 4 to verify data integrity.

Delete the migrator pod and upscale the tenant StatefulSet back to 1:

kubectl delete po data-migrator-2 -n minio
kubectl scale statefulset defaulttenant-ss-0 -n minio --replicas=1

Verify the tenant is working, and you can log into MinIO in the Web UI, and that the MinIO data is shown there.

Verify the default tenant PV is using the desired storage class:
```
kubectl get pvc 0-defaulttenant-ss-0-0 -n minio -o jsonpath='{.spec.storageClassName}{"\n"}'
```
Then delete the intermediate PV:
```
kubectl delete pvc tenant-migrate-pvc -n minio  # this should also delete the corresponding PV
```

The file size is too large to upload files to MinIO using the Web-UI⚓︎

The ingress controller limits the maximum body size for HTTP requests. If you encounter an error like 413 Request Entity Too Large, you will need to increase the limit in the ingress controller configuration. When using the ingress-nginx controller, this can be done by adding or editing the relevant annotation on that ingress resource. For example, to set the maximum body size to 500Mi, you can add the following annotation:

annotations:
  nginx.ingress.kubernetes.io/proxy-body-size: "500m"

Alternatively use the MinIO command line client and port forwarding to upload bigger files.

The MinIO tenant is not getting ready⚓︎

If you have an error message such as

Get "https://operator.minio.svc.cluster.local:4222/webhook/v1/getenv/minio/defaulttenant?key=MINIO_ARGS": x509: certificate has expired or is not yet valid: current time 2024-06-11T15:49:46Z is after 2024-05-15T12:39:48Z

MinIO's TLS certificate has expired and needs to be renewed.

Storage is slow⚓︎

You can try attaching another SSD to the node and use that as a dedicated storage class for MinIO. You can check the performance with:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75

MinIO has run out of space⚓︎

If MinIO has run out of space, you can either increase the size of the underlying PersistentVolume (if your StorageClass supports that) or delete old objects. You can also set up lifecycle rules to automatically delete old objects, this needs to be set up for each bucket individually and makes sense if you have objects that can be deleted after a certain time (e.g. old pipeline runs).