Skip to content

Storage Classes Documentation⚓︎

Context⚓︎

Important: This documentation describes the default storage configuration for self-managed Kubernetes clusters deployed with prokube. If you're running on managed Kubernetes services (GKE, EKS, AKS), you may want to use cloud-provider storage classes instead. However, prokube's setup works on any Kubernetes cluster that meets the prerequisites (kernel modules, hugepages, etc.).

Overview⚓︎

prokube provides three storage classes for different use cases, based on performance requirements, redundancy needs, and workload characteristics. All storage is provided through the OpenEBS ecosystem:

  • OpenEBS LocalPV (hostpath) for local, non-replicated storage
  • Mayastor for high-performance replicated block storage
  • mayastor-etcd-localpv for Mayastor's internal infrastructure (not for user workloads)

Both user-facing storage options are automatically configured during prokube deployment.

Available Storage Classes⚓︎

1. openebs-hostpath (Default)⚓︎

Provider: OpenEBS LocalPV
Type: Local, non-replicated storage
Default Storage Class: Yes

Description⚓︎

Uses a directory on the host node's filesystem (/var/openebs/local/) to provide persistent volumes. Data is stored locally on the node where the pod is scheduled and is not replicated across nodes.

Characteristics⚓︎

  • Performance: Fast local disk I/O
  • Redundancy: None - data exists only on one node
  • Volume Expansion: Not supported
  • Volume Binding: WaitForFirstConsumer (volume is created when pod is scheduled)
  • Reclaim Policy: Delete (volume deleted when PVC is deleted)
  • Size Enforcement: No - does not enforce storage limits set in PVC

When to Use⚓︎

  • Applications that handle their own data replication (e.g., Kafka, Cassandra)
  • Temporary or cache data
  • Single-node clusters
  • Development and testing environments
  • When maximum I/O performance is needed and redundancy is not required
  • Stateful workloads that are node-affine

When NOT to Use⚓︎

  • Critical data requiring redundancy
  • Applications that need to migrate between nodes
  • When you need strict storage quota enforcement
  • Production databases without application-level replication

Configuration⚓︎

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-hostpath
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/var/openebs/local"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

2. mayastor-three-replicas (Default for multi-node)⚓︎

Provider: OpenEBS Mayastor
Type: Replicated block storage
Default Storage Class: Yes (on multi-node clusters, replaces openebs-hostpath as default)
Minimum Cluster Size: 3 nodes

Description⚓︎

High-performance replicated storage using NVMe-oF (NVMe over Fabrics) protocol. Data is synchronously replicated across three nodes, providing both redundancy and high availability.

Characteristics⚓︎

  • Performance: Very high (uses polling which consumes CPU even when idle)
  • Redundancy: 3-way replication across nodes
  • Volume Expansion: Supported
  • Volume Binding: WaitForFirstConsumer
  • Protocol: NVMe-oF (nvmf) - always over network
  • Provisioning: Thin provisioning enabled
  • I/O Timeout: 60 seconds
  • CPU Usage: Higher due to polling mechanism
  • Data Locality: None guaranteed - volumes are always accessed over network (see Important Notes)

When to Use⚓︎

  • Production databases requiring high availability (MySQL, PostgreSQL, MariaDB)
  • Critical application data
  • Stateful applications that need to survive node failures
  • Applications requiring high I/O performance with redundancy
  • When you have 3+ node clusters
  • Object storage backends (e.g., MinIO when you want storage-level replication)
  • Applications that can tolerate network I/O latency for volume access

When NOT to Use⚓︎

  • Clusters with fewer than 3 nodes (replication requires 3+ nodes)
  • Applications that do their own replication (wastes resources)
  • When CPU resources are constrained (Mayastor uses polling)
  • Development/testing where redundancy is not needed
  • Latency-critical applications requiring guaranteed local disk access - use LocalPV instead for true local access (see Important Notes)

Configuration⚓︎

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mayastor-three-replicas
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
parameters:
  repl: '3'
  protocol: 'nvmf'
  ioTimeout: '60'
  thin: 'true'
provisioner: io.openebs.csi-mayastor
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

3. mayastor-no-redundancy⚓︎

Provider: OpenEBS Mayastor
Type: Non-replicated block storage
Default Storage Class: No

Description⚓︎

High-performance storage using Mayastor's NVMe-oF capabilities but without replication. Provides a single replica on one node, similar to LocalPV but with Mayastor's performance characteristics and volume management features.

Characteristics⚓︎

  • Performance: Very high (uses polling)
  • Redundancy: None - single replica
  • Volume Expansion: Supported
  • Volume Binding: WaitForFirstConsumer
  • Protocol: NVMe-oF (nvmf)
  • Provisioning: Thin provisioning enabled
  • I/O Timeout: 60 seconds

When to Use⚓︎

  • High-performance workloads without redundancy requirements
  • When you need volume expansion capabilities not available in LocalPV
  • Applications with their own replication that need Mayastor's performance
  • When you want Mayastor's volume management features without replication overhead
  • Temporary high-performance storage
  • Testing Mayastor capabilities

When NOT to Use⚓︎

  • Critical data requiring redundancy
  • When LocalPV performance is sufficient (to save CPU from polling)
  • Small clusters where CPU is constrained

Configuration⚓︎

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mayastor-no-redundancy
parameters:
  repl: '1'
  protocol: 'nvmf'
  ioTimeout: '60'
  thin: 'true'
provisioner: io.openebs.csi-mayastor
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

4. mayastor-etcd-localpv (Internal Use Only)⚓︎

This is simply an OpenEBS LocalPV storage class (same as openebs-hostpath) but with a separate base path (/var/local/mayastor/localpv-hostpath/etcd). It exists only for Mayastor's internal etcd StatefulSet and is deployed automatically with Mayastor.

Do not use this storage class for your workloads - use openebs-hostpath instead.

Storage Architecture Decision Tree⚓︎

Is this a single-node cluster?
├─ Yes → Use openebs-hostpath
└─ No (3+ nodes available)
   │
   └─ Does the application handle its own replication?
      ├─ Yes (e.g., Kafka, Cassandra, some databases)
      │  └─ Is maximum performance critical?
      │     ├─ Yes → Use mayastor-no-redundancy
      │     └─ No → Use openebs-hostpath
      │
      └─ No (application needs storage-level redundancy)
         └─ Use mayastor-three-replicas

Mayastor Architecture Notes⚓︎

Deployment in prokube⚓︎

In prokube's default configuration: - Mayastor CSI node plugin is deployed as a DaemonSet to all nodes automatically - Mayastor io-engine (storage engine) runs only on nodes labeled openebs.io/engine=mayastor (configured via infra/Ansible) - This separation allows you to have dedicated storage nodes with fast NVMe drives while all compute nodes can still access volumes - The setup is automated during prokube deployment - no manual CSI plugin configuration needed

Storage Nodes vs Consumer Nodes⚓︎

Mayastor distinguishes between two types of nodes:

  1. Storage Nodes (labeled openebs.io/engine=mayastor):
  2. Run the Mayastor io-engine (storage engine)
  3. Provide disk pools for actual data storage
  4. Must have dedicated block devices

  5. Typically 3+ nodes for replicated storage

  6. Consumer Nodes (any node in the cluster):

  7. The Mayastor CSI node plugin is automatically deployed to all nodes via DaemonSet
  8. Kernel modules (e.g., nvme_tcp) are loaded automatically
  9. Can access volumes over the network without being storage nodes
  10. Does NOT need dedicated block devices or special labels

Key Point: Pods can access Mayastor volumes from any node in the cluster. The CSI node plugin DaemonSet automatically runs on all nodes, so there's no special configuration needed. All volume access is over the network (NVMe-oF).

Network Access Always⚓︎

All Mayastor volume I/O goes through NVMe-oF network protocol, even if a replica exists on the same node. The CSI node plugin (deployed automatically on all nodes) handles connectivity. See Important Notes for details on the Mayastor 2.0 changes.

Component Usage in prokube⚓︎

Based on the main data storage README, here are the minimum storage requirements:

Component Volume Size Recommended Storage Class
Istio authservice 10 GiB mayastor-three-replicas
Keycloak PostgreSQL 8 GiB mayastor-three-replicas
Prometheus 10 GiB mayastor-three-replicas
MinIO 20+ GiB mayastor-three-replicas or openebs-hostpath*
Kubeflow MySQL 20 GiB mayastor-three-replicas
Katib MySQL 10 GiB mayastor-three-replicas
Kafka Variable openebs-hostpath

*MinIO can use either replicated storage (mayastor-three-replicas) or non-replicated (openebs-hostpath) depending on whether you want storage-level or application-level replication.

Total minimum: >80 GiB

Storage Engine Details⚓︎

OpenEBS LocalPV (Hostpath)⚓︎

OpenEBS LocalPV is the product name; "hostpath" refers to the storage type it uses (a directory on the host filesystem). In prokube, these terms are used interchangeably since hostpath is the only LocalPV type deployed.

Note: There are other OpenEBS LocalPV variants not used in prokube: - Device LocalPV: Uses entire block devices and enforces volume size limits, but requires complex device setup and is in alpha status. - LVM LocalPV: Uses LVM for volume management with size enforcement and snapshots, but requires pre-configured LVM volume groups on each node, adding operational complexity that conflicts with prokube's goal of minimal infrastructure prerequisites.

  • Pros:
  • Simple and fast
  • Low resource overhead
  • Works on any cluster size

  • Ideal for applications with built-in replication

  • Cons:

  • No size enforcement (can exceed PVC limit)
  • No redundancy
  • Data is lost if node fails
  • No volume migration
  • No CSI snapshot support

Mayastor⚓︎

  • Pros:
  • Very high performance (NVMe-oF)
  • Configurable replication (1, 2, or 3 replicas)
  • Volume expansion supported
  • CSI snapshot support

  • Enterprise-grade features

  • Cons:

  • High CPU usage due to polling (even when idle)
  • Requires at least 3 nodes for replication
  • Complex setup (requires dedicated block devices)
  • Takes complete ownership of devices

Important Notes⚓︎

  1. Default Storage Class: In multi-node deployments, mayastor-three-replicas is set as default. In single-node or when Mayastor is not available, openebs-hostpath is the default.

  2. Volume Binding Mode: All storage classes use WaitForFirstConsumer, meaning volumes are provisioned only when a pod is scheduled. This ensures volumes are created in the correct availability zone/node and that the pod's scheduling constraints are considered before volume creation.

  3. Network I/O Considerations:

  4. Important: As of Mayastor 2.0, the local: true parameter was deprecated and removed. This means Mayastor volumes are always accessed over the network (NVMe-oF), even if a replica exists on the local node.
  5. For latency-sensitive applications where data locality is critical, LocalPV (openebs-hostpath) may be preferred even without redundancy, as it guarantees local disk access with zero network overhead.

  6. Some HA databases and distributed systems optimize for local replica access (preferring to read from a local copy). These applications cannot benefit from such optimizations with Mayastor 2.0+.

  7. Mayastor Device Requirements: Mayastor completely takes over block devices. Each DiskPool can only use one device, though one node can have multiple pools. See infra repository for device preparation.

  8. Backups: Automated backups are not currently configured. LocalPV volumes can be backed up directly from the filesystem. Mayastor supports volume snapshots via the snapshotter, but exporting snapshots outside Kubernetes is not yet automated.

  9. CPU Considerations: Mayastor uses polling for high performance, which means CPU cores will show utilization even when there's no I/O activity. Plan CPU resources accordingly.

  10. Mayastor 2.0 Breaking Change: The local: true storage class parameter was removed in Mayastor 2.0. Older deployments using this parameter will need to migrate data to new storage classes without this parameter. Mayastor now always uses networked I/O (NVMe-oF) regardless of replica location. See UPDATING.md file for migration instructions.