Skip to content

Backup/Restore⚓︎

Overview⚓︎

As we need to backup (and restore) not only Kubernetes resources, but also application data (e.g. databases, object storage), we have implemented a custom backup and restore solution for prokube deployments. In its core it is based on Velero and adds modules for the applications that need to be backed up on application level. Some high level points about the backup solution:

  • Backups are intended to be used for disaster recovery and can (should) only be restored to a prokube deployment of the same version. They are not intended for upgrading between versions.
  • Backups need to be set up manually per deployment.
  • Backups require a CSI storage class to be used in the cluster. If no CSI storage class is available, backup functionality is reduced (e.g. no backup of Kubeflow Notebook volumes).
  • Backups are stored in a location external of the prokube cluster (e.g. S3 bucket).
  • The execution of backups is scheduled, usually daily, but manual backups can also be triggered.
  • Existing backups in the external storage location are automatically pruned based on the configured retention policy. Usually a 30 day period is used.
  • Restoring from a backup is a manual process. For instructions see below.

What is backed up⚓︎

  • Keycloak configuration and database
  • Kubeflow Profiles, Notebooks, Notebook Volumes, Katib experiments, Pipelines, KServe model servings
  • MinIO configuration and data
  • MLflow experiments, runs, registered models

Setup Backups⚓︎

As the backup storage location is external to the cluster and individual for each deployment, backups have to be set up manually. For instructions, see the paas/backup_restore directory in the prokube repository.

Things you might want to additionally backup on the host level⚓︎

  • SSH: backup authorized keys from ~/.ssh/authorized_keys
  • Network: backup the configuration
    • On Ubuntu /etc/netplan/00-installer-config.yaml

Restore from a Backup⚓︎

Restoring from a backup is a manual process. For instructions, see the paas/backup_restore directory in the prokube repository.