Skip to content

Notebooks⚓︎

Links to external documentation

→ Kubeflow Notebooks Documentation

Node affinity for notebooks⚓︎

By default, notebooks are scheduled on any node in the cluster. You can restrict this by setting node affinity in the kubeflow/project-overlays/jupyter-web-app/configs/spawner_ui_config.yaml file at the affinityConfig tag. For example:

affinityConfig:
  # If readonly, the default value will be the only option
  readOnly: false

  # value is a list of `configKey`s that we want to be selected by default
  value: ""

  # The list of available affinity configs
  options:
    - configKey: tesla-v100-node3
      displayName: Tesla V100 (node3)
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - node3 # Replace with your actual node label value
    - configKey: a100-node4
      displayName: A100 (node4)
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - node4 # Replace with your actual node label value

After ArgoCD has synced the changes, you might also need to restart the notebook controller:

kubectl rollout restart deploy notebook-controller-deployment -n kubeflow

After that, you can select the node affinity when creating a new notebook:

  • Open the new notebook dialog
  • Click on "Advanced Options"
  • Select the desired node affinity from the "Affinity Config" dropdown

Notebook culling⚓︎

Notebook culling is the automatic shutdown of idle notebooks. It is supported by Kubeflow's notebook controller.
The behaviour is controlled by env variables: 

ENABLE_CULLING: "true"
CULL_IDLE_TIME: "15" # Cull after this amount of minutes
IDLENESS_CHECK_PERIOD: "1" # Check every x minutes

These variables are set in notebook-controller-config-<somehash> configmap. You can edit it to enable culling and then restart the controller like this:

kubectl edit cm -n kubeflow notebook-controller-config-<somehash>
kubectl rollout restart deploy notebook-controller-deployment -n kubeflow

Note: This works only for JupyterLab notebooks because they report idleness, this doesn't work with VSCode or RStudio.