-
Lukas Metzner authored
We are reintroducing a feature originally present in v2.10.0 to prevent pods from getting stuck in the `pending` state in clusters with non-cloud nodes. This feature is now optional and can be enabled via the Helm Chart. By default, it remains disabled to avoid compatibility issues with Nomad clusters, which have a different CSI spec implementation. Learn more about it in #400.
Lukas Metzner authoredWe are reintroducing a feature originally present in v2.10.0 to prevent pods from getting stuck in the `pending` state in clusters with non-cloud nodes. This feature is now optional and can be enabled via the Helm Chart. By default, it remains disabled to avoid compatibility issues with Nomad clusters, which have a different CSI spec implementation. Learn more about it in #400.
Kubernetes Hetzner Cloud csi-driver
Getting Started
-
Create a read+write API token in the Hetzner Cloud Console.
-
Create a secret containing the token:
# secret.yml apiVersion: v1 kind: Secret metadata: name: hcloud namespace: kube-system stringData: token: YOURTOKENand apply it:
kubectl apply -f <secret.yml> -
Deploy the CSI driver and wait until everything is up and running:
Have a look at our Version Matrix to pick the correct version.
# Sync the Hetzner Cloud helm chart repository to your local computer. helm repo add hcloud https://charts.hetzner.cloud helm repo update hcloud # Install the latest version of the csi-driver chart. helm install hcloud-csi hcloud/hcloud-csi -n kube-systemAlternative: Using a plain manifest
kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.5.1/deploy/kubernetes/hcloud-csi.yml -
To verify everything is working, create a persistent volume claim and a pod which uses that volume:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: hcloud-volumes --- kind: Pod apiVersion: v1 metadata: name: my-csi-app spec: containers: - name: my-frontend image: busybox volumeMounts: - mountPath: "/data" name: my-csi-volume command: [ "sleep", "1000000" ] volumes: - name: my-csi-volume persistentVolumeClaim: claimName: csi-pvcOnce the pod is ready, exec a shell and check that your volume is mounted at
/data.kubectl exec -it my-csi-app -- /bin/sh
Alternative Kubelet Directory
Some Kubernetes distributions use a non-standard path for the Kubelet directory.
The csi-driver needs to know about this to successfully mount volumes. You can
configure this through the Helm Chart Value node.kubeletDir.
- Standard:
/var/lib/kubelet -
k0s:
/var/lib/k0s/kubelet -
microk8s:
/var/snap/microk8s/common/var/lib/kubelet
Volumes Encrypted with LUKS
To add encryption with LUKS you have to create a dedicate secret containing an encryption passphrase and duplicate the default hcloud-volumes storage class with added parameters referencing this secret:
apiVersion: v1
kind: Secret
metadata:
name: encryption-secret
namespace: kube-system
stringData:
encryption-passphrase: foobar
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hcloud-volumes-encrypted
provisioner: csi.hetzner.cloud
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
csi.storage.k8s.io/node-publish-secret-name: encryption-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-systemYour nodes might need to have cryptsetup installed to mount the volumes with LUKS.
Formatting Options
You can specify extra formatting options which are passed directly to mkfs.FSTYPE via the fsFormatOptions parameter in the storage class.
Example
parameters:
csi.storage.k8s.io/fstype: xfs
fsFormatOptions: "-i nrext64=1"XFS Filesystem
When using XFS as the filesystem type and no fsFormatOptions are set, we apply a default configuration to mkfs to ensure maximum compatibility with older Linux kernel versions. This configuration file is from the xfsprogs-extra alpine package and currently targets Linux 4.19.
Note
The targeted minimum Linux Kernel version may be raised in a minor update, we will announce this in the Release Notes.
If you set any options at all, it is your responsible to make sure that all default flags from mkfs.xfs are supported on your current Linux Kernel version or that you set the flags appropriately.
Upgrading
To upgrade the csi-driver version, you just need to apply the new manifests to your cluster.
In case of a new major version, there might be manual steps that you need to follow to upgrade the csi-driver. See the following section for a list of major updates and their required steps.
From v1 to v2
There are three breaking changes between v1.6 and v2.0 that require user intervention. Please take care to follow these steps, as otherwise the update might fail.
Before the rollout:
-
The secret containing the API token was renamed from
hcloud-csitohcloud. This change was made so both the cloud-controller-manager and the csi-driver can use the same secret. Check that you have a secrethcloudin the namespacekube-system, and that the secret contains the API token, as described in the section Getting Started:$ kubectl get secret -n kube-system hcloud -
We added a new field to our
CSIDriverresource to support CSI volume fsGroup policy management. This change requires a replacement of theCSIDriverobject. You need to manually delete the old object:$ kubectl delete csidriver csi.hetzner.cloudThe new
CSIDriverwill be installed when you apply the new manifests. -
Stop the old pods to make sure that only everything is replaced in order and no incompatible pods are running side-by-side:
$ kubectl delete statefulset -n kube-system hcloud-csi-controller $ kubectl delete daemonset -n kube-system hcloud-csi-node -
We changed the way the device path of mounted volumes is communicated to the node service. This requires changes to the
VolumeAttachmentobjects, where we need to add information to thestatus.attachmentMetadatafield. Execute the linked script to automatically add the required information. This requireskubectlversionv1.24+, even if your cluster is running v1.23.$ kubectl version $ curl -O https://raw.githubusercontent.com/hetznercloud/csi-driver/main/docs/v2-fix-volumeattachments/fix-volumeattachments.sh $ chmod +x ./fix-volumeattachments.sh $ ./fix-volumeattachments.sh
Rollout the new manifest:
$ kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.5.1/deploy/kubernetes/hcloud-csi.ymlAfter the rollout:
-
Delete the now unused secret
hcloud-csiin the namespacekube-system:$ kubectl delete secret -n kube-system hcloud-csi -
Remove old resources that have been replaced:
$ kubectl delete clusterrolebinding hcloud-csi $ kubectl delete clusterrole hcloud-csi $ kubectl delete serviceaccount -n kube-system hcloud-csi
Integration with Root Servers
Root servers can be part of the cluster, but the CSI plugin doesn't work there. To ensure proper topology evaluation, labels are needed to indicate whether a node is a cloud VM or a dedicated server from Robot. If you are using the hcloud-cloud-controller-manager version 1.21.0 or later, these labels are added automatically. Otherwise, you will need to label the nodes manually.
Adding labels manually
Cloud Servers
kubectl label nodes <node name> instance.hetzner.cloud/provided-by=cloudRoot Servers
kubectl label nodes <node name> instance.hetzner.cloud/provided-by=robotDEPRECATED: Old Label
We prefer that you use our new label. The label instance.hetzner.cloud/is-robot-server will be deprecated in future releases.
Cloud Servers
kubectl label nodes <node name> instance.hetzner.cloud/is-root-server=falseRoot Servers
kubectl label nodes <node name> instance.hetzner.cloud/is-root-server=truePods stuck in pending
The current behavior of the scheduler can cause Pods to be stuck in Pending when using the integration with Robot servers.
To address this behavior, you can set enableProvidedByTopology to true in the Helm Chart configuration. This setting prevents pods from being scheduled on nodes — specifically, Robot servers — where Hetzner volumes are unavailable. Enabling this option adds the instance.hetzner.cloud/provided-by label to the allowed topologies section of the storage classes that are created. Additionally, this label is included in the topologyKeys section of csinode objects, and a node affinity is set up for each persistent volume. This workaround does not work with the old label.
Warning
Once enabled, this feature cannot be easily disabled. It automatically adds required nodeAffinities to each volume and the topology keys to csinode objects. If the feature is later disabled, the topology keys are removed from the csinode objects, leaving volumes with required affinities that cannot be satisfied.
Note
After enabling this feature, the workaround for the Kubernetes upstream issue only works on newly created volumes, as old volumes are not updated with the required node affinity.
global:
enableProvidedByTopology: trueFurther information on the upstream issue can be found here.
Versioning policy
We aim to support the latest three versions of Kubernetes. When a Kubernetes version is marked as End Of Life, we will stop support for it and remove the version from our CI tests. This does not necessarily mean that the csi-driver does not still work with this version. We will not fix bugs related only to an unsupported version.
Current Kubernetes Releases: https://kubernetes.io/releases/