Skip to content

Upgrade to v0.8.0

Use this guide to upgrade Kubemox from a version below 0.8.0 to v0.8.X.

The commands assume two things:

  • The Helm release is named kubemox.
  • Your kubectl context points to the namespace of the release.

What changed

Five kinds are now namespaced

VirtualMachine, VirtualMachineSet, VirtualMachineTemplate, Container and StorageDownloadURL are now namespaced.

ProxmoxConnection and CustomCertificate stay cluster scoped. The steps below do not touch them.

Kubernetes cannot change the scope of a CRD in place. You must delete the five CRDs and re-create every object of those kinds.

A webhook rejects duplicate names

A validating webhook checks spec.name of each VirtualMachine, VirtualMachineTemplate and Container. It rejects the object when:

  • Another object in any namespace already uses the name on the same node and ProxmoxConnection.
  • More than one guest on that Proxmox node has the name.

Name rules:

  • VirtualMachine and VirtualMachineTemplate share one set of names. The comparison ignores case.
  • Container names are compared exactly, and only against other Containers.
  • The same name on different nodes is allowed.

Certificate of the webhook:

  • With cert-manager installed, cert-manager issues the certificate.
  • Without cert-manager, the operator issues a self-signed certificate. It stores the certificate in the <release>-webhook-cert Secret. It injects the CA into the webhook configuration.
  • Force one of the two with --set webhook.certManager=true|false.
  • Disable the webhook with --set webhook.enabled=false. The controllers then catch duplicate names. See the next section.

The oldest object owns a duplicate name

Two objects can still claim one name. This happens when they are created at the same moment, or when the webhook is disabled. Then:

  • The oldest object owns the guest.
  • The other objects get the condition NameConflict=True. The condition names the owner.
  • The other objects are not reconciled against Proxmox.
  • Deleting one of the other objects does not delete the guest.
  • When the owner is gone, the next object takes the name within about 30 seconds.

Snapshot policies lost namespaceSelector

A VirtualMachineSnapshotPolicy only snapshots VirtualMachines in its own namespace. It selects them with its labelSelector and its connectionRef.

  • Create one policy per namespace.
  • Remove namespaceSelector from your manifests. Some clients reject the field. Other clients drop it silently.

Snapshots follow the VirtualMachine

A VirtualMachineSnapshot looks up its VirtualMachine in its own namespace. It snapshots the guest named spec.name on spec.nodeName of that VirtualMachine.

  • Keep each snapshot in the same namespace as its VirtualMachine.
  • The connectionRef of the snapshot must name the connection of the VirtualMachine. If not, the snapshot reports an error and does nothing.

passwordFrom reads from the template namespace

cloudInitConfig.passwordFrom of a VirtualMachineTemplate now reads the Secret from the namespace of the template. Before, it read from the namespace of the operator. Copy the Secret into the namespace you restore the template into.

Upgrade steps

Danger

Deleting a VirtualMachine, Container or VirtualMachineTemplate while the operator runs deletes the guest on Proxmox. Do the steps in order. Do not skip step 1.

1. Stop the operator

kubectl scale deployment kubemox --replicas=0

2. Back up the objects

This also saves the VirtualMachineSnapshot objects as a record. You do not restore them. See step 3.

for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls virtualmachinesnapshots; do
  kubectl get "$kind.proxmox.alperen.cloud" -A -o yaml > "backup-$kind.yaml"
done

3. Delete the old objects and CRDs

First remove the finalizers. Then the CRD deletion does not touch Proxmox.

for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls; do
  kubectl get "$kind.proxmox.alperen.cloud" -o name | \
    xargs -r -I{} kubectl patch {} --type=merge -p '{"metadata":{"finalizers":null}}'
  kubectl delete crd "$kind.proxmox.alperen.cloud"
done

Kubernetes deletes the VirtualMachineSnapshot objects at the same time. Their owner, the VirtualMachine, is gone. This does not delete the snapshots on Proxmox.

Do not restore these objects in step 5. The operator would take each snapshot again, and Proxmox refuses a snapshot name that exists. The snapshots stay on Proxmox. Create a new VirtualMachineSnapshot only when you want a new snapshot.

4. Upgrade the chart

The upgrade installs the new CRDs. It also sets the operator back to 1 replica. The operator starts.

Set the image tag. Helm keeps the values of the old release. If you pinned image.tag before, the old operator keeps running on the new CRDs. Then every step 5 command fails with failed calling webhook.

helm repo update
helm upgrade kubemox alperencelik/kubemox --set image.tag=v0.8.0

Wait until the operator is ready. Check the image. The webhook runs inside the operator. Step 5 fails until the webhook is ready.

kubectl rollout status deployment/kubemox
kubectl get deployment kubemox -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

The second command must print alperencelik/kubemox:v0.8.0.

5. Restore the objects into one namespace

Pick the namespace. Then restore the five kinds. The command removes the server-set fields. It skips the objects that another Kubemox object owns:

  • A VirtualMachineSet re-creates its VirtualMachines.
  • A VirtualMachineTemplate re-creates its StorageDownloadURL.
  • Both adopt the existing guests on Proxmox by name.
NAMESPACE=default
STRIP='del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp,
            .metadata.generation, .metadata.managedFields, .metadata.finalizers, .status)'
for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls; do
  yq ".items[] | select(.metadata.ownerReferences == null) | $STRIP | split_doc" "backup-$kind.yaml" \
    | kubectl apply -n "$NAMESPACE" -f -
done

A kind with no objects prints error: no objects passed to apply. Ignore it.

If kubectl apply fails with failed calling webhook, wait a few seconds. Then run the command again. The webhook certificate was not ready yet. If it still fails after a minute, check the image from step 4.

6. Verify

kubectl get vm,vmset,vmt,lxc,sdu -n "$NAMESPACE"

Check that every object reaches its desired state. Check that no object has the condition NameConflict=True.