Upgrade to v0.8.0
Use this guide to upgrade Kubemox from a version below 0.8.0 to v0.8.X.
The commands assume two things:
- The Helm release is named
kubemox. - Your kubectl context points to the namespace of the release.
What changed
Five kinds are now namespaced
VirtualMachine, VirtualMachineSet, VirtualMachineTemplate, Container and StorageDownloadURL are now namespaced.
ProxmoxConnection and CustomCertificate stay cluster scoped. The steps below do not touch them.
Kubernetes cannot change the scope of a CRD in place. You must delete the five CRDs and re-create every object of those kinds.
A webhook rejects duplicate names
A validating webhook checks spec.name of each VirtualMachine, VirtualMachineTemplate and Container. It rejects the object when:
- Another object in any namespace already uses the name on the same node and
ProxmoxConnection. - More than one guest on that Proxmox node has the name.
Name rules:
VirtualMachineandVirtualMachineTemplateshare one set of names. The comparison ignores case.Containernames are compared exactly, and only against otherContainers.- The same name on different nodes is allowed.
Certificate of the webhook:
- With cert-manager installed, cert-manager issues the certificate.
- Without cert-manager, the operator issues a self-signed certificate. It stores the certificate in the
<release>-webhook-certSecret. It injects the CA into the webhook configuration. - Force one of the two with
--set webhook.certManager=true|false. - Disable the webhook with
--set webhook.enabled=false. The controllers then catch duplicate names. See the next section.
The oldest object owns a duplicate name
Two objects can still claim one name. This happens when they are created at the same moment, or when the webhook is disabled. Then:
- The oldest object owns the guest.
- The other objects get the condition
NameConflict=True. The condition names the owner. - The other objects are not reconciled against Proxmox.
- Deleting one of the other objects does not delete the guest.
- When the owner is gone, the next object takes the name within about 30 seconds.
Snapshot policies lost namespaceSelector
A VirtualMachineSnapshotPolicy only snapshots VirtualMachines in its own namespace. It selects them with its labelSelector and its connectionRef.
- Create one policy per namespace.
- Remove
namespaceSelectorfrom your manifests. Some clients reject the field. Other clients drop it silently.
Snapshots follow the VirtualMachine
A VirtualMachineSnapshot looks up its VirtualMachine in its own namespace. It snapshots the guest named spec.name on spec.nodeName of that VirtualMachine.
- Keep each snapshot in the same namespace as its VirtualMachine.
- The
connectionRefof the snapshot must name the connection of the VirtualMachine. If not, the snapshot reports an error and does nothing.
passwordFrom reads from the template namespace
cloudInitConfig.passwordFrom of a VirtualMachineTemplate now reads the Secret from the namespace of the template. Before, it read from the namespace of the operator. Copy the Secret into the namespace you restore the template into.
Upgrade steps
Danger
Deleting a VirtualMachine, Container or VirtualMachineTemplate while the operator runs deletes the guest on Proxmox. Do the steps in order. Do not skip step 1.
1. Stop the operator
kubectl scale deployment kubemox --replicas=0
2. Back up the objects
This also saves the VirtualMachineSnapshot objects as a record. You do not restore them. See step 3.
for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls virtualmachinesnapshots; do
kubectl get "$kind.proxmox.alperen.cloud" -A -o yaml > "backup-$kind.yaml"
done
3. Delete the old objects and CRDs
First remove the finalizers. Then the CRD deletion does not touch Proxmox.
for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls; do
kubectl get "$kind.proxmox.alperen.cloud" -o name | \
xargs -r -I{} kubectl patch {} --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl delete crd "$kind.proxmox.alperen.cloud"
done
Kubernetes deletes the VirtualMachineSnapshot objects at the same time. Their owner, the VirtualMachine, is gone. This does not delete the snapshots on Proxmox.
Do not restore these objects in step 5. The operator would take each snapshot again, and Proxmox refuses a snapshot name that exists. The snapshots stay on Proxmox. Create a new VirtualMachineSnapshot only when you want a new snapshot.
4. Upgrade the chart
The upgrade installs the new CRDs. It also sets the operator back to 1 replica. The operator starts.
Set the image tag. Helm keeps the values of the old release. If you pinned image.tag before, the old operator keeps running on the new CRDs. Then every step 5 command fails with failed calling webhook.
helm repo update
helm upgrade kubemox alperencelik/kubemox --set image.tag=v0.8.0
Wait until the operator is ready. Check the image. The webhook runs inside the operator. Step 5 fails until the webhook is ready.
kubectl rollout status deployment/kubemox
kubectl get deployment kubemox -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
The second command must print alperencelik/kubemox:v0.8.0.
5. Restore the objects into one namespace
Pick the namespace. Then restore the five kinds. The command removes the server-set fields. It skips the objects that another Kubemox object owns:
- A
VirtualMachineSetre-creates itsVirtualMachines. - A
VirtualMachineTemplatere-creates itsStorageDownloadURL. - Both adopt the existing guests on Proxmox by name.
NAMESPACE=default
STRIP='del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp,
.metadata.generation, .metadata.managedFields, .metadata.finalizers, .status)'
for kind in virtualmachines virtualmachinesets virtualmachinetemplates containers storagedownloadurls; do
yq ".items[] | select(.metadata.ownerReferences == null) | $STRIP | split_doc" "backup-$kind.yaml" \
| kubectl apply -n "$NAMESPACE" -f -
done
A kind with no objects prints error: no objects passed to apply. Ignore it.
If kubectl apply fails with failed calling webhook, wait a few seconds. Then run the command again. The webhook certificate was not ready yet. If it still fails after a minute, check the image from step 4.
6. Verify
kubectl get vm,vmset,vmt,lxc,sdu -n "$NAMESPACE"
Check that every object reaches its desired state. Check that no object has the condition NameConflict=True.