#Global Cluster Disaster Recovery

This page is the disaster recovery entry point for a global cluster that runs on Immutable Infrastructure. Deploy-time disaster recovery configuration is part of the global cluster installation procedure.

#Deployment Procedure

Use Optional Disaster Recovery Deployment in the installation guide when you create the primary and standby global clusters.

That deployment procedure is the authoritative source for the installation-time DR configuration, including:

• Primary and standby clusters use the same Kubernetes API server encryption provider configuration.
• The etcd server certificate SAN list includes both the primary and standby control plane VIPs and the Platform Access Address.
• Huawei DCS deployments reference the shared encryption provider Secret from DCSCluster.spec.encryptionProviderConfigRef.
• VMware vSphere deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Huawei Cloud Stack deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Bare Metal deployments reference the shared encryption provider Secret from BaremetalCluster.spec.encryptionProviderConfigRef and use the same Kubernetes ServiceAccount signing key on both sides.
• VMware vSphere and Huawei Cloud Stack create the dcs-import-extra-resources ConfigMap before installer import so the installation can preserve provider-specific resources. The name keeps the dcs prefix for historical installer compatibility. Huawei DCS uses the built-in provider resource migration unless extra resources must be imported.
• Bare Metal creates the dcs-import-extra-resources ConfigMap before installer import so the installation can preserve bare-metal and elemental resources required by handoff.
• The primary cluster installs global-etcd-sync with the standby cluster connection values after both installations succeed.

#Operational Scope

After the primary and standby clusters are installed, operate DR as a separate lifecycle process. Keep the installation manifests aligned with the installation guide, then use an approved operations runbook for the following tasks:

• Verify etcd-sync health and replication lag.
• Verify that the standby cluster can decrypt Kubernetes Secrets created on the primary cluster.
• Validate the primary and standby control plane VIPs and the platform access path before a planned failover.
• Execute failover and failback with an approved operations procedure.
• Reconcile provider-specific resources after a failover.

#Bare Metal DR Model

Bare Metal DR adds host-level state that does not exist in VM-based providers. The global cluster does not create physical machines; it manages long-lived MachineInventory objects and writes plan Secrets that are consumed by elemental-system-agent on each host. Because of that, a standby global cluster must be able to accept the existing system-agent token and must receive the workload cluster inventory and plan state without overwriting its own global inventory.

#Endpoint Rules

Use different system-agent endpoints for global hosts and workload-cluster hosts:

Both endpoints are platform ingress paths, not direct kube-apiserver :6443 endpoints. For the bootstrap KIND phase, global host registration uses the bootstrap host endpoint, normally https://<bootstrap-host-ip>:12443. Handoff changes only the installed global hosts to the final VIP endpoint.

For a post-handoff global host registration, set the system-agent server URL annotation to the current active global control-plane VIP. The elemental operator appends the /kubernetes/global path when it renders the system-agent endpoint. Use the primary VIP before failover and the standby VIP after failover.

apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: global-<purpose>
  namespace: cpaas-system
  annotations:
    baremetal.cluster.io/system-agent-server-url: https://<current-global-control-plane-vip>

#Shared Token Requirements

Existing hosts keep the baremetal-system-agent token that they received before failover. For that token to work on the standby cluster:

• Primary and standby must use the same /etc/kubernetes/pki/sa.key and /etc/kubernetes/pki/sa.pub.
• Primary and standby must use the same service-account-issuer and api-audiences values, normally https://kubernetes.default.svc.cluster.local.
• The ServiceAccount, token Secret, Role, and RoleBinding for baremetal-system-agent must be synchronized exactly to standby.
• Role/baremetal-system-agent.rules[].resourceNames must contain only plan Secret names. Do not grant namespace-wide Secret access and do not add registry, bootstrap, kubeconfig, or platform credential Secrets.

Verify the permission boundary on standby before failover.

TOKEN=$(kubectl -n cpaas-system get secret baremetal-system-agent-token -o jsonpath='{.data.token}' | base64 -d)

kubectl --token="${TOKEN}" -n cpaas-system auth can-i get secret <plan-secret-name>
kubectl --token="${TOKEN}" -n cpaas-system auth can-i patch secret <plan-secret-name>
kubectl --token="${TOKEN}" -n cpaas-system auth can-i get secret global-registry-auth
kubectl --token="${TOKEN}" -n cpaas-system auth can-i list secrets

Expected result: get and patch are allowed only for plan Secrets; global-registry-auth and namespace-wide list secrets are denied.

#etcd-sync Scope

Deploy etcd-sync on the standby cluster. The sync direction is one-way before failover:

source: primary global etcd
target: standby global etcd

Synchronize the workload-cluster resources that allow standby to continue reconciling existing Bare Metal clusters:

• Non-global CAPI resources: Cluster, Machine, MachineDeployment, KubeadmControlPlane, bootstrap templates, and related owner resources.
• Non-global Bare Metal provider resources: BaremetalCluster, BaremetalMachine, BaremetalMachineTemplate, and MachineInventoryPool.
• Non-global elemental resources: MachineInventory and MachineRegistration.
• Plan Secrets, bootstrap data Secrets, workload kubeconfig Secrets, and workload credential Secrets that are required by the synchronized resources.
• Provider referenced Secrets that must exist on the target side, including the encryption-provider Secret referenced by BaremetalCluster.spec.encryptionProviderConfigRef when Bare Metal DR is enabled.
• Exact shared auth bundle: ServiceAccount/cpaas-system/baremetal-system-agent, Secret/cpaas-system/baremetal-system-agent-token, Role/cpaas-system/baremetal-system-agent, and RoleBinding/cpaas-system/baremetal-system-agent.

Do not synchronize:

• SeedImage.
• Standby Cluster/cpaas-system/global.
• Standby BaremetalCluster/cpaas-system/global.
• Standby BaremetalMachine, MachineInventoryPool, MachineInventory, and MachineRegistration objects that belong to the standby global cluster.
• Secrets from the primary cluster that are not required by workload cluster reconciliation.

#Failover Procedure

Use this sequence for a planned or declared failover:

1. Fence the primary global write path. Stop or isolate the controllers and platform entry points that can continue writing workload cluster resources. Do this at the owning CR or platform-resource layer where possible; scaling a Deployment alone can be reverted by its owner.

2. Wait until primary-to-standby etcd-sync is caught up, then freeze or stop it. Do not allow old primary data to overwrite new standby writes after failover.

3. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress.

4. Switch the platform domain to the standby platform entrance.

5. Verify DNS and ingress:

   dig +short <platform-domain>
   curl -kI https://<platform-domain>/kubernetes/global/version
   curl -kI https://<standby-control-plane-vip>/kubernetes/global/version

   A 401 response is acceptable; it proves the request reached the platform ingress.

6. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components.

7. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets.

#Post-Failover Validation

After standby is active, validate in increasing risk order:

• Existing workload cluster remains Ready.
• All BaremetalMachine objects remain bound to the expected MachineInventory objects.
• Plan Secrets can be updated and the MachineInventory.status.plan feedback changes.
• Worker scale-out allocates an available inventory and updates Role/baremetal-system-agent.resourceNames with the new plan Secret.
• Worker scale-in runs a clean plan and releases inventory as designed.
• Control-plane replacement restores the desired KCP replica count and etcd membership.
• A new MachineRegistration and SeedImage created on standby can register a new host.
• A pre-failover workload ISO is usable only if its registration URL uses the platform domain.

#Bare Metal Troubleshooting

#Provider Notes

#See Also

For traditional-operating-system global cluster disaster recovery, see Global Cluster Disaster Recovery.

For installation and upgrade of the global cluster on Immutable Infrastructure, see:

• Installing the global Cluster
• Upgrading the global Cluster