Immutable Infrastructure

Creating Clusters on Bare Metal

This document explains how to create Kubernetes clusters on physical servers using the bare-metal provider. The workflow is YAML-only — there is no Fleet Essentials UI for bare-metal clusters at this time.

TOC

Prerequisites1. Required Plugin Installation2. Image Catalog Confirmed3. Network Connectivity4. TPM Decision5. Public Registry Credential (Only When Installing Platform Components Later)Cluster Creation WorkflowResolving Placeholder ValuesStep 1: Build the SeedImage and Register HostsStep 2: Create MachineInventoryPool ResourcesStep 3: Create the Control-Plane Cluster ResourcesStep 4: Deploy Worker NodesCluster VerificationUsing kubectlExpected ResultsCommon Failure ModesNext StepsAppendixComplete KubeadmControlPlane Configuration

Prerequisites

Before creating clusters, ensure all of the following prerequisites are met.

1. Required Plugin Installation

Install the following plugins on the global cluster:

  • Alauda Container Platform Kubeadm Provider
  • Alauda Container Platform Bare Metal Infrastructure Provider (umbrella chart that installs both the bare-metal manager and elemental-operator)

See the Installation Guide for details.

2. Image Catalog Confirmed

The bare-metal provider chart ships an elemental-image-catalog ConfigMap that maps Machine.spec.version to the elemental upgrade image used to (re)provision a node. You do not need to create this ConfigMap separately — confirm that the target Kubernetes version is present:

kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml

Every value used as Machine.spec.version (for both the control plane and worker MachineDeployment resources) must appear as a key in this ConfigMap, with the leading v preserved. The provider resolves the image at reprovision time by substituting the platform registry address for the registry portion of the entry. If a target version is missing, no reprovision plan is written and BaremetalMachine ends up in Failed / Reason=ImageCatalogMiss until the entry is added.

3. Network Connectivity

  • Every physical host must be able to reach global.platformUrl (elemental-system-agent registration, plan secret polling).
  • Every physical host must be able to pull from the platform registry (global.registry.address) for both elemental install (during the first boot) and elemental upgrade (during every reprovision).
  • The control-plane VIP must live in the same Layer-2 broadcast domain as the control-plane node IPs. The vrid chosen for the VIP must be unique within that broadcast domain.

If the target VM or physical host does not receive an address from DHCP while booted into the live ISO, configure the network manually from the host console before waiting for registration. Check the NetworkManager connection name first, then apply the site-specific address, gateway, and DNS values:

nmcli connection show
nmcli connection modify "Wired connection 1" \
  ipv4.method manual \
  ipv4.addresses 192.168.254.73/24 \
  ipv4.gateway 192.168.254.1 \
  ipv4.dns 192.168.16.19 \
  ipv4.ignore-auto-dns yes
nmcli connection up "Wired connection 1"

4. TPM Decision

Production hosts should keep MachineRegistration.spec.config.elemental.registration.emulate-tpm: false (or remove the field). For PoC and virtual-machine smoke tests, set emulate-tpm: true and emulated-tpm-seed: -1 so registration works without a real TPM.

5. Public Registry Credential (Only When Installing Platform Components Later)

public-registry-credential is not required to create a bare-metal cluster. It only becomes necessary when later platform components on the new workload cluster need to pull from a credentialed public registry. If your test scope ends at cluster + node Ready, you can ignore this prerequisite.

Cluster Creation Workflow

When using YAML, the workflow proceeds through five steps. Every step must be applied in the cpaas-system namespace.

StepApply or performResult
1MachineRegistration and SeedImageelemental-operator builds the bootable ISO.
2Boot the ISO on each physical hostThe host runs elemental install and registers as a MachineInventory.
3MachineInventoryPool resources, one per roleThe allowed control-plane and worker inventories are declared before CAPI creates Machines.
4BaremetalCluster, control-plane BaremetalMachineTemplate, KubeadmControlPlane, and ClusterCluster API starts the control-plane rollout using the control-plane pool.
5Worker KubeadmConfigTemplate, worker BaremetalMachineTemplate, and MachineDeploymentCluster API starts the worker rollout using the worker pool.
WARNING

Important Namespace Requirement

All bare-metal resources must be applied in the cpaas-system namespace. The provider and elemental-operator only reconcile objects in that namespace.

WARNING

Workload Cluster Naming

The workload cluster-name must not be global. That name is reserved for the global cluster, and reusing it causes the workload cluster's resources to collide with global cluster resources in cpaas-system. As a convention, keep the CAPI Cluster and BaremetalCluster named exactly <cluster-name>, and prefix dependent resources (KubeadmControlPlane, KubeadmConfigTemplate, MachineDeployment, machine templates, pools, registrations) with <cluster-name>-.

Resolving Placeholder Values

The example manifests below use <placeholder> syntax for environment-specific values:

PlaceholderSource of truthHow to retrieve
<cluster-name>Operator-supplied workload cluster name. Must not be global.Choose once and use consistently for Cluster, BaremetalCluster, pools, templates, and registration resources.
<kubernetes-version>elemental-image-catalog ConfigMap key (with leading v, for example v1.33.7-2).kubectl -n cpaas-system get cm elemental-image-catalog -o yaml
<base-image-iso>Same ConfigMap entry, with -iso appended to the repository. Tag/digest matches the catalog value.See the SeedImage step.
<registry-address>Platform registry (same value as global.registry.address on the install chart).kubectl get cluster global -n cpaas-system -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}' (when one exists).
<control-plane-vip> / <control-plane-port> / <vrid>Operator-supplied. The VIP must be free in the control-plane Layer-2 domain; <vrid> must be unique in that domain.n/a
<install-device>Target disk for elemental install, such as /dev/vda, /dev/sda, or /dev/nvme0n1.Confirm from the host console or inventory hardware data. Do not leave this implicit in virtualized tests.
<dns-image-tag> / <etcd-image-tag>Component versions baked into the bare-metal base image for <kubernetes-version>.See OS Support Matrix.
<dns-server>Operator-supplied when the first-boot registration path needs an explicit resolver.Prefer MachineRegistration.config.cloud-config or later kubeadm cloud-init. Do not write /etc/resolv.conf from SeedImage.cloud-config.
<ssh-authorized-keys>Operator-supplied OpenSSH public key — required for any interactive debugging on the node.n/a
<inventory-name>Allocated by elemental-operator after the host boots the ISO and registers. Cannot be predicted before registration.kubectl -n cpaas-system get machineinventories.elemental.cattle.io
<pool-name>Name of the MachineInventoryPool that claims an inventory.Use <cluster-name>-control-plane-pool or <cluster-name>-worker-pool in the examples below.
<control-plane-inventory-1/2/3> / <worker-inventory-1/2/3>Exact MachineInventory names selected for each role.Use the names from kubectl -n cpaas-system get machineinventories.elemental.cattle.io.
<control-plane-host-1/2/3> / <worker-host-1>Optional Kubernetes node hostnames applied during reprovision.Operator-supplied; omit hostname to fall back to the inventory name.
<pods-cidr> / <services-cidr> / <kube-ovn-join-cidr>Operator-supplied. Must not overlap with the host network, the global cluster's CIDRs, or any other CAPI cluster on the same global.n/a
<machine-name>Existing CAPI Machine name used for targeted removal or recovery.kubectl -n cpaas-system get machines.cluster.x-k8s.io
<new-replica-count>Desired MachineDeployment.spec.replicas value after scaling.Operator-supplied target worker count.
<new-template-name>New immutable template name for a changed BaremetalMachineTemplate or KubeadmConfigTemplate.Operator-supplied; create a new name instead of editing an existing template in place.
<new-kubernetes-version> / <new-tag> / <new-image>Target upgrade version and matching bare-metal base-image tag or resolved image.Use the release support matrix and confirm the final key exists in elemental-image-catalog.

Step 1: Build the SeedImage and Register Hosts

Create a MachineRegistration that describes the registration URL and first-install cloud-config, and a SeedImage that points elemental-operator at the matching ISO base image.

SeedImage.spec.baseImage is derived from the image catalog entry for the target Kubernetes version: take the catalog repository, append -iso, and keep the same tag or digest. For example, if elemental-image-catalog resolves v1.33.7-2 to <registry-address>/tkestack/baremetal-base-image:v0.0.0-beta-1.33.7-2, then SeedImage.spec.baseImage is <registry-address>/tkestack/baremetal-base-image-iso:v0.0.0-beta-1.33.7-2.

01-machineregistration-seedimage.yaml
apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: <cluster-name>-registration
  namespace: cpaas-system
  labels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
spec:
  # ${...} placeholders are expanded by elemental-register from SMBIOS data.
  # System UUID is preferred over Serial Number because it is more reliable
  # across virtualised test environments.
  machineName: "<cluster-name>-${System Information/UUID}"
  machineInventoryLabels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
    pool.baremetal.alauda.io/eligible: "true"
    elemental.cattle.io/serial-number: "${System Information/Serial Number}"
  machineInventoryAnnotations:
    elemental.cattle.io/machine-uuid: "${System Information/UUID}"
  config:
    elemental:
      install:
        device: <install-device>
        eject-cd: true
        reboot: true
        snapshotter:
          type: btrfs
          maxSnaps: 4
      registration:
        # Keep false for production hardware and for VM tests that already
        # expose stable SMBIOS identity. Set true only for lab hosts without
        # a usable TPM or stable identity source.
        emulate-tpm: false
---
apiVersion: elemental.cattle.io/v1beta1
kind: SeedImage
metadata:
  name: <cluster-name>-registration-iso
  namespace: cpaas-system
  labels:
    app.kubernetes.io/part-of: cluster-api-provider-baremetal
spec:
  type: iso
  baseImage: <base-image-iso>          # See the table above for derivation.
  registrationRef:
    apiVersion: elemental.cattle.io/v1beta1
    kind: MachineRegistration
    name: <cluster-name>-registration
    namespace: cpaas-system
  targetPlatform: linux/amd64
  size: 20Gi
  cleanupAfterMinutes: 120

Do not add /etc/resolv.conf to SeedImage.spec.cloud-config. Keep the ISO generic, and put site-specific resolver configuration in MachineRegistration.spec.config.cloud-config only when the first-boot registration path needs it. During the tested Global deployment flow, the ISO did not carry resolver files; node DNS was configured later by the kubeadm bootstrap data.

install.device, install.eject-cd, and install.reboot are intentional. If the target disk is omitted, elemental install can select the live ISO device in VM-based tests. If eject-cd or reboot is false, a host can remain in the live environment after the first install and never become usable inventory for Cluster API.

Apply the manifest and wait for the SeedImage build to finish:

kubectl apply -f 01-machineregistration-seedimage.yaml
kubectl -n cpaas-system get seedimage <cluster-name>-registration-iso -w

When status.state reaches Completed, fetch the download URL and ISO checksum:

kubectl -n cpaas-system get seedimage <cluster-name>-registration-iso \
  -o jsonpath='{.status.downloadURL}{"\n"}'
kubectl -n cpaas-system get seedimage <cluster-name>-registration-iso \
  -o jsonpath='{.status.checksumURL}{"\n"}'

Boot every target host from this ISO. elemental-register runs first (creates the MachineInventory and uploads observedNetwork), then elemental install writes the on-disk OS. After install completes, the host stays available for plan execution. If the live ISO environment has no DHCP address, configure NetworkManager manually on the host console as described in Network Connectivity before waiting for the MachineInventory.

Confirm registration:

kubectl -n cpaas-system get machineinventories.elemental.cattle.io

Every inventory you intend to use must:

  • Show Ready=True.
  • Have a non-empty status.plan.secretRef.name.
  • Have a spec.observedNetwork that matches the host's expected NIC (only required when you want the install-time IP to survive across reprovisions).

Record the exact MachineInventory names — they are referenced by name in the next step.

Step 2: Create MachineInventoryPool Resources

Create one pool per role. The pool reconciler validates that every member exists, computes capacity counters, and writes the baremetal.alauda.io/pool=<pool-name> annotation onto the inventory.

02-machineinventorypool.yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: <cluster-name>-control-plane-pool
  namespace: cpaas-system
spec:
  clusterName: <cluster-name>
  machineInventories:
    - name: <control-plane-inventory-1>
      hostname: <control-plane-host-1>    # Optional; falls back to the inventory name.
    - name: <control-plane-inventory-2>
      hostname: <control-plane-host-2>
    - name: <control-plane-inventory-3>
      hostname: <control-plane-host-3>
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: <cluster-name>-worker-pool
  namespace: cpaas-system
spec:
  clusterName: <cluster-name>
  machineInventories:
    - name: <worker-inventory-1>
      hostname: <worker-host-1>
    - name: <worker-inventory-2>
    - name: <worker-inventory-3>

Key parameters:

ParameterTypeDescriptionRequired
.spec.clusterNamestringThe workload cluster name this pool serves. A MachineInventory must not appear in two active pools.Yes
.spec.machineInventories[].namestringExact name of a registered MachineInventory.Yes
.spec.machineInventories[].hostnamestringHostname applied during reprovision. Defaults to the inventory name when omitted.No

Apply and verify:

kubectl apply -f 02-machineinventorypool.yaml
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io

A healthy pool reports Ready=True, total = len(spec.machineInventories), and available = total - allocated - preparing - reprovisioning - unavailable. Inventories listed in spec.machineInventories that fail validation (missing, plan secret missing, Ready=False) raise the pool's unavailable counter and surface in the MembersValid condition.

Size the control-plane pool to at least KubeadmControlPlane.spec.replicas. Size the worker pool to at least MachineDeployment.spec.replicas. For rolling upgrades the pool must hold the entire replica count — the provider uses delete-then-add semantics from the same pool, never both at once.

Step 3: Create the Control-Plane Cluster Resources

Create the BaremetalCluster (declares the control-plane VIP), the control-plane BaremetalMachineTemplate (points at the control-plane pool), the KubeadmControlPlane (replicas + kubeadm config), and the CAPI Cluster.

TIP

Full Configuration Reference

The example below uses a minimal KubeadmControlPlane. For the full hardening profile recommended in production — admission, audit, kubelet patches, encryption provider — see Complete KubeadmControlPlane Configuration in the Appendix.

03-cluster.yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: BaremetalCluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
spec:
  controlPlaneLoadBalancer:
    type: Internal           # External: skip alive deployment, supply your own LB.
    host: <control-plane-vip>
    port: <control-plane-port>
    vrid: <vrid>
  networkType: kube-ovn      # Phase-1 metadata only.
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: BaremetalMachineTemplate
metadata:
  name: <cluster-name>-control-plane-template
  namespace: cpaas-system
spec:
  template:
    spec:
      machineInventoryPoolRef:
        name: <cluster-name>-control-plane-pool
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <cluster-name>-control-plane
  namespace: cpaas-system
spec:
  replicas: 3
  version: <kubernetes-version>
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0            # Bare-metal does not over-provision; replace one at a time.
  machineTemplate:
    nodeDrainTimeout: 5m
    nodeVolumeDetachTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: BaremetalMachineTemplate
      name: <cluster-name>-control-plane-template
  kubeadmConfigSpec:
    users:
      - name: boot
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        sshAuthorizedKeys:
          - "<ssh-authorized-keys>"
    clusterConfiguration:
      imageRepository: <registry-address>/tkestack
      dns:
        imageTag: <dns-image-tag>
      etcd:
        local:
          imageTag: <etcd-image-tag>
      apiServer:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
      controllerManager:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
      scheduler:
        extraArgs:
          profiling: "false"
          tls-min-version: VersionTLS12
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
    joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster-name>
  namespace: cpaas-system
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: BaremetalCluster
    cpaas.io/kube-ovn-join-cidr: <kube-ovn-join-cidr>
    cpaas.io/sentry-deploy-type: Baremetal
    cpaas.io/alb-address-type: ClusterAddress
  labels:
    cluster-type: ProviderBaremetal
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - <pods-cidr>
    services:
      cidrBlocks:
        - <services-cidr>
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: <cluster-name>-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: BaremetalCluster
    name: <cluster-name>

Cluster annotations. The bare-metal provider relies on a small set of Cluster annotations during reconcile. Authoritative ones the operator must set:

AnnotationRequiredValue sourcePurpose
capi.cpaas.io/resource-group-versionYesLiteral infrastructure.cluster.x-k8s.io/v1beta1CAPI infrastructure binding.
capi.cpaas.io/resource-kindYesLiteral BaremetalClusterCAPI infrastructure binding.
cpaas.io/kube-ovn-join-cidrYesOperator-chosen /16 CIDR that does not overlap with pods/services or any other cluster.Kube-OVN inter-node tunnel.
cpaas.io/sentry-deploy-typeYesLiteral Baremetal.Marks the cluster for the bare-metal deploy profile.
cpaas.io/alb-address-typeYesLiteral ClusterAddress.ALB address mode used on bare-metal clusters.

BaremetalCluster parameters:

ParameterTypeDescriptionRequired
.spec.controlPlaneLoadBalancer.typestring (Internal / External)Internal deploys alive static pods on the control-plane nodes and manages the VIP. External skips alive and assumes an external LB already terminates <control-plane-vip>:<control-plane-port>.No (defaults to Internal)
.spec.controlPlaneLoadBalancer.hoststringControl-plane VIP. Backfilled into .spec.controlPlaneEndpoint.host when the endpoint is left empty.Yes
.spec.controlPlaneLoadBalancer.portint (1–65535)Control-plane port. Typically 6443.Yes
.spec.controlPlaneLoadBalancer.vridint (0–255)keepalived virtual_router_id. Must be unique in the control-plane Layer-2 domain. Only consumed when type=Internal.When type=Internal
.spec.controlPlaneEndpointobjectAPI server endpoint exposed to CAPI. Once set, must not change. Leave empty to let the reconciler backfill it from controlPlaneLoadBalancer.No
.spec.networkTypestringCNI hint. kube-ovn today (Phase-1 metadata only).No

Apply and watch:

kubectl apply -f 03-cluster.yaml
kubectl -n cpaas-system get clusters.cluster.x-k8s.io
kubectl -n cpaas-system get kubeadmcontrolplanes.controlplane.cluster.x-k8s.io
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w

Each new control-plane BaremetalMachine advances Pending → Allocated → Reprovisioning → Running. Watch:

  • BaremetalMachine.status.machineInventoryRef.name — which inventory was picked.
  • BaremetalMachine.status.planSecretRef.name — plan secret being driven. The secret carries baremetal.alauda.io/plan.type=reprovision.
  • MachineInventory.status.plan.state — Applied once the host completes cloud-init clean, elemental upgrade, reboot, and kubeadm init/join.
  • BaremetalCluster.status.conditions[EndpointReady] — true once the VIP is reachable.
WARNING

The bare-metal provider does not support single-node control planes. Provision at least three control-plane replicas (KubeadmControlPlane.spec.replicas: 3) so that alive can arbitrate the VIP and etcd retains quorum.

Step 4: Deploy Worker Nodes

After the control plane is Ready, create the worker BaremetalMachineTemplate, the worker KubeadmConfigTemplate, and the MachineDeployment. The full worker YAML and parameter table are in Managing Nodes on Bare Metal → Worker Node Deployment.

Cluster Verification

Using kubectl

# Cluster status
kubectl -n cpaas-system get cluster <cluster-name>

# Control plane progress
kubectl -n cpaas-system get kubeadmcontrolplane <cluster-name>-control-plane

# Per-machine state
kubectl -n cpaas-system get machines.cluster.x-k8s.io
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io

# Pool counters
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io

# Workload cluster Nodes (use the workload kubeconfig)
kubectl get nodes -o wide

Expected Results

A successfully created cluster shows:

  • Cluster.status.conditions[Ready]=True.
  • KubeadmControlPlane replicas all Ready.
  • Every BaremetalMachine.status.phase=Running and status.ready=true.
  • Every used MachineInventory.status.plan.state=Applied with baremetal.alauda.io/plan.type=reprovision on its plan secret.
  • MachineInventoryPool.status satisfies available + allocated + preparing + reprovisioning + unavailable = total.
  • Kubernetes Nodes Ready.

Common Failure Modes

SymptomLikely causeWhere to look
BaremetalMachine stuck in Pending, InventoryAllocated=False / Reason=PoolMissingBaremetalMachineTemplate.spec.template.spec.machineInventoryPoolRef.name points at a non-existent pool.Pool name + namespace.
BaremetalMachine stuck in Pending, InventoryAllocated=False / Reason=PoolExhaustedPool has no Available inventory left.MachineInventoryPool.status.available; allocation annotations on each member.
BaremetalMachine stuck after allocation, BootstrapReady=False / Reason=BootstrapWaitingKubeadmConfig has not produced a bootstrap data secret yet.Machine.spec.bootstrap.dataSecretName.
BaremetalMachine in Failed, ImageResolved=False / Reason=ImageCatalogMissThe target Machine.spec.version is not a key in elemental-image-catalog.kubectl -n cpaas-system get cm elemental-image-catalog -o yaml.
ImageResolved=False / Reason=ImageRegistryMissingNeither the optional cpaas.io/registry-address annotation nor the platform registry-credential Secret resolves a registry address.Cluster annotations; platform registry-credential Secret.
Reprovision never completes; MachineInventory.status.plan.state=Failedelemental upgrade failed (registry unreachable, TLS, disk full).Plan secret failed-output field; host serial console.
BaremetalCluster Ready, but the VIP is not reachablevrid collides with another cluster; control-plane VIP not in the L2 domain; firewall blocking VRRP.kubectl -n kube-system get pod -l app=alive, ip addr show, ipvsadm -Ln on a control-plane host.

For the full operator-side state machine reference (every condition reason and recovery action), see Provider Overview → clean / reprovision plans.

Next Steps

After creating a cluster:

Appendix

Complete KubeadmControlPlane Configuration

The hardened configuration recommended for production bare-metal clusters — admission control, audit policy, kubelet patches, encryption provider, and IPv6 bind addresses. Substitute the placeholders from the table in Resolving Placeholder Values.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <cluster-name>-control-plane
  namespace: cpaas-system
spec:
  replicas: 3
  version: <kubernetes-version>
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
  machineTemplate:
    nodeDrainTimeout: 5m
    nodeVolumeDetachTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: BaremetalMachineTemplate
      name: <cluster-name>-control-plane-template
  kubeadmConfigSpec:
    users:
      - name: boot
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        sshAuthorizedKeys:
          - "<ssh-authorized-keys>"
    files:
      - path: /etc/kubernetes/admission/psa-config.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: apiserver.config.k8s.io/v1
          kind: AdmissionConfiguration
          plugins:
          - name: PodSecurity
            configuration:
              apiVersion: pod-security.admission.config.k8s.io/v1
              kind: PodSecurityConfiguration
              defaults:
                enforce: "privileged"
                enforce-version: "latest"
                audit: "baseline"
                audit-version: "latest"
                warn: "baseline"
                warn-version: "latest"
              exemptions:
                usernames: []
                runtimeClasses: []
                namespaces:
                - kube-system
                - cpaas-system
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: "root:root"
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "protectKernelDefaults": true,
            "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
            "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key",
            "streamingConnectionIdleTimeout": "5m",
            "clientCAFile": "/etc/kubernetes/pki/ca.crt"
          }
      - path: /etc/kubernetes/audit/policy.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: audit.k8s.io/v1
          kind: Policy
          omitStages:
          - "RequestReceived"
          rules:
          - level: None
            users:
            - system:kube-controller-manager
            - system:kube-scheduler
            - system:serviceaccount:kube-system:endpoint-controller
            verbs: ["get", "update"]
            namespaces: ["kube-system"]
            resources:
            - group: ""
              resources: ["endpoints"]
          - level: None
            nonResourceURLs:
            - /healthz*
            - /version
            - /swagger*
          - level: None
            resources:
            - group: ""
              resources: ["events"]
          - level: None
            verbs: ["get", "list", "watch"]
          - level: None
            resources:
            - group: "coordination.k8s.io"
              resources: ["leases"]
          - level: None
            resources:
            - group: "authorization.k8s.io"
              resources: ["subjectaccessreviews", "selfsubjectaccessreviews"]
            - group: "authentication.k8s.io"
              resources: ["tokenreviews"]
          - level: Metadata
            resources:
            - group: ""
              resources: ["secrets", "configmaps"]
          - level: RequestResponse
            resources:
            - group: ""
            - group: "apps"
            - group: "rbac.authorization.k8s.io"
            - group: "storage.k8s.io"
            - group: "networking.k8s.io"
          - level: Metadata
    postKubeadmCommands:
      - chmod 600 /var/lib/kubelet/config.yaml
    clusterConfiguration:
      imageRepository: <registry-address>/tkestack
      dns:
        imageTag: <dns-image-tag>
      etcd:
        local:
          imageTag: <etcd-image-tag>
      apiServer:
        extraArgs:
          audit-log-format: json
          audit-log-maxage: "30"
          audit-log-maxbackup: "10"
          audit-log-maxsize: "200"
          profiling: "false"
          audit-log-mode: batch
          audit-log-path: /etc/kubernetes/audit/audit.log
          audit-policy-file: /etc/kubernetes/audit/policy.yaml
          tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
          admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml
          tls-min-version: VersionTLS12
          kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
        extraVolumes:
        - name: vol-dir-0
          hostPath: /etc/kubernetes
          mountPath: /etc/kubernetes
          pathType: Directory
      controllerManager:
        extraArgs:
          bind-address: "::"
          profiling: "false"
          tls-min-version: VersionTLS12
      scheduler:
        extraArgs:
          bind-address: "::"
          tls-min-version: VersionTLS12
          profiling: "false"
    initConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"
    joinConfiguration:
      patches:
        directory: /etc/kubernetes/patches
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "kube-ovn/role=master"

Worker bootstrap is symmetric — see Managing Nodes on Bare Metal → Bootstrap Template for the worker KubeadmConfigTemplate.