Cluster API on Kubevirt
Cluster API on Kubevirt
Install Cluster API on homelab cluster
➜ cluster-api clusterctl init --infrastructure kubevirt
Fetching providers
Skipping installing cert-manager as it is already installed
Installing provider="cluster-api" version="v1.8.1" targetNamespace="capi-system"
Installing provider="bootstrap-kubeadm" version="v1.8.1" targetNamespace="capi-kubeadm-bootstrap-system"
Installing provider="control-plane-kubeadm" version="v1.8.1" targetNamespace="capi-kubeadm-control-plane-system"
Installing provider="infrastructure-kubevirt" version="v0.1.9" targetNamespace="capk-system"
[KubeAPIWarningLogger] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "manager" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "manager" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "manager" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "manager" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Your management cluster has been initialized successfully!
You can now create your first workload cluster by running the following:
clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
## This gives us new ns
➜ cluster-api k get ns | grep cap
capi-kubeadm-bootstrap-system Active 67s
capi-kubeadm-control-plane-system Active 66s
capi-system Active 69s
capk-system Active 65s
## verify resources
➜ cluster-api k get po -n capi-system
NAME READY STATUS RESTARTS AGE
capi-controller-manager-68fbd598c5-6kgxc 1/1 Running 0 104s
➜ cluster-api k get po -n capk-system
NAME READY STATUS RESTARTS AGE
capk-controller-manager-795548fb75-2fxc4 1/1 Running 0 102s
➜ cluster-api k get po -n capi-kubeadm-control-plane-system
NAME READY STATUS RESTARTS AGE
capi-kubeadm-control-plane-controller-manager-79cf6494bf-kmkwf 1/1 Running 0 106s
➜ cluster-api k get po -n capi-kubeadm-bootstrap-system
NAME READY STATUS RESTARTS AGE
capi-kubeadm-bootstrap-controller-manager-554b87b54b-8zg62 1/1 Running 0 109s
Deploy Workload cluster
➜ cluster-api export CAPK_GUEST_K8S_VERSION="v1.26.0"
export CRI_PATH="/var/run/containerd/containerd.sock"
export NODE_VM_IMAGE_TEMPLATE="quay.io/capk/ubuntu-2004-container-disk:${CAPK_GUEST_K8S_VERSION}"
## Generate capi manifests
➜ cluster-api clusterctl generate cluster capi-quickstart --infrastructure="kubevirt" \
--flavor lb \
--kubernetes-version ${CAPK_GUEST_K8S_VERSION} \
--control-plane-machine-count=1 \
--worker-machine-count=1 > capi-quickstart.yaml
➜ cluster-api ls
capi-quickstart.yaml
The above capi config file looks like this
➜ cluster-api cat capi-quickstart.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: capi-quickstart
namespace: default
spec:
clusterNetwork:
pods:
cidrBlocks:
- 10.243.0.0/16
services:
cidrBlocks:
- 10.95.0.0/16
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: capi-quickstart-control-plane
namespace: default
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtCluster
name: capi-quickstart
namespace: default
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtCluster
metadata:
name: capi-quickstart
namespace: default
spec:
controlPlaneServiceTemplate:
spec:
type: LoadBalancer
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
name: capi-quickstart-control-plane
namespace: default
spec:
template:
spec:
virtualMachineBootstrapCheck:
checkStrategy: ssh
virtualMachineTemplate:
metadata:
namespace: default
spec:
runStrategy: Always
template:
spec:
domain:
cpu:
cores: 2
devices:
disks:
- disk:
bus: virtio
name: containervolume
networkInterfaceMultiqueue: true
memory:
guest: 4Gi
evictionStrategy: External
volumes:
- containerDisk:
image: quay.io/capk/ubuntu-2004-container-disk:v1.30.3 ## ==> verify version
name: containervolume
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: capi-quickstart-control-plane
namespace: default
spec:
kubeadmConfigSpec:
clusterConfiguration:
networking:
dnsDomain: capi-quickstart.default.local
podSubnet: 10.243.0.0/16
serviceSubnet: 10.95.0.0/16
initConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
joinConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
machineTemplate:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
name: capi-quickstart-control-plane
namespace: default
replicas: 1
version: v1.30.3 ## ==> verify version
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
template:
spec:
virtualMachineBootstrapCheck:
checkStrategy: ssh
virtualMachineTemplate:
metadata:
namespace: default
spec:
runStrategy: Always
template:
spec:
domain:
cpu:
cores: 2
devices:
disks:
- disk:
bus: virtio
name: containervolume
networkInterfaceMultiqueue: true
memory:
guest: 4Gi
evictionStrategy: External
volumes:
- containerDisk:
image: quay.io/capk/ubuntu-2004-container-disk:v1.30.3 ## ==> verify version
name: containervolume
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
kubeletExtraArgs: {}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
clusterName: capi-quickstart
replicas: 1
selector:
matchLabels: null
template:
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: capi-quickstart-md-0
namespace: default
clusterName: capi-quickstart
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: KubevirtMachineTemplate
name: capi-quickstart-md-0
namespace: default
version: v1.30.3 ## ==> verify version
Applying the above file
➜ cluster-api k apply -f capi-quickstart.yaml
cluster.cluster.x-k8s.io/capi-quickstart created
kubevirtcluster.infrastructure.cluster.x-k8s.io/capi-quickstart created
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane created
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 created
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 created
We can check status
➜ cluster-api k get cluster
NAME CLUSTERCLASS PHASE AGE VERSION
capi-quickstart Provisioned 17s
## Status for vms
➜ cluster-api k get po
NAME READY STATUS RESTARTS AGE
virt-launcher-capi-quickstart-control-plane-sdzrh-q2jcv 3/3 Running 0 5m12s
virt-launcher-capi-quickstart-md-0-n9z5r-qztm2-sgcsq 3/3 Running 0 2m51s
➜ cluster-api k get vm
NAME AGE STATUS READY
capi-quickstart-control-plane-sdzrh 5m14s Running True
capi-quickstart-md-0-n9z5r-qztm2 2m53s Running True
Get the kubeconfig for the above cluster
➜ cluster-api clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig
## verify the kubeconfig file
➜ cluster-api k get no --kubeconfig=capi-quickstart.kubeconfig
NAME STATUS ROLES AGE VERSION
capi-quickstart-control-plane-sdzrh NotReady control-plane 7m49s v1.26.0
capi-quickstart-md-0-n9z5r-qztm2 NotReady <none> 6m42s v1.26.0
➜ cluster-api k get po -A --kubeconfig=capi-quickstart.kubeconfig
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-787d4945fb-7pl52 0/1 Pending 0 8m24s
kube-system coredns-787d4945fb-mkgkw 0/1 Pending 0 8m24s
kube-system etcd-capi-quickstart-control-plane-sdzrh 1/1 Running 0 8m38s
kube-system kube-apiserver-capi-quickstart-control-plane-sdzrh 1/1 Running 0 8m38s
kube-system kube-controller-manager-capi-quickstart-control-plane-sdzrh 1/1 Running 0 8m38s
kube-system kube-proxy-bmjhz 1/1 Running 0 7m32s
kube-system kube-proxy-cqwkg 1/1 Running 0 8m25s
kube-system kube-scheduler-capi-quickstart-control-plane-sdzrh 1/1 Running 0 8m38s
## Install CNI to that cluster
# Download the calico file
➜ cluster-api curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.4/manifests/calico.yaml -o calico-workload.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 229k 100 229k 0 0 257k 0 --:--:-- --:--:-- --:--:-- 257k
## Edit according the quickstart demo for kubevirt on capi site
➜ cluster-api sed -i -E 's|^( +)# (- name: CALICO_IPV4POOL_CIDR)$|\1\2|g;'\
's|^( +)# ( value: )"192.168.0.0/16"|\1\2"10.243.0.0/16"|g;'\
'/- name: CLUSTER_TYPE/{ n; s/( +value: ").+/\1k8s"/g };'\
'/- name: CALICO_IPV4POOL_IPIP/{ n; s/value: "Always"/value: "Never"/ };'\
'/- name: CALICO_IPV4POOL_VXLAN/{ n; s/value: "Never"/value: "Always"/};'\
'/# Set Felix endpoint to host default action to ACCEPT./a\ - name: FELIX_VXLANPORT\n value: "6789"' \
calico-workload.yaml
## Apply
➜ cluster-api k create -f calico-workload.yaml --kubeconfig=capi-quickstart.kubeconfig
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
## Verify
➜ cluster-api k get po -A --kubeconfig=capi-quickstart.kubeconfig
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c79bd4b6-89swr 1/1 Running 0 82s
kube-system calico-node-245qt 1/1 Running 0 82s
kube-system calico-node-l9c26 1/1 Running 0 82s
kube-system coredns-787d4945fb-7pl52 1/1 Running 0 12m
kube-system coredns-787d4945fb-mkgkw 1/1 Running 0 12m
kube-system etcd-capi-quickstart-control-plane-sdzrh 1/1 Running 0 12m
kube-system kube-apiserver-capi-quickstart-control-plane-sdzrh 1/1 Running 0 12m
kube-system kube-controller-manager-capi-quickstart-control-plane-sdzrh 1/1 Running 0 12m
kube-system kube-proxy-bmjhz 1/1 Running 0 11m
kube-system kube-proxy-cqwkg 1/1 Running 0 12m
kube-system kube-scheduler-capi-quickstart-control-plane-sdzrh 1/1 Running 0 12m
➜ cluster-api k get no --kubeconfig=capi-quickstart.kubeconfig
NAME STATUS ROLES AGE VERSION
capi-quickstart-control-plane-sdzrh Ready control-plane 12m v1.26.0
capi-quickstart-md-0-n9z5r-qztm2 Ready <none> 11m v1.26.0
We can verify full cluster condition by following
➜ cluster-api clusterctl describe cluster capi-quickstart
NAME READY SEVERITY REASON SINCE MESSAGE
Cluster/capi-quickstart True 12m
├─ClusterInfrastructure - KubevirtCluster/capi-quickstart True 14m
├─ControlPlane - KubeadmControlPlane/capi-quickstart-control-plane True 12m
│ └─Machine/capi-quickstart-control-plane-sdzrh True 12m
└─Workers
└─MachineDeployment/capi-quickstart-md-0 True 89s
└─Machine/capi-quickstart-md-0-n9z5r-qztm2 True 11m
Increase Replica for Worker node
## In the machinedeployment part, changed the replicas for md-0 nodes to 2
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
clusterName: capi-quickstart
replicas: 2 ## ==> this part
➜ cluster-api k apply -f capi-quickstart.yaml
cluster.cluster.x-k8s.io/capi-quickstart unchanged
kubevirtcluster.infrastructure.cluster.x-k8s.io/capi-quickstart unchanged
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane unchanged
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane unchanged
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 unchanged
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 configured
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 configured
➜ cluster-api k get vm
NAME AGE STATUS READY
capi-quickstart-control-plane-sdzrh 21m Running True
capi-quickstart-md-0-n9z5r-kqzb9 81s Running True
capi-quickstart-md-0-n9z5r-qztm2 19m Running True
➜ cluster-api k get po
NAME READY STATUS RESTARTS AGE
virt-launcher-capi-quickstart-control-plane-sdzrh-q2jcv 3/3 Running 0 21m
virt-launcher-capi-quickstart-md-0-n9z5r-kqzb9-wrr5q 3/3 Running 0 83s
virt-launcher-capi-quickstart-md-0-n9z5r-qztm2-sgcsq 3/3 Running 0 19m
## Verify the cluster
➜ cluster-api k get no --kubeconfig=capi-quickstart.kubeconfig
NAME STATUS ROLES AGE VERSION
capi-quickstart-control-plane-sdzrh Ready control-plane 19m v1.26.0
capi-quickstart-md-0-n9z5r-kqzb9 Ready <none> 70s v1.26.0
capi-quickstart-md-0-n9z5r-qztm2 Ready <none> 18m v1.26.0
➜ cluster-api k get po -A --kubeconfig=capi-quickstart.kubeconfig
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c79bd4b6-89swr 1/1 Running 0 9m8s
kube-system calico-node-245qt 1/1 Running 0 9m8s
kube-system calico-node-2sfcv 1/1 Running 0 76s
kube-system calico-node-l9c26 1/1 Running 0 9m8s
kube-system coredns-787d4945fb-7pl52 1/1 Running 0 19m
kube-system coredns-787d4945fb-mkgkw 1/1 Running 0 19m
kube-system etcd-capi-quickstart-control-plane-sdzrh 1/1 Running 0 20m
kube-system kube-apiserver-capi-quickstart-control-plane-sdzrh 1/1 Running 0 20m
kube-system kube-controller-manager-capi-quickstart-control-plane-sdzrh 1/1 Running 0 20m
kube-system kube-proxy-b64d5 1/1 Running 0 76s
kube-system kube-proxy-bmjhz 1/1 Running 0 18m
kube-system kube-proxy-cqwkg 1/1 Running 0 19m
kube-system kube-scheduler-capi-quickstart-control-plane-sdzrh 1/1 Running 0 20m
Remove node
## To remove the node, edit replicas to 1
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: capi-quickstart-md-0
namespace: default
spec:
clusterName: capi-quickstart
replicas: 1 ## ==> this part
➜ cluster-api k apply -f capi-quickstart.yaml
cluster.cluster.x-k8s.io/capi-quickstart unchanged
kubevirtcluster.infrastructure.cluster.x-k8s.io/capi-quickstart unchanged
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane unchanged
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane unchanged
kubevirtmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 unchanged
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 configured
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 configured
➜ cluster-api k get po
NAME READY STATUS RESTARTS AGE
virt-launcher-capi-quickstart-control-plane-sdzrh-q2jcv 3/3 Running 0 24m
virt-launcher-capi-quickstart-md-0-n9z5r-qztm2-sgcsq 3/3 Running 0 21m
➜ cluster-api k get vm
NAME AGE STATUS READY
capi-quickstart-control-plane-sdzrh 24m Running True
capi-quickstart-md-0-n9z5r-qztm2 21m Running True
## Verify cluster
➜ cluster-api k get no --kubeconfig=capi-quickstart.kubeconfig
NAME STATUS ROLES AGE VERSION
capi-quickstart-control-plane-sdzrh Ready control-plane 22m v1.26.0
capi-quickstart-md-0-n9z5r-qztm2 Ready <none> 21m v1.26.0
➜ cluster-api k get po -A --kubeconfig=capi-quickstart.kubeconfig
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c79bd4b6-89swr 1/1 Running 0 11m
kube-system calico-node-245qt 1/1 Running 0 11m
kube-system calico-node-l9c26 1/1 Running 0 11m
kube-system coredns-787d4945fb-7pl52 1/1 Running 0 22m
kube-system coredns-787d4945fb-mkgkw 1/1 Running 0 22m
kube-system etcd-capi-quickstart-control-plane-sdzrh 1/1 Running 0 22m
kube-system kube-apiserver-capi-quickstart-control-plane-sdzrh 1/1 Running 0 22m
kube-system kube-controller-manager-capi-quickstart-control-plane-sdzrh 1/1 Running 0 22m
kube-system kube-proxy-bmjhz 1/1 Running 0 21m
kube-system kube-proxy-cqwkg 1/1 Running 0 22m
kube-system kube-scheduler-capi-quickstart-control-plane-sdzrh 1/1 Running 0 22m
Upgrade Cluster Version - To be
## Note that:
## - KubevirtMachineTemplateSpec is immutable - probably need to set up new kubevirt vms with new version disks..
## - Cannot jump k8s versions -> KubeadmControlPlane.cluster.x-k8s.io "capi-quickstart-control-plane" is invalid: spec.version: Forbidden: cannot update Kubernetes version from v1.26.0 to v1.30.1