[RFC DOCUMENT 10/12] kubevirt-and-kvm: Add Upgrades page

[Andrea Bolognani <abologna@redhat.com>]

https://gitlab.com/abologna/kubevirt-and-kvm/-/blob/master/Upgrades.md

# Upgrades

The KubeVirt installation and upgrade process are entirely controlled
by an [operator][], which is a common pattern in the Kubernetes
world. The operator is a piece of software running in the cluster and
managing the lifecycle of other components, in this case KubeVirt.

## The operator

What it does:

* Manages the whole KubeVirt installation
* Keeps the cluster actively in sync with the desired state
* Upgrades KubeVirt with zero downtime

## Installation

Install the operator:

```bash
$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)
$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/$(LATEST)/kubevirt-operator.yaml
$ kubectl get pods -n kubevirt
NAME                                      READY   STATUS    RESTARTS   AGE
virt-operator-58cf9d6648-c7qph            1/1     Running   0          69s
virt-operator-58cf9d6648-pvzw2            1/1     Running   0          69s
```

Trigger the installation of KubeVirt:

```bash
$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)
$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr.yaml
$ kubectl get pods -n kubevirt
NAME                                      READY   STATUS    RESTARTS   AGE
virt-api-8bdd88557-fllhr                  1/1     Running   0          59s
virt-controller-55ccb8cdcb-5rtp6          1/1     Running   0          43s
virt-controller-55ccb8cdcb-v8kr9          1/1     Running   0          43s
virt-handler-67pns                        1/1     Running   0          43s
```

The process happens in two steps because the operator relies on the
KubeVirt [custom resource][] for information on the desired
installation, and will not do anything until that resource exists in
the cluster.

## Upgrade

The upgrading process is similar:

* Install the latest operator
* Reference the new version in the KubeVirt CustomResource to trigger
  the actual upgrade

```bash
$ kubectl.sh get kubevirt -n kubevirt kubevirt -o yaml
apiVersion: kubevirt.io/v1alpha3
kind: KubeVirt
metadata:
  name: kubevirt
spec:
  imageTag: v0.30
  certificateRotateStrategy: {}
  configuration: {}
  imagePullPolicy: IfNotPresent
```

Note the `imageTag` attribute: when present, the KubeVirt operator
will take steps to ensure that the version of KubeVirt that's
deployed on the cluster matches its value, which in our case will
trigger an upgrade.

The following chart explain the upgrade flow in more detail and shows
how the various components are affected:

![KubeVirt upgrade flow][Upgrades-Kubevirt]

KubeVirt is released as a complete suite: no individual
`virt-launcher` releases are planned. Everything is tested together,
everything is released together.

## QEMU and libvirt

The versions of QEMU and libvirt used for VMs are also tied to the
version of KubeVirt and are upgraded along with everything else.

* Migrations from old libvirt/QEMU to new libvirt/QEMU pairs are
  possible
* As soon as the new `virt-handler` and the new controller are rolled
  out, the cluster will only start VMIs with the new QEMU/libvirt
  versions

## Version compatibility

The virt stack is updated along with KubeVirt, which mitigates
compatibility concerns. As a rule of thumb, versions of QEMU and
libvirt older than a year or so are not taken into consideration.

Currently, the ability to perform backwards migation (eg. from a
newer version of QEMU to an older one) is not necessary, but that
could very well change as KubeVirt becomes more widely used.

[Upgrades-Kubevirt]: https://gitlab.com/abologna/kubevirt-and-kvm/-/blob/master/Images/Upgrades-Kubevirt.png
[custom resource]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
[operator]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/