[PATCH 43/43] Documentation: KVM: Add the VGICv5 IRS save/restore sequences

[Sascha Bischoff <Sascha.Bischoff@arm.com>]

When saving/restoring the state of the GICv5 IRS, it is important that
it happens in the correct order. Failure to do so will almost
certainly result in failing to restore a guest that is capable of
handling interrupts correctly.

On a save, the ISTs must be saved prior to saving the guest's memory
as the guest's LPI IST is written to guest memory. Conversely, on
restore the guest's memory must be restored prior to restoring the
ISTs.

It is important to restore the IRS MMIO registers by first restoring
the IRS_IDx registers as they define the capabilities of the IRS, and
are used as part of creating and managing ISTs and SPIs.

In order to restore the ISTs themselves, the IRS_IST_CFGR must be
restored prior to the IRS_IST_BASER. This is because KVM extracts
fields from the CFGR to determine the size and structure of the IRS
created by the guest. The IST itself is created as part of the write
to the IRS_IST_BASER. At this stage the remaining MMIO registers can
be restored.

Once the LPI IST has been created (by the aforementioned write to the
IRS_IST_BASER), the IST state can be restored using
KVM_DEV_ARM_VGIC_GRP_IST. The SPI IST gets extracted from a userspace
provided buffer, and is transferred to the host-allocated SPI IST. The
LPI IST is extracted from guest memory, and is written to the
host-allocated LPI IST.

As a general rule, the IRS_*_STATUSR registers can be ignored on
restore. They are not userspace writable.

Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
---
 .../virt/kvm/devices/arm-vgic-v5.rst          | 63 +++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/Documentation/virt/kvm/devices/arm-vgic-v5.rst b/Documentation/virt/kvm/devices/arm-vgic-v5.rst
index 38eef7cc63e3e..1c55f5040757d 100644
--- a/Documentation/virt/kvm/devices/arm-vgic-v5.rst
+++ b/Documentation/virt/kvm/devices/arm-vgic-v5.rst
@@ -201,3 +201,66 @@ Groups:
       -ENOMEM      Restoring IST state failed while tracking pending interrupts
       -ETIMEDOUT   An IRS save/VM operation timed out
       ===========  ============================================================
+
+IRS Save Sequence:
+------------------
+
+The following ordering should be followed when saving the virtual GICv5 and
+IRS:
+
+a) Save the ISTs by issuing KVM_GET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IST.
+   This MUST happen before the guest's memory is serialised as the LPI IST is
+   stored directly to guest memory.
+
+b) Save the IRS MMIO register state in the following order by issuing
+   KVM_GET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
+
+     1. Save IRS_IDR0-2 and IRS_IDR5-7 registers.
+     2. Save IRS_IST_CFGR.
+     3. Save IRS_IST_BASER.
+     4. Save the remaining global IRS MMIO registers.
+     5. For each PE:
+        - write IRS_PE_SELR
+        - save IRS_PE_CR0
+     6. For each SPI:
+        - write IRS_SPI_SELR
+        - save IRS_SPI_CFGR
+
+IRS Restore Sequence:
+---------------------
+
+The following ordering must be followed when restoring the virtual GICv5 and
+IRS:
+
+a) restore all guest memory and create vcpus
+b) provide the IRS base address by issuing KVM_SET_DEVICE_ATTR on
+   KVM_DEV_ARM_VGIC_GRP_ADDR
+c) initialise the GIC - this sets up the default state and creates the SPI
+   IST - by issuing KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_CTRL with
+   KVM_DEV_ARM_VGIC_CTRL_INIT
+d) restore the IRS MMIO register state in the following order by issuing
+   KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
+
+     1. Restore IRS_IDR0-2 and IRS_IDR5-7 registers.
+     2. Restore IRS_IST_CFGR.
+     3. Restore IRS_IST_BASER - this triggers KVM to create the LPI IST.
+
+e) restore the ISTs by issuing KVM_SET_DEVICE_ATTR on
+   KVM_DEV_ARM_VGIC_GRP_IST.
+f) restore the remaining IRS MMIO register state in the following order by
+   issuing KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
+
+     1. Restore the remaining global IRS MMIO registers.
+     2. For each PE:
+        - write IRS_PE_SELR
+        - restore IRS_PE_CR0
+     3. For each SPI:
+        - write IRS_SPI_SELR
+        - restore IRS_SPI_CFGR
+
+The various *_STATUSR registers are observational state in the current KVM
+implementation. Userspace may save them for validation or debugging purposes,
+but they are not required restore input and do not need to be replayed during
+restore.
+
+Then vcpus can be started.
-- 
2.34.1