[Qemu-devel] [QEMU-PPC] [PATCH V3 0/3] ppc/spapr: Add support for nested kvm-hv

[Qemu-devel] [QEMU-PPC] [PATCH V3 0/3] ppc/spapr: Add support for nested kvm-hv

[Suraj Jitindar Singh]

This patch series adds the qemu support for running nested kvm-hv on a
POWER9 platform with appropriate hypervisor support and migration of
these guests.
That is, the ability to run kvm-hv guests as guests of an operating system
which is itself a kvm-hv guest.

The host (L0 hypervisor) and level 1 guest (L1 guest hypervisor) require
the following patch series:
KVM: PPC: Book3S HV: Nested HV virtualization

And this patch series in the qemu instance running on the L0 hypervisor.

Patch series based on: ppc-for-3.1

The cap number is now in Paolo's tree so hopefully we can rely on it not
changing.

V2 -> V3:
- The enable cap ioctl no longer takes an enable field to indicate
  disable/enable, but just enables the cap. So update to match this
  kernel change. (The cap starts out disable by default)

Suraj Jitindar Singh (3):
  target/ppc: Update linux-headers for v4.19-rc7
  target/ppc: Add one reg id for ptcr
  ppc/spapr_caps: Add SPAPR_CAP_NESTED_KVM_HV

 hw/ppc/spapr.c                  |  2 ++
 hw/ppc/spapr_caps.c             | 32 ++++++++++++++++++++++++++++++++
 include/hw/ppc/spapr.h          |  5 ++++-
 linux-headers/asm-powerpc/kvm.h |  1 +
 linux-headers/linux/kvm.h       |  1 +
 target/ppc/kvm.c                | 12 ++++++++++++
 target/ppc/kvm_ppc.h            | 12 ++++++++++++
 target/ppc/translate_init.inc.c | 10 +++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

-- 
2.13.6

[Qemu-devel] [QEMU-PPC] [PATCH V3 1/3] target/ppc: Update linux-headers for v4.19-rc7

[Suraj Jitindar Singh]

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 linux-headers/asm-powerpc/kvm.h | 1 +
 linux-headers/linux/kvm.h       | 1 +
 2 files changed, 2 insertions(+)

diff --git a/linux-headers/asm-powerpc/kvm.h b/linux-headers/asm-powerpc/kvm.h
index 1b32b56a03..8c876c166e 100644
--- a/linux-headers/asm-powerpc/kvm.h
+++ b/linux-headers/asm-powerpc/kvm.h
@@ -634,6 +634,7 @@ struct kvm_ppc_cpu_char {
 
 #define KVM_REG_PPC_DEC_EXPIRY	(KVM_REG_PPC | KVM_REG_SIZE_U64 | 0xbe)
 #define KVM_REG_PPC_ONLINE	(KVM_REG_PPC | KVM_REG_SIZE_U32 | 0xbf)
+#define KVM_REG_PPC_PTCR	(KVM_REG_PPC | KVM_REG_SIZE_U64 | 0xc0)
 
 /* Transactional Memory checkpointed state:
  * This is all GPRs, all VSX regs and a subset of SPRs
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 66790724f1..d49767ad25 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -951,6 +951,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_HYPERV_TLBFLUSH 155
 #define KVM_CAP_S390_HPAGE_1M 156
 #define KVM_CAP_NESTED_STATE 157
+#define KVM_CAP_PPC_NESTED_HV 160
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
-- 
2.13.6

[Qemu-devel] [QEMU-PPC] [PATCH V3 2/3] target/ppc: Add one reg id for ptcr

[Suraj Jitindar Singh]

The ptcr (partition table control register) is used to store the address
and size of the partition table. For nested kvm-hv we have a level 1
guest register the location of it's partition table with the hypervisor.
Thus to support migration we need to be able to read this out of kvm
and restore it post migration.

Add the one reg id for the ptcr.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 target/ppc/translate_init.inc.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/target/ppc/translate_init.inc.c b/target/ppc/translate_init.inc.c
index 263e63cb03..487196800b 100644
--- a/target/ppc/translate_init.inc.c
+++ b/target/ppc/translate_init.inc.c
@@ -8197,11 +8197,11 @@ static void gen_spr_power9_mmu(CPUPPCState *env)
 {
 #if !defined(CONFIG_USER_ONLY)
     /* Partition Table Control */
-    spr_register_hv(env, SPR_PTCR, "PTCR",
-                    SPR_NOACCESS, SPR_NOACCESS,
-                    SPR_NOACCESS, SPR_NOACCESS,
-                    &spr_read_generic, &spr_write_ptcr,
-                    0x00000000);
+    spr_register_kvm_hv(env, SPR_PTCR, "PTCR",
+                        SPR_NOACCESS, SPR_NOACCESS,
+                        SPR_NOACCESS, SPR_NOACCESS,
+                        &spr_read_generic, &spr_write_ptcr,
+                        KVM_REG_PPC_PTCR, 0x00000000);
 #endif
 }
 
-- 
2.13.6

[Qemu-devel] [QEMU-PPC] [PATCH V3 3/3] ppc/spapr_caps: Add SPAPR_CAP_NESTED_KVM_HV

[Suraj Jitindar Singh]

Add the spapr cap SPAPR_CAP_NESTED_KVM_HV to be used to control the
availability of nested kvm-hv to the level 1 (L1) guest.

Assuming a hypervisor with support enabled an L1 guest can be allowed to
use the kvm-hv module (and thus run it's own kvm-hv guests) by setting:
-machine pseries,cap-nested-hv=true
or disabled with:
-machine pseries,cap-nested-hv=false

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 hw/ppc/spapr.c         |  2 ++
 hw/ppc/spapr_caps.c    | 32 ++++++++++++++++++++++++++++++++
 include/hw/ppc/spapr.h |  5 ++++-
 target/ppc/kvm.c       | 12 ++++++++++++
 target/ppc/kvm_ppc.h   | 12 ++++++++++++
 5 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 98868d893a..8ce97900e9 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1915,6 +1915,7 @@ static const VMStateDescription vmstate_spapr = {
         &vmstate_spapr_cap_sbbc,
         &vmstate_spapr_cap_ibs,
         &vmstate_spapr_irq_map,
+        &vmstate_spapr_cap_nested_kvm_hv,
         NULL
     }
 };
@@ -3886,6 +3887,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     smc->default_caps.caps[SPAPR_CAP_SBBC] = SPAPR_CAP_BROKEN;
     smc->default_caps.caps[SPAPR_CAP_IBS] = SPAPR_CAP_BROKEN;
     smc->default_caps.caps[SPAPR_CAP_HPT_MAXPAGESIZE] = 16; /* 64kiB */
+    smc->default_caps.caps[SPAPR_CAP_NESTED_KVM_HV] = SPAPR_CAP_OFF;
     spapr_caps_add_properties(smc, &error_abort);
     smc->irq = &spapr_irq_xics;
 }
diff --git a/hw/ppc/spapr_caps.c b/hw/ppc/spapr_caps.c
index aa605cea91..8e1d791c63 100644
--- a/hw/ppc/spapr_caps.c
+++ b/hw/ppc/spapr_caps.c
@@ -368,6 +368,28 @@ static void cap_hpt_maxpagesize_cpu_apply(sPAPRMachineState *spapr,
     ppc_hash64_filter_pagesizes(cpu, spapr_pagesize_cb, &maxshift);
 }
 
+static void cap_nested_kvm_hv_apply(sPAPRMachineState *spapr,
+                                    uint8_t val, Error **errp)
+{
+    if (!val) {
+        /* capability disabled by default */
+        return;
+    }
+
+    if (tcg_enabled()) {
+        error_setg(errp,
+                   "No Nested KVM-HV support in tcg, try cap-nested-hv=off");
+    } else if (kvm_enabled()) {
+        if (!kvmppc_has_cap_nested_kvm_hv()) {
+            error_setg(errp,
+"KVM implementation does not support Nested KVM-HV, try cap-nested-hv=off");
+        } else if (kvmppc_enable_nested_kvm_hv() < 0) {
+                error_setg(errp,
+"Error enabling cap-nested-hv with KVM, try cap-nested-hv=off");
+        }
+    }
+}
+
 sPAPRCapabilityInfo capability_table[SPAPR_CAP_NUM] = {
     [SPAPR_CAP_HTM] = {
         .name = "htm",
@@ -437,6 +459,15 @@ sPAPRCapabilityInfo capability_table[SPAPR_CAP_NUM] = {
         .apply = cap_hpt_maxpagesize_apply,
         .cpu_apply = cap_hpt_maxpagesize_cpu_apply,
     },
+    [SPAPR_CAP_NESTED_KVM_HV] = {
+        .name = "nested-hv",
+        .description = "Allow Nested KVM-HV",
+        .index = SPAPR_CAP_NESTED_KVM_HV,
+        .get = spapr_cap_get_bool,
+        .set = spapr_cap_set_bool,
+        .type = "bool",
+        .apply = cap_nested_kvm_hv_apply,
+    },
 };
 
 static sPAPRCapabilities default_caps_with_cpu(sPAPRMachineState *spapr,
@@ -564,6 +595,7 @@ SPAPR_CAP_MIG_STATE(dfp, SPAPR_CAP_DFP);
 SPAPR_CAP_MIG_STATE(cfpc, SPAPR_CAP_CFPC);
 SPAPR_CAP_MIG_STATE(sbbc, SPAPR_CAP_SBBC);
 SPAPR_CAP_MIG_STATE(ibs, SPAPR_CAP_IBS);
+SPAPR_CAP_MIG_STATE(nested_kvm_hv, SPAPR_CAP_NESTED_KVM_HV);
 
 void spapr_caps_init(sPAPRMachineState *spapr)
 {
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index ad4d7cfd97..bced85dd92 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -70,8 +70,10 @@ typedef enum {
 #define SPAPR_CAP_IBS                   0x05
 /* HPT Maximum Page Size (encoded as a shift) */
 #define SPAPR_CAP_HPT_MAXPAGESIZE       0x06
+/* Nested KVM-HV */
+#define SPAPR_CAP_NESTED_KVM_HV         0x07
 /* Num Caps */
-#define SPAPR_CAP_NUM                   (SPAPR_CAP_HPT_MAXPAGESIZE + 1)
+#define SPAPR_CAP_NUM                   (SPAPR_CAP_NESTED_KVM_HV + 1)
 
 /*
  * Capability Values
@@ -793,6 +795,7 @@ extern const VMStateDescription vmstate_spapr_cap_dfp;
 extern const VMStateDescription vmstate_spapr_cap_cfpc;
 extern const VMStateDescription vmstate_spapr_cap_sbbc;
 extern const VMStateDescription vmstate_spapr_cap_ibs;
+extern const VMStateDescription vmstate_spapr_cap_nested_kvm_hv;
 
 static inline uint8_t spapr_get_cap(sPAPRMachineState *spapr, int cap)
 {
diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c
index 30aeafa7de..8849180548 100644
--- a/target/ppc/kvm.c
+++ b/target/ppc/kvm.c
@@ -91,6 +91,7 @@ static int cap_ppc_pvr_compat;
 static int cap_ppc_safe_cache;
 static int cap_ppc_safe_bounds_check;
 static int cap_ppc_safe_indirect_branch;
+static int cap_ppc_nested_kvm_hv;
 
 static uint32_t debug_inst_opcode;
 
@@ -150,6 +151,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     cap_mmu_hash_v3 = kvm_vm_check_extension(s, KVM_CAP_PPC_MMU_HASH_V3);
     cap_resize_hpt = kvm_vm_check_extension(s, KVM_CAP_SPAPR_RESIZE_HPT);
     kvmppc_get_cpu_characteristics(s);
+    cap_ppc_nested_kvm_hv = kvm_vm_check_extension(s, KVM_CAP_PPC_NESTED_HV);
     /*
      * Note: setting it to false because there is not such capability
      * in KVM at this moment.
@@ -2422,6 +2424,16 @@ int kvmppc_get_cap_safe_indirect_branch(void)
     return cap_ppc_safe_indirect_branch;
 }
 
+bool kvmppc_has_cap_nested_kvm_hv(void)
+{
+    return !!cap_ppc_nested_kvm_hv;
+}
+
+int kvmppc_enable_nested_kvm_hv(void)
+{
+    return kvm_vm_enable_cap(kvm_state, KVM_CAP_PPC_NESTED_HV, 0);
+}
+
 bool kvmppc_has_cap_spapr_vfio(void)
 {
     return cap_spapr_vfio;
diff --git a/target/ppc/kvm_ppc.h b/target/ppc/kvm_ppc.h
index f696c6e498..797271114e 100644
--- a/target/ppc/kvm_ppc.h
+++ b/target/ppc/kvm_ppc.h
@@ -62,6 +62,8 @@ bool kvmppc_has_cap_mmu_hash_v3(void);
 int kvmppc_get_cap_safe_cache(void);
 int kvmppc_get_cap_safe_bounds_check(void);
 int kvmppc_get_cap_safe_indirect_branch(void);
+bool kvmppc_has_cap_nested_kvm_hv(void);
+int kvmppc_enable_nested_kvm_hv(void);
 int kvmppc_enable_hwrng(void);
 int kvmppc_put_books_sregs(PowerPCCPU *cpu);
 PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void);
@@ -320,6 +322,16 @@ static inline int kvmppc_get_cap_safe_indirect_branch(void)
     return 0;
 }
 
+static inline bool kvmppc_has_cap_nested_kvm_hv(void)
+{
+    return false;
+}
+
+static inline int kvmppc_enable_nested_kvm_hv(void)
+{
+    return -1;
+}
+
 static inline int kvmppc_enable_hwrng(void)
 {
     return -1;
-- 
2.13.6

Re: [Qemu-devel] [QEMU-PPC] [PATCH V3 0/3] ppc/spapr: Add support for nested kvm-hv

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Thu, Oct 11, 2018 at 05:16:06PM +1100, Suraj Jitindar Singh wrote:
> This patch series adds the qemu support for running nested kvm-hv on a
> POWER9 platform with appropriate hypervisor support and migration of
> these guests.
> That is, the ability to run kvm-hv guests as guests of an operating system
> which is itself a kvm-hv guest.
> 
> The host (L0 hypervisor) and level 1 guest (L1 guest hypervisor) require
> the following patch series:
> KVM: PPC: Book3S HV: Nested HV virtualization
> 
> And this patch series in the qemu instance running on the L0 hypervisor.
> 
> Patch series based on: ppc-for-3.1
> 
> The cap number is now in Paolo's tree so hopefully we can rely on it not
> changing.

Now that the kernel changes are in upstream master, can you resend a
final version of this?

> 
> V2 -> V3:
> - The enable cap ioctl no longer takes an enable field to indicate
>   disable/enable, but just enables the cap. So update to match this
>   kernel change. (The cap starts out disable by default)
> 
> Suraj Jitindar Singh (3):
>   target/ppc: Update linux-headers for v4.19-rc7
>   target/ppc: Add one reg id for ptcr
>   ppc/spapr_caps: Add SPAPR_CAP_NESTED_KVM_HV
> 
>  hw/ppc/spapr.c                  |  2 ++
>  hw/ppc/spapr_caps.c             | 32 ++++++++++++++++++++++++++++++++
>  include/hw/ppc/spapr.h          |  5 ++++-
>  linux-headers/asm-powerpc/kvm.h |  1 +
>  linux-headers/linux/kvm.h       |  1 +
>  target/ppc/kvm.c                | 12 ++++++++++++
>  target/ppc/kvm_ppc.h            | 12 ++++++++++++
>  target/ppc/translate_init.inc.c | 10 +++++-----
>  8 files changed, 69 insertions(+), 6 deletions(-)
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]