[PATCH] spapr: Clarify error and documentation for broken KVM XICS

[PATCH] spapr: Clarify error and documentation for broken KVM XICS

[Greg Kurz]

When starting an L2 KVM guest with `ic-mode=dual,kernel-irqchip=on`,
QEMU fails with:

KVM is too old to support ic-mode=dual,kernel-irqchip=on

This error message was introduced to detect older KVM versions that
didn't allow destruction and re-creation of the XICS KVM device that
we do at reboot. But it is actually the same issue that we get with
nested guests : when running under pseries, KVM currently provides
a genuine XICS device (not the XICS-on-XIVE device that we get
under powernv) which doesn't support destruction/re-creation.

This will eventually be fixed in KVM but in the meantime, update
the error message and documentation to mention the nested case.
While here, mention that in "No XIVE support in KVM" section that
this can also happen with "guest OSes supporting XIVE" since
we check this at init time before starting the guest.

Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
Buglink: https://bugs.launchpad.net/qemu/+bug/1890290
Signed-off-by: Greg Kurz <groug@kaod.org>
---
 docs/specs/ppc-spapr-xive.rst |    5 ++++-
 hw/ppc/spapr_irq.c            |   12 +++++++++---
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/docs/specs/ppc-spapr-xive.rst b/docs/specs/ppc-spapr-xive.rst
index 7199db730b82..7144347560f1 100644
--- a/docs/specs/ppc-spapr-xive.rst
+++ b/docs/specs/ppc-spapr-xive.rst
@@ -126,6 +126,9 @@ xics            XICS KVM       XICS emul.     XICS KVM
 
 (1) QEMU warns with ``warning: kernel_irqchip requested but unavailable:
     IRQ_XIVE capability must be present for KVM``
+    In some cases (old host kernels or KVM nested guests), one may hit a
+    QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
+    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
 (2) QEMU fails with ``kernel_irqchip requested but unavailable:
     IRQ_XIVE capability must be present for KVM``
 
@@ -148,7 +151,7 @@ xics            XICS KVM       XICS emul.     XICS KVM
     mode (XICS), either don't set the ic-mode machine property or try
     ic-mode=xics or ic-mode=dual``
 (4) QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
-    with ``KVM is too old to support ic-mode=dual,kernel-irqchip=on``
+    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
 
 
 XIVE Device tree properties
diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
index 2f8f7d62f875..72bb938375ef 100644
--- a/hw/ppc/spapr_irq.c
+++ b/hw/ppc/spapr_irq.c
@@ -139,6 +139,7 @@ SpaprIrq spapr_irq_dual = {
 
 static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
 {
+    ERRP_GUARD();
     MachineState *machine = MACHINE(spapr);
 
     /*
@@ -179,14 +180,19 @@ static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
 
     /*
      * On a POWER9 host, some older KVM XICS devices cannot be destroyed and
-     * re-created. Detect that early to avoid QEMU to exit later when the
-     * guest reboots.
+     * re-created. Same happens with KVM nested guests. Detect that early to
+     * avoid QEMU to exit later when the guest reboots.
      */
     if (kvm_enabled() &&
         spapr->irq == &spapr_irq_dual &&
         kvm_kernel_irqchip_required() &&
         xics_kvm_has_broken_disconnect(spapr)) {
-        error_setg(errp, "KVM is too old to support ic-mode=dual,kernel-irqchip=on");
+        error_setg(errp,
+            "KVM is incompatible with ic-mode=dual,kernel-irqchip=on");
+        error_append_hint(errp,
+            "This can happen with an old KVM or in a KVM nested guest.\n");
+        error_append_hint(errp,
+            "Try without kernel-irqchip or with kernel-irqchip=off.\n");
         return -1;
     }
 

Re: [PATCH] spapr: Clarify error and documentation for broken KVM XICS

[Cédric Le Goater]

On 8/5/20 5:47 PM, Greg Kurz wrote:
> When starting an L2 KVM guest with `ic-mode=dual,kernel-irqchip=on`,
> QEMU fails with:
> 
> KVM is too old to support ic-mode=dual,kernel-irqchip=on
> 
> This error message was introduced to detect older KVM versions that
> didn't allow destruction and re-creation of the XICS KVM device that
> we do at reboot. But it is actually the same issue that we get with
> nested guests : when running under pseries, KVM currently provides
> a genuine XICS device (not the XICS-on-XIVE device that we get
> under powernv) which doesn't support destruction/re-creation.
> 
> This will eventually be fixed in KVM but in the meantime, update
> the error message and documentation to mention the nested case.
> While here, mention that in "No XIVE support in KVM" section that
> this can also happen with "guest OSes supporting XIVE" since
> we check this at init time before starting the guest.
> 
> Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
> Buglink: https://bugs.launchpad.net/qemu/+bug/1890290
> Signed-off-by: Greg Kurz <groug@kaod.org>

Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C. 


> ---
>  docs/specs/ppc-spapr-xive.rst |    5 ++++-
>  hw/ppc/spapr_irq.c            |   12 +++++++++---
>  2 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/docs/specs/ppc-spapr-xive.rst b/docs/specs/ppc-spapr-xive.rst
> index 7199db730b82..7144347560f1 100644
> --- a/docs/specs/ppc-spapr-xive.rst
> +++ b/docs/specs/ppc-spapr-xive.rst
> @@ -126,6 +126,9 @@ xics            XICS KVM       XICS emul.     XICS KVM
>  
>  (1) QEMU warns with ``warning: kernel_irqchip requested but unavailable:
>      IRQ_XIVE capability must be present for KVM``
> +    In some cases (old host kernels or KVM nested guests), one may hit a
> +    QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
> +    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
>  (2) QEMU fails with ``kernel_irqchip requested but unavailable:
>      IRQ_XIVE capability must be present for KVM``
>  
> @@ -148,7 +151,7 @@ xics            XICS KVM       XICS emul.     XICS KVM
>      mode (XICS), either don't set the ic-mode machine property or try
>      ic-mode=xics or ic-mode=dual``
>  (4) QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
> -    with ``KVM is too old to support ic-mode=dual,kernel-irqchip=on``
> +    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
>  
>  
>  XIVE Device tree properties
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index 2f8f7d62f875..72bb938375ef 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -139,6 +139,7 @@ SpaprIrq spapr_irq_dual = {
>  
>  static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
>  {
> +    ERRP_GUARD();
>      MachineState *machine = MACHINE(spapr);
>  
>      /*
> @@ -179,14 +180,19 @@ static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
>  
>      /*
>       * On a POWER9 host, some older KVM XICS devices cannot be destroyed and
> -     * re-created. Detect that early to avoid QEMU to exit later when the
> -     * guest reboots.
> +     * re-created. Same happens with KVM nested guests. Detect that early to
> +     * avoid QEMU to exit later when the guest reboots.
>       */
>      if (kvm_enabled() &&
>          spapr->irq == &spapr_irq_dual &&
>          kvm_kernel_irqchip_required() &&
>          xics_kvm_has_broken_disconnect(spapr)) {
> -        error_setg(errp, "KVM is too old to support ic-mode=dual,kernel-irqchip=on");
> +        error_setg(errp,
> +            "KVM is incompatible with ic-mode=dual,kernel-irqchip=on");
> +        error_append_hint(errp,
> +            "This can happen with an old KVM or in a KVM nested guest.\n");
> +        error_append_hint(errp,
> +            "Try without kernel-irqchip or with kernel-irqchip=off.\n");
>          return -1;
>      }
>  
> 
> 

Re: [PATCH] spapr: Clarify error and documentation for broken KVM XICS

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4218 bytes --]

On Wed, Aug 05, 2020 at 05:47:16PM +0200, Greg Kurz wrote:
> When starting an L2 KVM guest with `ic-mode=dual,kernel-irqchip=on`,
> QEMU fails with:
> 
> KVM is too old to support ic-mode=dual,kernel-irqchip=on
> 
> This error message was introduced to detect older KVM versions that
> didn't allow destruction and re-creation of the XICS KVM device that
> we do at reboot. But it is actually the same issue that we get with
> nested guests : when running under pseries, KVM currently provides
> a genuine XICS device (not the XICS-on-XIVE device that we get
> under powernv) which doesn't support destruction/re-creation.
> 
> This will eventually be fixed in KVM but in the meantime, update
> the error message and documentation to mention the nested case.
> While here, mention that in "No XIVE support in KVM" section that
> this can also happen with "guest OSes supporting XIVE" since
> we check this at init time before starting the guest.
> 
> Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
> Buglink: https://bugs.launchpad.net/qemu/+bug/1890290
> Signed-off-by: Greg Kurz <groug@kaod.org>

Applied to ppc-for-5.2.

> ---
>  docs/specs/ppc-spapr-xive.rst |    5 ++++-
>  hw/ppc/spapr_irq.c            |   12 +++++++++---
>  2 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/docs/specs/ppc-spapr-xive.rst b/docs/specs/ppc-spapr-xive.rst
> index 7199db730b82..7144347560f1 100644
> --- a/docs/specs/ppc-spapr-xive.rst
> +++ b/docs/specs/ppc-spapr-xive.rst
> @@ -126,6 +126,9 @@ xics            XICS KVM       XICS emul.     XICS KVM
>  
>  (1) QEMU warns with ``warning: kernel_irqchip requested but unavailable:
>      IRQ_XIVE capability must be present for KVM``
> +    In some cases (old host kernels or KVM nested guests), one may hit a
> +    QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
> +    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
>  (2) QEMU fails with ``kernel_irqchip requested but unavailable:
>      IRQ_XIVE capability must be present for KVM``
>  
> @@ -148,7 +151,7 @@ xics            XICS KVM       XICS emul.     XICS KVM
>      mode (XICS), either don't set the ic-mode machine property or try
>      ic-mode=xics or ic-mode=dual``
>  (4) QEMU/KVM incompatibility due to device destruction in reset. QEMU fails
> -    with ``KVM is too old to support ic-mode=dual,kernel-irqchip=on``
> +    with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on``
>  
>  
>  XIVE Device tree properties
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index 2f8f7d62f875..72bb938375ef 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -139,6 +139,7 @@ SpaprIrq spapr_irq_dual = {
>  
>  static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
>  {
> +    ERRP_GUARD();
>      MachineState *machine = MACHINE(spapr);
>  
>      /*
> @@ -179,14 +180,19 @@ static int spapr_irq_check(SpaprMachineState *spapr, Error **errp)
>  
>      /*
>       * On a POWER9 host, some older KVM XICS devices cannot be destroyed and
> -     * re-created. Detect that early to avoid QEMU to exit later when the
> -     * guest reboots.
> +     * re-created. Same happens with KVM nested guests. Detect that early to
> +     * avoid QEMU to exit later when the guest reboots.
>       */
>      if (kvm_enabled() &&
>          spapr->irq == &spapr_irq_dual &&
>          kvm_kernel_irqchip_required() &&
>          xics_kvm_has_broken_disconnect(spapr)) {
> -        error_setg(errp, "KVM is too old to support ic-mode=dual,kernel-irqchip=on");
> +        error_setg(errp,
> +            "KVM is incompatible with ic-mode=dual,kernel-irqchip=on");
> +        error_append_hint(errp,
> +            "This can happen with an old KVM or in a KVM nested guest.\n");
> +        error_append_hint(errp,
> +            "Try without kernel-irqchip or with kernel-irqchip=off.\n");
>          return -1;
>      }
>  
> 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]