[Qemu-devel] [PATCH v4 0/3] sPAPR: EEH fixes

[Qemu-devel] [PATCH v4 0/3] sPAPR: EEH fixes

[Gavin Shan]

The series of patches fix couple of issues as below:

PATCH[1]: Don't enable EEH functionality on the emulated PCI devices,
          which shouldn't be affected during EEH recovery
PATCH[2]: Clear PE's frozen state on reboot. Some PEs in the guest
          might be put into offline because excessive EEH errors and
          those PEs are still in frozen state. Users want them to work
          again after reboot
PATCH[3]: Flush stable MSIx vectors during PE reset.

Changelog:
=========
v3 -> v4:
   * Added PATCH[1]
   * Separate function to reenable EEH functionality in PATCH[2]
   * Move the whole logic into sPAPR platform in PATCH[3]

Gavin Shan (3):
  sPAPR: Don't enable EEH on emulated PCI devices
  sPAPR: Reenable EEH functionality on reboot
  sPAPR: Clear stale MSIx table during EEH reset

 hw/ppc/spapr_pci.c      |  7 ++++++
 hw/ppc/spapr_pci_vfio.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 71 insertions(+), 1 deletion(-)

-- 
2.1.0

[Qemu-devel] [PATCH v4 1/3] sPAPR: Don't enable EEH on emulated PCI devices

[Gavin Shan]

There might have emulated PCI devices, together with VFIO PCI
devices under one PHB. The EEH capability shouldn't enabled
on emulated PCI devices.

The patch returns error when enabling EEH capability on emulated
PCI devices by RTAS call "ibm,set-eeh-option".

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 hw/ppc/spapr_pci.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index a139aea..a8f79d8 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -429,6 +429,7 @@ static void rtas_ibm_set_eeh_option(PowerPCCPU *cpu,
 {
     sPAPRPHBState *sphb;
     sPAPRPHBClass *spc;
+    PCIDevice *pdev;
     uint32_t addr, option;
     uint64_t buid;
     int ret;
@@ -446,6 +447,12 @@ static void rtas_ibm_set_eeh_option(PowerPCCPU *cpu,
         goto param_error_exit;
     }
 
+    pdev = pci_find_device(PCI_HOST_BRIDGE(sphb)->bus,
+                           (addr >> 16) & 0xFF, (addr >> 8) & 0xFF);
+    if (!pdev || !object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
+        goto param_error_exit;
+    }
+
     spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
     if (!spc->eeh_set_option) {
         goto param_error_exit;
-- 
2.1.0

[Qemu-devel] [PATCH v4 2/3] sPAPR: Reenable EEH functionality on reboot

[Gavin Shan]

When rebooting the guest, some PEs might be in frozen state. The
contained PCI devices won't work properly if their frozen states
aren't cleared in time. One case running into this situation would
be maximal EEH error times encountered in the guest.

The patch reenables the EEH functinality on PEs on PHB's reset
callback, which will clear their frozen states if needed.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
v4: Separate function to reenable EEH functionality
---
 hw/ppc/spapr_pci_vfio.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index 99a1be5..8104074 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -71,9 +71,26 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
                                 spapr_tce_get_iommu(tcet));
 }
 
+static void spapr_phb_vfio_eeh_reenable(sPAPRPHBVFIOState *svphb)
+{
+    struct vfio_eeh_pe_op op = {
+        .argsz = sizeof(op),
+        .op    = VFIO_EEH_PE_ENABLE
+    };
+
+    vfio_container_ioctl(&svphb->phb.iommu_as,
+                         svphb->iommugroupid, VFIO_EEH_PE_OP, &op);
+}
+
 static void spapr_phb_vfio_reset(DeviceState *qdev)
 {
-    /* Do nothing */
+    /*
+     * The PE might be in frozen state. To reenable the EEH
+     * functionality on it will clean the frozen state, which
+     * ensures that the contained PCI devices will work properly
+     * after reboot.
+     */
+    spapr_phb_vfio_eeh_reenable(SPAPR_PCI_VFIO_HOST_BRIDGE(qdev));
 }
 
 static int spapr_phb_vfio_eeh_set_option(sPAPRPHBState *sphb,
-- 
2.1.0

[Qemu-devel] [PATCH v4 3/3] sPAPR: Clear stale MSIx table during EEH reset

[Gavin Shan]

The PCI device MSIx table is cleaned out in hardware after EEH PE
reset. However, we still hold the stale MSIx entries in QEMU, which
should be cleared accordingly. Otherwise, we will run into another
(recursive) EEH error and the PCI devices contained in the PE have
to be offlined exceptionally.

The patch introduces function spapr_phb_vfio_eeh_pre_reset(), which
is called by sPAPR when asserting hot or fundamental reset, to clear
stale MSIx table for VFIO PCI devices before EEH PE reset so that
MSIx table could be restored properly after EEH PE reset.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
v4: Move the whole logic to platform code as suggested by Alex
---
 hw/ppc/spapr_pci_vfio.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index 8104074..cca45ed 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -19,6 +19,7 @@
 
 #include "hw/ppc/spapr.h"
 #include "hw/pci-host/spapr.h"
+#include "hw/pci/msix.h"
 #include "linux/vfio.h"
 #include "hw/vfio/vfio.h"
 
@@ -159,6 +160,49 @@ static int spapr_phb_vfio_eeh_get_state(sPAPRPHBState *sphb, int *state)
     return RTAS_OUT_SUCCESS;
 }
 
+static void spapr_phb_vfio_eeh_clear_dev_msix(PCIBus *bus,
+                                              PCIDevice *pdev,
+                                              void *opaque)
+{
+    /* Check if the device is VFIO PCI device */
+    if (!object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
+        return;
+    }
+
+    /*
+     * The MSIx table will be cleaned out by reset. We need
+     * disable it so that it can be reenabled properly. Also,
+     * the cached MSIx table should be cleared as it's not
+     * reflecting the contents in hardware.
+     */
+    if (msix_enabled(pdev)) {
+        uint16_t flags;
+
+        flags = pci_host_config_read_common(pdev,
+                                            pdev->msix_cap + PCI_MSIX_FLAGS,
+                                            pci_config_size(pdev), 2);
+        flags &= ~PCI_MSIX_FLAGS_ENABLE;
+        pci_host_config_write_common(pdev,
+                                     pdev->msix_cap + PCI_MSIX_FLAGS,
+                                     pci_config_size(pdev), flags, 2);
+    }
+
+    msix_reset(pdev);
+}
+
+static void spapr_phb_vfio_eeh_clear_bus_msix(PCIBus *bus, void *opaque)
+{
+       pci_for_each_device(bus, pci_bus_num(bus),
+                           spapr_phb_vfio_eeh_clear_dev_msix, NULL);
+}
+
+static void spapr_phb_vfio_eeh_pre_reset(sPAPRPHBState *sphb)
+{
+       PCIHostState *phb = PCI_HOST_BRIDGE(sphb);
+
+       pci_for_each_bus(phb->bus, spapr_phb_vfio_eeh_clear_bus_msix, NULL);
+}
+
 static int spapr_phb_vfio_eeh_reset(sPAPRPHBState *sphb, int option)
 {
     sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
@@ -170,9 +214,11 @@ static int spapr_phb_vfio_eeh_reset(sPAPRPHBState *sphb, int option)
         op.op = VFIO_EEH_PE_RESET_DEACTIVATE;
         break;
     case RTAS_SLOT_RESET_HOT:
+        spapr_phb_vfio_eeh_pre_reset(sphb);
         op.op = VFIO_EEH_PE_RESET_HOT;
         break;
     case RTAS_SLOT_RESET_FUNDAMENTAL:
+        spapr_phb_vfio_eeh_pre_reset(sphb);
         op.op = VFIO_EEH_PE_RESET_FUNDAMENTAL;
         break;
     default:
-- 
2.1.0

Re: [Qemu-devel] [PATCH v4 0/3] sPAPR: EEH fixes

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1342 bytes --]

On Thu, Jul 02, 2015 at 10:30:09AM +1000, Gavin Shan wrote:
> The series of patches fix couple of issues as below:
> 
> PATCH[1]: Don't enable EEH functionality on the emulated PCI devices,
>           which shouldn't be affected during EEH recovery
> PATCH[2]: Clear PE's frozen state on reboot. Some PEs in the guest
>           might be put into offline because excessive EEH errors and
>           those PEs are still in frozen state. Users want them to work
>           again after reboot
> PATCH[3]: Flush stable MSIx vectors during PE reset.
> 
> Changelog:
> =========
> v3 -> v4:
>    * Added PATCH[1]
>    * Separate function to reenable EEH functionality in PATCH[2]
>    * Move the whole logic into sPAPR platform in PATCH[3]
> 
> Gavin Shan (3):
>   sPAPR: Don't enable EEH on emulated PCI devices
>   sPAPR: Reenable EEH functionality on reboot
>   sPAPR: Clear stale MSIx table during EEH reset
> 
>  hw/ppc/spapr_pci.c      |  7 ++++++
>  hw/ppc/spapr_pci_vfio.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 71 insertions(+), 1 deletion(-)

Thanks, applied to spapr-next.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]