[Qemu-devel] [PATCH v8 0/4] EEH Support for VFIO PCI Device

[Qemu-devel] [PATCH v8 0/4] EEH Support for VFIO PCI Device

[Gavin Shan]

The series of patches adds support EEH for VFIO PCI devices on sPAPR platform.
It requires corresponding host kernel support. Also, it is based on top of
Alexey's VFIO-for-sPAPR git repository.

QEMU:   git://github.com/aik/qemu.git (branch: vfio)
Kernel: git://github.com/aik/linux.git (branch: vfio)
Kernel: http://linuxppc.10917.n7.nabble.com/PATCH-v8-0-3-EEH-Support-for-VFIO-PCI-Device-td82940.html

The implementations notes are below. Please comment.

* RTAS calls are received in spapr_pci.c, sanity check is done there. RTAS
  handlers handle what they can. If there is something it cannot handle and
  sPAPRPHBClass::eeh_handler callback is defined, it is called.
* sPAPRPHBClass::eeh_handler is only implemented for VFIO now. It does ioctl()
  to the IOMMU container fd to complete the call. Error codes from that ioctl()
  are transferred back to the guest.

Gavin Shan (4):
  sPAPR: Implement EEH RTAS calls
  headers: Update kernel header
  VFIO: Introduce helper vfio_pci_container_ioctl()
  sPAPR: Implement sPAPRPHBClass::eeh_handler

 hw/misc/vfio.c              |  31 ++++++
 hw/ppc/spapr_pci.c          | 248 ++++++++++++++++++++++++++++++++++++++++++++
 hw/ppc/spapr_pci_vfio.c     |  54 ++++++++++
 include/hw/misc/vfio.h      |   2 +
 include/hw/pci-host/spapr.h |   7 ++
 include/hw/ppc/spapr.h      |  33 ++++++
 linux-headers/linux/vfio.h  |  35 +++++++
 7 files changed, 410 insertions(+)

-- 
1.8.3.2

[Qemu-devel] [PATCH v8 1/4] sPAPR: Implement EEH RTAS calls

[Gavin Shan]

The emulation for EEH RTAS requests from guest isn't covered
by QEMU yet and the patch implements them.

The patch defines constants used by EEH RTAS calls and adds
callback sPAPRPHBClass::eeh_handler, which is going to be used
this way:

1. RTAS calls are received in spapr_pci.c, sanity check is done
   there.
2. RTAS handlers handle what they can. If there is something it
   cannot handle and sPAPRPHBClass::eeh_handler callback is defined,
   it is called.
3. sPAPRPHBClass::eeh_handler is only implemented for VFIO now. It
   does ioctl() to the IOMMU container fd to complete the call. Error
   codes from that ioctl() are transferred back to the guest.

This adds 6 RTAS handlers, all defined in SPAPR specification:

1) ibm,set-eeh-option: disables/enables EEH on a device, removes PE from
stopped state;
2) ibm,get-config-addr-info2 - returns fabric configuration address (upper
PCI bridge or PHB if there is no bridge);
3) ibm,read-slot-reset-state2 - retrieve PE state;
4) ibm,set-slot-reset - issue PE reset;
5) ibm,configure-pe - configure PCI bridges in the affected PE;
6) ibm,slot-error-detail - retrieve EEH error log;

All calls use fabric configuration address (a.k.a. PE address) as a target
address except ibm,get-config-addr-info2 and one case (enable EEH on the
specified PCI function) for ibm,set-eeh-option.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 hw/ppc/spapr_pci.c          | 248 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci-host/spapr.h |   7 ++
 include/hw/ppc/spapr.h      |  33 ++++++
 3 files changed, 288 insertions(+)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index a9f307a..423e4ff 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -422,6 +422,241 @@ static void rtas_ibm_query_interrupt_source_number(PowerPCCPU *cpu,
     rtas_st(rets, 2, 1);/* 0 == level; 1 == edge */
 }
 
+static int rtas_finish_eeh_request(sPAPRPHBState *sphb,
+                                   uint32_t req, uint32_t opt,
+                                   target_ulong rets)
+{
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    int ret;
+
+    ret = info->eeh_handler(sphb, req, opt);
+    if (ret >= 0) {
+        rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    } else {
+        rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+    }
+
+    return ret;
+}
+
+static void rtas_ibm_set_eeh_option(PowerPCCPU *cpu,
+                                    sPAPREnvironment *spapr,
+                                    uint32_t token, uint32_t nargs,
+                                    target_ulong args, uint32_t nret,
+                                    target_ulong rets)
+{
+    uint32_t addr, option;
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 4) || (nret != 1)) {
+        goto param_error_exit;
+    }
+
+    addr = rtas_ld(args, 0);
+    option = rtas_ld(args, 3);
+    switch (option) {
+    case RTAS_EEH_ENABLE:
+        if (!find_dev(spapr, buid, addr)) {
+            goto param_error_exit;
+        }
+        break;
+    case RTAS_EEH_DISABLE:
+    case RTAS_EEH_THAW_IO:
+    case RTAS_EEH_THAW_DMA:
+        break;
+    default:
+        goto param_error_exit;
+    }
+
+    rtas_finish_eeh_request(sphb, RTAS_EEH_REQ_SET_OPTION, option, rets);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_get_config_addr_info2(PowerPCCPU *cpu,
+                                           sPAPREnvironment *spapr,
+                                           uint32_t token, uint32_t nargs,
+                                           target_ulong args, uint32_t nret,
+                                           target_ulong rets)
+{
+    uint32_t addr, option;
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    PCIDevice *pdev;
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 4) || (nret != 2)) {
+        goto param_error_exit;
+    }
+
+    addr = rtas_ld(args, 0);
+    option = rtas_ld(args, 3);
+    if (option != RTAS_GET_PE_ADDR && option != RTAS_GET_PE_MODE) {
+        goto param_error_exit;
+    }
+
+    pdev = find_dev(spapr, buid, addr);
+    if (!pdev) {
+        goto param_error_exit;
+    }
+
+    /*
+     * For now, we always have bus level PE whose address
+     * has format "00BBSS00". The guest OS might regard
+     * PE address 0 as invalid. We avoid that simply by
+     * extending it with one.
+     */
+    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    if (option == RTAS_GET_PE_ADDR) {
+        rtas_st(rets, 1, (pci_bus_num(pdev->bus) << 16) + 1);
+    } else {
+        rtas_st(rets, 1, RTAS_PE_MODE_SHARED);
+    }
+
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_read_slot_reset_state2(PowerPCCPU *cpu,
+                                            sPAPREnvironment *spapr,
+                                            uint32_t token, uint32_t nargs,
+                                            target_ulong args, uint32_t nret,
+                                            target_ulong rets)
+{
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    int32_t ret;
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 3) || (nret != 4 && nret != 5)) {
+        goto param_error_exit;
+    }
+
+    ret = rtas_finish_eeh_request(sphb, RTAS_EEH_REQ_GET_STATE, 0, rets);
+    if (ret >= 0) {
+        rtas_st(rets, 1, ret);
+        rtas_st(rets, 2, RTAS_EEH_SUPPORT);
+        rtas_st(rets, 3, RTAS_EEH_PE_UNAVAIL_INFO);
+        if (nret >= 5) {
+            rtas_st(rets, 4, RTAS_EEH_PE_RECOVER_INFO);
+        }
+    }
+
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_set_slot_reset(PowerPCCPU *cpu,
+                                    sPAPREnvironment *spapr,
+                                    uint32_t token, uint32_t nargs,
+                                    target_ulong args, uint32_t nret,
+                                    target_ulong rets)
+{
+    uint32_t option;
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 4) || (nret != 1)) {
+        goto param_error_exit;
+    }
+
+    option = rtas_ld(args, 3);
+    if (option != RTAS_SLOT_RESET_DEACTIVATE &&
+        option != RTAS_SLOT_RESET_HOT &&
+        option != RTAS_SLOT_RESET_FUNDAMENTAL) {
+        goto param_error_exit;
+    }
+
+    rtas_finish_eeh_request(sphb, RTAS_EEH_REQ_RESET, option, rets);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_configure_pe(PowerPCCPU *cpu,
+                                  sPAPREnvironment *spapr,
+                                  uint32_t token, uint32_t nargs,
+                                  target_ulong args, uint32_t nret,
+                                  target_ulong rets)
+{
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 3) || (nret != 1)) {
+        goto param_error_exit;
+    }
+
+    rtas_finish_eeh_request(sphb, RTAS_EEH_REQ_CONFIGURE, 0, rets);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+/* To support it later */
+static void rtas_ibm_slot_error_detail(PowerPCCPU *cpu,
+                                       sPAPREnvironment *spapr,
+                                       uint32_t token, uint32_t nargs,
+                                       target_ulong args, uint32_t nret,
+                                       target_ulong rets)
+{
+    int option;
+    uint64_t buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    sPAPRPHBState *sphb = spapr_find_phb(spapr, buid);
+    sPAPRPHBClass *info = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+
+    if (!sphb || !info->eeh_handler) {
+        goto param_error_exit;
+    }
+
+    if ((nargs != 8) || (nret != 1)) {
+        goto param_error_exit;
+    }
+
+    option = rtas_ld(args, 7);
+    if (option != RTAS_SLOT_TEMP_ERR_LOG &&
+        option != RTAS_SLOT_PERM_ERR_LOG) {
+        goto param_error_exit;
+    }
+
+    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
 static int pci_spapr_swizzle(int slot, int pin)
 {
     return (slot + pin) % PCI_NUM_PINS;
@@ -941,6 +1176,19 @@ void spapr_pci_rtas_init(void)
                             rtas_ibm_query_interrupt_source_number);
         spapr_rtas_register("ibm,change-msi", rtas_ibm_change_msi);
     }
+
+    spapr_rtas_register("ibm,set-eeh-option",
+                        rtas_ibm_set_eeh_option);
+    spapr_rtas_register("ibm,get-config-addr-info2",
+                        rtas_ibm_get_config_addr_info2);
+    spapr_rtas_register("ibm,read-slot-reset-state2",
+                        rtas_ibm_read_slot_reset_state2);
+    spapr_rtas_register("ibm,set-slot-reset",
+                        rtas_ibm_set_slot_reset);
+    spapr_rtas_register("ibm,configure-pe",
+                        rtas_ibm_configure_pe);
+    spapr_rtas_register("ibm,slot-error-detail",
+                        rtas_ibm_slot_error_detail);
 }
 
 static void spapr_pci_register_types(void)
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index a695f12..d16f29b 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -51,6 +51,7 @@ struct sPAPRPHBClass {
     PCIHostBridgeClass parent_class;
 
     void (*finish_realize)(sPAPRPHBState *sphb, Error **errp);
+    int (*eeh_handler)(sPAPRPHBState *sphb, int req, int opt);
 };
 
 typedef struct sPAPRPHBState {
@@ -105,6 +106,12 @@ struct sPAPRPHBVFIOState {
 
 #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL
 
+/* EEH related requests */
+#define RTAS_EEH_REQ_SET_OPTION      0
+#define RTAS_EEH_REQ_GET_STATE       1
+#define RTAS_EEH_REQ_RESET           2
+#define RTAS_EEH_REQ_CONFIGURE       3
+
 static inline qemu_irq spapr_phb_lsi_qirq(struct sPAPRPHBState *phb, int pin)
 {
     return xics_get_qirq(spapr->icp, phb->lsi_table[pin].irq);
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index f1d307f..c9416c0 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -342,6 +342,39 @@ static inline int spapr_allocate_lsi(int hint)
     return spapr_allocate_irq(hint, true);
 }
 
+/* ibm,set-eeh-option */
+#define RTAS_EEH_DISABLE                       0
+#define RTAS_EEH_ENABLE                        1
+#define RTAS_EEH_THAW_IO                       2
+#define RTAS_EEH_THAW_DMA                      3
+
+/* ibm,get-config-addr-info2 */
+#define RTAS_GET_PE_ADDR                       0
+#define RTAS_GET_PE_MODE                       1
+#define RTAS_PE_MODE_NONE                      0
+#define RTAS_PE_MODE_NOT_SHARED                1
+#define RTAS_PE_MODE_SHARED                    2
+
+/* ibm,read-slot-reset-state2 */
+#define RTAS_EEH_PE_STATE_NORMAL               0
+#define RTAS_EEH_PE_STATE_RESET                1
+#define RTAS_EEH_PE_STATE_STOPPED_IO_DMA       2
+#define RTAS_EEH_PE_STATE_STOPPED_DMA          4
+#define RTAS_EEH_PE_STATE_UNAVAIL              5
+#define RTAS_EEH_NOT_SUPPORT                   0
+#define RTAS_EEH_SUPPORT                       1
+#define RTAS_EEH_PE_UNAVAIL_INFO               1000
+#define RTAS_EEH_PE_RECOVER_INFO               0
+
+/* ibm,set-slot-reset */
+#define RTAS_SLOT_RESET_DEACTIVATE             0
+#define RTAS_SLOT_RESET_HOT                    1
+#define RTAS_SLOT_RESET_FUNDAMENTAL            3
+
+/* ibm,slot-error-detail */
+#define RTAS_SLOT_TEMP_ERR_LOG                 1
+#define RTAS_SLOT_PERM_ERR_LOG                 2
+
 /* RTAS return codes */
 #define RTAS_OUT_SUCCESS            0
 #define RTAS_OUT_NO_ERRORS_FOUND    1
-- 
1.8.3.2

Re: [Qemu-devel] [PATCH v8 1/4] sPAPR: Implement EEH RTAS calls

[Alexander Graf]

On 05.06.14 08:53, Gavin Shan wrote:
> The emulation for EEH RTAS requests from guest isn't covered
> by QEMU yet and the patch implements them.
>
> The patch defines constants used by EEH RTAS calls and adds
> callback sPAPRPHBClass::eeh_handler, which is going to be used
> this way:
>
> 1. RTAS calls are received in spapr_pci.c, sanity check is done
>     there.
> 2. RTAS handlers handle what they can. If there is something it
>     cannot handle and sPAPRPHBClass::eeh_handler callback is defined,
>     it is called.
> 3. sPAPRPHBClass::eeh_handler is only implemented for VFIO now. It
>     does ioctl() to the IOMMU container fd to complete the call. Error
>     codes from that ioctl() are transferred back to the guest.
>
> This adds 6 RTAS handlers, all defined in SPAPR specification:
>
> 1) ibm,set-eeh-option: disables/enables EEH on a device, removes PE from
> stopped state;
> 2) ibm,get-config-addr-info2 - returns fabric configuration address (upper
> PCI bridge or PHB if there is no bridge);
> 3) ibm,read-slot-reset-state2 - retrieve PE state;
> 4) ibm,set-slot-reset - issue PE reset;
> 5) ibm,configure-pe - configure PCI bridges in the affected PE;
> 6) ibm,slot-error-detail - retrieve EEH error log;
>
> All calls use fabric configuration address (a.k.a. PE address) as a target
> address except ibm,get-config-addr-info2 and one case (enable EEH on the
> specified PCI function) for ibm,set-eeh-option.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>   hw/ppc/spapr_pci.c          | 248 ++++++++++++++++++++++++++++++++++++++++++++
>   include/hw/pci-host/spapr.h |   7 ++
>   include/hw/ppc/spapr.h      |  33 ++++++
>   3 files changed, 288 insertions(+)
>
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index a9f307a..423e4ff 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -422,6 +422,241 @@ static void rtas_ibm_query_interrupt_source_number(PowerPCCPU *cpu,
>       rtas_st(rets, 2, 1);/* 0 == level; 1 == edge */
>   }
>   
> +static int rtas_finish_eeh_request(sPAPRPHBState *sphb,
> +                                   uint32_t req, uint32_t opt,
> +                                   target_ulong rets)

The only thing I dislike about this patch is the name of this function. 
It doesn't finish the eeh request - it actually does it :).


Alex

Re: [Qemu-devel] [PATCH v8 1/4] sPAPR: Implement EEH RTAS calls

[Gavin Shan]

On Thu, Jun 05, 2014 at 02:09:14PM +0200, Alexander Graf wrote:
>
>On 05.06.14 08:53, Gavin Shan wrote:
>>The emulation for EEH RTAS requests from guest isn't covered
>>by QEMU yet and the patch implements them.
>>
>>The patch defines constants used by EEH RTAS calls and adds
>>callback sPAPRPHBClass::eeh_handler, which is going to be used
>>this way:
>>
>>1. RTAS calls are received in spapr_pci.c, sanity check is done
>>    there.
>>2. RTAS handlers handle what they can. If there is something it
>>    cannot handle and sPAPRPHBClass::eeh_handler callback is defined,
>>    it is called.
>>3. sPAPRPHBClass::eeh_handler is only implemented for VFIO now. It
>>    does ioctl() to the IOMMU container fd to complete the call. Error
>>    codes from that ioctl() are transferred back to the guest.
>>
>>This adds 6 RTAS handlers, all defined in SPAPR specification:
>>
>>1) ibm,set-eeh-option: disables/enables EEH on a device, removes PE from
>>stopped state;
>>2) ibm,get-config-addr-info2 - returns fabric configuration address (upper
>>PCI bridge or PHB if there is no bridge);
>>3) ibm,read-slot-reset-state2 - retrieve PE state;
>>4) ibm,set-slot-reset - issue PE reset;
>>5) ibm,configure-pe - configure PCI bridges in the affected PE;
>>6) ibm,slot-error-detail - retrieve EEH error log;
>>
>>All calls use fabric configuration address (a.k.a. PE address) as a target
>>address except ibm,get-config-addr-info2 and one case (enable EEH on the
>>specified PCI function) for ibm,set-eeh-option.
>>
>>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>---
>>  hw/ppc/spapr_pci.c          | 248 ++++++++++++++++++++++++++++++++++++++++++++
>>  include/hw/pci-host/spapr.h |   7 ++
>>  include/hw/ppc/spapr.h      |  33 ++++++
>>  3 files changed, 288 insertions(+)
>>
>>diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>>index a9f307a..423e4ff 100644
>>--- a/hw/ppc/spapr_pci.c
>>+++ b/hw/ppc/spapr_pci.c
>>@@ -422,6 +422,241 @@ static void rtas_ibm_query_interrupt_source_number(PowerPCCPU *cpu,
>>      rtas_st(rets, 2, 1);/* 0 == level; 1 == edge */
>>  }
>>+static int rtas_finish_eeh_request(sPAPRPHBState *sphb,
>>+                                   uint32_t req, uint32_t opt,
>>+                                   target_ulong rets)
>
>The only thing I dislike about this patch is the name of this
>function. It doesn't finish the eeh request - it actually does it :).
>

Ok. I'll change it to rtas_handle_eeh_request() :-)

Thanks,
Gavin

>
>Alex
>

[Qemu-devel] [PATCH v8 2/4] headers: Update kernel header

[Gavin Shan]

This updates kernel header (vfio.h) for EEH support on VFIO PCI
devices.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 linux-headers/linux/vfio.h | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
index f0aa97d..cf703d3 100644
--- a/linux-headers/linux/vfio.h
+++ b/linux-headers/linux/vfio.h
@@ -30,6 +30,9 @@
  */
 #define VFIO_DMA_CC_IOMMU		4
 
+/* Check if EEH is supported */
+#define VFIO_EEH			5
+
 /*
  * The IOCTL interface is designed for extensibility by embedding the
  * structure length (argsz) and flags into structures passed between
@@ -490,6 +493,38 @@ struct vfio_iommu_spapr_tce_reset {
 };
 #define VFIO_IOMMU_SPAPR_TCE_RESET	_IO(VFIO_TYPE, VFIO_BASE + 20)
 
+/*
+ * EEH PE operation struct provides ways to:
+ * - enable/disable EEH functionality;
+ * - unfreeze IO/DMA for frozen PE;
+ * - read PE state;
+ * - reset PE;
+ * - configure PE.
+ */
+struct vfio_eeh_pe_op {
+	__u32 argsz;
+	__u32 flags;
+	__u32 op;
+};
+
+#define VFIO_EEH_PE_DISABLE		0	/* Disable EEH functionality */
+#define VFIO_EEH_PE_ENABLE		1	/* Enable EEH functionality  */
+#define VFIO_EEH_PE_UNFREEZE_IO		2	/* Enable IO for frozen PE   */
+#define VFIO_EEH_PE_UNFREEZE_DMA	3	/* Enable DMA for frozen PE  */
+#define VFIO_EEH_PE_GET_STATE		4	/* PE state retrieval        */
+#define VFIO_EEH_PE_RESET_DEACTIVATE	5	/* Deassert PE reset         */
+#define VFIO_EEH_PE_RESET_HOT		6	/* Assert hot reset          */
+#define VFIO_EEH_PE_RESET_FUNDAMENTAL	7	/* Assert fundamental reset  */
+#define VFIO_EEH_PE_CONFIGURE		8	/* PE configuration          */
+
+#define VFIO_EEH_PE_STATE_NORMAL	0	/* PE in functional state    */
+#define VFIO_EEH_PE_STATE_RESET		1	/* PE reset in progress      */
+#define VFIO_EEH_PE_STATE_STOPPED	2	/* Stopped DMA and IO        */
+#define VFIO_EEH_PE_STATE_STOPPED_DMA	4	/* Stopped DMA only          */
+#define VFIO_EEH_PE_STATE_UNAVAIL	5	/* State unavailable         */
+
+#define VFIO_EEH_PE_OP			_IO(VFIO_TYPE, VFIO_BASE + 21)
+
 /* ***************************************************************** */
 
 #endif /* VFIO_H */
-- 
1.8.3.2

[Qemu-devel] [PATCH v8 3/4] VFIO: Introduce helper vfio_pci_container_ioctl()

[Gavin Shan]

The patch introduces helper function vfio_pci_container_ioctl() to
pass ioctl commands to the specified VFIO container that is identified
by IOMMU group id. On sPAPR platform, each container only has one
IOMMU group.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 hw/misc/vfio.c         | 31 +++++++++++++++++++++++++++++++
 include/hw/misc/vfio.h |  2 ++
 2 files changed, 33 insertions(+)

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index 0796abf..999d97d 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -4310,3 +4310,34 @@ put_group_exit:
 
     return n;
 }
+
+int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt)
+{
+    VFIOGroup *group;
+    int ret, fd = 0;
+
+    /* Search container's fd */
+    QLIST_FOREACH(group, &group_list, next) {
+        if (group->groupid == iommu_group_id) {
+            fd = group->container ? group->container->fd : 0;
+            break;
+        }
+    }
+
+    if (fd <= 0) {
+        return -ENOENT;
+    }
+
+    switch (req) {
+    case VFIO_EEH_PE_OP: {
+	struct vfio_eeh_pe_op op = { .argsz = sizeof(op), .op = opt };
+
+        ret = ioctl(fd, req, &op);
+        break;
+    }
+    default:
+        ret = -EINVAL;
+    }
+
+    return ret;
+}
diff --git a/include/hw/misc/vfio.h b/include/hw/misc/vfio.h
index 53ec665..dc92fae 100644
--- a/include/hw/misc/vfio.h
+++ b/include/hw/misc/vfio.h
@@ -30,4 +30,6 @@ static inline long vfio_kvm_notify(Notifier *n, unsigned request, void *data)
     return p.ret;
 }
 
+extern int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt);
+
 #endif
-- 
1.8.3.2

Re: [Qemu-devel] [PATCH v8 3/4] VFIO: Introduce helper vfio_pci_container_ioctl()

[Alexander Graf]

On 05.06.14 08:53, Gavin Shan wrote:
> The patch introduces helper function vfio_pci_container_ioctl() to
> pass ioctl commands to the specified VFIO container that is identified
> by IOMMU group id. On sPAPR platform, each container only has one
> IOMMU group.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>   hw/misc/vfio.c         | 31 +++++++++++++++++++++++++++++++
>   include/hw/misc/vfio.h |  2 ++
>   2 files changed, 33 insertions(+)
>
> diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> index 0796abf..999d97d 100644
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -4310,3 +4310,34 @@ put_group_exit:
>   
>       return n;
>   }
> +
> +int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt)
> +{
> +    VFIOGroup *group;
> +    int ret, fd = 0;
> +
> +    /* Search container's fd */
> +    QLIST_FOREACH(group, &group_list, next) {
> +        if (group->groupid == iommu_group_id) {
> +            fd = group->container ? group->container->fd : 0;
> +            break;
> +        }
> +    }
> +
> +    if (fd <= 0) {

fd 0 is a valid file descriptor.


Alex

Re: [Qemu-devel] [PATCH v8 3/4] VFIO: Introduce helper vfio_pci_container_ioctl()

[Gavin Shan]

On Thu, Jun 05, 2014 at 02:11:21PM +0200, Alexander Graf wrote:
>
>On 05.06.14 08:53, Gavin Shan wrote:
>>The patch introduces helper function vfio_pci_container_ioctl() to
>>pass ioctl commands to the specified VFIO container that is identified
>>by IOMMU group id. On sPAPR platform, each container only has one
>>IOMMU group.
>>
>>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>---
>>  hw/misc/vfio.c         | 31 +++++++++++++++++++++++++++++++
>>  include/hw/misc/vfio.h |  2 ++
>>  2 files changed, 33 insertions(+)
>>
>>diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
>>index 0796abf..999d97d 100644
>>--- a/hw/misc/vfio.c
>>+++ b/hw/misc/vfio.c
>>@@ -4310,3 +4310,34 @@ put_group_exit:
>>      return n;
>>  }
>>+
>>+int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt)
>>+{
>>+    VFIOGroup *group;
>>+    int ret, fd = 0;
>>+
>>+    /* Search container's fd */
>>+    QLIST_FOREACH(group, &group_list, next) {
>>+        if (group->groupid == iommu_group_id) {
>>+            fd = group->container ? group->container->fd : 0;
>>+            break;
>>+        }
>>+    }
>>+
>>+    if (fd <= 0) {
>
>fd 0 is a valid file descriptor.
>

Yep, I'll fix :)

Thanks,
Gavin

>
>Alex
>

Re: [Qemu-devel] [PATCH v8 3/4] VFIO: Introduce helper vfio_pci_container_ioctl()

[Alex Williamson]

On Thu, 2014-06-05 at 16:53 +1000, Gavin Shan wrote:
> The patch introduces helper function vfio_pci_container_ioctl() to
> pass ioctl commands to the specified VFIO container that is identified
> by IOMMU group id. On sPAPR platform, each container only has one
> IOMMU group.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  hw/misc/vfio.c         | 31 +++++++++++++++++++++++++++++++
>  include/hw/misc/vfio.h |  2 ++
>  2 files changed, 33 insertions(+)
> 
> diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
> index 0796abf..999d97d 100644
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -4310,3 +4310,34 @@ put_group_exit:
>  
>      return n;
>  }
> +
> +int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt)
> +{
> +    VFIOGroup *group;
> +    int ret, fd = 0;
> +
> +    /* Search container's fd */
> +    QLIST_FOREACH(group, &group_list, next) {
> +        if (group->groupid == iommu_group_id) {
> +            fd = group->container ? group->container->fd : 0;
> +            break;
> +        }
> +    }
> +
> +    if (fd <= 0) {
> +        return -ENOENT;
> +    }
> +
> +    switch (req) {
> +    case VFIO_EEH_PE_OP: {
> +	struct vfio_eeh_pe_op op = { .argsz = sizeof(op), .op = opt };
> +
> +        ret = ioctl(fd, req, &op);
> +        break;

In addition to fd 0 being valid, there's some white space issues here.

Passing an integer option is not very extensible, maybe a void* that
gets cast to an int* for VFIO_EEH_PE_OP would be better.  It's a qemu
internal API though, so I'm not going to sweat saving that problem for
the next user.  Thanks,

Alex

> +    }
> +    default:
> +        ret = -EINVAL;
> +    }
> +
> +    return ret;
> +}
> diff --git a/include/hw/misc/vfio.h b/include/hw/misc/vfio.h
> index 53ec665..dc92fae 100644
> --- a/include/hw/misc/vfio.h
> +++ b/include/hw/misc/vfio.h
> @@ -30,4 +30,6 @@ static inline long vfio_kvm_notify(Notifier *n, unsigned request, void *data)
>      return p.ret;
>  }
>  
> +extern int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt);
> +
>  #endif

Re: [Qemu-devel] [PATCH v8 3/4] VFIO: Introduce helper vfio_pci_container_ioctl()

[Gavin Shan]

On Thu, Jun 05, 2014 at 12:27:23PM -0600, Alex Williamson wrote:
>On Thu, 2014-06-05 at 16:53 +1000, Gavin Shan wrote:
>> The patch introduces helper function vfio_pci_container_ioctl() to
>> pass ioctl commands to the specified VFIO container that is identified
>> by IOMMU group id. On sPAPR platform, each container only has one
>> IOMMU group.
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  hw/misc/vfio.c         | 31 +++++++++++++++++++++++++++++++
>>  include/hw/misc/vfio.h |  2 ++
>>  2 files changed, 33 insertions(+)
>> 
>> diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
>> index 0796abf..999d97d 100644
>> --- a/hw/misc/vfio.c
>> +++ b/hw/misc/vfio.c
>> @@ -4310,3 +4310,34 @@ put_group_exit:
>>  
>>      return n;
>>  }
>> +
>> +int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt)
>> +{
>> +    VFIOGroup *group;
>> +    int ret, fd = 0;
>> +
>> +    /* Search container's fd */
>> +    QLIST_FOREACH(group, &group_list, next) {
>> +        if (group->groupid == iommu_group_id) {
>> +            fd = group->container ? group->container->fd : 0;
>> +            break;
>> +        }
>> +    }
>> +
>> +    if (fd <= 0) {
>> +        return -ENOENT;
>> +    }
>> +
>> +    switch (req) {
>> +    case VFIO_EEH_PE_OP: {
>> +	struct vfio_eeh_pe_op op = { .argsz = sizeof(op), .op = opt };
>> +
>> +        ret = ioctl(fd, req, &op);
>> +        break;
>
>In addition to fd 0 being valid, there's some white space issues here.
>

Thanks and I'll fix :-)

>Passing an integer option is not very extensible, maybe a void* that
>gets cast to an int* for VFIO_EEH_PE_OP would be better.  It's a qemu
>internal API though, so I'm not going to sweat saving that problem for
>the next user.  Thanks,
>

yep, I'll change accordingly.

Thanks,
Gavin

>Alex
>
>> +    }
>> +    default:
>> +        ret = -EINVAL;
>> +    }
>> +
>> +    return ret;
>> +}
>> diff --git a/include/hw/misc/vfio.h b/include/hw/misc/vfio.h
>> index 53ec665..dc92fae 100644
>> --- a/include/hw/misc/vfio.h
>> +++ b/include/hw/misc/vfio.h
>> @@ -30,4 +30,6 @@ static inline long vfio_kvm_notify(Notifier *n, unsigned request, void *data)
>>      return p.ret;
>>  }
>>  
>> +extern int vfio_pci_container_ioctl(int iommu_group_id, int req, int opt);
>> +
>>  #endif
>
>
>

[Qemu-devel] [PATCH v8 4/4] sPAPR: Implement sPAPRPHBClass::eeh_handler

[Gavin Shan]

The patch implements sPAPRPHBClass::eeh_handler so that the
EEH RTAS requests can be routed to VFIO for further handling.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 hw/ppc/spapr_pci_vfio.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index decf3dd..94bebc2 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -86,6 +86,59 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
                                               spapr_tce_get_iommu(tcet));
 }
 
+static int spapr_phb_vfio_eeh_handler(sPAPRPHBState *sphb, int req, int opt)
+{
+    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
+    int cmd;
+
+    switch (req) {
+    case RTAS_EEH_REQ_SET_OPTION:
+        switch (opt) {
+        case RTAS_EEH_DISABLE:
+            cmd = VFIO_EEH_PE_DISABLE;
+            break;
+        case RTAS_EEH_ENABLE:
+            cmd = VFIO_EEH_PE_ENABLE;
+            break;
+        case RTAS_EEH_THAW_IO:
+            cmd = VFIO_EEH_PE_UNFREEZE_IO;
+            break;
+        case RTAS_EEH_THAW_DMA:
+            cmd = VFIO_EEH_PE_UNFREEZE_DMA;
+            break;
+        default:
+            return -EINVAL;
+        }
+        break;
+    case RTAS_EEH_REQ_GET_STATE:
+        cmd = VFIO_EEH_PE_GET_STATE;
+        break;
+    case RTAS_EEH_REQ_RESET:
+        switch (opt) {
+        case RTAS_SLOT_RESET_DEACTIVATE:
+            cmd = VFIO_EEH_PE_RESET_DEACTIVATE;
+            break;
+        case RTAS_SLOT_RESET_HOT:
+            cmd = VFIO_EEH_PE_RESET_HOT;
+            break;
+        case RTAS_SLOT_RESET_FUNDAMENTAL:
+            cmd = VFIO_EEH_PE_RESET_FUNDAMENTAL;
+            break;
+        default:
+            return -EINVAL;
+        }
+        break;
+    case RTAS_EEH_REQ_CONFIGURE:
+        cmd = VFIO_EEH_PE_CONFIGURE;
+        break;
+    default:
+         return -EINVAL;
+    }
+
+    return vfio_pci_container_ioctl(svphb->iommugroupid,
+                                    VFIO_EEH_PE_OP, cmd);
+}
+
 static void spapr_phb_vfio_reset(DeviceState *qdev)
 {
     /* Do nothing */
@@ -99,6 +152,7 @@ static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
     dc->props = spapr_phb_vfio_properties;
     dc->reset = spapr_phb_vfio_reset;
     spc->finish_realize = spapr_phb_vfio_finish_realize;
+    spc->eeh_handler = spapr_phb_vfio_eeh_handler;
 }
 
 static const TypeInfo spapr_phb_vfio_info = {
-- 
1.8.3.2

Re: [Qemu-devel] [PATCH v8 4/4] sPAPR: Implement sPAPRPHBClass::eeh_handler

[Alexander Graf]

On 05.06.14 08:53, Gavin Shan wrote:
> The patch implements sPAPRPHBClass::eeh_handler so that the
> EEH RTAS requests can be routed to VFIO for further handling.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Just to make sure I grasp this correctly.

We have PHBs that are VFIO PHBs and PHBs that are emulation PHBs. 
Emulation PHBs can only have emulated devices attached. VFIO PHBs can 
only have a single IOMMU group's devices attached each. A container can 
not span 2 different VFIO PHBs.

Is this correct? If not the code below needs to take care of any 
abstraction we have in between and maybe limit scope a bit to make sure 
the guest doesn't kill devices on other PHBs :).


Alex

Re: [Qemu-devel] [PATCH v8 4/4] sPAPR: Implement sPAPRPHBClass::eeh_handler

[Alexey Kardashevskiy]

On 06/05/2014 11:00 PM, Alexander Graf wrote:
> 
> On 05.06.14 08:53, Gavin Shan wrote:
>> The patch implements sPAPRPHBClass::eeh_handler so that the
>> EEH RTAS requests can be routed to VFIO for further handling.
>>
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> 
> Just to make sure I grasp this correctly.
> 
> We have PHBs that are VFIO PHBs and PHBs that are emulation PHBs. Emulation
> PHBs can only have emulated devices attached. VFIO PHBs can only have a
> single IOMMU group's devices attached each. A container can not span 2
> different VFIO PHBs.
> 
> Is this correct? 


Yes, this is correct. PHBs are too cheap on SPAPR to think of any
additional complication here :)


> If not the code below needs to take care of any
> abstraction we have in between and maybe limit scope a bit to make sure the
> guest doesn't kill devices on other PHBs :).



-- 
Alexey

Re: [Qemu-devel] [PATCH v8 0/4] EEH Support for VFIO PCI Device

[Alexander Graf]

On 05.06.14 08:53, Gavin Shan wrote:
> The series of patches adds support EEH for VFIO PCI devices on sPAPR platform.
> It requires corresponding host kernel support. Also, it is based on top of
> Alexey's VFIO-for-sPAPR git repository.
>
> QEMU:   git://github.com/aik/qemu.git (branch: vfio)
> Kernel: git://github.com/aik/linux.git (branch: vfio)
> Kernel: http://linuxppc.10917.n7.nabble.com/PATCH-v8-0-3-EEH-Support-for-VFIO-PCI-Device-td82940.html
>
> The implementations notes are below. Please comment.

Looks pretty straight forward, but let's wait for Alex's comments too ;).


Alex

Re: [Qemu-devel] [PATCH v8 0/4] EEH Support for VFIO PCI Device

[Alex Williamson]

On Thu, 2014-06-05 at 14:12 +0200, Alexander Graf wrote:
> On 05.06.14 08:53, Gavin Shan wrote:
> > The series of patches adds support EEH for VFIO PCI devices on sPAPR platform.
> > It requires corresponding host kernel support. Also, it is based on top of
> > Alexey's VFIO-for-sPAPR git repository.
> >
> > QEMU:   git://github.com/aik/qemu.git (branch: vfio)
> > Kernel: git://github.com/aik/linux.git (branch: vfio)
> > Kernel: http://linuxppc.10917.n7.nabble.com/PATCH-v8-0-3-EEH-Support-for-VFIO-PCI-Device-td82940.html
> >
> > The implementations notes are below. Please comment.
> 
> Looks pretty straight forward, but let's wait for Alex's comments too ;).

Yep, it's looking pretty good to me.  Thanks,

Alex