[Qemu-devel] [PATCH v4 0/3] pcie: Add support for Single Root I/O Virtualization

[Qemu-devel] [PATCH v4 0/3] pcie: Add support for Single Root I/O Virtualization

[Knut Omang]

This patch set implements generic support for SR/IOV as an extension to the
core PCIe functionality, similar to the way other capabilities such as AER
is implemented.

There is no implementation of any device that provides
SR/IOV support included, but I have implemented a test
example which can be found together with this patch set here:

  git://github.com/knuto/qemu.git sriov_patches_v4

Testing with the example device was documented here:

  http://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg05110.html

Changes since v3:
  - Reworked 'pci: Update pci_regs header' to merge kernel version improvements
    with the current qemu version instead of copying from the kernel version.

Changes since v2:
  - Rebased onto 090d0bfd
  - Un-qdev'ified - avoids issues when resetting NUM_VFS
  - Fixed handling of vf_offset/vf_stride

Changes since v1:
  - Rebased on top of latest master, eliminating prereqs.
  - Implement proper support for VF_STRIDE, VF_OFFSET and SUP_PGSIZE
    Time better spent fixing it than explaining what the previous
    limitations were.
    - Added new first patch to fix pci bug related to this
  - Split out patch to pci_default_config_write to a separate patch 2
    to highlight bug fix.
  - Refactored out logic into new source files
    hw/pci/pcie_sriov.c include/hw/pci/pcie_sriov.h
    similar to pcie_aer.c/h.
  - Rename functions and introduce structs to better separate
    pf and vf functionality.
  - Replaced is_vf member with pci_is_vf() function abstraction
  - Fix numerous syntax, whitespace and comment issues
    according to Michael's review.
  - Fix memory leaks.
  - Removed igb example device - a rebased version available
    on github instead.

Knut Omang (3):
  pci: Make use of the devfn property when registering new devices
  pci: Update pci_regs header
  pcie: Add support for Single Root I/O Virtualization (SR/IOV)

 hw/pci/Makefile.objs                      |   2 +-
 hw/pci/pci.c                              | 101 ++++++++---
 hw/pci/pcie.c                             |   9 +-
 hw/pci/pcie_sriov.c                       | 271 ++++++++++++++++++++++++++++++
 include/hw/pci/pci.h                      |  11 +-
 include/hw/pci/pcie.h                     |   6 +
 include/hw/pci/pcie_sriov.h               |  55 ++++++
 include/qemu/typedefs.h                   |   2 +
 include/standard-headers/linux/pci_regs.h | 142 +++++++++++++---
 9 files changed, 545 insertions(+), 54 deletions(-)
 create mode 100644 hw/pci/pcie_sriov.c
 create mode 100644 include/hw/pci/pcie_sriov.h

--
2.4.3

[Qemu-devel] [PATCH v4 1/3] pci: Make use of the devfn property when registering new devices

[Knut Omang]

Without this, the devfn argument to pci_create_*()
does not affect the assigned devfn.

Needed to support (VF_STRIDE,VF_OFFSET) values other than (1,1)
for SR/IOV.

Signed-off-by: Knut Omang <knut.omang@oracle.com>
---
 hw/pci/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index ccea628..a5cc015 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1840,7 +1840,7 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
     bus = PCI_BUS(qdev_get_parent_bus(qdev));
     pci_dev = do_pci_register_device(pci_dev, bus,
                                      object_get_typename(OBJECT(qdev)),
-                                     pci_dev->devfn, errp);
+                                     object_property_get_int(OBJECT(qdev), "addr", NULL), errp);
     if (pci_dev == NULL)
         return;
 
-- 
2.4.3

Re: [Qemu-devel] [PATCH v4 1/3] pci: Make use of the devfn property when registering new devices

[Marcel Apfelbaum]

On 09/12/2015 03:36 PM, Knut Omang wrote:
> Without this, the devfn argument to pci_create_*()
> does not affect the assigned devfn.
>
> Needed to support (VF_STRIDE,VF_OFFSET) values other than (1,1)
> for SR/IOV.
>
> Signed-off-by: Knut Omang <knut.omang@oracle.com>
> ---
>   hw/pci/pci.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index ccea628..a5cc015 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -1840,7 +1840,7 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
>       bus = PCI_BUS(qdev_get_parent_bus(qdev));
>       pci_dev = do_pci_register_device(pci_dev, bus,
>                                        object_get_typename(OBJECT(qdev)),
> -                                     pci_dev->devfn, errp);
> +                                     object_property_get_int(OBJECT(qdev), "addr", NULL), errp);
Hi,

I don't get this, using object_property_get_int on "addr" should return the value of pci_dev->devfn,
can you please tell what exactly is not working?

Thanks,
Marcel



>       if (pci_dev == NULL)
>           return;
>
>

Re: [Qemu-devel] [PATCH v4 1/3] pci: Make use of the devfn property when registering new devices

[Knut Omang]

On Thu, 2015-09-17 at 14:11 +0300, Marcel Apfelbaum wrote:
> On 09/12/2015 03:36 PM, Knut Omang wrote:
> > Without this, the devfn argument to pci_create_*()
> > does not affect the assigned devfn.
> > 
> > Needed to support (VF_STRIDE,VF_OFFSET) values other than (1,1)
> > for SR/IOV.
> > 
> > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > ---
> >   hw/pci/pci.c | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > index ccea628..a5cc015 100644
> > --- a/hw/pci/pci.c
> > +++ b/hw/pci/pci.c
> > @@ -1840,7 +1840,7 @@ static void pci_qdev_realize(DeviceState
> > *qdev, Error **errp)
> >       bus = PCI_BUS(qdev_get_parent_bus(qdev));
> >       pci_dev = do_pci_register_device(pci_dev, bus,
> >                                       
> >  object_get_typename(OBJECT(qdev)),
> > -                                     pci_dev->devfn, errp);
> > +                                    
> >  object_property_get_int(OBJECT(qdev), "addr", NULL), errp);
> Hi,
> 
> I don't get this, using object_property_get_int on "addr" should
> return the value of pci_dev->devfn,
> can you please tell what exactly is not working?

The problem is that at that point pci_dev->devfn has not been set yet -
have commented on this before somewhere..

Knut

> Thanks,
> Marcel
> 
> 
> 
> >       if (pci_dev == NULL)
> >           return;
> > 
> > 
> 

Re: [Qemu-devel] [PATCH v4 1/3] pci: Make use of the devfn property when registering new devices

[Marcel Apfelbaum]

On 09/17/2015 03:12 PM, Knut Omang wrote:
> On Thu, 2015-09-17 at 14:11 +0300, Marcel Apfelbaum wrote:
>> On 09/12/2015 03:36 PM, Knut Omang wrote:
>>> Without this, the devfn argument to pci_create_*()
>>> does not affect the assigned devfn.
>>>
>>> Needed to support (VF_STRIDE,VF_OFFSET) values other than (1,1)
>>> for SR/IOV.
>>>
>>> Signed-off-by: Knut Omang <knut.omang@oracle.com>
>>> ---
>>>    hw/pci/pci.c | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
>>> index ccea628..a5cc015 100644
>>> --- a/hw/pci/pci.c
>>> +++ b/hw/pci/pci.c
>>> @@ -1840,7 +1840,7 @@ static void pci_qdev_realize(DeviceState
>>> *qdev, Error **errp)
>>>        bus = PCI_BUS(qdev_get_parent_bus(qdev));
>>>        pci_dev = do_pci_register_device(pci_dev, bus,
>>>
>>>   object_get_typename(OBJECT(qdev)),
>>> -                                     pci_dev->devfn, errp);
>>> +
>>>   object_property_get_int(OBJECT(qdev), "addr", NULL), errp);
>> Hi,
>>
>> I don't get this, using object_property_get_int on "addr" should
>> return the value of pci_dev->devfn,
>> can you please tell what exactly is not working?
>
> The problem is that at that point pci_dev->devfn has not been set yet -
> have commented on this before somewhere..
>

But "addr" property has the right value? Is indeed strange because it should
get the value from pci_dev->devfn.

Don't get me wrong, this patch is OK.
I just want to understand if we have a hidden bug somewhere.

Anyway,
Reviewed-by: Marcel Apfelbaum <marcel@redhat.com>

> Knut
>
>> Thanks,
>> Marcel
>>
>>
>>
>>>        if (pci_dev == NULL)
>>>            return;
>>>
>>>
>>

Re: [Qemu-devel] [PATCH v4 1/3] pci: Make use of the devfn property when registering new devices

[Knut Omang]

On Thu, 2015-09-17 at 15:41 +0300, Marcel Apfelbaum wrote:
> On 09/17/2015 03:12 PM, Knut Omang wrote:
> > On Thu, 2015-09-17 at 14:11 +0300, Marcel Apfelbaum wrote:
> > > On 09/12/2015 03:36 PM, Knut Omang wrote:
> > > > Without this, the devfn argument to pci_create_*()
> > > > does not affect the assigned devfn.
> > > > 
> > > > Needed to support (VF_STRIDE,VF_OFFSET) values other than (1,1)
> > > > for SR/IOV.
> > > > 
> > > > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > > > ---
> > > >    hw/pci/pci.c | 2 +-
> > > >    1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > > > index ccea628..a5cc015 100644
> > > > --- a/hw/pci/pci.c
> > > > +++ b/hw/pci/pci.c
> > > > @@ -1840,7 +1840,7 @@ static void pci_qdev_realize(DeviceState
> > > > *qdev, Error **errp)
> > > >        bus = PCI_BUS(qdev_get_parent_bus(qdev));
> > > >        pci_dev = do_pci_register_device(pci_dev, bus,
> > > > 
> > > >   object_get_typename(OBJECT(qdev)),
> > > > -                                     pci_dev->devfn, errp);
> > > > +
> > > >   object_property_get_int(OBJECT(qdev), "addr", NULL), errp);
> > > Hi,
> > > 
> > > I don't get this, using object_property_get_int on "addr" should
> > > return the value of pci_dev->devfn,
> > > can you please tell what exactly is not working?
> > 
> > The problem is that at that point pci_dev->devfn has not been set
> > yet -
> > have commented on this before somewhere..
> > 
> 
> But "addr" property has the right value? Is indeed strange because it
> should
> get the value from pci_dev->devfn.

In the current version, in pci_qdev_realize() 
the devfn argument to do_pci_register_device() is set to 
pci_dev->devfn. Then inside do_pci_register_device that devfn argument
is again used to set pci_dev->devfn.

When the SR/IOV code tries to add a second device (the first VF) at 
(devfn + offset + stride) to the bus, it fails because devfn is 0 and
that hits the PF in the devices[] array.

> Don't get me wrong, this patch is OK.
> I just want to understand if we have a hidden bug somewhere.

Yes, havent dived into the details for a while but it probably works
because the pci_dev->devfn argument is often -1 (don't care) and then
the code will just search for an available function number.

> Anyway,
> Reviewed-by: Marcel Apfelbaum <marcel@redhat.com>

Thanks,

Knut

> > Knut
> > 
> > > Thanks,
> > > Marcel
> > > 
> > > 
> > > 
> > > >        if (pci_dev == NULL)
> > > >            return;
> > > > 
> > > > 
> > > 
> 
> 

[Qemu-devel] [PATCH v4 2/3] pci: Update pci_regs header

[Knut Omang]

Merge in new definitions from kernel v4.2
Adds definition necessary to support emulated SR/IOV.

Signed-off-by: Knut Omang <knut.omang@oracle.com>
---
 include/standard-headers/linux/pci_regs.h | 142 +++++++++++++++++++++++++-----
 1 file changed, 118 insertions(+), 24 deletions(-)

diff --git a/include/standard-headers/linux/pci_regs.h b/include/standard-headers/linux/pci_regs.h
index 57e8c80..3d56c1e 100644
--- a/include/standard-headers/linux/pci_regs.h
+++ b/include/standard-headers/linux/pci_regs.h
@@ -304,13 +304,14 @@
 #define PCI_MSI_PENDING_64	20	/* Pending bits register for 32-bit devices */
 
 /* MSI-X registers */
-#define PCI_MSIX_FLAGS		2
-#define  PCI_MSIX_FLAGS_QSIZE	0x7FF
-#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15)
-#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14)
-#define PCI_MSIX_TABLE		4
-#define PCI_MSIX_PBA		8
+#define PCI_MSIX_FLAGS		2	/* Message Control */
+#define  PCI_MSIX_FLAGS_QSIZE	0x7FF	/* Table size */
+#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15) /* MSI-X enable */
+#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14) /* Mask all vectors for this function */
+#define PCI_MSIX_TABLE		4	/* Table offset */
+#define PCI_MSIX_PBA		8	/* Pending Bit Array offset */
 #define  PCI_MSIX_FLAGS_BIRMASK	(7 << 0)
+#define PCI_CAP_MSIX_SIZEOF	12	/* size of MSIX capability structure */
 
 /* MSI-X entry's format */
 #define PCI_MSIX_ENTRY_SIZE		16
@@ -517,31 +518,63 @@
 #define  PCI_EXP_OBFF_MSG	0x40000 /* New message signaling */
 #define  PCI_EXP_OBFF_WAKE	0x80000 /* Re-use WAKE# for OBFF */
 #define PCI_EXP_DEVCTL2		40	/* Device Control 2 */
-#define  PCI_EXP_DEVCTL2_ARI	0x20	/* Alternative Routing-ID */
-#define  PCI_EXP_IDO_REQ_EN	0x100	/* ID-based ordering request enable */
-#define  PCI_EXP_IDO_CMP_EN	0x200	/* ID-based ordering completion enable */
-#define  PCI_EXP_LTR_EN		0x400	/* Latency tolerance reporting */
-#define  PCI_EXP_OBFF_MSGA_EN	0x2000	/* OBFF enable with Message type A */
-#define  PCI_EXP_OBFF_MSGB_EN	0x4000	/* OBFF enable with Message type B */
-#define  PCI_EXP_OBFF_WAKE_EN	0x6000	/* OBFF using WAKE# signaling */
+#define  PCI_EXP_DEVCTL2_COMP_TIMEOUT	0x000f	/* Completion Timeout Value */
+#define  PCI_EXP_DEVCTL2_ARI		0x0020	/* Alternative Routing-ID */
+#define  PCI_EXP_DEVCTL2_IDO_REQ_EN	0x0100	/* Allow IDO for requests */
+#define  PCI_EXP_DEVCTL2_IDO_CMP_EN	0x0200	/* Allow IDO for completions */
+#define  PCI_EXP_DEVCTL2_LTR_EN		0x0400	/* Enable LTR mechanism */
+#define  PCI_EXP_DEVCTL2_OBFF_MSGA_EN	0x2000	/* Enable OBFF Message type A */
+#define  PCI_EXP_DEVCTL2_OBFF_MSGB_EN	0x4000	/* Enable OBFF Message type B */
+#define  PCI_EXP_DEVCTL2_OBFF_WAKE_EN	0x6000	/* OBFF using WAKE# signaling */
+#define PCI_EXP_DEVSTA2		42	/* Device Status 2 */
+#define PCI_CAP_EXP_ENDPOINT_SIZEOF_V2	44	/* v2 endpoints end here */
+#define PCI_EXP_LNKCAP2		44	/* Link Capabilities 2 */
+#define  PCI_EXP_LNKCAP2_SLS_2_5GB	0x00000002 /* Supported Speed 2.5GT/s */
+#define  PCI_EXP_LNKCAP2_SLS_5_0GB	0x00000004 /* Supported Speed 5.0GT/s */
+#define  PCI_EXP_LNKCAP2_SLS_8_0GB	0x00000008 /* Supported Speed 8.0GT/s */
+#define  PCI_EXP_LNKCAP2_CROSSLINK	0x00000100 /* Crosslink supported */
 #define PCI_EXP_LNKCTL2		48	/* Link Control 2 */
-#define PCI_EXP_SLTCTL2		56	/* Slot Control 2 */
+#define PCI_EXP_LNKSTA2		50	/* Link Status 2 */
+#define PCI_EXP_SLTCAP2		52	/* Slot Capabilities 2 */
+#define PCI_EXP_SLTCTL2 	56	/* Slot Control 2 */
+#define PCI_EXP_SLTSTA2		58	/* Slot Status 2 */
 
 /* Extended Capabilities (PCI-X 2.0 and Express) */
 #define PCI_EXT_CAP_ID(header)		(header & 0x0000ffff)
 #define PCI_EXT_CAP_VER(header)		((header >> 16) & 0xf)
 #define PCI_EXT_CAP_NEXT(header)	((header >> 20) & 0xffc)
 
-#define PCI_EXT_CAP_ID_ERR	1
-#define PCI_EXT_CAP_ID_VC	2
-#define PCI_EXT_CAP_ID_DSN	3
-#define PCI_EXT_CAP_ID_PWR	4
-#define PCI_EXT_CAP_ID_VNDR	11
-#define PCI_EXT_CAP_ID_ACS	13
-#define PCI_EXT_CAP_ID_ARI	14
-#define PCI_EXT_CAP_ID_ATS	15
-#define PCI_EXT_CAP_ID_SRIOV	16
-#define PCI_EXT_CAP_ID_LTR	24
+#define PCI_EXT_CAP_ID_ERR	0x01	/* Advanced Error Reporting */
+#define PCI_EXT_CAP_ID_VC	0x02	/* Virtual Channel Capability */
+#define PCI_EXT_CAP_ID_DSN	0x03	/* Device Serial Number */
+#define PCI_EXT_CAP_ID_PWR	0x04	/* Power Budgeting */
+#define PCI_EXT_CAP_ID_RCLD	0x05	/* Root Complex Link Declaration */
+#define PCI_EXT_CAP_ID_RCILC	0x06	/* Root Complex Internal Link Control */
+#define PCI_EXT_CAP_ID_RCEC	0x07	/* Root Complex Event Collector */
+#define PCI_EXT_CAP_ID_MFVC	0x08	/* Multi-Function VC Capability */
+#define PCI_EXT_CAP_ID_VC9	0x09	/* same as _VC */
+#define PCI_EXT_CAP_ID_RCRB	0x0A	/* Root Complex RB? */
+#define PCI_EXT_CAP_ID_VNDR	0x0B	/* Vendor-Specific */
+#define PCI_EXT_CAP_ID_CAC	0x0C	/* Config Access - obsolete */
+#define PCI_EXT_CAP_ID_ACS	0x0D	/* Access Control Services */
+#define PCI_EXT_CAP_ID_ARI	0x0E	/* Alternate Routing ID */
+#define PCI_EXT_CAP_ID_ATS	0x0F	/* Address Translation Services */
+#define PCI_EXT_CAP_ID_SRIOV	0x10	/* Single Root I/O Virtualization */
+#define PCI_EXT_CAP_ID_MRIOV	0x11	/* Multi Root I/O Virtualization */
+#define PCI_EXT_CAP_ID_MCAST	0x12	/* Multicast */
+#define PCI_EXT_CAP_ID_PRI	0x13	/* Page Request Interface */
+#define PCI_EXT_CAP_ID_AMD_XXX	0x14	/* Reserved for AMD */
+#define PCI_EXT_CAP_ID_REBAR	0x15	/* Resizable BAR */
+#define PCI_EXT_CAP_ID_DPA	0x16	/* Dynamic Power Allocation */
+#define PCI_EXT_CAP_ID_TPH	0x17	/* TPH Requester */
+#define PCI_EXT_CAP_ID_LTR	0x18	/* Latency Tolerance Reporting */
+#define PCI_EXT_CAP_ID_SECPCI	0x19	/* Secondary PCIe Capability */
+#define PCI_EXT_CAP_ID_PMUX	0x1A	/* Protocol Multiplexing */
+#define PCI_EXT_CAP_ID_PASID	0x1B	/* Process Address Space ID */
+#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PASID
+
+#define PCI_EXT_CAP_DSN_SIZEOF	12
+#define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
 
 /* Advanced Error Reporting */
 #define PCI_ERR_UNCOR_STATUS	4	/* Uncorrectable Error Status */
@@ -556,6 +589,11 @@
 #define  PCI_ERR_UNC_MALF_TLP	0x00040000	/* Malformed TLP */
 #define  PCI_ERR_UNC_ECRC	0x00080000	/* ECRC Error Status */
 #define  PCI_ERR_UNC_UNSUP	0x00100000	/* Unsupported Request */
+#define  PCI_ERR_UNC_ACSV	0x00200000	/* ACS Violation */
+#define  PCI_ERR_UNC_INTN	0x00400000	/* internal error */
+#define  PCI_ERR_UNC_MCBTLP	0x00800000	/* MC blocked TLP */
+#define  PCI_ERR_UNC_ATOMEG	0x01000000	/* Atomic egress blocked */
+#define  PCI_ERR_UNC_TLPPRE	0x02000000	/* TLP prefix blocked */
 #define PCI_ERR_UNCOR_MASK	8	/* Uncorrectable Error Mask */
 	/* Same bits as above */
 #define PCI_ERR_UNCOR_SEVER	12	/* Uncorrectable Error Severity */
@@ -657,6 +695,7 @@
 #define  PCI_ARI_CTRL_MFVC	0x0001	/* MFVC Function Groups Enable */
 #define  PCI_ARI_CTRL_ACS	0x0002	/* ACS Function Groups Enable */
 #define  PCI_ARI_CTRL_FG(x)	(((x) >> 4) & 7) /* Function Group */
+#define PCI_EXT_CAP_ARI_SIZEOF	8
 
 /* Address Translation Service */
 #define PCI_ATS_CAP		0x04	/* ATS Capability Register */
@@ -666,6 +705,29 @@
 #define  PCI_ATS_CTRL_ENABLE	0x8000	/* ATS Enable */
 #define  PCI_ATS_CTRL_STU(x)	((x) & 0x1f)	/* Smallest Translation Unit */
 #define  PCI_ATS_MIN_STU	12	/* shift of minimum STU block */
+#define PCI_EXT_CAP_ATS_SIZEOF	8
+
+/* Page Request Interface */
+#define PCI_PRI_CTRL		0x04	/* PRI control register */
+#define  PCI_PRI_CTRL_ENABLE	0x01	/* Enable */
+#define  PCI_PRI_CTRL_RESET	0x02	/* Reset */
+#define PCI_PRI_STATUS		0x06	/* PRI status register */
+#define  PCI_PRI_STATUS_RF	0x001	/* Response Failure */
+#define  PCI_PRI_STATUS_UPRGI	0x002	/* Unexpected PRG index */
+#define  PCI_PRI_STATUS_STOPPED	0x100	/* PRI Stopped */
+#define PCI_PRI_MAX_REQ		0x08	/* PRI max reqs supported */
+#define PCI_PRI_ALLOC_REQ	0x0c	/* PRI max reqs allowed */
+#define PCI_EXT_CAP_PRI_SIZEOF	16
+
+/* Process Address Space ID */
+#define PCI_PASID_CAP		0x04    /* PASID feature register */
+#define  PCI_PASID_CAP_EXEC	0x02	/* Exec permissions Supported */
+#define  PCI_PASID_CAP_PRIV	0x04	/* Privilege Mode Supported */
+#define PCI_PASID_CTRL		0x06    /* PASID control register */
+#define  PCI_PASID_CTRL_ENABLE	0x01	/* Enable bit */
+#define  PCI_PASID_CTRL_EXEC	0x02	/* Exec permissions Enable */
+#define  PCI_PASID_CTRL_PRIV	0x04	/* Privilege Mode Enable */
+#define PCI_EXT_CAP_PASID_SIZEOF	8
 
 /* Single Root I/O Virtualization */
 #define PCI_SRIOV_CAP		0x04	/* SR-IOV Capabilities */
@@ -697,6 +759,7 @@
 #define  PCI_SRIOV_VFM_MI	0x1	/* Dormant.MigrateIn */
 #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
 #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
+#define PCI_EXT_CAP_SRIOV_SIZEOF 64
 
 #define PCI_LTR_MAX_SNOOP_LAT	0x4
 #define PCI_LTR_MAX_NOSNOOP_LAT	0x6
@@ -713,7 +776,38 @@
 #define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
 #define  PCI_ACS_EC		0x20	/* P2P Egress Control */
 #define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_EGRESS_BITS	0x05	/* ACS Egress Control Vector Size */
 #define PCI_ACS_CTRL		0x06	/* ACS Control Register */
 #define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
 
+#define PCI_VSEC_HDR		4	/* extended cap - vendor-specific */
+#define  PCI_VSEC_HDR_LEN_SHIFT	20	/* shift for length field */
+
+/* SATA capability */
+#define PCI_SATA_REGS		4	/* SATA REGs specifier */
+#define  PCI_SATA_REGS_MASK	0xF	/* location - BAR#/inline */
+#define  PCI_SATA_REGS_INLINE	0xF	/* REGS in config space */
+#define PCI_SATA_SIZEOF_SHORT	8
+#define PCI_SATA_SIZEOF_LONG	16
+
+/* Resizable BARs */
+#define PCI_REBAR_CTRL		8	/* control register */
+#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # bars */
+#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # bars */
+
+/* Dynamic Power Allocation */
+#define PCI_DPA_CAP		4	/* capability register */
+#define  PCI_DPA_CAP_SUBSTATE_MASK	0x1F	/* # substates - 1 */
+#define PCI_DPA_BASE_SIZEOF	16	/* size with 0 substates */
+
+/* TPH Requester */
+#define PCI_TPH_CAP		4	/* capability register */
+#define  PCI_TPH_CAP_LOC_MASK	0x600	/* location mask */
+#define   PCI_TPH_LOC_NONE	0x000	/* no location */
+#define   PCI_TPH_LOC_CAP	0x200	/* in capability */
+#define   PCI_TPH_LOC_MSIX	0x400	/* in MSI-X */
+#define PCI_TPH_CAP_ST_MASK	0x07FF0000	/* st table mask */
+#define PCI_TPH_CAP_ST_SHIFT	16	/* st table shift */
+#define PCI_TPH_BASE_SIZEOF	12	/* size with no st table */
+
 #endif /* LINUX_PCI_REGS_H */
-- 
2.4.3

Re: [Qemu-devel] [PATCH v4 2/3] pci: Update pci_regs header

[Marcel Apfelbaum]

On 09/12/2015 03:36 PM, Knut Omang wrote:
> Merge in new definitions from kernel v4.2
> Adds definition necessary to support emulated SR/IOV.

In the meantime Paolo updated this header to 4.3-RC1, is this patch still needed?

Thanks,
Marcel

>
> Signed-off-by: Knut Omang <knut.omang@oracle.com>
> ---
>   include/standard-headers/linux/pci_regs.h | 142 +++++++++++++++++++++++++-----
>   1 file changed, 118 insertions(+), 24 deletions(-)
>
> diff --git a/include/standard-headers/linux/pci_regs.h b/include/standard-headers/linux/pci_regs.h
> index 57e8c80..3d56c1e 100644
> --- a/include/standard-headers/linux/pci_regs.h
> +++ b/include/standard-headers/linux/pci_regs.h
> @@ -304,13 +304,14 @@
>   #define PCI_MSI_PENDING_64	20	/* Pending bits register for 32-bit devices */
>
>   /* MSI-X registers */
> -#define PCI_MSIX_FLAGS		2
> -#define  PCI_MSIX_FLAGS_QSIZE	0x7FF
> -#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15)
> -#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14)
> -#define PCI_MSIX_TABLE		4
> -#define PCI_MSIX_PBA		8
> +#define PCI_MSIX_FLAGS		2	/* Message Control */
> +#define  PCI_MSIX_FLAGS_QSIZE	0x7FF	/* Table size */
> +#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15) /* MSI-X enable */
> +#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14) /* Mask all vectors for this function */
> +#define PCI_MSIX_TABLE		4	/* Table offset */
> +#define PCI_MSIX_PBA		8	/* Pending Bit Array offset */
>   #define  PCI_MSIX_FLAGS_BIRMASK	(7 << 0)
> +#define PCI_CAP_MSIX_SIZEOF	12	/* size of MSIX capability structure */
>
>   /* MSI-X entry's format */
>   #define PCI_MSIX_ENTRY_SIZE		16
> @@ -517,31 +518,63 @@
>   #define  PCI_EXP_OBFF_MSG	0x40000 /* New message signaling */
>   #define  PCI_EXP_OBFF_WAKE	0x80000 /* Re-use WAKE# for OBFF */
>   #define PCI_EXP_DEVCTL2		40	/* Device Control 2 */
> -#define  PCI_EXP_DEVCTL2_ARI	0x20	/* Alternative Routing-ID */
> -#define  PCI_EXP_IDO_REQ_EN	0x100	/* ID-based ordering request enable */
> -#define  PCI_EXP_IDO_CMP_EN	0x200	/* ID-based ordering completion enable */
> -#define  PCI_EXP_LTR_EN		0x400	/* Latency tolerance reporting */
> -#define  PCI_EXP_OBFF_MSGA_EN	0x2000	/* OBFF enable with Message type A */
> -#define  PCI_EXP_OBFF_MSGB_EN	0x4000	/* OBFF enable with Message type B */
> -#define  PCI_EXP_OBFF_WAKE_EN	0x6000	/* OBFF using WAKE# signaling */
> +#define  PCI_EXP_DEVCTL2_COMP_TIMEOUT	0x000f	/* Completion Timeout Value */
> +#define  PCI_EXP_DEVCTL2_ARI		0x0020	/* Alternative Routing-ID */
> +#define  PCI_EXP_DEVCTL2_IDO_REQ_EN	0x0100	/* Allow IDO for requests */
> +#define  PCI_EXP_DEVCTL2_IDO_CMP_EN	0x0200	/* Allow IDO for completions */
> +#define  PCI_EXP_DEVCTL2_LTR_EN		0x0400	/* Enable LTR mechanism */
> +#define  PCI_EXP_DEVCTL2_OBFF_MSGA_EN	0x2000	/* Enable OBFF Message type A */
> +#define  PCI_EXP_DEVCTL2_OBFF_MSGB_EN	0x4000	/* Enable OBFF Message type B */
> +#define  PCI_EXP_DEVCTL2_OBFF_WAKE_EN	0x6000	/* OBFF using WAKE# signaling */
> +#define PCI_EXP_DEVSTA2		42	/* Device Status 2 */
> +#define PCI_CAP_EXP_ENDPOINT_SIZEOF_V2	44	/* v2 endpoints end here */
> +#define PCI_EXP_LNKCAP2		44	/* Link Capabilities 2 */
> +#define  PCI_EXP_LNKCAP2_SLS_2_5GB	0x00000002 /* Supported Speed 2.5GT/s */
> +#define  PCI_EXP_LNKCAP2_SLS_5_0GB	0x00000004 /* Supported Speed 5.0GT/s */
> +#define  PCI_EXP_LNKCAP2_SLS_8_0GB	0x00000008 /* Supported Speed 8.0GT/s */
> +#define  PCI_EXP_LNKCAP2_CROSSLINK	0x00000100 /* Crosslink supported */
>   #define PCI_EXP_LNKCTL2		48	/* Link Control 2 */
> -#define PCI_EXP_SLTCTL2		56	/* Slot Control 2 */
> +#define PCI_EXP_LNKSTA2		50	/* Link Status 2 */
> +#define PCI_EXP_SLTCAP2		52	/* Slot Capabilities 2 */
> +#define PCI_EXP_SLTCTL2 	56	/* Slot Control 2 */
> +#define PCI_EXP_SLTSTA2		58	/* Slot Status 2 */
>
>   /* Extended Capabilities (PCI-X 2.0 and Express) */
>   #define PCI_EXT_CAP_ID(header)		(header & 0x0000ffff)
>   #define PCI_EXT_CAP_VER(header)		((header >> 16) & 0xf)
>   #define PCI_EXT_CAP_NEXT(header)	((header >> 20) & 0xffc)
>
> -#define PCI_EXT_CAP_ID_ERR	1
> -#define PCI_EXT_CAP_ID_VC	2
> -#define PCI_EXT_CAP_ID_DSN	3
> -#define PCI_EXT_CAP_ID_PWR	4
> -#define PCI_EXT_CAP_ID_VNDR	11
> -#define PCI_EXT_CAP_ID_ACS	13
> -#define PCI_EXT_CAP_ID_ARI	14
> -#define PCI_EXT_CAP_ID_ATS	15
> -#define PCI_EXT_CAP_ID_SRIOV	16
> -#define PCI_EXT_CAP_ID_LTR	24
> +#define PCI_EXT_CAP_ID_ERR	0x01	/* Advanced Error Reporting */
> +#define PCI_EXT_CAP_ID_VC	0x02	/* Virtual Channel Capability */
> +#define PCI_EXT_CAP_ID_DSN	0x03	/* Device Serial Number */
> +#define PCI_EXT_CAP_ID_PWR	0x04	/* Power Budgeting */
> +#define PCI_EXT_CAP_ID_RCLD	0x05	/* Root Complex Link Declaration */
> +#define PCI_EXT_CAP_ID_RCILC	0x06	/* Root Complex Internal Link Control */
> +#define PCI_EXT_CAP_ID_RCEC	0x07	/* Root Complex Event Collector */
> +#define PCI_EXT_CAP_ID_MFVC	0x08	/* Multi-Function VC Capability */
> +#define PCI_EXT_CAP_ID_VC9	0x09	/* same as _VC */
> +#define PCI_EXT_CAP_ID_RCRB	0x0A	/* Root Complex RB? */
> +#define PCI_EXT_CAP_ID_VNDR	0x0B	/* Vendor-Specific */
> +#define PCI_EXT_CAP_ID_CAC	0x0C	/* Config Access - obsolete */
> +#define PCI_EXT_CAP_ID_ACS	0x0D	/* Access Control Services */
> +#define PCI_EXT_CAP_ID_ARI	0x0E	/* Alternate Routing ID */
> +#define PCI_EXT_CAP_ID_ATS	0x0F	/* Address Translation Services */
> +#define PCI_EXT_CAP_ID_SRIOV	0x10	/* Single Root I/O Virtualization */
> +#define PCI_EXT_CAP_ID_MRIOV	0x11	/* Multi Root I/O Virtualization */
> +#define PCI_EXT_CAP_ID_MCAST	0x12	/* Multicast */
> +#define PCI_EXT_CAP_ID_PRI	0x13	/* Page Request Interface */
> +#define PCI_EXT_CAP_ID_AMD_XXX	0x14	/* Reserved for AMD */
> +#define PCI_EXT_CAP_ID_REBAR	0x15	/* Resizable BAR */
> +#define PCI_EXT_CAP_ID_DPA	0x16	/* Dynamic Power Allocation */
> +#define PCI_EXT_CAP_ID_TPH	0x17	/* TPH Requester */
> +#define PCI_EXT_CAP_ID_LTR	0x18	/* Latency Tolerance Reporting */
> +#define PCI_EXT_CAP_ID_SECPCI	0x19	/* Secondary PCIe Capability */
> +#define PCI_EXT_CAP_ID_PMUX	0x1A	/* Protocol Multiplexing */
> +#define PCI_EXT_CAP_ID_PASID	0x1B	/* Process Address Space ID */
> +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PASID
> +
> +#define PCI_EXT_CAP_DSN_SIZEOF	12
> +#define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
>
>   /* Advanced Error Reporting */
>   #define PCI_ERR_UNCOR_STATUS	4	/* Uncorrectable Error Status */
> @@ -556,6 +589,11 @@
>   #define  PCI_ERR_UNC_MALF_TLP	0x00040000	/* Malformed TLP */
>   #define  PCI_ERR_UNC_ECRC	0x00080000	/* ECRC Error Status */
>   #define  PCI_ERR_UNC_UNSUP	0x00100000	/* Unsupported Request */
> +#define  PCI_ERR_UNC_ACSV	0x00200000	/* ACS Violation */
> +#define  PCI_ERR_UNC_INTN	0x00400000	/* internal error */
> +#define  PCI_ERR_UNC_MCBTLP	0x00800000	/* MC blocked TLP */
> +#define  PCI_ERR_UNC_ATOMEG	0x01000000	/* Atomic egress blocked */
> +#define  PCI_ERR_UNC_TLPPRE	0x02000000	/* TLP prefix blocked */
>   #define PCI_ERR_UNCOR_MASK	8	/* Uncorrectable Error Mask */
>   	/* Same bits as above */
>   #define PCI_ERR_UNCOR_SEVER	12	/* Uncorrectable Error Severity */
> @@ -657,6 +695,7 @@
>   #define  PCI_ARI_CTRL_MFVC	0x0001	/* MFVC Function Groups Enable */
>   #define  PCI_ARI_CTRL_ACS	0x0002	/* ACS Function Groups Enable */
>   #define  PCI_ARI_CTRL_FG(x)	(((x) >> 4) & 7) /* Function Group */
> +#define PCI_EXT_CAP_ARI_SIZEOF	8
>
>   /* Address Translation Service */
>   #define PCI_ATS_CAP		0x04	/* ATS Capability Register */
> @@ -666,6 +705,29 @@
>   #define  PCI_ATS_CTRL_ENABLE	0x8000	/* ATS Enable */
>   #define  PCI_ATS_CTRL_STU(x)	((x) & 0x1f)	/* Smallest Translation Unit */
>   #define  PCI_ATS_MIN_STU	12	/* shift of minimum STU block */
> +#define PCI_EXT_CAP_ATS_SIZEOF	8
> +
> +/* Page Request Interface */
> +#define PCI_PRI_CTRL		0x04	/* PRI control register */
> +#define  PCI_PRI_CTRL_ENABLE	0x01	/* Enable */
> +#define  PCI_PRI_CTRL_RESET	0x02	/* Reset */
> +#define PCI_PRI_STATUS		0x06	/* PRI status register */
> +#define  PCI_PRI_STATUS_RF	0x001	/* Response Failure */
> +#define  PCI_PRI_STATUS_UPRGI	0x002	/* Unexpected PRG index */
> +#define  PCI_PRI_STATUS_STOPPED	0x100	/* PRI Stopped */
> +#define PCI_PRI_MAX_REQ		0x08	/* PRI max reqs supported */
> +#define PCI_PRI_ALLOC_REQ	0x0c	/* PRI max reqs allowed */
> +#define PCI_EXT_CAP_PRI_SIZEOF	16
> +
> +/* Process Address Space ID */
> +#define PCI_PASID_CAP		0x04    /* PASID feature register */
> +#define  PCI_PASID_CAP_EXEC	0x02	/* Exec permissions Supported */
> +#define  PCI_PASID_CAP_PRIV	0x04	/* Privilege Mode Supported */
> +#define PCI_PASID_CTRL		0x06    /* PASID control register */
> +#define  PCI_PASID_CTRL_ENABLE	0x01	/* Enable bit */
> +#define  PCI_PASID_CTRL_EXEC	0x02	/* Exec permissions Enable */
> +#define  PCI_PASID_CTRL_PRIV	0x04	/* Privilege Mode Enable */
> +#define PCI_EXT_CAP_PASID_SIZEOF	8
>
>   /* Single Root I/O Virtualization */
>   #define PCI_SRIOV_CAP		0x04	/* SR-IOV Capabilities */
> @@ -697,6 +759,7 @@
>   #define  PCI_SRIOV_VFM_MI	0x1	/* Dormant.MigrateIn */
>   #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut */
>   #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available */
> +#define PCI_EXT_CAP_SRIOV_SIZEOF 64
>
>   #define PCI_LTR_MAX_SNOOP_LAT	0x4
>   #define PCI_LTR_MAX_NOSNOOP_LAT	0x6
> @@ -713,7 +776,38 @@
>   #define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
>   #define  PCI_ACS_EC		0x20	/* P2P Egress Control */
>   #define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
> +#define PCI_ACS_EGRESS_BITS	0x05	/* ACS Egress Control Vector Size */
>   #define PCI_ACS_CTRL		0x06	/* ACS Control Register */
>   #define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
>
> +#define PCI_VSEC_HDR		4	/* extended cap - vendor-specific */
> +#define  PCI_VSEC_HDR_LEN_SHIFT	20	/* shift for length field */
> +
> +/* SATA capability */
> +#define PCI_SATA_REGS		4	/* SATA REGs specifier */
> +#define  PCI_SATA_REGS_MASK	0xF	/* location - BAR#/inline */
> +#define  PCI_SATA_REGS_INLINE	0xF	/* REGS in config space */
> +#define PCI_SATA_SIZEOF_SHORT	8
> +#define PCI_SATA_SIZEOF_LONG	16
> +
> +/* Resizable BARs */
> +#define PCI_REBAR_CTRL		8	/* control register */
> +#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # bars */
> +#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # bars */
> +
> +/* Dynamic Power Allocation */
> +#define PCI_DPA_CAP		4	/* capability register */
> +#define  PCI_DPA_CAP_SUBSTATE_MASK	0x1F	/* # substates - 1 */
> +#define PCI_DPA_BASE_SIZEOF	16	/* size with 0 substates */
> +
> +/* TPH Requester */
> +#define PCI_TPH_CAP		4	/* capability register */
> +#define  PCI_TPH_CAP_LOC_MASK	0x600	/* location mask */
> +#define   PCI_TPH_LOC_NONE	0x000	/* no location */
> +#define   PCI_TPH_LOC_CAP	0x200	/* in capability */
> +#define   PCI_TPH_LOC_MSIX	0x400	/* in MSI-X */
> +#define PCI_TPH_CAP_ST_MASK	0x07FF0000	/* st table mask */
> +#define PCI_TPH_CAP_ST_SHIFT	16	/* st table shift */
> +#define PCI_TPH_BASE_SIZEOF	12	/* size with no st table */
> +
>   #endif /* LINUX_PCI_REGS_H */
>

Re: [Qemu-devel] [PATCH v4 2/3] pci: Update pci_regs header

[Knut Omang]

On Wed, 2015-10-07 at 16:32 +0300, Marcel Apfelbaum wrote:
> On 09/12/2015 03:36 PM, Knut Omang wrote:
> > Merge in new definitions from kernel v4.2
> > Adds definition necessary to support emulated SR/IOV.
> 
> In the meantime Paolo updated this header to 4.3-RC1, is this patch
> still needed?
> 
> Thanks,
> Marcel

Quite right - it is gone after rebase.

Thanks,
Knut

> > 
> > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > ---
> >   include/standard-headers/linux/pci_regs.h | 142
> > +++++++++++++++++++++++++-----
> >   1 file changed, 118 insertions(+), 24 deletions(-)
> > 
> > diff --git a/include/standard-headers/linux/pci_regs.h
> > b/include/standard-headers/linux/pci_regs.h
> > index 57e8c80..3d56c1e 100644
> > --- a/include/standard-headers/linux/pci_regs.h
> > +++ b/include/standard-headers/linux/pci_regs.h
> > @@ -304,13 +304,14 @@
> >   #define PCI_MSI_PENDING_64	20	/* Pending bits
> > register for 32-bit devices */
> > 
> >   /* MSI-X registers */
> > -#define PCI_MSIX_FLAGS		2
> > -#define  PCI_MSIX_FLAGS_QSIZE	0x7FF
> > -#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15)
> > -#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14)
> > -#define PCI_MSIX_TABLE		4
> > -#define PCI_MSIX_PBA		8
> > +#define PCI_MSIX_FLAGS		2	/* Message Control
> > */
> > +#define  PCI_MSIX_FLAGS_QSIZE	0x7FF	/* Table size */
> > +#define  PCI_MSIX_FLAGS_ENABLE	(1 << 15) /* MSI-X enable */
> > +#define  PCI_MSIX_FLAGS_MASKALL	(1 << 14) /* Mask all
> > vectors for this function */
> > +#define PCI_MSIX_TABLE		4	/* Table offset */
> > +#define PCI_MSIX_PBA		8	/* Pending Bit Array
> > offset */
> >   #define  PCI_MSIX_FLAGS_BIRMASK	(7 << 0)
> > +#define PCI_CAP_MSIX_SIZEOF	12	/* size of MSIX
> > capability structure */
> > 
> >   /* MSI-X entry's format */
> >   #define PCI_MSIX_ENTRY_SIZE		16
> > @@ -517,31 +518,63 @@
> >   #define  PCI_EXP_OBFF_MSG	0x40000 /* New message signaling
> > */
> >   #define  PCI_EXP_OBFF_WAKE	0x80000 /* Re-use WAKE# for
> > OBFF */
> >   #define PCI_EXP_DEVCTL2		40	/* Device
> > Control 2 */
> > -#define  PCI_EXP_DEVCTL2_ARI	0x20	/* Alternative
> > Routing-ID */
> > -#define  PCI_EXP_IDO_REQ_EN	0x100	/* ID-based
> > ordering request enable */
> > -#define  PCI_EXP_IDO_CMP_EN	0x200	/* ID-based
> > ordering completion enable */
> > -#define  PCI_EXP_LTR_EN		0x400	/* Latency
> > tolerance reporting */
> > -#define  PCI_EXP_OBFF_MSGA_EN	0x2000	/* OBFF enable
> > with Message type A */
> > -#define  PCI_EXP_OBFF_MSGB_EN	0x4000	/* OBFF enable
> > with Message type B */
> > -#define  PCI_EXP_OBFF_WAKE_EN	0x6000	/* OBFF using
> > WAKE# signaling */
> > +#define  PCI_EXP_DEVCTL2_COMP_TIMEOUT	0x000f	/*
> > Completion Timeout Value */
> > +#define  PCI_EXP_DEVCTL2_ARI		0x0020	/*
> > Alternative Routing-ID */
> > +#define  PCI_EXP_DEVCTL2_IDO_REQ_EN	0x0100	/* Allow
> > IDO for requests */
> > +#define  PCI_EXP_DEVCTL2_IDO_CMP_EN	0x0200	/* Allow
> > IDO for completions */
> > +#define  PCI_EXP_DEVCTL2_LTR_EN		0x0400	/*
> > Enable LTR mechanism */
> > +#define  PCI_EXP_DEVCTL2_OBFF_MSGA_EN	0x2000	/*
> > Enable OBFF Message type A */
> > +#define  PCI_EXP_DEVCTL2_OBFF_MSGB_EN	0x4000	/*
> > Enable OBFF Message type B */
> > +#define  PCI_EXP_DEVCTL2_OBFF_WAKE_EN	0x6000	/* OBFF
> > using WAKE# signaling */
> > +#define PCI_EXP_DEVSTA2		42	/* Device Status
> > 2 */
> > +#define PCI_CAP_EXP_ENDPOINT_SIZEOF_V2	44	/* v2
> > endpoints end here */
> > +#define PCI_EXP_LNKCAP2		44	/* Link
> > Capabilities 2 */
> > +#define  PCI_EXP_LNKCAP2_SLS_2_5GB	0x00000002 /* Supported
> > Speed 2.5GT/s */
> > +#define  PCI_EXP_LNKCAP2_SLS_5_0GB	0x00000004 /* Supported
> > Speed 5.0GT/s */
> > +#define  PCI_EXP_LNKCAP2_SLS_8_0GB	0x00000008 /* Supported
> > Speed 8.0GT/s */
> > +#define  PCI_EXP_LNKCAP2_CROSSLINK	0x00000100 /* Crosslink
> > supported */
> >   #define PCI_EXP_LNKCTL2		48	/* Link Control
> > 2 */
> > -#define PCI_EXP_SLTCTL2		56	/* Slot Control 2
> > */
> > +#define PCI_EXP_LNKSTA2		50	/* Link Status 2
> > */
> > +#define PCI_EXP_SLTCAP2		52	/* Slot
> > Capabilities 2 */
> > +#define PCI_EXP_SLTCTL2 	56	/* Slot Control 2 */
> > +#define PCI_EXP_SLTSTA2		58	/* Slot Status 2
> > */
> > 
> >   /* Extended Capabilities (PCI-X 2.0 and Express) */
> >   #define PCI_EXT_CAP_ID(header)		(header &
> > 0x0000ffff)
> >   #define PCI_EXT_CAP_VER(header)		((header >> 16) &
> > 0xf)
> >   #define PCI_EXT_CAP_NEXT(header)	((header >> 20) & 0xffc)
> > 
> > -#define PCI_EXT_CAP_ID_ERR	1
> > -#define PCI_EXT_CAP_ID_VC	2
> > -#define PCI_EXT_CAP_ID_DSN	3
> > -#define PCI_EXT_CAP_ID_PWR	4
> > -#define PCI_EXT_CAP_ID_VNDR	11
> > -#define PCI_EXT_CAP_ID_ACS	13
> > -#define PCI_EXT_CAP_ID_ARI	14
> > -#define PCI_EXT_CAP_ID_ATS	15
> > -#define PCI_EXT_CAP_ID_SRIOV	16
> > -#define PCI_EXT_CAP_ID_LTR	24
> > +#define PCI_EXT_CAP_ID_ERR	0x01	/* Advanced Error
> > Reporting */
> > +#define PCI_EXT_CAP_ID_VC	0x02	/* Virtual Channel
> > Capability */
> > +#define PCI_EXT_CAP_ID_DSN	0x03	/* Device Serial
> > Number */
> > +#define PCI_EXT_CAP_ID_PWR	0x04	/* Power Budgeting
> > */
> > +#define PCI_EXT_CAP_ID_RCLD	0x05	/* Root Complex
> > Link Declaration */
> > +#define PCI_EXT_CAP_ID_RCILC	0x06	/* Root Complex
> > Internal Link Control */
> > +#define PCI_EXT_CAP_ID_RCEC	0x07	/* Root Complex
> > Event Collector */
> > +#define PCI_EXT_CAP_ID_MFVC	0x08	/* Multi-Function
> > VC Capability */
> > +#define PCI_EXT_CAP_ID_VC9	0x09	/* same as _VC */
> > +#define PCI_EXT_CAP_ID_RCRB	0x0A	/* Root Complex RB?
> > */
> > +#define PCI_EXT_CAP_ID_VNDR	0x0B	/* Vendor-Specific
> > */
> > +#define PCI_EXT_CAP_ID_CAC	0x0C	/* Config Access -
> > obsolete */
> > +#define PCI_EXT_CAP_ID_ACS	0x0D	/* Access Control
> > Services */
> > +#define PCI_EXT_CAP_ID_ARI	0x0E	/* Alternate Routing
> > ID */
> > +#define PCI_EXT_CAP_ID_ATS	0x0F	/* Address
> > Translation Services */
> > +#define PCI_EXT_CAP_ID_SRIOV	0x10	/* Single Root I/O
> > Virtualization */
> > +#define PCI_EXT_CAP_ID_MRIOV	0x11	/* Multi Root I/O
> > Virtualization */
> > +#define PCI_EXT_CAP_ID_MCAST	0x12	/* Multicast */
> > +#define PCI_EXT_CAP_ID_PRI	0x13	/* Page Request
> > Interface */
> > +#define PCI_EXT_CAP_ID_AMD_XXX	0x14	/* Reserved for
> > AMD */
> > +#define PCI_EXT_CAP_ID_REBAR	0x15	/* Resizable BAR
> > */
> > +#define PCI_EXT_CAP_ID_DPA	0x16	/* Dynamic Power
> > Allocation */
> > +#define PCI_EXT_CAP_ID_TPH	0x17	/* TPH Requester */
> > +#define PCI_EXT_CAP_ID_LTR	0x18	/* Latency Tolerance
> > Reporting */
> > +#define PCI_EXT_CAP_ID_SECPCI	0x19	/* Secondary PCIe
> > Capability */
> > +#define PCI_EXT_CAP_ID_PMUX	0x1A	/* Protocol
> > Multiplexing */
> > +#define PCI_EXT_CAP_ID_PASID	0x1B	/* Process Address
> > Space ID */
> > +#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PASID
> > +
> > +#define PCI_EXT_CAP_DSN_SIZEOF	12
> > +#define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
> > 
> >   /* Advanced Error Reporting */
> >   #define PCI_ERR_UNCOR_STATUS	4	/* Uncorrectable
> > Error Status */
> > @@ -556,6 +589,11 @@
> >   #define  PCI_ERR_UNC_MALF_TLP	0x00040000	/*
> > Malformed TLP */
> >   #define  PCI_ERR_UNC_ECRC	0x00080000	/* ECRC Error
> > Status */
> >   #define  PCI_ERR_UNC_UNSUP	0x00100000	/*
> > Unsupported Request */
> > +#define  PCI_ERR_UNC_ACSV	0x00200000	/* ACS
> > Violation */
> > +#define  PCI_ERR_UNC_INTN	0x00400000	/* internal
> > error */
> > +#define  PCI_ERR_UNC_MCBTLP	0x00800000	/* MC blocked
> > TLP */
> > +#define  PCI_ERR_UNC_ATOMEG	0x01000000	/* Atomic
> > egress blocked */
> > +#define  PCI_ERR_UNC_TLPPRE	0x02000000	/* TLP prefix
> > blocked */
> >   #define PCI_ERR_UNCOR_MASK	8	/* Uncorrectable Error
> > Mask */
> >   	/* Same bits as above */
> >   #define PCI_ERR_UNCOR_SEVER	12	/* Uncorrectable
> > Error Severity */
> > @@ -657,6 +695,7 @@
> >   #define  PCI_ARI_CTRL_MFVC	0x0001	/* MFVC Function
> > Groups Enable */
> >   #define  PCI_ARI_CTRL_ACS	0x0002	/* ACS Function
> > Groups Enable */
> >   #define  PCI_ARI_CTRL_FG(x)	(((x) >> 4) & 7) /* Function
> > Group */
> > +#define PCI_EXT_CAP_ARI_SIZEOF	8
> > 
> >   /* Address Translation Service */
> >   #define PCI_ATS_CAP		0x04	/* ATS Capability
> > Register */
> > @@ -666,6 +705,29 @@
> >   #define  PCI_ATS_CTRL_ENABLE	0x8000	/* ATS Enable
> > */
> >   #define  PCI_ATS_CTRL_STU(x)	((x) & 0x1f)	/*
> > Smallest Translation Unit */
> >   #define  PCI_ATS_MIN_STU	12	/* shift of minimum STU
> > block */
> > +#define PCI_EXT_CAP_ATS_SIZEOF	8
> > +
> > +/* Page Request Interface */
> > +#define PCI_PRI_CTRL		0x04	/* PRI control
> > register */
> > +#define  PCI_PRI_CTRL_ENABLE	0x01	/* Enable */
> > +#define  PCI_PRI_CTRL_RESET	0x02	/* Reset */
> > +#define PCI_PRI_STATUS		0x06	/* PRI status
> > register */
> > +#define  PCI_PRI_STATUS_RF	0x001	/* Response Failure
> > */
> > +#define  PCI_PRI_STATUS_UPRGI	0x002	/* Unexpected
> > PRG index */
> > +#define  PCI_PRI_STATUS_STOPPED	0x100	/* PRI Stopped
> > */
> > +#define PCI_PRI_MAX_REQ		0x08	/* PRI max reqs
> > supported */
> > +#define PCI_PRI_ALLOC_REQ	0x0c	/* PRI max reqs
> > allowed */
> > +#define PCI_EXT_CAP_PRI_SIZEOF	16
> > +
> > +/* Process Address Space ID */
> > +#define PCI_PASID_CAP		0x04    /* PASID feature
> > register */
> > +#define  PCI_PASID_CAP_EXEC	0x02	/* Exec permissions
> > Supported */
> > +#define  PCI_PASID_CAP_PRIV	0x04	/* Privilege Mode
> > Supported */
> > +#define PCI_PASID_CTRL		0x06    /* PASID control
> > register */
> > +#define  PCI_PASID_CTRL_ENABLE	0x01	/* Enable bit */
> > +#define  PCI_PASID_CTRL_EXEC	0x02	/* Exec
> > permissions Enable */
> > +#define  PCI_PASID_CTRL_PRIV	0x04	/* Privilege Mode
> > Enable */
> > +#define PCI_EXT_CAP_PASID_SIZEOF	8
> > 
> >   /* Single Root I/O Virtualization */
> >   #define PCI_SRIOV_CAP		0x04	/* SR-IOV
> > Capabilities */
> > @@ -697,6 +759,7 @@
> >   #define  PCI_SRIOV_VFM_MI	0x1	/* Dormant.MigrateIn
> > */
> >   #define  PCI_SRIOV_VFM_MO	0x2	/* Active.MigrateOut
> > */
> >   #define  PCI_SRIOV_VFM_AV	0x3	/* Active.Available
> > */
> > +#define PCI_EXT_CAP_SRIOV_SIZEOF 64
> > 
> >   #define PCI_LTR_MAX_SNOOP_LAT	0x4
> >   #define PCI_LTR_MAX_NOSNOOP_LAT	0x6
> > @@ -713,7 +776,38 @@
> >   #define  PCI_ACS_UF		0x10	/* Upstream
> > Forwarding */
> >   #define  PCI_ACS_EC		0x20	/* P2P Egress
> > Control */
> >   #define  PCI_ACS_DT		0x40	/* Direct
> > Translated P2P */
> > +#define PCI_ACS_EGRESS_BITS	0x05	/* ACS Egress
> > Control Vector Size */
> >   #define PCI_ACS_CTRL		0x06	/* ACS Control
> > Register */
> >   #define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress
> > Control Vector */
> > 
> > +#define PCI_VSEC_HDR		4	/* extended cap -
> > vendor-specific */
> > +#define  PCI_VSEC_HDR_LEN_SHIFT	20	/* shift for
> > length field */
> > +
> > +/* SATA capability */
> > +#define PCI_SATA_REGS		4	/* SATA REGs
> > specifier */
> > +#define  PCI_SATA_REGS_MASK	0xF	/* location -
> > BAR#/inline */
> > +#define  PCI_SATA_REGS_INLINE	0xF	/* REGS in config
> > space */
> > +#define PCI_SATA_SIZEOF_SHORT	8
> > +#define PCI_SATA_SIZEOF_LONG	16
> > +
> > +/* Resizable BARs */
> > +#define PCI_REBAR_CTRL		8	/* control register
> > */
> > +#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask
> > for # bars */
> > +#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for #
> > bars */
> > +
> > +/* Dynamic Power Allocation */
> > +#define PCI_DPA_CAP		4	/* capability register
> > */
> > +#define  PCI_DPA_CAP_SUBSTATE_MASK	0x1F	/* #
> > substates - 1 */
> > +#define PCI_DPA_BASE_SIZEOF	16	/* size with 0
> > substates */
> > +
> > +/* TPH Requester */
> > +#define PCI_TPH_CAP		4	/* capability register
> > */
> > +#define  PCI_TPH_CAP_LOC_MASK	0x600	/* location mask
> > */
> > +#define   PCI_TPH_LOC_NONE	0x000	/* no location */
> > +#define   PCI_TPH_LOC_CAP	0x200	/* in capability */
> > +#define   PCI_TPH_LOC_MSIX	0x400	/* in MSI-X */
> > +#define PCI_TPH_CAP_ST_MASK	0x07FF0000	/* st table
> > mask */
> > +#define PCI_TPH_CAP_ST_SHIFT	16	/* st table shift */
> > +#define PCI_TPH_BASE_SIZEOF	12	/* size with no st
> > table */
> > +
> >   #endif /* LINUX_PCI_REGS_H */
> > 
> 

[Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Knut Omang]

This patch provides the building blocks for creating an SR/IOV
PCIe Extended Capability header and register/unregister
SR/IOV Virtual Functions.

Signed-off-by: Knut Omang <knut.omang@oracle.com>
---
 hw/pci/Makefile.objs        |   2 +-
 hw/pci/pci.c                |  99 ++++++++++++----
 hw/pci/pcie.c               |   9 +-
 hw/pci/pcie_sriov.c         | 271 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/pci.h        |  11 +-
 include/hw/pci/pcie.h       |   6 +
 include/hw/pci/pcie_sriov.h |  55 +++++++++
 include/qemu/typedefs.h     |   2 +
 8 files changed, 426 insertions(+), 29 deletions(-)
 create mode 100644 hw/pci/pcie_sriov.c
 create mode 100644 include/hw/pci/pcie_sriov.h

diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
index 9f905e6..2226980 100644
--- a/hw/pci/Makefile.objs
+++ b/hw/pci/Makefile.objs
@@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
 common-obj-$(CONFIG_PCI) += shpc.o
 common-obj-$(CONFIG_PCI) += slotid_cap.o
 common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
-common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
+common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o pcie_sriov.o
 
 common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
 common-obj-$(CONFIG_ALL) += pci-stub.o
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index a5cc015..9c0eba1 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
 {
     uint8_t type;
 
+    /* PCIe virtual functions do not have their own BARs */
+    assert(!pci_is_vf(d));
+
     if (reg != PCI_ROM_SLOT)
         return PCI_BASE_ADDRESS_0 + reg * 4;
 
@@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
     }
 }
 
-static void pci_do_device_reset(PCIDevice *dev)
+static void pci_reset_regions(PCIDevice *dev)
 {
     int r;
+    if (pci_is_vf(dev)) {
+        return;
+    }
 
-    pci_device_deassert_intx(dev);
-    assert(dev->irq_state == 0);
-
-    /* Clear all writable bits */
-    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
-                                 pci_get_word(dev->wmask + PCI_COMMAND) |
-                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
-    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
-                                 pci_get_word(dev->wmask + PCI_STATUS) |
-                                 pci_get_word(dev->w1cmask + PCI_STATUS));
-    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
-    dev->config[PCI_INTERRUPT_LINE] = 0x0;
     for (r = 0; r < PCI_NUM_REGIONS; ++r) {
         PCIIORegion *region = &dev->io_regions[r];
         if (!region->size) {
@@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice *dev)
             pci_set_long(dev->config + pci_bar(dev, r), region->type);
         }
     }
+}
+
+static void pci_do_device_reset(PCIDevice *dev)
+{
+    qdev_reset_all(&dev->qdev);
+
+    dev->irq_state = 0;
+    pci_update_irq_status(dev);
+    pci_device_deassert_intx(dev);
+    assert(dev->irq_state == 0);
+
+    /* Clear all writable bits */
+    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
+                                 pci_get_word(dev->wmask + PCI_COMMAND) |
+                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
+    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
+                                 pci_get_word(dev->wmask + PCI_STATUS) |
+                                 pci_get_word(dev->w1cmask + PCI_STATUS));
+    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
+    dev->config[PCI_INTERRUPT_LINE] = 0x0;
+    pci_reset_regions(dev);
     pci_update_mappings(dev);
 
     msi_reset(dev);
@@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus *bus, PCIDevice *dev, Error **errp)
         dev->config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION;
     }
 
+    /* With SR/IOV and ARI, a device at function 0 need not be a multifunction
+     * device, as it may just be a VF that ended up with function 0 in
+     * the legacy PCI interpretation. Avoid failing in such cases:
+     */
+    if (pci_is_vf(dev) &&
+        dev->exp.sriov_vf.pf->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
+        return;
+    }
+
     /*
      * multifunction bit is interpreted in two ways as follows.
      *   - all functions must set the bit to 1.
@@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num,
     uint64_t wmask;
     pcibus_t size = memory_region_size(memory);
 
+    assert(!pci_is_vf(pci_dev)); /* VFs must use pcie_sriov_vf_register_bar */
     assert(region_num >= 0);
     assert(region_num < PCI_NUM_REGIONS);
     if (size & (size-1)) {
@@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num)
     return pci_dev->io_regions[region_num].addr;
 }
 
-static pcibus_t pci_bar_address(PCIDevice *d,
-				int reg, uint8_t type, pcibus_t size)
+
+static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
+                                        uint8_t type, pcibus_t size)
+{
+    pcibus_t new_addr;
+    if (!pci_is_vf(d)) {
+        int bar = pci_bar(d, reg);
+        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
+            new_addr = pci_get_quad(d->config + bar);
+        } else {
+            new_addr = pci_get_long(d->config + bar);
+        }
+    } else {
+        PCIDevice *pf = d->exp.sriov_vf.pf;
+        uint16_t sriov_cap = pf->exp.sriov_cap;
+        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
+        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
+        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
+        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) / vf_stride;
+
+        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
+            new_addr = pci_get_quad(pf->config + bar);
+        } else {
+            new_addr = pci_get_long(pf->config + bar);
+        }
+        new_addr += vf_num * size;
+    }
+    if (reg != PCI_ROM_SLOT) {
+        /* Preserve the rom enable bit */
+        new_addr &= ~(size - 1);
+    }
+    return new_addr;
+}
+
+pcibus_t pci_bar_address(PCIDevice *d,
+                         int reg, uint8_t type, pcibus_t size)
 {
     pcibus_t new_addr, last_addr;
-    int bar = pci_bar(d, reg);
     uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
     Object *machine = qdev_get_machine();
     ObjectClass *oc = object_get_class(machine);
@@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
         if (!(cmd & PCI_COMMAND_IO)) {
             return PCI_BAR_UNMAPPED;
         }
-        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
+        new_addr = pci_config_get_bar_addr(d, reg, type, size);
         last_addr = new_addr + size - 1;
         /* Check if 32 bit BAR wraps around explicitly.
          * TODO: make priorities correct and remove this work around.
@@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
     if (!(cmd & PCI_COMMAND_MEMORY)) {
         return PCI_BAR_UNMAPPED;
     }
-    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
-        new_addr = pci_get_quad(d->config + bar);
-    } else {
-        new_addr = pci_get_long(d->config + bar);
-    }
+    new_addr = pci_config_get_bar_addr(d, reg, type, size);
     /* the ROM slot has a specific enable bit */
     if (reg == PCI_ROM_SLOT && !(new_addr & PCI_ROM_ADDRESS_ENABLE)) {
         return PCI_BAR_UNMAPPED;
@@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val_in, int
 
     msi_write_config(d, addr, val_in, l);
     msix_write_config(d, addr, val_in, l);
+    pcie_sriov_config_write(d, addr, val_in, l);
 }
 
 /***********************************************************/
diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
index 6e28985..ba49c0f 100644
--- a/hw/pci/pcie.c
+++ b/hw/pci/pcie.c
@@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
      * Right now, only a device of function = 0 is allowed to be
      * hot plugged/unplugged.
      */
-    assert(PCI_FUNC(pci_dev->devfn) == 0);
+    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
 
     pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
                                PCI_EXP_SLTSTA_PDS);
@@ -265,10 +265,11 @@ void pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
                                          DeviceState *dev, Error **errp)
 {
     uint8_t *exp_cap;
+    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
 
-    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev, &exp_cap, errp);
+    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
 
-    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
+    pcie_cap_slot_push_attention_button(pdev);
 }
 
 /* pci express slot for pci express root/downstream port
@@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
     }
 
     /*
-     * If the slot is polulated, power indicator is off and power
+     * If the slot is populated, power indicator is off and power
      * controller is off, it is safe to detach the devices.
      */
     if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val & PCI_EXP_SLTCTL_PCC) &&
diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
new file mode 100644
index 0000000..14f31cc
--- /dev/null
+++ b/hw/pci/pcie_sriov.c
@@ -0,0 +1,271 @@
+/*
+ * pcie_sriov.c:
+ *
+ * Implementation of SR/IOV emulation support.
+ *
+ * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pci_bus.h"
+#include "qemu/range.h"
+
+//#define DEBUG_SRIOV
+#ifdef DEBUG_SRIOV
+# define SRIOV_DPRINTF(fmt, ...)                                         \
+    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ## __VA_ARGS__)
+#else
+# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
+#endif
+#define SRIOV_DEV_PRINTF(dev, fmt, ...)                                  \
+    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ## __VA_ARGS__)
+
+static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name);
+static void unregister_vfs(PCIDevice *dev);
+
+void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
+                        const char *vfname, uint16_t vf_dev_id,
+                        uint16_t init_vfs, uint16_t total_vfs,
+                        uint16_t vf_offset, uint16_t vf_stride)
+{
+    uint8_t *cfg = dev->config + offset;
+    uint8_t *wmask;
+
+    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
+                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
+    dev->exp.sriov_cap = offset;
+    dev->exp.sriov_pf.num_vfs = 0;
+    dev->exp.sriov_pf.vfname = g_strdup(vfname);
+    dev->exp.sriov_pf.vf = NULL;
+
+    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
+    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
+
+    /* Minimal page size support values required by the SR/IOV standard:
+     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
+     * Hard coded for simplicity in this version
+     */
+    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
+
+    /* Default is to use 4K pages, software can modify it
+     * to any of the supported bits
+     */
+    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
+
+    /* Set up device ID and initial/total number of VFs available */
+    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
+    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
+    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
+    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
+
+    /* Write enable control bits */
+    wmask = dev->wmask + offset;
+    pci_set_word(wmask + PCI_SRIOV_CTRL,
+                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE | PCI_SRIOV_CTRL_ARI);
+    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
+    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
+
+    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
+}
+
+void pcie_sriov_pf_exit(PCIDevice *dev)
+{
+    SRIOV_DPRINTF("\n");
+    unregister_vfs(dev);
+    g_free((char *)dev->exp.sriov_pf.vfname);
+    dev->exp.sriov_pf.vfname = NULL;
+}
+
+void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
+                               uint8_t type, dma_addr_t size)
+{
+    uint32_t addr;
+    uint64_t wmask;
+    uint16_t sriov_cap = dev->exp.sriov_cap;
+
+    assert(sriov_cap > 0);
+    assert(region_num >= 0);
+    assert(region_num < PCI_NUM_REGIONS);
+    assert(region_num != PCI_ROM_SLOT);
+
+    wmask = ~(size - 1);
+    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
+
+    pci_set_long(dev->config + addr, type);
+    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
+        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
+        pci_set_quad(dev->wmask + addr, wmask);
+        pci_set_quad(dev->cmask + addr, ~0ULL);
+    } else {
+        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
+        pci_set_long(dev->cmask + addr, 0xffffffff);
+    }
+    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
+}
+
+void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
+                                MemoryRegion *memory)
+{
+    PCIIORegion *r;
+    uint8_t type;
+    pcibus_t size = memory_region_size(memory);
+
+    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
+    assert(region_num >= 0);
+    assert(region_num < PCI_NUM_REGIONS);
+    type = dev->exp.sriov_vf.pf->exp.sriov_pf.vf_bar_type[region_num];
+
+    if (size & (size-1)) {
+        fprintf(stderr, "ERROR: PCI region size must be a power"
+                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n", type, size);
+        exit(1);
+    }
+
+    r = &dev->io_regions[region_num];
+    r->memory = memory;
+    r->address_space =
+        type & PCI_BASE_ADDRESS_SPACE_IO
+        ? dev->bus->address_space_io
+        : dev->bus->address_space_mem;
+    r->size = size;
+    r->type = type;
+
+    r->addr = pci_bar_address(dev, region_num, r->type, r->size);
+    if (r->addr != PCI_BAR_UNMAPPED) {
+        memory_region_add_subregion_overlap(r->address_space,
+                                            r->addr, r->memory, 1);
+    }
+}
+
+static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name)
+{
+    PCIDevice *dev = pci_create(pf->bus, devfn, name);
+    dev->exp.sriov_vf.pf = pf;
+    Error *local_err = NULL;
+
+    object_property_set_bool(OBJECT(&dev->qdev), true, "realized", &local_err);
+    if (local_err) {
+        fprintf(stderr, "Failed to enable: %s\n",
+                error_get_pretty(local_err));
+        error_free(local_err);
+        return NULL;
+    }
+
+    /* set vid/did according to sr/iov spec - they are not used */
+    pci_config_set_vendor_id(dev->config, 0xffff);
+    pci_config_set_device_id(dev->config, 0xffff);
+    return dev;
+}
+
+static void register_vfs(PCIDevice *dev)
+{
+    uint16_t num_vfs;
+    uint16_t i;
+    uint16_t sriov_cap = dev->exp.sriov_cap;
+    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
+    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
+    int32_t devfn = dev->devfn + vf_offset;
+
+    assert(sriov_cap > 0);
+    num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF);
+
+    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) * num_vfs);
+    assert(dev->exp.sriov_pf.vf);
+
+    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
+    for (i = 0; i < num_vfs; i++) {
+        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev->exp.sriov_pf.vfname);
+        if (!dev->exp.sriov_pf.vf[i]) {
+            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at devfn %x.%x\n", i,
+                             PCI_SLOT(devfn), PCI_FUNC(devfn));
+            num_vfs = i;
+            break;
+        }
+        devfn += vf_stride;
+    }
+    dev->exp.sriov_pf.num_vfs = num_vfs;
+}
+
+static void unregister_vfs(PCIDevice *dev)
+{
+    Error *local_err = NULL;
+    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
+    uint16_t i;
+    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n", num_vfs);
+    for (i = 0; i < num_vfs; i++) {
+        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]->qdev), false, "realized", &local_err);
+        if (local_err) {
+            fprintf(stderr, "Failed to unplug: %s\n",
+                    error_get_pretty(local_err));
+            error_free(local_err);
+        }
+    }
+    g_free(dev->exp.sriov_pf.vf);
+    dev->exp.sriov_pf.vf = NULL;
+    dev->exp.sriov_pf.num_vfs = 0;
+}
+
+void pcie_sriov_config_write(PCIDevice *dev, uint32_t address, uint32_t val, int len)
+{
+    uint32_t off;
+    uint16_t sriov_cap = dev->exp.sriov_cap;
+
+    if (!sriov_cap || address < sriov_cap) {
+        return;
+    }
+    off = address - sriov_cap;
+    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
+        return;
+    }
+
+    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val 0x%x len %d\n",
+                 sriov_cap, off, val, len);
+
+    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
+        if (dev->exp.sriov_pf.num_vfs) {
+            if (!(val & PCI_SRIOV_CTRL_VFE)) {
+                unregister_vfs(dev);
+            }
+        } else {
+            if (val & PCI_SRIOV_CTRL_VFE) {
+                register_vfs(dev);
+            }
+        }
+    }
+}
+
+
+/* Reset SR/IOV VF Enable bit to trigger an unregister of all VFs */
+void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
+{
+    uint16_t sriov_cap = dev->exp.sriov_cap;
+    if (sriov_cap) {
+        uint32_t val = pci_get_byte(dev->config + sriov_cap + PCI_SRIOV_CTRL);
+        if (val & PCI_SRIOV_CTRL_VFE) {
+            val &= ~PCI_SRIOV_CTRL_VFE;
+            pcie_sriov_config_write(dev, sriov_cap + PCI_SRIOV_CTRL, val, 1);
+        }
+    }
+}
+
+/* Add optional supported page sizes to the mask of supported page sizes */
+void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize)
+{
+    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
+    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
+
+    uint16_t sup_pgsize = pci_get_word(cfg + PCI_SRIOV_SUP_PGSIZE);
+
+    sup_pgsize |= opt_sup_pgsize;
+
+    /* Make sure the new bits are set, and that system page size
+     * also can be set to any of the new values according to spec:
+     */
+    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
+    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
+}
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 551cb3d..2e9d8ba 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -11,8 +11,6 @@
 /* PCI includes legacy ISA access.  */
 #include "hw/isa/isa.h"
 
-#include "hw/pci/pcie.h"
-
 /* PCI bus */
 
 #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) | ((func) & 0x07))
@@ -132,6 +130,7 @@ enum {
 #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
 
 #include "hw/pci/pci_regs.h"
+#include "hw/pci/pcie.h"
 
 /* PCI HEADER_TYPE */
 #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
@@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus *, void *, int);
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
 void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void *opaque);
 
+pcibus_t pci_bar_address(PCIDevice *d,
+                         int reg, uint8_t type, pcibus_t size);
+
 static inline void
 pci_set_byte(uint8_t *config, uint8_t val)
 {
@@ -672,6 +674,11 @@ static inline int pci_is_express(const PCIDevice *d)
     return d->cap_present & QEMU_PCI_CAP_EXPRESS;
 }
 
+static inline int pci_is_vf(const PCIDevice *d)
+{
+    return d->exp.sriov_vf.pf != NULL;
+}
+
 static inline uint32_t pci_config_size(const PCIDevice *d)
 {
     return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE : PCI_CONFIG_SPACE_SIZE;
diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
index b48a7a2..b09f79a 100644
--- a/include/hw/pci/pcie.h
+++ b/include/hw/pci/pcie.h
@@ -25,6 +25,7 @@
 #include "hw/pci/pci_regs.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_aer.h"
+#include "hw/pci/pcie_sriov.h"
 #include "hw/hotplug.h"
 
 typedef enum {
@@ -74,6 +75,11 @@ struct PCIExpressDevice {
     /* AER */
     uint16_t aer_cap;
     PCIEAERLog aer_log;
+
+    /* SR/IOV */
+    uint16_t sriov_cap;
+    PCIESriovPF sriov_pf;
+    PCIESriovVF sriov_vf;
 };
 
 #define COMPAT_PROP_PCP "power_controller_present"
diff --git a/include/hw/pci/pcie_sriov.h b/include/hw/pci/pcie_sriov.h
new file mode 100644
index 0000000..061f5cf
--- /dev/null
+++ b/include/hw/pci/pcie_sriov.h
@@ -0,0 +1,55 @@
+/*
+ * pcie_sriov.h:
+ *
+ * Implementation of SR/IOV emulation support.
+ *
+ * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_PCIE_SRIOV_H
+#define QEMU_PCIE_SRIOV_H
+
+
+
+struct PCIESriovPF {
+    uint16_t num_vfs;           /* Number of virtual functions created */
+    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for each VF bar */
+    const char *vfname;         /* Reference to the device type used for the VFs */
+    PCIDevice **vf;             /* Pointer to an array of num_vfs VF devices */
+};
+
+struct PCIESriovVF {
+    PCIDevice *pf;              /* Pointer back to owner physical function */
+};
+
+void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
+                        const char *vfname, uint16_t vf_dev_id,
+                        uint16_t init_vfs, uint16_t total_vfs,
+                        uint16_t vf_offset, uint16_t vf_stride);
+void pcie_sriov_pf_exit(PCIDevice *dev);
+
+/* Set up a VF bar in the SR/IOV bar area */
+void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
+                               uint8_t type, dma_addr_t size);
+
+/* Instantiate a bar for a VF */
+void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
+                                MemoryRegion *memory);
+
+/* Add optional supported page sizes to the mask of supported page sizes
+ * Page size values are interpreted as opt_sup_pgsize << 12.
+ */
+void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize);
+
+/* SR/IOV capability config write handler */
+void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
+                             uint32_t val, int len);
+
+/* Reset SR/IOV VF Enable bit to unregister all VFs */
+void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
+
+#endif /* QEMU_PCIE_SRIOV_H */
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index f8a9dd6..fa36b8c 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
 typedef struct PCIEAERErr PCIEAERErr;
 typedef struct PCIEAERLog PCIEAERLog;
 typedef struct PCIEAERMsg PCIEAERMsg;
+typedef struct PCIESriovPF PCIESriovPF;
+typedef struct PCIESriovVF PCIESriovVF;
 typedef struct PCIEPort PCIEPort;
 typedef struct PCIESlot PCIESlot;
 typedef struct PCIExpressDevice PCIExpressDevice;
-- 
2.4.3

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Marcel Apfelbaum]

On 09/12/2015 03:36 PM, Knut Omang wrote:
> This patch provides the building blocks for creating an SR/IOV
> PCIe Extended Capability header and register/unregister
> SR/IOV Virtual Functions.
>
> Signed-off-by: Knut Omang <knut.omang@oracle.com>
> ---
>   hw/pci/Makefile.objs        |   2 +-
>   hw/pci/pci.c                |  99 ++++++++++++----
>   hw/pci/pcie.c               |   9 +-
>   hw/pci/pcie_sriov.c         | 271 ++++++++++++++++++++++++++++++++++++++++++++
>   include/hw/pci/pci.h        |  11 +-
>   include/hw/pci/pcie.h       |   6 +
>   include/hw/pci/pcie_sriov.h |  55 +++++++++
>   include/qemu/typedefs.h     |   2 +
>   8 files changed, 426 insertions(+), 29 deletions(-)
>   create mode 100644 hw/pci/pcie_sriov.c
>   create mode 100644 include/hw/pci/pcie_sriov.h
>
> diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
> index 9f905e6..2226980 100644
> --- a/hw/pci/Makefile.objs
> +++ b/hw/pci/Makefile.objs
> @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
>   common-obj-$(CONFIG_PCI) += shpc.o
>   common-obj-$(CONFIG_PCI) += slotid_cap.o
>   common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
> -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o pcie_sriov.o
>
>   common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
>   common-obj-$(CONFIG_ALL) += pci-stub.o
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index a5cc015..9c0eba1 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
>   {
>       uint8_t type;
>
> +    /* PCIe virtual functions do not have their own BARs */
> +    assert(!pci_is_vf(d));
> +
>       if (reg != PCI_ROM_SLOT)
>           return PCI_BASE_ADDRESS_0 + reg * 4;
>
> @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
>       }
>   }
>
> -static void pci_do_device_reset(PCIDevice *dev)
> +static void pci_reset_regions(PCIDevice *dev)
>   {
>       int r;
> +    if (pci_is_vf(dev)) {
> +        return;
> +    }
>
> -    pci_device_deassert_intx(dev);
> -    assert(dev->irq_state == 0);
> -
> -    /* Clear all writable bits */
> -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> -                                 pci_get_word(dev->wmask + PCI_COMMAND) |
> -                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
> -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> -                                 pci_get_word(dev->wmask + PCI_STATUS) |
> -                                 pci_get_word(dev->w1cmask + PCI_STATUS));
> -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
>       for (r = 0; r < PCI_NUM_REGIONS; ++r) {
>           PCIIORegion *region = &dev->io_regions[r];
>           if (!region->size) {
> @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice *dev)
>               pci_set_long(dev->config + pci_bar(dev, r), region->type);
>           }
>       }
> +}
> +
> +static void pci_do_device_reset(PCIDevice *dev)
> +{
> +    qdev_reset_all(&dev->qdev);

Hi,
Thank you for resubmitting this series!

This is only a quick look, I hope I'll have more time next week to go over it again.


> +
> +    dev->irq_state = 0;

Are you sure we need the assignment above? It seems that the irq_state
should be modified only by the intx wrappers as  pci_device_deassert_intx and so.


> +    pci_update_irq_status(dev);

Why do we have to update the irq status on reset?


> +    pci_device_deassert_intx(dev);
> +    assert(dev->irq_state == 0);
> +
> +    /* Clear all writable bits */
> +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> +                                 pci_get_word(dev->wmask + PCI_COMMAND) |
> +                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
> +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> +                                 pci_get_word(dev->wmask + PCI_STATUS) |
> +                                 pci_get_word(dev->w1cmask + PCI_STATUS));
> +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> +    pci_reset_regions(dev);
>       pci_update_mappings(dev);
>
>       msi_reset(dev);
> @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus *bus, PCIDevice *dev, Error **errp)
>           dev->config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION;
>       }
>
> +    /* With SR/IOV and ARI, a device at function 0 need not be a multifunction
> +     * device, as it may just be a VF that ended up with function 0 in
> +     * the legacy PCI interpretation. Avoid failing in such cases:
> +     */
> +    if (pci_is_vf(dev) &&
> +        dev->exp.sriov_vf.pf->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
> +        return;
> +    }
> +
>       /*
>        * multifunction bit is interpreted in two ways as follows.
>        *   - all functions must set the bit to 1.
> @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num,
>       uint64_t wmask;
>       pcibus_t size = memory_region_size(memory);
>
> +    assert(!pci_is_vf(pci_dev)); /* VFs must use pcie_sriov_vf_register_bar */
>       assert(region_num >= 0);
>       assert(region_num < PCI_NUM_REGIONS);
>       if (size & (size-1)) {
> @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num)
>       return pci_dev->io_regions[region_num].addr;
>   }
>
> -static pcibus_t pci_bar_address(PCIDevice *d,
> -				int reg, uint8_t type, pcibus_t size)
> +
> +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
> +                                        uint8_t type, pcibus_t size)
> +{
> +    pcibus_t new_addr;
> +    if (!pci_is_vf(d)) {
> +        int bar = pci_bar(d, reg);
> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +            new_addr = pci_get_quad(d->config + bar);
> +        } else {
> +            new_addr = pci_get_long(d->config + bar);
> +        }
> +    } else {
> +        PCIDevice *pf = d->exp.sriov_vf.pf;
> +        uint16_t sriov_cap = pf->exp.sriov_cap;
> +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
> +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
> +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
> +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) / vf_stride;
> +
> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +            new_addr = pci_get_quad(pf->config + bar);
> +        } else {
> +            new_addr = pci_get_long(pf->config + bar);
> +        }
> +        new_addr += vf_num * size;
> +    }
> +    if (reg != PCI_ROM_SLOT) {
> +        /* Preserve the rom enable bit */
> +        new_addr &= ~(size - 1);
> +    }
> +    return new_addr;
> +}
> +
> +pcibus_t pci_bar_address(PCIDevice *d,
> +                         int reg, uint8_t type, pcibus_t size)
>   {
>       pcibus_t new_addr, last_addr;
> -    int bar = pci_bar(d, reg);
>       uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
>       Object *machine = qdev_get_machine();
>       ObjectClass *oc = object_get_class(machine);
> @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
>           if (!(cmd & PCI_COMMAND_IO)) {
>               return PCI_BAR_UNMAPPED;
>           }
> -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
> +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
>           last_addr = new_addr + size - 1;
>           /* Check if 32 bit BAR wraps around explicitly.
>            * TODO: make priorities correct and remove this work around.
> @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
>       if (!(cmd & PCI_COMMAND_MEMORY)) {
>           return PCI_BAR_UNMAPPED;
>       }
> -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> -        new_addr = pci_get_quad(d->config + bar);
> -    } else {
> -        new_addr = pci_get_long(d->config + bar);
> -    }
> +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
>       /* the ROM slot has a specific enable bit */
>       if (reg == PCI_ROM_SLOT && !(new_addr & PCI_ROM_ADDRESS_ENABLE)) {
>           return PCI_BAR_UNMAPPED;
> @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val_in, int
>
>       msi_write_config(d, addr, val_in, l);
>       msix_write_config(d, addr, val_in, l);
> +    pcie_sriov_config_write(d, addr, val_in, l);
>   }
>
>   /***********************************************************/
> diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
> index 6e28985..ba49c0f 100644
> --- a/hw/pci/pcie.c
> +++ b/hw/pci/pcie.c
> @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
>        * Right now, only a device of function = 0 is allowed to be
>        * hot plugged/unplugged.
>        */
> -    assert(PCI_FUNC(pci_dev->devfn) == 0);
> +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
>
>       pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
>                                  PCI_EXP_SLTSTA_PDS);
> @@ -265,10 +265,11 @@ void pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
>                                            DeviceState *dev, Error **errp)
>   {
>       uint8_t *exp_cap;
> +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
>
> -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev, &exp_cap, errp);
> +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
>
> -    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
> +    pcie_cap_slot_push_attention_button(pdev);
>   }

This chunk is not directly liked to the patch, I would put it in a different patch.

>
>   /* pci express slot for pci express root/downstream port
> @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
>       }
>
>       /*
> -     * If the slot is polulated, power indicator is off and power
> +     * If the slot is populated, power indicator is off and power
>        * controller is off, it is safe to detach the devices.
>        */
>       if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val & PCI_EXP_SLTCTL_PCC) &&


Same here. I am always happy to have this kind of types taken care of,
but I think a separate patch would be cleaner.

> diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
> new file mode 100644
> index 0000000..14f31cc
> --- /dev/null
> +++ b/hw/pci/pcie_sriov.c
> @@ -0,0 +1,271 @@
> +/*
> + * pcie_sriov.c:
> + *
> + * Implementation of SR/IOV emulation support.
> + *
> + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>

2015 :)

> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "hw/pci/pci.h"
> +#include "hw/pci/pcie.h"
> +#include "hw/pci/pci_bus.h"
> +#include "qemu/range.h"
> +
> +//#define DEBUG_SRIOV
> +#ifdef DEBUG_SRIOV
> +# define SRIOV_DPRINTF(fmt, ...)                                         \
> +    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ## __VA_ARGS__)
> +#else
> +# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
> +#endif
> +#define SRIOV_DEV_PRINTF(dev, fmt, ...)                                  \
> +    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ## __VA_ARGS__)
> +
> +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name);
> +static void unregister_vfs(PCIDevice *dev);
> +
> +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> +                        const char *vfname, uint16_t vf_dev_id,
> +                        uint16_t init_vfs, uint16_t total_vfs,
> +                        uint16_t vf_offset, uint16_t vf_stride)
> +{
> +    uint8_t *cfg = dev->config + offset;
> +    uint8_t *wmask;
> +
> +    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
> +                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
> +    dev->exp.sriov_cap = offset;
> +    dev->exp.sriov_pf.num_vfs = 0;
> +    dev->exp.sriov_pf.vfname = g_strdup(vfname);
> +    dev->exp.sriov_pf.vf = NULL;
> +
> +    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
> +    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
> +
> +    /* Minimal page size support values required by the SR/IOV standard:
> +     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
> +     * Hard coded for simplicity in this version
> +     */

A define (with the above very good explanation) would be preferred.

> +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
> +
> +    /* Default is to use 4K pages, software can modify it
> +     * to any of the supported bits
> +     */
> +    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
> +
> +    /* Set up device ID and initial/total number of VFs available */
> +    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
> +    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
> +    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
> +    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
> +
> +    /* Write enable control bits */
> +    wmask = dev->wmask + offset;
> +    pci_set_word(wmask + PCI_SRIOV_CTRL,
> +                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE | PCI_SRIOV_CTRL_ARI);
> +    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
> +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
> +
> +    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
> +}
> +
> +void pcie_sriov_pf_exit(PCIDevice *dev)
> +{
> +    SRIOV_DPRINTF("\n");
> +    unregister_vfs(dev);
> +    g_free((char *)dev->exp.sriov_pf.vfname);
> +    dev->exp.sriov_pf.vfname = NULL;
> +}
> +
> +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> +                               uint8_t type, dma_addr_t size)
> +{
> +    uint32_t addr;
> +    uint64_t wmask;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +
> +    assert(sriov_cap > 0);
> +    assert(region_num >= 0);
> +    assert(region_num < PCI_NUM_REGIONS);
> +    assert(region_num != PCI_ROM_SLOT);
> +
> +    wmask = ~(size - 1);
> +    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
> +
> +    pci_set_long(dev->config + addr, type);
> +    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
> +        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +        pci_set_quad(dev->wmask + addr, wmask);
> +        pci_set_quad(dev->cmask + addr, ~0ULL);
> +    } else {
> +        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
> +        pci_set_long(dev->cmask + addr, 0xffffffff);
> +    }
> +    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
> +}
> +
> +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> +                                MemoryRegion *memory)
> +{
> +    PCIIORegion *r;
> +    uint8_t type;
> +    pcibus_t size = memory_region_size(memory);
> +
> +    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
> +    assert(region_num >= 0);
> +    assert(region_num < PCI_NUM_REGIONS);
> +    type = dev->exp.sriov_vf.pf->exp.sriov_pf.vf_bar_type[region_num];
> +
> +    if (size & (size-1)) {

We already have a  is_power_of_2 func defined in include/qemu/host-utils.h

> +        fprintf(stderr, "ERROR: PCI region size must be a power"
> +                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n", type, size);

Maybe we should error_report instead of fprintf

> +        exit(1);
> +    }
> +
> +    r = &dev->io_regions[region_num];
> +    r->memory = memory;
> +    r->address_space =
> +        type & PCI_BASE_ADDRESS_SPACE_IO
> +        ? dev->bus->address_space_io
> +        : dev->bus->address_space_mem;
> +    r->size = size;
> +    r->type = type;
> +
> +    r->addr = pci_bar_address(dev, region_num, r->type, r->size);
> +    if (r->addr != PCI_BAR_UNMAPPED) {
> +        memory_region_add_subregion_overlap(r->address_space,
> +                                            r->addr, r->memory, 1);
> +    }
> +}
> +
> +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name)
> +{
> +    PCIDevice *dev = pci_create(pf->bus, devfn, name);
> +    dev->exp.sriov_vf.pf = pf;
> +    Error *local_err = NULL;
> +
> +    object_property_set_bool(OBJECT(&dev->qdev), true, "realized", &local_err);
> +    if (local_err) {
> +        fprintf(stderr, "Failed to enable: %s\n",
> +                error_get_pretty(local_err));
> +        error_free(local_err);

and error_report_err here

> +        return NULL;
> +    }
> +
> +    /* set vid/did according to sr/iov spec - they are not used */
> +    pci_config_set_vendor_id(dev->config, 0xffff);
> +    pci_config_set_device_id(dev->config, 0xffff);
> +    return dev;
> +}
> +
> +static void register_vfs(PCIDevice *dev)
> +{
> +    uint16_t num_vfs;
> +    uint16_t i;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
> +    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
> +    int32_t devfn = dev->devfn + vf_offset;
> +
> +    assert(sriov_cap > 0);
> +    num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF);
> +
> +    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) * num_vfs);
> +    assert(dev->exp.sriov_pf.vf);
> +
> +    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
> +    for (i = 0; i < num_vfs; i++) {
> +        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev->exp.sriov_pf.vfname);
> +        if (!dev->exp.sriov_pf.vf[i]) {
> +            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at devfn %x.%x\n", i,
> +                             PCI_SLOT(devfn), PCI_FUNC(devfn));
> +            num_vfs = i;
> +            break;
> +        }
> +        devfn += vf_stride;
> +    }
> +    dev->exp.sriov_pf.num_vfs = num_vfs;
> +}
> +
> +static void unregister_vfs(PCIDevice *dev)
> +{
> +    Error *local_err = NULL;
> +    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
> +    uint16_t i;
> +    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n", num_vfs);
> +    for (i = 0; i < num_vfs; i++) {
> +        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]->qdev), false, "realized", &local_err);
> +        if (local_err) {
> +            fprintf(stderr, "Failed to unplug: %s\n",
> +                    error_get_pretty(local_err));
> +            error_free(local_err);

error_report_err

> +        }
> +    }
> +    g_free(dev->exp.sriov_pf.vf);
> +    dev->exp.sriov_pf.vf = NULL;
> +    dev->exp.sriov_pf.num_vfs = 0;
> +}
> +
> +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address, uint32_t val, int len)
> +{
> +    uint32_t off;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +
> +    if (!sriov_cap || address < sriov_cap) {
> +        return;
> +    }
> +    off = address - sriov_cap;
> +    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
> +        return;
> +    }
> +
> +    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val 0x%x len %d\n",
> +                 sriov_cap, off, val, len);
> +
> +    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
> +        if (dev->exp.sriov_pf.num_vfs) {
> +            if (!(val & PCI_SRIOV_CTRL_VFE)) {
> +                unregister_vfs(dev);
> +            }
> +        } else {
> +            if (val & PCI_SRIOV_CTRL_VFE) {
> +                register_vfs(dev);
> +            }
> +        }
> +    }
> +}
> +
> +
> +/* Reset SR/IOV VF Enable bit to trigger an unregister of all VFs */
> +void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
> +{
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +    if (sriov_cap) {
> +        uint32_t val = pci_get_byte(dev->config + sriov_cap + PCI_SRIOV_CTRL);
> +        if (val & PCI_SRIOV_CTRL_VFE) {
> +            val &= ~PCI_SRIOV_CTRL_VFE;
> +            pcie_sriov_config_write(dev, sriov_cap + PCI_SRIOV_CTRL, val, 1);
> +        }
> +    }
> +}
> +
> +/* Add optional supported page sizes to the mask of supported page sizes */
> +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize)
> +{
> +    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
> +    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
> +
> +    uint16_t sup_pgsize = pci_get_word(cfg + PCI_SRIOV_SUP_PGSIZE);
> +
> +    sup_pgsize |= opt_sup_pgsize;
> +
> +    /* Make sure the new bits are set, and that system page size
> +     * also can be set to any of the new values according to spec:
> +     */
> +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
> +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
> +}
> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> index 551cb3d..2e9d8ba 100644
> --- a/include/hw/pci/pci.h
> +++ b/include/hw/pci/pci.h
> @@ -11,8 +11,6 @@
>   /* PCI includes legacy ISA access.  */
>   #include "hw/isa/isa.h"
>
> -#include "hw/pci/pcie.h"
> -
>   /* PCI bus */
>
>   #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) | ((func) & 0x07))
> @@ -132,6 +130,7 @@ enum {
>   #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
>
>   #include "hw/pci/pci_regs.h"
> +#include "hw/pci/pcie.h"

Why is it moved here?


>
>   /* PCI HEADER_TYPE */
>   #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
> @@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus *, void *, int);
>   AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
>   void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void *opaque);
>
> +pcibus_t pci_bar_address(PCIDevice *d,
> +                         int reg, uint8_t type, pcibus_t size);
> +
>   static inline void
>   pci_set_byte(uint8_t *config, uint8_t val)
>   {
> @@ -672,6 +674,11 @@ static inline int pci_is_express(const PCIDevice *d)
>       return d->cap_present & QEMU_PCI_CAP_EXPRESS;
>   }
>
> +static inline int pci_is_vf(const PCIDevice *d)
> +{
> +    return d->exp.sriov_vf.pf != NULL;
> +}
> +
>   static inline uint32_t pci_config_size(const PCIDevice *d)
>   {
>       return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE : PCI_CONFIG_SPACE_SIZE;
> diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
> index b48a7a2..b09f79a 100644
> --- a/include/hw/pci/pcie.h
> +++ b/include/hw/pci/pcie.h
> @@ -25,6 +25,7 @@
>   #include "hw/pci/pci_regs.h"
>   #include "hw/pci/pcie_regs.h"
>   #include "hw/pci/pcie_aer.h"
> +#include "hw/pci/pcie_sriov.h"
>   #include "hw/hotplug.h"
>
>   typedef enum {
> @@ -74,6 +75,11 @@ struct PCIExpressDevice {
>       /* AER */
>       uint16_t aer_cap;
>       PCIEAERLog aer_log;
> +
> +    /* SR/IOV */
> +    uint16_t sriov_cap;
> +    PCIESriovPF sriov_pf;
> +    PCIESriovVF sriov_vf;
>   };
>
>   #define COMPAT_PROP_PCP "power_controller_present"
> diff --git a/include/hw/pci/pcie_sriov.h b/include/hw/pci/pcie_sriov.h
> new file mode 100644
> index 0000000..061f5cf
> --- /dev/null
> +++ b/include/hw/pci/pcie_sriov.h
> @@ -0,0 +1,55 @@
> +/*
> + * pcie_sriov.h:
> + *
> + * Implementation of SR/IOV emulation support.
> + *
> + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>

2015

> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#ifndef QEMU_PCIE_SRIOV_H
> +#define QEMU_PCIE_SRIOV_H
> +
> +
> +
> +struct PCIESriovPF {
> +    uint16_t num_vfs;           /* Number of virtual functions created */
> +    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for each VF bar */
> +    const char *vfname;         /* Reference to the device type used for the VFs */
> +    PCIDevice **vf;             /* Pointer to an array of num_vfs VF devices */
> +};
> +
> +struct PCIESriovVF {
> +    PCIDevice *pf;              /* Pointer back to owner physical function */
> +};
> +
> +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> +                        const char *vfname, uint16_t vf_dev_id,
> +                        uint16_t init_vfs, uint16_t total_vfs,
> +                        uint16_t vf_offset, uint16_t vf_stride);
> +void pcie_sriov_pf_exit(PCIDevice *dev);
> +
> +/* Set up a VF bar in the SR/IOV bar area */
> +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> +                               uint8_t type, dma_addr_t size);
> +
> +/* Instantiate a bar for a VF */
> +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> +                                MemoryRegion *memory);
> +
> +/* Add optional supported page sizes to the mask of supported page sizes
> + * Page size values are interpreted as opt_sup_pgsize << 12.
> + */
> +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize);
> +
> +/* SR/IOV capability config write handler */
> +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> +                             uint32_t val, int len);
> +
> +/* Reset SR/IOV VF Enable bit to unregister all VFs */
> +void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
> +
> +#endif /* QEMU_PCIE_SRIOV_H */
> diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> index f8a9dd6..fa36b8c 100644
> --- a/include/qemu/typedefs.h
> +++ b/include/qemu/typedefs.h
> @@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
>   typedef struct PCIEAERErr PCIEAERErr;
>   typedef struct PCIEAERLog PCIEAERLog;
>   typedef struct PCIEAERMsg PCIEAERMsg;
> +typedef struct PCIESriovPF PCIESriovPF;
> +typedef struct PCIESriovVF PCIESriovVF;
>   typedef struct PCIEPort PCIEPort;
>   typedef struct PCIESlot PCIESlot;
>   typedef struct PCIExpressDevice PCIExpressDevice;
>


I hope I helped,

Thanks,
Marcel

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Knut Omang]

On Thu, 2015-09-17 at 14:49 +0300, Marcel Apfelbaum wrote:
> On 09/12/2015 03:36 PM, Knut Omang wrote:
> > This patch provides the building blocks for creating an SR/IOV
> > PCIe Extended Capability header and register/unregister
> > SR/IOV Virtual Functions.
> > 
> > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > ---
> >   hw/pci/Makefile.objs        |   2 +-
> >   hw/pci/pci.c                |  99 ++++++++++++----
> >   hw/pci/pcie.c               |   9 +-
> >   hw/pci/pcie_sriov.c         | 271
> > ++++++++++++++++++++++++++++++++++++++++++++
> >   include/hw/pci/pci.h        |  11 +-
> >   include/hw/pci/pcie.h       |   6 +
> >   include/hw/pci/pcie_sriov.h |  55 +++++++++
> >   include/qemu/typedefs.h     |   2 +
> >   8 files changed, 426 insertions(+), 29 deletions(-)
> >   create mode 100644 hw/pci/pcie_sriov.c
> >   create mode 100644 include/hw/pci/pcie_sriov.h
> > 
> > diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
> > index 9f905e6..2226980 100644
> > --- a/hw/pci/Makefile.objs
> > +++ b/hw/pci/Makefile.objs
> > @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
> >   common-obj-$(CONFIG_PCI) += shpc.o
> >   common-obj-$(CONFIG_PCI) += slotid_cap.o
> >   common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
> > -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > pcie_sriov.o
> > 
> >   common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
> >   common-obj-$(CONFIG_ALL) += pci-stub.o
> > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > index a5cc015..9c0eba1 100644
> > --- a/hw/pci/pci.c
> > +++ b/hw/pci/pci.c
> > @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
> >   {
> >       uint8_t type;
> > 
> > +    /* PCIe virtual functions do not have their own BARs */
> > +    assert(!pci_is_vf(d));
> > +
> >       if (reg != PCI_ROM_SLOT)
> >           return PCI_BASE_ADDRESS_0 + reg * 4;
> > 
> > @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
> >       }
> >   }
> > 
> > -static void pci_do_device_reset(PCIDevice *dev)
> > +static void pci_reset_regions(PCIDevice *dev)
> >   {
> >       int r;
> > +    if (pci_is_vf(dev)) {
> > +        return;
> > +    }
> > 
> > -    pci_device_deassert_intx(dev);
> > -    assert(dev->irq_state == 0);
> > -
> > -    /* Clear all writable bits */
> > -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > -                                 pci_get_word(dev->wmask +
> > PCI_COMMAND) |
> > -                                 pci_get_word(dev->w1cmask +
> > PCI_COMMAND));
> > -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > -                                 pci_get_word(dev->wmask +
> > PCI_STATUS) |
> > -                                 pci_get_word(dev->w1cmask +
> > PCI_STATUS));
> > -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> >       for (r = 0; r < PCI_NUM_REGIONS; ++r) {
> >           PCIIORegion *region = &dev->io_regions[r];
> >           if (!region->size) {
> > @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice
> > *dev)
> >               pci_set_long(dev->config + pci_bar(dev, r), region
> > ->type);
> >           }
> >       }
> > +}
> > +
> > +static void pci_do_device_reset(PCIDevice *dev)
> > +{
> > +    qdev_reset_all(&dev->qdev);
> 
> Hi,
> Thank you for resubmitting this series!
> 
> This is only a quick look, I hope I'll have more time next week to go
> over it again.
> 
> 
> > +
> > +    dev->irq_state = 0;
> 
> Are you sure we need the assignment above? It seems that the
> irq_state
> should be modified only by the intx wrappers as 
>  pci_device_deassert_intx and so.
> 
> 
> > +    pci_update_irq_status(dev);
> 
> Why do we have to update the irq status on reset?

I struggled a lot with interrupts and PF disapperance in earlier
versions, this may be unintended leftovers from rebase.

The intention was to avoid the qdev removal which caused problems with
unintended "hot unplug of the PF" at VF removal earlier, but after some
of the more recent QOM'ification all those problems disappeared.

I tried without these lines and the do_device_reset refactor and it
seems to work just fine, will fix that in v5, thanks!

> > +    pci_device_deassert_intx(dev);
> > +    assert(dev->irq_state == 0);
> > +
> > +    /* Clear all writable bits */
> > +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > +                                 pci_get_word(dev->wmask +
> > PCI_COMMAND) |
> > +                                 pci_get_word(dev->w1cmask +
> > PCI_COMMAND));
> > +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > +                                 pci_get_word(dev->wmask +
> > PCI_STATUS) |
> > +                                 pci_get_word(dev->w1cmask +
> > PCI_STATUS));
> > +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> > +    pci_reset_regions(dev);
> >       pci_update_mappings(dev);
> > 
> >       msi_reset(dev);
> > @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus
> > *bus, PCIDevice *dev, Error **errp)
> >           dev->config[PCI_HEADER_TYPE] |=
> > PCI_HEADER_TYPE_MULTI_FUNCTION;
> >       }
> > 
> > +    /* With SR/IOV and ARI, a device at function 0 need not be a
> > multifunction
> > +     * device, as it may just be a VF that ended up with function
> > 0 in
> > +     * the legacy PCI interpretation. Avoid failing in such cases:
> > +     */
> > +    if (pci_is_vf(dev) &&
> > +        dev->exp.sriov_vf.pf->cap_present &
> > QEMU_PCI_CAP_MULTIFUNCTION) {
> > +        return;
> > +    }
> > +
> >       /*
> >        * multifunction bit is interpreted in two ways as follows.
> >        *   - all functions must set the bit to 1.
> > @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int
> > region_num,
> >       uint64_t wmask;
> >       pcibus_t size = memory_region_size(memory);
> > 
> > +    assert(!pci_is_vf(pci_dev)); /* VFs must use
> > pcie_sriov_vf_register_bar */
> >       assert(region_num >= 0);
> >       assert(region_num < PCI_NUM_REGIONS);
> >       if (size & (size-1)) {
> > @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice
> > *pci_dev, int region_num)
> >       return pci_dev->io_regions[region_num].addr;
> >   }
> > 
> > -static pcibus_t pci_bar_address(PCIDevice *d,
> > -				int reg, uint8_t type, pcibus_t
> > size)
> > +
> > +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
> > +                                        uint8_t type, pcibus_t
> > size)
> > +{
> > +    pcibus_t new_addr;
> > +    if (!pci_is_vf(d)) {
> > +        int bar = pci_bar(d, reg);
> > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +            new_addr = pci_get_quad(d->config + bar);
> > +        } else {
> > +            new_addr = pci_get_long(d->config + bar);
> > +        }
> > +    } else {
> > +        PCIDevice *pf = d->exp.sriov_vf.pf;
> > +        uint16_t sriov_cap = pf->exp.sriov_cap;
> > +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
> > +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap +
> > PCI_SRIOV_VF_OFFSET);
> > +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap +
> > PCI_SRIOV_VF_STRIDE);
> > +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) /
> > vf_stride;
> > +
> > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +            new_addr = pci_get_quad(pf->config + bar);
> > +        } else {
> > +            new_addr = pci_get_long(pf->config + bar);
> > +        }
> > +        new_addr += vf_num * size;
> > +    }
> > +    if (reg != PCI_ROM_SLOT) {
> > +        /* Preserve the rom enable bit */
> > +        new_addr &= ~(size - 1);
> > +    }
> > +    return new_addr;
> > +}
> > +
> > +pcibus_t pci_bar_address(PCIDevice *d,
> > +                         int reg, uint8_t type, pcibus_t size)
> >   {
> >       pcibus_t new_addr, last_addr;
> > -    int bar = pci_bar(d, reg);
> >       uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
> >       Object *machine = qdev_get_machine();
> >       ObjectClass *oc = object_get_class(machine);
> > @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
> >           if (!(cmd & PCI_COMMAND_IO)) {
> >               return PCI_BAR_UNMAPPED;
> >           }
> > -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
> > +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
> >           last_addr = new_addr + size - 1;
> >           /* Check if 32 bit BAR wraps around explicitly.
> >            * TODO: make priorities correct and remove this work
> > around.
> > @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice
> > *d,
> >       if (!(cmd & PCI_COMMAND_MEMORY)) {
> >           return PCI_BAR_UNMAPPED;
> >       }
> > -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > -        new_addr = pci_get_quad(d->config + bar);
> > -    } else {
> > -        new_addr = pci_get_long(d->config + bar);
> > -    }
> > +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
> >       /* the ROM slot has a specific enable bit */
> >       if (reg == PCI_ROM_SLOT && !(new_addr &
> > PCI_ROM_ADDRESS_ENABLE)) {
> >           return PCI_BAR_UNMAPPED;
> > @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d,
> > uint32_t addr, uint32_t val_in, int
> > 
> >       msi_write_config(d, addr, val_in, l);
> >       msix_write_config(d, addr, val_in, l);
> > +    pcie_sriov_config_write(d, addr, val_in, l);
> >   }
> > 
> >   /***********************************************************/
> > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
> > index 6e28985..ba49c0f 100644
> > --- a/hw/pci/pcie.c
> > +++ b/hw/pci/pcie.c
> > @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler
> > *hotplug_dev, DeviceState *dev,
> >        * Right now, only a device of function = 0 is allowed to be
> >        * hot plugged/unplugged.
> >        */
> > -    assert(PCI_FUNC(pci_dev->devfn) == 0);
> > +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
> > 
> >       pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
> >                                  PCI_EXP_SLTSTA_PDS);
> > @@ -265,10 +265,11 @@ void
> > pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
> >                                            DeviceState *dev, Error
> > **errp)
> >   {
> >       uint8_t *exp_cap;
> > +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
> > 
> > -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev,
> > &exp_cap, errp);
> > +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
> > 
> > -    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
> > +    pcie_cap_slot_push_attention_button(pdev);
> >   }
> 
> This chunk is not directly liked to the patch, I would put it in a
> different patch.

ok

> >  /* pci express slot for pci express root/downstream port
> > @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
> >       }
> > 
> >       /*
> > -     * If the slot is polulated, power indicator is off and power
> > +     * If the slot is populated, power indicator is off and power
> >        * controller is off, it is safe to detach the devices.
> >        */
> >       if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val &
> > PCI_EXP_SLTCTL_PCC) &&
> 
> 
> Same here. I am always happy to have this kind of types taken care
> of,
> but I think a separate patch would be cleaner.

Would you like it in the patch set as a separate patch or should I
schedule it for a trivial patches set instead, maybe together with the
 PCI_DEVICE thing?

> > diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
> > new file mode 100644
> > index 0000000..14f31cc
> > --- /dev/null
> > +++ b/hw/pci/pcie_sriov.c
> > @@ -0,0 +1,271 @@
> > +/*
> > + * pcie_sriov.c:
> > + *
> > + * Implementation of SR/IOV emulation support.
> > + *
> > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> 
> 2015 :)

thanks,

> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2
> > or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#include "hw/pci/pci.h"
> > +#include "hw/pci/pcie.h"
> > +#include "hw/pci/pci_bus.h"
> > +#include "qemu/range.h"
> > +
> > +//#define DEBUG_SRIOV
> > +#ifdef DEBUG_SRIOV
> > +# define SRIOV_DPRINTF(fmt, ...)                                  
> >        \
> > +    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ##
> > __VA_ARGS__)
> > +#else
> > +# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
> > +#endif
> > +#define SRIOV_DEV_PRINTF(dev, fmt, ...)                           
> >        \
> > +    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ##
> > __VA_ARGS__)
> > +
> > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char
> > *name);
> > +static void unregister_vfs(PCIDevice *dev);
> > +
> > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > +                        const char *vfname, uint16_t vf_dev_id,
> > +                        uint16_t init_vfs, uint16_t total_vfs,
> > +                        uint16_t vf_offset, uint16_t vf_stride)
> > +{
> > +    uint8_t *cfg = dev->config + offset;
> > +    uint8_t *wmask;
> > +
> > +    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
> > +                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
> > +    dev->exp.sriov_cap = offset;
> > +    dev->exp.sriov_pf.num_vfs = 0;
> > +    dev->exp.sriov_pf.vfname = g_strdup(vfname);
> > +    dev->exp.sriov_pf.vf = NULL;
> > +
> > +    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
> > +    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
> > +
> > +    /* Minimal page size support values required by the SR/IOV
> > standard:
> > +     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
> > +     * Hard coded for simplicity in this version
> > +     */
> 
> A define (with the above very good explanation) would be preferred.

Will do

> > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
> > +
> > +    /* Default is to use 4K pages, software can modify it
> > +     * to any of the supported bits
> > +     */
> > +    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
> > +
> > +    /* Set up device ID and initial/total number of VFs available
> > */
> > +    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
> > +    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
> > +    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
> > +    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
> > +
> > +    /* Write enable control bits */
> > +    wmask = dev->wmask + offset;
> > +    pci_set_word(wmask + PCI_SRIOV_CTRL,
> > +                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE |
> > PCI_SRIOV_CTRL_ARI);
> > +    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
> > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
> > +
> > +    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
> > +}
> > +
> > +void pcie_sriov_pf_exit(PCIDevice *dev)
> > +{
> > +    SRIOV_DPRINTF("\n");
> > +    unregister_vfs(dev);
> > +    g_free((char *)dev->exp.sriov_pf.vfname);
> > +    dev->exp.sriov_pf.vfname = NULL;
> > +}
> > +
> > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > +                               uint8_t type, dma_addr_t size)
> > +{
> > +    uint32_t addr;
> > +    uint64_t wmask;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +
> > +    assert(sriov_cap > 0);
> > +    assert(region_num >= 0);
> > +    assert(region_num < PCI_NUM_REGIONS);
> > +    assert(region_num != PCI_ROM_SLOT);
> > +
> > +    wmask = ~(size - 1);
> > +    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
> > +
> > +    pci_set_long(dev->config + addr, type);
> > +    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
> > +        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +        pci_set_quad(dev->wmask + addr, wmask);
> > +        pci_set_quad(dev->cmask + addr, ~0ULL);
> > +    } else {
> > +        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
> > +        pci_set_long(dev->cmask + addr, 0xffffffff);
> > +    }
> > +    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
> > +}
> > +
> > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > +                                MemoryRegion *memory)
> > +{
> > +    PCIIORegion *r;
> > +    uint8_t type;
> > +    pcibus_t size = memory_region_size(memory);
> > +
> > +    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
> > +    assert(region_num >= 0);
> > +    assert(region_num < PCI_NUM_REGIONS);
> > +    type = dev->exp.sriov_vf.pf
> > ->exp.sriov_pf.vf_bar_type[region_num];
> > +
> > +    if (size & (size-1)) {
> 
> We already have a  is_power_of_2 func defined in include/qemu/host
> -utils.h

will fix,

> > +        fprintf(stderr, "ERROR: PCI region size must be a power"
> > +                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n",
> > type, size);
> 
> Maybe we should error_report instead of fprintf

Leftover from far too many rebases, will fix,

> > +        exit(1);
> > +    }
> > +
> > +    r = &dev->io_regions[region_num];
> > +    r->memory = memory;
> > +    r->address_space =
> > +        type & PCI_BASE_ADDRESS_SPACE_IO
> > +        ? dev->bus->address_space_io
> > +        : dev->bus->address_space_mem;
> > +    r->size = size;
> > +    r->type = type;
> > +
> > +    r->addr = pci_bar_address(dev, region_num, r->type, r->size);
> > +    if (r->addr != PCI_BAR_UNMAPPED) {
> > +        memory_region_add_subregion_overlap(r->address_space,
> > +                                            r->addr, r->memory,
> > 1);
> > +    }
> > +}
> > +
> > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char
> > *name)
> > +{
> > +    PCIDevice *dev = pci_create(pf->bus, devfn, name);
> > +    dev->exp.sriov_vf.pf = pf;
> > +    Error *local_err = NULL;
> > +
> > +    object_property_set_bool(OBJECT(&dev->qdev), true, "realized",
> > &local_err);
> > +    if (local_err) {
> > +        fprintf(stderr, "Failed to enable: %s\n",
> > +                error_get_pretty(local_err));
> > +        error_free(local_err);
> 
> and error_report_err here

ok

> > +        return NULL;
> > +    }
> > +
> > +    /* set vid/did according to sr/iov spec - they are not used */
> > +    pci_config_set_vendor_id(dev->config, 0xffff);
> > +    pci_config_set_device_id(dev->config, 0xffff);
> > +    return dev;
> > +}
> > +
> > +static void register_vfs(PCIDevice *dev)
> > +{
> > +    uint16_t num_vfs;
> > +    uint16_t i;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_VF_OFFSET);
> > +    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_VF_STRIDE);
> > +    int32_t devfn = dev->devfn + vf_offset;
> > +
> > +    assert(sriov_cap > 0);
> > +    num_vfs = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_NUM_VF);
> > +
> > +    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) *
> > num_vfs);
> > +    assert(dev->exp.sriov_pf.vf);
> > +
> > +    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
> > +    for (i = 0; i < num_vfs; i++) {
> > +        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev
> > ->exp.sriov_pf.vfname);
> > +        if (!dev->exp.sriov_pf.vf[i]) {
> > +            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at devfn
> > %x.%x\n", i,
> > +                             PCI_SLOT(devfn), PCI_FUNC(devfn));
> > +            num_vfs = i;
> > +            break;
> > +        }
> > +        devfn += vf_stride;
> > +    }
> > +    dev->exp.sriov_pf.num_vfs = num_vfs;
> > +}
> > +
> > +static void unregister_vfs(PCIDevice *dev)
> > +{
> > +    Error *local_err = NULL;
> > +    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
> > +    uint16_t i;
> > +    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n", num_vfs);
> > +    for (i = 0; i < num_vfs; i++) {
> > +        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]
> > ->qdev), false, "realized", &local_err);
> > +        if (local_err) {
> > +            fprintf(stderr, "Failed to unplug: %s\n",
> > +                    error_get_pretty(local_err));
> > +            error_free(local_err);
> 
> error_report_err
> 
> > +        }
> > +    }
> > +    g_free(dev->exp.sriov_pf.vf);
> > +    dev->exp.sriov_pf.vf = NULL;
> > +    dev->exp.sriov_pf.num_vfs = 0;
> > +}
> > +
> > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > uint32_t val, int len)
> > +{
> > +    uint32_t off;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +
> > +    if (!sriov_cap || address < sriov_cap) {
> > +        return;
> > +    }
> > +    off = address - sriov_cap;
> > +    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
> > +        return;
> > +    }
> > +
> > +    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val 0x%x
> > len %d\n",
> > +                 sriov_cap, off, val, len);
> > +
> > +    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
> > +        if (dev->exp.sriov_pf.num_vfs) {
> > +            if (!(val & PCI_SRIOV_CTRL_VFE)) {
> > +                unregister_vfs(dev);
> > +            }
> > +        } else {
> > +            if (val & PCI_SRIOV_CTRL_VFE) {
> > +                register_vfs(dev);
> > +            }
> > +        }
> > +    }
> > +}
> > +
> > +
> > +/* Reset SR/IOV VF Enable bit to trigger an unregister of all VFs
> > */
> > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
> > +{
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +    if (sriov_cap) {
> > +        uint32_t val = pci_get_byte(dev->config + sriov_cap +
> > PCI_SRIOV_CTRL);
> > +        if (val & PCI_SRIOV_CTRL_VFE) {
> > +            val &= ~PCI_SRIOV_CTRL_VFE;
> > +            pcie_sriov_config_write(dev, sriov_cap +
> > PCI_SRIOV_CTRL, val, 1);
> > +        }
> > +    }
> > +}
> > +
> > +/* Add optional supported page sizes to the mask of supported page
> > sizes */
> > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > opt_sup_pgsize)
> > +{
> > +    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
> > +    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
> > +
> > +    uint16_t sup_pgsize = pci_get_word(cfg +
> > PCI_SRIOV_SUP_PGSIZE);
> > +
> > +    sup_pgsize |= opt_sup_pgsize;
> > +
> > +    /* Make sure the new bits are set, and that system page size
> > +     * also can be set to any of the new values according to spec:
> > +     */
> > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
> > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
> > +}
> > diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> > index 551cb3d..2e9d8ba 100644
> > --- a/include/hw/pci/pci.h
> > +++ b/include/hw/pci/pci.h
> > @@ -11,8 +11,6 @@
> >   /* PCI includes legacy ISA access.  */
> >   #include "hw/isa/isa.h"
> > 
> > -#include "hw/pci/pcie.h"
> > -
> >   /* PCI bus */
> > 
> >   #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) | ((func)
> > & 0x07))
> > @@ -132,6 +130,7 @@ enum {
> >   #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
> > 
> >   #include "hw/pci/pci_regs.h"
> > +#include "hw/pci/pcie.h"
> 
> Why is it moved here?

Probably an unintended side effect of rebasing..

> >   /* PCI HEADER_TYPE */
> >   #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
> > @@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus *,
> > void *, int);
> >   AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
> >   void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void *opaque);
> > 
> > +pcibus_t pci_bar_address(PCIDevice *d,
> > +                         int reg, uint8_t type, pcibus_t size);
> > +
> >   static inline void
> >   pci_set_byte(uint8_t *config, uint8_t val)
> >   {
> > @@ -672,6 +674,11 @@ static inline int pci_is_express(const
> > PCIDevice *d)
> >       return d->cap_present & QEMU_PCI_CAP_EXPRESS;
> >   }
> > 
> > +static inline int pci_is_vf(const PCIDevice *d)
> > +{
> > +    return d->exp.sriov_vf.pf != NULL;
> > +}
> > +
> >   static inline uint32_t pci_config_size(const PCIDevice *d)
> >   {
> >       return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE :
> > PCI_CONFIG_SPACE_SIZE;
> > diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
> > index b48a7a2..b09f79a 100644
> > --- a/include/hw/pci/pcie.h
> > +++ b/include/hw/pci/pcie.h
> > @@ -25,6 +25,7 @@
> >   #include "hw/pci/pci_regs.h"
> >   #include "hw/pci/pcie_regs.h"
> >   #include "hw/pci/pcie_aer.h"
> > +#include "hw/pci/pcie_sriov.h"
> >   #include "hw/hotplug.h"
> > 
> >   typedef enum {
> > @@ -74,6 +75,11 @@ struct PCIExpressDevice {
> >       /* AER */
> >       uint16_t aer_cap;
> >       PCIEAERLog aer_log;
> > +
> > +    /* SR/IOV */
> > +    uint16_t sriov_cap;
> > +    PCIESriovPF sriov_pf;
> > +    PCIESriovVF sriov_vf;
> >   };
> > 
> >   #define COMPAT_PROP_PCP "power_controller_present"
> > diff --git a/include/hw/pci/pcie_sriov.h
> > b/include/hw/pci/pcie_sriov.h
> > new file mode 100644
> > index 0000000..061f5cf
> > --- /dev/null
> > +++ b/include/hw/pci/pcie_sriov.h
> > @@ -0,0 +1,55 @@
> > +/*
> > + * pcie_sriov.h:
> > + *
> > + * Implementation of SR/IOV emulation support.
> > + *
> > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> 
> 2015
> 
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2
> > or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#ifndef QEMU_PCIE_SRIOV_H
> > +#define QEMU_PCIE_SRIOV_H
> > +
> > +
> > +
> > +struct PCIESriovPF {
> > +    uint16_t num_vfs;           /* Number of virtual functions
> > created */
> > +    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for each
> > VF bar */
> > +    const char *vfname;         /* Reference to the device type
> > used for the VFs */
> > +    PCIDevice **vf;             /* Pointer to an array of num_vfs
> > VF devices */
> > +};
> > +
> > +struct PCIESriovVF {
> > +    PCIDevice *pf;              /* Pointer back to owner physical
> > function */
> > +};
> > +
> > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > +                        const char *vfname, uint16_t vf_dev_id,
> > +                        uint16_t init_vfs, uint16_t total_vfs,
> > +                        uint16_t vf_offset, uint16_t vf_stride);
> > +void pcie_sriov_pf_exit(PCIDevice *dev);
> > +
> > +/* Set up a VF bar in the SR/IOV bar area */
> > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > +                               uint8_t type, dma_addr_t size);
> > +
> > +/* Instantiate a bar for a VF */
> > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > +                                MemoryRegion *memory);
> > +
> > +/* Add optional supported page sizes to the mask of supported page
> > sizes
> > + * Page size values are interpreted as opt_sup_pgsize << 12.
> > + */
> > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > opt_sup_pgsize);
> > +
> > +/* SR/IOV capability config write handler */
> > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > +                             uint32_t val, int len);
> > +
> > +/* Reset SR/IOV VF Enable bit to unregister all VFs */
> > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
> > +
> > +#endif /* QEMU_PCIE_SRIOV_H */
> > diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> > index f8a9dd6..fa36b8c 100644
> > --- a/include/qemu/typedefs.h
> > +++ b/include/qemu/typedefs.h
> > @@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
> >   typedef struct PCIEAERErr PCIEAERErr;
> >   typedef struct PCIEAERLog PCIEAERLog;
> >   typedef struct PCIEAERMsg PCIEAERMsg;
> > +typedef struct PCIESriovPF PCIESriovPF;
> > +typedef struct PCIESriovVF PCIESriovVF;
> >   typedef struct PCIEPort PCIEPort;
> >   typedef struct PCIESlot PCIESlot;
> >   typedef struct PCIExpressDevice PCIExpressDevice;
> > 
> 
> 
> I hope I helped,

Definitely, 
thanks for the review!

Knut

> Thanks,
> Marcel
> 

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Marcel Apfelbaum]

On 09/17/2015 05:10 PM, Knut Omang wrote:
> On Thu, 2015-09-17 at 14:49 +0300, Marcel Apfelbaum wrote:
>> On 09/12/2015 03:36 PM, Knut Omang wrote:
>>> This patch provides the building blocks for creating an SR/IOV
>>> PCIe Extended Capability header and register/unregister
>>> SR/IOV Virtual Functions.
>>>
>>> Signed-off-by: Knut Omang <knut.omang@oracle.com>
>>> ---
>>>    hw/pci/Makefile.objs        |   2 +-
>>>    hw/pci/pci.c                |  99 ++++++++++++----
>>>    hw/pci/pcie.c               |   9 +-
>>>    hw/pci/pcie_sriov.c         | 271
>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>    include/hw/pci/pci.h        |  11 +-
>>>    include/hw/pci/pcie.h       |   6 +
>>>    include/hw/pci/pcie_sriov.h |  55 +++++++++
>>>    include/qemu/typedefs.h     |   2 +
>>>    8 files changed, 426 insertions(+), 29 deletions(-)
>>>    create mode 100644 hw/pci/pcie_sriov.c
>>>    create mode 100644 include/hw/pci/pcie_sriov.h
>>>
>>> diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
>>> index 9f905e6..2226980 100644
>>> --- a/hw/pci/Makefile.objs
>>> +++ b/hw/pci/Makefile.objs
>>> @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
>>>    common-obj-$(CONFIG_PCI) += shpc.o
>>>    common-obj-$(CONFIG_PCI) += slotid_cap.o
>>>    common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
>>> -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
>>> +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
>>> pcie_sriov.o
>>>
>>>    common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
>>>    common-obj-$(CONFIG_ALL) += pci-stub.o
>>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
>>> index a5cc015..9c0eba1 100644
>>> --- a/hw/pci/pci.c
>>> +++ b/hw/pci/pci.c
>>> @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
>>>    {
>>>        uint8_t type;
>>>
>>> +    /* PCIe virtual functions do not have their own BARs */
>>> +    assert(!pci_is_vf(d));
>>> +
>>>        if (reg != PCI_ROM_SLOT)
>>>            return PCI_BASE_ADDRESS_0 + reg * 4;
>>>
>>> @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
>>>        }
>>>    }
>>>
>>> -static void pci_do_device_reset(PCIDevice *dev)
>>> +static void pci_reset_regions(PCIDevice *dev)
>>>    {
>>>        int r;
>>> +    if (pci_is_vf(dev)) {
>>> +        return;
>>> +    }
>>>
>>> -    pci_device_deassert_intx(dev);
>>> -    assert(dev->irq_state == 0);
>>> -
>>> -    /* Clear all writable bits */
>>> -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
>>> -                                 pci_get_word(dev->wmask +
>>> PCI_COMMAND) |
>>> -                                 pci_get_word(dev->w1cmask +
>>> PCI_COMMAND));
>>> -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
>>> -                                 pci_get_word(dev->wmask +
>>> PCI_STATUS) |
>>> -                                 pci_get_word(dev->w1cmask +
>>> PCI_STATUS));
>>> -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
>>> -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
>>>        for (r = 0; r < PCI_NUM_REGIONS; ++r) {
>>>            PCIIORegion *region = &dev->io_regions[r];
>>>            if (!region->size) {
>>> @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice
>>> *dev)
>>>                pci_set_long(dev->config + pci_bar(dev, r), region
>>> ->type);
>>>            }
>>>        }
>>> +}
>>> +
>>> +static void pci_do_device_reset(PCIDevice *dev)
>>> +{
>>> +    qdev_reset_all(&dev->qdev);
>>
>> Hi,
>> Thank you for resubmitting this series!
>>
>> This is only a quick look, I hope I'll have more time next week to go
>> over it again.
>>
>>
>>> +
>>> +    dev->irq_state = 0;
>>
>> Are you sure we need the assignment above? It seems that the
>> irq_state
>> should be modified only by the intx wrappers as
>>   pci_device_deassert_intx and so.
>>
>>
>>> +    pci_update_irq_status(dev);
>>
>> Why do we have to update the irq status on reset?
>
> I struggled a lot with interrupts and PF disapperance in earlier
> versions, this may be unintended leftovers from rebase.
>
> The intention was to avoid the qdev removal which caused problems with
> unintended "hot unplug of the PF" at VF removal earlier, but after some
> of the more recent QOM'ification all those problems disappeared.
>
> I tried without these lines and the do_device_reset refactor and it
> seems to work just fine, will fix that in v5, thanks!
>
>>> +    pci_device_deassert_intx(dev);
>>> +    assert(dev->irq_state == 0);
>>> +
>>> +    /* Clear all writable bits */
>>> +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
>>> +                                 pci_get_word(dev->wmask +
>>> PCI_COMMAND) |
>>> +                                 pci_get_word(dev->w1cmask +
>>> PCI_COMMAND));
>>> +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
>>> +                                 pci_get_word(dev->wmask +
>>> PCI_STATUS) |
>>> +                                 pci_get_word(dev->w1cmask +
>>> PCI_STATUS));
>>> +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
>>> +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
>>> +    pci_reset_regions(dev);
>>>        pci_update_mappings(dev);
>>>
>>>        msi_reset(dev);
>>> @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus
>>> *bus, PCIDevice *dev, Error **errp)
>>>            dev->config[PCI_HEADER_TYPE] |=
>>> PCI_HEADER_TYPE_MULTI_FUNCTION;
>>>        }
>>>
>>> +    /* With SR/IOV and ARI, a device at function 0 need not be a
>>> multifunction
>>> +     * device, as it may just be a VF that ended up with function
>>> 0 in
>>> +     * the legacy PCI interpretation. Avoid failing in such cases:
>>> +     */
>>> +    if (pci_is_vf(dev) &&
>>> +        dev->exp.sriov_vf.pf->cap_present &
>>> QEMU_PCI_CAP_MULTIFUNCTION) {
>>> +        return;
>>> +    }
>>> +
>>>        /*
>>>         * multifunction bit is interpreted in two ways as follows.
>>>         *   - all functions must set the bit to 1.
>>> @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int
>>> region_num,
>>>        uint64_t wmask;
>>>        pcibus_t size = memory_region_size(memory);
>>>
>>> +    assert(!pci_is_vf(pci_dev)); /* VFs must use
>>> pcie_sriov_vf_register_bar */
>>>        assert(region_num >= 0);
>>>        assert(region_num < PCI_NUM_REGIONS);
>>>        if (size & (size-1)) {
>>> @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice
>>> *pci_dev, int region_num)
>>>        return pci_dev->io_regions[region_num].addr;
>>>    }
>>>
>>> -static pcibus_t pci_bar_address(PCIDevice *d,
>>> -				int reg, uint8_t type, pcibus_t
>>> size)
>>> +
>>> +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
>>> +                                        uint8_t type, pcibus_t
>>> size)
>>> +{
>>> +    pcibus_t new_addr;
>>> +    if (!pci_is_vf(d)) {
>>> +        int bar = pci_bar(d, reg);
>>> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
>>> +            new_addr = pci_get_quad(d->config + bar);
>>> +        } else {
>>> +            new_addr = pci_get_long(d->config + bar);
>>> +        }
>>> +    } else {
>>> +        PCIDevice *pf = d->exp.sriov_vf.pf;
>>> +        uint16_t sriov_cap = pf->exp.sriov_cap;
>>> +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
>>> +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap +
>>> PCI_SRIOV_VF_OFFSET);
>>> +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap +
>>> PCI_SRIOV_VF_STRIDE);
>>> +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) /
>>> vf_stride;
>>> +
>>> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
>>> +            new_addr = pci_get_quad(pf->config + bar);
>>> +        } else {
>>> +            new_addr = pci_get_long(pf->config + bar);
>>> +        }
>>> +        new_addr += vf_num * size;
>>> +    }
>>> +    if (reg != PCI_ROM_SLOT) {
>>> +        /* Preserve the rom enable bit */
>>> +        new_addr &= ~(size - 1);
>>> +    }
>>> +    return new_addr;
>>> +}
>>> +
>>> +pcibus_t pci_bar_address(PCIDevice *d,
>>> +                         int reg, uint8_t type, pcibus_t size)
>>>    {
>>>        pcibus_t new_addr, last_addr;
>>> -    int bar = pci_bar(d, reg);
>>>        uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
>>>        Object *machine = qdev_get_machine();
>>>        ObjectClass *oc = object_get_class(machine);
>>> @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
>>>            if (!(cmd & PCI_COMMAND_IO)) {
>>>                return PCI_BAR_UNMAPPED;
>>>            }
>>> -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
>>> +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
>>>            last_addr = new_addr + size - 1;
>>>            /* Check if 32 bit BAR wraps around explicitly.
>>>             * TODO: make priorities correct and remove this work
>>> around.
>>> @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice
>>> *d,
>>>        if (!(cmd & PCI_COMMAND_MEMORY)) {
>>>            return PCI_BAR_UNMAPPED;
>>>        }
>>> -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
>>> -        new_addr = pci_get_quad(d->config + bar);
>>> -    } else {
>>> -        new_addr = pci_get_long(d->config + bar);
>>> -    }
>>> +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
>>>        /* the ROM slot has a specific enable bit */
>>>        if (reg == PCI_ROM_SLOT && !(new_addr &
>>> PCI_ROM_ADDRESS_ENABLE)) {
>>>            return PCI_BAR_UNMAPPED;
>>> @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d,
>>> uint32_t addr, uint32_t val_in, int
>>>
>>>        msi_write_config(d, addr, val_in, l);
>>>        msix_write_config(d, addr, val_in, l);
>>> +    pcie_sriov_config_write(d, addr, val_in, l);
>>>    }
>>>
>>>    /***********************************************************/
>>> diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
>>> index 6e28985..ba49c0f 100644
>>> --- a/hw/pci/pcie.c
>>> +++ b/hw/pci/pcie.c
>>> @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler
>>> *hotplug_dev, DeviceState *dev,
>>>         * Right now, only a device of function = 0 is allowed to be
>>>         * hot plugged/unplugged.
>>>         */
>>> -    assert(PCI_FUNC(pci_dev->devfn) == 0);
>>> +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
>>>
>>>        pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
>>>                                   PCI_EXP_SLTSTA_PDS);
>>> @@ -265,10 +265,11 @@ void
>>> pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
>>>                                             DeviceState *dev, Error
>>> **errp)
>>>    {
>>>        uint8_t *exp_cap;
>>> +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
>>>
>>> -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev,
>>> &exp_cap, errp);
>>> +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
>>>
>>> -    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
>>> +    pcie_cap_slot_push_attention_button(pdev);
>>>    }
>>
>> This chunk is not directly liked to the patch, I would put it in a
>> different patch.
>
> ok
>
>>>   /* pci express slot for pci express root/downstream port
>>> @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
>>>        }
>>>
>>>        /*
>>> -     * If the slot is polulated, power indicator is off and power
>>> +     * If the slot is populated, power indicator is off and power
>>>         * controller is off, it is safe to detach the devices.
>>>         */
>>>        if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val &
>>> PCI_EXP_SLTCTL_PCC) &&
>>
>>
>> Same here. I am always happy to have this kind of types taken care
>> of,
>> but I think a separate patch would be cleaner.
>
> Would you like it in the patch set as a separate patch or should I
> schedule it for a trivial patches set instead, maybe together with the
>   PCI_DEVICE thing?

Hi,

I think in the same patch set  would be just fine.

Thanks,
Marcel

>
[...]

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Knut Omang]

On Thu, 2015-09-17 at 16:10 +0200, Knut Omang wrote:
> On Thu, 2015-09-17 at 14:49 +0300, Marcel Apfelbaum wrote:
> > On 09/12/2015 03:36 PM, Knut Omang wrote:
> > > This patch provides the building blocks for creating an SR/IOV
> > > PCIe Extended Capability header and register/unregister
> > > SR/IOV Virtual Functions.
> > > 
> > > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > > ---
> > >   hw/pci/Makefile.objs        |   2 +-
> > >   hw/pci/pci.c                |  99 ++++++++++++----
> > >   hw/pci/pcie.c               |   9 +-
> > >   hw/pci/pcie_sriov.c         | 271
> > > ++++++++++++++++++++++++++++++++++++++++++++
> > >   include/hw/pci/pci.h        |  11 +-
> > >   include/hw/pci/pcie.h       |   6 +
> > >   include/hw/pci/pcie_sriov.h |  55 +++++++++
> > >   include/qemu/typedefs.h     |   2 +
> > >   8 files changed, 426 insertions(+), 29 deletions(-)
> > >   create mode 100644 hw/pci/pcie_sriov.c
> > >   create mode 100644 include/hw/pci/pcie_sriov.h
> > > 
> > > diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
> > > index 9f905e6..2226980 100644
> > > --- a/hw/pci/Makefile.objs
> > > +++ b/hw/pci/Makefile.objs
> > > @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
> > >   common-obj-$(CONFIG_PCI) += shpc.o
> > >   common-obj-$(CONFIG_PCI) += slotid_cap.o
> > >   common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
> > > -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > > +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > > pcie_sriov.o
> > > 
> > >   common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
> > >   common-obj-$(CONFIG_ALL) += pci-stub.o
> > > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > > index a5cc015..9c0eba1 100644
> > > --- a/hw/pci/pci.c
> > > +++ b/hw/pci/pci.c
> > > @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
> > >   {
> > >       uint8_t type;
> > > 
> > > +    /* PCIe virtual functions do not have their own BARs */
> > > +    assert(!pci_is_vf(d));
> > > +
> > >       if (reg != PCI_ROM_SLOT)
> > >           return PCI_BASE_ADDRESS_0 + reg * 4;
> > > 
> > > @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice
> > > *dev)
> > >       }
> > >   }
> > > 
> > > -static void pci_do_device_reset(PCIDevice *dev)
> > > +static void pci_reset_regions(PCIDevice *dev)
> > >   {
> > >       int r;
> > > +    if (pci_is_vf(dev)) {
> > > +        return;
> > > +    }
> > > 
> > > -    pci_device_deassert_intx(dev);
> > > -    assert(dev->irq_state == 0);
> > > -
> > > -    /* Clear all writable bits */
> > > -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > > -                                 pci_get_word(dev->wmask +
> > > PCI_COMMAND) |
> > > -                                 pci_get_word(dev->w1cmask +
> > > PCI_COMMAND));
> > > -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > > -                                 pci_get_word(dev->wmask +
> > > PCI_STATUS) |
> > > -                                 pci_get_word(dev->w1cmask +
> > > PCI_STATUS));
> > > -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > > -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> > >       for (r = 0; r < PCI_NUM_REGIONS; ++r) {
> > >           PCIIORegion *region = &dev->io_regions[r];
> > >           if (!region->size) {
> > > @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice
> > > *dev)
> > >               pci_set_long(dev->config + pci_bar(dev, r), region
> > > ->type);
> > >           }
> > >       }
> > > +}
> > > +
> > > +static void pci_do_device_reset(PCIDevice *dev)
> > > +{
> > > +    qdev_reset_all(&dev->qdev);
> > 
> > Hi,
> > Thank you for resubmitting this series!
> > 
> > This is only a quick look, I hope I'll have more time next week to
> > go
> > over it again.
> > 
> > 
> > > +
> > > +    dev->irq_state = 0;
> > 
> > Are you sure we need the assignment above? It seems that the
> > irq_state
> > should be modified only by the intx wrappers as 
> >  pci_device_deassert_intx and so.
> > 
> > 
> > > +    pci_update_irq_status(dev);
> > 
> > Why do we have to update the irq status on reset?
> 
> I struggled a lot with interrupts and PF disapperance in earlier
> versions, this may be unintended leftovers from rebase.
> 
> The intention was to avoid the qdev removal which caused problems
> with
> unintended "hot unplug of the PF" at VF removal earlier, but after
> some
> of the more recent QOM'ification all those problems disappeared.
> 
> I tried without these lines and the do_device_reset refactor and it
> seems to work just fine, will fix that in v5, thanks!
> 
> > > +    pci_device_deassert_intx(dev);
> > > +    assert(dev->irq_state == 0);
> > > +
> > > +    /* Clear all writable bits */
> > > +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > > +                                 pci_get_word(dev->wmask +
> > > PCI_COMMAND) |
> > > +                                 pci_get_word(dev->w1cmask +
> > > PCI_COMMAND));
> > > +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > > +                                 pci_get_word(dev->wmask +
> > > PCI_STATUS) |
> > > +                                 pci_get_word(dev->w1cmask +
> > > PCI_STATUS));
> > > +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > > +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> > > +    pci_reset_regions(dev);
> > >       pci_update_mappings(dev);
> > > 
> > >       msi_reset(dev);
> > > @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus
> > > *bus, PCIDevice *dev, Error **errp)
> > >           dev->config[PCI_HEADER_TYPE] |=
> > > PCI_HEADER_TYPE_MULTI_FUNCTION;
> > >       }
> > > 
> > > +    /* With SR/IOV and ARI, a device at function 0 need not be a
> > > multifunction
> > > +     * device, as it may just be a VF that ended up with
> > > function
> > > 0 in
> > > +     * the legacy PCI interpretation. Avoid failing in such
> > > cases:
> > > +     */
> > > +    if (pci_is_vf(dev) &&
> > > +        dev->exp.sriov_vf.pf->cap_present &
> > > QEMU_PCI_CAP_MULTIFUNCTION) {
> > > +        return;
> > > +    }
> > > +
> > >       /*
> > >        * multifunction bit is interpreted in two ways as follows.
> > >        *   - all functions must set the bit to 1.
> > > @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int
> > > region_num,
> > >       uint64_t wmask;
> > >       pcibus_t size = memory_region_size(memory);
> > > 
> > > +    assert(!pci_is_vf(pci_dev)); /* VFs must use
> > > pcie_sriov_vf_register_bar */
> > >       assert(region_num >= 0);
> > >       assert(region_num < PCI_NUM_REGIONS);
> > >       if (size & (size-1)) {
> > > @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice
> > > *pci_dev, int region_num)
> > >       return pci_dev->io_regions[region_num].addr;
> > >   }
> > > 
> > > -static pcibus_t pci_bar_address(PCIDevice *d,
> > > -				int reg, uint8_t type, pcibus_t
> > > size)
> > > +
> > > +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
> > > +                                        uint8_t type, pcibus_t
> > > size)
> > > +{
> > > +    pcibus_t new_addr;
> > > +    if (!pci_is_vf(d)) {
> > > +        int bar = pci_bar(d, reg);
> > > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > > +            new_addr = pci_get_quad(d->config + bar);
> > > +        } else {
> > > +            new_addr = pci_get_long(d->config + bar);
> > > +        }
> > > +    } else {
> > > +        PCIDevice *pf = d->exp.sriov_vf.pf;
> > > +        uint16_t sriov_cap = pf->exp.sriov_cap;
> > > +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
> > > +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap
> > > +
> > > PCI_SRIOV_VF_OFFSET);
> > > +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap
> > > +
> > > PCI_SRIOV_VF_STRIDE);
> > > +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) /
> > > vf_stride;
> > > +
> > > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > > +            new_addr = pci_get_quad(pf->config + bar);
> > > +        } else {
> > > +            new_addr = pci_get_long(pf->config + bar);
> > > +        }
> > > +        new_addr += vf_num * size;
> > > +    }
> > > +    if (reg != PCI_ROM_SLOT) {
> > > +        /* Preserve the rom enable bit */
> > > +        new_addr &= ~(size - 1);
> > > +    }
> > > +    return new_addr;
> > > +}
> > > +
> > > +pcibus_t pci_bar_address(PCIDevice *d,
> > > +                         int reg, uint8_t type, pcibus_t size)
> > >   {
> > >       pcibus_t new_addr, last_addr;
> > > -    int bar = pci_bar(d, reg);
> > >       uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
> > >       Object *machine = qdev_get_machine();
> > >       ObjectClass *oc = object_get_class(machine);
> > > @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice
> > > *d,
> > >           if (!(cmd & PCI_COMMAND_IO)) {
> > >               return PCI_BAR_UNMAPPED;
> > >           }
> > > -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
> > > +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
> > >           last_addr = new_addr + size - 1;
> > >           /* Check if 32 bit BAR wraps around explicitly.
> > >            * TODO: make priorities correct and remove this work
> > > around.
> > > @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice
> > > *d,
> > >       if (!(cmd & PCI_COMMAND_MEMORY)) {
> > >           return PCI_BAR_UNMAPPED;
> > >       }
> > > -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > > -        new_addr = pci_get_quad(d->config + bar);
> > > -    } else {
> > > -        new_addr = pci_get_long(d->config + bar);
> > > -    }
> > > +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
> > >       /* the ROM slot has a specific enable bit */
> > >       if (reg == PCI_ROM_SLOT && !(new_addr &
> > > PCI_ROM_ADDRESS_ENABLE)) {
> > >           return PCI_BAR_UNMAPPED;
> > > @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d,
> > > uint32_t addr, uint32_t val_in, int
> > > 
> > >       msi_write_config(d, addr, val_in, l);
> > >       msix_write_config(d, addr, val_in, l);
> > > +    pcie_sriov_config_write(d, addr, val_in, l);
> > >   }
> > > 
> > >   /***********************************************************/
> > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
> > > index 6e28985..ba49c0f 100644
> > > --- a/hw/pci/pcie.c
> > > +++ b/hw/pci/pcie.c
> > > @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler
> > > *hotplug_dev, DeviceState *dev,
> > >        * Right now, only a device of function = 0 is allowed to
> > > be
> > >        * hot plugged/unplugged.
> > >        */
> > > -    assert(PCI_FUNC(pci_dev->devfn) == 0);
> > > +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
> > > 
> > >       pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
> > >                                  PCI_EXP_SLTSTA_PDS);
> > > @@ -265,10 +265,11 @@ void
> > > pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
> > >                                            DeviceState *dev,
> > > Error
> > > **errp)
> > >   {
> > >       uint8_t *exp_cap;
> > > +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
> > > 
> > > -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev,
> > > &exp_cap, errp);
> > > +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
> > > 
> > > -   
> > >  pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
> > > +    pcie_cap_slot_push_attention_button(pdev);
> > >   }
> > 
> > This chunk is not directly liked to the patch, I would put it in a
> > different patch.
> 
> ok
> 
> > >  /* pci express slot for pci express root/downstream port
> > > @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice
> > > *dev,
> > >       }
> > > 
> > >       /*
> > > -     * If the slot is polulated, power indicator is off and
> > > power
> > > +     * If the slot is populated, power indicator is off and
> > > power
> > >        * controller is off, it is safe to detach the devices.
> > >        */
> > >       if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val &
> > > PCI_EXP_SLTCTL_PCC) &&
> > 
> > 
> > Same here. I am always happy to have this kind of types taken care
> > of,
> > but I think a separate patch would be cleaner.
> 
> Would you like it in the patch set as a separate patch or should I
> schedule it for a trivial patches set instead, maybe together with
> the
>  PCI_DEVICE thing?
> 
> > > diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
> > > new file mode 100644
> > > index 0000000..14f31cc
> > > --- /dev/null
> > > +++ b/hw/pci/pcie_sriov.c
> > > @@ -0,0 +1,271 @@
> > > +/*
> > > + * pcie_sriov.c:
> > > + *
> > > + * Implementation of SR/IOV emulation support.
> > > + *
> > > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> > 
> > 2015 :)
> 
> thanks,
> 
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL, version
> > > 2
> > > or later.
> > > + * See the COPYING file in the top-level directory.
> > > + *
> > > + */
> > > +
> > > +#include "hw/pci/pci.h"
> > > +#include "hw/pci/pcie.h"
> > > +#include "hw/pci/pci_bus.h"
> > > +#include "qemu/range.h"
> > > +
> > > +//#define DEBUG_SRIOV
> > > +#ifdef DEBUG_SRIOV
> > > +# define SRIOV_DPRINTF(fmt, ...)                                
> > >   
> > >        \
> > > +    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ##
> > > __VA_ARGS__)
> > > +#else
> > > +# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
> > > +#endif
> > > +#define SRIOV_DEV_PRINTF(dev, fmt, ...)                         
> > >   
> > >        \
> > > +    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ##
> > > __VA_ARGS__)
> > > +
> > > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const
> > > char
> > > *name);
> > > +static void unregister_vfs(PCIDevice *dev);
> > > +
> > > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > > +                        const char *vfname, uint16_t vf_dev_id,
> > > +                        uint16_t init_vfs, uint16_t total_vfs,
> > > +                        uint16_t vf_offset, uint16_t vf_stride)
> > > +{
> > > +    uint8_t *cfg = dev->config + offset;
> > > +    uint8_t *wmask;
> > > +
> > > +    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
> > > +                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
> > > +    dev->exp.sriov_cap = offset;
> > > +    dev->exp.sriov_pf.num_vfs = 0;
> > > +    dev->exp.sriov_pf.vfname = g_strdup(vfname);
> > > +    dev->exp.sriov_pf.vf = NULL;
> > > +
> > > +    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
> > > +    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
> > > +
> > > +    /* Minimal page size support values required by the SR/IOV
> > > standard:
> > > +     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
> > > +     * Hard coded for simplicity in this version
> > > +     */
> > 
> > A define (with the above very good explanation) would be preferred.
> 
> Will do
> 
> > > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
> > > +
> > > +    /* Default is to use 4K pages, software can modify it
> > > +     * to any of the supported bits
> > > +     */
> > > +    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
> > > +
> > > +    /* Set up device ID and initial/total number of VFs
> > > available
> > > */
> > > +    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
> > > +    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
> > > +    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
> > > +    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
> > > +
> > > +    /* Write enable control bits */
> > > +    wmask = dev->wmask + offset;
> > > +    pci_set_word(wmask + PCI_SRIOV_CTRL,
> > > +                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE |
> > > PCI_SRIOV_CTRL_ARI);
> > > +    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
> > > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
> > > +
> > > +    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
> > > +}
> > > +
> > > +void pcie_sriov_pf_exit(PCIDevice *dev)
> > > +{
> > > +    SRIOV_DPRINTF("\n");
> > > +    unregister_vfs(dev);
> > > +    g_free((char *)dev->exp.sriov_pf.vfname);
> > > +    dev->exp.sriov_pf.vfname = NULL;
> > > +}
> > > +
> > > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > > +                               uint8_t type, dma_addr_t size)
> > > +{
> > > +    uint32_t addr;
> > > +    uint64_t wmask;
> > > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > > +
> > > +    assert(sriov_cap > 0);
> > > +    assert(region_num >= 0);
> > > +    assert(region_num < PCI_NUM_REGIONS);
> > > +    assert(region_num != PCI_ROM_SLOT);
> > > +
> > > +    wmask = ~(size - 1);
> > > +    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
> > > +
> > > +    pci_set_long(dev->config + addr, type);
> > > +    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
> > > +        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > > +        pci_set_quad(dev->wmask + addr, wmask);
> > > +        pci_set_quad(dev->cmask + addr, ~0ULL);
> > > +    } else {
> > > +        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
> > > +        pci_set_long(dev->cmask + addr, 0xffffffff);
> > > +    }
> > > +    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
> > > +}
> > > +
> > > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > > +                                MemoryRegion *memory)
> > > +{
> > > +    PCIIORegion *r;
> > > +    uint8_t type;
> > > +    pcibus_t size = memory_region_size(memory);
> > > +
> > > +    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
> > > +    assert(region_num >= 0);
> > > +    assert(region_num < PCI_NUM_REGIONS);
> > > +    type = dev->exp.sriov_vf.pf
> > > ->exp.sriov_pf.vf_bar_type[region_num];
> > > +
> > > +    if (size & (size-1)) {
> > 
> > We already have a  is_power_of_2 func defined in include/qemu/host
> > -utils.h
> 
> will fix,
> 
> > > +        fprintf(stderr, "ERROR: PCI region size must be a power"
> > > +                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n",
> > > type, size);
> > 
> > Maybe we should error_report instead of fprintf
> 
> Leftover from far too many rebases, will fix,
> 
> > > +        exit(1);
> > > +    }
> > > +
> > > +    r = &dev->io_regions[region_num];
> > > +    r->memory = memory;
> > > +    r->address_space =
> > > +        type & PCI_BASE_ADDRESS_SPACE_IO
> > > +        ? dev->bus->address_space_io
> > > +        : dev->bus->address_space_mem;
> > > +    r->size = size;
> > > +    r->type = type;
> > > +
> > > +    r->addr = pci_bar_address(dev, region_num, r->type, r
> > > ->size);
> > > +    if (r->addr != PCI_BAR_UNMAPPED) {
> > > +        memory_region_add_subregion_overlap(r->address_space,
> > > +                                            r->addr, r->memory,
> > > 1);
> > > +    }
> > > +}
> > > +
> > > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const
> > > char
> > > *name)
> > > +{
> > > +    PCIDevice *dev = pci_create(pf->bus, devfn, name);
> > > +    dev->exp.sriov_vf.pf = pf;
> > > +    Error *local_err = NULL;
> > > +
> > > +    object_property_set_bool(OBJECT(&dev->qdev), true,
> > > "realized",
> > > &local_err);
> > > +    if (local_err) {
> > > +        fprintf(stderr, "Failed to enable: %s\n",
> > > +                error_get_pretty(local_err));
> > > +        error_free(local_err);
> > 
> > and error_report_err here
> 
> ok
> 
> > > +        return NULL;
> > > +    }
> > > +
> > > +    /* set vid/did according to sr/iov spec - they are not used
> > > */
> > > +    pci_config_set_vendor_id(dev->config, 0xffff);
> > > +    pci_config_set_device_id(dev->config, 0xffff);
> > > +    return dev;
> > > +}
> > > +
> > > +static void register_vfs(PCIDevice *dev)
> > > +{
> > > +    uint16_t num_vfs;
> > > +    uint16_t i;
> > > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > > +    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap +
> > > PCI_SRIOV_VF_OFFSET);
> > > +    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap +
> > > PCI_SRIOV_VF_STRIDE);
> > > +    int32_t devfn = dev->devfn + vf_offset;
> > > +
> > > +    assert(sriov_cap > 0);
> > > +    num_vfs = pci_get_word(dev->config + sriov_cap +
> > > PCI_SRIOV_NUM_VF);
> > > +
> > > +    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) *
> > > num_vfs);
> > > +    assert(dev->exp.sriov_pf.vf);
> > > +
> > > +    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
> > > +    for (i = 0; i < num_vfs; i++) {
> > > +        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev
> > > ->exp.sriov_pf.vfname);
> > > +        if (!dev->exp.sriov_pf.vf[i]) {
> > > +            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at
> > > devfn
> > > %x.%x\n", i,
> > > +                             PCI_SLOT(devfn), PCI_FUNC(devfn));
> > > +            num_vfs = i;
> > > +            break;
> > > +        }
> > > +        devfn += vf_stride;
> > > +    }
> > > +    dev->exp.sriov_pf.num_vfs = num_vfs;
> > > +}
> > > +
> > > +static void unregister_vfs(PCIDevice *dev)
> > > +{
> > > +    Error *local_err = NULL;
> > > +    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
> > > +    uint16_t i;
> > > +    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n",
> > > num_vfs);
> > > +    for (i = 0; i < num_vfs; i++) {
> > > +        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]
> > > ->qdev), false, "realized", &local_err);
> > > +        if (local_err) {
> > > +            fprintf(stderr, "Failed to unplug: %s\n",
> > > +                    error_get_pretty(local_err));
> > > +            error_free(local_err);
> > 
> > error_report_err
> > 
> > > +        }
> > > +    }
> > > +    g_free(dev->exp.sriov_pf.vf);
> > > +    dev->exp.sriov_pf.vf = NULL;
> > > +    dev->exp.sriov_pf.num_vfs = 0;
> > > +}
> > > +
> > > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > > uint32_t val, int len)
> > > +{
> > > +    uint32_t off;
> > > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > > +
> > > +    if (!sriov_cap || address < sriov_cap) {
> > > +        return;
> > > +    }
> > > +    off = address - sriov_cap;
> > > +    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
> > > +        return;
> > > +    }
> > > +
> > > +    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val
> > > 0x%x
> > > len %d\n",
> > > +                 sriov_cap, off, val, len);
> > > +
> > > +    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
> > > +        if (dev->exp.sriov_pf.num_vfs) {
> > > +            if (!(val & PCI_SRIOV_CTRL_VFE)) {
> > > +                unregister_vfs(dev);
> > > +            }
> > > +        } else {
> > > +            if (val & PCI_SRIOV_CTRL_VFE) {
> > > +                register_vfs(dev);
> > > +            }
> > > +        }
> > > +    }
> > > +}
> > > +
> > > +
> > > +/* Reset SR/IOV VF Enable bit to trigger an unregister of all
> > > VFs
> > > */
> > > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
> > > +{
> > > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > > +    if (sriov_cap) {
> > > +        uint32_t val = pci_get_byte(dev->config + sriov_cap +
> > > PCI_SRIOV_CTRL);
> > > +        if (val & PCI_SRIOV_CTRL_VFE) {
> > > +            val &= ~PCI_SRIOV_CTRL_VFE;
> > > +            pcie_sriov_config_write(dev, sriov_cap +
> > > PCI_SRIOV_CTRL, val, 1);
> > > +        }
> > > +    }
> > > +}
> > > +
> > > +/* Add optional supported page sizes to the mask of supported
> > > page
> > > sizes */
> > > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > > opt_sup_pgsize)
> > > +{
> > > +    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
> > > +    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
> > > +
> > > +    uint16_t sup_pgsize = pci_get_word(cfg +
> > > PCI_SRIOV_SUP_PGSIZE);
> > > +
> > > +    sup_pgsize |= opt_sup_pgsize;
> > > +
> > > +    /* Make sure the new bits are set, and that system page size
> > > +     * also can be set to any of the new values according to
> > > spec:
> > > +     */
> > > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
> > > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
> > > +}
> > > diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> > > index 551cb3d..2e9d8ba 100644
> > > --- a/include/hw/pci/pci.h
> > > +++ b/include/hw/pci/pci.h
> > > @@ -11,8 +11,6 @@
> > >   /* PCI includes legacy ISA access.  */
> > >   #include "hw/isa/isa.h"
> > > 
> > > -#include "hw/pci/pcie.h"
> > > -
> > >   /* PCI bus */
> > > 
> > >   #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) |
> > > ((func)
> > > & 0x07))
> > > @@ -132,6 +130,7 @@ enum {
> > >   #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
> > > 
> > >   #include "hw/pci/pci_regs.h"
> > > +#include "hw/pci/pcie.h"
> > 
> > Why is it moved here?
> 
> Probably an unintended side effect of rebasing..

Actually it is due to the need for PCI_NUM_REGIONS (declared above in
pci.h now) to declare the PCIESriovPF data type in pcie_sriov.h, which
I think should naturally be included by pcie.h similar to pcie_aer.h

Knut

> 
> > >   /* PCI HEADER_TYPE */
> > >   #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
> > > @@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus
> > > *,
> > > void *, int);
> > >   AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
> > >   void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void
> > > *opaque);
> > > 
> > > +pcibus_t pci_bar_address(PCIDevice *d,
> > > +                         int reg, uint8_t type, pcibus_t size);
> > > +
> > >   static inline void
> > >   pci_set_byte(uint8_t *config, uint8_t val)
> > >   {
> > > @@ -672,6 +674,11 @@ static inline int pci_is_express(const
> > > PCIDevice *d)
> > >       return d->cap_present & QEMU_PCI_CAP_EXPRESS;
> > >   }
> > > 
> > > +static inline int pci_is_vf(const PCIDevice *d)
> > > +{
> > > +    return d->exp.sriov_vf.pf != NULL;
> > > +}
> > > +
> > >   static inline uint32_t pci_config_size(const PCIDevice *d)
> > >   {
> > >       return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE :
> > > PCI_CONFIG_SPACE_SIZE;
> > > diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
> > > index b48a7a2..b09f79a 100644
> > > --- a/include/hw/pci/pcie.h
> > > +++ b/include/hw/pci/pcie.h
> > > @@ -25,6 +25,7 @@
> > >   #include "hw/pci/pci_regs.h"
> > >   #include "hw/pci/pcie_regs.h"
> > >   #include "hw/pci/pcie_aer.h"
> > > +#include "hw/pci/pcie_sriov.h"
> > >   #include "hw/hotplug.h"
> > > 
> > >   typedef enum {
> > > @@ -74,6 +75,11 @@ struct PCIExpressDevice {
> > >       /* AER */
> > >       uint16_t aer_cap;
> > >       PCIEAERLog aer_log;
> > > +
> > > +    /* SR/IOV */
> > > +    uint16_t sriov_cap;
> > > +    PCIESriovPF sriov_pf;
> > > +    PCIESriovVF sriov_vf;
> > >   };
> > > 
> > >   #define COMPAT_PROP_PCP "power_controller_present"
> > > diff --git a/include/hw/pci/pcie_sriov.h
> > > b/include/hw/pci/pcie_sriov.h
> > > new file mode 100644
> > > index 0000000..061f5cf
> > > --- /dev/null
> > > +++ b/include/hw/pci/pcie_sriov.h
> > > @@ -0,0 +1,55 @@
> > > +/*
> > > + * pcie_sriov.h:
> > > + *
> > > + * Implementation of SR/IOV emulation support.
> > > + *
> > > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> > 
> > 2015
> > 
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL, version
> > > 2
> > > or later.
> > > + * See the COPYING file in the top-level directory.
> > > + *
> > > + */
> > > +
> > > +#ifndef QEMU_PCIE_SRIOV_H
> > > +#define QEMU_PCIE_SRIOV_H
> > > +
> > > +
> > > +
> > > +struct PCIESriovPF {
> > > +    uint16_t num_vfs;           /* Number of virtual functions
> > > created */
> > > +    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for
> > > each
> > > VF bar */
> > > +    const char *vfname;         /* Reference to the device type
> > > used for the VFs */
> > > +    PCIDevice **vf;             /* Pointer to an array of
> > > num_vfs
> > > VF devices */
> > > +};
> > > +
> > > +struct PCIESriovVF {
> > > +    PCIDevice *pf;              /* Pointer back to owner
> > > physical
> > > function */
> > > +};
> > > +
> > > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > > +                        const char *vfname, uint16_t vf_dev_id,
> > > +                        uint16_t init_vfs, uint16_t total_vfs,
> > > +                        uint16_t vf_offset, uint16_t vf_stride);
> > > +void pcie_sriov_pf_exit(PCIDevice *dev);
> > > +
> > > +/* Set up a VF bar in the SR/IOV bar area */
> > > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > > +                               uint8_t type, dma_addr_t size);
> > > +
> > > +/* Instantiate a bar for a VF */
> > > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > > +                                MemoryRegion *memory);
> > > +
> > > +/* Add optional supported page sizes to the mask of supported
> > > page
> > > sizes
> > > + * Page size values are interpreted as opt_sup_pgsize << 12.
> > > + */
> > > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > > opt_sup_pgsize);
> > > +
> > > +/* SR/IOV capability config write handler */
> > > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > > +                             uint32_t val, int len);
> > > +
> > > +/* Reset SR/IOV VF Enable bit to unregister all VFs */
> > > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
> > > +
> > > +#endif /* QEMU_PCIE_SRIOV_H */
> > > diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> > > index f8a9dd6..fa36b8c 100644
> > > --- a/include/qemu/typedefs.h
> > > +++ b/include/qemu/typedefs.h
> > > @@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
> > >   typedef struct PCIEAERErr PCIEAERErr;
> > >   typedef struct PCIEAERLog PCIEAERLog;
> > >   typedef struct PCIEAERMsg PCIEAERMsg;
> > > +typedef struct PCIESriovPF PCIESriovPF;
> > > +typedef struct PCIESriovVF PCIESriovVF;
> > >   typedef struct PCIEPort PCIEPort;
> > >   typedef struct PCIESlot PCIESlot;
> > >   typedef struct PCIExpressDevice PCIExpressDevice;
> > > 
> > 
> > 
> > I hope I helped,
> 
> Definitely, 
> thanks for the review!
> 
> Knut
> 
> > Thanks,
> > Marcel
> > 
> 

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Marcel Apfelbaum]

On 09/12/2015 03:36 PM, Knut Omang wrote:
> This patch provides the building blocks for creating an SR/IOV
> PCIe Extended Capability header and register/unregister
> SR/IOV Virtual Functions.
>
> Signed-off-by: Knut Omang <knut.omang@oracle.com>
> ---
>   hw/pci/Makefile.objs        |   2 +-
>   hw/pci/pci.c                |  99 ++++++++++++----
>   hw/pci/pcie.c               |   9 +-
>   hw/pci/pcie_sriov.c         | 271 ++++++++++++++++++++++++++++++++++++++++++++
>   include/hw/pci/pci.h        |  11 +-
>   include/hw/pci/pcie.h       |   6 +
>   include/hw/pci/pcie_sriov.h |  55 +++++++++
>   include/qemu/typedefs.h     |   2 +
>   8 files changed, 426 insertions(+), 29 deletions(-)
>   create mode 100644 hw/pci/pcie_sriov.c
>   create mode 100644 include/hw/pci/pcie_sriov.h
>
> diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
> index 9f905e6..2226980 100644
> --- a/hw/pci/Makefile.objs
> +++ b/hw/pci/Makefile.objs
> @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
>   common-obj-$(CONFIG_PCI) += shpc.o
>   common-obj-$(CONFIG_PCI) += slotid_cap.o
>   common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
> -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o pcie_sriov.o
>
>   common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
>   common-obj-$(CONFIG_ALL) += pci-stub.o
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index a5cc015..9c0eba1 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
>   {
>       uint8_t type;
>
> +    /* PCIe virtual functions do not have their own BARs */
> +    assert(!pci_is_vf(d));
> +
>       if (reg != PCI_ROM_SLOT)
>           return PCI_BASE_ADDRESS_0 + reg * 4;
>
> @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
>       }
>   }
>
> -static void pci_do_device_reset(PCIDevice *dev)
> +static void pci_reset_regions(PCIDevice *dev)
>   {
>       int r;
> +    if (pci_is_vf(dev)) {
> +        return;
> +    }
>
> -    pci_device_deassert_intx(dev);
> -    assert(dev->irq_state == 0);
> -
> -    /* Clear all writable bits */
> -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> -                                 pci_get_word(dev->wmask + PCI_COMMAND) |
> -                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
> -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> -                                 pci_get_word(dev->wmask + PCI_STATUS) |
> -                                 pci_get_word(dev->w1cmask + PCI_STATUS));
> -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
>       for (r = 0; r < PCI_NUM_REGIONS; ++r) {
>           PCIIORegion *region = &dev->io_regions[r];
>           if (!region->size) {
> @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice *dev)
>               pci_set_long(dev->config + pci_bar(dev, r), region->type);
>           }
>       }
> +}
> +
> +static void pci_do_device_reset(PCIDevice *dev)
> +{
> +    qdev_reset_all(&dev->qdev);
> +
> +    dev->irq_state = 0;
> +    pci_update_irq_status(dev);
> +    pci_device_deassert_intx(dev);
> +    assert(dev->irq_state == 0);
> +
> +    /* Clear all writable bits */
> +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> +                                 pci_get_word(dev->wmask + PCI_COMMAND) |
> +                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
> +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> +                                 pci_get_word(dev->wmask + PCI_STATUS) |
> +                                 pci_get_word(dev->w1cmask + PCI_STATUS));
> +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> +    pci_reset_regions(dev);
>       pci_update_mappings(dev);
>
>       msi_reset(dev);
> @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus *bus, PCIDevice *dev, Error **errp)
>           dev->config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION;
>       }
>
> +    /* With SR/IOV and ARI, a device at function 0 need not be a multifunction
> +     * device, as it may just be a VF that ended up with function 0 in
> +     * the legacy PCI interpretation. Avoid failing in such cases:
> +     */
> +    if (pci_is_vf(dev) &&
> +        dev->exp.sriov_vf.pf->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) {
> +        return;
> +    }
> +
>       /*
>        * multifunction bit is interpreted in two ways as follows.
>        *   - all functions must set the bit to 1.
> @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num,
>       uint64_t wmask;
>       pcibus_t size = memory_region_size(memory);
>
> +    assert(!pci_is_vf(pci_dev)); /* VFs must use pcie_sriov_vf_register_bar */
>       assert(region_num >= 0);
>       assert(region_num < PCI_NUM_REGIONS);
>       if (size & (size-1)) {
> @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num)
>       return pci_dev->io_regions[region_num].addr;
>   }
>
> -static pcibus_t pci_bar_address(PCIDevice *d,
> -				int reg, uint8_t type, pcibus_t size)
> +
> +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
> +                                        uint8_t type, pcibus_t size)
> +{
> +    pcibus_t new_addr;
> +    if (!pci_is_vf(d)) {
> +        int bar = pci_bar(d, reg);
> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +            new_addr = pci_get_quad(d->config + bar);
> +        } else {
> +            new_addr = pci_get_long(d->config + bar);
> +        }
> +    } else {
> +        PCIDevice *pf = d->exp.sriov_vf.pf;
> +        uint16_t sriov_cap = pf->exp.sriov_cap;
> +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
> +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
> +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
> +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) / vf_stride;
> +
> +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +            new_addr = pci_get_quad(pf->config + bar);
> +        } else {
> +            new_addr = pci_get_long(pf->config + bar);
> +        }
> +        new_addr += vf_num * size;
> +    }
> +    if (reg != PCI_ROM_SLOT) {
> +        /* Preserve the rom enable bit */
> +        new_addr &= ~(size - 1);
> +    }
> +    return new_addr;
> +}
> +
> +pcibus_t pci_bar_address(PCIDevice *d,
> +                         int reg, uint8_t type, pcibus_t size)
>   {
>       pcibus_t new_addr, last_addr;
> -    int bar = pci_bar(d, reg);
>       uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
>       Object *machine = qdev_get_machine();
>       ObjectClass *oc = object_get_class(machine);
> @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
>           if (!(cmd & PCI_COMMAND_IO)) {
>               return PCI_BAR_UNMAPPED;
>           }
> -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
> +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
>           last_addr = new_addr + size - 1;
>           /* Check if 32 bit BAR wraps around explicitly.
>            * TODO: make priorities correct and remove this work around.
> @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
>       if (!(cmd & PCI_COMMAND_MEMORY)) {
>           return PCI_BAR_UNMAPPED;
>       }
> -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> -        new_addr = pci_get_quad(d->config + bar);
> -    } else {
> -        new_addr = pci_get_long(d->config + bar);
> -    }
> +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
>       /* the ROM slot has a specific enable bit */
>       if (reg == PCI_ROM_SLOT && !(new_addr & PCI_ROM_ADDRESS_ENABLE)) {
>           return PCI_BAR_UNMAPPED;
> @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val_in, int
>
>       msi_write_config(d, addr, val_in, l);
>       msix_write_config(d, addr, val_in, l);
> +    pcie_sriov_config_write(d, addr, val_in, l);
>   }
>
>   /***********************************************************/
> diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
> index 6e28985..ba49c0f 100644
> --- a/hw/pci/pcie.c
> +++ b/hw/pci/pcie.c
> @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
>        * Right now, only a device of function = 0 is allowed to be
>        * hot plugged/unplugged.
>        */
> -    assert(PCI_FUNC(pci_dev->devfn) == 0);
> +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
>
>       pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
>                                  PCI_EXP_SLTSTA_PDS);
> @@ -265,10 +265,11 @@ void pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
>                                            DeviceState *dev, Error **errp)
>   {
>       uint8_t *exp_cap;
> +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
>
> -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev, &exp_cap, errp);
> +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
>
> -    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
> +    pcie_cap_slot_push_attention_button(pdev);
>   }
>
>   /* pci express slot for pci express root/downstream port
> @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
>       }
>
>       /*
> -     * If the slot is polulated, power indicator is off and power
> +     * If the slot is populated, power indicator is off and power
>        * controller is off, it is safe to detach the devices.
>        */
>       if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val & PCI_EXP_SLTCTL_PCC) &&
> diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
> new file mode 100644
> index 0000000..14f31cc
> --- /dev/null
> +++ b/hw/pci/pcie_sriov.c
> @@ -0,0 +1,271 @@
> +/*
> + * pcie_sriov.c:
> + *
> + * Implementation of SR/IOV emulation support.
> + *
> + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "hw/pci/pci.h"
> +#include "hw/pci/pcie.h"
> +#include "hw/pci/pci_bus.h"
> +#include "qemu/range.h"
> +
> +//#define DEBUG_SRIOV
> +#ifdef DEBUG_SRIOV
> +# define SRIOV_DPRINTF(fmt, ...)                                         \
> +    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ## __VA_ARGS__)
> +#else
> +# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
> +#endif
> +#define SRIOV_DEV_PRINTF(dev, fmt, ...)                                  \
> +    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ## __VA_ARGS__)
> +
> +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name);
> +static void unregister_vfs(PCIDevice *dev);
> +
> +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> +                        const char *vfname, uint16_t vf_dev_id,
> +                        uint16_t init_vfs, uint16_t total_vfs,
> +                        uint16_t vf_offset, uint16_t vf_stride)
> +{
> +    uint8_t *cfg = dev->config + offset;
> +    uint8_t *wmask;
> +
> +    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
> +                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
> +    dev->exp.sriov_cap = offset;
> +    dev->exp.sriov_pf.num_vfs = 0;
> +    dev->exp.sriov_pf.vfname = g_strdup(vfname);
> +    dev->exp.sriov_pf.vf = NULL;
> +
> +    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
> +    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
> +
> +    /* Minimal page size support values required by the SR/IOV standard:
> +     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
> +     * Hard coded for simplicity in this version
> +     */
> +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
> +
> +    /* Default is to use 4K pages, software can modify it
> +     * to any of the supported bits
> +     */
> +    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
> +
> +    /* Set up device ID and initial/total number of VFs available */
> +    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
> +    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
> +    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
> +    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
> +
> +    /* Write enable control bits */
> +    wmask = dev->wmask + offset;
> +    pci_set_word(wmask + PCI_SRIOV_CTRL,
> +                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE | PCI_SRIOV_CTRL_ARI);
> +    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
> +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
> +
> +    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
> +}
> +
> +void pcie_sriov_pf_exit(PCIDevice *dev)
> +{
> +    SRIOV_DPRINTF("\n");
> +    unregister_vfs(dev);
> +    g_free((char *)dev->exp.sriov_pf.vfname);
> +    dev->exp.sriov_pf.vfname = NULL;
> +}
> +
> +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> +                               uint8_t type, dma_addr_t size)
> +{
> +    uint32_t addr;
> +    uint64_t wmask;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +
> +    assert(sriov_cap > 0);
> +    assert(region_num >= 0);
> +    assert(region_num < PCI_NUM_REGIONS);
> +    assert(region_num != PCI_ROM_SLOT);
> +
> +    wmask = ~(size - 1);
> +    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
> +
> +    pci_set_long(dev->config + addr, type);
> +    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
> +        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> +        pci_set_quad(dev->wmask + addr, wmask);
> +        pci_set_quad(dev->cmask + addr, ~0ULL);
> +    } else {
> +        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
> +        pci_set_long(dev->cmask + addr, 0xffffffff);
> +    }
> +    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
> +}
> +
> +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> +                                MemoryRegion *memory)
> +{
> +    PCIIORegion *r;
> +    uint8_t type;
> +    pcibus_t size = memory_region_size(memory);
> +
> +    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
> +    assert(region_num >= 0);
> +    assert(region_num < PCI_NUM_REGIONS);
> +    type = dev->exp.sriov_vf.pf->exp.sriov_pf.vf_bar_type[region_num];
> +
> +    if (size & (size-1)) {
> +        fprintf(stderr, "ERROR: PCI region size must be a power"
> +                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n", type, size);
> +        exit(1);
> +    }
> +
> +    r = &dev->io_regions[region_num];
> +    r->memory = memory;
> +    r->address_space =
> +        type & PCI_BASE_ADDRESS_SPACE_IO
> +        ? dev->bus->address_space_io
> +        : dev->bus->address_space_mem;
> +    r->size = size;
> +    r->type = type;
> +
> +    r->addr = pci_bar_address(dev, region_num, r->type, r->size);
> +    if (r->addr != PCI_BAR_UNMAPPED) {
> +        memory_region_add_subregion_overlap(r->address_space,
> +                                            r->addr, r->memory, 1);
> +    }
> +}
> +
> +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char *name)
> +{
> +    PCIDevice *dev = pci_create(pf->bus, devfn, name);
> +    dev->exp.sriov_vf.pf = pf;
> +    Error *local_err = NULL;
> +
> +    object_property_set_bool(OBJECT(&dev->qdev), true, "realized", &local_err);
> +    if (local_err) {
> +        fprintf(stderr, "Failed to enable: %s\n",
> +                error_get_pretty(local_err));
> +        error_free(local_err);
> +        return NULL;
> +    }
> +
> +    /* set vid/did according to sr/iov spec - they are not used */
> +    pci_config_set_vendor_id(dev->config, 0xffff);
> +    pci_config_set_device_id(dev->config, 0xffff);
> +    return dev;
> +}
> +
> +static void register_vfs(PCIDevice *dev)
> +{
> +    uint16_t num_vfs;
> +    uint16_t i;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_OFFSET);
> +    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_VF_STRIDE);
> +    int32_t devfn = dev->devfn + vf_offset;
> +
> +    assert(sriov_cap > 0);
> +    num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF);
> +
> +    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) * num_vfs);
> +    assert(dev->exp.sriov_pf.vf);
> +
> +    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
> +    for (i = 0; i < num_vfs; i++) {
> +        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev->exp.sriov_pf.vfname);
> +        if (!dev->exp.sriov_pf.vf[i]) {
> +            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at devfn %x.%x\n", i,
> +                             PCI_SLOT(devfn), PCI_FUNC(devfn));
> +            num_vfs = i;
> +            break;
> +        }
> +        devfn += vf_stride;
> +    }
> +    dev->exp.sriov_pf.num_vfs = num_vfs;
> +}
> +
> +static void unregister_vfs(PCIDevice *dev)
> +{
> +    Error *local_err = NULL;
> +    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
> +    uint16_t i;
> +    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n", num_vfs);
> +    for (i = 0; i < num_vfs; i++) {
> +        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]->qdev), false, "realized", &local_err);
> +        if (local_err) {
> +            fprintf(stderr, "Failed to unplug: %s\n",
> +                    error_get_pretty(local_err));
> +            error_free(local_err);
> +        }
> +    }
> +    g_free(dev->exp.sriov_pf.vf);
> +    dev->exp.sriov_pf.vf = NULL;
> +    dev->exp.sriov_pf.num_vfs = 0;
> +}
> +
> +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address, uint32_t val, int len)
> +{
> +    uint32_t off;
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +
> +    if (!sriov_cap || address < sriov_cap) {
> +        return;
> +    }
> +    off = address - sriov_cap;
> +    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
> +        return;
> +    }
> +
> +    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val 0x%x len %d\n",
> +                 sriov_cap, off, val, len);
> +
> +    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
> +        if (dev->exp.sriov_pf.num_vfs) {
> +            if (!(val & PCI_SRIOV_CTRL_VFE)) {
> +                unregister_vfs(dev);
> +            }
> +        } else {
> +            if (val & PCI_SRIOV_CTRL_VFE) {
> +                register_vfs(dev);
> +            }
> +        }
> +    }
> +}
> +
> +
> +/* Reset SR/IOV VF Enable bit to trigger an unregister of all VFs */
> +void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
> +{
> +    uint16_t sriov_cap = dev->exp.sriov_cap;
> +    if (sriov_cap) {
> +        uint32_t val = pci_get_byte(dev->config + sriov_cap + PCI_SRIOV_CTRL);
> +        if (val & PCI_SRIOV_CTRL_VFE) {
> +            val &= ~PCI_SRIOV_CTRL_VFE;
> +            pcie_sriov_config_write(dev, sriov_cap + PCI_SRIOV_CTRL, val, 1);
> +        }
> +    }
> +}
> +
> +/* Add optional supported page sizes to the mask of supported page sizes */
> +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize)
> +{
> +    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
> +    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
> +
> +    uint16_t sup_pgsize = pci_get_word(cfg + PCI_SRIOV_SUP_PGSIZE);
> +
> +    sup_pgsize |= opt_sup_pgsize;
> +
> +    /* Make sure the new bits are set, and that system page size
> +     * also can be set to any of the new values according to spec:
> +     */
> +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
> +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
> +}
> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> index 551cb3d..2e9d8ba 100644
> --- a/include/hw/pci/pci.h
> +++ b/include/hw/pci/pci.h
> @@ -11,8 +11,6 @@
>   /* PCI includes legacy ISA access.  */
>   #include "hw/isa/isa.h"
>
> -#include "hw/pci/pcie.h"
> -
>   /* PCI bus */
>
>   #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) | ((func) & 0x07))
> @@ -132,6 +130,7 @@ enum {
>   #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
>
>   #include "hw/pci/pci_regs.h"
> +#include "hw/pci/pcie.h"
>
>   /* PCI HEADER_TYPE */
>   #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
> @@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus *, void *, int);
>   AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
>   void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void *opaque);
>
> +pcibus_t pci_bar_address(PCIDevice *d,
> +                         int reg, uint8_t type, pcibus_t size);
> +
>   static inline void
>   pci_set_byte(uint8_t *config, uint8_t val)
>   {
> @@ -672,6 +674,11 @@ static inline int pci_is_express(const PCIDevice *d)
>       return d->cap_present & QEMU_PCI_CAP_EXPRESS;
>   }
>
> +static inline int pci_is_vf(const PCIDevice *d)
> +{
> +    return d->exp.sriov_vf.pf != NULL;
> +}
> +
>   static inline uint32_t pci_config_size(const PCIDevice *d)
>   {
>       return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE : PCI_CONFIG_SPACE_SIZE;
> diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
> index b48a7a2..b09f79a 100644
> --- a/include/hw/pci/pcie.h
> +++ b/include/hw/pci/pcie.h
> @@ -25,6 +25,7 @@
>   #include "hw/pci/pci_regs.h"
>   #include "hw/pci/pcie_regs.h"
>   #include "hw/pci/pcie_aer.h"
> +#include "hw/pci/pcie_sriov.h"
>   #include "hw/hotplug.h"
>
>   typedef enum {
> @@ -74,6 +75,11 @@ struct PCIExpressDevice {
>       /* AER */
>       uint16_t aer_cap;
>       PCIEAERLog aer_log;
> +
> +    /* SR/IOV */
> +    uint16_t sriov_cap;
> +    PCIESriovPF sriov_pf;
> +    PCIESriovVF sriov_vf;
>   };
>
>   #define COMPAT_PROP_PCP "power_controller_present"
> diff --git a/include/hw/pci/pcie_sriov.h b/include/hw/pci/pcie_sriov.h
> new file mode 100644
> index 0000000..061f5cf
> --- /dev/null
> +++ b/include/hw/pci/pcie_sriov.h
> @@ -0,0 +1,55 @@
> +/*
> + * pcie_sriov.h:
> + *
> + * Implementation of SR/IOV emulation support.
> + *
> + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#ifndef QEMU_PCIE_SRIOV_H
> +#define QEMU_PCIE_SRIOV_H
> +
> +
> +
> +struct PCIESriovPF {
> +    uint16_t num_vfs;           /* Number of virtual functions created */
> +    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for each VF bar */
> +    const char *vfname;         /* Reference to the device type used for the VFs */
> +    PCIDevice **vf;             /* Pointer to an array of num_vfs VF devices */
> +};
> +
> +struct PCIESriovVF {
> +    PCIDevice *pf;              /* Pointer back to owner physical function */
> +};
> +
> +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> +                        const char *vfname, uint16_t vf_dev_id,
> +                        uint16_t init_vfs, uint16_t total_vfs,
> +                        uint16_t vf_offset, uint16_t vf_stride);
> +void pcie_sriov_pf_exit(PCIDevice *dev);
> +
> +/* Set up a VF bar in the SR/IOV bar area */
> +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> +                               uint8_t type, dma_addr_t size);
> +
> +/* Instantiate a bar for a VF */
> +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> +                                MemoryRegion *memory);
> +
> +/* Add optional supported page sizes to the mask of supported page sizes
> + * Page size values are interpreted as opt_sup_pgsize << 12.
> + */
> +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize);
> +
> +/* SR/IOV capability config write handler */
> +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> +                             uint32_t val, int len);
> +
> +/* Reset SR/IOV VF Enable bit to unregister all VFs */
> +void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
> +
> +#endif /* QEMU_PCIE_SRIOV_H */
> diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> index f8a9dd6..fa36b8c 100644
> --- a/include/qemu/typedefs.h
> +++ b/include/qemu/typedefs.h
> @@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
>   typedef struct PCIEAERErr PCIEAERErr;
>   typedef struct PCIEAERLog PCIEAERLog;o
>   typedef struct PCIEAERMsg PCIEAERMsg;
> +typedef struct PCIESriovPF PCIESriovPF;
> +typedef struct PCIESriovVF PCIESriovVF;
>   typedef struct PCIEPort PCIEPort;
>   typedef struct PCIESlot PCIESlot;
>   typedef struct PCIExpressDevice PCIExpressDevice;
>

I reviewed the patch and it looks OK to me (besides the other comments in the thread).
Since we don't have code using it yet and the tests are only an RFC, I suggest adding
to the series a document file explaining how to use it (template code from the RFC)
and the current limitations/issues.

Thanks,
Marcel

Re: [Qemu-devel] [PATCH v4 3/3] pcie: Add support for Single Root I/O Virtualization (SR/IOV)

[Knut Omang]

On Wed, 2015-10-07 at 18:06 +0300, Marcel Apfelbaum wrote:
> On 09/12/2015 03:36 PM, Knut Omang wrote:
> > This patch provides the building blocks for creating an SR/IOV
> > PCIe Extended Capability header and register/unregister
> > SR/IOV Virtual Functions.
> > 
> > Signed-off-by: Knut Omang <knut.omang@oracle.com>
> > ---
> >   hw/pci/Makefile.objs        |   2 +-
> >   hw/pci/pci.c                |  99 ++++++++++++----
> >   hw/pci/pcie.c               |   9 +-
> >   hw/pci/pcie_sriov.c         | 271
> > ++++++++++++++++++++++++++++++++++++++++++++
> >   include/hw/pci/pci.h        |  11 +-
> >   include/hw/pci/pcie.h       |   6 +
> >   include/hw/pci/pcie_sriov.h |  55 +++++++++
> >   include/qemu/typedefs.h     |   2 +
> >   8 files changed, 426 insertions(+), 29 deletions(-)
> >   create mode 100644 hw/pci/pcie_sriov.c
> >   create mode 100644 include/hw/pci/pcie_sriov.h
> > 
> > diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs
> > index 9f905e6..2226980 100644
> > --- a/hw/pci/Makefile.objs
> > +++ b/hw/pci/Makefile.objs
> > @@ -3,7 +3,7 @@ common-obj-$(CONFIG_PCI) += msix.o msi.o
> >   common-obj-$(CONFIG_PCI) += shpc.o
> >   common-obj-$(CONFIG_PCI) += slotid_cap.o
> >   common-obj-$(CONFIG_PCI) += pci_host.o pcie_host.o
> > -common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > +common-obj-$(CONFIG_PCI) += pcie.o pcie_aer.o pcie_port.o
> > pcie_sriov.o
> > 
> >   common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
> >   common-obj-$(CONFIG_ALL) += pci-stub.o
> > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > index a5cc015..9c0eba1 100644
> > --- a/hw/pci/pci.c
> > +++ b/hw/pci/pci.c
> > @@ -153,6 +153,9 @@ int pci_bar(PCIDevice *d, int reg)
> >   {
> >       uint8_t type;
> > 
> > +    /* PCIe virtual functions do not have their own BARs */
> > +    assert(!pci_is_vf(d));
> > +
> >       if (reg != PCI_ROM_SLOT)
> >           return PCI_BASE_ADDRESS_0 + reg * 4;
> > 
> > @@ -211,22 +214,13 @@ void pci_device_deassert_intx(PCIDevice *dev)
> >       }
> >   }
> > 
> > -static void pci_do_device_reset(PCIDevice *dev)
> > +static void pci_reset_regions(PCIDevice *dev)
> >   {
> >       int r;
> > +    if (pci_is_vf(dev)) {
> > +        return;
> > +    }
> > 
> > -    pci_device_deassert_intx(dev);
> > -    assert(dev->irq_state == 0);
> > -
> > -    /* Clear all writable bits */
> > -    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > -                                 pci_get_word(dev->wmask +
> > PCI_COMMAND) |
> > -                                 pci_get_word(dev->w1cmask +
> > PCI_COMMAND));
> > -    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > -                                 pci_get_word(dev->wmask +
> > PCI_STATUS) |
> > -                                 pci_get_word(dev->w1cmask +
> > PCI_STATUS));
> > -    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > -    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> >       for (r = 0; r < PCI_NUM_REGIONS; ++r) {
> >           PCIIORegion *region = &dev->io_regions[r];
> >           if (!region->size) {
> > @@ -240,6 +234,27 @@ static void pci_do_device_reset(PCIDevice
> > *dev)
> >               pci_set_long(dev->config + pci_bar(dev, r), region
> > ->type);
> >           }
> >       }
> > +}
> > +
> > +static void pci_do_device_reset(PCIDevice *dev)
> > +{
> > +    qdev_reset_all(&dev->qdev);
> > +
> > +    dev->irq_state = 0;
> > +    pci_update_irq_status(dev);
> > +    pci_device_deassert_intx(dev);
> > +    assert(dev->irq_state == 0);
> > +
> > +    /* Clear all writable bits */
> > +    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
> > +                                 pci_get_word(dev->wmask +
> > PCI_COMMAND) |
> > +                                 pci_get_word(dev->w1cmask +
> > PCI_COMMAND));
> > +    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
> > +                                 pci_get_word(dev->wmask +
> > PCI_STATUS) |
> > +                                 pci_get_word(dev->w1cmask +
> > PCI_STATUS));
> > +    dev->config[PCI_CACHE_LINE_SIZE] = 0x0;
> > +    dev->config[PCI_INTERRUPT_LINE] = 0x0;
> > +    pci_reset_regions(dev);
> >       pci_update_mappings(dev);
> > 
> >       msi_reset(dev);
> > @@ -771,6 +786,15 @@ static void pci_init_multifunction(PCIBus
> > *bus, PCIDevice *dev, Error **errp)
> >           dev->config[PCI_HEADER_TYPE] |=
> > PCI_HEADER_TYPE_MULTI_FUNCTION;
> >       }
> > 
> > +    /* With SR/IOV and ARI, a device at function 0 need not be a
> > multifunction
> > +     * device, as it may just be a VF that ended up with function
> > 0 in
> > +     * the legacy PCI interpretation. Avoid failing in such cases:
> > +     */
> > +    if (pci_is_vf(dev) &&
> > +        dev->exp.sriov_vf.pf->cap_present &
> > QEMU_PCI_CAP_MULTIFUNCTION) {
> > +        return;
> > +    }
> > +
> >       /*
> >        * multifunction bit is interpreted in two ways as follows.
> >        *   - all functions must set the bit to 1.
> > @@ -962,6 +986,7 @@ void pci_register_bar(PCIDevice *pci_dev, int
> > region_num,
> >       uint64_t wmask;
> >       pcibus_t size = memory_region_size(memory);
> > 
> > +    assert(!pci_is_vf(pci_dev)); /* VFs must use
> > pcie_sriov_vf_register_bar */
> >       assert(region_num >= 0);
> >       assert(region_num < PCI_NUM_REGIONS);
> >       if (size & (size-1)) {
> > @@ -1060,11 +1085,44 @@ pcibus_t pci_get_bar_addr(PCIDevice
> > *pci_dev, int region_num)
> >       return pci_dev->io_regions[region_num].addr;
> >   }
> > 
> > -static pcibus_t pci_bar_address(PCIDevice *d,
> > -				int reg, uint8_t type, pcibus_t
> > size)
> > +
> > +static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg,
> > +                                        uint8_t type, pcibus_t
> > size)
> > +{
> > +    pcibus_t new_addr;
> > +    if (!pci_is_vf(d)) {
> > +        int bar = pci_bar(d, reg);
> > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +            new_addr = pci_get_quad(d->config + bar);
> > +        } else {
> > +            new_addr = pci_get_long(d->config + bar);
> > +        }
> > +    } else {
> > +        PCIDevice *pf = d->exp.sriov_vf.pf;
> > +        uint16_t sriov_cap = pf->exp.sriov_cap;
> > +        int bar = sriov_cap + PCI_SRIOV_BAR + reg * 4;
> > +        uint16_t vf_offset = pci_get_word(pf->config + sriov_cap +
> > PCI_SRIOV_VF_OFFSET);
> > +        uint16_t vf_stride = pci_get_word(pf->config + sriov_cap +
> > PCI_SRIOV_VF_STRIDE);
> > +        uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) /
> > vf_stride;
> > +
> > +        if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +            new_addr = pci_get_quad(pf->config + bar);
> > +        } else {
> > +            new_addr = pci_get_long(pf->config + bar);
> > +        }
> > +        new_addr += vf_num * size;
> > +    }
> > +    if (reg != PCI_ROM_SLOT) {
> > +        /* Preserve the rom enable bit */
> > +        new_addr &= ~(size - 1);
> > +    }
> > +    return new_addr;
> > +}
> > +
> > +pcibus_t pci_bar_address(PCIDevice *d,
> > +                         int reg, uint8_t type, pcibus_t size)
> >   {
> >       pcibus_t new_addr, last_addr;
> > -    int bar = pci_bar(d, reg);
> >       uint16_t cmd = pci_get_word(d->config + PCI_COMMAND);
> >       Object *machine = qdev_get_machine();
> >       ObjectClass *oc = object_get_class(machine);
> > @@ -1075,7 +1133,7 @@ static pcibus_t pci_bar_address(PCIDevice *d,
> >           if (!(cmd & PCI_COMMAND_IO)) {
> >               return PCI_BAR_UNMAPPED;
> >           }
> > -        new_addr = pci_get_long(d->config + bar) & ~(size - 1);
> > +        new_addr = pci_config_get_bar_addr(d, reg, type, size);
> >           last_addr = new_addr + size - 1;
> >           /* Check if 32 bit BAR wraps around explicitly.
> >            * TODO: make priorities correct and remove this work
> > around.
> > @@ -1090,11 +1148,7 @@ static pcibus_t pci_bar_address(PCIDevice
> > *d,
> >       if (!(cmd & PCI_COMMAND_MEMORY)) {
> >           return PCI_BAR_UNMAPPED;
> >       }
> > -    if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > -        new_addr = pci_get_quad(d->config + bar);
> > -    } else {
> > -        new_addr = pci_get_long(d->config + bar);
> > -    }
> > +    new_addr = pci_config_get_bar_addr(d, reg, type, size);
> >       /* the ROM slot has a specific enable bit */
> >       if (reg == PCI_ROM_SLOT && !(new_addr &
> > PCI_ROM_ADDRESS_ENABLE)) {
> >           return PCI_BAR_UNMAPPED;
> > @@ -1228,6 +1282,7 @@ void pci_default_write_config(PCIDevice *d,
> > uint32_t addr, uint32_t val_in, int
> > 
> >       msi_write_config(d, addr, val_in, l);
> >       msix_write_config(d, addr, val_in, l);
> > +    pcie_sriov_config_write(d, addr, val_in, l);
> >   }
> > 
> >   /***********************************************************/
> > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
> > index 6e28985..ba49c0f 100644
> > --- a/hw/pci/pcie.c
> > +++ b/hw/pci/pcie.c
> > @@ -253,7 +253,7 @@ void pcie_cap_slot_hotplug_cb(HotplugHandler
> > *hotplug_dev, DeviceState *dev,
> >        * Right now, only a device of function = 0 is allowed to be
> >        * hot plugged/unplugged.
> >        */
> > -    assert(PCI_FUNC(pci_dev->devfn) == 0);
> > +    assert(PCI_FUNC(pci_dev->devfn) == 0 || pci_is_vf(pci_dev));
> > 
> >       pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA,
> >                                  PCI_EXP_SLTSTA_PDS);
> > @@ -265,10 +265,11 @@ void
> > pcie_cap_slot_hot_unplug_request_cb(HotplugHandler *hotplug_dev,
> >                                            DeviceState *dev, Error
> > **errp)
> >   {
> >       uint8_t *exp_cap;
> > +    PCIDevice *pdev = PCI_DEVICE(hotplug_dev);
> > 
> > -    pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev,
> > &exp_cap, errp);
> > +    pcie_cap_slot_hotplug_common(pdev, dev, &exp_cap, errp);
> > 
> > -    pcie_cap_slot_push_attention_button(PCI_DEVICE(hotplug_dev));
> > +    pcie_cap_slot_push_attention_button(pdev);
> >   }
> > 
> >   /* pci express slot for pci express root/downstream port
> > @@ -408,7 +409,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
> >       }
> > 
> >       /*
> > -     * If the slot is polulated, power indicator is off and power
> > +     * If the slot is populated, power indicator is off and power
> >        * controller is off, it is safe to detach the devices.
> >        */
> >       if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val &
> > PCI_EXP_SLTCTL_PCC) &&
> > diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c
> > new file mode 100644
> > index 0000000..14f31cc
> > --- /dev/null
> > +++ b/hw/pci/pcie_sriov.c
> > @@ -0,0 +1,271 @@
> > +/*
> > + * pcie_sriov.c:
> > + *
> > + * Implementation of SR/IOV emulation support.
> > + *
> > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2
> > or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#include "hw/pci/pci.h"
> > +#include "hw/pci/pcie.h"
> > +#include "hw/pci/pci_bus.h"
> > +#include "qemu/range.h"
> > +
> > +//#define DEBUG_SRIOV
> > +#ifdef DEBUG_SRIOV
> > +# define SRIOV_DPRINTF(fmt, ...)                                  
> >        \
> > +    fprintf(stderr, "%s:%d " fmt, __func__, __LINE__, ##
> > __VA_ARGS__)
> > +#else
> > +# define SRIOV_DPRINTF(fmt, ...) do {} while (0)
> > +#endif
> > +#define SRIOV_DEV_PRINTF(dev, fmt, ...)                           
> >        \
> > +    SRIOV_DPRINTF("%s:%x "fmt, (dev)->name, (dev)->devfn, ##
> > __VA_ARGS__)
> > +
> > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char
> > *name);
> > +static void unregister_vfs(PCIDevice *dev);
> > +
> > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > +                        const char *vfname, uint16_t vf_dev_id,
> > +                        uint16_t init_vfs, uint16_t total_vfs,
> > +                        uint16_t vf_offset, uint16_t vf_stride)
> > +{
> > +    uint8_t *cfg = dev->config + offset;
> > +    uint8_t *wmask;
> > +
> > +    pcie_add_capability(dev, PCI_EXT_CAP_ID_SRIOV, 1,
> > +                        offset, PCI_EXT_CAP_SRIOV_SIZEOF);
> > +    dev->exp.sriov_cap = offset;
> > +    dev->exp.sriov_pf.num_vfs = 0;
> > +    dev->exp.sriov_pf.vfname = g_strdup(vfname);
> > +    dev->exp.sriov_pf.vf = NULL;
> > +
> > +    pci_set_word(cfg + PCI_SRIOV_VF_OFFSET, vf_offset);
> > +    pci_set_word(cfg + PCI_SRIOV_VF_STRIDE, vf_stride);
> > +
> > +    /* Minimal page size support values required by the SR/IOV
> > standard:
> > +     * 0x553 << 12 = 0x553000 = 4K + 8K + 64K + 256K + 1M + 4M
> > +     * Hard coded for simplicity in this version
> > +     */
> > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, 0x553);
> > +
> > +    /* Default is to use 4K pages, software can modify it
> > +     * to any of the supported bits
> > +     */
> > +    pci_set_word(cfg + PCI_SRIOV_SYS_PGSIZE, 0x1);
> > +
> > +    /* Set up device ID and initial/total number of VFs available
> > */
> > +    pci_set_word(cfg + PCI_SRIOV_VF_DID, vf_dev_id);
> > +    pci_set_word(cfg + PCI_SRIOV_INITIAL_VF, init_vfs);
> > +    pci_set_word(cfg + PCI_SRIOV_TOTAL_VF, total_vfs);
> > +    pci_set_word(cfg + PCI_SRIOV_NUM_VF, 0);
> > +
> > +    /* Write enable control bits */
> > +    wmask = dev->wmask + offset;
> > +    pci_set_word(wmask + PCI_SRIOV_CTRL,
> > +                 PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE |
> > PCI_SRIOV_CTRL_ARI);
> > +    pci_set_word(wmask + PCI_SRIOV_NUM_VF, 0xffff);
> > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, 0x553);
> > +
> > +    qdev_prop_set_bit(&dev->qdev, "multifunction", true);
> > +}
> > +
> > +void pcie_sriov_pf_exit(PCIDevice *dev)
> > +{
> > +    SRIOV_DPRINTF("\n");
> > +    unregister_vfs(dev);
> > +    g_free((char *)dev->exp.sriov_pf.vfname);
> > +    dev->exp.sriov_pf.vfname = NULL;
> > +}
> > +
> > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > +                               uint8_t type, dma_addr_t size)
> > +{
> > +    uint32_t addr;
> > +    uint64_t wmask;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +
> > +    assert(sriov_cap > 0);
> > +    assert(region_num >= 0);
> > +    assert(region_num < PCI_NUM_REGIONS);
> > +    assert(region_num != PCI_ROM_SLOT);
> > +
> > +    wmask = ~(size - 1);
> > +    addr = sriov_cap + PCI_SRIOV_BAR + region_num * 4;
> > +
> > +    pci_set_long(dev->config + addr, type);
> > +    if (!(type & PCI_BASE_ADDRESS_SPACE_IO) &&
> > +        type & PCI_BASE_ADDRESS_MEM_TYPE_64) {
> > +        pci_set_quad(dev->wmask + addr, wmask);
> > +        pci_set_quad(dev->cmask + addr, ~0ULL);
> > +    } else {
> > +        pci_set_long(dev->wmask + addr, wmask & 0xffffffff);
> > +        pci_set_long(dev->cmask + addr, 0xffffffff);
> > +    }
> > +    dev->exp.sriov_pf.vf_bar_type[region_num] = type;
> > +}
> > +
> > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > +                                MemoryRegion *memory)
> > +{
> > +    PCIIORegion *r;
> > +    uint8_t type;
> > +    pcibus_t size = memory_region_size(memory);
> > +
> > +    assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */
> > +    assert(region_num >= 0);
> > +    assert(region_num < PCI_NUM_REGIONS);
> > +    type = dev->exp.sriov_vf.pf
> > ->exp.sriov_pf.vf_bar_type[region_num];
> > +
> > +    if (size & (size-1)) {
> > +        fprintf(stderr, "ERROR: PCI region size must be a power"
> > +                " of two - type=0x%x, size=0x%"FMT_PCIBUS"\n",
> > type, size);
> > +        exit(1);
> > +    }
> > +
> > +    r = &dev->io_regions[region_num];
> > +    r->memory = memory;
> > +    r->address_space =
> > +        type & PCI_BASE_ADDRESS_SPACE_IO
> > +        ? dev->bus->address_space_io
> > +        : dev->bus->address_space_mem;
> > +    r->size = size;
> > +    r->type = type;
> > +
> > +    r->addr = pci_bar_address(dev, region_num, r->type, r->size);
> > +    if (r->addr != PCI_BAR_UNMAPPED) {
> > +        memory_region_add_subregion_overlap(r->address_space,
> > +                                            r->addr, r->memory,
> > 1);
> > +    }
> > +}
> > +
> > +static PCIDevice *register_vf(PCIDevice *pf, int devfn, const char
> > *name)
> > +{
> > +    PCIDevice *dev = pci_create(pf->bus, devfn, name);
> > +    dev->exp.sriov_vf.pf = pf;
> > +    Error *local_err = NULL;
> > +
> > +    object_property_set_bool(OBJECT(&dev->qdev), true, "realized",
> > &local_err);
> > +    if (local_err) {
> > +        fprintf(stderr, "Failed to enable: %s\n",
> > +                error_get_pretty(local_err));
> > +        error_free(local_err);
> > +        return NULL;
> > +    }
> > +
> > +    /* set vid/did according to sr/iov spec - they are not used */
> > +    pci_config_set_vendor_id(dev->config, 0xffff);
> > +    pci_config_set_device_id(dev->config, 0xffff);
> > +    return dev;
> > +}
> > +
> > +static void register_vfs(PCIDevice *dev)
> > +{
> > +    uint16_t num_vfs;
> > +    uint16_t i;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +    uint16_t vf_offset = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_VF_OFFSET);
> > +    uint16_t vf_stride = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_VF_STRIDE);
> > +    int32_t devfn = dev->devfn + vf_offset;
> > +
> > +    assert(sriov_cap > 0);
> > +    num_vfs = pci_get_word(dev->config + sriov_cap +
> > PCI_SRIOV_NUM_VF);
> > +
> > +    dev->exp.sriov_pf.vf = g_malloc(sizeof(PCIDevice *) *
> > num_vfs);
> > +    assert(dev->exp.sriov_pf.vf);
> > +
> > +    SRIOV_DEV_PRINTF(dev, "creating %d vf devs\n", num_vfs);
> > +    for (i = 0; i < num_vfs; i++) {
> > +        dev->exp.sriov_pf.vf[i] = register_vf(dev, devfn, dev
> > ->exp.sriov_pf.vfname);
> > +        if (!dev->exp.sriov_pf.vf[i]) {
> > +            SRIOV_DEV_PRINTF(dev, "Failed to create VF %d at devfn
> > %x.%x\n", i,
> > +                             PCI_SLOT(devfn), PCI_FUNC(devfn));
> > +            num_vfs = i;
> > +            break;
> > +        }
> > +        devfn += vf_stride;
> > +    }
> > +    dev->exp.sriov_pf.num_vfs = num_vfs;
> > +}
> > +
> > +static void unregister_vfs(PCIDevice *dev)
> > +{
> > +    Error *local_err = NULL;
> > +    uint16_t num_vfs = dev->exp.sriov_pf.num_vfs;
> > +    uint16_t i;
> > +    SRIOV_DEV_PRINTF(dev, "Unregistering %d vf devs\n", num_vfs);
> > +    for (i = 0; i < num_vfs; i++) {
> > +        object_property_set_bool(OBJECT(&dev->exp.sriov_pf.vf[i]
> > ->qdev), false, "realized", &local_err);
> > +        if (local_err) {
> > +            fprintf(stderr, "Failed to unplug: %s\n",
> > +                    error_get_pretty(local_err));
> > +            error_free(local_err);
> > +        }
> > +    }
> > +    g_free(dev->exp.sriov_pf.vf);
> > +    dev->exp.sriov_pf.vf = NULL;
> > +    dev->exp.sriov_pf.num_vfs = 0;
> > +}
> > +
> > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > uint32_t val, int len)
> > +{
> > +    uint32_t off;
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +
> > +    if (!sriov_cap || address < sriov_cap) {
> > +        return;
> > +    }
> > +    off = address - sriov_cap;
> > +    if (off >= PCI_EXT_CAP_SRIOV_SIZEOF) {
> > +        return;
> > +    }
> > +
> > +    SRIOV_DEV_PRINTF(dev, "cap at 0x%x sriov offset 0x%x val 0x%x
> > len %d\n",
> > +                 sriov_cap, off, val, len);
> > +
> > +    if (range_covers_byte(off, len, PCI_SRIOV_CTRL)) {
> > +        if (dev->exp.sriov_pf.num_vfs) {
> > +            if (!(val & PCI_SRIOV_CTRL_VFE)) {
> > +                unregister_vfs(dev);
> > +            }
> > +        } else {
> > +            if (val & PCI_SRIOV_CTRL_VFE) {
> > +                register_vfs(dev);
> > +            }
> > +        }
> > +    }
> > +}
> > +
> > +
> > +/* Reset SR/IOV VF Enable bit to trigger an unregister of all VFs
> > */
> > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev)
> > +{
> > +    uint16_t sriov_cap = dev->exp.sriov_cap;
> > +    if (sriov_cap) {
> > +        uint32_t val = pci_get_byte(dev->config + sriov_cap +
> > PCI_SRIOV_CTRL);
> > +        if (val & PCI_SRIOV_CTRL_VFE) {
> > +            val &= ~PCI_SRIOV_CTRL_VFE;
> > +            pcie_sriov_config_write(dev, sriov_cap +
> > PCI_SRIOV_CTRL, val, 1);
> > +        }
> > +    }
> > +}
> > +
> > +/* Add optional supported page sizes to the mask of supported page
> > sizes */
> > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > opt_sup_pgsize)
> > +{
> > +    uint8_t *cfg = dev->config + dev->exp.sriov_cap;
> > +    uint8_t *wmask = dev->wmask + dev->exp.sriov_cap;
> > +
> > +    uint16_t sup_pgsize = pci_get_word(cfg +
> > PCI_SRIOV_SUP_PGSIZE);
> > +
> > +    sup_pgsize |= opt_sup_pgsize;
> > +
> > +    /* Make sure the new bits are set, and that system page size
> > +     * also can be set to any of the new values according to spec:
> > +     */
> > +    pci_set_word(cfg + PCI_SRIOV_SUP_PGSIZE, sup_pgsize);
> > +    pci_set_word(wmask + PCI_SRIOV_SYS_PGSIZE, sup_pgsize);
> > +}
> > diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> > index 551cb3d..2e9d8ba 100644
> > --- a/include/hw/pci/pci.h
> > +++ b/include/hw/pci/pci.h
> > @@ -11,8 +11,6 @@
> >   /* PCI includes legacy ISA access.  */
> >   #include "hw/isa/isa.h"
> > 
> > -#include "hw/pci/pcie.h"
> > -
> >   /* PCI bus */
> > 
> >   #define PCI_DEVFN(slot, func)   ((((slot) & 0x1f) << 3) | ((func)
> > & 0x07))
> > @@ -132,6 +130,7 @@ enum {
> >   #define QEMU_PCI_VGA_IO_HI_SIZE 0x20
> > 
> >   #include "hw/pci/pci_regs.h"
> > +#include "hw/pci/pcie.h"
> > 
> >   /* PCI HEADER_TYPE */
> >   #define  PCI_HEADER_TYPE_MULTI_FUNCTION 0x80
> > @@ -421,6 +420,9 @@ typedef AddressSpace *(*PCIIOMMUFunc)(PCIBus *,
> > void *, int);
> >   AddressSpace *pci_device_iommu_address_space(PCIDevice *dev);
> >   void pci_setup_iommu(PCIBus *bus, PCIIOMMUFunc fn, void *opaque);
> > 
> > +pcibus_t pci_bar_address(PCIDevice *d,
> > +                         int reg, uint8_t type, pcibus_t size);
> > +
> >   static inline void
> >   pci_set_byte(uint8_t *config, uint8_t val)
> >   {
> > @@ -672,6 +674,11 @@ static inline int pci_is_express(const
> > PCIDevice *d)
> >       return d->cap_present & QEMU_PCI_CAP_EXPRESS;
> >   }
> > 
> > +static inline int pci_is_vf(const PCIDevice *d)
> > +{
> > +    return d->exp.sriov_vf.pf != NULL;
> > +}
> > +
> >   static inline uint32_t pci_config_size(const PCIDevice *d)
> >   {
> >       return pci_is_express(d) ? PCIE_CONFIG_SPACE_SIZE :
> > PCI_CONFIG_SPACE_SIZE;
> > diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h
> > index b48a7a2..b09f79a 100644
> > --- a/include/hw/pci/pcie.h
> > +++ b/include/hw/pci/pcie.h
> > @@ -25,6 +25,7 @@
> >   #include "hw/pci/pci_regs.h"
> >   #include "hw/pci/pcie_regs.h"
> >   #include "hw/pci/pcie_aer.h"
> > +#include "hw/pci/pcie_sriov.h"
> >   #include "hw/hotplug.h"
> > 
> >   typedef enum {
> > @@ -74,6 +75,11 @@ struct PCIExpressDevice {
> >       /* AER */
> >       uint16_t aer_cap;
> >       PCIEAERLog aer_log;
> > +
> > +    /* SR/IOV */
> > +    uint16_t sriov_cap;
> > +    PCIESriovPF sriov_pf;
> > +    PCIESriovVF sriov_vf;
> >   };
> > 
> >   #define COMPAT_PROP_PCP "power_controller_present"
> > diff --git a/include/hw/pci/pcie_sriov.h
> > b/include/hw/pci/pcie_sriov.h
> > new file mode 100644
> > index 0000000..061f5cf
> > --- /dev/null
> > +++ b/include/hw/pci/pcie_sriov.h
> > @@ -0,0 +1,55 @@
> > +/*
> > + * pcie_sriov.h:
> > + *
> > + * Implementation of SR/IOV emulation support.
> > + *
> > + * Copyright (c) 2014 Knut Omang <knut.omang@oracle.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2
> > or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#ifndef QEMU_PCIE_SRIOV_H
> > +#define QEMU_PCIE_SRIOV_H
> > +
> > +
> > +
> > +struct PCIESriovPF {
> > +    uint16_t num_vfs;           /* Number of virtual functions
> > created */
> > +    uint8_t vf_bar_type[PCI_NUM_REGIONS];  /* Store type for each
> > VF bar */
> > +    const char *vfname;         /* Reference to the device type
> > used for the VFs */
> > +    PCIDevice **vf;             /* Pointer to an array of num_vfs
> > VF devices */
> > +};
> > +
> > +struct PCIESriovVF {
> > +    PCIDevice *pf;              /* Pointer back to owner physical
> > function */
> > +};
> > +
> > +void pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset,
> > +                        const char *vfname, uint16_t vf_dev_id,
> > +                        uint16_t init_vfs, uint16_t total_vfs,
> > +                        uint16_t vf_offset, uint16_t vf_stride);
> > +void pcie_sriov_pf_exit(PCIDevice *dev);
> > +
> > +/* Set up a VF bar in the SR/IOV bar area */
> > +void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num,
> > +                               uint8_t type, dma_addr_t size);
> > +
> > +/* Instantiate a bar for a VF */
> > +void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num,
> > +                                MemoryRegion *memory);
> > +
> > +/* Add optional supported page sizes to the mask of supported page
> > sizes
> > + * Page size values are interpreted as opt_sup_pgsize << 12.
> > + */
> > +void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t
> > opt_sup_pgsize);
> > +
> > +/* SR/IOV capability config write handler */
> > +void pcie_sriov_config_write(PCIDevice *dev, uint32_t address,
> > +                             uint32_t val, int len);
> > +
> > +/* Reset SR/IOV VF Enable bit to unregister all VFs */
> > +void pcie_sriov_pf_disable_vfs(PCIDevice *dev);
> > +
> > +#endif /* QEMU_PCIE_SRIOV_H */
> > diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> > index f8a9dd6..fa36b8c 100644
> > --- a/include/qemu/typedefs.h
> > +++ b/include/qemu/typedefs.h
> > @@ -53,6 +53,8 @@ typedef struct PCIDevice PCIDevice;
> >   typedef struct PCIEAERErr PCIEAERErr;
> >   typedef struct PCIEAERLog PCIEAERLog;o
> >   typedef struct PCIEAERMsg PCIEAERMsg;
> > +typedef struct PCIESriovPF PCIESriovPF;
> > +typedef struct PCIESriovVF PCIESriovVF;
> >   typedef struct PCIEPort PCIEPort;
> >   typedef struct PCIESlot PCIESlot;
> >   typedef struct PCIExpressDevice PCIExpressDevice;
> > 
> 
> I reviewed the patch and it looks OK to me (besides the other
> comments in the thread).
> Since we don't have code using it yet and the tests are only an RFC,
> I suggest adding
> to the series a document file explaining how to use it (template code
> from the RFC)
> and the current limitations/issues.

Good idea, the 

docs/pci_expander_bridge.txt

file seems like a good template,

Thanks

Knut

> Thanks,
> Marcel
> 
> 
>