SR-IOV function passthrough to a PV domU

SR-IOV function passthrough to a PV domU

[Chris Reeves]

I'm trying to pass through an SR-IOV virtual function to a PV domU.
The pci passthrough seems to work, but the virtual function never shows
up on the pci bus of the domU.  All I see is a message stating:
[  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00

I'm using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and domU
and Xen 4.0.  I've recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG option
set, but I don't get any additional output when attaching the device.

Is it possible to use pci passthrough with SR-IOV virtual functions to a
PV domU?

I noticed in this thread:
http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
That someone mentioned a specific SR-IOV pcifront driver.
Does anyone know where this driver may be available?

For what it's worth, I've also tried kernel 2.6.34 with the xen patches
from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
the same results.

Thanks,
Chris

Re: SR-IOV function passthrough to a PV domU

[Konrad Rzeszutek Wilk]

On Tue, Jun 15, 2010 at 03:15:35PM -0600, Chris Reeves wrote:
> I'm trying to pass through an SR-IOV virtual function to a PV domU.
> The pci passthrough seems to work, but the virtual function never shows
> up on the pci bus of the domU.  All I see is a message stating:
> [  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
> 
> I'm using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and domU
> and Xen 4.0.  I've recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG option
> set, but I don't get any additional output when attaching the device.
> 
> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?

Yes. But your Dom0 kernel MUST be the updated PV-OPS version. You need
to get the latest pv-ops kernel. Details at:
http://wiki.xensource.com/xenwiki/XenParavirtOps

> 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?

In the PV-OPS tree.
> 
> For what it's worth, I've also tried kernel 2.6.34 with the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.

Yeah, no surprise. They don't have the fixes for Xen PCIBack working
with SR-IOV hardware.

Re: SR-IOV function passthrough to a PV domU

[Keir Fraser]

The pciback driver in dom0 needs patching.

See http://xenbits.xensource.com/linux-2.6.18-xen.hg changeset 999, for
example. I would imagine some vendor kernels may be missing this, although I
thought Novell had it. Jan might be able to say (cc'ed).

 -- Keir

On 15/06/2010 22:15, "Chris Reeves" <chrisr@spectralogic.com> wrote:

> I'm trying to pass through an SR-IOV virtual function to a PV domU.
> The pci passthrough seems to work, but the virtual function never shows
> up on the pci bus of the domU.  All I see is a message stating:
> [  342.002602] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
> 
> I'm using kernel 2.6.31.12-0.1-xen from OpenSUSE on the dom0 and domU
> and Xen 4.0.  I've recompiled with the CONFIG_XEN_PCIDEV_FE_DEBUG option
> set, but I don't get any additional output when attaching the device.
> 
> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?
> 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?
> 
> For what it's worth, I've also tried kernel 2.6.34 with the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.
> 
> Thanks,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: SR-IOV function passthrough to a PV domU

[Ian Pratt]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> PV domU?

Yes, though you need to use a dom0 kernel with the attached pciback patch.

XCP 0.5 has the relevant patch.

Ian

 
> I noticed in this thread:
> http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> That someone mentioned a specific SR-IOV pcifront driver.
> Does anyone know where this driver may be available?
> 
> For what it's worth, I've also tried kernel 2.6.34 with the xen patches
> from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> the same results.
> 
> Thanks,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #2: 00-pciback --]
[-- Type: application/octet-stream, Size: 3000 bytes --]

diff --git a/drivers/xen/pciback/conf_space_header.c b/drivers/xen/pciback/conf_space_header.c
index 3ae7da1..66a46dd 100644
--- a/drivers/xen/pciback/conf_space_header.c
+++ b/drivers/xen/pciback/conf_space_header.c
@@ -74,17 +74,7 @@ static int rom_write(struct pci_dev *dev, int offset, u32 value, void *data)
 	/* A write to obtain the length must happen as a 32-bit write.
 	 * This does not (yet) support writing individual bytes
 	 */
-	if (value == ~PCI_ROM_ADDRESS_ENABLE)
-		bar->which = 1;
-	else {
-		u32 tmpval;
-		pci_read_config_dword(dev, offset, &tmpval);
-		if (tmpval != bar->val && value == bar->val) {
-			/* Allow restoration of bar value. */
-			pci_write_config_dword(dev, offset, bar->val);
-		}
-		bar->which = 0;
-	}
+	bar->which = (value == ~PCI_ROM_ADDRESS_ENABLE);
 
 	/* Do we need to support enabling/disabling the rom address here? */
 
@@ -108,17 +98,7 @@ static int bar_write(struct pci_dev *dev, int offset, u32 value, void *data)
 	/* A write to obtain the length must happen as a 32-bit write.
 	 * This does not (yet) support writing individual bytes
 	 */
-	if (value == ~0)
-		bar->which = 1;
-	else {
-		u32 tmpval;
-		pci_read_config_dword(dev, offset, &tmpval);
-		if (tmpval != bar->val && value == bar->val) {
-			/* Allow restoration of bar value. */
-			pci_write_config_dword(dev, offset, bar->val);
-		}
-		bar->which = 0;
-	}
+	bar->which = (value == ~0);
 
 	return 0;
 }
@@ -126,6 +106,10 @@ static int bar_write(struct pci_dev *dev, int offset, u32 value, void *data)
 static int bar_read(struct pci_dev *dev, int offset, u32 * value, void *data)
 {
 	struct pci_bar_info *bar = data;
+	int idx = (offset - 0x10) >> 2;
+
+	if (idx > PCI_STD_RESOURCE_END )
+		idx = PCI_ROM_RESOURCE;
 
 	if (unlikely(!bar)) {
 		printk(KERN_WARNING "pciback: driver data not found for %s\n",
@@ -133,7 +117,8 @@ static int bar_read(struct pci_dev *dev, int offset, u32 * value, void *data)
 		return XEN_PCI_ERR_op_failed;
 	}
 
-	*value = bar->which ? bar->len_val : bar->val;
+	*value = bar->which ? ~(pci_resource_len(dev, idx)-1) : pci_resource_start(dev, idx);
+	*value |= pci_resource_flags(dev, idx) & 0xf;
 
 	return 0;
 }
@@ -194,6 +179,22 @@ static int interrupt_read(struct pci_dev *dev, int offset, u8 * value,
 	return 0;
 }
 
+static int vendor_read(struct pci_dev *dev, int offset, u16 * value,
+		       void *data)
+{
+	*value = dev->vendor;
+
+	return 0;
+}
+
+static int device_read(struct pci_dev *dev, int offset, u16 * value,
+		       void *data)
+{
+	*value = dev->device;
+
+	return 0;
+}
+
 static int bist_write(struct pci_dev *dev, int offset, u8 value, void *data)
 {
 	u8 cur_value;
@@ -213,6 +214,16 @@ out:
 
 static const struct config_field header_common[] = {
 	{
+	 .offset    = PCI_VENDOR_ID,
+	 .size      = 2,
+	 .u.w.read  = vendor_read
+	},
+	{
+	 .offset    = PCI_DEVICE_ID,
+	 .size      = 2,
+	 .u.w.read  = device_read
+	},
+	{
 	 .offset    = PCI_COMMAND,
 	 .size      = 2,
 	 .u.w.read  = pciback_read_config_word,

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: SR-IOV function passthrough to a PV domU

[Pasi Kärkkäinen]

On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt wrote:
> > Is it possible to use pci passthrough with SR-IOV virtual functions to a
> > PV domU?
> 
> Yes, though you need to use a dom0 kernel with the attached pciback patch.
> 
> XCP 0.5 has the relevant patch.
> 

I wonder if pvops dom0 kernels have this patch? 

-- Pasi

> 
>  
> > I noticed in this thread:
> > http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00082.html
> > That someone mentioned a specific SR-IOV pcifront driver.
> > Does anyone know where this driver may be available?
> > 
> > For what it's worth, I've also tried kernel 2.6.34 with the xen patches
> > from http://code.google.com/p/gentoo-xen-kernel/downloads/list and I get
> > the same results.
> > 
> > Thanks,
> > Chris
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel


> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: SR-IOV function passthrough to a PV domU

[Keir Fraser]

On 16/06/2010 07:44, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

> On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt wrote:
>>> Is it possible to use pci passthrough with SR-IOV virtual functions to a
>>> PV domU?
>> 
>> Yes, though you need to use a dom0 kernel with the attached pciback patch.
>> 
>> XCP 0.5 has the relevant patch.
>> 
> 
> I wonder if pvops dom0 kernels have this patch?

I think it does, in the .31 branch at least.

 -- Keir

RE: SR-IOV function passthrough to a PV domU

[Jan Beulich]

>>> On 15.06.10 at 23:31, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>  Is it possible to use pci passthrough with SR-IOV virtual functions to a
>> PV domU?
> 
> Yes, though you need to use a dom0 kernel with the attached pciback patch.

That patch is an almost complete revert of c/s 999 - is that really the
right thing to have?

Jan

Re: SR-IOV function passthrough to a PV domU

[Jan Beulich]

>>> On 15.06.10 at 23:29, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> The pciback driver in dom0 needs patching.
> 
> See http://xenbits.xensource.com/linux-2.6.18-xen.hg changeset 999, for
> example. I would imagine some vendor kernels may be missing this, although I
> thought Novell had it. Jan might be able to say (cc'ed).

The 2.6.31 based tree of ours doesn't have it, but the 2.6.32 and
2.6.34 based ones do.

I would consider adding that c/s to the .31 tree, but only if I knew it
indeed helps (with you saying .34 - though it's not really clear what
patch level you have, since I understand it's not a SuSE kernel you're
using - doesn't work for you either it doesn't seem like it would).

For the .32 tree we definitely know this works for others, and I have
no reason to believe that it got broken with the update to .34.

Jan

Re: SR-IOV function passthrough to a PV domU

[Keir Fraser]

On 16/06/2010 08:59, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 15.06.10 at 23:31, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>>  Is it possible to use pci passthrough with SR-IOV virtual functions to a
>>> PV domU?
>> 
>> Yes, though you need to use a dom0 kernel with the attached pciback patch.
> 
> That patch is an almost complete revert of c/s 999 - is that really the
> right thing to have?

I expect Ian attached my patch to pciback. C/s 999 reverts that and applies
Intel's alternative patch in its place, since we decided that was a slightly
better fix. Really either patch would do.

The existence of Intel's patch was raised by you, Jan, so I assumed you must
already have the Intel fix in your trees.

 -- Keir

> Jan

Re: SR-IOV function passthrough to a PV domU

[Jan Beulich]

>>> On 16.06.10 at 10:37, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> The existence of Intel's patch was raised by you, Jan, so I assumed you must
> already have the Intel fix in your trees.

Yes, we started having it as a tentative patch around the release of
11.2 (using the .31 kernel). It never got validated there, so it never
got committed to the repo for that tree. It did get validated for .32
(SLE11 SP1), and then when I saw your version of the patch appear
I submitted ours (or really Intel's) as a replacement. And I think I
even had submitted it quite a while earlier, but it didn't get applied
for a reason I no longer remember. Or maybe I had just asked Intel
to submit it, and they never did...

Jan

Re: SR-IOV function passthrough to a PV domU

[Konrad Rzeszutek Wilk]

On Wed, Jun 16, 2010 at 08:09:12AM +0100, Keir Fraser wrote:
> On 16/06/2010 07:44, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > On Tue, Jun 15, 2010 at 10:31:02PM +0100, Ian Pratt wrote:
> >>> Is it possible to use pci passthrough with SR-IOV virtual functions to a
> >>> PV domU?
> >> 
> >> Yes, though you need to use a dom0 kernel with the attached pciback patch.
> >> 
> >> XCP 0.5 has the relevant patch.
> >> 
> > 
> > I wonder if pvops dom0 kernels have this patch?
> 
> I think it does, in the .31 branch at least.

And .32.

Re: SR-IOV function passthrough to a PV domU

[Chris Reeves]

Thanks for the help!  I used the patch Ian attached, and I could
successfully pass through the VFs.  I'll try changeset 999 next time if
that is the preferred solution.

Jan, I can try c/s 999 on my setup sooner if you'd be interested in the
results.

Thanks again,
Chris

Re: SR-IOV function passthrough to a PV domU

[Keir Fraser]

On 17/06/2010 01:15, "Chris Reeves" <chrisr@spectralogic.com> wrote:

> Thanks for the help!  I used the patch Ian attached, and I could
> successfully pass through the VFs.  I'll try changeset 999 next time if
> that is the preferred solution.
> 
> Jan, I can try c/s 999 on my setup sooner if you'd be interested in the
> results.

I think whatever works for you is just fine to carry on with. As Jan says, a
suitable patch is already applied to more recent Novell kernels.

 -- Keir

> Thanks again,
> Chris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel