PCI Passthrough and PV-OPS

PCI Passthrough and PV-OPS

[Kinsella, Ray]

[-- Attachment #1.1: Type: text/plain, Size: 2351 bytes --]

Hi all,

I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it.
In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo.

I am using dual port SR-IOV NIC's to test, passing through physical functions only.
Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci.
The guest detects the new device and loads the driver to service the NIC, as you would expect.

The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest.
In the Xen log, VT-d is producing errors similar to this one;

(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 06h] PTE Read access is not set
(XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584
(XEN)     root_entry = ffff83016ff38000
(XEN)     root_entry[3] = 14d90a001
(XEN)     context = ffff83014d90a000
(XEN)     context[0] = 102_152209005
(XEN)     l4 = ffff830152209000
(XEN)     l4_index = 0
(XEN)     l4[0] = 152208003
(XEN)     l3 = ffff830152208000
(XEN)     l3_index = 1
(XEN)     l3[1] = 0
(XEN)     l3[1] not present

My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above?

Has anyone else encountered this issue?

Thanks,

Ray Kinsella
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.


[-- Attachment #1.2: Type: text/html, Size: 8410 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI Passthrough and PV-OPS

[Konrad Rzeszutek Wilk]

On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote:
> Hi all,
> 
> I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it.
> In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo.

Do you have 'iommu=soft' in your guest configuration?

> 
> I am using dual port SR-IOV NIC's to test, passing through physical functions only.
> Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci.
> The guest detects the new device and loads the driver to service the NIC, as you would expect.
> 
> The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest.
> In the Xen log, VT-d is producing errors similar to this one;
> 
> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584
> (XEN)     root_entry = ffff83016ff38000
> (XEN)     root_entry[3] = 14d90a001
> (XEN)     context = ffff83014d90a000
> (XEN)     context[0] = 102_152209005
> (XEN)     l4 = ffff830152209000
> (XEN)     l4_index = 0
> (XEN)     l4[0] = 152208003
> (XEN)     l3 = ffff830152208000
> (XEN)     l3_index = 1
> (XEN)     l3[1] = 0
> (XEN)     l3[1] not present
> 
> My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above?
> 
> Has anyone else encountered this issue?

Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel command line argument.

RE: PCI Passthrough and PV-OPS

[Kinsella, Ray]

Hi Konrad,

Thanks for your response.
My understanding of this solution is that we are essentially switching off VT-d and using a software solution instead (bounce buffers?).
Am I correct?.

Ray Kinsella

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Thursday, March 03, 2011 3:03 PM
To: Kinsella, Ray
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] PCI Passthrough and PV-OPS

On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote:
> Hi all,
> 
> I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it.
> In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo.

Do you have 'iommu=soft' in your guest configuration?

> 
> I am using dual port SR-IOV NIC's to test, passing through physical functions only.
> Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci.
> The guest detects the new device and loads the driver to service the NIC, as you would expect.
> 
> The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest.
> In the Xen log, VT-d is producing errors similar to this one;
> 
> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584
> (XEN)     root_entry = ffff83016ff38000
> (XEN)     root_entry[3] = 14d90a001
> (XEN)     context = ffff83014d90a000
> (XEN)     context[0] = 102_152209005
> (XEN)     l4 = ffff830152209000
> (XEN)     l4_index = 0
> (XEN)     l4[0] = 152208003
> (XEN)     l3 = ffff830152208000
> (XEN)     l3_index = 1
> (XEN)     l3[1] = 0
> (XEN)     l3[1] not present
> 
> My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above?
> 
> Has anyone else encountered this issue?

Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel command line argument.
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Re: PCI Passthrough and PV-OPS

[Konrad Rzeszutek Wilk]

On Thu, Mar 03, 2011 at 07:07:48PM +0000, Kinsella, Ray wrote:
> Hi Konrad,
> 
> Thanks for your response.
> My understanding of this solution is that we are essentially switching off VT-d and using a software solution instead (bounce buffers?).

No.
> Am I correct?.

There are _two_ swiotlb implementation in the pvops kernel.

The baremetal is to be used under baremetal and it won't be turned
on when running DomU/Dom0.

The other, xen-swiotlb is turned on by default when dom0 boots. And
it is turned on for DomU if the user specified 'iommu=soft'.

The xen-swiotlb has two modes: bounce buffer for devices that need it (which nowadays is USB), and
translate the PFNs to MFNs (and vice-versa). We want to use the second functionality
which hooks up in the PCI DMA. You can turn on the bounce buffer for everything
if you specify 'swiotlb=force', but there is no point of doing that.

So in essence you have Xen hypervisor doing VT-D, dom0 and domU both using
the Linux DMA API wherein the Xen-SWIOTLB is hooked up. The Xen-SWIOTLB runs
in translation mode and when a device is setup using the DMA API it provides
the MFNs instead of the PFNs to the driver.

> 
> Ray Kinsella
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
> Sent: Thursday, March 03, 2011 3:03 PM
> To: Kinsella, Ray
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] PCI Passthrough and PV-OPS
> 
> On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote:
> > Hi all,
> > 
> > I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it.
> > In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo.
> 
> Do you have 'iommu=soft' in your guest configuration?
> 
> > 
> > I am using dual port SR-IOV NIC's to test, passing through physical functions only.
> > Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci.
> > The guest detects the new device and loads the driver to service the NIC, as you would expect.
> > 
> > The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest.
> > In the Xen log, VT-d is producing errors similar to this one;
> > 
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000
> > (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> > (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584
> > (XEN)     root_entry = ffff83016ff38000
> > (XEN)     root_entry[3] = 14d90a001
> > (XEN)     context = ffff83014d90a000
> > (XEN)     context[0] = 102_152209005
> > (XEN)     l4 = ffff830152209000
> > (XEN)     l4_index = 0
> > (XEN)     l4[0] = 152208003
> > (XEN)     l3 = ffff830152208000
> > (XEN)     l3_index = 1
> > (XEN)     l3[1] = 0
> > (XEN)     l3[1] not present
> > 
> > My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above?
> > 
> > Has anyone else encountered this issue?
> 
> Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel command line argument.
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
> 
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel