From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Kinsella, Ray" Subject: PCI Passthrough and PV-OPS Date: Thu, 3 Mar 2011 14:25:47 +0000 Message-ID: <7D07DB03D5AECA4CB23F2CA801AA720BF9694916@irsmsx502.ger.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2115902537==" Return-path: Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org --===============2115902537== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_7D07DB03D5AECA4CB23F2CA801AA720BF9694916irsmsx502gercor_" --_000_7D07DB03D5AECA4CB23F2CA801AA720BF9694916irsmsx502gercor_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux P= V Guests, and I was wondering if anyone else had seen it. In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Je= remy Fitzhartridge's Git repo. I am using dual port SR-IOV NIC's to test, passing through physical functio= ns only. Passthrough appears to work, I can passthrough the NIC and it will appear i= n the guest with lspci. The guest detects the new device and loads the driver to service the NIC, a= s you would expect. The guest is complaining about the NIC being hung, the message "Detected Tx= Unit Hang..." is appearing in the system log on the guest. In the Xen log, VT-d is producing errors similar to this one; (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault add= r 7c584000, iommu reg =3D ffff82c3fff57000 (XEN) DMAR:[fault reason 06h] PTE Read access is not set (XEN) print_vtd_entries: iommu =3D ffff83016fffa5f0 bdf =3D 3:0.0 gmfn =3D = 7c584 (XEN) root_entry =3D ffff83016ff38000 (XEN) root_entry[3] =3D 14d90a001 (XEN) context =3D ffff83014d90a000 (XEN) context[0] =3D 102_152209005 (XEN) l4 =3D ffff830152209000 (XEN) l4_index =3D 0 (XEN) l4[0] =3D 152208003 (XEN) l3 =3D ffff830152208000 (XEN) l3_index =3D 1 (XEN) l3[1] =3D 0 (XEN) l3[1] not present My understanding of what is happening here is that page table mappings are = missing from VT-d's page table, mapping machine physical addresses to my PV= guest's physical addresses. I have had a look at iommu_populate_page_table= and I dumped the mappings as they are being setup but couldn't see a mappi= ng for the GPA 0x7c584000 above? Has anyone else encountered this issue? Thanks, Ray Kinsella -------------------------------------------------------------- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others = is strictly prohibited. If you are not the intended recipient, please conta= ct the sender and delete all copies. --_000_7D07DB03D5AECA4CB23F2CA801AA720BF9694916irsmsx502gercor_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi all,

I am having a problem with Xen 4.0.1, PCI Passthrough = with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it.

In both Dom 0 and Dom U’s I am using 64bit PV-OP= S 2.6.32.26 Kernels from Jeremy Fitzhartridge’s Git repo.

I am using dual port SR-IOV NIC’s to test, passi= ng through physical functions only.

Passthrough appears to work, I can passthrough the NIC= and it will appear in the guest with lspci.

The guest detects the new device and loads the driver = to service the NIC, as you would expect.

The guest is complaining about the NIC being hung, the message “Detected Tx Unit Hang...” is appearing in the system log on the guest.

In the Xen log, VT-d is producing errors similar to th= is one;

(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary P= ending Fault

(XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg =3D ffff82c3fff57000

(XEN) DMAR:[fault reason 06h] PTE Read access is not s= et

(XEN) print_vtd_entries: iommu =3D ffff83016fffa5f0 bd= f =3D 3:0.0 gmfn =3D 7c584

(XEN) root_entry =3D ffff83016= ff38000

(XEN) root_entry[3] =3D 14d90a= 001

(XEN) context =3D ffff83014d90= a000

(XEN) context[0] =3D 102_15220= 9005

(XEN) l4 =3D ffff830152209000<= o:p>

(XEN) l4_index =3D 0

(XEN) l4[0] =3D 152208003=

(XEN) l3 =3D ffff830152208000<= o:p>

(XEN) l3_index =3D 1

(XEN) l3[1] =3D 0

(XEN) l3[1] not present

My understanding of what is happening here is that page table mappings are missing from VT-d’s page table, mapping machine ph= ysical addresses to my PV guest’s physical addresses. I h= ave had a look at iommu_populate_page_table and I dumped the mappings as they a= re being setup but couldn’t see a mapping for the GPA 0x7c584000 above?<= o:p>

Has anyone else encountered this is= sue?

Thanks,

Ray Kinsella

--------------------------------------------------= ------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others = is strictly prohibited. If you are not the intended recipient, please conta= ct the sender and delete all copies.
--_000_7D07DB03D5AECA4CB23F2CA801AA720BF9694916irsmsx502gercor_-- --===============2115902537== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============2115902537==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: PCI Passthrough and PV-OPS Date: Thu, 3 Mar 2011 10:03:12 -0500 Message-ID: <20110303150312.GA8024@dumpdata.com> References: <7D07DB03D5AECA4CB23F2CA801AA720BF9694916@irsmsx502.ger.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <7D07DB03D5AECA4CB23F2CA801AA720BF9694916@irsmsx502.ger.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Kinsella, Ray" Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote: > Hi all, > > I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it. > In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo. Do you have 'iommu=soft' in your guest configuration? > > I am using dual port SR-IOV NIC's to test, passing through physical functions only. > Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci. > The guest detects the new device and loads the driver to service the NIC, as you would expect. > > The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest. > In the Xen log, VT-d is producing errors similar to this one; > > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000 > (XEN) DMAR:[fault reason 06h] PTE Read access is not set > (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584 > (XEN) root_entry = ffff83016ff38000 > (XEN) root_entry[3] = 14d90a001 > (XEN) context = ffff83014d90a000 > (XEN) context[0] = 102_152209005 > (XEN) l4 = ffff830152209000 > (XEN) l4_index = 0 > (XEN) l4[0] = 152208003 > (XEN) l3 = ffff830152208000 > (XEN) l3_index = 1 > (XEN) l3[1] = 0 > (XEN) l3[1] not present > > My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above? > > Has anyone else encountered this issue? Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel command line argument. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Kinsella, Ray" Subject: RE: PCI Passthrough and PV-OPS Date: Thu, 3 Mar 2011 19:07:48 +0000 Message-ID: <7D07DB03D5AECA4CB23F2CA801AA720BF9694C38@irsmsx502.ger.corp.intel.com> References: <7D07DB03D5AECA4CB23F2CA801AA720BF9694916@irsmsx502.ger.corp.intel.com> <20110303150312.GA8024@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20110303150312.GA8024@dumpdata.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Konrad, Thanks for your response. My understanding of this solution is that we are essentially switching off = VT-d and using a software solution instead (bounce buffers?). Am I correct?. Ray Kinsella -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20 Sent: Thursday, March 03, 2011 3:03 PM To: Kinsella, Ray Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] PCI Passthrough and PV-OPS On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote: > Hi all, >=20 > I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux= PV Guests, and I was wondering if anyone else had seen it. > In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from = Jeremy Fitzhartridge's Git repo. Do you have 'iommu=3Dsoft' in your guest configuration? >=20 > I am using dual port SR-IOV NIC's to test, passing through physical funct= ions only. > Passthrough appears to work, I can passthrough the NIC and it will appear= in the guest with lspci. > The guest detects the new device and loads the driver to service the NIC,= as you would expect. >=20 > The guest is complaining about the NIC being hung, the message "Detected = Tx Unit Hang..." is appearing in the system log on the guest. > In the Xen log, VT-d is producing errors similar to this one; >=20 > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault a= ddr 7c584000, iommu reg =3D ffff82c3fff57000 > (XEN) DMAR:[fault reason 06h] PTE Read access is not set > (XEN) print_vtd_entries: iommu =3D ffff83016fffa5f0 bdf =3D 3:0.0 gmfn = =3D 7c584 > (XEN) root_entry =3D ffff83016ff38000 > (XEN) root_entry[3] =3D 14d90a001 > (XEN) context =3D ffff83014d90a000 > (XEN) context[0] =3D 102_152209005 > (XEN) l4 =3D ffff830152209000 > (XEN) l4_index =3D 0 > (XEN) l4[0] =3D 152208003 > (XEN) l3 =3D ffff830152208000 > (XEN) l3_index =3D 1 > (XEN) l3[1] =3D 0 > (XEN) l3[1] not present >=20 > My understanding of what is happening here is that page table mappings ar= e missing from VT-d's page table, mapping machine physical addresses to my = PV guest's physical addresses. I have had a look at iommu_populate_page_tab= le and I dumped the mappings as they are being setup but couldn't see a map= ping for the GPA 0x7c584000 above? >=20 > Has anyone else encountered this issue? Yes. And it was fixed by running a PV guest with 'iommu=3Dsoft' as Linux ke= rnel command line argument. -------------------------------------------------------------- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others = is strictly prohibited. If you are not the intended recipient, please conta= ct the sender and delete all copies. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: PCI Passthrough and PV-OPS Date: Thu, 3 Mar 2011 15:25:09 -0500 Message-ID: <20110303202509.GA14899@dumpdata.com> References: <7D07DB03D5AECA4CB23F2CA801AA720BF9694916@irsmsx502.ger.corp.intel.com> <20110303150312.GA8024@dumpdata.com> <7D07DB03D5AECA4CB23F2CA801AA720BF9694C38@irsmsx502.ger.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <7D07DB03D5AECA4CB23F2CA801AA720BF9694C38@irsmsx502.ger.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Kinsella, Ray" Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org On Thu, Mar 03, 2011 at 07:07:48PM +0000, Kinsella, Ray wrote: > Hi Konrad, > > Thanks for your response. > My understanding of this solution is that we are essentially switching off VT-d and using a software solution instead (bounce buffers?). No. > Am I correct?. There are _two_ swiotlb implementation in the pvops kernel. The baremetal is to be used under baremetal and it won't be turned on when running DomU/Dom0. The other, xen-swiotlb is turned on by default when dom0 boots. And it is turned on for DomU if the user specified 'iommu=soft'. The xen-swiotlb has two modes: bounce buffer for devices that need it (which nowadays is USB), and translate the PFNs to MFNs (and vice-versa). We want to use the second functionality which hooks up in the PCI DMA. You can turn on the bounce buffer for everything if you specify 'swiotlb=force', but there is no point of doing that. So in essence you have Xen hypervisor doing VT-D, dom0 and domU both using the Linux DMA API wherein the Xen-SWIOTLB is hooked up. The Xen-SWIOTLB runs in translation mode and when a device is setup using the DMA API it provides the MFNs instead of the PFNs to the driver. > > Ray Kinsella > > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Thursday, March 03, 2011 3:03 PM > To: Kinsella, Ray > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] PCI Passthrough and PV-OPS > > On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote: > > Hi all, > > > > I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux PV Guests, and I was wondering if anyone else had seen it. > > In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from Jeremy Fitzhartridge's Git repo. > > Do you have 'iommu=soft' in your guest configuration? > > > > > I am using dual port SR-IOV NIC's to test, passing through physical functions only. > > Passthrough appears to work, I can passthrough the NIC and it will appear in the guest with lspci. > > The guest detects the new device and loads the driver to service the NIC, as you would expect. > > > > The guest is complaining about the NIC being hung, the message "Detected Tx Unit Hang..." is appearing in the system log on the guest. > > In the Xen log, VT-d is producing errors similar to this one; > > > > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault > > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault addr 7c584000, iommu reg = ffff82c3fff57000 > > (XEN) DMAR:[fault reason 06h] PTE Read access is not set > > (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584 > > (XEN) root_entry = ffff83016ff38000 > > (XEN) root_entry[3] = 14d90a001 > > (XEN) context = ffff83014d90a000 > > (XEN) context[0] = 102_152209005 > > (XEN) l4 = ffff830152209000 > > (XEN) l4_index = 0 > > (XEN) l4[0] = 152208003 > > (XEN) l3 = ffff830152208000 > > (XEN) l3_index = 1 > > (XEN) l3[1] = 0 > > (XEN) l3[1] not present > > > > My understanding of what is happening here is that page table mappings are missing from VT-d's page table, mapping machine physical addresses to my PV guest's physical addresses. I have had a look at iommu_populate_page_table and I dumped the mappings as they are being setup but couldn't see a mapping for the GPA 0x7c584000 above? > > > > Has anyone else encountered this issue? > > Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel command line argument. > -------------------------------------------------------------- > Intel Shannon Limited > Registered in Ireland > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare > Registered Number: 308263 > Business address: Dromore House, East Park, Shannon, Co. Clare > > This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel