From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lin, Ray" Subject: PCI devices remap from dom0 to domU with pv-ops kernel Date: Wed, 13 Oct 2010 10:49:12 -0600 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1959932082==" Return-path: Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" , "bruce.edge@gmail.com" List-Id: xen-devel@lists.xenproject.org --===============1959932082== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_EB4C61A1A2501842A04B573FE42B14D601319D07DEcosmail02lsic_" --_000_EB4C61A1A2501842A04B573FE42B14D601319D07DEcosmail02lsic_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Konrad, We try to remap the pci devices from dom0 to domU by assigning the pci = in the domU configuration file. Such as pci=3D"['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']" In hvm kernel, the domU would see the pci devices remapped like this, dom0 pci device 01:00.0 =3D> domU pci device 00:11.0 dom0 pci device 01:00.1 =3D> domU pci device 00:10.0 dom0 pci device 04:00.0 =3D> domU pci device 00:13.0 dom0 pci device 04:00.1 =3D> domU pci device 00:12.0 But pv-ops kernel seems ignore the remapping. Is there any way (kernel con= figuration setting ??) to make it work ? Thanks, Ray --_000_EB4C61A1A2501842A04B573FE42B14D601319D07DEcosmail02lsic_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Konrad,

We try to remap the pci devices fr= om dom0 to domU by assigning the pci in the domU configuration file. Such a= s

pci=3D"['01:00.0@11', '01:00.= 1@10', '04:00.0@13', '04:00.1@12']"

In hvm kernel, the domU would see the pci devices rem= apped like this,

dom0 pci device 01:00.0 = ; =3D> domU pci device 00:11.0

dom0 pci device 01:00.1 =3D&= gt; domU pci device 00:10.0

dom0 pci device 04:00.0 =3D&= gt; domU pci device 00:13.0

dom0 pci device 04:00.1 =3D&= gt; domU pci device 00:12.0

But pv-ops kernel seems ignore the remapping. Is the= re any way (kernel configuration setting ??) to make it work ?

Thanks,

Ray

--_000_EB4C61A1A2501842A04B573FE42B14D601319D07DEcosmail02lsic_-- --===============1959932082== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1959932082==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel Date: Wed, 13 Oct 2010 14:51:49 -0400 Message-ID: <20101013185149.GA14056@dumpdata.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Lin, Ray" Cc: "xen-devel@lists.xensource.com" , "bruce.edge@gmail.com" List-Id: xen-devel@lists.xenproject.org On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: > > Konrad, > > We try to remap the pci devices from dom0 to domU by assigning the pci in the domU configuration file. Such as > > pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']" > > In hvm kernel, the domU would see the pci devices remapped like this, > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > dom0 pci device 01:00.1 => domU pci device 00:10.0 > dom0 pci device 04:00.0 => domU pci device 00:13.0 > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ? Not completly. When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire. > > Thanks, > Ray > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lin, Ray" Subject: RE: PCI devices remap from dom0 to domU with pv-ops kernel Date: Thu, 14 Oct 2010 11:26:03 -0600 Message-ID: References: <20101013185149.GA14056@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101013185149.GA14056@dumpdata.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" , "bruce.edge@gmail.com" List-Id: xen-devel@lists.xenproject.org Konrad, We choose "Virtual" and the dom0/domU show like this. It doesn't remap to= what we want. You mentioned about "write your own plugin that would do rem= apping per your desire". Could you elaborate it a bit ? Thanks. Dom0: kaan-18:~# lspci -n | grep 11f8 07:00.0 0c04: 11f8:8032 (rev 08) 07:00.1 0c04: 11f8:8032 (rev 08) 07:00.2 0c04: 11f8:8032 (rev 08) 07:00.3 0c04: 11f8:8032 (rev 08) DomU: kaan-18-dpm:~# lspci -n 00:00.0 0c04: 11f8:8032 (rev 08) 00:00.1 0c04: 11f8:8032 (rev 08) 00:00.2 0c04: 11f8:8032 (rev 08) 00:00.3 0c04: 11f8:8032 (rev 08) -Ray -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20 Sent: Wednesday, October 13, 2010 11:52 AM To: Lin, Ray Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: >=20 > Konrad, >=20 > We try to remap the pci devices from dom0 to domU by assigning the=20 > pci in the domU configuration file. Such as >=20 > pci=3D"['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']" >=20 > In hvm kernel, the domU would see the pci devices remapped like this, >=20 > dom0 pci device 01:00.0 =3D> domU pci device 00:11.0 > dom0 pci device 01:00.1 =3D> domU pci device 00:10.0 > dom0 pci device 04:00.0 =3D> domU pci device 00:13.0 > dom0 pci device 04:00.1 =3D> domU pci device 00:12.0 >=20 > But pv-ops kernel seems ignore the remapping. Is there any way (kernel c= onfiguration setting ??) to make it work ? Not completly. When you compile your Xen PCI backend, you should see an option whether you= want Virtual PCI, Passthrough or some other type. By default it will pick = the Virtual one so that it just enumerates from zero the PCI devices. You c= ould write your own plugin that would do remaping per your desire. >=20 > Thanks, > Ray >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: RE: PCI devices remap from dom0 to domU with pv-ops kernel Date: Thu, 14 Oct 2010 13:49:09 -0400 Message-ID: <20101014174909.GB7312@dumpdata.com> References: <20101013185149.GA14056@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Lin, Ray" Cc: "xen-devel@lists.xensource.com" , "bruce.edge@gmail.com" List-Id: xen-devel@lists.xenproject.org On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote: > > Konrad, > > We choose "Virtual" and the dom0/domU show like this. It doesn't remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks. Sure. Look in drivers/xen/pciback. There are four files of interest: vpci.c slot.c passthrough.c controller.c By default, the vpci.c is selected, where in the domU the devices start from 00:00.0. The passthrough is obvious. The slot.c is where we have only two bus and all the devices are stuck under the slots. So a maximum of 64 devices, and the device function is ignored: This is what it would show up (I think, you might want to double-check this): 00:01.0:0c04: 11f8:8032 (rev 08) 00:02.0:0c04: 11f8:8032 (rev 08) 00:03.0:0c04: 11f8:8032 (rev 08) 00:04.0:0c04: 11f8:8032 (rev 08) The controller, is "PCI "Controller" Backend - virtualize PCI bus topology based on PCI controllers. Devices under the same PCI controller are exposed on the same virtual domain:bus. Within a bus, device slots are virtualized to compact the bus. " (from controller.c) I think based on what you said earlier, you just want to use the "slot" backend: CONFIG_XEN_PCIDEV_BACKEND_SLOT=y > > Dom0: > kaan-18:~# lspci -n | grep 11f8 > 07:00.0 0c04: 11f8:8032 (rev 08) > 07:00.1 0c04: 11f8:8032 (rev 08) > 07:00.2 0c04: 11f8:8032 (rev 08) > 07:00.3 0c04: 11f8:8032 (rev 08) > > DomU: > kaan-18-dpm:~# lspci -n > 00:00.0 0c04: 11f8:8032 (rev 08) > 00:00.1 0c04: 11f8:8032 (rev 08) > 00:00.2 0c04: 11f8:8032 (rev 08) > 00:00.3 0c04: 11f8:8032 (rev 08) > > > -Ray > > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Wednesday, October 13, 2010 11:52 AM > To: Lin, Ray > Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com > Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel > > On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: > > > > Konrad, > > > > We try to remap the pci devices from dom0 to domU by assigning the > > pci in the domU configuration file. Such as > > > > pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']" > > > > In hvm kernel, the domU would see the pci devices remapped like this, > > > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > > dom0 pci device 01:00.1 => domU pci device 00:10.0 > > dom0 pci device 04:00.0 => domU pci device 00:13.0 > > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ? > > Not completly. > > When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire. > > > > Thanks, > > Ray > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lin, Ray" Subject: RE: RE: PCI devices remap from dom0 to domU with pv-ops kernel Date: Thu, 14 Oct 2010 11:54:33 -0600 Message-ID: References: <20101014174909.GB7312@dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20101014174909.GB7312@dumpdata.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" , "bruce.edge@gmail.com" List-Id: xen-devel@lists.xenproject.org Thanks. I'll look into it. -Ray=20 -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]=20 Sent: Thursday, October 14, 2010 10:49 AM To: Lin, Ray Cc: xen-devel@lists.xensource.com; bruce.edge@gmail.com Subject: Re: [Xen-devel] RE: PCI devices remap from dom0 to domU with pv-op= s kernel On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote: >=20 > Konrad, >=20 > We choose "Virtual" and the dom0/domU show like this. It doesn't remap = to what we want. You mentioned about "write your own plugin that would do r= emapping per your desire". Could you elaborate it a bit ? Thanks. Sure. Look in drivers/xen/pciback. There are four files of interest: vpci.c slot.c passthrough.c controller.c By default, the vpci.c is selected, where in the domU the devices start fro= m 00:00.0. The passthrough is obvious.=20 The slot.c is where we have only two bus and all the devices are stuck unde= r the slots. So a maximum of 64 devices, and the device function is ignored: This is what it would show up (I think, you might want to double-check this= ): 00:01.0:0c04: 11f8:8032 (rev 08) 00:02.0:0c04: 11f8:8032 (rev 08) 00:03.0:0c04: 11f8:8032 (rev 08) 00:04.0:0c04: 11f8:8032 (rev 08) The controller, is "PCI "Controller" Backend - virtualize PCI bus topology = based on PCI controllers. Devices under the same PCI controller are expos= ed on the same virtual domain:bus. Within a bus, device slots are virtual= ized to compact the bus. " (from controller.c) I think based on what you said earlier, you just want to use the "slot" bac= kend: CONFIG_XEN_PCIDEV_BACKEND_SLOT=3Dy >=20 > Dom0: > kaan-18:~# lspci -n | grep 11f8 > 07:00.0 0c04: 11f8:8032 (rev 08) > 07:00.1 0c04: 11f8:8032 (rev 08) > 07:00.2 0c04: 11f8:8032 (rev 08) > 07:00.3 0c04: 11f8:8032 (rev 08) >=20 > DomU: > kaan-18-dpm:~# lspci -n > 00:00.0 0c04: 11f8:8032 (rev 08) > 00:00.1 0c04: 11f8:8032 (rev 08) > 00:00.2 0c04: 11f8:8032 (rev 08) > 00:00.3 0c04: 11f8:8032 (rev 08) >=20 >=20 > -Ray >=20 > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Wednesday, October 13, 2010 11:52 AM > To: Lin, Ray > Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com > Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel >=20 > On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: > >=20 > > Konrad, > >=20 > > We try to remap the pci devices from dom0 to domU by assigning=20 > > the pci in the domU configuration file. Such as > >=20 > > pci=3D"['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']" > >=20 > > In hvm kernel, the domU would see the pci devices remapped like=20 > > this, > >=20 > > dom0 pci device 01:00.0 =3D> domU pci device 00:11.0 > > dom0 pci device 01:00.1 =3D> domU pci device 00:10.0 > > dom0 pci device 04:00.0 =3D> domU pci device 00:13.0 > > dom0 pci device 04:00.1 =3D> domU pci device 00:12.0 > >=20 > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel= configuration setting ??) to make it work ? >=20 > Not completly. >=20 > When you compile your Xen PCI backend, you should see an option whether y= ou want Virtual PCI, Passthrough or some other type. By default it will pic= k the Virtual one so that it just enumerates from zero the PCI devices. You= could write your own plugin that would do remaping per your desire. > >=20 > > Thanks, > > Ray > >=20 > >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel