PCI devices remap from dom0 to domU with pv-ops kernel

PCI devices remap from dom0 to domU with pv-ops kernel

[Lin, Ray]

[-- Attachment #1.1: Type: text/plain, Size: 645 bytes --]


Konrad,

    We try to remap the pci devices from dom0 to domU by assigning the pci in the domU configuration file. Such as

    pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']"

In hvm kernel, the domU would see the pci devices remapped like this,

    dom0 pci device 01:00.0   => domU pci device 00:11.0
    dom0 pci device 01:00.1  =>  domU pci device 00:10.0
    dom0 pci device 04:00.0  =>  domU pci device 00:13.0
    dom0 pci device 04:00.1  =>  domU pci device 00:12.0

 But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?

Thanks,
Ray



[-- Attachment #1.2: Type: text/html, Size: 1889 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI devices remap from dom0 to domU with pv-ops kernel

[Konrad Rzeszutek Wilk]

On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:
> 
> Konrad,
> 
>     We try to remap the pci devices from dom0 to domU by assigning the pci in the domU configuration file. Such as
> 
>     pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']"
> 
> In hvm kernel, the domU would see the pci devices remapped like this,
> 
>     dom0 pci device 01:00.0   => domU pci device 00:11.0
>     dom0 pci device 01:00.1  =>  domU pci device 00:10.0
>     dom0 pci device 04:00.0  =>  domU pci device 00:13.0
>     dom0 pci device 04:00.1  =>  domU pci device 00:12.0
> 
>  But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?

Not completly.

When you compile your Xen PCI backend, you should see an option
whether you want Virtual PCI, Passthrough or some other type. By default
it will pick the Virtual one so that it just enumerates from
zero the PCI devices. You could write your own plugin that would
do remaping per your desire.
> 
> Thanks,
> Ray
> 
> 

RE: PCI devices remap from dom0 to domU with pv-ops kernel

[Lin, Ray]

Konrad,

  We choose "Virtual" and the dom0/domU show like this. It doesn't remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks.

Dom0:
kaan-18:~# lspci -n | grep 11f8
07:00.0 0c04: 11f8:8032 (rev 08)
07:00.1 0c04: 11f8:8032 (rev 08)
07:00.2 0c04: 11f8:8032 (rev 08)
07:00.3 0c04: 11f8:8032 (rev 08)

DomU:
kaan-18-dpm:~# lspci -n
00:00.0 0c04: 11f8:8032 (rev 08)
00:00.1 0c04: 11f8:8032 (rev 08)
00:00.2 0c04: 11f8:8032 (rev 08)
00:00.3 0c04: 11f8:8032 (rev 08)


-Ray

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Wednesday, October 13, 2010 11:52 AM
To: Lin, Ray
Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com
Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel

On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:
> 
> Konrad,
> 
>     We try to remap the pci devices from dom0 to domU by assigning the 
> pci in the domU configuration file. Such as
> 
>     pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']"
> 
> In hvm kernel, the domU would see the pci devices remapped like this,
> 
>     dom0 pci device 01:00.0   => domU pci device 00:11.0
>     dom0 pci device 01:00.1  =>  domU pci device 00:10.0
>     dom0 pci device 04:00.0  =>  domU pci device 00:13.0
>     dom0 pci device 04:00.1  =>  domU pci device 00:12.0
> 
>  But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?

Not completly.

When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire.
> 
> Thanks,
> Ray
> 
> 

Re: RE: PCI devices remap from dom0 to domU with pv-ops kernel

[Konrad Rzeszutek Wilk]

On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote:
> 
> Konrad,
> 
>   We choose "Virtual" and the dom0/domU show like this. It doesn't remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks.

Sure. Look in drivers/xen/pciback. There are four files of interest:

vpci.c
slot.c
passthrough.c
controller.c

By default, the vpci.c is selected, where in the domU the devices start
from 00:00.0. The passthrough is obvious. 

The slot.c is where we have only two bus and all the devices are stuck under the slots.
So a maximum of 64 devices, and the device function is ignored:

This is what it would show up (I think, you might want to double-check this):
00:01.0:0c04: 11f8:8032 (rev 08)
00:02.0:0c04: 11f8:8032 (rev 08)
00:03.0:0c04: 11f8:8032 (rev 08)
00:04.0:0c04: 11f8:8032 (rev 08)

The controller, is "PCI "Controller" Backend - virtualize PCI bus topology based on PCI
 controllers.  Devices under the same PCI controller are exposed on the
 same virtual domain:bus.  Within a bus, device slots are virtualized
 to compact the bus. " (from controller.c)

I think based on what you said earlier, you just want to use the "slot" backend:
CONFIG_XEN_PCIDEV_BACKEND_SLOT=y
> 
> Dom0:
> kaan-18:~# lspci -n | grep 11f8
> 07:00.0 0c04: 11f8:8032 (rev 08)
> 07:00.1 0c04: 11f8:8032 (rev 08)
> 07:00.2 0c04: 11f8:8032 (rev 08)
> 07:00.3 0c04: 11f8:8032 (rev 08)
> 
> DomU:
> kaan-18-dpm:~# lspci -n
> 00:00.0 0c04: 11f8:8032 (rev 08)
> 00:00.1 0c04: 11f8:8032 (rev 08)
> 00:00.2 0c04: 11f8:8032 (rev 08)
> 00:00.3 0c04: 11f8:8032 (rev 08)
> 
> 
> -Ray
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
> Sent: Wednesday, October 13, 2010 11:52 AM
> To: Lin, Ray
> Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com
> Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel
> 
> On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:
> > 
> > Konrad,
> > 
> >     We try to remap the pci devices from dom0 to domU by assigning the 
> > pci in the domU configuration file. Such as
> > 
> >     pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']"
> > 
> > In hvm kernel, the domU would see the pci devices remapped like this,
> > 
> >     dom0 pci device 01:00.0   => domU pci device 00:11.0
> >     dom0 pci device 01:00.1  =>  domU pci device 00:10.0
> >     dom0 pci device 04:00.0  =>  domU pci device 00:13.0
> >     dom0 pci device 04:00.1  =>  domU pci device 00:12.0
> > 
> >  But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?
> 
> Not completly.
> 
> When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire.
> > 
> > Thanks,
> > Ray
> > 
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: RE: PCI devices remap from dom0 to domU with pv-ops kernel

[Lin, Ray]

Thanks. I'll look into it.


-Ray 

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Thursday, October 14, 2010 10:49 AM
To: Lin, Ray
Cc: xen-devel@lists.xensource.com; bruce.edge@gmail.com
Subject: Re: [Xen-devel] RE: PCI devices remap from dom0 to domU with pv-ops kernel

On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote:
> 
> Konrad,
> 
>   We choose "Virtual" and the dom0/domU show like this. It doesn't remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks.

Sure. Look in drivers/xen/pciback. There are four files of interest:

vpci.c
slot.c
passthrough.c
controller.c

By default, the vpci.c is selected, where in the domU the devices start from 00:00.0. The passthrough is obvious. 

The slot.c is where we have only two bus and all the devices are stuck under the slots.
So a maximum of 64 devices, and the device function is ignored:

This is what it would show up (I think, you might want to double-check this):
00:01.0:0c04: 11f8:8032 (rev 08)
00:02.0:0c04: 11f8:8032 (rev 08)
00:03.0:0c04: 11f8:8032 (rev 08)
00:04.0:0c04: 11f8:8032 (rev 08)

The controller, is "PCI "Controller" Backend - virtualize PCI bus topology based on PCI  controllers.  Devices under the same PCI controller are exposed on the  same virtual domain:bus.  Within a bus, device slots are virtualized  to compact the bus. " (from controller.c)

I think based on what you said earlier, you just want to use the "slot" backend:
CONFIG_XEN_PCIDEV_BACKEND_SLOT=y
> 
> Dom0:
> kaan-18:~# lspci -n | grep 11f8
> 07:00.0 0c04: 11f8:8032 (rev 08)
> 07:00.1 0c04: 11f8:8032 (rev 08)
> 07:00.2 0c04: 11f8:8032 (rev 08)
> 07:00.3 0c04: 11f8:8032 (rev 08)
> 
> DomU:
> kaan-18-dpm:~# lspci -n
> 00:00.0 0c04: 11f8:8032 (rev 08)
> 00:00.1 0c04: 11f8:8032 (rev 08)
> 00:00.2 0c04: 11f8:8032 (rev 08)
> 00:00.3 0c04: 11f8:8032 (rev 08)
> 
> 
> -Ray
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Wednesday, October 13, 2010 11:52 AM
> To: Lin, Ray
> Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com
> Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel
> 
> On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:
> > 
> > Konrad,
> > 
> >     We try to remap the pci devices from dom0 to domU by assigning 
> > the pci in the domU configuration file. Such as
> > 
> >     pci="['01:00.0@11', '01:00.1@10', '04:00.0@13', '04:00.1@12']"
> > 
> > In hvm kernel, the domU would see the pci devices remapped like 
> > this,
> > 
> >     dom0 pci device 01:00.0   => domU pci device 00:11.0
> >     dom0 pci device 01:00.1  =>  domU pci device 00:10.0
> >     dom0 pci device 04:00.0  =>  domU pci device 00:13.0
> >     dom0 pci device 04:00.1  =>  domU pci device 00:12.0
> > 
> >  But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?
> 
> Not completly.
> 
> When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire.
> > 
> > Thanks,
> > Ray
> > 
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel