Xen PCI passthrough & OpenWRT domU

Xen PCI passthrough & OpenWRT domU

[Jean Baptiste Favre]

Hello,
I'm trying to make Openwrt running as domU with network cards PCI
passthrough.

First part is OK, OpenWRT runs fine as x86 PV domU, but not PCI passthrough.

Dom0 runs debian Xen packages.
# cat /proc/cmdline
placeholder root=/dev/mapper/system-root ro console=tty0 console=hvc0
console=ttyS0,115200n8 xen-pciback.permissive xen-pciback.hide=(04:00.0)
pci=resource_alignment=04:00.0 quiet

# lspci
...
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E
Gigabit Ethernet Controller (rev 12)

# uname -a
Linux remus 2.6.32-5-xen-amd64 #1 SMP Fri Dec 10 17:41:50 UTC 2010
x86_64 GNU/Linux

OpenWRT uses upstream kernel 2.6.32.27 (but I also tested 2.6.35.9
without any changes).

domU informations:
# cat /proc/cmdline
root=/dev/xvda2 rw swiotlb=force console=hvc0 xencons=tty

# lspci
[empty]

# dmesg | grep -i pci
Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
PCI: Fatal: No config space access function found
PCI: System does not support PCI
PCI: System does not support PCI
PCI: CLS 0 bytes, default 128
XENBUS: Device with no driver: device/pci/0

# dmesg | grep -i xen
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000010000000 (usable)
  #0 [00014c6000 - 00014d4000]   XEN PAGETABLES ==> [00014c6000 -
00014d4000]
  #3 [0001483000 - 00014c6000]   XEN START INFO ==> [0001483000 -
00014c6000]
Booting paravirtualized kernel on Xen
Xen version: 4.0.1 (preserve-AD)
Xen: using vcpu_info placement
Kernel command line: root=/dev/xvda2 rw swiotlb=force console=hvc0
xencons=tty
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
xen_balloon: Initialising balloon driver.
Switching to clocksource xen
Initialising Xen virtual ethernet driver.
XENBUS: Device with no driver: device/pci/0
XENBUS: Device with no driver: device/console/0

So, it seems that domU sees PCI device (and sky2 module is loaded), but
do not recognize it.
I have tried to compile PCI frontend driver, which is not by default,
but not sure I succeeded. How can I check it ?

Another question, are there any requirement in terms of kernel version
regarding the version of Xen I use ? If yes, how can I check that too ?

Sorry for long email, but I also ask it on OpenWRT devel list without
answer for now.
Regards,
JB

Re: Xen PCI passthrough & OpenWRT domU

[Pasi Kärkkäinen]

On Wed, Dec 29, 2010 at 12:04:17PM +0100, Jean Baptiste Favre wrote:
> Hello,
> I'm trying to make Openwrt running as domU with network cards PCI
> passthrough.
> 
> First part is OK, OpenWRT runs fine as x86 PV domU, but not PCI passthrough.
> 
> Dom0 runs debian Xen packages.
> # cat /proc/cmdline
> placeholder root=/dev/mapper/system-root ro console=tty0 console=hvc0
> console=ttyS0,115200n8 xen-pciback.permissive xen-pciback.hide=(04:00.0)
> pci=resource_alignment=04:00.0 quiet
> 
> # lspci
> ...
> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E
> Gigabit Ethernet Controller (rev 12)
> 
> # uname -a
> Linux remus 2.6.32-5-xen-amd64 #1 SMP Fri Dec 10 17:41:50 UTC 2010
> x86_64 GNU/Linux
> 
> OpenWRT uses upstream kernel 2.6.32.27 (but I also tested 2.6.35.9
> without any changes).
> 

Upstream Linux 2.6.32.27 doesn't have xen-pcifront driver.
Upstream 2.6.35.9 doesn't have it either.

xen-pcifront driver was merged to upcoming upstream 2.6.37 kernel.


> domU informations:
> # cat /proc/cmdline
> root=/dev/xvda2 rw swiotlb=force console=hvc0 xencons=tty
> 
> # lspci
> [empty]
> 

You should see the PCI device with lspci. 
I guess you don't have xen-pcifront driver in use.


> # dmesg | grep -i pci
> Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
> PCI: Fatal: No config space access function found
> PCI: System does not support PCI
> PCI: System does not support PCI
> PCI: CLS 0 bytes, default 128
> XENBUS: Device with no driver: device/pci/0
> 
> # dmesg | grep -i xen
>  Xen: 0000000000000000 - 00000000000a0000 (usable)
>  Xen: 00000000000a0000 - 0000000000100000 (reserved)
>  Xen: 0000000000100000 - 0000000010000000 (usable)
>   #0 [00014c6000 - 00014d4000]   XEN PAGETABLES ==> [00014c6000 -
> 00014d4000]
>   #3 [0001483000 - 00014c6000]   XEN START INFO ==> [0001483000 -
> 00014c6000]
> Booting paravirtualized kernel on Xen
> Xen version: 4.0.1 (preserve-AD)
> Xen: using vcpu_info placement
> Kernel command line: root=/dev/xvda2 rw swiotlb=force console=hvc0
> xencons=tty
> Xen: using vcpuop timer interface
> installing Xen timer for CPU 0
> xen_balloon: Initialising balloon driver.
> Switching to clocksource xen
> Initialising Xen virtual ethernet driver.
> XENBUS: Device with no driver: device/pci/0
> XENBUS: Device with no driver: device/console/0
> 
> So, it seems that domU sees PCI device (and sky2 module is loaded), but
> do not recognize it.
> I have tried to compile PCI frontend driver, which is not by default,
> but not sure I succeeded. How can I check it ?
> 
> Another question, are there any requirement in terms of kernel version
> regarding the version of Xen I use ? If yes, how can I check that too ?
> 

xen-pcifront driver is required in the domU kernel.

> Sorry for long email, but I also ask it on OpenWRT devel list without
> answer for now.

See: http://wiki.xensource.com/xenwiki/XenPCIpassthrough

-- Pasi

Re: Xen PCI passthrough & OpenWRT domU

[Jean Baptiste Favre]

Hello Pasi,

Le 29/12/2010 13:07, Pasi Kärkkäinen a écrit :
> On Wed, Dec 29, 2010 at 12:04:17PM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> I'm trying to make Openwrt running as domU with network cards PCI
>> passthrough.
>>
>> First part is OK, OpenWRT runs fine as x86 PV domU, but not PCI passthrough.
>>
>> Dom0 runs debian Xen packages.
>> # cat /proc/cmdline
>> placeholder root=/dev/mapper/system-root ro console=tty0 console=hvc0
>> console=ttyS0,115200n8 xen-pciback.permissive xen-pciback.hide=(04:00.0)
>> pci=resource_alignment=04:00.0 quiet
>>
>> # lspci
>> ...
>> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E
>> Gigabit Ethernet Controller (rev 12)
>>
>> # uname -a
>> Linux remus 2.6.32-5-xen-amd64 #1 SMP Fri Dec 10 17:41:50 UTC 2010
>> x86_64 GNU/Linux
>>
>> OpenWRT uses upstream kernel 2.6.32.27 (but I also tested 2.6.35.9
>> without any changes).
>>
> 
> Upstream Linux 2.6.32.27 doesn't have xen-pcifront driver.
> Upstream 2.6.35.9 doesn't have it either.
> 
> xen-pcifront driver was merged to upcoming upstream 2.6.37 kernel.

So, I'll try with 2.6.37 :)


>> domU informations:
>> # cat /proc/cmdline
>> root=/dev/xvda2 rw swiotlb=force console=hvc0 xencons=tty
>>
>> # lspci
>> [empty]
>>
> 
> You should see the PCI device with lspci. 
> I guess you don't have xen-pcifront driver in use.

That's what I think, but do not know how to check xen-pcifront use


>> # dmesg | grep -i pci
>> Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
>> PCI: Fatal: No config space access function found
>> PCI: System does not support PCI
>> PCI: System does not support PCI
>> PCI: CLS 0 bytes, default 128
>> XENBUS: Device with no driver: device/pci/0
>>
>> # dmesg | grep -i xen
>>  Xen: 0000000000000000 - 00000000000a0000 (usable)
>>  Xen: 00000000000a0000 - 0000000000100000 (reserved)
>>  Xen: 0000000000100000 - 0000000010000000 (usable)
>>   #0 [00014c6000 - 00014d4000]   XEN PAGETABLES ==> [00014c6000 -
>> 00014d4000]
>>   #3 [0001483000 - 00014c6000]   XEN START INFO ==> [0001483000 -
>> 00014c6000]
>> Booting paravirtualized kernel on Xen
>> Xen version: 4.0.1 (preserve-AD)
>> Xen: using vcpu_info placement
>> Kernel command line: root=/dev/xvda2 rw swiotlb=force console=hvc0
>> xencons=tty
>> Xen: using vcpuop timer interface
>> installing Xen timer for CPU 0
>> xen_balloon: Initialising balloon driver.
>> Switching to clocksource xen
>> Initialising Xen virtual ethernet driver.
>> XENBUS: Device with no driver: device/pci/0
>> XENBUS: Device with no driver: device/console/0
>>
>> So, it seems that domU sees PCI device (and sky2 module is loaded), but
>> do not recognize it.
>> I have tried to compile PCI frontend driver, which is not by default,
>> but not sure I succeeded. How can I check it ?
>>
>> Another question, are there any requirement in terms of kernel version
>> regarding the version of Xen I use ? If yes, how can I check that too ?
>>
> 
> xen-pcifront driver is required in the domU kernel.
> 
>> Sorry for long email, but I also ask it on OpenWRT devel list without
>> answer for now.
> 
> See: http://wiki.xensource.com/xenwiki/XenPCIpassthrough

Already saw this page. This is my reference for these tries :)


Thanks a lot for answer,
JB

Re: Xen PCI passthrough & OpenWRT domU

[Jean Baptiste Favre]

Hello everybody,

Just a quick mail to inform you that I succeed to make PCI passthrough
run with OpenWRT trunk and newly released Linux kernel 2.6.37 :)

I'll now write an howto

Regards,
JB


Le 29/12/2010 14:11, Jean Baptiste Favre a écrit :
> Hello Pasi,
> 
> Le 29/12/2010 13:07, Pasi Kärkkäinen a écrit :
>> On Wed, Dec 29, 2010 at 12:04:17PM +0100, Jean Baptiste Favre wrote:
>>> Hello,
>>> I'm trying to make Openwrt running as domU with network cards PCI
>>> passthrough.
>>>
>>> First part is OK, OpenWRT runs fine as x86 PV domU, but not PCI passthrough.
>>>
>>> Dom0 runs debian Xen packages.
>>> # cat /proc/cmdline
>>> placeholder root=/dev/mapper/system-root ro console=tty0 console=hvc0
>>> console=ttyS0,115200n8 xen-pciback.permissive xen-pciback.hide=(04:00.0)
>>> pci=resource_alignment=04:00.0 quiet
>>>
>>> # lspci
>>> ...
>>> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E
>>> Gigabit Ethernet Controller (rev 12)
>>>
>>> # uname -a
>>> Linux remus 2.6.32-5-xen-amd64 #1 SMP Fri Dec 10 17:41:50 UTC 2010
>>> x86_64 GNU/Linux
>>>
>>> OpenWRT uses upstream kernel 2.6.32.27 (but I also tested 2.6.35.9
>>> without any changes).
>>>
>>
>> Upstream Linux 2.6.32.27 doesn't have xen-pcifront driver.
>> Upstream 2.6.35.9 doesn't have it either.
>>
>> xen-pcifront driver was merged to upcoming upstream 2.6.37 kernel.
> 
> So, I'll try with 2.6.37 :)
> 
> 
>>> domU informations:
>>> # cat /proc/cmdline
>>> root=/dev/xvda2 rw swiotlb=force console=hvc0 xencons=tty
>>>
>>> # lspci
>>> [empty]
>>>
>>
>> You should see the PCI device with lspci. 
>> I guess you don't have xen-pcifront driver in use.
> 
> That's what I think, but do not know how to check xen-pcifront use
> 
> 
>>> # dmesg | grep -i pci
>>> Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
>>> PCI: Fatal: No config space access function found
>>> PCI: System does not support PCI
>>> PCI: System does not support PCI
>>> PCI: CLS 0 bytes, default 128
>>> XENBUS: Device with no driver: device/pci/0
>>>
>>> # dmesg | grep -i xen
>>>  Xen: 0000000000000000 - 00000000000a0000 (usable)
>>>  Xen: 00000000000a0000 - 0000000000100000 (reserved)
>>>  Xen: 0000000000100000 - 0000000010000000 (usable)
>>>   #0 [00014c6000 - 00014d4000]   XEN PAGETABLES ==> [00014c6000 -
>>> 00014d4000]
>>>   #3 [0001483000 - 00014c6000]   XEN START INFO ==> [0001483000 -
>>> 00014c6000]
>>> Booting paravirtualized kernel on Xen
>>> Xen version: 4.0.1 (preserve-AD)
>>> Xen: using vcpu_info placement
>>> Kernel command line: root=/dev/xvda2 rw swiotlb=force console=hvc0
>>> xencons=tty
>>> Xen: using vcpuop timer interface
>>> installing Xen timer for CPU 0
>>> xen_balloon: Initialising balloon driver.
>>> Switching to clocksource xen
>>> Initialising Xen virtual ethernet driver.
>>> XENBUS: Device with no driver: device/pci/0
>>> XENBUS: Device with no driver: device/console/0
>>>
>>> So, it seems that domU sees PCI device (and sky2 module is loaded), but
>>> do not recognize it.
>>> I have tried to compile PCI frontend driver, which is not by default,
>>> but not sure I succeeded. How can I check it ?
>>>
>>> Another question, are there any requirement in terms of kernel version
>>> regarding the version of Xen I use ? If yes, how can I check that too ?
>>>
>>
>> xen-pcifront driver is required in the domU kernel.
>>
>>> Sorry for long email, but I also ask it on OpenWRT devel list without
>>> answer for now.
>>
>> See: http://wiki.xensource.com/xenwiki/XenPCIpassthrough
> 
> Already saw this page. This is my reference for these tries :)
> 
> 
> Thanks a lot for answer,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Xen PCI passthrough & OpenWRT domU

[Konrad Rzeszutek Wilk]

On Wed, Jan 05, 2011 at 04:27:34PM +0100, Jean Baptiste Favre wrote:
> Hello everybody,
> 
> Just a quick mail to inform you that I succeed to make PCI passthrough
> run with OpenWRT trunk and newly released Linux kernel 2.6.37 :)

Woot! Great! 
> 
> I'll now write an howto

Thanks. Would it make sense to augment the ones on in the Xen Wiki?

Re: Xen PCI passthrough & OpenWRT domU

[Jean Baptiste Favre]

Hello Konrad,
As announced, here's my howto installing Openwrt as Xen domU with PCI
passthrough, using kernel 2.6.37.

French version:
http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.fr

English version:
http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en

Any comment welcome,
Regards,
JB

Le 06/01/2011 17:46, Konrad Rzeszutek Wilk a écrit :
> On Wed, Jan 05, 2011 at 04:27:34PM +0100, Jean Baptiste Favre wrote:
>> Hello everybody,
>>
>> Just a quick mail to inform you that I succeed to make PCI passthrough
>> run with OpenWRT trunk and newly released Linux kernel 2.6.37 :)
> 
> Woot! Great! 
>>
>> I'll now write an howto
> 
> Thanks. Would it make sense to augment the ones on in the Xen Wiki?
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>