Re: how to PCI-passthrough 1 card to a DomU when another @ Dom0 uses the same driver?

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Feb 01, 2011 at 09:03:31PM -0800, dev001 wrote:
> it was suggested (thx!) @ my OP
> (http://lists.xensource.com/archives/html/xen-users/2011-02/msg00013.html)
> might be more appropriate here ... hope that's the case (?)
> 
> my Dom0 host runs,
> 
> uname -a
>         Linux test 2.6.34.8-7-xen #1 SMP 2011-01-20 21:58:39 +0100
>         x86_64 x86_64 x86_64 GNU/Linux
> lsb_release -d
>         Description:    openSUSE 11.3 (x86_64)
> rpm -qa | grep -i Xen-4
>         xen-4.0.1_21326_02-0.3.1.x86_64
> 
> i've got 3 NICs,
> 
> lspci | grep -i eth
>         02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>         RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
>         03:00.0 Ethernet controller: Marvell Technology Group Ltd.
>         88E8053 PCI-E Gigabit Ethernet Controller (rev 20)
>         04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>         RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
> 
> where 04:00.0 is assigned as the internal-LAN/Bridge port, and 02:00.0 &
> 03:00.0 are PCIe cards that I intend to PCI-passthrough to a DomU guest.
> 
> so far, I've:
> 
> cat /boot/grub/menu.lst
>         ...
>     kernel /xen.gz ... iommu=verbose dom0_vcpus_pin=true sched=credit
>     cpuidle=1 cpufreq=xen clocksource=acpi numa=on
>     module /vmlinuz-xen ... elevator=cfq clocksource=xen mce=off
>     nomodeset pciback.permissive
>     pciback.hide=(0000:02:00.0)(0000:03:00.0)

>     module /initrd-xen
>         ...
> 
> cat /etc/modprobe.d/99-local.conf
>         options pciback hide=(0000:02:00.0)(0000:03:00.0)
>         install sky2       /sbin/modprobe pciback ; /sbin/modprobe
>         --first-time --ignore-install sky2
>         install r8169      /sbin/modprobe pciback ; /sbin/modprobe
>         --first-time --ignore-install r8169
> 
> cat /etc/init.d/boot.local
>         modprobe pciback hide='(02:00.0)(03.00.0)'

Hmm, the 03.00 is incorrect. Should be 03:00.0

>         echo -n 0000:02:00.0 > sys/bus/pci/drivers/pciback/permissive
>         echo -n 0000:03:00.0 > sys/bus/pci/drivers/pciback/permissive
> 
> @ Dom0 boot, i only see ONE card seized,
> 
> dmesg | egrep -i seiz
>         [    2.081944] pciback 0000:03:00.0: seizing device
> 
> and, of course as a result, for a guest,

So if you look in /sys/pci .. something for 02:00.0 is there any driver
that is using it? Or is it attached ot pciback?

> 
> cat test.cfg
>         ...
>         extra = 'textmode=1 xencons=xvc0 elevator=noop iommu=soft'
>         pci   = [ '02:00.0', '03:00.0']
>         ...
> 
> xm create test.cfg
>         Error: pci: PCI Backend and pci-stub don't own device
>         0000:02:00.0
> 
> 
> if i simply replace the 04:00.0 Realtek card with a non-r8619 driver
> card, e.g.,
> 
>         04:00.0 Ethernet controller: Atheros Communications
>         AR8121/AR8113/AR8114 Gigabit or Fast Ethernet (rev b0)
> 
> and repeat the experiment, then both cards are CORRECTLY seized,
> 
> dmesg | egrep -i seiz
>         [    3.027654] pciback 0000:02:00.0: seizing device
>         [    3.321515] pciback 0000:03:00.0: seizing device
> 
> and I can subsequently launch the DomU, with both cards successfully
> passed through.
> 
> i.e., using two cards with the same driver type FAILS to allow one of
> the cards to be seized.
> 
> i suspect this has to do with the driver not being unloadable at the
> time of device seizure, as it's in use.

Well, the pciback.hide=(..) during bootup should have taken care of that.

Oh, unless the pciback is a _module_ at which point it would not do it.
> 
> if that's the case -- how do i configure this so that the _same_ driver
> can be used for _both_ the Dom0 card instance, and then appropriately
> seized for the PCI-passthru target'd 2nd card?

Did you follow the rules for doing it detailed in the VT-d howto? Just use
the pciback instead of pci-stub in it.

What happens if you 'rmmod sky2' or blacklist the sky2 driver?
> 
> or, is this another issue & there's addition/other config necessary?
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel