Hiding pci devices from dom0 drivers

Hiding pci devices from dom0 drivers

[Kay, Allen M]

Keir,

Currently we had to manually make the following change in linux config
before building dom0 kernel to enable pciback.hide functionality in
dom0.

-CONFIG_XEN_PCIDEV_BACKEND=m
+CONFIG_XEN_PCIDEV_BACKEND=y

What do you think about having this change check into your tree so that
we can get rid of this manual step?  If you are ok with it, we can send
you a patch for it.

Allen

Re: Hiding pci devices from dom0 drivers

[Keir Fraser]

On 24/9/07 08:10, "Kay, Allen M" <allen.m.kay@intel.com> wrote:

> Currently we had to manually make the following change in linux config
> before building dom0 kernel to enable pciback.hide functionality in
> dom0.
> 
> -CONFIG_XEN_PCIDEV_BACKEND=m
> +CONFIG_XEN_PCIDEV_BACKEND=y
> 
> What do you think about having this change check into your tree so that
> we can get rid of this manual step?  If you are ok with it, we can send
> you a patch for it.

It's already '=y' in the -xen0 kernel. -xen is a modular kernel config to
keep the base kernel a reasonable size. PCI device hiding is still an
esoteric option that most users will not use.

 -- Keir

Re: Hiding pci devices from dom0 drivers

[Ian Campbell]

On Mon, 2007-09-24 at 08:27 +0100, Keir Fraser wrote:
> On 24/9/07 08:10, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> 
> > Currently we had to manually make the following change in linux config
> > before building dom0 kernel to enable pciback.hide functionality in
> > dom0.
> > 
> > -CONFIG_XEN_PCIDEV_BACKEND=m
> > +CONFIG_XEN_PCIDEV_BACKEND=y
> > 
> > What do you think about having this change check into your tree so that
> > we can get rid of this manual step?  If you are ok with it, we can send
> > you a patch for it.
> 
> It's already '=y' in the -xen0 kernel. -xen is a modular kernel config to
> keep the base kernel a reasonable size. PCI device hiding is still an
> esoteric option that most users will not use.

You can also get the same effect as pciback.hide entirely from
userspace:

        # cat pciback 
        
        #!/bin/sh
        
        if [ $# -eq 0 ] ; then
           echo "Require a PCI device as parameter"
           exit 1
        fi
        
        for pcidev in $@ ; do
            if [ -h /sys/bus/pci/devices/"$pcidev"/driver ] ; then
        
                echo "Unbinding $pcidev from" $(basename $(readlink /sys/bus/pci/devices/"$pcidev"/driver))
                echo -n "$pcidev" > /sys/bus/pci/devices/"$pcidev"/driver/unbind
            fi
            echo "Binding $pcidev to pciback"
            echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/new_slot
            echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/bind
        done

RE: Hiding pci devices from dom0 drivers

[Kay, Allen M]

What are your experiences with the reliability of the following
"pciback" script to dynamically "hide" pci devices after boot?

I found it cause system hangs about 50% of the time.

I'm also having some issues with xen0 kernel.  It has e1000 driver as
builtin.  This causes problems since I need to use the latest e1000
driver from sourceforge so I need to modify xen0 linux config to mark
e1000 as "m".

In any case, the path of least resistence for me right now is to come up
with a set of instructions to ask people to download linux-2.6.18-xen.hg
manually, move it to xen-unstable.hg, turn on PCIDEV_BACKEND='y' in
config file, and then do "make install" in xen-unstable.hg.

Let me know if there is a simpler way to do this.

Also

>-----Original Message-----
>From: Ian Campbell [mailto:Ian.Campbell@XenSource.com] 
>Sent: Monday, September 24, 2007 1:13 AM
>To: Keir Fraser
>Cc: Kay, Allen M; xen-devel@lists.xensource.com; Han, Weidong; 
>You, Yongkang
>Subject: Re: [Xen-devel] Hiding pci devices from dom0 drivers
>
>On Mon, 2007-09-24 at 08:27 +0100, Keir Fraser wrote:
>> On 24/9/07 08:10, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>> 
>> > Currently we had to manually make the following change in 
>linux config
>> > before building dom0 kernel to enable pciback.hide functionality in
>> > dom0.
>> > 
>> > -CONFIG_XEN_PCIDEV_BACKEND=m
>> > +CONFIG_XEN_PCIDEV_BACKEND=y
>> > 
>> > What do you think about having this change check into your 
>tree so that
>> > we can get rid of this manual step?  If you are ok with 
>it, we can send
>> > you a patch for it.
>> 
>> It's already '=y' in the -xen0 kernel. -xen is a modular 
>kernel config to
>> keep the base kernel a reasonable size. PCI device hiding is still an
>> esoteric option that most users will not use.
>
>You can also get the same effect as pciback.hide entirely from
>userspace:
>
>        # cat pciback 
>        
>        #!/bin/sh
>        
>        if [ $# -eq 0 ] ; then
>           echo "Require a PCI device as parameter"
>           exit 1
>        fi
>        
>        for pcidev in $@ ; do
>            if [ -h /sys/bus/pci/devices/"$pcidev"/driver ] ; then
>        
>                echo "Unbinding $pcidev from" $(basename 
>$(readlink /sys/bus/pci/devices/"$pcidev"/driver))
>                echo -n "$pcidev" > 
>/sys/bus/pci/devices/"$pcidev"/driver/unbind
>            fi
>            echo "Binding $pcidev to pciback"
>            echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/new_slot
>            echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/bind
>        done
>        
>

RE: Hiding pci devices from dom0 drivers

[Ian Campbell]

On Thu, 2007-10-04 at 11:53 -0700, Kay, Allen M wrote:
> What are your experiences with the reliability of the following
> "pciback" script to dynamically "hide" pci devices after boot?
> 
> I found it cause system hangs about 50% of the time.

I've never had a problem with it. It can hang on unbind if the device in
question is in use but as soon as you free up the device (i.e. close the
application which accesses it) the script will continue.

> I'm also having some issues with xen0 kernel.  It has e1000 driver as
> builtin.  This causes problems since I need to use the latest e1000
> driver from sourceforge so I need to modify xen0 linux config to mark
> e1000 as "m".

The -xen0 is intended to be a mostly statically compiled kernel with
drivers for the most common hardware. If you want a modular kernel you
should be using the -xen kernel.

> In any case, the path of least resistence for me right now is to come up
> with a set of instructions to ask people to download linux-2.6.18-xen.hg
> manually, move it to xen-unstable.hg, turn on PCIDEV_BACKEND='y' in
> config file, and then do "make install" in xen-unstable.hg.

The xen-unstable build system will do the download phase for you. You
can also use the "make CONFIGMODE=menuconfig linux-2.6-xen0-config"
style targets.

Ian.

RE: Hiding pci devices from dom0 drivers

[You, Yongkang]

>I've never had a problem with it. It can hang on unbind if the 
>device in
>question is in use but as soon as you free up the device (i.e. 
>close the
>application which accesses it) the script will continue.

Ian, I often met system crashing and rebooting several times when using
hiding pci script to hide a PCIe NIC. Is it a bug or expectable normal
condition?

Following is the serial console log when system crashing. 
----------------
Unable to handle kernel paging request at ffffc20000080000 RIP:
 [<ffffffff880eaeed>] :e1000:e1000_check_for_link+0x13/0x369
PGD dae067 PUD dad067 PMD dac067 PTE 0
Oops: 0000 [1] SMP
CPU 0
Modules linked in: nfs lockd nfs_acl bridge autofs4 hidp rfcomm l2cap
bluetoothePid: 0, comm: swapper Tainted: GF     2.6.18.8-xen #1
RIP: e030:[<ffffffff880eaeed>]  [<ffffffff880eaeed>]
:e1000:e1000_check_for_lin9RSP: e02b:ffffffff8052ee40  EFLAGS: 00010246
RAX: 00000000ffffffff RBX: ffff8800000a47f0 RCX: ffff8800000a4530
RDX: 0000000000000000 RSI: ffffc20000080000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ef75cbd37cda3791 R09: ffffffffb8f63bae
R10: 000000003116eb91 R11: 0000000000000000 R12: ffff8800000a4000
R13: ffff88001fe62dc0 R14: ffffffff8052ee90 R15: ffffffffff578000
FS:  00002af533322ac0(0000) GS:ffffffff804d9000(0000)
knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process swapper (pid: 0, threadinfo ffffffff804f0000, task
ffffffff80461180)
Stack:  ffffffff804f1fd8 ffff8800000a4500 0000000000000100
ffffffff880e4b8e
 ffffffff80559500 ffffffff80559500 0000000000000100 ffffffff880e4b6a
 ffff8800000a4500 ffffffff8023a04c ffffffff8052ee90 ffffffff8052ee90
Call Trace:
 <IRQ> [<ffffffff880e4b8e>] :e1000:e1000_watchdog+0x24/0x622
 [<ffffffff880e4b6a>] :e1000:e1000_watchdog+0x0/0x622
 [<ffffffff8023a04c>] run_timer_softirq+0x175/0x1fd
 [<ffffffff80235ff3>] __do_softirq+0x7b/0x10d
 [<ffffffff8020b098>] call_softirq+0x1c/0x28
 [<ffffffff8020ce01>] do_softirq+0x62/0xd9
 [<ffffffff8020cca0>] do_IRQ+0x68/0x71
 [<ffffffff8034d985>] evtchn_do_upcall+0xee/0x165
 [<ffffffff8020abce>] do_hypervisor_callback+0x1e/0x2c
 <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
 [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
 [<ffffffff8020eec9>] raw_safe_halt+0xb7/0xdd
 [<ffffffff80207040>] init+0x0/0x334
 [<ffffffff802082c3>] xen_idle+0x67/0x79
 [<ffffffff8020837f>] cpu_idle+0xaa/0xcd
 [<ffffffff804f97ea>] start_kernel+0x260/0x262
 [<ffffffff804f922b>] _sinittext+0x22b/0x231


Code: 8b 0e 44 8b 46 08 83 f8 01 77 30 83 fa 01 8b ae 80 01 00 00
RIP  [<ffffffff880eaeed>] :e1000:e1000_check_for_link+0x13/0x369
 RSP <ffffffff8052ee40>
CR2: ffffc20000080000
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!
 (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
----------------------

Best Regards,
Yongkang You
 

>-----Original Message-----
>From: Ian Campbell [mailto:Ian.Campbell@XenSource.com] 
>Sent: Friday, October 05, 2007 4:32 PM
>To: Kay, Allen M
>Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong; 
>You, Yongkang
>Subject: RE: [Xen-devel] Hiding pci devices from dom0 drivers
>
>On Thu, 2007-10-04 at 11:53 -0700, Kay, Allen M wrote:
>> What are your experiences with the reliability of the following
>> "pciback" script to dynamically "hide" pci devices after boot?
>> 
>> I found it cause system hangs about 50% of the time.
>
>I've never had a problem with it. It can hang on unbind if the 
>device in
>question is in use but as soon as you free up the device (i.e. 
>close the
>application which accesses it) the script will continue.
>
>> I'm also having some issues with xen0 kernel.  It has e1000 driver as
>> builtin.  This causes problems since I need to use the latest e1000
>> driver from sourceforge so I need to modify xen0 linux config to mark
>> e1000 as "m".
>
>The -xen0 is intended to be a mostly statically compiled kernel with
>drivers for the most common hardware. If you want a modular kernel you
>should be using the -xen kernel.
>
>> In any case, the path of least resistence for me right now 
>is to come up
>> with a set of instructions to ask people to download 
>linux-2.6.18-xen.hg
>> manually, move it to xen-unstable.hg, turn on PCIDEV_BACKEND='y' in
>> config file, and then do "make install" in xen-unstable.hg.
>
>The xen-unstable build system will do the download phase for you. You
>can also use the "make CONFIGMODE=menuconfig linux-2.6-xen0-config"
>style targets.
>
>Ian.
>