pciback bug "secondary bus reset failed"

pciback bug "secondary bus reset failed"

[Alex Zeffertt]

Hi,

I'm trying to passthrough a PCI network card to a domU guest, but I'm hitting a 
bug that prevents the IRQ being unbound from dom0.

Here are the steps I am taking:

  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
  * unbind from current driver in dom0 and rebind to pciback:

      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
      modprobe pciback
      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
      echo -n $pci > /sys/bus/pci/drivers/pciback/bind

  * start guest VM

I then get the following on the dom0 console

   pciback 0000:06:00.0: secondary bus reset failed for device - all functions \
   need to be co-assigned - err: 6
   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
   secondary bus reset unsuccessful;
   pciback 0000:06:00.0: FLR not performed for device

I don't understand the above, but the upshot is that pirq_guest_unbind() does 
not get called.  Later when the guest does an "ifconfig up" the IRQ gets sent to 
dom0 instead of the domU, and this causes the following to appear on the dom0 
console:

   irq 21: nobody cared (try booting with the "irqpoll" option)
   Pid: 0, comm: swapper Tainted: G          2.6.27.42-0.1.1.xs5.6.0.44.111158xen
    #1
    [<c0154517>] __report_bad_irq+0x27/0x90
    [<c015487c>] note_interrupt+0x2fc/0x330
    [<f009e92d>] ? usb_hcd_irq+0x4d/0xe0 [usbcore]
    [<c0153951>] ? handle_IRQ_event+0x31/0x90
    [<c0155204>] handle_level_irq+0xe4/0x110
    [<c0107733>] do_IRQ+0x43/0x90

I have ioapic_ack=old on the Xen command line.  This causes Xen to EOI the 
interrupt straight away instead of waiting for a guest to EOI and forwarding 
that on to the IO APIC.  If I remove this then since no guest handles the 
interrupt, all lower priority IRQs get blocked forever and I lose access to my 
hard disk!

Any help would be much appreciated.

Regards,

Alex Zeffertt

Re: pciback bug "secondary bus reset failed"

[Pasi Kärkkäinen]

On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt wrote:
> Hi,
>
> I'm trying to passthrough a PCI network card to a domU guest, but I'm 
> hitting a bug that prevents the IRQ being unbound from dom0.
>
> Here are the steps I am taking:
>
>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>  * unbind from current driver in dom0 and rebind to pciback:
>
>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>      modprobe pciback
>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>
>  * start guest VM
>
> I then get the following on the dom0 console
>
>   pciback 0000:06:00.0: secondary bus reset failed for device - all functions \
>   need to be co-assigned - err: 6
>   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
>   secondary bus reset unsuccessful;
>   pciback 0000:06:00.0: FLR not performed for device
>

Please check this wiki page:
http://wiki.xensource.com/xenwiki/XenPCIpassthrough

It has a chapter about the "all functions need to be co-assigned" error..

-- Pasi

> I don't understand the above, but the upshot is that pirq_guest_unbind() 
> does not get called.  Later when the guest does an "ifconfig up" the IRQ 
> gets sent to dom0 instead of the domU, and this causes the following to 
> appear on the dom0 console:
>
>   irq 21: nobody cared (try booting with the "irqpoll" option)
>   Pid: 0, comm: swapper Tainted: G          2.6.27.42-0.1.1.xs5.6.0.44.111158xen
>    #1
>    [<c0154517>] __report_bad_irq+0x27/0x90
>    [<c015487c>] note_interrupt+0x2fc/0x330
>    [<f009e92d>] ? usb_hcd_irq+0x4d/0xe0 [usbcore]
>    [<c0153951>] ? handle_IRQ_event+0x31/0x90
>    [<c0155204>] handle_level_irq+0xe4/0x110
>    [<c0107733>] do_IRQ+0x43/0x90
>
> I have ioapic_ack=old on the Xen command line.  This causes Xen to EOI 
> the interrupt straight away instead of waiting for a guest to EOI and 
> forwarding that on to the IO APIC.  If I remove this then since no guest 
> handles the interrupt, all lower priority IRQs get blocked forever and I 
> lose access to my hard disk!
>
> Any help would be much appreciated.
>
> Regards,
>
> Alex Zeffertt
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: pciback bug "secondary bus reset failed"

[Alex Zeffertt]

Pasi Kärkkäinen wrote:
> On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt wrote:
>> Hi,
>>
>> I'm trying to passthrough a PCI network card to a domU guest, but I'm 
>> hitting a bug that prevents the IRQ being unbound from dom0.
>>
>> Here are the steps I am taking:
>>
>>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>>  * unbind from current driver in dom0 and rebind to pciback:
>>
>>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>>      modprobe pciback
>>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>>
>>  * start guest VM
>>
>> I then get the following on the dom0 console
>>
>>   pciback 0000:06:00.0: secondary bus reset failed for device - all functions \
>>   need to be co-assigned - err: 6
>>   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
>>   secondary bus reset unsuccessful;
>>   pciback 0000:06:00.0: FLR not performed for device
>>
> 
> Please check this wiki page:
> http://wiki.xensource.com/xenwiki/XenPCIpassthrough
> 
> It has a chapter about the "all functions need to be co-assigned" error..
> 
> -- Pasi

Hi Pasi,

Thanks for your reply.  I think I understand what is happening now.  My card is 
too old to have the Function Level Reset functionality.  This leaves the 
following options:


  1. Assign all the cards that share the PCI bus (0000:06) to the same guest. 
This enables the FLR functionality to be replaced by the bridge's "Secondary Bus 
Reset" functionality

  - or -

  2. Apply the patch referred to on 
http://wiki.xensource.com/xenwiki/XenPCIpassthrough to bypass the FLR


If I can't do number #1 because some cards are in use by dom0 then I guess #2 is 
the only option.  But does #2 require that dom0 has never accessed the card? 
Also, what if you reboot the domU - does the fact that you can't do an FLR on 
the card mean that there will be unpredictable results the 2nd time domU tries 
to initialise it?

Thanks again,

Alex

Re: pciback bug "secondary bus reset failed"

[Weidong Han]

Alex Zeffertt wrote:
> Pasi Kärkkäinen wrote:
>   
>> On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt wrote:
>>     
>>> Hi,
>>>
>>> I'm trying to passthrough a PCI network card to a domU guest, but I'm 
>>> hitting a bug that prevents the IRQ being unbound from dom0.
>>>
>>> Here are the steps I am taking:
>>>
>>>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>>>  * unbind from current driver in dom0 and rebind to pciback:
>>>
>>>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>>>      modprobe pciback
>>>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>>>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>>>
>>>  * start guest VM
>>>
>>> I then get the following on the dom0 console
>>>
>>>   pciback 0000:06:00.0: secondary bus reset failed for device - all functions \
>>>   need to be co-assigned - err: 6
>>>   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
>>>   secondary bus reset unsuccessful;
>>>   pciback 0000:06:00.0: FLR not performed for device
>>>
>>>       
>> Please check this wiki page:
>> http://wiki.xensource.com/xenwiki/XenPCIpassthrough
>>
>> It has a chapter about the "all functions need to be co-assigned" error..
>>
>> -- Pasi
>>     
>
> Hi Pasi,
>
> Thanks for your reply.  I think I understand what is happening now.  My card is 
> too old to have the Function Level Reset functionality.  This leaves the 
> following options:
>
>
>   1. Assign all the cards that share the PCI bus (0000:06) to the same guest. 
> This enables the FLR functionality to be replaced by the bridge's "Secondary Bus 
> Reset" functionality
>
>   - or -
>
>   2. Apply the patch referred to on 
> http://wiki.xensource.com/xenwiki/XenPCIpassthrough to bypass the FLR
>
>
> If I can't do number #1 because some cards are in use by dom0 then I guess #2 is 
> the only option.  But does #2 require that dom0 has never accessed the card? 
>   
No. Dom0 still can access the card. Dom0 just doesn't perform FLR on it 
to avoid your issue.
> Also, what if you reboot the domU - does the fact that you can't do an FLR on 
> the card mean that there will be unpredictable results the 2nd time domU tries 
> to initialise it?
>   
Yes. FLR is required by pci passthrough. your NIC only can be 
co-assigned (#1). #2 is just a temporary workaround for your try.

Regards,
Weidong
> Thanks again,
>
> Alex
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: pciback bug "secondary bus reset failed"

[Alex Zeffertt]

Weidong Han wrote:
> Alex Zeffertt wrote:
>> Also, what if you reboot the domU - does the fact that you can't do an FLR on 
>> the card mean that there will be unpredictable results the 2nd time domU tries 
>> to initialise it?
>>   
> Yes. FLR is required by pci passthrough. your NIC only can be 
> co-assigned (#1). #2 is just a temporary workaround for your try.
> 

Is there a way of finding out from a running system whether a PCI card supports 
Feature Level Reset?

Thanks,

Alex

Re: pciback bug "secondary bus reset failed"

[Ian Campbell]

On Tue, 2010-05-25 at 09:42 +0100, Alex Zeffertt wrote:
> Weidong Han wrote:
> > Alex Zeffertt wrote:
> >> Also, what if you reboot the domU - does the fact that you can't do an FLR on 
> >> the card mean that there will be unpredictable results the 2nd time domU tries 
> >> to initialise it?
> >>   
> > Yes. FLR is required by pci passthrough. your NIC only can be 
> > co-assigned (#1). #2 is just a temporary workaround for your try.
> > 
> 
> Is there a way of finding out from a running system whether a PCI card supports 
> Feature Level Reset?

I think "lspci -vv" will include "FLReset+" in the DevCap field.

Ian.