Re: Fail to probe qla2xxx fiber channel card while doing pci hotplug

[Jiang Liu <liuj97@gmail.com>]

On 09/19/2012 09:39 PM, Bjorn Helgaas wrote:
> On Tue, Sep 18, 2012 at 7:50 PM, Yijing Wang <wangyijing@huawei.com> wrote:
>> On 2012/9/19 1:54, Bjorn Helgaas wrote:
>>> On Mon, Sep 17, 2012 at 6:06 AM, Yijing Wang <wangyijing@huawei.com> wrote:
>>>> On 2012/9/16 11:30, Bjorn Helgaas wrote:
>>>>> On Sat, Sep 15, 2012 at 4:22 AM, Yijing Wang <wangyijing@huawei.com> wrote:
>>>>>> Hi all,
>>>>>>    I encountered a very strange problem when I hot plug a fiber channel card(using qla2xxx driver).
>>>>>> I did the hotplug in arch x86 machine, using pciehp driver for hotplug, this platform supports pci hot-plug triggering from both
>>>>>> sysfs and attention button. If a hot-plug slot is empty when system boot-up, then hotplug FC card in this slot is ok.
>>>>>> If a hot-plug slot has been embeded a FC card when system boot-up, hot-remove this card is ok, but hot-add this card will fail.
>>>>>> I used
>>>>>> #modprobe qla2xxx ql2xextended_error_logging=0x7fffffff
>>>>>> to get all probe info. As bellow:
>>>>>>
>>>>>> Can anyone give me any suggestion for this problem?
>>>>>
>>>>> It sounds like you did this:
>>>>>
>>>>>   1) Power down system
>>>>>   2) Remove FC card from slot
>>>>>   3) Boot system
>>>>>   4) Hot-add FC card
>>>>>   5) Load qla2xxx driver
>>>>>   6) qla2xxx driver claims FC card
>>>>>   7) FC card works correctly
>>>>>
>>>>>   8) Power down system
>>>>>   9) Install FC card in slot
>>>>>  10) Boot system
>>>>>  11) Load qla2xxx driver
>>>>>  12) qla2xxx driver claims FC card
>>>>>  13) FC card works correctly
>>>> I rmmod qla2xxx driver here and modprobe qla2xxx ql2xextended_error_logging=0x1e400000 again for get errors info
>>>> Also I modprobe pciehp pciehp_debug=1 for getting debug info
>>>>>  14) Hot-remove card
>>>>>  15) Hot-add card
>>>>>  16) qla2xxx driver claims FC card
>>>>>  17) FC card does not work
>>>>>
>>>>> and I assume the dmesg log you included is just from steps 15 and 16
>>>>> (correct me if I'm wrong).
>>>>>
>>>>> It would be useful to see the entire log showing all these events so
>>>>> we can compare the working cases with the non-working one.  If you use
>>>>> the pciehp_debug module parameter, we should also see some pciehp
>>>>> events that would help me understand that driver.
>>>>>
>>>>
>>>> Hi Bjorn,
>>>>    Thanks for your comments very much!
>>>>
>>>> My steps:
>>>> 1) power down system
>>>> 2) Install FC card in slot
>>>> 3) Boot system
>>>> 4) Load qla2xxx driver
>>>> 5) qla2xxx driver claims FC card
>>>> 6) FC card works correctly(at least probe return ok, I don't know qla2xxx driver much..)
>>>> 7) rmmod qla2xxx
>>>> 8) modprobe qla2xxx ql2xextended_error_logging=0x1e400000(for get errors info)
>>>> 9) modprobe pciehp pciehp_debug=1
>>>> 10) Hot-remove card
>>>> 11) Hot-add card
>>>> 12) qla2xxx driver claims FC card fail(probe return fail, setup chip fail)
>>>> --------------------------------------so this is failed situation----------
>>>>
>>>> --------------------------------------continue to hot-add fc card into empty slot(also support pci hp)
>>>> 13) Install FC card in empty slot
>>>> 14) Hot-add card
>>>> 15) qla2xxx driver claims FC card ok (probe return ok)
>>>>
>>>> btw:
>>>> If fc card firmware version 4.03, everything is ok (hot-plug in any slots(empty or not))
>>>> fc card firmware version is 4.04 or 5.04 , situation as same as 1)--->12)
>>>
>>> Thanks.  The FW change is a good clue.  If everything works with
>>> version 4.03, but it doesn't work with version 4.04, it's likely to be
>>> a FW problem, not a Linux PCI core problem.
>>>
>>> Here's what I see from your logs.  In slot 4 (bus 08), the card was
>>> present before boot, you removed it, re-added it, and it failed after
>>> being re-added.  Slot 3 (bus 06) was empty at boot, you hot-added a
>>> card, and it worked.  Here are the resources available on those two
>>> buses and the boot-time config of the first device in slot 4:
>>>
>>>       pci 0000:00:07.0: PCI bridge to [bus 06-07]
>>>       pci 0000:00:07.0:   bridge window [io  0xc000-0xcfff]
>>>       pci 0000:00:07.0:   bridge window [mem 0xf9000000-0xf9ffffff]
>>>       pci 0000:00:07.0:   bridge window [mem 0xf1000000-0xf1ffffff 64bit pref]
>>>       pci 0000:00:09.0: PCI bridge to [bus 08-09]
>>>       pci 0000:00:09.0:   bridge window [io  0xb000-0xbfff]
>>>       pci 0000:00:09.0:   bridge window [mem 0xf8000000-0xf8ffffff]
>>>       pci 0000:00:09.0:   bridge window [mem 0xf0000000-0xf0ffffff 64bit pref]
>>>       pci 0000:08:00.0: [1077:2532] type 00 class 0x0c0400
>>>       pci 0000:08:00.0: reg 10: [io  0xb100-0xb1ff]
>>>       pci 0000:08:00.0: reg 14: [mem 0xf8084000-0xf8087fff 64bit]
>>>       pci 0000:08:00.0: reg 30: [mem 0xf8040000-0xf807ffff pref]
>>>
>>> After you remove and re-add the card in slot 4, it starts with
>>> uninitialized BARs as expected, then we assign resources to it.  It's
>>> sort of interesting that the BIOS had originally put the ROM (reg 30)
>>> in the non-prefetchable window, while after the hot-add, Linux places
>>> it in the prefetchable window.  Either should work, and in fact the
>>> card you added in slot 3 *does* work with its ROM in the prefetchable
>>> window.
>>>
>>>       pci 0000:08:00.0: [1077:2532] type 00 class 0x0c0400
>>>       pci 0000:08:00.0: reg 10: [io  0x0000-0x00ff]
>>>       pci 0000:08:00.0: reg 14: [mem 0x00000000-0x00003fff 64bit]
>>>       pci 0000:08:00.0: reg 30: [mem 0x00000000-0x0003ffff pref]
>>>       pci 0000:08:00.0: BAR 0: assigned [io  0xb000-0xb0ff]
>>>       pci 0000:08:00.0: BAR 1: assigned [mem 0xf8000000-0xf8003fff 64bit]
>>>       pci 0000:08:00.0: BAR 6: assigned [mem 0xf0000000-0xf003ffff pref]
>>>       qla2xxx [0000:08:00.0]-0098:10: Failed to load segment 0 of firmware.
>>>       qla2xxx [0000:08:00.0]-d008:10: No buffer available for dump.
>>>       qla2xxx [0000:08:00.0]-008f:10: Failed to load segment 0 of firmware.
>>>       qla2xxx [0000:08:00.0]-00cf:10: Setup chip ****FAILED****.
>>>
>>> When you hot-add the card in slot 3, it starts with uninitialized BARs
>>> as expected, but again, we assign valid resources to it:
>>>
>>>       pci 0000:06:00.0: [1077:2532] type 00 class 0x0c0400
>>>       pci 0000:06:00.0: reg 10: [io  0x0000-0x00ff]
>>>       pci 0000:06:00.0: reg 14: [mem 0x00000000-0x00003fff 64bit]
>>>       pci 0000:06:00.0: reg 30: [mem 0x00000000-0x0003ffff pref]
>>>       pci 0000:06:00.0: BAR 0: assigned [io  0xc000-0xc0ff]
>>>       pci 0000:06:00.0: BAR 1: assigned [mem 0xf9000000-0xf9003fff 64bit]
>>>       pci 0000:06:00.0: BAR 6: assigned [mem 0xf1000000-0xf103ffff pref]
>>>
>>> I don't see anything wrong from a PCI perspective.  I suspect
>>> something strange in the card firmware.
>>>
>>> If you do figure out something wrong in PCI, let me know.
>>>
>>> Bjorn
>>>
>>
>> Hi Bjorn,
>>    Thanks for your detailed analysis very much!
>>
>> We compared the two situations after BIOS initialization, and found Max Payload Size in DEVCTRL is 256B
>> if FC card had been installed, if the slot is empty, Max Payload Size is 128B. We force it to be 128B when
>> FC card installed when system boot up. Finally pci hotplug becomes ok. So I suspect maybe our PCIe hardware
>> has problem supporting 256B.
> 
> Ah, this sounds like something I've been worried about for a while,
> i.e., do we handle MPS correctly when we hot-add devices?
> 
> Yijing, I'm not quite clear on what you're observing.  I guess you're
> saying that if an FC card is installed at boot, the BIOS sets MPS to
> 256, and that if no FC card is installed, the BIOS sets MPS to 128?
> You haven't mentioned any Linux boot options, so I assume you haven't
> tried any.  Does "pci=pcie_bus_safe" make any difference?
> 
> Jon, here's a pointer to the beginning of the thread:
> http://marc.info/?l=linux-pci&m=134770460302298&w=2 (full dmesg log at
> http://marc.info/?l=linux-scsi&m=134788365823217&w=2).  I'm not sure
> we have enough in the dmesg log to diagnose an issue like this.  I
> wonder if it would be useful to log the current setting, so we could
> notice BIOS default differences like this one.

Hi Yijing,
	It's possible that the issue is caused by pcie_bus_configure_settings() instead of
hardware flaw. By default, pcie_bus_config is setting to PCIE_BUS_TUNE_OFF, which means 
all PCIe devices' Max Payload Size if configured by BIOS and OS won't change it.
	So could you please help to:
	1) add "pci=pcie_bus_safe" kernel option and check whether the behavior changes.
	2) Print out Max Payload Size configuration for all PCIe devices along the path from
the hod-added card to corresponding root port.
	3) tracing executing of pcie_bus_configure_settings().
	Thanks!
	Gerry
card to the