lpfc driver loading is rebooting the box: 2.6.33 and above.

lpfc driver loading is rebooting the box: 2.6.33 and above.

[Keith Mannthey]

I moved forwarded and a server started rebooting while loading the
kernel current kernels. 

I isolated it to the lpfc driver. 

Upsteam kernels tried so far:

3.6.34-rc1
3.4.33-git15 

With these kernels I see the following when the driver loads: 

"
[ 276.643991] Emulex LightPulse Fibre Channel SCSI driver 8.3.9
[ 276.643996] Copyright(c) 2004-2009 Emulex. All rights reserved.
[ 276.644058] lpfc 0000:1a:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
[ 276.644067] lpfc 0000:1a:00.0: setting latency timer to 64
[ 276.645003] scsi5 : on PCI bus 1a device 00 irq 30
[ 277.480150] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 278.167137] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 278.167171] alloc irq_desc for 72 on node -1
[ 278.167174] alloc kstat_irqs on node -1
[ 278.167181] alloc irq_2_iommu on node -1
[ 278.167195] lpfc 0000:1a:00.0: irq 72 for MSI/MSI-X
[ 278.857352] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 279.547573] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 280.237779] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 280.927994] lpfc 0000:1a:00.0: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 280.927999] lpfc 0000:1a:00.0: 0:1477 Failed to set up hba
[ 281.028380] lpfc 0000:1a:00.0: PCI INT A disabled
[ 281.028411] alloc irq_desc for 37 on node -1
[ 281.028412] alloc kstat_irqs on node -1
[ 281.028416] alloc irq_2_iommu on node -1
[ 281.028422] lpfc 0000:1a:00.1: PCI INT B -> GSI 37 (level, low) -> IRQ 37
[ 281.028427] lpfc 0000:1a:00.1: setting latency timer to 64
[ 281.028939] scsi6 : on PCI bus 1a device 01 irq 37
[ 281.717949] lpfc 0000:1a:00.1: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 282.408169] lpfc 0000:1a:00.1: 0:0440 Adapter failed to init, READ_REV has missing revision information.
[ 282.408210] lpfc 0000:1a:00.1: irq 72 for MSI/MSI-X
[ 283.098375] lpfc 0000:1a:00.1: 0:0440 Adapter failed to inelm
 "

A little while (less then 30 seconds) after loading the driver (rmmod
sometimes extends the life of the box). I get a hardware level pci fault
and my box reboots. 

My FW has been constant during this at 282a4 (current I believe) and
lspci reports the card as

1a:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
1a:00.1 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)

The system is a 2 socket Intel based server IBM x3550m2.  


I forwarded ported some of the older versions of the driver 8.3.7 and
8.3.6 and that didn't not fix the issue, I am not convinced it is a
direct driver change that is causing this.  I also tried "lpfc_use_msi"
and "lpfc_sli_mode" modules options without any affect on behavior. 

Then I started testing whole kernels, .config file remained as close as
possible. After some boots it seem 2.6.33-rc8 and before works. It is a
relatively small window of change and I am still digging but I thought I
would post and see if there were any ideas or suggestions out there. 

Anyone seen anything like this?  

Thanks,
  Keith Mannthey
  LTC FS-Dev 



--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: lpfc driver loading is rebooting the box: 2.6.33 and above.

[Keith Mannthey]

On Wed, 2010-03-10 at 13:29 -0800, Keith Mannthey wrote:
> I moved forwarded and a server started rebooting while loading the
> kernel current kernels. 
> 
> I isolated it to the lpfc driver. 
> 
> Upsteam kernels tried so far:
> 
> 3.6.34-rc1
> 3.4.33-git15 
Update on the issue. It gets weird. 

2.6.33 stock worked ok. 

A different .config file was working just fine on an identical setup so
I started peeling away at .config changes. 


I really have no idea about this option change and it will have to be
punted to those more in the know. 

the diff between a good and a bad kernel is....

@@ -130,7 +130,7 @@
 CONFIG_RD_BZIP2=y
 # CONFIG_RD_LZMA is not set
 # CONFIG_RD_LZO is not set
-# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
+CONFIG_CC_OPTIMIZE_FOR_SIZE=y
 CONFIG_SYSCTL=y
 CONFIG_ANON_INODES=y
 CONFIG_EMBEDDED=y

This is some gcc optimization flag change. In the working config I have
CONFIG_CC_OPTIMIZE_FOR_SIZE=y.  If I don't optimize I get the badness. 


 gcc -v
Using built-in specs.
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info
--mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64
--enable-languages=c,c++,objc,fortran,obj-c++,java,ada
--enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3
--enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/
--with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap
--with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit
--enable-libstdcxx-allocator=new --disable-libstdcxx-pch
--enable-version-specific-runtime-libs --program-suffix=-4.3
--enable-linux-futex --without-system-libunwind --with-cpu=generic
--build=x86_64-suse-linux
Thread model: posix
gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)


Thanks,
  Keith Mannthey 
  LTC LTC-FS