Recurring qla2xxx crashes (maybe APIC related)

Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

Hi.  I've been having recurring problems with the qla2xxx driver or
firmware lockups.  Seems to happen out of the blue, with nothing special 
going on on the SAN.

The servers are IBM BladeCenter HS21 8853A2Gs, with dual-port QLA2422
cards connected to a dual-fabric topology.  They are running Ubuntu
6.06, kernel 2.6.22.19 with some OCFS2 patches applied.  qla2xxx driver
version is 8.01.07-k7, loaded with params qlport_down_retry=35 and
ql2xextended_error_logging=1.  Firmware is the latest from QLogic's FTP.

When they crash, the following is logged:

Apr 21 09:50:33 xander kernel: APIC error on CPU3: 00(40)
Apr 21 09:51:18 xander kernel: qla2xxx_eh_abort(1): aborting sp ffff81010ae4c7c0 from RISC. pid=1024761.
Apr 21 09:51:48 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
Apr 21 09:51:48 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
Apr 21 09:51:48 xander kernel: qla2xxx 0000:08:01.0: Mailbox command timeout occured. Issuing ISP abort.
Apr 21 09:51:48 xander kernel: qla2xxx 0000:08:01.0: Performing ISP error recovery - ha= ffff81021f5ec530.
Apr 21 09:51:48 xander kernel: scsi(1): **** Load RISC code ****
Apr 21 09:51:48 xander kernel: scsi(1): Verifying Checksum of loaded RISC code.
Apr 21 09:51:48 xander kernel: scsi(1): Checksum OK, start firmware.
Apr 21 09:51:48 xander kernel: scsi(1): Issue init firmware.
Apr 21 09:51:49 xander kernel: scsi(1): fcport-0 - port retry count: 34 remaining
Apr 21 09:51:49 xander kernel: scsi(1): fcport-1 - port retry count: 34 remaining
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous P2P MODE received.
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous LOOP UP (4 Gbps).
Apr 21 09:51:49 xander kernel: qla2xxx 0000:08:01.0: LOOP UP detected (4 Gbps).
Apr 21 09:51:49 xander kernel: scsi(1): F/W Ready - OK 
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE.
Apr 21 09:51:49 xander kernel: scsi(1): Port database changed ffff 0006 0000.
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0000/0007/0b00.
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0001/0007/0b00.
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0004/0600.
Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0007/0b00.
Apr 21 09:51:49 xander kernel: scsi(1): fw_state=3 curr time=102a04c2d.
Apr 21 09:51:49 xander kernel: qla2x00_restart_isp(): Start configure loop, status = 0
Apr 21 09:51:49 xander kernel: scsi(1): Configure loop -- dpc flags =0x4080048
Apr 21 09:51:49 xander kernel: scsi(1): RSCN queue entry[0] = [00/000000].
Apr 21 09:51:49 xander kernel: scsi(1): device_resync: rscn overflow.
Apr 21 09:51:50 xander kernel: scsi(1): RFT_ID failed, completion status (280).
Apr 21 09:51:50 xander kernel: scsi(1): Register FC-4 TYPE failed.
Apr 21 09:51:50 xander kernel: scsi(1): RFF_ID failed, completion status (280).
Apr 21 09:51:50 xander kernel: scsi(1): fcport-0 - port retry count: 33 remaining
Apr 21 09:51:50 xander kernel: scsi(1): fcport-1 - port retry count: 33 remaining
Apr 21 09:51:50 xander kernel: scsi(1): Register FC-4 Features failed.
Apr 21 09:51:50 xander kernel: scsi(1): RNN_ID failed, completion status (280).
Apr 21 09:51:50 xander kernel: scsi(1): Register Node Name failed.
Apr 21 09:51:50 xander kernel: scsi(1): GID_PT failed, completion status (6380).
Apr 21 09:51:50 xander kernel: scsi(1): GA_NXT failed, rejected request:
Apr 21 09:51:50 xander kernel: 0   1   2   3   4   5   6   7   8   9  Ah  Bh  Ch  Dh  Eh  Fh
Apr 21 09:51:50 xander kernel: --------------------------------------------------------------
Apr 21 09:51:50 xander kernel: 14  00  00  00  00  70  26  1f  02  00  00  00  10  08  00  00
Apr 21 09:51:50 xander kernel: qla2xxx 0000:08:01.0: SNS scan failed -- assuming zero-entry result...
Apr 21 09:51:50 xander kernel: qla24xx_fabric_logout(1): failed to complete IOCB -- completion status (2)  ioparam=0/810031.
Apr 21 09:51:50 xander kernel: scsi(1): LOOP READY
Apr 21 09:51:50 xander kernel: qla2x00_restart_isp(): Configure loop done, status = 0x0
Apr 21 09:51:50 xander kernel: APIC error on CPU4: 00(40)
Apr 21 09:51:50 xander kernel: qla2x00_abort_isp(1): exiting.
Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): **** FAILED. mbx0=54, mbx1=0, mbx2=1f58, cmd=54 ****
Apr 21 09:51:50 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
Apr 21 09:51:50 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
Apr 21 09:51:50 xander kernel: qla24xx_abort_command(1): failed to issue IOCB (100).
Apr 21 09:51:50 xander kernel: qla2xxx_eh_abort(1): abort_command mbx failed.
Apr 21 09:51:50 xander kernel: qla2xxx 0000:08:01.0: scsi(1:1:5): Abort command issued -- 0 fa2f9 2002.
Apr 21 09:51:51 xander kernel: scsi(1): fcport-0 - port retry count: 32 remaining
Apr 21 09:51:51 xander kernel: scsi(1): fcport-1 - port retry count: 32 remaining
Apr 21 09:51:52 xander kernel: scsi(1): fcport-0 - port retry count: 31 remaining
Apr 21 09:51:52 xander kernel: scsi(1): fcport-1 - port retry count: 31 remaining
Apr 21 09:51:53 xander kernel: scsi(1): fcport-0 - port retry count: 30 remaining
Apr 21 09:51:53 xander kernel: scsi(1): fcport-1 - port retry count: 30 remaining
Apr 21 09:51:54 xander kernel: scsi(1): fcport-0 - port retry count: 29 remaining
Apr 21 09:51:54 xander kernel: scsi(1): fcport-1 - port retry count: 29 remaining
Apr 21 09:51:55 xander kernel: scsi(1): fcport-0 - port retry count: 28 remaining
Apr 21 09:51:55 xander kernel: scsi(1): fcport-1 - port retry count: 28 remaining
Apr 21 09:51:56 xander kernel: scsi(1): fcport-0 - port retry count: 27 remaining
Apr 21 09:51:56 xander kernel: scsi(1): fcport-1 - port retry count: 27 remaining
Apr 21 09:51:57 xander kernel: scsi(1): fcport-0 - port retry count: 26 remaining
Apr 21 09:51:57 xander kernel: scsi(1): fcport-1 - port retry count: 26 remaining
Apr 21 09:51:58 xander kernel: scsi(1): fcport-0 - port retry count: 25 remaining
Apr 21 09:51:58 xander kernel: scsi(1): fcport-1 - port retry count: 25 remaining
Apr 21 09:51:59 xander kernel: scsi(1): fcport-0 - port retry count: 24 remaining
Apr 21 09:51:59 xander kernel: scsi(1): fcport-1 - port retry count: 24 remaining
Apr 21 09:52:00 xander kernel: scsi(1): fcport-0 - port retry count: 23 remaining
Apr 21 09:52:00 xander kernel: scsi(1): fcport-1 - port retry count: 23 remaining
Apr 21 09:52:01 xander kernel: scsi(1): fcport-0 - port retry count: 22 remaining
Apr 21 09:52:01 xander kernel: scsi(1): fcport-1 - port retry count: 22 remaining
Apr 21 09:52:02 xander kernel: scsi(1): fcport-0 - port retry count: 21 remaining
Apr 21 09:52:02 xander kernel: scsi(1): fcport-1 - port retry count: 21 remaining
Apr 21 09:52:03 xander kernel: scsi(1): fcport-0 - port retry count: 20 remaining
Apr 21 09:52:03 xander kernel: scsi(1): fcport-1 - port retry count: 20 remaining
Apr 21 09:52:04 xander kernel: scsi(1): fcport-0 - port retry count: 19 remaining
Apr 21 09:52:04 xander kernel: scsi(1): fcport-1 - port retry count: 19 remaining
Apr 21 09:52:05 xander kernel: scsi(1): fcport-0 - port retry count: 18 remaining
Apr 21 09:52:05 xander kernel: scsi(1): fcport-1 - port retry count: 18 remaining
Apr 21 09:52:06 xander kernel: scsi(1): fcport-0 - port retry count: 17 remaining
Apr 21 09:52:06 xander kernel: scsi(1): fcport-1 - port retry count: 17 remaining
Apr 21 09:52:07 xander kernel: scsi(1): fcport-0 - port retry count: 16 remaining
Apr 21 09:52:07 xander kernel: scsi(1): fcport-1 - port retry count: 16 remaining
Apr 21 09:52:09 xander kernel: scsi(1): fcport-0 - port retry count: 15 remaining
Apr 21 09:52:09 xander kernel: scsi(1): fcport-1 - port retry count: 15 remaining
Apr 21 09:52:10 xander kernel: scsi(1): fcport-0 - port retry count: 14 remaining
Apr 21 09:52:10 xander kernel: scsi(1): fcport-1 - port retry count: 14 remaining
Apr 21 09:52:11 xander kernel: scsi(1): fcport-0 - port retry count: 13 remaining
Apr 21 09:52:11 xander kernel: scsi(1): fcport-1 - port retry count: 13 remaining
Apr 21 09:52:12 xander kernel: scsi(1): fcport-0 - port retry count: 12 remaining
Apr 21 09:52:12 xander kernel: scsi(1): fcport-1 - port retry count: 12 remaining
Apr 21 09:52:13 xander kernel: scsi(1): fcport-0 - port retry count: 11 remaining
Apr 21 09:52:13 xander kernel: scsi(1): fcport-1 - port retry count: 11 remaining
Apr 21 09:52:14 xander kernel: scsi(1): fcport-0 - port retry count: 10 remaining
Apr 21 09:52:14 xander kernel: scsi(1): fcport-1 - port retry count: 10 remaining
Apr 21 09:52:15 xander kernel: scsi(1): fcport-0 - port retry count: 9 remaining
Apr 21 09:52:15 xander kernel: scsi(1): fcport-1 - port retry count: 9 remaining
Apr 21 09:52:16 xander kernel: scsi(1): fcport-0 - port retry count: 8 remaining
Apr 21 09:52:16 xander kernel: scsi(1): fcport-1 - port retry count: 8 remaining
Apr 21 09:52:17 xander kernel: scsi(1): fcport-0 - port retry count: 7 remaining
Apr 21 09:52:17 xander kernel: scsi(1): fcport-1 - port retry count: 7 remaining
Apr 21 09:52:18 xander kernel: scsi(1): fcport-0 - port retry count: 6 remaining
Apr 21 09:52:18 xander kernel: scsi(1): fcport-1 - port retry count: 6 remaining
Apr 21 09:52:19 xander kernel: scsi(1): fcport-0 - port retry count: 5 remaining
Apr 21 09:52:19 xander kernel: scsi(1): fcport-1 - port retry count: 5 remaining
Apr 21 09:52:20 xander kernel: scsi(1): fcport-0 - port retry count: 4 remaining
Apr 21 09:52:20 xander kernel: scsi(1): fcport-1 - port retry count: 4 remaining
Apr 21 09:52:21 xander kernel: scsi(1): fcport-0 - port retry count: 3 remaining
Apr 21 09:52:21 xander kernel: scsi(1): fcport-1 - port retry count: 3 remaining
Apr 21 09:52:22 xander kernel: scsi(1): fcport-0 - port retry count: 2 remaining
Apr 21 09:52:22 xander kernel: scsi(1): fcport-1 - port retry count: 2 remaining
Apr 21 09:52:23 xander kernel: scsi(1): fcport-0 - port retry count: 1 remaining
Apr 21 09:52:23 xander kernel: scsi(1): fcport-1 - port retry count: 1 remaining
Apr 21 09:52:24 xander kernel: scsi(1): fcport-0 - port retry count: 0 remaining
Apr 21 09:52:24 xander kernel: scsi(1): fcport-1 - port retry count: 0 remaining

It varies on which CPU the APIC error happens, but after that it's
always the same:  qla2xxx complaining and attempting to restart the
firmware without any success, and I/O service never recovers.  Soon
thereafter other cluster members fences out the problematic machine by
rebooting it.

Any ideas on what could cause this, or how to track down the problem?

Regards,
-- 
Tore Anderson

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Fri, 25 Apr 2008, Tore Anderson wrote:

> Hi.  I've been having recurring problems with the qla2xxx driver or
> firmware lockups.  Seems to happen out of the blue, with nothing special 
> going on on the SAN.
> 
> The servers are IBM BladeCenter HS21 8853A2Gs, with dual-port QLA2422
> cards connected to a dual-fabric topology.  They are running Ubuntu
> 6.06, kernel 2.6.22.19 with some OCFS2 patches applied.  qla2xxx driver
> version is 8.01.07-k7, loaded with params qlport_down_retry=35 and
> ql2xextended_error_logging=1.  Firmware is the latest from QLogic's FTP.
> 
> When they crash, the following is logged:
> 
> Apr 21 09:50:33 xander kernel: APIC error on CPU3: 00(40)

There's a slew of problem reports noted on the web with this 'APIC
error' signature...  From the qla2xxx driver perspective the following
logs show the classic 'no interrupts being routed' failures:

I/O needs to be aborted, request issued:

> Apr 21 09:51:18 xander kernel: qla2xxx_eh_abort(1): aborting sp ffff81010ae4c7c0 from RISC. pid=1024761.

Request times out, only recourse is for the driver to perform a full
RISC reset:

> Apr 21 09:51:48 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
> Apr 21 09:51:48 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
> Apr 21 09:51:48 xander kernel: qla2xxx 0000:08:01.0: Mailbox command timeout occured. Issuing ISP abort.
> Apr 21 09:51:48 xander kernel: qla2xxx 0000:08:01.0: Performing ISP error recovery - ha= ffff81021f5ec530.
> Apr 21 09:51:48 xander kernel: scsi(1): **** Load RISC code ****
> Apr 21 09:51:48 xander kernel: scsi(1): Verifying Checksum of loaded RISC code.
> Apr 21 09:51:48 xander kernel: scsi(1): Checksum OK, start firmware.
> Apr 21 09:51:48 xander kernel: scsi(1): Issue init firmware.
> Apr 21 09:51:49 xander kernel: scsi(1): fcport-0 - port retry count: 34 remaining
> Apr 21 09:51:49 xander kernel: scsi(1): fcport-1 - port retry count: 34 remaining
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous P2P MODE received.
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous LOOP UP (4 Gbps).
> Apr 21 09:51:49 xander kernel: qla2xxx 0000:08:01.0: LOOP UP detected (4 Gbps).
> Apr 21 09:51:49 xander kernel: scsi(1): F/W Ready - OK 
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE.

Note, the driver is in 'polling' mode during error recovery...

> Apr 21 09:51:49 xander kernel: scsi(1): Port database changed ffff 0006 0000.
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0000/0007/0b00.
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0001/0007/0b00.
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0004/0600.
> Apr 21 09:51:49 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0007/0b00.
> Apr 21 09:51:49 xander kernel: scsi(1): fw_state=3 curr time=102a04c2d.
> Apr 21 09:51:49 xander kernel: qla2x00_restart_isp(): Start configure loop, status = 0
> Apr 21 09:51:49 xander kernel: scsi(1): Configure loop -- dpc flags =0x4080048
> Apr 21 09:51:49 xander kernel: scsi(1): RSCN queue entry[0] = [00/000000].
> Apr 21 09:51:49 xander kernel: scsi(1): device_resync: rscn overflow.
> Apr 21 09:51:50 xander kernel: scsi(1): RFT_ID failed, completion status (280).
> Apr 21 09:51:50 xander kernel: scsi(1): Register FC-4 TYPE failed.
> Apr 21 09:51:50 xander kernel: scsi(1): RFF_ID failed, completion status (280).
> Apr 21 09:51:50 xander kernel: scsi(1): fcport-0 - port retry count: 33 remaining
> Apr 21 09:51:50 xander kernel: scsi(1): fcport-1 - port retry count: 33 remaining
> Apr 21 09:51:50 xander kernel: scsi(1): Register FC-4 Features failed.
> Apr 21 09:51:50 xander kernel: scsi(1): RNN_ID failed, completion status (280).
> Apr 21 09:51:50 xander kernel: scsi(1): Register Node Name failed.
> Apr 21 09:51:50 xander kernel: scsi(1): GID_PT failed, completion status (6380).
> Apr 21 09:51:50 xander kernel: scsi(1): GA_NXT failed, rejected request:
> Apr 21 09:51:50 xander kernel: 0   1   2   3   4   5   6   7   8   9  Ah  Bh  Ch  Dh  Eh  Fh
> Apr 21 09:51:50 xander kernel: --------------------------------------------------------------
> Apr 21 09:51:50 xander kernel: 14  00  00  00  00  70  26  1f  02  00  00  00  10  08  00  00
> Apr 21 09:51:50 xander kernel: qla2xxx 0000:08:01.0: SNS scan failed -- assuming zero-entry result...
> Apr 21 09:51:50 xander kernel: qla24xx_fabric_logout(1): failed to complete IOCB -- completion status (2)  ioparam=0/810031.
> Apr 21 09:51:50 xander kernel: scsi(1): LOOP READY
> Apr 21 09:51:50 xander kernel: qla2x00_restart_isp(): Configure loop done, status = 0x0
> Apr 21 09:51:50 xander kernel: APIC error on CPU4: 00(40)
> Apr 21 09:51:50 xander kernel: qla2x00_abort_isp(1): exiting.
> Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
> Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
> Apr 21 09:51:50 xander kernel: qla2x00_mailbox_command(1): **** FAILED. mbx0=54, mbx1=0, mbx2=1f58, cmd=54 ****
> Apr 21 09:51:50 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
> Apr 21 09:51:50 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
> Apr 21 09:51:50 xander kernel: qla24xx_abort_command(1): failed to issue IOCB (100).
> Apr 21 09:51:50 xander kernel: qla2xxx_eh_abort(1): abort_command mbx failed.

Error recovery (RISC reset) completes, transition to normal INTx
processing and continue.  Next abort request comes down:

> Apr 21 09:51:50 xander kernel: qla2xxx 0000:08:01.0: scsi(1:1:5): Abort command issued -- 0 fa2f9 2002.
> Apr 21 09:51:51 xander kernel: scsi(1): fcport-0 - port retry count: 32 remaining
...
> Apr 21 09:52:24 xander kernel: scsi(1): fcport-0 - port retry count: 0 remaining
> Apr 21 09:52:24 xander kernel: scsi(1): fcport-1 - port retry count: 0 remaining

So do the abort requests fail with the similar signature (timeout)?

> It varies on which CPU the APIC error happens, but after that it's
> always the same:  qla2xxx complaining and attempting to restart the
> firmware without any success, and I/O service never recovers.  Soon
> thereafter other cluster members fences out the problematic machine by
> rebooting it.
> 
> Any ideas on what could cause this, or how to track down the problem?

There's a blanket suggestion that has helped others (perhaps by
ignoring the problem), disable the APIC:

	apm=force noapic acpi=off pci=noacpi

but that seems like a bandaid.  I'd suggest you work this through your
IBM support contract, if possible.

BTW: I'd like to take a look at several failure iterations, could you
send the messages file during the failures...

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

Hi Andrew,

* Andrew Vasquez

> There's a slew of problem reports noted on the web with this 'APIC
> error' signature...  From the qla2xxx driver perspective the following
> logs show the classic 'no interrupts being routed' failures:

Yes - I suspect this might not have anything to do with the HBA at all.
It is a bit odd that it is always the qla2xxx driver that runs into
trouble, for instance will I/O to the local hard drives continue to work
(which is fortunate as that's where I have the kernel logs).

> So do the abort requests fail with the similar signature (timeout)?

The log isn't edited so if it doesn't say then I don't know.

I/O service never recovers after the crash, so the multipath maps blocks
all I/O until the machine is rebooted (which the remaining cluster
members take care of within a minute).

> There's a blanket suggestion that has helped others (perhaps by
> ignoring the problem), disable the APIC:
> 
> 	apm=force noapic acpi=off pci=noacpi
> 
> but that seems like a bandaid.  I'd suggest you work this through your
> IBM support contract, if possible.

I will try to do both, thank you for the suggestions.  I fear IBM will
hang up on me for not running SuSE or Red Hat, though...

> BTW: I'd like to take a look at several failure iterations, could you
> send the messages file during the failures...

Okay, sent you the (unedited) kern.log since the last log rotation.  It
contains several crash events, as well as the bootup messages (left them
in there in case there's anything interesting for you to see).

I have many more crash events in the rotated logs.  If you want I can
send you those too (maybe off list due to their size), just say so.
They all look the same, though:  APIC errors followed by qla2xxx
attempting to fix it, but the rports never recover and in the end the
machine is rebooted by another cluster node.

Regards,
-- 
Tore Anderson

[-- Attachment #2: kern.log.gz --]
[-- Type: application/x-gzip, Size: 46847 bytes --]

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Fri, 25 Apr 2008, Tore Anderson wrote:

> * Andrew Vasquez
> 
> > There's a slew of problem reports noted on the web with this 'APIC
> > error' signature...  From the qla2xxx driver perspective the following
> > logs show the classic 'no interrupts being routed' failures:
> 
> Yes - I suspect this might not have anything to do with the HBA at all.
> It is a bit odd that it is always the qla2xxx driver that runs into
> trouble, for instance will I/O to the local hard drives continue to work
> (which is fortunate as that's where I have the kernel logs).
> 
> > So do the abort requests fail with the similar signature (timeout)?
> 
> The log isn't edited so if it doesn't say then I don't know.
> 
> I/O service never recovers after the crash, so the multipath maps blocks
> all I/O until the machine is rebooted (which the remaining cluster
> members take care of within a minute).

Hmm, MSI is enabled:

	qla2xxx 0000:08:01.1: Found an ISP2422, irq 26, iobase 0xffffc20000c54000
	qla2xxx 0000:08:01.1: Configuring PCI space...
	qla2xxx 0000:08:01.1: Configure NVRAM parameters...
	qla2xxx 0000:08:01.1: Verifying loaded RISC code...
	scsi(2): **** Load RISC code ****
	scsi(2): Verifying Checksum of loaded RISC code.
	scsi(2): Checksum OK, start firmware.
	qla2xxx 0000:08:01.1: Allocated (64 KB) for EFT...
	qla2xxx 0000:08:01.1: Allocated (1413 KB) for firmware dump...
	scsi(2): Issue init firmware.
	qla2xxx 0000:08:01.1: MSI: Enabled.
	...

could you try disabling MSI via 'pci=nomsi' (I believe), we've dealt
with a large number of problem reports where customers reported 'odd'
behaviours (no interrupt routining) with several motherboard chipsets.
At least it could be another useful datapoint...

> > There's a blanket suggestion that has helped others (perhaps by
> > ignoring the problem), disable the APIC:
> > 
> > 	apm=force noapic acpi=off pci=noacpi
> > 
> > but that seems like a bandaid.  I'd suggest you work this through your
> > IBM support contract, if possible.
> 
> I will try to do both, thank you for the suggestions.  I fear IBM will
> hang up on me for not running SuSE or Red Hat, though...
> 
> > BTW: I'd like to take a look at several failure iterations, could you
> > send the messages file during the failures...
> 
> Okay, sent you the (unedited) kern.log since the last log rotation.  It
> contains several crash events, as well as the bootup messages (left them
> in there in case there's anything interesting for you to see).
> 
> I have many more crash events in the rotated logs.  If you want I can
> send you those too (maybe off list due to their size), just say so.
> They all look the same, though:  APIC errors followed by qla2xxx
> attempting to fix it, but the rports never recover and in the end the
> machine is rebooted by another cluster node.

Let's force the driver to operating in INTx mode...

--
av

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

Hi again,

* Andrew Vasquez

> Hmm, MSI is enabled:

Indeed - seems to be the default behaviour.

> could you try disabling MSI via 'pci=nomsi' (I believe), we've dealt 
> with a large number of problem reports where customers reported 'odd'
> behaviours (no interrupt routining) with several motherboard
> chipsets. At least it could be another useful datapoint...

The system has now been up for 58 hours straight with MSI disabled per
your suggestion.  This is almost a new record, usually it has crashed
once per day or more.  I won't open the champagne before a week more has
passed, but this looks _very_ promising!

Thank you very much for your help in tracking this down!  Be sure to let
me know if you want any more information and I'll see what I can do.
The system isn't yet back in full production, so I have a small window
of opportunity to reproduce the problem, test patches, and so on.

Regards,
-- 
Tore Anderson

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Mon, 28 Apr 2008, Tore Anderson wrote:

> * Andrew Vasquez
> 
> > Hmm, MSI is enabled:
> 
> Indeed - seems to be the default behaviour.
> 
> > could you try disabling MSI via 'pci=nomsi' (I believe), we've dealt 
> > with a large number of problem reports where customers reported 'odd'
> > behaviours (no interrupt routining) with several motherboard
> > chipsets. At least it could be another useful datapoint...
> 
> The system has now been up for 58 hours straight with MSI disabled per
> your suggestion.  This is almost a new record, usually it has crashed
> once per day or more.  I won't open the champagne before a week more has
> passed, but this looks _very_ promising!

That's good news...

> Thank you very much for your help in tracking this down!  Be sure to let
> me know if you want any more information and I'll see what I can do.
> The system isn't yet back in full production, so I have a small window
> of opportunity to reproduce the problem, test patches, and so on.

In general, most MSI/MSI-X machine/chipset quirks (global/bus-level
disabling) are handled by the PCI layer.  I'd be curious to see if any
of the 30+ updates to drivers/pci/quirk.c since 2.6.22 have addressed
this issue via DECLARE_PCI_FIXUP_FINAL()???

Could you send the output of 'lspci -vvx'?

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

Hi,

* Andrew Vasquez

> That's good news...

Very good news indeed - it's still up.

> In general, most MSI/MSI-X machine/chipset quirks (global/bus-level
> disabling) are handled by the PCI layer.  I'd be curious to see if any
> of the 30+ updates to drivers/pci/quirk.c since 2.6.22 have addressed
> this issue via DECLARE_PCI_FIXUP_FINAL()???
> 
> Could you send the output of 'lspci -vvx'?

Certainly!  Attached.

Regards,
-- 
Tore Anderson

[-- Attachment #2: lspci --]
[-- Type: text/plain, Size: 42807 bytes --]

00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 0
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x4
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 d8 25 44 05 10 00 b1 00 00 06 10 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 01 00 00

00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 2-3 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=07, subordinate=0b, sec-latency=0
	I/O behind bridge: 00005000-00005fff
	Memory behind bridge: d5000000-d6ffffff
	Prefetchable memory behind bridge: 00000000d1000000-00000000d10fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 2
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x8
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 f7 25 47 01 10 00 b1 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 07 0b 00 50 50 00 00
20: 00 d5 f0 d6 01 d1 01 d1 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Bus: primary=00, secondary=0f, subordinate=0f, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00000  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 3
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x0
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 e3 25 07 01 10 00 b1 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 0f 0f 00 f0 00 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 4 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=0c, subordinate=0d, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 4
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x4
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 e4 25 47 01 10 00 b1 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 0c 0d 00 f0 00 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=0e, subordinate=0e, sec-latency=0
	I/O behind bridge: 00006000-0000ffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 5
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 e5 25 47 01 10 00 b1 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 0e 0e 00 60 f0 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 6 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
	Memory behind bridge: da000000-dcffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 6
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x1
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 e6 25 47 01 10 00 b1 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 03 04 00 f0 00 00 20
20: 00 da f0 dc f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev b1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=05, subordinate=06, sec-latency=0
	Memory behind bridge: d7000000-d9ffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/1 Enable-
		Address: fee00020  Data: 0000
	Capabilities: [6c] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 7
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x1
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [100] Advanced Error Reporting
00: 86 80 e7 25 47 01 10 00 b1 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 05 06 00 f0 00 00 20
20: 00 d7 f0 d9 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 03 00

00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA Engine (rev b1)
	Subsystem: IBM Unknown device 02dd
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 5
	Region 0: Memory at fe700000 (64-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
		Address: fee00000  Data: 0000
	Capabilities: [6c] Express Unknown type IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed unknown, Width x0, ASPM unknown, Port 0
		Link: Latency L0s <64ns, L1 <1us
		Link: ASPM Disabled CommClk- ExtSynch-
		Link: Speed unknown, Width x0
00: 86 80 38 1a 46 01 10 00 b1 00 80 08 00 00 00 00
10: 04 00 70 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00

00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
	Subsystem: IBM Unknown device 02dd
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f0 25 00 00 00 00 b1 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f0 25 00 00 00 00 b1 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f0 25 00 00 00 00 b1 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f1 25 00 00 00 00 b1 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f3 25 00 00 00 00 b1 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f5 25 00 00 00 00 b1 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
	Subsystem: Intel Corporation Unknown device 8086
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 86 80 f6 25 00 00 00 00 b1 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 86 80
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 00004000-00004fff
	Memory behind bridge: dd000000-deffffff
	Prefetchable memory behind bridge: 00000000d1100000-00000000d11fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [40] Express Root Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
		Device: Latency L0s unlimited, L1 unlimited
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 1
		Link: Latency L0s <1us, L1 <4us
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4
		Root: Correctable- Non-Fatal- Fatal- PME-
	Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
		Address: 00000000  Data: 0000
	Capabilities: [90] Subsystem: Gammagraphx, Inc. Unknown device 0000
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100] Virtual Channel
	Capabilities: [180] Unknown (5)
00: 86 80 90 26 47 01 10 00 09 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 02 02 00 40 40 00 00
20: 00 dd f0 de 11 d1 11 d1 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 01 03 00

00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) (prog-if 00 [UHCI])
	Subsystem: IBM Unknown device 02dd
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 18
	Region 4: I/O ports at 2200 [size=32]
00: 86 80 88 26 05 00 80 02 09 00 03 0c 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 22 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 00

00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) (prog-if 00 [UHCI])
	Subsystem: IBM Unknown device 02dd
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at 2600 [size=32]
00: 86 80 89 26 05 00 80 02 09 00 03 0c 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 26 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 00 00 00 00 00 00 00 00 07 02 00 00

00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) (prog-if 00 [UHCI])
	Subsystem: IBM Unknown device 02dd
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin C routed to IRQ 19
	Region 4: I/O ports at 2800 [size=32]
00: 86 80 8a 26 05 00 80 02 09 00 03 0c 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 28 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 00 00 00 00 00 00 00 00 05 03 00 00

00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) (prog-if 20 [EHCI])
	Subsystem: IBM Unknown device 02dd
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin D routed to IRQ 23
	Region 0: Memory at f9000000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port
00: 86 80 8c 26 06 00 90 02 09 20 03 0c 00 00 00 00
10: 00 00 00 f9 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 50 00 00 00 00 00 00 00 07 04 00 00

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: f8000000-f8ffffff
	Prefetchable memory behind bridge: 00000000f0000000-00000000f7ffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B-
	Capabilities: [50] Subsystem: Gammagraphx, Inc. Unknown device 0000
00: 86 80 4e 24 47 01 10 00 d9 01 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 20 30 30 80 22
20: 00 f8 f0 f8 01 f0 f1 f7 00 00 00 00 00 00 00 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 ff 00 0f 00

00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
00: 86 80 70 26 47 01 00 02 09 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA Storage Controller IDE (rev 09) (prog-if 80 [Master])
	Subsystem: IBM Unknown device 02dd
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 0
	Region 0: I/O ports at 01f0 [size=8]
	Region 1: I/O ports at 03f4 [size=1]
	Region 2: I/O ports at 0170 [size=8]
	Region 3: I/O ports at 0374 [size=1]
	Region 4: I/O ports at 2a00 [size=16]
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 86 80 80 26 45 00 b8 02 09 80 01 01 00 00 00 00
10: 01 00 00 00 01 00 00 00 01 00 00 00 01 00 00 00
20: 01 2a 00 00 00 00 00 00 00 00 00 00 14 10 dd 02
30: 00 00 00 00 70 00 00 00 00 00 00 00 00 02 00 00

01:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) (prog-if 00 [VGA])
	Subsystem: IBM Unknown device 0305
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (2000ns min), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f0000000 (32-bit, prefetchable) [size=128M]
	Region 1: I/O ports at 3000 [size=256]
	Region 2: Memory at f8000000 (32-bit, non-prefetchable) [size=64K]
	[virtual] Expansion ROM at f8020000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 02 10 5e 51 87 01 90 02 02 00 00 03 10 40 00 00
10: 08 00 00 f0 01 30 00 00 00 00 00 f8 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 05 03
30: 00 00 00 00 50 00 00 00 00 00 00 00 0b 01 08 00

02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1064E PCI-Express Fusion-MPT SAS (rev 02)
	Subsystem: IBM Unknown device 0306
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: I/O ports at 4000 [size=256]
	Region 1: Memory at deffc000 (64-bit, non-prefetchable) [size=16K]
	Region 3: Memory at defe0000 (64-bit, non-prefetchable) [size=64K]
	[virtual] Expansion ROM at d1100000 [disabled] [size=1M]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Express Endpoint IRQ 0
		Device: Supported: MaxPayload 4096 bytes, PhantFunc 0, ExtTag+
		Device: Latency L0s <64ns, L1 <1us
		Device: AtnBtn- AtnInd- PwrInd-
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
		Link: Latency L0s <64ns, L1 <1us
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4
	Capabilities: [98] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1
		Vector table: BAR=1 offset=00002000
		PBA: BAR=1 offset=00003000
	Capabilities: [100] Advanced Error Reporting
00: 00 10 56 00 47 01 10 00 02 00 00 01 10 00 00 00
10: 01 40 00 00 04 c0 ff de 00 00 00 00 04 00 fe de
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 06 03
30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 00

03:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=03, secondary=04, subordinate=04, sec-latency=64
	Memory behind bridge: da000000-dcffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [60] Express PCI/PCI-X Bridge IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
		Device: Latency L0s <64ns, L1 <1us
		Device: AtnBtn- AtnInd- PwrInd-
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 0
		Link: Latency L0s <4us, L1 <4us
		Link: ASPM Disabled CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x1
	Capabilities: [90] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=03:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=0 CommitmentLimit=0
		Downstream: Capacity=0 CommitmentLimit=0
	Capabilities: [b0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [14c] Power Budgeting
	Capabilities: [160] Device Serial Number c8-ad-da-fe-ff-5e-14-00
00: 66 11 03 01 47 01 10 00 c3 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 03 04 04 40 f1 01 00 22
20: 00 da f0 dc f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 60 00 00 00 00 00 00 00 00 00 03 00

04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
	Subsystem: IBM Unknown device 0327
	Control: I/O- Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (16000ns min), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at da000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [40] PCI-X non-bridge device
		Command: DPERE- ERO- RBC=4096 OST=8
		Status: Dev=04:00.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=512 DMOST=8 DMCRS=32 RSCEM- 266MHz- 533MHz-
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 00000000feeff00c  Data: 41e1
00: e4 14 ac 16 5e 01 b0 02 12 00 00 02 10 40 00 00
10: 04 00 00 da 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 27 03
30: 00 00 00 00 40 00 00 00 00 00 00 00 07 01 40 00

05:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=05, secondary=06, subordinate=06, sec-latency=64
	Memory behind bridge: d7000000-d9ffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [60] Express PCI/PCI-X Bridge IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
		Device: Latency L0s <64ns, L1 <1us
		Device: AtnBtn- AtnInd- PwrInd-
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 0
		Link: Latency L0s <4us, L1 <4us
		Link: ASPM Disabled CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x1
	Capabilities: [90] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=05:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=0 CommitmentLimit=0
		Downstream: Capacity=0 CommitmentLimit=0
	Capabilities: [b0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [14c] Power Budgeting
	Capabilities: [160] Device Serial Number ca-ad-da-fe-ff-5e-14-00
00: 66 11 03 01 47 01 10 00 c3 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 05 06 06 40 f1 01 00 22
20: 00 d7 f0 d9 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 60 00 00 00 00 00 00 00 00 00 03 00

06:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
	Subsystem: IBM Unknown device 0327
	Control: I/O- Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (16000ns min), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 19
	Region 0: Memory at d8000000 (64-bit, non-prefetchable) [size=32M]
	Capabilities: [40] PCI-X non-bridge device
		Command: DPERE- ERO- RBC=4096 OST=8
		Status: Dev=06:00.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=512 DMOST=8 DMCRS=32 RSCEM- 266MHz- 533MHz-
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
	Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 00000000feeff00c  Data: 41e9
00: e4 14 ac 16 5e 01 b0 02 12 00 00 02 10 40 00 00
10: 04 00 00 d8 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 14 10 27 03
30: 00 00 00 00 40 00 00 00 00 00 00 00 07 01 40 00

07:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=07, secondary=09, subordinate=0b, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [44] Express Upstream Port IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: AtnBtn- AtnInd- PwrInd-
		Device: SlotPowerLimit 0.000000
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x8
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Subsystem: Gammagraphx, Inc. Unknown device 0000
	Capabilities: [100] Advanced Error Reporting
00: 86 80 00 35 47 01 10 00 01 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 07 09 0b 00 f0 00 00 20
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 44 00 00 00 00 00 00 00 00 00 03 00

07:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
	I/O behind bridge: 00005000-00005fff
	Memory behind bridge: d5000000-d6ffffff
	Prefetchable memory behind bridge: 00000000d1000000-00000000d10fffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: AtnBtn- AtnInd- PwrInd-
		Device: Errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
		Link: Latency L0s unlimited, L1 unlimited
		Link: ASPM Disabled CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x8
	Capabilities: [6c] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Subsystem: Gammagraphx, Inc. Unknown device 0000
	Capabilities: [d8] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=00:00.3 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=65535 CommitmentLimit=65535
		Downstream: Capacity=65535 CommitmentLimit=65535
	Capabilities: [100] Advanced Error Reporting
00: 86 80 0c 35 47 01 10 00 01 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 07 08 08 40 50 50 a0 02
20: 00 d5 f0 d6 01 d1 01 d1 00 00 00 00 00 00 00 00
30: 00 00 00 00 44 00 00 00 00 00 00 00 00 00 03 00

08:01.0 Fibre Channel: QLogic Corp. QLA2422 Fibre Channel Adapter (rev 02)
	Subsystem: QLogic Corp. Unknown device 0136
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 128 (16000ns min), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	Region 0: I/O ports at 5000 [size=256]
	Region 1: Memory at d6fff000 (64-bit, non-prefetchable) [size=4K]
	[virtual] Expansion ROM at d1000000 [disabled] [size=256K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [4c] PCI-X non-bridge device
		Command: DPERE- ERO+ RBC=2048 OST=8
		Status: Dev=08:01.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=4096 DMOST=12 DMCRS=128 RSCEM- 266MHz+ 533MHz-
	Capabilities: [64] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [74] Vital Product Data
00: 77 10 22 24 57 01 30 02 02 00 04 0c 10 80 80 00
10: 01 50 00 00 04 f0 ff d6 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 77 10 36 01
30: 00 00 00 00 44 00 00 00 00 00 00 00 07 01 40 00

08:01.1 Fibre Channel: QLogic Corp. QLA2422 Fibre Channel Adapter (rev 02)
	Subsystem: QLogic Corp. Unknown device 0136
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 128 (16000ns min), Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 26
	Region 0: I/O ports at 5100 [size=256]
	Region 1: Memory at d6ffe000 (64-bit, non-prefetchable) [size=4K]
	[virtual] Expansion ROM at d1040000 [disabled] [size=256K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [4c] PCI-X non-bridge device
		Command: DPERE- ERO+ RBC=2048 OST=8
		Status: Dev=08:01.1 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=4096 DMOST=12 DMCRS=128 RSCEM- 266MHz+ 533MHz-
	Capabilities: [64] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [74] Vital Product Data
00: 77 10 22 24 57 01 30 02 02 00 04 0c 10 80 80 00
10: 01 51 00 00 04 e0 ff d6 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 77 10 36 01
30: 00 00 00 00 44 00 00 00 00 00 00 00 0b 02 40 00

09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=09, secondary=0b, subordinate=0b, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [44] Express Downstream Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
		Link: Latency L0s <512ns, L1 unlimited
		Link: ASPM Disabled CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x0
	Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Subsystem: Gammagraphx, Inc. Unknown device 0000
	Capabilities: [100] Advanced Error Reporting
00: 86 80 10 35 47 01 10 00 01 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 09 0b 0b 00 f0 00 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 44 00 00 00 00 00 00 00 00 00 03 00

09:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=09, secondary=0a, subordinate=0a, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [44] Express Downstream Port (Slot-) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
		Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
		Link: Latency L0s <512ns, L1 unlimited
		Link: ASPM Disabled CommClk+ ExtSynch-
		Link: Speed 2.5Gb/s, Width x0
	Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Subsystem: Gammagraphx, Inc. Unknown device 0000
	Capabilities: [100] Advanced Error Reporting
00: 86 80 14 35 47 01 10 00 01 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 09 0a 0a 00 f0 00 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 44 00 00 00 00 00 00 00 00 00 03 00

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Tue, 29 Apr 2008, Tore Anderson wrote:

> > That's good news...
> 
> Very good news indeed - it's still up.
> 
> > In general, most MSI/MSI-X machine/chipset quirks (global/bus-level
> > disabling) are handled by the PCI layer.  I'd be curious to see if any
> > of the 30+ updates to drivers/pci/quirk.c since 2.6.22 have addressed
> > this issue via DECLARE_PCI_FIXUP_FINAL()???
> > 
> > Could you send the output of 'lspci -vvx'?
> 
> Certainly!  Attached.

Hmm, nothing really stands out on the quirks-front.  I know there's
several HS21s locally here, I'll try to get some cycles on the box to
see if there's something deeper going on...

We did notice that the RISC firmware being loaded is quite dated (15
months):

	QLogic Fibre Channel HBA Driver: 8.01.07-k7-debug
	QLogic QMC2462S - IBM eServer BC 4Gb FC Expansion Card SFF
	ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:08:01.0 hdma+, host#=1, fw=4.00.27 [IP] 

At a minimum, our FW folks suggested 4.01.00 should be used to address
any known residual MSI issues at the RISC-side.  You can download the
latest 24xx firmware (4.04.00) here:

	ftp://ftp.qlogic.com/outgoing/linux/firmware/ql2400_fw.bin

I'll do what I can to try to reproduce this locally with our HW.

Thanks,
AV

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

Hi,

* Andrew Vasquez

> We did notice that the RISC firmware being loaded is quite dated (15 
> months):

I did ensure that we were running the latest firmware when trying to
track down the problem.  From what I can tell 4.00.27 was the latest
available on your FTP up until the 23rd of April.  I simply didn't check
to see if any updates were published in the last week.

> At a minimum, our FW folks suggested 4.01.00 should be used to
> address any known residual MSI issues at the RISC-side.  You can
> download the latest 24xx firmware (4.04.00) here:

The sysadmins have requested that I leave the box alone for a few days
more so we can tell our customer that we're 100% confident we have a
viable workaround for the stability problems.  I'll reboot with MSI
enabled and FW 4.04.00 on Friday, and will let you know how it's going
on Monday.

I hope you're okay with waiting until then.

Regards,
-- 
Tore Anderson

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Wed, 30 Apr 2008, Tore Anderson wrote:

> * Andrew Vasquez
> 
> > We did notice that the RISC firmware being loaded is quite dated (15 
> > months):
> 
> I did ensure that we were running the latest firmware when trying to
> track down the problem.  From what I can tell 4.00.27 was the latest
> available on your FTP up until the 23rd of April.  I simply didn't check
> to see if any updates were published in the last week.

Ok, I see where the confusion lies...  I've always updated
ql2400_fw.bin to the latest FW version, and simply renamed older
versions to ql2400_fw.bin.X.YY.ZZ.  To avoid any future uncertainty,
I've changed the layout a bit and now archive older firmware versions
in an 'old' directory...

> > At a minimum, our FW folks suggested 4.01.00 should be used to
> > address any known residual MSI issues at the RISC-side.  You can
> > download the latest 24xx firmware (4.04.00) here:
> 
> The sysadmins have requested that I leave the box alone for a few days
> more so we can tell our customer that we're 100% confident we have a
> viable workaround for the stability problems.  I'll reboot with MSI
> enabled and FW 4.04.00 on Friday, and will let you know how it's going
> on Monday.
> 
> I hope you're okay with waiting until then.

That's fine.

Thanks again,
Andrew Vasquez

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

Hi Andrew,

* Andrew Vasquez

> Ok, I see where the confusion lies...  I've always updated
> ql2400_fw.bin to the latest FW version, and simply renamed older
> versions to ql2400_fw.bin.X.YY.ZZ.  To avoid any future uncertainty,
> I've changed the layout a bit and now archive older firmware versions
> in an 'old' directory...

I don't...

- ql2400_fw.bin is dated the 23rd of April 2008.
- The latest ql2400_fw.bin.X.YY.ZZ (now found in old/) is 4.00.27, dated
the 26th of February 2007.

How can I interpret this in any other way but to assume that 4.00.27 was
the latest available from the 26th of February 2007 to the 23rd of April
2008, at which point it was replaced with the current version (4.04.00)?

Anyway.  The problematic box has now been running with FW 4.04.00 and
MSI active for 60 hours now, so it looks like the old FW was the cause
of the instability.  I'll leave it running and will let you know if any
problems occur, though.

Thanks again for your help!

Regards,
-- 
Tore Anderson

Re: Recurring qla2xxx crashes (maybe APIC related)

[Tore Anderson]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

* Tore Anderson

> Anyway.  The problematic box has now been running with FW 4.04.00 and
> MSI active for 60 hours now, so it looks like the old FW was the
> cause of the instability.  I'll leave it running and will let you
> know if any problems occur, though.

It lasted 72 hours and 25 minutes with the new firmware.  More than what
the average was with the old one, but still not very impressive.

I'm attaching the kernel log from the entire period.

Now I'm running with the new FW and pci=nomsi.  I expect this to work
just as fine as with the old FW - I will let you know if not.

Regards,
-- 
Tore Anderson

[-- Attachment #2: kern.log --]
[-- Type: text/plain, Size: 100189 bytes --]

May  2 21:11:02 xander kernel: Linux version 2.6.22.19-ocfs2-rdac (root@xander) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Mon Mar 10 08:37:04 CET 2008
May  2 21:11:02 xander kernel: Command line: root=/dev/md0 elevator=deadline selinux=0 ro console=tty1 console=ttyS1,19200 
May  2 21:11:02 xander kernel: BIOS-provided physical RAM map:
May  2 21:11:02 xander kernel: BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
May  2 21:11:02 xander kernel: BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved)
May  2 21:11:02 xander kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
May  2 21:11:02 xander kernel: BIOS-e820: 0000000000100000 - 00000000cffbcd00 (usable)
May  2 21:11:02 xander kernel: BIOS-e820: 00000000cffbcd00 - 00000000cffd0000 (ACPI data)
May  2 21:11:02 xander kernel: BIOS-e820: 00000000cffd0000 - 00000000d0000000 (reserved)
May  2 21:11:02 xander kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
May  2 21:11:02 xander kernel: BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
May  2 21:11:02 xander kernel: BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
May  2 21:11:02 xander kernel: Entering add_active_range(0, 0, 157) 0 entries of 3200 used
May  2 21:11:02 xander kernel: Entering add_active_range(0, 256, 851900) 1 entries of 3200 used
May  2 21:11:02 xander kernel: Entering add_active_range(0, 1048576, 2293760) 2 entries of 3200 used
May  2 21:11:02 xander kernel: end_pfn_map = 2293760
May  2 21:11:02 xander kernel: DMI 2.4 present.
May  2 21:11:02 xander kernel: ACPI: RSDP 000FDFD0, 0024 (r2 IBM   )
May  2 21:11:02 xander kernel: ACPI: XSDT CFFCFF00, 003C (r1 IBM    SERBLADE     1001 IBM  45444F43)
May  2 21:11:02 xander kernel: ACPI: FACP CFFCFE40, 0084 (r2 IBM    SERBLADE     1001 IBM  45444F43)
May  2 21:11:02 xander kernel: ACPI: DSDT CFFBCD2B, 28D1 (r2 IBM    SERBLADE     1000 INTL 20060707)
May  2 21:11:02 xander kernel: ACPI: FACS CFFCFCC0, 0040
May  2 21:11:02 xander kernel: ACPI: APIC CFFCFD40, 00C8 (r1 IBM    SERBLADE     1001 IBM  45444F43)
May  2 21:11:02 xander kernel: ACPI: MCFG CFFCFD00, 003C (r1 IBM    SERBLADE     1001 IBM  45444F43)
May  2 21:11:02 xander kernel: No NUMA configuration found
May  2 21:11:02 xander kernel: Faking a node at 0000000000000000-0000000230000000
May  2 21:11:02 xander kernel: Entering add_active_range(0, 0, 157) 0 entries of 3200 used
May  2 21:11:02 xander kernel: Entering add_active_range(0, 256, 851900) 1 entries of 3200 used
May  2 21:11:02 xander kernel: Entering add_active_range(0, 1048576, 2293760) 2 entries of 3200 used
May  2 21:11:02 xander kernel: Bootmem setup node 0 0000000000000000-0000000230000000
May  2 21:11:02 xander kernel: Zone PFN ranges:
May  2 21:11:02 xander kernel: DMA             0 ->     4096
May  2 21:11:02 xander kernel: DMA32        4096 ->  1048576
May  2 21:11:02 xander kernel: Normal    1048576 ->  2293760
May  2 21:11:02 xander kernel: early_node_map[3] active PFN ranges
May  2 21:11:02 xander kernel: 0:        0 ->      157
May  2 21:11:02 xander kernel: 0:      256 ->   851900
May  2 21:11:02 xander kernel: 0:  1048576 ->  2293760
May  2 21:11:02 xander kernel: On node 0 totalpages: 2096985
May  2 21:11:02 xander kernel: DMA zone: 56 pages used for memmap
May  2 21:11:02 xander kernel: DMA zone: 1002 pages reserved
May  2 21:11:02 xander kernel: DMA zone: 2939 pages, LIFO batch:0
May  2 21:11:02 xander kernel: DMA32 zone: 14280 pages used for memmap
May  2 21:11:02 xander kernel: DMA32 zone: 833524 pages, LIFO batch:31
May  2 21:11:02 xander kernel: Normal zone: 17024 pages used for memmap
May  2 21:11:02 xander kernel: Normal zone: 1228160 pages, LIFO batch:31
May  2 21:11:02 xander kernel: ACPI: PM-Timer IO Port: 0x588
May  2 21:11:02 xander kernel: ACPI: Local APIC address 0xfee00000
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
May  2 21:11:02 xander kernel: Processor #0 (Bootup-CPU)
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x04] enabled)
May  2 21:11:02 xander kernel: Processor #4
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
May  2 21:11:02 xander kernel: Processor #1
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
May  2 21:11:02 xander kernel: Processor #2
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
May  2 21:11:02 xander kernel: Processor #3
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
May  2 21:11:02 xander kernel: Processor #5
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
May  2 21:11:02 xander kernel: Processor #6
May  2 21:11:02 xander kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
May  2 21:11:02 xander kernel: Processor #7
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x04] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x05] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x06] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: LAPIC_NMI (acpi_id[0x07] dfl dfl lint[0x1])
May  2 21:11:02 xander kernel: ACPI: IOAPIC (id[0x0e] address[0xfec00000] gsi_base[0])
May  2 21:11:02 xander kernel: IOAPIC[0]: apic_id 14, address 0xfec00000, GSI 0-23
May  2 21:11:02 xander kernel: ACPI: IOAPIC (id[0x0d] address[0xfec80000] gsi_base[24])
May  2 21:11:02 xander kernel: IOAPIC[1]: apic_id 13, address 0xfec80000, GSI 24-47
May  2 21:11:02 xander kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May  2 21:11:02 xander kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May  2 21:11:02 xander kernel: ACPI: IRQ0 used by override.
May  2 21:11:02 xander kernel: ACPI: IRQ2 used by override.
May  2 21:11:02 xander kernel: ACPI: IRQ9 used by override.
May  2 21:11:02 xander kernel: Setting APIC routing to flat
May  2 21:11:02 xander kernel: Using ACPI (MADT) for SMP configuration information
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 000000000009d000 - 00000000000a0000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000000a0000 - 00000000000e0000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000000e0000 - 0000000000100000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000cffbc000 - 00000000cffbd000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000cffbd000 - 00000000cffd0000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000cffd0000 - 00000000d0000000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000d0000000 - 00000000e0000000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000e0000000 - 00000000f0000000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000f0000000 - 00000000fec00000
May  2 21:11:02 xander kernel: swsusp: Registered nosave memory region: 00000000fec00000 - 0000000100000000
May  2 21:11:02 xander kernel: Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
May  2 21:11:02 xander kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs
May  2 21:11:02 xander kernel: PERCPU: Allocating 37768 bytes of per cpu data
May  2 21:11:02 xander kernel: Built 1 zonelists.  Total pages: 2064623
May  2 21:11:02 xander kernel: Kernel command line: root=/dev/md0 elevator=deadline selinux=0 ro console=tty1 console=ttyS1,19200 
May  2 21:11:02 xander kernel: Initializing CPU#0
May  2 21:11:02 xander kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
May  2 21:11:02 xander kernel: time.c: Detected 2000.122 MHz processor.
May  2 21:11:02 xander kernel: Console: colour VGA+ 80x25
May  2 21:11:02 xander kernel: Checking aperture...
May  2 21:11:02 xander kernel: Calgary: detecting Calgary via BIOS EBDA area
May  2 21:11:02 xander kernel: Calgary: Unable to locate Rio Grande table in EBDA - bailing!
May  2 21:11:02 xander kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
May  2 21:11:02 xander kernel: Placing software IO TLB between 0x108d000 - 0x508d000
May  2 21:11:02 xander kernel: Memory: 8188676k/9175040k available (2032k kernel code, 199264k reserved, 946k data, 296k init)
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4003.29 BogoMIPS (lpj=8006595)
May  2 21:11:02 xander kernel: Security Framework v1.0.0 initialized
May  2 21:11:02 xander kernel: SELinux:  Disabled at boot.
May  2 21:11:02 xander kernel: Capability LSM initialized
May  2 21:11:02 xander kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May  2 21:11:02 xander kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
May  2 21:11:02 xander kernel: Mount-cache hash table entries: 256
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 0/0 -> Node 0
May  2 21:11:02 xander kernel: using mwait in idle threads.
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 0
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 0
May  2 21:11:02 xander kernel: CPU0: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: SMP alternatives: switching to UP code
May  2 21:11:02 xander kernel: ACPI: Core revision 20070126
May  2 21:11:02 xander kernel: Using local APIC timer interrupts.
May  2 21:11:02 xander kernel: result 20834602
May  2 21:11:02 xander kernel: Detected 20.834 MHz APIC timer.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 1/8 APIC 0x4
May  2 21:11:02 xander kernel: Initializing CPU#1
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.51 BogoMIPS (lpj=8001021)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 1/4 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 1
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 0
May  2 21:11:02 xander kernel: CPU1: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#1]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 2/8 APIC 0x1
May  2 21:11:02 xander kernel: Initializing CPU#2
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.52 BogoMIPS (lpj=8001042)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 2/1 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 0
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 1
May  2 21:11:02 xander kernel: CPU2: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#2]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 3/8 APIC 0x2
May  2 21:11:02 xander kernel: Initializing CPU#3
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.41 BogoMIPS (lpj=8000824)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 3/2 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 0
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 2
May  2 21:11:02 xander kernel: CPU3: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#3]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 4/8 APIC 0x3
May  2 21:11:02 xander kernel: Initializing CPU#4
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.48 BogoMIPS (lpj=8000967)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 4/3 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 0
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 3
May  2 21:11:02 xander kernel: CPU4: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#4]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 5/8 APIC 0x5
May  2 21:11:02 xander kernel: Initializing CPU#5
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.49 BogoMIPS (lpj=8000989)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 5/5 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 1
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 1
May  2 21:11:02 xander kernel: CPU5: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#5]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 6/8 APIC 0x6
May  2 21:11:02 xander kernel: Initializing CPU#6
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.48 BogoMIPS (lpj=8000973)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 6/6 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 1
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 2
May  2 21:11:02 xander kernel: CPU6: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#6]: passed.
May  2 21:11:02 xander kernel: SMP alternatives: switching to SMP code
May  2 21:11:02 xander kernel: Booting processor 7/8 APIC 0x7
May  2 21:11:02 xander kernel: Initializing CPU#7
May  2 21:11:02 xander kernel: Calibrating delay using timer specific routine.. 4000.48 BogoMIPS (lpj=8000978)
May  2 21:11:02 xander kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May  2 21:11:02 xander kernel: CPU: L2 cache: 4096K
May  2 21:11:02 xander kernel: CPU 7/7 -> Node 0
May  2 21:11:02 xander kernel: CPU: Physical Processor ID: 1
May  2 21:11:02 xander kernel: CPU: Processor Core ID: 3
May  2 21:11:02 xander kernel: CPU7: Thermal monitoring enabled (TM1)
May  2 21:11:02 xander kernel: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz stepping 07
May  2 21:11:02 xander kernel: checking TSC synchronization [CPU#0 -> CPU#7]: passed.
May  2 21:11:02 xander kernel: Brought up 8 CPUs
May  2 21:11:02 xander kernel: migration_cost=18,9698
May  2 21:11:02 xander kernel: NET: Registered protocol family 16
May  2 21:11:02 xander kernel: ACPI: bus type pci registered
May  2 21:11:02 xander kernel: PCI: Using MMCONFIG at e0000000 - e0ffffff
May  2 21:11:02 xander kernel: ACPI: Interpreter enabled
May  2 21:11:02 xander kernel: ACPI: (supports S0 S4 S5)
May  2 21:11:02 xander kernel: ACPI: Using IOAPIC for interrupt routing
May  2 21:11:02 xander kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
May  2 21:11:02 xander kernel: PCI: Probing PCI hardware (bus 00)
May  2 21:11:02 xander kernel: PCI: Transparent bridge - 0000:00:1e.0
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCIX._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCI9.PCIA._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCI9.PCIB._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI3._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIE._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIF._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIS._PRT]
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP00] (IRQs *5)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP01] (IRQs *7)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP02] (IRQs *11)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP03] (IRQs *7)
May  2 21:11:02 xander kernel: ACPI: Blank IRQ resource
May  2 21:11:02 xander kernel: ACPI: Resource is not an IRQ entry
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP04] (IRQs) *0, disabled.
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP05] (IRQs *7)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP06] (IRQs *11)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt Link [LP07] (IRQs *7)
May  2 21:11:02 xander kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
May  2 21:11:02 xander kernel: pnp: PnP ACPI init
May  2 21:11:02 xander kernel: ACPI: bus type pnp registered
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0A08
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0C02
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0501
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0501
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0003
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0003
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0200
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0B00
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0800
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0C04
May  2 21:11:02 xander kernel: pnp: ACPI device : hid IPI0001
May  2 21:11:02 xander kernel: pnp: ACPI device : hid PNP0C02
May  2 21:11:02 xander kernel: pnp: PnP ACPI: found 12 devices
May  2 21:11:02 xander kernel: ACPI: ACPI bus type pnp unregistered
May  2 21:11:02 xander kernel: usbcore: registered new interface driver usbfs
May  2 21:11:02 xander kernel: usbcore: registered new interface driver hub
May  2 21:11:02 xander kernel: usbcore: registered new device driver usb
May  2 21:11:02 xander kernel: PCI: Using ACPI for IRQ routing
May  2 21:11:02 xander kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
May  2 21:11:02 xander kernel: NET: Registered protocol family 8
May  2 21:11:02 xander kernel: NET: Registered protocol family 20
May  2 21:11:02 xander kernel: PCI-GART: No AMD northbridge found.
May  2 21:11:02 xander kernel: pnp: the driver 'system' has been registered
May  2 21:11:02 xander kernel: pnp: match found with the PnP device '00:01' and the driver 'system'
May  2 21:11:02 xander kernel: pnp: 00:01: ioport range 0xc00-0xc1f has been reserved
May  2 21:11:02 xander kernel: Time: tsc clocksource has been installed.
May  2 21:11:02 xander kernel: pnp: 00:01: ioport range 0xc60-0xc67 has been reserved
May  2 21:11:02 xander kernel: pnp: match found with the PnP device '00:0a' and the driver 'system'
May  2 21:11:02 xander kernel: pnp: 00:0a: ioport range 0xc28-0xc28 has been reserved
May  2 21:11:02 xander kernel: pnp: 00:0a: ioport range 0xc2c-0xc2c has been reserved
May  2 21:11:02 xander kernel: pnp: match found with the PnP device '00:0b' and the driver 'system'
May  2 21:11:02 xander kernel: pnp: 00:0b: iomem range 0xe0000000-0xefffffff could not be reserved
May  2 21:11:02 xander kernel: pnp: 00:0b: iomem range 0xfed1c000-0xfed1ffff has been reserved
May  2 21:11:02 xander kernel: pnp: 00:0b: iomem range 0xfe000000-0xfe01ffff has been reserved
May  2 21:11:02 xander kernel: pnp: 00:0b: iomem range 0xfe020000-0xfe6fffff has been reserved
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:09:00.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:09:01.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:07:00.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:07:00.3
May  2 21:11:02 xander kernel: IO window: 5000-5fff
May  2 21:11:02 xander kernel: MEM window: d5000000-d6ffffff
May  2 21:11:02 xander kernel: PREFETCH window: d1000000-d10fffff
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:02.0
May  2 21:11:02 xander kernel: IO window: 5000-5fff
May  2 21:11:02 xander kernel: MEM window: d5000000-d6ffffff
May  2 21:11:02 xander kernel: PREFETCH window: d1000000-d10fffff
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:03.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:04.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:05.0
May  2 21:11:02 xander kernel: IO window: 6000-ffff
May  2 21:11:02 xander kernel: MEM window: disabled.
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:03:00.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: da000000-dcffffff
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:06.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: da000000-dcffffff
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:05:00.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: d7000000-d9ffffff
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:07.0
May  2 21:11:02 xander kernel: IO window: disabled.
May  2 21:11:02 xander kernel: MEM window: d7000000-d9ffffff
May  2 21:11:02 xander kernel: PREFETCH window: disabled.
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:1c.0
May  2 21:11:02 xander kernel: IO window: 4000-4fff
May  2 21:11:02 xander kernel: MEM window: dd000000-deffffff
May  2 21:11:02 xander kernel: PREFETCH window: d1100000-d11fffff
May  2 21:11:02 xander kernel: PCI: Bridge: 0000:00:1e.0
May  2 21:11:02 xander kernel: IO window: 3000-3fff
May  2 21:11:02 xander kernel: MEM window: f8000000-f8ffffff
May  2 21:11:02 xander kernel: PREFETCH window: f0000000-f7ffffff
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:02.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:07:00.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:09:00.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:09:01.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:07:00.3 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:03.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:04.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:05.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:06.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:03:00.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:07.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:05:00.0 to 64
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1c.0 to 64
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1e.0 to 64
May  2 21:11:02 xander kernel: NET: Registered protocol family 2
May  2 21:11:02 xander kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
May  2 21:11:02 xander kernel: TCP established hash table entries: 1048576 (order: 12, 25165824 bytes)
May  2 21:11:02 xander kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
May  2 21:11:02 xander kernel: TCP: Hash tables configured (established 1048576 bind 65536)
May  2 21:11:02 xander kernel: TCP reno registered
May  2 21:11:02 xander kernel: checking if image is initramfs... it is
May  2 21:11:02 xander kernel: Freeing initrd memory: 3286k freed
May  2 21:11:02 xander kernel: audit: initializing netlink socket (disabled)
May  2 21:11:02 xander kernel: audit(1209755421.636:1): initialized
May  2 21:11:02 xander kernel: VFS: Disk quotas dquot_6.5.1
May  2 21:11:02 xander kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May  2 21:11:02 xander kernel: io scheduler noop registered
May  2 21:11:02 xander kernel: io scheduler anticipatory registered
May  2 21:11:02 xander kernel: io scheduler deadline registered (default)
May  2 21:11:02 xander kernel: io scheduler cfq registered
May  2 21:11:02 xander kernel: Boot video device is 0000:01:01.0
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:02.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:02.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:02.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:03.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:03.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:03.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:04.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:04.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:04.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:05.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:05.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:05.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:06.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:06.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:06.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:07.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:07.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:07.0:pcie01]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1c.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:1c.0:pcie00]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:00:1c.0:pcie03]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:07:00.0 to 64
May  2 21:11:02 xander kernel: Allocate Port Service[0000:07:00.0:pcie10]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:07:00.0:pcie11]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:09:00.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:09:00.0:pcie20]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:09:00.0:pcie21]
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:09:01.0 to 64
May  2 21:11:02 xander kernel: assign_interrupt_mode Found MSI capability
May  2 21:11:02 xander kernel: Allocate Port Service[0000:09:01.0:pcie20]
May  2 21:11:02 xander kernel: Allocate Port Service[0000:09:01.0:pcie21]
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:02.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:03.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:04.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:05.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:06.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: aer_init: AER service init fails - No ACPI _OSC support
May  2 21:11:02 xander kernel: aer: probe of 0000:00:07.0:pcie01 failed with error 1
May  2 21:11:02 xander kernel: Real Time Clock Driver v1.12ac
May  2 21:11:02 xander kernel: Linux agpgart interface v0.102 (c) Dave Jones
May  2 21:11:02 xander kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
May  2 21:11:02 xander kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May  2 21:11:02 xander kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May  2 21:11:02 xander kernel: pnp: the driver 'serial' has been registered
May  2 21:11:02 xander kernel: pnp: match found with the PnP device '00:02' and the driver 'serial'
May  2 21:11:02 xander kernel: 00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May  2 21:11:02 xander kernel: pnp: match found with the PnP device '00:03' and the driver 'serial'
May  2 21:11:02 xander kernel: 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May  2 21:11:02 xander kernel: RAMDISK driver initialized: 16 RAM disks of 65536K size 1024 blocksize
May  2 21:11:02 xander kernel: pnp: the driver 'i8042 kbd' has been registered
May  2 21:11:02 xander kernel: pnp: the driver 'i8042 aux' has been registered
May  2 21:11:02 xander kernel: pnp: the driver 'i8042 kbd' has been unregistered
May  2 21:11:02 xander kernel: pnp: the driver 'i8042 aux' has been unregistered
May  2 21:11:02 xander kernel: PNP: No PS/2 controller found. Probing ports directly.
May  2 21:11:02 xander kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
May  2 21:11:02 xander kernel: mice: PS/2 mouse device common for all mice
May  2 21:11:02 xander kernel: TCP bic registered
May  2 21:11:02 xander kernel: NET: Registered protocol family 1
May  2 21:11:02 xander kernel: NET: Registered protocol family 17
May  2 21:11:02 xander kernel: Freeing unused kernel memory: 296k freed
May  2 21:11:02 xander kernel: loop: module loaded
May  2 21:11:02 xander kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
May  2 21:11:02 xander kernel: md: raid1 personality registered for level 1
May  2 21:11:02 xander kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
May  2 21:11:02 xander kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
May  2 21:11:02 xander kernel: Probing IDE interface ide0...
May  2 21:11:02 xander kernel: Probing IDE interface ide1...
May  2 21:11:02 xander kernel: SCSI subsystem initialized
May  2 21:11:02 xander kernel: Fusion MPT base driver 3.04.04
May  2 21:11:02 xander kernel: Copyright (c) 1999-2007 LSI Logic Corporation
May  2 21:11:02 xander kernel: Fusion MPT SAS Host driver 3.04.04
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
May  2 21:11:02 xander kernel: mptbase: Initiating ioc0 bringup
May  2 21:11:02 xander kernel: ioc0: SAS1064E: Capabilities={Initiator}
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:02:00.0 to 64
May  2 21:11:02 xander kernel: scsi0 : ioc0: LSISAS1064E, FwRev=000a0f00h, Ports=1, MaxQ=511, IRQ=16
May  2 21:11:02 xander kernel: scsi 0:0:0:0: Direct-Access     IBM-ESXS MAY2073RC        T107 PQ: 0 ANSI: 5
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] 143374000 512-byte hardware sectors (73407 MB)
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Write Protect is off
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Mode Sense: d7 00 00 08
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] 143374000 512-byte hardware sectors (73407 MB)
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Write Protect is off
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Mode Sense: d7 00 00 08
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sda: sda1 sda2
May  2 21:11:02 xander kernel: sd 0:0:0:0: [sda] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 0:0:1:0: Direct-Access     IBM-ESXS MAY2073RC        T107 PQ: 0 ANSI: 5
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] 143374000 512-byte hardware sectors (73407 MB)
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Write Protect is off
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Mode Sense: d7 00 00 08
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] 143374000 512-byte hardware sectors (73407 MB)
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Write Protect is off
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Mode Sense: d7 00 00 08
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdb: sdb1 sdb2
May  2 21:11:02 xander kernel: sd 0:0:1:0: [sdb] Attached SCSI disk
May  2 21:11:02 xander kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.5.11 (June 4, 2007)
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 17 (level, low) -> IRQ 17
May  2 21:11:02 xander kernel: eth0: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem da000000, IRQ 17, node addr 00145edaadc8
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 19 (level, low) -> IRQ 19
May  2 21:11:02 xander kernel: eth1: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem d8000000, IRQ 19, node addr 00145edaadca
May  2 21:11:02 xander kernel: ACPI: Processor [CPU0] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU1] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU2] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU3] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU4] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU5] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU6] (supports 8 throttling states)
May  2 21:11:02 xander kernel: ACPI: Processor [CPU7] (supports 8 throttling states)
May  2 21:11:02 xander kernel: md: md0 stopped.
May  2 21:11:02 xander kernel: md: bind<sdb1>
May  2 21:11:02 xander kernel: md: bind<sda1>
May  2 21:11:02 xander kernel: raid1: raid set md0 active with 2 out of 2 mirrors
May  2 21:11:02 xander kernel: md: md1 stopped.
May  2 21:11:02 xander kernel: md: bind<sdb2>
May  2 21:11:02 xander kernel: md: bind<sda2>
May  2 21:11:02 xander kernel: raid1: raid set md1 active with 2 out of 2 mirrors
May  2 21:11:02 xander kernel: device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
May  2 21:11:02 xander kernel: Attempting manual resume
May  2 21:11:02 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:11:02 xander kernel: EXT3-fs: mounted filesystem with ordered data mode.
May  2 21:11:02 xander kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
May  2 21:11:02 xander kernel: USB Universal Host Controller Interface driver v3.0
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 18 (level, low) -> IRQ 18
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1d.0 to 64
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.0: irq 18, io base 0x00002200
May  2 21:11:02 xander kernel: usb usb1: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 1-0:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 1-0:1.0: 2 ports detected
May  2 21:11:02 xander kernel: input: PC Speaker as /class/input/input0
May  2 21:11:02 xander kernel: intel_rng: FWH not detected
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 21 (level, low) -> IRQ 21
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1d.1 to 64
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.1: irq 21, io base 0x00002600
May  2 21:11:02 xander kernel: usb usb2: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 2-0:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 2-0:1.0: 2 ports detected
May  2 21:11:02 xander kernel: QLogic Fibre Channel HBA Driver
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 19 (level, low) -> IRQ 19
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1d.2 to 64
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
May  2 21:11:02 xander kernel: uhci_hcd 0000:00:1d.2: irq 19, io base 0x00002800
May  2 21:11:02 xander kernel: usb usb3: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 3-0:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 3-0:1.0: 2 ports detected
May  2 21:11:02 xander kernel: usb 2-1: new full speed USB device using uhci_hcd and address 2
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 23
May  2 21:11:02 xander kernel: PCI: Setting latency timer of device 0000:00:1d.7 to 64
May  2 21:11:02 xander kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
May  2 21:11:02 xander kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
May  2 21:11:02 xander kernel: ehci_hcd 0000:00:1d.7: debug port 1
May  2 21:11:02 xander kernel: usb 2-1: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 2-1:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 2-1:1.0: 2 ports detected
May  2 21:11:02 xander kernel: PCI: cache line size of 32 is not supported by device 0000:00:1d.7
May  2 21:11:02 xander kernel: ehci_hcd 0000:00:1d.7: irq 23, io mem 0xf9000000
May  2 21:11:02 xander kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May  2 21:11:02 xander kernel: usb usb4: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 4-0:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 4-0:1.0: 5 ports detected
May  2 21:11:02 xander kernel: usb 2-1: USB disconnect, address 2
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:08:01.0[A] -> GSI 25 (level, low) -> IRQ 25
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Found an ISP2422, irq 25, iobase 0xffffc20000c38000
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Configuring PCI space...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Configure NVRAM parameters...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Verifying loaded RISC code...
May  2 21:11:02 xander kernel: scsi(1): **** Load RISC code ****
May  2 21:11:02 xander kernel: scsi(1): Verifying Checksum of loaded RISC code.
May  2 21:11:02 xander kernel: scsi(1): Checksum OK, start firmware.
May  2 21:11:02 xander kernel: usb 4-3: new high speed USB device using ehci_hcd and address 2
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Allocated (64 KB) for EFT...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: Allocated (1413 KB) for firmware dump...
May  2 21:11:02 xander kernel: scsi(1): Issue init firmware.
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: MSI: Enabled.
May  2 21:11:02 xander kernel: DEBUG: detect hba 1 at address = ffff810224314530
May  2 21:11:02 xander kernel: usb 4-3: configuration #1 chosen from 1 choice
May  2 21:11:02 xander kernel: hub 4-3:1.0: USB hub found
May  2 21:11:02 xander kernel: hub 4-3:1.0: 2 ports detected
May  2 21:11:02 xander kernel: scsi1 : qla2xxx
May  2 21:11:02 xander kernel: scsi(1): Asynchronous P2P MODE received.
May  2 21:11:02 xander kernel: scsi(1): Asynchronous LOOP UP (4 Gbps).
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: LOOP UP detected (4 Gbps).
May  2 21:11:02 xander kernel: scsi(1): Asynchronous PORT UPDATE.
May  2 21:11:02 xander kernel: scsi(1): Port database changed ffff 0006 0000.
May  2 21:11:02 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0000/0004/0600.
May  2 21:11:02 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0000/0007/0b00.
May  2 21:11:02 xander kernel: scsi(1): qla2x00_reset_marker()
May  2 21:11:02 xander kernel: scsi(1): qla2x00_loop_resync()
May  2 21:11:02 xander kernel: scsi(1): F/W Ready - OK 
May  2 21:11:02 xander kernel: scsi(1): fw_state=3 curr time=fffef2a5.
May  2 21:11:02 xander kernel: scsi(1): Configure loop -- dpc flags =0x40800e0
May  2 21:11:02 xander kernel: scsi(1): RSCN queue entry[0] = [00/000000].
May  2 21:11:02 xander kernel: scsi(1): device_resync: rscn overflow.
May  2 21:11:02 xander kernel: scsi(1): RFT_ID exiting normally.
May  2 21:11:02 xander kernel: scsi(1): RFF_ID exiting normally.
May  2 21:11:02 xander kernel: scsi(1): RNN_ID exiting normally.
May  2 21:11:02 xander kernel: scsi(1): RSNN_NN exiting normally.
May  2 21:11:02 xander kernel: scsi(1): GID_PT entry - nn 200000e08b94d035 pn 210000e08b94d035 portid=620100.
May  2 21:11:02 xander kernel: scsi(1): GID_PT entry - nn 50060160c1e0a27d pn 5006016141e0a27d portid=630000.
May  2 21:11:02 xander kernel: scsi(1): GID_PT entry - nn 50060160c1e0a27d pn 5006016941e0a27d portid=630100.
May  2 21:11:02 xander kernel: scsi(1): GPSC failed, rejected request:
May  2 21:11:02 xander kernel: 0   1   2   3   4   5   6   7   8   9  Ah  Bh  Ch  Dh  Eh  Fh
May  2 21:11:02 xander kernel: --------------------------------------------------------------
May  2 21:11:02 xander kernel: 01  00  00  00  fa  01  00  00  80  01  00  00  00  01  00  00
May  2 21:11:02 xander kernel: scsi(1): GPSC command unsupported, disabling query...
May  2 21:11:02 xander kernel: scsi(1): device wrap (630100)
May  2 21:11:02 xander kernel: scsi(1): Trying Fabric Login w/loop id 0x0081 for port 630000.
May  2 21:11:02 xander kernel: scsi(1): 5006016141e0a27d -- unsupported FM port operating speed (0000).
May  2 21:11:02 xander kernel: scsi(1): Trying Fabric Login w/loop id 0x0082 for port 630100.
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: scsi(1:0:0:0): Queue depth adjusted-up to 4.
May  2 21:11:02 xander kernel: scsi 1:0:0:0: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdc: unknown partition table
May  2 21:11:02 xander kernel: scsi(1): 5006016941e0a27d -- unsupported FM port operating speed (0000).
May  2 21:11:02 xander kernel: scsi(1): LOOP READY
May  2 21:11:02 xander kernel: scsi(1): qla2x00_loop_resync - end
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: 
May  2 21:11:02 xander kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k7-debug
May  2 21:11:02 xander kernel: QLogic QMC2462S - IBM eServer BC 4Gb FC Expansion Card SFF
May  2 21:11:02 xander kernel: ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:08:01.0 hdma+, host#=1, fw=4.04.00 [IP] 
May  2 21:11:02 xander kernel: ACPI: PCI Interrupt 0000:08:01.1[B] -> GSI 26 (level, low) -> IRQ 26
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Found an ISP2422, irq 26, iobase 0xffffc20000c54000
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Configuring PCI space...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Configure NVRAM parameters...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Verifying loaded RISC code...
May  2 21:11:02 xander kernel: scsi(2): **** Load RISC code ****
May  2 21:11:02 xander kernel: scsi(2): Verifying Checksum of loaded RISC code.
May  2 21:11:02 xander kernel: scsi(2): Checksum OK, start firmware.
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Allocated (64 KB) for EFT...
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: Allocated (1413 KB) for firmware dump...
May  2 21:11:02 xander kernel: scsi(2): Issue init firmware.
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: MSI: Enabled.
May  2 21:11:02 xander kernel: DEBUG: detect hba 2 at address = ffff810224340530
May  2 21:11:02 xander kernel: scsi2 : qla2xxx
May  2 21:11:02 xander kernel: sd 1:0:0:0: [sdc] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 1:0:0:1: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: scsi(2): Asynchronous P2P MODE received.
May  2 21:11:02 xander kernel: scsi(2): Asynchronous LOOP UP (4 Gbps).
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdd:<6>qla2xxx 0000:08:01.1: LOOP UP detected (4 Gbps).
May  2 21:11:02 xander kernel: scsi(2): Asynchronous PORT UPDATE.
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: scsi(2): Port database changed ffff 0006 0000.
May  2 21:11:02 xander kernel: scsi(2): Asynchronous PORT UPDATE ignored 0000/0004/0600.
May  2 21:11:02 xander kernel: sd 1:0:0:1: [sdd] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 1:0:0:2: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: scsi(2): Asynchronous PORT UPDATE ignored 0000/0007/0b00.
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sde: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:0:2: [sde] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 1:0:0:3: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: scsi(2): qla2x00_reset_marker()
May  2 21:11:02 xander kernel: scsi(2): qla2x00_loop_resync()
May  2 21:11:02 xander kernel: scsi(2): F/W Ready - OK 
May  2 21:11:02 xander kernel: scsi(2): fw_state=3 curr time=fffef4bc.
May  2 21:11:02 xander kernel: scsi(2): Configure loop -- dpc flags =0x40800e0
May  2 21:11:02 xander kernel: scsi(2): RSCN queue entry[0] = [00/000000].
May  2 21:11:02 xander kernel: scsi(2): device_resync: rscn overflow.
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Write Protect is off
May  2 21:11:02 xander kernel: scsi(2): RFT_ID exiting normally.
May  2 21:11:02 xander kernel: scsi(2): RFF_ID exiting normally.
May  2 21:11:02 xander kernel: scsi(2): RNN_ID exiting normally.
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: scsi(2): RSNN_NN exiting normally.
May  2 21:11:02 xander kernel: sdf:scsi(2): GID_PT entry - nn 200100e08bb4d035 pn 210100e08bb4d035 portid=620100.
May  2 21:11:02 xander kernel: scsi(2): GID_PT entry - nn 50060160c1e0a27d pn 5006016041e0a27d portid=630000.
May  2 21:11:02 xander kernel: scsi(2): GID_PT entry - nn 50060160c1e0a27d pn 5006016841e0a27d portid=630100.
May  2 21:11:02 xander kernel: scsi(2): GPSC failed, rejected request:
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: 0   1   2   3   4   5   6   7   8   9  Ah  Bh  Ch  Dh  Eh  Fh
May  2 21:11:02 xander kernel: sd 1:0:0:3: [sdf] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 1:0:0:4: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: --------------------------------------------------------------
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: 01  00  00  00  fa  01  00  00  80  01  00  00  00  01  00  00
May  2 21:11:02 xander kernel: scsi(2): GPSC command unsupported, disabling query...
May  2 21:11:02 xander kernel: scsi(2): device wrap (630100)
May  2 21:11:02 xander kernel: scsi(2): Trying Fabric Login w/loop id 0x0081 for port 630000.
May  2 21:11:02 xander kernel: scsi(2): 5006016041e0a27d -- unsupported FM port operating speed (0000).
May  2 21:11:02 xander kernel: scsi(2): Trying Fabric Login w/loop id 0x0082 for port 630100.
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: scsi(2:0:0:0): Queue depth adjusted-up to 4.
May  2 21:11:02 xander kernel: scsi 2:0:0:0: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sdg:scsi(2): 5006016841e0a27d -- unsupported FM port operating speed (0000).
May  2 21:11:02 xander kernel: scsi(2): LOOP READY
May  2 21:11:02 xander kernel: scsi(2): qla2x00_loop_resync - end
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: 
May  2 21:11:02 xander kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k7-debug
May  2 21:11:02 xander kernel: QLogic QMC2462S - IBM eServer BC 4Gb FC Expansion Card SFF
May  2 21:11:02 xander kernel: ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:08:01.1 hdma+, host#=2, fw=4.04.00 [IP] 
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Write Protect is off
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:4: [sdg] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: scsi 1:0:0:5: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdh: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:0: [sdh] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: scsi 2:0:0:1: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdi: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:5: [sdi] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdj: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:1: [sdj] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 1:0:0:6: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: scsi 2:0:0:2: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdk:<5>sd 2:0:0:2: [sdl] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:0:6: [sdk] Attached SCSI disk
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.0: scsi(1:0:1:0): Queue depth adjusted-up to 4.
May  2 21:11:02 xander kernel: scsi 1:0:1:0: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Write Protect is off
May  2 21:11:02 xander kernel: sdl:<7>sd 1:0:1:0: [sdm] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:2: [sdl] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: scsi 2:0:0:3: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sdm:<5>sd 2:0:0:3: [sdn] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:0: [sdm] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdn:<5>scsi 1:0:1:1: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:0:3: [sdn] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdo: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:1: [sdo] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:0:4: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Write Protect is off
May  2 21:11:02 xander kernel: scsi 1:0:1:2: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdq:<7>sd 2:0:0:4: [sdp] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:2: [sdq] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdp:<5>scsi 1:0:1:3: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:4: [sdp] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: scsi 2:0:0:5: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdr: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:1:3: [sdr] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sds:<5>scsi 1:0:1:4: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Write Protect is off
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:5: [sds] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: scsi 2:0:0:6: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdt: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 1:0:1:4: [sdt] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: scsi 1:0:1:5: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:0:6: [sdu] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdu:<5>sd 1:0:1:5: [sdv] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdv:<5>sd 2:0:0:6: [sdu] Attached SCSI disk
May  2 21:11:02 xander kernel: qla2xxx 0000:08:01.1: scsi(2:0:1:0): Queue depth adjusted-up to 4.
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 1:0:1:5: [sdv] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:0: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: scsi 1:0:1:6: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Write Protect is off
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdw:<5>sd 1:0:1:6: [sdx] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdx: unknown partition table
May  2 21:11:02 xander kernel: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:0: [sdw] Attached SCSI disk
May  2 21:11:02 xander kernel: sd 1:0:1:6: [sdx] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:1: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdy: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:1: [sdy] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:2: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] 524288000 512-byte hardware sectors (268435 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdz: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:2: [sdz] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:3: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] 377487360 512-byte hardware sectors (193274 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdaa: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:3: [sdaa] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:4: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] 62914560 512-byte hardware sectors (32212 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdab: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:4: [sdab] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:5: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] 10485760 512-byte hardware sectors (5369 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdac: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:5: [sdac] Attached SCSI disk
May  2 21:11:02 xander kernel: scsi 2:0:1:6: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] 31457280 512-byte hardware sectors (16106 MB)
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Write Protect is off
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Mode Sense: 87 00 00 08
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
May  2 21:11:02 xander kernel: sdad: unknown partition table
May  2 21:11:02 xander kernel: sd 2:0:1:6: [sdad] Attached SCSI disk
May  2 21:11:02 xander kernel: device-mapper: multipath: version 1.0.5 loaded
May  2 21:11:02 xander kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
May  2 21:11:02 xander kernel: EXT3 FS on md0, internal journal
May  2 21:11:02 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:11:02 xander kernel: EXT3 FS on dm-1, internal journal
May  2 21:11:02 xander kernel: EXT3-fs: mounted filesystem with ordered data mode.
May  2 21:11:02 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:11:02 xander kernel: EXT3 FS on dm-2, internal journal
May  2 21:11:02 xander kernel: EXT3-fs: mounted filesystem with ordered data mode.
May  2 21:11:02 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:11:02 xander kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
May  2 21:11:02 xander kernel: EXT3 FS on dm-3, internal journal
May  2 21:11:02 xander kernel: EXT3-fs: mounted filesystem with ordered data mode.
May  2 21:11:02 xander kernel: Adding 16777208k swap on /dev/mapper/vg0-swap.  Priority:-1 extents:1 across:16777208k
May  2 21:11:02 xander kernel: Ethernet Channel Bonding Driver: v3.1.3 (June 13, 2007)
May  2 21:11:02 xander kernel: bonding: ARP monitoring set to 2500 ms, validate none, with 1 target(s): 87.238.48.193
May  2 21:11:02 xander kernel: bnx2: eth0: using MSI
May  2 21:11:02 xander kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
May  2 21:11:02 xander kernel: bonding: bond0: making interface eth0 the new active one.
May  2 21:11:02 xander kernel: bonding: bond0: first active interface up!
May  2 21:11:02 xander kernel: bonding: bond0: enslaving eth0 as an active interface with an up link.
May  2 21:11:02 xander kernel: bnx2: eth1: using MSI
May  2 21:11:02 xander kernel: bnx2: eth0 NIC Link is Down
May  2 21:11:02 xander kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
May  2 21:11:02 xander kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
May  2 21:11:02 xander kernel: bonding: bond0: enslaving eth1 as a backup interface with an up link.
May  2 21:11:02 xander kernel: bnx2: eth1 NIC Link is Down
May  2 21:11:02 xander kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
May  2 21:11:02 xander kernel: OCFS2 Node Manager 1.3.3
May  2 21:11:02 xander kernel: OCFS2 DLM 1.3.3
May  2 21:11:02 xander kernel: OCFS2 DLMFS 1.3.3
May  2 21:11:02 xander kernel: OCFS2 User DLM kernel interface loaded
May  2 21:12:08 xander kernel: o2net: accepted connection from node anya (num 3) at 192.168.52.16:7777
May  2 21:12:08 xander kernel: o2net: accepted connection from node rupert (num 2) at 192.168.52.15:7777
May  2 21:12:08 xander kernel: o2net: accepted connection from node spike (num 1) at 192.168.52.14:7777
May  2 21:12:11 xander kernel: OCFS2 1.3.3
May  2 21:12:11 xander kernel: ocfs2_dlm: Nodes in domain ("C9EDFC5A3F9B43AD8FBBECDCF1764EEB"): 0 1 2 3 
May  2 21:12:11 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:12:11 xander kernel: ocfs2: Mounting device (253,6) on (node 0, slot 1) with ordered data mode.
May  2 21:12:16 xander kernel: ocfs2_dlm: Nodes in domain ("A824FEDB0123417DB10959D47B8A70B5"): 0 1 2 3 
May  2 21:12:16 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:12:16 xander kernel: ocfs2: Mounting device (253,4) on (node 0, slot 0) with ordered data mode.
May  2 21:12:20 xander kernel: ocfs2_dlm: Nodes in domain ("69293E0D9AFB41DB9583F44FCC4ADFE4"): 0 1 2 3 
May  2 21:12:20 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:12:20 xander kernel: ocfs2: Mounting device (253,10) on (node 0, slot 1) with ordered data mode.
May  2 21:12:25 xander kernel: ocfs2_dlm: Nodes in domain ("86B1F86A2FC64FB78ECFB9EFE178E1AB"): 0 1 2 3 
May  2 21:12:25 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:12:25 xander kernel: ocfs2: Mounting device (253,7) on (node 0, slot 2) with ordered data mode.
May  2 21:12:29 xander kernel: ocfs2_dlm: Nodes in domain ("68D15727ECF34B9789FFFED12E041950"): 0 1 2 3 
May  2 21:12:29 xander kernel: kjournald starting.  Commit interval 5 seconds
May  2 21:12:29 xander kernel: ocfs2: Mounting device (253,5) on (node 0, slot 0) with ordered data mode.
May  2 21:12:50 xander kernel: ipmi message handler version 39.1
May  2 21:12:50 xander kernel: IPMI System Interface driver.
May  2 21:12:50 xander kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xc28, slave address 0x20, irq 0
May  2 21:12:50 xander kernel: ipmi: Found new BMC (man_id: 0x000002,  prod_id: 0x000d, dev_id: 0x20)
May  2 21:12:50 xander kernel: IPMI kcs interface initialized
May  2 21:12:50 xander kernel: ipmi device interface
May  4 01:06:02 xander kernel: md: data-check of RAID array md0
May  4 01:06:02 xander kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
May  4 01:06:02 xander kernel: md: delaying data-check of md1 until md0 has finished (they share one or more physical units)
May  4 01:06:02 xander kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
May  4 01:06:02 xander kernel: md: using 128k window, over a total of 13084800 blocks.
May  4 01:10:04 xander kernel: md: md0: data-check done.
May  4 01:10:04 xander kernel: md: data-check of RAID array md1
May  4 01:10:04 xander kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
May  4 01:10:04 xander kernel: RAID1 conf printout:
May  4 01:10:04 xander kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
May  4 01:10:04 xander kernel: md: using 128k window, over a total of 58596992 blocks.
May  4 01:10:04 xander kernel: --- wd:2 rd:2
May  4 01:10:04 xander kernel: disk 0, wo:0, o:1, dev:sda1
May  4 01:10:04 xander kernel: disk 1, wo:0, o:1, dev:sdb1
May  4 01:36:10 xander kernel: md: md1: data-check done.
May  4 01:36:10 xander kernel: RAID1 conf printout:
May  4 01:36:10 xander kernel: --- wd:2 rd:2
May  4 01:36:10 xander kernel: disk 0, wo:0, o:1, dev:sda2
May  4 01:36:10 xander kernel: disk 1, wo:0, o:1, dev:sdb2
May  5 21:36:03 xander kernel: APIC error on CPU4: 00(40)
May  5 21:36:34 xander kernel: qla2xxx_eh_abort(1): aborting sp ffff810224adf4c0 from RISC. pid=1510613.
May  5 21:37:04 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
May  5 21:37:04 xander kernel: qla2x00_mailbox_command(1): timeout calling abort_isp
May  5 21:37:04 xander kernel: qla2xxx 0000:08:01.0: Mailbox command timeout occured. Issuing ISP abort.
May  5 21:37:04 xander kernel: qla2xxx 0000:08:01.0: Performing ISP error recovery - ha= ffff810224314530.
May  5 21:37:04 xander kernel: scsi(1): **** Load RISC code ****
May  5 21:37:04 xander kernel: scsi(1): Verifying Checksum of loaded RISC code.
May  5 21:37:04 xander kernel: scsi(1): Checksum OK, start firmware.
May  5 21:37:04 xander kernel: scsi(1): fcport-0 - port retry count: 34 remaining
May  5 21:37:04 xander kernel: scsi(1): fcport-1 - port retry count: 34 remaining
May  5 21:37:04 xander kernel: scsi(1): Issue init firmware.
May  5 21:37:05 xander kernel: scsi(1): Asynchronous P2P MODE received.
May  5 21:37:05 xander kernel: scsi(1): Asynchronous LOOP UP (4 Gbps).
May  5 21:37:05 xander kernel: qla2xxx 0000:08:01.0: LOOP UP detected (4 Gbps).
May  5 21:37:05 xander kernel: scsi(1): Asynchronous PORT UPDATE.
May  5 21:37:05 xander kernel: scsi(1): Port database changed ffff 0006 0000.
May  5 21:37:05 xander kernel: scsi(1): F/W Ready - OK 
May  5 21:37:05 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0000/0007/0b00.
May  5 21:37:05 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0001/0007/0b00.
May  5 21:37:05 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0004/0600.
May  5 21:37:05 xander kernel: scsi(1): Asynchronous PORT UPDATE ignored 0002/0007/0b00.
May  5 21:37:05 xander kernel: scsi(1): fw_state=3 curr time=103e1ae99.
May  5 21:37:05 xander kernel: qla2x00_restart_isp(): Start configure loop, status = 0
May  5 21:37:06 xander kernel: scsi(1): fcport-0 - port retry count: 33 remaining
May  5 21:37:06 xander kernel: scsi(1): fcport-1 - port retry count: 33 remaining
May  5 21:37:06 xander kernel: scsi(1): Configure loop -- dpc flags =0x4080048
May  5 21:37:06 xander kernel: scsi(1): RSCN queue entry[0] = [00/000000].
May  5 21:37:06 xander kernel: scsi(1): device_resync: rscn overflow.
May  5 21:37:06 xander kernel: scsi(1): RFT_ID failed, completion status (280).
May  5 21:37:06 xander kernel: scsi(1): Register FC-4 TYPE failed.
May  5 21:37:06 xander kernel: scsi(1): RFF_ID failed, completion status (280).
May  5 21:37:06 xander kernel: scsi(1): Register FC-4 Features failed.
May  5 21:37:06 xander kernel: scsi(1): RNN_ID failed, completion status (280).
May  5 21:37:06 xander kernel: scsi(1): Register Node Name failed.
May  5 21:37:06 xander kernel: scsi(1): GID_PT failed, completion status (6380).
May  5 21:37:06 xander kernel: scsi(1): GA_NXT failed, rejected request:
May  5 21:37:06 xander kernel: 0   1   2   3   4   5   6   7   8   9  Ah  Bh  Ch  Dh  Eh  Fh
May  5 21:37:06 xander kernel: --------------------------------------------------------------
May  5 21:37:06 xander kernel: 14  00  00  00  00  90  7e  24  02  00  00  00  10  08  00  00
May  5 21:37:06 xander kernel: qla2xxx 0000:08:01.0: SNS scan failed -- assuming zero-entry result...
May  5 21:37:06 xander kernel: qla24xx_fabric_logout(1): failed to complete IOCB -- completion status (2)  ioparam=0/810031.
May  5 21:37:06 xander kernel: scsi(1): LOOP READY
May  5 21:37:06 xander kernel: qla2x00_restart_isp(): Configure loop done, status = 0x0
May  5 21:37:06 xander kernel: qla2x00_abort_isp(1): exiting.
May  5 21:37:06 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
May  5 21:37:06 xander kernel: qla2x00_mailbox_command(1): finished abort_isp
May  5 21:37:06 xander kernel: qla2x00_mailbox_command(1): **** FAILED. mbx0=54, mbx1=0, mbx2=2418, cmd=54 ****
May  5 21:37:06 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
May  5 21:37:06 xander kernel: qla2x00_issue_iocb(1): failed rval 0x100
May  5 21:37:06 xander kernel: qla24xx_abort_command(1): failed to issue IOCB (100).
May  5 21:37:06 xander kernel: qla2xxx_eh_abort(1): abort_command mbx failed.
May  5 21:37:06 xander kernel: qla2xxx 0000:08:01.0: scsi(1:1:5): Abort command issued -- 0 170cd5 2002.
May  5 21:37:06 xander kernel: APIC error on CPU5: 00(40)
May  5 21:37:07 xander kernel: scsi(1): fcport-0 - port retry count: 32 remaining
May  5 21:37:07 xander kernel: scsi(1): fcport-1 - port retry count: 32 remaining
May  5 21:37:08 xander kernel: scsi(1): fcport-0 - port retry count: 31 remaining
May  5 21:37:08 xander kernel: scsi(1): fcport-1 - port retry count: 31 remaining
May  5 21:37:09 xander kernel: scsi(1): fcport-0 - port retry count: 30 remaining
May  5 21:37:09 xander kernel: scsi(1): fcport-1 - port retry count: 30 remaining
May  5 21:37:10 xander kernel: scsi(1): fcport-0 - port retry count: 29 remaining
May  5 21:37:10 xander kernel: scsi(1): fcport-1 - port retry count: 29 remaining
May  5 21:37:11 xander kernel: scsi(1): fcport-0 - port retry count: 28 remaining
May  5 21:37:11 xander kernel: scsi(1): fcport-1 - port retry count: 28 remaining
May  5 21:37:12 xander kernel: scsi(1): fcport-0 - port retry count: 27 remaining
May  5 21:37:12 xander kernel: scsi(1): fcport-1 - port retry count: 27 remaining
May  5 21:37:13 xander kernel: scsi(1): fcport-0 - port retry count: 26 remaining
May  5 21:37:13 xander kernel: scsi(1): fcport-1 - port retry count: 26 remaining
May  5 21:37:14 xander kernel: scsi(1): fcport-0 - port retry count: 25 remaining
May  5 21:37:14 xander kernel: scsi(1): fcport-1 - port retry count: 25 remaining
May  5 21:37:15 xander kernel: scsi(1): fcport-0 - port retry count: 24 remaining
May  5 21:37:15 xander kernel: scsi(1): fcport-1 - port retry count: 24 remaining
May  5 21:37:17 xander kernel: scsi(1): fcport-0 - port retry count: 23 remaining
May  5 21:37:17 xander kernel: scsi(1): fcport-1 - port retry count: 23 remaining
May  5 21:37:18 xander kernel: scsi(1): fcport-0 - port retry count: 22 remaining
May  5 21:37:18 xander kernel: scsi(1): fcport-1 - port retry count: 22 remaining
May  5 21:37:19 xander kernel: scsi(1): fcport-0 - port retry count: 21 remaining
May  5 21:37:19 xander kernel: scsi(1): fcport-1 - port retry count: 21 remaining
May  5 21:37:20 xander kernel: scsi(1): fcport-0 - port retry count: 20 remaining
May  5 21:37:20 xander kernel: scsi(1): fcport-1 - port retry count: 20 remaining
May  5 21:37:21 xander kernel: scsi(1): fcport-0 - port retry count: 19 remaining
May  5 21:37:21 xander kernel: scsi(1): fcport-1 - port retry count: 19 remaining
May  5 21:37:22 xander kernel: scsi(1): fcport-0 - port retry count: 18 remaining
May  5 21:37:22 xander kernel: scsi(1): fcport-1 - port retry count: 18 remaining
May  5 21:37:23 xander kernel: scsi(1): fcport-0 - port retry count: 17 remaining
May  5 21:37:23 xander kernel: scsi(1): fcport-1 - port retry count: 17 remaining
May  5 21:37:24 xander kernel: scsi(1): fcport-0 - port retry count: 16 remaining
May  5 21:37:24 xander kernel: scsi(1): fcport-1 - port retry count: 16 remaining
May  5 21:37:25 xander kernel: scsi(1): fcport-0 - port retry count: 15 remaining
May  5 21:37:25 xander kernel: scsi(1): fcport-1 - port retry count: 15 remaining
May  5 21:37:26 xander kernel: scsi(1): fcport-0 - port retry count: 14 remaining
May  5 21:37:26 xander kernel: scsi(1): fcport-1 - port retry count: 14 remaining
May  5 21:37:27 xander kernel: scsi(1): fcport-0 - port retry count: 13 remaining
May  5 21:37:27 xander kernel: scsi(1): fcport-1 - port retry count: 13 remaining
May  5 21:37:28 xander kernel: scsi(1): fcport-0 - port retry count: 12 remaining
May  5 21:37:28 xander kernel: scsi(1): fcport-1 - port retry count: 12 remaining
May  5 21:37:29 xander kernel: scsi(1): fcport-0 - port retry count: 11 remaining
May  5 21:37:29 xander kernel: scsi(1): fcport-1 - port retry count: 11 remaining
May  5 21:37:30 xander kernel: scsi(1): fcport-0 - port retry count: 10 remaining
May  5 21:37:30 xander kernel: scsi(1): fcport-1 - port retry count: 10 remaining
May  5 21:37:31 xander kernel: scsi(1): fcport-0 - port retry count: 9 remaining
May  5 21:37:31 xander kernel: scsi(1): fcport-1 - port retry count: 9 remaining
May  5 21:37:32 xander kernel: scsi(1): fcport-0 - port retry count: 8 remaining
May  5 21:37:32 xander kernel: scsi(1): fcport-1 - port retry count: 8 remaining
May  5 21:37:33 xander kernel: scsi(1): fcport-0 - port retry count: 7 remaining
May  5 21:37:33 xander kernel: scsi(1): fcport-1 - port retry count: 7 remaining
May  5 21:37:34 xander kernel: scsi(1): fcport-0 - port retry count: 6 remaining
May  5 21:37:34 xander kernel: scsi(1): fcport-1 - port retry count: 6 remaining
May  5 21:37:35 xander kernel: scsi(1): fcport-0 - port retry count: 5 remaining
May  5 21:37:35 xander kernel: scsi(1): fcport-1 - port retry count: 5 remaining
May  5 21:37:37 xander kernel: scsi(1): fcport-0 - port retry count: 4 remaining
May  5 21:37:37 xander kernel: scsi(1): fcport-1 - port retry count: 4 remaining
May  5 21:37:38 xander kernel: scsi(1): fcport-0 - port retry count: 3 remaining
May  5 21:37:38 xander kernel: scsi(1): fcport-1 - port retry count: 3 remaining
May  5 21:37:39 xander kernel: scsi(1): fcport-0 - port retry count: 2 remaining
May  5 21:37:39 xander kernel: scsi(1): fcport-1 - port retry count: 2 remaining
May  5 21:37:40 xander kernel: scsi(1): fcport-0 - port retry count: 1 remaining
May  5 21:37:40 xander kernel: scsi(1): fcport-1 - port retry count: 1 remaining
May  5 21:37:41 xander kernel: scsi(1): fcport-0 - port retry count: 0 remaining
May  5 21:37:41 xander kernel: scsi(1): fcport-1 - port retry count: 0 remaining
May  5 21:37:44 xander kernel: rport-1:0-0: blocked FC remote port time out: removing target and saving binding
May  5 21:37:44 xander kernel: rport-1:0-1: blocked FC remote port time out: removing target and saving binding
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:45 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:0:2: scsi: Device offlined - not ready after error recovery
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: device-mapper: multipath: Failing path 65:80.
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:46 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:0:2: rejecting I/O to dead device
May  5 21:37:47 xander kernel: device-mapper: multipath: Failing path 8:64.
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: rejecting I/O to dead device
May  5 21:37:47 xander kernel: scsi 1:0:1:5: [sdv] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
May  5 21:37:47 xander kernel: end_request: I/O error, dev sdv, sector 2087680
May  5 21:37:47 xander kernel: scsi 1:0:0:2: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
May  5 21:37:47 xander kernel: end_request: I/O error, dev sde, sector 1586128

Re: Recurring qla2xxx crashes (maybe APIC related)

[Andrew Vasquez]

On Mon, 05 May 2008, Tore Anderson wrote:

> * Tore Anderson
> 
> > Anyway.  The problematic box has now been running with FW 4.04.00 and
> > MSI active for 60 hours now, so it looks like the old FW was the
> > cause of the instability.  I'll leave it running and will let you
> > know if any problems occur, though.
> 
> It lasted 72 hours and 25 minutes with the new firmware.  More than what
> the average was with the old one, but still not very impressive.
> 
> I'm attaching the kernel log from the entire period.

I managed to get an HS21 configuration (with 2422 mezz card) running
within the lab (with latest linux-2.6.git tree) against some IBM
storage.  It's been running now for 12+ hours with MSI enabled.  I'll
let it run for several days (if no failures), then restart with
2.6.22.19 for a weekend run...

> Now I'm running with the new FW and pci=nomsi.  I expect this to work
> just as fine as with the old FW - I will let you know if not.

Hrrm...  still haven't discounted some potential hardware problems...

Thanks for the update,
Andrew Vasquez