problems with scsi_transport

public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed

* problems with scsi_transport_fc and qla2xxx
@ 2006-02-27 12:05 Maxim Kozover
  2006-02-27 18:05 ` Maxim Kozover
  0 siblings, 1 reply; 19+ messages in thread
From: Maxim Kozover @ 2006-02-27 12:05 UTC (permalink / raw)
  To: linux-kernel

Hi!
I'm checking 2.6.16-rc5 with 2 QLogic 2312 adapters using qla2xxx
driver from 2.6.16-rc5.
As with earlier kernels, I think > 2.6.12 (since scsi_transport_fc
gained functionality) I have the following problem.
2 scsi hosts available, 4 and 5 (for QLogic).
I disconnect the cable from one of QLogic cards. After timeout I have
the message
rport-4:0-0: blocked FC remote port time out: removing target and saving binding
and appropriate SCSI devices that came from adapter 4 disappear from
/proc/scsi/scsi.
So far, so good.
I reconnect the cable, the directory
/sys/class/fc_remote_ports/rport-4:0-1 appears along with the old
ones rport-4:0-0 and rport-5:0-0, so currently I have 3.
However, no automatic rescan appears on adapter 4.
What's worse, if I try echo "0 1 0" > /sys/class/scsi_host/host4/scan
the process is stuck.
Please advise.

Thanks,

Maxim.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-02-27 12:05 problems with scsi_transport_fc and qla2xxx Maxim Kozover
@ 2006-02-27 18:05 ` Maxim Kozover
  2006-03-01  6:14   ` Andrew Morton
  0 siblings, 1 reply; 19+ messages in thread
From: Maxim Kozover @ 2006-02-27 18:05 UTC (permalink / raw)
  To: linux-kernel

Hi!
Most of the problem seems to be a QLogic driver problem.
HBAs are connected to target via FC switch.

1. If I have several LUNs on each HBA, with QLogic only 1 directory
per adapter (for LUN 0) is created in /sys/class/fc_remote_ports,
while with Emulex a directory for every LUN is created.

2. The situation I described occurs with QLogic only if the cable
connecting between HBA and switch is pulled out/in. If I
connect/disconnect the cable between switch and target, disks come
back.

3. With Emulex in both cases disks come back.

However, both with Emulex and QLogic stale directories in
/sys/classfc_remote_ports are left.
For example, with Emulex if I had in the beginning
rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-0  rport-7:0-1  rport-7:0-2
then disconnected adapter 7, got
rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-0  rport-7:0-2
(7-0-0 and 7-0-2 didn't disappear while 7-0-1 did)
connected 7 back
rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-2  rport-7:0-4
rport-7:0-5 rport-7:0-6
(7-0-0 disappeared, but 7-0-2 is still here).

Thanks,

Maxim.

MK> Hi!
MK> I'm checking 2.6.16-rc5 with 2 QLogic 2312 adapters using qla2xxx
MK> driver from 2.6.16-rc5.
MK> As with earlier kernels, I think > 2.6.12 (since scsi_transport_fc
MK> gained functionality) I have the following problem.
MK> 2 scsi hosts available, 4 and 5 (for QLogic).
MK> I disconnect the cable from one of QLogic cards. After timeout I have
MK> the message
MK> rport-4:0-0: blocked FC remote port time out: removing target and saving binding
MK> and appropriate SCSI devices that came from adapter 4 disappear from
MK> /proc/scsi/scsi.
MK> So far, so good.
MK> I reconnect the cable, the directory
MK> /sys/class/fc_remote_ports/rport-4:0-1 appears along with the old
MK> ones rport-4:0-0 and rport-5:0-0, so currently I have 3.
MK> However, no automatic rescan appears on adapter 4.
MK> What's worse, if I try echo "0 1 0" > /sys/class/scsi_host/host4/scan
MK> the process is stuck.
MK> Please advise.

MK> Thanks,

MK> Maxim.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-02-27 18:05 ` Maxim Kozover
@ 2006-03-01  6:14   ` Andrew Morton
  2006-03-01 20:25     ` Stefan Kaltenbrunner
  0 siblings, 1 reply; 19+ messages in thread
From: Andrew Morton @ 2006-03-01  6:14 UTC (permalink / raw)
  To: Maxim Kozover; +Cc: linux-kernel, linux-scsi, Andrew Vasquez

Maxim Kozover <maximkoz@netvision.net.il> wrote:
>
> Hi!

(cc's added)

> Most of the problem seems to be a QLogic driver problem.
> HBAs are connected to target via FC switch.
> 
> 1. If I have several LUNs on each HBA, with QLogic only 1 directory
> per adapter (for LUN 0) is created in /sys/class/fc_remote_ports,
> while with Emulex a directory for every LUN is created.
> 
> 2. The situation I described occurs with QLogic only if the cable
> connecting between HBA and switch is pulled out/in. If I
> connect/disconnect the cable between switch and target, disks come
> back.
> 
> 3. With Emulex in both cases disks come back.
> 
> However, both with Emulex and QLogic stale directories in
> /sys/classfc_remote_ports are left.
> For example, with Emulex if I had in the beginning
> rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-0  rport-7:0-1  rport-7:0-2
> then disconnected adapter 7, got
> rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-0  rport-7:0-2
> (7-0-0 and 7-0-2 didn't disappear while 7-0-1 did)
> connected 7 back
> rport-6:0-0  rport-6:0-1  rport-6:0-2  rport-7:0-2  rport-7:0-4
> rport-7:0-5 rport-7:0-6
> (7-0-0 disappeared, but 7-0-2 is still here).
> 
> Thanks,
> 
> Maxim.
> 
> MK> Hi!
> MK> I'm checking 2.6.16-rc5 with 2 QLogic 2312 adapters using qla2xxx
> MK> driver from 2.6.16-rc5.
> MK> As with earlier kernels, I think > 2.6.12 (since scsi_transport_fc
> MK> gained functionality) I have the following problem.
> MK> 2 scsi hosts available, 4 and 5 (for QLogic).
> MK> I disconnect the cable from one of QLogic cards. After timeout I have
> MK> the message
> MK> rport-4:0-0: blocked FC remote port time out: removing target and saving binding
> MK> and appropriate SCSI devices that came from adapter 4 disappear from
> MK> /proc/scsi/scsi.
> MK> So far, so good.
> MK> I reconnect the cable, the directory
> MK> /sys/class/fc_remote_ports/rport-4:0-1 appears along with the old
> MK> ones rport-4:0-0 and rport-5:0-0, so currently I have 3.
> MK> However, no automatic rescan appears on adapter 4.
> MK> What's worse, if I try echo "0 1 0" > /sys/class/scsi_host/host4/scan
> MK> the process is stuck.
> MK> Please advise.
> 
> MK> Thanks,
> 
> MK> Maxim.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-03-01  6:14   ` Andrew Morton
@ 2006-03-01 20:25     ` Stefan Kaltenbrunner
  2006-03-01 21:08       ` Andrew Vasquez
  0 siblings, 1 reply; 19+ messages in thread
From: Stefan Kaltenbrunner @ 2006-03-01 20:25 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Maxim Kozover, linux-kernel, linux-scsi, Andrew Vasquez

Andrew Morton wrote:
> Maxim Kozover <maximkoz@netvision.net.il> wrote:
> 
>>Hi!
> 
> 
> (cc's added)
> 
> 
>>Most of the problem seems to be a QLogic driver problem.
>>HBAs are connected to target via FC switch.
>>
>>1. If I have several LUNs on each HBA, with QLogic only 1 directory
>>per adapter (for LUN 0) is created in /sys/class/fc_remote_ports,
>>while with Emulex a directory for every LUN is created.
>>
>>2. The situation I described occurs with QLogic only if the cable
>>connecting between HBA and switch is pulled out/in. If I
>>connect/disconnect the cable between switch and target, disks come
>>back.

I can confirm that very problem (pulling the cable between HBA and
switch results in only LUN 0 or nothing coming back afterward) on
2.6.15.4 here too.


Stefan

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-03-01 20:25     ` Stefan Kaltenbrunner
@ 2006-03-01 21:08       ` Andrew Vasquez
  2006-03-02 16:32         ` Maxim Kozover
  0 siblings, 1 reply; 19+ messages in thread
From: Andrew Vasquez @ 2006-03-01 21:08 UTC (permalink / raw)
  To: Stefan Kaltenbrunner
  Cc: Andrew Morton, Maxim Kozover, linux-kernel, linux-scsi

On Wed, 01 Mar 2006, Stefan Kaltenbrunner wrote:

> Andrew Morton wrote:
> > Maxim Kozover <maximkoz@netvision.net.il> wrote:
> > 
> >>Hi!
> > 
> > 
> > (cc's added)
> > 
> > 
> >>Most of the problem seems to be a QLogic driver problem.
> >>HBAs are connected to target via FC switch.
> >>
> >>1. If I have several LUNs on each HBA, with QLogic only 1 directory
> >>per adapter (for LUN 0) is created in /sys/class/fc_remote_ports,
> >>while with Emulex a directory for every LUN is created.
> >>
> >>2. The situation I described occurs with QLogic only if the cable
> >>connecting between HBA and switch is pulled out/in. If I
> >>connect/disconnect the cable between switch and target, disks come
> >>back.
> 
> I can confirm that very problem (pulling the cable between HBA and
> switch results in only LUN 0 or nothing coming back afterward) on
> 2.6.15.4 here too.

Please try recent 2.6.16-rcX kernels as there have been a number of
patches submitted since 2.6.15 which (attempt to) address most of
these holes -- I'm still trying to get additional details on Maxim's.

387f96b4d9391bf3ce6928fb9cd90c9c7df37291 [PATCH] qla2xxx: Close window on race between rport removal and fcport transition.
77427f514f88143bfef41ba8c1e624bc45f42297 [SCSI] qla2xxx: Drop legacy 'bypass lun scan for tape device' code.
052c40c83b4ca37be226112049b60097cb9961e1 [SCSI] qla2xxx: Correct issue where the rport's upcall was not being made after relogin.
d97994dc1fddcbb8212b745d9c9c9ce96262155c [SCSI] qla2xxx: Correct synchronization issues during rport addition/deletion.
79f89a4296ff22f09baf538d4ff2a6d0c3097a73 [SCSI] qla2xxx: Disable port-type RSCN handling via driver state-machine.

Regards,
Andrew Vasquez

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re:Re: problems with scsi_transport_fc and qla2xxx
  2006-03-01 21:08       ` Andrew Vasquez
@ 2006-03-02 16:32         ` Maxim Kozover
  2006-03-02 17:38           ` Andrew Vasquez
  2006-03-06 20:00           ` Mike Snitzer
  0 siblings, 2 replies; 19+ messages in thread
From: Maxim Kozover @ 2006-03-02 16:32 UTC (permalink / raw)
  To: Andrew Vasquez
  Cc: Stefan Kaltenbrunner, Andrew Morton, linux-kernel, linux-scsi

Hi Andrew!
Today I tested disconnecting QLogic port.
Adapter 4 is connected via switch to a storage and 3 LUNs are seen via
the adapter.
Only 1 rport is created (for FCP Target) while in Emulex case there
were 3: (Fabric Port, Directory Server and FCP Target, FCP Initiator).
# ls /sys/class/fc_remote_ports/
rport-4:0-0
# cat /sys/class/fc_remote_ports/*/roles
FCP Target

Default dev_loss_tmo is 6 (1+5) while in Emulex case the default was 35.

After disconnecting the cable between the HBA and the switch
qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
 rport-4:0-0: blocked FC remote port time out: removing target and saving binding

# ls /sys/class/fc_remote_ports/
rport-4:0-0
# cat /sys/class/fc_remote_ports/*/roles
unknown

Relevant scsi devices are removed from /proc/scsi/scsi.

After reconnecting the cable
qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).

# ls /sys/class/fc_remote_ports/
rport-4:0-0
# cat /sys/class/fc_remote_ports/*/roles
FCP Target

However, scsi devices don't reappear in /proc/scsi/scsi.
When I issue rescan, the command is stuck
echo - - - > /sys/class/scsi_host/host4/scan

Please advise.

Thanks,

Maxim.

I'm including /var/log/messages output:
# egrep -v 'Vendor|cron' /var/log/messages
Mar  2 19:13:49 multipath kernel: QLogic Fibre Channel HBA Driver
Mar  2 19:13:49 multipath kernel: GSI 24 sharing vector 0xE9 and IRQ 24
Mar  2 19:13:49 multipath kernel: ACPI: PCI Interrupt 0000:03:01.0[A] -> GSI 48 (level, low) -> IRQ 233
Mar  2 19:13:49 multipath kernel: qla2xxx 0000:03:01.0: Found an ISP2312, irq 233, iobase 0xffffc20000004000
Mar  2 19:13:49 multipath kernel: qla2xxx 0000:03:01.0: Configuring PCI space...
Mar  2 19:13:49 multipath kernel: qla2xxx 0000:03:01.0: Configure NVRAM parameters...
Mar  2 19:13:49 multipath kernel: qla2xxx 0000:03:01.0: Verifying loaded RISC code...
Mar  2 19:13:49 multipath kernel: qla2xxx 0000:03:01.0: Waiting for LIP to complete...
Mar  2 19:13:51 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
Mar  2 19:13:51 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
Mar  2 19:13:51 multipath kernel: qla2xxx 0000:03:01.0: Topology - (F_Port), Host Loop address 0xffff
Mar  2 19:13:52 multipath kernel: scsi4 : qla2xxx
Mar  2 19:13:52 multipath kernel: qla2xxx 0000:03:01.0:
Mar  2 19:13:52 multipath kernel:  QLogic Fibre Channel HBA Driver: 8.01.04-k
Mar  2 19:13:52 multipath kernel:   QLogic QLA2340 -
Mar  2 19:13:52 multipath kernel:   ISP2312: PCI-X (133 MHz) @ 0000:03:01.0 hdma-, host#=4, fw=3.03.18 IPX
Mar  2 19:13:52 multipath kernel: GSI 25 sharing vector 0x32 and IRQ 25
Mar  2 19:13:52 multipath kernel: ACPI: PCI Interrupt 0000:06:01.0[A] -> GSI 72 (level, low) -> IRQ 50
Mar  2 19:13:52 multipath kernel: qla2xxx 0000:06:01.0: Configuring PCI space...
Mar  2 19:13:52 multipath kernel: qla2xxx 0000:06:01.0: Configure NVRAM parameters...
Mar  2 19:13:52 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  2 19:13:52 multipath kernel: SCSI device sdb: 33554432 512-byte hdwr sectors (17180 MB)
Mar  2 19:13:52 multipath kernel: sdb: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdb: drive cache: write through
Mar  2 19:13:52 multipath kernel: SCSI device sdb: 33554432 512-byte hdwr sectors (17180 MB)
Mar  2 19:13:52 multipath kernel: sdb: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdb: drive cache: write through
Mar  2 19:13:52 multipath kernel:  sdb: unknown partition table
Mar  2 19:13:52 multipath kernel: sd 4:0:0:0: Attached scsi disk sdb
Mar  2 19:13:52 multipath kernel: qla2xxx 0000:06:01.0: Verifying loaded RISC code...
Mar  2 19:13:52 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  2 19:13:52 multipath scsi.agent[4288]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:0
Mar  2 19:13:52 multipath kernel: SCSI device sdc: 67108864 512-byte hdwr sectors (34360 MB)
Mar  2 19:13:52 multipath kernel: sdc: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdc: drive cache: write through
Mar  2 19:13:52 multipath kernel: SCSI device sdc: 67108864 512-byte hdwr sectors (34360 MB)
Mar  2 19:13:52 multipath kernel: sdc: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdc: drive cache: write through
Mar  2 19:13:52 multipath kernel:  sdc: unknown partition table
Mar  2 19:13:52 multipath kernel: sd 4:0:0:1: Attached scsi disk sdc
Mar  2 19:13:52 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  2 19:13:52 multipath kernel: sdd : very big device. try to use READ CAPACITY(16).
Mar  2 19:13:52 multipath kernel: SCSI device sdd: 9797894144 512-byte hdwr sectors (5016522 MB)
Mar  2 19:13:52 multipath kernel: sdd: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdd: drive cache: write through
Mar  2 19:13:52 multipath kernel: sdd : very big device. try to use READ CAPACITY(16).
Mar  2 19:13:52 multipath kernel: SCSI device sdd: 9797894144 512-byte hdwr sectors (5016522 MB)
Mar  2 19:13:52 multipath kernel: sdd: Write Protect is off
Mar  2 19:13:52 multipath kernel: SCSI device sdd: drive cache: write through
Mar  2 19:13:52 multipath scsi.agent[4313]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:1
Mar  2 19:13:52 multipath kernel:  sdd: unknown partition table
Mar  2 19:13:52 multipath kernel: sd 4:0:0:2: Attached scsi disk sdd
Mar  2 19:13:52 multipath scsi.agent[4331]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:2
Mar  2 19:13:52 multipath kernel: qla2xxx 0000:06:01.0: Waiting for LIP to complete...
Mar  2 19:14:12 multipath kernel: qla2xxx 0000:06:01.0: Cable is unplugged...
Mar  2 19:14:12 multipath kernel: scsi5 : qla2xxx
Mar  2 19:14:12 multipath kernel: qla2xxx 0000:06:01.0:
Mar  2 19:14:12 multipath kernel:  QLogic Fibre Channel HBA Driver: 8.01.04-k
Mar  2 19:14:12 multipath kernel:   QLogic QLA2340 -
Mar  2 19:14:12 multipath kernel:   ISP2312: PCI-X (133 MHz) @ 0000:06:01.0 hdma-, host#=5, fw=3.03.18 IPX
Mar  2 19:14:17 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  2 19:14:18 multipath last message repeated 19 times
Mar  2 19:22:02 multipath kernel: qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
Mar  2 19:22:08 multipath kernel:  rport-4:0-0: blocked FC remote port time out: removing target and saving binding
Mar  2 19:24:04 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
Mar  2 19:24:04 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re:Re: problems with scsi_transport_fc and qla2xxx
  2006-03-02 16:32         ` Maxim Kozover
@ 2006-03-02 17:38           ` Andrew Vasquez
  2006-03-02 22:15             ` Maxim Kozover
  2006-03-06 20:00           ` Mike Snitzer
  1 sibling, 1 reply; 19+ messages in thread
From: Andrew Vasquez @ 2006-03-02 17:38 UTC (permalink / raw)
  To: Maxim Kozover
  Cc: Stefan Kaltenbrunner, Andrew Morton, linux-kernel, linux-scsi

On Thu, 02 Mar 2006, Maxim Kozover wrote:

> Today I tested disconnecting QLogic port.
> Adapter 4 is connected via switch to a storage and 3 LUNs are seen via
> the adapter.
> Only 1 rport is created (for FCP Target) while in Emulex case there
> were 3: (Fabric Port, Directory Server and FCP Target, FCP Initiator).

That's correct, we currently don't make an upcall for the SNS server
port nor the switch fabric port.

> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> FCP Target
> 
> Default dev_loss_tmo is 6 (1+5) while in Emulex case the default was 35.
> 
> After disconnecting the cable between the HBA and the switch
> qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
>  rport-4:0-0: blocked FC remote port time out: removing target and saving binding
> 
> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> unknown
> 
> Relevant scsi devices are removed from /proc/scsi/scsi.
> 
> After reconnecting the cable
> qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
> qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
> 
> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> FCP Target
> 
> However, scsi devices don't reappear in /proc/scsi/scsi.
> When I issue rescan, the command is stuck
> echo - - - > /sys/class/scsi_host/host4/scan
> 
> Please advise.

Could you add the enable-debug patch I sent you earlier and retry the
test?  Again forward the relevent snippets from var/log/messages.

Here's the patch again.

--
av

---

diff --git a/drivers/scsi/qla2xxx/qla_dbg.h b/drivers/scsi/qla2xxx/qla_dbg.h
index 935a59a..632f653 100644
--- a/drivers/scsi/qla2xxx/qla_dbg.h
+++ b/drivers/scsi/qla2xxx/qla_dbg.h
@@ -9,6 +9,7 @@
  */
 /* #define QL_DEBUG_LEVEL_1  */ /* Output register accesses to COM1 */
 /* #define QL_DEBUG_LEVEL_2  */ /* Output error msgs to COM1 */
+#define QL_DEBUG_LEVEL_2   /* Output error msgs to COM1 */
 /* #define QL_DEBUG_LEVEL_3  */ /* Output function trace msgs to COM1 */
 /* #define QL_DEBUG_LEVEL_4  */ /* Output NVRAM trace msgs to COM1 */
 /* #define QL_DEBUG_LEVEL_5  */ /* Output ring trace msgs to COM1 */
diff --git a/drivers/scsi/qla2xxx/qla_settings.h b/drivers/scsi/qla2xxx/qla_settings.h
index 363205c..b2e22b0 100644
--- a/drivers/scsi/qla2xxx/qla_settings.h
+++ b/drivers/scsi/qla2xxx/qla_settings.h
@@ -8,7 +8,7 @@
  * Compile time Options:
  *     0 - Disable and 1 - Enable
  */
-#define DEBUG_QLA2100		0	/* For Debug of qla2x00 */
+#define DEBUG_QLA2100		1	/* For Debug of qla2x00 */
 
 #define USE_ABORT_TGT		1	/* Use Abort Target mbx cmd */
 



^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: Re: problems with scsi_transport_fc and qla2xxx
  2006-03-02 17:38           ` Andrew Vasquez
@ 2006-03-02 22:15             ` Maxim Kozover
  2006-03-06 19:06               ` Andrew Vasquez
  0 siblings, 1 reply; 19+ messages in thread
From: Maxim Kozover @ 2006-03-02 22:15 UTC (permalink / raw)
  To: Andrew Vasquez
  Cc: Stefan Kaltenbrunner, Andrew Morton, linux-kernel, linux-scsi

Hi Andrew!
Please see the log with debug-patch.
The module is loaded with option qlport_down_retry=1.
Adapter 4 is connected to switch, adapter 5 doesn't have cable attached.
After reconnecting the cable the disks don't reappear and rescan is stuck.
Before applying your patches ghost rport was staying, now it's OK.

Thanks,

Maxim.

# egrep -v 'Vendor|cron' /var/log/messages
Mar  3 01:05:17 multipath kernel: QLogic Fibre Channel HBA Driver
Mar  3 01:05:17 multipath kernel: GSI 24 sharing vector 0xE9 and IRQ 24
Mar  3 01:05:17 multipath kernel: ACPI: PCI Interrupt 0000:03:01.0[A] -> GSI 48 (level, low) -> IRQ 233
Mar  3 01:05:17 multipath kernel: qla2xxx 0000:03:01.0: Found an ISP2312, irq 233, iobase 0xffffc20000004000
Mar  3 01:05:17 multipath kernel: qla2xxx 0000:03:01.0: Configuring PCI space...
Mar  3 01:05:17 multipath kernel: qla2xxx 0000:03:01.0: Configure NVRAM parameters...
Mar  3 01:05:17 multipath kernel: qla2xxx 0000:03:01.0: Verifying loaded RISC code...
Mar  3 01:05:17 multipath kernel: scsi(4): **** Load RISC code ****
Mar  3 01:05:17 multipath kernel: scsi(4): Verifying Checksum of loaded RISC code.
Mar  3 01:05:17 multipath kernel: scsi(4): Checksum OK, start firmware.
Mar  3 01:05:17 multipath kernel: scsi(4): Issue init firmware.
Mar  3 01:05:17 multipath kernel: qla2xxx 0000:03:01.0: Waiting for LIP to complete...
Mar  3 01:05:19 multipath kernel: scsi(4): Asynchronous LIP RESET (f7f7).
Mar  3 01:05:19 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
Mar  3 01:05:19 multipath kernel: scsi(4): Asynchronous P2P MODE received.
Mar  3 01:05:20 multipath kernel: scsi(4): Asynchronous LOOP UP (2 Gbps).
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
Mar  3 01:05:20 multipath kernel: scsi(4): Asynchronous PORT UPDATE.
Mar  3 01:05:20 multipath kernel: scsi(4): Port database changed ffff 0006 0000.
Mar  3 01:05:20 multipath kernel: scsi(4): F/W Ready - OK
Mar  3 01:05:20 multipath kernel: scsi(4): fw_state=3 curr time=ffff3dac.
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:03:01.0: Topology - (F_Port), Host Loop address 0xffff
Mar  3 01:05:20 multipath kernel: scsi(4): Configure loop -- dpc flags =0x4080040
Mar  3 01:05:20 multipath kernel: scsi(4): RSCN queue entry[0] = [00/000000].
Mar  3 01:05:20 multipath kernel: scsi(4): device_resync: rscn overflow.
Mar  3 01:05:20 multipath kernel: scsi(4): RFT_ID exiting normally.
Mar  3 01:05:20 multipath kernel: scsi(4): RFF_ID exiting normally.
Mar  3 01:05:20 multipath kernel: scsi(4): RNN_ID exiting normally.
Mar  3 01:05:20 multipath kernel: scsi(4): RSNN_NN exiting normally.
Mar  3 01:05:20 multipath kernel: scsi(4): GID_PT entry - nn 200000e08b079a69 pn 210000e08b079a69 portid=010700.
Mar  3 01:05:20 multipath kernel: scsi(4): GID_PT entry - nn 2000001738279c00 pn 1000001738279c11 portid=010200.
Mar  3 01:05:20 multipath kernel: scsi(4): device wrap (010200)
Mar  3 01:05:20 multipath kernel: scsi(4): Trying Fabric Login w/loop id 0x0081 for port 010200.
Mar  3 01:05:20 multipath kernel: scsi(4): LOOP READY
Mar  3 01:05:20 multipath kernel: DEBUG: detect hba 4 at address = ffff81006f5f8548
Mar  3 01:05:20 multipath kernel: scsi4 : qla2xxx
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:03:01.0:
Mar  3 01:05:20 multipath kernel:  QLogic Fibre Channel HBA Driver: 8.01.04-k-debug
Mar  3 01:05:20 multipath kernel:   QLogic QLA2340 -
Mar  3 01:05:20 multipath kernel:   ISP2312: PCI-X (133 MHz) @ 0000:03:01.0 hdma-, host#=4, fw=3.03.18 IPX
Mar  3 01:05:20 multipath kernel: GSI 25 sharing vector 0x32 and IRQ 25
Mar  3 01:05:20 multipath kernel: ACPI: PCI Interrupt 0000:06:01.0[A] -> GSI 72 (level, low) -> IRQ 50
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:06:01.0: Found an ISP2312, irq 50, iobase 0xffffc20000006000
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:06:01.0: Configuring PCI space...
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:06:01.0: Configure NVRAM parameters...
Mar  3 01:05:20 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  3 01:05:20 multipath kernel: SCSI device sdb: 33554432 512-byte hdwr sectors (17180 MB)
Mar  3 01:05:20 multipath kernel: sdb: Write Protect is off
Mar  3 01:05:20 multipath kernel: SCSI device sdb: drive cache: write through
Mar  3 01:05:20 multipath kernel: SCSI device sdb: 33554432 512-byte hdwr sectors (17180 MB)
Mar  3 01:05:20 multipath kernel: sdb: Write Protect is off
Mar  3 01:05:20 multipath kernel: SCSI device sdb: drive cache: write through
Mar  3 01:05:20 multipath kernel:  sdb: unknown partition table
Mar  3 01:05:20 multipath kernel: sd 4:0:0:0: Attached scsi disk sdb
Mar  3 01:05:20 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  3 01:05:20 multipath kernel: SCSI device sdc: 67108864 512-byte hdwr sectors (34360 MB)
Mar  3 01:05:20 multipath kernel: sdc: Write Protect is off
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:06:01.0: Verifying loaded RISC code...
Mar  3 01:05:20 multipath kernel: scsi(5): **** Load RISC code ****
Mar  3 01:05:20 multipath kernel: SCSI device sdc: drive cache: write through
Mar  3 01:05:20 multipath kernel: SCSI device sdc: 67108864 512-byte hdwr sectors (34360 MB)
Mar  3 01:05:20 multipath kernel: sdc: Write Protect is off
Mar  3 01:05:20 multipath kernel: SCSI device sdc: drive cache: write through
Mar  3 01:05:20 multipath kernel:  sdc: unknown partition table
Mar  3 01:05:20 multipath kernel: sd 4:0:0:1: Attached scsi disk sdc
Mar  3 01:05:20 multipath scsi.agent[4295]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:0
Mar  3 01:05:20 multipath kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar  3 01:05:20 multipath kernel: scsi(5): Verifying Checksum of loaded RISC code.
Mar  3 01:05:20 multipath kernel: sdd : very big device. try to use READ CAPACITY(16).
Mar  3 01:05:20 multipath kernel: SCSI device sdd: 9797894144 512-byte hdwr sectors (5016522 MB)
Mar  3 01:05:20 multipath kernel: sdd: Write Protect is off
Mar  3 01:05:20 multipath scsi.agent[4330]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:1
Mar  3 01:05:20 multipath kernel: SCSI device sdd: drive cache: write through
Mar  3 01:05:20 multipath scsi.agent[4342]: disk at /devices/pci0000:00/0000:00:02.0/0000:01:00.2/0000:03:01.0/host4/rport-4:0-0/target4:0:0/4:0:0:2
Mar  3 01:05:20 multipath kernel: sdd : very big device. try to use READ CAPACITY(16).
Mar  3 01:05:20 multipath kernel: SCSI device sdd: 9797894144 512-byte hdwr sectors (5016522 MB)
Mar  3 01:05:20 multipath kernel: scsi(5): Checksum OK, start firmware.
Mar  3 01:05:20 multipath kernel: sdd: Write Protect is off
Mar  3 01:05:20 multipath kernel: SCSI device sdd: drive cache: write through
Mar  3 01:05:20 multipath kernel:  sdd: unknown partition table
Mar  3 01:05:20 multipath kernel: sd 4:0:0:2: Attached scsi disk sdd
Mar  3 01:05:20 multipath kernel: scsi(5): Issue init firmware.
Mar  3 01:05:20 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0006/7ee5.
Mar  3 01:05:20 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0007/7ee5.
Mar  3 01:05:20 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0004/7ee5.
Mar  3 01:05:20 multipath kernel: qla2xxx 0000:06:01.0: Waiting for LIP to complete...
Mar  3 01:05:21 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0081/0006/7ee5.
Mar  3 01:05:41 multipath kernel: qla2xxx 0000:06:01.0: Cable is unplugged...
Mar  3 01:05:41 multipath kernel: scsi(5): fw_state=4 curr time=ffff522d.
Mar  3 01:05:41 multipath kernel: scsi(5): Firmware ready **** FAILED ****.
Mar  3 01:05:41 multipath kernel: DEBUG: detect hba 5 at address = ffff81006e8f0548
Mar  3 01:05:41 multipath kernel: scsi5 : qla2xxx
Mar  3 01:05:41 multipath kernel: qla2xxx 0000:06:01.0:
Mar  3 01:05:41 multipath kernel:  QLogic Fibre Channel HBA Driver: 8.01.04-k-debug
Mar  3 01:05:41 multipath kernel:   QLogic QLA2340 -
Mar  3 01:05:41 multipath kernel:   ISP2312: PCI-X (133 MHz) @ 0000:06:01.0 hdma-, host#=5, fw=3.03.18 IPX
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:46 multipath kernel: qla2x00_mailbox_command(5): **** FAILED. mbx0=4005, mbx1=0, mbx2=6e92, cmd=6b ****
Mar  3 01:05:46 multipath kernel: qla2x00_get_link_status(5): failed=102.
Mar  3 01:05:46 multipath kernel: qla2xxx 0000:06:01.0: Unable to retrieve host statistics (65535).
Mar  3 01:05:50 multipath kernel: scsi(5): Loop Down - aborting the queues before time expire
-------------------------------
Disconnecting the cable
Mar  3 01:06:34 multipath kernel: scsi(4): Asynchronous LOOP DOWN (2).
Mar  3 01:06:34 multipath kernel: qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
Mar  3 01:06:34 multipath kernel: scsi(4): fcport-0 - port retry count: 0 remaining
Mar  3 01:06:40 multipath kernel:  rport-4:0-0: blocked FC remote port time out: removing target and saving binding
Mar  3 01:06:42 multipath kernel: scsi(4): Loop Down - aborting the queues before time expire
Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous LIP RESET (f7f7).
Mar  3 01:07:22 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
Mar  3 01:07:22 multipath kernel: scsi(4): qla2x00_reset_marker()
Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous P2P MODE received.
Mar  3 01:07:22 multipath kernel: scsi(4): qla2x00_reset_marker()
Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous LOOP UP (2 Gbps).
Mar  3 01:07:22 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous PORT UPDATE.
Mar  3 01:07:22 multipath kernel: scsi(4): Port database changed ffff 0006 7ee5.
Mar  3 01:07:22 multipath kernel: scsi(4): qla2x00_loop_resync()
Mar  3 01:07:22 multipath kernel: scsi(4): F/W Ready - OK
Mar  3 01:07:22 multipath kernel: scsi(4): fw_state=3 curr time=ffffb533.
Mar  3 01:07:22 multipath kernel: scsi(4): Configure loop -- dpc flags =0x4090060
Mar  3 01:07:22 multipath kernel: scsi(4): RSCN queue entry[0] = [00/000000].
Mar  3 01:07:22 multipath kernel: scsi(4): device_resync: rscn overflow.
Mar  3 01:07:22 multipath kernel: scsi(4): RFT_ID exiting normally.
Mar  3 01:07:22 multipath kernel: scsi(4): RFF_ID exiting normally.
Mar  3 01:07:22 multipath kernel: scsi(4): RNN_ID exiting normally.
Mar  3 01:07:22 multipath kernel: scsi(4): RSNN_NN exiting normally.
Mar  3 01:07:22 multipath kernel: scsi(4): GID_PT entry - nn 200000e08b079a69 pn 210000e08b079a69 portid=010700.
Mar  3 01:07:22 multipath kernel: scsi(4): GID_PT entry - nn 2000001738279c00 pn 1000001738279c11 portid=010200.
Mar  3 01:07:22 multipath kernel: scsi(4): device wrap (010200)
Mar  3 01:07:22 multipath kernel: scsi(4): Trying Fabric Login w/loop id 0x0081 for port 010200.
Mar  3 01:07:22 multipath kernel: scsi(4): LOOP READY
Mar  3 01:07:22 multipath kernel: scsi(4): qla2x00_loop_resync - end
Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0081/0007/7ee5.
Mar  3 01:07:22 multipath kernel: scsi(4:0:0): status_entry: Port Down pid=43, compl status=0x29, port state=0x4
Mar  3 01:07:22 multipath kernel: scsi(4): Port login retry: 1000001738279c11, id = 0x0081 retry cnt=8
Mar  3 01:07:23 multipath kernel: scsi(4): fcport-0 - port retry count: 0 remaining
Mar  3 01:07:23 multipath kernel: scsi(4): qla2x00_port_login()
Mar  3 01:07:23 multipath kernel: scsi(4): Trying Fabric Login w/loop id 0x0081 for port 010200.
Mar  3 01:07:23 multipath kernel: scsi(4): port login OK: logged in ID 0x81
Mar  3 01:07:23 multipath kernel: scsi(4): qla2x00_port_login - end
Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0006/0001.
Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0007/0001.
Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0004/0001.
Mar  3 01:07:24 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0081/0006/0001.
Mar  3 01:09:56 multipath kernel: scsi(5): Loop down exceed 4 mins - restarting queues.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-03-02 22:15             ` Maxim Kozover
@ 2006-03-06 19:06               ` Andrew Vasquez
  0 siblings, 0 replies; 19+ messages in thread
From: Andrew Vasquez @ 2006-03-06 19:06 UTC (permalink / raw)
  To: Maxim Kozover
  Cc: Stefan Kaltenbrunner, Andrew Morton, linux-kernel, linux-scsi

On Fri, 03 Mar 2006, Maxim Kozover wrote:

> Please see the log with debug-patch.
> The module is loaded with option qlport_down_retry=1.
> Adapter 4 is connected to switch, adapter 5 doesn't have cable attached.
> After reconnecting the cable the disks don't reappear and rescan is stuck.
> Before applying your patches ghost rport was staying, now it's OK.

Before you try the patch I sent earlier, could you send be the output
from the following:

	# echo t > /proc/sysrq-trigger

Thanks,
Andrew Vasquez

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re: problems with scsi_transport_fc and qla2xxx
  2006-03-02 16:32         ` Maxim Kozover
  2006-03-02 17:38           ` Andrew Vasquez
@ 2006-03-06 20:00           ` Mike Snitzer
  2006-03-06 20:09             ` Maxim Kozover
  1 sibling, 1 reply; 19+ messages in thread
From: Mike Snitzer @ 2006-03-06 20:00 UTC (permalink / raw)
  To: Maxim Kozover
  Cc: Andrew Vasquez, Stefan Kaltenbrunner, Andrew Morton, linux-kernel,
	linux-scsi

On 3/2/06, Maxim Kozover <maximkoz@netvision.net.il> wrote:
> Hi Andrew!
> Today I tested disconnecting QLogic port.
> Adapter 4 is connected via switch to a storage and 3 LUNs are seen via
> the adapter.
> Only 1 rport is created (for FCP Target) while in Emulex case there
> were 3: (Fabric Port, Directory Server and FCP Target, FCP Initiator).
> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> FCP Target
>
> Default dev_loss_tmo is 6 (1+5) while in Emulex case the default was 35.
>
> After disconnecting the cable between the HBA and the switch
> qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
>  rport-4:0-0: blocked FC remote port time out: removing target and saving binding
>
> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> unknown
>
> Relevant scsi devices are removed from /proc/scsi/scsi.
>
> After reconnecting the cable
> qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
> qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
>
> # ls /sys/class/fc_remote_ports/
> rport-4:0-0
> # cat /sys/class/fc_remote_ports/*/roles
> FCP Target
>
> However, scsi devices don't reappear in /proc/scsi/scsi.
> When I issue rescan, the command is stuck
> echo - - - > /sys/class/scsi_host/host4/scan

Historically the qlogic driver rescan is a 2-phase process:
1) schedule the rescan, e.g.: echo scsi-qlascan > /proc/scsi/qla2xxx/4
2) rescan, e.g.: echo - - - > /sys/class/scsi_host/host4/scan

BUT, I've just used scsi-qlascan to discover _new_ devices... not
existing devices that experienced FC connection loss.  I assume the
qla driver _should_ just bring those lost devices back?  But does the
historic 2-phase rescan for new devices speak to why the qlogic driver
doesn't automagically bring the old devices back?  Or has the latest
qlogic driver in mainline advanced past this 2-phase requirement in
general?

regards,
Mike

Mike

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re: problems with scsi_transport_fc and qla2xxx
  2006-03-06 20:00           ` Mike Snitzer
@ 2006-03-06 20:09             ` Maxim Kozover
  2006-03-06 21:28               ` Andrew Vasquez
  0 siblings, 1 reply; 19+ messages in thread
From: Maxim Kozover @ 2006-03-06 20:09 UTC (permalink / raw)
  To: Mike Snitzer; +Cc: Andrew Vasquez, linux-kernel, linux-scsi

Hi Mike!
Unfortunately I don't have the directory /proc/scsi/qla2xxx.
However the target sees PRLI from the host again after reconnecting
the cable between the initiator and the switch.
Does it mean the rediscovering new devices on initiator side is
already done?

Thanks,

Maxim.

MS> Historically the qlogic driver rescan is a 2-phase process:
MS> 1) schedule the rescan, e.g.: echo scsi-qlascan > /proc/scsi/qla2xxx/4
MS> 2) rescan, e.g.: echo - - - > /sys/class/scsi_host/host4/scan

MS> BUT, I've just used scsi-qlascan to discover _new_ devices... not
MS> existing devices that experienced FC connection loss.  I assume the
MS> qla driver _should_ just bring those lost devices back?  But does the
MS> historic 2-phase rescan for new devices speak to why the qlogic driver
MS> doesn't automagically bring the old devices back?  Or has the latest
MS> qlogic driver in mainline advanced past this 2-phase requirement in
MS> general?

MS> regards,
MS> Mike

MS> Mike

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: problems with scsi_transport_fc and qla2xxx
  2006-03-06 20:09             ` Maxim Kozover
@ 2006-03-06 21:28               ` Andrew Vasquez
  2006-03-06 20:57                 ` Maxim Kozover
  2006-03-06 21:09                 ` Maxim Kozover
  0 siblings, 2 replies; 19+ messages in thread
From: Andrew Vasquez @ 2006-03-06 21:28 UTC (permalink / raw)
  To: Maxim Kozover; +Cc: Mike Snitzer, linux-kernel, linux-scsi

On Mon, 06 Mar 2006, Maxim Kozover wrote:

> Unfortunately I don't have the directory /proc/scsi/qla2xxx.

The two stage discovery process has not been needed since FC transport
integration.  Instead, the driver simply makes up-calls to signal
rport visiblity (add on PLOGI/PRLI; delete on LOGO/cable-pull/etc).

> However the target sees PRLI from the host again after reconnecting
> the cable between the initiator and the switch.
> Does it mean the rediscovering new devices on initiator side is
> already done?
> 

Yes, after plugging the cable back in, the driver rediscovers ports:

	Mar  3 01:07:22 multipath kernel: scsi(4): RSNN_NN exiting normally.
	Mar  3 01:07:22 multipath kernel: scsi(4): GID_PT entry - nn 200000e08b079a69 pn 210000e08b079a69 portid=010700.
	Mar  3 01:07:22 multipath kernel: scsi(4): GID_PT entry - nn 2000001738279c00 pn 1000001738279c11 portid=010200.
	Mar  3 01:07:22 multipath kernel: scsi(4): device wrap (010200)

Initiates PLOGI/PRLI: 

	Mar  3 01:07:22 multipath kernel: scsi(4): Trying Fabric Login w/loop id 0x0081 for port 010200.

And upcall via fc_remote_port_add() is done.

	Mar  3 01:07:22 multipath kernel: scsi(4): LOOP READY
	Mar  3 01:07:22 multipath kernel: scsi(4): qla2x00_loop_resync - end

Firmware then notifies software that the port has logged out:

	Mar  3 01:07:22 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0081/0007/7ee5.
	Mar  3 01:07:22 multipath kernel: scsi(4:0:0): status_entry: Port Down pid=43, compl status=0x29, port state=0x4

A CDB also returns with a completion status of PORT_LOGGED_OUT.  From
the driver's DPC routine (process-context), the upcall to
fc_remote_port_delete() is issued:

Driver attempts a relogin:

	Mar  3 01:07:22 multipath kernel: scsi(4): Port login retry: 1000001738279c11, id = 0x0081 retry cnt=8
	Mar  3 01:07:23 multipath kernel: scsi(4): fcport-0 - port retry count: 0 remaining
	Mar  3 01:07:23 multipath kernel: scsi(4): qla2x00_port_login()
	Mar  3 01:07:23 multipath kernel: scsi(4): Trying Fabric Login w/loop id 0x0081 for port 010200.

Relogin complete

	Mar  3 01:07:23 multipath kernel: scsi(4): port login OK: logged in ID 0x81

Upcall to fc_remote_port_add() done.

	Mar  3 01:07:23 multipath kernel: scsi(4): qla2x00_port_login - end
	Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0006/0001.
	Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0007/0001.
	Mar  3 01:07:23 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0000/0004/0001.
	Mar  3 01:07:24 multipath kernel: scsi(4): Asynchronous PORT UPDATE ignored 0081/0006/0001.

I also noticed that scsi_transport_fc.c::fc_user_scan() is not called
with the host_lock held... hmm..  could you try out the patch I sent
earlier and provide the results.

Also, could you send the "echo t > /proc/..." output after the cable
has been reinserted, but, before the 'echo "- - -" > /sys/class' scan
is initiated.

thanks,
av

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re: problems with scsi_transport_fc and qla2xxx
  2006-03-06 21:28               ` Andrew Vasquez
@ 2006-03-06 20:57                 ` Maxim Kozover
  2006-03-06 21:09                 ` Maxim Kozover
  1 sibling, 0 replies; 19+ messages in thread
From: Maxim Kozover @ 2006-03-06 20:57 UTC (permalink / raw)
  To: Andrew Vasquez; +Cc: Mike Snitzer, linux-kernel, linux-scsi

Hi Andrew!
AV> I also noticed that scsi_transport_fc.c::fc_user_scan() is not called
AV> with the host_lock held... hmm..  could you try out the patch I sent
AV> earlier and provide the results.

AV> Also, could you send the "echo t > /proc/..." output after the cable
AV> has been reinserted, but, before the 'echo "- - -" > /sys/class' scan
AV> is initiated.

AV> thanks,
AV> av

Here's sysrq output after reconnecting cable without manual disk
rescan. Before applying a patch.
The same lock exists:
#001:             [ffff81006ee20080] {scsi_host_alloc}
.. held by:         scsi_wq_4: 4255 [ffff81006f9147b0, 110]
... acquired at:               scsi_scan_target+0x51/0x87 [scsi_mod]

Thanks,

Maxim.

SysRq : Show State

                                                       sibling
  task                 PC          pid father child younger older
init          S ffff81007fdd3d78     0     1      0     2               (NOTLB)
ffff81007fdd3d78 0000000000000000 0000000000000282 0000000000000000
       ffff81007f6126d8 0000000000000000 0000000000000000 000000000000cc00
       ffff81007fdc4770 0000000000000ee6
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
migration/0   S ffff81007fdd7ec8     0     2      1             3       (L-TLB)
ffff81007fdd7ec8 000000017ead3830 ffff810003018b00 ffff81000301f800
       0000000000000002 0000000000000000 0000000000000000 0000000000000096
       ffff81007fdc4040 0000000000000d38
Call Trace: <ffffffff801255b3>{__wake_up_common+67}
       <ffffffff8012687c>{migration_thread+360} <ffffffff80126714>{migration_thread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
ksoftirqd/0   S ffff81007fddbf08     0     3      1             4     2 (L-TLB)
ffff81007fddbf08 ffff81007ebd1080 000000008031dafb 0000000000000000
       0000000000000000 0000000000000000 0000000000000000 000000000000c6b5
       ffff81007fdc67b0 00000000000000b9
Call Trace: <ffffffff801309a5>{ksoftirqd+0} <ffffffff801309dd>{ksoftirqd+56}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
watchdog/0    S ffff81007fdddea8     0     4      1             5     3 (L-TLB)
ffff81007fdddea8 ffffffff8038f000 0000000000000000 ffff81007fddde68
       0000000000000200 0000000000000000 0000000000000000 000000007fdddf08
       ffff81007fdc6080 0000000000000789
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8014cb3d>{watchdog+0}
       <ffffffff8031e54a>{schedule_timeout+138} <ffffffff8013435d>{process_timeout+0}
       <ffffffff80134645>{msleep_interruptible+46} <ffffffff8014cb8c>{watchdog+79}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
migration/1   S ffff81007fddfec8     0     5      1             6     4 (L-TLB)
ffff81007fddfec8 000000007a26c080 ffff810003020b00 ffff810003017800
       0000000000000002 0000000000000000 0000000000000001 0000000100000096
       ffff81007fdc77f0 00000000000010bc
Call Trace: <ffffffff801255b3>{__wake_up_common+67}
       <ffffffff8012687c>{migration_thread+360} <ffffffff80126714>{migration_thread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
ksoftirqd/1   S ffff810003363f08     0     6      1             7     5 (L-TLB)
ffff810003363f08 ffff81007ed8e7f0 000000008031dafb 0000000000000000
       0000000000000000 0000000100000000 0000000000000001 00000001000014e5
       ffff81007fdc70c0 0000000000000075
Call Trace: <ffffffff801309a5>{ksoftirqd+0} <ffffffff801309dd>{ksoftirqd+56}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
watchdog/1    S ffff81007fd85ea8     0     7      1             8     6 (L-TLB)
ffff81007fd85ea8 ffff810003364100 0000000000000000 ffff81007fd85e68
       0000000000000200 0000000000000000 0000000000000001 000000017fd85f08
       ffff810003364830 000000000000008d
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8014cb3d>{watchdog+0}
       <ffffffff8031e54a>{schedule_timeout+138} <ffffffff8013435d>{process_timeout+0}
       <ffffffff80134645>{msleep_interruptible+46} <ffffffff8014cb8c>{watchdog+79}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
events/0      S ffff810037f3de98     0     8      1             9     7 (L-TLB)
ffff810037f3de98 0000000000000286 ffffffff801339a1 0000000000000286
       0000000000000286 0000000300000000 0000000000000000 0000000000000000
       ffff810037fef870 0000000000002478
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
events/1      S ffff810037e57e98     0     9      1            10     8 (L-TLB)
ffff810037e57e98 0000000000000246 0000000000000246 ffff81007e992800
       000000007e9929e8 0000000000000000 ffff810037e46520 0000000100000000
       ffff810037fef140 0000000000000134
Call Trace: <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ae41>{worker_thread+211} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
khelper       S ffff810037f39e98     0    10      1            11     9 (L-TLB)
ffff810037f39e98 ffff81006ee95b80 0000000000000001 ffff81006ee95b98
       0000000000000000 0000000000000000 ffff810037e46420 0000000000000000
       ffff810037fee770 00000000000001c2
Call Trace: <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ae41>{worker_thread+211} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kthread       S ffff810037c2de98     0    11      1    15     181    10 (L-TLB)
ffff810037c2de98 ffff81006fbe5b68 0000000000000001 ffff81006fbe5b80
       0000000000000000 0000000300000000 0000000000000001 0000000100000000
       ffff810037fee040 0000000000000151
Call Trace: <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ae41>{worker_thread+211} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kblockd/0     S ffff810037dc3e98     0    15     11            16       (L-TLB)
ffff810037dc3e98 ffff81007f99c000 ffff81007f618ce8 ffff81007f618ce8
       ffffffff801dd758 0000000300000000 0000000000000000 0000000000000000
       ffff81007f580830 0000000000001488
Call Trace: <ffffffff801dd758>{elv_next_request+62}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kblockd/1     S ffff810037dc7e98     0    16     11            17    15 (L-TLB)
ffff810037dc7e98 ffff81007f99c000 ffff81007f618ce8 ffff81007f618ce8
       ffffffff801dd758 0000000300000000 0000000000000001 0000000100000000
       ffff81007f580100 000000000000142c
Call Trace: <ffffffff801dd758>{elv_next_request+62}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kacpid        S ffff810037de7e98     0    17     11           115    16 (L-TLB)
ffff810037de7e98 0000000000000000 0000000000000006 00000000000006cc
       0000000000000000 0000000000000000 0000000000000001 0000000137de7e28
       ffff810003369870 00000000000001cc
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
khubd         S ffff810037dede28     0   115     11           179    17 (L-TLB)
ffff810037dede28 0000000000000001 800001807f5107b0 ffff81007e4af800
       0000000000100100 0000000000000000 00000000ffffffff 00000001ffffffff
       ffff81007f5107b0 00000000000006a4
Call Trace: <ffffffff80287ff6>{hub_port_status+83} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8028921a>{hub_thread+0} <ffffffff80289d06>{hub_thread+2796}
       <ffffffff8031dafb>{thread_return+100} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8028921a>{hub_thread+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
pdflush       S ffff81007f773ec8     0   179     11           180   115 (L-TLB)
ffff81007f773ec8 ffff81007f773e28 ffff810037fee040 0000000000000000
       000000007f505870 0000000000000000 0000000000000292 000000010301f800
       ffff81007f505870 00000000000000d2
Call Trace: <ffffffff80125835>{set_user_nice+261} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80153f7f>{pdflush+220} <ffffffff80153ea3>{pdflush+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
pdflush       S ffff81007f76fec8     0   180     11           182   179 (L-TLB)
ffff81007f76fec8 ffffffff8039aaa0 ffffffff80133a6a 0000000000000286
       00000000000035c8 0000000000000000 ffff81007fdd3dc8 00000001fffffffc
       ffff81007f517140 0000000000000652
Call Trace: <ffffffff80133a6a>{__mod_timer+168} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80153f7f>{pdflush+220} <ffffffff80153ea3>{pdflush+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
aio/0         S ffff81007f659e98     0   182     11           183   180 (L-TLB)
ffff81007f659e98 ffff81007f51a9c0 0000000000000009 00000000000009ba
       0000000000000002 ffff81007fdd4088 0000000000000000 000000007f659e28
       ffff81007f7240c0 0000000000000239
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kswapd0       S ffff81007f4fdeb8     0   181      1           431    11 (L-TLB)
ffff81007f4fdeb8 ffff81007f4fddf8 0000000000000001 0000000000000001
       0000000000000000 0000000000000000 0000000000000296 000000010301f800
       ffff81007f540080 00000000000002fa
Call Trace: <ffffffff801363fb>{do_notify_parent+404}
       <ffffffff8015769b>{kswapd+247} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8012e510>{do_exit+2116} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8010b846>{child_rip+8} <ffffffff801575a4>{kswapd+0}
       <ffffffff8010b83e>{child_rip+0}
aio/1         S ffff81007f6f9e98     0   183     11           258   182 (L-TLB)
ffff81007f6f9e98 ffff81007f6cd660 0000000000000009 00000000000009ba
       0000000000000002 ffff81007fdd4088 0000000000000001 000000017f6f9e28
       ffff81007f687100 000000000000015c
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kseriod       S ffff81007f6dbe88     0   258     11           316   183 (L-TLB)
ffff81007f6dbe88 ffff81007f78cda0 ffffffff8031ed71 0000000000000000
       0000000000000000 0000000000008080 0000000000000000 00000000803ce7f0
       ffff81007f682770 000000000000062d
Call Trace: <ffffffff8031ed71>{__mutex_unlock_slowpath+476}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8025133b>{serio_thread+664}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff802510a3>{serio_thread+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
kpsmoused     S ffff81007f7e3e98     0   316     11           337   258 (L-TLB)
ffff81007f7e3e98 ffff81007ff59ba0 0000000000000009 00000000000009ba
       0000000000000002 ffff81007fdd4088 0000000000000000 000000007f7e3e28
       ffff81007f564040 000000000000024c
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
ata/0         S ffff81007ec65e98     0   337     11           338   316 (L-TLB)
ffff81007ec65e98 0000000000000000 0000000000000006 00000000000006cc
       0000000000000000 0000000000000000 0000000000000000 000000007ec65e28
       ffff81007f64e7b0 00000000000002ed
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
ata/1         S ffff81007ec67e98     0   338     11           344   337 (L-TLB)
ffff81007ec67e98 ffffffff8031f917 0000000000000010 0000000000000246
       000000007ec67df8 0000000000000000 ffff81007f6b7ae0 0000000100000000
       ffff81007f682040 00000000000051f3
Call Trace: <ffffffff8031f917>{_spin_unlock_irqrestore+11}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_eh_0     S ffff81007ec73e98     0   344     11           345   338 (L-TLB)
ffff81007ec73e98 0000000000000000 0000000000000006 00000000000006cc
       0000000000000000 0000000000000000 0000000000000001 000000017ec73e28
       ffff81007f64e080 00000000000000c6
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031dafb>{thread_return+100}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0}
       <ffffffff88005e10>{:scsi_mod:scsi_error_handler+100}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_eh_1     S ffff81007ec13e98     0   345     11           382   344 (L-TLB)
ffff81007ec13e98 0000000100000001 0000000000000006 00000000000006cc
       0000000000000001 0000000000000001 0000000000000001 000000017ec13e28
       ffff81007f60e7b0 00000000000000ba
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff88005e10>{:scsi_mod:scsi_error_handler+100}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_eh_3     S ffff81007ec1fe98     0   382     11           385   345 (L-TLB)
ffff81007ec1fe98 0000000000000000 0000000000000009 00000000000009ba
       00000000732f0000 0000000000000000 0000000000000001 000000017ec1fe28
       ffff81007f7247f0 00000000000000a3
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031dafb>{thread_return+100}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0}
       <ffffffff88005e10>{:scsi_mod:scsi_error_handler+100}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_wq_3     S ffff81007f58fe98     0   385     11           417   382 (L-TLB)
ffff81007f58fe98 2222222222222222 2222222222222222 2222222222222222
       2222222222222222 0000000300000000 0000000000000001 0000000100000000
       ffff81007f7dc870 0000000000000843
Call Trace: <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kmirrord      S ffff81007ef15e98     0   417     11          1935   385 (L-TLB)
ffff81007ef15e98 0000000000000011 ffff81007ef15eb8 ffff81007ef0e320
       ffff81007edc7870 0000000000000282 0000000000000001 0000000100000100
       ffff81007edc7870 00000000000004d0
Call Trace: <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kjournald     S ffff81007ede1e88     0   431      1          1358   181 (L-TLB)
ffff81007ede1e88 ffff81007edc95f8 ffffffff88087585 00000fdc00000000
       000000006ebc6024 0000000000000000 ffff81007f62a000 000000017278ddf0
       ffff81007ed8e7f0 00000000000001b6
Call Trace: <ffffffff88087585>{:jbd:journal_commit_transaction+4225}
       <ffffffff88089467>{:jbd:kjournald+466} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff8012e510>{do_exit+2116}
       <ffffffff88089290>{:jbd:commit_timeout+0} <ffffffff8010b846>{child_rip+8}
       <ffffffff88089295>{:jbd:kjournald+0} <ffffffff8010b83e>{child_rip+0}
udevd         S ffff81007da91d78     0  1358      1          2104   431 (NOTLB)
ffff81007da91d78 0000000000000206 0000000000000206 0000000000000001
       0000004400000001 ffff81000000cc00 0000000000000001 0000000100000286
       ffff81007edc8100 0000000000001f20
Call Trace: <ffffffff80151fd1>{__alloc_pages+102} <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff802bc43c>{datagram_poll+42}
       <ffffffff802bc412>{datagram_poll+0} <ffffffff8018189c>{do_select+944}
       <ffffffff801a5313>{proc_delete_inode+0} <ffffffff80181431>{__pollwait+0}
       <ffffffff8031f883>{_spin_lock_irqsave+11} <ffffffff80181b7a>{sys_select+651}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8016fddd>{sys_read+69}
       <ffffffff8010a70a>{system_call+126}
kauditd       S ffff81007d5f7ea8     0  1935     11          2043   417 (L-TLB)
ffff81007d5f7ea8 00000000000009ba 0000000000000002 ffff81007f70e908
       0000000000000001 ffff81007d5f7e28 0000000000000000 0000000000000000
       ffff81007e562140 0000000000000140
Call Trace: <ffffffff80149c98>{kauditd_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80149c98>{kauditd_thread+0} <ffffffff80149dd7>{kauditd_thread+319}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
kmpathd/0     S ffff81007d031e98     0  2043     11          2044  1935 (L-TLB)
ffff81007d031e98 2d18130000000006 0000000000000008 00000000000008c0
       00000000000000db 0000000000000000 0000000000000001 000000007d031e28
       ffff81007e58d7b0 000000000000011c
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031dafb>{thread_return+100}
       <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ae41>{worker_thread+211} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
kmpathd/1     S ffff81007d027e98     0  2044     11          4224  2043 (L-TLB)
ffff81007d027e98 0000000000000000 0000000000000008 00000000000008c0
       000000007da3a890 0000000000000000 0000000000000001 000000017d027e28
       ffff81007ead57f0 00000000000001b8
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031dafb>{thread_return+100}
       <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013ae41>{worker_thread+211} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ddeb>{kthread+200}
       <ffffffff8010b846>{child_rip+8} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013dd23>{kthread+0} <ffffffff8010b83e>{child_rip+0}
kjournald     S ffff81007a0a9e88     0  2104      1          2716  1358 (L-TLB)
ffff81007a0a9e88 0000003000000010 ffff81007a0a9e98 ffff81007a0a9dd8
       0000000079fcfb28 0000000000000000 0000000000000005 0000000100000000
       ffff81007e9de080 0000000000000922
Call Trace: <ffffffff88089467>{:jbd:kjournald+466} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff88089290>{:jbd:commit_timeout+0} <ffffffff8010b846>{child_rip+8}
       <ffffffff88089295>{:jbd:kjournald+0} <ffffffff8010b83e>{child_rip+0}
dhclient      S ffff810079229d78     0  2716      1          2760  2104 (NOTLB)
ffff810079229d78 0000000000000000 0000004400000001 ffff81000000cc00
       000200d00000015c 0000000000000282 0000000000000000 00000000000000d0
       ffff81007a0d9040 0000000000001125
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff802bc412>{datagram_poll+0}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff80181b7a>{sys_select+651} <ffffffff8010a70a>{system_call+126}
syslogd       R  running task       0  2760      1          2764  2716 (NOTLB)
klogd         S ffff81007906fbd8     0  2764      1          2775  2760 (NOTLB)
ffff81007906fbd8 ffff81007edc8830 ffff810037fdac80 ffff810037c1e690
       0000000078715000 000000005b169863 0000000000000000 000000007906fbf8
       ffff81007ebd1080 00000000000004c0
Call Trace: <ffffffff80123eda>{try_to_wake_up+1035}
       <ffffffff8031e4de>{schedule_timeout+30} <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff8013e2f6>{prepare_to_wait_exclusive+21} <ffffffff803177cf>{unix_wait_for_peer+163}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff80125609>{__wake_up+54}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff803182c8>{unix_dgram_sendmsg+950}
       <ffffffff802b5561>{do_sock_write+196} <ffffffff802b56b1>{sock_aio_write+79}
       <ffffffff8031dafb>{thread_return+100} <ffffffff8016fbe9>{do_sync_write+201}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8016fd0d>{vfs_write+231} <ffffffff8016fe4b>{sys_write+69}
       <ffffffff8010a70a>{system_call+126}
irqbalance    S ffff8100791f1e98     0  2775      1          2787  2764 (NOTLB)
ffff8100791f1e98 00000002540be400 ffffffff8013031b 00000000440caf80
       000000000005b900 0000000000019a0c 0000000000000001 00000001bbf35151
       ffff81007e484040 0000000000011881
Call Trace: <ffffffff8013031b>{getnstimeofday+16} <ffffffff8014059e>{enqueue_hrtimer+93}
       <ffffffff801406ac>{hrtimer_start+195} <ffffffff8031f1fd>{schedule_hrtimer+36}
       <ffffffff80140933>{hrtimer_nanosleep+91} <ffffffff80137877>{do_sigaction+568}
       <ffffffff80136d74>{sigprocmask+216} <ffffffff80140a2d>{sys_nanosleep+85}
       <ffffffff8010a70a>{system_call+126}
portmap       S ffff810079063e78     0  2787      1          2807  2775 (NOTLB)
ffff810079063e78 0000000000000001 0000004400000001 ffff81000000cc00
       00000000802b4b33 0000000000000000 ffff81000000cc00 00000001000000d0
       ffff81007eece0c0 000000000000098b
Call Trace: <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff80181f2c>{do_sys_poll+610}
       <ffffffff80181431>{__pollwait+0} <ffffffff80182049>{sys_poll+74}
       <ffffffff8010a70a>{system_call+126}
rpc.statd     S ffff81007958bd78     0  2807      1          2846  2787 (NOTLB)
ffff81007958bd78 0000000000000001 0000004400000001 ffff81000000cc00
       000200d0802b516e 0000000000000286 0000000000000001 00000001000000d0
       ffff81007e4f0100 00000000000019d8
Call Trace: <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff802e4789>{tcp_poll+42}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff8013e451>{bit_waitqueue+53} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
rpc.idmapd    S ffff810078d43e88     0  2846      1          2937  2807 (NOTLB)
ffff810078d43e88 ffff81007e5300c0 ffffffff80160d90 ffff810002e3bf90
       000000008015b7f2 0000000000000000 0000000000000001 0000000100000246
       ffff81007ec83870 0000000000000816
Call Trace: <ffffffff80160d90>{page_add_file_rmap+36}
       <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff801953e9>{sys_epoll_wait+400}
       <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8010a70a>{system_call+126}
acpid         S ffff810078849d78     0  2937      1          2968  2846 (NOTLB)
ffff810078849d78 ffff810078849e88 ffffffff8014fef4 0000000000000000
       0000000000000286 0000000000000001 0000000000000001 0000000100000001
       ffff81007ec71100 0000000000005e59
Call Trace: <ffffffff8014fef4>{__generic_file_aio_write_nolock+873}
       <ffffffff80151fd1>{__alloc_pages+102} <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8031f883>{_spin_lock_irqsave+11} <ffffffff8013e1d6>{add_wait_queue+18}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff80181b7a>{sys_select+651} <ffffffff8016fe4b>{sys_write+69}
       <ffffffff8010a70a>{system_call+126}
cupsd         S ffff810078a9bd78     0  2968      1          3073  2937 (NOTLB)
ffff810078a9bd78 0000000000000000 0000000000000282 0000000000000014
       ffffffff802b4ff1 0000000000000000 0000000000000000 000000000000cc00
       ffff81007e58d080 0000000000000865
Call Trace: <ffffffff802b4ff1>{sock_sendmsg+265} <ffffffff801339a1>{lock_timer_base+27}
       <ffffffff8031e54a>{schedule_timeout+138} <ffffffff8013435d>{process_timeout+0}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff80181b7a>{sys_select+651} <ffffffff8010a70a>{system_call+126}
sshd          S ffff8100761bbd78     0  3073      1          3094  2968 (NOTLB)
ffff8100761bbd78 0000000000000000 0000000000000286 0000000000000005
       ffff81007da0f480 0000000000000001 0000000000000001 000000010000cc00
       ffff81007edc7140 0000000000003c68
Call Trace: <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff802e4789>{tcp_poll+42}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff80181b7a>{sys_select+651} <ffffffff8010a70a>{system_call+126}
xinetd        S ffff81007626dd78     0  3094      1          3113  3073 (NOTLB)
ffff81007626dd78 0000000000000000 0000000000000282 0000000000000046
       00000000802b4ff1 0000000000000000 0000004400000001 000000000000cc00
       ffff81007ed1b7b0 00000000000007c9
Call Trace: <ffffffff80151fd1>{__alloc_pages+102} <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff802e4789>{tcp_poll+42}
       <ffffffff8018189c>{do_select+944} <ffffffff80181431>{__pollwait+0}
       <ffffffff80181b7a>{sys_select+651} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8010a70a>{system_call+126}
sendmail      S ffff8100769b5d78     0  3113      1          3121  3094 (NOTLB)
ffff8100769b5d78 0000000000000000 0000000000000286 0000000000000033
       000000006ed12014 0000000000000000 0000004400000001 000000010000cc00
       ffff81007e5300c0 0000000000002cfd
Call Trace: <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
sendmail      S ffff810076c6ff68     0  3121      1          3170  3113 (NOTLB)
ffff810076c6ff68 0000000000000000 0000000000030002 0000000000000000
       00000000440caeea 0000000000000000 0000000000000001 00000001798e6698
       ffff81007e614870 000000000000fcc1
Call Trace: <ffffffff80137caa>{sys_pause+23} <ffffffff8010a70a>{system_call+126}
gpm           S ffff8100771b1d78     0  3170      1          3229  3121 (NOTLB)
ffff8100771b1d78 ffff81007232c830 ffff81007f763568 0000000000000000
       0000000000000282 0000000000000000 0000000000000000 0000000000000000
       ffff81007f53c040 0000000000000149
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff802b4d59>{sock_map_fd+305} <ffffffff8010a70a>{system_call+126}
htt           S ffff810076f63ea8     0  3229      1  3230    3242  3170 (NOTLB)
ffff810076f63ea8 0000000000000000 ffff81000337b0c0 0000000001200011
       0000000000000000 0000000000000000 ffffffff8013b8c1 0000000076aae010
       ffff81000337b0c0 00000000000024e9
Call Trace: <ffffffff8013b8c1>{attach_pid+28} <ffffffff801ed478>{__up_read+16}
       <ffffffff8032163c>{do_page_fault+1000} <ffffffff8012f1ef>{do_wait+2585}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8010a70a>{system_call+126}
htt_server    S ffff810077227e78     0  3230   3229                     (NOTLB)
ffff810077227e78 0000000000000000 0000000000000000 0000000000000000
       0000000000000282 0000000000000000 ffffffff8015b7f2 0000000000000000
       ffff81007e310140 0000000000000f32
Call Trace: <ffffffff8015b7f2>{__handle_mm_fault+1372}
       <ffffffff8031f883>{_spin_lock_irqsave+11} <ffffffff801339a1>{lock_timer_base+27}
       <ffffffff8031e54a>{schedule_timeout+138} <ffffffff8013435d>{process_timeout+0}
       <ffffffff80181f2c>{do_sys_poll+610} <ffffffff80181431>{__pollwait+0}
       <ffffffff80182049>{sys_poll+74} <ffffffff8010a70a>{system_call+126}
cannaserver   S ffff810076733d78     0  3242      1          3254  3229 (NOTLB)
ffff810076733d78 ffffffff8038f000 0000000000000000 0000000000000000
       0000000000000282 ffff810076733d88 0000000000000000 0000000000000000
       ffff81007e739100 0000000000000633
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
crond         S ffff810073dbfe98     0  3254      1          3295  3242 (NOTLB)
ffff810073dbfe98 0000000df8475800 ffffffff8013031b 00000000440caf79
       00000000000d47d4 0000000000018d58 0000000000000000 00000000bbf35151
       ffff81007e6997f0 000000000000130f
Call Trace: <ffffffff8013031b>{getnstimeofday+16} <ffffffff8014059e>{enqueue_hrtimer+93}
       <ffffffff801406ac>{hrtimer_start+195} <ffffffff8031f1fd>{schedule_hrtimer+36}
       <ffffffff80140933>{hrtimer_nanosleep+91} <ffffffff80137877>{do_sigaction+568}
       <ffffffff80136d74>{sigprocmask+216} <ffffffff80140a2d>{sys_nanosleep+85}
       <ffffffff8010a70a>{system_call+126}
xfs           S ffff810073641d78     0  3295      1          3314  3254 (NOTLB)
ffff810073641d78 ffff810003364100 0000000000000001 0000000000000000
       0000000000000282 0000000000000000 0000000000000001 0000000100000000
       ffff81007e5247f0 00000000000003d1
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
atd           S ffff8100737f1e98     0  3314      1          3333  3295 (NOTLB)
ffff8100737f1e98 00000045d964b800 ffffffff8013031b 00000000440caeed
       0000000000077199 0000000000007b32 0000000000000000 00000000bbf35151
       ffff81007f60e080 00000000000036c3
Call Trace: <ffffffff8013031b>{getnstimeofday+16} <ffffffff8014059e>{enqueue_hrtimer+93}
       <ffffffff801406ac>{hrtimer_start+195} <ffffffff8031f1fd>{schedule_hrtimer+36}
       <ffffffff80140933>{hrtimer_nanosleep+91} <ffffffff80137877>{do_sigaction+568}
       <ffffffff80136d74>{sigprocmask+216} <ffffffff80140a2d>{sys_nanosleep+85}
       <ffffffff8010a70a>{system_call+126}
dbus-daemon-1 S ffff810072a7de78     0  3333      1          3343  3314 (NOTLB)
ffff810072a7de78 ffff810000000000 ffff810072a7ddd8 0000000000000000
       0000000000000286 0000000000000000 0000000000000001 0000000100000001
       ffff810079e48040 00000000000013d2
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff80181f2c>{do_sys_poll+610}
       <ffffffff80181431>{__pollwait+0} <ffffffff80182049>{sys_poll+74}
       <ffffffff8010a70a>{system_call+126}
cups-config-d S ffff810072b33e78     0  3343      1          3363  3333 (NOTLB)
ffff810072b33e78 0000000000000000 0000000000000000 0000000000000000
       0000000000000282 ffff810002cbf750 0000000000000000 0000000000000000
       ffff81007e549140 00000000000045c9
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff80181f2c>{do_sys_poll+610}
       <ffffffff80181431>{__pollwait+0} <ffffffff80182049>{sys_poll+74}
       <ffffffff8010a70a>{system_call+126}
login         S ffff810072bc5ea8     0  3363      1  3967    3364  3343 (NOTLB)
ffff810072bc5ea8 0000000000000000 ffff81007a26c7b0 0000000001200011
       0000000000000000 ffff810072bc5e18 0000000000000000 000000007ede2ff8
       ffff81007a26c7b0 0000000000000b53
Call Trace: <ffffffff801ed478>{__up_read+16} <ffffffff8032163c>{do_page_fault+1000}
       <ffffffff8012f1ef>{do_wait+2585} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80137b93>{sys_rt_sigaction+97} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8010a70a>{system_call+126}
mingetty      S ffff8100726bfda8     0  3364      1          3365  3363 (NOTLB)
ffff8100726bfda8 0000000000000020 0000000000000004 0000000000000410
       0000000000000246 0000000000000000 0000000000000001 00000000726bfd38
       ffff81007e549870 000000000001a64c
Call Trace: <ffffffff80123eda>{try_to_wake_up+1035}
       <ffffffff8031e4de>{schedule_timeout+30} <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff80236612>{read_chan+990}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8031f47f>{__down_failed+53}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80231317>{tty_read+136}
       <ffffffff8016fa82>{vfs_read+209} <ffffffff8016fddd>{sys_read+69}
       <ffffffff8010a70a>{system_call+126}
mingetty      S ffff810072babda8     0  3365      1          3368  3364 (NOTLB)
ffff810072babda8 0000000000000020 ffff81007e2e1800 0000000000000246
       0000000000000246 0000000000000000 ffffffff8012bce6 0000000100000246
       ffff81007e0eb770 00000000000054ec
Call Trace: <ffffffff8012bce6>{release_console_sem+377}
       <ffffffff8031e4de>{schedule_timeout+30} <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff80236612>{read_chan+990}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80230fce>{tty_ldisc_deref+101} <ffffffff80231317>{tty_read+136}
       <ffffffff8016fa82>{vfs_read+209} <ffffffff8016fddd>{sys_read+69}
       <ffffffff8010a70a>{system_call+126}
mingetty      S ffff810072badda8     0  3368      1          3369  3365 (NOTLB)
ffff810072badda8 0000000000000020 ffff81007e2e1000 0000000000000246
       0000000000000246 0000000000000000 ffffffff8012bce6 0000000100000246
       ffff810079df1770 0000000000008895
Call Trace: <ffffffff8012bce6>{release_console_sem+377}
       <ffffffff8031e4de>{schedule_timeout+30} <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff80236612>{read_chan+990}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80230fce>{tty_ldisc_deref+101} <ffffffff80231317>{tty_read+136}
       <ffffffff8016fa82>{vfs_read+209} <ffffffff8016fddd>{sys_read+69}
       <ffffffff8010a70a>{system_call+126}
mingetty      S ffff810072bbbda8     0  3369      1          3371  3368 (NOTLB)
ffff810072bbbda8 0000000000000020 ffff81007e288c00 0000000000000246
       0000000000000246 0000000000000000 ffffffff8012bce6 0000000100000246
       ffff810079c90100 0000000000008342
Call Trace: <ffffffff8012bce6>{release_console_sem+377}
       <ffffffff8031e4de>{schedule_timeout+30} <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff8013e1d6>{add_wait_queue+18} <ffffffff80236612>{read_chan+990}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80230fce>{tty_ldisc_deref+101} <ffffffff80231317>{tty_read+136}
       <ffffffff8016fa82>{vfs_read+209} <ffffffff8016fddd>{sys_read+69}
       <ffffffff8010a70a>{system_call+126}
mingetty      S ffff810072bc7da8     0  3371      1          3581  3369 (NOTLB)
ffff810072bc7da8 0000000000000020 ffff81007e288000 0000000000000246
       0000000000000246 0000000000004e7d 0000000000000000 0000000000000246
       ffff81007eece7f0 0000000000004fa0
Call Trace: <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8031f883>{_spin_lock_irqsave+11} <ffffffff8013e1d6>{add_wait_queue+18}
       <ffffffff80236612>{read_chan+990} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80230fce>{tty_ldisc_deref+101}
       <ffffffff80231317>{tty_read+136} <ffffffff8016fa82>{vfs_read+209}
       <ffffffff8016fddd>{sys_read+69} <ffffffff8010a70a>{system_call+126}
mingetty      S ffff8100721e7da8     0  3581      1          3584  3371 (NOTLB)
ffff8100721e7da8 0000000000000020 ffff81007e49fa00 0000000000000246
       0000000000000246 0000000000004e7d 0000000000000000 0000000000000246
       ffff81007ead3100 000000000000be06
Call Trace: <ffffffff8031e4de>{schedule_timeout+30}
       <ffffffff8031f883>{_spin_lock_irqsave+11} <ffffffff8013e1d6>{add_wait_queue+18}
       <ffffffff80236612>{read_chan+990} <ffffffff80125562>{default_wake_function+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80230fce>{tty_ldisc_deref+101}
       <ffffffff80231317>{tty_read+136} <ffffffff8016fa82>{vfs_read+209}
       <ffffffff8016fddd>{sys_read+69} <ffffffff8010a70a>{system_call+126}
gdm-binary    S ffff810072337e78     0  3584      1  3845    4253  3581 (NOTLB)
ffff810072337e78 0000000000000000 0000000000000282 0000000000000001
       0000000003017800 0000000000000000 0000004400000001 000000000000cc00
       ffff8100721bb830 0000000000000518
Call Trace: <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff80181f2c>{do_sys_poll+610}
       <ffffffff80181431>{__pollwait+0} <ffffffff80182049>{sys_poll+74}
       <ffffffff8010a70a>{system_call+126}
gdm-binary    S ffff81007215dde8     0  3845   3584  3846               (NOTLB)
ffff81007215dde8 000000003ad45f4e 0000000000000020 0000000000000000
       0000000000000009 000000000000099c 0000000000000001 0000000171bcc0f8
       ffff81007ede3040 0000000000000519
Call Trace: <ffffffff8017ad22>{pipe_wait+118} <ffffffff8013e39e>{autoremove_wake_function+0}
       <ffffffff8013e39e>{autoremove_wake_function+0} <ffffffff8017b086>{pipe_readv+658}
       <ffffffff8017b130>{pipe_read+26} <ffffffff8016fa82>{vfs_read+209}
       <ffffffff8016fddd>{sys_read+69} <ffffffff8010a70a>{system_call+126}
X             S ffff81007206bd78     0  3846   3845          3966       (NOTLB)
ffff81007206bd78 ffffffff8038f000 ffff81007206bdc8 0000000000000000
       0000000000000282 0000000000000000 0000000000000000 000000007ea25800
       ffff81007232c830 000000000000a27d
Call Trace: <ffffffff8031f883>{_spin_lock_irqsave+11}
       <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff8018189c>{do_select+944}
       <ffffffff80181431>{__pollwait+0} <ffffffff80181b7a>{sys_select+651}
       <ffffffff8010a70a>{system_call+126}
gdmgreeter    S ffff810070303e78     0  3966   3845                3846 (NOTLB)
ffff810070303e78 ffff81007232c830 ffff810070303dd8 0000000000000000
       0000000000000282 0000000000000000 0000000000000000 0000000000000000
       ffff810079894870 000000000000015a
Call Trace: <ffffffff801339a1>{lock_timer_base+27} <ffffffff8031e54a>{schedule_timeout+138}
       <ffffffff8013435d>{process_timeout+0} <ffffffff80181f2c>{do_sys_poll+610}
       <ffffffff80181431>{__pollwait+0} <ffffffff80182049>{sys_poll+74}
       <ffffffff8010a70a>{system_call+126}
bash          R  running task       0  3967   3363                     (NOTLB)
scsi_eh_4     S ffff81006ee2be98     0  4224     11          4255  2044 (L-TLB)
ffff81006ee2be98 0000000000000000 0000000000000008 00000000000008c0
       0000000000000000 0000000000000000 0000000000000000 000000006ee2be28
       ffff81007f5407b0 0000000000000121
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff88005e10>{:scsi_mod:scsi_error_handler+100}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
qla2xxx_4_dpc S ffff81007f4ede18     0  4253      1          4408  3584 (L-TLB)
ffff81007f4ede18 0000000000004000 ffff81007f985a00 0000000000000000
       0000000000200200 00000000ffffdb57 0000000000000000 000000006ee234e8
       ffff81006fbf9830 00000000000a624e
Call Trace: <ffffffff8031f7b7>{__down_interruptible+203}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013ab66>{__queue_work+78}
       <ffffffff8031f4b9>{__down_failed_interruptible+53} <ffffffff881b952c>{:qla2xxx:.text.lock.qla_os+15}
       <ffffffff8010b846>{child_rip+8} <ffffffff881b8cb8>{:qla2xxx:qla2x00_do_dpc+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_wq_4     D ffff81006ee95a88     0  4255     11          4264  4224 (L-TLB)
ffff81006ee95a88 0001122000011220 0000000000011220 ffff81007628a780
       ffff81007f985a00 ffff81007f9d8580 0000000000000000 0000000000000246
       ffff81006f9147b0 0000000000001045
Call Trace: <ffffffff8031dc2a>{wait_for_completion+165}
       <ffffffff80125562>{default_wake_function+0} <ffffffff80125562>{default_wake_function+0}
       <ffffffff801e025d>{blk_execute_rq+242} <ffffffff80174151>{bio_alloc_bioset+198}
       <ffffffff801deb05>{blk_recount_segments+124} <ffffffff88006cf9>{:scsi_mod:scsi_execute+216}
       <ffffffff88006d96>{:scsi_mod:scsi_execute_req+120} <ffffffff88009411>{:scsi_mod:scsi_probe_and_add_lun+523}
       <ffffffff88009160>{:scsi_mod:scsi_alloc_target+512}
       <ffffffff88009d1d>{:scsi_mod:__scsi_scan_target+196}
       <ffffffff8031ea88>{__mutex_lock_slowpath+768} <ffffffff8800a2ea>{:scsi_mod:scsi_scan_target+111}
       <ffffffff88046aac>{:scsi_transport_fc:fc_scsi_scan_rport+0}
       <ffffffff8013ad24>{run_workqueue+161} <ffffffff8013ad6e>{worker_thread+0}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013ae73>{worker_thread+261}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_eh_5     S ffff81006eef5e98     0  4264     11          4410  4255 (L-TLB)
ffff81006eef5e98 0000000000000005 0000000000000009 00000000000009ba
       0000000000000002 0000000000000000 0000000000000001 000000016eef5e28
       ffff81007e739830 000000000000012c
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031dafb>{thread_return+100}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0}
       <ffffffff88005e10>{:scsi_mod:scsi_error_handler+100}
       <ffffffff801255b3>{__wake_up_common+67} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff88005dac>{:scsi_mod:scsi_error_handler+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}
qla2xxx_5_dpc S ffff81006fa1fe18     0  4408      1                4253 (L-TLB)
ffff81006fa1fe18 ffffffff8038f000 0000000000000000 000000000000004c
       00000000722a8980 0000000000000000 ffff8100722a8980 0000000000000000
       ffff81007e5240c0 0000000000000182
Call Trace: <ffffffff80123600>{activate_task+140} <ffffffff80123eda>{try_to_wake_up+1035}
       <ffffffff8031f7b7>{__down_interruptible+203} <ffffffff80125562>{default_wake_function+0}
       <ffffffff8031f4b9>{__down_failed_interruptible+53} <ffffffff881b952c>{:qla2xxx:.text.lock.qla_os+15}
       <ffffffff8010b846>{child_rip+8} <ffffffff881b8cb8>{:qla2xxx:qla2x00_do_dpc+0}
       <ffffffff8010b83e>{child_rip+0}
scsi_wq_5     S ffff81006fa99e98     0  4410     11                4264 (L-TLB)
ffff81006fa99e98 00005555556a41d8 0000000000000009 00000000000009ba
       00002b693ef0b820 00002b693ef0b810 0000000000000001 000000016fa99e28
       ffff81007e6990c0 0000000000000174
Call Trace: <ffffffff8031f92f>{_spin_unlock_irq+7} <ffffffff8031f92f>{_spin_unlock_irq+7}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ad6e>{worker_thread+0} <ffffffff8013ae41>{worker_thread+211}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff80125562>{default_wake_function+0} <ffffffff8013de14>{keventd_create_kthread+0}
       <ffffffff8013ddeb>{kthread+200} <ffffffff8010b846>{child_rip+8}
       <ffffffff8013de14>{keventd_create_kthread+0} <ffffffff8013dd23>{kthread+0}
       <ffffffff8010b83e>{child_rip+0}

Showing all blocking locks in the system:
S            init:    1 [ffff81007fdc4770, 116] (not blocked on mutex)
S     migration/0:    2 [ffff81007fdc4040,   0] (not blocked on mutex)
S     ksoftirqd/0:    3 [ffff81007fdc67b0, 135] (not blocked on mutex)
S      watchdog/0:    4 [ffff81007fdc6080,   0] (not blocked on mutex)
S     migration/1:    5 [ffff81007fdc77f0,   0] (not blocked on mutex)
S     ksoftirqd/1:    6 [ffff81007fdc70c0, 134] (not blocked on mutex)
S      watchdog/1:    7 [ffff810003364830,   0] (not blocked on mutex)
S        events/0:    8 [ffff810037fef870, 110] (not blocked on mutex)
S        events/1:    9 [ffff810037fef140, 110] (not blocked on mutex)
S         khelper:   10 [ffff810037fee770, 110] (not blocked on mutex)
S         kthread:   11 [ffff810037fee040, 110] (not blocked on mutex)
S       kblockd/0:   15 [ffff81007f580830, 110] (not blocked on mutex)
S       kblockd/1:   16 [ffff81007f580100, 110] (not blocked on mutex)
S          kacpid:   17 [ffff810003369870, 114] (not blocked on mutex)
S           khubd:  115 [ffff81007f5107b0, 110] (not blocked on mutex)
S         pdflush:  179 [ffff81007f505870, 120] (not blocked on mutex)
S         pdflush:  180 [ffff81007f517140, 115] (not blocked on mutex)
S           aio/0:  182 [ffff81007f7240c0, 110] (not blocked on mutex)
S         kswapd0:  181 [ffff81007f540080, 117] (not blocked on mutex)
S           aio/1:  183 [ffff81007f687100, 111] (not blocked on mutex)
S         kseriod:  258 [ffff81007f682770, 110] (not blocked on mutex)
S       kpsmoused:  316 [ffff81007f564040, 111] (not blocked on mutex)
S           ata/0:  337 [ffff81007f64e7b0, 114] (not blocked on mutex)
S           ata/1:  338 [ffff81007f682040, 110] (not blocked on mutex)
S       scsi_eh_0:  344 [ffff81007f64e080, 114] (not blocked on mutex)
S       scsi_eh_1:  345 [ffff81007f60e7b0, 114] (not blocked on mutex)
S       scsi_eh_3:  382 [ffff81007f7247f0, 111] (not blocked on mutex)
S       scsi_wq_3:  385 [ffff81007f7dc870, 111] (not blocked on mutex)
S        kmirrord:  417 [ffff81007edc7870, 115] (not blocked on mutex)
S       kjournald:  431 [ffff81007ed8e7f0, 115] (not blocked on mutex)
S           udevd: 1358 [ffff81007edc8100, 111] (not blocked on mutex)
S         kauditd: 1935 [ffff81007e562140, 110] (not blocked on mutex)
S       kmpathd/0: 2043 [ffff81007e58d7b0, 112] (not blocked on mutex)
S       kmpathd/1: 2044 [ffff81007ead57f0, 112] (not blocked on mutex)
S       kjournald: 2104 [ffff81007e9de080, 119] (not blocked on mutex)
S        dhclient: 2716 [ffff81007a0d9040, 116] (not blocked on mutex)
R         syslogd: 2760 [ffff81007edc8830, 116] (not blocked on mutex)
S           klogd: 2764 [ffff81007ebd1080, 115] (not blocked on mutex)
S      irqbalance: 2775 [ffff81007e484040, 116] (not blocked on mutex)
S         portmap: 2787 [ffff81007eece0c0, 116] (not blocked on mutex)
S       rpc.statd: 2807 [ffff81007e4f0100, 122] (not blocked on mutex)
S      rpc.idmapd: 2846 [ffff81007ec83870, 116] (not blocked on mutex)
S           acpid: 2937 [ffff81007ec71100, 119] (not blocked on mutex)
S           cupsd: 2968 [ffff81007e58d080, 116] (not blocked on mutex)
S            sshd: 3073 [ffff81007edc7140, 117] (not blocked on mutex)
S          xinetd: 3094 [ffff81007ed1b7b0, 115] (not blocked on mutex)
S        sendmail: 3113 [ffff81007e5300c0, 116] (not blocked on mutex)
S        sendmail: 3121 [ffff81007e614870, 119] (not blocked on mutex)
S             gpm: 3170 [ffff81007f53c040, 115] (not blocked on mutex)
S             htt: 3229 [ffff81000337b0c0, 121] (not blocked on mutex)
S      htt_server: 3230 [ffff81007e310140, 117] (not blocked on mutex)
S     cannaserver: 3242 [ffff81007e739100, 116] (not blocked on mutex)
S           crond: 3254 [ffff81007e6997f0, 116] (not blocked on mutex)
S             xfs: 3295 [ffff81007e5247f0, 116] (not blocked on mutex)
S             atd: 3314 [ffff81007f60e080, 118] (not blocked on mutex)
S   dbus-daemon-1: 3333 [ffff810079e48040, 116] (not blocked on mutex)
S cups-config-dae: 3343 [ffff81007e549140, 119] (not blocked on mutex)
S           login: 3363 [ffff81007a26c7b0, 116] (not blocked on mutex)
S        mingetty: 3364 [ffff81007e549870, 118] (not blocked on mutex)
S        mingetty: 3365 [ffff81007e0eb770, 120] (not blocked on mutex)
S        mingetty: 3368 [ffff810079df1770, 121] (not blocked on mutex)
S        mingetty: 3369 [ffff810079c90100, 121] (not blocked on mutex)
S        mingetty: 3371 [ffff81007eece7f0, 118] (not blocked on mutex)
S        mingetty: 3581 [ffff81007ead3100, 122] (not blocked on mutex)
S      gdm-binary: 3584 [ffff8100721bb830, 115] (not blocked on mutex)
?      gdm-binary: 3845 [ffff81007ede3040, 117] (not blocked on mutex)
S               X: 3846 [ffff81007232c830, 115] (not blocked on mutex)
S      gdmgreeter: 3966 [ffff810079894870, 115] (not blocked on mutex)
R            bash: 3967 [ffff81000337b7f0, 116] (not blocked on mutex)
S       scsi_eh_4: 4224 [ffff81007f5407b0, 112] (not blocked on mutex)
S   qla2xxx_4_dpc: 4253 [ffff81006fbf9830, 100] (not blocked on mutex)
D       scsi_wq_4: 4255 [ffff81006f9147b0, 110] (not blocked on mutex)
S       scsi_eh_5: 4264 [ffff81007e739830, 111] (not blocked on mutex)
S   qla2xxx_5_dpc: 4408 [ffff81007e5240c0, 100] (not blocked on mutex)
S       scsi_wq_5: 4410 [ffff81007e6990c0, 111] (not blocked on mutex)

---------------------------
| showing all locks held: |
---------------------------

#001:             [ffff81006ee20080] {scsi_host_alloc}
.. held by:         scsi_wq_4: 4255 [ffff81006f9147b0, 110]
... acquired at:               scsi_scan_target+0x51/0x87 [scsi_mod]

=============================================



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re: problems with scsi_transport_fc and qla2xxx
  2006-03-06 21:28               ` Andrew Vasquez
  2006-03-06 20:57                 ` Maxim Kozover
@ 2006-03-06 21:09                 ` Maxim Kozover
       [not found]                   ` <20060306232831.GS6278@andrew-vasquezs-powerbook-g4-15.local>
  1 sibling, 1 reply; 19+ messages in thread
From: Maxim Kozover @ 2006-03-06 21:09 UTC (permalink / raw)
  To: Andrew Vasquez; +Cc: Mike Snitzer, linux-kernel, linux-scsi

Hi Andrew!
After applying the patch the same lock exists:
#001:             [ffff81006edc4080] {scsi_host_alloc}
.. held by:         scsi_wq_4: 4255 [ffff81007edaf770, 110]
... acquired at:               scsi_scan_target+0x51/0x87 [scsi_mod]

Thanks,

Maxim.

^ permalink raw reply	[flat|nested] 19+ messages in thread

[parent not found: <20060306232831.GS6278@andrew-vasquezs-powerbook-g4-15.local>]

[parent not found: <1219491790.20060307124035@netvision.net.il>]

[parent not found: <20060307172227.GE6275@andrew-vasquezs-powerbook-g4-15.local>]

[parent not found: <1343850424.20060307231141@netvision.net.il>]

[parent not found: <20060308080050.GF9956@andrew-vasquezs-powerbook-g4-15.local>]

[parent not found: <20060308154341.GA1779@andrew-vasquezs-powerbook-g4-15.local>]

[parent not found: <1502511597.20060308213247@netvision.net.il>]

[parent not found: <20060310231344.GB641@andrew-vasquezs-powerbook-g4-15.local>]

* Re[8]: problems with scsi_transport_fc and qla2xxx
       [not found]                                 ` <20060310231344.GB641@andrew-vasquezs-powerbook-g4-15.local>
@ 2006-03-11 21:10                                   ` Maxim Kozover
  2006-03-12  9:28                                     ` Arjan van de Ven
  2006-03-13 23:19                                     ` Andrew Vasquez
  0 siblings, 2 replies; 19+ messages in thread
From: Maxim Kozover @ 2006-03-11 21:10 UTC (permalink / raw)
  To: Andrew Vasquez; +Cc: linux-kernel, linux-scsi

Hi Andrew!
Congratulations! The kernel from scsi-rc-fixes git and your patch are
working.
By the way, could you, please, tell me how I get only scsi patches
from the git repository, cause I got the whole kernel by using
cg-clone http://kernel.org/pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6.git

Now the process looks like following:
Mar 11 23:54:22 multipath kernel: qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
Mar 11 23:54:28 multipath kernel:  rport-4:0-0: blocked FC remote port time out:
 removing target and saving binding
Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
Mar 11 23:54:59 multipath kernel:  4:0:0:0: timing out command, waited 22s

And the disks appear.
Could you tell me, please, where this 22sec timeout came from?

Again, congratulations for good work!

Thanks much,

Maxim.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re[8]: problems with scsi_transport_fc and qla2xxx
  2006-03-11 21:10                                   ` Re[8]: " Maxim Kozover
@ 2006-03-12  9:28                                     ` Arjan van de Ven
  2006-03-12 11:46                                       ` Maxim Kozover
  2006-03-13 23:19                                     ` Andrew Vasquez
  1 sibling, 1 reply; 19+ messages in thread
From: Arjan van de Ven @ 2006-03-12  9:28 UTC (permalink / raw)
  To: Maxim Kozover; +Cc: linux-scsi, linux-kernel, Andrew Vasquez

On Sun, 2006-03-12 at 00:10 +0300, Maxim Kozover wrote:
> Hi Andrew!
> Congratulations! The kernel from scsi-rc-fixes git and your patch are
> working.
> By the way, could you, please, tell me how I get only scsi patches
> from the git repository, cause I got the whole kernel by using
> cg-clone http://kernel.org/pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6.git
> 
> Now the process looks like following:
> Mar 11 23:54:22 multipath kernel: qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
> Mar 11 23:54:28 multipath kernel:  rport-4:0-0: blocked FC remote port time out:
>  removing target and saving binding
> Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
> Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
> Mar 11 23:54:59 multipath kernel:  4:0:0:0: timing out command, waited 22s
> 
> And the disks appear.
> Could you tell me, please, where this 22sec timeout came from?

looks like your fiber fabric decided to renegotiate, and halfway it went
for a coffee and donuts break to not upset the union rules :)

I've seen LOOP negotiations take 10+ seconds before, and that is on a
really simple setup.... so nothing super special 


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re: Re[8]: problems with scsi_transport_fc and qla2xxx
  2006-03-12  9:28                                     ` Arjan van de Ven
@ 2006-03-12 11:46                                       ` Maxim Kozover
  0 siblings, 0 replies; 19+ messages in thread
From: Maxim Kozover @ 2006-03-12 11:46 UTC (permalink / raw)
  To: Arjan van de Ven; +Cc: linux-scsi, linux-kernel, Andrew Vasquez

OK, Arjan, thanks!

Maxim.

AvdV> looks like your fiber fabric decided to renegotiate, and halfway it went
AvdV> for a coffee and donuts break to not upset the union rules :)

AvdV> I've seen LOOP negotiations take 10+ seconds before, and that is on a
AvdV> really simple setup.... so nothing super special 



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Re[8]: problems with scsi_transport_fc and qla2xxx
  2006-03-11 21:10                                   ` Re[8]: " Maxim Kozover
  2006-03-12  9:28                                     ` Arjan van de Ven
@ 2006-03-13 23:19                                     ` Andrew Vasquez
  2006-03-20 10:45                                       ` Maxim Kozover
  1 sibling, 1 reply; 19+ messages in thread
From: Andrew Vasquez @ 2006-03-13 23:19 UTC (permalink / raw)
  To: Maxim Kozover; +Cc: linux-kernel, linux-scsi, Michael Reed, James.Smart

On Sun, 12 Mar 2006, Maxim Kozover wrote:

> Congratulations! The kernel from scsi-rc-fixes git and your patch are
> working.

Actually Mike R. and James S. deserve the credit for the composite
patch which consists of:

1) [PATCH] FC transport : Avoid device offline cases by stalling aborts until device unblocked
   http://marc.theaimsgroup.com/?l=linux-scsi&m=114225658724378&w=2

2) Serialize scan work during fc_remote_port_delete() so rport removal
doesn't deadlock midlayer scans.  The problem you were seeing.  (Mike
R.)

3) rport race fixes during removal (James S.).

> By the way, could you, please, tell me how I get only scsi patches
> from the git repository, cause I got the whole kernel by using
> cg-clone http://kernel.org/pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6.git
> 
> Now the process looks like following:
> Mar 11 23:54:22 multipath kernel: qla2xxx 0000:03:01.0: LOOP DOWN detected (2).
> Mar 11 23:54:28 multipath kernel:  rport-4:0-0: blocked FC remote port time out:
>  removing target and saving binding
> Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LIP reset occured (f7f7).
> Mar 11 23:54:37 multipath kernel: qla2xxx 0000:03:01.0: LOOP UP detected (2 Gbps).
> Mar 11 23:54:59 multipath kernel:  4:0:0:0: timing out command, waited 22s
> 
> And the disks appear.
> Could you tell me, please, where this 22sec timeout came from?

Essentially there's currently several issues with rport consumers
making delete() calls during mid-layer scanning.

I'm hoping at a minimum we can get Mike R's fixes into 2.6.16, and
address the additional races going forward...  James/Mike?

Here's a minimal the serialize scan-work patch, could you check to see
that this addresses your issue?  Start with any latest linux-2.6.git
tree.

Thanks,
Andrew

---

diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
index 929032e..3d09920 100644
--- a/drivers/scsi/scsi_transport_fc.c
+++ b/drivers/scsi/scsi_transport_fc.c
@@ -1649,6 +1649,8 @@ fc_remote_port_delete(struct fc_rport  *
 		return;
 	}
 
+	/* flush any scan work */ /* which can sleep */
+	scsi_flush_work(rport_to_shost(rport));
 	scsi_target_block(&rport->dev);
 
 	/* cap the length the devices can be blocked until they are deleted */

^ permalink raw reply related	[flat|nested] 19+ messages in thread

* Re: Re: Re[8]: problems with scsi_transport_fc and qla2xxx
  2006-03-13 23:19                                     ` Andrew Vasquez
@ 2006-03-20 10:45                                       ` Maxim Kozover
  0 siblings, 0 replies; 19+ messages in thread
From: Maxim Kozover @ 2006-03-20 10:45 UTC (permalink / raw)
  To: Andrew Vasquez; +Cc: linux-kernel, linux-scsi, Michael Reed, James.Smart

Hi Andrew!
Unfortunately I see that scan-work patch is not included in
2.6.16 and the usual lock appears:
#001:             [ffff8100708a8080] {scsi_host_alloc}
.. held by:         scsi_wq_4: 3912 [ffff810071edf870, 110]
... acquired at:               scsi_scan_target+0x51/0x87 [scsi_mod]

Applying the patch you sent solves the problem, i.e. disks appear again after
22 sec timeout (why?).

Thanks,

Maxim.

Tuesday, March 14, 2006, 2:19:03 AM, you wrote:

AV> diff --git a/drivers/scsi/scsi_transport_fc.c
AV> b/drivers/scsi/scsi_transport_fc.c
AV> index 929032e..3d09920 100644
AV> --- a/drivers/scsi/scsi_transport_fc.c
AV> +++ b/drivers/scsi/scsi_transport_fc.c
AV> @@ -1649,6 +1649,8 @@ fc_remote_port_delete(struct fc_rport  *
AV>                 return;
AV>         }
AV>  
AV> +       /* flush any scan work */ /* which can sleep */
AV> +       scsi_flush_work(rport_to_shost(rport));
AV>         scsi_target_block(&rport->dev);
AV>  
AV>         /* cap the length the devices can be blocked until they are deleted */



^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2006-03-20 11:46 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-02-27 12:05 problems with scsi_transport_fc and qla2xxx Maxim Kozover
2006-02-27 18:05 ` Maxim Kozover
2006-03-01  6:14   ` Andrew Morton
2006-03-01 20:25     ` Stefan Kaltenbrunner
2006-03-01 21:08       ` Andrew Vasquez
2006-03-02 16:32         ` Maxim Kozover
2006-03-02 17:38           ` Andrew Vasquez
2006-03-02 22:15             ` Maxim Kozover
2006-03-06 19:06               ` Andrew Vasquez
2006-03-06 20:00           ` Mike Snitzer
2006-03-06 20:09             ` Maxim Kozover
2006-03-06 21:28               ` Andrew Vasquez
2006-03-06 20:57                 ` Maxim Kozover
2006-03-06 21:09                 ` Maxim Kozover
     [not found]                   ` <20060306232831.GS6278@andrew-vasquezs-powerbook-g4-15.local>
     [not found]                     ` <1219491790.20060307124035@netvision.net.il>
     [not found]                       ` <20060307172227.GE6275@andrew-vasquezs-powerbook-g4-15.local>
     [not found]                         ` <1343850424.20060307231141@netvision.net.il>
     [not found]                           ` <20060308080050.GF9956@andrew-vasquezs-powerbook-g4-15.local>
     [not found]                             ` <20060308154341.GA1779@andrew-vasquezs-powerbook-g4-15.local>
     [not found]                               ` <1502511597.20060308213247@netvision.net.il>
     [not found]                                 ` <20060310231344.GB641@andrew-vasquezs-powerbook-g4-15.local>
2006-03-11 21:10                                   ` Re[8]: " Maxim Kozover
2006-03-12  9:28                                     ` Arjan van de Ven
2006-03-12 11:46                                       ` Maxim Kozover
2006-03-13 23:19                                     ` Andrew Vasquez
2006-03-20 10:45                                       ` Maxim Kozover

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox