qla2xxx: Shutdown Hang / OOPS with 2.6.16-rc1 + qla2xxx driver via mm4 patch

[Michael Reed <mdr@sgi.com>]

Hi Andrew,

I'm running 2.6.16-rc1 + your driver from mm4.  The entire mm4 patch
causes the kernel to not compile for my ia64 platform.

I'm observing a system hang at shutdown.  Some dpc threads appear to
be in the state below and the associated scsi_wq threads are all as shown.
Smells like a deadlock.

It appears as though the four patches you referenced in email are already
applied.

Have I got the right code?

--

Additional testing also showed an OOPS (below) with the same code.
I was running lots of i/o and did switchdisable / switchenable.

Mike


[0]kdb> btp 932
Stack traceback for pid 932
0xe0000034f6610000      932        1  0    1   D  0xe0000034f6610380  qla2300_5_dpc
0xa00000010071b670 schedule+0x2290
        args (0xe00000b003914500, 0xa0000001000cac90, 0x309, 0xa0000001000cac80, 0x1)
0xa0000001000caca0 flush_cpu_workqueue+0x140
        args (0xe00000b003914500, 0x250, 0xe0000034f6610000, 0xa0000001000cae00, 0x288)
0xa0000001000cae00 flush_workqueue+0xc0
        args (0xe0000030070fb0c0, 0x2, 0xa0000001000cafc0, 0x184, 0x205)
0xa0000001000cafc0 flush_scheduled_work+0x20
        args (0xa00000010055c200, 0x389, 0x389)
0xa00000010055c200 fc_remote_port_rolechg+0x1a0
        args (0xe000003011f36e00, 0x1, 0xe000003011f37078, 0xe0000034f67f0000, 0xa0000001005729b0)
0xa0000001005729b0 qla2x00_reg_remote_port+0x3b0
        args (0xe0000034f2b0375c, 0xe0000034f2b03700, 0xe0000034f2b03778, 0xe000003011f36e00, 0xe0000034f6617d54)
0xa000000100572b10 qla2x00_update_fcport+0xd0
        args (0xe0000034f67f04b8, 0xe0000034f2b03700, 0xa000000100572c80, 0x30b, 0xe0000034f2b03730)
0xa000000100572c80 qla2x00_fabric_dev_login+0x140
        args (0xe0000034f67f04b8, 0xe0000034f2b03700, 0x0, 0xa000000100575220, 0x694)
0xa000000100575220 qla2x00_configure_loop+0x2320
        args (0xe0000034f67f04b8, 0xe0000034f2b03700, 0xe0000034f67f26e8, 0x0, 0xe0000030178f3980)
0xa0000001005774f0 qla2x00_loop_resync+0x1b0
        args (0xfe, 0x0, 0xa00000010056c3d0, 0x30b, 0xa00000010056ba30)
0xa00000010056c3d0 qla2x00_do_dpc+0xb50
        args (0xe0000034f67f04b8, 0x4018090, 0x4018000, 0xa000000100011070, 0x286)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100842f90, 0xe0000034f67f04b8, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100842f90, 0xe0000034f67f04b8)
[0]kdb> btp 934
Stack traceback for pid 934
0xe00000b07a0e0000      934       19  0    2   D  0xe00000b07a0e0380  scsi_wq_5
0xa00000010071b670 schedule+0x2290
        args (0xe00000b003914500, 0xa0000001000cac90, 0x309, 0xa0000001000cac80, 0x1)
0xa0000001000caca0 flush_cpu_workqueue+0x140
        args (0xe00000b003914500, 0x24e, 0xe00000b07a0e0000, 0xa0000001000cae00, 0x288)
0xa0000001000cae00 flush_workqueue+0xc0
        args (0xe0000030070fb0c0, 0x2, 0xa0000001000cafc0, 0x184, 0x205)
0xa0000001000cafc0 flush_scheduled_work+0x20
        args (0xa0000001005578d0, 0x307, 0x307)
0xa0000001005578d0 fc_rport_tgt_remove+0xf0
        args (0xe000003012942200, 0xe000003012942478, 0xe0000034f67f0000, 0xa000000100557a10, 0x308)
0xa000000100557a10 fc_shost_remove_rports+0xf0
        args (0xe0000034f67f0000, 0xe0000034f67f0058, 0xe000003012942200, 0xa0000001000ca490, 0x40c)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f76ed980, 0xe0000034f661c890, 0xe0000034f661c888, 0xe0000034f67f0000, 0xa000000100842540)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f76ed980, 0xe0000034f76ed998, 0xe0000034f76ed9a8, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b07a0e7dc0, 0xe00000b07a0e0000, 0xe0000034f76ed980, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)


======================
====================== start of oops data
======================

duck /root# WARNING: Switching to polling CMC handler; error records may be lost
WARNING: Switching to polling CPE handler; error records may be lost
qla2300 0012:00:02.1: LOOP DOWN detected (2).
qla2300 0012:00:02.0: LOOP DOWN detected (2).
qla2300 0012:00:01.1: LOOP DOWN detected (2).
qla2300 0012:00:01.0: LOOP DOWN detected (2).
qla2300 0002:01:04.1: LOOP DOWN detected (2).
qla2300 0002:01:06.1: LOOP DOWN detected (2).
qla2300 0002:01:04.0: LOOP DOWN detected (2).
qla2300 0002:01:06.0: LOOP DOWN detected (2).
qla2300 0002:01:04.1: LIP reset occured (f7f7).
qla2300 0012:00:02.1: LIP reset occured (f800).
qla2300 0012:00:01.1: LIP reset occured (f800).
qla2300 0012:00:02.0: LIP reset occured (f7f7).
qla2300 0012:00:01.0: LIP reset occured (f7f7).
qla2300 0002:01:06.1: LIP reset occured (f800).
qla2300 0002:01:04.0: LIP reset occured (f7f7).
qla2300 0002:01:06.0: LIP reset occured (f800).
qla2300 0012:00:01.0: LIP reset occured (f7f7).
qla2300 0002:01:04.0: LIP reset occured (f7f7).
qla2300 0012:00:02.0: LIP reset occured (f7f7).
qla2300 0012:00:01.1: LIP reset occured (f7f7).
qla2300 0012:00:02.1: LIP reset occured (f7f7).
qla2300 0002:01:04.1: LIP reset occured (f800).
qla2300 0002:01:06.1: LIP reset occured (f7f7).
qla2300 0002:01:06.0: LIP reset occured (f7f7).
qla2300 0012:00:02.0: LOOP UP detected (2 Gbps).
qla2300 0012:00:01.0: LOOP UP detected (2 Gbps).
qla2300 0012:00:02.1: LOOP UP detected (2 Gbps).
qla2300 0012:00:01.1: LOOP UP detected (2 Gbps).
qla2300 0002:01:04.0: LOOP UP detected (2 Gbps).
qla2300 0002:01:04.1: LOOP UP detected (2 Gbps).
qla2300 0002:01:06.1: LOOP UP detected (2 Gbps).
qla2300 0002:01:06.0: LOOP UP detected (2 Gbps).
Unable to handle kernel NULL pointer dereference (address 000000000000003c)
scsi_wq_8[1805]: Oops 11012296146944 [1]
Modules linked in: ipv6 nfs lockd sunrpc sg

Pid: 1805, CPU 3, comm:            scsi_wq_8
psr : 00001210081a2018 ifs : 800000000000050c ip  : [<a00000010056fd61>]    Not tainted
ip is at qla2x00_queuecommand+0x301/0x360
unat: 0000000000000000 pfs : 000000000000050c rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr  : 0000000000005941
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f
csd : 0000000000000000 ssd : 0000000000000000
b0  : a00000010056fac0 b6  : a00000010056fa60 b7  : a000000100547980
f6  : 000000000000000000000 f7  : 1003e0000000000000100
f8  : 1003e000000000000003c f9  : 10002a000000000000000
f10 : 0fffeccccccccc8c00000 f11 : 1003e0000000000000000
r1  : a000000100d1c990 r2  : a000000100b29a80 r3  : e0000034f6636ba8
r8  : 0000000000000001 r9  : 000000000001da46 r10 : 000000000001da45
r11 : 000000000001da45 r12 : e00000b0786c7bb0 r13 : e00000b0786c0000
r14 : 0000000000000002 r15 : 000000000000003c r16 : e00000b007fa3090
r17 : e00000b07a0e8140 r18 : e00000b072ebbd48 r19 : 000000000001da46
r20 : 000000000001da45 r21 : 000000000001da45 r22 : e00000b072ebbd65
r23 : 00000000000000a0 r24 : a00000010056fa60 r25 : a0000001009b7c58
r26 : a0000001009b7c20 r27 : e00000b07a0e80e8 r28 : a000000100b298e8
r29 : e00000b007fa3000 r30 : 000000000000000c r31 : a000000100b3485d

Call Trace:
 [<a000000100012ae0>] show_stack+0x40/0xa0
                                sp=e00000b0786c7740 bsp=e00000b0786c15c0
 [<a0000001000133f0>] show_regs+0x850/0x8a0
                                sp=e00000b0786c7910 bsp=e00000b0786c1568
 [<a000000100037870>] die+0x210/0x2c0
                                sp=e00000b0786c7910 bsp=e00000b0786c1520
 [<a00000010005cc90>] ia64_do_page_fault+0xa10/0xba0
                                sp=e00000b0786c7930 bsp=e00000b0786c14d0
 [<a00000010000bbe0>] ia64_leave_kernel+0x0/0x290
                                sp=e00000b0786c79e0 bsp=e00000b0786c14d0
 [<a00000010056fd60>] qla2x00_queuecommand+0x300/0x360
                                sp=e00000b0786c7bb0 bsp=e00000b0786c1470
 [<a00000010053ab40>] scsi_dispatch_cmd+0x480/0x540
                                sp=e00000b0786c7bb0 bsp=e00000b0786c1430
 [<a0000001005477b0>] scsi_request_fn+0x650/0x820
                                sp=e00000b0786c7bb0 bsp=e00000b0786c13e0
 [<a0000001003cdb90>] __generic_unplug_device+0x90/0xc0
                                sp=e00000b0786c7bb0 bsp=e00000b0786c13c0
 [<a0000001003ce0d0>] generic_unplug_device+0x30/0x60
                                sp=e00000b0786c7bb0 bsp=e00000b0786c1398
 [<a0000001003ce180>] blk_execute_rq_nowait+0x80/0xa0
                                sp=e00000b0786c7bb0 bsp=e00000b0786c1358
 [<a0000001003ce300>] blk_execute_rq+0x160/0x1e0
                                sp=e00000b0786c7bb0 bsp=e00000b0786c12f8
 [<a000000100546d10>] scsi_execute+0x170/0x1c0
                                sp=e00000b0786c7c60 bsp=e00000b0786c12a0
 [<a000000100546e50>] scsi_execute_req+0xf0/0x180
                                sp=e00000b0786c7c60 bsp=e00000b0786c1238
 [<a00000010054b330>] __scsi_scan_target+0x4b0/0xb60
                                sp=e00000b0786c7c70 bsp=e00000b0786c11c8
 [<a00000010054bfb0>] scsi_scan_target+0xd0/0x100
                                sp=e00000b0786c7cd0 bsp=e00000b0786c1178
 [<a000000100558c70>] fc_scsi_scan_rport+0x50/0x80
                                sp=e00000b0786c7cd0 bsp=e00000b0786c1150
 [<a0000001000ca490>] run_workqueue+0x1b0/0x280
                                sp=e00000b0786c7cd0 bsp=e00000b0786c1110
 [<a0000001000cbb90>] worker_thread+0x1d0/0x260
                                sp=e00000b0786c7cd0 bsp=e00000b0786c10e0
 [<a0000001000d3980>] kthread+0x220/0x2a0
                                sp=e00000b0786c7d50 bsp=e00000b0786c1098
 [<a000000100011070>] kernel_thread_helper+0xd0/0x100
                                sp=e00000b0786c7e30 bsp=e00000b0786c1070
 [<a000000100009120>] start_kernel_thread+0x20/0x40
                                sp=e00000b0786c7e30 bsp=e00000b0786c1070

Entering kdb (current=0xe00000b0786c0000, pid 1805) on processor 3 Oops: <NULL>
due to oops @ 0xa00000010056fd61
 psr: 0x00001210081a2018   ifs: 0x800000000000050c    ip: 0xa00000010056fd60
unat: 0x0000000000000000   pfs: 0x000000000000050c   rsc: 0x0000000000000003
rnat: 0x0000000000000000  bsps: 0x0000000000000000    pr: 0x0000000000005941
ldrs: 0x0000000000000000   ccv: 0x0000000000000000  fpsr: 0x0009804c8a70433f
  b0: 0xa00000010056fac0    b6: 0xa00000010056fa60    b7: 0xa000000100547980
  r1: 0xa000000100d1c990    r2: 0xa000000100b29a80    r3: 0xe0000034f6636ba8
  r8: 0x0000000000000001    r9: 0x000000000001da46   r10: 0x000000000001da45
 r11: 0x000000000001da45   r12: 0xe00000b0786c7bb0   r13: 0xe00000b0786c0000
 r14: 0x0000000000000002   r15: 0x000000000000003c   r16: 0xe00000b007fa3090
 r17: 0xe00000b07a0e8140   r18: 0xe00000b072ebbd48   r19: 0x000000000001da46
 r20: 0x000000000001da45   r21: 0x000000000001da45   r22: 0xe00000b072ebbd65
 r23: 0x00000000000000a0   r24: 0xa00000010056fa60   r25: 0xa0000001009b7c58
 r26: 0xa0000001009b7c20   r27: 0xe00000b07a0e80e8   r28: 0xa000000100b298e8
 r29: 0xe00000b007fa3000   r30: 0x000000000000000c   r31: 0xa000000100b3485d
&regs = e00000b0786c79f0
[3]kdb>

[3]kdb> btp 1803
Stack traceback for pid 1803
0xe00000b078070000     1803        1  0    3   R  0xe00000b078070380  qla2300_8_dpc
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a0eb3e8, 0x4000, 0xa000000100718f10, 0x38a, 0x0)
0xa000000100718f20 __down+0x260
        args (0xe00000b07a0eb3e0, 0x10081a6018, 0xe00000b07a0eb3e8, 0xe00000b078070000, 0xa00000010057beb0)
0xa00000010057beb0 qla2x00_mailbox_command+0x490
        args (0x2ef8, 0xe00000b078077cf8, 0xe00000b07a0e84b8, 0xc00000880fe02000, 0x1)
0xa00000010057f3f0 qla2x00_get_port_database+0x1f0
        args (0xe00000b07a0f0300, 0xe00000b0786ccb80, 0x0, 0xe00000b07a0e84b8, 0xa000000100572bb0)
0xa000000100572bb0 qla2x00_fabric_dev_login+0x70
        args (0xe00000b07a0e84b8, 0xe00000b0786ccb80, 0x0, 0xa000000100575220, 0x694)
0xa000000100575220 qla2x00_configure_loop+0x2320
        args (0xe00000b07a0e84b8, 0xe00000b0786ccb80, 0xe00000b07a0ea6e8, 0x0, 0xe00000b0786ce580)
0xa0000001005774f0 qla2x00_loop_resync+0x1b0
        args (0xfe, 0x0, 0xa00000010056c3d0, 0x30b, 0xa00000010056ba30)
0xa00000010056c3d0 qla2x00_do_dpc+0xb50
        args (0xe00000b07a0e84b8, 0x4000090, 0x4000000, 0xa000000100011070, 0x286)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100842f90, 0xe00000b07a0e84b8, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100842f90, 0xe00000b07a0e84b8)



[3]kdb> btp 755
Stack traceback for pid 755
0xe00000b003fb0000      755       19  0    0   D  0xe00000b003fb0380  scsi_wq_2
0xa00000010071b670 schedule+0x2290
        args (0xe00000b003fb7bd0, 0xa00000010071bb90, 0x309, 0xe0000034f66c7960, 0x1)
0xa00000010071bba0 wait_for_completion+0x140
        args (0xe00000b003fb7bc8, 0xe00000b003fb7b90, 0xe00000b003fb7bd0, 0xa0000001003ce310, 0x611)
0xa0000001003ce310 blk_execute_rq+0x170
        args (0xe0000034f66c7778, 0x0, 0xe0000034f53f7af0, 0x1, 0xe00000b003fb7c50)
0xa000000100546d10 scsi_execute+0x170
        args (0xe0000034f679d808, 0xe00000b003fb7c8c, 0xe0000034f53f7af0, 0x4000000, 0x1000)
0xa000000100546e50 scsi_execute_req+0xf0
        args (0xe0000034f679d800, 0xe00000b003fb7c8c, 0x2, 0xe0000034f74c7000, 0x1000)
0xa00000010054b330 __scsi_scan_target+0x4b0
        args (0xe00000b003fb7c84, 0x1000, 0xe0000034f74c7000, 0x1, 0x1)
0xa00000010054bfb0 scsi_scan_target+0xd0
        args (0xe0000034f7b0b650, 0x0, 0x8, 0xffffffffffffffff, 0x1)
0xa000000100558c70 fc_scsi_scan_rport+0x50
        args (0xe0000034f7b0b62c, 0xe0000034f7b0b650, 0xa0000001000ca490, 0x40c, 0x286)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f6538800, 0xe0000034f7b0b8e0, 0xe0000034f7b0b8d8, 0xe0000034f7b0b600, 0xa000000100842560)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f6538800, 0xe0000034f6538818, 0xe0000034f6538828, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b003fb7dc0, 0xe00000b003fb0000, 0xe0000034f6538800, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)
[3]kdb> btp 925
Stack traceback for pid 925
0xe00000b07a0a8000      925       19  0    1   D  0xe00000b07a0a8380  scsi_wq_3
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a0afbd0, 0xa00000010071bb90, 0x309, 0xe0000034f67bdc40, 0x1)
0xa00000010071bba0 wait_for_completion+0x140
        args (0xe00000b07a0afbc8, 0xe00000b07a0afb90, 0xe00000b07a0afbd0, 0xa0000001003ce310, 0x611)
0xa0000001003ce310 blk_execute_rq+0x170
        args (0xe0000034f67bda58, 0x0, 0xe0000034f52f7c10, 0x1, 0xe00000b07a0afc50)
0xa000000100546d10 scsi_execute+0x170
        args (0xe0000034f6863808, 0xe00000b07a0afc8c, 0xe0000034f52f7c10, 0x4000000, 0x1000)
0xa000000100546e50 scsi_execute_req+0xf0
        args (0xe0000034f6863800, 0xe00000b07a0afc8c, 0x2, 0xe0000034f74c6000, 0x1000)
0xa00000010054b330 __scsi_scan_target+0x4b0
        args (0xe00000b07a0afc84, 0x1000, 0xe0000034f74c6000, 0x1, 0x1)
0xa00000010054bfb0 scsi_scan_target+0xd0
        args (0xe0000034f66fced0, 0x0, 0x8, 0xffffffffffffffff, 0x1)
0xa000000100558c70 fc_scsi_scan_rport+0x50
        args (0xe0000034f66fceac, 0xe0000034f66fced0, 0xa0000001000ca490, 0x40c, 0x286)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f6538380, 0xe0000034f66fd160, 0xe0000034f66fd158, 0xe0000034f66fce80, 0xa000000100842560)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f6538380, 0xe0000034f6538398, 0xe0000034f65383a8, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b07a0afdc0, 0xe00000b07a0a8000, 0xe0000034f6538380, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)
[3]kdb> btp 1049
Stack traceback for pid 1049
0xe00000b07a340000     1049       19  0    0   D  0xe00000b07a340380  scsi_wq_4
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a347bd0, 0xa00000010071bb90, 0x309, 0xe00000b003af2ea0, 0x1)
0xa00000010071bba0 wait_for_completion+0x140
        args (0xe00000b07a347bc8, 0xe00000b07a347b90, 0xe00000b07a347bd0, 0xa0000001003ce310, 0x611)
0xa0000001003ce310 blk_execute_rq+0x170
        args (0xe00000b003af2cb8, 0x0, 0xe0000034f52f5930, 0x1, 0xe00000b07a347c50)
0xa000000100546d10 scsi_execute+0x170
        args (0xe00000b07a358808, 0xe00000b07a347c8c, 0xe0000034f52f5930, 0x4000000, 0x1000)
0xa000000100546e50 scsi_execute_req+0xf0
        args (0xe00000b07a358800, 0xe00000b07a347c8c, 0x2, 0xe00000300e3f4000, 0x1000)
0xa00000010054b330 __scsi_scan_target+0x4b0
        args (0xe00000b07a347c84, 0x1000, 0xe00000300e3f4000, 0x1, 0x1)
0xa00000010054bfb0 scsi_scan_target+0xd0
        args (0xe0000034f684b6d0, 0x0, 0x8, 0xffffffffffffffff, 0x1)
0xa000000100558c70 fc_scsi_scan_rport+0x50
        args (0xe0000034f684b6ac, 0xe0000034f684b6d0, 0xa0000001000ca490, 0x40c, 0x286)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f7beb880, 0xe0000034f684b960, 0xe0000034f684b958, 0xe0000034f684b680, 0xa000000100842560)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f7beb880, 0xe0000034f7beb898, 0xe0000034f7beb8a8, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b07a347dc0, 0xe00000b07a340000, 0xe0000034f7beb880, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)
[3]kdb> btp 1271
Stack traceback for pid 1271
0xe00000b07a210000     1271       19  0    1   D  0xe00000b07a210380  scsi_wq_5
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a217bd0, 0xa00000010071bb90, 0x309, 0xe0000034f62bed20, 0x1)
0xa00000010071bba0 wait_for_completion+0x140
        args (0xe00000b07a217bc8, 0xe00000b07a217b90, 0xe00000b07a217bd0, 0xa0000001003ce310, 0x611)
0xa0000001003ce310 blk_execute_rq+0x170
        args (0xe0000034f62beb38, 0x0, 0xe0000034f52f5a50, 0x1, 0xe00000b07a217c50)
0xa000000100546d10 scsi_execute+0x170
        args (0xe0000034f6861008, 0xe00000b07a217c8c, 0xe0000034f52f5a50, 0x4000000, 0x1000)
0xa000000100546e50 scsi_execute_req+0xf0
        args (0xe0000034f6861000, 0xe00000b07a217c8c, 0x2, 0xe0000034f74c5000, 0x1000)
0xa00000010054b330 __scsi_scan_target+0x4b0
        args (0xe00000b07a217c84, 0x1000, 0xe0000034f74c5000, 0x1, 0x1)
0xa00000010054bfb0 scsi_scan_target+0xd0
        args (0xe0000034f65d05d0, 0x0, 0x8, 0xffffffffffffffff, 0x1)
0xa000000100558c70 fc_scsi_scan_rport+0x50
        args (0xe0000034f65d05ac, 0xe0000034f65d05d0, 0xa0000001000ca490, 0x40c, 0x286)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f7beb280, 0xe0000034f65d0860, 0xe0000034f65d0858, 0xe0000034f65d0580, 0xa000000100842560)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f7beb280, 0xe0000034f7beb298, 0xe0000034f7beb2a8, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b07a217dc0, 0xe00000b07a210000, 0xe0000034f7beb280, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)
[3]kdb> btp 1445
Stack traceback for pid 1445
0xe00000b003f80000     1445        1  0    2   R  0xe00000b003f80380  qla2300_6_dpc
0xa00000010071b670 schedule+0x2290
        args (0xe00000b0780bf3e8, 0x4000, 0xa000000100718f10, 0x38a, 0x0)
0xa000000100718f20 __down+0x260
        args (0xe00000b0780bf3e0, 0x1008126038, 0xe00000b0780bf3e8, 0xe00000b003f80000, 0xa00000010057beb0)
0xa00000010057beb0 qla2x00_mailbox_command+0x490
        args (0x2ef8, 0xe00000b003f87d00, 0xe00000b0780bc4b8, 0xc00000880fe00000, 0x1)
0xa00000010057df00 qla2x00_issue_iocb+0xc0
        args (0xe00000b0780bc4b8, 0xe00000b0780c4203, 0xb1000020780c4200, 0xa000000100588e60, 0x50e)
0xa000000100588e60 qla2x00_gnn_id+0x360
        args (0xe00000b0780bc4b8, 0xe00000b0731743c0, 0x1b, 0xe00000b07a2f8000, 0xe00000b0780c4200)
0xa000000100574680 qla2x00_configure_loop+0x1780
        args (0xe00000b0780bc4b8, 0x0, 0x0, 0xe00000b073174000, 0x4000)
0xa0000001005774f0 qla2x00_loop_resync+0x1b0
        args (0xfe, 0x0, 0xa00000010056c3d0, 0x30b, 0xa00000010056ba30)
0xa00000010056c3d0 qla2x00_do_dpc+0xb50
        args (0xe00000b0780bc4b8, 0x4000090, 0x4000000, 0xa000000100011070, 0x286)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100842f90, 0xe00000b0780bc4b8, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100842f90, 0xe00000b0780bc4b8)
[3]kdb> btp 1581
Stack traceback for pid 1581
0xe00000b003ff0000     1581        1  0    3   R  0xe00000b003ff0380  qla2300_7_dpc
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a2ff3e8, 0x4000, 0xa000000100718f10, 0x38a, 0x0)
0xa000000100718f20 __down+0x260
        args (0xe00000b07a2ff3e0, 0x10081a6018, 0xe00000b07a2ff3e8, 0xe00000b003ff0000, 0xa00000010057beb0)
0xa00000010057beb0 qla2x00_mailbox_command+0x490
        args (0x2ef8, 0xe00000b003ff7cc0, 0xe00000b07a2fc4b8, 0xc00000880fe01000, 0x1)
0xa00000010057d000 qla2x00_login_fabric+0x140
        args (0xe00000b003ff7ccc, 0x90, 0x13, 0xf, 0x0)
0xa000000100572090 qla2x00_fabric_login+0x90
        args (0xe00000b07a2fc4b8, 0xe00000b0039adf80, 0xe00000b003ff7d66, 0xe00000b0039adfb4, 0xe00000b0039adfb2)
0xa000000100572b70 qla2x00_fabric_dev_login+0x30
        args (0xe00000b07a2fc4b8, 0xe00000b0039adf80, 0xe00000b003ff7d66, 0xa000000100575220, 0x694)
0xa000000100575220 qla2x00_configure_loop+0x2320
        args (0xe00000b07a2fc4b8, 0xe00000b0039adf80, 0xe00000b07a2fe6e8, 0x0, 0xe00000b07a353180)
0xa0000001005774f0 qla2x00_loop_resync+0x1b0
        args (0xfe, 0x0, 0xa00000010056c3d0, 0x30b, 0xa00000010056ba30)
0xa00000010056c3d0 qla2x00_do_dpc+0xb50
        args (0xe00000b07a2fc4b8, 0x4000090, 0x4000000, 0xa000000100011070, 0x286)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100842f90, 0xe00000b07a2fc4b8, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100842f90, 0xe00000b07a2fc4b8)
[3]kdb> btp 1583
Stack traceback for pid 1583
0xe00000b07a160000     1583       19  0    3   R  0xe00000b07a160380  scsi_wq_7
0xa00000010071b670 schedule+0x2290
        args (0xe00000b07a167bd0, 0xa00000010071bb90, 0x309, 0xe00000b078ad2260, 0x1)
0xa00000010071bba0 wait_for_completion+0x140
        args (0xe00000b07a167bc8, 0xe00000b07a167b90, 0xe00000b07a167bd0, 0xa0000001003ce310, 0x611)
0xa0000001003ce310 blk_execute_rq+0x170
        args (0xe00000b078ad2078, 0x0, 0xe00000b07666d5d0, 0x1, 0xe00000b07a167c50)
0xa000000100546d10 scsi_execute+0x170
        args (0xe00000b005e98808, 0xe00000b07a167c8c, 0xe00000b07666d5d0, 0x4000000, 0x1000)
0xa000000100546e50 scsi_execute_req+0xf0
        args (0xe00000b005e98800, 0xe00000b07a167c8c, 0x2, 0xe00000b006a05000, 0x1000)
0xa00000010054b330 __scsi_scan_target+0x4b0
        args (0xe00000b07a167c84, 0x1000, 0xe00000b006a05000, 0x1, 0x1)
0xa00000010054bfb0 scsi_scan_target+0xd0
        args (0xe00000b0780eb6d0, 0x0, 0x6, 0xffffffffffffffff, 0x1)
0xa000000100558c70 fc_scsi_scan_rport+0x50
        args (0xe00000b0780eb6ac, 0xe00000b0780eb6d0, 0xa0000001000ca490, 0x40c, 0x286)
0xa0000001000ca490 run_workqueue+0x1b0
        args (0xe0000034f6e4a880, 0xe00000b0780eb960, 0xe00000b0780eb958, 0xe00000b0780eb680, 0xa000000100842560)
0xa0000001000cbb90 worker_thread+0x1d0
        args (0xe0000034f6e4a880, 0xe0000034f6e4a898, 0xe0000034f6e4a8a8, 0xa0000001000d3980, 0x491)
0xa0000001000d3980 kthread+0x220
        args (0xe0000034f7b2fa00, 0xe00000b07a167dc0, 0xe00000b07a160000, 0xe0000034f6e4a880, 0xa000000100844440)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100844920, 0xe0000034f7b2fa00, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100844920, 0xe0000034f7b2fa00)
[3]kdb> btp 1951
Stack traceback for pid 1951
0xe00000b003fa8000     1951        1  0    3   R  0xe00000b003fa8380  qla2300_9_dpc
0xa00000010071b670 schedule+0x2290
        args (0xe00000b0787133e8, 0x4000, 0xa000000100718f10, 0x38a, 0x0)
0xa000000100718f20 __down+0x260
        args (0xe00000b0787133e0, 0x10081a6018, 0xe00000b0787133e8, 0xe00000b003fa8000, 0xa00000010057beb0)
0xa00000010057beb0 qla2x00_mailbox_command+0x490
        args (0x2ef8, 0xe00000b003fafd00, 0xe00000b0787104b8, 0xc00000880fe03000, 0x1)
0xa00000010057df00 qla2x00_issue_iocb+0xc0
        args (0xe00000b0787104b8, 0xe00000b07a1ac203, 0xb10000207a1ac200, 0xa000000100588e60, 0x50e)
0xa000000100588e60 qla2x00_gnn_id+0x360
        args (0xe00000b0787104b8, 0xe00000b0712b80dc, 0x2, 0xe00000b07a1c4000, 0xe00000b07a1ac200)
0xa000000100574680 qla2x00_configure_loop+0x1780
        args (0xe00000b0787104b8, 0x0, 0x1d, 0xe00000b0712b8000, 0x4000)
0xa0000001005774f0 qla2x00_loop_resync+0x1b0
        args (0xfd, 0x0, 0xa00000010056c3d0, 0x30b, 0xa00000010056ba30)
0xa00000010056c3d0 qla2x00_do_dpc+0xb50
        args (0xe00000b0787104b8, 0x4000090, 0x4000000, 0xa000000100011070, 0x286)
0xa000000100011070 kernel_thread_helper+0xd0
        args (0xa000000100842f90, 0xe00000b0787104b8, 0xa000000100009120, 0x2, 0xa000000100d1c990)
0xa000000100009120 start_kernel_thread+0x20
        args (0xa000000100842f90, 0xe00000b0787104b8)
[3]kdb>