qlogicpti driver hangs the machine

qlogicpti driver hangs the machine

[Meelis Roos]

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in 
Sparcstation 5 (sparc32) with etch and sarge installers, then tried 
another in E3500. Both tries on SS5 hung when loadin qlogicpti module. 
E3500 conf doesn't autoload SBus drivers and the card was properly 
inserted only after installation, so I tried to modprobe qlogicpti after 
booting up. This also results in a hang but outputs that much on console 
before hanging. Maybe the error handling spinlock lockup gives a clue to 
someone...

[ 3570.576359] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
[ 3571.002975] qlogicpti0: [Fast Wide, using single ended interface]
[ 3571.078001] scsi2 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
[ 3571.667582] scsi 2:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
[ 3571.768584] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3571.852563] sd 2:0:2:0: [sdg] Write Protect is off
[ 3571.907932] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3571.973016] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.082467] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3572.166373] sd 2:0:2:0: [sdg] Write Protect is off
[ 3572.221718] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3572.286814] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.394250]  sdg: unknown partition table
[ 3572.455839] sd 2:0:2:0: [sdg] Attached SCSI disk
[ 3572.511006] sd 2:0:2:0: Attached scsi generic sg7 type 0
[ 3578.075788] qlogicpti0: Aborting cmd for tgt[2] lun[0]
[ 3578.137633] qlogicpti0: Aborting cmd for tgt[2] lun[1]
[ 3696.467069] BUG: spinlock lockup on CPU#7, scsi_eh_2/1737, fffff8004d0f8060
[ 3696.548319] Call Trace:
[ 3696.577498]  [0000000000590e5c] _raw_spin_lock+0xfc/0x160
[ 3696.642105]  [000000000069dc5c] _spin_lock_irqsave+0x1c/0x40
[ 3696.709816]  [00000000005d5090] scsi_send_eh_cmnd+0x70/0x280
[ 3696.777530]  [00000000005d52c4] scsi_eh_tur+0x24/0xa0
[ 3696.837958]  [00000000005d5f8c] scsi_error_handler+0x2cc/0x3c0
[ 3696.907783]  [000000000047676c] kthread+0x4c/0x80
[ 3696.964032]  [000000000042707c] kernel_thread+0x3c/0x60
[ 3697.026543]  [0000000000476694] kthreadd+0x174/0x200
[ 3697.086064]   CPU[  6]: TSTATE[0000009980009602] TPC[000000000069dc40] TNPC[000000000069dc44] TASK[NULL:-1]
[ 3697.202709]              TPC[_spin_lock_irqsave+0x0/0x40] O7[complete+0x14/0x60] I7[scsi_eh_done+0x24/0x40] RPC[qpti_intr+0]
[ 3697.359973] * CPU[  7]: TSTATE[0000000080009625] TPC[0000000000427074] TNPC[0000000000427078] TASK[scsi_eh_2:1737]
[ 3697.483975]              TPC[kernel_thread+0x34/0x60] O7[kernel_thread+0x8/0x60] I7[kthreadd+0x174/0x200] RPC[0x0]

-- 
Meelis Roos (mroos@ut.ee)      http://www.cs.ut.ee/~mroos/

Re: qlogicpti driver hangs the machine

[Matthew Wilcox]

On Tue, Sep 23, 2008 at 06:06:03PM +0300, Meelis Roos wrote:
> I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in 
> Sparcstation 5 (sparc32) with etch and sarge installers, then tried 
> another in E3500. Both tries on SS5 hung when loadin qlogicpti module. 
> E3500 conf doesn't autoload SBus drivers and the card was properly 
> inserted only after installation, so I tried to modprobe qlogicpti after 
> booting up. This also results in a hang but outputs that much on console 
> before hanging. Maybe the error handling spinlock lockup gives a clue to 
> someone...

What kernel version is this trace from?

> [ 3570.576359] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
> [ 3571.002975] qlogicpti0: [Fast Wide, using single ended interface]
> [ 3571.078001] scsi2 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
> [ 3571.667582] scsi 2:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
> [ 3571.768584] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
> [ 3571.852563] sd 2:0:2:0: [sdg] Write Protect is off
> [ 3571.907932] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
> [ 3571.973016] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [ 3572.082467] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
> [ 3572.166373] sd 2:0:2:0: [sdg] Write Protect is off
> [ 3572.221718] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
> [ 3572.286814] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> [ 3572.394250]  sdg: unknown partition table
> [ 3572.455839] sd 2:0:2:0: [sdg] Attached SCSI disk
> [ 3572.511006] sd 2:0:2:0: Attached scsi generic sg7 type 0
> [ 3578.075788] qlogicpti0: Aborting cmd for tgt[2] lun[0]
> [ 3578.137633] qlogicpti0: Aborting cmd for tgt[2] lun[1]
> [ 3696.467069] BUG: spinlock lockup on CPU#7, scsi_eh_2/1737, fffff8004d0f8060
> [ 3696.548319] Call Trace:
> [ 3696.577498]  [0000000000590e5c] _raw_spin_lock+0xfc/0x160
> [ 3696.642105]  [000000000069dc5c] _spin_lock_irqsave+0x1c/0x40
> [ 3696.709816]  [00000000005d5090] scsi_send_eh_cmnd+0x70/0x280
> [ 3696.777530]  [00000000005d52c4] scsi_eh_tur+0x24/0xa0
> [ 3696.837958]  [00000000005d5f8c] scsi_error_handler+0x2cc/0x3c0
> [ 3696.907783]  [000000000047676c] kthread+0x4c/0x80
> [ 3696.964032]  [000000000042707c] kernel_thread+0x3c/0x60
> [ 3697.026543]  [0000000000476694] kthreadd+0x174/0x200
> [ 3697.086064]   CPU[  6]: TSTATE[0000009980009602] TPC[000000000069dc40] TNPC[000000000069dc44] TASK[NULL:-1]
> [ 3697.202709]              TPC[_spin_lock_irqsave+0x0/0x40] O7[complete+0x14/0x60] I7[scsi_eh_done+0x24/0x40] RPC[qpti_intr+0]
> [ 3697.359973] * CPU[  7]: TSTATE[0000000080009625] TPC[0000000000427074] TNPC[0000000000427078] TASK[scsi_eh_2:1737]
> [ 3697.483975]              TPC[kernel_thread+0x34/0x60] O7[kernel_thread+0x8/0x60] I7[kthreadd+0x174/0x200] RPC[0x0]
> 
> -- 
> Meelis Roos (mroos@ut.ee)      http://www.cs.ut.ee/~mroos/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: qlogicpti driver hangs the machine

[Meelis Roos]

> What kernel version is this trace from?

Oops, forgot - todays 2.6.27-rc7+git.

-- 
Meelis Roos (mroos@linux.ee)

Re: qlogicpti driver hangs the machine

[David Miller]

From: Matthew Wilcox <matthew@wil.cx>
Date: Tue, 23 Sep 2008 09:59:18 -0600

> On Tue, Sep 23, 2008 at 06:06:03PM +0300, Meelis Roos wrote:
> > I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in 
> > Sparcstation 5 (sparc32) with etch and sarge installers, then tried 
> > another in E3500. Both tries on SS5 hung when loadin qlogicpti module. 
> > E3500 conf doesn't autoload SBus drivers and the card was properly 
> > inserted only after installation, so I tried to modprobe qlogicpti after 
> > booting up. This also results in a hang but outputs that much on console 
> > before hanging. Maybe the error handling spinlock lockup gives a clue to 
> > someone...
> 
> What kernel version is this trace from?

FWIW, I personally haven't tested this driver in a long time, did
anyone give it at least a simple test when you made those
->slave_configure changes to this driver?

I'm talking about this one:

commit 9ec76fbf7d6da3e98070a7059699d0ca019b0c9b
Author: Matthew Wilcox <matthew@wil.cx>
Date:   Mon Oct 1 08:07:52 2007 -0600

    [SCSI] qlogicpti: Add a slave_configure method
    
    By configuring targets in slave_configure, we can eliminate a shadow
    queuecommand, a shadow scsi_done, a write to the host template, abuse of
    SCp->Message and SCp->Status, a use of kmap_atomic() and sniffing the
    results of INQUIRY.
    
    Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
    Acked-by: David S. Miller <davem@davemloft.net>
    Signed-off-by: James Bottomley <James.Bottomley@SteelEye.com>

Re: qlogicpti driver hangs the machine

[Matthew Wilcox]

On Tue, Sep 23, 2008 at 09:25:13PM -0700, David Miller wrote:
> From: Matthew Wilcox <matthew@wil.cx>
> Date: Tue, 23 Sep 2008 09:59:18 -0600
> 
> > On Tue, Sep 23, 2008 at 06:06:03PM +0300, Meelis Roos wrote:
> > > I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in 
> > > Sparcstation 5 (sparc32) with etch and sarge installers, then tried 
> > > another in E3500. Both tries on SS5 hung when loadin qlogicpti module. 
> > > E3500 conf doesn't autoload SBus drivers and the card was properly 
> > > inserted only after installation, so I tried to modprobe qlogicpti after 
> > > booting up. This also results in a hang but outputs that much on console 
> > > before hanging. Maybe the error handling spinlock lockup gives a clue to 
> > > someone...
> > 
> > What kernel version is this trace from?
> 
> FWIW, I personally haven't tested this driver in a long time, did
> anyone give it at least a simple test when you made those
> ->slave_configure changes to this driver?

Not that I know of, but I do have an incriminating email from you where
you promise to pick up the pieces if anything explodes ;-)

There've also been commits from Jens, Boaz and Tomo that could plausibly
be related (I think we can discount the one from Adrian).  Meelis, can
you run through the various patches to qlogicpti, backing them out one
at a time?  I worry that some infrastructure may have been taken away by
followups to these patches, so it may not be feasible.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: qlogicpti driver hangs the machine

[David Miller]

From: Matthew Wilcox <matthew@wil.cx>
Date: Tue, 23 Sep 2008 22:42:11 -0600

> On Tue, Sep 23, 2008 at 09:25:13PM -0700, David Miller wrote:
> > From: Matthew Wilcox <matthew@wil.cx>
> > Date: Tue, 23 Sep 2008 09:59:18 -0600
> > 
> > > On Tue, Sep 23, 2008 at 06:06:03PM +0300, Meelis Roos wrote:
> > > > I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in 
> > > > Sparcstation 5 (sparc32) with etch and sarge installers, then tried 
> > > > another in E3500. Both tries on SS5 hung when loadin qlogicpti module. 
> > > > E3500 conf doesn't autoload SBus drivers and the card was properly 
> > > > inserted only after installation, so I tried to modprobe qlogicpti after 
> > > > booting up. This also results in a hang but outputs that much on console 
> > > > before hanging. Maybe the error handling spinlock lockup gives a clue to 
> > > > someone...
> > > 
> > > What kernel version is this trace from?
> > 
> > FWIW, I personally haven't tested this driver in a long time, did
> > anyone give it at least a simple test when you made those
> > ->slave_configure changes to this driver?
> 
> Not that I know of, but I do have an incriminating email from you where
> you promise to pick up the pieces if anything explodes ;-)

And here I am :-)

> There've also been commits from Jens, Boaz and Tomo that could plausibly
> be related (I think we can discount the one from Adrian).  Meelis, can
> you run through the various patches to qlogicpti, backing them out one
> at a time?  I worry that some infrastructure may have been taken away by
> followups to these patches, so it may not be feasible.

I know, I looked at those too.  Backing many of them out is infeasible,
especially the scatterlist ones.

I'm about to stick a qlogicpti card into my ultra2 and try to reproduce
this.

Re: qlogicpti driver hangs the machine

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 23 Sep 2008 21:53:02 -0700 (PDT)

> I'm about to stick a qlogicpti card into my ultra2 and try to reproduce
> this.

Unfortunately, the driver works just fine for me, in fact it's
checking the root filesystem right now :-)

Meelis, if Matthew isn't able to give you some debugging patches or
other stuff to try, I'll try to get this fixed when I get back from
the netfilter workshop in Paris in 2 weeks.

Re: qlogicpti driver hangs the machine

[Boaz Harrosh]

David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 23 Sep 2008 21:53:02 -0700 (PDT)
> 
>> I'm about to stick a qlogicpti card into my ultra2 and try to reproduce
>> this.
> 
> Unfortunately, the driver works just fine for me, in fact it's
> checking the root filesystem right now :-)
> 
> Meelis, if Matthew isn't able to give you some debugging patches or
> other stuff to try, I'll try to get this fixed when I get back from
> the netfilter workshop in Paris in 2 weeks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small
problem.

In the load_cmd() there is a compound loop where the first 4 sg's are
set then the rest are set into a memory structure in group of 7 sg's.

Well the second 7-group and on is a bug because sg pointer does not advance.
This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed
---
git diff --stat -p
 drivers/scsi/qlogicpti.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/scsi/qlogicpti.c b/drivers/scsi/qlogicpti.c
index 4a1cf63..9053508 100644
--- a/drivers/scsi/qlogicpti.c
+++ b/drivers/scsi/qlogicpti.c
@@ -914,6 +914,7 @@ static inline int load_cmd(struct scsi_cmnd *Cmnd, struct Command_Entry *cmd,
 				ds[i].d_count = sg_dma_len(s);
 			}
 			sg_count -= n;
+			sg = s;
 		}
 	} else {
 		cmd->dataseg[0].d_base = 0;

Re: qlogicpti driver hangs the machine

[David Miller]

From: Boaz Harrosh <bharrosh@panasas.com>
Date: Wed, 24 Sep 2008 12:00:22 +0300

> Subject: [PATCH] fix fall out of sg-chaining patch
> 
> I've reviewed all patches since Matthew's, and I find one small
> problem.
> 
> In the load_cmd() there is a compound loop where the first 4 sg's are
> set then the rest are set into a memory structure in group of 7 sg's.
> 
> Well the second 7-group and on is a bug because sg pointer does not advance.
> This is a fall out from Jens's patch.
> 
> I'm not sure if this is it but it should be fixed

Definitely a bug, and I looked at the patch that addded this code,
specifically audited this stuff, and didn't notice the problem. :-)

Meelis, see if this patch helps out your case, thanks!

> ---
> git diff --stat -p
>  drivers/scsi/qlogicpti.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/scsi/qlogicpti.c b/drivers/scsi/qlogicpti.c
> index 4a1cf63..9053508 100644
> --- a/drivers/scsi/qlogicpti.c
> +++ b/drivers/scsi/qlogicpti.c
> @@ -914,6 +914,7 @@ static inline int load_cmd(struct scsi_cmnd *Cmnd, struct Command_Entry *cmd,
>  				ds[i].d_count = sg_dma_len(s);
>  			}
>  			sg_count -= n;
> +			sg = s;
>  		}
>  	} else {
>  		cmd->dataseg[0].d_base = 0;
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: qlogicpti driver hangs the machine

[Meelis Roos]

> Unfortunately, the driver works just fine for me, in fact it's
> checking the root filesystem right now :-)

Hmm, it might to do with the disk that I tried with qlogicpti's on both 
systems. It's an externa SUN4.2G disk in unipack enclosure. I don't know 
about its health, only that it shows up in probe-scsi-all PROM command.

-- 
Meelis Roos (mroos@ut.ee)      http://www.cs.ut.ee/~mroos/

Re: qlogicpti driver hangs the machine

[Meelis Roos]

> Meelis, see if this patch helps out your case, thanks!

Will try soon.

-- 
Meelis Roos (mroos@ut.ee)      http://www.cs.ut.ee/~mroos/

Re: qlogicpti driver hangs the machine

[Meelis Roos]

> Subject: [PATCH] fix fall out of sg-chaining patch
> 
> I've reviewed all patches since Matthew's, and I find one small
> problem.
> 
> In the load_cmd() there is a compound loop where the first 4 sg's are
> set then the rest are set into a memory structure in group of 7 sg's.
> 
> Well the second 7-group and on is a bug because sg pointer does not advance.
> This is a fall out from Jens's patch.
> 
> I'm not sure if this is it but it should be fixed

This definitely helps,
# modprobe qlogicpti
[  271.742828] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
[  272.168804] qlogicpti0: [Fast Wide, using single ended interface]
[  272.243809] scsi3 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
[  272.835116] scsi 3:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
[  272.942031] sd 3:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[  273.026123] sd 3:0:2:0: [sdg] Write Protect is off
[  273.087748] sd 3:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  273.198290] sd 3:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[  273.282354] sd 3:0:2:0: [sdg] Write Protect is off
[  273.342641] sd 3:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  273.450154]  sdg: unknown partition table
[  273.511662] sd 3:0:2:0: [sdg] Attached SCSI disk
[  273.566006] sd 3:0:2:0: Attached scsi generic sg7 type 0
[  279.133183] qlogicpti0: Aborting cmd for tgt[2] lun[0]
[  279.193222] qlogicpti0: Aborting cmd for tgt[2] lun[1]
[  279.259841] qlogicpti0: Resetting SCSI bus!

and here it finishes and sdg seems to be working fine. There are two 
strange command aborts followed by a bus reset, but it recovers and 
seems to work (reading whole disc with dd seems to work at least).

Thanks!

-- 
Meelis Roos (mroos@ut.ee)      http://www.cs.ut.ee/~mroos/

Re: qlogicpti driver hangs the machine

[David Miller]

From: Meelis Roos <mroos@ut.ee>
Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)

> > Subject: [PATCH] fix fall out of sg-chaining patch
> > 
> > I've reviewed all patches since Matthew's, and I find one small
> > problem.
> > 
> > In the load_cmd() there is a compound loop where the first 4 sg's are
> > set then the rest are set into a memory structure in group of 7 sg's.
> > 
> > Well the second 7-group and on is a bug because sg pointer does not advance.
> > This is a fall out from Jens's patch.
> > 
> > I'm not sure if this is it but it should be fixed
> 
> This definitely helps,

Can we get this bug fix merged if it isn't already?

Thanks.

Re: qlogicpti driver hangs the machine

[Jens Axboe]

On Sun, Sep 28 2008, David Miller wrote:
> From: Meelis Roos <mroos@ut.ee>
> Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)
> 
> > > Subject: [PATCH] fix fall out of sg-chaining patch
> > > 
> > > I've reviewed all patches since Matthew's, and I find one small
> > > problem.
> > > 
> > > In the load_cmd() there is a compound loop where the first 4 sg's are
> > > set then the rest are set into a memory structure in group of 7 sg's.
> > > 
> > > Well the second 7-group and on is a bug because sg pointer does not advance.
> > > This is a fall out from Jens's patch.
> > > 
> > > I'm not sure if this is it but it should be fixed
> > 
> > This definitely helps,
> 
> Can we get this bug fix merged if it isn't already?

I'll take care of it.

-- 
Jens Axboe

Re: qlogicpti driver hangs the machine

[James Bottomley]

On Mon, 2008-09-29 at 08:33 +0200, Jens Axboe wrote:
> On Sun, Sep 28 2008, David Miller wrote:
> > From: Meelis Roos <mroos@ut.ee>
> > Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)
> > 
> > > > Subject: [PATCH] fix fall out of sg-chaining patch
> > > > 
> > > > I've reviewed all patches since Matthew's, and I find one small
> > > > problem.
> > > > 
> > > > In the load_cmd() there is a compound loop where the first 4 sg's are
> > > > set then the rest are set into a memory structure in group of 7 sg's.
> > > > 
> > > > Well the second 7-group and on is a bug because sg pointer does not advance.
> > > > This is a fall out from Jens's patch.
> > > > 
> > > > I'm not sure if this is it but it should be fixed
> > > 
> > > This definitely helps,
> > 
> > Can we get this bug fix merged if it isn't already?
> 
> I'll take care of it.

Actually, I already have it in rc fixes which are due out this morning.

James