RE: 2.5.59-dcl2

RE: 2.5.59-dcl2

[Matt_Domsch]

> From: Mark Haverkamp [mailto:markh@osdl.org]
> I sent a bug report the the kernel bugzilla a while ago. It 
> got assigned
> to someone at IBM by default.  This is a problem we have been having
> between our megaraid cards and an IBM enclosure for some time.  I get
> around it by disabling report luns in my kernel configuration.  
> 
> Could you take a look at the bug and let me know if I've 
> included enough
> information to make sense.  I included debug output for the 
> 1.18 and 2.0 driver.
> 
> http://bugme.osdl.org/show_bug.cgi?id=183
> 
> or
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=183

Mark, your description in the bug appears accurate.  Here's what happens:

1) REPORT_LUNS is sent to the IBM enclosure, command times out.
2) scsi_unjam_host() runs, trying the error handlers...
3) megaraid abort handler, then the reset handler called (3 times), none of
which do anything at all because the command has already been issued to the
firmware, and the card itself isn't getting reset.  Both those routines
return failure.
4) so scsi_unjam_host() offlines the device
5) scsi_decide_disposition() sees device is offline and returns success,
freeing the command for future use again.
6) command eventually completes and the mega_rundoneq() calls
cmd->scsi_done() on it, but cmd has been filled with 0x5a's so it Oopses.


The scsi mid-layer shouldn't free a command that hasn't actually been
aborted/reset because it *could* come back from the firmware after the
timeout has expired, and the driver has a reference to it (need
refcounting...)  This could potentially lead to an exhaustion of the command
pool though, if a command *never* comes back.

How long does it take for the IBM enclosure to return REPORT LUNS?  Since
this works on aic7xxx within the timeout period, I'm guessing the megaraid
firmware takes a long time to deal with it since it's a pass-through device.

I believe there is a way to issue an adapter reset command to the megaraid
firmware, though neither driver series 1.18 or 2.00 do so presently.
Copying Atul for insight as to what effects this would have on the
controller and commands in flight...

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer, Lead Engineer, Architect
Dell Linux Solutions www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com

Re: 2.5.59-dcl2

[Mike Anderson]

Matt_Domsch@Dell.com [Matt_Domsch@Dell.com] wrote:
> > From: Mark Haverkamp [mailto:markh@osdl.org]
> > I sent a bug report the the kernel bugzilla a while ago. It 
> > got assigned
> > to someone at IBM by default.  This is a problem we have been having
> > between our megaraid cards and an IBM enclosure for some time.  I get
> > around it by disabling report luns in my kernel configuration.  
> > 
> > Could you take a look at the bug and let me know if I've 
> > included enough
> > information to make sense.  I included debug output for the 
> > 1.18 and 2.0 driver.
> > 
> > http://bugme.osdl.org/show_bug.cgi?id=183
> > 
> > or
> > 
> > http://bugzilla.kernel.org/show_bug.cgi?id=183
> 
> Mark, your description in the bug appears accurate.  Here's what happens:
> 
> 1) REPORT_LUNS is sent to the IBM enclosure, command times out.
> 2) scsi_unjam_host() runs, trying the error handlers...
> 3) megaraid abort handler, then the reset handler called (3 times), none of
> which do anything at all because the command has already been issued to the
> firmware, and the card itself isn't getting reset.  Both those routines
> return failure.
> 4) so scsi_unjam_host() offlines the device
> 5) scsi_decide_disposition() sees device is offline and returns success,
> freeing the command for future use again.
> 6) command eventually completes and the mega_rundoneq() calls
> cmd->scsi_done() on it, but cmd has been filled with 0x5a's so it Oopses.
> 
> 
> The scsi mid-layer shouldn't free a command that hasn't actually been
> aborted/reset because it *could* come back from the firmware after the
> timeout has expired, and the driver has a reference to it (need
> refcounting...)  This could potentially lead to an exhaustion of the command
> pool though, if a command *never* comes back.
> 

This is bad to return the command if the eh cannot cancel it from
the LLDD. Then again the eh routines provided by the driver are not very
useful. We could change the error handler to mark cmds that have and
have not been canceled and then at the end do something with the
un-canceled ones. The do something is unclear.

> How long does it take for the IBM enclosure to return REPORT LUNS?  Since
> this works on aic7xxx within the timeout period, I'm guessing the megaraid
> firmware takes a long time to deal with it since it's a pass-through device.
> 
> I believe there is a way to issue an adapter reset command to the megaraid
> firmware, though neither driver series 1.18 or 2.00 do so presently.
> Copying Atul for insight as to what effects this would have on the
> controller and commands in flight...
> 

It would be good to add a eh function that could cancel a command.

-andmike
--
Michael Anderson
andmike@us.ibm.com

RE: 2.5.59-dcl2

[Mark Haverkamp]

On Tue, 2003-01-28 at 21:35, Matt_Domsch@Dell.com wrote:
> > From: Mark Haverkamp [mailto:markh@osdl.org]
> > I sent a bug report the the kernel bugzilla a while ago. It 
> > got assigned
> > to someone at IBM by default.  This is a problem we have been having
> > between our megaraid cards and an IBM enclosure for some time.  I get
> > around it by disabling report luns in my kernel configuration.  
> > 
> > Could you take a look at the bug and let me know if I've 
> > included enough
> > information to make sense.  I included debug output for the 
> > 1.18 and 2.0 driver.
> > 
> > http://bugme.osdl.org/show_bug.cgi?id=183
> > 
> > or
> > 
> > http://bugzilla.kernel.org/show_bug.cgi?id=183
> 
> Mark, your description in the bug appears accurate.  Here's what happens:
> 
> 1) REPORT_LUNS is sent to the IBM enclosure, command times out.
> 2) scsi_unjam_host() runs, trying the error handlers...
> 3) megaraid abort handler, then the reset handler called (3 times), none of
> which do anything at all because the command has already been issued to the
> firmware, and the card itself isn't getting reset.  Both those routines
> return failure.
> 4) so scsi_unjam_host() offlines the device
> 5) scsi_decide_disposition() sees device is offline and returns success,
> freeing the command for future use again.
> 6) command eventually completes and the mega_rundoneq() calls
> cmd->scsi_done() on it, but cmd has been filled with 0x5a's so it Oopses.
> 
> 
> The scsi mid-layer shouldn't free a command that hasn't actually been
> aborted/reset because it *could* come back from the firmware after the
> timeout has expired, and the driver has a reference to it (need
> refcounting...)  This could potentially lead to an exhaustion of the command
> pool though, if a command *never* comes back.
> 
> How long does it take for the IBM enclosure to return REPORT LUNS?  Since
> this works on aic7xxx within the timeout period, I'm guessing the megaraid
> firmware takes a long time to deal with it since it's a pass-through device.

I'm not sure how long the report luns takes on the megaraid card.  As an
experiment I made the timeout about 30 seconds and the command still
timed out.  Is there a chance that the response is getting lost in the
adapter and doesn't get returned to the driver until another command is
issued that does complete?

> 
> I believe there is a way to issue an adapter reset command to the megaraid
> firmware, though neither driver series 1.18 or 2.00 do so presently.
> Copying Atul for insight as to what effects this would have on the
> controller and commands in flight...
> 
> Thanks,
> Matt
> 
> --
> Matt Domsch
> Sr. Software Engineer, Lead Engineer, Architect
> Dell Linux Solutions www.dell.com/linux
> Linux on Dell mailing lists @ http://lists.us.dell.com
-- 
Mark Haverkamp <markh@osdl.org>

Re: 2.5.59-dcl2

[Luben Tuikov]

Mike Anderson wrote:
>>
>>The scsi mid-layer shouldn't free a command that hasn't actually been
>>aborted/reset because it *could* come back from the firmware after the
>>timeout has expired, and the driver has a reference to it (need
>>refcounting...)  This could potentially lead to an exhaustion of the command
>>pool though, if a command *never* comes back.
>>
> 
> 
> This is bad to return the command if the eh cannot cancel it from
> the LLDD. Then again the eh routines provided by the driver are not very
> useful. We could change the error handler to mark cmds that have and
> have not been canceled and then at the end do something with the
> un-canceled ones. The do something is unclear.

1. Right, if the eh_abort_handler() (read: eh_cancel_command())
could *not* complete cancellation of the command, SCSI Core
should *not* repossess it.

OTOH, after successfull eh_device_reset() (use your imagination: UYI),
or eh_host_reset() (again UYI), LLDD is in its full right to *repossess*
all pending LLDD status commands. (See bottom of text.)

2. There are two ways to implement object states.  One is to ``mark them''
via an object variable which represents the state and the other is to
use containers* /representing/ the different states.

* Queues, lists, etc.

The first method is quite, quite trivial and worth no discussion except
to mention that it just adds more cruft to the object itself.

The other is what would normally be implemented in any self respecting program,
and has many, many advantages, e.g. one can use a priority queues, on top of it
being a state queue, etc.

For this reason what you'd want to do is use a per device variable
	struct scsi_device::struct list_head lost_cmds;
where
	struct scsi_cmnd::struct list_head list;    /* scsi_cmnd participates in queue lists */
is hooked. (The comment is copied verbatim from my patches, now in scsi-combined-2.5.)

When a command couldn't be cancelled (eh_* returned failure), you *could* move
the command into that queue and make SCSI Core be aware of it, e.g. when freeing
the device.  I said ``could'' since this is a policy issue and will have to be
decided by more than one knowlegable scsi developer of linux-scsi.

As to LLDD: after ABORT TASK TMF, no further responses should be sent from
the target device to the initiator port regarding that task.  The initiator port
is the LLDD or SCSI Core *if* LLDD does its own queuing.  ((Strictly speaking
the Initiator Port is always the LLDD, but if it does its own queuing it will have
to notice the ABORT TASK TMF.)) What this means is that SCSI Core *can* make some
assumptions.

-- 
Luben

RE: 2.5.59-dcl2

[Mukker, Atul]

What might be happening here is, FW is retrying the command for some
iterations and each of them times out. The total timeout is greater than the
expected window.

A SCSI trace would tell us exactly what is going on there.

Thanks
-Atul Mukker

-----Original Message-----
From: Matt_Domsch@Dell.com [mailto:Matt_Domsch@Dell.com]
Sent: Wednesday, January 29, 2003 12:35 AM
To: markh@osdl.org
Cc: andmike@us.ibm.com; linux-scsi@vger.kernel.org; atulm@lsil.com
Subject: RE: 2.5.59-dcl2


> From: Mark Haverkamp [mailto:markh@osdl.org]
> I sent a bug report the the kernel bugzilla a while ago. It 
> got assigned
> to someone at IBM by default.  This is a problem we have been having
> between our megaraid cards and an IBM enclosure for some time.  I get
> around it by disabling report luns in my kernel configuration.  
> 
> Could you take a look at the bug and let me know if I've 
> included enough
> information to make sense.  I included debug output for the 
> 1.18 and 2.0 driver.
> 
> http://bugme.osdl.org/show_bug.cgi?id=183
> 
> or
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=183

Mark, your description in the bug appears accurate.  Here's what happens:

1) REPORT_LUNS is sent to the IBM enclosure, command times out.
2) scsi_unjam_host() runs, trying the error handlers...
3) megaraid abort handler, then the reset handler called (3 times), none of
which do anything at all because the command has already been issued to the
firmware, and the card itself isn't getting reset.  Both those routines
return failure.
4) so scsi_unjam_host() offlines the device
5) scsi_decide_disposition() sees device is offline and returns success,
freeing the command for future use again.
6) command eventually completes and the mega_rundoneq() calls
cmd->scsi_done() on it, but cmd has been filled with 0x5a's so it Oopses.


The scsi mid-layer shouldn't free a command that hasn't actually been
aborted/reset because it *could* come back from the firmware after the
timeout has expired, and the driver has a reference to it (need
refcounting...)  This could potentially lead to an exhaustion of the command
pool though, if a command *never* comes back.

How long does it take for the IBM enclosure to return REPORT LUNS?  Since
this works on aic7xxx within the timeout period, I'm guessing the megaraid
firmware takes a long time to deal with it since it's a pass-through device.

I believe there is a way to issue an adapter reset command to the megaraid
firmware, though neither driver series 1.18 or 2.00 do so presently.
Copying Atul for insight as to what effects this would have on the
controller and commands in flight...

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer, Lead Engineer, Architect
Dell Linux Solutions www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com

RE: 2.5.59-dcl2

[Mark Haverkamp]

On Wed, 2003-01-29 at 11:41, Mukker, Atul wrote:
> What might be happening here is, FW is retrying the command for some
> iterations and each of them times out. The total timeout is greater than the
> expected window.
> 
> A SCSI trace would tell us exactly what is going on there.

Unfortunately we don't have a SCSI analyzer here.

Mark.
> 
> Thanks
> -Atul Mukker
> 
> -----Original Message-----
> From: Matt_Domsch@Dell.com [mailto:Matt_Domsch@Dell.com]
> Sent: Wednesday, January 29, 2003 12:35 AM
> To: markh@osdl.org
> Cc: andmike@us.ibm.com; linux-scsi@vger.kernel.org; atulm@lsil.com
> Subject: RE: 2.5.59-dcl2
> 
> 
> > From: Mark Haverkamp [mailto:markh@osdl.org]
> > I sent a bug report the the kernel bugzilla a while ago. It 
> > got assigned
> > to someone at IBM by default.  This is a problem we have been having
> > between our megaraid cards and an IBM enclosure for some time.  I get
> > around it by disabling report luns in my kernel configuration.  
> > 
> > Could you take a look at the bug and let me know if I've 
> > included enough
> > information to make sense.  I included debug output for the 
> > 1.18 and 2.0 driver.
> > 
> > http://bugme.osdl.org/show_bug.cgi?id=183
> > 
> > or
> > 
> > http://bugzilla.kernel.org/show_bug.cgi?id=183
> 
> Mark, your description in the bug appears accurate.  Here's what happens:
> 
> 1) REPORT_LUNS is sent to the IBM enclosure, command times out.
> 2) scsi_unjam_host() runs, trying the error handlers...
> 3) megaraid abort handler, then the reset handler called (3 times), none of
> which do anything at all because the command has already been issued to the
> firmware, and the card itself isn't getting reset.  Both those routines
> return failure.
> 4) so scsi_unjam_host() offlines the device
> 5) scsi_decide_disposition() sees device is offline and returns success,
> freeing the command for future use again.
> 6) command eventually completes and the mega_rundoneq() calls
> cmd->scsi_done() on it, but cmd has been filled with 0x5a's so it Oopses.
> 
> 
> The scsi mid-layer shouldn't free a command that hasn't actually been
> aborted/reset because it *could* come back from the firmware after the
> timeout has expired, and the driver has a reference to it (need
> refcounting...)  This could potentially lead to an exhaustion of the command
> pool though, if a command *never* comes back.
> 
> How long does it take for the IBM enclosure to return REPORT LUNS?  Since
> this works on aic7xxx within the timeout period, I'm guessing the megaraid
> firmware takes a long time to deal with it since it's a pass-through device.
> 
> I believe there is a way to issue an adapter reset command to the megaraid
> firmware, though neither driver series 1.18 or 2.00 do so presently.
> Copying Atul for insight as to what effects this would have on the
> controller and commands in flight...
> 
> Thanks,
> Matt
> 
> --
> Matt Domsch
> Sr. Software Engineer, Lead Engineer, Architect
> Dell Linux Solutions www.dell.com/linux
> Linux on Dell mailing lists @ http://lists.us.dell.com
-- 
Mark Haverkamp <markh@osdl.org>