Re: [2.6.27.25] Hang in SCSI sync cache when a disk is removed--?

[Paul Smith <paul@mad-scientist.net>]

On Thu, 2009-07-02 at 10:41 -0700, Mike Anderson wrote:
> > As I mentioned, when we pull one of the disks from the EXP3000 the IO
> > subsystem completely hangs.  Since we're running on a ramdisk this
> > doesn't hang our system completely, but any attempt to do any disk IO
> > thereafter hangs, so we have to power-cycle the blade (because reboot
> > tries to write to the disks).  This quite reproducible in our
> > environment BUT it is very timing-sensitive, as shown below.  If we
> > enable too much logging, etc. it goes away.
> 
> Have you tried a minimum level of logging like the following without the
> error going away?
> "sysctl -w dev.scsi.logging_level=4100"

We've been enabling some logging at the mptlinux driver level, but not
the generic SCSI level.  We'll give this a try.

> Can you run "cat /sys/class/scsi_host/*/state" when you are in the hung
> state?

We'll try this as well.  One thing we did discover was that the device
entry in /sys/class/scsi_device/* is already gone when the hang occurs
so we can't retrieve the state of the device that way.

> If the host is in recovery no IOs will move forward. I assume if you can
> get a run with the 4100 level of logging it will show a host reset sent,
> but no waking up host to restart (unless the reset is being generated for
> other reasons outside of the scsi error handler).

This seems likely but the question is, why isn't it waking up again?

Unfortunately one of the folks working on this with me left for the US
holiday weekend so we'll have to take it up again on Monday morning.
Thanks for your reply, Mike!