Re: [BUG] 2.6.39.1 crash in scsi_dispatch_cmd()

Roland Dreier
2011-07-06T08:07:06Z

> > I booted with slub_debug=FZUP and the 6b pattern in RAX pretty much
> > does prove that this is a use-after-free issue. Any thoughts about
> > how to pin this down before I muddle on in my lowbrow way?
>
> Alan Stern came up with a patch that could fix this:
>
> http://marc.info/?l=linux-kernel&m=130963676907731&w=2

Thanks! It seems my crash is actually a different problem (or perhaps
the same problem in different code), since I'm not going through SCSI
directly but rather a dm-multipath device on top of SCSI disks. But
it does seem that it is another case of the block queue elevator
getting freed while requests can still be submitted.

In my case, to reproduce this I have to hold the multipath device file
open with something like "cat > /dev/dm-X" and then kill the
underlying drive. What I think is happening (although I haven't
traced all the layers to be sure) is that then the multipath daemon
notices that all the paths to the disk are lost and tries to kill the
multipath device, which ends up in dm.c:dm_destroy().

This ends up in blk_cleanup_queue() which frees q->elevator, which of
course leads to the crash.

What's not clear to me is how things are supposed to work. It seems
that the dm stuff at least is missing a lot of required reference
counting, to make sure that some structure sticks around to reject IOs
to the device after it is destroy but while it is still open. But I
don't understand why people don't hit this more since it is completely
reproducible for me with a fairly normal setup (hot-remove a multipath
device that some process has open).

Alan Stern's patch looks a bit fishy -- the scsi_free_queue() is moved
earlier than the

/* cause the request function to reject all I/O requests */
sdev->request_queue->queuedata = NULL;

which seems to leave a small window where the use-after-free can
happen, and it's not clear to me why the scsi_free_queue() has to move
at all.

Thanks,
- R.