Re: A unresponsive file system can hang all I/O in the system on linux-2.6.23-rc6 (dirty_thresh problem?)

Daniel Phillips
2007-09-29T01:27:47Z

On Friday 28 September 2007 06:35, Peter Zijlstra wrote:
> ,,,it would be grand (and dangerous) if we could provide for a
> button that would just kill off all outstanding pages against a dead
> device.

Substitute "resources" for "pages" and you begin to get an idea of how
tricky that actually is. That said, this is exactly what we have done
with ddsnap, for the simple reason that our users, now emboldened by
being able to stop or terminate the user space part, felt justified in
expecting that the system continue as if nothing had happened, and
furthermore, be able to restart ddsnap without a hiccup. (Otherwise
known as a sysop's diety-given right to kill.)

So this is what we do in the specific case of ddsnap:

* When we detect some nasty state change such as our userspace
control daemon disappearing on us, we go poking around and
explicitly release every semaphore that the device driver could
possibly wait on forever (interestingly they are all in our own
driver except for BKL, which is just an artifact of device mapper
not having gone over to unlock_ioctl for no good reason that I
know of).

* Then at the points were the driver falls through some lock thus
released, we check our "ready" flag, and if it indicates "busted",
proceed with wherever cleanup is needed at that point.

Does not sound like an approach one would expect to work reliably, does
it? But there just may be some general principle to be ferretted out
here. (Anyone who has ideas on how bits of this procedure could be
abstracted, please do not hesitate to step boldly forth into the
limelight.)

Incidentally, only a small subset of locks needed special handling as
above. Most can be shown to have no way to block forever, short of an
outright bug.

I shudder to think how much work it would be to bring every driver in
the kernel up to such a standard, particularly if user space components
are involved, as with USB. On the other hand, every driver fixed is
one less driver that sucks. The next one to emerge from the pipeline
will most likely be NBD, which we have been working on in fits and
starts for a while. Look for it to morph into "ddbd", with cross-node
distributed data awareness, in addition to perforning its current job
without deadlocking.

Regards,

Daniel