Re: A unresponsive file system can hang all I/O in the system on linux-2.6.23-rc6 (dirty_thresh problem?)

[Daniel Phillips <phillips@phunq.net>]

On Friday 28 September 2007 06:35, Peter Zijlstra wrote:
> ,,,it would be grand (and dangerous) if we could provide for a
> button that would just kill off all outstanding pages against a dead
> device.

Substitute "resources" for "pages" and you begin to get an idea of how 
tricky that actually is.  That said, this is exactly what we have done 
with ddsnap, for the simple reason that our users, now emboldened by 
being able to stop or terminate the user space part, felt justified in 
expecting that the system continue as if nothing had happened, and 
furthermore, be able to restart ddsnap without a hiccup.  (Otherwise 
known as a sysop's diety-given right to kill.)

So this is what we do in the specific case of ddsnap:

   * When we detect some nasty state change such as our userspace
      control daemon disappearing on us, we go poking around and
      explicitly release every semaphore that the device driver could
      possibly wait on forever (interestingly they are all in our own
      driver except for BKL, which is just an artifact of device mapper
      not having gone over to unlock_ioctl for no good reason that I
      know of).

   * Then at the points were the driver falls through some lock thus
      released, we check our "ready" flag, and if it indicates "busted",
      proceed with wherever cleanup is needed at that point.

Does not sound like an approach one would expect to work reliably, does 
it?  But there just may be some general principle to be ferretted out 
here.  (Anyone who has ideas on how bits of this procedure could be 
abstracted, please do not hesitate to step boldly forth into the 
limelight.)

Incidentally, only a small subset of locks needed special handling as 
above.  Most can be shown to have no way to block forever, short of an 
outright bug.

I shudder to think how much work it would be to bring every driver in 
the kernel up to such a standard, particularly if user space components 
are involved, as with USB.  On the other hand, every driver fixed is 
one less driver that sucks.  The next one to emerge from the pipeline 
will most likely be NBD, which we have been working on in fits and 
starts for a while.  Look for it to morph into "ddbd", with cross-node 
distributed data awareness, in addition to perforning its current job 
without deadlocking.

Regards,

Daniel