Re: [patch 4/5] fail-injection capability for disk IO

[Jens Axboe <axboe@suse.de>]

On Wed, Aug 23 2006, Ric Wheeler wrote:
> Jens Axboe wrote:
> >On Wed, Aug 23 2006, Andrew Morton wrote:
> >
> >>On Wed, 23 Aug 2006 14:03:55 +0200
> >>Jens Axboe <axboe@suse.de> wrote:
> >>
> >>
> >>>On Wed, Aug 23 2006, Akinobu Mita wrote:
> >>>
> >>>>This patch provides fail-injection capability for disk IO.
> >>>>
> >>>>Boot option:
> >>>>
> >>>>	fail_make_request=<probability>,<interval>,<times>,<space>
> >>>>
> >>>>	<probability>
> >>>>
> >>>>		specifies how often it should fail in percent.
> >>>>
> >>>>	<interval>
> >>>>
> >>>>		specifies the interval of failures.
> >>>>
> >>>>	<times>
> >>>>
> >>>>		specifies how many times failures may happen at most.
> >>>>
> >>>>	<space>
> >>>>
> >>>>		specifies the size of free space where disk IO can be issued
> >>>>		safely in bytes.
> >>>>
> >>>>Example:
> >>>>
> >>>>	fail_make_request=100,10,-1,0
> >>>>
> >>>>generic_make_request() fails once per 10 times.
> >>>
> >>>Hmm dunno, seems a pretty useless feature to me.
> >>
> >>We need it.  What is the FS/VFS/VM behaviour in the presence of IO
> >>errors?  Nobody knows, because we rarely test it.  Those few times where
> >>people _do_ test it (the hard way), bad things tend to happen.  reiserfs
> >>(for example) likes to go wobble, wobble, wobble, BUG.
> >
> >
> >You misunderstood me - a global parameter is useless, as it makes it
> >pretty impossible for people to use this for any sort of testing (unless
> >it's very specialized). I didn't say a feature to test io errors was
> >useless!
> >
> >
> >>>Wouldn't it make a lot
> >>>more sense to do this per-queue instead of a global entity?
> >>
> >>Yes, I think so.  /sys/block/sda/sda2/make-it-fail.
> >
> >
> >Precisely.
> >
> 
> I think that this is very useful for testing file systems.
> 
> What this will miss is the error path through the lower levels of the IO 
> path (i.e., the libata/SCSI error handling confusion that Mark Lord has 
> been working on patches for would need some error injection at or below 
> the libata level).
> 
> We currently test this whole path with either weird fault injection gear 
> to hit the s-ata bus or the old fashion pile of moderately flaky disks 
> that we try hard not to fix or totally kill.
> 
> It would be really useful to get something (target mode SW disk? libata 
> or other low level error injection?) to test this whole path in software...

Yes, this approach only tests the layer(s) above the device. To simulate
hardware failure or timeouts, I _think_ scsi_debug can already help you
quite a bit. If not, it should be easy enough to extend do add these
sorts of things.

-- 
Jens Axboe