Fio ignoring runtime after an error?

Fio ignoring runtime after an error?

[Carl Zwanzig]

Hello,

I think I've run across a case where, during a time-limited test, a soft error occurred on the device and fio continued to run -way- past (overnight) the time limit.

The rough scenario is:
fio test - ok
fio test - ok
fio test - starts
  soft error
fio never ends (and is still pushing traffic as reported by the disk system)

I haven't had a chance to dive into the code yet (fio 2.0.14, linux 2.6.32-279.14.1.el6.x86_64), but can anyone think of a reason why this would happen?

The command line looks like:
fio --name='raid0;16;randread;8;16' --rw=randread --filename=/dev/ethdrv/e107.1 --iodepth=16 --bs=8k --rwmixread=70 --norandommap --group_reporting --time_based --minimal --direct=1 --fill_device=1 --ioengine=libaio --ramp_time=5 --runtime=35 --randrepeat=0

Hopefully, I'll get the clue to inject an error into the disk system and run this down, but that'll take time I don't really have.

Thanks,

z!

Re: Fio ignoring runtime after an error?

[Greg Freemyer]

On Tue, Apr 16, 2013 at 1:54 PM, Carl Zwanzig <cpz@coraid.com> wrote:
> Hopefully, I'll get the clue to inject an error into the disk system and run this down, but that'll take time I don't really have.

hdparm --make-bad-sector   can be used to inject an error in a ata drive.

See http://lwn.net/Articles/269721/ for details

Greg

Re: Fio ignoring runtime after an error?

[Jens Axboe]

On Tue, Apr 16 2013, Carl Zwanzig wrote:
> Hello,
> 
> I think I've run across a case where, during a time-limited test, a soft error occurred on the device and fio continued to run -way- past (overnight) the time limit.
> 
> The rough scenario is:
> fio test - ok
> fio test - ok
> fio test - starts
>   soft error
> fio never ends (and is still pushing traffic as reported by the disk system)
> 
> I haven't had a chance to dive into the code yet (fio 2.0.14, linux 2.6.32-279.14.1.el6.x86_64), but can anyone think of a reason why this would happen?
> 
> The command line looks like:
> fio --name='raid0;16;randread;8;16' --rw=randread --filename=/dev/ethdrv/e107.1 --iodepth=16 --bs=8k --rwmixread=70 --norandommap --group_reporting --time_based --minimal --direct=1 --fill_device=1 --ioengine=libaio --ramp_time=5 --runtime=35 --randrepeat=0
> 
> Hopefully, I'll get the clue to inject an error into the disk system and run this down, but that'll take time I don't really have.

What kind of error? For fill_device=1, fio will clear ENOSPC, but should
not clear other errors. Apart from that, there should not be any
interactions between errors and runtime/time_based.

-- 
Jens Axboe