Re: Every 13 hours sees a spike in fio CPU consumption, and a drop in IOPS/throughput

[Jens Axboe <jens.axboe@oracle.com>]

On Fri, Oct 30 2009, Chris Worley wrote:
> On Fri, Oct 30, 2009 at 9:20 AM, Chris Worley <worleys@gmail.com> wrote:
> > I'm running fio benchmarks for 120 hours:
> >
> > fio --rw=randrw --bs=4k --rwmixread=60 --numjobs=64 --iodepth=64
> > --sync=0 --direct=1 --randrepeat=0 --ioengine=libaio
> > --filename=/dev/sde --filename=/dev/sdf --name=test --loops=10000
> > --size=322122547200 --runtime=432000 --group_reporting
> >
> > The fio threads are generally reporting 100K IOPS and each of the 64
> > fio threads uses less than 10% CPU.
> >
> > But, every 13 hours (nearly to the minute), for ~200 seconds, the FIO
> > threads start consuming large amounts of CPU, and the IOPS drop to
> > ~40K:
> 
> Josh is brilliant:
> 
> (322122547200[bytes per test]*64[threads])/(4096[block
> size]*100000[IOPS]*3600[seconds per hour])=13 hours
> 
> ...it's restarting the next loop and recalculating the random maps
> every 13 hours

Awesome! It should just give it or re-clean the random map, that is
likely where the bug is. Looking forward to a tested patch :-)

-- 
Jens Axboe