fadvise and small random reads

fadvise and small random reads

[Animesh K Trivedi1]

Hi all,

I'm running the random read test on a 64MB file with 4K block size. The
file is small enough to fit in the buffer cache of the system (8 GB DRAM).
But default fadvise calls to OS pass two flags (POSIX_FADV_DONTNEED, and
POSIX_FADV_RANDOM). With these two settings OS does not try to cache any
reads and they all go to the disk. I get ~280 IOPS with 99% disk
utilization. And that is fine. But even when I turn fadvise_hint=0, it
shows the similar behaviour. Upon further investigation I noticed that
irrespective of settings, POSIX_FADV_DONTNEED is always passed to OS
(filesetup.c, line 361). Should not it also be restricted when
--fadvise_hint is set to zero ?

If I don't pass any hint to OS, then after stabilizing whole  file is
cached in the buffer and IOPS reaches upto 700+K (which I was expecting for
small file sizes). Here is the snipped sample output :

fio --fadvise_hint=0 --ioengine=psync --direct=0 --rw=randread --bs=4K
--size=64M --numjob=1 --runtime=60 --time_based --group_reporting
--name=file
file: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=psync, iodepth=1
fio 1.54
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [3043M/0K /s] [761K/0  iops] [eta 00m:00s]
file: (groupid=0, jobs=1): err= 0: pid=3820
  read : io=121156MB, bw=2019.3MB/s, iops=516931 , runt= 60000msec
......
Disk stats (read/write):
  sda: ios=7742/82, merge=0/4060, ticks=34608/9112, in_queue=43712,
util=32.27%

Am I missing something ?

cheers,
--
Animesh

PS ~  I am not on the list so please cc me in the reply.

Re: fadvise and small random reads

[Jens Axboe]

On 2011-05-20 16:42, Animesh K Trivedi1 wrote:
> 
> Hi all,
> 
> I'm running the random read test on a 64MB file with 4K block size. The
> file is small enough to fit in the buffer cache of the system (8 GB DRAM).
> But default fadvise calls to OS pass two flags (POSIX_FADV_DONTNEED, and
> POSIX_FADV_RANDOM). With these two settings OS does not try to cache any
> reads and they all go to the disk. I get ~280 IOPS with 99% disk
> utilization. And that is fine. But even when I turn fadvise_hint=0, it
> shows the similar behaviour. Upon further investigation I noticed that
> irrespective of settings, POSIX_FADV_DONTNEED is always passed to OS
> (filesetup.c, line 361). Should not it also be restricted when
> --fadvise_hint is set to zero ?

FADV_DONTNEED is used to invalidate the mapping cache so that buffered
runs are "identical" and not influenced by previous cache hotness of the
page cache.

FADV_DONTNEED should not have any effects on the caching later done,
whereas FADV_RANDOM definitely will impact caching or read-ahead.

> If I don't pass any hint to OS, then after stabilizing whole  file is
> cached in the buffer and IOPS reaches upto 700+K (which I was expecting for
> small file sizes). Here is the snipped sample output :
> 
> fio --fadvise_hint=0 --ioengine=psync --direct=0 --rw=randread --bs=4K
> --size=64M --numjob=1 --runtime=60 --time_based --group_reporting
> --name=file
> file: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=psync, iodepth=1
> fio 1.54
> Starting 1 process
> Jobs: 1 (f=1): [r] [100.0% done] [3043M/0K /s] [761K/0  iops] [eta 00m:00s]
> file: (groupid=0, jobs=1): err= 0: pid=3820
>   read : io=121156MB, bw=2019.3MB/s, iops=516931 , runt= 60000msec
> ......
> Disk stats (read/write):
>   sda: ios=7742/82, merge=0/4060, ticks=34608/9112, in_queue=43712,
> util=32.27%
> 
> Am I missing something ?

You can always disable the cache clear hint with invalidate=0.

-- 
Jens Axboe