memory caching

memory caching

[N. Harake]

I would like to disable completely the caching in memory, trying to measure real throughput read/write and iops on filesystem (not raw access).
Other than direct=1 is there any flag I could use to obtain such complete bypass caching?

Regards
H. N. Harake

Re: memory caching

[N. Harake]

I noticed when I am using --sync=1, IOPS go down to 15K and throughput to
53MB (That�s horrible for the HW I am using), removing --sync I got 150K
IOPS and almost 800MB, which make more sense. What does --sync does, and
hope what I am doing should be enough to avoid any kind of memory caching..

Regards
H. N. Harake

fio --name=job --directory=/lvm --size=1G --iodepth=1 --sync=1
--filesize=1G --direct=1 --numjobs=64 --bs=4096 --rw=randwrite
--group_reporting

On 3/15/11 9:27 AM, "N. Harake" <hussein@cscs.ch> wrote:

>
>I would like to disable completely the caching in memory, trying to
>measure real throughput read/write and iops on filesystem (not raw
>access).
>Other than direct=1 is there any flag I could use to obtain such complete
>bypass caching?
>
>Regards
>H. N. Harake
>--
>To unsubscribe from this list: send the line "unsubscribe fio" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: memory caching

[Jens Axboe]

On 2011-03-15 13:36, N. Harake wrote:
> 
> I noticed when I am using --sync=1, IOPS go down to 15K and throughput to
> 53MB (That�s horrible for the HW I am using), removing --sync I got 150K
> IOPS and almost 800MB, which make more sense. What does --sync does, and
> hope what I am doing should be enough to avoid any kind of memory caching..
> 
> Regards
> H. N. Harake
> 
> fio --name=job --directory=/lvm --size=1G --iodepth=1 --sync=1
> --filesize=1G --direct=1 --numjobs=64 --bs=4096 --rw=randwrite
> --group_reporting

If your goal is to maximize iops for testing, use the raw block device.
Use direct=1, and ioengine=libaio. Set depth large, ala:

--iodepth=128 --iodepth_batch_complete=8 --iodepth_batch_submit=8


-- 
Jens Axboe

RE: memory caching

[N. Harake]

Thanks for your reply, indeed my interest to get the max iops but on a filesystem not a raw device, 
I understand the penalty but that would be more realstic for production systems. I will try these changes and keep you updated .

All the best
H. N. Harake

________________________________________
From: Jens Axboe [jaxboe@fusionio.com]
Sent: Tuesday, March 15, 2011 9:28 PM
To: N. Harake
Cc: fio@vger.kernel.org
Subject: Re: memory caching

On 2011-03-15 13:36, N. Harake wrote:
>
> I noticed when I am using --sync=1, IOPS go down to 15K and throughput to
> 53MB (That¹s horrible for the HW I am using), removing --sync I got 150K
> IOPS and almost 800MB, which make more sense. What does --sync does, and
> hope what I am doing should be enough to avoid any kind of memory caching..
>
> Regards
> H. N. Harake
>
> fio --name=job --directory=/lvm --size=1G --iodepth=1 --sync=1
> --filesize=1G --direct=1 --numjobs=64 --bs=4096 --rw=randwrite
> --group_reporting

If your goal is to maximize iops for testing, use the raw block device.
Use direct=1, and ioengine=libaio. Set depth large, ala:

--iodepth=128 --iodepth_batch_complete=8 --iodepth_batch_submit=8


--
Jens Axboe

Re: memory caching

[Jens Axboe]

On 2011-03-15 21:46, N. Harake wrote:
> 
> Thanks for your reply, indeed my interest to get the max iops but on a
> filesystem not a raw device, I understand the penalty but that would
> be more realstic for production systems. I will try these changes and
> keep you updated .

It's always a good idea to do the raw block device test first, that'll
at least give you an idea of the best possible result. On a file system,
you'll get the best results with O_DIRECT if the file has been
pre-allocated first. Otherwise you end up punting to buffered IO for
filling new blocks.

So that would mean adding --overwrite=1. And get rid of the O_SYNC. If
your workload will utilize lots of processes or threads, use what you
already have with a high numjobs. But an libaio with fewer threads and
higher depth will get you better performance.

-- 
Jens Axboe

RE: memory caching

[N. Harake]

Things improved but maybe too much :), I was expecting a 100K IOPS not more

server1:~ # fio --name=job --directory=/lvm --size=1G --filesize=1G --iodepth=128 --iodepth_batch_complete=8 --iodepth_batch_submit=8 --ioengine=libaio --direct=1 --numjobs=64 --bs= 4096 --rw=randwrite --group_reporting
Jobs: 50 (f=50): [wwwwww_wwwwwww_wwwwwwwwww_wwwwwwwwwww_www_www_ww_www___ww_w__w_w] [98.3% done] [0K/570.8M /s] [0 /143K iops] [eta 00m:02s]

  write: io=65536MB, bw=584771KB/s, iops=146192 , runt=114761msec

H. Harake

________________________________________
From: Jens Axboe [jaxboe@fusionio.com]
Sent: Tuesday, March 15, 2011 9:28 PM
To: N. Harake
Cc: fio@vger.kernel.org
Subject: Re: memory caching

On 2011-03-15 13:36, N. Harake wrote:
>
> I noticed when I am using --sync=1, IOPS go down to 15K and throughput to
> 53MB (That¹s horrible for the HW I am using), removing --sync I got 150K
> IOPS and almost 800MB, which make more sense. What does --sync does, and
> hope what I am doing should be enough to avoid any kind of memory caching..
>
> Regards
> H. N. Harake
>
> fio --name=job --directory=/lvm --size=1G --iodepth=1 --sync=1
> --filesize=1G --direct=1 --numjobs=64 --bs=4096 --rw=randwrite
> --group_reporting

If your goal is to maximize iops for testing, use the raw block device.
Use direct=1, and ioengine=libaio. Set depth large, ala:

--iodepth=128 --iodepth_batch_complete=8 --iodepth_batch_submit=8


--
Jens Axboe

Re: memory caching

[Jens Axboe]

On 2011-03-15 21:54, N. Harake wrote:
> 
> Things improved but maybe too much :), I was expecting a 100K IOPS not more
> 
> server1:~ # fio --name=job --directory=/lvm --size=1G --filesize=1G --iodepth=128 --iodepth_batch_complete=8 --iodepth_batch_submit=8 --ioengine=libaio --direct=1 --numjobs=64 --bs= 4096 --rw=randwrite --group_reporting
> Jobs: 50 (f=50): [wwwwww_wwwwwww_wwwwwwwwww_wwwwwwwwwww_www_www_ww_www___ww_w__w_w] [98.3% done] [0K/570.8M /s] [0 /143K iops] [eta 00m:02s]
> 
>   write: io=65536MB, bw=584771KB/s, iops=146192 , runt=114761msec

If you use numjobs=64, you probably don't want 128 for io depth. Not
very realistic, and you'll likely end up running slower than you could.

-- 
Jens Axboe

Re: memory caching

[Jens Axboe]

On 2011-03-15 09:27, N. Harake wrote:
> 
> I would like to disable completely the caching in memory, trying to
> measure real throughput read/write and iops on filesystem (not raw
> access).  Other than direct=1 is there any flag I could use to obtain
> such complete bypass caching?

direct=1 is the one to use, this controls whether you go through the
page cache or not.

-- 
Jens Axboe