From: Ankit Jain <jankit@suse.de>
To: Christoph Hellwig <hch@infradead.org>
Cc: Dave Chinner <david@fromorbit.com>,
Al Viro <viro@zeniv.linux.org.uk>,
bcrl@kvack.org, linux-fsdevel@vger.kernel.org,
linux-aio@kvack.org, linux-kernel@vger.kernel.org,
Jan Kara <jack@suse.cz>
Subject: Re: [RFC][PATCH] Make io_submit non-blocking
Date: Fri, 27 Jul 2012 01:22:10 +0530 [thread overview]
Message-ID: <50119FEA.4060806@suse.de> (raw)
In-Reply-To: <20120724225026.GA21691@infradead.org>
[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]
On 07/25/2012 04:20 AM, Christoph Hellwig wrote:
> On Wed, Jul 25, 2012 at 08:31:10AM +1000, Dave Chinner wrote:
>> FWIW, if you are going to change generic code, you need to present
>> results for other filesystems as well (xfs, btrfs are typical), as
>> they may not have the same problems as ext4 or react the same way to
>> your change. The result might simply be "it is 20% slower"....
>
> And most importantly block devices, as they are one of the biggest
> use cases of AIO. With an almost no-op get_blocks callback I can't
> see how this change would provide any gain there.
I tried running fio against a block device, disk partition and a
ramdisk. I ran this with a single job though. For disks, bandwidth
seems to stay nearly the same with submit latencies getting better.
And for ramdisk, bandwidth also sees improvement. I should probably
be doing better tests, any suggestions on what or how I can test?
For block devices, if the patch doesn't make it worse, at least, then
that should be good enough?
------ disk -------
submit latencies(usec)
B/w iops runtime min max avg std dev
Read :
Old: 417335 B/s 101 252668msec 4 231 40.03 21.66
New: 419099 B/s 102 251282msec 0 169 8.20 6.95
Write:
Old: 412667 B/s 100 252668msec 3 272 47.65 24.58
New: 415481 B/s 101 251282msec 0 134 7.95 7.11
------ ramdisk -------
submit latencies(usec)
B/w iops runtime min max avg std dev
Read:
Old: 708235KB/s 177058 1227msec 1 51 1.61 0.72
New: 822157KB/s 205539 1059msec 0 14 0.38 0.52
Write:
Old: 710510KB/s 177627 1227msec 2 46 2.33 0.81
New: 821658KB/s 205414 1059msec 0 24 0.40 0.53
Full fio results are attached, and I dropped cache before running
the tests.
--
Ankit Jain
SUSE Labs
[-- Attachment #2: raw-disk-new.log --]
[-- Type: text/x-log, Size: 2601 bytes --]
random_rw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.8-9-gfb9f0
Starting 1 process
random_rw: (groupid=0, jobs=1): err= 0: pid=2109: Thu Jul 26 17:14:55 2012
read : io=102844KB, bw=419099 B/s, iops=102 , runt=251282msec
slat (usec): min=0 , max=169 , avg= 8.20, stdev= 6.95
clat (usec): min=335 , max=3356.7K, avg=255054.47, stdev=158234.29
lat (usec): min=342 , max=3356.7K, avg=255063.32, stdev=158234.33
clat percentiles (msec):
| 1.00th=[ 8], 5.00th=[ 50], 10.00th=[ 84], 20.00th=[ 130],
| 30.00th=[ 169], 40.00th=[ 204], 50.00th=[ 237], 60.00th=[ 269],
| 70.00th=[ 306], 80.00th=[ 351], 90.00th=[ 437], 95.00th=[ 529],
| 99.00th=[ 791], 99.50th=[ 914], 99.90th=[ 1237], 99.95th=[ 1483],
| 99.99th=[ 2073]
bw (KB/s) : min= 111, max= 646, per=100.00%, avg=410.90, stdev=84.69
write: io=101956KB, bw=415481 B/s, iops=101 , runt=251282msec
slat (usec): min=0 , max=134 , avg= 7.95, stdev= 7.11
clat (usec): min=189 , max=928209 , avg=58138.79, stdev=76776.72
lat (usec): min=194 , max=928221 , avg=58147.37, stdev=76776.86
clat percentiles (usec):
| 1.00th=[ 498], 5.00th=[ 828], 10.00th=[ 1624], 20.00th=[ 4960],
| 30.00th=[12352], 40.00th=[22144], 50.00th=[33536], 60.00th=[46848],
| 70.00th=[63232], 80.00th=[90624], 90.00th=[148480], 95.00th=[203776],
| 99.00th=[370688], 99.50th=[460800], 99.90th=[643072], 99.95th=[716800],
| 99.99th=[831488]
bw (KB/s) : min= 31, max= 864, per=100.00%, avg=408.11, stdev=111.34
lat (usec) : 250=0.02%, 500=0.54%, 750=1.27%, 1000=1.51%
lat (msec) : 2=2.39%, 4=3.60%, 10=4.63%, 20=5.96%, 50=13.51%
lat (msec) : 100=14.18%, 250=27.95%, 500=21.04%, 750=2.78%, 1000=0.46%
lat (msec) : 2000=0.15%, >=2000=0.01%
cpu : usr=0.51%, sys=1.52%, ctx=52135, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=25711/w=25489/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=102844KB, aggrb=409KB/s, minb=409KB/s, maxb=409KB/s, mint=251282msec, maxt=251282msec
WRITE: io=101956KB, aggrb=405KB/s, minb=405KB/s, maxb=405KB/s, mint=251282msec, maxt=251282msec
fio rand-rw-disk-2-raw.fio --output=/home/radical/src/play/ios-test/logs-with-drop-cache/ad6d29a/raw-disk-2-raw-ad6d29a.log --max-jobs=2 --latency-log --bandwidth-log
ad6d29a sent upstream
[-- Attachment #3: raw-disk-old.log --]
[-- Type: text/x-log, Size: 2663 bytes --]
random_rw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.8-9-gfb9f0
Starting 1 process
random_rw: (groupid=0, jobs=1): err= 0: pid=2117: Thu Jul 26 17:53:41 2012
read : io=102976KB, bw=417335 B/s, iops=101 , runt=252668msec
slat (usec): min=4 , max=231 , avg=40.03, stdev=21.66
clat (usec): min=236 , max=4075.6K, avg=254175.39, stdev=158853.64
lat (usec): min=339 , max=4075.7K, avg=254216.22, stdev=158853.33
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 51], 10.00th=[ 85], 20.00th=[ 131],
| 30.00th=[ 169], 40.00th=[ 204], 50.00th=[ 237], 60.00th=[ 269],
| 70.00th=[ 306], 80.00th=[ 351], 90.00th=[ 433], 95.00th=[ 529],
| 99.00th=[ 766], 99.50th=[ 906], 99.90th=[ 1270], 99.95th=[ 1500],
| 99.99th=[ 3261]
bw (KB/s) : min= 83, max= 624, per=100.00%, avg=409.49, stdev=91.68
write: io=101824KB, bw=412667 B/s, iops=100 , runt=252668msec
slat (usec): min=3 , max=272 , avg=47.65, stdev=24.58
clat (usec): min=139 , max=1248.8K, avg=60442.70, stdev=82817.74
lat (usec): min=198 , max=1248.9K, avg=60491.15, stdev=82817.11
clat percentiles (usec):
| 1.00th=[ 438], 5.00th=[ 812], 10.00th=[ 1704], 20.00th=[ 5280],
| 30.00th=[13376], 40.00th=[23168], 50.00th=[34560], 60.00th=[47872],
| 70.00th=[66048], 80.00th=[91648], 90.00th=[150528], 95.00th=[209920],
| 99.00th=[403456], 99.50th=[505856], 99.90th=[798720], 99.95th=[897024],
| 99.99th=[1073152]
bw (KB/s) : min= 7, max= 808, per=100.00%, avg=405.72, stdev=121.51
lat (usec) : 250=0.03%, 500=0.79%, 750=1.35%, 1000=1.21%
lat (msec) : 2=2.29%, 4=3.59%, 10=4.36%, 20=5.90%, 50=13.45%
lat (msec) : 100=14.37%, 250=28.08%, 500=21.27%, 750=2.73%, 1000=0.42%
lat (msec) : 2000=0.17%, >=2000=0.01%
cpu : usr=0.54%, sys=1.44%, ctx=52211, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=25744/w=25456/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=102976KB, aggrb=407KB/s, minb=407KB/s, maxb=407KB/s, mint=252668msec, maxt=252668msec
WRITE: io=101824KB, aggrb=402KB/s, minb=402KB/s, maxb=402KB/s, mint=252668msec, maxt=252668msec
fio rand-rw-disk-2-raw.fio --output=/home/radical/src/play/ios-test/logs-with-drop-cache/918227b/raw-disk-2-raw-918227b.log --max-jobs=2 --latency-log --bandwidth-log
918227b Merge tag 'fbdev-fixes-for-3.5-2' of git://github.com/schandinat/linux-2.6
[-- Attachment #4: raw-rd-new.log --]
[-- Type: text/x-log, Size: 2477 bytes --]
random_rw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.8-9-gfb9f0
Starting 1 process
random_rw: (groupid=0, jobs=1): err= 0: pid=2179: Thu Jul 26 17:14:57 2012
read : io=870664KB, bw=822157KB/s, iops=205539 , runt= 1059msec
slat (usec): min=0 , max=14 , avg= 0.38, stdev= 0.52
clat (usec): min=44 , max=294 , avg=76.67, stdev=12.77
lat (usec): min=45 , max=296 , avg=77.10, stdev=12.84
clat percentiles (usec):
| 1.00th=[ 70], 5.00th=[ 71], 10.00th=[ 72], 20.00th=[ 73],
| 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 75], 60.00th=[ 76],
| 70.00th=[ 76], 80.00th=[ 77], 90.00th=[ 79], 95.00th=[ 86],
| 99.00th=[ 97], 99.50th=[ 107], 99.90th=[ 255], 99.95th=[ 266],
| 99.99th=[ 286]
bw (KB/s) : min=819368, max=826656, per=100.00%, avg=823012.00, stdev=5153.39
write: io=870136KB, bw=821658KB/s, iops=205414 , runt= 1059msec
slat (usec): min=0 , max=24 , avg= 0.40, stdev= 0.53
clat (usec): min=42 , max=292 , avg=77.34, stdev=12.81
lat (usec): min=43 , max=293 , avg=77.79, stdev=12.89
clat percentiles (usec):
| 1.00th=[ 70], 5.00th=[ 72], 10.00th=[ 73], 20.00th=[ 74],
| 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 76], 60.00th=[ 76],
| 70.00th=[ 77], 80.00th=[ 78], 90.00th=[ 80], 95.00th=[ 87],
| 99.00th=[ 98], 99.50th=[ 107], 99.90th=[ 262], 99.95th=[ 270],
| 99.99th=[ 286]
bw (KB/s) : min=819368, max=825328, per=100.00%, avg=822348.00, stdev=4214.36
lat (usec) : 50=0.01%, 100=99.35%, 250=0.41%, 500=0.24%
cpu : usr=24.76%, sys=74.76%, ctx=114, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=217666/w=217534/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=870664KB, aggrb=822156KB/s, minb=822156KB/s, maxb=822156KB/s, mint=1059msec, maxt=1059msec
WRITE: io=870136KB, aggrb=821658KB/s, minb=821658KB/s, maxb=821658KB/s, mint=1059msec, maxt=1059msec
Disk stats (read/write):
ram0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
fio rand-rw-rd-raw.fio --output=/home/radical/src/play/ios-test/logs-with-drop-cache/ad6d29a/raw-rd-raw-ad6d29a.log --max-jobs=2 --latency-log --bandwidth-log
ad6d29a sent upstream
[-- Attachment #5: raw-rd-old.log --]
[-- Type: text/x-log, Size: 2548 bytes --]
random_rw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.8-9-gfb9f0
Starting 1 process
random_rw: (groupid=0, jobs=1): err= 0: pid=2192: Thu Jul 26 17:53:52 2012
read : io=869004KB, bw=708235KB/s, iops=177058 , runt= 1227msec
slat (usec): min=1 , max=51 , avg= 1.61, stdev= 0.72
clat (usec): min=17 , max=425 , avg=87.51, stdev=17.51
lat (usec): min=19 , max=432 , avg=89.20, stdev=17.83
clat percentiles (usec):
| 1.00th=[ 79], 5.00th=[ 81], 10.00th=[ 82], 20.00th=[ 83],
| 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60.00th=[ 86],
| 70.00th=[ 87], 80.00th=[ 89], 90.00th=[ 91], 95.00th=[ 98],
| 99.00th=[ 111], 99.50th=[ 118], 99.90th=[ 374], 99.95th=[ 390],
| 99.99th=[ 406]
bw (KB/s) : min=707912, max=711328, per=100.00%, avg=709620.00, stdev=2415.48
write: io=871796KB, bw=710510KB/s, iops=177627 , runt= 1227msec
slat (usec): min=2 , max=46 , avg= 2.33, stdev= 0.81
clat (usec): min=14 , max=425 , avg=87.62, stdev=17.61
lat (usec): min=16 , max=435 , avg=90.05, stdev=18.07
clat percentiles (usec):
| 1.00th=[ 79], 5.00th=[ 81], 10.00th=[ 82], 20.00th=[ 83],
| 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60.00th=[ 87],
| 70.00th=[ 87], 80.00th=[ 89], 90.00th=[ 91], 95.00th=[ 98],
| 99.00th=[ 111], 99.50th=[ 118], 99.90th=[ 378], 99.95th=[ 390],
| 99.99th=[ 406]
bw (KB/s) : min=709360, max=717872, per=100.00%, avg=713616.00, stdev=6018.89
lat (usec) : 20=0.01%, 50=0.01%, 100=95.59%, 250=4.04%, 500=0.36%
cpu : usr=31.57%, sys=67.94%, ctx=125, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=217251/w=217949/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=869004KB, aggrb=708234KB/s, minb=708234KB/s, maxb=708234KB/s, mint=1227msec, maxt=1227msec
WRITE: io=871796KB, aggrb=710510KB/s, minb=710510KB/s, maxb=710510KB/s, mint=1227msec, maxt=1227msec
Disk stats (read/write):
ram0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
fio rand-rw-rd-raw.fio --output=/home/radical/src/play/ios-test/logs-with-drop-cache/918227b/raw-rd-raw-918227b.log --max-jobs=2 --latency-log --bandwidth-log
918227b Merge tag 'fbdev-fixes-for-3.5-2' of git://github.com/schandinat/linux-2.6
next prev parent reply other threads:[~2012-07-26 19:52 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-24 11:41 [RFC][PATCH] Make io_submit non-blocking Ankit Jain
2012-07-24 12:34 ` Rajat Sharma
2012-07-24 12:34 ` Rajat Sharma
2012-07-24 20:27 ` Theodore Ts'o
2012-07-24 20:27 ` Theodore Ts'o
2012-07-24 22:31 ` Dave Chinner
2012-07-24 22:50 ` Christoph Hellwig
2012-07-24 22:50 ` Christoph Hellwig
2012-07-24 23:08 ` Zach Brown
2012-07-24 23:08 ` Zach Brown
2012-07-26 19:52 ` Ankit Jain [this message]
2012-07-26 21:43 ` Zach Brown
2012-07-25 20:12 ` Ankit Jain
2012-07-24 22:37 ` Zach Brown
2012-07-24 22:37 ` Zach Brown
2012-07-25 20:17 ` Ankit Jain
2012-07-25 20:17 ` Ankit Jain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50119FEA.4060806@suse.de \
--to=jankit@suse.de \
--cc=bcrl@kvack.org \
--cc=david@fromorbit.com \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=linux-aio@kvack.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.