* [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
@ 2008-10-07 10:03 Andrea Righi
2008-10-14 9:17 ` Dong-Jae Kang
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
0 siblings, 2 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-07 10:03 UTC (permalink / raw)
To: Balbir Singh, Paul Menage
Cc: agk, akpm, axboe, baramsori72, Carl Henrik Lunde, dave,
Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
The objective of the i/o controller is to improve i/o performance
predictability of different cgroups sharing the same block devices.
Respect to other priority/weight-based solutions the approach used by this
controller is to explicitly choke applications' requests that directly (or
indirectly) generate i/o activity in the system.
The direct bandwidth and/or iops limiting method has the advantage of improving
the performance predictability at the cost of reducing, in general, the overall
performance of the system (in terms of throughput).
Detailed informations about design, its goal and usage are described in the
documentation.
Patchset against 2.6.27-rc5-mm1:
[PATCH 0/6] cgroup: block device i/o controller (v11)
[PATCH 1/6] i/o controller documentation
[PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
[PATCH 3/6] i/o controller infrastructure
[PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
[PATCH 5/6] i/o controller instrumentation: accounting and throttling
[PATCH 6/6] export per-task i/o throttling statistics to userspace
The all-in-one patch (and previous versions) can be found at:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/
There are no significant changes respect to v10, I've only implemented/fixed
some suggestions I received.
Changelog: (v10 -> v11)
* report per block device i/o statistics (total bytes read/written and iops)
in blockio.stat for i/o limited cgroups
* distinct bandwidth and iops statistics: both in blockio.throttlecnt and
/proc/PID/io-throttle-stat (suggested by David Radford)
* merge res_counter_ratelimit functionality into res_counter, to avoid code
duplication (suggested by Paul Manage)
* use kernel-doc style for documenting struct res_counter attributes
(suggested by Randy Dunalp)
* udpated documentation
Thanks to all for the feedback!
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi @ 2008-10-14 9:17 ` Dong-Jae Kang [not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2008-10-14 9:56 ` Andrea Righi [not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 1 sibling, 2 replies; 13+ messages in thread From: Dong-Jae Kang @ 2008-10-14 9:17 UTC (permalink / raw) To: Andrea Righi Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 9545 bytes --] Hi, Andrea thank you for your contribution to communitythese days, I am testing several IO controllers in container ML,dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and yourio-throttle(v11) I have several question about io-throttlebelow is my test reusult of io-throttle(v11) with xdd 6.5But, I think I have something wrong, as showed in resultIn direct IO mode, Only read operation was controlled by io-throttleCan you check my test procedure and result and comments to me about that additionally, your testing shell script(run_io_throttle_test.sh) forio-throttle was not updated for new io-throttleso, it could be operated after I fixed it ------------------------------------------------------------------------------------ Test System Information Computer Name, localhost.localdomain, User Name, rootOS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008Machine hardware type, i686Number of processors on this system, 1Page size in bytes, 4096Number of physical pages, 515885Megabytes of physical memory, 2015Target[0] Q[0], /dev/sdbPer-pass time limit in seconds, 30Blocksize in bytes, 512Request size, 128, blocks, 65536, bytesNumber of Requests, 16384Number of MegaBytes, 512 or 1024Direct I/O, disabled or enableSeek pattern, sequentialQueue Depth, 1 - Test Procedure ï¬ mkdir /dev/blockioctlï¬ mount -t cgroup -o blockio cgroup /dev/blockioctlï¬ mkdir /dev/blockioctl/cgroup-1ï¬ mkdir /dev/blockioctl/cgroup-2ï¬ mkdir /dev/blockioctl/cgroup-3ï¬ echo /dev/sdb:$((1024*1024)):0:0 >/dev/blockioctl/cgroup-1/blockio.bandwidth-maxï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >/dev/blockioctl/cgroup-2/blockio.bandwidth-maxï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >/dev/blockioctl/cgroup-3/blockio.bandwidth-maxï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasksï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasksï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasksï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb-blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30-verbose –dio(enable or disable) - setting status information [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max8 16 1048576 0 0 0 13016[root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max8 16 2097152 0 0 0 11763[root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max8 16 3145728 0 0 0 11133 - Test Resultxdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655360 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655361 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655360 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655361 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655360 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655361 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio –verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655360 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655361 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655360 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655361 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655360 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655361 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536 xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655360 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655361 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655360 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655361 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655360 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655361 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655360 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655361 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655360 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655361 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655360 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655361 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536 Best Regards,Dong-Jae Kang 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>> The objective of the i/o controller is to improve i/o performance> predictability of different cgroups sharing the same block devices.>> Respect to other priority/weight-based solutions the approach used by this> controller is to explicitly choke applications' requests that directly (or> indirectly) generate i/o activity in the system.>> The direct bandwidth and/or iops limiting method has the advantage of improving> the performance predictability at the cost of reducing, in general, the overall> performance of the system (in terms of throughput).>> Detailed informations about design, its goal and usage are described in the> documentation.>> Patchset against 2.6.27-rc5-mm1:>> [PATCH 0/6] cgroup: block device i/o controller (v11)> [PATCH 1/6] i/o controller documentation> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter> [PATCH 3/6] i/o controller infrastructure> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity> [PATCH 5/6] i/o controller instrumentation: accounting and throttling> [PATCH 6/6] export per-task i/o throttling statistics to userspace>> The all-in-one patch (and previous versions) can be found at:> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>> There are no significant changes respect to v10, I've only implemented/fixed> some suggestions I received.>> Changelog: (v10 -> v11)>> * report per block device i/o statistics (total bytes read/written and iops)> in blockio.stat for i/o limited cgroups> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and> /proc/PID/io-throttle-stat (suggested by David Radford)> * merge res_counter_ratelimit functionality into res_counter, to avoid code> duplication (suggested by Paul Manage)> * use kernel-doc style for documenting struct res_counter attributes> (suggested by Randy Dunalp)> * udpated documentation>> Thanks to all for the feedback!> -Andrea>ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2008-10-14 9:56 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2008-10-14 9:56 UTC (permalink / raw) To: Dong-Jae Kang Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage, Carl Henrik Lunde, Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh, fernando-gVGce1chcLdL9jVzuh4AOg, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw, subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti, containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI, ngupta-hpIqsD4AKlfQT0dZR+AlfA Dong-Jae Kang wrote: > Hi, Andrea > > thank you for your contribution to community > these days, I am testing several IO controllers in container ML, > dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your > io-throttle(v11) Thanks! this is surely a valuable task. > > I have several question about io-throttle > below is my test reusult of io-throttle(v11) with xdd 6.5 > But, I think I have something wrong, as showed in result > In direct IO mode, Only read operation was controlled by io-throttle > Can you check my test procedure and result and comments to me about that Your procedure is correct. Anyway, you found a known bug in io-throttle v11. If you want to properly use it you need to mount the memory controller together with blockio, since currently blockio depends on it to retrieve the owner of a page during writes in submit_bio(). As reported in: [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity this is no more than a hack and in perspective a more generic framework able to provide this functionality should be used (i.e. bio-cgroup). I'll fix this issue in the next version of io-throttle (probably I'll try to rewrite io-throttle on top of bio-cgroup), but for now the workaround is to mount the cgroupfs using -o blockio,memory (at least). > > additionally, your testing shell script(run_io_throttle_test.sh) for > io-throttle was not updated for new io-throttle > so, it could be operated after I fixed it The testing of iops limiting is not yet implemented and I don't have a very good testcase for this, but I can share with you a small script that I'm using to check if iops limiting is working or not, if you're interested. Thanks, -Andrea > > ----------------------------------------------------------------------------------- > - Test System Information > > Computer Name, localhost.localdomain, User Name, root > OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008 > Machine hardware type, i686 > Number of processors on this system, 1 > Page size in bytes, 4096 > Number of physical pages, 515885 > Megabytes of physical memory, 2015 > Target[0] Q[0], /dev/sdb > Per-pass time limit in seconds, 30 > Blocksize in bytes, 512 > Request size, 128, blocks, 65536, bytes > Number of Requests, 16384 > Number of MegaBytes, 512 or 1024 > Direct I/O, disabled or enable > Seek pattern, sequential > Queue Depth, 1 > > - Test Procedure > > mkdir /dev/blockioctl > mount -t cgroup -o blockio cgroup /dev/blockioctl > mkdir /dev/blockioctl/cgroup-1 > mkdir /dev/blockioctl/cgroup-2 > mkdir /dev/blockioctl/cgroup-3 > echo /dev/sdb:$((1024*1024)):0:0 > > /dev/blockioctl/cgroup-1/blockio.bandwidth-max > echo /dev/sdb:$((2*1024*1024)):0:0 > > /dev/blockioctl/cgroup-2/blockio.bandwidth-max > echo /dev/sdb:$((3*1024*1024)):0:0 > > /dev/blockioctl/cgroup-3/blockio.bandwidth-max > in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks > in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks > in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks > in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb > -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30 > -verbose –dio(enable or disable) > > - setting status information > > [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max > 8 16 1048576 0 0 0 13016 > [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max > 8 16 2097152 0 0 0 11763 > [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max > 8 16 3145728 0 0 0 11133 > > - Test Result > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 1 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 1 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 1 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio –verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 1 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 1 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 1 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 1 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 1 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 1 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 1 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 1 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 1 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > > Best Regards, > Dong-Jae Kang > > > 2008/10/7 Andrea Righi <righi.andrea@gmail.com>: >> The objective of the i/o controller is to improve i/o performance >> predictability of different cgroups sharing the same block devices. >> >> Respect to other priority/weight-based solutions the approach used by this >> controller is to explicitly choke applications' requests that directly (or >> indirectly) generate i/o activity in the system. >> >> The direct bandwidth and/or iops limiting method has the advantage of improving >> the performance predictability at the cost of reducing, in general, the overall >> performance of the system (in terms of throughput). >> >> Detailed informations about design, its goal and usage are described in the >> documentation. >> >> Patchset against 2.6.27-rc5-mm1: >> >> [PATCH 0/6] cgroup: block device i/o controller (v11) >> [PATCH 1/6] i/o controller documentation >> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter >> [PATCH 3/6] i/o controller infrastructure >> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> [PATCH 5/6] i/o controller instrumentation: accounting and throttling >> [PATCH 6/6] export per-task i/o throttling statistics to userspace >> >> The all-in-one patch (and previous versions) can be found at: >> http://download.systemimager.org/~arighi/linux/patches/io-throttle/ >> >> There are no significant changes respect to v10, I've only implemented/fixed >> some suggestions I received. >> >> Changelog: (v10 -> v11) >> >> * report per block device i/o statistics (total bytes read/written and iops) >> in blockio.stat for i/o limited cgroups >> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and >> /proc/PID/io-throttle-stat (suggested by David Radford) >> * merge res_counter_ratelimit functionality into res_counter, to avoid code >> duplication (suggested by Paul Manage) >> * use kernel-doc style for documenting struct res_counter attributes >> (suggested by Randy Dunalp) >> * udpated documentation >> >> Thanks to all for the feedback! >> -Andrea _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-14 9:17 ` Dong-Jae Kang [not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2008-10-14 9:56 ` Andrea Righi [not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2008-10-15 6:13 ` Dong-Jae Kang 1 sibling, 2 replies; 13+ messages in thread From: Andrea Righi @ 2008-10-14 9:56 UTC (permalink / raw) To: Dong-Jae Kang Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel Dong-Jae Kang wrote: > Hi, Andrea > > thank you for your contribution to community > these days, I am testing several IO controllers in container ML, > dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your > io-throttle(v11) Thanks! this is surely a valuable task. > > I have several question about io-throttle > below is my test reusult of io-throttle(v11) with xdd 6.5 > But, I think I have something wrong, as showed in result > In direct IO mode, Only read operation was controlled by io-throttle > Can you check my test procedure and result and comments to me about that Your procedure is correct. Anyway, you found a known bug in io-throttle v11. If you want to properly use it you need to mount the memory controller together with blockio, since currently blockio depends on it to retrieve the owner of a page during writes in submit_bio(). As reported in: [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity this is no more than a hack and in perspective a more generic framework able to provide this functionality should be used (i.e. bio-cgroup). I'll fix this issue in the next version of io-throttle (probably I'll try to rewrite io-throttle on top of bio-cgroup), but for now the workaround is to mount the cgroupfs using -o blockio,memory (at least). > > additionally, your testing shell script(run_io_throttle_test.sh) for > io-throttle was not updated for new io-throttle > so, it could be operated after I fixed it The testing of iops limiting is not yet implemented and I don't have a very good testcase for this, but I can share with you a small script that I'm using to check if iops limiting is working or not, if you're interested. Thanks, -Andrea > > ----------------------------------------------------------------------------------- > - Test System Information > > Computer Name, localhost.localdomain, User Name, root > OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008 > Machine hardware type, i686 > Number of processors on this system, 1 > Page size in bytes, 4096 > Number of physical pages, 515885 > Megabytes of physical memory, 2015 > Target[0] Q[0], /dev/sdb > Per-pass time limit in seconds, 30 > Blocksize in bytes, 512 > Request size, 128, blocks, 65536, bytes > Number of Requests, 16384 > Number of MegaBytes, 512 or 1024 > Direct I/O, disabled or enable > Seek pattern, sequential > Queue Depth, 1 > > - Test Procedure > > mkdir /dev/blockioctl > mount -t cgroup -o blockio cgroup /dev/blockioctl > mkdir /dev/blockioctl/cgroup-1 > mkdir /dev/blockioctl/cgroup-2 > mkdir /dev/blockioctl/cgroup-3 > echo /dev/sdb:$((1024*1024)):0:0 > > /dev/blockioctl/cgroup-1/blockio.bandwidth-max > echo /dev/sdb:$((2*1024*1024)):0:0 > > /dev/blockioctl/cgroup-2/blockio.bandwidth-max > echo /dev/sdb:$((3*1024*1024)):0:0 > > /dev/blockioctl/cgroup-3/blockio.bandwidth-max > in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks > in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks > in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks > in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb > -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30 > -verbose –dio(enable or disable) > > - setting status information > > [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max > 8 16 1048576 0 0 0 13016 > [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max > 8 16 2097152 0 0 0 11763 > [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max > 8 16 3145728 0 0 0 11133 > > - Test Result > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 1 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 1 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 1 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio –verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 1 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 1 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 1 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 1 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 1 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 1 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 1 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 1 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 1 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > > Best Regards, > Dong-Jae Kang > > > 2008/10/7 Andrea Righi <righi.andrea@gmail.com>: >> The objective of the i/o controller is to improve i/o performance >> predictability of different cgroups sharing the same block devices. >> >> Respect to other priority/weight-based solutions the approach used by this >> controller is to explicitly choke applications' requests that directly (or >> indirectly) generate i/o activity in the system. >> >> The direct bandwidth and/or iops limiting method has the advantage of improving >> the performance predictability at the cost of reducing, in general, the overall >> performance of the system (in terms of throughput). >> >> Detailed informations about design, its goal and usage are described in the >> documentation. >> >> Patchset against 2.6.27-rc5-mm1: >> >> [PATCH 0/6] cgroup: block device i/o controller (v11) >> [PATCH 1/6] i/o controller documentation >> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter >> [PATCH 3/6] i/o controller infrastructure >> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> [PATCH 5/6] i/o controller instrumentation: accounting and throttling >> [PATCH 6/6] export per-task i/o throttling statistics to userspace >> >> The all-in-one patch (and previous versions) can be found at: >> http://download.systemimager.org/~arighi/linux/patches/io-throttle/ >> >> There are no significant changes respect to v10, I've only implemented/fixed >> some suggestions I received. >> >> Changelog: (v10 -> v11) >> >> * report per block device i/o statistics (total bytes read/written and iops) >> in blockio.stat for i/o limited cgroups >> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and >> /proc/PID/io-throttle-stat (suggested by David Radford) >> * merge res_counter_ratelimit functionality into res_counter, to avoid code >> duplication (suggested by Paul Manage) >> * use kernel-doc style for documenting struct res_counter attributes >> (suggested by Randy Dunalp) >> * udpated documentation >> >> Thanks to all for the feedback! >> -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2008-10-15 6:13 ` Dong-Jae Kang 0 siblings, 0 replies; 13+ messages in thread From: Dong-Jae Kang @ 2008-10-15 6:13 UTC (permalink / raw) To: righi.andrea-Re5JQEeQqe8AvxtiuMwx3w Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage, Carl Henrik Lunde, Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh, fernando-gVGce1chcLdL9jVzuh4AOg, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw, subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti, containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI, ngupta-hpIqsD4AKlfQT0dZR+AlfA Hi, Andrea Thank you for your comments > Dong-Jae Kang wrote: >> Hi, Andrea >> >> thank you for your contribution to community >> these days, I am testing several IO controllers in container ML, >> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your >> io-throttle(v11) > > Thanks! this is surely a valuable task. > >> >> I have several question about io-throttle >> below is my test reusult of io-throttle(v11) with xdd 6.5 >> But, I think I have something wrong, as showed in result >> In direct IO mode, Only read operation was controlled by io-throttle >> Can you check my test procedure and result and comments to me about that > > Your procedure is correct. Anyway, you found a known bug in io-throttle > v11. If you want to properly use it you need to mount the memory > controller together with blockio, since currently blockio depends on it > to retrieve the owner of a page during writes in submit_bio(). > > As reported in: > > [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity > > this is no more than a hack and in perspective a more generic framework > able to provide this functionality should be used (i.e. bio-cgroup). But, In my opinion, it is some strange that the bandwidth of write operation in direct IO mode was not controlled by io-throttle I think [PATCH -mm 4/6] is for control of buffered IO. Do I misunderstand it? > I'll fix this issue in the next version of io-throttle (probably I'll > try to rewrite io-throttle on top of bio-cgroup), but for now the > workaround is to mount the cgroupfs using -o blockio,memory (at least). > Oh, it sounds good. I look foward to your next io-throttle >> >> additionally, your testing shell script(run_io_throttle_test.sh) for >> io-throttle was not updated for new io-throttle >> so, it could be operated after I fixed it > > The testing of iops limiting is not yet implemented and I don't have a > very good testcase for this, but I can share with you a small script that > I'm using to check if iops limiting is working or not, if you're interested. thank you, Andrea. I want to check iops limiting of io-throttle is working well. Have a nice day... > >> >> ----------------------------------------------------------------------------------- >> - Test System Information >> >> Computer Name, localhost.localdomain, User Name, root >> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008 >> Machine hardware type, i686 >> Number of processors on this system, 1 >> Page size in bytes, 4096 >> Number of physical pages, 515885 >> Megabytes of physical memory, 2015 >> Target[0] Q[0], /dev/sdb >> Per-pass time limit in seconds, 30 >> Blocksize in bytes, 512 >> Request size, 128, blocks, 65536, bytes >> Number of Requests, 16384 >> Number of MegaBytes, 512 or 1024 >> Direct I/O, disabled or enable >> Seek pattern, sequential >> Queue Depth, 1 >> >> - Test Procedure >> >> mkdir /dev/blockioctl >> mount -t cgroup -o blockio cgroup /dev/blockioctl >> mkdir /dev/blockioctl/cgroup-1 >> mkdir /dev/blockioctl/cgroup-2 >> mkdir /dev/blockioctl/cgroup-3 >> echo /dev/sdb:$((1024*1024)):0:0 > >> /dev/blockioctl/cgroup-1/blockio.bandwidth-max >> echo /dev/sdb:$((2*1024*1024)):0:0 > >> /dev/blockioctl/cgroup-2/blockio.bandwidth-max >> echo /dev/sdb:$((3*1024*1024)):0:0 > >> /dev/blockioctl/cgroup-3/blockio.bandwidth-max >> in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks >> in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks >> in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks >> in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb >> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30 >> -verbose –dio(enable or disable) >> >> - setting status information >> >> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max >> 8 16 1048576 0 0 0 13016 >> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max >> 8 16 2097152 0 0 0 11763 >> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max >> 8 16 3145728 0 0 0 11133 >> >> - Test Result >> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 >> -mbytes 512 -timelimit 30 -dio -verbose >> >> cgroup-1 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 31522816 481 30.005 1.051 16.03 0.0624 >> 0.00 read 65536 >> 0 1 31522816 481 30.005 1.051 16.03 0.0624 >> 0.00 read 65536 >> 1 1 31522816 481 30.005 1.051 16.03 0.0624 >> 0.00 read 65536 >> >> cgroup-2 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 62980096 961 30.001 2.099 32.03 0.0312 >> 0.00 read 65536 >> 0 1 62980096 961 30.001 2.099 32.03 0.0312 >> 0.00 read 65536 >> 1 1 62980096 961 30.001 2.099 32.03 0.0312 >> 0.00 read 65536 >> >> cgroup-3 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 94437376 1441 30.003 3.148 48.03 0.0208 >> 0.00 read 65536 >> 0 1 94437376 1441 30.003 3.148 48.03 0.0208 >> 0.00 read 65536 >> 1 1 94437376 1441 30.003 3.148 48.03 0.0208 >> 0.00 read 65536 >> >> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 >> -mbytes 512 -timelimit 30 -dio –verbose >> >> cgroup-1 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 640221184 9769 30.097 21.272 324.58 0.0031 >> 0.00 write 65536 >> 0 1 640221184 9769 30.097 21.272 324.58 0.0031 >> 0.00 write 65536 >> 1 1 640221184 9769 30.097 21.272 324.58 0.0031 >> 0.00 write 65536 >> >> cgroup-2 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 633798656 9671 30.001 21.126 322.36 0.0031 >> 0.00 write 65536 >> 0 1 633798656 9671 30.001 21.126 322.36 0.0031 >> 0.00 write 65536 >> 1 1 633798656 9671 30.001 21.126 322.36 0.0031 >> 0.00 write 65536 >> >> cgroup-3 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 630652928 9623 30.001 21.021 320.76 0.0031 >> 0.00 write 65536 >> 0 1 630652928 9623 30.001 21.021 320.76 0.0031 >> 0.00 write 65536 >> 1 1 630652928 9623 30.001 21.021 320.76 0.0031 >> 0.00 write 65536 >> >> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 >> -mbytes 1024 -timelimit 30 -verbose >> >> cgroup-1 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 70123520 1070 30.150 2.326 35.49 0.0282 >> 0.00 read 65536 >> 0 1 70123520 1070 30.150 2.326 35.49 0.0282 >> 0.00 read 65536 >> 1 1 70123520 1070 30.150 2.326 35.49 0.0282 >> 0.00 read 65536 >> >> cgroup-2 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 70844416 1081 30.063 2.357 35.96 0.0278 >> 0.00 read 65536 >> 0 1 70844416 1081 30.063 2.357 35.96 0.0278 >> 0.00 read 65536 >> 1 1 70844416 1081 30.063 2.357 35.96 0.0278 >> 0.00 read 65536 >> >> cgroup-3 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 72155136 1101 30.204 2.389 36.45 0.0274 >> 0.00 read 65536 >> 0 1 72155136 1101 30.204 2.389 36.45 0.0274 >> 0.00 read 65536 >> 1 1 72155136 1101 30.204 2.389 36.45 0.0274 >> 0.00 read 65536 >> >> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 >> -mbytes 1024 -timelimit 30 -verbose >> >> cgroup-1 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 818610176 12491 30.031 27.258 415.93 0.0024 >> 0.00 write 65536 >> 0 1 818610176 12491 30.031 27.258 415.93 0.0024 >> 0.00 write 65536 >> 1 1 818610176 12491 30.031 27.258 415.93 0.0024 >> 0.00 write 65536 >> >> cgroup-2 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 848494592 12947 30.066 28.221 430.62 0.0023 >> 0.00 write 65536 >> 0 1 848494592 12947 30.066 28.221 430.62 0.0023 >> 0.00 write 65536 >> 1 1 848494592 12947 30.066 28.221 430.62 0.0023 >> 0.00 write 65536 >> >> cgroup-3 >> >> T Q Bytes Ops Time Rate IOPS Latency >> %CPU OP_Type ReqSize >> 0 1 786563072 12002 30.078 26.151 399.03 0.0025 >> 0.00 write 65536 >> 0 1 786563072 12002 30.078 26.151 399.03 0.0025 >> 0.00 write 65536 >> 1 1 786563072 12002 30.078 26.151 399.03 0.0025 >> 0.00 write 65536 >> >> Best Regards, >> Dong-Jae Kang >> >> >> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>: >>> The objective of the i/o controller is to improve i/o performance >>> predictability of different cgroups sharing the same block devices. >>> >>> Respect to other priority/weight-based solutions the approach used by this >>> controller is to explicitly choke applications' requests that directly (or >>> indirectly) generate i/o activity in the system. >>> >>> The direct bandwidth and/or iops limiting method has the advantage of improving >>> the performance predictability at the cost of reducing, in general, the overall >>> performance of the system (in terms of throughput). >>> >>> Detailed informations about design, its goal and usage are described in the >>> documentation. >>> >>> Patchset against 2.6.27-rc5-mm1: >>> >>> [PATCH 0/6] cgroup: block device i/o controller (v11) >>> [PATCH 1/6] i/o controller documentation >>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter >>> [PATCH 3/6] i/o controller infrastructure >>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling >>> [PATCH 6/6] export per-task i/o throttling statistics to userspace >>> >>> The all-in-one patch (and previous versions) can be found at: >>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/ >>> >>> There are no significant changes respect to v10, I've only implemented/fixed >>> some suggestions I received. >>> >>> Changelog: (v10 -> v11) >>> >>> * report per block device i/o statistics (total bytes read/written and iops) >>> in blockio.stat for i/o limited cgroups >>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and >>> /proc/PID/io-throttle-stat (suggested by David Radford) >>> * merge res_counter_ratelimit functionality into res_counter, to avoid code >>> duplication (suggested by Paul Manage) >>> * use kernel-doc style for documenting struct res_counter attributes >>> (suggested by Randy Dunalp) >>> * udpated documentation >>> >>> Thanks to all for the feedback! >>> -Andrea > -- ------------------------------------------------------------------------------------------------- DONG-JAE, KANG Senior Member of Engineering Staff Internet Platform Research Dept, S/W Content Research Lab Electronics and Telecommunications Research Institute(ETRI) 138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA Phone : 82-42-860-1561 Fax : 82-42-860-6699 Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN) ------------------------------------------------------------------------------------------------- _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-14 9:56 ` Andrea Righi [not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2008-10-15 6:13 ` Dong-Jae Kang 2008-10-16 9:00 ` Andrea Righi [not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 2 replies; 13+ messages in thread From: Dong-Jae Kang @ 2008-10-15 6:13 UTC (permalink / raw) To: righi.andrea Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 12215 bytes --] Hi, AndreaThank you for your comments > Dong-Jae Kang wrote:>> Hi, Andrea>>>> thank you for your contribution to community>> these days, I am testing several IO controllers in container ML,>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your>> io-throttle(v11)>> Thanks! this is surely a valuable task.>>>>> I have several question about io-throttle>> below is my test reusult of io-throttle(v11) with xdd 6.5>> But, I think I have something wrong, as showed in result>> In direct IO mode, Only read operation was controlled by io-throttle>> Can you check my test procedure and result and comments to me about that>> Your procedure is correct. Anyway, you found a known bug in io-throttle> v11. If you want to properly use it you need to mount the memory> controller together with blockio, since currently blockio depends on it> to retrieve the owner of a page during writes in submit_bio().>> As reported in:>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>> this is no more than a hack and in perspective a more generic framework> able to provide this functionality should be used (i.e. bio-cgroup). But, In my opinion,it is some strange that the bandwidth of write operation in direct IOmode was not controlled by io-throttleI think [PATCH -mm 4/6] is for control of buffered IO.Do I misunderstand it? > I'll fix this issue in the next version of io-throttle (probably I'll> try to rewrite io-throttle on top of bio-cgroup), but for now the> workaround is to mount the cgroupfs using -o blockio,memory (at least).> Oh, it sounds good.I look foward to your next io-throttle >>>> additionally, your testing shell script(run_io_throttle_test.sh) for>> io-throttle was not updated for new io-throttle>> so, it could be operated after I fixed it>> The testing of iops limiting is not yet implemented and I don't have a> very good testcase for this, but I can share with you a small script that> I'm using to check if iops limiting is working or not, if you're interested. thank you, Andrea.I want to check iops limiting of io-throttle is working well.Have a nice day...>>>>> ----------------------------------------------------------------------------------->> - Test System Information>>>> Computer Name, localhost.localdomain, User Name, root>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008>> Machine hardware type, i686>> Number of processors on this system, 1>> Page size in bytes, 4096>> Number of physical pages, 515885>> Megabytes of physical memory, 2015>> Target[0] Q[0], /dev/sdb>> Per-pass time limit in seconds, 30>> Blocksize in bytes, 512>> Request size, 128, blocks, 65536, bytes>> Number of Requests, 16384>> Number of MegaBytes, 512 or 1024>> Direct I/O, disabled or enable>> Seek pattern, sequential>> Queue Depth, 1>>>> - Test Procedure>>>> ï¬ mkdir /dev/blockioctl>> ï¬ mount -t cgroup -o blockio cgroup /dev/blockioctl>> ï¬ mkdir /dev/blockioctl/cgroup-1>> ï¬ mkdir /dev/blockioctl/cgroup-2>> ï¬ mkdir /dev/blockioctl/cgroup-3>> ï¬ echo /dev/sdb:$((1024*1024)):0:0 >>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max>> ï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks>> ï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks>> ï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks>> ï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30>> -verbose –dio(enable or disable)>>>> - setting status information>>>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max>> 8 16 1048576 0 0 0 13016>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max>> 8 16 2097152 0 0 0 11763>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max>> 8 16 3145728 0 0 0 11133>>>> - Test Result>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 1 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 1 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 1 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio –verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 1 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 1 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 1 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>>>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 1 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 1 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 1 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 1 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 1 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 1 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>>>> Best Regards,>> Dong-Jae Kang>>>>>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>>> The objective of the i/o controller is to improve i/o performance>>> predictability of different cgroups sharing the same block devices.>>>>>> Respect to other priority/weight-based solutions the approach used by this>>> controller is to explicitly choke applications' requests that directly (or>>> indirectly) generate i/o activity in the system.>>>>>> The direct bandwidth and/or iops limiting method has the advantage of improving>>> the performance predictability at the cost of reducing, in general, the overall>>> performance of the system (in terms of throughput).>>>>>> Detailed informations about design, its goal and usage are described in the>>> documentation.>>>>>> Patchset against 2.6.27-rc5-mm1:>>>>>> [PATCH 0/6] cgroup: block device i/o controller (v11)>>> [PATCH 1/6] i/o controller documentation>>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>>> [PATCH 3/6] i/o controller infrastructure>>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling>>> [PATCH 6/6] export per-task i/o throttling statistics to userspace>>>>>> The all-in-one patch (and previous versions) can be found at:>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>>>>>> There are no significant changes respect to v10, I've only implemented/fixed>>> some suggestions I received.>>>>>> Changelog: (v10 -> v11)>>>>>> * report per block device i/o statistics (total bytes read/written and iops)>>> in blockio.stat for i/o limited cgroups>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>>> /proc/PID/io-throttle-stat (suggested by David Radford)>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code>>> duplication (suggested by Paul Manage)>>> * use kernel-doc style for documenting struct res_counter attributes>>> (suggested by Randy Dunalp)>>> * udpated documentation>>>>>> Thanks to all for the feedback!>>> -Andrea> -- ------------------------------------------------------------------------------------------------- DONG-JAE, KANG Senior Member of Engineering Staff Internet Platform Research Dept, S/W Content Research Lab Electronics and Telecommunications Research Institute(ETRI) 138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA Phone : 82-42-860-1561 Fax : 82-42-860-6699 Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)-------------------------------------------------------------------------------------------------ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-15 6:13 ` Dong-Jae Kang @ 2008-10-16 9:00 ` Andrea Righi [not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 0 replies; 13+ messages in thread From: Andrea Righi @ 2008-10-16 9:00 UTC (permalink / raw) To: Dong-Jae Kang Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel Dong-Jae Kang wrote: >>> I have several question about io-throttle >>> below is my test reusult of io-throttle(v11) with xdd 6.5 >>> But, I think I have something wrong, as showed in result >>> In direct IO mode, Only read operation was controlled by io-throttle >>> Can you check my test procedure and result and comments to me about that >> Your procedure is correct. Anyway, you found a known bug in io-throttle >> v11. If you want to properly use it you need to mount the memory >> controller together with blockio, since currently blockio depends on it >> to retrieve the owner of a page during writes in submit_bio(). >> >> As reported in: >> >> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> >> this is no more than a hack and in perspective a more generic framework >> able to provide this functionality should be used (i.e. bio-cgroup). > > But, In my opinion, > it is some strange that the bandwidth of write operation in direct IO > mode was not controlled by io-throttle > I think [PATCH -mm 4/6] is for control of buffered IO. > Do I misunderstand it? The approach is the same both for buffered and direct IO. With buffered IO, writes are accounted in submit_bio() and throttling is performed in balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always performed in submit_bio() and throttling in submit_page_section(). That's the same also for AIO, accounting in submit_bio() and throttling in io_submit_one(), returning -EAGAIN in this case when the IO limits are exceeded (instead of sleeping). We never sleep in submit_bio() during writes and to account the IO activity we always look at the owner (cgroup) of the first page in the struct bio, also when the IO context is the cgroup of the current task (as in the direct IO case). To correctly retrieve the owner of each page I've used the memory controller functionality. This part should be improved to use a more generic framework and remove the dependency of the memory controller. Or better, doesn't impose to mount blockio and memory controller both to the same mountpoint. -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2008-10-16 9:00 ` Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2008-10-16 9:00 UTC (permalink / raw) To: Dong-Jae Kang Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage, Carl Henrik Lunde, Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh, fernando-gVGce1chcLdL9jVzuh4AOg, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw, subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti, containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI, ngupta-hpIqsD4AKlfQT0dZR+AlfA Dong-Jae Kang wrote: >>> I have several question about io-throttle >>> below is my test reusult of io-throttle(v11) with xdd 6.5 >>> But, I think I have something wrong, as showed in result >>> In direct IO mode, Only read operation was controlled by io-throttle >>> Can you check my test procedure and result and comments to me about that >> Your procedure is correct. Anyway, you found a known bug in io-throttle >> v11. If you want to properly use it you need to mount the memory >> controller together with blockio, since currently blockio depends on it >> to retrieve the owner of a page during writes in submit_bio(). >> >> As reported in: >> >> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> >> this is no more than a hack and in perspective a more generic framework >> able to provide this functionality should be used (i.e. bio-cgroup). > > But, In my opinion, > it is some strange that the bandwidth of write operation in direct IO > mode was not controlled by io-throttle > I think [PATCH -mm 4/6] is for control of buffered IO. > Do I misunderstand it? The approach is the same both for buffered and direct IO. With buffered IO, writes are accounted in submit_bio() and throttling is performed in balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always performed in submit_bio() and throttling in submit_page_section(). That's the same also for AIO, accounting in submit_bio() and throttling in io_submit_one(), returning -EAGAIN in this case when the IO limits are exceeded (instead of sleeping). We never sleep in submit_bio() during writes and to account the IO activity we always look at the owner (cgroup) of the first page in the struct bio, also when the IO context is the cgroup of the current task (as in the direct IO case). To correctly retrieve the owner of each page I've used the memory controller functionality. This part should be improved to use a more generic framework and remove the dependency of the memory controller. Or better, doesn't impose to mount blockio and memory controller both to the same mountpoint. -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2008-10-14 9:17 ` Dong-Jae Kang 2008-11-03 7:01 ` Gui Jianfeng 1 sibling, 0 replies; 13+ messages in thread From: Dong-Jae Kang @ 2008-10-14 9:17 UTC (permalink / raw) To: Andrea Righi Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage, Carl Henrik Lunde, Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh, fernando-gVGce1chcLdL9jVzuh4AOg, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw, subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti, containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI, ngupta-hpIqsD4AKlfQT0dZR+AlfA Hi, Andrea thank you for your contribution to community these days, I am testing several IO controllers in container ML, dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your io-throttle(v11) I have several question about io-throttle below is my test reusult of io-throttle(v11) with xdd 6.5 But, I think I have something wrong, as showed in result In direct IO mode, Only read operation was controlled by io-throttle Can you check my test procedure and result and comments to me about that additionally, your testing shell script(run_io_throttle_test.sh) for io-throttle was not updated for new io-throttle so, it could be operated after I fixed it ----------------------------------------------------------------------------------- - Test System Information Computer Name, localhost.localdomain, User Name, root OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008 Machine hardware type, i686 Number of processors on this system, 1 Page size in bytes, 4096 Number of physical pages, 515885 Megabytes of physical memory, 2015 Target[0] Q[0], /dev/sdb Per-pass time limit in seconds, 30 Blocksize in bytes, 512 Request size, 128, blocks, 65536, bytes Number of Requests, 16384 Number of MegaBytes, 512 or 1024 Direct I/O, disabled or enable Seek pattern, sequential Queue Depth, 1 - Test Procedure mkdir /dev/blockioctl mount -t cgroup -o blockio cgroup /dev/blockioctl mkdir /dev/blockioctl/cgroup-1 mkdir /dev/blockioctl/cgroup-2 mkdir /dev/blockioctl/cgroup-3 echo /dev/sdb:$((1024*1024)):0:0 > /dev/blockioctl/cgroup-1/blockio.bandwidth-max echo /dev/sdb:$((2*1024*1024)):0:0 > /dev/blockioctl/cgroup-2/blockio.bandwidth-max echo /dev/sdb:$((3*1024*1024)):0:0 > /dev/blockioctl/cgroup-3/blockio.bandwidth-max in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30 -verbose –dio(enable or disable) - setting status information [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max 8 16 1048576 0 0 0 13016 [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max 8 16 2097152 0 0 0 11763 [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max 8 16 3145728 0 0 0 11133 - Test Result xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 -mbytes 512 -timelimit 30 -dio -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536 0 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536 1 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536 0 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536 1 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536 0 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536 1 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 -mbytes 512 -timelimit 30 -dio –verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536 0 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536 1 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536 0 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536 1 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536 0 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536 1 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536 xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 -mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536 0 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536 1 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536 0 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536 1 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536 0 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536 1 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 -mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536 0 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536 1 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536 0 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536 1 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize 0 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536 0 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536 1 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536 Best Regards, Dong-Jae Kang 2008/10/7 Andrea Righi <righi.andrea@gmail.com>: > > The objective of the i/o controller is to improve i/o performance > predictability of different cgroups sharing the same block devices. > > Respect to other priority/weight-based solutions the approach used by this > controller is to explicitly choke applications' requests that directly (or > indirectly) generate i/o activity in the system. > > The direct bandwidth and/or iops limiting method has the advantage of improving > the performance predictability at the cost of reducing, in general, the overall > performance of the system (in terms of throughput). > > Detailed informations about design, its goal and usage are described in the > documentation. > > Patchset against 2.6.27-rc5-mm1: > > [PATCH 0/6] cgroup: block device i/o controller (v11) > [PATCH 1/6] i/o controller documentation > [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter > [PATCH 3/6] i/o controller infrastructure > [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity > [PATCH 5/6] i/o controller instrumentation: accounting and throttling > [PATCH 6/6] export per-task i/o throttling statistics to userspace > > The all-in-one patch (and previous versions) can be found at: > http://download.systemimager.org/~arighi/linux/patches/io-throttle/ > > There are no significant changes respect to v10, I've only implemented/fixed > some suggestions I received. > > Changelog: (v10 -> v11) > > * report per block device i/o statistics (total bytes read/written and iops) > in blockio.stat for i/o limited cgroups > * distinct bandwidth and iops statistics: both in blockio.throttlecnt and > /proc/PID/io-throttle-stat (suggested by David Radford) > * merge res_counter_ratelimit functionality into res_counter, to avoid code > duplication (suggested by Paul Manage) > * use kernel-doc style for documenting struct res_counter attributes > (suggested by Randy Dunalp) > * udpated documentation > > Thanks to all for the feedback! > -Andrea > _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2008-10-14 9:17 ` Dong-Jae Kang @ 2008-11-03 7:01 ` Gui Jianfeng [not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org> 1 sibling, 1 reply; 13+ messages in thread From: Gui Jianfeng @ 2008-11-03 7:01 UTC (permalink / raw) To: Andrea Righi; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA Hi Andrea, I'v tested io-throttle(v11), it seems io-throttle doesn't work well when a process writes(delay-write) a small file(<50M may be). it can't throttle the request process accurately. I guess the following reason causes this problem. The write-request process terminates too early to see all accounting calculated in pdflush. So io-throttle can't throttle enough time for that process. Am i right? -- Regards Gui Jianfeng ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org> @ 2008-11-10 8:51 ` Andrea Righi [not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 13+ messages in thread From: Andrea Righi @ 2008-11-10 8:51 UTC (permalink / raw) To: Gui Jianfeng; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA On 2008-11-03 08:01, Gui Jianfeng wrote: > Hi Andrea, > > I'v tested io-throttle(v11), it seems io-throttle doesn't work well > when a process writes(delay-write) a small file(<50M may be). it can't > throttle the request process accurately. > I guess the following reason causes this problem. > The write-request process terminates too early to see all accounting > calculated in pdflush. So io-throttle can't throttle > enough time for that process. Am i right? Hi Gui, sorry for my late first of all. You're right in part. The fact is that with asynchronous IO (no O_DIRECT more exactly) the accounting is performed when the blocks are submitted to the IO subsystem (submit_bio) and the throttling is performed post-facto when the same process or the other processes in the same cgroup write additional pages mapped to the limited device (throttling is performed in balance_dirty_pages_ratelimited_nr). Anyway, if the process did a single small write in memory the operation is not accounted until pdflush writes the data back to the block device. When pdflush (or the process itself) performs the writeback the operation is accounted and next reads/writes that come from the same cgroup can now be throttled according to the defined limits. So, the accounting is not lost, this information is only used later to throttle the next operations that will cause IO requests. IO spikes in this way are not prevented. With an early version of io-throttle I did both throttling and accounting in balance_dirty_pages_ratelimited_nr (when pages are written to memory), this can prevent IO spikes, but only if dirty_ratio is set to small values. Hope this clarify, -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) [not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2008-11-10 9:52 ` Gui Jianfeng 0 siblings, 0 replies; 13+ messages in thread From: Gui Jianfeng @ 2008-11-10 9:52 UTC (permalink / raw) To: righi.andrea-Re5JQEeQqe8AvxtiuMwx3w Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA Andrea Righi wrote: > On 2008-11-03 08:01, Gui Jianfeng wrote: >> Hi Andrea, >> >> I'v tested io-throttle(v11), it seems io-throttle doesn't work well >> when a process writes(delay-write) a small file(<50M may be). it can't >> throttle the request process accurately. >> I guess the following reason causes this problem. >> The write-request process terminates too early to see all accounting >> calculated in pdflush. So io-throttle can't throttle >> enough time for that process. Am i right? > > Hi Gui, > > sorry for my late first of all. You're right in part. The fact is that > with asynchronous IO (no O_DIRECT more exactly) the accounting is > performed when the blocks are submitted to the IO subsystem (submit_bio) > and the throttling is performed post-facto when the same process or the > other processes in the same cgroup write additional pages mapped to the > limited device (throttling is performed in > balance_dirty_pages_ratelimited_nr). > > Anyway, if the process did a single small write in memory the operation > is not accounted until pdflush writes the data back to the block device. > When pdflush (or the process itself) performs the writeback the > operation is accounted and next reads/writes that come from the same > cgroup can now be throttled according to the defined limits. > > So, the accounting is not lost, this information is only used later to > throttle the next operations that will cause IO requests. IO spikes in > this way are not prevented. With an early version of io-throttle I did > both throttling and accounting in balance_dirty_pages_ratelimited_nr > (when pages are written to memory), this can prevent IO spikes, but only > if dirty_ratio is set to small values. Hi, Andrea, Thanks for your detailed explanation, I'v got your point. I think it makes more sense if throttling can be fair among other tasks. ;-) > > Hope this clarify, > -Andrea > > > -- Regards Gui Jianfeng ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH -mm 0/6] cgroup: block device i/o controller (v11) @ 2008-10-07 10:03 Andrea Righi 0 siblings, 0 replies; 13+ messages in thread From: Andrea Righi @ 2008-10-07 10:03 UTC (permalink / raw) To: Balbir Singh, Paul Menage Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Carl Henrik Lunde, Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, fernando-gVGce1chcLdL9jVzuh4AOg, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw, subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti, containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-kernel-u79uwXL29TY76Z2rM5mHXA, dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8, matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI, ngupta-hpIqsD4AKlfQT0dZR+AlfA The objective of the i/o controller is to improve i/o performance predictability of different cgroups sharing the same block devices. Respect to other priority/weight-based solutions the approach used by this controller is to explicitly choke applications' requests that directly (or indirectly) generate i/o activity in the system. The direct bandwidth and/or iops limiting method has the advantage of improving the performance predictability at the cost of reducing, in general, the overall performance of the system (in terms of throughput). Detailed informations about design, its goal and usage are described in the documentation. Patchset against 2.6.27-rc5-mm1: [PATCH 0/6] cgroup: block device i/o controller (v11) [PATCH 1/6] i/o controller documentation [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter [PATCH 3/6] i/o controller infrastructure [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity [PATCH 5/6] i/o controller instrumentation: accounting and throttling [PATCH 6/6] export per-task i/o throttling statistics to userspace The all-in-one patch (and previous versions) can be found at: http://download.systemimager.org/~arighi/linux/patches/io-throttle/ There are no significant changes respect to v10, I've only implemented/fixed some suggestions I received. Changelog: (v10 -> v11) * report per block device i/o statistics (total bytes read/written and iops) in blockio.stat for i/o limited cgroups * distinct bandwidth and iops statistics: both in blockio.throttlecnt and /proc/PID/io-throttle-stat (suggested by David Radford) * merge res_counter_ratelimit functionality into res_counter, to avoid code duplication (suggested by Paul Manage) * use kernel-doc style for documenting struct res_counter attributes (suggested by Randy Dunalp) * udpated documentation Thanks to all for the feedback! -Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2008-11-10 9:52 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi
2008-10-14 9:17 ` Dong-Jae Kang
[not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-14 9:56 ` Andrea Righi
2008-10-14 9:56 ` Andrea Righi
[not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-15 6:13 ` Dong-Jae Kang
2008-10-15 6:13 ` Dong-Jae Kang
2008-10-16 9:00 ` Andrea Righi
[not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-16 9:00 ` Andrea Righi
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-14 9:17 ` Dong-Jae Kang
2008-11-03 7:01 ` Gui Jianfeng
[not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
2008-11-10 8:51 ` Andrea Righi
[not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-11-10 9:52 ` Gui Jianfeng
-- strict thread matches above, loose matches on Subject: below --
2008-10-07 10:03 Andrea Righi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.