* [PATCH -mm 0/6] cgroup: block device i/o controller (v11) @ 2008-10-07 10:03 Andrea Righi 2008-10-14 9:17 ` Dong-Jae Kang 0 siblings, 1 reply; 5+ messages in thread From: Andrea Righi @ 2008-10-07 10:03 UTC (permalink / raw) To: Balbir Singh, Paul Menage Cc: agk, akpm, axboe, baramsori72, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel The objective of the i/o controller is to improve i/o performance predictability of different cgroups sharing the same block devices. Respect to other priority/weight-based solutions the approach used by this controller is to explicitly choke applications' requests that directly (or indirectly) generate i/o activity in the system. The direct bandwidth and/or iops limiting method has the advantage of improving the performance predictability at the cost of reducing, in general, the overall performance of the system (in terms of throughput). Detailed informations about design, its goal and usage are described in the documentation. Patchset against 2.6.27-rc5-mm1: [PATCH 0/6] cgroup: block device i/o controller (v11) [PATCH 1/6] i/o controller documentation [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter [PATCH 3/6] i/o controller infrastructure [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity [PATCH 5/6] i/o controller instrumentation: accounting and throttling [PATCH 6/6] export per-task i/o throttling statistics to userspace The all-in-one patch (and previous versions) can be found at: http://download.systemimager.org/~arighi/linux/patches/io-throttle/ There are no significant changes respect to v10, I've only implemented/fixed some suggestions I received. Changelog: (v10 -> v11) * report per block device i/o statistics (total bytes read/written and iops) in blockio.stat for i/o limited cgroups * distinct bandwidth and iops statistics: both in blockio.throttlecnt and /proc/PID/io-throttle-stat (suggested by David Radford) * merge res_counter_ratelimit functionality into res_counter, to avoid code duplication (suggested by Paul Manage) * use kernel-doc style for documenting struct res_counter attributes (suggested by Randy Dunalp) * udpated documentation Thanks to all for the feedback! -Andrea ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi @ 2008-10-14 9:17 ` Dong-Jae Kang 2008-10-14 9:56 ` Andrea Righi 0 siblings, 1 reply; 5+ messages in thread From: Dong-Jae Kang @ 2008-10-14 9:17 UTC (permalink / raw) To: Andrea Righi Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 9545 bytes --] Hi, Andrea thank you for your contribution to communitythese days, I am testing several IO controllers in container ML,dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and yourio-throttle(v11) I have several question about io-throttlebelow is my test reusult of io-throttle(v11) with xdd 6.5But, I think I have something wrong, as showed in resultIn direct IO mode, Only read operation was controlled by io-throttleCan you check my test procedure and result and comments to me about that additionally, your testing shell script(run_io_throttle_test.sh) forio-throttle was not updated for new io-throttleso, it could be operated after I fixed it ------------------------------------------------------------------------------------ Test System Information Computer Name, localhost.localdomain, User Name, rootOS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008Machine hardware type, i686Number of processors on this system, 1Page size in bytes, 4096Number of physical pages, 515885Megabytes of physical memory, 2015Target[0] Q[0], /dev/sdbPer-pass time limit in seconds, 30Blocksize in bytes, 512Request size, 128, blocks, 65536, bytesNumber of Requests, 16384Number of MegaBytes, 512 or 1024Direct I/O, disabled or enableSeek pattern, sequentialQueue Depth, 1 - Test Procedure ï¬ mkdir /dev/blockioctlï¬ mount -t cgroup -o blockio cgroup /dev/blockioctlï¬ mkdir /dev/blockioctl/cgroup-1ï¬ mkdir /dev/blockioctl/cgroup-2ï¬ mkdir /dev/blockioctl/cgroup-3ï¬ echo /dev/sdb:$((1024*1024)):0:0 >/dev/blockioctl/cgroup-1/blockio.bandwidth-maxï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >/dev/blockioctl/cgroup-2/blockio.bandwidth-maxï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >/dev/blockioctl/cgroup-3/blockio.bandwidth-maxï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasksï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasksï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasksï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb-blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30-verbose –dio(enable or disable) - setting status information [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max8 16 1048576 0 0 0 13016[root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max8 16 2097152 0 0 0 11763[root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max8 16 3145728 0 0 0 11133 - Test Resultxdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655360 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655361 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655360 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655361 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655360 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655361 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio –verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655360 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655361 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655360 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655361 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655360 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655361 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536 xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655360 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655361 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655360 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655361 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655360 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655361 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536 xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose cgroup-1 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655360 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655361 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536 cgroup-2 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655360 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655361 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536 cgroup-3 T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655360 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655361 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536 Best Regards,Dong-Jae Kang 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>> The objective of the i/o controller is to improve i/o performance> predictability of different cgroups sharing the same block devices.>> Respect to other priority/weight-based solutions the approach used by this> controller is to explicitly choke applications' requests that directly (or> indirectly) generate i/o activity in the system.>> The direct bandwidth and/or iops limiting method has the advantage of improving> the performance predictability at the cost of reducing, in general, the overall> performance of the system (in terms of throughput).>> Detailed informations about design, its goal and usage are described in the> documentation.>> Patchset against 2.6.27-rc5-mm1:>> [PATCH 0/6] cgroup: block device i/o controller (v11)> [PATCH 1/6] i/o controller documentation> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter> [PATCH 3/6] i/o controller infrastructure> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity> [PATCH 5/6] i/o controller instrumentation: accounting and throttling> [PATCH 6/6] export per-task i/o throttling statistics to userspace>> The all-in-one patch (and previous versions) can be found at:> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>> There are no significant changes respect to v10, I've only implemented/fixed> some suggestions I received.>> Changelog: (v10 -> v11)>> * report per block device i/o statistics (total bytes read/written and iops)> in blockio.stat for i/o limited cgroups> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and> /proc/PID/io-throttle-stat (suggested by David Radford)> * merge res_counter_ratelimit functionality into res_counter, to avoid code> duplication (suggested by Paul Manage)> * use kernel-doc style for documenting struct res_counter attributes> (suggested by Randy Dunalp)> * udpated documentation>> Thanks to all for the feedback!> -Andrea>ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-14 9:17 ` Dong-Jae Kang @ 2008-10-14 9:56 ` Andrea Righi 2008-10-15 6:13 ` Dong-Jae Kang 0 siblings, 1 reply; 5+ messages in thread From: Andrea Righi @ 2008-10-14 9:56 UTC (permalink / raw) To: Dong-Jae Kang Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel Dong-Jae Kang wrote: > Hi, Andrea > > thank you for your contribution to community > these days, I am testing several IO controllers in container ML, > dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your > io-throttle(v11) Thanks! this is surely a valuable task. > > I have several question about io-throttle > below is my test reusult of io-throttle(v11) with xdd 6.5 > But, I think I have something wrong, as showed in result > In direct IO mode, Only read operation was controlled by io-throttle > Can you check my test procedure and result and comments to me about that Your procedure is correct. Anyway, you found a known bug in io-throttle v11. If you want to properly use it you need to mount the memory controller together with blockio, since currently blockio depends on it to retrieve the owner of a page during writes in submit_bio(). As reported in: [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity this is no more than a hack and in perspective a more generic framework able to provide this functionality should be used (i.e. bio-cgroup). I'll fix this issue in the next version of io-throttle (probably I'll try to rewrite io-throttle on top of bio-cgroup), but for now the workaround is to mount the cgroupfs using -o blockio,memory (at least). > > additionally, your testing shell script(run_io_throttle_test.sh) for > io-throttle was not updated for new io-throttle > so, it could be operated after I fixed it The testing of iops limiting is not yet implemented and I don't have a very good testcase for this, but I can share with you a small script that I'm using to check if iops limiting is working or not, if you're interested. Thanks, -Andrea > > ----------------------------------------------------------------------------------- > - Test System Information > > Computer Name, localhost.localdomain, User Name, root > OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008 > Machine hardware type, i686 > Number of processors on this system, 1 > Page size in bytes, 4096 > Number of physical pages, 515885 > Megabytes of physical memory, 2015 > Target[0] Q[0], /dev/sdb > Per-pass time limit in seconds, 30 > Blocksize in bytes, 512 > Request size, 128, blocks, 65536, bytes > Number of Requests, 16384 > Number of MegaBytes, 512 or 1024 > Direct I/O, disabled or enable > Seek pattern, sequential > Queue Depth, 1 > > - Test Procedure > > mkdir /dev/blockioctl > mount -t cgroup -o blockio cgroup /dev/blockioctl > mkdir /dev/blockioctl/cgroup-1 > mkdir /dev/blockioctl/cgroup-2 > mkdir /dev/blockioctl/cgroup-3 > echo /dev/sdb:$((1024*1024)):0:0 > > /dev/blockioctl/cgroup-1/blockio.bandwidth-max > echo /dev/sdb:$((2*1024*1024)):0:0 > > /dev/blockioctl/cgroup-2/blockio.bandwidth-max > echo /dev/sdb:$((3*1024*1024)):0:0 > > /dev/blockioctl/cgroup-3/blockio.bandwidth-max > in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks > in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks > in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks > in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb > -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30 > -verbose –dio(enable or disable) > > - setting status information > > [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max > 8 16 1048576 0 0 0 13016 > [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max > 8 16 2097152 0 0 0 11763 > [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max > 8 16 3145728 0 0 0 11133 > > - Test Result > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 0 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > 1 1 31522816 481 30.005 1.051 16.03 0.0624 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 0 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > 1 1 62980096 961 30.001 2.099 32.03 0.0312 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 0 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > 1 1 94437376 1441 30.003 3.148 48.03 0.0208 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 512 -timelimit 30 -dio –verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 0 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > 1 1 640221184 9769 30.097 21.272 324.58 0.0031 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 0 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > 1 1 633798656 9671 30.001 21.126 322.36 0.0031 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 0 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > 1 1 630652928 9623 30.001 21.021 320.76 0.0031 > 0.00 write 65536 > > xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 0 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > 1 1 70123520 1070 30.150 2.326 35.49 0.0282 > 0.00 read 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 0 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > 1 1 70844416 1081 30.063 2.357 35.96 0.0278 > 0.00 read 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 0 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > 1 1 72155136 1101 30.204 2.389 36.45 0.0274 > 0.00 read 65536 > > xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128 > -mbytes 1024 -timelimit 30 -verbose > > cgroup-1 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 0 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > 1 1 818610176 12491 30.031 27.258 415.93 0.0024 > 0.00 write 65536 > > cgroup-2 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 0 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > 1 1 848494592 12947 30.066 28.221 430.62 0.0023 > 0.00 write 65536 > > cgroup-3 > > T Q Bytes Ops Time Rate IOPS Latency > %CPU OP_Type ReqSize > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 0 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > 1 1 786563072 12002 30.078 26.151 399.03 0.0025 > 0.00 write 65536 > > Best Regards, > Dong-Jae Kang > > > 2008/10/7 Andrea Righi <righi.andrea@gmail.com>: >> The objective of the i/o controller is to improve i/o performance >> predictability of different cgroups sharing the same block devices. >> >> Respect to other priority/weight-based solutions the approach used by this >> controller is to explicitly choke applications' requests that directly (or >> indirectly) generate i/o activity in the system. >> >> The direct bandwidth and/or iops limiting method has the advantage of improving >> the performance predictability at the cost of reducing, in general, the overall >> performance of the system (in terms of throughput). >> >> Detailed informations about design, its goal and usage are described in the >> documentation. >> >> Patchset against 2.6.27-rc5-mm1: >> >> [PATCH 0/6] cgroup: block device i/o controller (v11) >> [PATCH 1/6] i/o controller documentation >> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter >> [PATCH 3/6] i/o controller infrastructure >> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> [PATCH 5/6] i/o controller instrumentation: accounting and throttling >> [PATCH 6/6] export per-task i/o throttling statistics to userspace >> >> The all-in-one patch (and previous versions) can be found at: >> http://download.systemimager.org/~arighi/linux/patches/io-throttle/ >> >> There are no significant changes respect to v10, I've only implemented/fixed >> some suggestions I received. >> >> Changelog: (v10 -> v11) >> >> * report per block device i/o statistics (total bytes read/written and iops) >> in blockio.stat for i/o limited cgroups >> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and >> /proc/PID/io-throttle-stat (suggested by David Radford) >> * merge res_counter_ratelimit functionality into res_counter, to avoid code >> duplication (suggested by Paul Manage) >> * use kernel-doc style for documenting struct res_counter attributes >> (suggested by Randy Dunalp) >> * udpated documentation >> >> Thanks to all for the feedback! >> -Andrea ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-14 9:56 ` Andrea Righi @ 2008-10-15 6:13 ` Dong-Jae Kang 2008-10-16 9:00 ` Andrea Righi 0 siblings, 1 reply; 5+ messages in thread From: Dong-Jae Kang @ 2008-10-15 6:13 UTC (permalink / raw) To: righi.andrea Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 12215 bytes --] Hi, AndreaThank you for your comments > Dong-Jae Kang wrote:>> Hi, Andrea>>>> thank you for your contribution to community>> these days, I am testing several IO controllers in container ML,>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your>> io-throttle(v11)>> Thanks! this is surely a valuable task.>>>>> I have several question about io-throttle>> below is my test reusult of io-throttle(v11) with xdd 6.5>> But, I think I have something wrong, as showed in result>> In direct IO mode, Only read operation was controlled by io-throttle>> Can you check my test procedure and result and comments to me about that>> Your procedure is correct. Anyway, you found a known bug in io-throttle> v11. If you want to properly use it you need to mount the memory> controller together with blockio, since currently blockio depends on it> to retrieve the owner of a page during writes in submit_bio().>> As reported in:>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>> this is no more than a hack and in perspective a more generic framework> able to provide this functionality should be used (i.e. bio-cgroup). But, In my opinion,it is some strange that the bandwidth of write operation in direct IOmode was not controlled by io-throttleI think [PATCH -mm 4/6] is for control of buffered IO.Do I misunderstand it? > I'll fix this issue in the next version of io-throttle (probably I'll> try to rewrite io-throttle on top of bio-cgroup), but for now the> workaround is to mount the cgroupfs using -o blockio,memory (at least).> Oh, it sounds good.I look foward to your next io-throttle >>>> additionally, your testing shell script(run_io_throttle_test.sh) for>> io-throttle was not updated for new io-throttle>> so, it could be operated after I fixed it>> The testing of iops limiting is not yet implemented and I don't have a> very good testcase for this, but I can share with you a small script that> I'm using to check if iops limiting is working or not, if you're interested. thank you, Andrea.I want to check iops limiting of io-throttle is working well.Have a nice day...>>>>> ----------------------------------------------------------------------------------->> - Test System Information>>>> Computer Name, localhost.localdomain, User Name, root>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008>> Machine hardware type, i686>> Number of processors on this system, 1>> Page size in bytes, 4096>> Number of physical pages, 515885>> Megabytes of physical memory, 2015>> Target[0] Q[0], /dev/sdb>> Per-pass time limit in seconds, 30>> Blocksize in bytes, 512>> Request size, 128, blocks, 65536, bytes>> Number of Requests, 16384>> Number of MegaBytes, 512 or 1024>> Direct I/O, disabled or enable>> Seek pattern, sequential>> Queue Depth, 1>>>> - Test Procedure>>>> ï¬ mkdir /dev/blockioctl>> ï¬ mount -t cgroup -o blockio cgroup /dev/blockioctl>> ï¬ mkdir /dev/blockioctl/cgroup-1>> ï¬ mkdir /dev/blockioctl/cgroup-2>> ï¬ mkdir /dev/blockioctl/cgroup-3>> ï¬ echo /dev/sdb:$((1024*1024)):0:0 >>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max>> ï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks>> ï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks>> ï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks>> ï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30>> -verbose –dio(enable or disable)>>>> - setting status information>>>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max>> 8 16 1048576 0 0 0 13016>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max>> 8 16 2097152 0 0 0 11763>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max>> 8 16 3145728 0 0 0 11133>>>> - Test Result>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 1 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 1 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 1 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio –verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 1 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 1 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 1 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>>>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 1 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 1 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 1 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 1 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 1 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 1 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>>>> Best Regards,>> Dong-Jae Kang>>>>>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>>> The objective of the i/o controller is to improve i/o performance>>> predictability of different cgroups sharing the same block devices.>>>>>> Respect to other priority/weight-based solutions the approach used by this>>> controller is to explicitly choke applications' requests that directly (or>>> indirectly) generate i/o activity in the system.>>>>>> The direct bandwidth and/or iops limiting method has the advantage of improving>>> the performance predictability at the cost of reducing, in general, the overall>>> performance of the system (in terms of throughput).>>>>>> Detailed informations about design, its goal and usage are described in the>>> documentation.>>>>>> Patchset against 2.6.27-rc5-mm1:>>>>>> [PATCH 0/6] cgroup: block device i/o controller (v11)>>> [PATCH 1/6] i/o controller documentation>>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>>> [PATCH 3/6] i/o controller infrastructure>>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling>>> [PATCH 6/6] export per-task i/o throttling statistics to userspace>>>>>> The all-in-one patch (and previous versions) can be found at:>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>>>>>> There are no significant changes respect to v10, I've only implemented/fixed>>> some suggestions I received.>>>>>> Changelog: (v10 -> v11)>>>>>> * report per block device i/o statistics (total bytes read/written and iops)>>> in blockio.stat for i/o limited cgroups>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>>> /proc/PID/io-throttle-stat (suggested by David Radford)>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code>>> duplication (suggested by Paul Manage)>>> * use kernel-doc style for documenting struct res_counter attributes>>> (suggested by Randy Dunalp)>>> * udpated documentation>>>>>> Thanks to all for the feedback!>>> -Andrea> -- ------------------------------------------------------------------------------------------------- DONG-JAE, KANG Senior Member of Engineering Staff Internet Platform Research Dept, S/W Content Research Lab Electronics and Telecommunications Research Institute(ETRI) 138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA Phone : 82-42-860-1561 Fax : 82-42-860-6699 Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)-------------------------------------------------------------------------------------------------ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11) 2008-10-15 6:13 ` Dong-Jae Kang @ 2008-10-16 9:00 ` Andrea Righi 0 siblings, 0 replies; 5+ messages in thread From: Andrea Righi @ 2008-10-16 9:00 UTC (permalink / raw) To: Dong-Jae Kang Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde, dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi, Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto, Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya, containers, linux-kernel Dong-Jae Kang wrote: >>> I have several question about io-throttle >>> below is my test reusult of io-throttle(v11) with xdd 6.5 >>> But, I think I have something wrong, as showed in result >>> In direct IO mode, Only read operation was controlled by io-throttle >>> Can you check my test procedure and result and comments to me about that >> Your procedure is correct. Anyway, you found a known bug in io-throttle >> v11. If you want to properly use it you need to mount the memory >> controller together with blockio, since currently blockio depends on it >> to retrieve the owner of a page during writes in submit_bio(). >> >> As reported in: >> >> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity >> >> this is no more than a hack and in perspective a more generic framework >> able to provide this functionality should be used (i.e. bio-cgroup). > > But, In my opinion, > it is some strange that the bandwidth of write operation in direct IO > mode was not controlled by io-throttle > I think [PATCH -mm 4/6] is for control of buffered IO. > Do I misunderstand it? The approach is the same both for buffered and direct IO. With buffered IO, writes are accounted in submit_bio() and throttling is performed in balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always performed in submit_bio() and throttling in submit_page_section(). That's the same also for AIO, accounting in submit_bio() and throttling in io_submit_one(), returning -EAGAIN in this case when the IO limits are exceeded (instead of sleeping). We never sleep in submit_bio() during writes and to account the IO activity we always look at the owner (cgroup) of the first page in the struct bio, also when the IO context is the cgroup of the current task (as in the direct IO case). To correctly retrieve the owner of each page I've used the memory controller functionality. This part should be improved to use a more generic framework and remove the dependency of the memory controller. Or better, doesn't impose to mount blockio and memory controller both to the same mountpoint. -Andrea ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2008-10-16 9:01 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi 2008-10-14 9:17 ` Dong-Jae Kang 2008-10-14 9:56 ` Andrea Righi 2008-10-15 6:13 ` Dong-Jae Kang 2008-10-16 9:00 ` Andrea Righi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox