* [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
@ 2008-10-07 10:03 Andrea Righi
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-14 9:17 ` Dong-Jae Kang
0 siblings, 2 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-07 10:03 UTC (permalink / raw)
To: Balbir Singh, Paul Menage
Cc: agk, akpm, axboe, baramsori72, Carl Henrik Lunde, dave,
Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
The objective of the i/o controller is to improve i/o performance
predictability of different cgroups sharing the same block devices.
Respect to other priority/weight-based solutions the approach used by this
controller is to explicitly choke applications' requests that directly (or
indirectly) generate i/o activity in the system.
The direct bandwidth and/or iops limiting method has the advantage of improving
the performance predictability at the cost of reducing, in general, the overall
performance of the system (in terms of throughput).
Detailed informations about design, its goal and usage are described in the
documentation.
Patchset against 2.6.27-rc5-mm1:
[PATCH 0/6] cgroup: block device i/o controller (v11)
[PATCH 1/6] i/o controller documentation
[PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
[PATCH 3/6] i/o controller infrastructure
[PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
[PATCH 5/6] i/o controller instrumentation: accounting and throttling
[PATCH 6/6] export per-task i/o throttling statistics to userspace
The all-in-one patch (and previous versions) can be found at:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/
There are no significant changes respect to v10, I've only implemented/fixed
some suggestions I received.
Changelog: (v10 -> v11)
* report per block device i/o statistics (total bytes read/written and iops)
in blockio.stat for i/o limited cgroups
* distinct bandwidth and iops statistics: both in blockio.throttlecnt and
/proc/PID/io-throttle-stat (suggested by David Radford)
* merge res_counter_ratelimit functionality into res_counter, to avoid code
duplication (suggested by Paul Manage)
* use kernel-doc style for documenting struct res_counter attributes
(suggested by Randy Dunalp)
* udpated documentation
Thanks to all for the feedback!
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
@ 2008-10-07 10:03 Andrea Righi
0 siblings, 0 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-07 10:03 UTC (permalink / raw)
To: Balbir Singh, Paul Menage
Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Carl Henrik Lunde,
Divyesh Shah, eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w,
fernando-gVGce1chcLdL9jVzuh4AOg,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw,
subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI,
ngupta-hpIqsD4AKlfQT0dZR+AlfA
The objective of the i/o controller is to improve i/o performance
predictability of different cgroups sharing the same block devices.
Respect to other priority/weight-based solutions the approach used by this
controller is to explicitly choke applications' requests that directly (or
indirectly) generate i/o activity in the system.
The direct bandwidth and/or iops limiting method has the advantage of improving
the performance predictability at the cost of reducing, in general, the overall
performance of the system (in terms of throughput).
Detailed informations about design, its goal and usage are described in the
documentation.
Patchset against 2.6.27-rc5-mm1:
[PATCH 0/6] cgroup: block device i/o controller (v11)
[PATCH 1/6] i/o controller documentation
[PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
[PATCH 3/6] i/o controller infrastructure
[PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
[PATCH 5/6] i/o controller instrumentation: accounting and throttling
[PATCH 6/6] export per-task i/o throttling statistics to userspace
The all-in-one patch (and previous versions) can be found at:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/
There are no significant changes respect to v10, I've only implemented/fixed
some suggestions I received.
Changelog: (v10 -> v11)
* report per block device i/o statistics (total bytes read/written and iops)
in blockio.stat for i/o limited cgroups
* distinct bandwidth and iops statistics: both in blockio.throttlecnt and
/proc/PID/io-throttle-stat (suggested by David Radford)
* merge res_counter_ratelimit functionality into res_counter, to avoid code
duplication (suggested by Paul Manage)
* use kernel-doc style for documenting struct res_counter attributes
(suggested by Randy Dunalp)
* udpated documentation
Thanks to all for the feedback!
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
@ 2008-10-14 9:17 ` Dong-Jae Kang
2008-11-03 7:01 ` Gui Jianfeng
1 sibling, 0 replies; 13+ messages in thread
From: Dong-Jae Kang @ 2008-10-14 9:17 UTC (permalink / raw)
To: Andrea Righi
Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage,
Carl Henrik Lunde, Divyesh Shah,
eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh,
fernando-gVGce1chcLdL9jVzuh4AOg,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw,
subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI,
ngupta-hpIqsD4AKlfQT0dZR+AlfA
Hi, Andrea
thank you for your contribution to community
these days, I am testing several IO controllers in container ML,
dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your
io-throttle(v11)
I have several question about io-throttle
below is my test reusult of io-throttle(v11) with xdd 6.5
But, I think I have something wrong, as showed in result
In direct IO mode, Only read operation was controlled by io-throttle
Can you check my test procedure and result and comments to me about that
additionally, your testing shell script(run_io_throttle_test.sh) for
io-throttle was not updated for new io-throttle
so, it could be operated after I fixed it
-----------------------------------------------------------------------------------
- Test System Information
Computer Name, localhost.localdomain, User Name, root
OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008
Machine hardware type, i686
Number of processors on this system, 1
Page size in bytes, 4096
Number of physical pages, 515885
Megabytes of physical memory, 2015
Target[0] Q[0], /dev/sdb
Per-pass time limit in seconds, 30
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 16384
Number of MegaBytes, 512 or 1024
Direct I/O, disabled or enable
Seek pattern, sequential
Queue Depth, 1
- Test Procedure
mkdir /dev/blockioctl
mount -t cgroup -o blockio cgroup /dev/blockioctl
mkdir /dev/blockioctl/cgroup-1
mkdir /dev/blockioctl/cgroup-2
mkdir /dev/blockioctl/cgroup-3
echo /dev/sdb:$((1024*1024)):0:0 >
/dev/blockioctl/cgroup-1/blockio.bandwidth-max
echo /dev/sdb:$((2*1024*1024)):0:0 >
/dev/blockioctl/cgroup-2/blockio.bandwidth-max
echo /dev/sdb:$((3*1024*1024)):0:0 >
/dev/blockioctl/cgroup-3/blockio.bandwidth-max
in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks
in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks
in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks
in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb
-blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30
-verbose –dio(enable or disable)
- setting status information
[root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max
8 16 1048576 0 0 0 13016
[root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max
8 16 2097152 0 0 0 11763
[root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max
8 16 3145728 0 0 0 11133
- Test Result
xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
-mbytes 512 -timelimit 30 -dio -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 31522816 481 30.005 1.051 16.03 0.0624
0.00 read 65536
0 1 31522816 481 30.005 1.051 16.03 0.0624
0.00 read 65536
1 1 31522816 481 30.005 1.051 16.03 0.0624
0.00 read 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 62980096 961 30.001 2.099 32.03 0.0312
0.00 read 65536
0 1 62980096 961 30.001 2.099 32.03 0.0312
0.00 read 65536
1 1 62980096 961 30.001 2.099 32.03 0.0312
0.00 read 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 94437376 1441 30.003 3.148 48.03 0.0208
0.00 read 65536
0 1 94437376 1441 30.003 3.148 48.03 0.0208
0.00 read 65536
1 1 94437376 1441 30.003 3.148 48.03 0.0208
0.00 read 65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
-mbytes 512 -timelimit 30 -dio –verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 640221184 9769 30.097 21.272 324.58 0.0031
0.00 write 65536
0 1 640221184 9769 30.097 21.272 324.58 0.0031
0.00 write 65536
1 1 640221184 9769 30.097 21.272 324.58 0.0031
0.00 write 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 633798656 9671 30.001 21.126 322.36 0.0031
0.00 write 65536
0 1 633798656 9671 30.001 21.126 322.36 0.0031
0.00 write 65536
1 1 633798656 9671 30.001 21.126 322.36 0.0031
0.00 write 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 630652928 9623 30.001 21.021 320.76 0.0031
0.00 write 65536
0 1 630652928 9623 30.001 21.021 320.76 0.0031
0.00 write 65536
1 1 630652928 9623 30.001 21.021 320.76 0.0031
0.00 write 65536
xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
-mbytes 1024 -timelimit 30 -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 70123520 1070 30.150 2.326 35.49 0.0282
0.00 read 65536
0 1 70123520 1070 30.150 2.326 35.49 0.0282
0.00 read 65536
1 1 70123520 1070 30.150 2.326 35.49 0.0282
0.00 read 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 70844416 1081 30.063 2.357 35.96 0.0278
0.00 read 65536
0 1 70844416 1081 30.063 2.357 35.96 0.0278
0.00 read 65536
1 1 70844416 1081 30.063 2.357 35.96 0.0278
0.00 read 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 72155136 1101 30.204 2.389 36.45 0.0274
0.00 read 65536
0 1 72155136 1101 30.204 2.389 36.45 0.0274
0.00 read 65536
1 1 72155136 1101 30.204 2.389 36.45 0.0274
0.00 read 65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
-mbytes 1024 -timelimit 30 -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 818610176 12491 30.031 27.258 415.93 0.0024
0.00 write 65536
0 1 818610176 12491 30.031 27.258 415.93 0.0024
0.00 write 65536
1 1 818610176 12491 30.031 27.258 415.93 0.0024
0.00 write 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 848494592 12947 30.066 28.221 430.62 0.0023
0.00 write 65536
0 1 848494592 12947 30.066 28.221 430.62 0.0023
0.00 write 65536
1 1 848494592 12947 30.066 28.221 430.62 0.0023
0.00 write 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 786563072 12002 30.078 26.151 399.03 0.0025
0.00 write 65536
0 1 786563072 12002 30.078 26.151 399.03 0.0025
0.00 write 65536
1 1 786563072 12002 30.078 26.151 399.03 0.0025
0.00 write 65536
Best Regards,
Dong-Jae Kang
2008/10/7 Andrea Righi <righi.andrea@gmail.com>:
>
> The objective of the i/o controller is to improve i/o performance
> predictability of different cgroups sharing the same block devices.
>
> Respect to other priority/weight-based solutions the approach used by this
> controller is to explicitly choke applications' requests that directly (or
> indirectly) generate i/o activity in the system.
>
> The direct bandwidth and/or iops limiting method has the advantage of improving
> the performance predictability at the cost of reducing, in general, the overall
> performance of the system (in terms of throughput).
>
> Detailed informations about design, its goal and usage are described in the
> documentation.
>
> Patchset against 2.6.27-rc5-mm1:
>
> [PATCH 0/6] cgroup: block device i/o controller (v11)
> [PATCH 1/6] i/o controller documentation
> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
> [PATCH 3/6] i/o controller infrastructure
> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
> [PATCH 5/6] i/o controller instrumentation: accounting and throttling
> [PATCH 6/6] export per-task i/o throttling statistics to userspace
>
> The all-in-one patch (and previous versions) can be found at:
> http://download.systemimager.org/~arighi/linux/patches/io-throttle/
>
> There are no significant changes respect to v10, I've only implemented/fixed
> some suggestions I received.
>
> Changelog: (v10 -> v11)
>
> * report per block device i/o statistics (total bytes read/written and iops)
> in blockio.stat for i/o limited cgroups
> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and
> /proc/PID/io-throttle-stat (suggested by David Radford)
> * merge res_counter_ratelimit functionality into res_counter, to avoid code
> duplication (suggested by Paul Manage)
> * use kernel-doc style for documenting struct res_counter attributes
> (suggested by Randy Dunalp)
> * udpated documentation
>
> Thanks to all for the feedback!
> -Andrea
>
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
@ 2008-10-14 9:17 ` Dong-Jae Kang
[not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-14 9:56 ` Andrea Righi
1 sibling, 2 replies; 13+ messages in thread
From: Dong-Jae Kang @ 2008-10-14 9:17 UTC (permalink / raw)
To: Andrea Righi
Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde,
dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 9545 bytes --]
Hi, Andrea
thank you for your contribution to communitythese days, I am testing several IO controllers in container ML,dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and yourio-throttle(v11)
I have several question about io-throttlebelow is my test reusult of io-throttle(v11) with xdd 6.5But, I think I have something wrong, as showed in resultIn direct IO mode, Only read operation was controlled by io-throttleCan you check my test procedure and result and comments to me about that
additionally, your testing shell script(run_io_throttle_test.sh) forio-throttle was not updated for new io-throttleso, it could be operated after I fixed it
------------------------------------------------------------------------------------ Test System Information
Computer Name, localhost.localdomain, User Name, rootOS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008Machine hardware type, i686Number of processors on this system, 1Page size in bytes, 4096Number of physical pages, 515885Megabytes of physical memory, 2015Target[0] Q[0], /dev/sdbPer-pass time limit in seconds, 30Blocksize in bytes, 512Request size, 128, blocks, 65536, bytesNumber of Requests, 16384Number of MegaBytes, 512 or 1024Direct I/O, disabled or enableSeek pattern, sequentialQueue Depth, 1
- Test Procedure
ï¬ mkdir /dev/blockioctlï¬ mount -t cgroup -o blockio cgroup /dev/blockioctlï¬ mkdir /dev/blockioctl/cgroup-1ï¬ mkdir /dev/blockioctl/cgroup-2ï¬ mkdir /dev/blockioctl/cgroup-3ï¬ echo /dev/sdb:$((1024*1024)):0:0 >/dev/blockioctl/cgroup-1/blockio.bandwidth-maxï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >/dev/blockioctl/cgroup-2/blockio.bandwidth-maxï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >/dev/blockioctl/cgroup-3/blockio.bandwidth-maxï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasksï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasksï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasksï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb-blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30-verbose –dio(enable or disable)
- setting status information
[root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max8 16 1048576 0 0 0 13016[root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max8 16 2097152 0 0 0 11763[root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max8 16 3145728 0 0 0 11133
- Test Resultxdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655360 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 655361 1 31522816 481 30.005 1.051 16.03 0.0624 0.00 read 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655360 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 655361 1 62980096 961 30.001 2.099 32.03 0.0312 0.00 read 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655360 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 655361 1 94437376 1441 30.003 3.148 48.03 0.0208 0.00 read 65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio –verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655360 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 655361 1 640221184 9769 30.097 21.272 324.58 0.0031 0.00 write 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655360 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 655361 1 633798656 9671 30.001 21.126 322.36 0.0031 0.00 write 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655360 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 655361 1 630652928 9623 30.001 21.021 320.76 0.0031 0.00 write 65536
xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655360 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 655361 1 70123520 1070 30.150 2.326 35.49 0.0282 0.00 read 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655360 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 655361 1 70844416 1081 30.063 2.357 35.96 0.0278 0.00 read 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655360 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 655361 1 72155136 1101 30.204 2.389 36.45 0.0274 0.00 read 65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024 -timelimit 30 -verbose
cgroup-1
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655360 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 655361 1 818610176 12491 30.031 27.258 415.93 0.0024 0.00 write 65536
cgroup-2
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655360 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 655361 1 848494592 12947 30.066 28.221 430.62 0.0023 0.00 write 65536
cgroup-3
T Q Bytes Ops Time Rate IOPS Latency%CPU OP_Type ReqSize0 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655360 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 655361 1 786563072 12002 30.078 26.151 399.03 0.0025 0.00 write 65536
Best Regards,Dong-Jae Kang
2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>> The objective of the i/o controller is to improve i/o performance> predictability of different cgroups sharing the same block devices.>> Respect to other priority/weight-based solutions the approach used by this> controller is to explicitly choke applications' requests that directly (or> indirectly) generate i/o activity in the system.>> The direct bandwidth and/or iops limiting method has the advantage of improving> the performance predictability at the cost of reducing, in general, the overall> performance of the system (in terms of throughput).>> Detailed informations about design, its goal and usage are described in the> documentation.>> Patchset against 2.6.27-rc5-mm1:>> [PATCH 0/6] cgroup: block device i/o controller (v11)> [PATCH 1/6] i/o controller documentation> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter> [PATCH 3/6] i/o controller infrastructure> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity> [PATCH 5/6] i/o controller instrumentation: accounting and throttling> [PATCH 6/6] export per-task i/o throttling statistics to userspace>> The all-in-one patch (and previous versions) can be found at:> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>> There are no significant changes respect to v10, I've only implemented/fixed> some suggestions I received.>> Changelog: (v10 -> v11)>> * report per block device i/o statistics (total bytes read/written and iops)> in blockio.stat for i/o limited cgroups> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and> /proc/PID/io-throttle-stat (suggested by David Radford)> * merge res_counter_ratelimit functionality into res_counter, to avoid code> duplication (suggested by Paul Manage)> * use kernel-doc style for documenting struct res_counter attributes> (suggested by Randy Dunalp)> * udpated documentation>> Thanks to all for the feedback!> -Andrea>ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2008-10-14 9:56 ` Andrea Righi
0 siblings, 0 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-14 9:56 UTC (permalink / raw)
To: Dong-Jae Kang
Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage,
Carl Henrik Lunde, Divyesh Shah,
eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh,
fernando-gVGce1chcLdL9jVzuh4AOg,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw,
subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI,
ngupta-hpIqsD4AKlfQT0dZR+AlfA
Dong-Jae Kang wrote:
> Hi, Andrea
>
> thank you for your contribution to community
> these days, I am testing several IO controllers in container ML,
> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your
> io-throttle(v11)
Thanks! this is surely a valuable task.
>
> I have several question about io-throttle
> below is my test reusult of io-throttle(v11) with xdd 6.5
> But, I think I have something wrong, as showed in result
> In direct IO mode, Only read operation was controlled by io-throttle
> Can you check my test procedure and result and comments to me about that
Your procedure is correct. Anyway, you found a known bug in io-throttle
v11. If you want to properly use it you need to mount the memory
controller together with blockio, since currently blockio depends on it
to retrieve the owner of a page during writes in submit_bio().
As reported in:
[PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
this is no more than a hack and in perspective a more generic framework
able to provide this functionality should be used (i.e. bio-cgroup).
I'll fix this issue in the next version of io-throttle (probably I'll
try to rewrite io-throttle on top of bio-cgroup), but for now the
workaround is to mount the cgroupfs using -o blockio,memory (at least).
>
> additionally, your testing shell script(run_io_throttle_test.sh) for
> io-throttle was not updated for new io-throttle
> so, it could be operated after I fixed it
The testing of iops limiting is not yet implemented and I don't have a
very good testcase for this, but I can share with you a small script that
I'm using to check if iops limiting is working or not, if you're interested.
Thanks,
-Andrea
>
> -----------------------------------------------------------------------------------
> - Test System Information
>
> Computer Name, localhost.localdomain, User Name, root
> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008
> Machine hardware type, i686
> Number of processors on this system, 1
> Page size in bytes, 4096
> Number of physical pages, 515885
> Megabytes of physical memory, 2015
> Target[0] Q[0], /dev/sdb
> Per-pass time limit in seconds, 30
> Blocksize in bytes, 512
> Request size, 128, blocks, 65536, bytes
> Number of Requests, 16384
> Number of MegaBytes, 512 or 1024
> Direct I/O, disabled or enable
> Seek pattern, sequential
> Queue Depth, 1
>
> - Test Procedure
>
> mkdir /dev/blockioctl
> mount -t cgroup -o blockio cgroup /dev/blockioctl
> mkdir /dev/blockioctl/cgroup-1
> mkdir /dev/blockioctl/cgroup-2
> mkdir /dev/blockioctl/cgroup-3
> echo /dev/sdb:$((1024*1024)):0:0 >
> /dev/blockioctl/cgroup-1/blockio.bandwidth-max
> echo /dev/sdb:$((2*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-2/blockio.bandwidth-max
> echo /dev/sdb:$((3*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-3/blockio.bandwidth-max
> in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks
> in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks
> in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks
> in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb
> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30
> -verbose –dio(enable or disable)
>
> - setting status information
>
> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max
> 8 16 1048576 0 0 0 13016
> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max
> 8 16 2097152 0 0 0 11763
> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max
> 8 16 3145728 0 0 0 11133
>
> - Test Result
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
> 0 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
> 1 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
> 0 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
> 1 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
> 1 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
>
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio –verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
> 1 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
> 1 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
> 1 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
>
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024 -timelimit 30 -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
> 1 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
> 1 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
> 1 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
>
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024 -timelimit 30 -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
> 1 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
> 1 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
> 1 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
>
> Best Regards,
> Dong-Jae Kang
>
>
> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:
>> The objective of the i/o controller is to improve i/o performance
>> predictability of different cgroups sharing the same block devices.
>>
>> Respect to other priority/weight-based solutions the approach used by this
>> controller is to explicitly choke applications' requests that directly (or
>> indirectly) generate i/o activity in the system.
>>
>> The direct bandwidth and/or iops limiting method has the advantage of improving
>> the performance predictability at the cost of reducing, in general, the overall
>> performance of the system (in terms of throughput).
>>
>> Detailed informations about design, its goal and usage are described in the
>> documentation.
>>
>> Patchset against 2.6.27-rc5-mm1:
>>
>> [PATCH 0/6] cgroup: block device i/o controller (v11)
>> [PATCH 1/6] i/o controller documentation
>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
>> [PATCH 3/6] i/o controller infrastructure
>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling
>> [PATCH 6/6] export per-task i/o throttling statistics to userspace
>>
>> The all-in-one patch (and previous versions) can be found at:
>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/
>>
>> There are no significant changes respect to v10, I've only implemented/fixed
>> some suggestions I received.
>>
>> Changelog: (v10 -> v11)
>>
>> * report per block device i/o statistics (total bytes read/written and iops)
>> in blockio.stat for i/o limited cgroups
>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and
>> /proc/PID/io-throttle-stat (suggested by David Radford)
>> * merge res_counter_ratelimit functionality into res_counter, to avoid code
>> duplication (suggested by Paul Manage)
>> * use kernel-doc style for documenting struct res_counter attributes
>> (suggested by Randy Dunalp)
>> * udpated documentation
>>
>> Thanks to all for the feedback!
>> -Andrea
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
2008-10-14 9:17 ` Dong-Jae Kang
[not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2008-10-14 9:56 ` Andrea Righi
2008-10-15 6:13 ` Dong-Jae Kang
[not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
1 sibling, 2 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-14 9:56 UTC (permalink / raw)
To: Dong-Jae Kang
Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde,
dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
Dong-Jae Kang wrote:
> Hi, Andrea
>
> thank you for your contribution to community
> these days, I am testing several IO controllers in container ML,
> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your
> io-throttle(v11)
Thanks! this is surely a valuable task.
>
> I have several question about io-throttle
> below is my test reusult of io-throttle(v11) with xdd 6.5
> But, I think I have something wrong, as showed in result
> In direct IO mode, Only read operation was controlled by io-throttle
> Can you check my test procedure and result and comments to me about that
Your procedure is correct. Anyway, you found a known bug in io-throttle
v11. If you want to properly use it you need to mount the memory
controller together with blockio, since currently blockio depends on it
to retrieve the owner of a page during writes in submit_bio().
As reported in:
[PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
this is no more than a hack and in perspective a more generic framework
able to provide this functionality should be used (i.e. bio-cgroup).
I'll fix this issue in the next version of io-throttle (probably I'll
try to rewrite io-throttle on top of bio-cgroup), but for now the
workaround is to mount the cgroupfs using -o blockio,memory (at least).
>
> additionally, your testing shell script(run_io_throttle_test.sh) for
> io-throttle was not updated for new io-throttle
> so, it could be operated after I fixed it
The testing of iops limiting is not yet implemented and I don't have a
very good testcase for this, but I can share with you a small script that
I'm using to check if iops limiting is working or not, if you're interested.
Thanks,
-Andrea
>
> -----------------------------------------------------------------------------------
> - Test System Information
>
> Computer Name, localhost.localdomain, User Name, root
> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008
> Machine hardware type, i686
> Number of processors on this system, 1
> Page size in bytes, 4096
> Number of physical pages, 515885
> Megabytes of physical memory, 2015
> Target[0] Q[0], /dev/sdb
> Per-pass time limit in seconds, 30
> Blocksize in bytes, 512
> Request size, 128, blocks, 65536, bytes
> Number of Requests, 16384
> Number of MegaBytes, 512 or 1024
> Direct I/O, disabled or enable
> Seek pattern, sequential
> Queue Depth, 1
>
> - Test Procedure
>
> mkdir /dev/blockioctl
> mount -t cgroup -o blockio cgroup /dev/blockioctl
> mkdir /dev/blockioctl/cgroup-1
> mkdir /dev/blockioctl/cgroup-2
> mkdir /dev/blockioctl/cgroup-3
> echo /dev/sdb:$((1024*1024)):0:0 >
> /dev/blockioctl/cgroup-1/blockio.bandwidth-max
> echo /dev/sdb:$((2*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-2/blockio.bandwidth-max
> echo /dev/sdb:$((3*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-3/blockio.bandwidth-max
> in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks
> in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks
> in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks
> in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb
> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30
> -verbose –dio(enable or disable)
>
> - setting status information
>
> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max
> 8 16 1048576 0 0 0 13016
> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max
> 8 16 2097152 0 0 0 11763
> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max
> 8 16 3145728 0 0 0 11133
>
> - Test Result
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
> 0 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
> 1 1 31522816 481 30.005 1.051 16.03 0.0624
> 0.00 read 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
> 0 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
> 1 1 62980096 961 30.001 2.099 32.03 0.0312
> 0.00 read 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
> 1 1 94437376 1441 30.003 3.148 48.03 0.0208
> 0.00 read 65536
>
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio –verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
> 1 1 640221184 9769 30.097 21.272 324.58 0.0031
> 0.00 write 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
> 1 1 633798656 9671 30.001 21.126 322.36 0.0031
> 0.00 write 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
> 1 1 630652928 9623 30.001 21.021 320.76 0.0031
> 0.00 write 65536
>
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024 -timelimit 30 -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
> 1 1 70123520 1070 30.150 2.326 35.49 0.0282
> 0.00 read 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
> 1 1 70844416 1081 30.063 2.357 35.96 0.0278
> 0.00 read 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
> 1 1 72155136 1101 30.204 2.389 36.45 0.0274
> 0.00 read 65536
>
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024 -timelimit 30 -verbose
>
> cgroup-1
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
> 1 1 818610176 12491 30.031 27.258 415.93 0.0024
> 0.00 write 65536
>
> cgroup-2
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
> 1 1 848494592 12947 30.066 28.221 430.62 0.0023
> 0.00 write 65536
>
> cgroup-3
>
> T Q Bytes Ops Time Rate IOPS Latency
> %CPU OP_Type ReqSize
> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
> 1 1 786563072 12002 30.078 26.151 399.03 0.0025
> 0.00 write 65536
>
> Best Regards,
> Dong-Jae Kang
>
>
> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:
>> The objective of the i/o controller is to improve i/o performance
>> predictability of different cgroups sharing the same block devices.
>>
>> Respect to other priority/weight-based solutions the approach used by this
>> controller is to explicitly choke applications' requests that directly (or
>> indirectly) generate i/o activity in the system.
>>
>> The direct bandwidth and/or iops limiting method has the advantage of improving
>> the performance predictability at the cost of reducing, in general, the overall
>> performance of the system (in terms of throughput).
>>
>> Detailed informations about design, its goal and usage are described in the
>> documentation.
>>
>> Patchset against 2.6.27-rc5-mm1:
>>
>> [PATCH 0/6] cgroup: block device i/o controller (v11)
>> [PATCH 1/6] i/o controller documentation
>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
>> [PATCH 3/6] i/o controller infrastructure
>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling
>> [PATCH 6/6] export per-task i/o throttling statistics to userspace
>>
>> The all-in-one patch (and previous versions) can be found at:
>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/
>>
>> There are no significant changes respect to v10, I've only implemented/fixed
>> some suggestions I received.
>>
>> Changelog: (v10 -> v11)
>>
>> * report per block device i/o statistics (total bytes read/written and iops)
>> in blockio.stat for i/o limited cgroups
>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and
>> /proc/PID/io-throttle-stat (suggested by David Radford)
>> * merge res_counter_ratelimit functionality into res_counter, to avoid code
>> duplication (suggested by Paul Manage)
>> * use kernel-doc style for documenting struct res_counter attributes
>> (suggested by Randy Dunalp)
>> * udpated documentation
>>
>> Thanks to all for the feedback!
>> -Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
@ 2008-10-15 6:13 ` Dong-Jae Kang
0 siblings, 0 replies; 13+ messages in thread
From: Dong-Jae Kang @ 2008-10-15 6:13 UTC (permalink / raw)
To: righi.andrea-Re5JQEeQqe8AvxtiuMwx3w
Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage,
Carl Henrik Lunde, Divyesh Shah,
eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh,
fernando-gVGce1chcLdL9jVzuh4AOg,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw,
subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI,
ngupta-hpIqsD4AKlfQT0dZR+AlfA
Hi, Andrea
Thank you for your comments
> Dong-Jae Kang wrote:
>> Hi, Andrea
>>
>> thank you for your contribution to community
>> these days, I am testing several IO controllers in container ML,
>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your
>> io-throttle(v11)
>
> Thanks! this is surely a valuable task.
>
>>
>> I have several question about io-throttle
>> below is my test reusult of io-throttle(v11) with xdd 6.5
>> But, I think I have something wrong, as showed in result
>> In direct IO mode, Only read operation was controlled by io-throttle
>> Can you check my test procedure and result and comments to me about that
>
> Your procedure is correct. Anyway, you found a known bug in io-throttle
> v11. If you want to properly use it you need to mount the memory
> controller together with blockio, since currently blockio depends on it
> to retrieve the owner of a page during writes in submit_bio().
>
> As reported in:
>
> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>
> this is no more than a hack and in perspective a more generic framework
> able to provide this functionality should be used (i.e. bio-cgroup).
But, In my opinion,
it is some strange that the bandwidth of write operation in direct IO
mode was not controlled by io-throttle
I think [PATCH -mm 4/6] is for control of buffered IO.
Do I misunderstand it?
> I'll fix this issue in the next version of io-throttle (probably I'll
> try to rewrite io-throttle on top of bio-cgroup), but for now the
> workaround is to mount the cgroupfs using -o blockio,memory (at least).
>
Oh, it sounds good.
I look foward to your next io-throttle
>>
>> additionally, your testing shell script(run_io_throttle_test.sh) for
>> io-throttle was not updated for new io-throttle
>> so, it could be operated after I fixed it
>
> The testing of iops limiting is not yet implemented and I don't have a
> very good testcase for this, but I can share with you a small script that
> I'm using to check if iops limiting is working or not, if you're interested.
thank you, Andrea.
I want to check iops limiting of io-throttle is working well.
Have a nice day...
>
>>
>> -----------------------------------------------------------------------------------
>> - Test System Information
>>
>> Computer Name, localhost.localdomain, User Name, root
>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008
>> Machine hardware type, i686
>> Number of processors on this system, 1
>> Page size in bytes, 4096
>> Number of physical pages, 515885
>> Megabytes of physical memory, 2015
>> Target[0] Q[0], /dev/sdb
>> Per-pass time limit in seconds, 30
>> Blocksize in bytes, 512
>> Request size, 128, blocks, 65536, bytes
>> Number of Requests, 16384
>> Number of MegaBytes, 512 or 1024
>> Direct I/O, disabled or enable
>> Seek pattern, sequential
>> Queue Depth, 1
>>
>> - Test Procedure
>>
>> mkdir /dev/blockioctl
>> mount -t cgroup -o blockio cgroup /dev/blockioctl
>> mkdir /dev/blockioctl/cgroup-1
>> mkdir /dev/blockioctl/cgroup-2
>> mkdir /dev/blockioctl/cgroup-3
>> echo /dev/sdb:$((1024*1024)):0:0 >
>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max
>> echo /dev/sdb:$((2*1024*1024)):0:0 >
>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max
>> echo /dev/sdb:$((3*1024*1024)):0:0 >
>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max
>> in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks
>> in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks
>> in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks
>> in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb
>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30
>> -verbose –dio(enable or disable)
>>
>> - setting status information
>>
>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max
>> 8 16 1048576 0 0 0 13016
>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max
>> 8 16 2097152 0 0 0 11763
>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max
>> 8 16 3145728 0 0 0 11133
>>
>> - Test Result
>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
>> -mbytes 512 -timelimit 30 -dio -verbose
>>
>> cgroup-1
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 31522816 481 30.005 1.051 16.03 0.0624
>> 0.00 read 65536
>> 0 1 31522816 481 30.005 1.051 16.03 0.0624
>> 0.00 read 65536
>> 1 1 31522816 481 30.005 1.051 16.03 0.0624
>> 0.00 read 65536
>>
>> cgroup-2
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 62980096 961 30.001 2.099 32.03 0.0312
>> 0.00 read 65536
>> 0 1 62980096 961 30.001 2.099 32.03 0.0312
>> 0.00 read 65536
>> 1 1 62980096 961 30.001 2.099 32.03 0.0312
>> 0.00 read 65536
>>
>> cgroup-3
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
>> 0.00 read 65536
>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208
>> 0.00 read 65536
>> 1 1 94437376 1441 30.003 3.148 48.03 0.0208
>> 0.00 read 65536
>>
>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
>> -mbytes 512 -timelimit 30 -dio –verbose
>>
>> cgroup-1
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
>> 0.00 write 65536
>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031
>> 0.00 write 65536
>> 1 1 640221184 9769 30.097 21.272 324.58 0.0031
>> 0.00 write 65536
>>
>> cgroup-2
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
>> 0.00 write 65536
>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031
>> 0.00 write 65536
>> 1 1 633798656 9671 30.001 21.126 322.36 0.0031
>> 0.00 write 65536
>>
>> cgroup-3
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
>> 0.00 write 65536
>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031
>> 0.00 write 65536
>> 1 1 630652928 9623 30.001 21.021 320.76 0.0031
>> 0.00 write 65536
>>
>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
>> -mbytes 1024 -timelimit 30 -verbose
>>
>> cgroup-1
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
>> 0.00 read 65536
>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282
>> 0.00 read 65536
>> 1 1 70123520 1070 30.150 2.326 35.49 0.0282
>> 0.00 read 65536
>>
>> cgroup-2
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
>> 0.00 read 65536
>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278
>> 0.00 read 65536
>> 1 1 70844416 1081 30.063 2.357 35.96 0.0278
>> 0.00 read 65536
>>
>> cgroup-3
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
>> 0.00 read 65536
>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274
>> 0.00 read 65536
>> 1 1 72155136 1101 30.204 2.389 36.45 0.0274
>> 0.00 read 65536
>>
>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
>> -mbytes 1024 -timelimit 30 -verbose
>>
>> cgroup-1
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
>> 0.00 write 65536
>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024
>> 0.00 write 65536
>> 1 1 818610176 12491 30.031 27.258 415.93 0.0024
>> 0.00 write 65536
>>
>> cgroup-2
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
>> 0.00 write 65536
>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023
>> 0.00 write 65536
>> 1 1 848494592 12947 30.066 28.221 430.62 0.0023
>> 0.00 write 65536
>>
>> cgroup-3
>>
>> T Q Bytes Ops Time Rate IOPS Latency
>> %CPU OP_Type ReqSize
>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
>> 0.00 write 65536
>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025
>> 0.00 write 65536
>> 1 1 786563072 12002 30.078 26.151 399.03 0.0025
>> 0.00 write 65536
>>
>> Best Regards,
>> Dong-Jae Kang
>>
>>
>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:
>>> The objective of the i/o controller is to improve i/o performance
>>> predictability of different cgroups sharing the same block devices.
>>>
>>> Respect to other priority/weight-based solutions the approach used by this
>>> controller is to explicitly choke applications' requests that directly (or
>>> indirectly) generate i/o activity in the system.
>>>
>>> The direct bandwidth and/or iops limiting method has the advantage of improving
>>> the performance predictability at the cost of reducing, in general, the overall
>>> performance of the system (in terms of throughput).
>>>
>>> Detailed informations about design, its goal and usage are described in the
>>> documentation.
>>>
>>> Patchset against 2.6.27-rc5-mm1:
>>>
>>> [PATCH 0/6] cgroup: block device i/o controller (v11)
>>> [PATCH 1/6] i/o controller documentation
>>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
>>> [PATCH 3/6] i/o controller infrastructure
>>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling
>>> [PATCH 6/6] export per-task i/o throttling statistics to userspace
>>>
>>> The all-in-one patch (and previous versions) can be found at:
>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/
>>>
>>> There are no significant changes respect to v10, I've only implemented/fixed
>>> some suggestions I received.
>>>
>>> Changelog: (v10 -> v11)
>>>
>>> * report per block device i/o statistics (total bytes read/written and iops)
>>> in blockio.stat for i/o limited cgroups
>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and
>>> /proc/PID/io-throttle-stat (suggested by David Radford)
>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code
>>> duplication (suggested by Paul Manage)
>>> * use kernel-doc style for documenting struct res_counter attributes
>>> (suggested by Randy Dunalp)
>>> * udpated documentation
>>>
>>> Thanks to all for the feedback!
>>> -Andrea
>
--
-------------------------------------------------------------------------------------------------
DONG-JAE, KANG
Senior Member of Engineering Staff
Internet Platform Research Dept, S/W Content Research Lab
Electronics and Telecommunications Research Institute(ETRI)
138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA
Phone : 82-42-860-1561 Fax : 82-42-860-6699
Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)
-------------------------------------------------------------------------------------------------
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
2008-10-14 9:56 ` Andrea Righi
@ 2008-10-15 6:13 ` Dong-Jae Kang
[not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-16 9:00 ` Andrea Righi
[not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
1 sibling, 2 replies; 13+ messages in thread
From: Dong-Jae Kang @ 2008-10-15 6:13 UTC (permalink / raw)
To: righi.andrea
Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde,
dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 12215 bytes --]
Hi, AndreaThank you for your comments
> Dong-Jae Kang wrote:>> Hi, Andrea>>>> thank you for your contribution to community>> these days, I am testing several IO controllers in container ML,>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your>> io-throttle(v11)>> Thanks! this is surely a valuable task.>>>>> I have several question about io-throttle>> below is my test reusult of io-throttle(v11) with xdd 6.5>> But, I think I have something wrong, as showed in result>> In direct IO mode, Only read operation was controlled by io-throttle>> Can you check my test procedure and result and comments to me about that>> Your procedure is correct. Anyway, you found a known bug in io-throttle> v11. If you want to properly use it you need to mount the memory> controller together with blockio, since currently blockio depends on it> to retrieve the owner of a page during writes in submit_bio().>> As reported in:>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>> this is no more than a hack and in perspective a more generic framework> able to provide this functionality should be used (i.e. bio-cgroup).
But, In my opinion,it is some strange that the bandwidth of write operation in direct IOmode was not controlled by io-throttleI think [PATCH -mm 4/6] is for control of buffered IO.Do I misunderstand it?
> I'll fix this issue in the next version of io-throttle (probably I'll> try to rewrite io-throttle on top of bio-cgroup), but for now the> workaround is to mount the cgroupfs using -o blockio,memory (at least).>
Oh, it sounds good.I look foward to your next io-throttle
>>>> additionally, your testing shell script(run_io_throttle_test.sh) for>> io-throttle was not updated for new io-throttle>> so, it could be operated after I fixed it>> The testing of iops limiting is not yet implemented and I don't have a> very good testcase for this, but I can share with you a small script that> I'm using to check if iops limiting is working or not, if you're interested.
thank you, Andrea.I want to check iops limiting of io-throttle is working well.Have a nice day...>>>>> ----------------------------------------------------------------------------------->> - Test System Information>>>> Computer Name, localhost.localdomain, User Name, root>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008>> Machine hardware type, i686>> Number of processors on this system, 1>> Page size in bytes, 4096>> Number of physical pages, 515885>> Megabytes of physical memory, 2015>> Target[0] Q[0], /dev/sdb>> Per-pass time limit in seconds, 30>> Blocksize in bytes, 512>> Request size, 128, blocks, 65536, bytes>> Number of Requests, 16384>> Number of MegaBytes, 512 or 1024>> Direct I/O, disabled or enable>> Seek pattern, sequential>> Queue Depth, 1>>>> - Test Procedure>>>> ï¬ mkdir /dev/blockioctl>> ï¬ mount -t cgroup -o blockio cgroup /dev/blockioctl>> ï¬ mkdir /dev/blockioctl/cgroup-1>> ï¬ mkdir /dev/blockioctl/cgroup-2>> ï¬ mkdir /dev/blockioctl/cgroup-3>> ï¬ echo /dev/sdb:$((1024*1024)):0:0 >>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((2*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max>> ï¬ echo /dev/sdb:$((3*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max>> ï¬ in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks>> ï¬ in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks>> ï¬ in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks>> ï¬ in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 ) -timelimit 30>> -verbose –dio(enable or disable)>>>> - setting status information>>>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max>> 8 16 1048576 0 0 0 13016>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max>> 8 16 2097152 0 0 0 11763>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max>> 8 16 3145728 0 0 0 11133>>>> - Test Result>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 0 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>> 1 1 31522816 481 30.005 1.051 16.03 0.0624>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 0 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>> 1 1 62980096 961 30.001 2.099 32.03 0.0312>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 0 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>> 1 1 94437376 1441 30.003 3.148 48.03 0.0208>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio –verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 0 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>> 1 1 640221184 9769 30.097 21.272 324.58 0.0031>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 0 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>> 1 1 633798656 9671 30.001 21.126 322.36 0.0031>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 0 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>> 1 1 630652928 9623 30.001 21.021 320.76 0.0031>> 0.00 write 65536>>>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 0 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>> 1 1 70123520 1070 30.150 2.326 35.49 0.0282>> 0.00 read 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 0 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>> 1 1 70844416 1081 30.063 2.357 35.96 0.0278>> 0.00 read 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 0 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>> 1 1 72155136 1101 30.204 2.389 36.45 0.0274>> 0.00 read 65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024 -timelimit 30 -verbose>>>> cgroup-1>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 0 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>> 1 1 818610176 12491 30.031 27.258 415.93 0.0024>> 0.00 write 65536>>>> cgroup-2>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 0 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>> 1 1 848494592 12947 30.066 28.221 430.62 0.0023>> 0.00 write 65536>>>> cgroup-3>>>> T Q Bytes Ops Time Rate IOPS Latency>> %CPU OP_Type ReqSize>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 0 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>> 1 1 786563072 12002 30.078 26.151 399.03 0.0025>> 0.00 write 65536>>>> Best Regards,>> Dong-Jae Kang>>>>>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>>> The objective of the i/o controller is to improve i/o performance>>> predictability of different cgroups sharing the same block devices.>>>>>> Respect to other priority/weight-based solutions the approach used by this>>> controller is to explicitly choke applications' requests that directly (or>>> indirectly) generate i/o activity in the system.>>>>>> The direct bandwidth and/or iops limiting method has the advantage of improving>>> the performance predictability at the cost of reducing, in general, the overall>>> performance of the system (in terms of throughput).>>>>>> Detailed informations about design, its goal and usage are described in the>>> documentation.>>>>>> Patchset against 2.6.27-rc5-mm1:>>>>>> [PATCH 0/6] cgroup: block device i/o controller (v11)>>> [PATCH 1/6] i/o controller documentation>>> [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>>> [PATCH 3/6] i/o controller infrastructure>>> [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>>> [PATCH 5/6] i/o controller instrumentation: accounting and throttling>>> [PATCH 6/6] export per-task i/o throttling statistics to userspace>>>>>> The all-in-one patch (and previous versions) can be found at:>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>>>>>> There are no significant changes respect to v10, I've only implemented/fixed>>> some suggestions I received.>>>>>> Changelog: (v10 -> v11)>>>>>> * report per block device i/o statistics (total bytes read/written and iops)>>> in blockio.stat for i/o limited cgroups>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>>> /proc/PID/io-throttle-stat (suggested by David Radford)>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code>>> duplication (suggested by Paul Manage)>>> * use kernel-doc style for documenting struct res_counter attributes>>> (suggested by Randy Dunalp)>>> * udpated documentation>>>>>> Thanks to all for the feedback!>>> -Andrea>
-- ------------------------------------------------------------------------------------------------- DONG-JAE, KANG Senior Member of Engineering Staff Internet Platform Research Dept, S/W Content Research Lab Electronics and Telecommunications Research Institute(ETRI) 138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA Phone : 82-42-860-1561 Fax : 82-42-860-6699 Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)-------------------------------------------------------------------------------------------------ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2008-10-16 9:00 ` Andrea Righi
0 siblings, 0 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-16 9:00 UTC (permalink / raw)
To: Dong-Jae Kang
Cc: randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA, Paul Menage,
Carl Henrik Lunde, Divyesh Shah,
eric.rannaud-Re5JQEeQqe8AvxtiuMwx3w, Balbir Singh,
fernando-gVGce1chcLdL9jVzuh4AOg,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, agk-9JcytcrH/bA+uJoB2kUjGw,
subrata-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
axboe-tSWWG44O7X1aa/9Udqfwiw, Marco Innocenti,
containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8,
matt-cT2on/YLNlBWk0Htik3J/w, roberto-5KDOxZqKugI,
ngupta-hpIqsD4AKlfQT0dZR+AlfA
Dong-Jae Kang wrote:
>>> I have several question about io-throttle
>>> below is my test reusult of io-throttle(v11) with xdd 6.5
>>> But, I think I have something wrong, as showed in result
>>> In direct IO mode, Only read operation was controlled by io-throttle
>>> Can you check my test procedure and result and comments to me about that
>> Your procedure is correct. Anyway, you found a known bug in io-throttle
>> v11. If you want to properly use it you need to mount the memory
>> controller together with blockio, since currently blockio depends on it
>> to retrieve the owner of a page during writes in submit_bio().
>>
>> As reported in:
>>
>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>>
>> this is no more than a hack and in perspective a more generic framework
>> able to provide this functionality should be used (i.e. bio-cgroup).
>
> But, In my opinion,
> it is some strange that the bandwidth of write operation in direct IO
> mode was not controlled by io-throttle
> I think [PATCH -mm 4/6] is for control of buffered IO.
> Do I misunderstand it?
The approach is the same both for buffered and direct IO. With buffered
IO, writes are accounted in submit_bio() and throttling is performed in
balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always
performed in submit_bio() and throttling in submit_page_section(). That's
the same also for AIO, accounting in submit_bio() and throttling in
io_submit_one(), returning -EAGAIN in this case when the IO limits are
exceeded (instead of sleeping).
We never sleep in submit_bio() during writes and to account the IO activity
we always look at the owner (cgroup) of the first page in the struct bio,
also when the IO context is the cgroup of the current task (as in the direct
IO case). To correctly retrieve the owner of each page I've used the memory
controller functionality. This part should be improved to use a more generic
framework and remove the dependency of the memory controller. Or better, doesn't
impose to mount blockio and memory controller both to the same mountpoint.
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
2008-10-15 6:13 ` Dong-Jae Kang
[not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2008-10-16 9:00 ` Andrea Righi
1 sibling, 0 replies; 13+ messages in thread
From: Andrea Righi @ 2008-10-16 9:00 UTC (permalink / raw)
To: Dong-Jae Kang
Cc: Balbir Singh, Paul Menage, agk, akpm, axboe, Carl Henrik Lunde,
dave, Divyesh Shah, eric.rannaud, fernando, Hirokazu Takahashi,
Li Zefan, Marco Innocenti, matt, ngupta, randy.dunlap, roberto,
Ryo Tsuruta, Satoshi UCHIDA, subrata, yoshikawa.takuya,
containers, linux-kernel
Dong-Jae Kang wrote:
>>> I have several question about io-throttle
>>> below is my test reusult of io-throttle(v11) with xdd 6.5
>>> But, I think I have something wrong, as showed in result
>>> In direct IO mode, Only read operation was controlled by io-throttle
>>> Can you check my test procedure and result and comments to me about that
>> Your procedure is correct. Anyway, you found a known bug in io-throttle
>> v11. If you want to properly use it you need to mount the memory
>> controller together with blockio, since currently blockio depends on it
>> to retrieve the owner of a page during writes in submit_bio().
>>
>> As reported in:
>>
>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>>
>> this is no more than a hack and in perspective a more generic framework
>> able to provide this functionality should be used (i.e. bio-cgroup).
>
> But, In my opinion,
> it is some strange that the bandwidth of write operation in direct IO
> mode was not controlled by io-throttle
> I think [PATCH -mm 4/6] is for control of buffered IO.
> Do I misunderstand it?
The approach is the same both for buffered and direct IO. With buffered
IO, writes are accounted in submit_bio() and throttling is performed in
balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always
performed in submit_bio() and throttling in submit_page_section(). That's
the same also for AIO, accounting in submit_bio() and throttling in
io_submit_one(), returning -EAGAIN in this case when the IO limits are
exceeded (instead of sleeping).
We never sleep in submit_bio() during writes and to account the IO activity
we always look at the owner (cgroup) of the first page in the struct bio,
also when the IO context is the cgroup of the current task (as in the direct
IO case). To correctly retrieve the owner of each page I've used the memory
controller functionality. This part should be improved to use a more generic
framework and remove the dependency of the memory controller. Or better, doesn't
impose to mount blockio and memory controller both to the same mountpoint.
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-14 9:17 ` Dong-Jae Kang
@ 2008-11-03 7:01 ` Gui Jianfeng
[not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
1 sibling, 1 reply; 13+ messages in thread
From: Gui Jianfeng @ 2008-11-03 7:01 UTC (permalink / raw)
To: Andrea Righi; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Hi Andrea,
I'v tested io-throttle(v11), it seems io-throttle doesn't work well
when a process writes(delay-write) a small file(<50M may be). it can't
throttle the request process accurately.
I guess the following reason causes this problem.
The write-request process terminates too early to see all accounting
calculated in pdflush. So io-throttle can't throttle
enough time for that process. Am i right?
--
Regards
Gui Jianfeng
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
@ 2008-11-10 8:51 ` Andrea Righi
[not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
0 siblings, 1 reply; 13+ messages in thread
From: Andrea Righi @ 2008-11-10 8:51 UTC (permalink / raw)
To: Gui Jianfeng; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
On 2008-11-03 08:01, Gui Jianfeng wrote:
> Hi Andrea,
>
> I'v tested io-throttle(v11), it seems io-throttle doesn't work well
> when a process writes(delay-write) a small file(<50M may be). it can't
> throttle the request process accurately.
> I guess the following reason causes this problem.
> The write-request process terminates too early to see all accounting
> calculated in pdflush. So io-throttle can't throttle
> enough time for that process. Am i right?
Hi Gui,
sorry for my late first of all. You're right in part. The fact is that
with asynchronous IO (no O_DIRECT more exactly) the accounting is
performed when the blocks are submitted to the IO subsystem (submit_bio)
and the throttling is performed post-facto when the same process or the
other processes in the same cgroup write additional pages mapped to the
limited device (throttling is performed in
balance_dirty_pages_ratelimited_nr).
Anyway, if the process did a single small write in memory the operation
is not accounted until pdflush writes the data back to the block device.
When pdflush (or the process itself) performs the writeback the
operation is accounted and next reads/writes that come from the same
cgroup can now be throttled according to the defined limits.
So, the accounting is not lost, this information is only used later to
throttle the next operations that will cause IO requests. IO spikes in
this way are not prevented. With an early version of io-throttle I did
both throttling and accounting in balance_dirty_pages_ratelimited_nr
(when pages are written to memory), this can prevent IO spikes, but only
if dirty_ratio is set to small values.
Hope this clarify,
-Andrea
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)
[not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
@ 2008-11-10 9:52 ` Gui Jianfeng
0 siblings, 0 replies; 13+ messages in thread
From: Gui Jianfeng @ 2008-11-10 9:52 UTC (permalink / raw)
To: righi.andrea-Re5JQEeQqe8AvxtiuMwx3w
Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Andrea Righi wrote:
> On 2008-11-03 08:01, Gui Jianfeng wrote:
>> Hi Andrea,
>>
>> I'v tested io-throttle(v11), it seems io-throttle doesn't work well
>> when a process writes(delay-write) a small file(<50M may be). it can't
>> throttle the request process accurately.
>> I guess the following reason causes this problem.
>> The write-request process terminates too early to see all accounting
>> calculated in pdflush. So io-throttle can't throttle
>> enough time for that process. Am i right?
>
> Hi Gui,
>
> sorry for my late first of all. You're right in part. The fact is that
> with asynchronous IO (no O_DIRECT more exactly) the accounting is
> performed when the blocks are submitted to the IO subsystem (submit_bio)
> and the throttling is performed post-facto when the same process or the
> other processes in the same cgroup write additional pages mapped to the
> limited device (throttling is performed in
> balance_dirty_pages_ratelimited_nr).
>
> Anyway, if the process did a single small write in memory the operation
> is not accounted until pdflush writes the data back to the block device.
> When pdflush (or the process itself) performs the writeback the
> operation is accounted and next reads/writes that come from the same
> cgroup can now be throttled according to the defined limits.
>
> So, the accounting is not lost, this information is only used later to
> throttle the next operations that will cause IO requests. IO spikes in
> this way are not prevented. With an early version of io-throttle I did
> both throttling and accounting in balance_dirty_pages_ratelimited_nr
> (when pages are written to memory), this can prevent IO spikes, but only
> if dirty_ratio is set to small values.
Hi, Andrea,
Thanks for your detailed explanation, I'v got your point.
I think it makes more sense if throttling can be fair among other tasks. ;-)
>
> Hope this clarify,
> -Andrea
>
>
>
--
Regards
Gui Jianfeng
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2008-11-10 9:52 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-10-07 10:03 [PATCH -mm 0/6] cgroup: block device i/o controller (v11) Andrea Righi
[not found] ` <1223373818-13687-1-git-send-email-righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-14 9:17 ` Dong-Jae Kang
2008-11-03 7:01 ` Gui Jianfeng
[not found] ` <490EA1B4.9070600-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
2008-11-10 8:51 ` Andrea Righi
[not found] ` <4917F62A.20800-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-11-10 9:52 ` Gui Jianfeng
2008-10-14 9:17 ` Dong-Jae Kang
[not found] ` <2891419e0810140217l70f233bbr3b08760188458c35-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-14 9:56 ` Andrea Righi
2008-10-14 9:56 ` Andrea Righi
2008-10-15 6:13 ` Dong-Jae Kang
[not found] ` <2891419e0810142313p1aa16295ne2aa8ac3a87be491-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-10-16 9:00 ` Andrea Righi
2008-10-16 9:00 ` Andrea Righi
[not found] ` <48F46CB6.3010904-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2008-10-15 6:13 ` Dong-Jae Kang
-- strict thread matches above, loose matches on Subject: below --
2008-10-07 10:03 Andrea Righi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.