[PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[Andrea Righi]

The objective of the i/o controller is to improve i/o performance
predictability of different cgroups sharing the same block devices.

Respect to other priority/weight-based solutions the approach used by this
controller is to explicitly choke applications' requests that directly (or
indirectly) generate i/o activity in the system.

The direct bandwidth and/or iops limiting method has the advantage of improving
the performance predictability at the cost of reducing, in general, the overall
performance of the system (in terms of throughput).

Detailed informations about design, its goal and usage are described in the
documentation.

Patchset against 2.6.27-rc5-mm1:

  [PATCH 0/6] cgroup: block device i/o controller (v11)
  [PATCH 1/6] i/o controller documentation
  [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
  [PATCH 3/6] i/o controller infrastructure
  [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
  [PATCH 5/6] i/o controller instrumentation: accounting and throttling
  [PATCH 6/6] export per-task i/o throttling statistics to userspace

The all-in-one patch (and previous versions) can be found at:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There are no significant changes respect to v10, I've only implemented/fixed
some suggestions I received.

Changelog: (v10 -> v11)

* report per block device i/o statistics (total bytes read/written and iops)
  in blockio.stat for i/o limited cgroups
* distinct bandwidth and iops statistics: both in blockio.throttlecnt and
  /proc/PID/io-throttle-stat (suggested by David Radford)
* merge res_counter_ratelimit functionality into res_counter, to avoid code
  duplication (suggested by Paul Manage)
* use kernel-doc style for documenting struct res_counter attributes
  (suggested by Randy Dunalp)
* udpated documentation

Thanks to all for the feedback!
-Andrea

Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[Dong-Jae Kang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 9545 bytes --]

Hi, Andrea
thank you for your contribution to communitythese days, I am testing several IO controllers in container ML,dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and yourio-throttle(v11)
I have several question about io-throttlebelow is my test reusult of io-throttle(v11) with xdd 6.5But, I think I have something wrong, as showed in resultIn direct IO mode, Only read operation was controlled by io-throttleCan you check my  test procedure and result and comments to me about that
additionally, your testing shell script(run_io_throttle_test.sh) forio-throttle was not updated for new io-throttleso, it could be operated after I fixed it
------------------------------------------------------------------------------------ Test System Information
Computer Name, localhost.localdomain, User Name, rootOS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008Machine hardware type, i686Number of processors on this system, 1Page size in bytes, 4096Number of physical pages, 515885Megabytes of physical memory, 2015Target[0] Q[0], /dev/sdbPer-pass time limit in seconds, 30Blocksize in bytes, 512Request size, 128, blocks, 65536, bytesNumber of Requests, 16384Number of MegaBytes, 512 or 1024Direct I/O, disabled or enableSeek pattern, sequentialQueue Depth, 1
- Test Procedure
	mkdir /dev/blockioctl	mount -t cgroup -o blockio cgroup /dev/blockioctl	mkdir /dev/blockioctl/cgroup-1	mkdir /dev/blockioctl/cgroup-2	mkdir /dev/blockioctl/cgroup-3	echo /dev/sdb:$((1024*1024)):0:0 >/dev/blockioctl/cgroup-1/blockio.bandwidth-max	echo /dev/sdb:$((2*1024*1024)):0:0 >/dev/blockioctl/cgroup-2/blockio.bandwidth-max	echo /dev/sdb:$((3*1024*1024)):0:0 >/dev/blockioctl/cgroup-3/blockio.bandwidth-max	in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks	in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks	in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks	in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb-blocksize 512 -reqsize 128 -mbytes 1024( or 512 )  -timelimit 30-verbose –dio(enable or disable)
- setting status information
[root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max8 16 1048576 0 0 0 13016[root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max8 16 2097152 0 0 0 11763[root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max8 16 3145728 0 0 0 11133
- Test Resultxdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio -verbose
cgroup-1
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      31522816      481    30.005     1.051      16.03    0.0624  0.00   read       655360  1      31522816      481    30.005     1.051      16.03    0.0624  0.00   read       655361  1      31522816      481    30.005     1.051      16.03    0.0624  0.00   read       65536
cgroup-2
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      62980096      961    30.001     2.099      32.03    0.0312  0.00   read       655360  1      62980096      961    30.001     2.099      32.03    0.0312  0.00   read       655361  1      62980096      961    30.001     2.099      32.03    0.0312  0.00   read       65536
cgroup-3
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      94437376     1441    30.003     3.148      48.03    0.0208  0.00   read       655360  1      94437376     1441    30.003     3.148      48.03    0.0208  0.00   read       655361  1      94437376     1441    30.003     3.148      48.03    0.0208  0.00   read       65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 512 -timelimit 30 -dio –verbose
cgroup-1
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     640221184     9769    30.097    21.272     324.58    0.0031  0.00   write       655360  1     640221184     9769    30.097    21.272     324.58    0.0031  0.00   write       655361  1     640221184     9769    30.097    21.272     324.58    0.0031  0.00   write       65536
cgroup-2
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     633798656     9671    30.001    21.126     322.36    0.0031  0.00   write       655360  1     633798656     9671    30.001    21.126     322.36    0.0031  0.00   write       655361  1     633798656     9671    30.001    21.126     322.36    0.0031  0.00   write       65536
cgroup-3
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     630652928     9623    30.001    21.021     320.76    0.0031  0.00   write       655360  1     630652928     9623    30.001    21.021     320.76    0.0031  0.00   write       655361  1     630652928     9623    30.001    21.021     320.76    0.0031  0.00   write       65536
xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024  -timelimit 30  -verbose
cgroup-1
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      70123520     1070    30.150     2.326      35.49    0.0282  0.00   read       655360  1      70123520     1070    30.150     2.326      35.49    0.0282  0.00   read       655361  1      70123520     1070    30.150     2.326      35.49    0.0282  0.00   read       65536
cgroup-2
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      70844416     1081    30.063     2.357      35.96    0.0278  0.00   read       655360  1      70844416     1081    30.063     2.357      35.96    0.0278  0.00   read       655361  1      70844416     1081    30.063     2.357      35.96    0.0278  0.00   read       65536
cgroup-3
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1      72155136     1101    30.204     2.389      36.45    0.0274  0.00   read       655360  1      72155136     1101    30.204     2.389      36.45    0.0274  0.00   read       655361  1      72155136     1101    30.204     2.389      36.45    0.0274  0.00   read       65536
xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128-mbytes 1024  -timelimit 30  -verbose
cgroup-1
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     818610176    12491    30.031    27.258     415.93    0.0024  0.00   write       655360  1     818610176    12491    30.031    27.258     415.93    0.0024  0.00   write       655361  1     818610176    12491    30.031    27.258     415.93    0.0024  0.00   write       65536
cgroup-2
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     848494592    12947    30.066    28.221     430.62    0.0023  0.00   write       655360  1     848494592    12947    30.066    28.221     430.62    0.0023  0.00   write       655361  1     848494592    12947    30.066    28.221     430.62    0.0023  0.00   write       65536
cgroup-3
T  Q       Bytes      Ops    Time      Rate      IOPS   Latency%CPU  OP_Type    ReqSize0  1     786563072    12002    30.078    26.151     399.03    0.0025  0.00   write       655360  1     786563072    12002    30.078    26.151     399.03    0.0025  0.00   write       655361  1     786563072    12002    30.078    26.151     399.03    0.0025  0.00   write       65536
Best Regards,Dong-Jae Kang

2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>> The objective of the i/o controller is to improve i/o performance> predictability of different cgroups sharing the same block devices.>> Respect to other priority/weight-based solutions the approach used by this> controller is to explicitly choke applications' requests that directly (or> indirectly) generate i/o activity in the system.>> The direct bandwidth and/or iops limiting method has the advantage of improving> the performance predictability at the cost of reducing, in general, the overall> performance of the system (in terms of throughput).>> Detailed informations about design, its goal and usage are described in the> documentation.>> Patchset against 2.6.27-rc5-mm1:>>  [PATCH 0/6] cgroup: block device i/o controller (v11)>  [PATCH 1/6] i/o controller documentation>  [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>  [PATCH 3/6] i/o controller infrastructure>  [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>  [PATCH 5/6] i/o controller instrumentation: accounting and throttling>  [PATCH 6/6] export per-task i/o throttling statistics to userspace>> The all-in-one patch (and previous versions) can be found at:> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>> There are no significant changes respect to v10, I've only implemented/fixed> some suggestions I received.>> Changelog: (v10 -> v11)>> * report per block device i/o statistics (total bytes read/written and iops)>  in blockio.stat for i/o limited cgroups> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>  /proc/PID/io-throttle-stat (suggested by David Radford)> * merge res_counter_ratelimit functionality into res_counter, to avoid code>  duplication (suggested by Paul Manage)> * use kernel-doc style for documenting struct res_counter attributes>  (suggested by Randy Dunalp)> * udpated documentation>> Thanks to all for the feedback!> -Andrea>ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[Andrea Righi]

Dong-Jae Kang wrote:
> Hi, Andrea
> 
> thank you for your contribution to community
> these days, I am testing several IO controllers in container ML,
> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your
> io-throttle(v11)

Thanks! this is surely a valuable task.

> 
> I have several question about io-throttle
> below is my test reusult of io-throttle(v11) with xdd 6.5
> But, I think I have something wrong, as showed in result
> In direct IO mode, Only read operation was controlled by io-throttle
> Can you check my  test procedure and result and comments to me about that

Your procedure is correct. Anyway, you found a known bug in io-throttle
v11. If you want to properly use it you need to mount the memory
controller together with blockio, since currently blockio depends on it
to retrieve the owner of a page during writes in submit_bio().

As reported in:

[PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity

this is no more than a hack and in perspective a more generic framework
able to provide this functionality should be used (i.e. bio-cgroup).

I'll fix this issue in the next version of io-throttle (probably I'll
try to rewrite io-throttle on top of bio-cgroup), but for now the
workaround is to mount the cgroupfs using -o blockio,memory (at least).

> 
> additionally, your testing shell script(run_io_throttle_test.sh) for
> io-throttle was not updated for new io-throttle
> so, it could be operated after I fixed it

The testing of iops limiting is not yet implemented and I don't have a
very good testcase for this, but I can share with you a small script that
I'm using to check if iops limiting is working or not, if you're interested.

Thanks,
-Andrea

> 
> -----------------------------------------------------------------------------------
> - Test System Information
> 
> Computer Name, localhost.localdomain, User Name, root
> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008
> Machine hardware type, i686
> Number of processors on this system, 1
> Page size in bytes, 4096
> Number of physical pages, 515885
> Megabytes of physical memory, 2015
> Target[0] Q[0], /dev/sdb
> Per-pass time limit in seconds, 30
> Blocksize in bytes, 512
> Request size, 128, blocks, 65536, bytes
> Number of Requests, 16384
> Number of MegaBytes, 512 or 1024
> Direct I/O, disabled or enable
> Seek pattern, sequential
> Queue Depth, 1
> 
> - Test Procedure
> 
> 	mkdir /dev/blockioctl
> 	mount -t cgroup -o blockio cgroup /dev/blockioctl
> 	mkdir /dev/blockioctl/cgroup-1
> 	mkdir /dev/blockioctl/cgroup-2
> 	mkdir /dev/blockioctl/cgroup-3
> 	echo /dev/sdb:$((1024*1024)):0:0 >
> /dev/blockioctl/cgroup-1/blockio.bandwidth-max
> 	echo /dev/sdb:$((2*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-2/blockio.bandwidth-max
> 	echo /dev/sdb:$((3*1024*1024)):0:0 >
> /dev/blockioctl/cgroup-3/blockio.bandwidth-max
> 	in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks
> 	in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks
> 	in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks
> 	in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb
> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 )  -timelimit 30
> -verbose –dio(enable or disable)
> 
> - setting status information
> 
> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max
> 8 16 1048576 0 0 0 13016
> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max
> 8 16 2097152 0 0 0 11763
> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max
> 8 16 3145728 0 0 0 11133
> 
> - Test Result
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio -verbose
> 
> cgroup-1
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      31522816      481    30.005     1.051      16.03    0.0624
>   0.00   read       65536
> 0  1      31522816      481    30.005     1.051      16.03    0.0624
>   0.00   read       65536
> 1  1      31522816      481    30.005     1.051      16.03    0.0624
>   0.00   read       65536
> 
> cgroup-2
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      62980096      961    30.001     2.099      32.03    0.0312
>   0.00   read       65536
> 0  1      62980096      961    30.001     2.099      32.03    0.0312
>   0.00   read       65536
> 1  1      62980096      961    30.001     2.099      32.03    0.0312
>   0.00   read       65536
> 
> cgroup-3
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      94437376     1441    30.003     3.148      48.03    0.0208
>   0.00   read       65536
> 0  1      94437376     1441    30.003     3.148      48.03    0.0208
>   0.00   read       65536
> 1  1      94437376     1441    30.003     3.148      48.03    0.0208
>   0.00   read       65536
> 
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 512 -timelimit 30 -dio –verbose
> 
> cgroup-1
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     640221184     9769    30.097    21.272     324.58    0.0031
>   0.00   write       65536
> 0  1     640221184     9769    30.097    21.272     324.58    0.0031
>   0.00   write       65536
> 1  1     640221184     9769    30.097    21.272     324.58    0.0031
>   0.00   write       65536
> 
> cgroup-2
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     633798656     9671    30.001    21.126     322.36    0.0031
>   0.00   write       65536
> 0  1     633798656     9671    30.001    21.126     322.36    0.0031
>   0.00   write       65536
> 1  1     633798656     9671    30.001    21.126     322.36    0.0031
>   0.00   write       65536
> 
> cgroup-3
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     630652928     9623    30.001    21.021     320.76    0.0031
>   0.00   write       65536
> 0  1     630652928     9623    30.001    21.021     320.76    0.0031
>   0.00   write       65536
> 1  1     630652928     9623    30.001    21.021     320.76    0.0031
>   0.00   write       65536
> 
> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024  -timelimit 30  -verbose
> 
> cgroup-1
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      70123520     1070    30.150     2.326      35.49    0.0282
>   0.00   read       65536
> 0  1      70123520     1070    30.150     2.326      35.49    0.0282
>   0.00   read       65536
> 1  1      70123520     1070    30.150     2.326      35.49    0.0282
>   0.00   read       65536
> 
> cgroup-2
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      70844416     1081    30.063     2.357      35.96    0.0278
>   0.00   read       65536
> 0  1      70844416     1081    30.063     2.357      35.96    0.0278
>   0.00   read       65536
> 1  1      70844416     1081    30.063     2.357      35.96    0.0278
>   0.00   read       65536
> 
> cgroup-3
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1      72155136     1101    30.204     2.389      36.45    0.0274
>   0.00   read       65536
> 0  1      72155136     1101    30.204     2.389      36.45    0.0274
>   0.00   read       65536
> 1  1      72155136     1101    30.204     2.389      36.45    0.0274
>   0.00   read       65536
> 
> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128
> -mbytes 1024  -timelimit 30  -verbose
> 
> cgroup-1
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     818610176    12491    30.031    27.258     415.93    0.0024
>   0.00   write       65536
> 0  1     818610176    12491    30.031    27.258     415.93    0.0024
>   0.00   write       65536
> 1  1     818610176    12491    30.031    27.258     415.93    0.0024
>   0.00   write       65536
> 
> cgroup-2
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     848494592    12947    30.066    28.221     430.62    0.0023
>   0.00   write       65536
> 0  1     848494592    12947    30.066    28.221     430.62    0.0023
>   0.00   write       65536
> 1  1     848494592    12947    30.066    28.221     430.62    0.0023
>   0.00   write       65536
> 
> cgroup-3
> 
> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency
> %CPU  OP_Type    ReqSize
> 0  1     786563072    12002    30.078    26.151     399.03    0.0025
>   0.00   write       65536
> 0  1     786563072    12002    30.078    26.151     399.03    0.0025
>   0.00   write       65536
> 1  1     786563072    12002    30.078    26.151     399.03    0.0025
>   0.00   write       65536
> 
> Best Regards,
> Dong-Jae Kang
> 
> 
> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:
>> The objective of the i/o controller is to improve i/o performance
>> predictability of different cgroups sharing the same block devices.
>>
>> Respect to other priority/weight-based solutions the approach used by this
>> controller is to explicitly choke applications' requests that directly (or
>> indirectly) generate i/o activity in the system.
>>
>> The direct bandwidth and/or iops limiting method has the advantage of improving
>> the performance predictability at the cost of reducing, in general, the overall
>> performance of the system (in terms of throughput).
>>
>> Detailed informations about design, its goal and usage are described in the
>> documentation.
>>
>> Patchset against 2.6.27-rc5-mm1:
>>
>>  [PATCH 0/6] cgroup: block device i/o controller (v11)
>>  [PATCH 1/6] i/o controller documentation
>>  [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter
>>  [PATCH 3/6] i/o controller infrastructure
>>  [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>>  [PATCH 5/6] i/o controller instrumentation: accounting and throttling
>>  [PATCH 6/6] export per-task i/o throttling statistics to userspace
>>
>> The all-in-one patch (and previous versions) can be found at:
>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/
>>
>> There are no significant changes respect to v10, I've only implemented/fixed
>> some suggestions I received.
>>
>> Changelog: (v10 -> v11)
>>
>> * report per block device i/o statistics (total bytes read/written and iops)
>>  in blockio.stat for i/o limited cgroups
>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and
>>  /proc/PID/io-throttle-stat (suggested by David Radford)
>> * merge res_counter_ratelimit functionality into res_counter, to avoid code
>>  duplication (suggested by Paul Manage)
>> * use kernel-doc style for documenting struct res_counter attributes
>>  (suggested by Randy Dunalp)
>> * udpated documentation
>>
>> Thanks to all for the feedback!
>> -Andrea

Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[Dong-Jae Kang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 12215 bytes --]

Hi, AndreaThank you for your comments
> Dong-Jae Kang wrote:>> Hi, Andrea>>>> thank you for your contribution to community>> these days, I am testing several IO controllers in container ML,>> dm-ioband by Ryo tsuruta(v1.7.0), 2-Layer CFQ by Satoshi and your>> io-throttle(v11)>> Thanks! this is surely a valuable task.>>>>> I have several question about io-throttle>> below is my test reusult of io-throttle(v11) with xdd 6.5>> But, I think I have something wrong, as showed in result>> In direct IO mode, Only read operation was controlled by io-throttle>> Can you check my  test procedure and result and comments to me about that>> Your procedure is correct. Anyway, you found a known bug in io-throttle> v11. If you want to properly use it you need to mount the memory> controller together with blockio, since currently blockio depends on it> to retrieve the owner of a page during writes in submit_bio().>> As reported in:>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>> this is no more than a hack and in perspective a more generic framework> able to provide this functionality should be used (i.e. bio-cgroup).
But, In my opinion,it is some strange that the bandwidth of write operation in direct IOmode was not controlled by io-throttleI think [PATCH -mm 4/6] is for control of buffered IO.Do I misunderstand it?
> I'll fix this issue in the next version of io-throttle (probably I'll> try to rewrite io-throttle on top of bio-cgroup), but for now the> workaround is to mount the cgroupfs using -o blockio,memory (at least).>
Oh, it sounds good.I look foward to your next io-throttle
>>>> additionally, your testing shell script(run_io_throttle_test.sh) for>> io-throttle was not updated for new io-throttle>> so, it could be operated after I fixed it>> The testing of iops limiting is not yet implemented and I don't have a> very good testcase for this, but I can share with you a small script that> I'm using to check if iops limiting is working or not, if you're interested.
thank you, Andrea.I want to check iops limiting of io-throttle is working well.Have a nice day...>>>>> ----------------------------------------------------------------------------------->> - Test System Information>>>> Computer Name, localhost.localdomain, User Name, root>> OS release and version, Linux 2.6.27-rc5-mm1 #1 SMP Thu Oct 9 18:27:09 KST 2008>> Machine hardware type, i686>> Number of processors on this system, 1>> Page size in bytes, 4096>> Number of physical pages, 515885>> Megabytes of physical memory, 2015>> Target[0] Q[0], /dev/sdb>> Per-pass time limit in seconds, 30>> Blocksize in bytes, 512>> Request size, 128, blocks, 65536, bytes>> Number of Requests, 16384>> Number of MegaBytes, 512 or 1024>> Direct I/O, disabled or enable>> Seek pattern, sequential>> Queue Depth, 1>>>> - Test Procedure>>>>      mkdir /dev/blockioctl>>      mount -t cgroup -o blockio cgroup /dev/blockioctl>>      mkdir /dev/blockioctl/cgroup-1>>      mkdir /dev/blockioctl/cgroup-2>>      mkdir /dev/blockioctl/cgroup-3>>      echo /dev/sdb:$((1024*1024)):0:0 >>> /dev/blockioctl/cgroup-1/blockio.bandwidth-max>>      echo /dev/sdb:$((2*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-2/blockio.bandwidth-max>>      echo /dev/sdb:$((3*1024*1024)):0:0 >>> /dev/blockioctl/cgroup-3/blockio.bandwidth-max>>      in terminal 1, echo $$ > /dev/blockioctl/cgroup-1/tasks>>      in terminal 2, echo $$ > /dev/blockioctl/cgroup-2/tasks>>      in terminal 3, echo $$ > /dev/blockioctl/cgroup-3/tasks>>      in each terminal, xdd.linux -op write( or read ) -targets 1 /dev/sdb>> -blocksize 512 -reqsize 128 -mbytes 1024( or 512 )  -timelimit 30>> -verbose –dio(enable or disable)>>>> - setting status information>>>> [root@localhost blockioctl]# cat ./cgroup-1/blockio.bandwidth-max>> 8 16 1048576 0 0 0 13016>> [root@localhost blockioctl]# cat ./cgroup-2/blockio.bandwidth-max>> 8 16 2097152 0 0 0 11763>> [root@localhost blockioctl]# cat ./cgroup-3/blockio.bandwidth-max>> 8 16 3145728 0 0 0 11133>>>> - Test Result>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>> 0  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>> 1  1      31522816      481    30.005     1.051      16.03    0.0624>>   0.00   read       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>> 0  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>> 1  1      62980096      961    30.001     2.099      32.03    0.0312>>   0.00   read       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>> 0  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>> 1  1      94437376     1441    30.003     3.148      48.03    0.0208>>   0.00   read       65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 512 -timelimit 30 -dio –verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>> 0  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>> 1  1     640221184     9769    30.097    21.272     324.58    0.0031>>   0.00   write       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>> 0  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>> 1  1     633798656     9671    30.001    21.126     322.36    0.0031>>   0.00   write       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>> 0  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>> 1  1     630652928     9623    30.001    21.021     320.76    0.0031>>   0.00   write       65536>>>> xdd.linux -op read -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024  -timelimit 30  -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>> 0  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>> 1  1      70123520     1070    30.150     2.326      35.49    0.0282>>   0.00   read       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>> 0  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>> 1  1      70844416     1081    30.063     2.357      35.96    0.0278>>   0.00   read       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>> 0  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>> 1  1      72155136     1101    30.204     2.389      36.45    0.0274>>   0.00   read       65536>>>> xdd.linux -op write -targets 1 /dev/sdb -blocksize 512 -reqsize 128>> -mbytes 1024  -timelimit 30  -verbose>>>> cgroup-1>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>> 0  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>> 1  1     818610176    12491    30.031    27.258     415.93    0.0024>>   0.00   write       65536>>>> cgroup-2>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>> 0  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>> 1  1     848494592    12947    30.066    28.221     430.62    0.0023>>   0.00   write       65536>>>> cgroup-3>>>> T  Q       Bytes      Ops    Time      Rate      IOPS   Latency>> %CPU  OP_Type    ReqSize>> 0  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>> 0  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>> 1  1     786563072    12002    30.078    26.151     399.03    0.0025>>   0.00   write       65536>>>> Best Regards,>> Dong-Jae Kang>>>>>> 2008/10/7 Andrea Righi <righi.andrea@gmail.com>:>>> The objective of the i/o controller is to improve i/o performance>>> predictability of different cgroups sharing the same block devices.>>>>>> Respect to other priority/weight-based solutions the approach used by this>>> controller is to explicitly choke applications' requests that directly (or>>> indirectly) generate i/o activity in the system.>>>>>> The direct bandwidth and/or iops limiting method has the advantage of improving>>> the performance predictability at the cost of reducing, in general, the overall>>> performance of the system (in terms of throughput).>>>>>> Detailed informations about design, its goal and usage are described in the>>> documentation.>>>>>> Patchset against 2.6.27-rc5-mm1:>>>>>>  [PATCH 0/6] cgroup: block device i/o controller (v11)>>>  [PATCH 1/6] i/o controller documentation>>>  [PATCH 2/6] introduce ratelimiting attributes and functionality to res_counter>>>  [PATCH 3/6] i/o controller infrastructure>>>  [PATCH 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity>>>  [PATCH 5/6] i/o controller instrumentation: accounting and throttling>>>  [PATCH 6/6] export per-task i/o throttling statistics to userspace>>>>>> The all-in-one patch (and previous versions) can be found at:>>> http://download.systemimager.org/~arighi/linux/patches/io-throttle/>>>>>> There are no significant changes respect to v10, I've only implemented/fixed>>> some suggestions I received.>>>>>> Changelog: (v10 -> v11)>>>>>> * report per block device i/o statistics (total bytes read/written and iops)>>>  in blockio.stat for i/o limited cgroups>>> * distinct bandwidth and iops statistics: both in blockio.throttlecnt and>>>  /proc/PID/io-throttle-stat (suggested by David Radford)>>> * merge res_counter_ratelimit functionality into res_counter, to avoid code>>>  duplication (suggested by Paul Manage)>>> * use kernel-doc style for documenting struct res_counter attributes>>>  (suggested by Randy Dunalp)>>> * udpated documentation>>>>>> Thanks to all for the feedback!>>> -Andrea>


-- -------------------------------------------------------------------------------------------------   DONG-JAE, KANG   Senior Member of Engineering Staff   Internet Platform Research Dept, S/W Content Research Lab   Electronics and Telecommunications Research Institute(ETRI)   138 Gajeongno, Yuseong-gu, Daejeon, 305-700 KOREA   Phone : 82-42-860-1561 Fax : 82-42-860-6699   Mobile : 82-10-9919-2353 E-mail : djkang@etri.re.kr (MSN)-------------------------------------------------------------------------------------------------ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH -mm 0/6] cgroup: block device i/o controller (v11)

[Andrea Righi]

Dong-Jae Kang wrote:
>>> I have several question about io-throttle
>>> below is my test reusult of io-throttle(v11) with xdd 6.5
>>> But, I think I have something wrong, as showed in result
>>> In direct IO mode, Only read operation was controlled by io-throttle
>>> Can you check my  test procedure and result and comments to me about that
>> Your procedure is correct. Anyway, you found a known bug in io-throttle
>> v11. If you want to properly use it you need to mount the memory
>> controller together with blockio, since currently blockio depends on it
>> to retrieve the owner of a page during writes in submit_bio().
>>
>> As reported in:
>>
>> [PATCH -mm 4/6] memcg: interface to charge the right cgroup of asynchronous i/o activity
>>
>> this is no more than a hack and in perspective a more generic framework
>> able to provide this functionality should be used (i.e. bio-cgroup).
> 
> But, In my opinion,
> it is some strange that the bandwidth of write operation in direct IO
> mode was not controlled by io-throttle
> I think [PATCH -mm 4/6] is for control of buffered IO.
> Do I misunderstand it?

The approach is the same both for buffered and direct IO. With buffered
IO, writes are accounted in submit_bio() and throttling is performed in
balance_dirty_pages_ratelimited_nr(). With direct IO, accounting is always
performed in submit_bio() and throttling in submit_page_section(). That's
the same also for AIO, accounting in submit_bio() and throttling in
io_submit_one(), returning -EAGAIN in this case when the IO limits are
exceeded (instead of sleeping).

We never sleep in submit_bio() during writes and to account the IO activity
we always look at the owner (cgroup) of the first page in the struct bio,
also when the IO context is the cgroup of the current task (as in the direct
IO case). To correctly retrieve the owner of each page I've used the memory
controller functionality. This part should be improved to use a more generic
framework and remove the dependency of the memory controller. Or better, doesn't
impose to mount blockio and memory controller both to the same mountpoint.

-Andrea