Re: performance drop after using blkcg

[Vivek Goyal <vgoyal@redhat.com>]

On Mon, Dec 10, 2012 at 08:28:54PM +0800, Zhao Shuai wrote:
> Hi,
> 
> I plan to use blkcg(proportional BW) in my system. But I encounter
> great performance drop after enabling blkcg.
> The testing tool is fio(version 2.0.7) and both the BW and IOPS fields
> are recorded. Two instances of fio program are carried out simultaneously,
> each opearting on a separate disk file (say /data/testfile1,
> /data/testfile2).
> System environment:
> kernel: 3.7.0-rc5
> CFQ's slice_idle is disabled(slice_idle=0) while group_idle is
> enabled(group_idle=8).
> 
> FIO configuration(e.g. "read") for the first fio program(say FIO1):
> 
> [global]
> description=Emulation of Intel IOmeter File Server Access Pattern
> 
> [iometer]
> bssplit=4k/30:8k/40:16k/30
> rw=read
> direct=1
> time_based
> runtime=180s
> ioengine=sync
> filename=/data/testfile1
> numjobs=32
> group_reporting
> 
> 
> result before using blkcg: (the value of BW is KB/s)
> 
>            FIO1 BW/IOPS    FIO2 BW/IOPS
> ---------------------------------------
> read       26799/2911      25861/2810
> write      138618/15071    138578/15069
> rw         72159/7838(r)   71851/7811(r)
>            72171/7840(w)   71799/7805(w)
> randread   4982/543        5370/585
> randwrite  5192/566        6010/654
> randrw     2369/258(r)     3027/330(r)
>            2369/258(w)     3016/328(w)
> 
> result after using blkcg(create two blkio cgroups with
> default blkio.weight(500) and put FIO1 and FIO2 into these
> cgroups respectively)

These results are with slice_idle=0?

> 
>            FIO1 BW/IOPS    FIO2 BW/IOPS
> ---------------------------------------
> read       36651/3985      36470/3943
> write      75738/8229      75641/8221
> rw         49169/5342(r)   49168/5346(r)
>            49200/5348(w)   49140/5341(w)
> randread   4876/532        4905/534
> randwrite  5535/603        5497/599
> randrw     2521/274(r)     2527/275(r)
>            2510/273(w)     2532/274(w)
> 
> Comparing with those results, we found greate performance drop
> (30%-40%) in some test cases(especially for the "write", "rw" case).
> Is it normal to see write/rw bandwidth decrease by 40% after using
> blkio-cgroup? If not, any way to improve or tune the performace?

What's the storage you are using. Looking at the speed of IO I would
guess it is not one of those rotational disks.

blkcg does cause the drop in performance (due to idling at group level).
Faster the storage or more the number of cgroups, drop becomes even
more visible.

Only optimization I could think of was disabling slice_idle and you
have already done that.

There might be some opporutnities to cut down the group idling in
some cases and lose on fairness but we will have to identify those
and modify code.

In general, do not use blkcg on faster storage. In current form it
is at best suitable for single rotational SATA/SAS disk. I have not
been able to figure out how to provide fairness without group idling.

Thanks
Vivek