rate-capped fio jobs in a CFQ group degrade performance of fio jobs in another CFQ group with the same weight

[Hou Tao <houtao1@huawei.com>]

Hi all,

During our test of the CFQ group schedule, we found a performance related problem.
Rate-capped fio jobs in a CFQ group will degrade the performance of fio jobs in
another CFQ group. Both of the CFQ groups have the same blkio.weight.

We launch two fios in difference terminals. The following is the content of
these two job files:

# cat a.job
[global]
bs=4k
ioengine=psync
iodepth=1
direct=1
rw=randwrite
time_based
runtime=30
cgroup_nodelete=1
group_reporting=1

[test1]
filename=test1.dat
size=2G
cgroup_weight=500
cgroup=testA
thread=1
numjobs=2

[test3]
stonewall
filename=test3.dat
size=2G
rate=4m
cgroup_weight=500
cgroup=testA
thread=1
numjobs=2


# cat b.job
[global]
bs=4k
ioengine=psync
iodepth=1
direct=1
rw=randwrite
time_based
runtime=60
cgroup_nodelete=1
group_reporting=1

[test2]
filename=test2.dat
size=2G
cgroup_weight=500
cgroup=testB
thread=1
numjobs=2

In the first 30 seconds, the iops of both "test1" and "test3" is ~5000 and
the rate is about 20MBps.

In the last 30 seconds, we start "test3" and the rate of "test3" is limited to
8MBps (4MBps * 2 jobs). The cap works fine, but the iops of "test2" also
degraded from ~5000 to ~2040 or lower. This may seems reasonable for CFQ group
schedule because it needs to dispatch disk time equally to these two CFQ group,
but it doesn't fully utilize the disk throughput.

It can be reproduced on 4.4 and 4.10.0-rc7, the version of fio is 2.2.4.
The configuration options of CFQ are untouched.

I am not sure whether this situation is a bug or not. Are there some
configuration options we can used to alleviate the performance degradation
problem ? And how CFQ group schedule handles the trace-off between fairness
and performance ?

Any suggestion will be appreciated.

Regards,

Tao