From: Gui Jianfeng <guijianfeng@cn.fujitsu.com>
To: Vivek Goyal <vgoyal@redhat.com>, Jens Axboe <axboe@kernel.dk>
Cc: linux kernel mailing list <linux-kernel@vger.kernel.org>,
Corrado Zoccolo <czoccolo@gmail.com>,
Chad Talbott <ctalbott@google.com>,
Nauman Rafique <nauman@google.com>,
Divyesh Shah <dpshah@google.com>,
jmoyer@redhat.com, Shaohua Li <shaohua.li@intel.com>,
guijianfeng@cn.fujitsu.com
Subject: [PATCH 0/6 v3] Introduce CFQ group hierarchical scheduling and "use_hierarchy" interface
Date: Mon, 27 Dec 2010 16:50:31 +0800 [thread overview]
Message-ID: <4D185357.7090304@cn.fujitsu.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 2643 bytes --]
Hi
Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
in the way that it puts all CFQ queues in a hidden group and schedules with other
CFQ group under their parent. The patchset is available here,
http://lkml.org/lkml/2010/8/30/30
Vivek think this approach isn't so instinct that we should treat CFQ queues
and groups at the same level. Here is the new approach for hierarchical
scheduling based on Vivek's suggestion. The most big change of CFQ is that
it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
queue scheduling just like CFQ group does. But I still give cfqq some jump
in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
queue and CFQ group use the same scheduling algorithm.
"use_hierarchy" interface is now added to switch between hierarchical mode
and flat mode. It works as memcg.
--
V2 -> V3 Changes:
- Starting from cfqd->grp_service_tree for both hierarchical mode and flat mode
- Avoid recursion when allocating cfqg and force dispatch logic
- Fix a bug when boosting vdisktime
- Adjusting total_weight accordingly when changing weight
- Change group slice calculation into a hierarchical way
- Keep flat mode rather than deleting it first then adding it later
- kfree the parent cfqg if there nobody references to it
- Simplify select_queue logic by using some wrap function
- Make "use_hierarchy" interface work as memcg
- Make use of time_before() for vdisktime compare
- Update Document
- Fix some code style problems
--
V1 -> V2 Changes:
- Raname "struct io_sched_entity" to "struct cfq_entity" and don't differentiate
queue_entity and group_entity, just use cfqe instead.
- Give newly added cfqq a small vdisktime jump accord to its ioprio.
- Make flat mode as default CFQ group scheduling mode.
- Introduce "use_hierarchy" interface.
- Update blkio cgroup documents
Benchmarks:
I made use of Vivek's iostest to perform some benchmarks on my box. I tested different workloads with
idling on and off. I didn't see any performance drop comparing to vanilla kernel. The attached files
are some performance numbers on vanilla Kernel and patched Kernel.
[PATCH 1/6 v3] cfq-iosched: Introduce cfq_entity for CFQ queue
[PATCH 2/6 v3] cfq-iosched: Introduce cfq_entity for CFQ group
[PATCH 3/6 v3] cfq-iosched: Introduce vdisktime and io weight for CFQ queue
[PATCH 4/6 v3] cfq-iosched: Extract some common code of service tree handling for CFQ queue and CFQ group
[PATCH 5/6 v3] cfq-iosched: CFQ group hierarchical scheduling and use_hierarchy interface
[PATCH 6/6 v3] blkio-cgroup: Document for blkio.use_hierarchy interface
[-- Attachment #2: Hier_v3.log --]
[-- Type: text/plain, Size: 5318 bytes --]
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
brr 1 1 748 748 749 747 2992
brr 1 2 256 566 865 896 2583
brr 1 4 154 318 560 1084 2116
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
brr 1 1 380 728 1074 1443 3625
brr 1 2 201 427 713 1252 2593
brr 1 4 146 327 551 1111 2135
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
bsr 1 1 7797 7799 7788 7805 31189
bsr 1 2 3026 6433 9643 9953 29055
bsr 1 4 2302 4582 7507 13557 27948
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
bsr 1 1 6288 10419 15332 20832 52871
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drr 1 1 757 757 757 755 3026
drr 1 2 233 547 829 919 2528
drr 1 4 143 306 574 1071 2094
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drr 1 1 371 729 1080 1429 3609
drr 1 2 192 409 793 1260 2654
drr 1 4 153 316 578 1066 2113
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drw 1 1 1529 1530 1531 1532 6122
drw 1 2 521 931 1185 1224 3861
drw 1 4 150 382 988 1588 3108
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drw 1 1 529 1243 1887 2778 6437
drw 1 2 296 717 1259 1817 4089
drw 1 4 164 403 937 1662 3166
[-- Attachment #3: Vanilla.log --]
[-- Type: text/plain, Size: 5424 bytes --]
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
brr 1 1 759 759 760 758 3036
brr 1 2 224 549 849 903 2525
brr 1 4 146 307 558 1090 2101
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
brr 1 1 375 734 1080 1426 3615
brr 1 2 168 418 767 1233 2586
brr 1 4 151 333 581 1068 2133
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
bsr 1 1 8000 8015 8027 8159 32201
bsr 1 2 3200 6302 9612 10018 29132
bsr 1 4 2153 4408 7857 13590 28008
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
bsr 1 1 6988 11575 17883 23389 59835
bsr 1 2 6808 11453 15979 19838 54078
bsr 1 4 5352 10986 14529 18189 49056
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drr 1 1 736 737 737 735 2945
drr 1 2 214 550 854 912 2530
drr 1 4 148 312 591 1069 2120
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drr 1 1 371 722 1059 1423 3575
drr 1 2 173 412 805 1258 2648
drr 1 4 155 305 555 1075 2090
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drw 1 1 1491 1491 1491 1491 5964
drw 1 2 556 869 1082 1201 3708
drw 1 4 160 335 830 1690 3015
Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------
job Set NR cgrp1 cgrp2 cgrp3 cgrp4 total
--- --- -- -----------------------------------
drw 1 1 545 1343 2016 2743 6647
drw 1 2 268 696 1361 1942 4267
drw 1 4 140 357 775 1880 3152
reply other threads:[~2010-12-27 8:50 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D185357.7090304@cn.fujitsu.com \
--to=guijianfeng@cn.fujitsu.com \
--cc=axboe@kernel.dk \
--cc=ctalbott@google.com \
--cc=czoccolo@gmail.com \
--cc=dpshah@google.com \
--cc=jmoyer@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nauman@google.com \
--cc=shaohua.li@intel.com \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox