Re: 2.6.12-rc2-mm3

Andrew Morton
2005-04-12T05:25:36Z

Andrew Morton <akpm@osdl.org> wrote:
>
> - The anticipatory I/O scheduler has always been fairly useless with SCSI
> disks which perform tagged command queueing. There's a patch here from Jens
> which is designed to fix that up by constraining the number of requests
> which we'll leave pending in the device.
>
> The depth currently defaults to 1. Tunable in
> /sys/block/hdX/queue/iosched/queue_depth
>
> This patch hasn't been performance tested at all yet. If you think it is
> misbehaving (the usual symptom is processes stuck in D state) then please
> report it, then boot with `elevator=cfq' or `elevator=deadline' to work
> around it.

So it turns out that patch was broken. I've fixed it locally and the
results are good, but odd.

The machine is a 4GB x86_64 with aic79xx controllers and MAXTOR
ATLAS10K4_73WLS disks. ext2 filesystem.

The workload is continuous pagecache writeback versus
read-lots-of-little-files:

while true
do
dd if=/dev/zero of=/mnt/sdb2/x bs=40M count=100 conv=notrunc
done

versus

find /mnt/sdb2/linux-2.4.25 -type f | xargs cat > /dev/null

we measure how long the find+cat takes.

2.6.12-rc2, as, tcq depth=2: 7.241 seconds
2.6.12-rc2, as, tcq depth=64: 12.172 seconds
2.6.12-rc2+patch,as, tcq depth=64: 7.199 seconds
2.6.12-rc2, cfq2, tcq depth=64: much more than 5 minutes
2.6.12-rc2, cfq3, tcq depth=64: much more than 5 minutes

So

- The effects of tcq on AS are much less disastrous than I thought they
were. Do I have the wrong workload? Memory fails me. Or did we fix the
anticipatory scheduler?

- as-limit-queue-depth.patch fixes things right up anyway. Seems to be
doing the right thing.

- CFQ is seriously, seriously read-starved on this workload.

CFQ2:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 3 1116 25504 4868 3854008 4 0 8 61976 1112 291 0 4 39 58
0 4 1112 24992 4868 3855120 0 568 4 53804 1124 452 0 4 54 43
0 4 1112 24032 4868 3856004 0 0 8 44652 1110 303 0 3 45 53
0 2 1112 25912 4872 3854164 0 0 4 51108 1122 321 0 3 52 45
2 3 1112 24312 4872 3855728 0 0 32 52240 1113 300 0 4 44 52
1 3 1112 25728 4876 3854432 0 0 20 48128 1118 296 0 3 58 39
0 2 1112 23872 4876 3856336 0 0 4 48136 1116 288 0 4 47 49
0 4 1112 25856 4876 3854300 0 4 16 50260 1117 294 0 3 55 42

CFQ3:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 5 1008 25888 4204 3845820 0 0 12 50544 1119 116 0 3 49 48
0 5 1008 24096 4204 3847520 0 0 8 51200 1112 110 0 3 49 48
0 5 1008 25824 4204 3845820 0 0 8 54816 1117 120 0 4 49 48
0 5 1008 25440 4204 3846160 0 0 8 52880 1113 115 0 3 49 48
0 5 1008 25888 4208 3845748 0 0 16 51024 1121 116 0 3 49 48