[problem] raid performance loss with 2.6.26-rc8 on 32-bit x86 (bisected)

[Dan Williams <dan.j.williams@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2138 bytes --]

Hello,

Prompted by a report from a user I have bisected a performance loss
apparently introduced by commit 54a6eb5c (mm: use two zonelist that are
filtered by GFP mask).  The test is simple sequential writes to a 4 disk
raid5 array.  Performance should be about 20% greater than 2.6.25 due to
commit 8b3e6cdc (md: introduce get_priority_stripe() to improve raid456
write performance).  The sample data below shows sporadic performance
starting at 54a6eb5c.  The '+' indicates where I hand applied 8b3e6cdc.

revision   2.6.25.8-fc8 2.6.25.9+ dac1d27b+ 18ea7e71+ 54a6eb5c+ 2.6.26-rc1 2.6.26-rc8
           138          168       169       167       177       149        144
           140          168       172       170       109       138        142
           142          165       169       164       119       138        129
           144          168       169       171       120       139        135
           142          165       174       166       165       122        154
MB/s (avg) 141          167       171       168       138       137        141
% change   0%           18%       21%       19%       -2%       -3%        0%
result     base         good      good      good      [bad]     bad        bad

Notable observations:
1/ This problem does not reproduce when ARCH=x86_64, i.e. 2.6.26-rc8 and
54a6eb5c show consistent performance at 170MB/s.
2/ Single drive performance appears to be unaffected
3/ A quick test shows that raid0 performance is also sporadic:
   2147483648 bytes (2.1 GB) copied, 7.72408 s, 278 MB/s
   2147483648 bytes (2.1 GB) copied, 7.78478 s, 276 MB/s
   2147483648 bytes (2.1 GB) copied, 11.0323 s, 195 MB/s
   2147483648 bytes (2.1 GB) copied, 8.41244 s, 255 MB/s
   2147483648 bytes (2.1 GB) copied, 30.7649 s, 69.8 MB/s

System/Test configuration:
(2) Intel(R) Xeon(R) CPU 5150
mem=1024M
CONFIG_HIGHMEM4G=y (full config attached)
mdadm --create /dev/md0 /dev/sd[b-e] -n 4 -l 5 --assume-clean
for i in `seq 1 5`; do dd if=/dev/zero of=/dev/md0 bs=1024k count=2048; done

Neil suggested CONFIG_NOHIGHMEM=y, I will give that a shot tomorrow.
Other suggestions / experiments?

Thanks,
Dan

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 22024 bytes --]