XFS buffered sequential read performance low after kernel upgrade

Stan Hoeppner
2010-03-12T11:46:06Z

Hello,

I'm uncertain whether this is the best place to bring this up. I've been
lurking a short while and it seems almost all posts here deal with dev
issues. On the off chance this is an appropriate forum, here goes.

I believe I recently ran into my first "issue" with XFS. Up to now I've
been pleased as punch with XFS' performance and features. I rolled a new
kernel the other day going from vanilla kernel.org 2.6.31.1 to 2.6.32.9.
This is an i386 binary small mem kernel running on a dual Intel P6 class
system. I tried 2.6.33 but apparently my version of gcc in Debian Lenny is
too old. Anyway, I'm noticing what I believe to be a fairly substantial
decrease in sequential read performance after upgrading my kernel.

The SUT (system under test) has one single platter WD 500GB 7.2K rpm SATA
disk, Sil 3512 chip, sata_sil driver, NCQ disabled, elevator=deadline, disk
carved as follows and with the following mount options and xfs_info:

Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext2 92M 5.9M 81M 7% /boot
/dev/sda2 ext2 33G 6.9G 25G 22% /
/dev/sda6 xfs 94G 2.0G 92G 3% /home
/dev/sda7 xfs 94G 20G 74G 21% /samba

/dev/sda1 /boot ext2 defaults 0 1
/dev/sda2 / ext2 errors=remount-ro 0 2
/dev/sda5 none swap sw 0 0
/dev/sda6 /home xfs defaults 0 0
/dev/sda7 /samba xfs defaults 0 0

~$ xfs_info /home
meta-data=/dev/sda6 isize=256 agcount=4, agsize=6103694 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=24414775, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=11921, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0

~$ xfs_info /samba
meta-data=/dev/sda7 isize=256 agcount=4, agsize=6103694 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=24414775, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=11921, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0

hdparm test results for the filesystems:

/dev/sda2:
Timing O_DIRECT disk reads: 236 MB in 3.01 seconds = 78.48 MB/sec
/dev/sda2:
Timing buffered disk reads: 172 MB in 3.03 seconds = 56.68 MB/sec

/dev/sda6:
Timing O_DIRECT disk reads: 238 MB in 3.00 seconds = 79.21 MB/sec
/dev/sda6:
Timing buffered disk reads: 116 MB in 3.03 seconds = 38.27 MB/sec

/dev/sda7:
Timing O_DIRECT disk reads: 238 MB in 3.01 seconds = 79.10 MB/sec
/dev/sda7:
Timing buffered disk reads: 114 MB in 3.00 seconds = 37.99 MB/sec

Note that XFS is giving up almost 20MB/s to EXT2 in the hdparm read tests
through the Linux buffer cache. IIRC hdparm is supposed to ignore the
filesystem, according to the man page, but I don't think this is true given
the that the O_DIRECT read performance for the EXT2 partition and both XFS
partitions is identical. Going through the buffer cache cuts XFS read
performance in half compared to O_DIRECT. EXT2 fares much better, losing
about a third of its O_DIRECT performance to the buffer cache.

Some dd read tests:

~$ dd if=/dev/sda2 of=/dev/null bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.69972 s, 71.9 MB/s

~$ dd if=/dev/sda6 of=/dev/null bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 7.53601 s, 54.4 MB/s

~$ dd if=/dev/sda7 of=/dev/null bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 8.25571 s, 49.6 MB/s

Same thing with dd. The XFS partitions lag behind EXT2 by about 20MB/s,
although the overall numbers are better for dd than hdparm which seems to be
my general experience using the two utils.

Some small dd write tests:
EXT2
~$ dd if=/dev/zero of=/test.xfs bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.72976 s, 71.5 MB/s

XFS
~$ dd if=/dev/zero of=/home/stan/test.xfs bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.97482 s, 69.6 MB/s

XFS
~$ dd if=/dev/zero of=/samba/test.xfs bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.91914 s, 68.8 MB/s

XFS seems to keep right up with EXT2 in the write tests. The 35GB EXT2
partition is on the outer edge of the platter, following inward by the 100GB
XFS /home partition and then the 100GB /samba partition. I think the slight
performance difference in the dd write tests is mostly due to partition
placement on the platter.

I don't recall doing any testing as formal as this with the old kernel.
However, I don't recall sub 40MB/s XFS read numbers from hdparm. That
really surprised me when I went kicking the tires on the new kernel. IIRC,
on the previous kernel, buffered sequential read performance was pretty much
the same for EXT2 and XFS, with XFS showing a small but significant lead in
write performance.

So, finally, my question: Is there a known issue with XFS performance in
kernel 2.6.32.x, or is there something I need to tweak manually in the mount
options or other in 2.6.32.x that was automatic in the previous kernel? I
created the filessytems on 2.6.21.1 if that has any bearing.

Thanks in advance for any answers or advice you may be able to provide.