Re: Poor read performance on high-end server

[Freek Dijkstra <Freek.Dijkstra@sara.nl>]

Hi all,

Thanks a lot for the great feedback from before the weekend. Since one
of my colleagues needed the machine, I could only do the tests today.

In short: just installing 2.6.35 did make some difference, but I was
mostly impressed with the speedup gained by the hardware acceleration of
the crc32c_intel module.

Here is some quick data.

Reference figures:
16* single disk (theoretical limit):     4092 MiByte/s
fio data layer tests (achievable limit): 3250 MiByte/s
ZFS performance:                         2505 MiByte/s

BtrFS figures:
IOzone on 2.6.32:                         919 MiByte/s
fio btrfs tests on 2.6.35:               1460 MiByte/s
IOzone on 2.6.35 with crc32c:            1250 MiByte/s
IOzone on 2.6.35 with crc32c_intel:      1629 MiByte/s
IOzone on 2.6.35, using -o nodatasum:    1955 MiByte/s

For those finding this message and want a howto: the easiest way to use
crc32c_intel is to add the module name to /etc/modules:
 # echo "crc32c_intel" >> /etc/modules
 # reboot

Now the next step for us is to tune the block sizes. We only did that
preliminary, but now that we have a good knowledge of what software to
use, we can start tuning that in more detail.

If there is interest on this list, I'll gladly post our results here.


Jens Axboe wrote:

>>> Also, I didn't see Chris mention this, but if you have a newer intel box
>>> you can use hw accellerated crc32c instead. For some reason my test box
>>> always loads crc32c and not crc32c-intel, so I need to do that manually.
> 
> it is pretty annoying to have to do it manually. Sometimes
> you forget. And it's not possible to de-select CRC32C and have
> the intel variant loaded.

You can, but only if you first unmount the partition:

 # unmount /mnt/mybtrfsdisk
 # rmmod btrfs
 # rmmod libcrc32c
 # rmmod crc32c
 # modprobe crc32c_intel
 # mount -t btrfs /dev/sda1 /mnt/mybtrfsdisk




We encountered a small bug: the btrfs partition with RAID0 that was made
on 2.6.32 did not mount after a reboot or after unmounting. Running
btrfsck fixes this, but after a next umount, we had to run btrfsck
again. After recreating the btrfs partition on 2.6.35, all was well.
btrfs partitions that don't use (software) RAID work fine.

~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd3
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

~# dmesg | tail
device fsid ec4d518ec61d4496-81e5aeda2d8ef7b5 devid 1 transid 69 /dev/sdd
btrfs: use ssd allocation scheme
btrfs: failed to read the system array on sdd
btrfs: open_ctree failed

~# btrfsck /dev/sdd
found 550511136768 bytes used err is 0
total csum bytes: 536870912
total tree bytes: 755322880
total fs tree bytes: 77824
btree space waste bytes: 169152328
file data blocks allocated: 549755813888
 referenced 549755813888
Btrfs Btrfs v0.19

~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd3
[and it mounts fine now]


Regards,
Freek Dijkstra
SARA High Performance Computing and Networking