30 TB RAID6 + XFS slow write performance

John Bokma
2011-07-18T19:58:57Z

Dear list members,

A customer of mine is currently struggling with the performance of a 30
TB RAID6 which uses XFS as the filing system. I am somewhat sure it's
not XFS that's causing the performance issue but my expertise is not XFS
nor RAID; I just wrote the software that after moving to the larger RAID
(from a much smaller one, ~ 3TB, using ext3) suddenly seems to have a
huge drop in write performance.

The software I wrote writes many small (50-150K) files in parallel (100+
processes), thousands of times per hour. Writing a file of 50-150K now
and then seems to take between 30 and 90 seconds, and more rarely can
take over 200 seconds (several times an hour).

When all processes are stopped and restarted again the 30-90 seconds
delay start happening when about 16-20+ processes are running.

To me this sounds like something has been configured wrong. I already
recommended my customer to find someone who is capable of configuring
the RAID correctly; to me it sounds like a hardware/configuration issue.

Any insights are very welcome.

Hardware:
card: MegaRAID SAS 9260-16i
disks: 14x Barracuda® XT ST33000651AS 3TB (2 hot spares).
RAID6
~ 30TB

Thanks for reading,
John