Looking for Linux XFS file system performance tuning tips for LSI 9271-8i + 8 SSD's RAID0

Looking for Linux XFS file system performance tuning tips for LSI 9271-8i + 8 SSD's RAID0

[rkj]

I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's.  I am using 
CentOS 6.3 on a Supermicro X9SAE-V motherboard with Intel Xeon E3-1275V2 CPU 
and 32GB 1600 MHz ECC RAM.  My application is fast sensor data store and 
forward with UDP based file transfer using multiple 10GbE interfaces.  So I 
do not have any concurrent loading, I am mainly interested in optimizing 
sequential read/write performance. 

Raw performance as measured by Gnome Disk Utility is around 4GB/s sustained 
read/write.  With XFS buffer IO, my sequential writes max out at about  2.5 
GB/s.  With Direct IO, the sequential writes are around 3.5 GB/s but I 
noticed a drop-off in sequential reads for smaller record sizes.  I am 
trying to get the XFS sequential read/writes as close to 4 GB/s as possible. 

I have documented all of the various mkfs.xfs options I have tried, fstab 
mount options, iozone results, etc. in this forum thread: 

http://www.xtremesystems.org/forums/showthread.php?284853-Looking-for-Linux- 
file-system-performance-tuning-tips-for-LSI-9271-8i-8-SSD-s-RAID0 

Please let me know if you have any suggestions.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Looking for Linux XFS file system performance tuning tips for LSI 9271-8i + 8 SSD's RAID0

[Linda Walsh]

rkj@softhome.net wrote:
> 
> I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's...
>... With Direct IO, the sequential writes are around 
> 3.5 GB/s but I noticed a drop-off in sequential reads for smaller record 
> sizes.
----
1) Are you using the realtime partitions?
2) are you using an empty file space with max log size
3) what are you using to do write's with, "dd"?
4) have you tried pre-allocating the space, running xfs_fsr on the
file, to get it to 1 extent in size,
then run dd with the conv=nocreat,notrunc option?

Those are some things to try for maximums.
I notice ~30% hit if I go through the linux buffer system as well.
Have you tried different write sizes?

i.e. 32M, 64M, 128 256... i.e. I notice optimal sizes after
which things fall off...

do you run 'time' on "dd" as well and see how much cpu it is using?

If you use bash, I find setting TIMEFORMAT to:

"%2Rsec %2Uusr %2Ssys (%P%% cpu)"

to be a useful format (1 line, and shows cpu % usage)...

just some random thoughts...


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Looking for Linux XFS file system performance tuning tips for LSI9271-8i + 8 SSD's RAID0

[Dave Chinner]

On Sun, Feb 03, 2013 at 01:36:48PM -0700, rkj@softhome.net wrote:
> 
> I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's.  I am
> using CentOS 6.3 on a Supermicro X9SAE-V motherboard with Intel Xeon
> E3-1275V2 CPU and 32GB 1600 MHz ECC RAM.  My application is fast
> sensor data store and forward with UDP based file transfer using
> multiple 10GbE interfaces.  So I do not have any concurrent loading,
> I am mainly interested in optimizing sequential read/write
> performance.
>
> Raw performance as measured by Gnome Disk Utility is around 4GB/s
> sustained read/write.

I don't know what that does - probably lots of concurrent IO to drive
deep queue depths to get the absolute maximum possible from the
device....

> With XFS buffer IO, my sequential writes max
> out at about  2.5 GB/s.

CPU bound on single threaded IO, I'd guess.

> With Direct IO, the sequential writes are
> around 3.5 GB/s but I noticed a drop-off in sequential reads for
> smaller record sizes.

Almost certainly IO latency bound on single threaded IO.

> I am trying to get the XFS sequential
> read/writes as close to 4 GB/s as possible.

Time to go look up how to use async IO or multithreaded direct
IO.

FWIW, the best benchmark is your application - none of what you've
talked about even come close to modelling the data flow a
network-disk-network store-and-forward system needs, and a data
rates of 4GB/s you are going to benchmark the network devices
flowing data at the same time you do disk IO....

> I have documented all of the various mkfs.xfs options I have tried,
> fstab mount options, iozone results, etc. in this forum thread:

Configuration changes won't make any difference to data IO latency
or CPU usage. IOWs, SSDs don't magically solve the problem of having
to optimise the way the applications/benchmarks do IO and so no
amount of tweaking the filesystem will get you to your goal if the
application is deficient...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs