XFS Tunables for High Speed Linux SW RAID5 Systems?

XFS Tunables for High Speed Linux SW RAID5 Systems?

[Justin Piszcz]

Hi,

I was wondering if the XFS folks can recommend any optimizations for high 
speed disk arrays using RAID5?

As well as if there is anything else I can do on the MD side?

fs.xfs.restrict_chown = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.error_level = 3
fs.xfs.xfssyncd_centisecs = 3000
fs.xfs.inherit_sync = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_noatime = 1
fs.xfs.xfsbufd_centisecs = 100
fs.xfs.age_buffer_centisecs = 1500
fs.xfs.inherit_nosymlinks = 0
fs.xfs.rotorstep = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.stats_clear = 0

There is also vm/dirty tunable in /proc.

I was wondering what are some things to tune for speed?  I've already 
tuned the MD layer but is there anything with XFS I can also tune?

echo "Setting read-ahead to 64MB for /dev/md3"
blockdev --setra 65536 /dev/md3

echo "Setting stripe_cache_size to 16MB for /dev/md3"
echo 16384 > /sys/block/md3/md/stripe_cache_size

(also set max_sectors_kb) to 128K (chunk size) and disable NCQ

Justin.

Re: XFS Tunables for High Speed Linux SW RAID5 Systems?

[Eric Sandeen]

Justin Piszcz wrote:
> Hi,
> 
> I was wondering if the XFS folks can recommend any optimizations for high 
> speed disk arrays using RAID5?
> 
> As well as if there is anything else I can do on the MD side?
> 
> fs.xfs.restrict_chown = 1
> fs.xfs.irix_sgid_inherit = 0
> fs.xfs.irix_symlink_mode = 0

OT but...

/me wonders if some of these could go away by now... :)

-Eric

> fs.xfs.panic_mask = 0
> fs.xfs.error_level = 3
> fs.xfs.xfssyncd_centisecs = 3000
> fs.xfs.inherit_sync = 1
> fs.xfs.inherit_nodump = 1
> fs.xfs.inherit_noatime = 1
> fs.xfs.xfsbufd_centisecs = 100
> fs.xfs.age_buffer_centisecs = 1500
> fs.xfs.inherit_nosymlinks = 0
> fs.xfs.rotorstep = 1
> fs.xfs.inherit_nodefrag = 1
> fs.xfs.stats_clear = 0
> 
> There is also vm/dirty tunable in /proc.
> 
> I was wondering what are some things to tune for speed?  I've already 
> tuned the MD layer but is there anything with XFS I can also tune?
> 
> echo "Setting read-ahead to 64MB for /dev/md3"
> blockdev --setra 65536 /dev/md3
> 
> echo "Setting stripe_cache_size to 16MB for /dev/md3"
> echo 16384 > /sys/block/md3/md/stripe_cache_size
> 
> (also set max_sectors_kb) to 128K (chunk size) and disable NCQ
> 
> Justin.
> 
> 

Re: XFS Tunables for High Speed Linux SW RAID5 Systems?

[David Chinner]

On Fri, Jun 15, 2007 at 04:36:07PM -0400, Justin Piszcz wrote:
> Hi,
> 
> I was wondering if the XFS folks can recommend any optimizations for high 
> speed disk arrays using RAID5?

[sysctls snipped]

None of those options will make much difference to performance.
mkfs parameters are the big ticket item here....


> There is also vm/dirty tunable in /proc.

That changes benchmark times by starting writeback earlier, but
doesn't affect actual writeback speed.

> I was wondering what are some things to tune for speed?  I've already 
> tuned the MD layer but is there anything with XFS I can also tune?
> 
> echo "Setting read-ahead to 64MB for /dev/md3"
> blockdev --setra 65536 /dev/md3

Why so large? That's likely to cause readahead thrashing problems
under low memory....

> echo "Setting stripe_cache_size to 16MB for /dev/md3"
> echo 16384 > /sys/block/md3/md/stripe_cache_size
> 
> (also set max_sectors_kb) to 128K (chunk size) and disable NCQ

Why do that? You want XFS to issue large I/Os and the block layer
to split them across all the disks. i.e. you are preventing full
stripe writes from occurring by doing that.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS Tunables for High Speed Linux SW RAID5 Systems?

[Justin Piszcz]

Dave,

Questions inline and below.

On Mon, 18 Jun 2007, David Chinner wrote:

> On Fri, Jun 15, 2007 at 04:36:07PM -0400, Justin Piszcz wrote:
>> Hi,
>>
>> I was wondering if the XFS folks can recommend any optimizations for high
>> speed disk arrays using RAID5?
>
> [sysctls snipped]
>
> None of those options will make much difference to performance.
> mkfs parameters are the big ticket item here....
>
>
>> There is also vm/dirty tunable in /proc.
>
> That changes benchmark times by starting writeback earlier, but
> doesn't affect actual writeback speed.
>
>> I was wondering what are some things to tune for speed?  I've already
>> tuned the MD layer but is there anything with XFS I can also tune?
>>
>> echo "Setting read-ahead to 64MB for /dev/md3"
>> blockdev --setra 65536 /dev/md3
This proved to give the fastest performance, I have always used 4GB then 
recently 8GB of memory in the machine.

http://www.rhic.bnl.gov/hepix/talks/041019pm/schoen.pdf

See page 13.

>
> Why so large? That's likely to cause readahead thrashing problems
> under low memory....
>
>> echo "Setting stripe_cache_size to 16MB for /dev/md3"
>> echo 16384 > /sys/block/md3/md/stripe_cache_size
>>
>> (also set max_sectors_kb) to 128K (chunk size) and disable NCQ
>
> Why do that? You want XFS to issue large I/Os and the block layer
> to split them across all the disks. i.e. you are preventing full
> stripe writes from occurring by doing that.
I use a 128k stripe, what should I use for the max_sectors_kb? I read that 
128kb was optimal.

Can you please comment on all of the optimizations below?

#!/bin/bash

# source profile
. /etc/profile

echo "Optimizing RAID Arrays..."


# This step must come first.
# See: http://www.3ware.com/KB/article.aspx?id=11050

echo "Setting max_sectors_kb to chunk size of RAID5 arrays..."
for i in sdc sdd sde sdf sdg sdh sdi sdj sdk sdl
do
   echo "Setting /dev/$i to 128K..."
   echo 128 > /sys/block/"$i"/queue/max_sectors_kb
done

echo "Setting read-ahead to 64MB for /dev/md3"
blockdev --setra 65536 /dev/md3

echo "Setting stripe_cache_size to 16MB for /dev/md3"
echo 16384 > /sys/block/md3/md/stripe_cache_size

# if you use more than the default 64kb stripe with raid5
# this feature is broken so you need to limit it to 30MB/s
# neil has a patch, not sure when it will be merged.
echo "Setting minimum and maximum resync speed to 30MB/s..."
echo 30000 > /sys/block/md0/md/sync_speed_min
echo 30000 > /sys/block/md0/md/sync_speed_max
echo 30000 > /sys/block/md1/md/sync_speed_min
echo 30000 > /sys/block/md1/md/sync_speed_max
echo 30000 > /sys/block/md2/md/sync_speed_min
echo 30000 > /sys/block/md2/md/sync_speed_max
echo 30000 > /sys/block/md3/md/sync_speed_min
echo 30000 > /sys/block/md3/md/sync_speed_max

# Disable NCQ.
echo "Disabling NCQ..."
for i in sdc sdd sde sdf sdg sdh sdi sdj sdk sdl
do
   echo "Disabling NCQ on $i"
   echo 1 > /sys/block/"$i"/device/queue_depth
done

Re: XFS Tunables for High Speed Linux SW RAID5 Systems?

[David Greaves]

David Chinner wrote:
> On Fri, Jun 15, 2007 at 04:36:07PM -0400, Justin Piszcz wrote:
>> Hi,
>>
>> I was wondering if the XFS folks can recommend any optimizations for high 
>> speed disk arrays using RAID5?
> 
> [sysctls snipped]
> 
> None of those options will make much difference to performance.
> mkfs parameters are the big ticket item here....

Is there anywhere you can point to that expands on this?

Is there anything raid specific that would be worth including in the Wiki?

David