Are there some alignment settings when creating filesystem on RAID5 array which can improve performance?

Are there some alignment settings when creating filesystem on RAID5 array which can improve performance?

[hank peng]

Hi, everybody:
I have a machine and centos 5.2 installed on it. I have created a
RAID5 array with five SATA disks, then I create LVM on this array.
#pvcreate /dev/md0
#vgcreate myvg /dev/md0
#lvcreate -n mylv myvg
As you see, I want to use XFS or ReiserFS on "mylv" volume. I wonder
if there are some alignment settings when creating such file systems
so that a better R/W performance can be achieved. Thx in advance.


-- 
The simplest is not all best but the best is surely the simplest!

RE: Are there some alignment settings when creating filesystem on RAID5 array which can improve performance?

[David Lethe]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of hank peng
> Sent: Sunday, January 04, 2009 7:08 PM
> To: linux-raid
> Subject: Are there some alignment settings when creating filesystem on
> RAID5 array which can improve performance?
> 
> Hi, everybody:
> I have a machine and centos 5.2 installed on it. I have created a
> RAID5 array with five SATA disks, then I create LVM on this array.
> #pvcreate /dev/md0
> #vgcreate myvg /dev/md0
> #lvcreate -n mylv myvg
> As you see, I want to use XFS or ReiserFS on "mylv" volume. I wonder
> if there are some alignment settings when creating such file systems
> so that a better R/W performance can be achieved. Thx in advance.
> 
> 
> --
> The simplest is not all best but the best is surely the simplest!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Yes, but be specific.  How do you use the volume?   Do you want to optimize for IOPs or
raw file transfer?  What is mix of random vs sequential for both reads and writes.
What is block/chunk size of volume, raidset, filesystem?  Reveal your
journaling settings.  What are you getting now, and how did you measure it?   Are things
like NCQ/TCQ/read-write cache enabled or disabled?  

How are the disks physically attached?  PCI(-X) or PCIe, how many channels  are you using, or are
the disks attached via USB ports.  

Need more info.

Re: Are there some alignment settings when creating filesystem on RAID5 array which can improve performance?

[hank peng]

2009/1/5 David Lethe <david@santools.com>:
>> -----Original Message-----
>> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> owner@vger.kernel.org] On Behalf Of hank peng
>> Sent: Sunday, January 04, 2009 7:08 PM
>> To: linux-raid
>> Subject: Are there some alignment settings when creating filesystem on
>> RAID5 array which can improve performance?
>>
>> Hi, everybody:
>> I have a machine and centos 5.2 installed on it. I have created a
>> RAID5 array with five SATA disks, then I create LVM on this array.
>> #pvcreate /dev/md0
>> #vgcreate myvg /dev/md0
>> #lvcreate -n mylv myvg
>> As you see, I want to use XFS or ReiserFS on "mylv" volume. I wonder
>> if there are some alignment settings when creating such file systems
>> so that a better R/W performance can be achieved. Thx in advance.
>>
>>
>> --
>> The simplest is not all best but the best is surely the simplest!
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> Yes, but be specific.  How do you use the volume?   Do you want to optimize for IOPs or
> raw file transfer?  What is mix of random vs sequential for both reads and writes.
> What is block/chunk size of volume, raidset, filesystem?  Reveal your
> journaling settings.  What are you getting now, and how did you measure it?   Are things
> like NCQ/TCQ/read-write cache enabled or disabled?
>
> How are the disks physically attached?  PCI(-X) or PCIe, how many channels  are you using, or are
> the disks attached via USB ports.
>
I am new to this area, so I'm not quite familiar with some words what
you mentioned.
The machine has a SATA controller (chip is Marvell 6081) attached on
PCI-X bus. Five SATA II disks are attached to it.
Each disk has 500G space.
The following is my procedure:
#mdadm -C /dev/md0 -l5 -n5 /dev/sd{a,b,c,d,e}
After recovery is done, I do this:
#pvcreate /dev/md0
#vgcreate myvg /dev/md0
#lvcreate -n mylv -L 1000G myvg
#mkfs.xfs /dev/myvg/mylv  or  #mkfs.reiserfs /dev/myvg/mylv
mount this file system and begin to use it.
I mainly want to optimise its sequential write performace, IOPs is not
my concern.
SATA driver is provided by marvell, so NCQ/TCQ/read-write is default.


> Need more info.
>
>
>



-- 
The simplest is not all best but the best is surely the simplest!

Re: Are there some alignment settings when creating filesystem on RAID5 array which can improve performance?

[Michal Soltys]

hank peng wrote:
> I am new to this area, so I'm not quite familiar with some words what
> you mentioned.
> The machine has a SATA controller (chip is Marvell 6081) attached on
> PCI-X bus. Five SATA II disks are attached to it.
> Each disk has 500G space.
> The following is my procedure:
> #mdadm -C /dev/md0 -l5 -n5 /dev/sd{a,b,c,d,e}
> After recovery is done, I do this:
> #pvcreate /dev/md0
> #vgcreate myvg /dev/md0
> #lvcreate -n mylv -L 1000G myvg
> #mkfs.xfs /dev/myvg/mylv  or  #mkfs.reiserfs /dev/myvg/mylv
> mount this file system and begin to use it.
> I mainly want to optimise its sequential write performace, IOPs is not
> my concern.

When you create PV, it consists of usually initial 192KiB area with 
metadata (it can be controlled by --metadatasize option of pvcreate). 
Then extents follow (4MiB by default). As far as alignment is 
considered, the best case scenario is when extents are aligned with 
raid's stripe. It's possible in your case, but not generally - as 
extents must be power of 2.

Try:

pvcreate --metadatasize 250K /dev/md0 (250K will be rounded up properly)

...and verify

pvs /dev/md0 -o+pe_start

...you should get 256.00K under 1st PE. 256K, as it's your stripe's size 
(md defaults to 64KiB chunk, and you haven't altered it).

Most filesystems allow setting stripe and chunk parameters - ext{2,3,4}, 
xfs - to name a few. They are used to e.g. setup their structures more 
optimally, and avoid read-modify-write if at all possible. I don't know 
if reiser has such settings, but xfs certailny does (look for su/sw 
options of mkfs.xfs).

Note, that when you create filesystems on logical volumes, they will not 
detect under-the-lvm raid structure - you have to set that manually. If 
extents are not aligned, then any settings related to stripe will be 
meaningless (as filesystem assumes it starts are stripe boundary 
itself). Chunk size will [/might] still be useful though.

Another easily forgotten parameter is LV's readahead. If not set 
explicitly, it will default to 256, which is quite small value. You can 
change it with blockdev or lvchange (permanently with the latter). RA 
set on md0 directly doesn't matter afaik, unless you also plan to setup 
filesystems directly on it.

Check out /sys/class/block/md0/md/stripe_cache_size (or /sys/block/.. if 
you use old sysfs layout) and increase it, if you have memory to spare.

Increasing RA and stripe_cache_size can provide very significant boost. 
Forgetting about the former is often a cause for complains about lvm 
performance (when compared to md used directly).

There's definitely more to it (like specific filesystem's creation and 
mount options, or more basically - filesystem choice). Best wait for 
David's input.