how to use mkfs.ext3 "stride=" on LVM on RAID correctly?

how to use mkfs.ext3 "stride=" on LVM on RAID correctly?

[Tomasz Chmielewski]

Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem 
placed on LVM, which was placed on Linux RAID-5.

Luckily, it was possible to alleviate much of this problem by removing 
internal bitmap from RAID-5 and by using anticipatory IO scheduler.


As suggested earlier by Theodore Tso in "very poor ext3 write 
performance on big filesystems" thread on linux-fsdevel list, I should 
have used "mke2fs -E stride option to make sure all of the bitmaps
don't get concentrated on one hard drive spindle".


Although the fine mkfs.ext3 manual gives some basic information on how 
to do it if you place the whole filesystem on a RAID array, it is not 
clear to me how it should be done correctly if I want to create a ext3 
filesystem on LVM on RAID-5.

Any helpful hints?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: how to use mkfs.ext3 "stride=" on LVM on RAID correctly?

[Jon Nelson]

On Thu, Feb 28, 2008 at 3:38 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem
>  placed on LVM, which was placed on Linux RAID-5.
>
>  Luckily, it was possible to alleviate much of this problem by removing
>  internal bitmap from RAID-5 and by using anticipatory IO scheduler.
>
>
>  As suggested earlier by Theodore Tso in "very poor ext3 write
>  performance on big filesystems" thread on linux-fsdevel list, I should
>  have used "mke2fs -E stride option to make sure all of the bitmaps
>  don't get concentrated on one hard drive spindle".
>
>
>  Although the fine mkfs.ext3 manual gives some basic information on how
>  to do it if you place the whole filesystem on a RAID array, it is not
>  clear to me how it should be done correctly if I want to create a ext3
>  filesystem on LVM on RAID-5.
>
>  Any helpful hints?

I can't help with the stride stuff, as I use LVM and that makes the
stride stuff utterly useless anyway. However, a suggestion I have for
you which I found helped a great deal was to place the bitmap on some
/other/ device's filesystem (I used a /etc/bitmaps/). Obviously, don't
put it on a FS which is part of the raid, I typically divide each
device into 2 or more raids, but only if I don't have enough devices
for /system/ and /data/ -style pools.

-- 
Jon

Re: how to use mkfs.ext3 "stride=" on LVM on RAID correctly?

[Michal Soltys]

Tomasz Chmielewski wrote:

> Although the fine mkfs.ext3 manual gives some basic information on how 
> to do it if you place the whole filesystem on a RAID array, it is not 
> clear to me how it should be done correctly if I want to create a ext3 
> filesystem on LVM on RAID-5.
> 
> Any helpful hints?

You should be careful with all the alignemnts. Extents should be properly 
aligned with - at the very least - stripes, and preferably with whole stripe 
width (not always possible though). To check where lvm extents, you can use 
i.e.:

dmsetup table
pvs -o+pe_start --units s
losetup and hexedit experiments (silly, but I used to do it like that in the 
past, before I realised about other options :o )

If you don't need default 4M granularity, it's nice to increase it (vgcreate 
-s option ; I use 256M and 512M sized extents). You can alter the beginning 
of the extents with --metadatasize option of pvcreate (overally it always 
rounds up to the next 64K multiple). Be sure to verify with one of the 
commands above.

Moreover - if you use partitionable raid and create lvm in one of the 
partitions, make sure partition is properly aligned as well.