mdadm raid6->LVM->ext4 setup

mdadm raid6->LVM->ext4 setup

[Zoltan Szecsei]

Hi,
with the kind help from some of you on this list, I am finally 
formatting the filesystem.
I'd be grateful if any of you would like to comment on my chosen 
chunk/stripe/stride/ etc values, as from tomorrow I will be loading TBs 
of 400MB images onto this system, so would prefer to be coerced into 
making changes before then.

Thanks & regards,
Zoltan



***  Setup:
8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 
x86_64 GNU/Linux
mdadm - v3.1.4 - 31st August 2010
   LVM version:     2.02.54(1) (2009-10-26)
   Library version: 1.02.39 (2009-10-26)
   Driver version:  4.15.0
mke2fs 1.41.11 (14-Mar-2010)

***  Commands used:
mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean 
--bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6 
--raid-devices=8 /dev/sd[abcdefgh]
mdadm --detail --scan >> /etc/mdadm.conf
mdadm --assemble /dev/md0
pvcreate --metadatasize 512 /dev/md0
vgcreate geo_vg0 /dev/md0
lvcreate -n geo_lvm0 -l 100%FREE geo_vg0
mkfs -t ext4  -m 1-b 4096 -E stride=128,stripe-width=768 
/dev/geo_vg0/geo_lvm0


*** Results:
root@gs0:/home/geograph# mdadm -D /dev/md0
/dev/md0:
         Version : 1.2
   Creation Time : Tue Nov 16 17:28:37 2010
      Raid Level : raid6
      Array Size : 11721077760 (11178.09 GiB 12002.38 GB)
   Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
    Raid Devices : 8
   Total Devices : 8
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Tue Nov 16 19:44:10 2010
           State : active
  Active Devices : 8
Working Devices : 8
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 512K

            Name : gs0:0  (local to host gs0)
            UUID : c35e7642:7d78c385:61a26db1:89a5ace5
          Events : 3

     Number   Major   Minor   RaidDevice State
        0       8        0        0      active sync   /dev/sda
        1       8       16        1      active sync   /dev/sdb
        2       8       32        2      active sync   /dev/sdc
        3       8       48        3      active sync   /dev/sdd
        4       8       64        4      active sync   /dev/sde
        5       8       80        5      active sync   /dev/sdf
        6       8       96        6      active sync   /dev/sdg
        7       8      112        7      active sync   /dev/sdh
root@gs0:/home/geograph#
*******************************************************************
root@gs0:/home/geograph# pvdisplay
   --- Physical volume ---
   PV Name               /dev/md0
   VG Name               geo_vg0
   PV Size               10.92 TiB / not usable 0
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              2861463
   Free PE               0
   Allocated PE          2861463
   PV UUID               oE4Itq-rAwe-RQzK-JTnn-XAVk-vObv-ORx5HO

root@gs0:/home/geograph# vgdisplay
   --- Volume group ---
   VG Name               geo_vg0
   System ID
   Format                lvm2
   Metadata Areas        1
   Metadata Sequence No  2
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                1
   Open LV               1
   Max PV                0
   Cur PV                1
   Act PV                1
   VG Size               10.92 TiB
   PE Size               4.00 MiB
   Total PE              2861463
   Alloc PE / Size       2861463 / 10.92 TiB
   Free  PE / Size       0 / 0
   VG UUID               0S3VyN-aMu4-U60o-hz3z-R6RL-hGUd-nJHCTg

root@gs0:/home/geograph# lvdisplay
   --- Logical volume ---
   LV Name                /dev/geo_vg0/geo_lvm0
   VG Name                geo_vg0
   LV UUID                dnhY9R-mGO3-oRg6-0EP7-i6kt-2dIW-R9028r
   LV Write Access        read/write
   LV Status              available
   # open                 1
   LV Size                10.92 TiB
   Current LE             2861463
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     12288
   Block device           252:0

root@gs0:/home/geograph#

*********************************************************

root@gs0:/home/geograph# echo `date` > zz_mkfs ; mkfs -t ext4 -m 1 -b 
4096 -E stride=128,stripe-width=768 /dev/geo_vg0/geo_lvm0 ; echo `date` 
 >> zz_mkfs
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=768 blocks
732536832 inodes, 2930138112 blocks
29301381 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
89421 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
         4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 
78675968,
         102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
         2560000000



[still running]



-- 

===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.

65 Main Road, Muizenberg 7945
Western Cape, South Africa.

34° 6'16.35"S 18°28'5.62"E

Tel: +27-21-7884897  Mobile: +27-83-6004028
Fax: +27-86-6115323     www.geograph.co.za
===========================================



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3260 - Release Date: 11/16/10

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm raid6->LVM->ext4 setup

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

On Tue, 16 Nov 2010 20:08:44 +0200
Zoltan Szecsei <zoltans@geograph.co.za> wrote:

> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.

Excuse me, but how is that? This is a 2-port controller, so do you use port
multipliers, or 4 cards? (The latter would work an order of magnitude faster.)

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: mdadm raid6->LVM->ext4 setup

[Zoltan Szecsei]

On 2010-11-16 20:16, Roman Mamedov wrote:
> On Tue, 16 Nov 2010 20:08:44 +0200
> Zoltan Szecsei<zoltans@geograph.co.za>  wrote:
>
>    
>> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
>>      
> Excuse me, but how is that? This is a 2-port controller, so do you use port
> multipliers, or 4 cards? (The latter would work an order of magnitude faster.)
>
>    

The chassis has 2 * 4 port multipliers

http://www.chyangfun.com/pro01_2_3.asp

Regards,
Zoltan



-- 

===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.

65 Main Road, Muizenberg 7945
Western Cape, South Africa.

34° 6'16.35"S 18°28'5.62"E

Tel: +27-21-7884897  Mobile: +27-83-6004028
Fax: +27-86-6115323     www.geograph.co.za
===========================================



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3260 - Release Date: 11/16/10

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm raid6->LVM->ext4 setup

[Luca Berra]

On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
> ***  Setup:
> 8 * 5900RPM Seagte 2TB drives on SiI3132 PCIx x1 card.
> Linux gs0 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 
> x86_64 GNU/Linux
> mdadm - v3.1.4 - 31st August 2010
>   LVM version:     2.02.54(1) (2009-10-26)
>   Library version: 1.02.39 (2009-10-26)
>   Driver version:  4.15.0
> mke2fs 1.41.11 (14-Mar-2010)
>
> ***  Commands used:
> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean 
> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6 
> --raid-devices=8 /dev/sd[abcdefgh]
> mdadm --detail --scan >> /etc/mdadm.conf
> mdadm --assemble /dev/md0
as i said this is not useful
> pvcreate --metadatasize 512 /dev/md0
you should use --dataalignment, not --metadatasize, also you should
align to a full stripe
> vgcreate geo_vg0 /dev/md0
> lvcreate -n geo_lvm0 -l 100%FREE geo_vg0
> mkfs -t ext4  -m 1-b 4096 -E stride=128,stripe-width=768 
> /dev/geo_vg0/geo_lvm0
-- 
Luca Berra -- bluca@comedia.it

Re: mdadm raid6->LVM->ext4 setup

[Michal Soltys]

On 10-11-16 23:17, Luca Berra wrote:
> On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
>>
>> *** Commands used:
>> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
>> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
>> --raid-devices=8 /dev/sd[abcdefgh]
>> mdadm --detail --scan >> /etc/mdadm.conf
>> mdadm --assemble /dev/md0
> as i said this is not useful
>> pvcreate --metadatasize 512 /dev/md0
> you should use --dataalignment, not --metadatasize, also you should

Also with reference to metadatasize - keep in mind the value is rounded 
up to the /next/ 64k boundary (or it was a while ago, perhaps it's 
changed now). So in your case, 512 would really result with 576.

With reference to --dataalignment - there's also --dataalignoffset 
option (although not really important in your setup).

You might consider adjusting 
/sys/block/<yourmddevice>/md/stripe_cache_size to higher values when you 
have everything set. It tends to give quite a performance boost.

Re: mdadm raid6->LVM->ext4 setup

[Zoltan Szecsei]

On 2010-11-17 01:39, Michal Soltys wrote:
> On 10-11-16 23:17, Luca Berra wrote:
>> On Tue, Nov 16, 2010 at 08:08:44PM +0200, Zoltan Szecsei wrote:
>>>
>>> *** Commands used:
>>> mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean
>>> --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6
>>> --raid-devices=8 /dev/sd[abcdefgh]
>>> mdadm --detail --scan >> /etc/mdadm.conf
>>> mdadm --assemble /dev/md0
>> as i said this is not useful
Accepted, but I also understood "not harmfull". I tried it out of 
curiosity, it took a nano-second to complete, so didn't try to "undo it".
>>> pvcreate --metadatasize 512 /dev/md0
>> you should use --dataalignment, not --metadatasize, also you should
I checked out man pvcreate & lvm.conf had all the stuff below in it, so 
I assumed it would sort itself out.
md_component_detection = 1
md_chunk_alignment = 1
data_alignment_detection = 1
data_alignment_offset_detection = 1

>
> Also with reference to metadatasize - keep in mind the value is 
> rounded up to the /next/ 64k boundary (or it was a while ago, perhaps 
> it's changed now). So in your case, 512 would really result with 576.
>
> With reference to --dataalignment - there's also --dataalignoffset 
> option (although not really important in your setup).
>
> You might consider adjusting 
> /sys/block/<yourmddevice>/md/stripe_cache_size to higher values when 
> you have everything set. It tends to give quite a performance boost.

I had trouble getting my head around all the calculations, mainly 
because I do not know if my sector size of 512 is falsely reported or 
not (so I just accepted it as 512).

I am loading the array as we speak, with 200MB to 400MB tiff files. In 
the past 11 hours approx 995GB was loaded, so that calculates to a write 
speed of 25MB/s

I have no problem stopping & deleting this if you feel that this 
represents problems in my setup, but then please can someone do the 
alignment calculations & command syntax for me.
Regards to all,
Zoltan







-- 

===========================================
Zoltan Szecsei PrGISc [PGP0031]
Geograph (Pty) Ltd.
P.O. Box 7, Muizenberg 7950, South Africa.

65 Main Road, Muizenberg 7945
Western Cape, South Africa.

34° 6'16.35"S 18°28'5.62"E

Tel: +27-21-7884897  Mobile: +27-83-6004028
Fax: +27-86-6115323     www.geograph.co.za
===========================================



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1153 / Virus Database: 424/3261 - Release Date: 11/16/10

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html