Setup for Performance

Setup for Performance

[Andrew Dunn]

I will preface this by saying I only need about 100MB/s out of my array
because I access it via a gigabit crossover cable.

I am backing up all of my information right now (~4TB) with the
intention of re-creating this array with a larger chunk size and
possibly tweaking the file system a little bit.

My original array was a raid6 of 9 WD caviar black drives, the chunk
size was 64k. I use USAS-AOC-L8i controllers to address all of my drives
and the TLER setting on the drives is enabled for 7 seconds.

storrgie@ALEXANDRIA:~$ sudo mdadm -D /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Wed Oct 14 19:59:46 2009
     Raid Level : raid6
     Array Size : 6837319552 (6520.58 GiB 7001.42 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 9
  Total Devices : 9
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Nov  2 16:58:43 2009
          State : active
 Active Devices : 9
Working Devices : 9
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 64K

           UUID : 53dadda1:c58785d5:613e2239:070da8c8 (local to host
ALEXANDRIA)
         Events : 0.649527

    Number   Major   Minor   RaidDevice State
       0       8       65        0      active sync   /dev/sde1
       1       8       81        1      active sync   /dev/sdf1
       2       8       97        2      active sync   /dev/sdg1
       3       8      113        3      active sync   /dev/sdh1
       4       8      129        4      active sync   /dev/sdi1
       5       8      145        5      active sync   /dev/sdj1
       6       8      161        6      active sync   /dev/sdk1
       7       8      177        7      active sync   /dev/sdl1
       8       8      193        8      active sync   /dev/sdm1

I have noticed slow rebuilding time when I first created the array and
intermittent lockups while writing large data sets.

Per some reading I was thinking of adjusting my chunk size to 1024k, and
trying to figure out the weird stuff required when creating a file system.

Questions:

Should I have the TLER on my drives enabled? (WDTLER, seven seconds)

Is 1024k chunk size going to be a good choice for my purposes? (I store
use this for storage of files that are 4MiB to 16GiB)

Is ext4 the ideal file system for my purposes?

Should I be investigating into the file system stripe size and chunk
size or let mkfs choose these for me? If I need to, please be kind to
point me in a good direction as I am new to this lower level file system
stuff.

Can I change the properties of my file system in place (ext4 or other)
so that I can tweak the stripe size when I add more drives and grow the
array?

Should I be asking any other questions?

Thanks a ton, this is the first mailing list I have ever subscribed, I
am really excited to see what you all say.

-- 
Andrew Dunn
http://agdunn.net

Re: Setup for Performance

[Richard Scobie]

Andrew Dunn wrote:

> My original array was a raid6 of 9 WD caviar black drives, the chunk
> size was 64k. I use USAS-AOC-L8i controllers to address all of my drives
> and the TLER setting on the drives is enabled for 7 seconds.

> I have noticed slow rebuilding time when I first created the array and
> intermittent lockups while writing large data sets.

Does the USAS-AOC-L8i use a Marvell controller?

If so, someone was posting here recently that they have problems with it 
locking up under md RAID loads. Have a look over the past couple of 
months archives.

Regards,

Richard

Re: Setup for Performance

[Andrew Dunn]

Just quick searching around:

"Using a LSISAS 1068E I/O processor with Fusion-MPT (Message Passing
Technology) for optimum RAID performance and a PCI Express host
interface for increased bandwidth, these UIO internal RAID adapters
deliver an intelligent and robust RAID solution."

http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html

I am not sure if iozone was used correctly here:

storrgie@ALEXANDRIA:/mnt/ARRAY$ iozone -s 4096
    Run began: Mon Nov  2 18:02:41 2009

    File size set to 4096 KB
    Command line used: iozone -s 4096
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
                                                            random 
random    bkwd   record   stride                                  
              KB  reclen   write rewrite    read    reread    read  
write    read  rewrite     read   fwrite frewrite   fread  freread
            4096       4  670491 1309061  2432098  2163764 2205712
1148325 2123381  1300736  2214241   647788  1131832 2419425  2079181

I did not set any of the file system settings when creating my array's
file system, I simply did a mkfs.ext4 on the device. This leads me to
believe that I lost some of the performance due to my laziness.


Richard Scobie wrote:
> Andrew Dunn wrote:
>
>> My original array was a raid6 of 9 WD caviar black drives, the chunk
>> size was 64k. I use USAS-AOC-L8i controllers to address all of my drives
>> and the TLER setting on the drives is enabled for 7 seconds.
>
>> I have noticed slow rebuilding time when I first created the array and
>> intermittent lockups while writing large data sets.
>
> Does the USAS-AOC-L8i use a Marvell controller?
>
> If so, someone was posting here recently that they have problems with
> it locking up under md RAID loads. Have a look over the past couple of
> months archives.
>
> Regards,
>
> Richard

-- 
Andrew Dunn
http://agdunn.net