What layout with 6x2TB, Intel Atom?

What layout with 6x2TB, Intel Atom?

[Mathias Burén]

Hi list,

I've an Intel Atom 330 running 32-bit Ubuntu, ~3.5GB RAM. The intent
is to store large media files (>4GB) for streaming by maximum 3
clients at any time. Very read heavy.
6x 2TB HDDs are available, the crux is that 3 HDDs are on the on-board
SATA controller, the other 3 are on a Marvel PCI-E adapter

cat /proc/mdstat

Personalities : [raid10]
md125 : active raid10 sdg[1] sdf[0]
      1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
      [>....................]  resync =  1.1% (22074880/1953382400)
finish=642.2min speed=50114K/sec

md126 : active raid10 sde[1] sdd[0]
      1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
      [>....................]  resync =  1.6% (31473408/1953382400)
finish=398.6min speed=80354K/sec

md127 : active raid10 sdc[1] sdb[0]
      1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
      [>....................]  resync =  1.1% (22630912/1953382400)
finish=617.9min speed=52073K/sec

unused devices: <none>

create commands:
  105  sudo mdadm --verbose --create md0 --level=10 --layout=f2
--raid-devices=2 /dev/sdb /dev/sdc
  106  sudo mdadm --verbose --create md1 --level=10 --layout=f2
--raid-devices=2 /dev/sdd /dev/sde
  107  sudo mdadm --verbose --create md2 --level=10 --layout=f2
--raid-devices=2 /dev/sdf /dev/sdg
  111  sudo mdadm --verbose --create md3 --level=10 --layout=f2
--raid-devices=3 /dev/md/md0 /dev/md/md1 /dev/md/md2
  118  sudo mdadm --stop /dev/md124 # to stop seek of death until the
above finishes resyncing

I read that the far layout gives faster read performance compared to
standard RAID1 in a 2 HDD array, so that's why I used it. I then
created yet another RAID10 f2 layout of the 3x 2 disk RAID10 volumes,
giving me /dev/md124. Filesystem doesn't matter, I need something
that's stable with good performance over the RAID set.
I just realized that out of 12 TB raw space I now only have 3TB
available, heh. So I suppose I'm going for a different layout.

Any opinions?

Cheers,
Mathias

Re: What layout with 6x2TB, Intel Atom?

[Mathias Burén]

On 4 February 2013 20:27, Mathias Burén <mathias.buren@gmail.com> wrote:
> Hi list,
>
> I've an Intel Atom 330 running 32-bit Ubuntu, ~3.5GB RAM. The intent
> is to store large media files (>4GB) for streaming by maximum 3
> clients at any time. Very read heavy.
> 6x 2TB HDDs are available, the crux is that 3 HDDs are on the on-board
> SATA controller, the other 3 are on a Marvel PCI-E adapter
>
> cat /proc/mdstat
>
> Personalities : [raid10]
> md125 : active raid10 sdg[1] sdf[0]
>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>       [>....................]  resync =  1.1% (22074880/1953382400)
> finish=642.2min speed=50114K/sec
>
> md126 : active raid10 sde[1] sdd[0]
>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>       [>....................]  resync =  1.6% (31473408/1953382400)
> finish=398.6min speed=80354K/sec
>
> md127 : active raid10 sdc[1] sdb[0]
>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>       [>....................]  resync =  1.1% (22630912/1953382400)
> finish=617.9min speed=52073K/sec
>
> unused devices: <none>
>
> create commands:
>   105  sudo mdadm --verbose --create md0 --level=10 --layout=f2
> --raid-devices=2 /dev/sdb /dev/sdc
>   106  sudo mdadm --verbose --create md1 --level=10 --layout=f2
> --raid-devices=2 /dev/sdd /dev/sde
>   107  sudo mdadm --verbose --create md2 --level=10 --layout=f2
> --raid-devices=2 /dev/sdf /dev/sdg
>   111  sudo mdadm --verbose --create md3 --level=10 --layout=f2
> --raid-devices=3 /dev/md/md0 /dev/md/md1 /dev/md/md2
>   118  sudo mdadm --stop /dev/md124 # to stop seek of death until the
> above finishes resyncing
>
> I read that the far layout gives faster read performance compared to
> standard RAID1 in a 2 HDD array, so that's why I used it. I then
> created yet another RAID10 f2 layout of the 3x 2 disk RAID10 volumes,
> giving me /dev/md124. Filesystem doesn't matter, I need something
> that's stable with good performance over the RAID set.
> I just realized that out of 12 TB raw space I now only have 3TB
> available, heh. So I suppose I'm going for a different layout.
>
> Any opinions?
>
> Cheers,
> Mathias


I obviously meant to create a RAID0, not a RAID10, over the 3 RAID10 devices.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What layout with 6x2TB, Intel Atom?

[Mathias Burén]

On 4 February 2013 20:29, Mathias Burén <mathias.buren@gmail.com> wrote:
> On 4 February 2013 20:27, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Hi list,
>>
>> I've an Intel Atom 330 running 32-bit Ubuntu, ~3.5GB RAM. The intent
>> is to store large media files (>4GB) for streaming by maximum 3
>> clients at any time. Very read heavy.
>> 6x 2TB HDDs are available, the crux is that 3 HDDs are on the on-board
>> SATA controller, the other 3 are on a Marvel PCI-E adapter
>>

I was clearly not thinking when doing this. Now it's a 6 disk RAID10
with far layout, 2 copies. Is the default chunk size of 512 a good
option here?

$ sudo mdadm --verbose --create md0 --level=10 --layout=f2
--raid-devices=6 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
mdadm: chunk size defaults to 512K
mdadm: size set to 1953382912K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md0 started.


$ cat /proc/mdstat
Personalities : [raid10] [raid0]
md127 : active raid10 sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]
      5860147200 blocks super 1.2 512K chunks 2 far-copies [6/6] [UUUUUU]
      [>....................]  resync =  0.0% (775616/5860147200)
finish=629.5min speed=155123K/sec

unused devices: <none>
@ion:~$ sudo mdadm -D /dev/md/md0
/dev/md/md0:
        Version : 1.2
  Creation Time : Mon Feb  4 20:40:22 2013
     Raid Level : raid10
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

    Update Time : Mon Feb  4 20:40:22 2013
          State : clean, resyncing
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0

         Layout : far=2
     Chunk Size : 512K

  Resync Status : 0% complete

           Name : ion:md0  (local to host ion)
           UUID : 438e9a1a:bf840b03:05c36383:c965cb39
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       48        2      active sync   /dev/sdd
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       8       96        5      active sync   /dev/sdg

Kind regards,
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What layout with 6x2TB, Intel Atom?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Mon, 4 Feb 2013 20:27:42 +0000
Mathias Burén <mathias.buren@gmail.com> wrote:

> I read that the far layout gives faster read performance compared to
> standard RAID1 in a 2 HDD array, so that's why I used it. I then
> created yet another RAID10 f2 layout of the 3x 2 disk RAID10 volumes,
> giving me /dev/md124. Filesystem doesn't matter, I need something
> that's stable with good performance over the RAID set.
> I just realized that out of 12 TB raw space I now only have 3TB
> available, heh. So I suppose I'm going for a different layout.
> 
> Any opinions?

Hello,

Why won't you even consider RAID6? To me it seems like the perfect choice,
given the details you described (mostly reads, large files, few clients).
And the disk count is perfect, not too few and not too many, so to say :D

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: What layout with 6x2TB, Intel Atom?

[Steven Haigh]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On 5/02/2013 8:12 AM, Roman Mamedov wrote:
> On Mon, 4 Feb 2013 20:27:42 +0000
> Mathias Burén <mathias.buren@gmail.com> wrote:
>
>> I read that the far layout gives faster read performance compared to
>> standard RAID1 in a 2 HDD array, so that's why I used it. I then
>> created yet another RAID10 f2 layout of the 3x 2 disk RAID10 volumes,
>> giving me /dev/md124. Filesystem doesn't matter, I need something
>> that's stable with good performance over the RAID set.
>> I just realized that out of 12 TB raw space I now only have 3TB
>> available, heh. So I suppose I'm going for a different layout.
>>
>> Any opinions?
>
> Hello,
>
> Why won't you even consider RAID6? To me it seems like the perfect choice,
> given the details you described (mostly reads, large files, few clients).
> And the disk count is perfect, not too few and not too many, so to say :D
>

I was going to say the same. I run a 4 x 2Tb SATA RAID6. The read speeds 
will *easily* saturate GigE. Random seeks slow things down quite a bit - 
but you'll get that with any setup except SSD.

If its mainly sequential reads, then RAID6 will be fine.

And as I know someone will ask... Why 4 drives in RAID6? Firstly, you 
still get the redundancy of surviving 2 drive failures - but also when I 
start to run low on space I can easily add another 2Tb disk and reshape 
to 5 drives, then 6. My chassis has 8 x SATA hot swap bays - 2 x 80Gb 
RAID1 leaving the other 6 for the RAID6. By the time I fill up 6 x 2Tb 
drives, there will be other cheap multi-Tb drives to use.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4240 bytes --]

Re: What layout with 6x2TB, Intel Atom?

[Stan Hoeppner]

Forgot to reply-all.

On 2/4/2013 2:29 PM, Mathias Burén wrote:
> On 4 February 2013 20:27, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Hi list,
>>
>> I've an Intel Atom 330 running 32-bit Ubuntu, ~3.5GB RAM. The intent
>> is to store large media files (>4GB) for streaming by maximum 3
>> clients at any time. Very read heavy.
>> 6x 2TB HDDs are available, the crux is that 3 HDDs are on the on-board
>> SATA controller, the other 3 are on a Marvel PCI-E adapter
>>
>> cat /proc/mdstat
>>
>> Personalities : [raid10]
>> md125 : active raid10 sdg[1] sdf[0]
>>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>>       [>....................]  resync =  1.1% (22074880/1953382400)
>> finish=642.2min speed=50114K/sec
>>
>> md126 : active raid10 sde[1] sdd[0]
>>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>>       [>....................]  resync =  1.6% (31473408/1953382400)
>> finish=398.6min speed=80354K/sec
>>
>> md127 : active raid10 sdc[1] sdb[0]
>>       1953382400 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
>>       [>....................]  resync =  1.1% (22630912/1953382400)
>> finish=617.9min speed=52073K/sec
>>
>> unused devices: <none>
>>
>> create commands:
>>   105  sudo mdadm --verbose --create md0 --level=10 --layout=f2
>> --raid-devices=2 /dev/sdb /dev/sdc
>>   106  sudo mdadm --verbose --create md1 --level=10 --layout=f2
>> --raid-devices=2 /dev/sdd /dev/sde
>>   107  sudo mdadm --verbose --create md2 --level=10 --layout=f2
>> --raid-devices=2 /dev/sdf /dev/sdg
>>   111  sudo mdadm --verbose --create md3 --level=10 --layout=f2
>> --raid-devices=3 /dev/md/md0 /dev/md/md1 /dev/md/md2
>>   118  sudo mdadm --stop /dev/md124 # to stop seek of death until the
>> above finishes resyncing
>>
>> I read that the far layout gives faster read performance compared to
>> standard RAID1 in a 2 HDD array, so that's why I used it. I then
>> created yet another RAID10 f2 layout of the 3x 2 disk RAID10 volumes,
>> giving me /dev/md124. Filesystem doesn't matter, I need something
>> that's stable with good performance over the RAID set.
>> I just realized that out of 12 TB raw space I now only have 3TB
>> available, heh. So I suppose I'm going for a different layout.
>>
>> Any opinions?

First, the Atom 330 is a 64 bit dual core chip.  Use a 64 bit kernel
(preferably 3.0 or later, 2.6.39 minimum) and user space, unless you
have an old 32 bit app that won't run.  The 64 bit kernel will give
better performance for an IO server, yes, even if you have less than 4GB
of RAM.

Second, given your workload description, it is a good match for RAID6,
which will give you 8TB capacity.  Given the slow rotational speed of
those disks the default 512KB chunk should be fine as it will tend to
minimize seeks.  With those drives head movement is far more expensive
that rotation.

Third, use XFS, and format with alignment to the md device:
~$ mkfs.xfs -d su=512k,sw=4

Fourth, even if your Marvell SATA card is PCIe x1 1.0 and limited to
250MB/s, 500MB/s duplex, that shouldn't cause problems in daily use,
though it may very well slow down RAID operations such as rebuilds and
reshapes.  If it's a 2.0 card operating in 2.0 mode, it's 500/1000 MB/s,
exceeding the bandwidth of the 3 drives, and is ample.

-- 
Stan

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What layout with 6x2TB, Intel Atom?

[Roy Sigurd Karlsbakk]

> Third, use XFS, and format with alignment to the md device:
> ~$ mkfs.xfs -d su=512k,sw=4

Why? ext4 should be just as good for that, and better for metadata

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What layout with 6x2TB, Intel Atom?

[Roy Sigurd Karlsbakk]

> Fourth, even if your Marvell SATA card is PCIe x1 1.0 and limited to
> 250MB/s, 500MB/s duplex, that shouldn't cause problems in daily use,
> though it may very well slow down RAID operations such as rebuilds and
> reshapes. If it's a 2.0 card operating in 2.0 mode, it's 500/1000
> MB/s, exceeding the bandwidth of the 3 drives, and is ample.

250MB/s duplex or not. A 1Gbps ethernet link is 1Gbps either its duplex or not.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What layout with 6x2TB, Intel Atom?

[Stan Hoeppner]

On 2/4/2013 6:01 PM, Roy Sigurd Karlsbakk wrote:
>> Third, use XFS, and format with alignment to the md device:
>> ~$ mkfs.xfs -d su=512k,sw=4
> 
> Why? ext4 should be just as good for that, 

XFS generally yields better performance with concurrency, and is simply
a better filesystem across the board WRT performance, features, and
tools.  And XFS has been designed and optimized since 1994 specifically
for storing and streaming large files, making it a perfect fit for the
OP's 4GB files.

Why use "just as good" when one can use the best?

> and better for metadata

XFS metadata performance has exceeded that of EXT4 since mainline kernel
2.6.39 (May 2011) for workloads with medium to large concurrency (2-4
threads or more).  It's single thread metadata performance is now not
far behind EXT4.  Thus journal metadata performance is no longer a
reason to avoid XFS.

Regardless, metadata performance is irrelevant to the OP's stated workload.

-- 
Stan