Remote NAS

Remote NAS

[adfas asd]

A couple questions about MD RAID10:

Right now I'm RAIDing two 2TB drives in RAID10offset2. Need to add some drives and maybe I'll remote them using NAS (GB ethernet), for fire and theft protection.
 
So could the BIOS actually boot this if 2 drives are in the machine and 2 are in the garage? Is there a way to specify in mdadm that the two in the garage are mirrored from the two in the system?

Is GB ethernet NAS noticably slower than eSATA?  Is it reliable?
 
What if I use the mobo's e-SATA port to add 5 external drives locally. This would require a port multiplier. I presume it would not be bootable, since a port multiplier needs OS support? Or would something go in initrd.img?  Or would I need an individual boot drive to get things up?
 
Using the array to store MythTV recordings (very large files), but when it does commercial flagging (lots of writes) the array is so busy that GUI response is very slow.  I've set readahead cache to optimal (4096, after testing most sizes), so read should be OK.  I know far is optimized for read, so I chose offset in hopes it would bring up write speed.  Is there a way to speed up writes?



      

Re: Remote NAS

[adfas asd]

No one has any idea on my questions?


--- On Wed, 9/16/09, adfas asd <chimera_god@yahoo.com> wrote:

> From: adfas asd <chimera_god@yahoo.com>
> Subject: Remote NAS
> To: linux-raid@vger.kernel.org
> Date: Wednesday, September 16, 2009, 9:21 AM
> A couple questions about MD RAID10:
> 
> Right now I'm RAIDing two 2TB drives in RAID10offset2. Need
> to add some drives and maybe I'll remote them using NAS (GB
> ethernet), for fire and theft protection.
>  
> So could the BIOS actually boot this if 2 drives are in the
> machine and 2 are in the garage? Is there a way to specify
> in mdadm that the two in the garage are mirrored from the
> two in the system?
> 
> Is GB ethernet NAS noticably slower than eSATA?  Is it
> reliable?
>  
> What if I use the mobo's e-SATA port to add 5 external
> drives locally. This would require a port multiplier. I
> presume it would not be bootable, since a port multiplier
> needs OS support? Or would something go in initrd.img? 
> Or would I need an individual boot drive to get things up?
>  
> Using the array to store MythTV recordings (very large
> files), but when it does commercial flagging (lots of
> writes) the array is so busy that GUI response is very
> slow.  I've set readahead cache to optimal (4096, after
> testing most sizes), so read should be OK.  I know far
> is optimized for read, so I chose offset in hopes it would
> bring up write speed.  Is there a way to speed up
> writes?
> 
> 
> 
>       
> 


      

Re: Remote NAS

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]

On Tue Sep 22, 2009 at 12:38:37PM -0700, adfas asd wrote:

> > Right now I'm RAIDing two 2TB drives in RAID10offset2. Need
> > to add some drives and maybe I'll remote them using NAS (GB
> > ethernet), for fire and theft protection.
> >  
> > So could the BIOS actually boot this if 2 drives are in the
> > machine and 2 are in the garage? Is there a way to specify
> > in mdadm that the two in the garage are mirrored from the
> > two in the system?
> > 
The BIOS boots from a single drive, and won't boot from RAID10, so
presumably you already have a non-RAID (or RAID-1) boot partition.
Certainly I don't see why mdadm can't start up an array set up like this
though.  As for setting up how they're mirrored, it sounds like you want
a RAID-1 mirror over the top of a pair of RAID10 arrays.

> > Is GB ethernet NAS noticably slower than eSATA?  Is it
> > reliable?
> >  
It's technically slower, certainly - eSATA could be 3Gb compared to the
1Gb on ethernet.  On top of that you'll have added latency, so I'd be
surprised if you didn't notice the speed difference.  As for reliability
- I don't see why it shouldn't be (providing you're using decent
cabling).

> > What if I use the mobo's e-SATA port to add 5 external
> > drives locally. This would require a port multiplier. I
> > presume it would not be bootable, since a port multiplier
> > needs OS support? Or would something go in initrd.img? 
> > Or would I need an individual boot drive to get things up?
> >  
Depends on exactly how the BIOS sees is - some seem to see a single
drive off the multiplier, so can boot off that one, whereas others won't
see anything.  The initrd question is immaterial, as this is only read
after the BIOS has called the boot loader, but certainly you could load
any necessary drivers at that point.

> > Using the array to store MythTV recordings (very large
> > files), but when it does commercial flagging (lots of
> > writes) the array is so busy that GUI response is very
> > slow.  I've set readahead cache to optimal (4096, after
> > testing most sizes), so read should be OK.  I know far
> > is optimized for read, so I chose offset in hopes it would
> > bring up write speed.  Is there a way to speed up
> > writes?
> > 
I'm not sure offset is really noticeably faster than far for writes.
Commercial flagging shouldn't be writing to the recordings drive though
- it's just inserting database metadata.  If your MythTV database is on
the same array/drives then you'll certainly get a big gain from moving
it to a separate disk.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Remote NAS

[adfas asd]

Thanks Robin.

--- On Tue, 9/22/09, Robin Hill <robin@robinhill.me.uk> wrote:
> The BIOS boots from a single drive, and won't boot from
> RAID10, so
> presumably you already have a non-RAID (or RAID-1) boot
> partition.

I have my only two drives set up as RAID10offset2 (WD 2TB each), and it boots just fine for some reason.


> > > What if I use the mobo's e-SATA port to add 5
> external
> > > drives locally. This would require a port
> multiplier. I
> > > presume it would not be bootable, since a port
> multiplier
> > > needs OS support? Or would something go in
> initrd.img? 
> > > Or would I need an individual boot drive to get
> things up?
> > >  
> Depends on exactly how the BIOS sees is - some seem to see
> a single
> drive off the multiplier, so can boot off that one, whereas
> others won't
> see anything.  The initrd question is immaterial, as
> this is only read
> after the BIOS has called the boot loader, but certainly
> you could load
> any necessary drivers at that point.

It's an Asus P5N7A-VM mobo, and of course Asus doesn't know the answer to this question.  I have in mind the Addonics port "software" multiplier, which is a pure hardware multiplexer.  (AD5SARPM-E with eSATA & Sii3726 chip)

I guess the answer is to just buy and try it.  Reluctant to though, owing to the expense.  I am surprised no one else has tried this.

 
> I'm not sure offset is really noticeably faster than far
> for writes.
> Commercial flagging shouldn't be writing to the recordings
> drive though
> - it's just inserting database metadata.  If your
> MythTV database is on
> the same array/drives then you'll certainly get a big gain
> from moving
> it to a separate disk.

Commercial flagging is writing like hell to the array, according to iotop.  I don't know how else to assess what's going on.  The drives are constantly hammering, and my Myth menu moves get very slow during flagging, even with the latest 2TB disk drives and RAID10.




      

Re: Remote NAS

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 3477 bytes --]

On Tue Sep 22, 2009 at 02:56:08PM -0700, adfas asd wrote:

> Thanks Robin.
> 
> --- On Tue, 9/22/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > The BIOS boots from a single drive, and won't boot from RAID10, so
> > presumably you already have a non-RAID (or RAID-1) boot partition.
> 
> I have my only two drives set up as RAID10offset2 (WD 2TB each), and
> it boots just fine for some reason.
> 
Yes, with a 2-drive RAID10o2 layout, one of the drives contains all
blocks in normal order, the other doesn't (see
http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10)
so if the one drive fails then your system will be non-bootable.  A
RAID10n2 or RAID1 (the layout's the same) would be a better choice.

> > > > What if I use the mobo's e-SATA port to add 5 external
> > > > drives locally. This would require a port multiplier. I
> > > > presume it would not be bootable, since a port multiplier
> > > > needs OS support? Or would something go in initrd.img? 
> > > > Or would I need an individual boot drive to get things up?
> > > >  
> > Depends on exactly how the BIOS sees is - some seem to see a single
> > drive off the multiplier, so can boot off that one, whereas others
> > won't see anything.  The initrd question is immaterial, as
> > this is only read after the BIOS has called the boot loader, but
> > certainly you could load any necessary drivers at that point.
> 
> It's an Asus P5N7A-VM mobo, and of course Asus doesn't know the answer
> to this question.  I have in mind the Addonics port "software"
> multiplier, which is a pure hardware multiplexer.  (AD5SARPM-E with
> eSATA & Sii3726 chip)
> 
> I guess the answer is to just buy and try it.  Reluctant to though,
> owing to the expense.  I am surprised no one else has tried this.
> 
I certainly haven't anyway.  Is there any reason you can't leave a
couple of drives internal though?  You could definitely boot from those
then.
>  
> > I'm not sure offset is really noticeably faster than far for
> > writes. Commercial flagging shouldn't be writing to the recordings
> > drive though - it's just inserting database metadata.  If your
> > MythTV database is on the same array/drives then you'll certainly
> > get a big gain from moving it to a separate disk.
> 
> Commercial flagging is writing like hell to the array, according to
> iotop.  I don't know how else to assess what's going on.  The drives
> are constantly hammering, and my Myth menu moves get very slow during
> flagging, even with the latest 2TB disk drives and RAID10.
> 
Are you transcoding as well?  The flagging itself shouldn't write, but
if you're transcoding as well (to cut the commercials) then it will have
to write.  And slow menu moves would suggest database/CPU holdups -
there should be no need to read from the recordings disk (actually, it
may have to check the file exists, but that's it).  You didn't say
whether your database is on the array or not (if you're booting from the
array then I'd guess so though).

If your aim is to improve performance then I'd recommend using a single
(relatively small) drive for the main OS and database, and then one (or
more) drives for the recordings.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Remote NAS

[Keld Jørn Simonsen]

On Wed, Sep 23, 2009 at 08:57:10AM +0100, Robin Hill wrote:
> On Tue Sep 22, 2009 at 02:56:08PM -0700, adfas asd wrote:
> 
> > Thanks Robin.
> > 
> > --- On Tue, 9/22/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > > The BIOS boots from a single drive, and won't boot from RAID10, so
> > > presumably you already have a non-RAID (or RAID-1) boot partition.
> > 
> > I have my only two drives set up as RAID10offset2 (WD 2TB each), and
> > it boots just fine for some reason.
> > 
> Yes, with a 2-drive RAID10o2 layout, one of the drives contains all
> blocks in normal order, the other doesn't (see
> http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10)
> so if the one drive fails then your system will be non-bootable.  A
> RAID10n2 or RAID1 (the layout's the same) would be a better choice.

There is a setup described at 

http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk

You can substitute raid10,o2 for raid10,f2 for the root partitions etc.

Anyway, raid10,f2 should be faster than raid10,o2, for at least reads,
while for writes it is about the same performance given that you employ
a file system.

Best regards
keld

Re: Remote NAS

[adfas asd]

--- On Wed, 9/23/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > Commercial flagging is writing like hell to the array,
> according to
> > iotop.  I don't know how else to assess what's
> going on.  The drives
> > are constantly hammering, and my Myth menu moves get
> very slow during
> > flagging, even with the latest 2TB disk drives and
> RAID10.
> > 
> Are you transcoding as well?  The flagging itself
> shouldn't write, but
> if you're transcoding as well (to cut the commercials) then
> it will have
> to write.  And slow menu moves would suggest
> database/CPU holdups -
> there should be no need to read from the recordings disk
> (actually, it
> may have to check the file exists, but that's it). 
> You didn't say
> whether your database is on the array or not (if you're
> booting from the
> array then I'd guess so though).

Not transcoding at the same time, just the standard commercial flagging.  The drive light is constantly on.  CPU is pretty busy (3GHz) reaching 180% when simultaneously flagging 2 videos.

The database in in the / partition of the array, and the videos are in the /home partition on it.  No idea why the array is so busy.


> If your aim is to improve performance then I'd recommend
> using a single
> (relatively small) drive for the main OS and database, and
> then one (or
> more) drives for the recordings.

My goal is to have two large drives in the HTPC and two out in the garage NASed, in RAID10 so that if my HTPC is stolen or there's a fire I'll still have my data.  But no one here seems to know how to specify the HTPC drives as one side of the mirror and the garage drives as the other.  

Needless to say I'd like performance to be good, but that's starting to look hopeless.



      

Re: Remote NAS

[adfas asd]

--- On Wed, 9/23/09, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> There is a setup described at 
> 
> http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
> 
> You can substitute raid10,o2 for raid10,f2 for the root
> partitions etc.
> 
> Anyway, raid10,f2 should be faster than raid10,o2, for at
> least reads,
> while for writes it is about the same performance given
> that you employ
> a file system.

Thanks keld.  I used this procedure, which does about the same thing and seems more clear, adapting it to RAID10-o2:
http://www.howtoforge.com/software-raid1-grub-boot-debian-etch

I could find no performance comparison with far and offset, and knew that far improved on reads, so since offset came later inferred that it would have improved writes.  No objective data to confirm otherwise.




      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Remote NAS

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

On Wed Sep 23, 2009 at 07:52:59AM -0700, adfas asd wrote:

> 
> --- On Wed, 9/23/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > > Commercial flagging is writing like hell to the array,
> > according to
> > > iotop.  I don't know how else to assess what's
> > going on.  The drives
> > > are constantly hammering, and my Myth menu moves get
> > very slow during
> > > flagging, even with the latest 2TB disk drives and
> > RAID10.
> > > 
> > Are you transcoding as well?  The flagging itself
> > shouldn't write, but
> > if you're transcoding as well (to cut the commercials) then
> > it will have
> > to write.  And slow menu moves would suggest
> > database/CPU holdups -
> > there should be no need to read from the recordings disk
> > (actually, it
> > may have to check the file exists, but that's it). 
> > You didn't say
> > whether your database is on the array or not (if you're
> > booting from the
> > array then I'd guess so though).
> 
> Not transcoding at the same time, just the standard commercial
> flagging.  The drive light is constantly on.  CPU is pretty busy
> (3GHz) reaching 180% when simultaneously flagging 2 videos.
> 
> The database in in the / partition of the array, and the videos are in
> the /home partition on it.  No idea why the array is so busy.
> 
If they're both on the array, then that's the problem.  The partitioning
doesn't make difference to the issue - the drive heads are having to
seek frantically between reading the data file and writing the database.

> 
> > If your aim is to improve performance then I'd recommend
> > using a single
> > (relatively small) drive for the main OS and database, and
> > then one (or
> > more) drives for the recordings.
> 
> My goal is to have two large drives in the HTPC and two out in the
> garage NASed, in RAID10 so that if my HTPC is stolen or there's a fire
> I'll still have my data.  But no one here seems to know how to specify
> the HTPC drives as one side of the mirror and the garage drives as the
> other.  
> 
You have two drives in RAID10 currently, and you want to go to four.
Are you meaning to increase the capacity as well or do you just want the
two remote drives to be a mirror of your current two drives?

> Needless to say I'd like performance to be good, but that's starting
> to look hopeless.
> 
I'd recommend getting two smaller drives, and having two separate
mirrored pairs - the smaller array for the OS and database and the
larger one for recording.  Keep one of each size within the HTPC and one
of each in the garage.  You can also use the write-mostly option to make
the kernel read from the local drives where possible.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Remote NAS

[adfas asd]

--- On Wed, 9/23/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > My goal is to have two large drives in the HTPC and
> two out in the
> > garage NASed, in RAID10 so that if my HTPC is stolen
> or there's a fire
> > I'll still have my data.  But no one here seems
> to know how to specify
> > the HTPC drives as one side of the mirror and the
> garage drives as the
> > other.  
> > 
> You have two drives in RAID10 currently, and you want to go
> to four.
> Are you meaning to increase the capacity as well or do you
> just want the
> two remote drives to be a mirror of your current two
> drives?

Both more capacity, and remote storage.


> > Needless to say I'd like performance to be good, but
> that's starting
> > to look hopeless.
> > 
> I'd recommend getting two smaller drives, and having two
> separate
> mirrored pairs - the smaller array for the OS and database
> and the
> larger one for recording.  Keep one of each size
> within the HTPC and one
> of each in the garage.  You can also use the
> write-mostly option to make
> the kernel read from the local drives where possible.

But what I'm saying there doesn't seem to be a way in mdadm to specify --one side-- of the mirror here, and the other side, there.

Thanks for the tip on write-mostly.



      

Re: Remote NAS

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On Wed Sep 23, 2009 at 08:44:41AM -0700, adfas asd wrote:

> --- On Wed, 9/23/09, Robin Hill <robin@robinhill.me.uk> wrote:
> > > My goal is to have two large drives in the HTPC and
> > two out in the
> > > garage NASed, in RAID10 so that if my HTPC is stolen
> > or there's a fire
> > > I'll still have my data.  But no one here seems
> > to know how to specify
> > > the HTPC drives as one side of the mirror and the
> > garage drives as the
> > > other.  
> > > 
> > You have two drives in RAID10 currently, and you want to go
> > to four.
> > Are you meaning to increase the capacity as well or do you
> > just want the
> > two remote drives to be a mirror of your current two
> > drives?
> 
> Both more capacity, and remote storage.
> 
Okay, so you want the local drives to be (effectively) a RAID-0, and
then mirrored to the remote drives?

> 
> > > Needless to say I'd like performance to be good, but
> > that's starting
> > > to look hopeless.
> > > 
> > I'd recommend getting two smaller drives, and having two
> > separate
> > mirrored pairs - the smaller array for the OS and database
> > and the
> > larger one for recording.  Keep one of each size
> > within the HTPC and one
> > of each in the garage.  You can also use the
> > write-mostly option to make
> > the kernel read from the local drives where possible.
> 
> But what I'm saying there doesn't seem to be a way in mdadm to specify
> --one side-- of the mirror here, and the other side, there.
> 
I've no idea what you mean here.  You specify two block devices for the
mirror - what difference does it make where (or what) they are?  And if
you want RAID10 then (in the create order) drives 0 and 2 are local and
1 and 3 remote.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Remote NAS

[Keld Jørn Simonsen]

On Wed, Sep 23, 2009 at 07:59:09AM -0700, adfas asd wrote:
> --- On Wed, 9/23/09, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> > There is a setup described at 
> > 
> > http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
> > 
> > You can substitute raid10,o2 for raid10,f2 for the root
> > partitions etc.
> > 
> > Anyway, raid10,f2 should be faster than raid10,o2, for at
> > least reads,
> > while for writes it is about the same performance given
> > that you employ
> > a file system.
> 
> Thanks keld.  I used this procedure, which does about the same thing and seems more clear, adapting it to RAID10-o2:
> http://www.howtoforge.com/software-raid1-grub-boot-debian-etch
> 
> I could find no performance comparison with far and offset, and knew that far improved on reads, so since offset came later inferred that it would have improved writes.  No objective data to confirm otherwise.

There are a number of performance tests at:
http://linux-raid.osdl.org/index.php/Performance

Writes are about the same for RAID10 types and RAID1.
This is also what theory would tell for random writes, given that they
are random, and that the elevator algorithm of the file system optimizes
the writing. 

Yes, offset came after far, but Neil Brown said that offset was
implemented to make Linux MD align to RAID standards. It was not
necessarily meant to be better than far. 

I think it is hard to beat raid10 far on reads (but then I am also the
one that invented the layout). As said for random reading and writing
the theory says that all mirrored raid types perform equally, and tests
verify this. Maybe far layout even has an edge for random reading, in
theory it should, but I have not seen tests really verifying that.

best regards
Keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Remote NAS

[adfas asd]

--- On Wed, 9/23/09, Keld Jørn Simonsen <keld@dkuug.dk> wrote:
> Writes are about the same for RAID10 types and RAID1.
> This is also what theory would tell for random writes,
> given that they
> are random, and that the elevator algorithm of the file
> system optimizes
> the writing. 
> 
> Yes, offset came after far, but Neil Brown said that offset
> was
> implemented to make Linux MD align to RAID standards. It
> was not
> necessarily meant to be better than far. 
> 
> I think it is hard to beat raid10 far on reads (but then I
> am also the
> one that invented the layout). As said for random reading
> and writing
> the theory says that all mirrored raid types perform
> equally, and tests
> verify this. Maybe far layout even has an edge for random
> reading, in
> theory it should, but I have not seen tests really
> verifying that.

Excellent, thanks Keld.




      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Remote NAS

[adfas asd]

--- On Wed, 9/23/09, Robin Hill <robin@robinhill.me.uk> wrote:
> Obviously, but in the case of a RAID1 mirror, then this is
> immaterial
> (as either side of the mirror is identical), and in the
> RAID10 case it
> is controlled by the position of the drives in the array
> (which is
> controlled by the order they're given in the create
> statement).
> 
> > When I set up my NAS it's likely those remote drives
> will be sdc and
> > sdd (HTPC drives would be sda and sdb), so the RAID10
> create order
> > wouldn't help.
> > 
> Of course the create order helps - it's the _only_ way to
> control what
> chunks of the data go on which drive.  If your HTPC
> ends up with local
> drives sda and sdb, and remote drives sdc and sdd, then
> do:
> 
>     mdadm -C /dev/md0 -l 10 -p n2 -n 4 /dev/sda
> /dev/sdc /dev/sdb /dev/sdd
> 
> Then you'll end up with a RAID10 with all data held on both
> the sda/sdb
> and sdc/sdd pairs (so either pair can drop out of the array
> without
> losing any data).  The data layout will be:
> 
>     sda  sdb  sdc  sdd
>    
> A1   A2   A1   A2
>    
> A3   A4   A3   A4
>     ...
> 
> The equivalent happens for f2 or o2 arrays - though they're
> not strictly
> mirrored in those cases (they contain the same data but
> with a differing
> layout).
> 
> As I say though, this setup will still result in poor
> performance as you
> have the database and recordings on the same drives. 
> And to make them
> bootable you'd want to partition the drives first and
> create a separate
> 4-way RAID1 mirror for /boot.

YIKES, I'd better read this when I'm -not- drunk...

I take away that my main performance problem is in not separating by drive, the Myth database from the affected video.  Given these listserv transactions, I guess I have to resort to a 2.5" drive with the root filesystem and the Myth database, and the space-divided array for video storage.  I guess I can't RAID10 the root drive with the garage, as it might not be bootable, and you probably can't boot to a NAS drive anyway.   But this doesn't stop me from backing up the root drive onto the array.

It seems that I -can- control the role of the drives using your command above.  This is good news.

I will reexamine this tomorrow... 


      

RE: Remote NAS

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of adfas asd
> Sent: Wednesday, September 16, 2009 11:22 AM
> To: linux-raid@vger.kernel.org
> Subject: Remote NAS
> 
> A couple questions about MD RAID10:
> 
> Right now I'm RAIDing two 2TB drives in RAID10offset2. Need to add some
> drives and maybe I'll remote them using NAS (GB ethernet), for fire and
> theft protection.
> 
> So could the BIOS actually boot this if 2 drives are in the machine and 2
> are in the garage? Is there a way to specify in mdadm that the two in the
> garage are mirrored from the two in the system?
> 
> Is GB ethernet NAS noticably slower than eSATA?  Is it reliable?
> 
> What if I use the mobo's e-SATA port to add 5 external drives locally.
> This would require a port multiplier. I presume it would not be bootable,
> since a port multiplier needs OS support? Or would something go in
> initrd.img?  Or would I need an individual boot drive to get things up?
> 
> Using the array to store MythTV recordings (very large files), but when it
> does commercial flagging (lots of writes) the array is so busy that GUI
> response is very slow.  I've set readahead cache to optimal (4096, after
> testing most sizes), so read should be OK.  I know far is optimized for
> read, so I chose offset in hopes it would bring up write speed.  Is there
> a way to speed up writes?

	I'm not running MythTv, but I faced similar challenges when building
my video server.  RAID10 and similar solutions are great for many purposes,
but there are some fundamental considerations when choosing an array setup
in this situation which I think you should review carefully.

	1. Clearly you are concerned about a loss of data should something
catastrophic happen to one site or the other.  RAID mirroring certainly
addresses such a concern.  However, no matter what the array topology, RAID
systems are not fault free.  They are fault tolerant.  Of course, no system
is perfectly fault free, but a RAID solution only addresses hardware
failures.  By far the most common cause of data loss is not hardware
failure, but human error.  A RAID system cannot prevent a user from
accidentally overwriting data on an array.  Maddeningly, I lost several
favorite programs on separate occasions due to such errors before I had a
good backup solution in place.  I have only lost 1 program as a result of
array issues - that after a RAID growth which went sour unknowingly and a
subsequent write was corrupted.  The point I am making here is to prevent
data loss, one really needs a backup solution in addition to whatever array
solution one chooses.  You haven't mentioned a backup method (although
perhaps you have one), so perhaps a better solution than mirroring on a NAS
with the write mostly option would be to create a backup system in your
garage and run an rsync on a regular basis in order to assure data
retention.  That way, you have some period of time after having made a nasty
error to easily correct it before it becomes etched in stone.  This also
circumvents any performance issues associated with more complex RAID
solutions.

	2.  The nature of your data is a consideration.  As others have
mentioned, you might want to break up your partition structure to
accommodate the data types.  In particular, video data is huge, but does not
need to be read or written very quickly.  MPEG-II data only streams
typically at about 10 - 15 Mbps for 1080i, or at most 40 Mbps for 1080p
video on average per stream.  MPEG-IV streams even more slowly.  Database
writes and reads, however, are much, much smaller but can potentially fill a
3000 Mbps pipe.  I really agree with others here that you need to carefully
consider a different type of partition - perhaps not a RAID array at all -
for your database.  I run a tarball task every day to backup all the
important system data which is not on the RAID array over to a file on the
array, which in turn gets backed up every morning at 04:00 via rsync to the
remote backup system.  The video server employs a RAID6 array, and the
backup a RAID5 array.  Both systems boot off small drives whose contents are
copied to cold drives sitting on a shelf in case of boot drive failure.  In
your case, I think I would personally employ the small boot drive plus an
extra drive - perhaps a high performance 120G to 250G drive - just for your
database.  If you really feel the need, you could mirror the database drive,
but unless I am wrong, I think in your case the database can be easily
rebuilt if it is reverted to an older copy, so I would think a daily rsync
to your array would work well for you also.  The drives for the arrays can
be very inexpensive low performance drives with as little power consumption
as possible.

RE: Remote NAS

[Leslie Rhorer]

> Using the array to store MythTV recordings (very large files), but when it
> does commercial flagging (lots of writes) the array is so busy that GUI
> response is very slow.  I've set readahead cache to optimal (4096, after
> testing most sizes), so read should be OK.  I know far is optimized for
> read, so I chose offset in hopes it would bring up write speed.  Is there
> a way to speed up writes?

	Oh, on a follow up to my previous message, if you do create a
separate drive system for your database, and drive write performance is
still an issue, you might consider a non redundant striped array (RAID0 or
perhaps an LVM volume) for the database partition.  Two or three small,
relatively inexpensive, fairly high performance disks can be glued together
into a rather small array with a very small chunk size for absolute maximum
read and write capability of database files.  Note the video and backup
arrays can employ port multipliers and fairly inexpensive drive controllers
for maximum economy, while the database array should probably be a multilane
solution with a fairly high performance drive controller.

RE: Remote NAS

[Leslie Rhorer]

> > > You have two drives in RAID10 currently, and you want to go
> > > to four.
> > > Are you meaning to increase the capacity as well or do you
> > > just want the
> > > two remote drives to be a mirror of your current two
> > > drives?
> >
> > Both more capacity, and remote storage.
> >
> Okay, so you want the local drives to be (effectively) a RAID-0, and
> then mirrored to the remote drives?
> 
> >
> > > > Needless to say I'd like performance to be good, but
> > > that's starting
> > > > to look hopeless.
> > > >
> > > I'd recommend getting two smaller drives, and having two
> > > separate
> > > mirrored pairs - the smaller array for the OS and database
> > > and the
> > > larger one for recording.  Keep one of each size
> > > within the HTPC and one
> > > of each in the garage.  You can also use the
> > > write-mostly option to make
> > > the kernel read from the local drives where possible.
> >
> > But what I'm saying there doesn't seem to be a way in mdadm to specify
> > --one side-- of the mirror here, and the other side, there.
> >
> I've no idea what you mean here.  You specify two block devices for the
> mirror - what difference does it make where (or what) they are?  And if

	Well, he doesn't want to wind up with both a primary copy and a
mirror copy of any of the drive elements in one physical location.

> you want RAID10 then (in the create order) drives 0 and 2 are local and
> 1 and 3 remote.

	In his situation (assuming he doesn't take my advice and separate
the remote system entirely), wouldn't he be better served to create a RAIDn
array on both systems and then create a RAID1 from the local and NAS array
with the write mostly option?

RE: Remote NAS

[Leslie Rhorer]

> > Not transcoding at the same time, just the standard commercial
> > flagging.  The drive light is constantly on.  CPU is pretty busy
> > (3GHz) reaching 180% when simultaneously flagging 2 videos.
> >
> > The database in in the / partition of the array, and the videos are in
> > the /home partition on it.  No idea why the array is so busy.
> >
> If they're both on the array, then that's the problem.  The partitioning
> doesn't make difference to the issue - the drive heads are having to
> seek frantically between reading the data file and writing the database.

	You are perfectly correct that having the database and the video
files on the same array presents a very significant bottleneck, but I think
something else must be going on, as well.  If the CPU were having to wait
for drive seeks, then the CPU utilization would never be that high.  What's
more, while I am not intimately familiar with MythTV's database structure, I
don't think database writes are done very much.  Typically, in such
applications, one only marks the audio / video transition points, where the
video fades to black and the audio squelches and then again when a black
transitions to live video.  The rest of time should ordinarily be spent
entirely in memory (other than reading the video file, of course), analyzing
the video data to search for such transitions and decide whether the
transition in question is the entry or exit point for a group of
commercials.  I would expect a write to the database would only occur after
examining several minutes worth of video data - probably several seconds
worth of reading video.

	I cannot say for certain, but I suspect he might see significant
performance gains by increasing the size of his memory and perhaps his swap
partition, as well.  He also might do well to adjust the priority of the
threads handling the commercial flagging.  He hasn't mentioned what his
actual drive performance is, but his main complaint seems to me not to be
drive system performance but CPU loading.

RE: Remote NAS

[Leslie Rhorer]

> > > Commercial flagging is writing like hell to the array,
> > according to
> > > iotop.

	You might be a little more quantitative about "writing like hell".
How many MBps?  See my last message.

> > I don't know how else to assess what's
> > going on.  The drives

	You might look at the output of `watch iotstat -m 1 2`, as well.
Iotop breaks down disk usage by application.  Iostat shows the total
utilization of each disk element by all processes.

> > > are constantly hammering, and my Myth menu moves get
> > very slow during
> > > flagging, even with the latest 2TB disk drives and
> > RAID10.

	Unless perhaps your swap is on the array, as well, that doesn't
really suggest a drive system bottleneck.  What size is your swap, your main
memory, and how much memory and swap do you show being utilized during
flagging?  For any system doing any sort of video analysis, I suggest at
least 2G of memory.  You say this is a 3GHz CPU, but how many cores and what
width?  I recommend at least a dual core and a 64 bit, unless you have a
RISC based system.

	I have a very different setup than you, and I don't have any Linux
based commercial scan tools handy at the moment, but I have Video Redo on a
Windows XP 64 Pro machine with a dual core 3.2GHz CPU and 8G of RAM.  When I
run a commercial scan on a single video, the system slams both CPUs to more
than 50% and gulps more than 2G of real memory.  Meanwhile, the system is
reading steadily off the video array, but only at around 8MBps.  Even 20
MBps would not be tasking for the most inefficient of array topologies, and
I expect not even with a moderate amount of database activity, constant
seeking notwithstanding.

	Even so, having a completely separate cache group for the database
not touched by any reads or writes of the video files definitely can't hurt.
If you separate the database onto an independant high performance drive then
its entire 32MB is dedicated to caching the database.  If you employ a
striped array, then the database cache is N x 32MB, where N is the number of
drives.  This cannot hurt menu responsiveness of the MythTV menu.  (I use
Series III TiVos fed by the video server, so menu operations are on a
separate machine from file manipulations, including commercial elimination.)
I definitely suggest having plenty of memory to prevent any significant
amount of swapping, and having the swap, the database, and the video files
on three separate drive subsystems.

> Not transcoding at the same time, just the standard commercial flagging.
> The drive light is constantly on.

	That's to be expected.  After all, it should only take a few
microseconds to inspect a pair of successive frames for a black transition
before the system reads the next frame.  Even allowing for caching, the
system should be reading the array pretty much continuously in human terms.

> CPU is pretty busy (3GHz) reaching 180%
> when simultaneously flagging 2 videos.

	Again, see my last message.  It doesn't sound to me like you are
having a particularly large drive througput problem.  Of course, it can't
hurt to optimize the drive subsystem, but I am skeptical it will do much to
address your issue.

RE: Remote NAS

[adfas asd]

I am puzzled by why write-mostly.  Ostensibly Gb ethernet is supposed to nearly equal the data transfer rate of the controller cards, and be many times the speed of actual data transfer from the drives.  

So shouldn't a NAS have plenty of bandwidth to accommodate most any drive operations?

I will be setting up a small high-perf drive for / (including database) so the only data on the array will be /home (including large videos).

Seems like there is no need to specify write-mostly?


--- On Sun, 9/27/09, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>     In his situation (assuming he doesn't
> take my advice and separate
> the remote system entirely), wouldn't he be better served
> to create a RAIDn
> array on both systems and then create a RAID1 from the
> local and NAS array
> with the write mostly option?




      

Re: Remote NAS

[Thomas Fjellstrom]

On Mon September 28 2009, adfas asd wrote:
> I am puzzled by why write-mostly.  Ostensibly Gb ethernet is supposed to
>  nearly equal the data transfer rate of the controller cards, and be many
>  times the speed of actual data transfer from the drives.
> 
> So shouldn't a NAS have plenty of bandwidth to accommodate most any drive
>  operations?

My GbE lan gets about 90MiB/s, my pcie sata card has shown upwards of 
400MiB/s. I haven't done full tests on the sata card yet, but it clearly is 
able to more than saturate a single GbE connection.

> I will be setting up a small high-perf drive for / (including database) so
>  the only data on the array will be /home (including large videos).
> 
> Seems like there is no need to specify write-mostly?
> 
> --- On Sun, 9/27/09, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
> >     In his situation (assuming he doesn't
> > take my advice and separate
> > the remote system entirely), wouldn't he be better served
> > to create a RAIDn
> > array on both systems and then create a RAID1 from the
> > local and NAS array
> > with the write mostly option?
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

RE: Remote NAS

[Leslie Rhorer]

> I am puzzled by why write-mostly.  Ostensibly Gb ethernet is supposed to
> nearly equal the data transfer rate of the controller cards, and be many
> times the speed of actual data transfer from the drives.

Not at all.  Some multilane solutions can manage close to 300MBps per drive.
The absolute maximum throughput of a Gig-E is right at 120MBps.  OTOH, 50 -
60 MBps may be far more than you need, in which case Gig-E is fine.

> So shouldn't a NAS have plenty of bandwidth to accommodate most any drive
> operations?

For your case, perhaps, but you were the one concerned about performance.

 
> I will be setting up a small high-perf drive for / (including database) so
> the only data on the array will be /home (including large videos).

	Why are you putting the array in /home?  Does MythTV require it?  If
not, personally, I would recommend a separate directory for the recordings
and not in /home.  I mount my array in /RAID and put the recordings in
/RAID/Recordings.

> Seems like there is no need to specify write-mostly?

	For your setup it may or may not result in some performance gain.
Certainly reading the material strictly from the local drives will probably
result in higher throughput, but a lower througput for reading the videos
during commercial flagging might actually result in better responsiveness of
your MythTV menus.

RE: Remote NAS

[Leslie Rhorer]

> Thanks for the thoughtful and comprehensive advice.
> 
> Yes I do regular backups from machine to machine (one to the other), but
> my video library is getting large and it's impractical to back that up so
> I look to RAID.

	You need to think that through a bit.  A RAID10 (or other mirrored
solution) requires just as many drives as a backup server solution.  Doing a
JOBD solution on both systems and then backing up from one to the other
takes no more drive space than a RAID10.  I am a belt and suspenders sort of
guy, so I personally employ fault tolerant RAID arrays on both systems, but
a backup JOBD array provides essentially as much robustness as a RAID10
solution, as long as one can live with the primary system going down for a
period of time when a drive is lost and live with the time it takes to
transfer the material back from the backup to the main system after a drive
is replaced.  In my case, I eliminate the down time by adding an extra three
parity drives, and employing RAID6 on the main array and RAID5 on the
backup.

> I've thought about backing up to DVD-RW, but I'd need
> several so will likely just make deep storage volumes. (very occasional)

	Several?  Backing up 1T of storage would require over 100 dual layer
DVDs.

> I don't trust tape backup of any type, and blu-ray write is a losing
> concept when disk space is so cheep.

	Agreed, plus multi-terrabyte tape drives are very expensive, and the
media is as expensive as a hard drive.  Blu-Ray is even worse.  Much worse.

 
> I hear you on the human error aspect, and to have the garage as sort of an
> offline storage is interesting, but it would take forever to back up my
> data there.

	No it doesn't.  It's true the initial backup will take a while
(possibly up to 10 hours per terrabyte, or possibly as little as 3 hours per
terrabyte over a Gig-E link), but then so will building the initial array.
Whether you copy the data via rsync (or other file copy utility) or via md
when building the array, it's going to take the same amount of time.  It's
not like you have to sit and watch the process, though, or like the system
is down during the copy.  Activity can continue on your main system
normally, with some performance hit, of course.

> Not familiar with rsync and will look into it.  Maybe it's a
> diff sort of copy.

	One does not do a full backup of the data on a daily basis, no
matter what.  With video in particular, the bulk of the data never changes
once it is written to disk.  Once a video file is copied over to the backup,
it never needs to be copied again.  Rsync offers a large number of options
for copying based upon various criteria.  Take a look at the following from
my video server:

RAID-Server:/usr/bin# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             110G  5.3G  105G   5% /
tmpfs                 1.7G     0  1.7G   0% /lib/init/rw
udev                   10M  132K  9.9M   2% /dev
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/md0              7.3T  6.4T  957G  88% /RAID
/dev/hda4             788M  948K  747M   1% /etc/mdadm/bitmap
Backup:/Backup        8.2T  6.5T  1.8T  79% /Backup

	The /Backup directory is a remote share using NFS from my backup
server.  Now take a look at this morning's backup to the /Backup directory
performed by the backup server ( I do the backup from the backup server so
the rsync process uses very little CPU time on the video server):

receiving incremental file list
Mail/
Mail/lrhorer
           0   0%    0.00kB/s    0:00:00       106.36M  69%  101.44MB/s
0:00:00       152.08M 100%  108.80MB/s    0:00:01 (xfer#1,
to-check=1391/1469)
Mail_Folders/
Personal_Folders/Leslie/Outlook Files/Outlook Shared File.pst
           0   0%    0.00kB/s    0:00:00           582   0%    0.43kB/s
27:37:05        42.68M 100%   22.51MB/s    0:00:01 (xfer#2,
to-check=1005/5430)
Personal_Folders/Leslie/Quicken/2009.QDF
           0   0%    0.00kB/s    0:00:00         8.73M 100%   13.40MB/s
0:00:00 (xfer#3, to-check=1037/5863)
Server-Main/Movies/
Server-Main/Movies/Unverified TiVo Movies.csv
           0   0%    0.00kB/s    0:00:00            84 100%   82.03kB/s
0:00:00 (xfer#4, to-check=1019/24852)

Number of files: 31734
Number of files transferred: 4
Total file size: 6974.09G bytes
Total transferred file size: 203.49M bytes Literal data: 900.42K bytes
Matched data: 202.59M bytes File list size: 896.65K File list generation
time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent:
152.50K Total bytes received: 1.90M

sent 152.50K bytes  received 1.90M bytes  241.03K bytes/sec total size is
6974.09G  speedup is 3404008.11

RE: Remote NAS

[adfas asd]

--- On Sun, 9/27/09, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>     You might be a little more quantitative
> about "writing like hell".
> How many MBps?  See my last message.

I just started a mythcommflag on a video, and it is not behaving the same.  It's acting more like ppl here would expect with ~3.7MB/s read and 3.5KB/s write.  I'll have to wait for a normal automatic job and do these checks.


>     Unless perhaps your swap is on the
> array, as well, that doesn't
> really suggest a drive system bottleneck.  What size
> is your swap, your main
> memory, and how much memory and swap do you show being
> utilized during
> flagging?  For any system doing any sort of video
> analysis, I suggest at
> least 2G of memory.  You say this is a 3GHz CPU, but
> how many cores and what
> width?  I recommend at least a dual core and a 64 bit,
> unless you have a
> RISC based system.

Yes swap is on the same array, different part.  4G of memory and an E8400 Intel CPU (2 cores).  Debian 64bit OS, and self-compiles MythTV-fixes 0.21.  For the array 2 each WD Green 2TB drives.





      

Re: Remote NAS

[adfas asd]

Interesting, thanks.

--- On Mon, 9/28/09, Thomas Fjellstrom <tfjellstrom@shaw.ca> wrote:
> > I am puzzled by why write-mostly.  Ostensibly Gb
> ethernet is supposed to
> >  nearly equal the data transfer rate of the
> controller cards, and be many
> >  times the speed of actual data transfer from the
> drives.
> > 
> > So shouldn't a NAS have plenty of bandwidth to
> accommodate most any drive
> >  operations?
> 
> My GbE lan gets about 90MiB/s, my pcie sata card has shown
> upwards of 
> 400MiB/s. I haven't done full tests on the sata card yet,
> but it clearly is 
> able to more than saturate a single GbE connection.



      

RE: Remote NAS

[Leslie Rhorer]

> --- On Sun, 9/27/09, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
> >     You might be a little more quantitative
> > about "writing like hell".
> > How many MBps?  See my last message.
> 
> I just started a mythcommflag on a video, and it is not behaving the same.
> It's acting more like ppl here would expect with ~3.7MB/s read and 3.5KB/s
> write.  I'll have to wait for a normal automatic job and do these checks.

	Hmm.  It's odd the process would provide different performance when
automated.  It's possible there's a problem at the application layer.

 
> >     Unless perhaps your swap is on the
> > array, as well, that doesn't
> > really suggest a drive system bottleneck.  What size
> > is your swap, your main
> > memory, and how much memory and swap do you show being
> > utilized during
> > flagging?  For any system doing any sort of video
> > analysis, I suggest at
> > least 2G of memory.  You say this is a 3GHz CPU, but
> > how many cores and what
> > width?  I recommend at least a dual core and a 64 bit,
> > unless you have a
> > RISC based system.
> 
> Yes swap is on the same array, different part.  4G of memory and an E8400
> Intel CPU (2 cores).  Debian 64bit OS, and self-compiles MythTV-fixes
> 0.21.  For the array 2 each WD Green 2TB drives.

	Do you have a GUI installed on the box?  Gnome has a sort of nifty
resource monitor which shows swap and memory usage in real time.  If not,
you can use watch along with vmstat and free to see what's happening with
your memory usage.  If you are writing to the swap file and the database and
both are on your array, it could definitely affect performance, especially
since it is possible almost as much data is being written to the swap file
as is being read from the video files.  I believe there are also free tools
out there which monitor I/O and memory usage for specific applications and /
or threads.  A Google search may be in order if the less specific tools
point in that direction.

	It's possible 4G of memory is insufficient, although I would not
expect so for this application.  If the swap file is quiescent, then memory
size is not the issue, but if the system is swapping a great deal, then a
memory upgrade is definitely indicated, and indeed is a very inexpensive
fix.