Real Time Mirroring of a NAS

Real Time Mirroring of a NAS

[andy liebman]

Hi,

I'm looking for a way to create a real-time mirror of a NAS. In other 
words, say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB 
drives). I want to mirror it in real time to a completely separate 5.5 
TB NAS. RSYNCing in the background is not an option. The two NAS boxes 
need to hold identical data at all times. It is NOT necessary that the 
data be accessed from both NAS boxes simultaneously. One is simply a 
backup of the other.

I guess one option might be to use Redhat's Global File System (GFS). 
Although there seems to be an 8TB limit for a GFS filesystem, which 
could be an issue down the road.

But would the following approach work:

Can I export NAS B as a SAN or ISCSI target, connect the two machines 
with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on 
NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of 
thing done?

Andy Liebman

Re: Real Time Mirroring of a NAS

[Joshua Baker-LePain]

On Fri, 7 Apr 2006 at 6:47am, andy liebman wrote

> I'm looking for a way to create a real-time mirror of a NAS. In other words, 
> say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB drives). I want 
> to mirror it in real time to a completely separate 5.5 TB NAS. RSYNCing in 
> the background is not an option. The two NAS boxes need to hold identical 
> data at all times. It is NOT necessary that the data be accessed from both 
> NAS boxes simultaneously. One is simply a backup of the other.
>
> I guess one option might be to use Redhat's Global File System (GFS). 
> Although there seems to be an 8TB limit for a GFS filesystem, which could be 
> an issue down the road.

I don't have any experience with it, but I've often seen DRBD mentioned 
for just this sort of situation.  I'd look into that.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Real Time Mirroring of a NAS

[Tuomas Leikola]

> > I'm looking for a way to create a real-time mirror of a NAS. In other words,
> > say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB drives). I want
> > to mirror it in real time to a completely separate 5.5 TB NAS. RSYNCing in
> > the background is not an option. The two NAS boxes need to hold identical
> > data at all times. It is NOT necessary that the data be accessed from both
> > NAS boxes simultaneously. One is simply a backup of the other.
> >
> I don't have any experience with it, but I've often seen DRBD mentioned
> for just this sort of situation.  I'd look into that.
>

Linux NBD (and md on top) is a simpler solution for the same thing.
I'd look into that also :)

Re: Real Time Mirroring of a NAS

[John Stoffel]

andy> I'm looking for a way to create a real-time mirror of a NAS. In
andy> other words, say I have a 5.5 TB NAS (3ware 16-drive array,
andy> RAID-5, 500 GB drives). I want to mirror it in real time to a
andy> completely separate 5.5 TB NAS. RSYNCing in the background is
andy> not an option. The two NAS boxes need to hold identical data at
andy> all times. It is NOT necessary that the data be accessed from
andy> both NAS boxes simultaneously. One is simply a backup of the
andy> other.

How much data loss can you stand?  Or more accurately, how much lag in
copying data between the arrays can you stand for your Disaster
recovery issues?  How far apart are the two systems?  Is the setup
write intensive, or read intensive?

Why is rsync not an option?  Not upto date enough?  *grin*  

One option might be to use inotify to watch filesystem A and to copy
data that changes to filesystem B as quickly as possible.  

andy> I guess one option might be to use Redhat's Global File System
andy> (GFS).  Although there seems to be an 8TB limit for a GFS
andy> filesystem, which could be an issue down the road.

How large are you planning on growing your filesystem?  Why can't it
be split?  What are your assumptions?   

andy> But would the following approach work:

andy> Can I export NAS B as a SAN or ISCSI target, connect the two
andy> machines with, say, mryinet cards or 10 GbE TOE cards, mount the
andy> NAS B volume on NAS A, and create a RAID-1 mirror of the two
andy> volumes? Is this kind of thing done?

I guess I'd look into mirroring the underlying disk subsystems and
then putting the NAS on top of that.  If the NAS box goes down, you
can then just fire up NAS on the backup server and continue. 

Mostly, I don't understand what problem you're trying to solve, so I
can't offer good suggestions because the solution changes depending on
assumptions.  

I already assume you don't have tons of money to spend here.  *grin*

John

Re: Real Time Mirroring of a NAS

[Jim Klimov]

Hello andy,

al> Can I export NAS B as a SAN or ISCSI target, connect the two machines
Am I right in the assumption that your NASes are Linux boxes? :)

Did you take a look at Linux "Network block devices" (nbd/enbd)? They
might be what you need: you'd get a raw device on one of the servers
to use in a mirror along with a local device. The NBD page mentioned
some setups for high-availability services where an active server
clones itself to a backup server and vice-versa, whichever was active
most recently.

I'm not sure about performance though...

al> with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on
al> NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of
al> thing done?

Are you sure you need 10GbE? My experience with a 10-drive 3Ware 8506
array in RAID5 shows that reads from it usually fit in 500-700Mbit/s.
And it's a very busy popular fileserver, so I guess it's close to the
hardware limits of our array.

-- 
Best regards,
 Jim Klimov                            mailto:klimov@2ka.mipt.ru

Re: Real Time Mirroring of a NAS

[andy liebman]

klimov@2ka.mipt.ru wrote:
> Hello andy,
> 
> al> Can I export NAS B as a SAN or ISCSI target, connect the two machines
> Am I right in the assumption that your NASes are Linux boxes? :)
> 
> Did you take a look at Linux "Network block devices" (nbd/enbd)? They
> might be what you need: you'd get a raw device on one of the servers
> to use in a mirror along with a local device. The NBD page mentioned
> some setups for high-availability services where an active server
> clones itself to a backup server and vice-versa, whichever was active
> most recently.
> 
> I'm not sure about performance though...
> 
> al> with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on
> al> NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of
> al> thing done?
> 
> Are you sure you need 10GbE? My experience with a 10-drive 3Ware 8506
> array in RAID5 shows that reads from it usually fit in 500-700Mbit/s.
> And it's a very busy popular fileserver, so I guess it's close to the
> hardware limits of our array.
> 

Thanks for your reply, and the suggestions of others. I'm going to look 
into both NBD and DRBD.

Actually, I see that my idea to export an iSCSI target from Server B, 
mount it on A, and just create a RAID1 array with the two block devices 
must be very similar to what DRBD is doing, but my guess is that DRBD, 
with it's "heartbeat" signal, is probably more robust at error handling. 
I'd love to hear from somebody who has experience with DRBD.

By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array, I 
can get sequential reads that top 600 MBs/sec. That's megabytes, not 
megabits. And write speeds are close to 400 MB/sec with the new faster 
on-board XOR processing. And random reads are at least 200 MB/sec. So, 
10 GbE is a must, really.

Andy

Re: Real Time Mirroring of a NAS

[Maurice Hilarius]

andy liebman wrote:
> ..
> Thanks for your reply, and the suggestions of others. I'm going to
> look into both NBD and DRBD.
>
> Actually, I see that my idea to export an iSCSI target from Server B,
> mount it on A, and just create a RAID1 array with the two block
> devices must be very similar to what DRBD is doing, but my guess is
> that DRBD, with it's "heartbeat" signal, is probably more robust at
> error handling. I'd love to hear from somebody who has experience with
> DRBD.
>
> By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array,
> I can get sequential reads that top 600 MBs/sec. That's megabytes, not
> megabits. And write speeds are close to 400 MB/sec with the new faster
> on-board XOR processing. And random reads are at least 200 MB/sec. So,
> 10 GbE is a must, really.
>
> Andy
>
Hi Andy.

A couple of other suggestions that may prove helpful:

1) EVMS
http://evms.sourceforge.net/


2) Lustre
http://www.clusterfs.com/
http://www.lustre.org/



-- 

With our best regards,


Maurice W. Hilarius        Telephone: 01-780-456-9771
Hard Data Ltd.  FAX:       01-780-456-9772
11060 - 166 Avenue         email:maurice@harddata.com
Edmonton, AB, Canada       http://www.harddata.com/
   T5X 1Y3

Re: Real Time Mirroring of a NAS

[Tim Moore]

andy liebman wrote:
 > ...
> By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array, I 
> can get sequential reads that top 600 MBs/sec. That's megabytes, not 
> megabits. And write speeds are close to 400 MB/sec with the new faster 
> on-board XOR processing. And random reads are at least 200 MB/sec. So, 
> 10 GbE is a must, really.

A 400MB/s network input stream means either 1x10GbE or 4xGbE's to 
PCI-X/PCI-e bus, then to disk, then to PCI-X/PCI-e bus, then across another 
net to the backup system, so the I/O subsystem alone must handle 800MB/s.

3ware may actually do a single 400MB/s stream, but what about 4x100MB/s or 
8x50MB/s?  It's really hard to do that much I/O _and_ provide data 
integrity.  If your data rates and the mirroring requirements are that 
critical you might want to look at the fast NAS systems coming out (eg- 
agami.com) built on top of AMD's HT architecture.

Cheers,


> 
> Andy
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
  | for direct mail add "private_" in front of user name