Backups using RAID1

Backups using RAID1

[Drew]

Hi there,

In recent months I've been exploring low cost solutions for backing up
the data on my home media server. I already run three raid-1 arrays on
this rig to protect against drive failures and a fellow linux user
suggested I consider using the existing raid, just yank the 'backup'
disks as needed, and drop in fresh disks to replace the yanked unit.

I realize I'm treading on unsupported ground here but what suggestions
can people give me to minimize the risk of data loss?

-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: Backups using RAID1

[Drew]

On Wed, Nov 19, 2008 at 12:13 PM, Linux <linuxlist@gmail.com> wrote:
>> In recent months I've been exploring low cost solutions for backing up
>> the data on my home media server. I already run three raid-1 arrays on
>> this rig to protect against drive failures and a fellow linux user
>> suggested I consider using the existing raid, just yank the 'backup'
>> disks as needed, and drop in fresh disks to replace the yanked unit.
>
> I guess this is a little risky. Plugging disks too often does not  sound good.
> Is "rsync" not enough for you?

A year ago my old 10GB Travan & 4 tapes was enough to backup my OS and
data. In the last year I've seen my data storage jump from 40GB to 2TB
courtesy of a MythTV installation. I'm still figuring out how to
manage this large an amount of data and backup solutions are now on
the table as it's been indicated to me that loss of important 'media'
will result in a greatly lowered WAF(*).


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

(*) WAF: Wife Acceptance Factor

Re: Backups using RAID1

[Max Waterman]

Of course, you could always *not* yank them; instead just fail them
(after unmounting/syncing/whatever) and leave the actual disk in the
machine.

Purists would likely argue this isn't true backup or something, but
backup can be taken to different extremes - it all depends on what
risks you want to protect yourself against.

If you don't care about location-based risks (eg fire), then I don't
see why you would bother removing the drives. Leaving disks in the
machine basically only protects you against 'oops' moments (rm -rf and
such like)., but not much else.

The advantage in RAID1 is that it makes a copy constantly, so it takes
no time to create the backup - using other methods (rsync, tape,
rdiff-backup) with a huge amount of data, this time can be
prohibitive.

Also, I'd say that plugging/unplugging disks would historically be a
problem, but SATA shouldn't be, IMO. Also, there are solutions
specifically designed for plugging/unplugging - which makes the point
moot - so you might consider one of those.

Having saidd that, this is Just my opinion, and I'm no expert...

Max.

2008/11/20 Drew <drew.kay@gmail.com>:
> On Wed, Nov 19, 2008 at 12:13 PM, Linux <linuxlist@gmail.com> wrote:
>>> In recent months I've been exploring low cost solutions for backing up
>>> the data on my home media server. I already run three raid-1 arrays on
>>> this rig to protect against drive failures and a fellow linux user
>>> suggested I consider using the existing raid, just yank the 'backup'
>>> disks as needed, and drop in fresh disks to replace the yanked unit.
>>
>> I guess this is a little risky. Plugging disks too often does not  sound good.
>> Is "rsync" not enough for you?
>
> A year ago my old 10GB Travan & 4 tapes was enough to backup my OS and
> data. In the last year I've seen my data storage jump from 40GB to 2TB
> courtesy of a MythTV installation. I'm still figuring out how to
> manage this large an amount of data and backup solutions are now on
> the table as it's been indicated to me that loss of important 'media'
> will result in a greatly lowered WAF(*).
>
>
> --
> Drew
>
> "Nothing in life is to be feared. It is only to be understood."
> --Marie Curie
>
> (*) WAF: Wife Acceptance Factor
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Backups using RAID1

[David Greaves]

Max Waterman wrote:
> Of course, you could always *not* yank them; instead just fail them
> (after unmounting/syncing/whatever) and leave the actual disk in the
> machine.

Also worth mentioning is that you can have a 3-way RAID1.

So adding a disk, allowing it to sync and then removing it again means you are
never left with just a single point of failure.

You can of course send the 'backup' disk(s) to sleep too.

David

Re: Backups using RAID1

[Drew]

> If you don't care about location-based risks (eg fire), then I don't
> see why you would bother removing the drives. Leaving disks in the
> machine basically only protects you against 'oops' moments (rm -rf and
> such like)., but not much else.

In this instance location based risks (fire, earthquake) are a
concern. My original idea when I started exploring backup ideas was
something I could leave unattended to start when I went to bed and if
for some reason I was forced to evacuate in the wee hours of the
morning all I had to do was yank the drives from the server and leave.

As far as oops moments, only the applications have direct access to
files on disk. All user access to disks is via Samba and I've enabled
the recycle bin vfs module.

> The advantage in RAID1 is that it makes a copy constantly, so it takes
> no time to create the backup - using other methods (rsync, tape,
> rdiff-backup) with a huge amount of data, this time can be
> prohibitive.

That was part of why I was looking at RAID for the backup. I've also
had a few suggestions about getting an external eSATA drive and
leaving it plugged in overnight. Just have a cron job do a nightly
rsync or such and *if* I have to evacuate, hopefully rsync will be
complete.

> Also, I'd say that plugging/unplugging disks would historically be a
> problem, but SATA shouldn't be, IMO. Also, there are solutions
> specifically designed for plugging/unplugging - which makes the point
> moot - so you might consider one of those.

My SATA controller supports hot plugging so I'm not worried there.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: Backups using RAID1

[Jon Nelson]

Why not consider something like drbd or even something like this:

use AoE or iSCSI or NBD or *whatever* that exports a block device over
the network, and use *that* device in a bitmap-backed raid1.

(thus, your raid1 consists of /dev/sdX and /dev/etherd/e0.5 )


-- 
Jon

Re: Backups using RAID1

[Max Waterman]

2008/11/20 Drew <drew.kay@gmail.com>:
>> The advantage in RAID1 is that it makes a copy constantly, so it takes
>> no time to create the backup - using other methods (rsync, tape,
>> rdiff-backup) with a huge amount of data, this time can be
>> prohibitive.
>
> That was part of why I was looking at RAID for the backup. I've also
> had a few suggestions about getting an external eSATA drive and
> leaving it plugged in overnight. Just have a cron job do a nightly
> rsync or such and *if* I have to evacuate, hopefully rsync will be
> complete.

Well, I would have thought making your eSATA drive part of the RAID1
array would be plausible....even a good solution.

>
>> Also, I'd say that plugging/unplugging disks would historically be a
>> problem, but SATA shouldn't be, IMO. Also, there are solutions
>> specifically designed for plugging/unplugging - which makes the point
>> moot - so you might consider one of those.
>
> My SATA controller supports hot plugging so I'm not worried there.

I don't think the controller was the concern, though I could be wrong
- my take was it was more that the connectors might wear out or
something along those lines. Either a removable hard drive chassis or
an external eSATA drive would seem to fit the bill from that point of
view...

Max.

Re: Backups using RAID1

[Bill Davidsen]

Drew wrote:
>> If you don't care about location-based risks (eg fire), then I don't
>> see why you would bother removing the drives. Leaving disks in the
>> machine basically only protects you against 'oops' moments (rm -rf and
>> such like)., but not much else.
>>     
>
> In this instance location based risks (fire, earthquake) are a
> concern. My original idea when I started exploring backup ideas was
> something I could leave unattended to start when I went to bed and if
> for some reason I was forced to evacuate in the wee hours of the
> morning all I had to do was yank the drives from the server and leave.
>
> As far as oops moments, only the applications have direct access to
> files on disk. All user access to disks is via Samba and I've enabled
> the recycle bin vfs module.
>
>   
>> The advantage in RAID1 is that it makes a copy constantly, so it takes
>> no time to create the backup - using other methods (rsync, tape,
>> rdiff-backup) with a huge amount of data, this time can be
>> prohibitive.
>>     
>
> That was part of why I was looking at RAID for the backup. I've also
> had a few suggestions about getting an external eSATA drive and
> leaving it plugged in overnight. Just have a cron job do a nightly
> rsync or such and *if* I have to evacuate, hopefully rsync will be
> complete.
>
>   
If you are that paranoid about the backup, get two and use a different 
one each night. You can run a cron job to back stuff up (I've done it) 
every half hour or so, given three checks: (a) is the last one finished, 
(b) is the last modified time > 30 minutes (ie. is it done), and (c) has 
it been modified more recently than the last backup (touch a file at the 
end of backup).

Having dealt with both fire and earthquake, I doubt your wife will worry 
about the recordings, just getting the people and pets out, and whatever 
paperwork you have in your fireproof safe (in case time is tight).
>> Also, I'd say that plugging/unplugging disks would historically be a
>> problem, but SATA shouldn't be, IMO. Also, there are solutions
>> specifically designed for plugging/unplugging - which makes the point
>> moot - so you might consider one of those.
>>     
>
> My SATA controller supports hot plugging so I'm not worried there.
>
>
>   


-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Backups using RAID1

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Mon Nov 17, 2008 at 07:13:07AM -0800, Drew wrote:

> Hi there,
> 
> In recent months I've been exploring low cost solutions for backing up
> the data on my home media server. I already run three raid-1 arrays on
> this rig to protect against drive failures and a fellow linux user
> suggested I consider using the existing raid, just yank the 'backup'
> disks as needed, and drop in fresh disks to replace the yanked unit.
> 
> I realize I'm treading on unsupported ground here but what suggestions
> can people give me to minimize the risk of data loss?
> 
Firstly, a disclaimer - I've not tried this myself, so there's no
personal experience behind these suggestions.

Ideally you should unmount the array before failing & removing the
drive.  If this is not possible then pick a time when the IO is likely
to be low - there's always going be a risk of corruption (especially
with something like a database), but modern journalling filesystems
usually cope with this okay.

As has been said elsewhere, you can have more than 2 disks in a RAID-1
array so don't leave yourself at risk by having a single disk in the
array.  Whenever you want to do a backup, add in the new disk and, once
it's been fully synced, fail & remove it (or leave 3 disks in the array
& just pull one when you want the backup, replacing it with another).

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Backups using RAID1

[David Greaves]

Robin Hill wrote:
> Ideally you should unmount the array before failing & removing the
> drive.  If this is not possible then pick a time when the IO is likely
> to be low - there's always going be a risk of corruption (especially
> with something like a database), but modern journalling filesystems
> usually cope with this okay.


If you can't unmount then essentially you want to snapshot the
filesystem/databases - some provide facilities for this.

xfs_freeze springs to mind.

At the very least you could do
sync;mdadm --fail
but that's playing with fire...

David

-- 
"Don't worry, you'll be fine; I saw it work in a cartoon once..."