Raid1 backup solution.

Raid1 backup solution.

[Daniel Reurich]

Hi Guys.

I'm considering implementing a rotating offsite backup solution using
raid 1.  This solution uses 1 or more internal drives and 2 external
e-sata harddrives.  The raid setup would be a whole disk partitionable
raid1 volume.

The idea is that by swapping the external drives,  I can have a
boot-able ready to run offsite backup of the machine, as well as
redundancy on the machine itself.  Backups of the important data would
be replicated via an incremental daily backup process onto the raid
volume itself.  

The part that concerns me is how to get a clean removal of the drive
being swapped out, and how will the raid handle having a stale drive
inserted/re-added.

I have been considering a couple of ways to handle this:

1) Power the machine down to swap the drives.  This has the advantage
that the backup is always in a clean bootable state with filesystem
consistency pretty much guaranteed.

2) Use mdadm to fail and remove the drives, and then re-add the newly
attached stale drive.  (Perhaps a udev rule could be made handle the
re-add).  The disadvantage is this will potentially leave the backup in
an inconsistent and possibly un-bootable state unless there is a way to
quiesce and sync disk activity before the removal.  It will also mark
the drive as failed and require     It's advantage is that the machine
doesn't need to be turned off.

3) Hot pull the drive e-sata cable, then power down the drive.  This is
likely to leave the filesystems in really nasty state if there just
happens to be a write going on at the time.

My preference is for option 2 as option 1 may not always be feasible due
to the downtime, but I'm wondering about how best to handle the re-add,
as I suspect that the metadata on the failed then removed drive, would
make it more difficult to re-add the drive into the array.

If option 1 was used (cold swap), how would md handle assembly with the
stale but not failed member disk?  Would it simply force a resync, or
would it fail the disk and require manual intervention to re-add it.

Any thoughts on my hair brained scheme would be appreciated.

Daniel Reurich.

Re: Raid1 backup solution.

[Daniel Reurich]

On Tue, 2010-03-02 at 22:50 +1300, Daniel Reurich wrote:
> Hi Guys.
> 
> I'm considering implementing a rotating offsite backup solution using
> raid 1.  This solution uses 1 or more internal drives and 2 external
> e-sata harddrives.  The raid setup would be a whole disk partitionable
> raid1 volume.
> 
> The idea is that by swapping the external drives,  I can have a
> boot-able ready to run offsite backup of the machine, as well as
> redundancy on the machine itself.  Backups of the important data would
> be replicated via an incremental daily backup process onto the raid
> volume itself.  
> 
> The part that concerns me is how to get a clean removal of the drive
> being swapped out, and how will the raid handle having a stale drive
> inserted/re-added.
> 
> I have been considering a couple of ways to handle this:
> 
> 1) Power the machine down to swap the drives.  This has the advantage
> that the backup is always in a clean bootable state with filesystem
> consistency pretty much guaranteed.
> 
> 2) Use mdadm to fail and remove the drives, and then re-add the newly
> attached stale drive.  (Perhaps a udev rule could be made handle the
> re-add).  The disadvantage is this will potentially leave the backup 
with an inconsistent filesystem and possibly have some corrupted files
unless there is a way to programmatically queisce all write filesystem
write activity and sync the disk before the removal.  

> It will also mark the drive as failed and require mdadm --re-add to
> insert the spare drive.      It's advantage is that the machine
> doesn't need to be turned off.
> 
> 3) Hot pull the drive e-sata cable, then power down the drive.  This is
> likely to leave the filesystems in really nasty state if there just
> happens to be a write going on at the time.

Actually scrap 3 as on re-reading it goes against all sensibility.
> 
> My preference is for option 2 as option 1 may not always be feasible due
> to the downtime, but I'm wondering about how best to handle the re-add,
> as I suspect that the metadata on the failed then removed drive, would
> make it more difficult to re-add the drive into the array.
> 
> If option 1 was used (cold swap), how would md handle assembly with the
> stale but not failed member disk?  Would it simply force a resync, or
> would it fail the disk and require manual intervention to re-add it.
> 
> Any thoughts on my hair brained scheme would be appreciated.
> 
> Daniel Reurich.

Re: Raid1 backup solution.

[Neil Brown]

On Tue, 02 Mar 2010 22:50:07 +1300
Daniel Reurich <daniel@centurion.net.nz> wrote:

> Hi Guys.
> 
> I'm considering implementing a rotating offsite backup solution using
> raid 1.  This solution uses 1 or more internal drives and 2 external
> e-sata harddrives.  The raid setup would be a whole disk partitionable
> raid1 volume.
> 
> The idea is that by swapping the external drives,  I can have a
> boot-able ready to run offsite backup of the machine, as well as
> redundancy on the machine itself.  Backups of the important data would
> be replicated via an incremental daily backup process onto the raid
> volume itself.  
> 
> The part that concerns me is how to get a clean removal of the drive
> being swapped out, and how will the raid handle having a stale drive
> inserted/re-added.
> 
> I have been considering a couple of ways to handle this:
> 
> 1) Power the machine down to swap the drives.  This has the advantage
> that the backup is always in a clean bootable state with filesystem
> consistency pretty much guaranteed.
> 
> 2) Use mdadm to fail and remove the drives, and then re-add the newly
> attached stale drive.  (Perhaps a udev rule could be made handle the
> re-add).  The disadvantage is this will potentially leave the backup in
> an inconsistent and possibly un-bootable state unless there is a way to
> quiesce and sync disk activity before the removal.  It will also mark
> the drive as failed and require     It's advantage is that the machine
> doesn't need to be turned off.

How could it fail to boot?

If your machine crashes, it still boots - right?
So if you fail the drive at any random time, then it is like a crash, and
should still boot.

I would:
  sync ; sync; mdadm /dev/mdX -f /dev/whatever
  unplug the device
  mdadm /dev/mdX --remove detached

Also, if you want two rotating backups I would create two stacked raid1s.

mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
mkfs -j /dev/md1

Then when you add either device back in, it will just resync the bits that
have changed since that device was last attached.  Make sure you add the
device to the correct array of course.

NeilBrown


> 
> 3) Hot pull the drive e-sata cable, then power down the drive.  This is
> likely to leave the filesystems in really nasty state if there just
> happens to be a write going on at the time.
> 
> My preference is for option 2 as option 1 may not always be feasible due
> to the downtime, but I'm wondering about how best to handle the re-add,
> as I suspect that the metadata on the failed then removed drive, would
> make it more difficult to re-add the drive into the array.
> 
> If option 1 was used (cold swap), how would md handle assembly with the
> stale but not failed member disk?  Would it simply force a resync, or
> would it fail the disk and require manual intervention to re-add it.
> 
> Any thoughts on my hair brained scheme would be appreciated.
> 
> Daniel Reurich.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid1 backup solution.

[Michael Evans]

On Tue, Mar 2, 2010 at 2:05 AM, Daniel Reurich <daniel@centurion.net.nz> wrote:
> On Tue, 2010-03-02 at 22:50 +1300, Daniel Reurich wrote:
>> Hi Guys.
>>
>> I'm considering implementing a rotating offsite backup solution using
>> raid 1.  This solution uses 1 or more internal drives and 2 external
>> e-sata harddrives.  The raid setup would be a whole disk partitionable
>> raid1 volume.
>>
>> The idea is that by swapping the external drives,  I can have a
>> boot-able ready to run offsite backup of the machine, as well as
>> redundancy on the machine itself.  Backups of the important data would
>> be replicated via an incremental daily backup process onto the raid
>> volume itself.
>>
>> The part that concerns me is how to get a clean removal of the drive
>> being swapped out, and how will the raid handle having a stale drive
>> inserted/re-added.
>>
>> I have been considering a couple of ways to handle this:
>>
>> 1) Power the machine down to swap the drives.  This has the advantage
>> that the backup is always in a clean bootable state with filesystem
>> consistency pretty much guaranteed.
>>
>> 2) Use mdadm to fail and remove the drives, and then re-add the newly
>> attached stale drive.  (Perhaps a udev rule could be made handle the
>> re-add).  The disadvantage is this will potentially leave the backup
> with an inconsistent filesystem and possibly have some corrupted files
> unless there is a way to programmatically queisce all write filesystem
> write activity and sync the disk before the removal.
>
>> It will also mark the drive as failed and require mdadm --re-add to
>> insert the spare drive.      It's advantage is that the machine
>> doesn't need to be turned off.
>>
>> 3) Hot pull the drive e-sata cable, then power down the drive.  This is
>> likely to leave the filesystems in really nasty state if there just
>> happens to be a write going on at the time.
>
> Actually scrap 3 as on re-reading it goes against all sensibility.
>>
>> My preference is for option 2 as option 1 may not always be feasible due
>> to the downtime, but I'm wondering about how best to handle the re-add,
>> as I suspect that the metadata on the failed then removed drive, would
>> make it more difficult to re-add the drive into the array.
>>
>> If option 1 was used (cold swap), how would md handle assembly with the
>> stale but not failed member disk?  Would it simply force a resync, or
>> would it fail the disk and require manual intervention to re-add it.
>>
>> Any thoughts on my hair brained scheme would be appreciated.
>>
>> Daniel Reurich.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Why don't you just sync ; sync; sync; wait a few seconds and then
mdadm --fail /dev/whatever the device you want to remove is; then
actually remove it from the array?

In the event you need to boot or read from the failed 'snapshot' of
the array you can add --force to the mdadm --assemble to cause it to
accept the failed member as a valid array member (just make sure there
are no other members of that array present).  It will then become the
current master set for that fork of your storage versions.  If you
need to add that drive back to the original array be sure to
--zero-superblock for that device first, otherwise the versions may
confuse things.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid1 backup solution.

[Goswin von Brederlow]

Neil Brown <neilb@suse.de> writes:

> On Tue, 02 Mar 2010 22:50:07 +1300
> Daniel Reurich <daniel@centurion.net.nz> wrote:
>
>> Hi Guys.
>> 
>> I'm considering implementing a rotating offsite backup solution using
>> raid 1.  This solution uses 1 or more internal drives and 2 external
>> e-sata harddrives.  The raid setup would be a whole disk partitionable
>> raid1 volume.
>> 
>> The idea is that by swapping the external drives,  I can have a
>> boot-able ready to run offsite backup of the machine, as well as
>> redundancy on the machine itself.  Backups of the important data would
>> be replicated via an incremental daily backup process onto the raid
>> volume itself.  
>> 
>> The part that concerns me is how to get a clean removal of the drive
>> being swapped out, and how will the raid handle having a stale drive
>> inserted/re-added.
>> 
>> I have been considering a couple of ways to handle this:
>> 
>> 1) Power the machine down to swap the drives.  This has the advantage
>> that the backup is always in a clean bootable state with filesystem
>> consistency pretty much guaranteed.
>> 
>> 2) Use mdadm to fail and remove the drives, and then re-add the newly
>> attached stale drive.  (Perhaps a udev rule could be made handle the
>> re-add).  The disadvantage is this will potentially leave the backup in
>> an inconsistent and possibly un-bootable state unless there is a way to
>> quiesce and sync disk activity before the removal.  It will also mark
>> the drive as failed and require     It's advantage is that the machine
>> doesn't need to be turned off.
>
> How could it fail to boot?
>
> If your machine crashes, it still boots - right?
> So if you fail the drive at any random time, then it is like a crash, and
> should still boot.
>
> I would:
>   sync ; sync; mdadm /dev/mdX -f /dev/whatever
>   unplug the device
>   mdadm /dev/mdX --remove detached

You can improve on that with lvm and suspending the filesystems. That
ensures that nothing starts just between the sysnc and --fail. But if
you want to suspend / too (in case it isn't read-only) you will need an
tmpfs or ramdisk holding everything needed to suspend and fail the disk.

Having / and /usr read-only will basically garanty a bootable system
alraedy so suspending might be overkill.

> Also, if you want two rotating backups I would create two stacked raid1s.
>
> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> mkfs -j /dev/md1
>
> Then when you add either device back in, it will just resync the bits that
> have changed since that device was last attached.  Make sure you add the
> device to the correct array of course.
>
> NeilBrown

MfG
        Goswin