mdadm break / restore soft mirror

mdadm break / restore soft mirror

[Brett Maton]

Hi, 



  Question for you guys. 



  A brief history: 

  RHEL 4 AS 

  I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard



  methods mean that a restore is impossibly slow due to the sheer volume of files. 

  Solution, raw backup /restore of the device.  However the partition is permanently being accessed. 



  Proposed solution is to use software raid mirror.  Before backup starts, break the soft mirror unmount and backup partition



  restore soft mirror and let it resync / rebuild itself. 



  Would the above intentional break/fix of the mirror cause any problems? 



Regards, 

Brett 



Brett Maton 

Linux Consultant 

RHCE #805007238628267 



-----------------------------------------------------------------------------------------------

This email, and the contents contained within, are private

and confidential between the sender and the individual(s)

to whom it is addressed.

The contents do not necessarily represent the views of Simplyhealth Group

Limited.



In the event of misdirection, the recipient is prohibited from 

using, copying or disseminating it or any information contained in it.

If you have received this email in error please notify Simplyhealth

immediately by telephone on 0845 075 2020, or by email to 

network.administrator@simplyhealth.co.uk



Simplyhealth Group Limited is registered and incorporated in England and Wales  

as a company limited by guarantee. Its registered office is at Hambleden House, 

Waterloo Court, Andover, Hampshire, SP10 1LQ, registered no. 5445654. We 

may record or monitor telephone calls to help improve our service and protect our 

members.

-----------------------------------------------------------------------------------------------

Re: mdadm break / restore soft mirror

[Richard Scobie]

Brett Maton wrote:
> Hi, 
> 
>   Question for you guys. 
> 
>   A brief history: 
>   RHEL 4 AS 
>   I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
> 
>   methods mean that a restore is impossibly slow due to the sheer volume of files. 
>   Solution, raw backup /restore of the device.  However the partition is permanently being accessed. 
> 
>   Proposed solution is to use software raid mirror.  Before backup starts, break the soft mirror unmount and backup partition
> 
>   restore soft mirror and let it resync / rebuild itself. 
> 
>   Would the above intentional break/fix of the mirror cause any problems? 
> 

Is there a reason you can't use rsync to just update changes from the 
previous backup, once you have your initial one?

Regards,

Richard

Re: mdadm break / restore soft mirror

[Neil Brown]

On Wednesday December 12, Brett.Maton@simplyhealth.co.uk wrote:
> Hi, 
> 
>   Question for you guys. 
> 
>   A brief history: 
>   RHEL 4 AS 
>   I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
> 
>   methods mean that a restore is impossibly slow due to the sheer volume of files. 
>   Solution, raw backup /restore of the device.  However the partition is permanently being accessed. 
> 
>   Proposed solution is to use software raid mirror.  Before backup starts, break the soft mirror unmount and backup partition
> 
>   restore soft mirror and let it resync / rebuild itself. 
> 
>   Would the above intentional break/fix of the mirror cause any problems? 

No, it should work fine.

If you can be certain that the device that you break out of the mirror
is never altered, then you could add an internal bitmap while the
array is split and the rebuild will go much faster.
However even mounting a device readonly will sometimes alter the
content (e.g. if ext3 needs to replay the journal) so you need to be
very careful.

NeilBrown

Re: mdadm break / restore soft mirror

[Jeff Breidenbach]

>Proposed solution is to use software raid mirror.  Before backup starts,
>break the soft mirror unmount and backup partition

I use this method for backup once a week.

One challenge is drives aren't great at steaming data quickly (for the resync)
while also doing a lot of random access. Having a little extra redundancy (think
3 or more drives in the RAID-1) can really help.

>If you can be certain that the device that you break out of the mirror
>is never altered, then you could add an internal bitmap while the
>array is split and the rebuild will go much faster.

Is this also a viable speedup for the "kep rotating backup drives through
the array" strategy? If so, how much speedup are we talking about? Assume
the array changes by 1% before a backup drive gets rotated in again.

Jeff

Re: mdadm break / restore soft mirror

[Neil Brown]

On Wednesday December 12, jeff@jab.org wrote:
> 
> >If you can be certain that the device that you break out of the mirror
> >is never altered, then you could add an internal bitmap while the
> >array is split and the rebuild will go much faster.
> 
> Is this also a viable speedup for the "kep rotating backup drives through
> the array" strategy? If so, how much speedup are we talking about? Assume
> the array changes by 1% before a backup drive gets rotated in again.
> 

Not really...

The bitmap only records areas of the array that have changed since one
particular moment in time.  For rotating backs you would really want
several moments in time.
Whenever the array is non-degraded, the bitmap forgets any old state.

What you could do is set the number of devices in the array to 3 so
they it always appears to be degraded, then rotate your backup drives
through the array.  The number of dirty bits in the bitmap will
steadily grow and so resyncs will take longer.  Once it crosses some
threshold you set the array back to having 2 devices to that it looks
non-degraded and clean the bitmap.  Then each device will need a full
resync after which you will get away with partial resyncs for a while.

Not ideal, but it might work.

If 1% changes each time, then you will initially get a 100 fold
speedup, dropping away after that.

NeilBrown

Re: mdadm break / restore soft mirror

[Jeff Breidenbach]

> What you could do is set the number of devices in the array to 3 so
> they it always appears to be degraded, then rotate your backup drives
> through the array.  The number of dirty bits in the bitmap will
> steadily grow and so resyncs will take longer.  Once it crosses some
> threshold you set the array back to having 2 devices to that it looks
> non-degraded and clean the bitmap.  Then each device will need a full
> resync after which you will get away with partial resyncs for a while.

I don't undertand why clearing the bitmap causes a rebuild of
all devices. I think I have a conceptual misunderstanding.  Consider
a RAID-1 and three physical disks involved, A,B,C

1) A and B are in the RAID, everything is synced
2) Create a bitmap on the array
3) Fail + remove B
4) Hot add C, wait for C to sync
5) Fail + remove C
6) Hot add B, wait for B to resync
7) Goto step 3

I understand that after a while we might want to clean the bitmap
and that would trigger a full resync for drives B and C. I don't
understand why it would ever cause a resync for drive A.

Re: mdadm break / restore soft mirror

[Neil Brown]

On Thursday December 13, jeff@jab.org wrote:
> > What you could do is set the number of devices in the array to 3 so
> > they it always appears to be degraded, then rotate your backup drives
> > through the array.  The number of dirty bits in the bitmap will
> > steadily grow and so resyncs will take longer.  Once it crosses some
> > threshold you set the array back to having 2 devices to that it looks
> > non-degraded and clean the bitmap.  Then each device will need a full
> > resync after which you will get away with partial resyncs for a while.
> 
> I don't undertand why clearing the bitmap causes a rebuild of
> all devices. I think I have a conceptual misunderstanding.  Consider
> a RAID-1 and three physical disks involved, A,B,C
> 
> 1) A and B are in the RAID, everything is synced
> 2) Create a bitmap on the array
> 3) Fail + remove B
> 4) Hot add C, wait for C to sync
> 5) Fail + remove C
> 6) Hot add B, wait for B to resync
> 7) Goto step 3
> 
> I understand that after a while we might want to clean the bitmap
> and that would trigger a full resync for drives B and C. I don't
> understand why it would ever cause a resync for drive A.

You are exactly correct.  That is what I meant, though I probably
didn't express it very clearly.

After you clean out the bitmap, any devices that are not in the array
at that time will need a full resync to come back in to the array.

NeilBrown

Re: mdadm break / restore soft mirror

[Jeff Breidenbach]

So the obvious follow up question is: for this scenario does it make sense to
only resync the difference between the two bitmaps? E.g. Drive A will have
a current bitmap, B will have a stale bitmap. Presumably one could get away
with just resyncing the difference.

Or is this too much of special case to consider?

Jeff

Re: mdadm break / restore soft mirror

[Jeff Breidenbach]

On Dec 14, 2007 11:13 AM, Jeff Breidenbach <jeff@jab.org> wrote:
> So the obvious follow up question is: for this scenario does it make sense to
> only resync the difference between the two bitmaps?

Never mind, I see why this won't work.

Re: mdadm break / restore soft mirror

[Bill Davidsen]

Brett Maton wrote:
> Hi, 
>
>   Question for you guys. 
>
>   A brief history: 
>   RHEL 4 AS 
>   I have a partition with way to many small files on (Usually around a couple of million) that needs to be backed up, standard
>
>   methods mean that a restore is impossibly slow due to the sheer volume of files. 
>   Solution, raw backup /restore of the device.  However the partition is permanently being accessed. 
>
>   Proposed solution is to use software raid mirror.  Before backup starts, break the soft mirror unmount and backup partition
>
>   restore soft mirror and let it resync / rebuild itself. 
>
>   Would the above intentional break/fix of the mirror cause any problems? 
>   

Probably. If by "accessed" you mean read-only, you can do this, but if 
the data is changing you have a serious problem that the data on the 
disk and queued in memory may leave that part on the disk as an 
inconsistent data set. If there is a means of backing up a set of files 
which are changing other than stopping the updates in a known valid 
state, it's not something which I've seen really work in all cases.

DM has some snapshot capabilities, but in fact they have the same 
limitation, the data on a partition can be backed up, but unless you can 
ensure that the data is in a consistent state when it's frozen, your 
backup will have some small possibility of failure. Database programs 
have ways to freeze the data to do backups, but if an application 
doesn't have a means to force the data on the disk valid, it will only 
be a "pretty good" backup.

I suggest looking at things like rsync, which will not solve the 
changing data problem, but may do the backup quickly enough to be as 
useful as what you propose. Of course a full backup is likely to take a 
long time however you do it.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

RE: mdadm break / restore soft mirror

[Brett Maton]

>>Proposed solution is to use software raid mirror.  Before backup starts,

>>break the soft mirror unmount and backup partition



>I use this method for backup once a week.



>One challenge is drives aren't great at steaming data quickly (for the resync)

>while also doing a lot of random access. Having a little extra redundancy (think

>3 or more drives in the RAID-1) can really help.



>>If you can be certain that the device that you break out of the mirror

>>is never altered, then you could add an internal bitmap while the

>>array is split and the rebuild will go much faster.



When the device is split from the mirror it will only be read as a raw device by the backup

software (NetBackup) and not actually mounted during the process so it really shouldn't be

modified in anyway at all.

Also during the backup window files will only be being created on remaining device (reports pdf's) so random access shouldn't be a problem either.



How do I create the internal bitmap?  man mdadm didn't shed any light and my brief excursion into google wasn't much more helpful.



The version I have installed is mdadm-1.12.0-5.i386 from RedHat which would appear to be way out of date!



Thanks for you help,

Brett

-----------------------------------------------------------------------------------------------

This email, and the contents contained within, are private

and confidential between the sender and the individual(s)

to whom it is addressed.

The contents do not necessarily represent the views of Simplyhealth Group

Limited.



In the event of misdirection, the recipient is prohibited from 

using, copying or disseminating it or any information contained in it.

If you have received this email in error please notify Simplyhealth

immediately by telephone on 0845 075 2020, or by email to 

network.administrator@simplyhealth.co.uk



Simplyhealth Group Limited is registered and incorporated in England and Wales  

as a company limited by guarantee. Its registered office is at Hambleden House, 

Waterloo Court, Andover, Hampshire, SP10 1LQ, registered no. 5445654. We 

may record or monitor telephone calls to help improve our service and protect our 

members.

-----------------------------------------------------------------------------------------------

RE: mdadm break / restore soft mirror

[Neil Brown]

On Thursday December 13, Brett.Maton@simplyhealth.co.uk wrote:

> How do I create the internal bitmap?  man mdadm didn't shed any
> light and my brief excursion into google wasn't much more helpful. 

  mdadm --grow --bitmap=internal /dev/mdX

> 
> The version I have installed is mdadm-1.12.0-5.i386 from RedHat
> which would appear to be way out of date! 

WAY!  mdadm 2.0 would be an absolute minimum, and linux 2.6.13 as an
absolute minimum, probably something closer to 2.6.20 would be a good
idea.

NeilBRown