question: no-bitmap RAID1 with off-site drive

question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Hello,

Sorry if this has been answered before,
but I'm having trouble finding the
relevant messages by searching.  Several
solutions are floating around.

Want to configure a simple no-bitmap
RAID1 with three-drives, one always
stored off-site.

Will rotate only two or three times a
year so I'd rather have the no-bitmap
setup where the incoming drive gets a
full resync--takes just eight hours.
Have a daily incremental cloud-backup,
which allows infrequent visits to the
safe-deposit box.

What are the correct approach and
commands for this?  Will never
have three drives together on-site.

I'm nervous about using --fail --remove
to take the mirror copy offline, fearing
that it could put the backup in an
unusable state.  Thinking a --stop
might be cleanest approach.

Wondering if --grow --raid-devices=1 --force
after the drive is removed and then
--grow --add --raid-devices=2 with
the returning drive is correct or if a
--raid-devices=3 with no --grow on
rotation is better.

I'm looking for the most correct way to
go, but don't have enough experience of
'mdadm' to have a sense of what it
should be.

Thank you for any help!

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Would like to have the option to zero the
superblock of an incoming drive and have
it treated as a virgin replacement.

I tried this (I think) by running

   mdadm --add /dev/md2 /dev/sde

It worked, but assigned the new drive
as device number [2], which seems
incorrect.  I tried failing/removing the
drive followed by a

   mdadm --remove /dev/md2 detached

and repeated the --add with a zero-superblock
drive but it does the same thing every time.
Only way to get it to reuse drive slot 1
was to do a 

   mdadm --grow --raid-devices=1 --force /dev/md2

followed by a

   mdadm --grow --add --raid-devices=2 /dev/md2 /dev/sde

Is this the expected behavior?  I find it
confusing and probably don't understand
it properly.

Kernel is 3.1.8.

Re: question: no-bitmap RAID1 with off-site drive

[Adam Goryachev]

On 12/11/12 09:14, starlight.2012q4@binnacle.cx wrote:
> Would like to have the option to zero the
> superblock of an incoming drive and have
> it treated as a virgin replacement.
>
> I tried this (I think) by running
>
>    mdadm --add /dev/md2 /dev/sde
>
> It worked, but assigned the new drive
> as device number [2], which seems
> incorrect.  I tried failing/removing the
> drive followed by a
>
>    mdadm --remove /dev/md2 detached
>
> and repeated the --add with a zero-superblock
> drive but it does the same thing every time.
> Only way to get it to reuse drive slot 1
> was to do a 
>
>    mdadm --grow --raid-devices=1 --force /dev/md2
>
> followed by a
>
>    mdadm --grow --add --raid-devices=2 /dev/md2 /dev/sde
>
> Is this the expected behavior?  I find it
> confusing and probably don't understand
> it properly.
>
I think (from what I understand) is that you will do the following:
1) Two HDD in raid1 installed, synced, working
2) Install third HDD
3) mdadm --manage /dev/md2 --add /dev/sde (add it as a spare)
4) mdadm --grow /dev/md2 --raid-devices=3 (add the third disk to the
array, and let it sync)
5) madam --manage /dev/md2 --fail /dev/sdd (fail the disk you want to
remove)
6) mdadm --manage /dev/md2 --remove /dev/sdd (remove the disk from the
array)
7) mdadm --grow /dev/md2 --raid-devices=2 (reconfigure the array to only
have two disks)
8) Physically remove sdd and send offsite

The above process will ensure that you ALWAYS have at least two copies
of your live data (ie, two working, fully synced members of the array).

I have seen that there are some new features to make a drive as "want
replacement", which will do some magic to copy all the data to a spare
device without actually removing the drive until the spare is synced or
similar. However, I think this is only in very recent kernel, and you
might need to do some work in finding the usage details etc.

Hope the above helps.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

At 02:40 PM 11/12/2012 +1100, Adam Goryachev wrote:
>5) madam --manage /dev/md2 --fail /dev/sdd (fail the disk you want to remove)
>6) mdadm --manage /dev/md2 --remove /dev/sdd (remove the disk from the array)

So this sequence will not harm the image
on the drive being roatated offsite?
Absolutely 100% for certain?  If there
is any chance of a problem I can
--stop the array before removing the
drive.  The volume is used for a Bacula
network backup that is often idle.

The third drive will never be on site
(think about the risk), but that just
means that the array will grow from
2->1 and then from 1->2 instead
of from 2->3 and 3->2.

Re: question: no-bitmap RAID1 with off-site drive

[Drew]

> The third drive will never be on site
> (think about the risk), but that just
> means that the array will grow from
> 2->1 and then from 1->2 instead
> of from 2->3 and 3->2.

I'd argue your risk from a failing disk during resync is significantly
greater then your risk of facility failure during the same window. Not
sure what drives are pricing where you are but adding a forth into the
rotation is cheap insurance. Two for the array, one syncing, one
offsite.

Some recent blogs (
http://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2019/
for example ) suggest that modern drives will encounter unrecoverable
read errors approx every 12TB read. That works out to approx every
6-12 resyncs based on 1-2TB drives.


-- 
Drew

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Good point but not an issue here.

Have daily offsite cloud incremental backup
in the mix here (i.e. forth drive).

The array does nothing but receive
incremental Bacula archives once a day
in the early AM.  So there's no
moving target data loss potential.


At 09:56 AM 11/12/2012 -0800, Drew wrote:
>> The third drive will never be on site
>> (think about the risk), but that just
>> means that the array will grow from
>> 2->1 and then from 1->2 instead
>> of from 2->3 and 3->2.
>
>I'd argue your risk from a failing disk during
>resync is significantly greater then your risk
>of facility failure during the same window. Not
>sure what drives are pricing where you are but
>adding a forth into the rotation is cheap
>insurance. Two for the array, one syncing, one
>offsite.
>
>Some recent blogs (
>http://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2
>019/
>for example ) suggest that modern drives will
>encounter unrecoverable read errors approx every
>12TB read. That works out to approx every
>6-12 resyncs based on 1-2TB drives.
>
>
>-- 
>Drew
>--
>To unsubscribe from this list: send the line "unsubscribe 
>linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  
>http://vger.kernel.org/majordomo-info.html

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

After some pondering, think I've figure it out.

Best way to go is to set it up with

  --create --level=1 --raid-devices=2

for the initial pair of drives, then

  --fail --remove

the rotate-out drive, then

  --add

the alternate drive.  Now there will be
three drive slots with one "removed" and two
"active".

To rotate a drive off-site

   --fail --remove

go to the off-site, swap the drives and on return

   --re-add

the rotate-in drive.

This way the 'mdadm' UUID labels will stay on the
drives and 'mdadm' will warn against mistakes
such as trying to use the wrong drive for
a mirror pair.  Will always have one "removed"
drive.

The

   --grow --raid-devices=1 --force

and

   --grow --raid-devices=2 --add

would be used only if a drive fails and needs to
be replaced.

If anyone disagrees please advise.

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Probably this is more elegant:

   mdadm --create --level=1 --raid-devices=2 --spare-devices=1 /dev/sdd /dev/sde missing

At 12:26 PM 11/12/2012 -0500, starlight.2012q4@binnacle.cx wrote:
>After some pondering, think I've figure it out.
>
>Best way to go is to set it up with
>
>  --create --level=1 --raid-devices=2
>
>for the initial pair of drives, then
>
>  --fail --remove
>
>the rotate-out drive, then
>
>  --add
>
>the alternate drive.  Now there will be
>three drive slots with one "removed" and two
>"active".

Re: question: no-bitmap RAID1 with off-site drive

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

On Mon, 12 Nov 2012 13:11:50 -0500 starlight.2012q4@binnacle.cx wrote:

> Probably this is more elegant:
> 
>    mdadm --create --level=1 --raid-devices=2 --spare-devices=1 /dev/sdd /dev/sde missing

That is fairly meaningless ... I'm not sure it even works.

While it makes sense for raid devices to be missing, it doesn't make sense
for spares to be missing.  Spares are just just any devices in the array
which aren't active.  If they are missing, then they aren't spare.

NeilBrown


> 
> At 12:26 PM 11/12/2012 -0500, starlight.2012q4@binnacle.cx wrote:
> >After some pondering, think I've figure it out.
> >
> >Best way to go is to set it up with
> >
> >  --create --level=1 --raid-devices=2
> >
> >for the initial pair of drives, then
> >
> >  --fail --remove
> >
> >the rotate-out drive, then
> >
> >  --add
> >
> >the alternate drive.  Now there will be
> >three drive slots with one "removed" and two
> >"active".
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Thank you!  Still working to wrap my brain
around 'md'.  Will stick with the first
approach.

Amusingly I performed it by accident,
was confused and frustrated by result,
then eventually realized it was ideal.




At 07:31 AM 11/13/2012 +1100, NeilBrown wrote:
>On Mon, 12 Nov 2012 13:11:50 -0500 starlight.2012q4@binnacle.cx 
>wrote:
>
>> Probably this is more elegant:
>> 
>>    mdadm --create --level=1 --raid-devices=2 --spare-devices=1 
>/dev/sdd /dev/sde missing
>
>That is fairly meaningless ... I'm not sure it even works.
>
>While it makes sense for raid devices to be missing, it doesn't
>make sense for spares to be missing.  Spares are just
>any devices in the array which aren't active.  If they
>are missing, then they aren't spare.
>
>NeilBrown
>
>
>> 
>> At 12:26 PM 11/12/2012 -0500, starlight.2012q4@binnacle.cx 
>wrote:
>> >After some pondering, think I've figure it out.
>> >
>> >Best way to go is to set it up with
>> >
>> >  --create --level=1 --raid-devices=2
>> >
>> >for the initial pair of drives, then
>> >
>> >  --fail --remove
>> >
>> >the rotate-out drive, then
>> >
>> >  --add
>> >
>> >the alternate drive.  Now there will be
>> >three drive slots with one "removed" and two
>> >"active".
>> 
>> --
>
>

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

Started playing with the third drive
and my invented approach may not
be correct.  That, as recommended,
using --grow to eliminate the offsite
"removed" drive and put back the
rotate-in drive is right.

It looks like MD re-initializes the
superblock (different UUID) when
something like

  mdadm --grow --add --raid-devices=3 /dev/md3 /dev/sde

is run, even if /dev/sde has a previous
superblock from the same array.  However
I only tested it with a drive image that
had not been fully synced and from the
same (not different) array.  Running

   mdadm --re-add /dev/md3 /dev/sde

results 'mdadm' refusing the
drive that had been from the
same array (though not fully synced).

Appears then that 'mdadm' may not
check the superblock to see if the
drive came from a different array
and should not be overwritten, and
instead prefers to just zap it.

Can anyone confirm or deny the
possibility?  I can manually run
an --examine and check the array
name as a precaution when rotating
drives if 'mdadm' doesn't perform
the check.  Want to avoid inserting
a offsite drive in the wrong array.




At 12:26 PM 11/12/2012 -0500, starlight.2012q4@binnacle.cx wrote:
>After some pondering, think I've figure it out.
>
>Best way to go is to set it up with
>
>  --create --level=1 --raid-devices=2
>
>for the initial pair of drives, then
>
>  --fail --remove
>
>the rotate-out drive, then
>
>  --add
>
>the alternate drive.  Now there will be
>three drive slots with one "removed" and two
>"active".
>
>To rotate a drive off-site
>
>   --fail --remove
>
>go to the off-site, swap the drives and on return
>
>   --re-add
>
>the rotate-in drive.
>
>This way the 'mdadm' UUID labels will stay on the
>drives and 'mdadm' will warn against mistakes
>such as trying to use the wrong drive for
>a mirror pair.  Will always have one "removed"
>drive.
>
>The
>
>   --grow --raid-devices=1 --force
>
>and
>
>   --grow --raid-devices=2 --add
>
>would be used only if a drive fails and needs to
>be replaced.
>
>If anyone disagrees please advise.

Re: question: no-bitmap RAID1 with off-site drive

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3192 bytes --]

On Sun, 18 Nov 2012 19:11:56 -0500 starlight.2012q4@binnacle.cx wrote:

> Started playing with the third drive
> and my invented approach may not
> be correct.  That, as recommended,
> using --grow to eliminate the offsite
> "removed" drive and put back the
> rotate-in drive is right.
> 
> It looks like MD re-initializes the
> superblock (different UUID) when
> something like
> 
>   mdadm --grow --add --raid-devices=3 /dev/md3 /dev/sde
> 
> is run, even if /dev/sde has a previous
> superblock from the same array.  However
> I only tested it with a drive image that
> had not been fully synced and from the
> same (not different) array.  Running
> 
>    mdadm --re-add /dev/md3 /dev/sde
> 
> results 'mdadm' refusing the
> drive that had been from the
> same array (though not fully synced).
> 
> Appears then that 'mdadm' may not
> check the superblock to see if the
> drive came from a different array
> and should not be overwritten, and
> instead prefers to just zap it.

Correct.  When you ask mdadm to --add device to an array, that is what it
will do.  If the metadata looks particularly good it might do a re-add for
you, but if not it will just erase it and write some more.

> 
> Can anyone confirm or deny the
> possibility?  I can manually run
> an --examine and check the array
> name as a precaution when rotating
> drives if 'mdadm' doesn't perform
> the check.  Want to avoid inserting
> a offsite drive in the wrong array.
> 

We might be able to make "mdadm -I" do what you want ... find out which array
it "should" be a member of and auto-add it.
But that will currently fail for an out-of-date member with no bitmap.

If you applied
http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=75a410f6226d1e3ef441bbb8cd4f198d5de5cf5b

and put
  policy action=force-spare
in mdadm.conf, it might work.

NeilBrown


> 
> 
> 
> At 12:26 PM 11/12/2012 -0500, starlight.2012q4@binnacle.cx wrote:
> >After some pondering, think I've figure it out.
> >
> >Best way to go is to set it up with
> >
> >  --create --level=1 --raid-devices=2
> >
> >for the initial pair of drives, then
> >
> >  --fail --remove
> >
> >the rotate-out drive, then
> >
> >  --add
> >
> >the alternate drive.  Now there will be
> >three drive slots with one "removed" and two
> >"active".
> >
> >To rotate a drive off-site
> >
> >   --fail --remove
> >
> >go to the off-site, swap the drives and on return
> >
> >   --re-add
> >
> >the rotate-in drive.
> >
> >This way the 'mdadm' UUID labels will stay on the
> >drives and 'mdadm' will warn against mistakes
> >such as trying to use the wrong drive for
> >a mirror pair.  Will always have one "removed"
> >drive.
> >
> >The
> >
> >   --grow --raid-devices=1 --force
> >
> >and
> >
> >   --grow --raid-devices=2 --add
> >
> >would be used only if a drive fails and needs to
> >be replaced.
> >
> >If anyone disagrees please advise.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question: no-bitmap RAID1 with off-site drive

[starlight.2012q4]

At 01:31 PM 11/27/2012 +1100, NeilBrown wrote:
>> 
>> Appears then that 'mdadm' may not
>> check the superblock to see if the
>> drive came from a different array
>> and should not be overwritten, and
>> instead prefers to just zap it.
>
>Correct.  When you ask mdadm to --add device to
>an array, that is what it will do.  If the
>metadata looks particularly good it might do a 
>re-add for you, but if not it will just erase
>it and write some more.

Thank you very much for the confirmation.

>> Can anyone confirm or deny the
>> possibility?  I can manually run
>> an --examine and check the array
>> name as a precaution when rotating
>> drives if 'mdadm' doesn't perform
>> the check.  Want to avoid inserting
>> a offsite drive in the wrong array.
>
>We might be able to make "mdadm -I" do what
>you want ... find out which array
>it "should" be a member of and auto-add it.
>But that will currently fail for an
>out-of-date member with no bitmap.
>

Actually I was thinking more about 'mdadm'
not destroying an old-but-good mirror
copy from a different array.  So if I
accidently pull the wrong drive out of
the safe deposit box, then 'mdadm' would
refuse to zap and re-label the drive on
an attempt to insert it in the wrong
array.  An explicit --zero-superblock
would have to be run before a drive could
be re-used in any array with a different
name.

Not a big deal though.  I will be sure
to --examine all rotate-in drives
carefully before --grow --add'ing them.