reshape success story

reshape success story

[Florian Dazinger]

hi,
because there are so many "help"/"didn't work" mails here, I want to
give positive feedback:

Reshape of a 4-disk RAID6 (250GB each) to RADI5 worked perfectly, now
having a 3-disk RAID5 + hot spare.
the backup-file was on a NFS4-share ;)
duration: ca. 26h (cannot say for sure, it finished during night)

alas, I'm not able to tell the max. filesize of the backup-file (mdadm
deletes it afterwards), but when I looked at it in the middle of the
process, it was about 30MB. Unlike what I excpected from the man-page,
I had network traffic during the *whole* reshape-process, not just at
the beginning. It would be interesting to know, how fast the reshape
process would have been, if the backup-file were on a local disk...

kernel-2.6.36
mdadm-3.1.4

thank you very much for all the good work and user support on this list!!
cheers, florian

Re: reshape success story

[John Robinson]

On 31/10/2010 13:41, Florian Dazinger wrote:
[...]
> Unlike what I excpected from the man-page,
> I had network traffic during the *whole* reshape-process, not just at
> the beginning.

Perhaps the man page needs updating then. The backup file is only used 
at the beginning for grows, or at the end for shrinks, but a same-size 
reshape (as yours was, going from 4-disc RAID-6 to 3-disc RAID-5) needs 
to back up everything because there's no spare space.

Of course when the man page section on reshaping and the use of the 
backup file was originally written, changing RAID level wasn't 
supported, and nor were shrinks, so the backup file was only used for 
grows, so it was only used at the beginning.

If I've got the above right (someone please correct me if I'm not) 
perhaps I could make a modest contribution (for a change) by updating/ 
patching the man page...

Actually that makes me wonder: the man page says spare devices can be 
used for the backup if there are any. Is that still true with all the 
grows, shrinks and level-changing reshape options we have now? I'd 
expect that method of backup to be (slightly) faster than putting the 
backup on a filesystem (even on a local disc).

> thank you very much for all the good work and user support on this list!!

It is nice to hear positive feedback, but at the same time I tend to 
think that since there must be millions of users of md/mdadm, it's 
pretty encouraging that there are only one or two "arghs" per day...

Cheers,

John.

Re: reshape success story

[Neil Brown]

On Sun, 31 Oct 2010 14:19:13 +0000
John Robinson <john.robinson@anonymous.org.uk> wrote:

> On 31/10/2010 13:41, Florian Dazinger wrote:
> [...]
> > Unlike what I excpected from the man-page,
> > I had network traffic during the *whole* reshape-process, not just at
> > the beginning.
> 
> Perhaps the man page needs updating then. The backup file is only used 
> at the beginning for grows, or at the end for shrinks, but a same-size 
> reshape (as yours was, going from 4-disc RAID-6 to 3-disc RAID-5) needs 
> to back up everything because there's no spare space.
> 
> Of course when the man page section on reshaping and the use of the 
> backup file was originally written, changing RAID level wasn't 
> supported, and nor were shrinks, so the backup file was only used for 
> grows, so it was only used at the beginning.
> 
> If I've got the above right (someone please correct me if I'm not) 
> perhaps I could make a modest contribution (for a change) by updating/ 
> patching the man page...

That would certainly be appreciated.   Your understanding appear to be
correct!

> 
> Actually that makes me wonder: the man page says spare devices can be 
> used for the backup if there are any. Is that still true with all the 
> grows, shrinks and level-changing reshape options we have now? I'd 
> expect that method of backup to be (slightly) faster than putting the 
> backup on a filesystem (even on a local disc).

I think mdadm insists on a backup file for shrinks and same-size
transformations.  It probably could use a spare, but I think there is
generally less likely to be one - when you are growing an array there is
almost certainly an array to grow to.

Write to a file on a local disk should be just as fast as writing to a raw
device .... I guess there could be a little bit of filesystem overhead, but I
doubt you would be able to measure it.


> 
> > thank you very much for all the good work and user support on this list!!
> 
> It is nice to hear positive feedback, but at the same time I tend to 
> think that since there must be millions of users of md/mdadm, it's 
> pretty encouraging that there are only one or two "arghs" per day...

:-)

Certainly - positive feedback is always welcome!

NeilBrown



> 
> Cheers,
> 
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reshape success story

[John Robinson]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On 31/10/2010 15:46, Neil Brown wrote:
> On Sun, 31 Oct 2010 14:19:13 +0000
> John Robinson<john.robinson@anonymous.org.uk>  wrote:
[...]
>> Perhaps the man page needs updating then.
[...]
>> If I've got the above right (someone please correct me if I'm not)
>> perhaps I could make a modest contribution (for a change) by updating/
>> patching the man page...
>
> That would certainly be appreciated.   Your understanding appear to be
> correct!

Is the attached of any use? I started with 3.1.4. I've fixed a couple of 
typos as well as hopefully improving the explanations about backup files 
and reshapes, and added a couple of your remarks about metadata types 
from another thread. Some of the text was cribbed from your blog about 
reshaping.

Cheers,

John.

[-- Attachment #2: mdadm.8.in.patch --]
[-- Type: text/plain, Size: 6109 bytes --]

--- a/mdadm.8.in	2010-08-31 08:21:13.000000000 +0100
+++ b/mdadm.8.in	2010-11-02 01:05:44.000000000 +0000
@@ -322,16 +322,20 @@
 ..
 Use the original 0.90 format superblock.  This format limits arrays to
 28 component devices and limits component devices of levels 1 and
-greater to 2 terabytes.
+greater to 2 terabytes.  It is also possible for there to be confusion
+about whether the superblock applies to a whole device or just the
+last partition, if the partition starts on a 64K boundary.
 .ie '{DEFAULT_METADATA}'0.90'
 .IP "1, 1.0, 1.1, 1.2"
 .el
 .IP "1, 1.0, 1.1, 1.2 default"
 ..
 Use the new version-1 format superblock.  This has few restrictions.
-The different sub-versions store the superblock at different locations
-on the device, either at the end (for 1.0), at the start (for 1.1) or
-4K from the start (for 1.2).  "1" is equivalent to "1.0".
+It can easily be moved between hosts with different endian-ness, and a
+recovery operation can be checkpointed and restarted.  The different
+sub-versions store the superblock at different locations on the
+device, either at the end (for 1.0), at the start (for 1.1) or 4K from
+the start (for 1.2).  "1" is equivalent to "1.0".
 'if '{DEFAULT_METADATA}'1.2'  "default" is equivalent to "1.2".
 .IP ddf
 Use the "Industry Standard" DDF (Disk Data Format) format defined by
@@ -493,7 +497,7 @@
 The default is
 .BR left\-symmetric .
 
-It is also possibly to cause RAID5 to use a RAID4-like layout by
+It is also possible to cause RAID5 to use a RAID4-like layout by
 choosing
 .BR parity\-first ,
 or
@@ -660,11 +664,11 @@
 .BR \-\-backup\-file=
 This is needed when
 .B \-\-grow
-is used to increase the number of
-raid-devices in a RAID5 if there are no spare devices available.
-See the GROW MODE section below on RAID\-DEVICES CHANGES.  The file
-should be stored on a separate device, not on the RAID array being
-reshaped.
+is used to increase the number of raid-devices in a RAID5 or RAID6 if
+there are no spare devices available, or to shrink, change RAID level
+or layout.  See the GROW MODE section below on RAID\-DEVICES CHANGES.
+The file must be stored on a separate device, not on the RAID array
+being reshaped.
 
 .TP
 .BR \-\-array-size= ", " \-Z
@@ -883,12 +887,14 @@
 .BR \-\-backup\-file=
 If
 .B \-\-backup\-file
-was used to grow the number of raid-devices in a RAID5, and the system
-crashed during the critical section, then the same
+was used when requesting a grow, shrink, RAID level change or other
+reshape, and the system crashed during the critical section, then the
+same
 .B \-\-backup\-file
 must be presented to
 .B \-\-assemble
-to allow possibly corrupted data to be restored.
+to allow possibly corrupted data to be restored, and the reshape
+to be completed.
 
 .TP
 .BR \-U ", " \-\-update=
@@ -2171,27 +2177,36 @@
 inaccessible.  The integrity of any data can then be checked before
 the non-reversible reduction in the number of devices is request.
 
-When relocating the first few stripes on a RAID5, it is not possible
-to keep the data on disk completely consistent and crash-proof.  To
-provide the required safety, mdadm disables writes to the array while
-this "critical section" is reshaped, and takes a backup of the data
-that is in that section.  This backup is normally stored in any spare
-devices that the array has, however it can also be stored in a
-separate file specified with the
+When relocating the first few stripes on a RAID5 or RAID6, it is not
+possible to keep the data on disk completely consistent and
+crash-proof.  To provide the required safety, mdadm disables writes to
+the array while this "critical section" is reshaped, and takes a
+backup of the data that is in that section.  For grows, this backup may be
+stored in any spare devices that the array has, however it can also be
+stored in a separate file specified with the
 .B \-\-backup\-file
-option.  If this option is used, and the system does crash during the
-critical period, the same file must be passed to
+option, and is required to be specified for shrinks, RAID level
+changes and layout changes.  If this option is used, and the system
+does crash during the critical period, the same file must be passed to
 .B \-\-assemble
-to restore the backup and reassemble the array.
+to restore the backup and reassemble the array.  When shrinking rather
+than growing the array, the reshape is done from the end towards the
+beginning, so the "critical section" is at the end of the reshape.
 
 .SS LEVEL CHANGES
 
 Changing the RAID level of any array happens instantaneously.  However
-in the RAID to RAID6 case this requires a non-standard layout of the
+in the RAID5 to RAID6 case this requires a non-standard layout of the
 RAID6 data, and in the RAID6 to RAID5 case that non-standard layout is
-required before the change can be accomplish.  So while the level
+required before the change can be accomplished.  So while the level
 change is instant, the accompanying layout change can take quite a
-long time.
+long time.  A
+.B \-\-backup\-file
+is required.  If the array is not simultaneously being grown or
+shrunk, so that the array size will remain the same - for example,
+reshaping a 3-drive RAID5 into a 4-drive RAID6 - the backup file will
+be used not just for a "cricital section" but throughout the reshape
+operation, as described below under LAYOUT CHANGES.
 
 .SS CHUNK-SIZE AND LAYOUT CHANGES
 
@@ -2200,10 +2215,13 @@
 To ensure against data loss in the case of a crash, a
 .B --backup-file
 must be provided for these changes.  Small sections of the array will
-be copied to the backup file while they are being rearranged.
+be copied to the backup file while they are being rearranged.  This
+means that all the data is copied twice, once to the backup and once
+to the new layout on the array, so this type of reshape will go very
+slowly.
 
 If the reshape is interrupted for any reason, this backup file must be
-make available to
+made available to
 .B "mdadm --assemble"
 so the array can be reassembled.  Consequently the file cannot be
 stored on the device being reshaped.

Upgraded grub, now confused about mirrored /boot

[Guy Watkins]

Hello,

	I upgraded my system from Red Hat FC10 to FC11.  The instructions
say to run this command:
/sbin/grub-install BOOTDEVICE

And if it fails, run this:
/sbin/grub-install --recheck /dev/sda

However, my boot disk (/boot) is mirrored on 4 disks and I think (or hope)
all 4 are bootable.  The mirrors were created at install time many years ago
when I installed FC5.  No idea if it really made more than 1 bootable.  I
have assumed that if sda failed, I could still boot from sdb, sdc or sdd.
And I do understand that I might need to remove sda first, depending on the
type of failure.  Lucky for me, no drive has failed yet and I don't recall
if I tested booting off of any other disks.

I do have this on the kernel line:
md-mod.start_dirty_degraded=1

So, what do I do now?  Run that command on all 4 disks?  Or run it on
/dev/md0?

Oh, 4 way mirror is not because I am paranoid.  I have 4 disks partitioned
alike, so I figured I would use all 4 disks just for the symmetry.  OCD
maybe, but not paranoid.  :)

# df -k /boot
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0                256586    221526     21812  92% /boot

md0 : active raid1 sdd1[0] sda1[3] sdc1[2] sdb1[1]
      264960 blocks [4/4] [UUUU]
      bitmap: 0/33 pages [0KB], 4KB chunk

Thanks,
Guy

RE: Upgraded grub, now confused about mirrored /boot

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Guy Watkins
> Sent: Tuesday, November 02, 2010 1:11 AM
> To: 'Linux RAID'
> Subject: Upgraded grub, now confused about mirrored /boot
> 
> Hello,
> 
> 	I upgraded my system from Red Hat FC10 to FC11.  The instructions
> say to run this command:
> /sbin/grub-install BOOTDEVICE
> 
> And if it fails, run this:
> /sbin/grub-install --recheck /dev/sda
> 
> However, my boot disk (/boot) is mirrored on 4 disks and I think (or hope)
> all 4 are bootable.  The mirrors were created at install time many years
> ago
> when I installed FC5.  No idea if it really made more than 1 bootable.  I
> have assumed that if sda failed, I could still boot from sdb, sdc or sdd.
> And I do understand that I might need to remove sda first, depending on
> the
> type of failure.  Lucky for me, no drive has failed yet and I don't recall
> if I tested booting off of any other disks.
> 
> I do have this on the kernel line:
> md-mod.start_dirty_degraded=1
> 
> So, what do I do now?  Run that command on all 4 disks?  Or run it on
> /dev/md0?
> 
> Oh, 4 way mirror is not because I am paranoid.  I have 4 disks partitioned
> alike, so I figured I would use all 4 disks just for the symmetry.  OCD
> maybe, but not paranoid.  :)
> 
> # df -k /boot
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md0                256586    221526     21812  92% /boot
> 
> md0 : active raid1 sdd1[0] sda1[3] sdc1[2] sdb1[1]
>       264960 blocks [4/4] [UUUU]
>       bitmap: 0/33 pages [0KB], 4KB chunk

GRUB2 only supports RAID on 0.90 superblock arrays.  If your array has other
than 0.90 superblocks, it won't work, and you will need to convert to 0.90
superblocks in order to continue.  Once done, you should install GRUB2 to
all four drives.  Test booting from all four.

Re: Upgraded grub, now confused about mirrored /boot

[John Robinson]

On 02/11/2010 06:11, Guy Watkins wrote:
> Hello,
>
> 	I upgraded my system from Red Hat FC10 to FC11.  The instructions
> say to run this command:
> /sbin/grub-install BOOTDEVICE
>
> And if it fails, run this:
> /sbin/grub-install --recheck /dev/sda
>
> However, my boot disk (/boot) is mirrored on 4 disks and I think (or hope)
> all 4 are bootable.  The mirrors were created at install time many years ago
> when I installed FC5.  No idea if it really made more than 1 bootable.  I
> have assumed that if sda failed, I could still boot from sdb, sdc or sdd.
> And I do understand that I might need to remove sda first, depending on the
> type of failure.  Lucky for me, no drive has failed yet and I don't recall
> if I tested booting off of any other disks.
>
> I do have this on the kernel line:
> md-mod.start_dirty_degraded=1
>
> So, what do I do now?  Run that command on all 4 disks?  Or run it on
> /dev/md0?

I don't know which grub version you get in FC10/11, but in CentOS 5 
(with grub 0.97), grub-install is a clever little script which does all 
the work for you, so you just
   /sbin/grub-install /dev/md0
and it will install grub on all of /dev/md0's constituent drives and 
generally get everything right.

You should be fine with metadata 0.90 or 1.0 as both store the 
superblock at the end of the device. 1.1 and 1.2 probably won't work 
because their constituent partitions don't look like bare filesystems.

What actually happens is that the BIOS boots off the first live disc, 
and so does grub 0.97, neither has any inherent support for RAID but 
they don't have to if it's a RAID-1 mirror because both (or all) parts 
of the mirror can be used on their own, at least for reading enough to 
boot with.

You should probably test booting up with your first drive disconnected, 
just to be sure!

Cheers,

John.

RE: Upgraded grub, now confused about mirrored /boot

[Guy Watkins]

It worked!
/sbin/grub-install /dev/md0

Thanks,
Guy

} -----Original Message-----
} From: John Robinson [mailto:john.robinson@anonymous.org.uk]
} Sent: Tuesday, November 02, 2010 11:24 AM
} To: Guy Watkins
} Cc: 'Linux RAID'
} Subject: Re: Upgraded grub, now confused about mirrored /boot
} 
} On 02/11/2010 06:11, Guy Watkins wrote:
} > Hello,
} >
} > 	I upgraded my system from Red Hat FC10 to FC11.  The instructions
} > say to run this command:
} > /sbin/grub-install BOOTDEVICE
} >
} > And if it fails, run this:
} > /sbin/grub-install --recheck /dev/sda
} >
} > However, my boot disk (/boot) is mirrored on 4 disks and I think (or
} hope)
} > all 4 are bootable.  The mirrors were created at install time many years
} ago
} > when I installed FC5.  No idea if it really made more than 1 bootable.
} I
} > have assumed that if sda failed, I could still boot from sdb, sdc or
} sdd.
} > And I do understand that I might need to remove sda first, depending on
} the
} > type of failure.  Lucky for me, no drive has failed yet and I don't
} recall
} > if I tested booting off of any other disks.
} >
} > I do have this on the kernel line:
} > md-mod.start_dirty_degraded=1
} >
} > So, what do I do now?  Run that command on all 4 disks?  Or run it on
} > /dev/md0?
} 
} I don't know which grub version you get in FC10/11, but in CentOS 5
} (with grub 0.97), grub-install is a clever little script which does all
} the work for you, so you just
}    /sbin/grub-install /dev/md0
} and it will install grub on all of /dev/md0's constituent drives and
} generally get everything right.
} 
} You should be fine with metadata 0.90 or 1.0 as both store the
} superblock at the end of the device. 1.1 and 1.2 probably won't work
} because their constituent partitions don't look like bare filesystems.
} 
} What actually happens is that the BIOS boots off the first live disc,
} and so does grub 0.97, neither has any inherent support for RAID but
} they don't have to if it's a RAID-1 mirror because both (or all) parts
} of the mirror can be used on their own, at least for reading enough to
} boot with.
} 
} You should probably test booting up with your first drive disconnected,
} just to be sure!
} 
} Cheers,
} 
} John.

Re: reshape success story

[Neil Brown]

On Tue, 02 Nov 2010 01:14:27 +0000
John Robinson <john.robinson@anonymous.org.uk> wrote:

> On 31/10/2010 15:46, Neil Brown wrote:
> > On Sun, 31 Oct 2010 14:19:13 +0000
> > John Robinson<john.robinson@anonymous.org.uk>  wrote:
> [...]
> >> Perhaps the man page needs updating then.
> [...]
> >> If I've got the above right (someone please correct me if I'm not)
> >> perhaps I could make a modest contribution (for a change) by updating/
> >> patching the man page...
> >
> > That would certainly be appreciated.   Your understanding appear to be
> > correct!
> 
> Is the attached of any use? I started with 3.1.4. I've fixed a couple of 
> typos as well as hopefully improving the explanations about backup files 
> and reshapes, and added a couple of your remarks about metadata types 
> from another thread. Some of the text was cribbed from your blog about 
> reshaping.
> 
> Cheers,
> 
> John.

Great!  Thanks.

I made a couple of very minor changes - one of which was a nearby typo that I
spotted.

http://neil.brown.name/git?p=mdadm;a=commitdiff;h=cd19c0cf1c52ce765bf27791b5ef0ee4cdc4c8db

Thanks,
NeilBrown