Recovering from default FC6 install

Recovering from default FC6 install

[Bill Davidsen]

I tried something new on a test system, using the install partitioning 
tools to partition the disk. I had three drives and went with RAID-1 for 
boot, and RAID-5+LVM for the rest. After the install was complete I 
noted that it was solid busy on the drives, and found that the base RAID 
appears to have been created (a) with no superblock and (b) with no 
bitmap. That last is an issue, as a test system it WILL be getting hung 
and rebooted, and recovering the 1.5TB took hours.

Is there an easy way to recover this? The LVM dropped on it has a lot of 
partitions, and there is a lot of data in them asfter several hours of 
feeding with GigE, so I can't readily back up and recreate by hand.

Suggestions?

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Recovering from default FC6 install

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 2720 bytes --]

On Sun, 2006-11-12 at 01:00 -0500, Bill Davidsen wrote:
> I tried something new on a test system, using the install partitioning 
> tools to partition the disk. I had three drives and went with RAID-1 for 
> boot, and RAID-5+LVM for the rest. After the install was complete I 
> noted that it was solid busy on the drives, and found that the base RAID 
> appears to have been created (a) with no superblock and (b) with no 
> bitmap. That last is an issue, as a test system it WILL be getting hung 
> and rebooted, and recovering the 1.5TB took hours.
> 
> Is there an easy way to recover this? The LVM dropped on it has a lot of 
> partitions, and there is a lot of data in them asfter several hours of 
> feeding with GigE, so I can't readily back up and recreate by hand.
> 
> Suggestions?

First, the Fedora installer *always* creates persistent arrays, so I'm
not sure what is making you say it didn't, but they should be
persistent.

So, assuming that they are persistent, just recreate the arrays in place
as version 1.0 superblocks with internal bitmap.  I did that exact thing
on my FC6 machine I was testing with (raid1, not raid5, but no biggie
there) and it worked fine.  The detailed list of instructions:

Reboot with a rescue CD, skip the finding of the installation, when you
are at a prompt, use mdadm to examine the raid superblocks so you get
all the pertinent data such as chunk size on the raid5 and ordering of
constituent drives in the raid5 right.  Then recreate the arrays as
version 1.0 superblocks with internal write intent bitmaps.  Then mount
the partitions manually, bind mount things like /dev and /proc into
wherever you mounted the root filesystem, edit the mdadm.conf on the
root filesystem and remove the ARRAY lines (the GUIDs will be wrong
now), use mdadm -Db or mdadm -Eb to get new ARRAY lines and append them
to the mdadm.conf (possibly altering the device names for the arrays,
and if you use -E remember to correct the printout of the GUID in the
array line, it's 10:8:8:6 instead of 8:8:8:8), patch mkinitrd with
something like the attached patch, patch /etc/rc.d/rc.sysinit with
something like the other attached patch (or leave this patch out but
manually add the correct auto= parameter to your ARRAY lines in the
mdadm.conf), chroot into the root filesystem, remake your initrd image,
fdisk the drives and switch the linux partition types from raid
autodetect to plain linux, reboot, and you are done.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[-- Attachment #1.2: mkinitrd.patch --]
[-- Type: text/x-patch, Size: 796 bytes --]

--- /sbin/mkinitrd	2006-09-28 12:51:28.000000000 -0400
+++ mkinitrd	2006-11-12 10:28:31.000000000 -0500
@@ -1096,6 +1096,13 @@
     mknod $MNTIMAGE/dev/efirtc c 10 136
 fi
 
+if [ -n "$raiddevices" ]; then
+    inst /sbin/mdadm.static "$MNTIMAGE/bin/mdadm"
+    if [ -f /etc/mdadm.conf ]; then
+        cp $verbose /etc/mdadm.conf "$MNTIMAGE/etc/mdadm.conf"
+    fi
+fi
+
 # FIXME -- this can really go poorly with clvm or duplicate vg names.
 # nash should do lvm probing for us and write its own configs.
 if [ -n "$vg_list" ]; then
@@ -1234,8 +1241,7 @@
 
 if [ -n "$raiddevices" ]; then
     for dev in $raiddevices; do
-        cp -a /dev/${dev} $MNTIMAGE/dev
-        emit "raidautorun /dev/${dev}"
+        emit "mdadm -As --auto=yes /dev/${dev}"
     done
 fi
 

[-- Attachment #1.3: rc.sysinit.patch --]
[-- Type: text/x-patch, Size: 372 bytes --]

--- /etc/rc.d/rc.sysinit	2006-10-04 18:14:53.000000000 -0400
+++ rc.sysinit	2006-11-12 10:29:03.000000000 -0500
@@ -403,7 +403,7 @@
 update_boot_stage RCraid
 [ -x /sbin/nash ] && echo "raidautorun /dev/md0" | nash --quiet
 if [ -f /etc/mdadm.conf ]; then
-    /sbin/mdadm -A -s
+    /sbin/mdadm -A -s --auto=yes
 fi
 
 # Device mapper & related initialization

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Recovering from default FC6 install

[Bill Davidsen]

Doug Ledford wrote:

>On Sun, 2006-11-12 at 01:00 -0500, Bill Davidsen wrote:
>  
>
>>I tried something new on a test system, using the install partitioning 
>>tools to partition the disk. I had three drives and went with RAID-1 for 
>>boot, and RAID-5+LVM for the rest. After the install was complete I 
>>noted that it was solid busy on the drives, and found that the base RAID 
>>appears to have been created (a) with no superblock and (b) with no 
>>bitmap. That last is an issue, as a test system it WILL be getting hung 
>>and rebooted, and recovering the 1.5TB took hours.
>>
>>Is there an easy way to recover this? The LVM dropped on it has a lot of 
>>partitions, and there is a lot of data in them asfter several hours of 
>>feeding with GigE, so I can't readily back up and recreate by hand.
>>
>>Suggestions?
>>    
>>
>
>First, the Fedora installer *always* creates persistent arrays, so I'm
>not sure what is making you say it didn't, but they should be
>persistent.
>  
>
I got the detail on the md device, then -E on the components, and got a 
"no super block found" message, which made me think it wasn't there. 
Given that, I didn't have much hope for the part which starts "assuming 
that they are persistent" but I do thank you for the information, I'm 
sure it will be useful.

I did try recreating, from the running FC6 rather than the rescue, since 
the large data was on it's own RAID and I could umount the f/s and stop 
the array. Alas, I think a "grow" is needed somewhere, after 
configuration, start, and mount of the f/s on RAID-5, e2fsck told me my 
data was toast. Shortest time to solution was to recreate the f/s and 
reload the data.

The RAID-1 stuff is small, a total rebuild is acceptable in the case of 
a failure.

FC install suggestion: more optional control over the RAID features 
during creation. Maybe there's an "advanced features" button in the 
install and I just missed it, but there should be, since the non-average 
user might be able to do useful things with the chunk size, and specify 
a bitmap. I would think that a bitmap would be the default on large 
arrays, assuming that >1TB is still large for the moment.

Instructions and attachments save for future use, trimmed here.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Recovering from default FC6 install

[Neil Brown]

On Sunday November 12, dledford@redhat.com wrote:
> 
> So, assuming that they are persistent, just recreate the arrays in place
> as version 1.0 superblocks with internal bitmap.  I did that exact thing
> on my FC6 machine I was testing with (raid1, not raid5, but no biggie
> there) and it worked fine.  The detailed list of instructions:
> 

You can add a bitmap to an array while it is active.
  mdadm --grow --bitmap=internal /dev/mdX

(Providing the kernel is new enough - I'm not sure when that went in,
but it was a while ago).

NeilBrown

Re: Recovering from default FC6 install

[Bill Davidsen]

Neil Brown wrote:
> On Sunday November 12, dledford@redhat.com wrote:
>   
>> So, assuming that they are persistent, just recreate the arrays in place
>> as version 1.0 superblocks with internal bitmap.  I did that exact thing
>> on my FC6 machine I was testing with (raid1, not raid5, but no biggie
>> there) and it worked fine.  The detailed list of instructions:
>>
>>     
>
> You can add a bitmap to an array while it is active.
>   mdadm --grow --bitmap=internal /dev/mdX
>   

Thank you!!! Works like a charm!
> (Providing the kernel is new enough - I'm not sure when that went in,
> but it was a while ago).
The latest FC6 kernel is 2.6.18 something, should be okay.


-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979