Lenny to squeeze mdadm problem + e2label problem

Lenny to squeeze mdadm problem + e2label problem

[Manuel Arostegui]

Hey all,

Whilst trying to upgrade one box from Debian Lenny to Debian Squeeze I
have run into some problems with mdadm and the RAID labeling.

Kernel: 2.6.32-41
mdadm version: v3.1.4
Disks: 4

RAID devices:
/dev/md0 - boot - RAID1 - ext3
/dev/md1 swap
/dev/md2 - "/" - RAID1 - ext3
/dev/md3 - "/var/lib/mysql" - RAID10 - xfs

When doing the dist-upgrade and replacing the kernel + udev the system
changes the RAID devices to md124, md125, md126 etc.
As far as I have read, this makes sense - as mdadm is changing
superblock and has to recalculate the whole RAID. This is big problem,
if you have devices into /etc/fstab, like we do.

As this is kinda from the past, we have tried to label the RAID
devices in order to boot up using labels. As in the future migrations,
no one knows if this can arise again and using labels would prevent us
from having these kind of problems.

We used e2label for the ext3 partitions and xfs_admin for the XFS one.

Everything was fine when checking with blkid. We could see the labels
fine in /dev/mdX.
When rebooting, the machine won't boot up:

ALERT! /dev/disk/by-uuid/c018-etc-etc... does not exist


If we re-check the labels we can see all the devices got the same
label (the one assigned to md0, md1 etc), so probably grub is trying
to boot from single devices, and not from the md0 which is /boot
This is blkid's output:

/dev/sdb1: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
SEC_TYPE="ext2" TYPE="ext3"
/dev/sdb2: LABEL="swap" UUID="af027073-b59e-4886-9e5a-e3bad38ae466" TYPE="swap"
/dev/sdb3: LABEL="/" UUID="c8e78ff4-b165-45d4-9382-ed669e5650dc"
SEC_TYPE="ext2" TYPE="ext3"
/dev/sdc1: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
SEC_TYPE="ext2" TYPE="ext3"
/dev/sdc2: LABEL="swap" UUID="af027073-b59e-4886-9e5a-e3bad38ae466" TYPE="swap"
/dev/sdc3: LABEL="/" UUID="c8e78ff4-b165-45d4-9382-ed669e5650dc"
SEC_TYPE="ext2" TYPE="ext3"
/dev/sdd1: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
SEC_TYPE="ext2" TYPE="ext3"
/dev/sdd2: LABEL="swap" UUID="af027073-b59e-4886-9e5a-e3bad38ae466" TYPE="swap"
/dev/sdd3: LABEL="/" UUID="c8e78ff4-b165-45d4-9382-ed669e5650dc"
SEC_TYPE="ext2" TYPE="ext3"

<snip>

/dev/md0: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
SEC_TYPE="ext2" TYPE="ext3"
/dev/md1: LABEL="swap" UUID="af027073-b59e-4886-9e5a-e3bad38ae466" TYPE="swap"
/dev/md2: LABEL="/" UUID="c8e78ff4-b165-45d4-9382-ed669e5650dc"
SEC_TYPE="ext2" TYPE="ext3"
/dev/md3: LABEL="mysql" UUID="e627f664-f929-4be8-815e-9e3cf559e512" TYPE="xfs"


As we have to migrate loads of machines, using labels is the fastest
and cleanest way to do it. Just create the labels and change
/etc/fstab across all the machines and voilá. But we cannot get it
running. Is there any workaround to work with labels and avoid all the
disks in the raid to get them as well?.
Using UUID doesn't seem like a good idea, as they may change with
future dist-upgrades, mdadm upgrades etc.

Has someone had same/similar problem with mdadm?

Thanks a lot
Manuel
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Lenny to squeeze mdadm problem + e2label problem

[John Robinson]

On 22/02/2012 11:44, Manuel Arostegui wrote:
> Hey all,
>
> Whilst trying to upgrade one box from Debian Lenny to Debian Squeeze I
> have run into some problems with mdadm and the RAID labeling.
>
> Kernel: 2.6.32-41
> mdadm version: v3.1.4
> Disks: 4
>
> RAID devices:
> /dev/md0 - boot - RAID1 - ext3
> /dev/md1 swap
> /dev/md2 - "/" - RAID1 - ext3
> /dev/md3 - "/var/lib/mysql" - RAID10 - xfs
>
> When doing the dist-upgrade and replacing the kernel + udev the system
> changes the RAID devices to md124, md125, md126 etc.
> As far as I have read, this makes sense - as mdadm is changing
> superblock and has to recalculate the whole RAID. This is big problem,
> if you have devices into /etc/fstab, like we do.

mdadm doesn't change superblocks in the manner you suggest - there isn't 
even an option yet to --update=metadata. I don't know Debian but it 
sounds like the dist-upgrade must be doing something odd. Your device 
names ought to be being preserved. Does doing a dist-upgrade involve a 
reboot part-way through? It ought to have your mdadm.conf in its initrd, 
or at the very least preserve your hostname, and maybe it doesn't.

The nearest I have seen is booting a system with a rescue CD when 
existing arrays get auto-assembled with high numbers (from 127 down) 
which can cause problems the next time you reboot back into the original 
system (because the on-disc metadata now thinks the array prefers to be 
e.g. md127 instead of md0). This is all to do with a missing mdadm.conf 
and/or the hostname not matching the arrays' homehost.

[...]
> If we re-check the labels we can see all the devices got the same
> label (the one assigned to md0, md1 etc), so probably grub is trying
> to boot from single devices, and not from the md0 which is /boot

Grub always boots from single devices. This is why you have to use 
metadata 0.90 or 1.0 with RAID-1 for /boot. Generally the BIOS and 
therefore grub will boot from the first available drive, so if your 
normal first drive dies, the second drive becomes the first and you can 
still boot. Having said that, grub2 has real support for md RAID so you 
can boot off any metadata and RAID level with it, as long as the BIOS 
can read grub off the first drive.

> This is blkid's output:
>
> /dev/sdb1: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
> SEC_TYPE="ext2" TYPE="ext3"
[...]
> /dev/md0: LABEL="boot" UUID="6d15111c-d2d9-4636-b2a5-7954ddcbcae2"
> SEC_TYPE="ext2" TYPE="ext3"
[...]
>
> As we have to migrate loads of machines, using labels is the fastest
> and cleanest way to do it. Just create the labels and change
> /etc/fstab across all the machines and voilá. But we cannot get it
> running. Is there any workaround to work with labels and avoid all the
> disks in the raid to get them as well?.

As you can see, it doesn't work. This will always happen with mdadm 
metadata version 0.90 and 1.0 with RAID-1 and is the only way grub can 
boot at all.

> Using UUID doesn't seem like a good idea, as they may change with
> future dist-upgrades, mdadm upgrades etc.

Using UUID won't work anyway, for the same reasons as using the label 
doesn't work.

> Has someone had same/similar problem with mdadm?

It's not really a mdadm problem, I think you need to find a way to fix 
whatever it is that's causing your arrays to be renamed/renumbered.

The only other thing that occurs to me is to check whether the upgrade 
process gives you a symlink like /dev/md/d0 which points to /dev/md127 
(and similarly for your original md1-3), and if it does, fix up your 
Lenny systems to have those symlinks and use them in your fstab.

Sorry I can't give any more positive help.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html