raid, lvm, and reused disks crashing initramfs

raid, lvm, and reused disks crashing initramfs

[Andrew K. Bressen]

Hi--

I have a linux system running a pair of raid-1 drives, hda and hdc,
off of the motherboard IDE controller. 

Almost all the space on the disks is in md1. 
(There's a tiny md0 from an old configuration, the system used
to boot off of it, but now it's empty and doesn't get mounted).

On top of md1 is lvm, a single PV (horivg) 
with a bunch of logical volumes in it.

About a year ago, I replaced hda with a newer drive. 

Recently, wanting more disk space, I took the old hda and added
it back to the machine as hdc. I repartitioned, put lvm on it, 
creating a sigle PV (hori2vg) with two LVs in it. I copied files
onto it and such, and all was well. Until a power outage and reboot.

Then, the system crashed into initramfs. Exploring, it became
evident that initramfs was trying to mount the old disk
as root. This behavior occurred if the old disk was hdc or hdd. 

Remove the disk, system boots normally.

(Except that X11 no longer works. But I'm just going to choose
to believe that that's unrelated, even though I didn't change anything,
and even though that also occurred when the system was hosed a year 
ago by being booted with hda and hdc reversed.)

The system is an athlon running debian etch; 
kernel 2.6.22 (debian 2.6.22-3-k7 build).

I'd like to put that disk back in and have my system work. 

Any ideas about what's going on and how to fix it?

  --thanks for any clues
  --a k bressen

Re: raid, lvm, and reused disks crashing initramfs

[Bill Davidsen]

Andrew K. Bressen wrote:
> Hi--
>
> I have a linux system running a pair of raid-1 drives, hda and hdc,
> off of the motherboard IDE controller. 
>
> Almost all the space on the disks is in md1. 
> (There's a tiny md0 from an old configuration, the system used
> to boot off of it, but now it's empty and doesn't get mounted).
>
> On top of md1 is lvm, a single PV (horivg) 
> with a bunch of logical volumes in it.
>
> About a year ago, I replaced hda with a newer drive. 
>
> Recently, wanting more disk space, I took the old hda and added
> it back to the machine as hdc. I repartitioned, put lvm on it, 
> creating a sigle PV (hori2vg) with two LVs in it. I copied files
> onto it and such, and all was well. Until a power outage and reboot.
>
> Then, the system crashed into initramfs. Exploring, it became
> evident that initramfs was trying to mount the old disk
> as root. This behavior occurred if the old disk was hdc or hdd. 
>
> Remove the disk, system boots normally.
>
> (Except that X11 no longer works. But I'm just going to choose
> to believe that that's unrelated, even though I didn't change anything,
> and even though that also occurred when the system was hosed a year 
> ago by being booted with hda and hdc reversed.)
>
> The system is an athlon running debian etch; 
> kernel 2.6.22 (debian 2.6.22-3-k7 build).
>
> I'd like to put that disk back in and have my system work. 
>
> Any ideas about what's going on and how to fix it?
>   

You have stale configuration somewhere which causes the drive to be 
mounted. Do you have all of your mdadm mounts using UUID of the 
superblock? Did you zero the superblock(s) on the old drive? At this 
point you might have to zero the whole drive if you're already using 
UUID to start your arrays.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

uuids missing

[Andrew K. Bressen]

Bill Davidsen <davidsen@tmr.com> writes:
> You have stale configuration somewhere which causes the drive to be
> mounted.

I checked uuid's using blkid, I get the following: 

hda1 and hdc1, which are md0, have the same uuid, and md0 its own uuid. 

hda2 and hdc2, which are md1 (rootfs) HAVE NO UUID, nor does md1!
The pv and lv's on md1 do have uuid.

The old disk ALSO has no uuid on its partition (/dev/sde1), 
though it does have them for its pv and lv's. 

So the old disk, the root raid partitions, and the md device for root,
all have no uuids. How do I add them? Did they used to be there
and need to be recovered from someplace? 

(in re questions earlier, 1: I don't use uuid's in my fstab, 
I use /dev/mapper/vgname-lvname, but I don't think that can be relavent 
since the problem occurs in initramfs before the filesystem 
containing the fstab gets mounted. 2: I did not zero
anything on the old drive because I didn't think anything would survive
a repartition, pvcreate, lvcreate, and putting stuff on the drive)

  --thanks much for more clues!
  --akb

Re: uuids missing

[Andrew K. Bressen]

To recap config:
md0, a  60MB unused raid on hda1 and hdc1
md1, a 300GB raid with root fs on hda2 and hdc2

/dev/sde, not a raid disk. It used to be a raid disk, but went through
repartition, pvcreate, and had a bunch of lv's made on it. 
The system doesn't try to assemble it into a raid device. 

With sde on motherboard controller (as hdb or hdd), 
initramfs crashes trying to mount it as root. 

With sde in a usb shoebox (sde), system boots fine, 
including starting & mounting volumes on sde. 

blkid shows a uuid for hda1 and hdc1 and md0
mdadm shows a uuid for md0 that is _different_ from the uuid reported by blkid

Why on earth would mdadm uuid's and blkid uuid's differ on an md device? 

blkid does not show a uuid for hda2 and hdc2 and md1; what the heck? 
mdadm shows a uuid for md1.

Why doesn't blkid show uuid's for these devices? 

blkid does not show a uuid for sde1
that doesn't seem odd since it is not raid'ed.
pvscan does show a uuid for sde1, which makes sense since the pv
probably has it's own uuid. 

I do not believe I am using the old raidtools.

> You *might* find joy through changing the partition type of the old
> disk if it's old self-assembling raid. Maybe retype it from fd (Linux

Disktype is linux, and I also tried it as type swap. 
I can  try it as lvm later, but if it initramfs tried to mount
a swap disk as root, I suspect partition label doesn't figure
in to this. Though I certainly could be wrong on that. 

>> So the old disk, the root raid partitions, and the md device for root,
>> all have no uuids. How do I add them? Did they used to be there
>> and need to be recovered from someplace?

> No, I would worry about changing it now, maybe Neil has a clue what's
> safe. For now let's see if the partition type change fixes things.

  --thanks for the continued thoughts!
  --akb