Problems with RAID1 on cobalt raq3

Problems with RAID1 on cobalt raq3

[John P. Looney]

 I've been battling with getting RAID1 working on the root filesystem of a
Cobalt raq3 (running RedHat 7.3) for about a week.

 The current problem is that the Raq3 PROM will not mount a root
filesystem different to the one that it loads the kernel from initially.

 The PROM seems to use a variable "set_boot_dev" to get a partition name.
It mounts this partition, gets a file /boot/vmlinux.gz from it, unzips it,
and runs.

 I have hda2 (my original root partition) and hdg1 parts of a mirror setup
as /dev/md1, which is to be mounted as /. I used the "failed-disk"
directive to create md1 as / - however, after booting, md1 is mounted on /
fine, but I cannot add hda2 to this with raidhotadd:

[root@midir root]# raidhotadd /dev/md1 /dev/hda2                   
md: can not import hda2, has active inodes!
md: error, md_import_device() returned -16
/dev/md1: can not hot-add disk: invalid argument.

 Does anyone have any idea what could be happening, or how I could resolve
this ?

 I think I'm using lilo to some degree, as when the cobalt boots, it loads
a 2.2.16 kernel from it's PROM, and spits out:

VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 76k freed
Warning: unable to open an initial console.
BOOTLOADER: Mapping in physical locations
BOOTLOADER: load_addr=0xc1804000 ret_data=0xc1a050ac
BOOTLOADER: opening "/boot/vmlinux.gz"
BOOTLOADER: reading "/boot/vmlinux.gz"
BOOTLOADER: read 1254176bytes
BOOTLOADER: unmounting /
BOOTLOADER: calling reboot notifiers
stopping all md devices.
flushing ide devices: hdg 
disabling network interfaces.
resetting sym53c8xx scsi bus(es)
sym53c875-0: detaching ...
sym53c875-0: resetting chip
BOOTLOADER: mapping 16M-32M for ride home
BOOTLOADER: disabling interupts
BOOTLOADER: flushing cache
BOOTLOADER: Leap of faith!
Back in ramcode: done
Second stage kernel: Decompressing - done
  extract_header_info:
    ehead: 0x01500000
    shead: 0x01747eb0
    Machine: 0x00000003
    Kernel Entry: 0xc0105000
    Program header size: 0x00000020
    Number of Program headers: 1
    Section Header size: 0x00000028
    Number of section headers: 23
  relocate_and_zero:
    Offset: 0x00001000
    VAddr: 0xc0100000
    PAddr: 0xc0100000
    Filesize: 0x0023f6a0
    Memsize: 0x0027aa58
    Flags: 0x00000007
    relocate: src = 0x01501000, dest = 0x00100000, size = 0x0023f6a0
  Kernel cmd line: console=ttyS0,115200 ip=off 
  Booting kernel...
Linux version 2.4.18-4jp (root@jlooney) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.2BIOS-provided physical RAM map:

 But in case it is a problem with the lilo.conf:

     disk=/dev/md1
        bios=0x80
        sectors=63
        heads=16
        cylinders=39560

     boot=/dev/hda
     map=/boot/map
     install=/boot/boot.b
     image=/boot/boot/vmlinuz-2.4.18-3
         root=/dev/md1
         read-only
         label=LinuxRaid

 Should that work ?

John

Re: Problems with RAID1 on cobalt raq3

[Duncan Laurie]

On Fri, 16 Aug 2002 08:02:38 -0700, John P. Looney wrote:

>  I've been battling with getting RAID1 working on the root filesystem of a
> Cobalt raq3 (running RedHat 7.3) for about a week.
> 
>  The current problem is that the Raq3 PROM will not mount a root
> filesystem different to the one that it loads the kernel from initially.
> 
>  The PROM seems to use a variable "set_boot_dev" to get a partition name.
> It mounts this partition, gets a file /boot/vmlinux.gz from it, unzips it,
> and runs.
> 
>  I have hda2 (my original root partition) and hdg1 parts of a mirror setup
> as /dev/md1, which is to be mounted as /. I used the "failed-disk"
> directive to create md1 as / - however, after booting, md1 is mounted on /
> fine, but I cannot add hda2 to this with raidhotadd:
> 
> [root@midir root]# raidhotadd /dev/md1 /dev/hda2                   
> md: can not import hda2, has active inodes!
> md: error, md_import_device() returned -16
> /dev/md1: can not hot-add disk: invalid argument.
> 
>  Does anyone have any idea what could be happening, or how I could resolve
> this ?
> 
>  I think I'm using lilo to some degree, as when the cobalt boots, it loads
> a 2.2.16 kernel from it's PROM, and spits out:
> 

Nope, no LILO used in the cobalt bootloader.  You have it mostly figured
out though.  The variable written to with "set_boot_dev" is stored in
CMOS, and it is passed to the linux zero-page on boot as ORIG_ROOT_DEV.
(see linux-2.4/Documentation/i386/zero-page.txt)  However, this does not
have the same effect as passing "root=/dev/md1" to the kernel.

 

On your box, what is this variable set to?

> VFS: Mounted root (ext2 filesystem) readonly.
> Freeing unused kernel memory: 76k freed
> Warning: unable to open an initial console.
> BOOTLOADER: Mapping in physical locations
> BOOTLOADER: load_addr=0xc1804000 ret_data=0xc1a050ac
> BOOTLOADER: opening "/boot/vmlinux.gz"
> BOOTLOADER: reading "/boot/vmlinux.gz"
> BOOTLOADER: read 1254176bytes
> BOOTLOADER: unmounting /
> BOOTLOADER: calling reboot notifiers
> stopping all md devices.
> flushing ide devices: hdg 
> disabling network interfaces.
> resetting sym53c8xx scsi bus(es)
> sym53c875-0: detaching ...
> sym53c875-0: resetting chip
> BOOTLOADER: mapping 16M-32M for ride home
> BOOTLOADER: disabling interupts
> BOOTLOADER: flushing cache
> BOOTLOADER: Leap of faith!
> Back in ramcode: done
> Second stage kernel: Decompressing - done
>   extract_header_info:
>     ehead: 0x01500000
>     shead: 0x01747eb0
>     Machine: 0x00000003
>     Kernel Entry: 0xc0105000
>     Program header size: 0x00000020
>     Number of Program headers: 1
>     Section Header size: 0x00000028
>     Number of section headers: 23
>   relocate_and_zero:
>     Offset: 0x00001000
>     VAddr: 0xc0100000
>     PAddr: 0xc0100000
>     Filesize: 0x0023f6a0
>     Memsize: 0x0027aa58
>     Flags: 0x00000007
>     relocate: src = 0x01501000, dest = 0x00100000, size = 0x0023f6a0
>   Kernel cmd line: console=ttyS0,115200 ip=off 
>   Booting kernel...
> Linux version 2.4.18-4jp (root@jlooney) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.2BIOS-provided physical RAM map:
> 
>  But in case it is a problem with the lilo.conf:
> 
>      disk=/dev/md1
>         bios=0x80
>         sectors=63
>         heads=16
>         cylinders=39560
> 
>      boot=/dev/hda
>      map=/boot/map
>      install=/boot/boot.b
>      image=/boot/boot/vmlinuz-2.4.18-3
>          root=/dev/md1
>          read-only
>          label=LinuxRaid
> 
>  Should that work ?
> 
> John
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
 

Re: Problems with RAID1 on cobalt raq3

[Duncan Laurie]

On Fri, 16 Aug 2002 08:02:38 -0700, John P. Looney wrote:
>  I've been battling with getting RAID1 working on the root filesystem of a
> Cobalt raq3 (running RedHat 7.3) for about a week.
> 
>  The current problem is that the Raq3 PROM will not mount a root
> filesystem different to the one that it loads the kernel from initially.
> 
>  The PROM seems to use a variable "set_boot_dev" to get a partition name.
> It mounts this partition, gets a file /boot/vmlinux.gz from it, unzips it,
> and runs.
> 
>  I have hda2 (my original root partition) and hdg1 parts of a mirror setup
> as /dev/md1, which is to be mounted as /. I used the "failed-disk"
> directive to create md1 as / - however, after booting, md1 is mounted on /
> fine, but I cannot add hda2 to this with raidhotadd:
> 
> [root@midir root]# raidhotadd /dev/md1 /dev/hda2                   
> md: can not import hda2, has active inodes!
> md: error, md_import_device() returned -16
> /dev/md1: can not hot-add disk: invalid argument.
> 
>  Does anyone have any idea what could be happening, or how I could resolve
> this ?

Oops, that got sent before I was finished writing it,
I have a couple more questions..

1/ where did hdg come from?  Is there a 3rd party IDE controller
in the PCI slot?  (some RaQ3s had only 1 IDE connector populated..)
If it is, then you might run into problems because most likely the
ROM kernel does not have a driver for it.

2/ the ext3 filesystem might also be confusing the ROM kernel.
If you want to try one that will recognize it look here:

ftp://ftp-eng.cobalt.com/pub/users/duncan/kernel/rom/2.9.37-ext3.rom

This will also give you better control over the boot process,
adding a "set_lparams" function to the boot menu that specifies
the parameters specific to the ROM kernel.  ("set_params' will
pass parameters to the kernel that is loaded from disk)  This
will let you set a different root device to grab a kernel from.

It also has some more built-in PCI IDE drivers, so it might
help with problem #1. (if it is a problem...)

-duncan

Re: Problems with RAID1 on cobalt raq3

[John P. Looney]

 Duncan, thanks for responding! I was on the cobalt-developers list, but
they don't seem that interested in such problems.

 First off set_boot_dev was used to set the boot device to hda2 (I tryed
hda1, and as you pointed out, you can't move the root device easily to
hda2 later).

 I had originally planned that hda1 was /boot and hda2 was / - in the end,
to get *anything* booting, I'd to copy the /boot files into a /boot
directory on the / filesystem, and write off hda1 as a waste of disk
space.

On Sat, Aug 17, 2002 at 04:09:40PM -0700, Duncan Laurie mentioned:
> 1/ where did hdg come from?  Is there a 3rd party IDE controller
> in the PCI slot?  (some RaQ3s had only 1 IDE connector populated..)
> If it is, then you might run into problems because most likely the
> ROM kernel does not have a driver for it.

 It's a third-party ATA-66 disk controller. I wanted some sort of
reliability in the box, and as you said, this raq3 has only a single IDE
channel.

> 2/ the ext3 filesystem might also be confusing the ROM kernel.
> If you want to try one that will recognize it look here:
> ftp://ftp-eng.cobalt.com/pub/users/duncan/kernel/rom/2.9.37-ext3.rom

 Oh, thanks...I was using 2.3.40, and it seemed to boot the kernel off
ext3 fine (perhaps treating it as ext2).

> This will also give you better control over the boot process,
> adding a "set_lparams" function to the boot menu that specifies
> the parameters specific to the ROM kernel.  ("set_params' will
> pass parameters to the kernel that is loaded from disk)  This
> will let you set a different root device to grab a kernel from.

 Woo-hoo. Exactly what I wanted. I think.

 So, do I do 

     set_boot_dev hda1                      (/boot)
     set_params "root=/dev/md0 ro"         

 And that should do what I've been trying to do ? Or do I have to change
the paramters to the ROM kernel also ?

> It also has some more built-in PCI IDE drivers, so it might
> help with problem #1. (if it is a problem...)

 Well, it's not a problem just yet - though it certainly would be if hda
failed, and the next time, it had to boot from hdg. 

John

Re: Problems with RAID1 on cobalt raq3

[John P. Looney]

On Sat, Aug 17, 2002 at 03:55:38PM -0700, Duncan Laurie mentioned:
> > [root@midir root]# raidhotadd /dev/md1 /dev/hda2                   
> > md: can not import hda2, has active inodes!
> > md: error, md_import_device() returned -16
> > /dev/md1: can not hot-add disk: invalid argument.

 I've just tryed to get RAID1 going on another non-cobalt host, and I seem
to be doing something simple wrong, as I get something quite similar here
too:

[root@jny02 /root]# raidhotadd  /dev/md0 /dev/hda5
/dev/md0: can not hot-add disk: invalid argument.
md: trying to hot-add hda5 to md0 ... 
md: can not import hda5, has active inodes!
md: error, md_import_device() returned -16
[root@jny02 /root]# mount
/dev/md0 on / type ext2 (rw)
none on /proc type proc (rw)
/dev/hda1 on /boot type ext2 (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
 
 This is likely because to get it to boot, like the cobalt, I'm actually
passing root=/dev/hda5 to the kernel, not /dev/md0. 

 When I used "root=/dev/md0", the kernel booted, mounted root, and then
moaned that it couldn't find init (so I had to wonder what filesystem it
ended up mounting). I tryed to setup my lilo.conf from;
    http://www.tldp.org/HOWTO/mini/Boot+Root+Raid+LILO-5.html
but because I don't quite understand what it's doing, I could be doing
something wrong:

 My lilo.conf is:

boot=/dev/hda
map=/boot/map
install=/boot/boot.b
linear
disk=/dev/md0
    bios=0x80
    sectors=63
    heads=255
    cylinders=5005
partition=/dev/md1
    start=63
image=/boot/bzImage
    label=2.4.18
    read-only
    root=/dev/md0

 Is there anything obviously wrong with that ?
                                                        
John                                                    

Re: Problems with RAID1 on cobalt raq3

[Duncan Laurie]

On Mon, 2002-08-19 at 01:38, John P. Looney wrote: 
>
>  First off set_boot_dev was used to set the boot device to hda2 (I 
> tryed hda1, and as you pointed out, you can't move the root device
> easily to hda2 later).
> 
>  I had originally planned that hda1 was /boot and hda2 was / - in the end,
> to get *anything* booting, I'd to copy the /boot files into a /boot
> directory on the / filesystem, and write off hda1 as a waste of disk
> space.

I think you want this to be md1, but to do that and have the kernel on
a separate device (not a member of the raid set) will require you to 
override the root device for the ROM kernel with set_lparams. 

I was incorrect in saying that "set_boot_dev md1" does not have the same 
effect as "set_params root=/dev/md1", what I was seeing is actually a bug 
in the ROM.  When you run "set_boot_dev md1" it updates the major/minor 
bytes in CMOS, but it does not set the variable that is used to create 
the zero-page.  This means it will have the stale value until it is 
rebooted. (this is with a 2.9.x ROM, I am not sure about 2.3.40) 

>  Woo-hoo. Exactly what I wanted. I think.
> 
>  So, do I do 
> 
>      set_boot_dev hda1                      (/boot)
>      set_params "root=/dev/md0 ro"         
> 
>  And that should do what I've been trying to do ? Or do I have to change
> the paramters to the ROM kernel also ?

I would try this:

	set_boot_dev md1
	[reboot here - bug workaround]
	set_lparams root=/dev/hda1

(this is assuming hda1 is /boot, and hda2+hdg1=md1 is /)

The problem with this solution is you no longer get the ability to boot
autonomously.  The different root device is not saved in CMOS so has to
be re-entered each time.

-duncan

Re: Problems with RAID1 on cobalt raq3

[John P. Looney]

On Mon, Aug 19, 2002 at 03:08:16PM +0100, John P. Looney mentioned:
>  This is likely because to get it to boot, like the cobalt, I'm actually
> passing root=/dev/hda5 to the kernel, not /dev/md0. 

 Just to solve this...the reason I was booting the box with
root=/dev/hda5, not /dev/md0 was because /dev/md0 wasn't booting - it
would barf with 'can't find init'.

 It turns out that this is because I was populating md0 with tar. Which
seems to have 'issues' with crosslinked files - for instance, it was
trying to make a hard link of glibc.so to hda - and failing. It was only
as I did it again with a friend present, that he spotted the errors, and
queried them. We noticed that the hard linked files just didn't exist on
the new rootfs.

 When we duplicated the filesystems with dump instead of tar, it worked
fine, I was able to tell lilo to use root=/dev/md0 and everything worked.

 Woohoo.

Kate