Kernal Panic when booting redhat. Raid devices mount fine under Knoppix

Kernal Panic when booting redhat. Raid devices mount fine under Knoppix

[Brandon Evans]

Using rsync, I mirrored one server to some raid 1 partitions on  another 
server running knoppix.  The Raid devices mount fine
under knoppix 3.3, but when I try to boot Redhat 7.3, the servers can 
not read the superblock on the root device (/dev/md1), so the system panics.
   This has happened on 2 servers so far.

The error I am getting is this:

EXT3-fs: unable to read superblock
mount:error 22 mounting ext3 flags

There is also something like this:  pivotroot: pivot_root 
(/sysroot,/sysroot/initrd) failed 2.  But I left my notes at home, so 
I'm not sure
  that is exactly what the error says.  The same migration process 
worked fine in the test environment, but on the live servers,
with more data (20 Gigs), the server produce the above errors.

Why is Knoppix able to mount the root partition, but Redhat not?

To rsync, I used these steps: 
http://hosttuls.com/whitepapers/SyncServerMigration.txt

Raidtab:

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/hda1
     raid-disk     0
     device          /dev/hdd1
     raid-disk     1
raiddev             /dev/md1
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/hda3
     raid-disk     0
     device          /dev/hdd2
     raid-disk     1



Fstab:

/dev/md1        /       ext3    defaults,usrquota,grpquota      1       1
/dev/md0             /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /proc                   proc    defaults        0 0
none                    /dev/shm                tmpfs   defaults        0 0
/dev/hda2               swap                    swap    defaults        0 0
/dev/cdrom              /mnt/cdrom              iso9660 
noauto,owner,kudzu,ro 0 0
/dev/fd0                /mnt/floppy             auto 
noauto,owner,kudzu 0 0



Lilo Entry:

image=/boot/vmlinuz-2.4.20-20.7smp
         label=rh-2.4.20-20r
         initrd=/boot/initrd-2.4.20-20.7smp.raid.img
         read-only
         root=/dev/md1



df:
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              112G   26G   80G  24% /
none                  112G   26G   80G  24% /proc
/dev/md0               99M   28M   65M  30% /boot
none                  112G   26G   80G  24% /dev/pts
none                  112G   26G   80G  24% /dev/shm
/proc                 112G   26G   80G  24% /proc





-- 
     Brandon E.

Raid array not clean (its barely used and a few weeks old.. how can i find out what went wrong?)

[James R Bamford]

Ok,

I'm still trying to get this system actually setup and working and i noticed
this on my dmesg today..

md: autorun ...
md: considering hdg1 ...
md:  adding hdg1 ...
md:  adding hde1 ...
md: created md0
md: bind<hde1,1>
md: bind<hdg1,2>
md: running: <hdg1><hde1>
md: hdg1's event counter: 0000002d
md: hde1's event counter: 0000002d
md: md0: raid array is not clean -- starting background reconstruction
md: raid1 personality registered as nr 3
md0: max total readahead window set to 124k
md0: 1 data-disks, max readahead per data-disk: 124k
raid1: device hdg1 operational as mirror 1
raid1: device hde1 operational as mirror 0
raid1: raid set md0 not clean; reconstructing mirrors
raid1: raid set md0 active with 2 out of 2 mirrors
md: updating md0 RAID superblock on device
md: hdg1 [events: 0000002e]<6>(write) hdg1's sb offset: 156288256
md: syncing RAID array md0
md: minimum _guaranteed_ reconstruction speed: 100 KB/sec/disc.
md: using maximum available idle IO bandwith (but not more than 10000
KB/sec) for reconstruction.
md: using 124k window, over a total of 156288256 blocks.
md: hde1 [events: 0000002e]<6>(write) hde1's sb offset: 156288256
md: ... autorun DONE.


What does it mean, is it anythign to worry about?!? i've been trying to get
wake on lan to work and have been blindly powering off the machine, i am
pretty certain that this was done before it has a chance to boot into linux
(no monitor connected tho as its a backup server) .. anyways is this
indicative of any hdd errors? how can you get more info on what caused the
problems or can't you!?

With this reconstruction will i have all my files, some of them, none of
them!? does it choose one disk and basically the sync the other one up to
it!?

Cheers

Jim

RE: Raid array not clean (its barely used and a few weeks old.. how can i find out what went wrong?)

[Guy]

This is normal, if...  Your system crashed.
The system is syncing the mirror.
Try this command to see the status:
cat /proc/mdstat

If your system resyncs each time you re-boot then some is wrong.
If you re-boot before it is done, it will start over from the beginning.

The default max speed if 10000K per second.

Issue this command to change the speed now.
echo 100000 > /proc/sys/dev/raid/speed_limit_max

If you want the change to be permanent, add these 2 lines to
/etc/sysctl.conf
# RAID rebuild max speed per device
dev.raid.speed_limit_max = 100000

The speed can be whatever you want.  I use 100000, which is much faster than
possible for my system.  Oh!  100000 is 100000K bytes per second per device.
So if you have 2 disks, they will attempt 100Meg per second each!

Based you the size of your disks, the resync will take just over 2 hours, at
10000K per second.  Your disks should be much faster.

Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of James R Bamford
Sent: Wednesday, December 10, 2003 8:03 PM
To: Linux-raid maillist
Subject: Raid array not clean (its barely used and a few weeks old.. how can
i find out what went wrong?)

Ok,

I'm still trying to get this system actually setup and working and i noticed
this on my dmesg today..

md: autorun ...
md: considering hdg1 ...
md:  adding hdg1 ...
md:  adding hde1 ...
md: created md0
md: bind<hde1,1>
md: bind<hdg1,2>
md: running: <hdg1><hde1>
md: hdg1's event counter: 0000002d
md: hde1's event counter: 0000002d
md: md0: raid array is not clean -- starting background reconstruction
md: raid1 personality registered as nr 3
md0: max total readahead window set to 124k
md0: 1 data-disks, max readahead per data-disk: 124k
raid1: device hdg1 operational as mirror 1
raid1: device hde1 operational as mirror 0
raid1: raid set md0 not clean; reconstructing mirrors
raid1: raid set md0 active with 2 out of 2 mirrors
md: updating md0 RAID superblock on device
md: hdg1 [events: 0000002e]<6>(write) hdg1's sb offset: 156288256
md: syncing RAID array md0
md: minimum _guaranteed_ reconstruction speed: 100 KB/sec/disc.
md: using maximum available idle IO bandwith (but not more than 10000
KB/sec) for reconstruction.
md: using 124k window, over a total of 156288256 blocks.
md: hde1 [events: 0000002e]<6>(write) hde1's sb offset: 156288256
md: ... autorun DONE.


What does it mean, is it anythign to worry about?!? i've been trying to get
wake on lan to work and have been blindly powering off the machine, i am
pretty certain that this was done before it has a chance to boot into linux
(no monitor connected tho as its a backup server) .. anyways is this
indicative of any hdd errors? how can you get more info on what caused the
problems or can't you!?

With this reconstruction will i have all my files, some of them, none of
them!? does it choose one disk and basically the sync the other one up to
it!?

Cheers

Jim

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Raid array not clean (its barely used and a few weeks old.. how can i find out what went wrong?)

[James R Bamford]

Thanks Guy..

I'll keep an eye on it.. its a shame there is no log kept of all raid
errors, warnings, successes so you could look back and see.. another thing
is its a shame its not more visible that there is a problem.. obviously i
could get it so consoles show info on startup but this being hidden probably
meant i rebooted many a time whilst it was resyncing without finishing..

WHat is the scheme for a resync will it just make one drive equal the
other.. my data all looks intact

Cheers

Jim


> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org]On Behalf Of Guy
> Sent: 11 December 2003 01:50
> To: 'James R Bamford'; 'Linux-raid maillist'
> Subject: RE: Raid array not clean (its barely used and a few weeks old..
> how can i find out what went wrong?)
>
>
> This is normal, if...  Your system crashed.
> The system is syncing the mirror.
> Try this command to see the status:
> cat /proc/mdstat
>
> If your system resyncs each time you re-boot then some is wrong.
> If you re-boot before it is done, it will start over from the beginning.
>
> The default max speed if 10000K per second.
>
> Issue this command to change the speed now.
> echo 100000 > /proc/sys/dev/raid/speed_limit_max
>
> If you want the change to be permanent, add these 2 lines to
> /etc/sysctl.conf
> # RAID rebuild max speed per device
> dev.raid.speed_limit_max = 100000
>
> The speed can be whatever you want.  I use 100000, which is much
> faster than
> possible for my system.  Oh!  100000 is 100000K bytes per second
> per device.
> So if you have 2 disks, they will attempt 100Meg per second each!
>
> Based you the size of your disks, the resync will take just over
> 2 hours, at
> 10000K per second.  Your disks should be much faster.
>
> Guy
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of James R Bamford
> Sent: Wednesday, December 10, 2003 8:03 PM
> To: Linux-raid maillist
> Subject: Raid array not clean (its barely used and a few weeks
> old.. how can
> i find out what went wrong?)
>
> Ok,
>
> I'm still trying to get this system actually setup and working
> and i noticed
> this on my dmesg today..
>
> md: autorun ...
> md: considering hdg1 ...
> md:  adding hdg1 ...
> md:  adding hde1 ...
> md: created md0
> md: bind<hde1,1>
> md: bind<hdg1,2>
> md: running: <hdg1><hde1>
> md: hdg1's event counter: 0000002d
> md: hde1's event counter: 0000002d
> md: md0: raid array is not clean -- starting background reconstruction
> md: raid1 personality registered as nr 3
> md0: max total readahead window set to 124k
> md0: 1 data-disks, max readahead per data-disk: 124k
> raid1: device hdg1 operational as mirror 1
> raid1: device hde1 operational as mirror 0
> raid1: raid set md0 not clean; reconstructing mirrors
> raid1: raid set md0 active with 2 out of 2 mirrors
> md: updating md0 RAID superblock on device
> md: hdg1 [events: 0000002e]<6>(write) hdg1's sb offset: 156288256
> md: syncing RAID array md0
> md: minimum _guaranteed_ reconstruction speed: 100 KB/sec/disc.
> md: using maximum available idle IO bandwith (but not more than 10000
> KB/sec) for reconstruction.
> md: using 124k window, over a total of 156288256 blocks.
> md: hde1 [events: 0000002e]<6>(write) hde1's sb offset: 156288256
> md: ... autorun DONE.
>
>
> What does it mean, is it anythign to worry about?!? i've been
> trying to get
> wake on lan to work and have been blindly powering off the machine, i am
> pretty certain that this was done before it has a chance to boot
> into linux
> (no monitor connected tho as its a backup server) .. anyways is this
> indicative of any hdd errors? how can you get more info on what caused the
> problems or can't you!?
>
> With this reconstruction will i have all my files, some of them, none of
> them!? does it choose one disk and basically the sync the other one up to
> it!?
>
> Cheers
>
> Jim
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Raid array not clean (its barely used and a few weeks old.. how can i find out what went wrong?)

[Guy]

Look in /var/log/messages
Or
grep " md: " /var/log/messages|more

I don't know the details of a sync.  But for RAID1, it bet it's mostly just
a copy.  It does need to handle changes while syncing.  Just don't know how
it handles it.

Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of James R Bamford
Sent: Thursday, December 11, 2003 7:08 PM
To: Guy; 'Linux-raid maillist'
Subject: RE: Raid array not clean (its barely used and a few weeks old.. how
can i find out what went wrong?)

Thanks Guy..

I'll keep an eye on it.. its a shame there is no log kept of all raid
errors, warnings, successes so you could look back and see.. another thing
is its a shame its not more visible that there is a problem.. obviously i
could get it so consoles show info on startup but this being hidden probably
meant i rebooted many a time whilst it was resyncing without finishing..

WHat is the scheme for a resync will it just make one drive equal the
other.. my data all looks intact

Cheers

Jim


> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org]On Behalf Of Guy
> Sent: 11 December 2003 01:50
> To: 'James R Bamford'; 'Linux-raid maillist'
> Subject: RE: Raid array not clean (its barely used and a few weeks old..
> how can i find out what went wrong?)
>
>
> This is normal, if...  Your system crashed.
> The system is syncing the mirror.
> Try this command to see the status:
> cat /proc/mdstat
>
> If your system resyncs each time you re-boot then some is wrong.
> If you re-boot before it is done, it will start over from the beginning.
>
> The default max speed if 10000K per second.
>
> Issue this command to change the speed now.
> echo 100000 > /proc/sys/dev/raid/speed_limit_max
>
> If you want the change to be permanent, add these 2 lines to
> /etc/sysctl.conf
> # RAID rebuild max speed per device
> dev.raid.speed_limit_max = 100000
>
> The speed can be whatever you want.  I use 100000, which is much
> faster than
> possible for my system.  Oh!  100000 is 100000K bytes per second
> per device.
> So if you have 2 disks, they will attempt 100Meg per second each!
>
> Based you the size of your disks, the resync will take just over
> 2 hours, at
> 10000K per second.  Your disks should be much faster.
>
> Guy
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of James R Bamford
> Sent: Wednesday, December 10, 2003 8:03 PM
> To: Linux-raid maillist
> Subject: Raid array not clean (its barely used and a few weeks
> old.. how can
> i find out what went wrong?)
>
> Ok,
>
> I'm still trying to get this system actually setup and working
> and i noticed
> this on my dmesg today..
>
> md: autorun ...
> md: considering hdg1 ...
> md:  adding hdg1 ...
> md:  adding hde1 ...
> md: created md0
> md: bind<hde1,1>
> md: bind<hdg1,2>
> md: running: <hdg1><hde1>
> md: hdg1's event counter: 0000002d
> md: hde1's event counter: 0000002d
> md: md0: raid array is not clean -- starting background reconstruction
> md: raid1 personality registered as nr 3
> md0: max total readahead window set to 124k
> md0: 1 data-disks, max readahead per data-disk: 124k
> raid1: device hdg1 operational as mirror 1
> raid1: device hde1 operational as mirror 0
> raid1: raid set md0 not clean; reconstructing mirrors
> raid1: raid set md0 active with 2 out of 2 mirrors
> md: updating md0 RAID superblock on device
> md: hdg1 [events: 0000002e]<6>(write) hdg1's sb offset: 156288256
> md: syncing RAID array md0
> md: minimum _guaranteed_ reconstruction speed: 100 KB/sec/disc.
> md: using maximum available idle IO bandwith (but not more than 10000
> KB/sec) for reconstruction.
> md: using 124k window, over a total of 156288256 blocks.
> md: hde1 [events: 0000002e]<6>(write) hde1's sb offset: 156288256
> md: ... autorun DONE.
>
>
> What does it mean, is it anythign to worry about?!? i've been
> trying to get
> wake on lan to work and have been blindly powering off the machine, i am
> pretty certain that this was done before it has a chance to boot
> into linux
> (no monitor connected tho as its a backup server) .. anyways is this
> indicative of any hdd errors? how can you get more info on what caused the
> problems or can't you!?
>
> With this reconstruction will i have all my files, some of them, none of
> them!? does it choose one disk and basically the sync the other one up to
> it!?
>
> Cheers
>
> Jim
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernal Panic when booting redhat. Raid devices mount fine underKnoppix

[Brandon Evans]

Norman Schmidt wrote:
> Brandon Evans schrieb:
> 
>> The Raid devices mount fine
>> under knoppix 3.3, but when I try to boot Redhat 7.3, the servers can 
>> not read the superblock on the root device (/dev/md1), so the system 
>> panics.
>> This has happened on 2 servers so far.
> 
> 
> What partition types do the partitions hda1 hda2 hdd1 hdd2 have?
>
All the partitions are ext3



> Can it mount root (or does it give a kernel panic - unable to mount 
> root)? Could you give the exact boot error messages?

Root does not mount.  The exact errors are still on my notes that I 
again left at home.
But I will be down at the data center today trying it again, and I will 
post them

I'm thinking (hoping) that the fact that I didnt run resize2fs on the 
/dev/mdX devices is the problem.
If it isnt, then I don't know what else it could be.




-- 
Thanks,
     Brandon E.

Re: Kernal Panic when booting redhat. Raid devices mount fine underKnoppix

[Dan Egli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brandon Evans wrote:

| Norman Schmidt wrote:
|
|> Brandon Evans schrieb:
|>
|>> The Raid devices mount fine
|>> under knoppix 3.3, but when I try to boot Redhat 7.3, the servers can
|>> not read the superblock on the root device (/dev/md1), so the system
|>> panics.
|>> This has happened on 2 servers so far.
|>
|>
|>
|> What partition types do the partitions hda1 hda2 hdd1 hdd2 have?
|>
| All the partitions are ext3
|
|

I think he was referring to the partiton type code (i.e. linux native, linux
rain, linux raid autodetect, etc....)


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQE/2P0qtwT22Jak4/4RAgfUAKCSp41RGj1TbzT5eCQVaKvINyyKMQCfVMQm
lg3jg+eHf9+y5pmAW4T0v1M=
=Wl99
-----END PGP SIGNATURE-----

Re: Kernal Panic when booting redhat. Raid devices mount fine underKnoppix

[Brandon Evans]

Norman Schmidt wrote:
> Brandon Evans schrieb:

For the record, the error I was getting was:

EXT3-fs: unable to read superblock
mount:error 22 mounting ext3

> Are you sure that the partition type is not set to fd?

Yes, they are all set to fd.


> Then the problem might be the older mechanism of RedHAT to start the raids.
> 
> Which bootloader do you use? I use lilo for my servers, that works with 
> root on raid1.

Thanks for all the info.  It will come in handy.

I was finally able to get the / partition to boot by switching the fs 
type from ext3 to ext2.

I was able to boot in ext2, I switched the fs back to ext3, rebooted, 
and now everything works fine.
I'm curious as to what happened there, but that question is for another 
list.





-- 
Thanks,
     Brandon E.