HELP - need more documentation on using partitioned (mdp-type) md devices

HELP - need more documentation on using partitioned (mdp-type) md devices

[Vinnie]

Hello all,

I am gradually getting through the process of setting up a partitioned 
RAID1 array, but having quite a time with it.

I applied Neil Brown's md-partitioning patches to my 2.4.18 kernel, and 
embarked on the journey.

I'm trying to get a pair of 40GB ATA100 IDE drives to work as a bootable
RAID1 array, as an mdp-type md device.

I'm actually getting pretty close now, I just can't get LILO (22.3.2) to 
write the boot sector to the array drives, pretty much any way I try to 
do it.

I'm not running /dev file system, since I've never been able to get that 
to work with my Red hat systems (though it seems like it would be really 
neat, if my systems wouldn't lock up after compiling a kernel with that 
feature enabled.  Are you supposed to delete all the entries in /dev, 
leaving just the /dev mount point, and let the kernel fill in the 
entries for you?).

So I had to manually create the devices.  I'll spare all the gruesome 
details of incorrect device names made along the way, experimenting with 
major and minor device numbers, etc.

Here's what I have under my /dev device tree for md-type entries:
(The first group, /dev/mdXX, were already there, I had to create the
rest by hand.  Well... by script... ;)

Can somebody who actually has this working (particularly if you were 
able to get LILO to write a boot record to the drives somehow) take a 
look at this list of devices and let me know what else needs to be 
listed under /dev?  From reading Neil's posts on the list, I may need 
md0p1 - md0p15 devices, etc., but not sure whether I should use major 9 
or major 60 for these.

MAIN md DEVICE NUMBERS
PERMS 
  UID	GID	MAJOR# 
MINOR# 
DEVICE
=================================================
brw-rw-- root	disk	9	0	/dev/md0
brw-rw-- root	disk	9	1	/dev/md1
...
brw-rw-- root	disk	9	31	/dev/md31

PARTITIONED MD DEVICES --
ASSIGNED sdX/hdX-STYLE DEVICE ID'S AND PARTITION #'S BY THE KERNEL WHEN 
THE md DEVICE STARTS - USE THIS DEVICE ID (/dev/mda, /dev/mdb) IN FDISK
(I had to make these by hand -- a 'cat /proc/devices' showed the device 
major number to be 60 for the mdp device on this machine)
PERMS 
  UID	GID	MAJOR# 
MINOR# 
DEVICE
=================================================
brw-rw-- root	disk	60	0	/dev/mda
brw-rw-- root	disk	60	1	/dev/mda1
...
brw-rw-- root	disk	60	15	/dev/mda15

(you would also need a set of /dev/mdb device numbers if you had a 
second partitioned md device, but I only have an mda right now):
PERMS 
  UID	GID	MAJOR# 
MINOR# 
DEVICE
=================================================
brw-rw-- root	disk	60	16	/dev/mdb
brw-rw-- root	disk	60	17	/dev/mdb1
...
brw-rw-- root	disk	60	31	/dev/mdb15

MDP-TYPE DEVICE AND PARTITION NUMBERS
I was starting to think I wouldn't need these until I actually got ext3 
filesystems created on the partitions, and booted up with these 
filesystems listed in /etc/fstab - journald apparently looks for these 
device names when starting the journals, per dmesg output.  But notice 
that the major and minor numbers are the same as used in the mdaXX 
devices above.
PERMS 
  UID	GID	MAJOR# 
MINOR# 
DEVICE
=================================================
brw-rw-- root	disk	60	0	/dev/mdp0
brw-rw-- root	disk	60	1	/dev/mdp0p1
...
brw-rw-- root	disk	60	15	/dev/mdp0p15

this would be for a 2nd partitioned md device, like mdb above:
brw-rw-- root	disk	60	16	/dev/mdp1
brw-rw-- root	disk	60	17	/dev/mdp1p1
...
brw-rw-- root	disk	60	31	/dev/mdp1p15


INTERESTING OBSERVATIONS SO FAR
1. Once you create the raid device ('mkraid /dev/md0') or otherwise 
start it, it is interesting that the drive geometry displays differently 
depending on whether you start fdisk with /dev/md0 or /dev/mda.   I 
finally realized that 'fdisk /dev/mda' is the logical way to do it, 
since the C/H/S displayed matches what the kernel sees the geometry to 
be on the pair of actual drives when the EIDE driver initializes.  But 
if you try to view a partition table you created under one device name, 
from the other device name, you'll probably see error messages about the 
partitions not ending on cylinder boundaries.  Creating the partition 
table with /dev/mda as the device name to launch with made the most 
sense, assuming the OS/kernel will talk to this drive as /dev/mda, and 
ext3 seems to like it just fine.


UP TO THIS POINT
By using an 'md=0,/dev/hde,/dev/hdg' added to the Grub command line for 
the kernel, the RAID1 array starts fine, and it will mount all the ext3 
filesystems I created partitions for on the mda device (since I have 
them all in /etc/fstab now, under a /ideraid mount point off the root fs 
on the main storage for the machine).

I did not have any luck with changing the partition types to FD for 
autodetection (it kept complaining about invalid superblocks) and was 
glad to find a good commandline example of manually specifying your md 
arrays on the kernel cmdline, in a previous post to the list.


PROBLEMS I CAN'T FIGURE OUT (HELP!!!)

LILO NOT COOPERATING
LILO 22.3.2 doesn't seem to like /dev/md0 as the boot= and disk= device. 
  Or /dev/mda, or /dev/mdp0 either.  I can specify /dev/hde or /dev/hdg 
(the actual IDE drives the RAID1 array is made of), but LILO won't 
consider it a RAID device in that case, and I didn't think you were 
supposed to try to write directly to a RAID member device by it's 
"direct name".

I also tried specifying /dev/mda1 (what will be the new /boot partition, 
marked bootable in partition table), but it still says 'Fatal: Not a 
partition name; no digits at the end'.

Currently the md0/mda partitioned RAID1 array is an add-on - I am using 
a large external SCSI-to-ATA100 RAID subsystem, which has a giant RAID5 
array presented as one big drive with 3 partitions - /dev/sda1 (/boot), 
/dev/sda2 (/), and /dev/sda3 (swap).  I'm trying to get everything moved 
over to this RAID1 pair of drives, but I want to make sure I can BOOT 
from the RAID1 md0/mda pair before I actually move everything over.  So 
in the meantime, I have a mountpoint on the main root fs called 
/ideraid, which is how I have mounted all the partitions of the md0/mda 
RAID1 array to experiment with.

If anybody (Neil?) can offer some advice on how to get LILO to write a 
boot sector to the RAID1 array pair, it would be greatly appreciated.  I 
am so close to getting this working!!!

I'm tempted to try stopping the md0 raid device, and fixing up lilo.conf 
to install LILO on /dev/hde (disk=/dev/hde and boot=/dev/hde), then 
start the array again and see if it can just sync the MBR/boot record 
from one drive to the other.

thanks,
vinnie

Re: HELP - need more documentation on using partitioned (mdp-type) md devices

[Neil Brown]

On Monday November 4, listacct1@lvwnet.com wrote:
> 
> If anybody (Neil?) can offer some advice on how to get LILO to write a 
> boot sector to the RAID1 array pair, it would be greatly appreciated.  I 
> am so close to getting this working!!!
> 

First, lilo thinks it knows what "/dev/md*" means, and gets all
confused when it turns out to be different.
So create device names that *don't* start /dev/md.
e.g.
   /dev/MDa == 60,0
   /dev/MDa1 == 60,1
   etc

Second, you have to tell lilo about the geometry of /dev/MD*, because
it doesn't know:

   boot=/dev/MDa
   disk=/dev/MDa
     bios=0x80
     sectors=63
     heads=255
     cylinders=1024
     partitions=/dev/MDa1
      start=63

The "start" is very important.  check what the start sector of MDa1
is, and use that.

Lilo should then work.

NeilBrown

Re: HELP - need more documentation on using partitioned (mdp-type) md devices

[Vinnie]

Neil Brown wrote:

 > On Monday November 4, listacct1@lvwnet.com wrote:
 >
 >>If anybody (Neil?) can offer some advice on how to get LILO to write a
 >>boot sector to the RAID1 array pair, it would be greatly appreciated.  I
 >>am so close to getting this working!!!
 >>
 >>
 >
 > First, lilo thinks it knows what "/dev/md*" means, and gets all
 > confused when it turns out to be different.
 > So create device names that *don't* start /dev/md.
 > e.g.
 >    /dev/MDa == 60,0
 >    /dev/MDa1 == 60,1
 >    etc
 >
 > Second, you have to tell lilo about the geometry of /dev/MD*, because
 > it doesn't know:
 >
 >    boot=/dev/MDa
 >    disk=/dev/MDa
 >      bios=0x80
 >      sectors=63
 >      heads=255
 >      cylinders=1024
 >      partitions=/dev/MDa1
 >       start=63
 >
 > The "start" is very important.  check what the start sector of MDa1
 > is, and use that.
 >
 > Lilo should then work.
 >
 > NeilBrown


thanks Neil!  Getting ready to try this out.  This looks really
promising, makes a lot of sense.

Vince

Re: HELP - need more documentation on using partitioned (mdp-type) md devices

[Vinnie]

Hi Neil!

Thanks for your help.  That did the trick!

I did have to change the partitions= to partition=, I didn't know any 
better but LILO did! ;)

Also once I got a successful LILO install (I did a -t -v first), and 
booted up, I got an error from LILO stating "EBDA Too Big".

I looked up EBDA in the README's and it says it is a thing that happens
sometimes with MultiProcessor motherboards, and suggested building LILO
with the LARGE_EBDA option.  I thought I'd try out just using the
'compact' statement in lilo.conf first, and that was enough to do it, I
got booted up successfully!

My test case told the kernel that the root fs was on the original 
external RAID subsystem, so really all I was testing was that I could 
actually boot from the RAID1 pair of IDE disks created with the 
partitioned md device.  Should be able to proceed on now to copying all 
files over to the filesystems on the RAID1 pair, so I can pull this 
external RAID subsystem offline and reconfigure it.

Thanks again Neil!  Couldn't have done it without your help!  Well, not 
in the next week or so, probably... ;)

vince


Vinnie wrote:

> Neil Brown wrote:
> 
>> On Monday November 4, listacct1@lvwnet.com wrote:
>>
>>> If anybody (Neil?) can offer some advice on how to get LILO to write 
>>> a boot sector to the RAID1 array pair, it would be greatly 
>>> appreciated.  I am so close to getting this working!!!
>>>
>>>
>>
>> First, lilo thinks it knows what "/dev/md*" means, and gets all
>> confused when it turns out to be different.
>> So create device names that *don't* start /dev/md.
>> e.g.
>>    /dev/MDa == 60,0
>>    /dev/MDa1 == 60,1
>>    etc
>>
>> Second, you have to tell lilo about the geometry of /dev/MD*, because
>> it doesn't know:
>>
>>    boot=/dev/MDa
>>    disk=/dev/MDa
>>      bios=0x80
>>      sectors=63
>>      heads=255
>>      cylinders=1024
>>      partitions=/dev/MDa1
>>       start=63
>>
>> The "start" is very important.  check what the start sector of MDa1
>> is, and use that.
>>
>> Lilo should then work.
>>
>> NeilBrown
> 
> 
> 
> thanks Neil!  Getting ready to try this out.  This looks really 
> promising, makes a lot of sense.
> 
> Vince
> 
> 
> 
>