[linux-lvm] Boot with LVM on RAID1 on entire disk

[linux-lvm] Boot with LVM on RAID1 on entire disk

[Ralph Jennings]

I would like to be able to RAID mirror my entire hard drive,
and run LVM on top of that (/, /boot, /usr, swap, MBR).  So
that if 1 of them dies, I only have to take out the good one,
put it in the hda rack, and reboot.

First off, I need the patch to Lilo to allow /, and /boot to
be LVM partitions.  Can somebody please point me to this?

Second, I need a way to tell pvcreate or lvcreate not to mess
with the MBR.  Some sort of paramater to one of them that I
can use to tell it not to use the first 512 bytes (or 4K, or
whatever it needs).  Or alternatively, some way to create an
LV which covers the MBR, which I can keep blank, by just not
writting to it.  I also saw mention of a patch for this in
the list before, but to an old version of LVM.  If this exists
for LVM 0.9.1b5, and kernel 2.4.2 can somebody point me to
this also?

Thanks, Ralph.

[linux-lvm] Boot with LVM on RAID1 on entire disk

[Ralph Jennings]

Apologies, first post was accedentaly sent as a thread to an
unrelated thread:

------------------------------------------------------------

I would like to be able to RAID mirror my entire hard drive,
and run LVM on top of that (/, /boot, /usr, swap, MBR).  So
that if 1 of them dies, I only have to take out the good one,
put it in the hda rack, and reboot.

First off, I need the patch to Lilo to allow /, and /boot to
be LVM partitions.  Can somebody please point me to this?

Second, I need a way to tell pvcreate or lvcreate not to mess
with the MBR.  Some sort of paramater to one of them that I
can use to tell it not to use the first 512 bytes (or 4K, or
whatever it needs).  Or alternatively, some way to create an
LV which covers the MBR, which I can keep blank, by just not
writting to it.  I also saw mention of a patch for this in
the list before, but to an old version of LVM.  If this exists
for LVM 0.9.1b5, and kernel 2.4.2 can somebody point me to
this also?

Is anybody working on getting this functionallity put into the
LVM source tree?  If not, any pointers on how I could do this?
(not sure if I could, but if nobody else is, I could allways
take a look and try)

Thanks, Ralph.

Re: [linux-lvm] Boot with LVM on RAID1 on entire disk

[Andreas Dilger]

Ralph writes:
> I would like to be able to RAID mirror my entire hard drive,
> and run LVM on top of that (/, /boot, /usr, swap, MBR).  So
> that if 1 of them dies, I only have to take out the good one,
> put it in the hda rack, and reboot.

Are you using MD RAID or hardware RAID?

> First off, I need the patch to Lilo to allow /, and /boot to
> be LVM partitions.  Can somebody please point me to this?

You need to boot from initrd (using lvmcreate_initrd to build it)
to have / in LVM.  My (relatively untested) LILO patch is at the
end.

> Second, I need a way to tell pvcreate or lvcreate not to mess
> with the MBR.  Some sort of paramater to one of them that I
> can use to tell it not to use the first 512 bytes (or 4K, or
> whatever it needs).  Or alternatively, some way to create an
> LV which covers the MBR, which I can keep blank, by just not
> writting to it.  I also saw mention of a patch for this in
> the list before, but to an old version of LVM.  If this exists
> for LVM 0.9.1b5, and kernel 2.4.2 can somebody point me to
> this also?

If you have disk partitions, LVM will not mess with the MBR.  If
you pvcreate on a whole disk, then it will overwrite the MBR.
lvcreate is completely irrelevant in this case.  If you really
need to leave the MBR alone, simply create a partition which is
the size of the whole disk, and then pvcreate that partition.

Unless you are terribly unlucky with the disk size, you will not lose any
usable disk space.  LVM can only utilize full multiples of the PE size
(4MB by default), and the LVM metadata (a few hundred kB) usually fits
in the remainder, leaving a extra unused space.  In the future we may
use some of this extra space for LVM metadata backups.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Boot with LVM on RAID1 on entire disk

[Andreas Dilger]

PS - here is that LILO patch I promised.

Note that I don't think it works with LVM on MD yet (I don't use/test MD),
although there is no real reason you couldn't do that.  You definitely can
NOT run a /boot partition on a striped MD or LVM device.  MD MUST be RAID 1
only and not RAID 0 or RAID 5.

Cheers, Andreas
============================================================================
diff -ru lilo-21.6.1/geometry.c lilo-21.6.1a/geometry.c
--- lilo-21.6.1/geometry.c	Sat Dec 16 15:09:21 2000
+++ lilo-21.6.1a/geometry.c	Tue Jan 30 13:40:02 2001
@@ -39,6 +39,15 @@
 #endif
 #endif
 
+struct lv_bmap {
+    __u32 lv_block;
+    __u32 lv_dev;
+};
+
+#ifndef LV_BMAP
+#define LV_BMAP				_IOWR(0xfe, 0x30, 1)
+#endif
+
 #ifndef HDIO_GETGEO
 #define HDIO_GETGEO HDIO_REQ
 #endif
@@ -265,6 +274,22 @@
 }
 
 
+void lvm_bmap(struct lv_bmap *lbm)
+{
+    DEVICE dev;
+    static int lvmfd = -1;
+
+    if (lvmfd == -1) {
+	lvmfd = dev_open(&dev, lbm->lv_dev, O_RDONLY);
+	if (lvmfd < 0)
+		die("can't open LVM device %#x\n", lbm->lv_dev);
+    }
+    if (ioctl(lvmfd, LV_BMAP, lbm) < 0) {
+	perror(__FUNCTION__);
+	pdie("LV_BMAP error or ioctl unsupported, can't have image in LVM.\n");
+    }
+}
+
 void geo_query_dev(GEOMETRY *geo,int device,int all)
 {
     DEVICE dev;
@@ -421,7 +450,9 @@
     DT_ENTRY *walk;
     int inherited,keep_cyls;
 
-    if (MAJOR(device) == MD_MAJOR) {
+    switch (MAJOR(device)) {
+    case MD_MAJOR:
+    {
         char mdxxx[11];
 	int md_fd, pass;
 	md_array_info_t md_array_info;
@@ -446,6 +477,19 @@
 	    }
 	}
 	close(md_fd);
+	break;
+    }
+    case MAJOR_LVM:
+    {
+	struct lv_bmap lbm;
+
+	lbm.lv_dev = device;
+	lbm.lv_block = 0;
+
+	lvm_bmap(&lbm);
+	device = lbm.lv_dev;
+	break;
+    }
     }
     for (walk = disktab; walk; walk = walk->next)
 	if (walk->device == device) break;
@@ -523,7 +567,8 @@
     if (fstat(geo->fd,&st) < 0) die("fstat %s: %s",name,strerror(errno));
     if (!S_ISREG(st.st_mode) && !S_ISBLK(st.st_mode))
 	die("%s: neither a reg. file nor a block dev.",name);
-    geo_get(geo,S_ISREG(st.st_mode) ? st.st_dev : st.st_rdev,user_dev,1);
+    geo->dev = S_ISREG(st.st_mode) ? st.st_dev : st.st_rdev;
+    geo_get(geo, geo->dev, user_dev, 1);
     geo->file = S_ISREG(st.st_mode);
     geo->boot = 0;
 #ifndef FIGETBSZ
@@ -549,16 +594,15 @@
 int geo_open_boot(GEOMETRY *geo,char *name)
 {
     struct stat st;
-    dev_t dev;
 
     if (stat(name,&st) < 0) die("stat %s: %s",name,strerror(errno));
     if (!S_ISREG(st.st_mode) && !S_ISBLK(st.st_mode))
 	die("%s: neither a reg. file nor a block dev.",name);
-    dev = S_ISREG(st.st_mode) ? st.st_dev : st.st_rdev;
-    if (MAJOR(dev) == MAJOR_FD) geo->fd = 0;
+    geo->dev = S_ISREG(st.st_mode) ? st.st_dev : st.st_rdev;
+    if (MAJOR(geo->dev) == MAJOR_FD) geo->fd = 0;
     else if ((geo->fd = open(name,O_NOACCESS)) < 0)
 	    die("open %s: %s",name,strerror(errno));
-    geo_get(geo,dev,-1,0);
+    geo_get(geo, geo->dev, -1, 0);
     geo->file = S_ISREG(st.st_mode);
     geo->boot = 1;
     geo->spb = 1;
@@ -600,6 +643,15 @@
 	if (ioctl(geo->fd,FIBMAP,&block) < 0) pdie("ioctl FIBMAP");
 	if (!block) {
 	    return 0;
+	}
+	if (MAJOR(geo->dev) == MAJOR_LVM) {
+	    struct lv_bmap lbm;
+
+	    lbm.lv_dev = geo->dev;
+	    lbm.lv_block = block;
+
+	    lvm_bmap(&lbm);
+	    block = lbm.lv_block;
 	}
     }
     sector = block*geo->spb+((offset/SECTOR_SIZE) % geo->spb);
diff -ru lilo-21.6.1/geometry.h lilo-21.6.1a/geometry.h
--- lilo-21.6.1/geometry.h	Sat Nov  6 14:09:40 1999
+++ lilo-21.6.1a/geometry.h	Tue Jan 23 18:16:09 2001
@@ -16,6 +16,7 @@
     int spb; /* sectors per block */
     int fd,file;
     int boot; /* non-zero after geo_open_boot */
+    dev_t dev;
 } GEOMETRY;
 
 typedef struct _dt_entry {
diff -ru lilo-21.6.1/lilo.h lilo-21.6.1a/lilo.h
--- lilo-21.6.1/lilo.h	Thu Sep 21 17:18:50 2000
+++ lilo-21.6.1a/lilo.h	Tue Jan 23 14:49:59 2001
@@ -39,8 +39,9 @@
 #define MAJOR_IDE4	34 /* IDE on fourth interface */
 #define MAJOR_ESDI	36 /* PS/2 ESDI drives */
 #define MAJOR_DAC960	48 /* First Mylex DAC960 PCI RAID controller */
-#define MAJOR_IDE5	55 /* IDE on fifth interface */
+#define MAJOR_IDE5	56 /* IDE on fifth interface */
 #define MAJOR_IDE6	57 /* IDE on sixth interface */
+#define MAJOR_LVM	58 /* Logical Volume Manager devices */
 #define COMPAQ_SMART2_MAJOR	72 /* First Smart/2 Major */
 
 #define MAX_IMAGES      ((SECTOR_SIZE*2-2)/sizeof(IMAGE_DESCR))


-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Boot with LVM on RAID1 on entire disk

[Ralph Jennings]

On Mon, Feb 26, 2001 at 03:35:38PM -0700, Andreas Dilger wrote:
> Ralph writes:
> > I would like to be able to RAID mirror my entire hard drive,
> > and run LVM on top of that (/, /boot, /usr, swap, MBR).  So
> > that if 1 of them dies, I only have to take out the good one,
> > put it in the hda rack, and reboot.
> 
> Are you using MD RAID or hardware RAID?

Software (md) RAID1 (mirroring).

> > First off, I need the patch to Lilo to allow /, and /boot to
> > be LVM partitions.  Can somebody please point me to this?
> 
> You need to boot from initrd (using lvmcreate_initrd to build it)
> to have / in LVM.  My (relatively untested) LILO patch is at the
> end.

That's fine, thanks much.

> > Second, I need a way to tell pvcreate or lvcreate not to mess
> > with the MBR.  Some sort of paramater to one of them that I
> > can use to tell it not to use the first 512 bytes (or 4K, or
> > whatever it needs).  Or alternatively, some way to create an
> > LV which covers the MBR, which I can keep blank, by just not
> > writting to it.  I also saw mention of a patch for this in
> > the list before, but to an old version of LVM.  If this exists
> > for LVM 0.9.1b5, and kernel 2.4.2 can somebody point me to
> > this also?
> 
> If you have disk partitions, LVM will not mess with the MBR.  If
> you pvcreate on a whole disk, then it will overwrite the MBR.
> lvcreate is completely irrelevant in this case.  If you really
> need to leave the MBR alone, simply create a partition which is
> the size of the whole disk, and then pvcreate that partition.

I don't want disk partitions, I want md0 to include the MBR
so that when a disk crashes (say hda), I just swap the hard
drive that is working (say hdc) into hda (I have HD racks to
easily swap them with), and raidhotadd a new good HD.  Rather
than swap the HD, make a boot floppy from a system that's not
crashed (which half the time requires me to build a kernel
as well), re-run lilo, and reboot again, then raidhotadd the
new good HD (to replace the one that crashed).

> Unless you are terribly unlucky with the disk size, you will not lose any
> usable disk space.  LVM can only utilize full multiples of the PE size
> (4MB by default), and the LVM metadata (a few hundred kB) usually fits
> in the remainder, leaving a extra unused space.  In the future we may
> use some of this extra space for LVM metadata backups.

OK, so I create an MBR LVM partition that is 4MB big, and
just never use it, I can live with that.  But it seems the
LVM stuff is at the front of the disk (as opposed to MD
stuff being at the end of the disk), so the LVM stuff just
overwrites the MBR anyway.

Has anybody got any hints for this?  I saw mention of a
patch that just ignored the first few KB of the hard drive
but it was for an old version (2.2 kernel).

Thank much, Ralph.

Re: [linux-lvm] Boot with LVM on RAID1 on entire disk

[Andreas Dilger]

Ralph writes:
> On Mon, Feb 26, 2001 at 03:35:38PM -0700, Andreas Dilger wrote:
> > Ralph writes:
> > > I would like to be able to RAID mirror my entire hard drive,
> > > and run LVM on top of that (/, /boot, /usr, swap, MBR).  So
> > > that if 1 of them dies, I only have to take out the good one,
> > > put it in the hda rack, and reboot.
> > 
> > Are you using MD RAID or hardware RAID?
> 
> Software (md) RAID1 (mirroring).

OK, it is possible to build LVM on MD as well.

> > > First off, I need the patch to Lilo to allow /, and /boot to
> > > be LVM partitions.  Can somebody please point me to this?
> > 
> > You need to boot from initrd (using lvmcreate_initrd to build it)
> > to have / in LVM.  My (relatively untested) LILO patch is at the
> > end.
> 
> That's fine, thanks much.

One thing that probably _doesn't_ work is /boot on LVM on MD, just because
I haven't had a need to do so.  Unless you have unusual requirements,
you can leave /boot as a partition if you like, or you can fix LILO
based on the patch I sent.

> I don't want disk partitions, I want md0 to include the MBR
> so that when a disk crashes (say hda), I just swap the hard
> drive that is working (say hdc) into hda (I have HD racks to
> easily swap them with), and raidhotadd a new good HD.  Rather
> than swap the HD, make a boot floppy from a system that's not
> crashed (which half the time requires me to build a kernel
> as well), re-run lilo, and reboot again, then raidhotadd the
> new good HD (to replace the one that crashed).

I see what you are getting at, but unfortunately LVM can't do this, and
it would be an incompatible change to the LVM setup.  If you really want
to do this, you _could_ code up a simple utility which copies the MBR
from the good disk to the new disk, and still leave LVM in a partition.
Probably a script doing "dd if=/dev/hda of=/dev/hdc bs=512 count=1"
is enough for a single-partition disk (don't quote me because I don't
know the internals of DOS parition tables).

You could do this at the same time you run the RAID resync, or even add
a new option to LILO to copy the MBR to another disk "mirror=/dev/hdc"
or something like that.  This avoids the need to rerun LILO whenever
you replace a disk.

> OK, so I create an MBR LVM partition that is 4MB big, and
> just never use it, I can live with that.  But it seems the
> LVM stuff is at the front of the disk (as opposed to MD
> stuff being at the end of the disk), so the LVM stuff just
> overwrites the MBR anyway.

The LVM PV information is at the beginning, so creating a "don't use"
LV doesn't help (you couldn't create it in the right spot anyways.

> Has anybody got any hints for this?  I saw mention of a
> patch that just ignored the first few KB of the hard drive
> but it was for an old version (2.2 kernel).

Haven't seen it, but it would be different than any other LVM system,
so you would need to patch all of your tools.  I think the "dd" copy of
the MBR should meet your needs.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Boot with LVM on RAID1 on entire disk

[Russell Coker]

On Tuesday 27 February 2001 03:07, Andreas Dilger wrote:
> One thing that probably _doesn't_ work is /boot on LVM on MD, just because
> I haven't had a need to do so.  Unless you have unusual requirements,
> you can leave /boot as a partition if you like, or you can fix LILO
> based on the patch I sent.

I suggest the following:
/dev/hda1 - 20M /boot
/dev/hda2 - rest of the disk

Same partitioning for /dev/hdc .

Then make /dev/hda2 and /dev/hdc2 a software RAID mirror which has LVM on top 
(thus leaving the MBR nice and free).

Install the Debian MBR (I'm sure it's available under other names for other 
distributions) on both /dev/hda and /dev/hdc.  It'll load a LILO boot block 
from an active partition on the same drive by default.
Make /dev/hda1 and /dev/hdc1 active using fdisk or the LILO "activate" 
program.
Then after running "lilo" do "cat /dev/hda1 > /dev/hdc1".  At this stage if 
/dev/hda fails then swapping /dev/hdc to /dev/hda should result in a bootable 
system!

I have CC'd John Coffman (current Lilo developer) on this email as I think 
it'll interest him.  Please remove his name from the CC list if you are going 
to make a non-LILO posting.

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page