* LVM & RAID10
@ 2008-11-25 1:08 Wojciech Pyczak
2008-11-25 2:52 ` Bean
0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-11-25 1:08 UTC (permalink / raw)
To: grub-devel
Hello!
I have some troubles setting up grub2 on my configuration. After I
bought a new computer I decided to "go wild" and setup everything on top
of LVM, with LVM itself on RAID array. I expected there will be some
problems and here I am :)
Few words about configuration - I have 2x WD 640GB HDDs, there is single
partition on each drive, those partitions are used to build RAID10
array. Aray itself consists of 3 drives (third one is "missing" since I
use it in my "old" computer), like I mentioned before it's level 10,
with 256kB chunks and far2 layuout. On top of that there is LVM2,
consisting of single "storage" group (32MB extents), divided into few
logical volumes, including volumes named "root" (/) and "boot" (/boot).
I used latest RescueCD to made this up and installed fresh gentoo. I
managed to build entire system, I only needed to boot it. So, I emerged
(using my own custom ebuild) Grub2 SVN version (successfully) and tried
to make everythig work but it appers life is not so easy :P. The only
clue I have is (completly mysterius to me) error message: "We don't
support multiple metadata areas".
AFAIK both LVM and RAID10 are supported and it's possible to boot grub
from LVM volume, am I wrong ?
Or perhaps I'm missing something ?
Is there any example of LVM/RAID config files ?
What happens if array (RAID 5, 6, 10) is incomplete (disk
failure/missing) - maybe that's the problem ? And in that case, how I'm
supposed to boot system in such situations ?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: LVM & RAID10
2008-11-25 1:08 LVM & RAID10 Wojciech Pyczak
@ 2008-11-25 2:52 ` Bean
2008-12-03 22:32 ` Wojciech Pyczak
0 siblings, 1 reply; 6+ messages in thread
From: Bean @ 2008-11-25 2:52 UTC (permalink / raw)
To: The development of GRUB 2
On Tue, Nov 25, 2008 at 9:08 AM, Wojciech Pyczak <nissarin@gmail.com> wrote:
> Hello!
>
> I have some troubles setting up grub2 on my configuration. After I bought a
> new computer I decided to "go wild" and setup everything on top of LVM, with
> LVM itself on RAID array. I expected there will be some problems and here I
> am :)
>
> Few words about configuration - I have 2x WD 640GB HDDs, there is single
> partition on each drive, those partitions are used to build RAID10 array.
> Aray itself consists of 3 drives (third one is "missing" since I use it in
> my "old" computer), like I mentioned before it's level 10, with 256kB chunks
> and far2 layuout. On top of that there is LVM2, consisting of single
> "storage" group (32MB extents), divided into few logical volumes, including
> volumes named "root" (/) and "boot" (/boot).
>
> I used latest RescueCD to made this up and installed fresh gentoo. I managed
> to build entire system, I only needed to boot it. So, I emerged (using my
> own custom ebuild) Grub2 SVN version (successfully) and tried to make
> everythig work but it appers life is not so easy :P. The only clue I have is
> (completly mysterius to me) error message: "We don't support multiple
> metadata areas".
Hi,
This error come from lvm, perhaps you can try raid10 first.
>
> AFAIK both LVM and RAID10 are supported and it's possible to boot grub from
> LVM volume, am I wrong ?
> Or perhaps I'm missing something ?
> Is there any example of LVM/RAID config files ?
> What happens if array (RAID 5, 6, 10) is incomplete (disk failure/missing) -
> maybe that's the problem ? And in that case, how I'm supposed to boot system
> in such situations ?
Yes, it should work out of the box, although there is a small catch.
If you're using raid5/raid6 and one/two of the disk is missing or
corrupted, you need to add module raid5rec/raid6rec using the --module
option in grub-install.
--
Bean
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: LVM & RAID10
2008-11-25 2:52 ` Bean
@ 2008-12-03 22:32 ` Wojciech Pyczak
2008-12-04 4:38 ` Bean
0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-12-03 22:32 UTC (permalink / raw)
To: The development of GRUB 2
>> (...)The only clue I have is
>> (completly mysterius to me) error message: "We don't support multiple
>> metadata areas".
>>
>
> Hi,
>
> This error come from lvm, perhaps you can try raid10 first.
>
>
I played a bit when I had free time and...
- I learned to ignore grub error messages :D
- RAID10 seems to work OK
- LVM @ RAID10 @ single partition (on each disk) results in above
error message but if I add small partition (so there are 2 partitions
per disk) there is no error. I tried with different number of LVs,
different LV names, anything.. could it be some error in grub after all ?
Anyway even if there is no such error grub doesn't work - it appers
that core.img doesn't include LVM module (I tried to specify it
manually) or just doesn't install at all, of course grub-setup return
some error (unknown root device or something..) but in the other hand I
saw the same error with root filesystem on RAID device and it actually
worked.
There is also another problem: when I installed grub on single HD
with LVM module in core.img, I couldn't see any LVM devices in the
system (only hdX, mdX..), I saw something similar on some mailing list
(debian ?) - removing LVM module then inserting it again fixed that
problem, however in my case it didn't worked.
I'm thinking about installing grub by hand, i.e. creating core.img
and copying it along with boot.img after mbr, will that work ? (I just
want to bypass grub-setup).
>
> Yes, it should work out of the box, although there is a small catch.
> If you're using raid5/raid6 and one/two of the disk is missing or
> corrupted, you need to add module raid5rec/raid6rec using the --module
> option in grub-install.
>
Just curiosity - why it's in separate modules ?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: LVM & RAID10
2008-12-03 22:32 ` Wojciech Pyczak
@ 2008-12-04 4:38 ` Bean
2008-12-05 21:11 ` Wojciech Pyczak
0 siblings, 1 reply; 6+ messages in thread
From: Bean @ 2008-12-04 4:38 UTC (permalink / raw)
To: The development of GRUB 2
On Thu, Dec 4, 2008 at 6:32 AM, Wojciech Pyczak <nissarin@gmail.com> wrote:
>
>>> (...)The only clue I have is
>>> (completly mysterius to me) error message: "We don't support multiple
>>> metadata areas".
>>>
>>
>> Hi,
>>
>> This error come from lvm, perhaps you can try raid10 first.
>>
>>
>
> I played a bit when I had free time and...
> - I learned to ignore grub error messages :D
> - RAID10 seems to work OK
> - LVM @ RAID10 @ single partition (on each disk) results in above error
> message but if I add small partition (so there are 2 partitions per disk)
> there is no error. I tried with different number of LVs, different LV names,
> anything.. could it be some error in grub after all ?
>
> Anyway even if there is no such error grub doesn't work - it appers that
> core.img doesn't include LVM module (I tried to specify it manually) or just
> doesn't install at all, of course grub-setup return some error (unknown root
> device or something..) but in the other hand I saw the same error with root
> filesystem on RAID device and it actually worked.
> There is also another problem: when I installed grub on single HD with
> LVM module in core.img, I couldn't see any LVM devices in the system (only
> hdX, mdX..), I saw something similar on some mailing list (debian ?) -
> removing LVM module then inserting it again fixed that problem, however in
> my case it didn't worked.
> I'm thinking about installing grub by hand, i.e. creating core.img and
> copying it along with boot.img after mbr, will that work ? (I just want to
> bypass grub-setup).
Hi,
Perhaps you can describe the steps used to create the LVM volume,
along with version number of tools. Let me see if the error can be
recreated in qemu.
>
>>
>> Yes, it should work out of the box, although there is a small catch.
>> If you're using raid5/raid6 and one/two of the disk is missing or
>> corrupted, you need to add module raid5rec/raid6rec using the --module
>> option in grub-install.
>>
>
> Just curiosity - why it's in separate modules ?
It's caused by size limitation of mbr, which is normally 63 sectors.
When you use the following modules:
biosdisk pc reiserfs lvm mdraid raid6 raid6rec
The size of core.img would exceed 63 sectors, so that it can't be
installed to mbr.
--
Bean
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: LVM & RAID10
2008-12-04 4:38 ` Bean
@ 2008-12-05 21:11 ` Wojciech Pyczak
2009-05-01 18:29 ` Wojciech Pyczak
0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-12-05 21:11 UTC (permalink / raw)
To: The development of GRUB 2
>
> Hi,
>
> Perhaps you can describe the steps used to create the LVM volume,
> along with version number of tools. Let me see if the error can be
> recreated in qemu.
>
OK. I've made some less chaotic tests:
Hardware (probably insignificant):
- AMD Phenom X3
- Gigabyte mobo (780G/SB700 AMD chipset)
- 2x WD SE-16 6400AAKS (before I started testing I zeroed with dd a
few GB of each drive - just in case..)
Software:
- System RescueCD (used to manage partitions, RAID/LVM volumes)
- Linux sysresccd 2.6.26.07-std111-amd64 #1 SMP Sun Oct 26 11:28:49
UTC 2008 x86_64 AMD Phenom(tm) 8450 Triple-Core Processor AuthenticAMD
GNU/Linux
- LVM version: 2.02.39 (2008-06-27), Library version: 1.02.27
(2008-06-25), Driver version: 4.13.0
- mdadm - v2.6.4 - 19th October 2007
- Gentoo custom ("stable amd64") stage3 -
http://www.funtoo.org/linux/amd64/funtoo-amd64-2008.12.05/ (used for
chrooting)
- Grub2 rev 1933 (latest), build on my current PC:
- ./configure --prefix=/usr --enable-grub-emu --enable-grub-fstest
LDFLAGS=-static
- gcc version 4.1.2 (Gentoo 4.1.2 p1.1)
- GNU Make 3.81
- bison (GNU Bison) 2.3
- sys-devel/binutils-2.18-r3
* IMO also insignificant, since previously I build grub on my new
system which used many "experimental" packages, such as GCC 4.3.2, etc. *
* Small note - I renamed grub binaries (in bin/sbin) to *.org and
made a wrapper script, so I would know arguments used by
grub-install/setup: *
#!/bin/sh
echo \#${0} ${@} 1>&2
exec ${0}.org ${@}
I used cfdisk to partition disks. Commands used to create RAID arrays
and LVM volumes:
mdadm --create /dev/md0 (md1) --raid-devices=3 --level=10
--layout=f2 --chunk=256 /dev/sd{a,b}1 (sd{a,b}2) missing
pvcreate /dev/md0
vgcreate test /dev/md0
lvcreate test -ntest -l1788
mdadm --assemble /dev/md0 (md1) /dev/sd{a,b}1 (sd{a,b}2)
pvscan
vgchange -ay
Nothing fancy...
Test procedure looked something like that:
- booting system with RescueCD
- partitioning, creating RAID/LVM, formatting, reboot
- after reboot:
- assembling RAID/LVM volumes
- mounting corresponding device
- unpacking stage3
- unpacking package containing grub
- creating missing directories (/boot/grub) and files (device.map,
simple grub.cfg)
- mouting proc, sys, dev
- grub-install/setup, whatever
===== Single partition, RAID10, reiserfs
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sda2 0 - 0 0 0 Empty
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sdb2 0 - 0 0 0 Empty
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
--------------------------------------------------------------------------------------------------------------------------
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 sda1[0] sdb1[1]
7325184 blocks 256K chunks 2 far-copies [3/2] [UU_]
unused devices: <none>
--------------------------------------------------------------------------------------------------------------------------
sysresccd grub # grub-install "(md0)"
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/md0
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(md0)/boot/grub biosdisk reiserfs pc pc raid mdraid
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.
(hd0) /dev/sda
(hd1) /dev/sdb
(hd2) /dev/sdc <= USB stick
--------------------------------------------------------------------------------------------------------------------------
Everything works (I copied memtest86 from RescueCD to /boot and managed
to boot it).
===== Single partion, LVM @ RAID10, reiserfs
Partitions/RAID like above
--------------------------------------------------------------------------------------------------------------------------
root@sysresccd /root % pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name test
PV Size 6.99 GB / not usable 1.50 MB
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 1788
Free PE 0
Allocated PE 1788
PV UUID 6i6MZH-7Lv7-ZcXb-yr2W-ShFz-2c13-srXd07
root@sysresccd /root % vgdisplay
--- Volume group ---
VG Name test
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 6.98 GB
PE Size 4.00 MB
Total PE 1788
Alloc PE / Size 1788 / 6.98 GB
Free PE / Size 0 / 0
VG UUID aCTAWb-xbpL-3oSp-1V2X-Oy3T-E2D7-gJoCZG
root@sysresccd /root % lvdisplay
--- Logical volume ---
LV Name /dev/test/test
VG Name test
LV UUID IzYAb2-9y4u-HqB2-27bD-ihvH-uCmn-2a1xlH
LV Write Access read/write
LV Status available
# open 2
LV Size 6.98 GB
Current LE 1788
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install '(md0)'
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules='biosdisk pc raid
mdraid lvm reiserfs'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
error: We don't support multiple metadata areas
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Grub works (after reboot) but I can't access filesystem, LVM module is
not present
====== Added second partition
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sda2 608 1215 608 4883760 da Non-FS data
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sdb2 608 1215 608 4883760 da Non-FS data
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
--------------------------------------------------------------------------------------------------------------------------
RAID/LVM like above
--------------------------------------------------------------------------------------------------------------------------
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules="biosdisk pc raid
mdraid lvm reiserfs"
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/mapper/test-test
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/mapper/test-test
error: no such partition
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Second partition is unsed, however grub stoped complaining about
"multiple metadata". Desipte that I still can't boot system.
===== Added RAID10 using second partition pair, I'll try to boot system
from it
sysresccd / # grub-mkimage -o /boot/grub/core.img biosdisk pc raid
mdraid lvm reiserfs
#/usr/bin/grub-mkimage -o /boot/grub/core.img biosdisk pc raid mdraid
lvm reiserfs
sysresccd / # grub-setup '(md1)'
#/usr/sbin/grub-setup (md1)
grub-setup: error: Core image is too big for embedding, but this is
required when
the root device is on a RAID array or LVM volume.
--------------------------------------------------------------------------------------------------------------------------
After seeing that error message I created core.img without lvm module
and rebooted computer (I was able to boot system),
then I loaded lvm module but I couldn't see any lvm volumes (despite
fact that I saw md0 containing it). I tried to rmmod/inmod
again but it didn't help.
OK so here are my thoughts - most ppl use ext2 for /boot, I used
reiserfs which has 4k larger module so LVM just don't fit info core.img.
Using grub-install/setup on LVM@RAID always ends up with some errors I
just couldn't see it before (those errors were discarted
somewhere along the way ?) but it doesn't explain why I couldn't see lvm
volumes desipe the fact that loaded lvm module from disk.
Perhaps I should try to create LVM on md1 and boot from md0 to see what
will happen ?
Anyway there is something wrong with LVM+RAID on single partition ;)
>
> It's caused by size limitation of mbr, which is normally 63 sectors.
> When you use the following modules:
>
> biosdisk pc reiserfs lvm mdraid raid6 raid6rec
>
> The size of core.img would exceed 63 sectors, so that it can't be
> installed to mbr.
>
>
Is it possible to use more sectors by leaving some unused space before
fisrt partion ?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: LVM & RAID10
2008-12-05 21:11 ` Wojciech Pyczak
@ 2009-05-01 18:29 ` Wojciech Pyczak
0 siblings, 0 replies; 6+ messages in thread
From: Wojciech Pyczak @ 2009-05-01 18:29 UTC (permalink / raw)
To: The development of GRUB 2
FYI: I just tested latest revision (2154) and there is no error message
about "multiple metadata areas". I suppose it was fixed (probably "by
accident" ;) ) in on of latest patches ("Partitions can start at zero" ?
just wild guess).
Anyway I only tested grub-probe on my current system (with 2 disks
attached in LVM @ RAID5 configuration), tomorrow I'll try to move my
system to this array and see what happens (well, due to all that changes
lately I don't even know if this revision can boot anything).
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-05-01 18:29 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-11-25 1:08 LVM & RAID10 Wojciech Pyczak
2008-11-25 2:52 ` Bean
2008-12-03 22:32 ` Wojciech Pyczak
2008-12-04 4:38 ` Bean
2008-12-05 21:11 ` Wojciech Pyczak
2009-05-01 18:29 ` Wojciech Pyczak
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.