LVM & RAID10

All of lore.kernel.org
 help / color / mirror / Atom feed

* LVM & RAID10
@ 2008-11-25  1:08 Wojciech Pyczak
  2008-11-25  2:52 ` Bean
  0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-11-25  1:08 UTC (permalink / raw)
  To: grub-devel

Hello!

I have some troubles setting up grub2 on my configuration. After I 
bought a new computer I decided to "go wild" and setup everything on top 
of LVM, with LVM itself on RAID array. I expected there will be some 
problems and here I am :)

Few words about configuration - I have 2x WD 640GB HDDs, there is single 
partition on each drive, those partitions are used to build RAID10 
array. Aray itself consists of 3 drives (third one is "missing" since I 
use it in my "old" computer), like I mentioned before it's level 10, 
with 256kB chunks and far2 layuout. On top of that there is LVM2, 
consisting of single "storage" group (32MB extents), divided into few 
logical volumes, including volumes named "root" (/) and "boot" (/boot).

I used latest RescueCD to made this up and installed fresh gentoo. I 
managed to build entire system, I only needed to boot it. So, I emerged 
(using my own custom ebuild) Grub2 SVN version (successfully) and tried 
to make everythig work but it appers life is not so easy :P. The only 
clue I have is (completly mysterius to me) error message: "We don't 
support multiple metadata areas".

AFAIK both LVM and RAID10 are supported and it's possible to boot grub 
from LVM volume, am I wrong ?
Or perhaps I'm missing something ?
Is there any example of LVM/RAID config files ?
What happens if array (RAID 5, 6, 10) is incomplete (disk 
failure/missing) - maybe that's the problem ? And in that case, how I'm 
supposed to boot system in such situations ?

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: LVM & RAID10
  2008-11-25  1:08 LVM & RAID10 Wojciech Pyczak
@ 2008-11-25  2:52 ` Bean
  2008-12-03 22:32   ` Wojciech Pyczak
  0 siblings, 1 reply; 6+ messages in thread
From: Bean @ 2008-11-25  2:52 UTC (permalink / raw)
  To: The development of GRUB 2

On Tue, Nov 25, 2008 at 9:08 AM, Wojciech Pyczak <nissarin@gmail.com> wrote:
> Hello!
>
> I have some troubles setting up grub2 on my configuration. After I bought a
> new computer I decided to "go wild" and setup everything on top of LVM, with
> LVM itself on RAID array. I expected there will be some problems and here I
> am :)
>
> Few words about configuration - I have 2x WD 640GB HDDs, there is single
> partition on each drive, those partitions are used to build RAID10 array.
> Aray itself consists of 3 drives (third one is "missing" since I use it in
> my "old" computer), like I mentioned before it's level 10, with 256kB chunks
> and far2 layuout. On top of that there is LVM2, consisting of single
> "storage" group (32MB extents), divided into few logical volumes, including
> volumes named "root" (/) and "boot" (/boot).
>
> I used latest RescueCD to made this up and installed fresh gentoo. I managed
> to build entire system, I only needed to boot it. So, I emerged (using my
> own custom ebuild) Grub2 SVN version (successfully) and tried to make
> everythig work but it appers life is not so easy :P. The only clue I have is
> (completly mysterius to me) error message: "We don't support multiple
> metadata areas".

Hi,

This error come from lvm, perhaps you can try raid10 first.

>
> AFAIK both LVM and RAID10 are supported and it's possible to boot grub from
> LVM volume, am I wrong ?
> Or perhaps I'm missing something ?
> Is there any example of LVM/RAID config files ?
> What happens if array (RAID 5, 6, 10) is incomplete (disk failure/missing) -
> maybe that's the problem ? And in that case, how I'm supposed to boot system
> in such situations ?

Yes, it should work out of the box, although there is a small catch.
If you're using raid5/raid6 and one/two of the disk is missing or
corrupted, you need to add module raid5rec/raid6rec using the --module
option in grub-install.

-- 
Bean



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: LVM & RAID10
  2008-11-25  2:52 ` Bean
@ 2008-12-03 22:32   ` Wojciech Pyczak
  2008-12-04  4:38     ` Bean
  0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-12-03 22:32 UTC (permalink / raw)
  To: The development of GRUB 2


>> (...)The only clue I have is
>> (completly mysterius to me) error message: "We don't support multiple
>> metadata areas".
>>      
>
> Hi,
>
> This error come from lvm, perhaps you can try raid10 first.
>
>    

I played a bit when I had free time and...
  - I learned to ignore grub error messages :D
  - RAID10 seems to work OK
  - LVM @ RAID10 @ single partition (on each disk) results in above 
error message but if I add small partition (so there are 2 partitions 
per disk) there is no error. I tried with different number of LVs, 
different LV names, anything.. could it be some error in grub after all ?

     Anyway even if there is no such error grub doesn't work - it appers 
that core.img doesn't include LVM module (I tried to specify it 
manually) or just doesn't install at all, of course grub-setup return 
some error (unknown root device or something..) but in the other hand I 
saw the same error with root filesystem on RAID device and it actually 
worked.
     There is also another problem: when I installed grub on single HD 
with LVM module in core.img, I couldn't see any LVM devices in the 
system (only hdX, mdX..), I saw something similar on some mailing list 
(debian ?) - removing LVM module then inserting it again fixed that 
problem, however in my case it didn't worked.
     I'm thinking about installing grub by hand, i.e. creating core.img 
and copying it along with boot.img after mbr,  will that work ? (I just 
want to bypass grub-setup).

>
> Yes, it should work out of the box, although there is a small catch.
> If you're using raid5/raid6 and one/two of the disk is missing or
> corrupted, you need to add module raid5rec/raid6rec using the --module
> option in grub-install.
>    
Just curiosity - why it's in separate modules ?




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: LVM & RAID10
  2008-12-03 22:32   ` Wojciech Pyczak
@ 2008-12-04  4:38     ` Bean
  2008-12-05 21:11       ` Wojciech Pyczak
  0 siblings, 1 reply; 6+ messages in thread
From: Bean @ 2008-12-04  4:38 UTC (permalink / raw)
  To: The development of GRUB 2

On Thu, Dec 4, 2008 at 6:32 AM, Wojciech Pyczak <nissarin@gmail.com> wrote:
>
>>> (...)The only clue I have is
>>> (completly mysterius to me) error message: "We don't support multiple
>>> metadata areas".
>>>
>>
>> Hi,
>>
>> This error come from lvm, perhaps you can try raid10 first.
>>
>>
>
> I played a bit when I had free time and...
>  - I learned to ignore grub error messages :D
>  - RAID10 seems to work OK
>  - LVM @ RAID10 @ single partition (on each disk) results in above error
> message but if I add small partition (so there are 2 partitions per disk)
> there is no error. I tried with different number of LVs, different LV names,
> anything.. could it be some error in grub after all ?
>
>    Anyway even if there is no such error grub doesn't work - it appers that
> core.img doesn't include LVM module (I tried to specify it manually) or just
> doesn't install at all, of course grub-setup return some error (unknown root
> device or something..) but in the other hand I saw the same error with root
> filesystem on RAID device and it actually worked.
>    There is also another problem: when I installed grub on single HD with
> LVM module in core.img, I couldn't see any LVM devices in the system (only
> hdX, mdX..), I saw something similar on some mailing list (debian ?) -
> removing LVM module then inserting it again fixed that problem, however in
> my case it didn't worked.
>    I'm thinking about installing grub by hand, i.e. creating core.img and
> copying it along with boot.img after mbr,  will that work ? (I just want to
> bypass grub-setup).

Hi,

Perhaps you can describe the steps used to create the LVM volume,
along with version number of tools. Let me see if the error can be
recreated in qemu.

>
>>
>> Yes, it should work out of the box, although there is a small catch.
>> If you're using raid5/raid6 and one/two of the disk is missing or
>> corrupted, you need to add module raid5rec/raid6rec using the --module
>> option in grub-install.
>>
>
> Just curiosity - why it's in separate modules ?

It's caused by size limitation of mbr, which is normally 63 sectors.
When you use the following modules:

biosdisk pc reiserfs lvm mdraid raid6 raid6rec

The size of core.img would exceed 63 sectors, so that it can't be
installed to mbr.

-- 
Bean



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: LVM & RAID10
  2008-12-04  4:38     ` Bean
@ 2008-12-05 21:11       ` Wojciech Pyczak
  2009-05-01 18:29         ` Wojciech Pyczak
  0 siblings, 1 reply; 6+ messages in thread
From: Wojciech Pyczak @ 2008-12-05 21:11 UTC (permalink / raw)
  To: The development of GRUB 2


>
> Hi,
>
> Perhaps you can describe the steps used to create the LVM volume,
> along with version number of tools. Let me see if the error can be
> recreated in qemu.
>
OK. I've made some less chaotic tests:

Hardware (probably insignificant):
  - AMD Phenom X3
  - Gigabyte mobo (780G/SB700 AMD chipset)
  - 2x WD SE-16 6400AAKS (before I started testing I zeroed with dd a 
few GB of each drive - just in case..)

Software:
  - System RescueCD (used to manage partitions, RAID/LVM volumes)
     - Linux sysresccd 2.6.26.07-std111-amd64 #1 SMP Sun Oct 26 11:28:49 
UTC 2008 x86_64 AMD Phenom(tm) 8450 Triple-Core Processor AuthenticAMD 
GNU/Linux
     - LVM version:     2.02.39 (2008-06-27), Library version: 1.02.27 
(2008-06-25), Driver version:  4.13.0
     - mdadm - v2.6.4 - 19th October 2007

- Gentoo custom ("stable amd64") stage3 - 
http://www.funtoo.org/linux/amd64/funtoo-amd64-2008.12.05/ (used for 
chrooting)

- Grub2 rev 1933 (latest), build on my current PC:
     - ./configure --prefix=/usr --enable-grub-emu --enable-grub-fstest 
LDFLAGS=-static
     - gcc version 4.1.2 (Gentoo 4.1.2 p1.1)
     - GNU Make 3.81
     - bison (GNU Bison) 2.3
     - sys-devel/binutils-2.18-r3
     * IMO also insignificant, since previously I build grub on my new 
system which used many "experimental" packages, such as GCC 4.3.2, etc. *
     * Small note - I renamed grub binaries (in bin/sbin) to *.org and 
made a wrapper script, so I would know arguments used by 
grub-install/setup: *
         #!/bin/sh
         echo \#${0} ${@} 1>&2
         exec ${0}.org ${@}


I used cfdisk to partition disks. Commands used to create RAID arrays 
and LVM volumes:
     mdadm --create /dev/md0 (md1) --raid-devices=3 --level=10 
--layout=f2 --chunk=256 /dev/sd{a,b}1 (sd{a,b}2) missing
     pvcreate /dev/md0
     vgcreate test /dev/md0
     lvcreate test -ntest -l1788

     mdadm --assemble /dev/md0 (md1) /dev/sd{a,b}1 (sd{a,b}2)
     pvscan
     vgchange -ay
Nothing fancy...


Test procedure looked something like that:
- booting system with RescueCD
- partitioning, creating RAID/LVM, formatting, reboot
- after reboot:
     - assembling RAID/LVM volumes
     - mounting corresponding device
     - unpacking stage3
     - unpacking package containing grub
     - creating missing directories (/boot/grub) and files (device.map, 
simple grub.cfg)
     - mouting proc, sys, dev
     - grub-install/setup, whatever



===== Single partition, RAID10, reiserfs
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

    Device Boot Start     End   #cyls    #blocks   Id  System
    /dev/sda1   *      0+    607     608-   4883728+  da  Non-FS data
    /dev/sda2          0       -       0          0    0  Empty
    /dev/sda3          0       -       0          0    0  Empty
    /dev/sda4          0       -       0          0    0  Empty

Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

    Device Boot Start     End   #cyls    #blocks   Id  System
    /dev/sdb1   *      0+    607     608-   4883728+  da  Non-FS data
    /dev/sdb2          0       -       0          0    0  Empty
    /dev/sdb3          0       -       0          0    0  Empty
    /dev/sdb4          0       -       0          0    0  Empty
--------------------------------------------------------------------------------------------------------------------------
  Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
  md0 : active raid10 sda1[0] sdb1[1]
        7325184 blocks 256K chunks 2 far-copies [3/2] [UU_]

        unused devices: <none>
--------------------------------------------------------------------------------------------------------------------------
sysresccd grub # grub-install "(md0)"
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device 
/boot/grub
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs 
--device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map 
--target=abstraction --device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive 
--device /dev/md0
#/usr/bin/grub-mkimage --output=/boot/grub/core.img 
--prefix=(md0)/boot/grub biosdisk reiserfs pc pc raid mdraid
#/usr/sbin/grub-setup --directory=/boot/grub 
--device-map=/boot/grub/device.map
(md0)
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(hd0)   /dev/sda
(hd1)   /dev/sdb
(hd2)   /dev/sdc <= USB stick
--------------------------------------------------------------------------------------------------------------------------
Everything works (I copied memtest86 from RescueCD to /boot and managed 
to boot it).





===== Single partion, LVM @ RAID10, reiserfs
Partitions/RAID like above
--------------------------------------------------------------------------------------------------------------------------
root@sysresccd /root % pvdisplay
--- Physical volume ---
PV Name               /dev/md0
VG Name               test
PV Size               6.99 GB / not usable 1.50 MB
Allocatable           yes (but full)
PE Size (KByte)       4096
Total PE              1788
Free PE               0
Allocated PE          1788
PV UUID               6i6MZH-7Lv7-ZcXb-yr2W-ShFz-2c13-srXd07

root@sysresccd /root % vgdisplay
--- Volume group ---
VG Name               test
System ID
Format                lvm2
Metadata Areas        2
Metadata Sequence No  2
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                1
Act PV                1
VG Size               6.98 GB
PE Size               4.00 MB
Total PE              1788
Alloc PE / Size       1788 / 6.98 GB
Free  PE / Size       0 / 0
VG UUID               aCTAWb-xbpL-3oSp-1V2X-Oy3T-E2D7-gJoCZG

root@sysresccd /root % lvdisplay
--- Logical volume ---
LV Name                /dev/test/test
VG Name                test
LV UUID                IzYAb2-9y4u-HqB2-27bD-ihvH-uCmn-2a1xlH
LV Write Access        read/write
LV Status              available
# open                 2
LV Size                6.98 GB
Current LE             1788
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:0
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install '(md0)'
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device 
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs 
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules='biosdisk pc raid 
mdraid lvm reiserfs'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device 
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs 
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map 
--target=abstraction --device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive 
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/bin/grub-mkimage --output=/boot/grub/core.img 
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs 
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub 
--device-map=/boot/grub/device.map
(md0)
error: We don't support multiple metadata areas
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Grub works (after reboot) but I can't access filesystem, LVM module is 
not present





====== Added second partition
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+    607     608-   4883728+  da  Non-FS data
/dev/sda2        608    1215     608    4883760   da  Non-FS data
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty

Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+    607     608-   4883728+  da  Non-FS data
/dev/sdb2        608    1215     608    4883760   da  Non-FS data
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
--------------------------------------------------------------------------------------------------------------------------
RAID/LVM like above
--------------------------------------------------------------------------------------------------------------------------
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device 
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs 
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules="biosdisk pc raid 
mdraid lvm reiserfs"
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device 
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs 
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map 
--target=abstraction --device /dev/mapper/test-test
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive 
--device /dev/mapper/test-test
error: no such partition
#/usr/bin/grub-mkimage --output=/boot/grub/core.img 
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs 
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub 
--device-map=/boot/grub/device.map
(md0)
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Second partition is unsed, however grub stoped complaining about 
"multiple metadata". Desipte that I still can't boot system.




===== Added RAID10 using second partition pair, I'll try to boot system 
from it
sysresccd / # grub-mkimage -o /boot/grub/core.img biosdisk pc raid 
mdraid lvm reiserfs
#/usr/bin/grub-mkimage -o /boot/grub/core.img biosdisk pc raid mdraid 
lvm reiserfs
sysresccd / # grub-setup '(md1)'
#/usr/sbin/grub-setup (md1)
grub-setup: error: Core image is too big for embedding, but this is 
required when
the root device is on a RAID array or LVM volume.
--------------------------------------------------------------------------------------------------------------------------
After seeing that error message I created core.img without lvm module 
and rebooted computer (I was able to boot system),
then I loaded lvm module but I couldn't see any lvm volumes (despite 
fact that I saw md0 containing it). I tried to rmmod/inmod
again but it didn't help.



OK so here are my thoughts - most ppl use ext2 for /boot, I used 
reiserfs which has 4k larger module so LVM just don't fit info core.img.
Using grub-install/setup on LVM@RAID always ends up with some errors I 
just couldn't see it before (those errors were discarted
somewhere along the way ?) but it doesn't explain why I couldn't see lvm 
volumes desipe the fact that loaded lvm module from disk.
Perhaps I should try to create LVM on md1 and boot from md0 to see what 
will happen ?
Anyway there is something wrong with LVM+RAID on single partition ;)

>
> It's caused by size limitation of mbr, which is normally 63 sectors.
> When you use the following modules:
>
> biosdisk pc reiserfs lvm mdraid raid6 raid6rec
>
> The size of core.img would exceed 63 sectors, so that it can't be
> installed to mbr.
>
>    
Is it possible to use more sectors by leaving some unused space before 
fisrt partion ?





^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: LVM & RAID10
  2008-12-05 21:11       ` Wojciech Pyczak
@ 2009-05-01 18:29         ` Wojciech Pyczak
  0 siblings, 0 replies; 6+ messages in thread
From: Wojciech Pyczak @ 2009-05-01 18:29 UTC (permalink / raw)
  To: The development of GRUB 2

FYI: I just tested latest revision (2154) and there is no error message
about "multiple metadata areas". I suppose it was fixed (probably "by
accident" ;) ) in on of latest patches ("Partitions can start at zero" ?
just wild guess). 

Anyway I only tested grub-probe on my current system (with 2 disks
attached in LVM @ RAID5 configuration), tomorrow I'll try to move my
system to this array and see what happens (well, due to all that changes
lately I don't even know if this revision can boot anything). 

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2009-05-01 18:29 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-11-25  1:08 LVM & RAID10 Wojciech Pyczak
2008-11-25  2:52 ` Bean
2008-12-03 22:32   ` Wojciech Pyczak
2008-12-04  4:38     ` Bean
2008-12-05 21:11       ` Wojciech Pyczak
2009-05-01 18:29         ` Wojciech Pyczak

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.