From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with archive (Exim 4.43) id 1L8hxx-00036P-Hm for mharc-grub-devel@gnu.org; Fri, 05 Dec 2008 16:12:01 -0500 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1L8hxu-000358-O9 for grub-devel@gnu.org; Fri, 05 Dec 2008 16:11:58 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1L8hxr-000323-RF for grub-devel@gnu.org; Fri, 05 Dec 2008 16:11:58 -0500 Received: from [199.232.76.173] (port=38902 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1L8hxr-000320-L6 for grub-devel@gnu.org; Fri, 05 Dec 2008 16:11:55 -0500 Received: from gv-out-0910.google.com ([216.239.58.184]:21773) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1L8hxq-0004P6-RV for grub-devel@gnu.org; Fri, 05 Dec 2008 16:11:55 -0500 Received: by gv-out-0910.google.com with SMTP id i36so132083gve.17 for ; Fri, 05 Dec 2008 13:11:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=V8voHjXzz/dJdksugI1/+HL9mJ5gnXw/2q0ok3smdFc=; b=HqCMfjnvrRNmeY3MpUddFwWeBUGyJqAtNnJWlXUvbsE1hz+8k9dKg0O64h3sbqGFEF wL3gms004xT/8SzGjYLBz6oPz6Wjq+T3e/E/hkHOrEVcWpuqKIHCptIx9uGjgVzpFgaV yplw2/0L66kLbBSqFa009uC09nG1vSN1giS5g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=nqNtjfkcO5H/lKFTOYAT5VPZ28ZMSXCy7DYSs6rXE1iJIdmxemo4jkJzJvlm8UH+c0 OSTOSlI+F1gEYjKuSw6Ur66y3ItVbvP14mlibZnbjjx5a3prSiO0pmlTGD9DjjgYEZk/ KcKq7okQ5bkI28Jc2wXjmbi2POjqH8TBRu+8Y= Received: by 10.223.106.68 with SMTP id w4mr435460fao.20.1228511512901; Fri, 05 Dec 2008 13:11:52 -0800 (PST) Received: from ?91.189.141.21? (host-141-21.jkns.pl [91.189.141.21]) by mx.google.com with ESMTPS id h2sm13231310fkh.9.2008.12.05.13.11.49 (version=SSLv3 cipher=RC4-MD5); Fri, 05 Dec 2008 13:11:51 -0800 (PST) Message-ID: <49399916.5020307@gmail.com> Date: Fri, 05 Dec 2008 22:11:50 +0100 From: Wojciech Pyczak User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); pl; rv:1.9.1b1pre) Gecko/20081006 Shredder/3.0a3 MIME-Version: 1.0 To: The development of GRUB 2 References: <492B500A.8050506@gmail.com> <493708F9.4000704@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6 (newer, 2) Subject: Re: LVM & RAID10 X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: The development of GRUB 2 List-Id: The development of GRUB 2 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2008 21:11:59 -0000 > > Hi, > > Perhaps you can describe the steps used to create the LVM volume, > along with version number of tools. Let me see if the error can be > recreated in qemu. > OK. I've made some less chaotic tests: Hardware (probably insignificant): - AMD Phenom X3 - Gigabyte mobo (780G/SB700 AMD chipset) - 2x WD SE-16 6400AAKS (before I started testing I zeroed with dd a few GB of each drive - just in case..) Software: - System RescueCD (used to manage partitions, RAID/LVM volumes) - Linux sysresccd 2.6.26.07-std111-amd64 #1 SMP Sun Oct 26 11:28:49 UTC 2008 x86_64 AMD Phenom(tm) 8450 Triple-Core Processor AuthenticAMD GNU/Linux - LVM version: 2.02.39 (2008-06-27), Library version: 1.02.27 (2008-06-25), Driver version: 4.13.0 - mdadm - v2.6.4 - 19th October 2007 - Gentoo custom ("stable amd64") stage3 - http://www.funtoo.org/linux/amd64/funtoo-amd64-2008.12.05/ (used for chrooting) - Grub2 rev 1933 (latest), build on my current PC: - ./configure --prefix=/usr --enable-grub-emu --enable-grub-fstest LDFLAGS=-static - gcc version 4.1.2 (Gentoo 4.1.2 p1.1) - GNU Make 3.81 - bison (GNU Bison) 2.3 - sys-devel/binutils-2.18-r3 * IMO also insignificant, since previously I build grub on my new system which used many "experimental" packages, such as GCC 4.3.2, etc. * * Small note - I renamed grub binaries (in bin/sbin) to *.org and made a wrapper script, so I would know arguments used by grub-install/setup: * #!/bin/sh echo \#${0} ${@} 1>&2 exec ${0}.org ${@} I used cfdisk to partition disks. Commands used to create RAID arrays and LVM volumes: mdadm --create /dev/md0 (md1) --raid-devices=3 --level=10 --layout=f2 --chunk=256 /dev/sd{a,b}1 (sd{a,b}2) missing pvcreate /dev/md0 vgcreate test /dev/md0 lvcreate test -ntest -l1788 mdadm --assemble /dev/md0 (md1) /dev/sd{a,b}1 (sd{a,b}2) pvscan vgchange -ay Nothing fancy... Test procedure looked something like that: - booting system with RescueCD - partitioning, creating RAID/LVM, formatting, reboot - after reboot: - assembling RAID/LVM volumes - mounting corresponding device - unpacking stage3 - unpacking package containing grub - creating missing directories (/boot/grub) and files (device.map, simple grub.cfg) - mouting proc, sys, dev - grub-install/setup, whatever ===== Single partition, RAID10, reiserfs Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data /dev/sda2 0 - 0 0 0 Empty /dev/sda3 0 - 0 0 0 Empty /dev/sda4 0 - 0 0 0 Empty Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data /dev/sdb2 0 - 0 0 0 Empty /dev/sdb3 0 - 0 0 0 Empty /dev/sdb4 0 - 0 0 0 Empty -------------------------------------------------------------------------------------------------------------------------- Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 sda1[0] sdb1[1] 7325184 blocks 256K chunks 2 far-copies [3/2] [UU_] unused devices: -------------------------------------------------------------------------------------------------------------------------- sysresccd grub # grub-install "(md0)" #/usr/sbin/grub-install (md0) #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device /boot/grub #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs --device /dev/md0 #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=abstraction --device /dev/md0 #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive --device /dev/md0 #/usr/bin/grub-mkimage --output=/boot/grub/core.img --prefix=(md0)/boot/grub biosdisk reiserfs pc pc raid mdraid #/usr/sbin/grub-setup --directory=/boot/grub --device-map=/boot/grub/device.map (md0) Installation finished. No error reported. This is the contents of the device map /boot/grub/device.map. Check if this is correct or not. If any of the lines is incorrect, fix it and re-run the script `grub-install'. (hd0) /dev/sda (hd1) /dev/sdb (hd2) /dev/sdc <= USB stick -------------------------------------------------------------------------------------------------------------------------- Everything works (I copied memtest86 from RescueCD to /boot and managed to boot it). ===== Single partion, LVM @ RAID10, reiserfs Partitions/RAID like above -------------------------------------------------------------------------------------------------------------------------- root@sysresccd /root % pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name test PV Size 6.99 GB / not usable 1.50 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 1788 Free PE 0 Allocated PE 1788 PV UUID 6i6MZH-7Lv7-ZcXb-yr2W-ShFz-2c13-srXd07 root@sysresccd /root % vgdisplay --- Volume group --- VG Name test System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 6.98 GB PE Size 4.00 MB Total PE 1788 Alloc PE / Size 1788 / 6.98 GB Free PE / Size 0 / 0 VG UUID aCTAWb-xbpL-3oSp-1V2X-Oy3T-E2D7-gJoCZG root@sysresccd /root % lvdisplay --- Logical volume --- LV Name /dev/test/test VG Name test LV UUID IzYAb2-9y4u-HqB2-27bD-ihvH-uCmn-2a1xlH LV Write Access read/write LV Status available # open 2 LV Size 6.98 GB Current LE 1788 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 -------------------------------------------------------------------------------------------------------------------------- sysresccd / # grub-install '(md0)' #/usr/sbin/grub-install (md0) #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device /boot/grub error: We don't support multiple metadata areas #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs --device /dev/mapper/test-test error: We don't support multiple metadata areas grub-probe: error: no mapping exists for `test-test' Auto-detection of a filesystem module failed. Please specify the module with the option `--modules' explicitly. -------------------------------------------------------------------------------------------------------------------------- sysresccd / # grub-install.org '(md0)' --modules='biosdisk pc raid mdraid lvm reiserfs' #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device /boot/grub error: We don't support multiple metadata areas #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs --device /dev/mapper/test-test error: We don't support multiple metadata areas grub-probe: error: no mapping exists for `test-test' #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=abstraction --device /dev/mapper/test-test error: We don't support multiple metadata areas #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive --device /dev/mapper/test-test error: We don't support multiple metadata areas #/usr/bin/grub-mkimage --output=/boot/grub/core.img --prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs biosdisk lvm #/usr/sbin/grub-setup --directory=/boot/grub --device-map=/boot/grub/device.map (md0) error: We don't support multiple metadata areas grub-setup: error: no mapping exists for `test-test' -------------------------------------------------------------------------------------------------------------------------- Grub works (after reboot) but I can't access filesystem, LVM module is not present ====== Added second partition Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data /dev/sda2 608 1215 608 4883760 da Non-FS data /dev/sda3 0 - 0 0 0 Empty /dev/sda4 0 - 0 0 0 Empty Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data /dev/sdb2 608 1215 608 4883760 da Non-FS data /dev/sdb3 0 - 0 0 0 Empty /dev/sdb4 0 - 0 0 0 Empty -------------------------------------------------------------------------------------------------------------------------- RAID/LVM like above -------------------------------------------------------------------------------------------------------------------------- #/usr/sbin/grub-install (md0) #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device /boot/grub error: no such partition #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs --device /dev/mapper/test-test error: no such partition grub-probe: error: no mapping exists for `test-test' Auto-detection of a filesystem module failed. Please specify the module with the option `--modules' explicitly. -------------------------------------------------------------------------------------------------------------------------- sysresccd / # grub-install.org '(md0)' --modules="biosdisk pc raid mdraid lvm reiserfs" #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device /boot/grub error: no such partition #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs --device /dev/mapper/test-test error: no such partition grub-probe: error: no mapping exists for `test-test' #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=abstraction --device /dev/mapper/test-test error: no such partition #/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive --device /dev/mapper/test-test error: no such partition #/usr/bin/grub-mkimage --output=/boot/grub/core.img --prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs biosdisk lvm #/usr/sbin/grub-setup --directory=/boot/grub --device-map=/boot/grub/device.map (md0) grub-setup: error: no mapping exists for `test-test' -------------------------------------------------------------------------------------------------------------------------- Second partition is unsed, however grub stoped complaining about "multiple metadata". Desipte that I still can't boot system. ===== Added RAID10 using second partition pair, I'll try to boot system from it sysresccd / # grub-mkimage -o /boot/grub/core.img biosdisk pc raid mdraid lvm reiserfs #/usr/bin/grub-mkimage -o /boot/grub/core.img biosdisk pc raid mdraid lvm reiserfs sysresccd / # grub-setup '(md1)' #/usr/sbin/grub-setup (md1) grub-setup: error: Core image is too big for embedding, but this is required when the root device is on a RAID array or LVM volume. -------------------------------------------------------------------------------------------------------------------------- After seeing that error message I created core.img without lvm module and rebooted computer (I was able to boot system), then I loaded lvm module but I couldn't see any lvm volumes (despite fact that I saw md0 containing it). I tried to rmmod/inmod again but it didn't help. OK so here are my thoughts - most ppl use ext2 for /boot, I used reiserfs which has 4k larger module so LVM just don't fit info core.img. Using grub-install/setup on LVM@RAID always ends up with some errors I just couldn't see it before (those errors were discarted somewhere along the way ?) but it doesn't explain why I couldn't see lvm volumes desipe the fact that loaded lvm module from disk. Perhaps I should try to create LVM on md1 and boot from md0 to see what will happen ? Anyway there is something wrong with LVM+RAID on single partition ;) > > It's caused by size limitation of mbr, which is normally 63 sectors. > When you use the following modules: > > biosdisk pc reiserfs lvm mdraid raid6 raid6rec > > The size of core.img would exceed 63 sectors, so that it can't be > installed to mbr. > > Is it possible to use more sectors by leaving some unused space before fisrt partion ?