getting a linux boot loader (preferably grub) installed on an intel imsm raid

getting a linux boot loader (preferably grub) installed on an intel imsm raid

[K. Posern]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]

Hi all,

/// The problem ///

There are binary packages out there that do the trick based on 
device-mapper (dmraid) (e.g. Ubuntu). But I would like to do it

	a) manually (because thats why I use Linux, because you can do things 
manually and you can learn a lot (even if you don't succeed in the end)

	b) based on mdadm-imsm. Just there seems really /no/ documentation out 
there on how to do it (or I successfully managed to google around it 
since days now ;).

I will be very glad for any bit of information that might help me to 
eliminate some of the below parameters.
And Again: I would be more than glad to contribute (back to the 
community) by assembling a wiki page about all this, once I figured out 
all the bits and pieces.

//// The situation ///

I am still trying to figure out how to install grub / grub2 on my intel 
imsm raid. And it's a striping RAID-0. Because the documentation out 
there is overall for RAID-1 (or 5) and the few bits I found for RAID-0, 
just do not work for me.

/// The questions ///

Maybe someone could shed some light on these open questions:

	a) Do I need to install it to the Volumne (/dev/md126) or to the 
Container (/dev/md127)? - My guess: The volume because it represents the 
RAID, hence should be what gets assembled for the int13 BIOS handler by 
the Option-ROM of the intel-sw-raid-controller.

	b) What device do I need to put in the /grub/boot/device.map file as (hd0):
	(/dev/md126, /dev/md127 or ...)?
	My guess: The same like a).

	c) Do you suggest grub1 (grub legacy) or grub2?

	    *** For grub2: The default way to install it: grub-install does not 
(seem to) work as grub2 then tries to find the /boot device itself and 
grub built from bzr does not (yet) support mdadm arrays with metadata 
v1.2 that are partitioned :(
	As I am quiet new to grub2 it would be very helpful to get an idea of 
what of the (many) tools to use and if possible with what parameters.
	Plus I wonder if with grub2 there will be problems because grub2 will 
try to assemble the RAID itsself.

	    *** For grub: I did quiet some installations but all my trials did 
always end in an Error 22 on "setup (hd0)" in the grub shell. <<< If 
grub would be your recommendation I can post the details!


Thanks a lot for any hint in advance!


K. Posern

P.S.: I would be still interested to hear how the two solutions that 
support an intel BIOS-supported (mdadm-imsm >=v3 and dmraid >=v1) raid 
compare to each other in terms of: speed (throughput), load/overhead 
(cpu/ram), stability, features.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3644 bytes --]

RE: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[Jiang, Dave]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of K. Posern
> Sent: Wednesday, August 18, 2010 9:56 PM
> To: linux-raid@vger.kernel.org
> Subject: getting a linux boot loader (preferably grub) installed on an
> intel imsm raid
> 
> 	b) based on mdadm-imsm. Just there seems really /no/
> documentation out
> there on how to do it (or I successfully managed to google around it
> since days now ;).

There is a section in regards to external metadata
https://raid.wiki.kernel.org/index.php/RAID_setup#External_Metadata

But I guess I need to update with additional information on how to install to it. 

Do you have Intel MSM Option ROM on your system? If so, then you create a RAID volume in the RAID optionROM and then when you do a install say for example Fedora 12, you specifiy you have "advanced storage device" (I think?) and you will see the RAID volume available for installation. I had issues with Fedora 13. There is a bug that causes crash when you attempt to install to an IMSM volume. And yes you install to the volume. The container can have 2 different RAID volumes. 
 
> P.S.: I would be still interested to hear how the two solutions that
> support an intel BIOS-supported (mdadm-imsm >=v3 and dmraid >=v1) raid
> compare to each other in terms of: speed (throughput), load/overhead
> (cpu/ram), stability, features.

Just FYI Intel is moving forward with MDRAID. 

Re: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[K. Posern]

[-- Attachment #1: Type: text/plain, Size: 3547 bytes --]

Hi Dave,

Thanks for your answer!

So it seems I might have found somebody to fight through this RAID/grub 
jungle? :)


/// about my hardware ///

What I have (and can offer as a test-platform): Sony VAIO notebook where
CTRL+I during startup brings me to
	Intel(R) Rapid Storage Technology - Option ROM - 9.5.0.1037
At the moment I have 1 RAID-0 on all 4 SATA drives.
I think this version (Sony limited?) only supports RAID-0 and maybe 1.

mdadm --examine on the container /dev/md127 gives me:
	Magic : Intel Raid ISM Cfg Sig
	Version : 1.2.01
	Faimily : ba0a95b0
	Generation : 0000025c

lspci shows me:
	00:1f.2 RAID bus controller: Intel Corporation Mobile 82801 SATA RAID 
Controller (rev. 05)


/// about Fedora (13) ////

My core goal was more to explore and document the MANUAL setup of 
grub/grub2 (for people compiling from source or using Gentoo).
Basically to document the knowledge needed to do-it-yourself in order to 
save people like me some precious time in collecting all the bits and 
pieces themselves.
For the several distributions out there: I would have probably just 
linked to their own HowTos once we get information that it works (like I 
got this information for Ubuntu for example).

I personally use Gentoo --> no fancy GUI/install-scripts, but everything 
manual :)

As I said: Ubuntu has a device-mapper based binary package of grub2 that 
seems to install itsself in a way that allows to boot from an Intel OROM 
RAID (fakeraid). But that doesn't help me with Gentoo.

But I am really stuck alone, so I am willing to throw a Fedora on my 
machine if in exchange I would get the grub commands to make grub boot 
my intel RAID-0 :)

 > Just FYI Intel is moving forward with MDRAID.
Nice :)

Do you happen to know if mdadm just integrated (parts of) the dmraid code?
Because that's one of the many /rumors/ I picked up ;)
(and that I would like to put to an end ;)

Let me know.

Thanks!

Knuth

---

On 19/08/10 13:49, Jiang, Dave wrote:
>> -----Original Message-----
>> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> owner@vger.kernel.org] On Behalf Of K. Posern
>> Sent: Wednesday, August 18, 2010 9:56 PM
>> To: linux-raid@vger.kernel.org
>> Subject: getting a linux boot loader (preferably grub) installed on an
>> intel imsm raid
>>
>> 	b) based on mdadm-imsm. Just there seems really /no/
>> documentation out
>> there on how to do it (or I successfully managed to google around it
>> since days now ;).
>
> There is a section in regards to external metadata
> https://raid.wiki.kernel.org/index.php/RAID_setup#External_Metadata
>
> But I guess I need to update with additional information on how to install to it.
>
> Do you have Intel MSM Option ROM on your system? If so, then you create a RAID volume in the RAID optionROM and then when you do a install say for example Fedora 12, you specifiy you have "advanced storage device" (I think?) and you will see the RAID volume available for installation. I had issues with Fedora 13. There is a bug that causes crash when you attempt to install to an IMSM volume. And yes you install to the volume. The container can have 2 different RAID volumes.
>
>> P.S.: I would be still interested to hear how the two solutions that
>> support an intel BIOS-supported (mdadm-imsm>=v3 and dmraid>=v1) raid
>> compare to each other in terms of: speed (throughput), load/overhead
>> (cpu/ram), stability, features.
>
> Just FYI Intel is moving forward with MDRAID.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3644 bytes --]

RE: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[Jiang, Dave]

> -----Original Message-----
> From: K. Posern [mailto:quickhelp@gmail.com]
> Sent: Thursday, August 19, 2010 11:33 AM
> To: Jiang, Dave
> Cc: linux-raid@vger.kernel.org
> Subject: Re: getting a linux boot loader (preferably grub) installed on
> an intel imsm raid
> I personally use Gentoo --> no fancy GUI/install-scripts, but
> everything
> manual :)
> 
> As I said: Ubuntu has a device-mapper based binary package of grub2
> that
> seems to install itsself in a way that allows to boot from an Intel
> OROM
> RAID (fakeraid). But that doesn't help me with Gentoo.
> 
> But I am really stuck alone, so I am willing to throw a Fedora on my
> machine if in exchange I would get the grub commands to make grub boot
> my intel RAID-0 :)

Well theoretically the OROM would present the RAID volume to the bootloader as a disk that it can read from. So if you mark your RAID volume for booting in BIOS, it should be able to pick up the bootloader from there right? Are you not able to install to /dev/md127 or whatever device node your RAID volume is on?

Also technically the imsm RAID is not fakeraid because it uses MDRAID instead of special driver to "fake" RAID in Linux. It's just Linux software RAID with OROM support. 
 
>  > Just FYI Intel is moving forward with MDRAID.
> Nice :)
> 
> Do you happen to know if mdadm just integrated (parts of) the dmraid
> code?
> Because that's one of the many /rumors/ I picked up ;)
> (and that I would like to put to an end ;)

I have seen some patches posted on this mailing list in regards to putting in DMRAID support so MDRAID can handle DMRAID arrays. I believe it's still work in progress. Neil can probably tell you more about that. 

Re: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[K. Posern]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

Hi Dave,

Thanks for your answer!

On 19/08/10 17:40, Jiang, Dave wrote:
>> -----Original Message-----
>> Subject: Re: getting a linux boot loader (preferably grub) installed on
>> But I am really stuck alone, so I am willing to throw a Fedora on my
>> machine if in exchange I would get the grub commands to make grub boot
>> my intel RAID-0 :)
>
> Well theoretically the OROM would present the RAID volume to the bootloader as a disk that it can read from. So if you mark your RAID volume for booting in BIOS, it should be able to pick up the bootloader from there right? Are you not able to install to /dev/md127 or whatever device node your RAID volume is on?
No. That is exactly the problem here. As mentioned in my original posting:

         *** For grub2: The default way to install it: grub-install 
/dev/md126 (= imsm volume) just returns 3 error messages.
      Even though the bzr version of grub2 does support mdadm.imsm raids 
with metada version >=1.2, it does /not/ support mdadm raids which are 
partitioned.
So I would need to put /dev/md126 into device.map (as the OROM of the 
intel fakeraid will assemble the raid-0 for grub to access it as a 
normal drive) and I need to install grub2 on the RAID-0 Volume 
/dev/md126! ... But how?
     As I am quiet new to grub2 it would be very helpful to get an idea 
of what of the (many) tools to use and if possible with what parameters.
     Plus I wonder if with grub2 there will be problems because grub2 
will try to assemble the RAID itsself.

         *** For grub: I did quite some installations but all my trials 
to install grub to /dev/md126 (= imsm Volume) did always end in an Error 
22 on "setup (hd0)" in the grub shell. <<< If grub would be your 
recommendation I can post the details!

> Also technically the imsm RAID is not fakeraid because it uses MDRAID instead of special driver to "fake" RAID in Linux. It's just Linux software RAID with OROM support.
I am not sure about the /official/ definition of the term "fakeraid". I 
would have said that because you can create/delete the RAID within the 
OROM (so without any Operating System (e.g. Software-RAID Program 
involved) --> "Fakeraid".

Actually: Is there any hardware support within the:
   00:1f.2 RAID bus controller: Intel Corporation Mobile 82801 SATA RAID 
Controller (rev. 05)
Like an xor chip that is used to outsource some of the work under 
windows, but not under linux (because as you stated: as under linux 
there is no special driver, but linux uses mdadm to access to access the 
container and the volume)?
If yes, are there any plans to develop a driver to outsource the xors to 
the RAID chip for Linux?

Knuth


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3644 bytes --]

RE: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[Jiang, Dave]

> -----Original Message-----
> From: K. Posern [mailto:quickhelp@gmail.com]
> Sent: Thursday, August 19, 2010 3:30 PM
> To: Jiang, Dave
> Cc: linux-raid@vger.kernel.org
> Subject: Re: getting a linux boot loader (preferably grub) installed on
> an intel imsm raid
>          *** For grub2: The default way to install it: grub-install
> /dev/md126 (= imsm volume) just returns 3 error messages.
>       Even though the bzr version of grub2 does support mdadm.imsm
> raids
> with metada version >=1.2, it does /not/ support mdadm raids which are
> partitioned.
> So I would need to put /dev/md126 into device.map (as the OROM of the
> intel fakeraid will assemble the raid-0 for grub to access it as a
> normal drive) and I need to install grub2 on the RAID-0 Volume
> /dev/md126! ... But how?
>      As I am quiet new to grub2 it would be very helpful to get an idea
> of what of the (many) tools to use and if possible with what
> parameters.
>      Plus I wonder if with grub2 there will be problems because grub2
> will try to assemble the RAID itsself.
> 
>          *** For grub: I did quite some installations but all my trials
> to install grub to /dev/md126 (= imsm Volume) did always end in an
> Error
> 22 on "setup (hd0)" in the grub shell. <<< If grub would be your
> recommendation I can post the details!

I will have to create a setup and see how that works with grub. I guess the better question would also be how do distros like SUSE or Fedora/RHEL perform this as they use grub as well....

> 
> > Also technically the imsm RAID is not fakeraid because it uses MDRAID
> instead of special driver to "fake" RAID in Linux. It's just Linux
> software RAID with OROM support.
> I am not sure about the /official/ definition of the term "fakeraid". I
> would have said that because you can create/delete the RAID within the
> OROM (so without any Operating System (e.g. Software-RAID Program
> involved) --> "Fakeraid".
> 
> Actually: Is there any hardware support within the:
>    00:1f.2 RAID bus controller: Intel Corporation Mobile 82801 SATA
> RAID
> Controller (rev. 05)
> Like an xor chip that is used to outsource some of the work under
> windows, but not under linux (because as you stated: as under linux
> there is no special driver, but linux uses mdadm to access to access
> the
> container and the volume)?
> If yes, are there any plans to develop a driver to outsource the xors
> to
> the RAID chip for Linux?

Yes. But you will need the Xeon class CPUs (C55xx and C35xx I think?) that has the IOAT DMA (CBDMA) engines to do this. If you have the IOAT DMA engines that has XOR caps, Linux can use the DMA engine to do XOR or PQ offloading. See drivers/dma/ioat/

RE: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[Jiang, Dave]

> -----Original Message-----
> From: K. Posern [mailto:quickhelp@gmail.com]
> Sent: Thursday, August 19, 2010 3:30 PM
> To: Jiang, Dave
> Cc: linux-raid@vger.kernel.org
> Subject: Re: getting a linux boot loader (preferably grub) installed on
> an intel imsm raid
> No. That is exactly the problem here. As mentioned in my original
> posting:
> 
>          *** For grub2: The default way to install it: grub-install
> /dev/md126 (= imsm volume) just returns 3 error messages.
>       Even though the bzr version of grub2 does support mdadm.imsm
> raids
> with metada version >=1.2, it does /not/ support mdadm raids which are
> partitioned.
> So I would need to put /dev/md126 into device.map (as the OROM of the
> intel fakeraid will assemble the raid-0 for grub to access it as a
> normal drive) and I need to install grub2 on the RAID-0 Volume
> /dev/md126! ... But how?
>      As I am quiet new to grub2 it would be very helpful to get an idea
> of what of the (many) tools to use and if possible with what
> parameters.
>      Plus I wonder if with grub2 there will be problems because grub2
> will try to assemble the RAID itsself.
> 
>          *** For grub: I did quite some installations but all my trials
> to install grub to /dev/md126 (= imsm Volume) did always end in an
> Error
> 22 on "setup (hd0)" in the grub shell. <<< If grub would be your
> recommendation I can post the details!

So I created a RAID0 like your setup. I have md0 as the container and md127 as the RAID0 volume. Fedora 12 installed grub on /dev/md127. The grub version from F12 is 0.97. I was able to rerun grub-install on /dev/md127 successfully. My device map contains:
(hd0)		/dev/md127

Do you have to use grub2?

Re: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[K. Posern]

[-- Attachment #1.1: Type: text/plain, Size: 739 bytes --]

Thanks again!

On 19/08/10 20:46, Jiang, Dave wrote:
>> -----Original Message-----
> So I created a RAID0 like your setup. I have md0 as the container and md127 as the RAID0 volume. Fedora 12 installed grub on /dev/md127. The grub version from F12 is 0.97. I was able to rerun grub-install on /dev/md127 successfully. My device map contains:
> (hd0)		/dev/md127
>
> Do you have to use grub2?
No.

Attached my results with grub-0.97 (-r10).
The error 22 I mentioned... maybe I am just not seeing it... like 
something /very/ obvious... ?

/dev/md/126 is my volume with the partitions, /dev/md127 is my container

I can give you ssh access to the machine, just send me your pub key.

XMPP chat: tormen@jabber.org

Knuth

[-- Attachment #1.2: grub-1.txt --]
[-- Type: text/plain, Size: 961 bytes --]


/vola # grub-install /dev/md126                                   $1 #367 root@sysresccd pts/1

expr: non-integer argument


    GNU GRUB  version 0.97  (640K lower / 9216K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename. ]
grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup  --stage2=/boot/grub/stage2 --prefix=/grub (hd0,)
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
 Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
 Running "install --stage2=/boot/grub/stage2 /grub/stage1 (hd0,0) /grub/stage2 p /grub/menu.lst "... failed

Error 22: No such partition
grub> quit

[-- Attachment #1.3: grub-2.txt --]
[-- Type: text/plain, Size: 1477 bytes --]


/vola # grub --device-map=/dev/null                $1 #367 root@sysresccd pts/1


    GNU GRUB  version 0.97  (640K lower / 9216K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename. ]

grub> device(hd0) /dev/md126

Error 27: Unrecognized command

grub> device (hd0) /dev/md126

grub> root (hd0,
 Possible partitions are:
   Partition num: 0,  Filesystem type is ext2fs, partition type 0x83
   Partition num: 2,  Filesystem type unknown, partition type 0x7
   Partition num: 4,  Filesystem type unknown, partition type 0x82
   Partition num: 5,  Filesystem type is ext2fs, partition type 0x83
   Partition num: 6,  Filesystem type unknown, partition type 0x83
   Partition num: 7,  Filesystem type unknown, partition type 0x83
   Partition num: 8,  Filesystem type unknown, partition type 0x83
   Partition num: 9,  Filesystem type unknown, partition type 0x7

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  18 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+18 p (hd0,0)/boot/grub/stage2
/boot/grub/menu.lst"... failed

Error 22: No such partition

grub>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3644 bytes --]

RE: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[gabe peters]

It's my first time using a mailing list in many years, hopefully i've
got this threaded properly. (Intended as a reply to "Jiang, Dave" and
"K. Posern")

I've got an intel MSM RAID0 set up nearly identical to K. Posern, my
differences are that i'm using an older revision controller (lspci
shows revision 02) - the 4 SATA drives are SSDs (though that shouldn't
matter), the imsm container is set up as /dev/md127 and the raid
volume (RAID-0) as /dev/md126. I am able to successfully read and
write from all partitions on the RAID-0 (I set up windows 7 on another
partition prior to attempting to get the linux partition(s) bootable)

Aside from the differences in the partition table, all errors I'm
receiving are the same. Running under Gentoo using the svn -9999
ebuild for grub2 (also tried grub in multiple ways receiving the same
error 22 message)

Incidental to the errors, my motivation to run with this configuration
is to maintain a dual boot with Windows 7 x64 Ultimate. I previously
had an mdadm software raid set up (RAID-1 /boot and RAID-0 /) which I
had no problem booting up with. But ideally i'd like to be able to use
Grub/Grub2 as the bootloader, and load both linux (Gentoo multilib
x86_64) and Windows 7 x64 from the RAID-0.

Any insight will be welcome, I suppose this message amounts to nothing
more than a simple "me, too!" though!

Re: getting a linux boot loader (preferably grub) installed on an intel imsm raid

[K. Posern]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

Hi Gabe,

On 21/08/10 03:53, gabe peters wrote:
 > Any insight will be welcome, I suppose this message amounts to nothing
 > more than a simple "me, too!" though!

I got it working! Thanks to the *great* support of phcoder from the 
grub2 community!

He developed a patch that allows the user a sane way to override 
auto-detection mechanisms by supplying right device.map. It should get 
merged to bzr in the next couple of days :) (it is not yet ideally 
coded, but works :)

--> Basically allowing me to use /dev/md126 (the imsm Volume) as a grub 
device, even though grub can't assemble it --> hence to use the int13 
OROM assembled version of the raid-0 :)

If you want to give it a try:
     # http://www.gnu.org/software/grub/grub-download.en.html
     bzr branch http://bzr.savannah.gnu.org/r/grub/trunk/grub
     # Apply this patch: http://pastebin.com/raw.php?i=tgfqp9Hn
     # e.g. download it to file "phcoder.patch"
     # and run cat phcoder.patch|patch -p 0 in the "grub" directory that 
you received by the bzr command

     # I think without any parameters it should be fine:
     ./configure
     make
     # this command triggered for me that /boot/grub got populated:
     # for me /dev/mdYOURVOLUME was /dev/md126 (the imsm Volume)
     ./grub-probe /dev/mdYOURVOLUME
     # make sure that /boot/grub/device.map contains:
	(hd0) /dev/mdYOURVOLUME
     ./grub-probe /dev/mdYOURVOLUME -v
     ./grub-probe -t partmap /boot -v
     # until know we changed NOTHING...
     # ... so if you can also do this:
     ./grub-install /dev/mdYOURVOLUME
     reboot

Hope this helps you as well.


Best,

Knuth


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3644 bytes --]