embedding area is unusually small... (GRUB2 on software RAID1)

embedding area is unusually small... (GRUB2 on software RAID1)

[Lapohos Tibor]

Hello, 

I successfully set up an Intel Matrix Raid device with a RAID1 and a RAID0 volume, each having a couple of partitions, but then I could not install GRUB2 on the RAID1 volume, which I wanted to use to boot from and mount as root. It turned out that the "IMSM" metadata is not supported in GRUB2 (v1.97.1) just yet, so I had to turn away from my original plan. 

To "imitate" the setup I originally wanded, I turned both of my drives into AHCI controlled devices in the BIOS (instead of RAID), and I partitioned them to obtain /dev/sda[12] and /dev/sdb[12]. 

Then I used /dev/sd[ab]1 to build a RAID1 set, and /dev/sd[ab]2 to create a RAID0 set using mdadm v 3.0.3: 

> mdadm -C /dev/md0 -v -e 0 -l 1 -n 2 /dev/sda1 /dev/sdb1 
> mdadm -C /dev/md1 -v -e 0 -l 0 -n 2 /dev/sda2 /dev/sdb2 

I set the metadata type to 0.90 because I would like to boot from it and allow the kernel to auto-detect the RAID devices while it's booting, in order to can get away from using an intitrd (I am building my own distribution based on CLFS x86_64 multilib). 

I used cfdisk to partition both of the /dev/md[01] devices, and I obtained /dev/md0p[123] and /dev/md1p[12]. The plan is to use /dev/md0p1 as a RAID1 root partition, and have the system boot from /dev/md0. I formatted /dev/md0p1 as 

> mk2efs -t ext4 -L OS /dev/md0p1 

To this point, things went smoothly. mdadm -D... and mdadm -E... did report back working devices as intended. Then mounted /dev/md0p1 on a directory called /root/os, and I did 

> grub-install --root-directory=/root/os /dev/md0 

or 

> grub-install --root-directory=/root/os "md0" 

and I got a warning and an error message: "Your embedding area is unusually small.  core.img won't fit in it." and "Embedding is not possible, but this is required when the root device is on a RAID array or LVM volume." 

What did I do wrong, and how do I fix it? Thanks ahead, 
Tibor

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Antonio Perez]

Lapohos Tibor wrote:

[...]
>> grub-install --root-directory=/root/os /dev/md0

> and I got a warning and an error message: "Your embedding area is
> unusually small.  core.img won't fit in it." and "Embedding is not
> possible, but this is required when the root device is on a RAID array or
> LVM volume."
 
> What did I do wrong, and how do I fix it? Thanks ahead,

Grub2 needs a space of about 55 sectors free after the MBR. It ussually is 
63 sectors. What's the output of:
	fdisk -lu /dev/sda
the start of the first partition should be 63 sectors or bigger to leave 
enough space for grub2 to be embedded.

However, the way you are making partitions is quite complex and the needed 
space could be somewhere else.

Please, stop repeating your question everywhere.

-- 
Antonio Perez

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Lapohos Tibor]

Thanks for your reply, Antonio.

--- On Wed, 1/6/10, Antonio Perez <ap23563m@gmx.com> wrote:

> What's the output of:
>     fdisk -lu /dev/sda

I will look at it tonight. Thanks.

> However, the way you are making partitions is quite complex
> and the needed space could be somewhere else.

Does this mean that this partitioning is not the right way to proceed? should I create all the partitions first and build them into so many /dev/md[01234] devices separately maybe?

> Please, stop repeating your question everywhere.

I am very sorry if I upset the community. I did not think that linux-raid, grub-devel and help-grub lists would represent the same experts. In fact, based on what I see on the GRUB sites is, that grub-devel experts are different from more like "legacy" supporting help-grub ones. Maybe a bit of clarification would help keeping the developers/maintainers email intake a bit lower. So I must ask: which one is the most appropriate for my question?

Thanks again,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Antonio Perez]

Lapohos Tibor wrote:

> Thanks for your reply, Antonio.
> 
> --- On Wed, 1/6/10, Antonio Perez <ap23563m@gmx.com> wrote:
> 
>> What's the output of:
>> fdisk -lu /dev/sda
> 
> I will look at it tonight. Thanks.

Good.

>> However, the way you are making partitions is quite complex
>> and the needed space could be somewhere else.

You did "mdadm -C /dev/md0 -v -e 0 -l 1 -n 2 /dev/sda1 /dev/sdb1"

Which is the usual way to set the /boot partition.

However, you did repartition md0, while still expecting to boot from md0.

That is "odd" to me. Just make a FS in md0 and use it as /boot (and root).
 
> Does this mean that this partitioning is not the right way to proceed?
> should I create all the partitions first and build them into so many
> /dev/md[01234] devices separately maybe?

I could tell you what seems to be a reasonable setup IMO:
	200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
	5Gb   --> {sda2,sdb2} --> md2 --> RAID1 --> use as /
	2GB   --> independent sda3, sdb3  --> use as swap

		Use the rest as needed. One option is:
	rest --> {sda3,sdb3} --> md3 --> RAID1 --> LVM --> crete as many individual 
		disks as needed/wanted with lvcreate.

Having only two disks means to use only RAID1. RAID0 is too risky IMO as any 
disk failure means a complete loss of data.

>> Please, stop repeating your question everywhere.
 
> I am very sorry if I upset the community. I did not think that linux-raid,
> grub-devel and help-grub lists would represent the same experts. In fact,
> based on what I see on the GRUB sites is, that grub-devel experts are
> different from more like "legacy" supporting help-grub ones. Maybe a bit
> of clarification would help keeping the developers/maintainers email
> intake a bit lower. So I must ask: which one is the most appropriate for
> my question?

As I am not an expert in any sense of the word, I really can't tell you 
where to post. I am a common user which is trying to help you. But, in this 
list alone, there are three post from you with exactly the same content and 
question. If you are not getting any feedback, perhaps you should re-think 
of a different way to make your questions. And, please, allow more time for 
an answer.
Thanks for your cooperation.

-- 
Antonio Perez

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Lapohos Tibor]

Thank you Antonio!

--- On Wed, 1/6/10, Antonio Perez <ap23563m@gmx.com> wrote:

> However, you did repartition md0, while still expecting to
> boot from md0.
> 
> That is "odd" to me. Just make a FS in md0 and use it as>
> /boot (and root).

I've been, and still am, puzzled by the efficiency of the calculations involved in setting up RAID, one way or the other. And what I refer to, goes down to number of clocks needed to perform the writing. As I was wondering down this path, I figured that the way I set it up would be the most efficient. Of course, I do not know enough about the actual kernel/raid code to be able to tell for sure. My intuition dictated it this way.

> > should I create all the partitions first and build
> > them into so many /dev/md[01234] devices separately maybe?
>
> I could tell you what seems to be a reasonable setup IMO:
>     200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
>     5Gb   --> {sda2,sdb2} --> md2 --> RAID1 --> use as /
>     ...

....and that's exactly what is also on my list, and your suggestion enforces my thoughts, thanks. Nevertheless, it still does not explain to me whether multiple md devices with more underlying disk partitions or more md partitions with fewer underlying disk partitions are the way to go. To be absolutely clear, is the

/dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
/dev/sd[ab]2 => /dev/md1 => /dev/md1p[123...] [RAID0]

or the

/dev/sd[ab][123...] => /dev/md[012...] [RAID1,RAID0]

setup the most stable, efficient and robust solution? The way I see it, there shall be a difference somewhere, somehow, right? About the first one, thanks to Michael E. pointing it out, we already know that one has to take care about choosing the patritions.

Furthermore, I think that the gain in speed of RAID0 would start vanishing if disk access is required on the RAID1 side all at the same time. In fact, the whole system's efficiency would degrade. The only way out would be to use 4 disks as

/dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
/dev/sd[cd]1 => /dev/md1 => /dev/md1p[123...] [RAID0]

Right? Keep in mind that I am not after what RAID10 can offer. I can afford giving up the data safety on the RAID0 side in exchange for speed.

> Having only two disks means to use only RAID1. RAID0 is too
> risky IMO as any disk failure means a complete loss of data.

I will be developing a numerical electromagnetics code, that will do quite some number crunching and dumping with 4xTesla C1060 GPUs. I will need the RAID0 for that purpose, for a scratch space, if you will. So it is a must have for me.

> I am a common user which is trying to help you.

Thanks, it's highly appreciated.

> But, in this list alone, there are three post from you with exactly the
> same content and question. 

That's really odd. I would never do that intentionally. As I wrote earlier, I did submit it to 3 lists, help-grub, grub-devel and linux-raid, but not to the same one 3 times. The subject is related to both RAID and GRUB, which is why I sent it 3 ways. My apologies again to all those who got it in 3 copies, whether by being subscribed to all 3 lists, or by a misterious triple submission of mine. 

> If you are not getting any feedback, perhaps you should re-think 
> of a different way to make your questions. And, please,
> allow more time for an answer.

I am a patient man, so not getting answers would not drive me to re-post the same message again.

> Thanks for your cooperation.

Absolutely. And I thank to all for your help.

All the best,
Tibor

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Antonio Perez]

Lapohos Tibor wrote:

[...]
>> > should I create all the partitions first and build
>> > them into so many /dev/md[01234] devices separately maybe?

>> I could tell you what seems to be a reasonable setup IMO:
>> 200Mb --> {sda1,sdb1} --> md1 --> RAID1 --> use as /boot
>> 5Gb   --> {sda2,sdb2} --> md2 --> RAID1 --> use as /
>> ...
> 
> ....and that's exactly what is also on my list, and your suggestion
> enforces my thoughts, thanks. Nevertheless, it still does not explain to
> me whether multiple md devices with more underlying disk partitions or
> more md partitions with fewer underlying disk partitions are the way to
> go. To be absolutely clear, is the
> 
> /dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
> /dev/sd[ab]2 => /dev/md1 => /dev/md1p[123...] [RAID0]

I never meant to say to re-partition the md device!!
Why is that needed in a small 200 MByte partition?
I really like the a KISS approach.
 
> or the

> /dev/sd[ab][123...] => /dev/md[012...] [RAID1,RAID0]

> setup the most stable, efficient and robust solution? The way I see it,
> there shall be a difference somewhere, somehow, right? About the first
> one, thanks to Michael E. pointing it out, we already know that one has to
> take care about choosing the patritions.

I can't say what differences or efficiency is there, but it doesn't work!!
So, what does it matter?
Boot your system with the SIMPLE two RAID I stated above, and then, play as 
you wish with as many raid levels and sub-partitions as you want with the 
*rest* of the disks.
 
> Furthermore, I think that the gain in speed of RAID0 would start vanishing
> if disk access is required on the RAID1 side all at the same time.

> In fact, the whole system's efficiency would degrade. The only way out
> would be to use 4 disks as
> 
> /dev/sd[ab]1 => /dev/md0 => /dev/md0p[123...] [RAID1]
> /dev/sd[cd]1 => /dev/md1 => /dev/md1p[123...] [RAID0]
> 
> Right? Keep in mind that I am not after what RAID10 can offer. I can
> afford giving up the data safety on the RAID0 side in exchange for speed.

Yes, right. Do you have four disks?

>> Having only two disks means to use only RAID1. RAID0 is too
>> risky IMO as any disk failure means a complete loss of data.

> I will be developing a numerical electromagnetics code, that will do quite
> some number crunching and dumping with 4xTesla C1060 GPUs. I will need the
> RAID0 for that purpose, for a scratch space, if you will. So it is a must
> have for me.

I was just warning about a possible loss of data, if that is not an issue in 
your system, do set up as many RAID0 as you need.

-- 
Antonio Perez

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Lapohos Tibor]

Thanks for all your help again.

Things worked out nicely, and they left me wondering about what had really happened.

Trial #1:
I wiped out the disks, repartitioned them using fdisk starting the first partition at the 2nd cylinder to obtain sd[ab][12], and made md0 = sd[ab]1 and md1 = sd[ab]2. Installed GRUB2 without a problem.

Trial #2:
Then I repeated this very same thing, only this time I further partitioned the md[01] drives. The GRUB installation worked, again. The only problem was that before needing the root partition to become available the kernel would not detect the RAID partititons, only the arrays. So md0p1 was not available to be mounted, and this way I would need an initrd in the process.

Trial #3!!
I repeated Trial #1, by starting the firsst partition at the first cylinder. It worked! Therefore, I'm suspecting that cfdisk could have been the culprit the first time. How, I don't know.

Thanks again for all the attention you gave me,
Tibor

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Antonio Perez]

Lapohos Tibor wrote:

> Thanks for all your help again.

YW, all is well that ends well.

[...]
> ....How, I don't know.

Did you check the "fdisk -lu /dev/sda" output?

> Thanks again for all the attention you gave me,

YW, and good luck with your "numerical electromagnetics code".

-- 
Antonio Perez

Re: embedding area is unusually small... (GRUB2 on software RAID1)

[Lapohos Tibor]

> > ....How, I don't know.
> 
> Did you check the "fdisk -lu /dev/sda" output?
> 
Yes. 63, so I still went on and repartitioned as described in Trial #[123].

I'm thinking that it could have been something left behind by the Intel Matrix RAID volume management... Maybe Neil Brown could give an explanation to this. I also noticed that if one re-partitions the exact same way, the RAID superblock information becomes available again, that is, partitioning or disassembling a RAID device does not threcken those records off the disks. Therefore, even if one does not create a partitioned md array, it will be, on its own! Actually it makes sense, since partitioning does not involve alteration of records throughout the disks... I just thought it would be interesting to share all this.

In the near future I will describe my story in a single post, maybe it will inspire some developers to address certain points in their documentation work.

Thanks again to All,
Tibor