Putting core.img anywhere

Putting core.img anywhere

[Jean-Christophe Haessig]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

Hi,

I have an unusual LVM setup where GRUB cannot be installed (see last
Threads on LVM) because there is no room for core.img.

The only place where boot data can be put is on a logical volume, but
the bootsector (one of diskboot.S or boot.S, I assume) isn't smart
enough to understand LVM or filesystems.

However, I can ensure that my /boot logical volume is not too far from
the beginning of the disk, and I thought about the following algorithm :

Every 512-byte chunk of core.img contains :
496 bytes  data
  2 bytes  magic value
  2 bytes  chunk number
  4 bytes  random integer
  4 bytes  random integer in next chunk
  4 bytes  on-disk sector where this chunk is located (serves as hint)

The boot sector is patched with the random integer found in the first
chunk, and the on-disk sector where it is located. On boot, this disk
sector is loaded and the boot sector code checks if the random integer
matches.
If yes, the disk sector and the expected random integer are updated with
the values found in the newly read block and the program proceeds with
the next block.
If the number doesn't match, the boot code scans the disk from sector 1
until it finds the sector having the correct random integer, expected
magic value and chunk number.

This algorithm should work, regardless from where core.img is stored. It
can even be on a filesystem, as long as file chunks match disk blocks
(which is the case for ext2, I think). Filesystems doing strange things
with files may not work, but usually they aren't used for /boot anyway.

I'm not used to assembly programming, but I'll give it a try. Which file
contains the bootcode I should edit ? boot.S or diskboot.S ? Any
comments ?

JC


[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Putting core.img anywhere

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

El sáb, 19-07-2008 a las 19:24 +0200, Jean-Christophe Haessig escribió:
> Hi,
Hi there!
> 
> I have an unusual LVM setup where GRUB cannot be installed (see last
> Threads on LVM) because there is no room for core.img.
This should have been fixed by the transition to LZMA as the compression
algorithm for PC - core.img should now be under 32K and embeddable in
the 32256 bytes available before the first MBR partition. PC+GPT schemes
might be clumsier, since you'd stamp the main GPT (though the system
keeps working with the backup GPT at the end of the disk, as one of my
laptops proves).

> (...)
> However, I can ensure that my /boot logical volume is not too far from
> the beginning of the disk, and I thought about the following algorithm :
How can you do so? Is there a way to force LVM to put a certain LV
within a particular region of a PV within the VG? At most, you could
split your disk in two partitions, one small and one big; and format
both as LVM PVs, then add them to the same VG. You can then force
the /boot LV within the VG to lie in the small PV, but if you take the
trouble to do this it's quite simpler to just create a non-LVM boot
partition.

> 
> Every 512-byte chunk of core.img contains :
> 496 bytes  data
>   2 bytes  magic value
>   2 bytes  chunk number
>   4 bytes  random integer
>   4 bytes  random integer in next chunk
>   4 bytes  on-disk sector where this chunk is located (serves as hint)
> 
> The boot sector is patched with the random integer found in the first
> chunk, and the on-disk sector where it is located. On boot, this disk
> sector is loaded and the boot sector code checks if the random integer
> matches.
> If yes, the disk sector and the expected random integer are updated with
> the values found in the newly read block and the program proceeds with
> the next block.
> If the number doesn't match, the boot code scans the disk from sector 1
> until it finds the sector having the correct random integer, expected
> magic value and chunk number.
Theoretically it would work, but this is heavy duty work we're talking
about - potentially reading the whole disk if a single file has moved
due to, say, a defrag. Besides, we could hit one of the several BIOS
disk size limits, so you would have to ensure (as you said above) that
core.img lies within the reasonable limits for your BIOS (8G? 32G?
128G?)
> 
> This algorithm should work, regardless from where core.img is stored. It
> can even be on a filesystem, as long as file chunks match disk blocks
> (which is the case for ext2, I think). Filesystems doing strange things
> with files may not work, but usually they aren't used for /boot anyway.
It would work as long as the filesystem stores 512-byte blocks together.
However, with tail-packing features _might_ pose a problem; and
filesystems that altered the data in any way, like NTFS compression or
some kind of inline checksumming/signing would be a no-go.


[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: Putting core.img anywhere

[Jean-Christophe Haessig]

[-- Attachment #1: Type: text/plain, Size: 2739 bytes --]

Le dimanche 20 juillet 2008 à 17:06 +0200, Javier Martín a écrit :
Hi,

> This should have been fixed by the transition to LZMA as the compression
> algorithm for PC - core.img should now be under 32K and embeddable in
> the 32256 bytes available before the first MBR partition.

As I stated earlier my disks are not DOS-partitioned (e.g.
pvcreate /dev/hda). I didn't want to have one useless level of the old
DOS partition scheme since LVM already does the job (better).

> > However, I can ensure that my /boot logical volume is not too far from
> > the beginning of the disk, and I thought about the following algorithm :
> How can you do so? Is there a way to force LVM to put a certain LV
> within a particular region of a PV within the VG?

Short answer : yes. Longer answer : LVM does not (yet) gratuitously move
LVs among PVs, LVs stay where they have been created and when you create
a LV on an empty PV, it is normally allocated at the beginning of the
disk in continuous extents. And yes, using pvmove you can move data
anywhere you like.

> split your disk in two partitions, one small and one big; and format
> both as LVM PVs, then add them to the same VG. You can then force
> the /boot LV within the VG to lie in the small PV, but if you take the
> trouble to do this it's quite simpler to just create a non-LVM boot
> partition.

Yes, and were I in that case, I wouldn't have problems anyway.

> Theoretically it would work, but this is heavy duty work we're talking
> about - potentially reading the whole disk if a single file has moved
> due to, say, a defrag.

That's why I included the random-number-chain feature. If the file is
accidentally moved, it can still be found by the bootsector.
Additionnally, GRUB could display a message about this to the user,
telling him that it is in degraded mode and that he should re-run some
utility to reindex the file.
As for reading the entire disk, an arbitrary upper limit could be put to
force the boot sector to fail.

>  Besides, we could hit one of the several BIOS
> disk size limits, so you would have to ensure (as you said above) that
> core.img lies within the reasonable limits for your BIOS (8G? 32G?
> 128G?)

I'm confident that my /boot LV does not exceed the first 200M of the
drive.

> It would work as long as the filesystem stores 512-byte blocks together.
> However, with tail-packing features _might_ pose a problem; and
> filesystems that altered the data in any way, like NTFS compression or
> some kind of inline checksumming/signing would be a no-go.

Sure, but who would use NTFS for their /boot ? Most filesystems should
work already, I think, otherwise even LILO couldn't work…

JC


[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Putting core.img anywhere

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 4567 bytes --]

El lun, 21-07-2008 a las 11:13 +0200, Jean-Christophe Haessig escribió:
> Le dimanche 20 juillet 2008 à 17:06 +0200, Javier Martín a écrit :
> Hi,
> 
> > This should have been fixed by the transition to LZMA as the compression
> > algorithm for PC - core.img should now be under 32K and embeddable in
> > the 32256 bytes available before the first MBR partition.
> 
> As I stated earlier my disks are not DOS-partitioned (e.g.
> pvcreate /dev/hda). I didn't want to have one useless level of the old
> DOS partition scheme since LVM already does the job (better).
Well, I can't dissent on that, but then your whole HD, from the first to
the last LBA block, is full with data that can move at LVM's whim.

> > > However, I can ensure that my /boot logical volume is not too far from
> > > the beginning of the disk, and I thought about the following algorithm :
> > How can you do so? Is there a way to force LVM to put a certain LV
> > within a particular region of a PV within the VG?
> 
> Short answer : yes. Longer answer : LVM does not (yet) gratuitously move
> LVs among PVs, LVs stay where they have been created and when you create
> a LV on an empty PV, it is normally allocated at the beginning of the
> disk in continuous extents. And yes, using pvmove you can move data
> anywhere you like.
You can move a LV to a certain PV, but apart from that there is not a
point in the LVM "specification" or "documentation" that I've seen that
can allow you to move data into _specific_ PEs _and_ keep it there:
pvmove does the former but does not guarantee they will still be there
in, say, a week: the only way I can think of would be to tie the /boot
LV to a PV near the start of the disk, but as you said, then you
wouldn't be in this dilemma.

> > Theoretically it would work, but this is heavy duty work we're talking
> > about - potentially reading the whole disk if a single file has moved
> > due to, say, a defrag.
> 
> That's why I included the random-number-chain feature. If the file is
> accidentally moved, it can still be found by the bootsector.
> Additionnally, GRUB could display a message about this to the user,
> telling him that it is in degraded mode and that he should re-run some
> utility to reindex the file.
> As for reading the entire disk, an arbitrary upper limit could be put to
> force the boot sector to fail.
First of all, LVM2 LVs don't have to keep consistency iIrc: its PEs can
be (though usually aren't) scattered all over the VG PVs, which means
that part of core.img could be in LBA blocks 300-330 but the other
fragment might be in blocks 814567-814592. In the worst case, your
system might have to read the whole disk _once per block_ in core.img,
which is about 50 blocks long. That might be a _big_ hit.

If that weren't enough, all your system would have to fit in the
already-cramped bootsector image, since you said that there is not a
single block in the HD available for embedding. In fact, do you know if
LVM leaves space in block #0 (its "superblock") for boot code like
GRUB's? If not, you can't even install GRUB, since there is no place to
install it to.

A suggestion: if you want to keep your no-partitions schema, why don't
you boot from a floppy, a CD or a USB pendrive? You would save a lot of
headaches...

> > It would work as long as the filesystem stores 512-byte blocks together.
> > However, with tail-packing features _might_ pose a problem; and
> > filesystems that altered the data in any way, like NTFS compression or
> > some kind of inline checksumming/signing would be a no-go.
> 
> Sure, but who would use NTFS for their /boot ? Most filesystems should
> work already, I think, otherwise even LILO couldn't work…
The same kind of geek* that does not partition his hard drives ;)
Besides, I was only using NTFS compression as an example: extN
filesystems also have a compression feature (though it's not very used).
Any other kind of inline addition/alteration of the data on store by the
FS (like the addition of a checksum each 128 bytes or so) or the
breakage of the assumption that _our_ blocks are still block-aligned on
the filesystem would break your system too. Nevertheless, those
assumptions, particularly the latter, usually hold, so your system is
already quite robust as things go.

Cheers!
Habbit

* I did the same once, formatting a whole disk reiserfs for a temporary
storage addition. I've also partitioned an OLD pc laptop with GPT, so
yeah, I kinda like experimenting with partitions

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: Putting core.img anywhere

[Robert Millan]

On Mon, Jul 21, 2008 at 11:13:08AM +0200, Jean-Christophe Haessig wrote:
> 
> As I stated earlier my disks are not DOS-partitioned (e.g.
> pvcreate /dev/hda). I didn't want to have one useless level

Turns out it wasn't as useless as you thought :-)

> since LVM already does the job (better).

LVM does the job of managing storage partitions, but not the "job" of reserving
a gap for GRUB to use.

Furthermore, the scheme you propose looks overly complicated [1] for the task
it's meant to accomplish [2].

Thinking about it, I think the root of your problem is that you would like
your storage system (LVM) to be completely unrelated to your boot system
(GRUB).  But BIOS legacy cruft won't let you do that.  So maybe a solution
that would work well for you is switching to Coreboot [3]?

[1] random-number-chain??
[2] we don't need anything fancy, just an area of disk that isn't used by
    anything else.
[3] http://www.coreboot.org/

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: Putting core.img anywhere

[Jean-Christophe Haessig]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

Le mardi 22 juillet 2008 à 23:47 +0200, Robert Millan a écrit :
>  
> > As I stated earlier my disks are not DOS-partitioned (e.g.
> > pvcreate /dev/hda). I didn't want to have one useless level
> 
> Turns out it wasn't as useless as you thought :-)

heh :)

> > since LVM already does the job (better).
> 
> LVM does the job of managing storage partitions, but not the "job" of reserving
> a gap for GRUB to use.

LVM leaves some sectors at the beginning of the disk. This can be chosen
at pvcreate time, AFAIK one can leave between 1 and 4 sectors before the
LVM header. Which is sufficient since LILO does fine with such a setup,
after all. 

> Furthermore, the scheme you propose looks overly complicated [1]

Complicated ?? You must be kidding me ! A ten-year old would understand
it. It is way more complicated to include filesystem support in a
bootloader.
In fact I thought about this scheme because one of the features of GRUB
(compared to LILO) was to work without having to reinstall each time a
kernel is altered. I included the random number chain scheme to keep the
thing woking even if the core.img file was moved by accident. Random
numbers were just here to minimize the risk of using a wrong block that
just happens to have the correct value. 
But using a simple blocklist (just as LILO does) to locate the core.img
would also work (and is quite less complicated), just a little less
robust. I wonder why this isn't alredady possible, BTW.

> maybe […] switching to Coreboot [3]?

Coreboot is, indeed, interesting.

JC

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Putting core.img anywhere

[Robert Millan]

On Wed, Jul 23, 2008 at 11:40:27AM +0200, Jean-Christophe Haessig wrote:
> > Furthermore, the scheme you propose looks overly complicated [1]
> 
> Complicated ?? You must be kidding me ! A ten-year old would understand
> it. It is way more complicated to include filesystem support in a
> bootloader.
> In fact I thought about this scheme because one of the features of GRUB
> (compared to LILO) was to work without having to reinstall each time a
> kernel is altered. I included the random number chain scheme to keep the
> thing woking even if the core.img file was moved by accident. Random
> numbers were just here to minimize the risk of using a wrong block that
> just happens to have the correct value. 
> But using a simple blocklist (just as LILO does) to locate the core.img
> would also work (and is quite less complicated), just a little less
> robust. I wonder why this isn't alredady possible, BTW.

Block lists aren't reliable, and a random number chain scheme looks overly
complex for solving a trivial problem (i.e. reserve enough raw space for
GRUB).

Please propose a scheme that is simple and robust.  It's not much to ask,
since we have that already by using partition tables.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)