Proposal for savedefault implementation

Proposal for savedefault implementation

[Pavel Roskin]

Hello!

I cannot promise I'll do the coding myself, but I've been thinking of  
a safe approach to savedefault implementation, and I'd like to share  
some of my thoughts.

GRUB should support only one kind of writing to the disk, namely  
writing to one sector at a specified offset.  Even that is not  
trivial, as we should try to support BIOS, openfirmware and ATA.  And  
maybe we should try to support RAID at some point.

There should be a utility capable of creating a "blessed sector" where  
GRUB would be allowed to write.  Choosing the location is up to that  
utility and the user running it.  It could be a bootloader area of a  
partition, a sector on the first track of the hard drive or a file on  
the filesystem.

The utility must ensure that GRUB can write to that sector safely at  
the boot time.  In particular, it should be indeed aligned at the  
sector boundary, it should not be compressed or fragmented.  If it's  
on RAID, the bootloader should know what to do.  Only GRUB and the  
utility should ever write to that sector.  Using any other tools  
should be discouraged.

What could be in the "blessed sector":

version of the utility that created it
minimal version of GRUB allowed to write
sector size
sector number on the drive
data length
modification time?
counter, which is incremented on write?
drives and sectors to write copies to?
settings, perhaps stored as 0-terminated strings in the format key=value

GRUB should read the sector first, perhaps using the filesystem.  It  
should check that its version is at least the required one.  Then it  
should re-read the sector by the sector number.  The data should be  
exactly the same.  That should allow writing by the sector number.

The command for writing should save variables from the environment.   
Likewise, reading from the file should copy variables to the  
environment.

-- 
Regards,
Pavel Roskin

Re: Proposal for savedefault implementation

[Bean]

On Thu, Mar 20, 2008 at 4:09 PM, Pavel Roskin <proski@gnu.org> wrote:
> Hello!
>
>  I cannot promise I'll do the coding myself, but I've been thinking of
>  a safe approach to savedefault implementation, and I'd like to share
>  some of my thoughts.
>
>  GRUB should support only one kind of writing to the disk, namely
>  writing to one sector at a specified offset.  Even that is not
>  trivial, as we should try to support BIOS, openfirmware and ATA.  And
>  maybe we should try to support RAID at some point.
>
>  There should be a utility capable of creating a "blessed sector" where
>  GRUB would be allowed to write.  Choosing the location is up to that
>  utility and the user running it.  It could be a bootloader area of a
>  partition, a sector on the first track of the hard drive or a file on
>  the filesystem.
>
>  The utility must ensure that GRUB can write to that sector safely at
>  the boot time.  In particular, it should be indeed aligned at the
>  sector boundary, it should not be compressed or fragmented.  If it's
>  on RAID, the bootloader should know what to do.  Only GRUB and the
>  utility should ever write to that sector.  Using any other tools
>  should be discouraged.
>
>  What could be in the "blessed sector":
>
>  version of the utility that created it
>  minimal version of GRUB allowed to write
>  sector size
>  sector number on the drive
>  data length
>  modification time?
>  counter, which is incremented on write?
>  drives and sectors to write copies to?
>  settings, perhaps stored as 0-terminated strings in the format key=value
>
>  GRUB should read the sector first, perhaps using the filesystem.  It
>  should check that its version is at least the required one.  Then it
>  should re-read the sector by the sector number.  The data should be
>  exactly the same.  That should allow writing by the sector number.
>
>  The command for writing should save variables from the environment.
>  Likewise, reading from the file should copy variables to the
>  environment.

Hi,

I think using a separate sector to store the information is not very
practical. First of all, we need to modify all of the file system
modules  to support this new feature, which require non trivial work
and testing. Besides, some file system doesn't have space for extra
sector, for example, fat12/fat16. It would be annoying if this
function can't be used on such fs.

Perhaps we can just embed the information in core.img itself. We use
blocklist to find out the disk location and write to it. As core.img
is normally too big to fit within a filesystem cluster, we don't have
to worry about inline data. Also, we can read from the sector and
verify its content first, so that we won't overwrite other area even
if blocklist give incorrect result.

About the format, i think storing key=value pairs is all we need.
Other information can be converted to such form. Inside grub2, we can
add a command to mark some variable as persistent, so that its value
will be written to disk automatically (or manually).

-- 
Bean

Re: Proposal for savedefault implementation

[Pavel Roskin]

On Thu, 2008-03-20 at 20:28 +0800, Bean wrote:

> I think using a separate sector to store the information is not very
> practical. First of all, we need to modify all of the file system
> modules  to support this new feature, which require non trivial work
> and testing.

I'm afraid you misunderstood something.  Are you talking about the
bootloader or about the utility for creating the sector?  The bootloader
should write only by the block number.  No filesystem code is involved.

The utility only needs to know which filesystems are safe to write to.
It should also know the sector number, but I think LILO can do it
somehow without any filesystem support.  Even GRUB should be able to do
it if it doesn't embed the core image.  The worst case scenario is that
we add some tracing code to our filesystem implementation (userspace
only), which would record the last sector accessed for reading.  This
way we'll know the location of the file by reading from it.

>  Besides, some file system doesn't have space for extra
> sector, for example, fat12/fat16. It would be annoying if this
> function can't be used on such fs.

What extra space are you talking about?  I actually think FAT is one of
the easiest filesystems to support because it has no checksums, no
compression, no tail packing and no sparse files.  A 512-byte long file
would always occupy exactly one sector.

> Perhaps we can just embed the information in core.img itself. We use
> blocklist to find out the disk location and write to it. As core.img
> is normally too big to fit within a filesystem cluster, we don't have
> to worry about inline data. Also, we can read from the sector and
> verify its content first, so that we won't overwrite other area even
> if blocklist give incorrect result.

I don't see much difference between using core.img and a separate file,
except that using a separate file gives more flexibility.

By the way, you are inconsistent here.  If writing a separate file
requires changes to all filesystems, how come writing to core.img
doesn't require it?

The only difference in my proposal is to have the blocklist (consisting
of one sector) in the writable file itself, not in the bootloader.

> About the format, i think storing key=value pairs is all we need.
> Other information can be converted to such form. Inside grub2, we can
> add a command to mark some variable as persistent, so that its value
> will be written to disk automatically (or manually).

OK, fine with me.

-- 
Regards,
Pavel Roskin

Re: Proposal for savedefault implementation

[Bean]

On Fri, Mar 21, 2008 at 12:26 AM, Pavel Roskin <proski@gnu.org> wrote:
> On Thu, 2008-03-20 at 20:28 +0800, Bean wrote:
>
>  > I think using a separate sector to store the information is not very
>  > practical. First of all, we need to modify all of the file system
>  > modules  to support this new feature, which require non trivial work
>  > and testing.
>
>  I'm afraid you misunderstood something.  Are you talking about the
>  bootloader or about the utility for creating the sector?  The bootloader
>  should write only by the block number.  No filesystem code is involved.
>
>  The utility only needs to know which filesystems are safe to write to.
>  It should also know the sector number, but I think LILO can do it
>  somehow without any filesystem support.  Even GRUB should be able to do
>  it if it doesn't embed the core image.  The worst case scenario is that
>  we add some tracing code to our filesystem implementation (userspace
>  only), which would record the last sector accessed for reading.  This
>  way we'll know the location of the file by reading from it.
>
>
>  >  Besides, some file system doesn't have space for extra
>  > sector, for example, fat12/fat16. It would be annoying if this
>  > function can't be used on such fs.
>
>  What extra space are you talking about?  I actually think FAT is one of
>  the easiest filesystems to support because it has no checksums, no
>  compression, no tail packing and no sparse files.  A 512-byte long file
>  would always occupy exactly one sector.

I'm sorry, i do misunderstand your point. i thought you mean saving
the information in some reserved sector of file system.

>
>
>  > Perhaps we can just embed the information in core.img itself. We use
>  > blocklist to find out the disk location and write to it. As core.img
>  > is normally too big to fit within a filesystem cluster, we don't have
>  > to worry about inline data. Also, we can read from the sector and
>  > verify its content first, so that we won't overwrite other area even
>  > if blocklist give incorrect result.
>
>  I don't see much difference between using core.img and a separate file,
>  except that using a separate file gives more flexibility.
>
>  By the way, you are inconsistent here.  If writing a separate file
>  requires changes to all filesystems, how come writing to core.img
>  doesn't require it?
>
>  The only difference in my proposal is to have the blocklist (consisting
>  of one sector) in the writable file itself, not in the bootloader.
>

Yes, using core.img and a separate file is not that different. but if
we use normal file, we must create a large enough one.

-- 
Bean

Re: Proposal for savedefault implementation

[Bean]

Hi,

I'm think about the following implementation, please see if it's all right.

Config information is saved in a file named `defaults' in the grub
directory, it's format is something like this:

GRUBENV \0
key1=value1 \0
key2=value2 \0
...
keyN=valueN \0 \0

This file should be at least 8192 bytes, this should be ok for most file system.

In grub2, we use load_exports to load global variables, and
save_exports to save them. For example:

load_exports

menuentry "aa" {
  default=0
  export default
  save_exports
}

menuentry "bb" {
  default=1
  export default
  save_exports
}

we can also add a new tool such as grub-editenv to edit the config file.

-- 
Bean

Re: Proposal for savedefault implementation

[Pavel Roskin]

On Fri, 2008-03-21 at 19:31 +0800, Bean wrote:
> Hi,
> 
> I'm think about the following implementation, please see if it's all right.
> 
> Config information is saved in a file named `defaults' in the grub
> directory, it's format is something like this:

It would be great not to hardcode the name, or at least make it possible
to override the name.

> GRUBENV \0
> key1=value1 \0
> key2=value2 \0
> ...
> keyN=valueN \0 \0
> 
> This file should be at least 8192 bytes, this should be ok for most file system.

Why this number?  Is it to avoid some filesystem issues, like tail
packing?  Or is it just to ensure that there is enough space for the
variables?

> In grub2, we use load_exports to load global variables, and
> save_exports to save them. For example:
> 
> load_exports
> 
> menuentry "aa" {
>   default=0
>   export default
>   save_exports
> }

It feels too verbose to me.  I think "export default=0" could do all
three operations at once.  Also, "default" or some other variable should
be set to the menu entry number automatically, so that the entries don't
need to be renumbered every time.

> we can also add a new tool such as grub-editenv to edit the config
> file.

Maybe I'm too obsessed with safety, but that tool should also check that
the "default" file is located in a place that is safe for GRUB to use.

-- 
Regards,
Pavel Roskin

Re: Proposal for savedefault implementation

[Bean]

On Sat, Mar 22, 2008 at 2:36 PM, Pavel Roskin <proski@gnu.org> wrote:
> On Fri, 2008-03-21 at 19:31 +0800, Bean wrote:
>  > Hi,
>  >
>  > I'm think about the following implementation, please see if it's all right.
>  >
>  > Config information is saved in a file named `defaults' in the grub
>  > directory, it's format is something like this:
>
>  It would be great not to hardcode the name, or at least make it possible
>  to override the name.

yes, we can optionally use a filename parameter:

load_exports [FILE]
save_exports [FILE]

It's even better if we don't use hardcode name at all, but due to a
bug in lexer, we can't do this:

load_exports ${prefix}/defaults

it's just not easy to specify a file in the grub directory.

>
>
>  > GRUBENV \0
>  > key1=value1 \0
>  > key2=value2 \0
>  > ...
>  > keyN=valueN \0 \0
>  >
>  > This file should be at least 8192 bytes, this should be ok for most file system.
>
>  Why this number?  Is it to avoid some filesystem issues, like tail
>  packing?  Or is it just to ensure that there is enough space for the
>  variables?
>

it's used to avoid tail packing.

>
>  > In grub2, we use load_exports to load global variables, and
>  > save_exports to save them. For example:
>  >
>  > load_exports
>  >
>  > menuentry "aa" {
>  >   default=0
>  >   export default
>  >   save_exports
>  > }
>
>  It feels too verbose to me.  I think "export default=0" could do all
>  three operations at once.  Also, "default" or some other variable should
>  be set to the menu entry number automatically, so that the entries don't
>  need to be renumbered every time.
>
>

Currently, the export command of grub2 doesn't support usage like
"export default=0", but it's trivial to add this feature.

i agree that the menu entry should be set automatically, and perhaps
the title is better than plain number. we can store the current menu
selection in a global variable like menu_title.

>  > we can also add a new tool such as grub-editenv to edit the config
>  > file.
>
>  Maybe I'm too obsessed with safety, but that tool should also check that
>  the "default" file is located in a place that is safe for GRUB to use.
>

It's a good idea.

-- 
Bean