Some ideas about new features of grub

Some ideas about new features of grub

[Bean]

Hi,

Here are some of my ideas about the new features of grub.

Move kernel to a module.
This make it possible to relocate the kernel. For example, we can use
it to move grub-pc to upper memory, and free conventional memory for
use by real mode os such as MS-DOS. grub can resides in memory even
after os take overs, and we can invoke it through interrupt hooks.

LUA integration.
LUA is quite powerful, it's more suitable to do complicated task than
sh script. For example, we can use it to detect os at runtime,
implement simple commands, or draw the graphic menu.

Read/Write file system support
We can implement two kind of fs drivers. The boot time driver is
read-only, but after entering normal mode, we can optionally load
another driver for write support. This approach has been used by EFI.
For example, it has a default FAT driver, but you can also load an
extended FAT driver
later.

Disk emulation.
Now that it has drivemap command, we can extended it to map hard disk
or cdrom image file, roughly equivalent to the memdisk of syslinux.

-- 
Bean

Re: Some ideas about new features of grub

[Pavel Roskin]

On Thu, 2009-07-02 at 16:48 +0800, Bean wrote:
> Hi,
> 
> Here are some of my ideas about the new features of grub.
> 
> Move kernel to a module.
> This make it possible to relocate the kernel. For example, we can use
> it to move grub-pc to upper memory, and free conventional memory for
> use by real mode os such as MS-DOS. grub can resides in memory even
> after os take overs, and we can invoke it through interrupt hooks.

I don't care about MS DOS.  Other OSes should not need GRUB.  If you
want GRUB to be a supervisor or a microkernel, it's better that GRUB
loads them instead of incorporating their functionality.

The only reason for GRUB to _be_ a supervisor or a microkernel would be
to implement some kind of TPM, and I don't think we should spend
development effort on that unless were can be sure that it won't be used
against our users.

> LUA integration.
> LUA is quite powerful, it's more suitable to do complicated task than
> sh script. For example, we can use it to detect os at runtime,
> implement simple commands, or draw the graphic menu.

Yes, I think LUA improvements should continue.  We may switch to a LUA
implementation of grub.cfg at some point.

> Read/Write file system support
> We can implement two kind of fs drivers. The boot time driver is
> read-only, but after entering normal mode, we can optionally load
> another driver for write support. This approach has been used by EFI.
> For example, it has a default FAT driver, but you can also load an
> extended FAT driver
> later.

I think it's pure featuritis.  There is no reason for a bootloader to
write to filesystems except to store it's state, which is already
implemented.  What would GRUB write?  Implementing and maintaining full
featured drivers would take a lot of effort.  I'd rather see someone
implement UUIDs for all filesystems.

> Disk emulation.
> Now that it has drivemap command, we can extended it to map hard disk
> or cdrom image file, roughly equivalent to the memdisk of syslinux.

Hard drives and CD-ROMs are usually large and would take a lot of space
in memory that would need to remain allocated.  I think we need a strong
case to start that effort.

I'd rather see an effort to support CD-ROM and other ATAPI devices
without disrupting BIOS access to the hard drives and floppies.  We also
need AHCI support.

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Duboucher Thomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Roskin a écrit :
>> LUA integration.
>> LUA is quite powerful, it's more suitable to do complicated task than
>> sh script. For example, we can use it to detect os at runtime,
>> implement simple commands, or draw the graphic menu.
> 
> Yes, I think LUA improvements should continue.  We may switch to a LUA
> implementation of grub.cfg at some point.

	Is there someone working on Lua now? The last commit is 6 months backward.
	I have some experience in the C part of Lua, perhaps I should write a
line or two. ;)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpM/rwACgkQBV7eXqefhqhI2wCeNmkXaQ9q+wMUNvv/GxbsA6uO
TooAoJ3ujoTRv7xI4OD75pUM09ngkcPH
=mVIq
-----END PGP SIGNATURE-----

Re: Some ideas about new features of grub

[Pavel Roskin]

On Thu, 2009-07-02 at 20:38 +0200, Duboucher Thomas wrote:

> Pavel Roskin a écrit :
> >> LUA integration.
> >> LUA is quite powerful, it's more suitable to do complicated task than
> >> sh script. For example, we can use it to detect os at runtime,
> >> implement simple commands, or draw the graphic menu.
> > 
> > Yes, I think LUA improvements should continue.  We may switch to a LUA
> > implementation of grub.cfg at some point.
> 
> 	Is there someone working on Lua now? The last commit is 6 months backward.
> 	I have some experience in the C part of Lua, perhaps I should write a
> line or two. ;)

The last commit in scripts/lua is dated May 18, 2009, one and a half
months ago.

There is a patch for adding file functions that wasn't committed yet for
some reason.

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Vladimir 'phcoder' Serbinenko]

On Thu, Jul 2, 2009 at 7:51 PM, Pavel Roskin<proski@gnu.org> wrote:
> On Thu, 2009-07-02 at 16:48 +0800, Bean wrote:
>> Hi,
>>
>> Here are some of my ideas about the new features of grub.
>>
>> Move kernel to a module.
>> This make it possible to relocate the kernel. For example, we can use
>> it to move grub-pc to upper memory, and free conventional memory for
>> use by real mode os such as MS-DOS. grub can resides in memory even
>> after os take overs, and we can invoke it through interrupt hooks.
>
> I don't care about MS DOS.  Other OSes should not need GRUB.  If you
> want GRUB to be a supervisor or a microkernel, it's better that GRUB
> loads them instead of incorporating their functionality.
>
This isn't about just msdos. Disk emulation may need this design
change. It may be used in future for booting proprietary OSes from
disk images.But it shouldn't matter for free OSes which either support
initrd in some form or such feature can be added to them. I wouldn't
spend a lot of effort to implement something which is useful only for
proprietary OSes. Leaving entire grub is perhaps a bit of overkill. I
would prefer another approach: compile kernel twice: once as normal
kernel and another time as kernel for persistent grub. Modules can be
the same. This approach has an advantage of not increasing core.img
size (we already not too far from the wall) while implementing kernel
feature. We should take more care about core.img size
But another reason
>> LUA integration.
>> LUA is quite powerful, it's more suitable to do complicated task than
>> sh script. For example, we can use it to detect os at runtime,
>> implement simple commands, or draw the graphic menu.
>
> Yes, I think LUA improvements should continue.  We may switch to a LUA
> implementation of grub.cfg at some point.
>
Even if LUA is great I don't agree with its usage as default. LUA is
under a bit different copyright license. While it's ok for it to be
incorporated and used you should be able to use grub even without LUA.
Additionally at least some feature aren't too hard to implement in sh
and I find sh more appropriate for menu generation.
>> Read/Write file system support
>> We can implement two kind of fs drivers. The boot time driver is
>> read-only, but after entering normal mode, we can optionally load
>> another driver for write support. This approach has been used by EFI.
>> For example, it has a default FAT driver, but you can also load an
>> extended FAT driver
>> later.
>
> I think it's pure featuritis.  There is no reason for a bootloader to
> write to filesystems except to store it's state, which is already
> implemented.  What would GRUB write?  Implementing and maintaining full
> featured drivers would take a lot of effort.
I agree. I have given this a thought and conclusion was that writing
isn't used enough to be well tested and so to leave an "experimental"
stage. And experimental writing driver is a risk of FS corruption. I
don't want to sacrifice reliability for not-so-needed features.
Additionally again it seems like a waste of time. This being said
you're free to implement it (it's your free time) and put it somewhere
else or perhaps in grub trunk but under a separate directory/branch
and which must be explicitely compiled.
> I'd rather see someone
> implement UUIDs for all filesystems.
>

>> Disk emulation.
>> Now that it has drivemap command, we can extended it to map hard disk
>> or cdrom image file, roughly equivalent to the memdisk of syslinux.
>
> Hard drives and CD-ROMs are usually large and would take a lot of space
> in memory that would need to remain allocated.  I think we need a strong
> case to start that effort.
>
Todays RAM is usually big enough. This feature would mainly be used to
boot install images and not normal OSes and install image + memory
needed for installer usually fits 1GiB. On the other hand small
laptops and netbook usualy have no CD. It would also be possible not
to store the image in RAM and read it directly from disk if grub's
drivers are left in memory one way or another. But this way there are
2 disk drivers working in the same time: OS and grub's one and this
leads to corruption. Fortunately DOS-like OS use BIOS and grub can
intercept it too. In this case however grub will conflict with dos
extenders. Protected-mode OSes would ignore disk emulation unless
special driver is used. In this case it would duty of this driver to
ensure no other disk driver is used when grub is called. This is an
important performance hit. And again this is mainly useful for
proprietary OSes. To support windows you can do it by writin windows
driver and porting freeldr to grub. For Mac OS X it caan be
implemented using xnu_ramdisk and some program to mount image and
chroot to it. And there is no point in supporting other proprietary
OSes
> I'd rather see an effort to support CD-ROM and other ATAPI devices
> without disrupting BIOS access to the hard drives and floppies.
You can't have 2 disk drivers operation on same device without taking
some risk. I would prefer ehci driver and floppy driver if someone
still needs them
>  We also
> need AHCI support.
>
Isn't it already the case?
> --
> Regards,
> Pavel Roskin
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: Some ideas about new features of grub

[Pavel Roskin]

On Thu, 2009-07-02 at 21:38 +0200, Vladimir 'phcoder' Serbinenko wrote:

> > I don't care about MS DOS.  Other OSes should not need GRUB.  If you
> > want GRUB to be a supervisor or a microkernel, it's better that GRUB
> > loads them instead of incorporating their functionality.
> >
> This isn't about just msdos. Disk emulation may need this design
> change. It may be used in future for booting proprietary OSes from
> disk images.But it shouldn't matter for free OSes which either support
> initrd in some form or such feature can be added to them. I wouldn't
> spend a lot of effort to implement something which is useful only for
> proprietary OSes. Leaving entire grub is perhaps a bit of overkill. I
> would prefer another approach: compile kernel twice: once as normal
> kernel and another time as kernel for persistent grub. Modules can be
> the same. This approach has an advantage of not increasing core.img
> size (we already not too far from the wall) while implementing kernel
> feature. We should take more care about core.img size
> But another reason

OK, whoever wants to implement that will need to provide a use case.

> >> LUA integration.
> >> LUA is quite powerful, it's more suitable to do complicated task than
> >> sh script. For example, we can use it to detect os at runtime,
> >> implement simple commands, or draw the graphic menu.
> >
> > Yes, I think LUA improvements should continue.  We may switch to a LUA
> > implementation of grub.cfg at some point.
> >
> Even if LUA is great I don't agree with its usage as default. LUA is
> under a bit different copyright license. While it's ok for it to be
> incorporated and used you should be able to use grub even without LUA.
> Additionally at least some feature aren't too hard to implement in sh
> and I find sh more appropriate for menu generation.

I really like the idea of looking for the kernels on the fly.  If we can
do it in sh, let's do it.

> > Hard drives and CD-ROMs are usually large and would take a lot of space
> > in memory that would need to remain allocated.  I think we need a strong
> > case to start that effort.
> >
> Todays RAM is usually big enough. This feature would mainly be used to
> boot install images and not normal OSes and install image + memory
> needed for installer usually fits 1GiB. On the other hand small
> laptops and netbook usualy have no CD. It would also be possible not
> to store the image in RAM and read it directly from disk if grub's
> drivers are left in memory one way or another.

I think keeping the whole GRUB in memory for that purpose would be a
major overkill.  I would probably create a file map on the disk (like,
sectors 0-10 of the file are in sectors 1000-1010 of the disk etc) and
leave a simple interrupt handler that would emulate a BIOS disk by using
that map.

But I don't see it as priority.

> > I'd rather see an effort to support CD-ROM and other ATAPI devices
> > without disrupting BIOS access to the hard drives and floppies.
> You can't have 2 disk drivers operation on same device without taking
> some risk.

Floppies are not the same device.  Yes, there is some risk accessing a
CD-ROM while BIOS accesses a hard drive on the same controller.  But I
think GRUB would never do it in parallel.

>  I would prefer ehci driver and floppy driver if someone
> still needs them

USB needs work, that's for sure.

Floppies can be supported via int 13 even if the ATA module is in use.
Or we can use int 40 to be extra safe.

> >  We also
> > need AHCI support.
> >
> Isn't it already the case?

I don't see it.

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Vladimir 'phcoder' Serbinenko]

>> > Hard drives and CD-ROMs are usually large and would take a lot of space
>> > in memory that would need to remain allocated.  I think we need a strong
>> > case to start that effort.
>> >
>> Todays RAM is usually big enough. This feature would mainly be used to
>> boot install images and not normal OSes and install image + memory
>> needed for installer usually fits 1GiB. On the other hand small
>> laptops and netbook usualy have no CD. It would also be possible not
>> to store the image in RAM and read it directly from disk if grub's
>> drivers are left in memory one way or another.
>
> I think keeping the whole GRUB in memory for that purpose would be a
> major overkill.  I would probably create a file map on the disk (like,
> sectors 0-10 of the file are in sectors 1000-1010 of the disk etc)
blocklists are unreliable, we both know it even if in this case it may
be less grave. Additionally even if you use blocklists you still need
device driver and keeping in memory device driver or device+fs driver
isn't of much difference. I prefer a clean approach whixh would ensure
that if a driver works in grub it will also work while let to provide
services. That's why I proposed double kernel compilation
> and
> leave a simple interrupt handler that would emulate a BIOS disk by using
> that map.
>
> But I don't see it as priority.
>
Neither do I but I want to ensure if someone implements it it will be
in a clean way
>> > I'd rather see an effort to support CD-ROM and other ATAPI devices
>> > without disrupting BIOS access to the hard drives and floppies.
>> You can't have 2 disk drivers operation on same device without taking
>> some risk.
>
> Floppies are not the same device.  Yes, there is some risk accessing a
> CD-ROM while BIOS accesses a hard drive on the same controller.  But I
> think GRUB would never do it in parallel.
Even so, drive may have some persistent states and I see this as
potentially unreliable
> Floppies can be supported via int 13 even if the ATA module is in use.
> Or we can use int 40 to be extra safe.
>
Is int 40 a hardware interrupt? If so additional advantage of
implementing it would be coreboot support
>> >  We also
>> > need AHCI support.
>> >
>> Isn't it already the case?
>
> I don't see it.
>
I guess you need to test it with ata.mod - perhaps it works
> --
> Regards,
> Pavel Roskin
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: Some ideas about new features of grub

[Pavel Roskin]

On Thu, 2009-07-02 at 23:37 +0200, Vladimir 'phcoder' Serbinenko wrote:

> > Floppies can be supported via int 13 even if the ATA module is in use.
> > Or we can use int 40 to be extra safe.
> >
> Is int 40 a hardware interrupt? If so additional advantage of
> implementing it would be coreboot support

No, int 40 is a software interrupt.  It's the same as int 13, but only
for floppies.  It won't work without BIOS.

> >> >  We also
> >> > need AHCI support.
> >> >
> >> Isn't it already the case?
> >
> > I don't see it.
> >
> I guess you need to test it with ata.mod - perhaps it works

I tested it on a system with Intel SATA controller (Dell Inspiron 1420)
and it didn't work at all.  The drive cannot be read in AHCI mode.  When
testing in compatibility SATA mode, the drive is seen correctly.

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Marco Gerards]

Pavel Roskin <proski@gnu.org> writes:

> On Thu, 2009-07-02 at 16:48 +0800, Bean wrote:
>> Hi,
>> 
>> Here are some of my ideas about the new features of grub.
>> 
>> Move kernel to a module.
>> This make it possible to relocate the kernel. For example, we can use
>> it to move grub-pc to upper memory, and free conventional memory for
>> use by real mode os such as MS-DOS. grub can resides in memory even
>> after os take overs, and we can invoke it through interrupt hooks.
>
> I don't care about MS DOS.  Other OSes should not need GRUB.  If you
> want GRUB to be a supervisor or a microkernel, it's better that GRUB
> loads them instead of incorporating their functionality.
>
> The only reason for GRUB to _be_ a supervisor or a microkernel would be
> to implement some kind of TPM, and I don't think we should spend
> development effort on that unless were can be sure that it won't be used
> against our users.
>
>> LUA integration.
>> LUA is quite powerful, it's more suitable to do complicated task than
>> sh script. For example, we can use it to detect os at runtime,
>> implement simple commands, or draw the graphic menu.
>
> Yes, I think LUA improvements should continue.  We may switch to a LUA
> implementation of grub.cfg at some point.

Please, don't.  I never had objections against lua, but lua should not
silently take over. 

>> Read/Write file system support
>> We can implement two kind of fs drivers. The boot time driver is
>> read-only, but after entering normal mode, we can optionally load
>> another driver for write support. This approach has been used by EFI.
>> For example, it has a default FAT driver, but you can also load an
>> extended FAT driver
>> later.
>
> I think it's pure featuritis.  There is no reason for a bootloader to
> write to filesystems except to store it's state, which is already
> implemented.  What would GRUB write?  Implementing and maintaining full
> featured drivers would take a lot of effort.  I'd rather see someone
> implement UUIDs for all filesystems.

Agreed.

>> Disk emulation.
>> Now that it has drivemap command, we can extended it to map hard disk
>> or cdrom image file, roughly equivalent to the memdisk of syslinux.
>
> Hard drives and CD-ROMs are usually large and would take a lot of space
> in memory that would need to remain allocated.  I think we need a strong
> case to start that effort.
>
> I'd rather see an effort to support CD-ROM and other ATAPI devices
> without disrupting BIOS access to the hard drives and floppies.  We also
> need AHCI support.

Right.  Although I do not mind if a separate module is added for
this.  Given that the person who adds it will maintain it properly.

--
Marco

Re: Some ideas about new features of grub

[Robert Millan]

On Thu, Jul 02, 2009 at 04:48:56PM +0800, Bean wrote:
> Hi,
> 
> Here are some of my ideas about the new features of grub.
> 
> Move kernel to a module.
> This make it possible to relocate the kernel. For example, we can use
> it to move grub-pc to upper memory, and free conventional memory for
> use by real mode os such as MS-DOS. grub can resides in memory even
> after os take overs, and we can invoke it through interrupt hooks.

It is already possible to relocate the kernel, in startup.S.  Also, we
can link the kernel in any address we want.

Turning the kernel into a module sounds like killing rescue mode.  Making
modules a prerequisite for rescue mode to work is NOT a good idea.

> LUA integration.
> LUA is quite powerful, it's more suitable to do complicated task than
> sh script. For example, we can use it to detect os at runtime,
> implement simple commands, or draw the graphic menu.

I don't mind LUA being supported, but I think it's unnecessarily big
for the task GRUB is usually going to perform (the canonical example
of that is in the default grub.cfg) in the majority of cases.  I'd
like to see *that* use case made more robust instead of switching to
something else to obtain a flexibility we don't currently need.

> Read/Write file system support
> We can implement two kind of fs drivers. The boot time driver is
> read-only, but after entering normal mode, we can optionally load
> another driver for write support. This approach has been used by EFI.
> For example, it has a default FAT driver, but you can also load an
> extended FAT driver
> later.

I don't mind write filesystem support as long as its infrastructure is
completely isolated from the kernel.  That said, I think Marco objected
to this before, so you'll have to talk with him about this.

> Disk emulation.
> Now that it has drivemap command, we can extended it to map hard disk
> or cdrom image file, roughly equivalent to the memdisk of syslinux.

Sounds like a killer feature, but would this actually work?  Once whatever
we're loading stops using the BIOS, it won't see the virtual drive anymore.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Bean]

Hi,

>> LUA integration.
>> LUA is quite powerful, it's more suitable to do complicated task than
>> sh script. For example, we can use it to detect os at runtime,
>> implement simple commands, or draw the graphic menu.
>
> I don't mind LUA being supported, but I think it's unnecessarily big
> for the task GRUB is usually going to perform (the canonical example
> of that is in the default grub.cfg) in the majority of cases.  I'd
> like to see *that* use case made more robust instead of switching to
> something else to obtain a flexibility we don't currently need.

With LUA, we can have a more user friendly interface. I like way rEFIt
works, it doesn't require configuration. At runtime, it detects os and
shows an icon for each of them. We can achieve similar goal using lua.
Of course, advanced user can write the menu manually, but for most
user, a smart auto-generated menu may be more appealing.

>
>> Read/Write file system support
>> We can implement two kind of fs drivers. The boot time driver is
>> read-only, but after entering normal mode, we can optionally load
>> another driver for write support. This approach has been used by EFI.
>> For example, it has a default FAT driver, but you can also load an
>> extended FAT driver
>> later.
>
> I don't mind write filesystem support as long as its infrastructure is
> completely isolated from the kernel.  That said, I think Marco objected
> to this before, so you'll have to talk with him about this.
>

A while ago, grub4dos add write support for fs. Actually, it's quite
limited and can only overwrite existing data. But already, people has
used it to implement something like 0PE (zero sized PE), which
generate the PE image on the fly. It's amazing what you can do which
such function. And I think it shouldn't cause security issue, as the
write driver needs to be loaded explicitly.

>> Disk emulation.
>> Now that it has drivemap command, we can extended it to map hard disk
>> or cdrom image file, roughly equivalent to the memdisk of syslinux.
>
> Sounds like a killer feature, but would this actually work?  Once whatever
> we're loading stops using the BIOS, it won't see the virtual drive anymore.

Yes, it's used for BIOS based os like MS-DOS/FreeDOS. There is still
some usage for them, for example, many low level maintainer tools
still runs in dos.

-- 
Bean

Re: Some ideas about new features of grub

[Robert Millan]

On Sun, Jul 05, 2009 at 11:13:25AM +0800, Bean wrote:
> Hi,
> 
> >> LUA integration.
> >> LUA is quite powerful, it's more suitable to do complicated task than
> >> sh script. For example, we can use it to detect os at runtime,
> >> implement simple commands, or draw the graphic menu.
> >
> > I don't mind LUA being supported, but I think it's unnecessarily big
> > for the task GRUB is usually going to perform (the canonical example
> > of that is in the default grub.cfg) in the majority of cases.  I'd
> > like to see *that* use case made more robust instead of switching to
> > something else to obtain a flexibility we don't currently need.
> 
> With LUA, we can have a more user friendly interface. I like way rEFIt
> works, it doesn't require configuration. At runtime, it detects os and
> shows an icon for each of them. We can achieve similar goal using lua.
> Of course, advanced user can write the menu manually, but for most
> user, a smart auto-generated menu may be more appealing.

But we have grub-mkconfig for that.  This is what distributions shipping
GRUB are using.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Pavel Roskin]

On Tue, 2009-07-07 at 20:39 +0200, Robert Millan wrote:

> > With LUA, we can have a more user friendly interface. I like way rEFIt
> > works, it doesn't require configuration. At runtime, it detects os and
> > shows an icon for each of them. We can achieve similar goal using lua.
> > Of course, advanced user can write the menu manually, but for most
> > user, a smart auto-generated menu may be more appealing.
> 
> But we have grub-mkconfig for that.  This is what distributions shipping
> GRUB are using.

That's what distributions shipping GRUB2 are using.  Fedora is totally
unaware of grub-mkconfig, so I have to rerun it every time I install a
new kernel.

grub-mkconfig won't pick up all filesystems, mounted and unmounted.
Processing unmounted filesystem would require a binary with GRUB
filesystem code linked into it, perhaps grub-fstest or a separate
binary.

And let's not forget about recovery disks.  That's where lua would
really shine.

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Robert Millan]

On Wed, Jul 08, 2009 at 02:19:14AM -0400, Pavel Roskin wrote:
> On Tue, 2009-07-07 at 20:39 +0200, Robert Millan wrote:
> 
> > > With LUA, we can have a more user friendly interface. I like way rEFIt
> > > works, it doesn't require configuration. At runtime, it detects os and
> > > shows an icon for each of them. We can achieve similar goal using lua.
> > > Of course, advanced user can write the menu manually, but for most
> > > user, a smart auto-generated menu may be more appealing.
> > 
> > But we have grub-mkconfig for that.  This is what distributions shipping
> > GRUB are using.
> 
> That's what distributions shipping GRUB2 are using.  Fedora is totally
> unaware of grub-mkconfig, so I have to rerun it every time I install a
> new kernel.
> 
> grub-mkconfig won't pick up all filesystems, mounted and unmounted.

It will if you install os-prober.  I think os-prober is only available on
debian yet, but it's not debian-specific.

> And let's not forget about recovery disks.  That's where lua would
> really shine.

Yes, it's a nice feature.  But it's not what most people will use GRUB for.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[BandiPat]

Robert Millan wrote:
> On Wed, Jul 08, 2009 at 02:19:14AM -0400, Pavel Roskin wrote:
>> On Tue, 2009-07-07 at 20:39 +0200, Robert Millan wrote:
>>
>>>> With LUA, we can have a more user friendly interface. I like way rEFIt
>>>> works, it doesn't require configuration. At runtime, it detects os and
>>>> shows an icon for each of them. We can achieve similar goal using lua.
>>>> Of course, advanced user can write the menu manually, but for most
>>>> user, a smart auto-generated menu may be more appealing.
>>> But we have grub-mkconfig for that.  This is what distributions shipping
>>> GRUB are using.
>> That's what distributions shipping GRUB2 are using.  Fedora is totally
>> unaware of grub-mkconfig, so I have to rerun it every time I install a
>> new kernel.
>>
>> grub-mkconfig won't pick up all filesystems, mounted and unmounted.
> 
> It will if you install os-prober.  I think os-prober is only available on
> debian yet, but it's not debian-specific.
> 
>> And let's not forget about recovery disks.  That's where lua would
>> really shine.
> 
> Yes, it's a nice feature.  But it's not what most people will use GRUB for.
> 
=======
Actually Zenwalk provides os-prober as well.  The gentleman that 
provides the installer of our Grub2 uses os-prober to detect all OS's 
installed, so they may be added to the original grub.cfg.  Works very 
well, although not perfect, but we are pretty pleased with the results 
thus far.

We also do not have to run any of the Grub2 programs after installing 
new kernels.  The developer of our installer patched Grub2 for Zenwalk, 
so that no changes were necessary after kernel updates.  He tried to 
offer this to you guys as well earlier, but got little response, so we 
use it for Zenwalk exclusively at the moment.

Regards,
Pat

-- 
        ---Zenwalk v6.x--Linux 2.6.30---
         Registered Linux User #225206
"Ever tried Zen computing?"  http://www.zenwalk.org

Re: Some ideas about new features of grub

[Bean]

On Sat, Jul 11, 2009 at 5:27 AM, BandiPat<magicpage91@earthlink.net> wrote:
> Actually Zenwalk provides os-prober as well.  The gentleman that provides
> the installer of our Grub2 uses os-prober to detect all OS's installed, so
> they may be added to the original grub.cfg.  Works very well, although not
> perfect, but we are pretty pleased with the results thus far.
>
> We also do not have to run any of the Grub2 programs after installing new
> kernels.  The developer of our installer patched Grub2 for Zenwalk, so that
> no changes were necessary after kernel updates.  He tried to offer this to
> you guys as well earlier, but got little response, so we use it for Zenwalk
> exclusively at the moment.

Hi,

IIRC, os-prober is a collection of shell script to detect os, but grub
already have them in util/grub.d. This method is based on linux, so
you need to enter linux in order to update the menu. On the other
hand, lua script generate the menu inside grub, there is no need to
enter any os.

Another problem is the drive number. It's impossible to decide bios
drive number from inside linux, so we can't insert the correct
drivemap command required to boot DOS/Windows from secondary drive.
This information must be gathered at boot time.

-- 
Bean

Re: Some ideas about new features of grub

[Michal Suchanek]

2009/7/11 Bean <bean123ch@gmail.com>:
> On Sat, Jul 11, 2009 at 5:27 AM, BandiPat<magicpage91@earthlink.net> wrote:
>> Actually Zenwalk provides os-prober as well.  The gentleman that provides
>> the installer of our Grub2 uses os-prober to detect all OS's installed, so
>> they may be added to the original grub.cfg.  Works very well, although not
>> perfect, but we are pretty pleased with the results thus far.
>>
>> We also do not have to run any of the Grub2 programs after installing new
>> kernels.  The developer of our installer patched Grub2 for Zenwalk, so that
>> no changes were necessary after kernel updates.  He tried to offer this to
>> you guys as well earlier, but got little response, so we use it for Zenwalk
>> exclusively at the moment.
>
> Hi,
>
> IIRC, os-prober is a collection of shell script to detect os, but grub
> already have them in util/grub.d. This method is based on linux, so
> you need to enter linux in order to update the menu. On the other
> hand, lua script generate the menu inside grub, there is no need to
> enter any os.
>
> Another problem is the drive number. It's impossible to decide bios
> drive number from inside linux, so we can't insert the correct
> drivemap command required to boot DOS/Windows from secondary drive.
> This information must be gathered at boot time.
>

The other problem is that scripts like os-prober that use linux
filesystem code for probing filesystems destroy your data.

Linux cannot mount journalled filesystems readonly. It always modifies
them. The extent of the modifications and the impact may vary
depending on the state of the filesystem (clean/mounted) and the way
you are going to use it in the future (resume the system/fresh boot)
but until Linux people implement readonly filesystem support any Linux
based os-probers cannot be recommended.

Thanks

Michal

Re: Some ideas about new features of grub

[Colin Watson]

On Sat, Jul 11, 2009 at 06:20:13PM +0200, Michal Suchanek wrote:
> The other problem is that scripts like os-prober that use linux
> filesystem code for probing filesystems destroy your data.
> 
> Linux cannot mount journalled filesystems readonly. It always modifies
> them. The extent of the modifications and the impact may vary
> depending on the state of the filesystem (clean/mounted) and the way
> you are going to use it in the future (resume the system/fresh boot)
> but until Linux people implement readonly filesystem support any Linux
> based os-probers cannot be recommended.

I'm pretty sure this is fixable with blockdev --setro. The main reason
we haven't got round to doing this in os-prober yet is that we need to
get round to adding blockdev to busybox (which is important for
os-prober's original purpose, the Debian installer).

-- 
Colin Watson                                       [cjwatson@ubuntu.com]

Re: Some ideas about new features of grub

[Robert Millan]

On Sat, Jul 11, 2009 at 07:32:36PM +0100, Colin Watson wrote:
> On Sat, Jul 11, 2009 at 06:20:13PM +0200, Michal Suchanek wrote:
> > The other problem is that scripts like os-prober that use linux
> > filesystem code for probing filesystems destroy your data.
> > 
> > Linux cannot mount journalled filesystems readonly. It always modifies
> > them. The extent of the modifications and the impact may vary
> > depending on the state of the filesystem (clean/mounted) and the way
> > you are going to use it in the future (resume the system/fresh boot)
> > but until Linux people implement readonly filesystem support any Linux
> > based os-probers cannot be recommended.
> 
> I'm pretty sure this is fixable with blockdev --setro. The main reason
> we haven't got round to doing this in os-prober yet is that we need to
> get round to adding blockdev to busybox (which is important for
> os-prober's original purpose, the Debian installer).

Does D-I use os-prober for anything other than GRUB ?  With grub-pc, it
doesn't need to invoke os-prober directly, installing it to the target
system would be enough.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Felix Zielcke]

Am Samstag, den 11.07.2009, 18:20 +0200 schrieb Michal Suchanek:
> 2009/7/11 Bean <bean123ch@gmail.com>:
> > On Sat, Jul 11, 2009 at 5:27 AM, BandiPat<magicpage91@earthlink.net> wrote:
> >> Actually Zenwalk provides os-prober as well.  The gentleman that provides
> >> the installer of our Grub2 uses os-prober to detect all OS's installed, so
> >> they may be added to the original grub.cfg.  Works very well, although not
> >> perfect, but we are pretty pleased with the results thus far.
> >>
> >> We also do not have to run any of the Grub2 programs after installing new
> >> kernels.  The developer of our installer patched Grub2 for Zenwalk, so that
> >> no changes were necessary after kernel updates.  He tried to offer this to
> >> you guys as well earlier, but got little response, so we use it for Zenwalk
> >> exclusively at the moment.
> >
> > Hi,
> >
> > IIRC, os-prober is a collection of shell script to detect os, but grub
> > already have them in util/grub.d. This method is based on linux, so
> > you need to enter linux in order to update the menu. On the other
> > hand, lua script generate the menu inside grub, there is no need to
> > enter any os.
> >
> > Another problem is the drive number. It's impossible to decide bios
> > drive number from inside linux, so we can't insert the correct
> > drivemap command required to boot DOS/Windows from secondary drive.
> > This information must be gathered at boot time.
> >
> 
> The other problem is that scripts like os-prober that use linux
> filesystem code for probing filesystems destroy your data.
> 
> Linux cannot mount journalled filesystems readonly. It always modifies
> them. The extent of the modifications and the impact may vary
> depending on the state of the filesystem (clean/mounted) and the way
> you are going to use it in the future (resume the system/fresh boot)
> but until Linux people implement readonly filesystem support any Linux
> based os-probers cannot be recommended.
> 

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/ext3.txt;hb=HEAD
ext3 has a noload option which disables journal recovery and with
ro,noload used it's really read-only.
I think it was added recently with 2.6.30 or something like that.
-- 
Felix Zielcke

Re: Some ideas about new features of grub

[Robert Millan]

On Sat, Jul 11, 2009 at 02:53:13PM +0800, Bean wrote:
> On Sat, Jul 11, 2009 at 5:27 AM, BandiPat<magicpage91@earthlink.net> wrote:
> > Actually Zenwalk provides os-prober as well.  The gentleman that provides
> > the installer of our Grub2 uses os-prober to detect all OS's installed, so
> > they may be added to the original grub.cfg.  Works very well, although not
> > perfect, but we are pretty pleased with the results thus far.
> >
> > We also do not have to run any of the Grub2 programs after installing new
> > kernels.  The developer of our installer patched Grub2 for Zenwalk, so that
> > no changes were necessary after kernel updates.  He tried to offer this to
> > you guys as well earlier, but got little response, so we use it for Zenwalk
> > exclusively at the moment.
> 
> Hi,
> 
> IIRC, os-prober is a collection of shell script to detect os, but grub
> already have them in util/grub.d.

They have a different purpose.  The other grub.d scripts are targeted at
generating the _native_ boot setup.  os-prober is meant for the rest.

Our own scripts for native boot setup are more robust than the os-prober
approach.  For example, they can determine if Linux will support UUIDs
by checking for /dev/by-uuid nodes, etc.

> Another problem is the drive number. It's impossible to decide bios
> drive number from inside linux, so we can't insert the correct
> drivemap command required to boot DOS/Windows from secondary drive.
> This information must be gathered at boot time.

This is not a problem anymore.  grub-mkconfig uses UUIDs in its default
setup, and avoids hardcoding BIOS drive numbers.  In the very weird
situations in which this is not possible, it aborts install rather than
hardcoding a drive number.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Pavel Roskin]

Quoting Bean <bean123ch@gmail.com>:

> IIRC, os-prober is a collection of shell script to detect os, but grub
> already have them in util/grub.d. This method is based on linux, so
> you need to enter linux in order to update the menu. On the other
> hand, lua script generate the menu inside grub, there is no need to
> enter any os.

I imagine most distros would want tight control over what appears in  
the menu.  Nobody would want their distro appear last in the list.  So  
osdetect.lua is a great thing, but it's not for everyone.

> Another problem is the drive number. It's impossible to decide bios
> drive number from inside linux, so we can't insert the correct
> drivemap command required to boot DOS/Windows from secondary drive.
> This information must be gathered at boot time.

It should be possible to find the partition using uuid or a unique  
file.  drivemap accepts partitions as arguments (it did last time I  
checked).

-- 
Regards,
Pavel Roskin

Re: Some ideas about new features of grub

[Robert Millan]

On Thu, Jul 02, 2009 at 04:48:56PM +0800, Bean wrote:
> 
> Read/Write file system support
> We can implement two kind of fs drivers. The boot time driver is
> read-only, but after entering normal mode, we can optionally load
> another driver for write support. This approach has been used by EFI.
> For example, it has a default FAT driver, but you can also load an
> extended FAT driver
> later.

This would have to be post-1.97, but as long as they're not enabled by
default in the build system, I really don't mind.  What does everyone
else think?

What we can't accept is making core functionality rely on rw filesystem
drivers.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Robert Millan]

On Thu, Jul 02, 2009 at 04:48:56PM +0800, Bean wrote:
> 
> LUA integration.
> LUA is quite powerful, it's more suitable to do complicated task than
> sh script. For example, we can use it to detect os at runtime,
> implement simple commands, or draw the graphic menu.

I feel similarly about LUA as I do about writable filesystems.  In fact I'm
considering a configure flag so that it's only enabled only when user
requests it.

But I know both Marco and Pavel feel strongly about this.  Please can you
comment?

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[adrian15]

Robert Millan escribió:
> On Thu, Jul 02, 2009 at 04:48:56PM +0800, Bean wrote:
>> LUA integration.
>> LUA is quite powerful, it's more suitable to do complicated task than
>> sh script. For example, we can use it to detect os at runtime,
>> implement simple commands, or draw the graphic menu.
> 
> I feel similarly about LUA as I do about writable filesystems.  In fact I'm
> considering a configure flag so that it's only enabled only when user
> requests it.
> 
> But I know both Marco and Pavel feel strongly about this.  Please can you
> comment?
> 

I am against a configure flag (if configure is the one from configure, 
make, make install) but a configure flag (as something specified in 
grub.cfg) is ok.

adrian15
-- 
Support free software. Donate to Super Grub Disk. Apoya el software 
libre. Dona a Super Grub Disk. http://www.supergrubdisk.org/index.php?pid=10

Re: Some ideas about new features of grub

[Vladimir 'phcoder' Serbinenko]

>> But I know both Marco and Pavel feel strongly about this.  Please can you
>> comment?
>>
>
> I am against a configure flag (if configure is the one from configure, make,
> make install) but a configure flag (as something specified in grub.cfg) is
> ok.

Writing to filesystems is dangerous. Currently only grub-setup and
save_env can potentially destroy data or metadata and this risk is
minimised by just overwriting relevant sectors. Would you trust with
your data to a write driver which hasn't been thoroughfully tested? I
wouldn't. In current ro drivers we regularly find mistakes. Writing
support makes driver more complex and so failure is more likely. Even
a slightest mistake like putting entry in a wrong place in btree can
have devastating effects (like OS not seeing half of the files).
Writing files is only marginally needed for non-essential features
which reduces the benefits and even decreases the chances of drivers
to be tested enough. Currently grub2 does well what's supposed to do -
boot OS. Making it write to FS makes it less reliable and worse at
what its primary goal.
How long is rw driver marked experimental in kernel? A year perhaps
and it's more likely to be good tested because it 5 minutes of its use
for root fs you do a lot of transactions. But grub, will you save
grub.cfg 100 times at row?
What are benefits?
Saving environement? It's done with save_env. Only problem are
checksumming FSes but 2 I'm aware about ZFS and btrfs reserve space
for booter so we can embed config. Or just ask to disable checksumming
on grubenv. THis can be done as a part of grubenv creation on such
filesystems.
Recovering OS? You should use recovery OS. You can put a kernel and
initrd together with your grub for emergency cases. If grub survives
disaster recovery kernel is likely to survive it too.
Installing OS? You really need an installer
Saving modifications done to configuration? It can be useful but it's
done rarely enough to repeat it from OS if needed. Especially that
grub would have trouble to modify config if script that generated is a
bit more difficult than just array of menuentries
I really don't want hundreds of reports "I wanted to save modified
entry but grub destroyed my FS".
-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: Some ideas about new features of grub

[Robert Millan]

On Sun, Aug 23, 2009 at 06:58:43PM +0200, Vladimir 'phcoder' Serbinenko wrote:
> >> But I know both Marco and Pavel feel strongly about this.  Please can you
> >> comment?
> >>
> >
> > I am against a configure flag (if configure is the one from configure, make,
> > make install) but a configure flag (as something specified in grub.cfg) is
> > ok.
> 
> Writing to filesystems is dangerous. [...]

I was talking about LUA (I sent another mail about filesystems).  I admit
it's a bit confusing cause they're both in the same thread and I didn't
update the subject.  Please bear with me :-)

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Some ideas about new features of grub

[Pavel Roskin]

On Sun, 2009-08-23 at 14:56 +0200, Robert Millan wrote:
> On Thu, Jul 02, 2009 at 04:48:56PM +0800, Bean wrote:
> > 
> > LUA integration.
> > LUA is quite powerful, it's more suitable to do complicated task than
> > sh script. For example, we can use it to detect os at runtime,
> > implement simple commands, or draw the graphic menu.
> 
> I feel similarly about LUA as I do about writable filesystems.  In fact I'm
> considering a configure flag so that it's only enabled only when user
> requests it.
> 
> But I know both Marco and Pavel feel strongly about this.  Please can you
> comment?

Ideally, I would prefer GRUB to have one scripting language.  I just
didn't have time and desire to argue against LUA inclusion.  Having more
code means more work for maintainers.

-- 
Regards,
Pavel Roskin