idea: library support for grub2

idea: library support for grub2

[Bean]

Hi,

Library is an archive file that contains unlinked object files. We can
use library function both statically or dynamically. In the former
case, we link the objects together to form an executable image, in the
later case, we load the object at runtime and resolve symbols, much
like the modules.

With library, the size of kernel can be reduced dramatically. Now, the
kernel only contains platform specific function. To build a kernel
image, we link the kernel and other required module, then link any
unresolved function from the library, this way, only the function that
are actually used is in the kernel.

As native library format of the building os could vary, we should use
a format that's specific to grub, perhaps something similar to cpio.

-- 
Bean

Re: idea: library support for grub2

[Pavel Roskin]

On Sat, 2009-07-04 at 09:48 +0800, Bean wrote:
> Hi,
> 
> Library is an archive file that contains unlinked object files. We can
> use library function both statically or dynamically. In the former
> case, we link the objects together to form an executable image, in the
> later case, we load the object at runtime and resolve symbols, much
> like the modules.
> 
> With library, the size of kernel can be reduced dramatically. Now, the
> kernel only contains platform specific function. To build a kernel
> image, we link the kernel and other required module, then link any
> unresolved function from the library, this way, only the function that
> are actually used is in the kernel.

What matters is the size of the linked and compressed kernel image
(core.img) and not the bare kernel (kernel.img).  Yes, we can have some
savings, but I don't expect anything dramatic.

We would only gain size reduction for the code that is in the kernel,
but is not used by any module linked into the kernel image.  I don't
think there will be a lot of such code.  Whatever modules we link, the
image will need to output formatted strings, handle filesystems,
partitions, disks.

> As native library format of the building os could vary, we should use
> a format that's specific to grub, perhaps something similar to cpio.

The price will be a greatly increased complexity of the build system.  I
think it would be better to simplify it first.

-- 
Regards,
Pavel Roskin

Re: idea: library support for grub2

[Bean]

On Sat, Jul 4, 2009 at 12:26 PM, Pavel Roskin<proski@gnu.org> wrote:
> On Sat, 2009-07-04 at 09:48 +0800, Bean wrote:
>> Hi,
>>
>> Library is an archive file that contains unlinked object files. We can
>> use library function both statically or dynamically. In the former
>> case, we link the objects together to form an executable image, in the
>> later case, we load the object at runtime and resolve symbols, much
>> like the modules.
>>
>> With library, the size of kernel can be reduced dramatically. Now, the
>> kernel only contains platform specific function. To build a kernel
>> image, we link the kernel and other required module, then link any
>> unresolved function from the library, this way, only the function that
>> are actually used is in the kernel.
>
> What matters is the size of the linked and compressed kernel image
> (core.img) and not the bare kernel (kernel.img).  Yes, we can have some
> savings, but I don't expect anything dramatic.

Hi,

The current size:
   text	   data	    bss	    dec	    hex	filename
   4036	      0	      0	   4036	    fc4	kernel_img-kern_misc.o
   3520	      0	      0	   3520	    dc0	kernel_img-kern_i386_pc_startup.o
   3440	   1504	      0	   4944	   1350	kernel_img-symlist.o
   3056	      0	   2064	   5120	   1400	kernel_img-kern_dl.o
   2900	      0	  24544	  27444	   6b34	kernel_img-kern_disk.o
   1384	     64	     40	   1488	    5d0	kernel_img-kern_fs.o
   1204	      4	     64	   1272	    4f8	kernel_img-kern_env.o
   1104	      0	     16	   1120	    460	kernel_img-kern_mm.o
   1052	      0	      0	   1052	    41c	kernel_img-kern_corecmd.o
    964	    320	      0	   1284	    504	kernel_img-kern_parser.o
    872	      0	    272	   1144	    478	kernel_img-kern_i386_pc_init.o
    748	     36	     64	    848	    350	kernel_img-kern_term.o
    572	      0	     16	    588	    24c	kernel_img-kern_partition.o
    556	      0	      0	    556	    22c	kernel_img-kern_file.o
    536	      0	      0	    536	    218	kernel_img-kern_device.o
    524	      0	      0	    524	    20c	kernel_img-kern_main.o
    432	      0	      0	    432	    1b0	kernel_img-kern_i386_dl.o
    384	      0	      0	    384	    180	kernel_img-kern_list.o
    336	      0	   2640	   2976	    ba0	kernel_img-kern_err.o
    288	      4	      0	    292	    124	kernel_img-term_i386_vga_common.o
    276	     20	    256	    552	    228	kernel_img-kern_rescue_reader.o
    264	     20	      0	    284	    11c	kernel_img-kern_rescue_parser.o
    208	      0	     32	    240	     f0	kernel_img-kern_i386_tsc.o
    192	      0	      0	    192	     c0	kernel_img-kern_i386_pc_mmap.o
    156	      0	      0	    156	     9c	kernel_img-kern_handler.o
    144	      0	      0	    144	     90	kernel_img-kern_command.o
     96	     16	      0	    112	     70	kernel_img-kern_reader.o
     92	     96	      0	    188	     bc	kernel_img-term_i386_pc_console.o
     56	      0	      0	     56	     38	kernel_img-kern_i386_pit.o
     52	      0	      0	     52	     34	kernel_img-kern_generic_millisleep.o
     32	      0	      0	     32	
20	kernel_img-kern_generic_rtc_get_time_ms.o
     24	      0	     16	     40	     28	kernel_img-kern_time.o

The biggest part is kern/misc.c, all are library function, many
function in startup can also be moved to library. And with library,
symlist.c is not needed as well, which saves another 3k.

>
> We would only gain size reduction for the code that is in the kernel,
> but is not used by any module linked into the kernel image.  I don't
> think there will be a lot of such code.  Whatever modules we link, the
> image will need to output formatted strings, handle filesystems,
> partitions, disks.
>
>> As native library format of the building os could vary, we should use
>> a format that's specific to grub, perhaps something similar to cpio.
>
> The price will be a greatly increased complexity of the build system.  I
> think it would be better to simplify it first.

Actually, I'm also thinking about unifying the object format. We can
write a program to convert from platform specific object file to an
unified elf format, this would simplify the build process for system
like osx, and the dynamic loader can be unified as well, which may
save some code.

-- 
Bean

Re: idea: library support for grub2

[Pavel Roskin]

Quoting Bean <bean123ch@gmail.com>:

> The biggest part is kern/misc.c, all are library function, many
> function in startup can also be moved to library.

It doesn't matter by itself.  If the modules linked into the the  
kernel image use most functions in that library, there will be little  
of no saving.

> And with library,
> symlist.c is not needed as well, which saves another 3k.

Perhaps I misunderstand something in your proposal.  I don't see how  
introducing a library would eliminate symlist.c.

We'll need a list of symbols for modules to use, whether those symbols  
are in the kernel or in the library.  In fact, we'll need two lists  
instead of one - one for the kernel and one for the library.

We could use hashes instead of symbol names to make the symlist  
shorter (but hashes won't compress as well, so it's not a sure win).   
We can enumerate the symbols - that would compress better.  Or we can  
resolve all kernel symbols in the modules and eliminate the symlist.   
But it's a completely separate issue.

> Actually, I'm also thinking about unifying the object format. We can
> write a program to convert from platform specific object file to an
> unified elf format, this would simplify the build process for system
> like osx, and the dynamic loader can be unified as well, which may
> save some code.

Yes, refactoring of the build system would be helpful.

-- 
Regards,
Pavel Roskin

Re: idea: library support for grub2

[Bean]

On Sat, Jul 4, 2009 at 10:27 PM, Pavel Roskin<proski@gnu.org> wrote:
> Quoting Bean <bean123ch@gmail.com>:
>
>> The biggest part is kern/misc.c, all are library function, many
>> function in startup can also be moved to library.
>
> It doesn't matter by itself.  If the modules linked into the the kernel
> image use most functions in that library, there will be little of no saving.
>

Many string and char function are rarely used, and the unicode
function is used only by some fs.

>> And with library,
>> symlist.c is not needed as well, which saves another 3k.
>
> Perhaps I misunderstand something in your proposal.  I don't see how
> introducing a library would eliminate symlist.c.
>
> We'll need a list of symbols for modules to use, whether those symbols are
> in the kernel or in the library.  In fact, we'll need two lists instead of
> one - one for the kernel and one for the library.
>
> We could use hashes instead of symbol names to make the symlist shorter (but
> hashes won't compress as well, so it's not a sure win).  We can enumerate
> the symbols - that would compress better.  Or we can resolve all kernel
> symbols in the modules and eliminate the symlist.  But it's a completely
> separate issue.

The reason for symlist is to export function from kernel, but we can
move the kernel to a module as well, therefore eliminating symlist
altogether. With library, we can walk the module list (including
kernel), and linked the unresolved function when generating core.img.

-- 
Bean

Re: idea: library support for grub2

[Bean]

>>> And with library,
>>> symlist.c is not needed as well, which saves another 3k.
>>
>> Perhaps I misunderstand something in your proposal.  I don't see how
>> introducing a library would eliminate symlist.c.
>>
>> We'll need a list of symbols for modules to use, whether those symbols are
>> in the kernel or in the library.  In fact, we'll need two lists instead of
>> one - one for the kernel and one for the library.
>>
>> We could use hashes instead of symbol names to make the symlist shorter (but
>> hashes won't compress as well, so it's not a sure win).  We can enumerate
>> the symbols - that would compress better.  Or we can resolve all kernel
>> symbols in the modules and eliminate the symlist.  But it's a completely
>> separate issue.
>
> The reason for symlist is to export function from kernel, but we can
> move the kernel to a module as well, therefore eliminating symlist
> altogether. With library, we can walk the module list (including
> kernel), and linked the unresolved function when generating core.img.

Hi,

Actually, perhaps the most space saving method is not to use module in
kernel. We could link kernel.img, modules and unresolved function into
a single image, this image only export one symbol, the firmware
structure, which is used to invoke platform specific function, such as
firmware.get_mmap, etc.

For external module, the function is loaded from library, although
this means there are two version of the function in memory, one static
linked in kernel, one dynamic loaded, but memory is generally not an
issue, it's the size of core.img that matters, and we don't want to
expand it  by adding the exported symbol table there.

There is also another usage for this method, now we can use two
different version of function with the same name. For example, we
could have two grub_printf, a simple version used in kernel, and a
full version in the library.

-- 
Bean