[RFC] split floppy support off biosdisk

[RFC] split floppy support off biosdisk

[Robert Millan]

Floppy support seems to be a constant source of trouble because it's the
only disk type that can't be probed for reliably without causing unacceptable
delays on some systems.

How would you think about splitting it off biosdisk into a separate module?

This will probably add some code redundancy, but it also saves a small
space in the typical core.img, which is a good trade-off IMHO.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Sun, 2008-06-15 at 19:48 +0200, Robert Millan wrote:
> Floppy support seems to be a constant source of trouble because it's the
> only disk type that can't be probed for reliably without causing unacceptable
> delays on some systems.
> 
> How would you think about splitting it off biosdisk into a separate module?
> 
> This will probably add some code redundancy, but it also saves a small
> space in the typical core.img, which is a good trade-off IMHO.

I'm afraid the savings are going to be tiny (if any) on the hard drive
side, whereas the code duplication will be significant.  The floppy
module won't have CD-ROM support, and that's it.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Bean]

On Mon, Jun 16, 2008 at 8:32 AM, Pavel Roskin <proski@gnu.org> wrote:
> On Sun, 2008-06-15 at 19:48 +0200, Robert Millan wrote:
>> Floppy support seems to be a constant source of trouble because it's the
>> only disk type that can't be probed for reliably without causing unacceptable
>> delays on some systems.
>>
>> How would you think about splitting it off biosdisk into a separate module?
>>
>> This will probably add some code redundancy, but it also saves a small
>> space in the typical core.img, which is a good trade-off IMHO.
>
> I'm afraid the savings are going to be tiny (if any) on the hard drive
> side, whereas the code duplication will be significant.  The floppy
> module won't have CD-ROM support, and that's it.

Hi,

The fd and hd device share many code, separating them is not a good
idea. Perhaps what we need is a way to optionally disable the floppy
device, for example, with a variable such as no_floppy.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Mon, Jun 16, 2008 at 02:30:38PM +0800, Bean wrote:
> 
> The fd and hd device share many code, separating them is not a good
> idea.

If code duplication is a problem, we could link the same code staticaly for
each module.  Size of biosdisk module is an issue, because it's always used
in core.img, but size of floppy module isn't, because it's never needed for
bootstrap.

> Perhaps what we need is a way to optionally disable the floppy
> device, for example, with a variable such as no_floppy.

But that makes biosdisk bigger, not smaller :-(

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Bean]

On Mon, Jun 16, 2008 at 5:15 PM, Robert Millan <rmh@aybabtu.com> wrote:
> On Mon, Jun 16, 2008 at 02:30:38PM +0800, Bean wrote:
>>
>> The fd and hd device share many code, separating them is not a good
>> idea.
>
> If code duplication is a problem, we could link the same code staticaly for
> each module.  Size of biosdisk module is an issue, because it's always used
> in core.img, but size of floppy module isn't, because it's never needed for
> bootstrap.
>
>> Perhaps what we need is a way to optionally disable the floppy
>> device, for example, with a variable such as no_floppy.
>
> But that makes biosdisk bigger, not smaller :-(

Hi,

Is size really that important ? I think even if you split the module,
it would only save very little space, which doesn't matter in the big
picture, unless you have a situation where the current size is very
closed to the upper limit. In that case, I suggest more dramatic
action, like replacing lzo with a strong compression algorithm, like
lzma, which would save a few thousand bytes.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

El lun, 16-06-2008 a las 19:03 +0800, Bean escribió:
> In that case, I suggest more dramatic
> action, like replacing lzo with a strong compression algorithm, like
> lzma, which would save a few thousand bytes.
> 
Hmm.. that could be done without too much fuss (except the part of
implementing the decompressor in assembly for each platform...), but
we'd probably need to test both the change of size in the decompressor
code and the gains in compression. Since lzma is quite complex, the
substitution might not pay off...

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Mon, Jun 16, 2008 at 7:20 PM, Javier Martín <lordhabbit@gmail.com> wrote:
> El lun, 16-06-2008 a las 19:03 +0800, Bean escribió:
>> In that case, I suggest more dramatic
>> action, like replacing lzo with a strong compression algorithm, like
>> lzma, which would save a few thousand bytes.
>>
> Hmm.. that could be done without too much fuss (except the part of
> implementing the decompressor in assembly for each platform...), but
> we'd probably need to test both the change of size in the decompressor
> code and the gains in compression. Since lzma is quite complex, the
> substitution might not pay off...

Hi,

Well, I did some testing a while ago, the result is in:

http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00114.html

Even with lzo compressed image core.img, lzma can save up to 5K.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

El lun, 16-06-2008 a las 19:47 +0800, Bean escribió:
> Well, I did some testing a while ago, the result is in:
> 
> http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00114.html
> 
> Even with lzo compressed image core.img, lzma can save up to 5K.
> 
You mean that even with an already compressed LZO image, LZMA can
further compress it another 5 KB? Then I guess we're in! The only
problem of LZMA that could become a blocker is it's expansive memory
use, but the core image is small enough not to need more that a few KBs
of memory for decompression, I hope.

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Mon, Jun 16, 2008 at 8:09 PM, Javier Martín <lordhabbit@gmail.com> wrote:
> El lun, 16-06-2008 a las 19:47 +0800, Bean escribió:
>> Well, I did some testing a while ago, the result is in:
>>
>> http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00114.html
>>
>> Even with lzo compressed image core.img, lzma can save up to 5K.
>>
> You mean that even with an already compressed LZO image, LZMA can
> further compress it another 5 KB? Then I guess we're in! The only
> problem of LZMA that could become a blocker is it's expansive memory
> use, but the core image is small enough not to need more that a few KBs
> of memory for decompression, I hope.

Hi,

But do we really need those space ? I think lzo is working fine, we
should avoid switching unless there is a good reason to do so.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Mon, 2008-06-16 at 14:09 +0200, Javier Martín wrote:
> El lun, 16-06-2008 a las 19:47 +0800, Bean escribió:
> > Well, I did some testing a while ago, the result is in:
> > 
> > http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00114.html
> > 
> > Even with lzo compressed image core.img, lzma can save up to 5K.
> > 
> You mean that even with an already compressed LZO image, LZMA can
> further compress it another 5 KB?

No, it was uncompressed.  GRUB 0.97 doesn't use LZO compression.

Anyway, we need to check savings on the GRUB 2 core image, which is much
shorter than stage2, and factor out the decompressor size.  My gut
feeling is that block sorting algorithms like lzma and bzip2 would be an
overkill, and even switching to gzip would be only marginally useful.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Bean]

On Mon, Jun 16, 2008 at 9:06 PM, Pavel Roskin <proski@gnu.org> wrote:
> On Mon, 2008-06-16 at 14:09 +0200, Javier Martín wrote:
>> El lun, 16-06-2008 a las 19:47 +0800, Bean escribió:
>> > Well, I did some testing a while ago, the result is in:
>> >
>> > http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00114.html
>> >
>> > Even with lzo compressed image core.img, lzma can save up to 5K.
>> >
>> You mean that even with an already compressed LZO image, LZMA can
>> further compress it another 5 KB?
>
> No, it was uncompressed.  GRUB 0.97 doesn't use LZO compression.

Yes, grub 0.97 don't use lzo, but I use external lzo tool to get the
result. It illustrates the result of lzo and lzma  algorithm on an
uncompressed file.

> Anyway, we need to check savings on the GRUB 2 core image, which is much
> shorter than stage2, and factor out the decompressor size.  My gut
> feeling is that block sorting algorithms like lzma and bzip2 would be an
> overkill, and even switching to gzip would be only marginally useful.

core.img is already compressed using lzo, so lzma's effect is not
apparent, but if we use lzma as the primary compressor, the saving
would be more significant.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

Quoting Bean <bean123ch@gmail.com>:

> On Mon, Jun 16, 2008 at 9:06 PM, Pavel Roskin <proski@gnu.org> wrote:
>> No, it was uncompressed.  GRUB 0.97 doesn't use LZO compression.
>
> Yes, grub 0.97 don't use lzo, but I use external lzo tool to get the
> result. It illustrates the result of lzo and lzma  algorithm on an
> uncompressed file.

I know all that.

>> Anyway, we need to check savings on the GRUB 2 core image, which is much
>> shorter than stage2, and factor out the decompressor size.  My gut
>> feeling is that block sorting algorithms like lzma and bzip2 would be an
>> overkill, and even switching to gzip would be only marginally useful.
>
> core.img is already compressed using lzo, so lzma's effect is not
> apparent, but if we use lzma as the primary compressor, the saving
> would be more significant.

I realize it.

Still, uncompressed core.img would be shorter than stage2 from GRUB 1,  
as a lot of code it modularized.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Mon, Jun 16, 2008 at 07:03:50PM +0800, Bean wrote:
> On Mon, Jun 16, 2008 at 5:15 PM, Robert Millan <rmh@aybabtu.com> wrote:
> > On Mon, Jun 16, 2008 at 02:30:38PM +0800, Bean wrote:
> >>
> >> The fd and hd device share many code, separating them is not a good
> >> idea.
> >
> > If code duplication is a problem, we could link the same code staticaly for
> > each module.  Size of biosdisk module is an issue, because it's always used
> > in core.img, but size of floppy module isn't, because it's never needed for
> > bootstrap.
> >
> >> Perhaps what we need is a way to optionally disable the floppy
> >> device, for example, with a variable such as no_floppy.
> >
> > But that makes biosdisk bigger, not smaller :-(
> 
> Hi,
> 
> Is size really that important ? I think even if you split the module,
> it would only save very little space, which doesn't matter in the big
> picture, unless you have a situation where the current size is very
> closed to the upper limit.

No, not really..  I think I'm being overzealous.  I'll see about using
that variable.

> In that case, I suggest more dramatic
> action, like replacing lzo with a strong compression algorithm, like
> lzma, which would save a few thousand bytes.

I recall this being discussed before.  Possibly this would make the
decompressor bigger, too.  Did you check the trade-off is good?

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Mon, Jun 16, 2008 at 01:20:27PM +0200, Javier Martín wrote:
> Hmm.. that could be done without too much fuss (except the part of
> implementing the decompressor in assembly for each platform...),

For other platforms than i386-pc, size is usually not a problem (well,
for coreboot it is, but there we have someone else to care about
compression).

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Mon, Jun 16, 2008 at 08:37:43PM +0800, Bean wrote:
> 
> But do we really need those space ?

The typical use case where space is a PITA is boot an LVM system without a
separate /boot partition.  There we don't have the ressort of using a
blocklist (which is already bad), so embedding core.img is mandatory, but
at the same time, we have to load lvm.mod in (possibly raid.mod as well),
and then you see where I'm going...

For example, with current CVS:

$ ./grub-mkimage -d . biosdisk pc ext2 lvm -o /tmp/core.img && du -b /tmp/core.img
30511   /tmp/core.img

$ ./grub-mkimage -d . biosdisk pc ext2 lvm raid -o /tmp/core.img && du -b /tmp/core.img
32940   /tmp/core.img

The first is close to the limit (32256 B) and the second surpasses it, so
boot from LVM on RAID with DOS label and ext2 is a use case we can't support
right now.

> I think lzo is working fine, we
> should avoid switching unless there is a good reason to do so.

ISTR Okuji didn't like it;  can't remember why.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Bean]

On Tue, Jun 17, 2008 at 5:31 AM, Robert Millan <rmh@aybabtu.com> wrote:
> On Mon, Jun 16, 2008 at 08:37:43PM +0800, Bean wrote:
>>
>> But do we really need those space ?
>
> The typical use case where space is a PITA is boot an LVM system without a
> separate /boot partition.  There we don't have the ressort of using a
> blocklist (which is already bad), so embedding core.img is mandatory, but
> at the same time, we have to load lvm.mod in (possibly raid.mod as well),
> and then you see where I'm going...
>
> For example, with current CVS:
>
> $ ./grub-mkimage -d . biosdisk pc ext2 lvm -o /tmp/core.img && du -b /tmp/core.img
> 30511   /tmp/core.img
>
> $ ./grub-mkimage -d . biosdisk pc ext2 lvm raid -o /tmp/core.img && du -b /tmp/core.img
> 32940   /tmp/core.img
>
> The first is close to the limit (32256 B) and the second surpasses it, so
> boot from LVM on RAID with DOS label and ext2 is a use case we can't support
> right now.

I run some tests on the your situation:

Current CVS:

/grub-mkimage -d . biosdisk pc ext2 lvm raid -o c1.img && du -b c1.img
32735   c1.img

Then I comment out the lzo compression in grub-mkimage with this patch:

diff --git a/util/i386/pc/grub-mkimage.c b/util/i386/pc/grub-mkimage.c
index 189ec4e..cb069ec 100644
--- a/util/i386/pc/grub-mkimage.c
+++ b/util/i386/pc/grub-mkimage.c
@@ -59,9 +59,11 @@ compress_kernel (char *kernel_img, size_t kernel_size,
   *core_img = xmalloc (kernel_size + kernel_size / 64 + 16 + 3);
   wrkmem = xmalloc (LZO1X_999_MEM_COMPRESS);

-  memcpy (*core_img, kernel_img, GRUB_KERNEL_MACHINE_RAW_SIZE);
+  //memcpy (*core_img, kernel_img, GRUB_KERNEL_MACHINE_RAW_SIZE);
+  memcpy (*core_img, kernel_img, kernel_size);

   grub_util_info ("compressing the core image");
+/*
   if (lzo1x_999_compress ((const lzo_byte *) (kernel_img
 					      + GRUB_KERNEL_MACHINE_RAW_SIZE),
 			  kernel_size - GRUB_KERNEL_MACHINE_RAW_SIZE,
@@ -70,10 +72,12 @@ compress_kernel (char *kernel_img, size_t kernel_size,
 			  &size, wrkmem)
       != LZO_E_OK)
     grub_util_error ("cannot compress the kernel image");
-
+*/
   free (wrkmem);

-  *core_size = (size_t) size + GRUB_KERNEL_MACHINE_RAW_SIZE;
+  //*core_size = (size_t) size + GRUB_KERNEL_MACHINE_RAW_SIZE;
+
+  *core_size = kernel_size;
 }

 static void

/grub-mkimage -d . biosdisk pc ext2 lvm raid -o c2.img && du -b c2.img
55572   c2.img

Finally, I use lzma to compress it:

lzma c2.img && du -b c2.img.lzma
26737   c2.img.lzma

The decompresser is about 2K, we should also exclude the uncompressed
part at the beginning of kernel, but there is still plenty room left.

>
>> I think lzo is working fine, we
>> should avoid switching unless there is a good reason to do so.
>
> ISTR Okuji didn't like it;  can't remember why.

He is concerned about compression ratio, but doesn't mind in the end:

http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00120.html

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

El mar, 17-06-2008 a las 10:03 +0800, Bean escribió:
> The decompresser is about 2K, we should also exclude the uncompressed
> part at the beginning of kernel, but there is still plenty room left.
The problem is not just the decompresser size itself, but how much
memory it uses. LZO is pretty special (though not unique) in that its
requisites in that area are very low; as low as just the source and
destination buffers. With LZMA (and other algorithms) we would need
intermediate buffers and other memory-consuming things like dictionaries
and such. I'm neither a compression expert nor an assembly guru, but the
tradeoff might not be up to the task.

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Tue, Jun 17, 2008 at 10:47 AM, Javier Martín <lordhabbit@gmail.com> wrote:
> El mar, 17-06-2008 a las 10:03 +0800, Bean escribió:
>> The decompresser is about 2K, we should also exclude the uncompressed
>> part at the beginning of kernel, but there is still plenty room left.
> The problem is not just the decompresser size itself, but how much
> memory it uses. LZO is pretty special (though not unique) in that its
> requisites in that area are very low; as low as just the source and
> destination buffers. With LZMA (and other algorithms) we would need
> intermediate buffers and other memory-consuming things like dictionaries
> and such. I'm neither a compression expert nor an assembly guru, but the
> tradeoff might not be up to the task.

Hi,

AFAIK, lzma decompresser don't need intermediate buffers, it operates
directly on the input and output buffer, for example, the lzmadecoder
in coreboot:

http://qa.coreboot.org/docs/doxygen/lzmadecode_8c-source.html

As for dictionary, it's optional, lzma don't use it by default.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Tue, Jun 17, 2008 at 10:03:15AM +0800, Bean wrote:
> >> I think lzo is working fine, we
> >> should avoid switching unless there is a good reason to do so.
> >
> > ISTR Okuji didn't like it;  can't remember why.
> 
> He is concerned about compression ratio, but doesn't mind in the end:
> 
> http://lists.gnu.org/archive/html/grub-devel/2007-12/msg00120.html

A problem I see here is that LZMA is licensed under GPLv2-only, so it'd
probably take a rewrite of the decompressor for us to use it, or ask the
author to make it "or later".

By the looks (no explicit license header in individual files) I doubt it
was an explicit decision to exclude GPLv3.  I'll try contact him.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Tue, Jun 17, 2008 at 02:22:15PM +0200, Robert Millan wrote:
> 
> A problem I see here is that LZMA is licensed under GPLv2-only, so it'd
> probably take a rewrite of the decompressor for us to use it, or ask the
> author to make it "or later".
> 
> By the looks (no explicit license header in individual files) I doubt it
> was an explicit decision to exclude GPLv3.  I'll try contact him.

Actually, after checking:

  http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility

it seems there's no problem.  But I'd like to see what Okuji thinks about it.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Tue, Jun 17, 2008 at 02:33:48PM +0200, Robert Millan wrote:
> On Tue, Jun 17, 2008 at 02:22:15PM +0200, Robert Millan wrote:
> > 
> > A problem I see here is that LZMA is licensed under GPLv2-only, so it'd
> > probably take a rewrite of the decompressor for us to use it, or ask the
> > author to make it "or later".
> > 
> > By the looks (no explicit license header in individual files) I doubt it
> > was an explicit decision to exclude GPLv3.  I'll try contact him.
> 
> Actually, after checking:
> 
>   http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility
> 
> it seems there's no problem.  But I'd like to see what Okuji thinks about it.

CCing him.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

El mar, 17-06-2008 a las 11:06 +0800, Bean escribió:
> AFAIK, lzma decompresser don't need intermediate buffers, it operates
> directly on the input and output buffer, for example, the lzmadecoder
> in coreboot:
I think you didn't see the intermediate buffer because it is hidden in
the Probs field passed through the decoderstate struct. I've googled my
way through the net to find what the hell "probs" mean, and I still
don't understand ten contiguous lines of the LZMA code. What I could
find is that 

http://qa.coreboot.org/docs/doxygen/lzma_8c-source.html

provides the decoder with about 32KiB of "scratchpad" memory and will
reject decompressing files which "require" (as somehow indicated by the
first 5 bytes of the LZMA stream) more than that. I don't know if that
space is used for dictionaries, trees, forests or a Matrix full of
virtual monkeys hitting keys at random until the decompressed stream is
achieved, but its size seems to impose constraints at the time of
compression (probably a maximum -n level) that merit investigation.

PS: have you tested the performance of other compression algorithms like
gzip-deflate (much simpler) or bzip2 (near LZMA for small files but with
less memory requisites)?


[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Tue, Jun 17, 2008 at 10:39 PM, Javier Martín <lordhabbit@gmail.com> wrote:
> El mar, 17-06-2008 a las 11:06 +0800, Bean escribió:
>> AFAIK, lzma decompresser don't need intermediate buffers, it operates
>> directly on the input and output buffer, for example, the lzmadecoder
>> in coreboot:
> I think you didn't see the intermediate buffer because it is hidden in
> the Probs field passed through the decoderstate struct. I've googled my
> way through the net to find what the hell "probs" mean, and I still
> don't understand ten contiguous lines of the LZMA code. What I could
> find is that
>
> http://qa.coreboot.org/docs/doxygen/lzma_8c-source.html
>
> provides the decoder with about 32KiB of "scratchpad" memory and will
> reject decompressing files which "require" (as somehow indicated by the
> first 5 bytes of the LZMA stream) more than that. I don't know if that
> space is used for dictionaries, trees, forests or a Matrix full of
> virtual monkeys hitting keys at random until the decompressed stream is
> achieved, but its size seems to impose constraints at the time of
> compression (probably a maximum -n level) that merit investigation.
>
> PS: have you tested the performance of other compression algorithms like
> gzip-deflate (much simpler) or bzip2 (near LZMA for small files but with
> less memory requisites)?

Hi,

I think memory is not a problem here, as grub2 can use upper memory:

src = raw data
dest = 0x100000
buf = dest + kernel_size

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Bean]

On Tue, Jun 17, 2008 at 10:39 PM, Javier Martín <lordhabbit@gmail.com> wrote:
> PS: have you tested the performance of other compression algorithms like
> gzip-deflate (much simpler) or bzip2 (near LZMA for small files but with
> less memory requisites)?

Hi,

I'm not against other compression algorithm, but lzma seems to be the
best, for example, for the previous c2.img:

bzip2 c2.img && du -b c2.img.bz2
29247   c2.img.bz2

gzip c2.img && du -b c2.img.gz
29825   c2.img.gz

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Tue, 2008-06-17 at 23:02 +0800, Bean wrote:

> I'm not against other compression algorithm, but lzma seems to be the
> best, for example, for the previous c2.img:
> 
> bzip2 c2.img && du -b c2.img.bz2
> 29247   c2.img.bz2
> 
> gzip c2.img && du -b c2.img.gz
> 29825   c2.img.gz

We need numbers for all compression algorithms for the code that would
actually be compressed, not the whole image, which includes the
multiboot header, the lzo uncompression implementation and some other
functions.

Anyway, it looks like gzip is doing a good job.  By the way, gzip
uncompression is used in Linux kernel on i386.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

El mar, 17-06-2008 a las 22:52 +0800, Bean escribió:
> Hi,
> 
> I think memory is not a problem here, as grub2 can use upper memory:
> 
> src = raw data
> dest = 0x100000
> buf = dest + kernel_size
> 
We could do that, but since the code would execute at the very first
stages, where we still don't have a memory map, we could overwrite
something important and wreak havoc. It's unlikely, since most computers
have a wide space at 1MB with most or all of the RAM, before the ACPI
structures and the BIOS area at the end. However, we should contemplate
the possibility of things going awry and prepare to receive bug reports
about failures on esoterically nonstandard computers, such as "GRUB
triple-faults my HP Vectra 75".
Alternatively, we could first check the BDA and, if we have enough
"lower" memory for the operation, use it before venturing into post-1MB
territory.

I'm currently checking the memory requisites of LZMA decompression
depending on the -1..9 compression setting, and will report back when
finished

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 1:23 AM, Javier Martín <lordhabbit@gmail.com> wrote:
> El mar, 17-06-2008 a las 22:52 +0800, Bean escribió:
>> Hi,
>>
>> I think memory is not a problem here, as grub2 can use upper memory:
>>
>> src = raw data
>> dest = 0x100000
>> buf = dest + kernel_size
>>
> We could do that, but since the code would execute at the very first
> stages, where we still don't have a memory map, we could overwrite
> something important and wreak havoc. It's unlikely, since most computers
> have a wide space at 1MB with most or all of the RAM, before the ACPI
> structures and the BIOS area at the end. However, we should contemplate
> the possibility of things going awry and prepare to receive bug reports
> about failures on esoterically nonstandard computers, such as "GRUB
> triple-faults my HP Vectra 75".
> Alternatively, we could first check the BDA and, if we have enough
> "lower" memory for the operation, use it before venturing into post-1MB
> territory.
>
> I'm currently checking the memory requisites of LZMA decompression
> depending on the -1..9 compression setting, and will report back when
> finished

Hi,

Well, actually current lzo decompresser uses similar method,
decompress the data directly to 0x100000. The compressed kernel is
only used in i386-pc platform, and space at 0x100000 should be
available, as Linux kernel will be loaded at this address as well.

About lzma setting, although I can't be sure, but the default seems to
gain the best result.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2817 bytes --]

El mié, 18-06-2008 a las 01:35 +0800, Bean escribió:
> On Wed, Jun 18, 2008 at 1:23 AM, Javier Martín <lordhabbit@gmail.com>
> wrote:
> > I'm currently checking the memory requisites of LZMA decompression
> > depending on the -1..9 compression setting, and will report back when
> > finished
> 
> Hi,
> 
> Well, actually current lzo decompresser uses similar method,
> decompress the data directly to 0x100000. The compressed kernel is
> only used in i386-pc platform, and space at 0x100000 should be
> available, as Linux kernel will be loaded at this address as well.
> 
> About lzma setting, although I can't be sure, but the default seems to
> gain the best result.
> 

jmp 1f

Here are the results of my "investigation": in large files (the kernel
tarball) LZMA is the absolute king, with some 3% gains over bzip2 at the
same -n compression level and more than 7% over gzip. However, memory
used by lzma for decompression increases with each -n compression level
from 2MB while reading a "-1" .lzma file to 34MB reading a "-9" file.
Memory usage for bzip2 also fluctuates, though less dramatically: from
1MB in -1 to 4 in -9. Mem usage of gzip is constant at 500KB.

1:

That was the theory: now for the actual use in _small_ files like
core.img. I found that bzip2 is a no-go, since it gives worse results
than gzip in this kind of files and is also more complex. LZMA shows
better results than gzip, but is the additional complexity worth the
savings? One would have to code both decompressors and check, but I
think the best option is switch to gzip, mainly because LZO compression
does _not_ save nearly anything!

ORIGINAL FILE: /boot/grub/core.img (28449 bytes in Ubuntu Hardy default)

CLvl     LZO-sav%    GZIP-sav%    LZMA-sav%
   1   28215-0.82   26647-6.33   26270-7.66
   2   28211-0.84   26643-6.35   26270-7.66
   3  *28211-0.84   26644-6.35   26185-7.96
   4   28211-0.84   26600-6.50   26185-7.96
   5   28211-0.84   26601-6.50   26185-7.96
   6   28211-0.84  *26596-6.52   26185-7.96
   7   28129-1.13   26596-6.52  *26185-7.96
   8   28129-1.13   26596-6.52   26185-7.96
   9   28129-1.13   26596-6.52   26185-7.96

* indicates the default compression level
Apps used: lzop v.1.01, GNU gzip 1.3.12, LZMA 4.43

With these results, I believe that the first conclusion is immediate: we
should dump LZO because even in its "best" compression level it saves
just 320 bytes, or 1.13%. Of the alternatives, while LZMA provides a
superior compression level, I think Gzip offers the best compromise
between code complexity and space savings. Besides, it is already used
in the Linux kernel for this very purpose, so we might even find that we
"just" need to copy, paste and adjust a bit.

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

Huge Oops!! Please forget my last, grandiloquent post about my
"investigation": it did not cross my mind that core.img was _already_
compressed with LZO, so the results are not fair to it. I will repeat
the test tomorrow with what could be a pseudo pre-compression core image
consisting of kernel.img, pc.mod, fshelp.mod and ext2.mod cat'ed
together, which should be similar enough to what grub-mkimage does.
Sorry for making everybody waste time (once again) due to my
short-sightedness... The worst part is that I digitally signed the last
message, so I cannot even deny I wrote it :-(

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Wed, 2008-06-18 at 05:14 +0200, Javier Martín wrote:

> ORIGINAL FILE: /boot/grub/core.img (28449 bytes in Ubuntu Hardy default)

That's an already compressed file!  Please see my posts in the thread.
We should be testing compression on the data we are actually
compressing.

That's what I'm getting with default compression settings:

44372 image
31195 image.lzo
24321 image.gz
24130 image.bz2
21858 image.lzma

The image was made by adding fopen(), fwrite() and fclose() before
lzo1x_999_compress() in grub-mkimage.c, compiling, installing GRUB and
running "grub-install /dev/sda"

The image is attached.

-- 
Regards,
Pavel Roskin

[-- Attachment #2: image.lzma --]
[-- Type: application/x-lzma, Size: 21858 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 11:49 AM, Pavel Roskin <proski@gnu.org> wrote:
> On Wed, 2008-06-18 at 05:14 +0200, Javier Martín wrote:
>
>> ORIGINAL FILE: /boot/grub/core.img (28449 bytes in Ubuntu Hardy default)
>
> That's an already compressed file!  Please see my posts in the thread.
> We should be testing compression on the data we are actually
> compressing.
>
> That's what I'm getting with default compression settings:
>
> 44372 image
> 31195 image.lzo
> 24321 image.gz
> 24130 image.bz2
> 21858 image.lzma
>
> The image was made by adding fopen(), fwrite() and fclose() before
> lzo1x_999_compress() in grub-mkimage.c, compiling, installing GRUB and
> running "grub-install /dev/sda"
>
> The image is attached.

Hi,

You can try Robert's case:

grub-mkimage -d . -o core.img pc ext2 lvm raid

The uncompressed part need to be added as well, which is 1280 byte.

My concern about other compression is that even if space problem can
be fixed now, but the limit is not far away, so we may have to worry
about it again when new code is added, lzma could be a way to solve
this once and for all.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Wed, 2008-06-18 at 14:55 +0800, Bean wrote:

> You can try Robert's case:
> 
> grub-mkimage -d . -o core.img pc ext2 lvm raid

50764  image
35090  image.lzo
27308  image.gz
27012  image.bz2
24470  image.lzma

> The uncompressed part need to be added as well, which is 1280 byte.

It's actually going to depend on the decompression algorithm.

> My concern about other compression is that even if space problem can
> be fixed now, but the limit is not far away, so we may have to worry
> about it again when new code is added, lzma could be a way to solve
> this once and for all.

We would save 7782 bytes going to gzip or 10620 bytes going to lzma.
There is not much difference as long as we abandon lzop in favor of
something else.

If I run string on the image to be compressed, I see all function names,
such as grub_biosdisk_get_diskinfo_int13_extensions.  I don't see why we
need function names there.  Function names starting with "grub_" take
4590 bytes.  It looks like we have some reserves for slimming down the
core.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 3:36 PM, Pavel Roskin <proski@gnu.org> wrote:
> If I run string on the image to be compressed, I see all function names,
> such as grub_biosdisk_get_diskinfo_int13_extensions.  I don't see why we
> need function names there.  Function names starting with "grub_" take
> 4590 bytes.  It looks like we have some reserves for slimming down the
> core.

Those symbols are used in the dynamic loading of modules. The module
is basically object file, it can have unresolved symbols. grub looks
up the symbol table to find the address when the module is loaded.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 3:36 PM, Pavel Roskin <proski@gnu.org> wrote:
> On Wed, 2008-06-18 at 14:55 +0800, Bean wrote:
>
>> You can try Robert's case:
>>
>> grub-mkimage -d . -o core.img pc ext2 lvm raid
>
> 50764  image
> 35090  image.lzo
> 27308  image.gz
> 27012  image.bz2
> 24470  image.lzma
>
>> The uncompressed part need to be added as well, which is 1280 byte.
>
> It's actually going to depend on the decompression algorithm.
>
>> My concern about other compression is that even if space problem can
>> be fixed now, but the limit is not far away, so we may have to worry
>> about it again when new code is added, lzma could be a way to solve
>> this once and for all.
>
> We would save 7782 bytes going to gzip or 10620 bytes going to lzma.
> There is not much difference as long as we abandon lzop in favor of
> something else.

Hi,

I'm sorry, I forget about an important module, biosdisk, would you run
the test again with the following modules ?

biosdisk pc ext2 lvm raid

The uncompressed part, without decompresser, is about 1K. If we count
the decompresser, I'd say it's between 1K to 3K.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 3559 bytes --]

El mar, 17-06-2008 a las 23:49 -0400, Pavel Roskin escribió:
> On Wed, 2008-06-18 at 05:14 +0200, Javier Martín wrote:
> 
> > ORIGINAL FILE: /boot/grub/core.img (28449 bytes in Ubuntu Hardy default)
> 
> That's an already compressed file!  Please see my posts in the thread.
> We should be testing compression on the data we are actually
> compressing.
Yeah, I noticed it _after_ posting and sent another message to the
effect. I repeated the tests, now with an uncompressed image. For the
"scientific method" guys wanting to repeat or extend the test, this is
how the it was performed:

/boot/grub$ cat kernel.img pc.mod fshelp.mod ext2.mod > ~/core.img
/boot/grub$ cd
~$ stat -c %s core.img
40412	# Size of the uncompressed file (for later)
~$ for n in {1..9}; do
	lzop core.img -c -$n > ./core.$n.lzo
	gzip core.img -c -$n > ./core.$n.gz
	bzip2 core.img -c -k -$n > ./core.$n.bz2
	lzma core.img -c -k -$n > ./core.$n.lzma
done 
~$ for comp in lzo gz bz2 lzma; do
	echo $comp
	for n in {1..9}; do
		echo $(stat -c %s ./core.$n.$comp)-$(dc -e "2 k 100 $(stat -c %
s ./core.$n.$comp) 404.12 / - p")
	done
done

For me, the results are:

CL     LZO-%sav    GZIP-%sav   BZIP2-%sav    LZMA-%sav
 1   29391-27.3   24171-40.2   22979-43.2   21569-46.7
 2   29279-27.6   23913-40.9        x       21567-46.7
 3  *     x       23785-41.2        x       20922-48.3
 4        x       23350-42.3        x            x
 5        x       23148-42.8        x       20924-48.3
 6        x      *23074-43.0        x            x
 7   25250-37.6   23071-43.0        x      *     x
 8   25163-37.8   23045-43.0        x       20929-48.3
 9   25162-37.8   23035-43.0  *     x            x
Where:
* indicates the default compression level
x indicates "same as above"
The original file size was 40412 bytes

I hope the results are a bit more useful this time. The "uncompressed
chunk" of nearly 1KiB was not taken into account, i.e. the whole file
was compressed by every algorithm. My conclusions are:
* LZO is pretty bad unless we set the maximum compression level, which
is not a problem because required memory and CPU is still nearly zero.
Seemingly, grub-mkimage already compresses with high CLs because the
core.img it generated with the same parameters than my cat'ed test image
was 25928 bytes long, with the uncompressed part and the decompressor.
* GZip is okay in its default compression level of -6, but, as LZO, we
don't lose nearly anything in terms of CPU and memory by pushing it to
-9. The particular advantage of this algorithm is that is already
implemented for this same purpose in the Linux kernel, so may be we
could use copy-paste to some point and relieve us some work. We'd save
up to 2 KiB in my test image, assuming similar decompressor size.
* BZip2 is also fine - its results are immutable throughout the CLs, I
suppose because the whole file fits in its -1 compression dictionary of
100 KiB. I don't think we'd benefit a lot from this particular algorithm
with files this small, and thus I prefer GZip over it.
* LZMA is the king of compression, and so we gain more bytes for code,
but at the cost of an implementation of unknown size - the C version of
the decompressor in the coreboot project seemed to be a bit big, but
maybe our local assembly gurus could tune it down. We'd save up to 4 KiB
in my test image, assuming similar decompressor size.

I hope I did not make any embarrassing mistakes this time :-) 


[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 7:40 PM, Javier Martín <lordhabbit@gmail.com> wrote:
> El mar, 17-06-2008 a las 23:49 -0400, Pavel Roskin escribió:
>> On Wed, 2008-06-18 at 05:14 +0200, Javier Martín wrote:
>>
>> > ORIGINAL FILE: /boot/grub/core.img (28449 bytes in Ubuntu Hardy default)
>>
>> That's an already compressed file!  Please see my posts in the thread.
>> We should be testing compression on the data we are actually
>> compressing.
> Yeah, I noticed it _after_ posting and sent another message to the
> effect. I repeated the tests, now with an uncompressed image. For the
> "scientific method" guys wanting to repeat or extend the test, this is
> how the it was performed:
>
> /boot/grub$ cat kernel.img pc.mod fshelp.mod ext2.mod > ~/core.img
> /boot/grub$ cd
> ~$ stat -c %s core.img
> 40412   # Size of the uncompressed file (for later)
> ~$ for n in {1..9}; do
>        lzop core.img -c -$n > ./core.$n.lzo
>        gzip core.img -c -$n > ./core.$n.gz
>        bzip2 core.img -c -k -$n > ./core.$n.bz2
>        lzma core.img -c -k -$n > ./core.$n.lzma
> done
> ~$ for comp in lzo gz bz2 lzma; do
>        echo $comp
>        for n in {1..9}; do
>                echo $(stat -c %s ./core.$n.$comp)-$(dc -e "2 k 100 $(stat -c %
> s ./core.$n.$comp) 404.12 / - p")
>        done
> done
>
> For me, the results are:
>
> CL     LZO-%sav    GZIP-%sav   BZIP2-%sav    LZMA-%sav
>  1   29391-27.3   24171-40.2   22979-43.2   21569-46.7
>  2   29279-27.6   23913-40.9        x       21567-46.7
>  3  *     x       23785-41.2        x       20922-48.3
>  4        x       23350-42.3        x            x
>  5        x       23148-42.8        x       20924-48.3
>  6        x      *23074-43.0        x            x
>  7   25250-37.6   23071-43.0        x      *     x
>  8   25163-37.8   23045-43.0        x       20929-48.3
>  9   25162-37.8   23035-43.0  *     x            x
> Where:
> * indicates the default compression level
> x indicates "same as above"
> The original file size was 40412 bytes
>
> I hope the results are a bit more useful this time. The "uncompressed
> chunk" of nearly 1KiB was not taken into account, i.e. the whole file
> was compressed by every algorithm. My conclusions are:
> * LZO is pretty bad unless we set the maximum compression level, which
> is not a problem because required memory and CPU is still nearly zero.
> Seemingly, grub-mkimage already compresses with high CLs because the
> core.img it generated with the same parameters than my cat'ed test image
> was 25928 bytes long, with the uncompressed part and the decompressor.
> * GZip is okay in its default compression level of -6, but, as LZO, we
> don't lose nearly anything in terms of CPU and memory by pushing it to
> -9. The particular advantage of this algorithm is that is already
> implemented for this same purpose in the Linux kernel, so may be we
> could use copy-paste to some point and relieve us some work. We'd save
> up to 2 KiB in my test image, assuming similar decompressor size.
> * BZip2 is also fine - its results are immutable throughout the CLs, I
> suppose because the whole file fits in its -1 compression dictionary of
> 100 KiB. I don't think we'd benefit a lot from this particular algorithm
> with files this small, and thus I prefer GZip over it.
> * LZMA is the king of compression, and so we gain more bytes for code,
> but at the cost of an implementation of unknown size - the C version of
> the decompressor in the coreboot project seemed to be a bit big, but
> maybe our local assembly gurus could tune it down. We'd save up to 4 KiB
> in my test image, assuming similar decompressor size.
>
> I hope I did not make any embarrassing mistakes this time :-)

Hi,

We also need to consider how close it's to the upper limit, 32256,
which is the available free space in the mbr. To access partition with
lvm and raid enabled, we need to use:

biosdisk pc ext2 fshelp lvm raid

If you rerun the test using the above modules, you will notice that
besides lzma, others are close to the upper limit.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Bean]

On Wed, Jun 18, 2008 at 9:12 PM, Bean <bean123ch@gmail.com> wrote:
> Hi,
>
> We also need to consider how close it's to the upper limit, 32256,
> which is the available free space in the mbr. To access partition with
> lvm and raid enabled, we need to use:
>
> biosdisk pc ext2 fshelp lvm raid
>
> If you rerun the test using the above modules, you will notice that
> besides lzma, others are close to the upper limit.

Also, ext2 is not the only fs that can be used to install linux, some
system may prefer to use xfs or reiserfs, which requires even more
space that ext2.

-- 
Bean

Re: [RFC] split floppy support off biosdisk

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

El mié, 18-06-2008 a las 21:12 +0800, Bean escribió:
> If you rerun the test using the above modules, you will notice that
> besides lzma, others are close to the upper limit.
Test rerun:
Modules cat'ed together: kernel.img biosdisk pc ext2 fshelp lvm raid
Uncompressed file size: 49644 bytes
grub-mkimage file size: 32032 bytes

CL     LZO-%sav    GZIP-%sav   BZIP2-%sav    LZMA-%sav
 1   34710-14.2   28556-29.4   26685-34.0   25208-37.7
 2   34642-14.3   28214-30.2   26685-34.0   25207-37.7
 3   34642-14.3   27998-30.8   26685-34.0   24314-39.9
 4   34642-14.3   27538-31.9   26685-34.0   24314-39.9
 5   34642-14.3   27230-32.7   26685-34.0   24322-39.9
 6   34642-14.3   27102-33.0   26685-34.0   24322-39.9
 7   29572-26.9   27099-33.0   26685-34.0   24322-39.9
 8   29407-27.3   27016-33.2   26685-34.0   24333-39.8
 9   29400-27.3   26998-33.2   26685-34.0   24333-39.8

Sorry, I'm running late and I have an exam today so I can't elaborate,
but you can see that gzip still saves 2.5Kib from LZO

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] split floppy support off biosdisk

[Isaac Dupree]

if it's not too much work to maintain multiple decompression algorithms, 
maybe we can have grub-install pick the weakest compression that still 
shrinks the image enough to fit?  Then only "complicated" setups will 
need LZMA's memory excesses.  (Might not be worth the effort though.)

-Isaac

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Wed, Jun 18, 2008 at 02:55:44PM +0800, Bean wrote:
> 
> You can try Robert's case:
> 
> grub-mkimage -d . -o core.img pc ext2 lvm raid

If you're into benchmarking, I'd suggest trying all permutations of
"biosdisk (pc|gpt) (ext2|fat|reiserfs|xfs) lvm raid".

(notice biosdisk too, I forgot to add that before)

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Wed, Jun 18, 2008 at 01:32:57PM -0400, Isaac Dupree wrote:
> if it's not too much work to maintain multiple decompression algorithms, 
> maybe we can have grub-install pick the weakest compression that still 
> shrinks the image enough to fit?  Then only "complicated" setups will 
> need LZMA's memory excesses.  (Might not be worth the effort though.)

I hope we can avoid having to do this.  The install scripts are complex
enough already...

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Yoshinori K. Okuji]

On Tuesday 17 June 2008 14:54:23 Robert Millan wrote:
> On Tue, Jun 17, 2008 at 02:33:48PM +0200, Robert Millan wrote:
> > On Tue, Jun 17, 2008 at 02:22:15PM +0200, Robert Millan wrote:
> > > A problem I see here is that LZMA is licensed under GPLv2-only, so it'd
> > > probably take a rewrite of the decompressor for us to use it, or ask
> > > the author to make it "or later".
> > >
> > > By the looks (no explicit license header in individual files) I doubt
> > > it was an explicit decision to exclude GPLv3.  I'll try contact him.
> >
> > Actually, after checking:
> >
> >   http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility
> >
> > it seems there's no problem.  But I'd like to see what Okuji thinks about
> > it.
>
> CCing him.

Which implementation of LZMA do you refer to? I think LZMA SDK is available 
under LGPL, and most of LZMA Utils is also under LGPL. So there should be no 
problem.

Okuji

Re: [RFC] split floppy support off biosdisk

[Yoshinori K. Okuji]

On Tuesday 17 June 2008 17:54:27 Pavel Roskin wrote:
> On Tue, 2008-06-17 at 23:02 +0800, Bean wrote:
> > I'm not against other compression algorithm, but lzma seems to be the
> > best, for example, for the previous c2.img:
> >
> > bzip2 c2.img && du -b c2.img.bz2
> > 29247   c2.img.bz2
> >
> > gzip c2.img && du -b c2.img.gz
> > 29825   c2.img.gz
>
> We need numbers for all compression algorithms for the code that would
> actually be compressed, not the whole image, which includes the
> multiboot header, the lzo uncompression implementation and some other
> functions.
>
> Anyway, it looks like gzip is doing a good job.  By the way, gzip
> uncompression is used in Linux kernel on i386.

Maybe as I said before, I already measured the size with LZO and gzip in PUPA, 
and LZO won (after adding the decompression code). That was why I chose LZO.

Okuji 

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Wed, Jun 18, 2008 at 07:52:26PM +0200, Yoshinori K. Okuji wrote:
> On Tuesday 17 June 2008 14:54:23 Robert Millan wrote:
> > On Tue, Jun 17, 2008 at 02:33:48PM +0200, Robert Millan wrote:
> > > On Tue, Jun 17, 2008 at 02:22:15PM +0200, Robert Millan wrote:
> > > > A problem I see here is that LZMA is licensed under GPLv2-only, so it'd
> > > > probably take a rewrite of the decompressor for us to use it, or ask
> > > > the author to make it "or later".
> > > >
> > > > By the looks (no explicit license header in individual files) I doubt
> > > > it was an explicit decision to exclude GPLv3.  I'll try contact him.
> > >
> > > Actually, after checking:
> > >
> > >   http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility
> > >
> > > it seems there's no problem.  But I'd like to see what Okuji thinks about
> > > it.
> >
> > CCing him.
> 
> Which implementation of LZMA do you refer to? I think LZMA SDK is available 
> under LGPL, and most of LZMA Utils is also under LGPL. So there should be no 
> problem.

I expected that LGPL 2.1 wouldn't be compatible with GPL 3, but it seems
that it is;  so unless I missed something there's no problem at all.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Thu, 2008-06-19 at 00:15 +0200, Robert Millan wrote:

> > Which implementation of LZMA do you refer to? I think LZMA SDK is available 
> > under LGPL, and most of LZMA Utils is also under LGPL. So there should be no 
> > problem.
> 
> I expected that LGPL 2.1 wouldn't be compatible with GPL 3, but it seems
> that it is;  so unless I missed something there's no problem at all.

It would be fine if we were just linking against lzma.  In this case,
software using GPLv3 would have at least the same rights as proprietary
software that is allowed to do that.

But we are going to incorporate the decompression function into the
bootloader, and that may not be allowed by LGPL.  I don't think it would
be allowed for proprietary software.  It would be a derived work
regardless of the library exceptions.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Yoshinori K. Okuji]

On Thursday 19 June 2008 01:56:52 Pavel Roskin wrote:
> On Thu, 2008-06-19 at 00:15 +0200, Robert Millan wrote:
> > > Which implementation of LZMA do you refer to? I think LZMA SDK is
> > > available under LGPL, and most of LZMA Utils is also under LGPL. So
> > > there should be no problem.
> >
> > I expected that LGPL 2.1 wouldn't be compatible with GPL 3, but it seems
> > that it is;  so unless I missed something there's no problem at all.
>
> It would be fine if we were just linking against lzma.  In this case,
> software using GPLv3 would have at least the same rights as proprietary
> software that is allowed to do that.
>
> But we are going to incorporate the decompression function into the
> bootloader, and that may not be allowed by LGPL.  I don't think it would
> be allowed for proprietary software.  It would be a derived work
> regardless of the library exceptions.

I took a look at lzma 4.32.6. All code in liblzmadec is licensed with LGPL 2.1 
or any later version. So where is the problem?

Okuji

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Thu, 2008-06-19 at 04:14 +0200, Yoshinori K. Okuji wrote:

> I took a look at lzma 4.32.6. All code in liblzmadec is licensed with LGPL 2.1 
> or any later version. So where is the problem?

Indeed, the "any later version" provision is there.  It wasn't mentioned
in the original discussion.  Then we are fine.

-- 
Regards,
Pavel Roskin

Re: [RFC] split floppy support off biosdisk

[Robert Millan]

On Wed, Jun 18, 2008 at 10:37:03PM -0400, Pavel Roskin wrote:
> On Thu, 2008-06-19 at 04:14 +0200, Yoshinori K. Okuji wrote:
> 
> > I took a look at lzma 4.32.6. All code in liblzmadec is licensed with LGPL 2.1 
> > or any later version. So where is the problem?
> 
> Indeed, the "any later version" provision is there.  It wasn't mentioned
> in the original discussion.  Then we are fine.

Maybe I didn't check in the right place.  Anyway, the matrix at:

  http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility

seems to indicate LGPL 2.1 and GPL 3 are compatible.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What good is a phone call… if you are unable to speak?
(as seen on /.)

Re: [RFC] split floppy support off biosdisk

[Pavel Roskin]

On Thu, 2008-06-19 at 14:20 +0200, Robert Millan wrote:
> On Wed, Jun 18, 2008 at 10:37:03PM -0400, Pavel Roskin wrote:
> > On Thu, 2008-06-19 at 04:14 +0200, Yoshinori K. Okuji wrote:
> > 
> > > I took a look at lzma 4.32.6. All code in liblzmadec is licensed with LGPL 2.1 
> > > or any later version. So where is the problem?
> > 
> > Indeed, the "any later version" provision is there.  It wasn't mentioned
> > in the original discussion.  Then we are fine.
> 
> Maybe I didn't check in the right place.  Anyway, the matrix at:
> 
>   http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility
> 
> seems to indicate LGPL 2.1 and GPL 3 are compatible.

You are right.  From
http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html:

3. You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library. To do
this, you must alter all the notices that refer to this License, so that
they refer to the ordinary GNU General Public License, version 2,
instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.)

-- 
Regards,
Pavel Roskin