[U-Boot-Users] LZMA support

[U-Boot-Users] LZMA support

[Luigi 'Comio' Mantellini]

Hi people,

I have a stupid question: why does u-boot not support the LZMA
algorithm?

(Anyway, I have a self-made and untested patch that introduces this
compression algorithm).

Thanks a lot.

luigi


Industrie Dial Face S.p.A.
Luigi Mantellini
R&D - Software
Industrie Dial Face S.p.A.
Via Canzo, 4 
20068 Peschiera Borromeo (MI), Italy
Tel.:   +39 02 5167 2813
Fax:    +39 02 5167 2459
E-mail: luigi.mantellini at idf-hit.com
GPG fingerprint: 3DD1 7B71 FBDF 6376
1B4A
                 B003 175F E979 907E
1650

[U-Boot-Users] LZMA support (patch)

[Luigi 'Comio' Mantellini]

Hi everyone,

this patch adds the lzma support with the following constraints:

 - it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release
 - the imported lzma sdk files are _not touched_
 - the lzma support is enabled defining the CONFIG_LZMA define
 - I haven't tested this patch
 - ?Use this patch at your own risk

The patch creates:

 - the lib_lzma directory with the import_lzmasdk.sh script and a simple
front-end to LzmaDecode() fuction
 - the include/lzma directory with some include files to use in the
U-Boot code

modifies:

 - common/do_bootm.c file
 - common/image.c file
 - include/image.h file

Please read the lib_lzma/README.txt file.

Luigi



On ven, 2008-03-28 at 16:42 +0100, Luigi 'Comio' Mantellini wrote:
> Hi people,
> 
> I have a stupid question: why does u-boot not support the LZMA
> algorithm?
> 
> (Anyway, I have a self-made and untested patch that introduces this
> compression algorithm).
> 
> Thanks a lot.
> 
> luigi
> 
> 
> Industrie Dial Face S.p.A.
> Luigi Mantellini
> R&D - Software
> Industrie Dial Face S.p.A.
> Via Canzo, 4 
> 20068 Peschiera Borromeo (MI), Italy
> Tel.:   +39 02 5167 2813
> Fax:    +39 02 5167 2459
> E-mail: luigi.mantellini at idf-hit.com
> GPG fingerprint: 3DD1 7B71 FBDF 6376
> 1B4A
>                  B003 175F E979 907E
> 1650
> 
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> U-Boot-Users mailing list
> U-Boot-Users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/u-boot-users
-- 
     ______       Luigi Mantellini
   .'______'.     R&D - Software
  (.'      '.)    Industrie Dial Face S.p.A.
  ( :=----=: )    Via Canzo, 4
  ('.______.')    20068 Peschiera Borromeo (MI), Italy
   '.______.'     Tel.: +39 02 5167 2813
                  Fax:  +39 02 5167 2459
Ind.  Dial Face   Email: luigi.mantellini at idf-hit.com
www.idf-hit.com   GPG fingerprint: 3DD1 7B71 FBDF 6376 1B4A
                                   B003 175F E979 907E 1650



-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-LZMA-support-for-kernel-images.patch
Type: application/mbox
Size: 15652 bytes
Desc: not available
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080328/1cb1fe9e/attachment.bin 

[U-Boot-Users] LZMA support (patch)

[Jerry Van Baren]

Luigi 'Comio' Mantellini wrote:
> Hi everyone,
> 
> this patch adds the lzma support with the following constraints:
> 
>  - it doesn't install the LZMA SDK files. I provide the import_lzma.sh
> script to import the needs file from a lzma sdk release
>  - the imported lzma sdk files are _not touched_
>  - the lzma support is enabled defining the CONFIG_LZMA define
>  - I haven't tested this patch
>  - ?Use this patch at your own risk
> 
> The patch creates:
> 
>  - the lib_lzma directory with the import_lzmasdk.sh script and a simple
> front-end to LzmaDecode() fuction
>  - the include/lzma directory with some include files to use in the
> U-Boot code
> 
> modifies:
> 
>  - common/do_bootm.c file
>  - common/image.c file
>  - include/image.h file
> 
> Please read the lib_lzma/README.txt file.
> 
> Luigi

Hi Luigi,

I have a stupid question in return: why do we want lzma (7zip) added to 
u-boot?  The linux/uboot world is dominated by gzip.  I realize 7zip has 
advantages over gzip albeit incremental, not revolutionary IIRC.

I am not aware of proposals or implementations of kernels compressed 
with lzma (could well be my ignorance).  Adding more code for minimal 
benefits is a hard sell (we already have plenty of bloat :-O).

Best regards,
gvb

[U-Boot-Users] LZMA support (patch)

[Hoi-Ho Chan]

On Fri, Mar 28, 2008 at 12:31 PM, Jerry Van Baren
<gerald.vanbaren@ge.com> wrote:
> Luigi 'Comio' Mantellini wrote:
>  > Hi everyone,
>  >
>  > this patch adds the lzma support with the following constraints:
>  >
>  >  - it doesn't install the LZMA SDK files. I provide the import_lzma.sh
>  > script to import the needs file from a lzma sdk release
>  >  - the imported lzma sdk files are _not touched_
>  >  - the lzma support is enabled defining the CONFIG_LZMA define
>  >  - I haven't tested this patch
>  >  - ?Use this patch at your own risk
>  >
>  > The patch creates:
>  >
>  >  - the lib_lzma directory with the import_lzmasdk.sh script and a simple
>  > front-end to LzmaDecode() fuction
>  >  - the include/lzma directory with some include files to use in the
>  > U-Boot code
>  >
>  > modifies:
>  >
>  >  - common/do_bootm.c file
>  >  - common/image.c file
>  >  - include/image.h file
>  >
>  > Please read the lib_lzma/README.txt file.
>  >
>  > Luigi
>
>  Hi Luigi,
>
>  I have a stupid question in return: why do we want lzma (7zip) added to
>  u-boot?  The linux/uboot world is dominated by gzip.  I realize 7zip has
>  advantages over gzip albeit incremental, not revolutionary IIRC.
>
>  I am not aware of proposals or implementations of kernels compressed
>  with lzma (could well be my ignorance).  Adding more code for minimal
>  benefits is a hard sell (we already have plenty of bloat :-O).
>
>  Best regards,
>  gvb
>
>

Would be great if it's being used for squashfs-lzma support.

Thanks
Donald

[U-Boot-Users] LZMA support (patch)

[Luigi 'Comio' Mantellini]

Hi Jerry,

see inline comments.

On ven, 2008-03-28 at 13:31 -0400, Jerry Van Baren wrote:
> Luigi 'Comio' Mantellini wrote:
> > Hi everyone,
> > 
> > this patch adds the lzma support with the following constraints:
> > 
> >  - it doesn't install the LZMA SDK files. I provide the import_lzma.sh
> > script to import the needs file from a lzma sdk release
> >  - the imported lzma sdk files are _not touched_
> >  - the lzma support is enabled defining the CONFIG_LZMA define
> >  - I haven't tested this patch
> >  - ?Use this patch at your own risk
> > 
> > The patch creates:
> > 
> >  - the lib_lzma directory with the import_lzmasdk.sh script and a simple
> > front-end to LzmaDecode() fuction
> >  - the include/lzma directory with some include files to use in the
> > U-Boot code
> > 
> > modifies:
> > 
> >  - common/do_bootm.c file
> >  - common/image.c file
> >  - include/image.h file
> > 
> > Please read the lib_lzma/README.txt file.
> > 
> > Luigi
> 
> Hi Luigi,
> 
> I have a stupid question in return: why do we want lzma (7zip) added to 
> u-boot?  The linux/uboot world is dominated by gzip.  I realize 7zip has 
> advantages over gzip albeit incremental, not revolutionary IIRC.
> 

LZMA usually performs better than gzip in terms compression ratio. My
application has a small flash memory and I need to keep two (2) kernel
images... 

From these observations born my need to save as much memory as possible.


Best regards,

luigi

> I am not aware of proposals or implementations of kernels compressed 
> with lzma (could well be my ignorance).  Adding more code for minimal 
> benefits is a hard sell (we already have plenty of bloat :-O).
> 
> Best regards,
> gvb

-- 
     ______       Luigi Mantellini
   .'______'.     R&D - Software
  (.'      '.)    Industrie Dial Face S.p.A.
  ( :=----=: )    Via Canzo, 4
  ('.______.')    20068 Peschiera Borromeo (MI), Italy
   '.______.'     Tel.: +39 02 5167 2813
                  Fax:  +39 02 5167 2459
Ind.  Dial Face   Email: luigi.mantellini at idf-hit.com
www.idf-hit.com   GPG fingerprint: 3DD1 7B71 FBDF 6376 1B4A
                                   B003 175F E979 907E 1650

[U-Boot-Users] LZMA support (patch)

[Wolfgang Denk]

In message <1206724722.23627.28.camel@localhost> you wrote:
> 
> this patch adds the lzma support with the following constraints:
> 
>  - it doesn't install the LZMA SDK files. I provide the import_lzma.sh
> script to import the needs file from a lzma sdk release

Sorry, but I will not accept this.

U-Boot has always been self-contained, and I don't intend to change
this.

Please make sure that a patch contains all files that belong to it.

Also, please explain what this new code is useful for. So far I don;t
see any real-life use cases ?


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"I like your game but we have to change the rules."

[U-Boot-Users] LZMA support (patch)

[Luigi 'Comio' Mantellini]

> In message <1206724722.23627.28.camel@localhost> you wrote:
>>
>> this patch adds the lzma support with the following
>> constraints:
>>
>>  - it doesn't install the LZMA SDK files. I provide the
>> import_lzma.sh
>> script to import the needs file from a lzma sdk release
>
> Sorry, but I will not accept this.
>
> U-Boot has always been self-contained, and I don't intend to
> change
> this.

Running the contained script the directory will be populated
with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can
resubmit the patch including all files. The LZMA SDK is
released under LGPL with exceptions but I don't understand if
the license is compatible with the u-boot's license. For this
reason I omitted  the sources from the SDK.

>
> Please make sure that a patch contains all files that belong to
> it.
>
> Also, please explain what this new code is useful for. So far I
> don;t
> see any real-life use cases ?
>
>

LMZA performs better than LZ on binary files. I will switch to
LZMA to save flash memory space on my appliance. Of course, the
LZMA is slower than LZ.

Anyway, if the LZMA SDK license is compatible with U-boot , I
think that LZMA should be chosen by firmware developer when the
flash memory space is a critical resource.

My 2 Cents,

luigi

> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev
> Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> wd at denx.de
> "I like your game but we have to change the rules."
>

[U-Boot-Users] LZMA support (patch)

[Wolfgang Denk]

In message <5566.213.156.35.235.1206739577.webmail@webmail.idf-hit.com> you wrote:
>
> Running the contained script the directory will be populated
> with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can
> resubmit the patch including all files. The LZMA SDK is
> released under LGPL with exceptions but I don't understand if
> the license is compatible with the u-boot's license. For this
> reason I omitted  the sources from the SDK.

This makes no sense to me.

Either the license is compatible, and the code can be included, or it
is not, then it cannot. In the latter case, the whole patch makes no
sense.

Please check for compatibility of licenses berfore resubmitting.

> LMZA performs better than LZ on binary files. I will switch to
> LZMA to save flash memory space on my appliance. Of course, the
> LZMA is slower than LZ.

But so far there is no code (board support) in  U-Boot  to  use  this
feature, right? And it's not used by the Linux kernel either?

And your patch does not even enable support for it in the mkimage
tool, so you cannot even create images that use the feature?

How exactly is this code supposed to be used?

> Anyway, if the LZMA SDK license is compatible with U-boot , I
> think that LZMA should be chosen by firmware developer when the
> flash memory space is a critical resource.

Frankly, if you are so short with flash you have a h/w dsign problem.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
core error - bus dumped

[U-Boot-Users] LZMA support (patch)

[Ulf Samuelsson]

>> Running the contained script the directory will be populated
>> with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can
>> resubmit the patch including all files. The LZMA SDK is
>> released under LGPL with exceptions but I don't understand if
>> the license is compatible with the u-boot's license. For this
>> reason I omitted  the sources from the SDK.
> 
> This makes no sense to me.
> 
> Either the license is compatible, and the code can be included, or it
> is not, then it cannot. In the latter case, the whole patch makes no
> sense.
> 

I think there should be no problem from a license point of view.

The home page says that you can use unmodified source
in your own code without having to release it to the public.

Since U-boot is open source to start with, this cannot be a problem.

It goes on and says that if the LZMA lib is modified,
then the resulting code has to be released as open source.
Again, no problem for U-boot, since it is open source to start with.

There are also commercial licenses available, but that is not applicable
since noone signed a commerical license.


> Please check for compatibility of licenses berfore resubmitting.
> 
>> LMZA performs better than LZ on binary files. I will switch to
>> LZMA to save flash memory space on my appliance. Of course, the
>> LZMA is slower than LZ.
> 
> But so far there is no code (board support) in  U-Boot  to  use  this
> feature, right? And it's not used by the Linux kernel either?
> 

There are kernel patches for LZMA compression available.
Can be selected at least in buildroot.

> And your patch does not even enable support for it in the mkimage
> tool, so you cannot even create images that use the feature?
> 

Maybe the patches for Linux creates an image???
Never saw any need to use it myself.

> How exactly is this code supposed to be used?
> 
>> Anyway, if the LZMA SDK license is compatible with U-boot , I
>> think that LZMA should be chosen by firmware developer when the
>> flash memory space is a critical resource.
> 
> Frankly, if you are so short with flash you have a h/w design problem.
> 

Lots of applications are cost sensitive, and shaving 50 cents here,
50 cents there could be important.
Hard to judge without knowing the details.

> Best regards,
> 
> Wolfgang Denk
> 


Best Regards
Ulf Samuelsson

[U-Boot-Users] LZMA support (patch)

[Luigi 'Comio' Mantellini]

Hi Wolfgang,

> In message
> <5566.213.156.35.235.1206739577.webmail@webmail.idf-hit.com>
> you wrote:
>>
>> Running the contained script the directory will be populated
>> with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I
>> can
>> resubmit the patch including all files. The LZMA SDK is
>> released under LGPL with exceptions but I don't understand if
>> the license is compatible with the u-boot's license. For this
>> reason I omitted  the sources from the SDK.
>
> This makes no sense to me.

I'm not expert in legalese questions.

>
> Either the license is compatible, and the code can be included,
> or it
> is not, then it cannot. In the latter case, the whole patch
> makes no
> sense.
>
> Please check for compatibility of licenses berfore
> resubmitting.

I think that the license is compatible (see the
http://www.7-zip.org/sdk.html). I just share my code with
community...
>
>> LMZA performs better than LZ on binary files. I will switch
>> to
>> LZMA to save flash memory space on my appliance. Of course,
>> the
>> LZMA is slower than LZ.
>
> But so far there is no code (board support) in  U-Boot  to  use
>  this
> feature, right? And it's not used by the Linux kernel either?
>

The code is used (eventually) by do_bootm routine to uncompress
a lzma-ed kernel image. The patch provides the opportune
image.c/cmd_bootm.c changes for this.

> And your patch does not even enable support for it in the
> mkimage
> tool, so you cannot even create images that use the feature?
>

mkimage doesn't compress anything! As well as gzipped images,
the lzma-ed images are created by an external tool (7z?):
mkimage adds just a simple identification header to identify
the used compression method. The proposed patch permits to
mkimage to set the opportune compression type (id=3) in the
uImage to recognize the lzma images.

> How exactly is this code supposed to be used?
>
>> Anyway, if the LZMA SDK license is compatible with U-boot , I
>> think that LZMA should be chosen by firmware developer when
>> the
>> flash memory space is a critical resource.
>
> Frankly, if you are so short with flash you have a h/w dsign
> problem.
>

The project constraints require a cheap flash memory to reduce
the final cost (and prize) of the device. This is not a design
problem...

luigi

> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev
> Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
> wd at denx.de
> core error - bus dumped
>

[U-Boot-Users] LZMA support (patch)

[Mike Frysinger]

On Friday 28 March 2008, Wolfgang Denk wrote:
> But so far there is no code (board support) in  U-Boot  to  use  this
> feature, right? And it's not used by the Linux kernel either?
>
> And your patch does not even enable support for it in the mkimage
> tool, so you cannot even create images that use the feature?

ive seen patches floating around to add lzma support akin to gzip/bzip2 in the 
uImage format.  makes sense to me, and some people are very interested in 
shaving off flash space in return for slightly longer boot time.  linux 
kernel is irrelevant here.  someone else doing this means i dont have to at 
some point ;).
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080328/1d8c1860/attachment.pgp 

[U-Boot-Users] LZMA support (patch)

[Luigi 'Comio' Mantellini]

Hi All,

I will rework the patch in order to include all files from LZMA
SDK and some fix. I'll send the patch on Tuesday (I'm in
vacation). These are the implemented features:

- LZMA SDK 4.57 Decode code (vanilla)
- A Buff2Buff wrapper to decode the image (LZMA_Alone format)
- Fixes for mkimage (option -C lzma) and do_bootm
- small footprint and small impact on the u-boot sources.

To use the LZMA algorithm: just #define CONFIG_LZMA in your
board config file.

Ciao.

luigi

> On Friday 28 March 2008, Wolfgang Denk wrote:
>> But so far there is no code (board support) in  U-Boot  to
>> use  this
>> feature, right? And it's not used by the Linux kernel either?
>>
>> And your patch does not even enable support for it in the
>> mkimage
>> tool, so you cannot even create images that use the feature?
>
> ive seen patches floating around to add lzma support akin to
> gzip/bzip2 in the
> uImage format.  makes sense to me, and some people are very
> interested in
> shaving off flash space in return for slightly longer boot
> time.  linux
> kernel is irrelevant here.  someone else doing this means i
> dont have to at
> some point ;).
> -mike
>

[U-Boot-Users] LZMA support (patch)

[Jean-Christophe PLAGNIOL-VILLARD]

On 21:21 Sat 29 Mar     , Luigi 'Comio' Mantellini wrote:
> Hi All,
> 
> I will rework the patch in order to include all files from LZMA
> SDK and some fix. I'll send the patch on Tuesday (I'm in
> vacation). These are the implemented features:
> 
> - LZMA SDK 4.57 Decode code (vanilla)
> - A Buff2Buff wrapper to decode the image (LZMA_Alone format)
> - Fixes for mkimage (option -C lzma) and do_bootm
> - small footprint and small impact on the u-boot sources.
> 
> To use the LZMA algorithm: just #define CONFIG_LZMA in your
> board config file.
Please update the README

Best Regards,
J.

[U-Boot-Users] LZMA support (patch)

[Harald Welte]

Hi Wolfgang, Luigi,

On Fri, Mar 28, 2008 at 11:33:58PM +0100, Wolfgang Denk wrote:

> Either the license is compatible, and the code can be included, or it
> is not, then it cannot. In the latter case, the whole patch makes no
> sense.

Given my expertise with license compliance, I volunteer to look into the
legal issues, to the point that if I have any doubt I will consult with
my legal counsel on that subject.

> > LMZA performs better than LZ on binary files. I will switch to
> > LZMA to save flash memory space on my appliance. Of course, the
> > LZMA is slower than LZ.
> 
> But so far there is no code (board support) in  U-Boot  to  use  this
> feature, right? And it's not used by the Linux kernel either?

As for board support / LZMA: I don't know.  But I'm pretty sure that
I've seen LZMA compressed u-boot images before.  LZMA is really popular
with cheap consumer-grade embedded linux equipment, particularly DSL CPE
and Wifi Routers.  Most of them don't use u-boot, but only use it for
the kernel image and the flash filesystem.  They're using quite simple
out-of-tree patches for that, though.

I think this has come up on lkml before, and Linus didn't like it very
much.  Probably related to the fact that many core kernel hackers look
at embedded as sort of a special case, when in reality I am quite sure
that the total quantity of embedded systems running Linux is
significantly larger than the number of Linux servers + desktops
together...

> And your patch does not even enable support for it in the mkimage
> tool, so you cannot even create images that use the feature?

That has obviously to be fixed.
 
> How exactly is this code supposed to be used?

some documentation would certainly be fine, yes.

> > Anyway, if the LZMA SDK license is compatible with U-boot , I
> > think that LZMA should be chosen by firmware developer when the
> > flash memory space is a critical resource.
> 
> Frankly, if you are so short with flash you have a h/w dsign problem.

Yes and no.  I tend to agree, but I also realize that there are economic
incentives to keep the bom cost as low as possible, especially in
large-quantity consumer equipment...

You can just as well claim: If you're using suboptimal compression and
wasting flash space, you have a software design problem ;)

I think it makes a lot of sense to add LZMA support to u-boot, but
obviously in a clean, consistent, documented way.  It doesn't hurt, will
not be compiled unless explicitly selected, and the API is zlib like,
i.e. the code changes it requires are non-intrusive.

Just my thoughs...

--
- Harald Welte <laforge@gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080406/432514cb/attachment.pgp