Dynamic Rubin compression performance in JFFS2 on iPAQ

Dynamic Rubin compression performance in JFFS2 on iPAQ

[Christian, Andrew]

We recently updated the iPAQ bootldr (version 2.15.0 on  www.handhelds.org)
to support booting the kernel out of JFFS2.  In the process of doing this,
we ran into difficulties uncompressing kernel images from non-sorted lists
of file fragments - in particular, we had troubles with DYNRUBIN
decompression.

That caused us to look at the benefits of the different compression
algorithms in JFFS2.  On a typical iPAQ with approximately 45 Mbytes of
uncompressed data, we see the following numbers:
				
        Fragments	    Compressed Uncompressed   Saved     Saved
				(bytes)     (bytes)    (bytes)     (%)

NONE          813         248473      248473          0      0%   
ZLIB        12287       20886586    46882145   25995559     55%   	
DYNRUBIN       70	         20864       21490        626	 3%
RTIME           5	            94         115         21     18%

Total       13175       21156017    47152223   25996206     55%

These numbers are representative of the different iPAQs we tested - in fact,
DYNRUBIN in this iPAQ saved the most bytes of any of the tested iPAQS.

Including DYNRUBIN compression in the iPAQ Linux kernel requires an
additional 5793 bytes (a compressed zImage) 

For the moment, we have added a CONFIG option to turn off DYNRUBIN
compression for iPAQs --- decompression is left turned on for backward
compatibility.

We're curious - what is the intended target of DYNRUBIN compression?  We
haven't seen any advantages from including it on the iPAQ, but it must be
advantageous for some system or data type.  Anyone?


	Brian Avery		<brian.avery@compaq.com>
	Andrew Christian	<andrew.christian@compaq.com>


Compaq Computer Corporation
Cambridge Research Laboratory

Re: Dynamic Rubin compression performance in JFFS2 on iPAQ

[Russ Dill]

On 27 Aug 2001 11:04:59 -0400, Christian, Andrew wrote:
> We recently updated the iPAQ bootldr (version 2.15.0 on  www.handhelds.org)
> to support booting the kernel out of JFFS2.  In the process of doing this,
> we ran into difficulties uncompressing kernel images from non-sorted lists
> of file fragments - in particular, we had troubles with DYNRUBIN
> decompression.
> 
> That caused us to look at the benefits of the different compression
> algorithms in JFFS2.  On a typical iPAQ with approximately 45 Mbytes of
> uncompressed data, we see the following numbers:
> 				
>         Fragments	    Compressed Uncompressed   Saved     Saved
> 				(bytes)     (bytes)    (bytes)     (%)
> 
> NONE          813         248473      248473          0      0%   
> ZLIB        12287       20886586    46882145   25995559     55%   	
> DYNRUBIN       70	         20864       21490        626	 3%
> RTIME           5	            94         115         21     18%
> 
> Total       13175       21156017    47152223   25996206     55%
> 
> These numbers are representative of the different iPAQs we tested - in fact,
> DYNRUBIN in this iPAQ saved the most bytes of any of the tested iPAQS.
> 
> Including DYNRUBIN compression in the iPAQ Linux kernel requires an
> additional 5793 bytes (a compressed zImage) 
> 
> For the moment, we have added a CONFIG option to turn off DYNRUBIN
> compression for iPAQs --- decompression is left turned on for backward
> compatibility.
> 
> We're curious - what is the intended target of DYNRUBIN compression?  We
> haven't seen any advantages from including it on the iPAQ, but it must be
> advantageous for some system or data type.  Anyone?

I've allready written jffs2 boot code. DYNRUBIN can compress something
that has been gzipped, but its not worth it. I think you save more space
if you just put a straight vmlinux in the jffs2 image. Also, turning on
the I-cache is very important for speed. Also, the code contains its own
deflate algorithm that saves a lot of space over including zlib. My
jffs2/cramfs code is here

http://russ.dhs.org/load_kernel.html

It will get updated as it gets integrated into blob

RE: Dynamic Rubin compression performance in JFFS2 on iPAQ

[Christian, Andrew]

I should have mentioned that we started with your stuff, and modified it.
The major change is to cache the scanned dirnodes so directory traversals
are faster, and allowed us to add an "ls" command from the bootloader.

	Brian Avery		<brian.avery@compaq.com>


-----Original Message-----
From: Russ Dill [mailto:Russ.Dill@asu.edu]
Sent: Monday, August 27, 2001 11:29 AM
To: Christian, Andrew
Cc: 'linux-mtd@lists.infradead.org'; Hicks, Jamey
Subject: Re: Dynamic Rubin compression performance in JFFS2 on iPAQ


On 27 Aug 2001 11:04:59 -0400, Christian, Andrew wrote:
> We recently updated the iPAQ bootldr (version 2.15.0 on
www.handhelds.org)
> to support booting the kernel out of JFFS2.  In the process of doing this,
> we ran into difficulties uncompressing kernel images from non-sorted lists
> of file fragments - in particular, we had troubles with DYNRUBIN
> decompression.
> 
> That caused us to look at the benefits of the different compression
> algorithms in JFFS2.  On a typical iPAQ with approximately 45 Mbytes of
> uncompressed data, we see the following numbers:
> 				
>         Fragments	    Compressed Uncompressed   Saved     Saved
> 				(bytes)     (bytes)    (bytes)     (%)
> 
> NONE          813         248473      248473          0      0%   
> ZLIB        12287       20886586    46882145   25995559     55%   	
> DYNRUBIN       70	         20864       21490        626	 3%
> RTIME           5	            94         115         21     18%
> 
> Total       13175       21156017    47152223   25996206     55%
> 
> These numbers are representative of the different iPAQs we tested - in
fact,
> DYNRUBIN in this iPAQ saved the most bytes of any of the tested iPAQS.
> 
> Including DYNRUBIN compression in the iPAQ Linux kernel requires an
> additional 5793 bytes (a compressed zImage) 
> 
> For the moment, we have added a CONFIG option to turn off DYNRUBIN
> compression for iPAQs --- decompression is left turned on for backward
> compatibility.
> 
> We're curious - what is the intended target of DYNRUBIN compression?  We
> haven't seen any advantages from including it on the iPAQ, but it must be
> advantageous for some system or data type.  Anyone?

I've allready written jffs2 boot code. DYNRUBIN can compress something
that has been gzipped, but its not worth it. I think you save more space
if you just put a straight vmlinux in the jffs2 image. Also, turning on
the I-cache is very important for speed. Also, the code contains its own
deflate algorithm that saves a lot of space over including zlib. My
jffs2/cramfs code is here

http://russ.dhs.org/load_kernel.html

It will get updated as it gets integrated into blob

RE: Dynamic Rubin compression performance in JFFS2 on iPAQ

[Russ Dill]

On 27 Aug 2001 12:09:41 -0400, Christian, Andrew wrote:
> I should have mentioned that we started with your stuff, and modified it.
> The major change is to cache the scanned dirnodes so directory traversals
> are faster, and allowed us to add an "ls" command from the bootloader.
> 
> 	Brian Avery		<brian.avery@compaq.com>
> 
> 

ok, so certian images aren't decompressing DYNRUBIN compressed files
properly? Were you aware of the usermode debugging mode? Send me over
what you have, along with the image that fails and the correctly
decompressed file.