squashfs-lzma bitbake recipe needed

squashfs-lzma bitbake recipe needed

[Johannes Schauer]

Hello openembedded devs!

Since quite some time now I'm into the creation of an offline wikipedia for embedded devices - specifically the openmoko neo1973 but of course this will work on every other linux too.
The most important ingredient for this to work is the use of the squashfs filesystem as the storage format as I did not want to create just another file format.
Together with lzma patches this solution provides compression ratios of 19-20% of the original size for wikipedia xhtml articles.
Out of a 1.3GB german wikipedia dump *every* article is seeked to, decompressed and read in only 0.57seconds on a openmoko neo1973.

the only "downside" of this solution is that the user will have to load the squashfs kernel module to be able to mount a wikipedia dump into his file system.
To minimize the efforts on installing the module i need help from your side in building a bitbake recipe for an easy to install ipk.

Of course such a module would not only benefit my work but many other possible scenarios where big amounts of read-only data is required on the small space, memory and processor power of embedded devices.

i already natively build the module on debian-armel on the neo1973 so i can confirm that everything works as expected and i already supply the ready to use armv4t module binaries for a 2.6.24 kernel here: http://rabenfrost.net/mokopedia/sqlzma_armv4t.tgz

you have to know that squashfs alone doesnt use lzma as they want to be included in mainline kernel which only uses gzip which performs much worse than lzma (compression of 33-35% and only slightly faster on decompression)
so one has to use some patches that were wrote for the slax distribution

there are two methods to build those modules:
first is to use the squashfs-lzma patches supplied here: http://www.squashfs-lzma.org/
as they are a bit unmaintained i wrote this howto to natively compile them against a 2.6.24 kernel source: http://mokopedia.mister-muffin.de/wiki/BuildingSquashfsLzma

second there are the scripts used by slax here: ftp://ftp.slax.org/source/slax/kernel/2.6.24.3/src-core/
the script supplied there builds the slax kernel together with aufs and squashfs-lzma kernel and is guaranteed to work
i modified the build.kernel script a bit for my purpose and removed the aufs parts and the kernel compilation and added a variable for the desired kernel tarball to unpack and use.
you can get the modified version here: http://pastebin.com/f29f6e053

in the end those two methods nearly work the same.
they load the squashfs and lzma source, patch those together with the kernel source and then build the modules as well as squashfs-tools with the supplied makefile

as you can see this process is rather complicated and i need guidance on how to best fit everything into a bitbake recipe as i'm not much into that.

thank you for your help!

josch

Re: squashfs-lzma bitbake recipe needed

[Rolf Leggewie]

Hi,

Johannes Schauer wrote:
> Since quite some time now I'm into the creation of an offline wikipedia for embedded devices

you should have sent that mail earlier.  I've been doing some work on
this and have already checked in a few recipes.

> The most important ingredient for this to work is the use of the squashfs filesystem as the storage format as I did not want to create just another file format.

I am not sure this is the most appropriate.  My tests indicate a size
increase of at least 100%.

I think my work should be pretty well documented in bug 3711.  There are
uncommitted bits and pieces I have flying around locally, but the
general direction is spelt out there.

Before starting work, you should always consult the bug tracker.  If
there is no ticket about a task yet, open a new one.  This saves us from
a lot of frustration and double work.

As far as lzma module is concerned, I think it's also been done
somewhere.  "rgrep lzma packages/" is your friend.

Regards

Rolf

Re: squashfs-lzma bitbake recipe needed

[Leon Woestenberg]

Hello,

On Wed, Jul 16, 2008 at 10:34 AM, Rolf Leggewie
<no2spam@nospam.arcornews.de> wrote:

>> The most important ingredient for this to work is the use of the squashfs filesystem as the storage format as I did not want to create just another file format.
>
I have used squashfs-lzma with OpenEmbedded for years now, mostly on
IXP4xx platforms.

I didn't know it was special, but it might be most of our kernels do
not have the patches in place, so thanks for any pointers/work there.

Really, squashfs-lzma should be mainlined.

Regards,
-- 
Leon

Re: squashfs-lzma bitbake recipe needed

[Leon Woestenberg]

Hello,

On Wed, Jul 16, 2008 at 9:52 AM, Johannes Schauer <j.schauer@email.de> wrote:
> as they are a bit unmaintained i wrote this howto to natively compile them against a 2.6.24 kernel source: http://mokopedia.mister-muffin.de/wiki/BuildingSquashfsLzma
>

I submitted my work agains the 2.6.24 and 2.6.25 trees of the IXP4xx
kernel tree:

http://svn.nslu2-linux.org/svnroot/kernel/trunk/patches/2.6.25/squashfs/

I am using it on other platforms as well.

Regards,
-- 
Leon