* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Phillip Lougher @ 2009-12-10 23:20 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: dedekind1, Andrew Morton, linux-embedded, linux-fsdevel,
linux-kernel, phillip.lougher, Tim Bird, Geert Uytterhoeven,
Felix Fietkau
In-Reply-To: <10f740e80912100201l6d83588evff46e95ec96d0aed@mail.gmail.com>
Geert Uytterhoeven wrote:
> Hi Phillip,
>
> On Thu, Dec 10, 2009 at 01:38, Phillip Lougher
> <phillip@lougher.demon.co.uk> wrote:
>> Artem Bityutskiy wrote:
>>> Did you consider using cryptoapi? UBIFS uses zlib/lzo in cryptoapi - it
>>> is a very clean way.
>
> Exactly my question, as that's why the Crypto API was extended with
> support for partial (de)compression in the first place ;-)
>
Your cryptoapi patches are on my "must do something about them" list, I've
not forgotten them :-) The lack of lzma in the crypotapi made moving
solely to the cryptoapi difficult, when I wanted to add lzma decompression
support.
The decompressor framework is my _solution_ to this problem. This allows
cryptoapi support to be added as additional decompression_wrapper, with
multiple compression types using the crptoapi wrapper as appropriate.
This has the advantage it allows Squashfs to use the cryptoapi (with
its advantages), without needlessly restricting Squashfs to the
compression algorithms supported by the cryptoapi.
Let's get this mainlined, and then we can work on getting your
cryptoapi patches integrated into that. That'll make me happy,
and hopefully you'll be happy too?
Thanks
Phillip
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Felix Fietkau @ 2009-12-10 22:20 UTC (permalink / raw)
To: Phillip Lougher
Cc: Geert Uytterhoeven, dedekind1, Andrew Morton, linux-embedded,
linux-fsdevel, linux-kernel, phillip.lougher, Tim Bird,
Geert Uytterhoeven
In-Reply-To: <4B21656B.1090607@lougher.demon.co.uk>
On 2009-12-10 10:17 PM, Phillip Lougher wrote:
> Geert Uytterhoeven wrote:
>>
>>> Yes I did consider using the cryptoapi, but this doesn't have support for
>>> lzma in mainline.
>>
>> IIRC, Felix Fietkau added support for that for OpenWRT...
>>
>
> Yes, but it isn't in mainline, and OpenWRT don't appear to have tried to submit
> it. IMHO the major problem with their patch is it uses a second private copy
> of lzma, rather than being a wrapper around the pre-existing lzma implementation.
> I don't think it's going to be easy to get a second lzma implementation accepted
> into mainline, when it took so long to get one accepted. I have nothing against
> the cryptoapi, but it doesn't seem likely to be getting lzma support anytime soon.
>
> As I previously said my aim is to use the pre-existing lzma implementation. Unless
> it is stupendously bad (which it isn't), that seems to be the quickest, easiest and
> best way to get lzma support into Squashfs.
The main reason why my implementation couldn't be done as a simple
wrapper around the existing implementation is that it reuses the
caller's split output buffers instead wasting precious RAM by allocating
a contiguous buffer big enough to cover the entire block.
Especially at bigger block sizes (which provide better compression),
this probably has noticeable effects on tiny embedded systems with
little RAM.
- Felix
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Phillip Lougher @ 2009-12-10 21:17 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: dedekind1, Andrew Morton, linux-embedded, linux-fsdevel,
linux-kernel, phillip.lougher, Tim Bird, Geert Uytterhoeven,
Felix Fietkau
In-Reply-To: <10f740e80912100201l6d83588evff46e95ec96d0aed@mail.gmail.com>
Geert Uytterhoeven wrote:
>
>> Yes I did consider using the cryptoapi, but this doesn't have support for
>> lzma in mainline.
>
> IIRC, Felix Fietkau added support for that for OpenWRT...
>
Yes, but it isn't in mainline, and OpenWRT don't appear to have tried to submit
it. IMHO the major problem with their patch is it uses a second private copy
of lzma, rather than being a wrapper around the pre-existing lzma implementation.
I don't think it's going to be easy to get a second lzma implementation accepted
into mainline, when it took so long to get one accepted. I have nothing against
the cryptoapi, but it doesn't seem likely to be getting lzma support anytime soon.
As I previously said my aim is to use the pre-existing lzma implementation. Unless
it is stupendously bad (which it isn't), that seems to be the quickest, easiest and
best way to get lzma support into Squashfs.
Phillip
^ permalink raw reply
* Squashfs performance (3.3 vs 4.0 in mainline)
From: Simon Kagstrom @ 2009-12-10 14:02 UTC (permalink / raw)
To: linux-embedded; +Cc: Phillip Lougher
Hi!
I'm running 2.6.31 on a slow ARM platform and using squashfs 4.0 as
shipped with the kernel. It's a Sharp LH79524, so the architecture
itself isn't mainlined yet.
Squashfs works well, but I've noted a performance regression compared
to the out-of-tree squashfs 3.3 which we ran on 2.6.23. The new kernel
is faster until the squashfs root filesystem is mounted, but after that
it goes downhill. Especially mounting takes a lot more time with the
new:
2.6.23 / 3.3:
[ 0.01]: [ 1.250000] RAMDISK: squashfs filesystem found at block 0
[ 0.01]: [ 1.260000] RAMDISK: Loading 2156KiB [1 disk] into ram disk... done.
[ 1.39]: [ 2.650000] VFS: Mounted root (squashfs filesystem) readonly.
[ 0.01]: [ 2.660000] Freeing init memory: 80K
2.6.31 / 4.0:
[ 0.00]: [ 1.080000] RAMDISK: squashfs filesystem found at block 0
[ 0.01]: [ 1.090000] RAMDISK: Loading 2372KiB [1 disk] into ram disk... done.
[ 2.79]: [ 3.880000] VFS: Mounted root (squashfs filesystem) readonly on device 1:0.
[ 0.01]: [ 3.890000] Freeing init memory: 80K
(The first column is the time difference compared to last timestamp).
Are there any general changes which could have caused this difference?
I've tried with various mksquashfs options, and -noI makes it somewhat
faster (and larger), but not near the old version.
// Simon
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Geert Uytterhoeven @ 2009-12-10 10:01 UTC (permalink / raw)
To: Phillip Lougher
Cc: dedekind1, Andrew Morton, linux-embedded, linux-fsdevel,
linux-kernel, phillip.lougher, Tim Bird, Geert Uytterhoeven,
Felix Fietkau
In-Reply-To: <4B2042FA.8080406@lougher.demon.co.uk>
Hi Phillip,
On Thu, Dec 10, 2009 at 01:38, Phillip Lougher
<phillip@lougher.demon.co.uk> wrote:
> Artem Bityutskiy wrote:
>>
>> Did you consider using cryptoapi? UBIFS uses zlib/lzo in cryptoapi - it
>> is a very clean way.
Exactly my question, as that's why the Crypto API was extended with
support for partial (de)compression in the first place ;-)
In addition, using the Crypto API has the advantage that Squashfs will
use hardware decompression if/when available.
> Yes I did consider using the cryptoapi, but this doesn't have support for
> lzma in mainline.
IIRC, Felix Fietkau added support for that for OpenWRT...
> My aim is add lzma filesystem support to Squashfs using the existing kernel
> lzma iplementation, and to make as few changes to that as necessary.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Phillip Lougher @ 2009-12-10 0:38 UTC (permalink / raw)
To: dedekind1
Cc: Andrew Morton, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
In-Reply-To: <1260352852.19669.1253.camel@localhost>
Artem Bityutskiy wrote:
>
> Did you consider using cryptoapi? UBIFS uses zlib/lzo in cryptoapi - it
> is a very clean way.
>
Yes I did consider using the cryptoapi, but this doesn't have support for
lzma in mainline.
My aim is add lzma filesystem support to Squashfs using the existing kernel
lzma iplementation, and to make as few changes to that as necessary.
Phillip
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Artem Bityutskiy @ 2009-12-09 10:00 UTC (permalink / raw)
To: Phillip Lougher
Cc: Andrew Morton, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
In-Reply-To: <4B1ED122.3040309@lougher.demon.co.uk>
On Tue, 2009-12-08 at 22:20 +0000, Phillip Lougher wrote:
> Andrew Morton wrote:
> > On Mon, 07 Dec 2009 02:25:08 +0000
> > Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
> >
> >> +
> >> +int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
> >> + struct buffer_head **bh, int b, int offset, int length, int srclength,
> >> + int pages)
> >
> > This isn't a very good function name. zlib_uncompress() now becomes a
> > kernel-wide identifier, but it's part of squashfs, not part of zlib.
> >
> > Maybe that gets fixed in a later patch. If so, thwap me.
> >
>
> Yes, they get fixed up in [PATCH 3/9] Squashfs: add a decompressor framework.
> That patch makes the functions static, and instead exports them via a
> suitably named decompressor ops structure.
>
> +const struct squashfs_decompressor squashfs_zlib_comp_ops = {
> + .init = zlib_init,
> + .free = zlib_free,
> + .decompress = zlib_uncompress,
> + .id = ZLIB_COMPRESSION,
> + .name = "zlib",
> + .supported = 1
> +};
>
> I split the patches up to make them easier to review. The first two patches
> move the zlib code out to a separate file (ready for adding the framework).
> The third patch adds the framework. At the time of the second patch, however,
> to not break compilation, the functions have to be global. In hindsight
> I should have made named the functions squashfs_xxx, and removed the squashfs_
> when they were made static in the third patch.
Did you consider using cryptoapi? UBIFS uses zlib/lzo in cryptoapi - it
is a very clean way.
--
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Phillip Lougher @ 2009-12-08 22:20 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-embedded, linux-fsdevel, linux-kernel, phillip.lougher,
tim.bird
In-Reply-To: <20091207145705.cd30cae8.akpm@linux-foundation.org>
Andrew Morton wrote:
> On Mon, 07 Dec 2009 02:25:08 +0000
> Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
>
>> +
>> +int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
>> + struct buffer_head **bh, int b, int offset, int length, int srclength,
>> + int pages)
>
> This isn't a very good function name. zlib_uncompress() now becomes a
> kernel-wide identifier, but it's part of squashfs, not part of zlib.
>
> Maybe that gets fixed in a later patch. If so, thwap me.
>
Yes, they get fixed up in [PATCH 3/9] Squashfs: add a decompressor framework.
That patch makes the functions static, and instead exports them via a
suitably named decompressor ops structure.
+const struct squashfs_decompressor squashfs_zlib_comp_ops = {
+ .init = zlib_init,
+ .free = zlib_free,
+ .decompress = zlib_uncompress,
+ .id = ZLIB_COMPRESSION,
+ .name = "zlib",
+ .supported = 1
+};
I split the patches up to make them easier to review. The first two patches
move the zlib code out to a separate file (ready for adding the framework).
The third patch adds the framework. At the time of the second patch, however,
to not break compilation, the functions have to be global. In hindsight
I should have made named the functions squashfs_xxx, and removed the squashfs_
when they were made static in the third patch.
Thanks
Phillip
^ permalink raw reply
* Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
From: Phillip Lougher @ 2009-12-08 17:59 UTC (permalink / raw)
To: Peter Korsgaard
Cc: linux-embedded, linux-fsdevel, linux-kernel, phillip.lougher,
tim.bird
In-Reply-To: <87tyw3p3vu.fsf@macbook.be.48ers.dk>
Peter Korsgaard wrote:
>>>>>> "root" == root <root@lougher.demon.co.uk> writes:
>
> root> Hi,
> root> The following patches add LZMA decompression support to Squashfs, using the
> root> in-kernel LZMA decompression library.
>
> root> The patches also add a decompression framework to Squashfs.
> root> This allows LZMA decompression to be added cleanly, and it allows
> root> additional decompressors to be easily added in the future.
>
> root> To enable the in-kernel LZMA decompression code to be used by Squashfs,
> root> there are two patches to the lzma code itself: one to make lzma available
> root> to non-init code, and one to make lzma reentrant. These are obviously
> root> not restricted to Squashfs, but are needed by any non-init code that
> root> may wish to use lzma compression.
>
> Nice, do you also have an mksquashfs with lzma support?
>
Yes, in Squashfs CVS (http://sourceforge.net/projects/squashfs/develop).
You'll need to edit the squashfs-tools Makefile to enable LZMA support.
The comments in the Makefile should, hopefully, explain how to
build LZMA support into Mksquashfs/Unsquashfs.
Once built-in, LZMA support can be specified using the -comp lzma option, i.e.
mksquashfs dir dir.img -comp lzma
Unsquashfs doesn't need any extra options, it automatically detects which
compression has been used.
You can tell which compression algorithms Mksquashfs/Unsquashfs support by
just typing the command on its own (i.e. mksquashfs, or unsquashfs). The
(de-)compressors available are displayed at the end of the output.
Phillip
^ permalink raw reply
* Re: [POWER] battery calibration parameters from sysfs
From: Mark Brown @ 2009-12-08 10:28 UTC (permalink / raw)
To: Brian Swetland
Cc: Rebecca Schultz Zavin, Mike Lockwood, Linus Walleij, cbou, dwmw2,
LKML, linux-embedded
In-Reply-To: <a55d774e0912072127u33cebd65xbfe341160c314a2e@mail.gmail.com>
On Mon, Dec 07, 2009 at 09:27:20PM -0800, Brian Swetland wrote:
> On Mon, Dec 7, 2009 at 8:56 AM, Mark Brown
> > I don't think the existing Android devices are much of an issue here,
> > it's not as though end users have the ability modify the firmware on
> > them (modulo the fairly small number of ADP devices). This is an issue
> > for people producing new devices who are already going to be doing some
> > development to get the kernel up and running on their hardware.
> My preference, from an Android point of view, would be to make sure we
> have some reasonable userspace abstraction and provide a default
> implementation that will do the right thing with an intelligent
> battery driver (our usual design for inhouse work). Then alternative
> implementations can be plugged in, should that not be sufficient.
It proably makes sense to have some of the performance tracking in there
by default - while there are some systems out there which are able to do
high quality battery tracking there's several previous generations of
monitoring technology out there which have substantial accuracy issues
(especially in the face of removable battery packs) but which will
report information to userspace. This makes it unwise to rely to
totally on the information you're getting from the kernel - the
smartness of a smart battery driver can vary greatly.
^ permalink raw reply
* Re: [POWER] battery calibration parameters from sysfs
From: Brian Swetland @ 2009-12-08 5:27 UTC (permalink / raw)
To: Mark Brown, Rebecca Schultz Zavin, Mike Lockwood
Cc: Linus Walleij, cbou, dwmw2, LKML, linux-embedded
In-Reply-To: <20091207165628.GA24981@rakim.wolfsonmicro.main>
On Mon, Dec 7, 2009 at 8:56 AM, Mark Brown
<broonie@opensource.wolfsonmicro.com> wrote:
>
>> One of the things we're facing is Android, which has
>> its userspace in plain Java JNI at the end of this link:
>> http://android.git.kernel.org/?p=platform/frameworks/base.git;a=blob;f=s
>> ervices/jni/com_android_server_BatteryService.cpp;h=8e7cadc6b680fc420d34
>> 1faa094c71922946fdab;hb=HEAD
>
> Something word wrapped that link but I think I found the same code.
>
>> If you browse down to line 275 you can see it parse the sysfs
>> attribute "capacity", then this propagates up to the battery
>> status indicator on *all* Android phones out there. So if
>> you want to run Android unmodified, this is what you need to
>> provide. They are effectively using the power sysfs as
>> their hardware abstraction layer in this case.
>
> Oh dear. Using the power sysfs as the hardware abstraction seems
> perfectly reasonable but assuming that a given battery driver is going
> to have this level of information doesn't match up with an awful lot of
> actual charger hardware. My main concern here is that battery
> performance monitoring has no pressing need to be in kernel and that
> pushing it into the kernel creates a barrier to implementing more
> advanced schemes in userspace, which is especially serious given how
> involved this needs to be in order to be accurate.
We should tidy up the userspace side to have better abstraction for
this. On a number of devices we do obtain all the information
necessary from the kernel driver (or, in some cases battery monitoring
on the baseband side that the kernel driver chats with). On other
devices we don't (Motorola Droid, for example uses a userspace battery
daemon).
>> Note: I'm not claiming that Android is doing it right or that
>> we can't modify this code or so, it's just that this is the way
>> a few million Android devices out there are actually doing it.
>
> I don't think the existing Android devices are much of an issue here,
> it's not as though end users have the ability modify the firmware on
> them (modulo the fairly small number of ADP devices). This is an issue
> for people producing new devices who are already going to be doing some
> development to get the kernel up and running on their hardware.
My preference, from an Android point of view, would be to make sure we
have some reasonable userspace abstraction and provide a default
implementation that will do the right thing with an intelligent
battery driver (our usual design for inhouse work). Then alternative
implementations can be plugged in, should that not be sufficient.
Brian
^ permalink raw reply
* Re: [PATCH 2/9] Squashfs: Factor out remaining zlib dependencies into separate wrapper file
From: Andrew Morton @ 2009-12-07 22:58 UTC (permalink / raw)
To: Phillip Lougher
Cc: linux-embedded, linux-fsdevel, linux-kernel, phillip.lougher,
tim.bird
In-Reply-To: <E1NHTHg-0001sR-PM@dylan.lougher.demon.co.uk>
On Mon, 07 Dec 2009 02:25:08 +0000
Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
> +void *zlib_init()
> ...
> +void zlib_free(void *strm)
More poorly-named globals.
^ permalink raw reply
* Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
From: Andrew Morton @ 2009-12-07 22:57 UTC (permalink / raw)
To: Phillip Lougher
Cc: linux-embedded, linux-fsdevel, linux-kernel, phillip.lougher,
tim.bird
In-Reply-To: <E1NHTHg-0001sM-NQ@dylan.lougher.demon.co.uk>
On Mon, 07 Dec 2009 02:25:08 +0000
Phillip Lougher <phillip@lougher.demon.co.uk> wrote:
> +++ b/fs/squashfs/zlib_wrapper.c
> @@ -0,0 +1,109 @@
> +/*
> + * Squashfs - a compressed read only filesystem for Linux
> + *
> + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
> + * Phillip Lougher <phillip@lougher.demon.co.uk>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2,
> + * or (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
> + *
> + * zlib_wrapper.c
> + */
> +
> +
> +#include <linux/mutex.h>
> +#include <linux/buffer_head.h>
> +#include <linux/zlib.h>
> +
> +#include "squashfs_fs.h"
> +#include "squashfs_fs_sb.h"
> +#include "squashfs_fs_i.h"
> +#include "squashfs.h"
> +
> +int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
> + struct buffer_head **bh, int b, int offset, int length, int srclength,
> + int pages)
This isn't a very good function name. zlib_uncompress() now becomes a
kernel-wide identifier, but it's part of squashfs, not part of zlib.
Maybe that gets fixed in a later patch. If so, thwap me.
^ permalink raw reply
* Re: [POWER] battery calibration parameters from sysfs
From: Mark Brown @ 2009-12-07 16:56 UTC (permalink / raw)
To: Linus Walleij; +Cc: cbou, dwmw2, LKML, linux-embedded, Brian Swetland
In-Reply-To: <A6D19A13FE030A409EC4362C172E091F0E0A1FA1@eseldmw101.eemea.ericsson.se>
On Mon, Dec 07, 2009 at 03:07:15PM +0100, Linus Walleij wrote:
> [Mark wrote]
> > These files should only be present if we have data for them.
> > Userspace can't be reliant on them at present since relatively few
> > systems seem to implement them, for example none of my laptops
> > have time_to, energy_ or capacity attributes.
> Well, yeah, we're not exactly in the laptop business.
> (But my HP laptop nc2400 has the "charge_*" attributes,
> 4990000 uAh in "charge_now" and this corresponds to
> what is shown in g-p-m converted over to Wh, but I don't
> know exactly where it's getting it from...)
Looking at what PCs are doing is often a useful guide as to what the
desktop userspace stacks will be trying to use - if you can't get the
information from a Dell or whatever then the chances are they'll cope
fine without it. Of course, laptop batteries tend to be rather more
fancy in terms of their embedded controllers if nothing else.
> One of the things we're facing is Android, which has
> its userspace in plain Java JNI at the end of this link:
> http://android.git.kernel.org/?p=platform/frameworks/base.git;a=blob;f=s
> ervices/jni/com_android_server_BatteryService.cpp;h=8e7cadc6b680fc420d34
> 1faa094c71922946fdab;hb=HEAD
Something word wrapped that link but I think I found the same code.
> If you browse down to line 275 you can see it parse the sysfs
> attribute "capacity", then this propagates up to the battery
> status indicator on *all* Android phones out there. So if
> you want to run Android unmodified, this is what you need to
> provide. They are effectively using the power sysfs as
> their hardware abstraction layer in this case.
Oh dear. Using the power sysfs as the hardware abstraction seems
perfectly reasonable but assuming that a given battery driver is going
to have this level of information doesn't match up with an awful lot of
actual charger hardware. My main concern here is that battery
performance monitoring has no pressing need to be in kernel and that
pushing it into the kernel creates a barrier to implementing more
advanced schemes in userspace, which is especially serious given how
involved this needs to be in order to be accurate.
I'm not sure how familiar you are with the issues surrounding trying to
do a voltage to charge mapping for a battery but it's much more complex
than a simple table if you want to get it accurate. There's a lot of
dependence on particular operating conditions and things do change as
the batteries age. There are systems out there that do the work
required to gather the information in hardware and it's definitely good
to report the information from them but that doesn't mean it's a good
idea to try to synthesise the information for other systems.
We should at least explore the possibility of getting Android to improve
their userspace code rather than put this sort of code into the kernel.
I've CCed Brian in in case he can comment on the possibility of doing
that.
> Note: I'm not claiming that Android is doing it right or that
> we can't modify this code or so, it's just that this is the way
> a few million Android devices out there are actually doing it.
I don't think the existing Android devices are much of an issue here,
it's not as though end users have the ability modify the firmware on
them (modulo the fairly small number of ADP devices). This is an issue
for people producing new devices who are already going to be doing some
development to get the kernel up and running on their hardware.
^ permalink raw reply
* RE: [POWER] battery calibration parameters from sysfs
From: Linus Walleij @ 2009-12-07 14:07 UTC (permalink / raw)
To: Mark Brown; +Cc: cbou, dwmw2, LKML, linux-embedded
In-Reply-To: <20091207114825.GA26965@rakim.wolfsonmicro.main>
Thanks for all the great attention Mark!
[Mark wrote]
> On Sat, Dec 05, 2009 at 02:08:11PM +0100, Linus Walleij wrote:
> > (...)
> > That's an easy solution of course, but then the sysfs files
> > specified by the power subsystem, i.e. all "charge_*",
> > "energy_*", "capacity" and "time_to_*" loose their meaning
> > and must be ignored by userspace.
>
> These files should only be present if we have data for them.
> Userspace can't be reliant on them at present since relatively few
> systems seem to implement them, for example none of my laptops
> have time_to, energy_ or capacity attributes.
Well, yeah, we're not exactly in the laptop business.
(But my HP laptop nc2400 has the "charge_*" attributes,
4990000 uAh in "charge_now" and this corresponds to
what is shown in g-p-m converted over to Wh, but I don't
know exactly where it's getting it from...)
One of the things we're facing is Android, which has
its userspace in plain Java JNI at the end of this link:
http://android.git.kernel.org/?p=platform/frameworks/base.git;a=blob;f=s
ervices/jni/com_android_server_BatteryService.cpp;h=8e7cadc6b680fc420d34
1faa094c71922946fdab;hb=HEAD
If you browse down to line 275 you can see it parse the sysfs
attribute "capacity", then this propagates up to the battery
status indicator on *all* Android phones out there. So if
you want to run Android unmodified, this is what you need to
provide. They are effectively using the power sysfs as
their hardware abstraction layer in this case.
Note: I'm not claiming that Android is doing it right or that
we can't modify this code or so, it's just that this is the way
a few million Android devices out there are actually doing it.
> > We have other odd code. Actually we have full software-
> > controlled CC/CV charging in our driver, and that
> > would *definately* go in such a library if it was
> > to end up in kernelspace. We have actually
> > pushed that to userspace, while I still tend to think
> > that the kernel (with right parameters) should be able
> > to charge a battery. But, well.
>
> As was previously discussed (in another thread) I do think there needs
> to be at least some in kernel part to charger code like this
> in order to
> ensure that we're robust against userspace failure and cope well with
> suspend and resume. There's the potential for serious hardware damage
> if the battery is mistreated.
You're right of course. In our case we have a hardware
watchdog protecting against overcharging etc, and charging will
halt when the watchdog is not periodically fed. So actually
we're on the safe side here. (I think!)
> There's other kernel filesystems that might be more
> approprite for this
> sort of stuff, trying to fit this into sysfs really does feel like far
> too much pain to be right.
You're right GKH already pointed me to configfs so that's
where it's going to sit if we do this thing.
We'd better write up some code to show how this would
work instead so I'll be back with that sooner or later.
Yours,
Linus Walleij
^ permalink raw reply
* Re: [POWER] battery calibration parameters from sysfs
From: Mark Brown @ 2009-12-07 11:48 UTC (permalink / raw)
To: Linus Walleij; +Cc: cbou, dwmw2, LKML, linux-embedded
In-Reply-To: <A6D19A13FE030A409EC4362C172E091F0E071EB9@eseldmw101.eemea.ericsson.se>
On Sat, Dec 05, 2009 at 02:08:11PM +0100, Linus Walleij wrote:
> [Mark Brown]
> > Isn't the standard thing here to handle this voltage to
> > capacity mapping in userspace if we're just extrapolating
> > from experimental results?
> That's an easy solution of course, but then the sysfs files
> specified by the power subsystem, i.e. all "charge_*",
> "energy_*", "capacity" and "time_to_*" loose their meaning
> and must be ignored by userspace.
These files should only be present if we have data for them. Userspace
can't be reliant on them at present since relatively few systems seem to
implement them, for example none of my laptops have time_to, energy_ or
capacity attributes.
> Also this was just an example, we have similar calibration
> for the temperature sensor, and thus the "temp" sysfs file
> also loose its meaning.
Sure, and with temperature sensors tables of design based information
might be more appropriate since it's not possible to gain experimenal
data from the running system in the way we can for batteries.
> Since there is a plethora of userspace apps that just
> interface these files directly (gnome-power-manager and
> the Android stack come to mind) all these will have to
> be patches to accept a calibrated value from somewhere
> else if we shall use them with our hardware.
At least GNOME seems to already be collecting historical statistics on
the battery performance. I believe it's factoring the results into the
values reported through the UI but I'd need to check.
> > Actually, one further thing here - if this functionality
> > is implemented in kernel then shouldn't it be a generic
> > feature rather than part of the driver? The idea of
> Surely, we'd be happy to do it that way if desired.
> What about drivers/power/battery_lib.c?
If we're going to do it at all. Ideally it'd just kick in automatically
when data isn't available so possibly it ought to be core code rather
than a library used by drivers.
> (And getting algorithms in place for gradually
> adjusting the capacity levels as compared to factory
> settings for PC batteries would perhaps end up in the
> same place then.)
I'm still not convinced that it's a good idea to put this into the
kernel.
So far as I can see the main case for doing it in kernel is existing
userspace - is there any other motivation I've overlooked? Like I say
I'd be somewhat surprised if userspace were relying on this data given
that we're not currently generating it and it's going to need at least
some userspace work to save and restore the data. There's also policy
issues about how often you do the monitoring and so on.
> We have other odd code. Actually we have full software-
> controlled CC/CV charging in our driver, and that
> would *definately* go in such a library if it was
> to end up in kernelspace. We have actually
> pushed that to userspace, while I still tend to think
> that the kernel (with right parameters) should be able
> to charge a battery. But, well.
As was previously discussed (in another thread) I do think there needs
to be at least some in kernel part to charger code like this in order to
ensure that we're robust against userspace failure and cope well with
suspend and resume. There's the potential for serious hardware damage
if the battery is mistreated.
> As for the calibration format, after reading up on the
> latest sysfs doc I saw:
There's other kernel filesystems that might be more approprite for this
sort of stuff, trying to fit this into sysfs really does feel like far
too much pain to be right.
^ permalink raw reply
* Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
From: Peter Korsgaard @ 2009-12-07 8:47 UTC (permalink / raw)
To: phillip
Cc: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
In-Reply-To: <E1NHYun-000287-Vq@dylan.lougher.demon.co.uk>
>>>>> "root" == root <root@lougher.demon.co.uk> writes:
root> Hi,
root> The following patches add LZMA decompression support to Squashfs, using the
root> in-kernel LZMA decompression library.
root> The patches also add a decompression framework to Squashfs.
root> This allows LZMA decompression to be added cleanly, and it allows
root> additional decompressors to be easily added in the future.
root> To enable the in-kernel LZMA decompression code to be used by Squashfs,
root> there are two patches to the lzma code itself: one to make lzma available
root> to non-init code, and one to make lzma reentrant. These are obviously
root> not restricted to Squashfs, but are needed by any non-init code that
root> may wish to use lzma compression.
Nice, do you also have an mksquashfs with lzma support?
--
Bye, Peter Korsgaard
^ permalink raw reply
* [PATCH 4/9] Squashfs: add decompressor entries for lzma and lzo
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Add knowledge of lzma/lzo compression formats to the decompressor
framework. For now these are added as unsupported. Without
these entries lzma/lzo compressed filesystems will be flagged as
having unknown compression which is undesirable.
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/decompressor.c | 10 ++++++++++
fs/squashfs/squashfs_fs.h | 4 +++-
2 files changed, 13 insertions(+), 1 deletions(-)
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 0072ccd..157478d 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -36,12 +36,22 @@
* Squashfs, allowing multiple decompressors to be easily supported
*/
+static const struct squashfs_decompressor squashfs_lzma_unsupported_comp_ops = {
+ NULL, NULL, NULL, LZMA_COMPRESSION, "lzma", 0
+};
+
+static const struct squashfs_decompressor squashfs_lzo_unsupported_comp_ops = {
+ NULL, NULL, NULL, LZO_COMPRESSION, "lzo", 0
+};
+
static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
NULL, NULL, NULL, 0, "unknown", 0
};
static const struct squashfs_decompressor *decompressor[] = {
&squashfs_zlib_comp_ops,
+ &squashfs_lzma_unsupported_comp_ops,
+ &squashfs_lzo_unsupported_comp_ops,
&squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
index 283daaf..36e1604 100644
--- a/fs/squashfs/squashfs_fs.h
+++ b/fs/squashfs/squashfs_fs.h
@@ -211,7 +211,9 @@ struct meta_index {
/*
* definitions for structures on disk
*/
-#define ZLIB_COMPRESSION 1
+#define ZLIB_COMPRESSION 1
+#define LZMA_COMPRESSION 2
+#define LZO_COMPRESSION 3
struct squashfs_super_block {
__le32 s_magic;
--
1.6.3.3
^ permalink raw reply related
* [PATCH 5/9] Squashfs: make decompressor init function pass superblock info
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/decompressor.h | 4 ++--
fs/squashfs/zlib_wrapper.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
index 778760c..7425f80 100644
--- a/fs/squashfs/decompressor.h
+++ b/fs/squashfs/decompressor.h
@@ -24,7 +24,7 @@
*/
struct squashfs_decompressor {
- void *(*init)(void);
+ void *(*init)(struct squashfs_sb_info *);
void (*free)(void *);
int (*decompress)(struct squashfs_sb_info *, void **,
struct buffer_head **, int, int, int, int, int);
@@ -35,7 +35,7 @@ struct squashfs_decompressor {
static inline void *squashfs_decompressor_init(struct squashfs_sb_info *msblk)
{
- return msblk->decompressor->init();
+ return msblk->decompressor->init(msblk);
}
static inline void squashfs_decompressor_free(struct squashfs_sb_info *msblk,
diff --git a/fs/squashfs/zlib_wrapper.c b/fs/squashfs/zlib_wrapper.c
index 381768c..4dd70e0 100644
--- a/fs/squashfs/zlib_wrapper.c
+++ b/fs/squashfs/zlib_wrapper.c
@@ -32,7 +32,7 @@
#include "squashfs.h"
#include "decompressor.h"
-static void *zlib_init(void)
+static void *zlib_init(struct squashfs_sb_info *dummy)
{
z_stream *stream = kmalloc(sizeof(z_stream), GFP_KERNEL);
if (stream == NULL)
--
1.6.3.3
^ permalink raw reply related
* [PATCH 2/9] Squashfs: Factor out remaining zlib dependencies into separate wrapper file
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Move zlib buffer init/destroy code into separate wrapper file. Also
make zlib z_stream field a void * removing the need to include zlib.h
for most files.
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/block.c | 1 -
fs/squashfs/cache.c | 1 -
fs/squashfs/dir.c | 1 -
fs/squashfs/export.c | 1 -
fs/squashfs/file.c | 1 -
fs/squashfs/fragment.c | 1 -
fs/squashfs/id.c | 1 -
fs/squashfs/inode.c | 1 -
fs/squashfs/namei.c | 1 -
fs/squashfs/squashfs.h | 2 +
fs/squashfs/squashfs_fs_sb.h | 2 +-
fs/squashfs/super.c | 14 +++------
fs/squashfs/symlink.c | 1 -
fs/squashfs/zlib_wrapper.c | 56 ++++++++++++++++++++++++++++++++---------
14 files changed, 51 insertions(+), 33 deletions(-)
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index 5cd3934..baf7624 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -31,7 +31,6 @@
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/buffer_head.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
index 40c98fa..57314be 100644
--- a/fs/squashfs/cache.c
+++ b/fs/squashfs/cache.c
@@ -51,7 +51,6 @@
#include <linux/sched.h>
#include <linux/spinlock.h>
#include <linux/wait.h>
-#include <linux/zlib.h>
#include <linux/pagemap.h>
#include "squashfs_fs.h"
diff --git a/fs/squashfs/dir.c b/fs/squashfs/dir.c
index 566b0ea..12b933a 100644
--- a/fs/squashfs/dir.c
+++ b/fs/squashfs/dir.c
@@ -30,7 +30,6 @@
#include <linux/fs.h>
#include <linux/vfs.h>
#include <linux/slab.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/export.c b/fs/squashfs/export.c
index 2b1b8fe..7f93d5a 100644
--- a/fs/squashfs/export.c
+++ b/fs/squashfs/export.c
@@ -39,7 +39,6 @@
#include <linux/vfs.h>
#include <linux/dcache.h>
#include <linux/exportfs.h>
-#include <linux/zlib.h>
#include <linux/slab.h>
#include "squashfs_fs.h"
diff --git a/fs/squashfs/file.c b/fs/squashfs/file.c
index 717767d..a25c506 100644
--- a/fs/squashfs/file.c
+++ b/fs/squashfs/file.c
@@ -47,7 +47,6 @@
#include <linux/string.h>
#include <linux/pagemap.h>
#include <linux/mutex.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/fragment.c b/fs/squashfs/fragment.c
index b5a2c15..7c90bbd 100644
--- a/fs/squashfs/fragment.c
+++ b/fs/squashfs/fragment.c
@@ -36,7 +36,6 @@
#include <linux/fs.h>
#include <linux/vfs.h>
#include <linux/slab.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/id.c b/fs/squashfs/id.c
index 3795b83..b7f64bc 100644
--- a/fs/squashfs/id.c
+++ b/fs/squashfs/id.c
@@ -34,7 +34,6 @@
#include <linux/fs.h>
#include <linux/vfs.h>
#include <linux/slab.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/inode.c b/fs/squashfs/inode.c
index 9101dbd..49daaf6 100644
--- a/fs/squashfs/inode.c
+++ b/fs/squashfs/inode.c
@@ -40,7 +40,6 @@
#include <linux/fs.h>
#include <linux/vfs.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/namei.c b/fs/squashfs/namei.c
index 9e39865..5266bd8 100644
--- a/fs/squashfs/namei.c
+++ b/fs/squashfs/namei.c
@@ -57,7 +57,6 @@
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/dcache.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index 988bdce..b3eaf87 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -71,6 +71,8 @@ extern struct inode *squashfs_iget(struct super_block *, long long,
extern int squashfs_read_inode(struct inode *, long long);
/* zlib_wrapper.c */
+extern void *zlib_init(void);
+extern void zlib_free(void *);
extern int zlib_uncompress(struct squashfs_sb_info *, void **,
struct buffer_head **, int, int, int, int, int);
diff --git a/fs/squashfs/squashfs_fs_sb.h b/fs/squashfs/squashfs_fs_sb.h
index c8c6561..23a67fa 100644
--- a/fs/squashfs/squashfs_fs_sb.h
+++ b/fs/squashfs/squashfs_fs_sb.h
@@ -64,7 +64,7 @@ struct squashfs_sb_info {
struct mutex read_data_mutex;
struct mutex meta_index_mutex;
struct meta_index *meta_index;
- z_stream stream;
+ void *stream;
__le64 *inode_lookup_table;
u64 inode_table;
u64 directory_table;
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
index 6c197ef..6c3429b 100644
--- a/fs/squashfs/super.c
+++ b/fs/squashfs/super.c
@@ -35,7 +35,6 @@
#include <linux/pagemap.h>
#include <linux/init.h>
#include <linux/module.h>
-#include <linux/zlib.h>
#include <linux/magic.h>
#include "squashfs_fs.h"
@@ -87,12 +86,9 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
}
msblk = sb->s_fs_info;
- msblk->stream.workspace = kmalloc(zlib_inflate_workspacesize(),
- GFP_KERNEL);
- if (msblk->stream.workspace == NULL) {
- ERROR("Failed to allocate zlib workspace\n");
+ msblk->stream = zlib_init();
+ if (msblk->stream == NULL)
goto failure;
- }
sblk = kzalloc(sizeof(*sblk), GFP_KERNEL);
if (sblk == NULL) {
@@ -292,17 +288,17 @@ failed_mount:
squashfs_cache_delete(msblk->block_cache);
squashfs_cache_delete(msblk->fragment_cache);
squashfs_cache_delete(msblk->read_page);
+ zlib_free(msblk->stream);
kfree(msblk->inode_lookup_table);
kfree(msblk->fragment_index);
kfree(msblk->id_table);
- kfree(msblk->stream.workspace);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
kfree(sblk);
return err;
failure:
- kfree(msblk->stream.workspace);
+ zlib_free(msblk->stream);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
return -ENOMEM;
@@ -346,10 +342,10 @@ static void squashfs_put_super(struct super_block *sb)
squashfs_cache_delete(sbi->block_cache);
squashfs_cache_delete(sbi->fragment_cache);
squashfs_cache_delete(sbi->read_page);
+ zlib_free(sbi->stream);
kfree(sbi->id_table);
kfree(sbi->fragment_index);
kfree(sbi->meta_index);
- kfree(sbi->stream.workspace);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
}
diff --git a/fs/squashfs/symlink.c b/fs/squashfs/symlink.c
index 83d8788..e80be20 100644
--- a/fs/squashfs/symlink.c
+++ b/fs/squashfs/symlink.c
@@ -36,7 +36,6 @@
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/pagemap.h>
-#include <linux/zlib.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
diff --git a/fs/squashfs/zlib_wrapper.c b/fs/squashfs/zlib_wrapper.c
index 486a2a7..8ebbbc7 100644
--- a/fs/squashfs/zlib_wrapper.c
+++ b/fs/squashfs/zlib_wrapper.c
@@ -31,21 +31,51 @@
#include "squashfs_fs_i.h"
#include "squashfs.h"
+void *zlib_init()
+{
+ z_stream *stream = kmalloc(sizeof(z_stream), GFP_KERNEL);
+ if (stream == NULL)
+ goto failed;
+ stream->workspace = kmalloc(zlib_inflate_workspacesize(),
+ GFP_KERNEL);
+ if (stream->workspace == NULL)
+ goto failed;
+
+ return stream;
+
+failed:
+ ERROR("Failed to allocate zlib workspace\n");
+ kfree(stream);
+ return NULL;
+}
+
+
+void zlib_free(void *strm)
+{
+ z_stream *stream = strm;
+
+ if (stream)
+ kfree(stream->workspace);
+ kfree(stream);
+}
+
+
int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
struct buffer_head **bh, int b, int offset, int length, int srclength,
int pages)
{
int zlib_err = 0, zlib_init = 0;
int avail, bytes, k = 0, page = 0;
+ z_stream *stream = msblk->stream;
mutex_lock(&msblk->read_data_mutex);
- msblk->stream.avail_out = 0;
- msblk->stream.avail_in = 0;
+ stream->avail_out = 0;
+ stream->avail_in = 0;
bytes = length;
do {
- if (msblk->stream.avail_in == 0 && k < b) {
+ if (stream->avail_in == 0 && k < b) {
avail = min(bytes, msblk->devblksize - offset);
bytes -= avail;
wait_on_buffer(bh[k]);
@@ -58,18 +88,18 @@ int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
continue;
}
- msblk->stream.next_in = bh[k]->b_data + offset;
- msblk->stream.avail_in = avail;
+ stream->next_in = bh[k]->b_data + offset;
+ stream->avail_in = avail;
offset = 0;
}
- if (msblk->stream.avail_out == 0 && page < pages) {
- msblk->stream.next_out = buffer[page++];
- msblk->stream.avail_out = PAGE_CACHE_SIZE;
+ if (stream->avail_out == 0 && page < pages) {
+ stream->next_out = buffer[page++];
+ stream->avail_out = PAGE_CACHE_SIZE;
}
if (!zlib_init) {
- zlib_err = zlib_inflateInit(&msblk->stream);
+ zlib_err = zlib_inflateInit(stream);
if (zlib_err != Z_OK) {
ERROR("zlib_inflateInit returned unexpected "
"result 0x%x, srclength %d\n",
@@ -79,9 +109,9 @@ int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
zlib_init = 1;
}
- zlib_err = zlib_inflate(&msblk->stream, Z_SYNC_FLUSH);
+ zlib_err = zlib_inflate(stream, Z_SYNC_FLUSH);
- if (msblk->stream.avail_in == 0 && k < b)
+ if (stream->avail_in == 0 && k < b)
put_bh(bh[k++]);
} while (zlib_err == Z_OK);
@@ -90,14 +120,14 @@ int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
goto release_mutex;
}
- zlib_err = zlib_inflateEnd(&msblk->stream);
+ zlib_err = zlib_inflateEnd(stream);
if (zlib_err != Z_OK) {
ERROR("zlib_inflate error, data probably corrupt\n");
goto release_mutex;
}
mutex_unlock(&msblk->read_data_mutex);
- return msblk->stream.total_out;
+ return stream->total_out;
release_mutex:
mutex_unlock(&msblk->read_data_mutex);
--
1.6.3.3
^ permalink raw reply related
* [PATCH 6/9] Squashfs: add support for LZMA compressed filesystems
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/Kconfig | 5 ++
fs/squashfs/Makefile | 1 +
fs/squashfs/decompressor.c | 4 +
fs/squashfs/lzma_wrapper.c | 151 ++++++++++++++++++++++++++++++++++++++++++++
fs/squashfs/squashfs.h | 3 +
5 files changed, 164 insertions(+), 0 deletions(-)
create mode 100644 fs/squashfs/lzma_wrapper.c
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index 25a00d1..0294aa2 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -26,6 +26,11 @@ config SQUASHFS
If unsure, say N.
+config SQUASHFS_LZMA
+ bool "Include support for LZMA compressed file systems"
+ depends on SQUASHFS
+ select DECOMPRESS_LZMA
+
config SQUASHFS_EMBEDDED
bool "Additional option for memory-constrained systems"
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index df8a19e..45aaefd 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -5,3 +5,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
+squashfs-$(CONFIG_SQUASHFS_LZMA) += lzma_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 157478d..0b6ad9b 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -50,7 +50,11 @@ static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
static const struct squashfs_decompressor *decompressor[] = {
&squashfs_zlib_comp_ops,
+#ifdef CONFIG_SQUASHFS_LZMA
+ &squashfs_lzma_comp_ops,
+#else
&squashfs_lzma_unsupported_comp_ops,
+#endif
&squashfs_lzo_unsupported_comp_ops,
&squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/lzma_wrapper.c b/fs/squashfs/lzma_wrapper.c
new file mode 100644
index 0000000..cef06d6
--- /dev/null
+++ b/fs/squashfs/lzma_wrapper.c
@@ -0,0 +1,151 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher <phillip@lougher.demon.co.uk>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * lzma_wrapper.c
+ */
+
+#include <asm/unaligned.h>
+#include <linux/buffer_head.h>
+#include <linux/mutex.h>
+#include <linux/vmalloc.h>
+#include <linux/decompress/unlzma.h>
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "squashfs.h"
+#include "decompressor.h"
+
+struct squashfs_lzma {
+ void *input;
+ void *output;
+};
+
+/* decompress_unlzma.c is currently non re-entrant... */
+DEFINE_MUTEX(lzma_mutex);
+
+/* decompress_unlzma.c doesn't provide any context in its callbacks... */
+static int lzma_error;
+
+static void error(char *m)
+{
+ ERROR("unlzma error: %s\n", m);
+ lzma_error = 1;
+}
+
+
+static void *lzma_init(struct squashfs_sb_info *msblk)
+{
+ struct squashfs_lzma *stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+ if (stream == NULL)
+ goto failed;
+ stream->input = vmalloc(msblk->block_size);
+ if (stream->input == NULL)
+ goto failed;
+ stream->output = vmalloc(msblk->block_size);
+ if (stream->output == NULL)
+ goto failed2;
+
+ return stream;
+
+failed2:
+ vfree(stream->input);
+failed:
+ ERROR("failed to allocate lzma workspace\n");
+ kfree(stream);
+ return NULL;
+}
+
+
+static void lzma_free(void *strm)
+{
+ struct squashfs_lzma *stream = strm;
+
+ if (stream) {
+ vfree(stream->input);
+ vfree(stream->output);
+ }
+ kfree(stream);
+}
+
+
+static int lzma_uncompress(struct squashfs_sb_info *msblk, void **buffer,
+ struct buffer_head **bh, int b, int offset, int length, int srclength,
+ int pages)
+{
+ struct squashfs_lzma *stream = msblk->stream;
+ void *buff = stream->input;
+ int avail, i, bytes = length, res;
+
+ mutex_lock(&lzma_mutex);
+
+ for (i = 0; i < b; i++) {
+ wait_on_buffer(bh[i]);
+ if (!buffer_uptodate(bh[i]))
+ goto block_release;
+
+ avail = min(bytes, msblk->devblksize - offset);
+ memcpy(buff, bh[i]->b_data + offset, avail);
+ buff += avail;
+ bytes -= avail;
+ offset = 0;
+ put_bh(bh[i]);
+ }
+
+ lzma_error = 0;
+ res = unlzma(stream->input, length, NULL, NULL, stream->output, NULL,
+ error);
+ if (res || lzma_error)
+ goto failed;
+
+ /* uncompressed size is stored in the LZMA header (5 byte offset) */
+ res = bytes = get_unaligned_le32(stream->input + 5);
+ for (i = 0, buff = stream->output; bytes && i < pages; i++) {
+ avail = min_t(int, bytes, PAGE_CACHE_SIZE);
+ memcpy(buffer[i], buff, avail);
+ buff += avail;
+ bytes -= avail;
+ }
+ if (bytes)
+ goto failed;
+
+ mutex_unlock(&lzma_mutex);
+ return res;
+
+block_release:
+ for (; i < b; i++)
+ put_bh(bh[i]);
+
+failed:
+ mutex_unlock(&lzma_mutex);
+
+ ERROR("lzma decompression failed, data probably corrupt\n");
+ return -EIO;
+}
+
+const struct squashfs_decompressor squashfs_lzma_comp_ops = {
+ .init = lzma_init,
+ .free = lzma_free,
+ .decompress = lzma_uncompress,
+ .id = LZMA_COMPRESSION,
+ .name = "lzma",
+ .supported = 1
+};
+
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index fe2587a..d094886 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -94,3 +94,6 @@ extern const struct address_space_operations squashfs_symlink_aops;
/* zlib_wrapper.c */
extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
+
+/* lzma wrapper.c */
+extern const struct squashfs_decompressor squashfs_lzma_comp_ops;
--
1.6.3.3
^ permalink raw reply related
* [PATCH 8/9] Squashfs: select DECOMPRESS_LZMA_NEEDED when including support for lzma
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/Kconfig | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index 0294aa2..7ec5d7e 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -30,6 +30,7 @@ config SQUASHFS_LZMA
bool "Include support for LZMA compressed file systems"
depends on SQUASHFS
select DECOMPRESS_LZMA
+ select DECOMPRESS_LZMA_NEEDED
config SQUASHFS_EMBEDDED
--
1.6.3.3
^ permalink raw reply related
* [PATCH 9/9] lzma: make lzma reentrant
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
The error function pointer used by lzma is global (file scope) which
prevents it being used concurrently. This patch removes the global
error pointer use, and instead passes it to all functions that need it.
The error function pointer is still used by bzip2 and inflate.
This patch moves the definition into the separate bzip2/inflate
header files. This prevents gcc from complaining about an
unused definition compiling lzma.
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
include/linux/decompress/bunzip2_mm.h | 1 +
include/linux/decompress/inflate_mm.h | 1 +
include/linux/decompress/mm.h | 1 -
lib/decompress_unlzma.c | 82 +++++++++++++++++----------------
4 files changed, 44 insertions(+), 41 deletions(-)
diff --git a/include/linux/decompress/bunzip2_mm.h b/include/linux/decompress/bunzip2_mm.h
index cac6fef..863efd0 100644
--- a/include/linux/decompress/bunzip2_mm.h
+++ b/include/linux/decompress/bunzip2_mm.h
@@ -7,6 +7,7 @@
#else
/* Compile for initramfs/initrd code only */
#define INIT __init
+static void(*error)(char *m);
#endif
#endif
diff --git a/include/linux/decompress/inflate_mm.h b/include/linux/decompress/inflate_mm.h
index ca4a2ae..87a742b 100644
--- a/include/linux/decompress/inflate_mm.h
+++ b/include/linux/decompress/inflate_mm.h
@@ -7,6 +7,7 @@
#else
/* Compile for initramfs/initrd code only */
#define INIT __init
+static void(*error)(char *m);
#endif
#endif
diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h
index 80f5ba4..32651e4 100644
--- a/include/linux/decompress/mm.h
+++ b/include/linux/decompress/mm.h
@@ -72,7 +72,6 @@ static void free(void *where)
#define large_malloc(a) vmalloc(a)
#define large_free(a) vfree(a)
-static void(*error)(char *m);
#define set_error_fn(x) error = x;
#define STATIC
diff --git a/lib/decompress_unlzma.c b/lib/decompress_unlzma.c
index a614b26..3e85763 100644
--- a/lib/decompress_unlzma.c
+++ b/lib/decompress_unlzma.c
@@ -89,7 +89,7 @@ static int nofill(void *buffer, unsigned int len)
}
/* Called twice: once at startup and once in rc_normalize() */
-static void INIT rc_read(struct rc *rc)
+static void INIT rc_read(struct rc *rc, void(*error)(char *x))
{
rc->buffer_size = rc->fill((char *)rc->buffer, LZMA_IOBUF_SIZE);
if (rc->buffer_size <= 0)
@@ -116,13 +116,13 @@ static inline void INIT rc_init(struct rc *rc,
rc->range = 0xFFFFFFFF;
}
-static inline void INIT rc_init_code(struct rc *rc)
+static inline void INIT rc_init_code(struct rc *rc, void(*error)(char *x))
{
int i;
for (i = 0; i < 5; i++) {
if (rc->ptr >= rc->buffer_end)
- rc_read(rc);
+ rc_read(rc, error);
rc->code = (rc->code << 8) | *rc->ptr++;
}
}
@@ -135,32 +135,33 @@ static inline void INIT rc_free(struct rc *rc)
}
/* Called twice, but one callsite is in inline'd rc_is_bit_0_helper() */
-static void INIT rc_do_normalize(struct rc *rc)
+static void INIT rc_do_normalize(struct rc *rc, void(*error)(char *x))
{
if (rc->ptr >= rc->buffer_end)
- rc_read(rc);
+ rc_read(rc, error);
rc->range <<= 8;
rc->code = (rc->code << 8) | *rc->ptr++;
}
-static inline void INIT rc_normalize(struct rc *rc)
+static inline void INIT rc_normalize(struct rc *rc, void(*error)(char *x))
{
if (rc->range < (1 << RC_TOP_BITS))
- rc_do_normalize(rc);
+ rc_do_normalize(rc, error);
}
/* Called 9 times */
/* Why rc_is_bit_0_helper exists?
*Because we want to always expose (rc->code < rc->bound) to optimizer
*/
-static inline uint32_t INIT rc_is_bit_0_helper(struct rc *rc, uint16_t *p)
+static inline uint32_t INIT rc_is_bit_0_helper(struct rc *rc, uint16_t *p,
+ void (*error)(char *x))
{
- rc_normalize(rc);
+ rc_normalize(rc, error);
rc->bound = *p * (rc->range >> RC_MODEL_TOTAL_BITS);
return rc->bound;
}
-static inline int INIT rc_is_bit_0(struct rc *rc, uint16_t *p)
+static inline int INIT rc_is_bit_0(struct rc *rc, uint16_t *p, void(*error)(char *x))
{
- uint32_t t = rc_is_bit_0_helper(rc, p);
+ uint32_t t = rc_is_bit_0_helper(rc, p, error);
return rc->code < t;
}
@@ -178,9 +179,9 @@ static inline void rc_update_bit_1(struct rc *rc, uint16_t *p)
}
/* Called 4 times in unlzma loop */
-static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int *symbol)
+static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int *symbol, void(*error)(char *x))
{
- if (rc_is_bit_0(rc, p)) {
+ if (rc_is_bit_0(rc, p, error)) {
rc_update_bit_0(rc, p);
*symbol *= 2;
return 0;
@@ -192,9 +193,9 @@ static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int *symbol)
}
/* Called once */
-static inline int INIT rc_direct_bit(struct rc *rc)
+static inline int INIT rc_direct_bit(struct rc *rc , void(*error)(char *x))
{
- rc_normalize(rc);
+ rc_normalize(rc, error);
rc->range >>= 1;
if (rc->code >= rc->range) {
rc->code -= rc->range;
@@ -205,13 +206,14 @@ static inline int INIT rc_direct_bit(struct rc *rc)
/* Called twice */
static inline void INIT
-rc_bit_tree_decode(struct rc *rc, uint16_t *p, int num_levels, int *symbol)
+rc_bit_tree_decode(struct rc *rc, uint16_t *p, int num_levels, int *symbol,
+ void(*error)(char *x))
{
int i = num_levels;
*symbol = 1;
while (i--)
- rc_get_bit(rc, p + *symbol, symbol);
+ rc_get_bit(rc, p + *symbol, symbol, error);
*symbol -= 1 << num_levels;
}
@@ -348,7 +350,8 @@ static inline void INIT copy_bytes(struct writer *wr,
static inline void INIT process_bit0(struct writer *wr, struct rc *rc,
struct cstate *cst, uint16_t *p,
int pos_state, uint16_t *prob,
- int lc, uint32_t literal_pos_mask) {
+ int lc, uint32_t literal_pos_mask,
+ void(*error)(char *x)) {
int mi = 1;
rc_update_bit_0(rc, prob);
prob = (p + LZMA_LITERAL +
@@ -366,7 +369,7 @@ static inline void INIT process_bit0(struct writer *wr, struct rc *rc,
match_byte <<= 1;
bit = match_byte & 0x100;
prob_lit = prob + 0x100 + bit + mi;
- if (rc_get_bit(rc, prob_lit, &mi)) {
+ if (rc_get_bit(rc, prob_lit, &mi, error)) {
if (!bit)
break;
} else {
@@ -377,7 +380,7 @@ static inline void INIT process_bit0(struct writer *wr, struct rc *rc,
}
while (mi < 0x100) {
uint16_t *prob_lit = prob + mi;
- rc_get_bit(rc, prob_lit, &mi);
+ rc_get_bit(rc, prob_lit, &mi, error);
}
write_byte(wr, mi);
if (cst->state < 4)
@@ -390,7 +393,8 @@ static inline void INIT process_bit0(struct writer *wr, struct rc *rc,
static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
struct cstate *cst, uint16_t *p,
- int pos_state, uint16_t *prob) {
+ int pos_state, uint16_t *prob,
+ void(*error)(char *x)) {
int offset;
uint16_t *prob_len;
int num_bits;
@@ -398,7 +402,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
rc_update_bit_1(rc, prob);
prob = p + LZMA_IS_REP + cst->state;
- if (rc_is_bit_0(rc, prob)) {
+ if (rc_is_bit_0(rc, prob, error)) {
rc_update_bit_0(rc, prob);
cst->rep3 = cst->rep2;
cst->rep2 = cst->rep1;
@@ -408,13 +412,13 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
} else {
rc_update_bit_1(rc, prob);
prob = p + LZMA_IS_REP_G0 + cst->state;
- if (rc_is_bit_0(rc, prob)) {
+ if (rc_is_bit_0(rc, prob, error)) {
rc_update_bit_0(rc, prob);
prob = (p + LZMA_IS_REP_0_LONG
+ (cst->state <<
LZMA_NUM_POS_BITS_MAX) +
pos_state);
- if (rc_is_bit_0(rc, prob)) {
+ if (rc_is_bit_0(rc, prob, error)) {
rc_update_bit_0(rc, prob);
cst->state = cst->state < LZMA_NUM_LIT_STATES ?
@@ -429,13 +433,13 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
rc_update_bit_1(rc, prob);
prob = p + LZMA_IS_REP_G1 + cst->state;
- if (rc_is_bit_0(rc, prob)) {
+ if (rc_is_bit_0(rc, prob, error)) {
rc_update_bit_0(rc, prob);
distance = cst->rep1;
} else {
rc_update_bit_1(rc, prob);
prob = p + LZMA_IS_REP_G2 + cst->state;
- if (rc_is_bit_0(rc, prob)) {
+ if (rc_is_bit_0(rc, prob, error)) {
rc_update_bit_0(rc, prob);
distance = cst->rep2;
} else {
@@ -453,7 +457,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
}
prob_len = prob + LZMA_LEN_CHOICE;
- if (rc_is_bit_0(rc, prob_len)) {
+ if (rc_is_bit_0(rc, prob_len, error)) {
rc_update_bit_0(rc, prob_len);
prob_len = (prob + LZMA_LEN_LOW
+ (pos_state <<
@@ -463,7 +467,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
} else {
rc_update_bit_1(rc, prob_len);
prob_len = prob + LZMA_LEN_CHOICE_2;
- if (rc_is_bit_0(rc, prob_len)) {
+ if (rc_is_bit_0(rc, prob_len, error)) {
rc_update_bit_0(rc, prob_len);
prob_len = (prob + LZMA_LEN_MID
+ (pos_state <<
@@ -479,7 +483,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
}
}
- rc_bit_tree_decode(rc, prob_len, num_bits, &len);
+ rc_bit_tree_decode(rc, prob_len, num_bits, &len, error);
len += offset;
if (cst->state < 4) {
@@ -494,7 +498,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
<< LZMA_NUM_POS_SLOT_BITS);
rc_bit_tree_decode(rc, prob,
LZMA_NUM_POS_SLOT_BITS,
- &pos_slot);
+ &pos_slot, error);
if (pos_slot >= LZMA_START_POS_MODEL_INDEX) {
int i, mi;
num_bits = (pos_slot >> 1) - 1;
@@ -507,7 +511,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
num_bits -= LZMA_NUM_ALIGN_BITS;
while (num_bits--)
cst->rep0 = (cst->rep0 << 1) |
- rc_direct_bit(rc);
+ rc_direct_bit(rc, error);
prob = p + LZMA_ALIGN;
cst->rep0 <<= LZMA_NUM_ALIGN_BITS;
num_bits = LZMA_NUM_ALIGN_BITS;
@@ -515,7 +519,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
i = 1;
mi = 1;
while (num_bits--) {
- if (rc_get_bit(rc, prob + mi, &mi))
+ if (rc_get_bit(rc, prob + mi, &mi, error))
cst->rep0 |= i;
i <<= 1;
}
@@ -537,7 +541,7 @@ STATIC int INIT unlzma(unsigned char *buf, int in_len,
int(*flush)(void*, unsigned int),
unsigned char *output,
int *posp,
- void(*error_fn)(char *x)
+ void(*error)(char *x)
)
{
struct lzma_header header;
@@ -553,8 +557,6 @@ STATIC int INIT unlzma(unsigned char *buf, int in_len,
unsigned char *inbuf;
int ret = -1;
- set_error_fn(error_fn);
-
if (buf)
inbuf = buf;
else
@@ -577,7 +579,7 @@ STATIC int INIT unlzma(unsigned char *buf, int in_len,
for (i = 0; i < sizeof(header); i++) {
if (rc.ptr >= rc.buffer_end)
- rc_read(&rc);
+ rc_read(&rc, error);
((unsigned char *)&header)[i] = *rc.ptr++;
}
@@ -622,17 +624,17 @@ STATIC int INIT unlzma(unsigned char *buf, int in_len,
for (i = 0; i < num_probs; i++)
p[i] = (1 << RC_MODEL_TOTAL_BITS) >> 1;
- rc_init_code(&rc);
+ rc_init_code(&rc, error);
while (get_pos(&wr) < header.dst_size) {
int pos_state = get_pos(&wr) & pos_state_mask;
uint16_t *prob = p + LZMA_IS_MATCH +
(cst.state << LZMA_NUM_POS_BITS_MAX) + pos_state;
- if (rc_is_bit_0(&rc, prob))
+ if (rc_is_bit_0(&rc, prob, error))
process_bit0(&wr, &rc, &cst, p, pos_state, prob,
- lc, literal_pos_mask);
+ lc, literal_pos_mask, error);
else {
- process_bit1(&wr, &rc, &cst, p, pos_state, prob);
+ process_bit1(&wr, &rc, &cst, p, pos_state, prob, error);
if (cst.rep0 == 0)
break;
}
--
1.6.3.3
^ permalink raw reply related
* [PATCH 3/9] Squashfs: add a decompressor framework
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
This adds a decompressor framework which allows multiple compression
algorithms to be cleanly supported.
Also update zlib wrapper and other code to use the new framework.
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
fs/squashfs/Makefile | 2 +-
fs/squashfs/block.c | 6 ++--
fs/squashfs/decompressor.c | 58 ++++++++++++++++++++++++++++++++++++++++++
fs/squashfs/decompressor.h | 55 +++++++++++++++++++++++++++++++++++++++
fs/squashfs/squashfs.h | 14 +++++-----
fs/squashfs/squashfs_fs_sb.h | 41 +++++++++++++++--------------
fs/squashfs/super.c | 45 ++++++++++++++++++-------------
fs/squashfs/zlib_wrapper.c | 17 ++++++++++--
8 files changed, 185 insertions(+), 53 deletions(-)
create mode 100644 fs/squashfs/decompressor.c
create mode 100644 fs/squashfs/decompressor.h
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index a397e6f..df8a19e 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
-squashfs-y += namei.o super.o symlink.o zlib_wrapper.o
+squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index baf7624..6f9914d 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -36,7 +36,7 @@
#include "squashfs_fs_sb.h"
#include "squashfs_fs_i.h"
#include "squashfs.h"
-
+#include "decompressor.h"
/*
* Read the metadata block length, this is stored in the first two
* bytes of the metadata block.
@@ -151,8 +151,8 @@ int squashfs_read_data(struct super_block *sb, void **buffer, u64 index,
}
if (compressed) {
- length = zlib_uncompress(msblk, buffer, bh, b, offset, length,
- srclength, pages);
+ length = squashfs_decompress(msblk, buffer, bh, b, offset,
+ length, srclength, pages);
if (length < 0)
goto read_failure;
} else {
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
new file mode 100644
index 0000000..0072ccd
--- /dev/null
+++ b/fs/squashfs/decompressor.c
@@ -0,0 +1,58 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher <phillip@lougher.demon.co.uk>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * decompressor.c
+ */
+
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <linux/buffer_head.h>
+
+#include "squashfs_fs.h"
+#include "squashfs_fs_sb.h"
+#include "squashfs_fs_i.h"
+#include "decompressor.h"
+#include "squashfs.h"
+
+/*
+ * This file (and decompressor.h) implements a decompressor framework for
+ * Squashfs, allowing multiple decompressors to be easily supported
+ */
+
+static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
+ NULL, NULL, NULL, 0, "unknown", 0
+};
+
+static const struct squashfs_decompressor *decompressor[] = {
+ &squashfs_zlib_comp_ops,
+ &squashfs_unknown_comp_ops
+};
+
+
+const struct squashfs_decompressor *squashfs_lookup_decompressor(int id)
+{
+ int i;
+
+ for (i = 0; decompressor[i]->id; i++)
+ if (id == decompressor[i]->id)
+ break;
+
+ return decompressor[i];
+}
diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
new file mode 100644
index 0000000..778760c
--- /dev/null
+++ b/fs/squashfs/decompressor.h
@@ -0,0 +1,55 @@
+#ifndef DECOMPRESSOR_H
+#define DECOMPRESSOR_H
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher <phillip@lougher.demon.co.uk>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * decompressor.h
+ */
+
+struct squashfs_decompressor {
+ void *(*init)(void);
+ void (*free)(void *);
+ int (*decompress)(struct squashfs_sb_info *, void **,
+ struct buffer_head **, int, int, int, int, int);
+ int id;
+ char *name;
+ int supported;
+};
+
+static inline void *squashfs_decompressor_init(struct squashfs_sb_info *msblk)
+{
+ return msblk->decompressor->init();
+}
+
+static inline void squashfs_decompressor_free(struct squashfs_sb_info *msblk,
+ void *s)
+{
+ if (msblk->decompressor)
+ msblk->decompressor->free(s);
+}
+
+static inline int squashfs_decompress(struct squashfs_sb_info *msblk,
+ void **buffer, struct buffer_head **bh, int b, int offset, int length,
+ int srclength, int pages)
+{
+ return msblk->decompressor->decompress(msblk, buffer, bh, b, offset,
+ length, srclength, pages);
+}
+#endif
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index b3eaf87..fe2587a 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -51,6 +51,9 @@ extern struct squashfs_cache_entry *squashfs_get_datablock(struct super_block *,
u64, int);
extern int squashfs_read_table(struct super_block *, void *, u64, int);
+/* decompressor.c */
+extern const struct squashfs_decompressor *squashfs_lookup_decompressor(int);
+
/* export.c */
extern __le64 *squashfs_read_inode_lookup_table(struct super_block *, u64,
unsigned int);
@@ -70,14 +73,8 @@ extern struct inode *squashfs_iget(struct super_block *, long long,
unsigned int);
extern int squashfs_read_inode(struct inode *, long long);
-/* zlib_wrapper.c */
-extern void *zlib_init(void);
-extern void zlib_free(void *);
-extern int zlib_uncompress(struct squashfs_sb_info *, void **,
- struct buffer_head **, int, int, int, int, int);
-
/*
- * Inodes and files operations
+ * Inodes, files and decompressor operations
*/
/* dir.c */
@@ -94,3 +91,6 @@ extern const struct inode_operations squashfs_dir_inode_ops;
/* symlink.c */
extern const struct address_space_operations squashfs_symlink_aops;
+
+/* zlib_wrapper.c */
+extern const struct squashfs_decompressor squashfs_zlib_comp_ops;
diff --git a/fs/squashfs/squashfs_fs_sb.h b/fs/squashfs/squashfs_fs_sb.h
index 23a67fa..7533350 100644
--- a/fs/squashfs/squashfs_fs_sb.h
+++ b/fs/squashfs/squashfs_fs_sb.h
@@ -52,25 +52,26 @@ struct squashfs_cache_entry {
};
struct squashfs_sb_info {
- int devblksize;
- int devblksize_log2;
- struct squashfs_cache *block_cache;
- struct squashfs_cache *fragment_cache;
- struct squashfs_cache *read_page;
- int next_meta_index;
- __le64 *id_table;
- __le64 *fragment_index;
- unsigned int *fragment_index_2;
- struct mutex read_data_mutex;
- struct mutex meta_index_mutex;
- struct meta_index *meta_index;
- void *stream;
- __le64 *inode_lookup_table;
- u64 inode_table;
- u64 directory_table;
- unsigned int block_size;
- unsigned short block_log;
- long long bytes_used;
- unsigned int inodes;
+ const struct squashfs_decompressor *decompressor;
+ int devblksize;
+ int devblksize_log2;
+ struct squashfs_cache *block_cache;
+ struct squashfs_cache *fragment_cache;
+ struct squashfs_cache *read_page;
+ int next_meta_index;
+ __le64 *id_table;
+ __le64 *fragment_index;
+ unsigned int *fragment_index_2;
+ struct mutex read_data_mutex;
+ struct mutex meta_index_mutex;
+ struct meta_index *meta_index;
+ void *stream;
+ __le64 *inode_lookup_table;
+ u64 inode_table;
+ u64 directory_table;
+ unsigned int block_size;
+ unsigned short block_log;
+ long long bytes_used;
+ unsigned int inodes;
};
#endif
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
index 6c3429b..3550aec 100644
--- a/fs/squashfs/super.c
+++ b/fs/squashfs/super.c
@@ -41,27 +41,35 @@
#include "squashfs_fs_sb.h"
#include "squashfs_fs_i.h"
#include "squashfs.h"
+#include "decompressor.h"
static struct file_system_type squashfs_fs_type;
static const struct super_operations squashfs_super_ops;
-static int supported_squashfs_filesystem(short major, short minor, short comp)
+static const struct squashfs_decompressor *supported_squashfs_filesystem(short
+ major, short minor, short id)
{
+ const struct squashfs_decompressor *decompressor;
+
if (major < SQUASHFS_MAJOR) {
ERROR("Major/Minor mismatch, older Squashfs %d.%d "
"filesystems are unsupported\n", major, minor);
- return -EINVAL;
+ return NULL;
} else if (major > SQUASHFS_MAJOR || minor > SQUASHFS_MINOR) {
ERROR("Major/Minor mismatch, trying to mount newer "
"%d.%d filesystem\n", major, minor);
ERROR("Please update your kernel\n");
- return -EINVAL;
+ return NULL;
}
- if (comp != ZLIB_COMPRESSION)
- return -EINVAL;
+ decompressor = squashfs_lookup_decompressor(id);
+ if (!decompressor->supported) {
+ ERROR("Filesystem uses \"%s\" compression. This is not "
+ "supported\n", decompressor->name);
+ return NULL;
+ }
- return 0;
+ return decompressor;
}
@@ -86,10 +94,6 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
}
msblk = sb->s_fs_info;
- msblk->stream = zlib_init();
- if (msblk->stream == NULL)
- goto failure;
-
sblk = kzalloc(sizeof(*sblk), GFP_KERNEL);
if (sblk == NULL) {
ERROR("Failed to allocate squashfs_super_block\n");
@@ -116,25 +120,25 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
goto failed_mount;
}
+ err = -EINVAL;
+
/* Check it is a SQUASHFS superblock */
sb->s_magic = le32_to_cpu(sblk->s_magic);
if (sb->s_magic != SQUASHFS_MAGIC) {
if (!silent)
ERROR("Can't find a SQUASHFS superblock on %s\n",
bdevname(sb->s_bdev, b));
- err = -EINVAL;
goto failed_mount;
}
- /* Check the MAJOR & MINOR versions and compression type */
- err = supported_squashfs_filesystem(le16_to_cpu(sblk->s_major),
+ /* Check the MAJOR & MINOR versions and lookup compression type */
+ msblk->decompressor = supported_squashfs_filesystem(
+ le16_to_cpu(sblk->s_major),
le16_to_cpu(sblk->s_minor),
le16_to_cpu(sblk->compression));
- if (err < 0)
+ if (msblk->decompressor == NULL)
goto failed_mount;
- err = -EINVAL;
-
/*
* Check if there's xattrs in the filesystem. These are not
* supported in this version, so warn that they will be ignored.
@@ -201,6 +205,10 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
err = -ENOMEM;
+ msblk->stream = squashfs_decompressor_init(msblk);
+ if (msblk->stream == NULL)
+ goto failed_mount;
+
msblk->block_cache = squashfs_cache_init("metadata",
SQUASHFS_CACHED_BLKS, SQUASHFS_METADATA_SIZE);
if (msblk->block_cache == NULL)
@@ -288,7 +296,7 @@ failed_mount:
squashfs_cache_delete(msblk->block_cache);
squashfs_cache_delete(msblk->fragment_cache);
squashfs_cache_delete(msblk->read_page);
- zlib_free(msblk->stream);
+ squashfs_decompressor_free(msblk, msblk->stream);
kfree(msblk->inode_lookup_table);
kfree(msblk->fragment_index);
kfree(msblk->id_table);
@@ -298,7 +306,6 @@ failed_mount:
return err;
failure:
- zlib_free(msblk->stream);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
return -ENOMEM;
@@ -342,7 +349,7 @@ static void squashfs_put_super(struct super_block *sb)
squashfs_cache_delete(sbi->block_cache);
squashfs_cache_delete(sbi->fragment_cache);
squashfs_cache_delete(sbi->read_page);
- zlib_free(sbi->stream);
+ squashfs_decompressor_free(sbi, sbi->stream);
kfree(sbi->id_table);
kfree(sbi->fragment_index);
kfree(sbi->meta_index);
diff --git a/fs/squashfs/zlib_wrapper.c b/fs/squashfs/zlib_wrapper.c
index 8ebbbc7..381768c 100644
--- a/fs/squashfs/zlib_wrapper.c
+++ b/fs/squashfs/zlib_wrapper.c
@@ -30,8 +30,9 @@
#include "squashfs_fs_sb.h"
#include "squashfs_fs_i.h"
#include "squashfs.h"
+#include "decompressor.h"
-void *zlib_init()
+static void *zlib_init(void)
{
z_stream *stream = kmalloc(sizeof(z_stream), GFP_KERNEL);
if (stream == NULL)
@@ -50,7 +51,7 @@ failed:
}
-void zlib_free(void *strm)
+static void zlib_free(void *strm)
{
z_stream *stream = strm;
@@ -60,7 +61,7 @@ void zlib_free(void *strm)
}
-int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
+static int zlib_uncompress(struct squashfs_sb_info *msblk, void **buffer,
struct buffer_head **bh, int b, int offset, int length, int srclength,
int pages)
{
@@ -137,3 +138,13 @@ release_mutex:
return -EIO;
}
+
+const struct squashfs_decompressor squashfs_zlib_comp_ops = {
+ .init = zlib_init,
+ .free = zlib_free,
+ .decompress = zlib_uncompress,
+ .id = ZLIB_COMPRESSION,
+ .name = "zlib",
+ .supported = 1
+};
+
--
1.6.3.3
^ permalink raw reply related
* [PATCH 07/9] lzma: Make lzma available to non initramfs/initrd code
From: Phillip Lougher @ 2009-12-07 8:37 UTC (permalink / raw)
To: akpm, linux-embedded, linux-fsdevel, linux-kernel,
phillip.lougher, tim.bird
Add a config option DECOMPRESS_LZMA_NEEDED which allows subsystems to
specify they need the unlzma code. Normally decompress_unlzma.c is
compiled with __init and unlzma is not exported to modules.
Move INIT definition into separate header files for bzip2/lzma/inflate
so it can be defined differently for each decompressor.
Signed-off-by: Phillip Lougher <phillip@lougher.demon.co.uk>
---
include/linux/decompress/bunzip2_mm.h | 12 ++++++++++++
include/linux/decompress/inflate_mm.h | 12 ++++++++++++
include/linux/decompress/mm.h | 3 ---
include/linux/decompress/unlzma_mm.h | 20 ++++++++++++++++++++
lib/Kconfig | 3 +++
lib/decompress_bunzip2.c | 1 +
lib/decompress_inflate.c | 1 +
lib/decompress_unlzma.c | 6 +++++-
8 files changed, 54 insertions(+), 4 deletions(-)
create mode 100644 include/linux/decompress/bunzip2_mm.h
create mode 100644 include/linux/decompress/inflate_mm.h
create mode 100644 include/linux/decompress/unlzma_mm.h
diff --git a/include/linux/decompress/bunzip2_mm.h b/include/linux/decompress/bunzip2_mm.h
new file mode 100644
index 0000000..cac6fef
--- /dev/null
+++ b/include/linux/decompress/bunzip2_mm.h
@@ -0,0 +1,12 @@
+#ifndef BUNZIP2_MM_H
+#define BUNZIP2_MM_H
+
+#ifdef STATIC
+/* Code active when included from pre-boot environment: */
+#define INIT
+#else
+/* Compile for initramfs/initrd code only */
+#define INIT __init
+#endif
+
+#endif
diff --git a/include/linux/decompress/inflate_mm.h b/include/linux/decompress/inflate_mm.h
new file mode 100644
index 0000000..ca4a2ae
--- /dev/null
+++ b/include/linux/decompress/inflate_mm.h
@@ -0,0 +1,12 @@
+#ifndef INFLATE_MM_H
+#define INFLATE_MM_H
+
+#ifdef STATIC
+/* Code active when included from pre-boot environment: */
+#define INIT
+#else
+/* Compile for initramfs/initrd code only */
+#define INIT __init
+#endif
+
+#endif
diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h
index 12ff8c3..80f5ba4 100644
--- a/include/linux/decompress/mm.h
+++ b/include/linux/decompress/mm.h
@@ -53,8 +53,6 @@ static void free(void *where)
#define set_error_fn(x)
-#define INIT
-
#else /* STATIC */
/* Code active when compiled standalone for use when loading ramdisk: */
@@ -77,7 +75,6 @@ static void free(void *where)
static void(*error)(char *m);
#define set_error_fn(x) error = x;
-#define INIT __init
#define STATIC
#include <linux/init.h>
diff --git a/include/linux/decompress/unlzma_mm.h b/include/linux/decompress/unlzma_mm.h
new file mode 100644
index 0000000..859287e
--- /dev/null
+++ b/include/linux/decompress/unlzma_mm.h
@@ -0,0 +1,20 @@
+#ifndef UNLZMA_MM_H
+#define UNLZMA_MM_H
+
+#ifdef STATIC
+
+/* Code active when included from pre-boot environment: */
+#define INIT
+
+#elif defined(CONFIG_DECOMPRESS_LZMA_NEEDED)
+
+/* Make it available to non initramfs/initrd code */
+#define INIT
+#include <linux/module.h>
+#else
+
+/* Compile for initramfs/initrd code only */
+#define INIT __init
+#endif
+
+#endif
diff --git a/lib/Kconfig b/lib/Kconfig
index bb1326d..25e7f28 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -117,6 +117,9 @@ config DECOMPRESS_BZIP2
config DECOMPRESS_LZMA
tristate
+config DECOMPRESS_LZMA_NEEDED
+ boolean
+
#
# Generic allocator support is selected if needed
#
diff --git a/lib/decompress_bunzip2.c b/lib/decompress_bunzip2.c
index 600f473..6eb6433 100644
--- a/lib/decompress_bunzip2.c
+++ b/lib/decompress_bunzip2.c
@@ -52,6 +52,7 @@
#include <linux/slab.h>
#endif /* STATIC */
+#include <linux/decompress/bunzip2_mm.h>
#include <linux/decompress/mm.h>
#ifndef INT_MAX
diff --git a/lib/decompress_inflate.c b/lib/decompress_inflate.c
index fc686c7..cb6bcab 100644
--- a/lib/decompress_inflate.c
+++ b/lib/decompress_inflate.c
@@ -23,6 +23,7 @@
#endif /* STATIC */
+#include <linux/decompress/inflate_mm.h>
#include <linux/decompress/mm.h>
#define GZIP_IOBUF_SIZE (16*1024)
diff --git a/lib/decompress_unlzma.c b/lib/decompress_unlzma.c
index ca82fde..a614b26 100644
--- a/lib/decompress_unlzma.c
+++ b/lib/decompress_unlzma.c
@@ -36,6 +36,7 @@
#include <linux/slab.h>
#endif /* STATIC */
+#include <linux/decompress/unlzma_mm.h>
#include <linux/decompress/mm.h>
#define MIN(a, b) (((a) < (b)) ? (a) : (b))
@@ -531,7 +532,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc,
-STATIC inline int INIT unlzma(unsigned char *buf, int in_len,
+STATIC int INIT unlzma(unsigned char *buf, int in_len,
int(*fill)(void*, unsigned int),
int(*flush)(void*, unsigned int),
unsigned char *output,
@@ -652,6 +653,9 @@ exit_1:
exit_0:
return ret;
}
+#if defined(CONFIG_DECOMPRESS_LZMA_NEEDED) && !defined(PREBOOT)
+EXPORT_SYMBOL(unlzma);
+#endif
#ifdef PREBOOT
STATIC int INIT decompress(unsigned char *buf, int in_len,
--
1.6.3.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox