* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Paul Mundt @ 2008-08-28 1:02 UTC (permalink / raw)
To: David Miller
Cc: bunk-DgEjT+Ai2ygdnm+yROfE0A,
torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
rusty-8n+1lVoiYb80n/F98K4Iww, Alan.Brunelle-VXdhtT5mjnY,
rjw-KKrjLPT3xs0, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
kernel-testers-u79uwXL29TY76Z2rM5mHXA,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
arjan-VuQAYsv1563Yd54FQh9/CA, mingo-X9Un+BFzKDI,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20080827.174605.85608276.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
On Wed, Aug 27, 2008 at 05:46:05PM -0700, David Miller wrote:
> From: Paul Mundt <lethal-M7jkjyW5wf5g9hUCZPvPmw@public.gmane.org>
> Date: Thu, 28 Aug 2008 09:32:13 +0900
>
> > On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote:
> > > CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if
> > > wanted with an arbitrary limit.
> >
> > In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is
> > only performed from do_IRQ(), which is sporadic at best, especially on
> > tickless. While it catches some things, it's not a complete solution in
> > and of iteslf.
>
> BTW, on sparc64 we have a stack overflow checker that runs via
> the profiling _mcount hook. So every function call we check
> if the stack is getting overused.
>
> If so, we jump onto a special static debugging stack and print
> the stack overflow message.
>
> And yes it works with IRQ stacks which is all that sparc64 uses
> nowadays.
>
> Perhaps this is useful enough to make generic.
Thanks for the pointer, I'll take a look at it!
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Greg Ungerer @ 2008-08-28 1:05 UTC (permalink / raw)
To: Paul Mundt, Adrian Bunk, Linus Torvalds, Rusty Russell,
"Alan D. Brunelle" <Alan.Brunel>
In-Reply-To: <20080827160052.GA15968@linux-sh.org>
Paul Mundt wrote:
> On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote:
>> On Tue, Aug 26, 2008 at 05:28:37PM -0700, Linus Torvalds wrote:
>>> On Wed, 27 Aug 2008, Adrian Bunk wrote:
>>>> When did we get callpaths like like nfs+xfs+md+scsi reliably
>>>> working with 4kB stacks on x86-32?
>>> XFS may never have been usable, but the rest, sure.
>>>
>>> And you seem to be making this whole argument an excuse to SUCK, adn an
>>> excuse to let gcc crap even more on our stack space.
>>>
>>> Why?
>>>
>>> Why aren't you saying that we should be able to do better? Instead, you
>>> seem to asking us to do even worse than we do now?
>> My main point is:
>> - getting 4kB stacks working reliably is a hard task
>> - having an eye on gcc increasing the stack usage, and fixing it if
>> required, is relatively easy
>>
>> If we should be able to do better at getting (and keeping) 4kB stacks
>> working, then coping with possible inlining problems caused by gcc
>> should not be a big problem for us.
>>
> Out of the architectures you've mentioned for 4k stacks, they also tend
> to do IRQ stacks, which is something you seem to have overlooked.
>
> In addition to that, debugging the runaway stack users on 4k tends to be
> easier anyways since you end up blowing the stack a lot sooner. On sh
> we've had pretty good luck with it, though most of our users are using
> fairly deterministic workloads and continually profiling the footprint.
> Anything that runs away or uses an insane amount of stack space needs to
> be fixed well before that anyways, so catching it sooner is always
> preferable. I imagine the same case is true for m68knommu (even sans IRQ
> stacks).
Yep, definitely true for m68knommu in my experience. I haven't had
any problems with 4k stacks recently. But yes the workloads do tend
to be constrained - and almost never use any of the more exotic
filesystems or drivers.
> Things might be more sensitive on x86, but it's certainly not something
> that's a huge problem for the various embedded platforms to wire up,
> whether they want to go the IRQ stack route or not.
>
> In any event, lack of support for something on embedded architectures in
> the kernel is more often due to apathy/utter indifference on the part of
> the architecture maintainer rather than being indicative of any intrinsic
> difficulty in supporting the thing in question. Most new "features" on the
> lesser maintained architectures tend to end up there either out of peer
> pressure or copying-and-pasting accidents rather than any sort of design.
> ;-)
Indeed :-)
Regards
Greg
------------------------------------------------------------------------
Greg Ungerer -- Chief Software Dude EMAIL: gerg@snapgear.com
Secure Computing Corporation PHONE: +61 7 3435 2888
825 Stanley St, FAX: +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Geert Uytterhoeven @ 2008-08-28 15:40 UTC (permalink / raw)
To: Jared Hulbert
Cc: Linux-kernel, linux-embedded, linux-mtd, Jörn Engel,
tim.bird, cotte, nickpiggin
In-Reply-To: <6934efce0808220951i5a2cd6f9t70f9c522eae6f1d6@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 2909 bytes --]
On Fri, 22 Aug 2008, Jared Hulbert wrote:
> > The version in SVN seems to be slightly older than the one you submitted?
>
> Oops. Okay I must have neglected to sync at the very end. Thanks.
>
> I forgot, there is also a git repo at
>
> git://git.infradead.org/users/jehulber/axfs.git
>
> > Which platform(s) do you use for testing?
>
> ARM, x86
>
> > I gave AxFS a try on PS3 (ppc64, always use big-endian 64-bit for testing new
> > code ;-).
>
> Smart. Hmmm, If only I had a PS3....
I heard you got one? ;-)
> > When mounting the image, I got the crash below:
> >
> > | attempt to access beyond end of device
> > | loop0: rw=0, want=4920, limit=4912
Interestingly, it also doesn't work on UserModeLinux (x86, 32-bit):
| attempt to access beyond end of device
| loop0: rw=0, want=24, limit=16
|
| EIP: 0073:[<0811ec67>] CPU: 0 Not tainted ESP: 007b:19515c38 EFLAGS: 00210212
| Not tainted
| EAX: 00000000 EBX: 00001000 ECX: 19484aa0 EDX: 190d9f0c
| ESI: 195ee000 EDI: 19515cd0 EBP: 19515c6c DS: 007b ES: 007b
| 08247af0: [<08069ba3>] show_regs+0xb4/0xb9
| 08247b1c: [<080591ee>] segv+0x222/0x23a
| 08247bbc: [<08059296>] segv_handler+0x90/0x9a
| 08247c68: [<080649b8>] sig_handler_common+0x63/0x72
| 08247ce0: [<08064cac>] sig_handler+0x31/0x3d
| 08247cec: [<08064c0b>] handle_signal+0x4c/0x7a
| 08247d0c: [<08066327>] hard_handler+0xf/0x14
| 08247d1c: [<005c0420>] 0x5c0420
|
| Kernel panic - not syncing: Kernel mode fault at addr 0x14, ip 0x811ec67
|
| EIP: 0073:[<4010a44e>] CPU: 0 Not tainted ESP: 007b:bfa323a0 EFLAGS: 00200246
| Not tainted
| EAX: ffffffda EBX: 080595f8 ECX: 080595c8 EDX: 080595d8
| ESI: c0ed0000 EDI: 00000000 EBP: bfa323d8 DS: 007b ES: 007b
| 08247a5c: [<08069ba3>] show_regs+0xb4/0xb9
| 08247a88: [<08059462>] panic_exit+0x25/0x3b
| 08247a9c: [<08083642>] notifier_call_chain+0x27/0x4c
| 08247ac4: [<0808367e>] __atomic_notifier_call_chain+0x17/0x19
| 08247ad4: [<08083695>] atomic_notifier_call_chain+0x15/0x17
| 08247af0: [<0806fd87>] panic+0x52/0xd8
| 08247b10: [<080591fc>] segv+0x230/0x23a
| 08247bbc: [<08059296>] segv_handler+0x90/0x9a
| 08247c68: [<080649b8>] sig_handler_common+0x63/0x72
| 08247ce0: [<08064cac>] sig_handler+0x31/0x3d
| 08247cec: [<08064c0b>] handle_signal+0x4c/0x7a
| 08247d0c: [<08066327>] hard_handler+0xf/0x14
| 08247d1c: [<005c0420>] 0x5c0420
Commandline is `mount image.axfs /mnt -o loop -t axfs'.
Is there something wrong with the axfs version you submitted, or with
mkfs.axfs?
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-28 16:16 UTC (permalink / raw)
To: Paul Mundt, Linus Torvalds, Rusty Russell, Alan D. Brunelle,
Rafael
In-Reply-To: <20080828003211.GA18893@linux-sh.org>
On Thu, Aug 28, 2008 at 09:32:13AM +0900, Paul Mundt wrote:
> On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote:
> > On Thu, Aug 28, 2008 at 01:00:52AM +0900, Paul Mundt wrote:
> > > On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote:
> > > In addition to that, debugging the runaway stack users on 4k tends to be
> > > easier anyways since you end up blowing the stack a lot sooner. On sh
> > > we've had pretty good luck with it, though most of our users are using
> > > fairly deterministic workloads and continually profiling the footprint.
> > > Anything that runs away or uses an insane amount of stack space needs to
> > > be fixed well before that anyways, so catching it sooner is always
> > > preferable. I imagine the same case is true for m68knommu (even sans IRQ
> > > stacks).
> >
> > CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if
> > wanted with an arbitrary limit.
> >
> In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is
> only performed from do_IRQ(), which is sporadic at best, especially on
> tickless. While it catches some things, it's not a complete solution in
> and of iteslf.
>
> In addition to this, there are even fewer platforms that support it than
> there are platforms that do 4k stacks. At first glance, it looks like
> it's only m32r, powerpc, sh, x86, and xtensa.
>...
As far as I can see the only architectures that optionally offer 4kB
stacks today are m68knommu, s390, sh and x86.
Did I miss some architectures or is 5 < 4 ;) ?
> Others support the Kconfig
> option, but don't seem to realize that it's not an option that the kernel
> does anything with by itself, and so don't actually do anything (ie,
> FRV).
Unless I miss anything these "others" include only FRV.
> > IMHO there seems to currently be a mismatch between it's maintainance
> > cost and the actual number of users. That's in my opinion the main
> > problem with it, no matter in which direction it gets resolved.
> >
> Perhaps that's true on x86, but in general I take issue with that. On sh
> we've had to do very little maintenance for it and most shipping products
> are using it today (at least on MMU-Linux, we don't bother with it on
> nommu). Most of the problems we ran in to with 4k stacks tended to be
> stuff that we wanted to fix for 8k anyways. I suspect that this case is
> true for the other embedded platforms also.
>...
Most stack issues are not platform or architecture specific.
The maintainance effort therefore mostly depends on whether a non-zero
number of architectures uses 4kB stacks.
And if something is considered to be important for small embedded
systems, but not supported on ARM, MIPS or PowerPC, then that's
a bit strange.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [RFC] Remove more code when IP_MULTICAST=n
From: Alexander Clouter @ 2008-08-28 20:25 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <48B432EF.1010403@am.sony.com>
Hi,
Tim Bird <tim.bird@am.sony.com> wrote:
>
> [snipped]
>
> In this particular case, I think we need to show that
> there are valid cases were an embedded product can use networking just
> fine, without IGMP support but with support for multicast. (I believe
> that's the slice that this particular patch makes). IMHO, what's
> needed is a test to show that this works. This can then be presented
> to the netdev guys, who are, after all much more experienced with the
> networking stuff. If they can show that multicast does indeed *need* IGMP
> to work correctly, then we need to back off. The last thing I want is
> to find out in the field that some streaming feature I've built into a
> Sony camera that relies on multicast won't work in lots of network
> configurations. If that's true, then David is doing me a favor by
> saying No. (Note that I might still make the decision to forego
> a feature for size reasons if the number of instances of non-workingness
> was small.)
>
As a networking sysadmin monkey I can tell you that:
* you _need_ IGMP to *join* multicast groups
* you do not need to know anything about multicast to send data to a
group (except to consider a sensible value for the TTL in your
IP header)[1]
* you need PIM support *route* multicast traffic[2]
So as long as your embedded device does not need to route or *receive* a
multicast stream then you do not need IGMP support.
A reasonable example of this is a IPTV (source) streaming device, such as a
DVB->IP gateway.
To be honest, especially so in an embedded device, multicast is going to
become more and more important. Zero configuration along with DNS and
even now a multicast enabled DHCP RFC are here. Multicast is dead handy
for devices to automatically discover each other without using dreadful
broadcast traffic that of course is restricted to the same VLAN.
Real PnP plus service discovery lives in the world of multicast, the lusers
want it and it's the only way to provision such facilities. My personal
recommendation is that patches that affect 'net/' go to netdev@.
Just the comments of a daytime networking sysadmin :)
Cheers
Alex
[1] an exception, iirc, is that you can speed up group creation with the
"Create Group Request" igmp message type, but I could be wrong;
this is most definiately optional though.
[2] if your router/switch does not support multicast then you will find it
floods your switch ports with the traffic, effectively making it
broadcast traffic
^ permalink raw reply
* [patch 0/3] [RFC] zlib crypto module
From: Geert Uytterhoeven @ 2008-08-29 13:41 UTC (permalink / raw)
To: linux-crypto, squashfs-devel; +Cc: linux-embedded, linux-kernel
These patches add a (de)compression module for the "zlib" format using the
crypto API:
[1] crypto: Add a zlib crypto module
[2] tcrypt: Add a self test for the zlib crypto module
[3] squashfs: Switch from zlib/inflate to "zlib" crypto module
The last patch is a proof-of-concept to make SquashFS 3.4 use this new zlib
crypto module. This makes it easier to e.g. change the decompression algorithm
in SquashFS or to make use of a hardware-accelerated zlib crypto module.
It can be extended to other compressed file systems, like e.g. AxFS and CRAMFS.
All comments are welcome. Thanks!
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* [patch 1/3] crypto: Add a zlib crypto module
From: Geert Uytterhoeven @ 2008-08-29 13:41 UTC (permalink / raw)
To: linux-crypto, squashfs-devel
Cc: linux-embedded, linux-kernel, Geert Uytterhoeven
In-Reply-To: <20080829134158.108976037@vixen.sonytel.be>
[-- Attachment #1: crypto/add-zlib-crypto-module.diff --]
[-- Type: TEXT/PLAIN, Size: 8561 bytes --]
From: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Add a (de)compression module for the "zlib" format using the crypto API.
While both the "zlib" and "deflate" crypto modules are implemented on top of
the zlib library, they differ in the following aspects:
- The "deflate" crypto module (used by IPSec and UBIFS) does not support
partial decompression, i.e. all compressed data has to be passed at once.
The "zlib" crypto module does support partial decompression;
zlib_decompress() will return -EAGAIN if not all compressed data has been
passed.
- The deflate crypto module uses the raw deflate data format (zlib is
initialized with a windowBits parameter of -DEFLATE_DEF_WINBITS = -11),
while e.g. squashfs and axfs use the zlib data format, with the default
windowBits parameter DEF_WBITS = 15.
Both parameters are incompatible with each other due to the different data
formats, as indicated by the sign of the windowbits parameter.
The absolute value of this parameter is the base two logarithm of the
maximum window size, and larger values are backwards compatible with
smaller values (as far as decompression is concerned).
TODO:
- As the crypto wrapper around the zlib library supports both compression and
decompression (and always allocates temporary space for both during
initialization; lazy allocation was removed in 2004 because IPComp calls
the crypto routines in non-process context), more memory is needed:
* decompression (inflate) needs only ca. 41 KiB
* compression (deflate) needs ca. 262 KiB!
It would be nice to allow having support for decompression only.
Signed-off-by: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
---
crypto/Kconfig | 8 ++
crypto/Makefile | 1
crypto/zlib.c | 216 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 225 insertions(+)
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -666,6 +666,14 @@ config CRYPTO_LZO
help
This is the LZO algorithm.
+config CRYPTO_ZLIB
+ tristate "Zlib compression algorithm"
+ select CRYPTO_ALGAPI
+ select ZLIB_INFLATE
+ select ZLIB_DEFLATE
+ help
+ This is the Zlib algorithm.
+
source "drivers/crypto/Kconfig"
endif # if CRYPTO
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -69,6 +69,7 @@ obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += mich
obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o
obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o
obj-$(CONFIG_CRYPTO_LZO) += lzo.o
+obj-$(CONFIG_CRYPTO_ZLIB) += zlib.o
obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
--- /dev/null
+++ b/crypto/zlib.c
@@ -0,0 +1,216 @@
+/*
+ * Cryptographic API.
+ *
+ * "zlib" crypto module, based on the "deflate" crypto module
+ *
+ * Copyright (c) 2003 James Morris <jmorris@intercode.com.au>
+ * Copyright 2008 Sony Corp.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * FIXME: deflate transforms will require up to a total of about 436k of kernel
+ * memory on i386 (390k for compression, the rest for decompression), as the
+ * current zlib kernel code uses a worst case pre-allocation system by default.
+ * This needs to be fixed so that the amount of memory required is properly
+ * related to the winbits and memlevel parameters.
+ */
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/crypto.h>
+#include <linux/zlib.h>
+#include <linux/vmalloc.h>
+#include <linux/interrupt.h>
+#include <linux/mm.h>
+#include <linux/net.h>
+#include <linux/slab.h>
+
+#define ZLIB_DEF_LEVEL Z_DEFAULT_COMPRESSION
+
+struct zlib_ctx {
+ struct z_stream_s comp_stream;
+ struct z_stream_s decomp_stream;
+ bool decomp_needs_reset;
+};
+
+static int zlib_comp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->comp_stream;
+
+ stream->workspace = vmalloc(zlib_deflate_workspacesize());
+ if (!stream->workspace ) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memset(stream->workspace, 0, zlib_deflate_workspacesize());
+ ret = zlib_deflateInit(stream, ZLIB_DEF_LEVEL);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ vfree(stream->workspace);
+ goto out;
+}
+
+static int zlib_decomp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->decomp_stream;
+
+ stream->workspace = kzalloc(zlib_inflate_workspacesize(), GFP_KERNEL);
+ if (!stream->workspace ) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = zlib_inflateInit(stream);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+ ctx->decomp_needs_reset = true;
+out:
+ return ret;
+out_free:
+ kfree(stream->workspace);
+ goto out;
+}
+
+static void zlib_comp_exit(struct zlib_ctx *ctx)
+{
+ zlib_deflateEnd(&ctx->comp_stream);
+ vfree(ctx->comp_stream.workspace);
+}
+
+static void zlib_decomp_exit(struct zlib_ctx *ctx)
+{
+ zlib_inflateEnd(&ctx->decomp_stream);
+ kfree(ctx->decomp_stream.workspace);
+}
+
+static int zlib_init(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+ int ret;
+
+ ret = zlib_comp_init(ctx);
+ if (ret)
+ goto out;
+ ret = zlib_decomp_init(ctx);
+ if (ret)
+ zlib_comp_exit(ctx);
+out:
+ return ret;
+}
+
+static void zlib_exit(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ zlib_comp_exit(ctx);
+ zlib_decomp_exit(ctx);
+}
+
+static int zlib_compress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret = 0;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ ret = zlib_deflateReset(stream);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+
+ ret = zlib_deflate(stream, Z_FINISH);
+ if (ret != Z_STREAM_END) {
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = 0;
+ *dlen = stream->total_out;
+out:
+ return ret;
+}
+
+static int zlib_decompress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret = 0;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+ unsigned long old_total_out;
+
+ if (dctx->decomp_needs_reset) {
+ ret = zlib_inflateReset(stream);
+ if (ret != Z_OK)
+ return -EINVAL;
+ dctx->decomp_needs_reset = false;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+ old_total_out = stream->total_out;
+
+ ret = zlib_inflate(stream, Z_SYNC_FLUSH);
+ switch (ret) {
+ case Z_STREAM_END:
+ dctx->decomp_needs_reset = true;
+ ret = 0;
+ break;
+
+ case Z_OK:
+ ret = -EAGAIN;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ *dlen = stream->total_out - old_total_out;
+ return ret;
+}
+
+static struct crypto_alg alg = {
+ .cra_name = "zlib",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zlib_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_list = LIST_HEAD_INIT(alg.cra_list),
+ .cra_init = zlib_init,
+ .cra_exit = zlib_exit,
+ .cra_u = { .compress = {
+ .coa_compress = zlib_compress,
+ .coa_decompress = zlib_decompress } }
+};
+
+static int __init zlib_mod_init(void)
+{
+ return crypto_register_alg(&alg);
+}
+
+static void __exit zlib_mod_fini(void)
+{
+ crypto_unregister_alg(&alg);
+}
+
+module_init(zlib_mod_init);
+module_exit(zlib_mod_fini);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Zlib Compression Algorithm");
+
--
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [patch 2/3] tcrypt: Add a self test for the zlib crypto module
From: Geert Uytterhoeven @ 2008-08-29 13:42 UTC (permalink / raw)
To: linux-crypto, squashfs-devel
Cc: linux-embedded, linux-kernel, Geert Uytterhoeven
In-Reply-To: <20080829134158.108976037@vixen.sonytel.be>
[-- Attachment #1: crypto/add-zlib-crypto-test.diff --]
[-- Type: TEXT/PLAIN, Size: 4872 bytes --]
From: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Signed-off-by: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
---
crypto/tcrypt.c | 9 ++++++
crypto/tcrypt.h | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 90 insertions(+)
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -1400,6 +1400,9 @@ static void do_test(void)
DEFLATE_DECOMP_TEST_VECTORS);
test_comp("lzo", lzo_comp_tv_template, lzo_decomp_tv_template,
LZO_COMP_TEST_VECTORS, LZO_DECOMP_TEST_VECTORS);
+ test_comp("zlib", zlib_comp_tv_template,
+ zlib_decomp_tv_template, ZLIB_COMP_TEST_VECTORS,
+ ZLIB_DECOMP_TEST_VECTORS);
test_hash("crc32c", crc32c_tv_template, CRC32C_TEST_VECTORS);
test_hash("hmac(md5)", hmac_md5_tv_template,
HMAC_MD5_TEST_VECTORS);
@@ -1701,6 +1704,12 @@ static void do_test(void)
test_hash("rmd320", rmd320_tv_template, RMD320_TEST_VECTORS);
break;
+ case 43:
+ test_comp("zlib", zlib_comp_tv_template,
+ zlib_decomp_tv_template, ZLIB_COMP_TEST_VECTORS,
+ ZLIB_DECOMP_TEST_VECTORS);
+ break;
+
case 100:
test_hash("hmac(md5)", hmac_md5_tv_template,
HMAC_MD5_TEST_VECTORS);
--- a/crypto/tcrypt.h
+++ b/crypto/tcrypt.h
@@ -8512,6 +8512,87 @@ static struct comp_testvec lzo_decomp_tv
};
/*
+ * Zlib test vectors (null-terminated strings).
+ * Params: winbits=DEF_WBITS, Z_DEFAULT_COMPRESSION, MAX_MEM_LEVEL.
+ */
+#define ZLIB_COMP_TEST_VECTORS 2
+#define ZLIB_DECOMP_TEST_VECTORS 2
+
+static struct comp_testvec zlib_comp_tv_template[] = {
+ {
+ .inlen = 70,
+ .outlen = 44,
+ .input = "Join us now and share the software "
+ "Join us now and share the software ",
+ .output = "\x78\x9c\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ }, {
+ .inlen = 191,
+ .outlen = 128,
+ .input = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ .output = "\x78\x9c\x5d\x8d\x31\x0e\xc2\x30"
+ "\x10\x04\xbf\xb2\x2f\xc8\x1f\x10"
+ "\x04\x09\x89\xc2\x85\x3f\x70\xb1"
+ "\x2f\xf8\x24\xdb\x67\xd9\x47\xc1"
+ "\xef\x49\x68\x12\x51\xae\x76\x67"
+ "\xd6\x27\x19\x88\x1a\xde\x85\xab"
+ "\x21\xf2\x08\x5d\x16\x1e\x20\x04"
+ "\x2d\xad\xf3\x18\xa2\x15\x85\x2d"
+ "\x69\xc4\x42\x83\x23\xb6\x6c\x89"
+ "\x71\x9b\xef\xcf\x8b\x9f\xcf\x33"
+ "\xca\x2f\xed\x62\xa9\x4c\x80\xff"
+ "\x13\xaf\x52\x37\xed\x0e\x52\x6b"
+ "\x59\x02\xd9\x4e\xe8\x7a\x76\x1d"
+ "\x02\x98\xfe\x8a\x87\x83\xa3\x4f"
+ "\x56\x8a\xb8\x9e\x8e\x5c\x57\xd3"
+ "\xa0\x79\xfa\x02\x2e\x32\x45\x4e",
+ },
+};
+
+static struct comp_testvec zlib_decomp_tv_template[] = {
+ {
+ .inlen = 128,
+ .outlen = 191,
+ .input = "\x78\x9c\x5d\x8d\x31\x0e\xc2\x30"
+ "\x10\x04\xbf\xb2\x2f\xc8\x1f\x10"
+ "\x04\x09\x89\xc2\x85\x3f\x70\xb1"
+ "\x2f\xf8\x24\xdb\x67\xd9\x47\xc1"
+ "\xef\x49\x68\x12\x51\xae\x76\x67"
+ "\xd6\x27\x19\x88\x1a\xde\x85\xab"
+ "\x21\xf2\x08\x5d\x16\x1e\x20\x04"
+ "\x2d\xad\xf3\x18\xa2\x15\x85\x2d"
+ "\x69\xc4\x42\x83\x23\xb6\x6c\x89"
+ "\x71\x9b\xef\xcf\x8b\x9f\xcf\x33"
+ "\xca\x2f\xed\x62\xa9\x4c\x80\xff"
+ "\x13\xaf\x52\x37\xed\x0e\x52\x6b"
+ "\x59\x02\xd9\x4e\xe8\x7a\x76\x1d"
+ "\x02\x98\xfe\x8a\x87\x83\xa3\x4f"
+ "\x56\x8a\xb8\x9e\x8e\x5c\x57\xd3"
+ "\xa0\x79\xfa\x02\x2e\x32\x45\x4e",
+ .output = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ }, {
+ .inlen = 44,
+ .outlen = 70,
+ .input = "\x78\x9c\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ .output = "Join us now and share the software "
+ "Join us now and share the software ",
+ },
+};
+
+/*
* Michael MIC test vectors from IEEE 802.11i
*/
#define MICHAEL_MIC_TEST_VECTORS 6
--
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* [patch 3/3] squashfs: Switch from zlib/inflate to "zlib" crypto module
From: Geert Uytterhoeven @ 2008-08-29 13:42 UTC (permalink / raw)
To: linux-crypto, squashfs-devel
Cc: linux-embedded, linux-kernel, Geert Uytterhoeven
In-Reply-To: <20080829134158.108976037@vixen.sonytel.be>
[-- Attachment #1: crypto/squashfs-use-crypto-api-for-orig.diff --]
[-- Type: TEXT/PLAIN, Size: 6165 bytes --]
From: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
Modify SquashFS 3.4 to use the "zlib" crypto module instead of making direct
calls to the zlib/inflate library
Signed-off-by: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
---
fs/Kconfig | 3 +
fs/squashfs/inode.c | 65 +++++++++++++++++++++--------------------
include/linux/squashfs_fs_sb.h | 3 +
3 files changed, 38 insertions(+), 33 deletions(-)
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -1350,7 +1350,8 @@ config CRAMFS
config SQUASHFS
tristate "SquashFS 3.4 - Squashed file system support"
- select ZLIB_INFLATE
+ select CRYPTO
+ select CRYPTO_ZLIB
help
Saying Y here includes support for SquashFS 3.4 (a Compressed
Read-Only File System). Squashfs is a highly compressed read-only
--- a/fs/squashfs/inode.c
+++ b/fs/squashfs/inode.c
@@ -23,7 +23,6 @@
#include <linux/squashfs_fs.h>
#include <linux/module.h>
-#include <linux/zlib.h>
#include <linux/fs.h>
#include <linux/squashfs_fs_sb.h>
#include <linux/squashfs_fs_i.h>
@@ -36,6 +35,9 @@
#include "squashfs.h"
+#define SQUASHFS_CRYPTO_ALG "zlib"
+
+
static struct dentry *squashfs_fh_to_dentry(struct super_block *s,
struct fid *fid, int fh_len, int fh_type);
static struct dentry *squashfs_fh_to_parent(struct super_block *s,
@@ -236,7 +238,10 @@ SQSH_EXTERN unsigned int squashfs_read_d
}
if (compressed) {
- int zlib_err = 0;
+ int error = 0;
+ void *next_out;
+ int avail_out;
+ unsigned int total_out = 0, dlen;
/*
* uncompress block
@@ -244,8 +249,8 @@ SQSH_EXTERN unsigned int squashfs_read_d
mutex_lock(&msblk->read_data_mutex);
- msblk->stream.next_out = buffer;
- msblk->stream.avail_out = srclength;
+ next_out = buffer;
+ avail_out = srclength;
for (bytes = 0; k < b; k++) {
avail_bytes = min(c_byte - bytes, msblk->devblksize - offset);
@@ -254,16 +259,8 @@ SQSH_EXTERN unsigned int squashfs_read_d
if (!buffer_uptodate(bh[k]))
goto release_mutex;
- msblk->stream.next_in = bh[k]->b_data + offset;
- msblk->stream.avail_in = avail_bytes;
-
if (k == 0) {
- zlib_err = zlib_inflateInit(&msblk->stream);
- if (zlib_err != Z_OK) {
- ERROR("zlib_inflateInit returned unexpected result 0x%x,"
- " srclength %d\n", zlib_err, srclength);
- goto release_mutex;
- }
+ total_out = 0;
if (avail_bytes == 0) {
offset = 0;
@@ -272,29 +269,32 @@ SQSH_EXTERN unsigned int squashfs_read_d
}
}
- zlib_err = zlib_inflate(&msblk->stream, Z_NO_FLUSH);
- if (zlib_err != Z_OK && zlib_err != Z_STREAM_END) {
- ERROR("zlib_inflate returned unexpected result 0x%x,"
- " srclength %d, avail_in %d, avail_out %d\n", zlib_err,
- srclength, msblk->stream.avail_in, msblk->stream.avail_out);
+ dlen = avail_out;
+ error = crypto_comp_decompress(msblk->tfm,
+ bh[k]->b_data + offset,
+ avail_bytes, next_out,
+ &dlen);
+ if (error && error != -EAGAIN) {
+ ERROR("crypto_comp_decompress returned "
+ "unexpected result %d, srclength %d, "
+ "avail_bytes %d, avail_out %d\n",
+ error, srclength, avail_bytes,
+ avail_out);
goto release_mutex;
}
+ next_out += dlen;
+ avail_out -= dlen;
+ total_out += dlen;
bytes += avail_bytes;
offset = 0;
brelse(bh[k]);
}
- if (zlib_err != Z_STREAM_END)
+ if (error)
goto release_mutex;
- zlib_err = zlib_inflateEnd(&msblk->stream);
- if (zlib_err != Z_OK) {
- ERROR("zlib_inflateEnd returned unexpected result 0x%x,"
- " srclength %d\n", zlib_err, srclength);
- goto release_mutex;
- }
- bytes = msblk->stream.total_out;
+ bytes = total_out;
mutex_unlock(&msblk->read_data_mutex);
} else {
int i;
@@ -1104,9 +1104,12 @@ static int squashfs_fill_super(struct su
}
msblk = s->s_fs_info;
- msblk->stream.workspace = vmalloc(zlib_inflate_workspacesize());
- if (msblk->stream.workspace == NULL) {
- ERROR("Failed to allocate zlib workspace\n");
+ msblk->tfm = crypto_alloc_comp(SQUASHFS_CRYPTO_ALG, 0,
+ CRYPTO_ALG_ASYNC);
+ if (IS_ERR(msblk->tfm)) {
+ ERROR("Failed to load %s crypto module\n",
+ SQUASHFS_CRYPTO_ALG);
+ msblk->tfm = NULL;
goto failure;
}
sblk = &msblk->sblk;
@@ -1273,7 +1276,7 @@ failed_mount:
vfree(msblk->read_page);
squashfs_cache_delete(msblk->block_cache);
kfree(msblk->fragment_index_2);
- vfree(msblk->stream.workspace);
+ crypto_free_comp(msblk->tfm);
kfree(s->s_fs_info);
s->s_fs_info = NULL;
return -EINVAL;
@@ -2084,7 +2087,7 @@ static void squashfs_put_super(struct su
kfree(sbi->fragment_index);
kfree(sbi->fragment_index_2);
kfree(sbi->meta_index);
- vfree(sbi->stream.workspace);
+ crypto_free_comp(sbi->tfm);
kfree(s->s_fs_info);
s->s_fs_info = NULL;
}
--- a/include/linux/squashfs_fs_sb.h
+++ b/include/linux/squashfs_fs_sb.h
@@ -24,6 +24,7 @@
*/
#include <linux/squashfs_fs.h>
+#include <linux/crypto.h>
struct squashfs_cache_entry {
long long block;
@@ -67,7 +68,7 @@ struct squashfs_sb_info {
struct mutex read_page_mutex;
struct mutex meta_index_mutex;
struct meta_index *meta_index;
- z_stream stream;
+ struct crypto_comp *tfm;
long long *inode_lookup_table;
int (*read_inode)(struct inode *i, squashfs_inode_t \
inode);
--
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 10/10] AXFS: axfs_uncompress.c
From: Geert Uytterhoeven @ 2008-08-29 13:48 UTC (permalink / raw)
To: Jared Hulbert
Cc: Artem Bityutskiy, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <6934efce0808210735w479075abga9f660e42ab8935d@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1113 bytes --]
Hi Jared,
On Thu, 21 Aug 2008, Jared Hulbert wrote:
> > If you want support for multiple decompression algorithms, you can switch
> > from using zlib_inflate*() directly to calling zlib through the crypto API.
> > Then you can call crypto_alloc_comp() with the correct decompression algorithm
> > name.
> >
> > For squashfs, I had to modify only ca. 40 lines of code.
>
> I definately want to support multiple formats. I have a flag in the
> superblock all ready to go for that. One problem. I'm not sure how
> to do it. Can you point us to some reference code?
Please take a look at the email with subject
[patch 3/3] squashfs: Switch from zlib/inflate to "zlib" crypto module
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* Re: [patch 1/3] crypto: Add a zlib crypto module
From: Herbert Xu @ 2008-08-30 6:23 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: linux-crypto, squashfs-devel, linux-embedded, linux-kernel
In-Reply-To: <20080829134418.874948211@vixen.sonytel.be>
On Fri, Aug 29, 2008 at 01:41:59PM +0000, Geert Uytterhoeven wrote:
> From: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
>
> Add a (de)compression module for the "zlib" format using the crypto API.
I think we can safely conclude that our current compression
interface sucks for what you're trying to achieve :)
> While both the "zlib" and "deflate" crypto modules are implemented on top of
> the zlib library, they differ in the following aspects:
> - The "deflate" crypto module (used by IPSec and UBIFS) does not support
> partial decompression, i.e. all compressed data has to be passed at once.
> The "zlib" crypto module does support partial decompression;
> zlib_decompress() will return -EAGAIN if not all compressed data has been
> passed.
> - The deflate crypto module uses the raw deflate data format (zlib is
> initialized with a windowBits parameter of -DEFLATE_DEF_WINBITS = -11),
> while e.g. squashfs and axfs use the zlib data format, with the default
> windowBits parameter DEF_WBITS = 15.
> Both parameters are incompatible with each other due to the different data
> formats, as indicated by the sign of the windowbits parameter.
> The absolute value of this parameter is the base two logarithm of the
> maximum window size, and larger values are backwards compatible with
> smaller values (as far as decompression is concerned).
Therefore I suggest that we change the interface rather than
mutilate the implementations.
So here are a few things we should add:
1) Separate alloc functions for comp/decomp to avoid the memory
wastage you've identified.
2) Provide parameters to these alloc functions through an opaque
pointer. The format of the paramters will be determined by the
name of the algorithm, i.e., if you want to change the parameters
then you should change the name as well (e.g., deflate => deflate2).
This removes the need to dupliate the implementation just because
you want 15 instead of -11.
3) Provide init/update/final (one set each for comp and decomp)
functions similar to crypto_hash, in addition to the current
comp/decomp functions.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch 1/3] crypto: Add a zlib crypto module
From: David Woodhouse @ 2008-08-30 6:39 UTC (permalink / raw)
To: Herbert Xu
Cc: Geert Uytterhoeven, linux-crypto, squashfs-devel, linux-embedded,
linux-kernel
In-Reply-To: <20080830062315.GA18441@gondor.apana.org.au>
On Sat, 2008-08-30 at 16:23 +1000, Herbert Xu wrote:
> On Fri, Aug 29, 2008 at 01:41:59PM +0000, Geert Uytterhoeven wrote:
> > From: Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com>
> >
> > Add a (de)compression module for the "zlib" format using the crypto API.
>
> I think we can safely conclude that our current compression
> interface sucks for what you're trying to achieve :)
The main thing that's missing for JFFS2 is
"Compress as much of this as you can into X bytes"
--
David Woodhouse Open Source Technology Centre
David.Woodhouse@intel.com Intel Corporation
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Geert Uytterhoeven @ 2008-09-02 15:37 UTC (permalink / raw)
To: Jared Hulbert
Cc: Linux-kernel, linux-embedded, linux-mtd, Jörn Engel,
tim.bird, cotte, nickpiggin
In-Reply-To: <Pine.LNX.4.64.0808281737150.28552@vixen.sonytel.be>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1925 bytes --]
On Thu, 28 Aug 2008, Geert Uytterhoeven wrote:
> On Fri, 22 Aug 2008, Jared Hulbert wrote:
> > > The version in SVN seems to be slightly older than the one you submitted?
> >
> > Oops. Okay I must have neglected to sync at the very end. Thanks.
> >
> > I forgot, there is also a git repo at
> >
> > git://git.infradead.org/users/jehulber/axfs.git
> >
> > > Which platform(s) do you use for testing?
> >
> > ARM, x86
> >
> > > I gave AxFS a try on PS3 (ppc64, always use big-endian 64-bit for testing new
> > > code ;-).
> >
> > Smart. Hmmm, If only I had a PS3....
>
> I heard you got one? ;-)
>
> > > When mounting the image, I got the crash below:
> > >
> > > | attempt to access beyond end of device
> > > | loop0: rw=0, want=4920, limit=4912
>
> Interestingly, it also doesn't work on UserModeLinux (x86, 32-bit):
I found what's wrong.
The size of an AxFS image created by mkfs.axfs is always n*4096+4 bytes large.
So when it wants to check the magic value in the last 4 bytes, the block layer
tries to read a whole 512-byte sector, which fails for loop-mounted images.
If you test on real FLASH, additional bytes after the end of the AxFS image can
be read, hence it works.
By padding the image with 508 zero bytes, I can mount it, on both PS3 (ppc64)
and UML (ai32). I can even read images created on PS3.
However, there still are weird things going on, like `find' not seeing all
files and directories, or just aborting, and `ls -lR' showing actual file
contents in its output.
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jared Hulbert @ 2008-09-02 16:44 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Linux-kernel, linux-embedded, linux-mtd, Jörn Engel,
tim.bird, cotte, nickpiggin
In-Reply-To: <Pine.LNX.4.64.0809021725480.3986@vixen.sonytel.be>
> I found what's wrong.
>
> The size of an AxFS image created by mkfs.axfs is always n*4096+4 bytes large.
> So when it wants to check the magic value in the last 4 bytes, the block layer
> tries to read a whole 512-byte sector, which fails for loop-mounted images.
>
> If you test on real FLASH, additional bytes after the end of the AxFS image can
> be read, hence it works.
>
> By padding the image with 508 zero bytes, I can mount it, on both PS3 (ppc64)
> and UML (ai32). I can even read images created on PS3.
Right. We haven't tested loopback since we added the magic end value.
How is one expected to read those last 4 bytes of a loopbacked file?
Are they unreadable? We can add the padding. I am just wondering if
this is a bug or a known limitation in the loopback handling or if
there is a different safer way of reading block devs with truncated
last blocks.
> However, there still are weird things going on, like `find' not seeing all
> files and directories, or just aborting, and `ls -lR' showing actual file
> contents in its output.
Do you see this behavior for all builds for just the PS3?
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jörn Engel @ 2008-09-02 17:15 UTC (permalink / raw)
To: Jared Hulbert
Cc: Geert Uytterhoeven, Linux-kernel, linux-embedded, linux-mtd,
tim.bird, cotte, nickpiggin
In-Reply-To: <6934efce0809020944s36dd7f82wec77c4189ef1b114@mail.gmail.com>
On Tue, 2 September 2008 09:44:19 -0700, Jared Hulbert wrote:
>
> How is one expected to read those last 4 bytes of a loopbacked file?
> Are they unreadable? We can add the padding. I am just wondering if
> this is a bug or a known limitation in the loopback handling or if
> there is a different safer way of reading block devs with truncated
> last blocks.
Can't you just include the final magic into the last block, thereby
making the size a clean multiple of 4k? It looks as if you have some
padding before the magic anyway. So you just have to make sure the
padding is at least 4 bytes and write the magic to the end of it. Apart
from solving this bug, it should also save you some space. ;)
Jörn
--
Invincibility is in oneself, vulnerability is in the opponent.
-- Sun Tzu
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jared Hulbert @ 2008-09-02 17:47 UTC (permalink / raw)
To: Jörn Engel
Cc: Geert Uytterhoeven, Linux-kernel, linux-embedded, linux-mtd,
tim.bird, cotte, nickpiggin
In-Reply-To: <20080902171500.GC19618@logfs.org>
>> How is one expected to read those last 4 bytes of a loopbacked file?
>> Are they unreadable? We can add the padding. I am just wondering if
>> this is a bug or a known limitation in the loopback handling or if
>> there is a different safer way of reading block devs with truncated
>> last blocks.
>
> Can't you just include the final magic into the last block, thereby
> making the size a clean multiple of 4k? It looks as if you have some
> padding before the magic anyway. So you just have to make sure the
> padding is at least 4 bytes and write the magic to the end of it. Apart
> from solving this bug, it should also save you some space. ;)
I'm going to have to look into this n*4K thing. The image doesn't
need to be aligned. There shouldn't be any last block to put the
magic in. But I haven't messed with the mkfs.axfs code for a while.
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Geert Uytterhoeven @ 2008-09-02 18:33 UTC (permalink / raw)
To: Jared Hulbert
Cc: Linux Kernel Development, linux-embedded, linux-mtd,
Jörn Engel, Tim Bird, cotte, nickpiggin
In-Reply-To: <6934efce0809020944s36dd7f82wec77c4189ef1b114@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 820 bytes --]
On Tue, 2 Sep 2008, Jared Hulbert wrote:
> > However, there still are weird things going on, like `find' not seeing all
> > files and directories, or just aborting, and `ls -lR' showing actual file
> > contents in its output.
>
> Do you see this behavior for all builds for just the PS3?
The `find' issue I saw on both PS3 (ppc64) and UML (ia32).
The `ls -lR' I tried on UML (ia32) only for now.
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
^ permalink raw reply
* Re: UIO not working on ppc405 onchip registers
From: Markus Brunner @ 2008-09-05 6:18 UTC (permalink / raw)
To: Ben Nizette; +Cc: linuxppc-dev@ozlabs.org, linux-embedded@vger.kernel.org
In-Reply-To: <1216713170.4004.243.camel@moss.renham>
On Tuesday 22 July 2008, Ben Nizette wrote:
> But I'll let you get back to solving the UIO problem at hand :-D
I already surrendered and created (hacked) a read/write driver, which let me
use the old userspace drivers I'm trying to port (with some changes of
course). This was way faster than understanding all the involved APIs and
creating glue code between them. And the amount of time it needs was easy to
estimate.
However accidently I ran over the "old school" userland mmap code
with /dev/mem. I already knew mapping with /dev/mem is possible but I haven't
thought this could make a difference.
http://ozlabs.org/pipermail/linuxppc-embedded/2006-August/023811.html
Also I was wrong with my assumption that uio worked well on the peripherals.
It worked (very) well on the leds, just to fool me!!! But it returned crap on
some version registers.
With a working and a non working version of mmap it should be rather easy to
trace this bug. Unfortunately I was already short of time before running into
this (and other) problems and it didn't get any better.
So I would need some help from someone with more experience to fix this.
Any instructions?
Markus
^ permalink raw reply
* Re: [Squashfs-devel] [patch 0/3] [RFC] zlib crypto module
From: Phillip Lougher @ 2008-09-06 4:41 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: linux-crypto, squashfs-devel, linux-kernel, linux-embedded
In-Reply-To: <20080829134158.108976037@vixen.sonytel.be>
Geert Uytterhoeven wrote:
> These patches add a (de)compression module for the "zlib" format using the
> crypto API:
> [1] crypto: Add a zlib crypto module
> [2] tcrypt: Add a self test for the zlib crypto module
> [3] squashfs: Switch from zlib/inflate to "zlib" crypto module
>
> The last patch is a proof-of-concept to make SquashFS 3.4 use this new zlib
> crypto module. This makes it easier to e.g. change the decompression algorithm
> in SquashFS or to make use of a hardware-accelerated zlib crypto module.
>
> It can be extended to other compressed file systems, like e.g. AxFS and CRAMFS.
>
> All comments are welcome. Thanks!
>
Moving Squashfs over to the crypto API is a good idea. When the zlib
crypto module code is in the mainline kernel moving Squashfs, AxFS and
CRAMFS over will be easy.
Are you planning to do the necessary work to get this (or a subsequent
version following the comments from Herbert Xu) into mainline?
Cheers
Phillip
^ permalink raw reply
* [PATCH] Add better resolution to initcall_debug timings
From: Tim Bird @ 2008-09-12 19:51 UTC (permalink / raw)
To: linux-embedded; +Cc: linux kernel
Change the time resolution for initcall_debug to microseconds,
from milliseconds. This is handy to determine which
initcalls you want to work on for faster booting.
One one of my test machines, over 90% of the initcalls are
less than a millisecond and (without this patch) these are
all reported as 0 msecs. Working on the 900 us ones is more
important than the 4 us ones.
With 'quiet' on the kernel command line, this adds no significant
overhead to kernel boot time.
Signed-off-by: Tim Bird <tim.bird@am.sony.com>
---
init/main.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/init/main.c b/init/main.c
index bdc8be0..f441cad 100644
--- a/init/main.c
+++ b/init/main.c
@@ -719,8 +719,8 @@ int do_one_initcall(initcall_t fn)
delta = ktime_sub(t1, t0);
print_fn_descriptor_symbol("initcall %s", fn);
- printk(" returned %d after %Ld msecs\n", result,
- (unsigned long long) delta.tv64 >> 20);
+ printk(" returned %d after %Ld usecs\n", result,
+ (unsigned long long) delta.tv64 >> 10);
}
msgbuf[0] = 0;
--
1.5.6
^ permalink raw reply related
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jared Hulbert @ 2008-09-12 20:17 UTC (permalink / raw)
To: Jamie Lokier
Cc: Greg Ungerer, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <20080822183713.GC24179@shareable.org>
> I think the "fast" in "fast synchronous" gives it away :-)
Yes, I suppose it does.
> I'm using Spansion MirrorBit S29GL128N, which reads at about 0.6 MByte/s.
I think you should get more like an order of magnitude higher.... Get
an expert to look at your timings in the bootloader. Make sure things
are cached too. ioremap_cached()...
> Not because they're good, but because that's what the board I'm coding
> for has on it. I presume they were cheap and familiar to the board
> designers. (There is 32MB of RAM to play with after all.)
>
> So start a sequence of Busybox processes from a shell script is noticable,
> if it reads from NOR each time.
>
> Oh, and it's a 166MHz ARM, so it's quite capable of decompressing
> faster than the NOR can deliver.
Depends on how you are measuring it. You ought to be able to get at
least 2 orders of magnitude higher read speeds with a good sync Flash.
Some of the newer stuff is even faster.
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jared Hulbert @ 2008-09-12 21:52 UTC (permalink / raw)
To: Jamie Lokier
Cc: Greg Ungerer, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <20080822184326.GE24179@shareable.org>
On Fri, Aug 22, 2008 at 11:43 AM, Jamie Lokier <jamie@shareable.org> wrote:
> Jamie Lokier wrote:
>> Jared Hulbert wrote:
>> > What kind of NOR you using? That is not what I measure with fast
>> > synchronous burst NOR's.
>>
>> I think the "fast" in "fast synchronous" gives it away :-)
>>
>> I'm using Spansion MirrorBit S29GL128N, which reads at about 0.6 MByte/s.
>
> By the way, what speeds do you get on fast synchronous burst NORs - and
> which chips are those?
I am only familiar with the Numonyx product line up.
If you are using a GL, you'll probably find our P33 a good fit and at
competitive prices to GL as I understand it. That's I think 50Mhz.
M18 is 100Mhz maybe a little higher. And we just announced our LPDDR
266 part, Velocity LP.
A good way of making a rough estimate of read performance is to
measure a cache miss latency and convert that to bandwidth. It's
usually fairly close.
32Byte cache size / 16 bit bus = 16 word transfers
memory controller latency (time from Load instruction to bus activity)
= ~300ns (upto 450ns for some processors)
initial latency (time to read first word) = ~100ns (60ns - 120ns)
clock frequency (time between words) = 50Mhz = 20ns per word
bus clean up = ~50ns
32Bytes = 300ns + 100ns + (16 - 1) * 20ns + 50ns = 750ns
= 32B/750ns
= 40MB/s
This is a very simple model and reality is much more nuanced. You
also need to check my assume numbers with the reality of your system.
Also this doesn't take copying the data to RAM into account which is
usually what you are really measuring. That's easy to model though.
A rule of thumb is to say that copying to RAM will reduce this value
less than 50% because RAM should be at least a little faster than NOR.
Nevertheless, If you can't use a simple calculation like this to
explain the numbers then you have a poorly configured bus timings or
have your cache off. Cache is important because it would make the
equation look like this, or worse.
4Bytes = 300ns + 100ns + 1 * 20ns + 50ns = 470ns
=4B/470ns
=8MB/s
For a PXA270 if you go with the defaults it can look like this.
2Bytes = 300ns + 250ns + 250ns = 800ns
= 2MB/s
So, If you are only getting 0.6MB/s out of your NOR..... You're using it wrong.
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jamie Lokier @ 2008-09-15 16:34 UTC (permalink / raw)
To: Jared Hulbert
Cc: Greg Ungerer, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <6934efce0809121452kd20deb6m268715f6ed83a71e@mail.gmail.com>
Jared Hulbert wrote:
> >> > What kind of NOR you using? That is not what I measure with fast
> >> > synchronous burst NOR's.
> >>
> >> I think the "fast" in "fast synchronous" gives it away :-)
> >>
> >> I'm using Spansion MirrorBit S29GL128N, which reads at about 0.6 MByte/s.
> >
> > By the way, what speeds do you get on fast synchronous burst NORs - and
> > which chips are those?
>
> I am only familiar with the Numonyx product line up.
>
> If you are using a GL, you'll probably find our P33 a good fit and at
> competitive prices to GL as I understand it. That's I think 50Mhz.
What's a GL? Never heard of it - all I can think of is OpenGL :-)
I'm using a Sigma Designs 862x media processor. It clocks at 166MHz
to main RAM, has an ARM internally to run Linux, and the intensive
work happens in coprocessors. The NOR is not on the RAM bus, it's on
a "peripheral bus". About the only thing I know about the bus is it's
16 bits wide - I have the schematic, but only the board supplier has
access to Sigma chip documentation.
> A rule of thumb is to say that copying to RAM will reduce this value
> less than 50% because RAM should be at least a little faster than NOR.
In our case, RAM is at least 100x faster :-)
> Nevertheless, If you can't use a simple calculation like this to
> explain the numbers then you have a poorly configured bus timings or
> have your cache off.
I'm not sure if cache is an option with this device - but would it
make a difference anyway? Launching executables like Busybox - those
are much larger than the cache anyway, so launch time is dominated by
bulk streaming copy speed. Thanks for the idea, I'll look into
whether it's possible to access this 'peripheral bus' through the
ARM's cache and see if that speeds up streaming read time.
> So, If you are only getting 0.6MB/s out of your NOR..... You're
> using it wrong.
Interesting, thanks. I'm not sure it's possible to change the way NOR
is being used with this chip, and it'll be a while before it's
economical to replace the board with a new design.
This is all very interesting - I had no prior experience with NOR, so
didn't know that 0.6MB/s was slow. It's fast compared with older
EEPROMs after all, and had imagined that people wanting fast flash
would use NAND.
On looking at the datasheet, I see it's quite a lot faster. I'm
suspecting the Sigma Designs perpheral bus and the way it's wired up
not doing it any favours. We already have the weirdness that we have
to patch the Linux CFI-0002 MTD code: the CPU locks up when polling
the erase status byte, until erase is finished. Unfortunately this is
difficult to change now - I'm programming hardware which is already
out in the field and cannot be redesigned.
Thanks for your thoughts.
-- Jamie
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jamie Lokier @ 2008-09-15 16:40 UTC (permalink / raw)
To: Jared Hulbert
Cc: Greg Ungerer, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <6934efce0809121317r418c80e8s4755669cc74975c4@mail.gmail.com>
Jared Hulbert wrote:
> > I think the "fast" in "fast synchronous" gives it away :-)
>
> Yes, I suppose it does.
>
> > I'm using Spansion MirrorBit S29GL128N, which reads at about 0.6 MByte/s.
>
> I think you should get more like an order of magnitude higher.... Get
> an expert to look at your timings in the bootloader. Make sure things
> are cached too. ioremap_cached()...
Yes, looking at the Spansion datasheet, if it were interfaced properly
it should be quite fast. (25ns access time for in-page 16-bit reads,
100ns for random reads).
I'll see if ioremap_cached() makes a difference to streaming read
performance.
The BSP suppliers have been quite cautious in places, flushing cache a
bit too often. (I'm not surprised - we had disk ext3 filesystem
corruption due to insufficient cache flushing in places too.)
> > Oh, and it's a 166MHz ARM, so it's quite capable of decompressing
> > faster than the NOR can deliver.
>
> Depends on how you are measuring it. You ought to be able to get at
> least 2 orders of magnitude higher read speeds with a good sync Flash.
> Some of the newer stuff is even faster.
Thanks.
Oh, how I look forward to the day of working with current kernels and
current hardware.
-- Jamie
^ permalink raw reply
* Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
From: Jared Hulbert @ 2008-09-15 19:43 UTC (permalink / raw)
To: Jamie Lokier
Cc: Greg Ungerer, Linux-kernel, linux-embedded, linux-mtd,
Jörn Engel, tim.bird, cotte, nickpiggin
In-Reply-To: <20080915163421.GA13631@shareable.org>
>> >> I'm using Spansion MirrorBit S29GL128N, which reads at about 0.6 MByte/s.
>>
>> If you are using a GL, you'll probably find our P33 a good fit and at
>> competitive prices to GL as I understand it. That's I think 50Mhz.
>
> What's a GL? Never heard of it - all I can think of is OpenGL :-)
S29_GL_128N
> I'm not sure if cache is an option with this device - but would it
> make a difference anyway?
Well the first read takes 100ns (plus the other chipset overhead
300ns) but other reads in a page are only an extra 25ns each. So your
benefit is not from having the entire executable in cache it's from
having the next 7 instructions in the cacheline for only an extra 25ns
each instead of 400ns.
> Interesting, thanks. I'm not sure it's possible to change the way NOR
> is being used with this chip, and it'll be a while before it's
> economical to replace the board with a new design.
Usually these things can be fixed in the bootloader or by hacking the
kernel to tweak the relevant chipset registers.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox