ARM: add xz kernel compression support

ARM: add xz kernel compression support

[Michael Opdenacker]

This patchset applies on Linus' tree

It adds xz support to the ARM tree. x86 already has it, so let's have it too!
My benchmarks below show that xz is better than lzma in terms of image size
and uncompression time.

My benchmarks below show that xz is better than lzma in terms of image size
and uncompression time.

lzma and xz remain pretty bad in terms of boot time, but they may be useful
in cases where RAM usage and kernel size are critical (booting on FPGAs in early
CPU design?).

However, on fast CPUs like OMAP4 with slow storage, xz is almost as good as
lzo, but uses much less space. Therefore, it can be a good compromise.

This patch and all other compression options were tested on two boards:
- OMAP4430 Panda board
- AT91SAM9623 USB-A9263 board from Calao Systems

Results on OMAP4 Panda
----------------------

Booting from MMC

Comp.           Size            Loading time    Uncompressing time      Total time
gzip            3355768         2.213376        0.501500                2.714876
lzma            2488144         1.647410        1.399552                3.046962
xz              2366192         1.566978        1.299516                2.866494
lzo             3697840         2.471497        0.160596                2.632093
None            6965644         4.626749        0                       4.626749

Results on Calao Systems USB-A9263 (AT91)
-----------------------------------------

Booting from NAND

Compression     Size            Loading time    Uncompressing time      Total time
gzip            2386936         5.843289        0.935495                6.778784
lzma            1794344         4.465542        6.513644                10.979186
xz              1725360         4.308605        4.816191                9.124796
lzo             2608624         6.351539        0.447336                6.798875
None            4647908         11.080560       0                       11.080560

Loading time: between "reading uImage" and "OK" (right before "Starting kernel")
Compression time measured by:
~/bin/grabserial -v -d /dev/ttyUSB0 -e 15 -t -m "Uncompressing Linux" -i "done," > booting-lzo.log

[PATCH 1/2] ARM: remove memmove from boot/compressed/string.c

[Michael Opdenacker]

This patch removes the memmove function from boot/compressed/string.c
- This definition conflicts with the one in lib/decompress_unxz.c
  This change is required to support xz compression on ARM.
- memmove is not used in any of the other decompressors
  in lib/decompress_*.c

Signed-off-by: Michael Opdenacker <michael.opdenacker@free-electrons.com>
---
 arch/arm/boot/compressed/string.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/arm/boot/compressed/string.c b/arch/arm/boot/compressed/string.c
index 36e53ef..369443a 100644
--- a/arch/arm/boot/compressed/string.c
+++ b/arch/arm/boot/compressed/string.c
@@ -40,6 +40,7 @@ void *memcpy(void *__dest, __const void *__src, size_t __n)
 	return __dest;
 }
 
+/*
 void *memmove(void *__dest, __const void *__src, size_t count)
 {
 	unsigned char *d = __dest;
@@ -55,6 +56,7 @@ void *memmove(void *__dest, __const void *__src, size_t count)
 		d[count] = s[count];
 	return __dest;
 }
+*/
 
 size_t strlen(const char *s)
 {
-- 
1.7.4.1

[PATCH 1/2] ARM: remove memmove from boot/compressed/string.c

[Jean-Christophe PLAGNIOL-VILLARD]

On 15:54 Thu 16 Feb     , Michael Opdenacker wrote:
> This patch removes the memmove function from boot/compressed/string.c
> - This definition conflicts with the one in lib/decompress_unxz.c
>   This change is required to support xz compression on ARM.
> - memmove is not used in any of the other decompressors
>   in lib/decompress_*.c
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@free-electrons.com>
> ---
>  arch/arm/boot/compressed/string.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/arm/boot/compressed/string.c b/arch/arm/boot/compressed/string.c
> index 36e53ef..369443a 100644
> --- a/arch/arm/boot/compressed/string.c
> +++ b/arch/arm/boot/compressed/string.c
> @@ -40,6 +40,7 @@ void *memcpy(void *__dest, __const void *__src, size_t __n)
>  	return __dest;
>  }
>  
> +/*
>  void *memmove(void *__dest, __const void *__src, size_t count)
>  {
>  	unsigned char *d = __dest;
> @@ -55,6 +56,7 @@ void *memmove(void *__dest, __const void *__src, size_t count)
>  		d[count] = s[count];
>  	return __dest;
>  }
> +*/
so why not drop it cleanly

Best Regards,
J.

[PATCH 1/2] ARM: remove memmove from boot/compressed/string.c

[Michael Opdenacker]

Hi Jean-Christophe,

On 02/16/2012 04:09 PM, Jean-Christophe PLAGNIOL-VILLARD wrote:
> On 15:54 Thu 16 Feb     , Michael Opdenacker wrote:
>> This patch removes the memmove function from boot/compressed/string.c
>> - This definition conflicts with the one in lib/decompress_unxz.c
>>   This change is required to support xz compression on ARM.
>> - memmove is not used in any of the other decompressors
>>   in lib/decompress_*.c
>>
>> Signed-off-by: Michael Opdenacker <michael.opdenacker@free-electrons.com>
>> ---
>>  arch/arm/boot/compressed/string.c |    2 ++
>>  1 files changed, 2 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/arm/boot/compressed/string.c b/arch/arm/boot/compressed/string.c
>> index 36e53ef..369443a 100644
>> --- a/arch/arm/boot/compressed/string.c
>> +++ b/arch/arm/boot/compressed/string.c
>> @@ -40,6 +40,7 @@ void *memcpy(void *__dest, __const void *__src, size_t __n)
>>  	return __dest;
>>  }
>>  
>> +/*
>>  void *memmove(void *__dest, __const void *__src, size_t count)
>>  {
>>  	unsigned char *d = __dest;
>> @@ -55,6 +56,7 @@ void *memmove(void *__dest, __const void *__src, size_t count)
>>  		d[count] = s[count];
>>  	return __dest;
>>  }
>> +*/
> so why not drop it cleanly
Oooops. Well spotted. You no longer see the obviously stupid stuff in
your own code.

I'll send an update soon.

Thanks!

Michael.

-- 
Michael Opdenacker, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
+33 484 253 396

[PATCH 1/2] ARM: remove memmove from boot/compressed/string.c

[Nicolas Pitre]

On Thu, 16 Feb 2012, Michael Opdenacker wrote:

> This patch removes the memmove function from boot/compressed/string.c

... breaking CONFIG_ARM_ATAG_DTB_COMPAT.

> - This definition conflicts with the one in lib/decompress_unxz.c
>   This change is required to support xz compression on ARM.

No, it's the decompress_unxz.c code which is wrong.

To work around it, just do:

#define memmove memmove

before including lib/decompress_unxz.c.

> - memmove is not used in any of the other decompressors
>   in lib/decompress_*.c

Doesn't mean it isn't used at all.

Here's a trick.  Next time you can do:

	git blame arch/arm/boot/compressed/string.c

That will list all the file content, annotated with references to the 
commits that brought them there.

Looking for memmove() you'll see that commit df4879fa260 is responsible 
for it. Its commit log says:

    ARM: zImage: gather some string functions into string.c

    This is a small subset of string functions needed by commits to 
    come.

OK, doing a full 'git log' and searching for that commit, just to back 
up to the next one you'll see:

    ARM: zImage: allow supplementing appended DTB with traditional ATAG  data

That explains why memmove() is required, and by which code.


Nicolas

[PATCH 2/2] ARM: add support for xz kernel decompression

[Michael Opdenacker]

Changes applied on Linus' tree

This patch adds support for xz compression on the ARM architecture.
xz achieves slighly higher compression than LZMA.

Signed-off-by: Michael Opdenacker <michael.opdenacker@free-electrons.com>
---
 arch/arm/Kconfig                        |    1 +
 arch/arm/boot/compressed/.gitignore     |    1 +
 arch/arm/boot/compressed/Makefile       |    3 ++-
 arch/arm/boot/compressed/decompress.c   |    4 ++++
 arch/arm/boot/compressed/piggy.xzkern.S |    6 ++++++
 5 files changed, 14 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/boot/compressed/piggy.xzkern.S

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a48aecc..c34ff1e 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -21,6 +21,7 @@ config ARM
 	select HAVE_KERNEL_GZIP
 	select HAVE_KERNEL_LZO
 	select HAVE_KERNEL_LZMA
+	select HAVE_KERNEL_XZ
 	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select PERF_USE_VMALLOC
diff --git a/arch/arm/boot/compressed/.gitignore b/arch/arm/boot/compressed/.gitignore
index e0936a1..d1977f4 100644
--- a/arch/arm/boot/compressed/.gitignore
+++ b/arch/arm/boot/compressed/.gitignore
@@ -3,6 +3,7 @@ lib1funcs.S
 piggy.gzip
 piggy.lzo
 piggy.lzma
+piggy.xzkern
 vmlinux
 vmlinux.lds
 
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index cf0a64c..5a2972b 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -92,6 +92,7 @@ SEDFLAGS	= s/TEXT_START/$(ZTEXTADDR)/;s/BSS_START/$(ZBSSADDR)/
 suffix_$(CONFIG_KERNEL_GZIP) = gzip
 suffix_$(CONFIG_KERNEL_LZO)  = lzo
 suffix_$(CONFIG_KERNEL_LZMA) = lzma
+suffix_$(CONFIG_KERNEL_XZ) = xzkern
 
 # Borrowed libfdt files for the ATAG compatibility mode
 
@@ -115,7 +116,7 @@ targets       := vmlinux vmlinux.lds \
 		 lib1funcs.o lib1funcs.S font.o font.c head.o misc.o $(OBJS)
 
 # Make sure files are removed during clean
-extra-y       += piggy.gzip piggy.lzo piggy.lzma lib1funcs.S $(libfdt) $(libfdt_hdrs)
+extra-y       += piggy.gzip piggy.lzo piggy.lzma piggy.xz lib1funcs.S $(libfdt) $(libfdt_hdrs)
 
 ifeq ($(CONFIG_FUNCTION_TRACER),y)
 ORIG_CFLAGS := $(KBUILD_CFLAGS)
diff --git a/arch/arm/boot/compressed/decompress.c b/arch/arm/boot/compressed/decompress.c
index 07be5a2..0ecd8b4 100644
--- a/arch/arm/boot/compressed/decompress.c
+++ b/arch/arm/boot/compressed/decompress.c
@@ -44,6 +44,10 @@ extern void error(char *);
 #include "../../../../lib/decompress_unlzma.c"
 #endif
 
+#ifdef CONFIG_KERNEL_XZ
+#include "../../../../lib/decompress_unxz.c"
+#endif
+
 int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x))
 {
 	return decompress(input, len, NULL, NULL, output, NULL, error);
diff --git a/arch/arm/boot/compressed/piggy.xzkern.S b/arch/arm/boot/compressed/piggy.xzkern.S
new file mode 100644
index 0000000..5703f30
--- /dev/null
+++ b/arch/arm/boot/compressed/piggy.xzkern.S
@@ -0,0 +1,6 @@
+	.section .piggydata,#alloc
+	.globl	input_data
+input_data:
+	.incbin	"arch/arm/boot/compressed/piggy.xzkern"
+	.globl	input_data_end
+input_data_end:
-- 
1.7.4.1

ARM: add xz kernel compression support

[Veli-Pekka Peltola]

Hi,

On 02/16/2012 04:54 PM, Michael Opdenacker wrote:
> It adds xz support to the ARM tree. x86 already has it, so let's
> have it too!

XZ support is already applied to Russell's for-next through patch
tracking system [1].

[1]:
http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=7001/2

--
Veli-Pekka Peltola

ARM: add xz kernel compression support

[Michael Opdenacker]

Hi Veli,

On 02/16/2012 04:37 PM, Veli-Pekka Peltola wrote:
> Hi,
>
> On 02/16/2012 04:54 PM, Michael Opdenacker wrote:
>> It adds xz support to the ARM tree. x86 already has it, so let's
>> have it too!
>
> XZ support is already applied to Russell's for-next through patch
> tracking system [1].
>
> [1]:
> http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=7001/2

Cool! Good to know that we already have this :-)
This was a good exercise anyway and not a complete waste of time as I
produced benchmarks in the process.

Thanks,

Michael.

-- 
Michael Opdenacker, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
+33 484 253 396

ARM: add xz kernel compression support

[Russell King - ARM Linux]

On Thu, Feb 16, 2012 at 03:54:15PM +0100, Michael Opdenacker wrote:
> This patchset applies on Linus' tree
> 
> It adds xz support to the ARM tree. x86 already has it, so let's have it too!
> My benchmarks below show that xz is better than lzma in terms of image size
> and uncompression time.

Have you checked what's in my tree?  Have you checked patch 7001/2 ?

ARM: add xz kernel compression support

[Michael Opdenacker]

On 02/16/2012 04:55 PM, Russell King - ARM Linux wrote:
> On Thu, Feb 16, 2012 at 03:54:15PM +0100, Michael Opdenacker wrote:
>> This patchset applies on Linus' tree
>>
>> It adds xz support to the ARM tree. x86 already has it, so let's have it too!
>> My benchmarks below show that xz is better than lzma in terms of image size
>> and uncompression time.
> Have you checked what's in my tree?  Have you checked patch 7001/2 ?
No, I haven't. Promise to do my homework better next time ;-)
Thanks for your time anyway.

Cheers,

Michael.

-- 
Michael Opdenacker, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
+33 484 253 396