Re: [PATCH 0/3] Update LZO compression

Re: [PATCH 0/3] Update LZO compression

[Seth Jennings]

> As requested by akpm I am sending my "lzo-update" branch at
> 
>   git://github.com/markus-oberhumer/linux.git lzo-update
> 
> to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".
> 
> You can also browse the branch at
> 
>   https://github.com/markus-oberhumer/linux/compare/lzo-update
> 
> and review the three patches at
> 
>   https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
>   https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
>   https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1

As this relates to my work on zcache, I just tested these patches on PPC64 and
they cause the LZO crypto module to fail its self-test:

[    0.521137] alg: comp: Compression test 1 failed for lzo-generic: output len = 62

I built the exact same kernel for x86_64 and all is fine.  I suspect an endianness
related bug, but I haven't looked at the code that closely yet.

Any ideas?  I'd be happy to test any potential fixes.

Seth

Re: [PATCH 0/3] Update LZO compression

[Markus F.X.J. Oberhumer]

On 2012-10-15 21:19, Seth Jennings wrote:
>> As requested by akpm I am sending my "lzo-update" branch at
>>
>>   git://github.com/markus-oberhumer/linux.git lzo-update
>>
>> to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".
>>
>> You can also browse the branch at
>>
>>   https://github.com/markus-oberhumer/linux/compare/lzo-update
>>
>> and review the three patches at
>>
>>   https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
>>   https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
>>   https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1
> 
> As this relates to my work on zcache, I just tested these patches on PPC64 and
> they cause the LZO crypto module to fail its self-test:
> 
> [    0.521137] alg: comp: Compression test 1 failed for lzo-generic: output len = 62
> 
> I built the exact same kernel for x86_64 and all is fine.  I suspect an endianness
> related bug, but I haven't looked at the code that closely yet.
> 
> Any ideas?  I'd be happy to test any potential fixes.

The crypto LZO test vectors had to be updated - this should land in linux-next
soon (or you can just pull from my branch).

BTW, this cannot have worked on x86_64 (or any other arch), so you probably
tested the wrong kernel.

Cheers,
Markus

> Seth

-- 
Markus Oberhumer, <markus@oberhumer.com>, http://www.oberhumer.com/

Re: [PATCH 0/3] Update LZO compression

[Seth Jennings]

On 10/15/2012 06:45 PM, Markus F.X.J. Oberhumer wrote:
> The crypto LZO test vectors had to be updated - this should land in linux-next
> soon (or you can just pull from my branch).

Thanks, I pulled from your branch and now it works fine.

I noticed that you made the test vectors longer.  Any particular
reason? Is there a new minimum length now or something?

I guess I'm not understanding why a change to the test vectors was needed.

Seth

[PATCH 0/3] Update LZO compression

[Markus F.X.J. Oberhumer]

As requested by akpm I am sending my "lzo-update" branch at

  git://github.com/markus-oberhumer/linux.git lzo-update

to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".

You can also browse the branch at

  https://github.com/markus-oberhumer/linux/compare/lzo-update

and review the three patches at

  https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
  https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
  https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1

Share and enjoy,
Markus


Markus F.X.J. Oberhumer (3):
  lib/lzo: Rename lzo1x_decompress.c to lzo1x_decompress_safe.c
  lib/lzo: Update LZO compression to current upstream version
  lib/lzo: Optimize code for CPUs with inefficient unaligned access

 include/linux/lzo.h             |   15 +-
 lib/lzo/Makefile                |    2 +-
 lib/lzo/lzo1x_compress.c        |  309 +++++++++++++++++++++------------------
 lib/lzo/lzo1x_decompress.c      |  255 --------------------------------
 lib/lzo/lzo1x_decompress_safe.c |  237 ++++++++++++++++++++++++++++++
 lib/lzo/lzodefs.h               |   34 ++++-
 6 files changed, 441 insertions(+), 411 deletions(-)
 delete mode 100644 lib/lzo/lzo1x_decompress.c
 create mode 100644 lib/lzo/lzo1x_decompress_safe.c

Re: [PATCH 0/3] Update LZO compression

[Andrew Morton]

On Sun, 7 Oct 2012 17:07:55 +0200
"Markus F.X.J. Oberhumer" <markus@oberhumer.com> wrote:

> As requested by akpm I am sending my "lzo-update" branch at
> 
>   git://github.com/markus-oberhumer/linux.git lzo-update
> 
> to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".
> 
> You can also browse the branch at
> 
>   https://github.com/markus-oberhumer/linux/compare/lzo-update
> 
> and review the three patches at
> 
>   https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
>   https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
>   https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1

The changes look OK to me.  Please ask Stephen to include the tree in
linux-next, for a 3.7 merge.



The changelog for patch 2/3 says:

: This commit updates the kernel LZO code to the current upsteam version
: which features a significant speed improvement - benchmarking the Calgary
: and Silesia test corpora typically shows a doubled performance in
: both compression and decompression on modern i386/x86_64/powerpc machines.


There are significant clients of the LZO library - crypto, btrfs,
jffs2, ubifs, squashfs and zcache.  So let's give all those people a cc
and ask that they test the LZO changes once they land in linux-next. 
For correctness and performance, please.

Re: [PATCH 0/3] Update LZO compression

[Markus F.X.J. Oberhumer]

Hi Stephen,

On 2012-10-09 21:26, Andrew Morton wrote:
> On Sun, 7 Oct 2012 17:07:55 +0200
> "Markus F.X.J. Oberhumer" <markus@oberhumer.com> wrote:
> 
>> As requested by akpm I am sending my "lzo-update" branch at
>>
>>   git://github.com/markus-oberhumer/linux.git lzo-update
>>
>> to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".
>>
>> You can also browse the branch at
>>
>>   https://github.com/markus-oberhumer/linux/compare/lzo-update
>>
>> and review the three patches at
>>
>>   https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
>>   https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
>>   https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1
> 
> The changes look OK to me.  Please ask Stephen to include the tree in
> linux-next, for a 3.7 merge.

I'd ask you to include my "lzo-update" branch in linux-next:

  git://github.com/markus-oberhumer/linux.git lzo-update

> The changelog for patch 2/3 says:
> 
> : This commit updates the kernel LZO code to the current upsteam version
> : which features a significant speed improvement - benchmarking the Calgary
> : and Silesia test corpora typically shows a doubled performance in
> : both compression and decompression on modern i386/x86_64/powerpc machines.
> 
> There are significant clients of the LZO library - crypto, btrfs,
> jffs2, ubifs, squashfs and zcache.  So let's give all those people a cc
> and ask that they test the LZO changes once they land in linux-next. 
> For correctness and performance, please.

The core compression and decompression code has been thoroughly tested, so I
do not expect major problems.

Good testing after the merge and feedback about build or performance issues
(and improvements!) is highly appreciated.

Many thanks,
Markus

-- 
Markus Oberhumer, <markus@oberhumer.com>, http://www.oberhumer.com/

Re: [PATCH 0/3] Update LZO compression

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

Hi Markus,

On Tue, 09 Oct 2012 21:54:59 +0200 "Markus F.X.J. Oberhumer" <markus@oberhumer.com> wrote:
>
> On 2012-10-09 21:26, Andrew Morton wrote:
> > On Sun, 7 Oct 2012 17:07:55 +0200
> > "Markus F.X.J. Oberhumer" <markus@oberhumer.com> wrote:
> > 
> >> As requested by akpm I am sending my "lzo-update" branch at
> >>
> >>   git://github.com/markus-oberhumer/linux.git lzo-update
> >>
> >> to lkml as a patch series created by "git format-patch -M v3.5..lzo-update".
> >>
> >> You can also browse the branch at
> >>
> >>   https://github.com/markus-oberhumer/linux/compare/lzo-update
> >>
> >> and review the three patches at
> >>
> >>   https://github.com/markus-oberhumer/linux/commit/7c979cebc0f93dc692b734c12665a6824d219c20
> >>   https://github.com/markus-oberhumer/linux/commit/10f6781c8591fe5fe4c8c733131915e5ae057826
> >>   https://github.com/markus-oberhumer/linux/commit/5f702781f158cb59075cfa97e5c21f52275057f1
> > 
> > The changes look OK to me.  Please ask Stephen to include the tree in
> > linux-next, for a 3.7 merge.
> 
> I'd ask you to include my "lzo-update" branch in linux-next:
> 
>   git://github.com/markus-oberhumer/linux.git lzo-update

I have added this from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgment of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
	Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

Legal Stuff:
By participating in linux-next, your subsystem tree contributions are
public and will be included in the linux-next trees.  You may be sent
e-mail messages indicating errors or other issues when the
patches/commits from your subsystem tree are merged and tested in
linux-next.  These messages may also be cross-posted to the linux-next
mailing list, the linux-kernel mailing list, etc.  The linux-next tree
project and IBM (my employer) make no warranties regarding the linux-next
project, the testing procedures, the results, the e-mails, etc.  If you
don't agree to these ground rules, let me know and I'll remove your tree
from participation in linux-next.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 0/3] Update LZO compression

[Arnd Bergmann]

On Tuesday 09 October 2012, Markus F.X.J. Oberhumer wrote:
> > 
> > : This commit updates the kernel LZO code to the current upsteam version
> > : which features a significant speed improvement - benchmarking the Calgary
> > : and Silesia test corpora typically shows a doubled performance in
> > : both compression and decompression on modern i386/x86_64/powerpc machines.
> > 
> > There are significant clients of the LZO library - crypto, btrfs,
> > jffs2, ubifs, squashfs and zcache.  So let's give all those people a cc
> > and ask that they test the LZO changes once they land in linux-next. 
> > For correctness and performance, please.
> 
> The core compression and decompression code has been thoroughly tested, so I
> do not expect major problems.
> 
> Good testing after the merge and feedback about build or performance issues
> (and improvements!) is highly appreciated.

The addition of the lzo tree to linux-next caused this problem for ARM
imx_v6_v7_defconfig:

In file included from /home/arnd/linux-arm/arch/arm/boot/compressed/decompress.c:40:0:
/home/arnd/linux-arm/arch/arm/boot/compressed/../../../../lib/decompress_unlzo.c:34:34: fatal error: lzo/lzo1x_decompress.c: No such file or directory

Since the file was renamed, anything including it needs to be updated to the
new file name.

Signed-off-by: Arnd Bergmann <arnd@arndb.de>

diff --git a/lib/decompress_unlzo.c b/lib/decompress_unlzo.c
index 4531294..960183d 100644
--- a/lib/decompress_unlzo.c
+++ b/lib/decompress_unlzo.c
@@ -31,7 +31,7 @@
  */
 
 #ifdef STATIC
-#include "lzo/lzo1x_decompress.c"
+#include "lzo/lzo1x_decompress_safe.c"
 #else
 #include <linux/decompress/unlzo.h>
 #endif

Re: [PATCH 0/3] Update LZO compression

[Markus F.X.J. Oberhumer]

Thanks Arnd,

On 2012-10-11 13:41, Arnd Bergmann wrote:
> On Tuesday 09 October 2012, Markus F.X.J. Oberhumer wrote:
>>>
>>> : This commit updates the kernel LZO code to the current upsteam version
>>> : which features a significant speed improvement - benchmarking the Calgary
>>> : and Silesia test corpora typically shows a doubled performance in
>>> : both compression and decompression on modern i386/x86_64/powerpc machines.
>>>
>>> There are significant clients of the LZO library - crypto, btrfs,
>>> jffs2, ubifs, squashfs and zcache.  So let's give all those people a cc
>>> and ask that they test the LZO changes once they land in linux-next. 
>>> For correctness and performance, please.
>>
>> The core compression and decompression code has been thoroughly tested, so I
>> do not expect major problems.
>>
>> Good testing after the merge and feedback about build or performance issues
>> (and improvements!) is highly appreciated.
> 
> The addition of the lzo tree to linux-next caused this problem for ARM
> imx_v6_v7_defconfig:
> 
> In file included from /home/arnd/linux-arm/arch/arm/boot/compressed/decompress.c:40:0:
> /home/arnd/linux-arm/arch/arm/boot/compressed/../../../../lib/decompress_unlzo.c:34:34: fatal error: lzo/lzo1x_decompress.c: No such file or directory
> 
> Since the file was renamed, anything including it needs to be updated to the
> new file name.

I will add that patch to my tree.

Cheers,
Markus

> 
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> 
> diff --git a/lib/decompress_unlzo.c b/lib/decompress_unlzo.c
> index 4531294..960183d 100644
> --- a/lib/decompress_unlzo.c
> +++ b/lib/decompress_unlzo.c
> @@ -31,7 +31,7 @@
>   */
>  
>  #ifdef STATIC
> -#include "lzo/lzo1x_decompress.c"
> +#include "lzo/lzo1x_decompress_safe.c"
>  #else
>  #include <linux/decompress/unlzo.h>
>  #endif

-- 
Markus Oberhumer, <markus@oberhumer.com>, http://www.oberhumer.com/

RE: [PATCH 0/3] Update LZO compression

[Dan Magenheimer]

> From: Markus F.X.J. Oberhumer [mailto:markus@oberhumer.com]
> Subject: Re: [PATCH 0/3] Update LZO compression
> 
> Thanks Arnd,
> 
> On 2012-10-11 13:41, Arnd Bergmann wrote:
> > On Tuesday 09 October 2012, Markus F.X.J. Oberhumer wrote:
> >>>
> >>> : This commit updates the kernel LZO code to the current upsteam version
> >>> : which features a significant speed improvement - benchmarking the Calgary
> >>> : and Silesia test corpora typically shows a doubled performance in
> >>> : both compression and decompression on modern i386/x86_64/powerpc machines.
> >>>
> >>> There are significant clients of the LZO library - crypto, btrfs,
> >>> jffs2, ubifs, squashfs and zcache.  So let's give all those people a cc
> >>> and ask that they test the LZO changes once they land in linux-next.
> >>> For correctness and performance, please.
> >>
> >> The core compression and decompression code has been thoroughly tested, so I
> >> do not expect major problems.
> >>
> >> Good testing after the merge and feedback about build or performance issues
> >> (and improvements!) is highly appreciated.
> >
> > The addition of the lzo tree to linux-next caused this problem for ARM
> > imx_v6_v7_defconfig:
> >
> > In file included from /home/arnd/linux-arm/arch/arm/boot/compressed/decompress.c:40:0:
> > /home/arnd/linux-arm/arch/arm/boot/compressed/../../../../lib/decompress_unlzo.c:34:34: fatal error:
> lzo/lzo1x_decompress.c: No such file or directory
> >
> > Since the file was renamed, anything including it needs to be updated to the
> > new file name.
> 
> I will add that patch to my tree.
> 
> Cheers,
> Markus

Sorry if I missed it (bad connectivity this week), but is someone going
to send a pull request to get this LZO update from linux-next into
Linus's tree?  The window is closing soon isn't it?

Dan