Re: Bug in Kernel's LZ4-HC in 64bit (kernel crypto library)

Re: Bug in Kernel's LZ4-HC in 64bit (kernel crypto library)

[Greg Kroah-Hartman]

On Sat, Jun 28, 2014 at 09:19:34PM +0530, Kiran Kankipati wrote:
> Hi Something similar to that I got inconsistency issue in LZ4-HC in 64bit mode.
> 
> I posted today in kernel bugzee:  https://bugzilla.kernel.org/show_bug.cgi?id=79091

I don't do bugzilla :)

Let's drag this back to lkml as well...

Ok, where do you see an overrun happening?  I know the lz4 code in the
kernel is "old" compared to upstream.  I am working on updating it to
the latest version, but that's not going to happen until 3.17.

In the mean time, did you note the lz4 patch that was merged yesterday,
it is commit id 4148c1f67abf823099b2d7db6851e4aea407f5ee in Linus's
tree.  Does that solve the issue you are seeing?

If not, do you have a data stream I can use to try to find and fix this?

thanks,

greg k-h

Re: Bug in Kernel's LZ4-HC in 64bit (kernel crypto library)

[Greg Kroah-Hartman]

On Sat, Jun 28, 2014 at 09:39:12PM +0530, Kiran Kankipati wrote:
> Wow I see. Thanks Greg. I seen the 3.15.2 change logs, found you are involved
> in the LZO issue.
> 
> I read about this issue in some news too that they found 20yr old bug in it.
> 
> So thought reporting to you via email.
> 
> I don't do bugzilla :)
> >> I see.
> 
> I am new to kernel foundation procedures. So sorry for bothering you.
> 
> --------
> Ok, where do you see an overrun happening?  I know the lz4 code in the
> kernel is "old" compared to upstream.  I am working on updating it to
> the latest version, but that's not going to happen until 3.17.
> 
> >> Here is the place and APIs I am using in my TrafficSqueezer code:
> 
> 
> void ts_comp(BYTE *flag, BYTE *pbuff, size_t *pbuff_len, BYTE *mid_buff, BYTE
> *lz4hc_wrkmem)
> {    size_t mid_buff_size = 0;
> 
>     if(lz4hc_compress( (const unsigned char *)pbuff, (*pbuff_len), (unsigned
> char *)mid_buff, &mid_buff_size, (void *)lz4hc_wrkmem)<0) return;
>        
>     if(((mid_buff_size+TS_FLAG_SIZE) < (*pbuff_len)) && (mid_buff_size!=0))
>     {    memcpy((BYTE *)pbuff, (BYTE *)mid_buff, mid_buff_size); (*pbuff_len)=
> mid_buff_size;
>         ts_set_pkt_tag(flag, TS_FLAG_COMP);
>         #ifdef CONFIG_TRAFFICSQUEEZER_DEBUG
>         printk("comp - prossd [LZ4-HC] [mid_buff_size: %zu]\n", mid_buff_size);
>         #endif
>     }
> }
> 
> bool ts_decomp(BYTE *pbuff, size_t *pbuff_len, BYTE *mid_buff)
> {  size_t mid_buff_size = TS_MAX_OPT_BUF_LEN;  //WARNING: DO NOT change this !!
> 
>     if(lz4_decompress_unknownoutputsize( (const unsigned char *)pbuff,
> (*pbuff_len), (unsigned char *)mid_buff,  &mid_buff_size)<0) { return false; }
>     if(mid_buff_size > TS_MAX_OPT_BUF_LEN) { return false; }
>    
>     memcpy((BYTE *)pbuff, (BYTE *)mid_buff, mid_buff_size); (*pbuff_len)=
> mid_buff_size;
> #ifdef CONFIG_TRAFFICSQUEEZER_DEBUG
>     printk("decomp - prossd [LZ4-HC] [mid_buff_size: %zu]\n", mid_buff_size);
> #endif  
>     return true;
> }

I don't understand, are you having problems on the compress side, or the
decompress side?  Where in the lz4 is a bug happening?

Do you have a data stream that I can use to run it through the lz4 code
to see where the problem is in the compress/decompress code?

> ------------
> If this fix scheduled for 3.17, I am glad enough to wait till then. Since I
> need this fix as a part of mainstream kernel release series :)

The fixes I've done will show up in 3.16-rc3 and in the next round of
stable kernel updates that will come out Monday, so you can pick those
releases up and test if you can't apply individual patches for a kernel
tree.

thanks,

greg k-h

Re: Bug in Kernel's LZ4-HC in 64bit (kernel crypto library)

[Greg Kroah-Hartman]

On Sat, Jun 28, 2014 at 10:02:19PM +0530, Kiran Kankipati wrote:
> I don't understand, are you having problems on the compress side, or the
> decompress side?  Where in the lz4 is a bug happening?
> >> Sorry sorry.
> I dont have a clue. Well the reason is, if compress is happening well and
> decomp is getting this issue. Then decomp is having a bug.
> If compression is corrupting data, then decomp may get this issue too. Hence I
> am not fully sure.
> But the error code happening in the decomp API.
> 
> 
> Do you have a data stream that I can use to run it through the lz4 code
> to see where the problem is in the compress/decompress code?
> >> Again sorry about it. Since  iam testing with real-time packets. I dont have
> it Greg.
> 
> I can copypaste the error logs of my TrafficSqueezer per-packet tests. Since  I
> am running a simulation test, I am doing each packet
> compression + decompression. This way I can test TrafficSqueezer stack. As well
> in our case we can test LZ4 too.
> 
> Kindly refer screenshot:
> 
> But please note, sometimes it works fine too. Sometimes I am getting this error
> :(
> 
> Its purely LZ4-HC error. Since the return code of
> lz4_decompress_unknownoutputsize() API is <0 i.e error !

As you are compressing and then decompressing, this doesn't look to be
the same "type" of error we have fixed recently, so I don't know what to
do here.

If you find a data stream that this can be tested with, please let me
know and I'll be glad to try it out.  Until then, I need more of a hint
as to the problem before I can do anything, sorry.

greg k-h