Re: kernel decompressor interface

["H. Peter Anvin" <hpa@zytor.com>]

On 03/30/2010 09:10 AM, Ferenc Wagner wrote:
> Hi,
> 
> While working with SquashFS code recently, I got the impression that the
> current decompress_fn interface isn't best suited for general use: it
> rules out real scatter/gather operation, which -- one hopes -- is a
> general feature of stream decompressors.  For example, if one has to
> decompress data from a series of buffer_heads into a bunch of (cache)
> pages (typical operation in compressed file systems), the inflate
> interface in zlib.h provides the possibility of changing input and
> output buffer addresses, but decompress_fn does not, necessitating extra
> memory copying.  On the other hand, the latter is admittedly simpler.
> 
> However, I feel like their advantages could be combined by making the
> fill() and flush() callbacks able to change the buffer addresses, like
> int fill(void **, unsigned int) for example.  Even better, some context
> could also be passed around, so that they can easily carry private state.
> 
> A cursory look at the LZMA decompressor implementation suggests that
> this would be straightforward change (although I haven't done it).  Do
> you think that
> 1. it could be done for other decompressors (lzo, bunzip) as well, and
> 2. it should be done in any form (interface replacement or extension)?

I think the event-based interfaces in zlib.h are pretty good, myself,
and since they are so widely deployed they are also widely mimiced,
which makes it a good interface to rely on.  A simpler "one shot"
interface can then be implemented as a wrapper (since it just means:
create the context, point it at the two buffers, and then tell it to to
run to the end.)

LZO has some super-compact variants that don't support start and stop
operation, but even the "full featured" versions are small enough, that
I don't think it matters for something as large as the Linux kernel.  It
mattered to me when I was dealing with self-decompressing objects on the
order of a few tens of kilobytes.

	-hpa