Re: Can compression at filesystem level improve overall performance?

[Tom Vier <tmv@comcast.net>]

On Mon, Mar 22, 2004 at 10:03:36PM -0500, Scott Young wrote:
> What I'm saying is that you can start writing uncompressed data to the
> drive while the yet-to-be-written data is being compressed.  The goal is

i like your race idea. one problem is all writes now take significant cpu
(depending on the method). i'm wondering about a daemon that (in userspace)
that scans for files that are worth compressing. that's what disk doubler
used to do. i'd be interested to see some benchmarks, to see if compression
has much effect on total throughput.

> As an example, take a web server that has HTTP compression enabled. 
> Instead of compressing the data once per request, simply store the
> compressed version and send that when the next request occurs.  The

the server should cache the compressed versions in files. i'm pretty sure
apache does.

> There could be a "live" derivative, where any change to the original
> file reflects in the derivative file, or they could be constant, where
> any change in the original does not reflect in the derivative.  When I
<snip>

i dunno about doing all this in the kernel. as it is, i wish the kernel
wasn't so complicated. 8) what about a preload lib that puts everything
through bitkeeper or cvs?

> developer wouldn't have to rerun gcc to run the executable.  The file
> would automagically be compiled from the sources.

ide's can do this. and do you really want to have it run every time you
save? i save often, many times when i know there are going to be problems
cuz i haven't finished what i'm working on. a "lint" button would be better,
imho.

> The more I think about it, it seems that derived-files could be
> implemented as plugin(s) with only one extra feature added to the core
> filesystem: copy-on-write.  (Copy-on-write could be extended to only
> write changes, and have the new version be constructed as a derived file
> from the original, thereby compactly maintaining multiple versions of a
> file and improving write performance, but I digress)

cow is something that i've wanted to have ever since using hard links for
kernel trees (cp -al).

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE