Page compression in lieu of swap?

Page compression in lieu of swap?

[Timothy Miller]

I did some searching of the kernel archives and the only things related to
the forthcoming idea had to do with compressing pages when writing to swap
and doing compressed disks.  Here's a different idea...

Inspired by my recent experiments in compressing kernel messages, I started
to wonder what else might benefit from compression, and the following idea
occurred to me:

Given the hideous amount of time required to access a disk, especially when
something else wants to access it, could there be a benefit to "swapping"
pages by compressing them to somewhere else in memory?  If we could achieve,
even say, 30% compression on pages, on average, then we could free up RAM
without having to do any I/O.  This would be the first line of defense
against a low-memory situation, finally resorting to actual disk access when
that becomes unworkable or for pages which can't be compressed enough for it
to help (which has a penalty worse than just writing to disk).  And
furthermore, if we were to swap first memory containing compressed pages, we
can reduce the total amount of I/O for swapping.

This would, of course, suck a lot of CPU, and in the case of a server
running many services where the CPU usage is pegged even when there's a lot
of swapping, it would be better to just swap as normal.  But in any case
where swapping is causing an increase in idle time, I would expect a
considerable benefit from being able to free up pages by making LRU pages
simply take up less space in RAM when they're not being used.

Comments?

Re: Page compression in lieu of swap?

[Barry K. Nathan]

On Sat, Apr 12, 2003 at 06:35:19PM -0400, Timothy Miller wrote:
> Given the hideous amount of time required to access a disk, especially when
> something else wants to access it, could there be a benefit to "swapping"
> pages by compressing them to somewhere else in memory?  If we could achieve,
[snip]
> 
> Comments?

This has been done before, on (Classic) Mac OS (the program's name was
RAM Doubler). It was *far* faster than Apple's swapping implementation,
although I don't know how much of that was due to the compression and
how much was due to Apple's horrid virtual memory implementation back in
the day. It also had some stability problems, but that could have been
due to the implementation quality rather than to the overall approach.

-Barry K. Nathan <barryn@pobox.com>

Re: Page compression in lieu of swap?

[Jan Knutar]

On Sunday 13 April 2003 01:35, Timothy Miller wrote:
> I did some searching of the kernel archives and the only things
> related to the forthcoming idea had to do with compressing pages when
> writing to swap and doing compressed disks.  Here's a different
> idea...

http://linuxcompressed.sourceforge.net/

This the same thing?

Re: Page compression in lieu of swap?

[Con Kolivas]

On Sun, 13 Apr 2003 10:06, Jan Knutar wrote:
> On Sunday 13 April 2003 01:35, Timothy Miller wrote:
> > I did some searching of the kernel archives and the only things
> > related to the forthcoming idea had to do with compressing pages when
> > writing to swap and doing compressed disks.  Here's a different
> > idea...
>
> http://linuxcompressed.sourceforge.net/
>
> This the same thing?

Yes it is and works very well. However it isn't smp or preemptible aware yet. 
I have a patch against -ck* as well, but it isn't popular because of preempt 
incompatibility.

Con

Re: Page compression in lieu of swap?

[Timothy Miller]

From: "Jan Knutar" <jk-lkml@sci.fi>

> On Sunday 13 April 2003 01:35, Timothy Miller wrote:
> > I did some searching of the kernel archives and the only things
> > related to the forthcoming idea had to do with compressing pages when
> > writing to swap and doing compressed disks.  Here's a different
> > idea...
>
> http://linuxcompressed.sourceforge.net/
>
> This the same thing?
>


I believe it is!

Anyone on the list checked it out?  Is it good?  Any benchmarks performed?
Do we want it?

If it doesn't introduce instability, then it could be a very good addition.
Perhaps at the beginning of the 2.7 phase?  If were to be in a develoment
kernel for its entire life, it would certainly get all the bugs worked out.

Does Linus have an opinion on it?

One of the things that makes me like Linux the best is that the contributors
really seem to push the envelope.  Of course, I don't know really how well
other OS's do in that regard, but I wouldn't be surprised if we were leaving
other OS's like Windows, Solaris, and MacOS in the dust with some of the
cool things being worked on like the O(1) scheduler, the anticipatory I/O
scheduler, etc.  Maybe others have such things.  <shrug>

Sorry about the rambling.

Re: Page compression in lieu of swap?

[Jan Knutar]

> Anyone on the list checked it out?  Is it good?  Any benchmarks
> performed? Do we want it?

Some benchmarks on the site. Seems to have a negative effect on kernel
compiles atleast, on machines with lots of memory... Looks like my 24 meg
gateway might just be on the border to benefit from it, unless its 133Mhz
overdrive processor (33Mhz isa bus.. wee) makes compression too
expensive to be beneficial...

RE: Page compression in lieu of swap?

[Perez-Gonzalez, Inaky]

> -----Original Message-----
> From: Timothy Miller [mailto:tmiller10@cfl.rr.com]
> 
> 
> Given the hideous amount of time required to access a disk, especially
when
> something else wants to access it, could there be a benefit to "swapping"
> pages by compressing them to somewhere else in memory?  If we could
achieve,
> even say, 30% compression on pages, on average, then we could free up RAM

I tried this sometime ago (2.2.x timeframe) for canning mozilla into an
small amount of memory and it was kind of doable - not too complicated,
in fact - the only thing is it would reduce the machine to a crawl some
times (I guess I did not know how to throttle the swap) - I even got it
working with bzip2 -9 [this was a pure exercise].

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)