First of all, thanks for your replies and helpful information.

Andrew Morton wrote:

> "Brett E." <brettspamacct@fastclick.com> wrote:
> 
>>I attached sar, slabinfo and /proc/meminfo data on the 2.6.5 machine.  I 
>>reproduce this behavior by simply untarring a 260meg file on a 
>>production server, the machine becomes sluggish as it swaps to disk.
> 
> 
> I see no swapout from the info which you sent.

pgpgout/s gives the total number of blocks paged out to disk per second, 
it peaks at 13,000 and hovers around 3,000 per the attachment.

> 
> A `vmstat 1' trace would be more useful.
Ok. Attached(ran this with swappiness set to 0 then 100). In both cases 
sar showed high paging in/out.

> 
> 
>>Is there a way to limit the cache so this machine, which has 1 gigabyte of 
>>memory, doesn't dip into swap?
> 
> 
> Decrease /proc/sys/vm/swappiness?
> 
> Swapout is good.  It frees up unused memory.  I run my desktop machines at
> swappiness=100.
> 

Swapping out is good, but when that's coupled with swapping in as is the 
case on my side, it creates a thrashing situation where we swap out to 
disk pages which are being used, we then immediately swap those pages 
back in, etc etc.. This creates lots of disk I/O which competes with the 
userland processes, resulting in a system slowing down to a crawl. I 
don't understand why it swaps in the first place when 400-500 megs are 
taken up by cache datastructures.

The usage pattern by the way is on a server which continuously hits a 
database and reads files so I don't know what "swappiness" should be set 
to exactly.  Every hour or so it wants to untar tarballs and by then the 
cache is large. From here, the system swaps in and out more while cache 
decreases. Basically, it should do what I believe Solaris does... simply 
reclaim cache and not swap.  Capping cache would be good too but the 
best solution IMO is to simply reclaim the cache on an as-needed basis 
before thinking about swapping.