Re: ~500 megs cached yet 2.6.5 goes into swap hell

Andy Isaacson
2004-04-29T22:27:15Z

On Thu, Apr 29, 2004 at 01:42:22PM -0700, Andrew Morton wrote:
> Andy Isaacson <adi@hexapodia.org> wrote:
> > What I want is for purely sequential workloads which far exceed cache
> > size (dd, updatedb, tar czf /backup/home.nightly.tar.gz /home) to avoid
> > thrashing my entire desktop out of memory. I DON'T CARE if the tar
> > completed in 45 minutes rather than 80. (It wouldn't, anyways, because
> > it only needs about 5 MB of cache to get every bit of the speedup it was
> > going to get.) But the additional latency when I un-xlock in the
> > morning is annoying, and there is no benefit.
>
> What kernel version are you using? If 2.6, what value of
> /proc/sys/vm/swappiness?

2.4.various, including 2.4.25 and 2.4.26. I haven't taken the 2.6
plunge yet. Running on various x86 including
- dual PIII 666 MHz 512 MB
- SpeedStep PIII 700 MHz 128 MB
- Athlon XP 2GHz 512 MB

> > For a more useful example, ideally I *should not be able to tell* that
> > "dd if=/hde1 of=/hdf1" is running.
>
> I just did a 4GB `dd if=/dev/sda of=/x bs=1M' on a 1GB 2.6.6-rc2-mm2
> swappiness=85 machine here and there was no swapout at all.
>
> Probably your machine has less memory. But without real, hard details
> nothing can be done.

I'm pleased to hear that 2.6 is apparently better behaved. In your
test, what was the impact on the file cache? It's a big improvement to
not be paging out to swap, but it's also important that sequential IO
not evict my cached build tree.

An interesting test would be to time a compilation of a source file with
a large number of includes. For example, building
linux-2.4.25/kernel/sysctl.c on my Athlon XP 2GHz, 512MB, 2.4.25 takes
2.8 seconds with (fairly) cold cache. (I didn't reboot, but I did take
fairly extreme measures to force stuff out.) It takes 0.54 seconds with
warm caches. After doing 1GB of sequential IO (wc -w /tmp/bigfile) I'm
back up to 2.08 seconds.

> > There is *no* benefit to cacheing
> > more than about 2 pages, under this workload.
>
> Sure, we could do better things with the large streaming files, although
> the risk of accidentally screwing up particular workloads is high.

Yeah, I agree. For example, I've occasionally used cat(1) or wc(1) to
prefetch files that I knew I was going to be accessing randomly; with my
hypothetical "sequential IO doesn't cause cacheing" it would be much
harder to do effective manual prefetching.

> But the use-once logic which we have in there at present does handle these
> cases quite well.

Where is the use-once logic available? Is it in mainstream 2.6 or only
in some development branches? I've not upgraded from 2.4 mostly because
I didn't see much benefits evident in the discussions, but improved
paging logic would be nice.

> > But with current kernels,
> > IME, that workload results in a gargantuan buffer cache and lots of
> > swapout of apps I was using 3 minutes ago. I've taken to walking away
> > for some coffee, coming back when it's done, and "sudo swapoff
> > /dev/hda3; sudo swapon -a" to avoid the latency that is so annoying when
> > trying to use bloaty apps.
>
> What kernel, what system specs, what swappiness setting?

2.4.25, Athlon XP 2 GHz, 512MB. I suppose you're not terribly
interested in 2.4. I'll see if I can reasonably upgrade, if you can
tell me what I should upgrade to for the good stuff.

-andy