Re: [PATCH] Optimize out pte_chain take three

William Lee Irwin III
2002-07-11T01:51:02Z

William Lee Irwin III wrote:
>> Phenomenally harsh.

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> No offence I hope. Just venting three year's VM frustration.

None taken. I believe it's a strong advisory to change direction,
so I have.

William Lee Irwin III wrote:
>> Your criteria are quantitative. I can't immediately measure all
>> of them but can go about collecting missing data immediately and post
>> as I go, then. Perhaps I'll even have helpers. =)

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> A lot of it should be fairly simple. We have tons of pagecache-intensive
> workloads. But we have gaps when it comes to the VM. In the area of
> page replacement.
> Can we fill those gaps with a reasonable amount of effort?

I'm not entirely sure, but I do have ideas of what I think would
exercise specific (sub)functions of the VM.

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> example 1: "run thirty processes which mmap a common 50000 page file and
> touch its pages in a random-but-always-the-same pattern at fifty pages
> per second. Then run (dbench|tiobench|kernel build|slocate|foo). Then
> see how many pages the three processes actually managed to touch."

Are we looking for "doesn't evict long-lived stuff" or "figures out
when the long-lived stuff finally died?" Maybe both would be good.

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> example 2: "run a process which mallocs 60% of physical memory and
> touches it randomly at 1000 pages/sec. run a second process
> which mallocs 60% of physical memory and touches it randomly at
> 500 pages/sec. Measure aggregate throughput for both processes".
> example 3: "example 2, but do some pagecache stuff as well".

I think 2 and 3 should be merged, this should basically see how the
parallel pagecache stuff disturbs the prior results.

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> example 4: "run a process which mmaps 60%-worth of memory readonly,
> another which mmaps 60%-worth of memory MAP_SHARED. First process
> touches 1000 pages/sec. Second process modifies 1000 pages/sec.
> Optimise for throughput"
> Scriptable things. Things which we can optimise for with a
> reasonable expectation that this will improve real workloads.

Sequential/random access is another useful variable here.

William Lee Irwin III wrote:
>> I've already gone about asking for help benchmarking dmc's pte_chain
>> space optimization, and I envision the following list of TODO items
>> being things you're more interested in:
>> What other missing data are you after and which of these should
>> be chucked?

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> Well there are two phases to this. One is to run workloads,
> and the other is to analyse them. I think workloads such as my
> lame ones above are worth thinking about and setting up first,
> don't you? (And a lot of this will come from me picking your
> brains ;). I can do the coding, although by /bin/sh skills
> are woeful.)
> After that comes the analysis. Looks like rmap will be merged in
> the next few days for test-and-eval, so we don't need to go through
> some great beforehand-justification exercise. But we do need
> a permanent toolkit and we do need a way of optimising the VM.

Okay, this is relatively high priority then.

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> I would suggest that the toolkit consist of two things:
> 1: A set of scenarios and associated scripts/tools such as my
> examples above (except more real-worldy) and
> 2: Permanent in-kernel instrumentation which allows us (and
> remote testers) to understand what is happening in there.

At long last!

William Lee Irwin III wrote:
>> As far as operating regions for page replacement go I see 3 obvious ones:
>> (1) lots of writeback with no swap
>> (2) churning clean pages with no swap
>> (3) swapping
>> And each of these with several proportions of memory sharing.
>> Sound reasonable?

On Wed, Jul 10, 2002 at 05:39:47PM -0700, Andrew Morton wrote:
> Sounds ideal. Care to flesh these out into real use cases?

(1) would be a system dedicated to data collection in real-life.
Lots of writeback without swap is also a common database
scenario. At any rate, it's intended to measure how effective
and/or efficient replacement is when writeback is required to
satisfy allocations.

(2) would be a system dedicated to distributing data in real life.
This would be a read-only database scenario or (ignoring
single vs. multiple files) webserving. It's intended to
measure how effective page replacement is when clean pages
must be discarded to satisfy allocations.

These two testcases would be constellations of sibling processes
mmapping a shared file-backed memory block larger than memory
(where it departs from database-land) and stomping all over it
in various patterns. Random, forward sequential, backward
sequential, mixtures of different patterns across processes,
mixtures of different patterns over time, varying frequencies
of access to different regions of the arena and checking to see
that proportions of memory being reclaimed corresponds to
frequency of access, and sudden shifts between (1) and (2) would
all be good things to measure, but the combinatorial explosion
of options here hurts.

The measurable criterion is of course throughput shoveling data.
But there are many useful bits to measure here, e.g. the amount
of cpu consumed by replacement, how much actually gets scanned
for that cpu cost, and how right or wrong the guesses were when
variable frequency access is used.

(3) this is going to be a "slightly overburdened desktop" workload
here it's difficult to get a notion of how to appropriately
simulate the workload, but I have a number of things in mind
as to how to measure some useful things for one known case:

(A) there are a number (300?) of tasks that start up, fault in
some crud, and sleep forever -- in fact the majority
They're just there in the background to chew up some
space with per-task and per-mm pinned pages.
(B) there are a number of tasks that are spawned when timer
goes off, then they stomp on a metric buttload of stuff
and exit until spawned for the next interval, e.g.
updatedb, and of course none of the data it uses is
useful for anything else and is all used just once.
This can be dcache, buffer cache, pagecache, or anything.
(C) there are a number of tasks that get prodded and fault on
minor amounts of stuff and have mostly clean mappings
but occasionally they'll all sleep for a long time
(e.g. like end-users sleeping through updatedb runs)
Basically, get a pool of processes, let them sleep,
shoot a random process with signals at random times,
and when shot, a process stomps over some clean data
with basically random access before going back to
sleep.

The goal is to throw (A) out the window, control how much is
ever taken away from (C) and given to (B), and keep (C), who
is the real user of the thing, from seeing horribly bad worst
cases like after updatedb runs at 3AM. Or, perhaps, figuring
out who the real users are is the whole problem...

Okay, even though the updatedb stuff can (and must) be solved
with dcache-specific stuff, the general problem remains as a
different kind of memory can be allocated in the same way.
Trying to fool the VM with different kinds of one-shot
allocations is probably the best variable here; specific timings
aren't really very interesting.

The number that comes out of this is of course the peak
pagefault rate of the tasks in class (C).

The other swapping case is Netscape vs. xmms:
One huge big fat memory hog is so bloated its working set alone
drives the machine to swapping. This is poked and prodded at
repeatedly, at which time it of course faults in a bunch more
garbage from its inordinately and unreasonably bloated beyond
belief working set. Now poor little innocent xmms is running
in what's essentially a fixed amount of memory and cpu but
can't lose the cpu or the disk for too long or mp3's will skip.

This test is meant to exercise the VM's control over how much cpu
and -disk bandwidth- it chews at one time so that well-behaved
users don't see unreasonable latencies as the VM swamps their
needed resources.

I suspect the way to automate it is to generate signals to wake
the bloated mem hog at random intervals, and that mem hog then
randomly stomps over a bunch of memory. The innocent victim
keeps a fixed-size arena where it sequentially slurps in fresh
files, generates gibberish from their contents into a dedicated
piece of its fixed-size arena, and then squirts out the data at
what it wants to be a fixed rate to some character device. So
the write buffer will always be dirty and the read buffer clean,
except it doesn't really mmap. Oh, and of course, it takes a
substantial but not overwhelming amount of cpu (25-40% for
ancient p200's or something?) to generate the stuff it writes.

Then the metric used is the variability in the read and write
rates and %cpu used of the victim.

Uh-oh, this stuff might take a while to write... any userspace helpers
around?

Cheers,
Bill