Re: zone state overhead

[Mel Gorman <mel@csn.ul.ie>]

On Tue, Sep 28, 2010 at 07:39:24AM -0500, Christoph Lameter wrote:
> On Tue, 28 Sep 2010, Shaohua Li wrote:
> 
> > In a 4 socket 64 CPU system, zone_nr_free_pages() takes about 5% ~ 10% cpu time
> > according to perf when memory pressure is high. The workload does something
> > like:
> > for i in `seq 1 $nr_cpu`
> > do
> >         create_sparse_file $SPARSE_FILE-$i $((10 * mem / nr_cpu))
> >         $USEMEM -f $SPARSE_FILE-$i -j 4096 --readonly $((10 * mem / nr_cpu)) &
> > done
> > this simply reads a sparse file for each CPU. Apparently the
> > zone->percpu_drift_mark is too big, and guess zone_page_state_snapshot() makes
> > a lot of cache bounce for ->vm_stat_diff[]. below is the zoneinfo for reference.
> > Is there any way to reduce the overhead?
> 

The overhead is higher than I would have expected. I would guess the
cache bounces are a real problem.

> I guess Mel could reduce the percpu_drift_mark? Or tune that with a
> reduction in the stat_threshold? The less the count can deviate the less
> the percpu_drift_mark has to be and the less we need calls to
> zone_page_state_snapshot.
> 

This is true. It's helpful to remember why this patch exists. Under heavy
memory pressure, large machines run the risk of live-locking because the
NR_FREE_PAGES gets out of sync. The test case mentioned above is under
memory pressure so it is potentially at risk. Ordinarily, we would be less
concerned with performance under heavy memory pressure and more concerned with
correctness of behaviour. The percpu_drift_mark is set at a point where the
risk is "real".  Lowering it will help performance but increase risk. Reducing
stat_threshold shifts the cost elsewhere by increasing the frequency the
vmstat counters are updated which I considered to be worse overall.

Which of these is better or is there an alternative suggestion on how
this livelock can be avoided?

As a heads up, I'm preparing for exams at the moment and while I'm online, I'm
not in the position to prototype patches and test them at the moment but can
review alternative proposals if people have them. I'm also out early next week.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>