Re: [PATCH 2/3] mm: page allocator: Calculate a better estimate of NR_FREE_PAGES when memory is low and kswapd is awake

[Mel Gorman <mel@csn.ul.ie>]

On Fri, Aug 20, 2010 at 12:22:33AM +0900, Minchan Kim wrote:
> On Thu, Aug 19, 2010 at 04:07:39PM +0100, Mel Gorman wrote:
> > On Thu, Aug 19, 2010 at 11:34:39PM +0900, Minchan Kim wrote:
> > > On Thu, Aug 19, 2010 at 03:09:46PM +0100, Mel Gorman wrote:
> > > > On Thu, Aug 19, 2010 at 11:01:50PM +0900, Minchan Kim wrote:
> > > > > On Thu, Aug 19, 2010 at 11:38:39AM +0100, Mel Gorman wrote:
> > > > > > On Thu, Aug 19, 2010 at 07:33:57PM +0900, Minchan Kim wrote:
> > > > > > > On Thu, Aug 19, 2010 at 5:06 PM, Mel Gorman <mel@csn.ul.ie> wrote:
> > > > > > > > On Wed, Aug 18, 2010 at 11:57:26PM +0900, Minchan Kim wrote:
> > > > > > > >> On Wed, Aug 18, 2010 at 09:51:23AM +0100, Mel Gorman wrote:
> > > > > > > >> > > What's a window low and min wmark? Maybe I can miss your point.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > The window is due to the fact kswapd is not awake yet. The window is because
> > > > > > > >> > kswapd might not be awake as NR_FREE_PAGES is higher than it should be. The
> > > > > > > >> > system is really somewhere between the low and min watermark but we are not
> > > > > > > >> > taking the accurate measure until kswapd gets woken up. The first allocation
> > > > > > > >> > to notice we are below the low watermark (be it due to vmstat refreshing or
> > > > > > > >> > that NR_FREE_PAGES happens to report we are below the watermark regardless of
> > > > > > > >> > any drift) wakes kswapd and other callers then take an accurate count hence
> > > > > > > >> > "we could breach the watermark but I'm expecting it can only happen for at
> > > > > > > >> > worst one allocation".
> > > > > > > >>
> > > > > > > >> Right. I misunderstood your word.
> > > > > > > >> One more question.
> > > > > > > >>
> > > > > > > >> Could you explain live lock scenario?
> > > > > > > >>
> > > > > > > >
> > > > > > > > Lets say
> > > > > > > >
> > > > > > > > NR_FREE_PAGES     = 256
> > > > > > > > Actual free pages = 8
> > > > > > > >
> > > > > > > > The PCP lists get refilled in patch taking all 8 pages. Now there are
> > > > > > > > zero free pages. Reclaim kicks in but to reclaim any pages it needs to
> > > > > > > > clean something but all the pages are on a network-backed filesystem. To
> > > > > > > > clean them, it must transmit on the network so it tries to allocate some
> > > > > > > > buffers.
> > > > > > > >
> > > > > > > > The livelock is that to free some memory, an allocation must succeed but
> > > > > > > > for an allocation to succeed, some memory must be freed. The system
> > > > > > > 
> > > > > > > Yes. I understood this as livelock but at last VM will kill victim
> > > > > > > process then it can allocate free pages.
> > > > > > 
> > > > > > And if the exit path for the OOM kill needs to allocate a page what
> > > > > > should it do?
> > > > > 
> > > > > Yeah. It might be livelock. 
> > > > > Then, let's rethink the problem. 
> > > > > 
> > > > > The problem is following as. 
> > > > > 
> > > > > 1. Process A try to allocate the page
> > > > > 2. VM try to reclaim the page for process A
> > > > > 3. VM reclaims some pages but it remains on PCP so can't allocate pages for A
> > > > > 4. VM try to kill process B
> > > > > 5. The exit path need new pages for exiting process B
> > > > > 6. Livelock happens(I am not sure but we need any warning if it really happens at least)
> > > > > 
> > > > 
> > > > The problem this patch is concerned with is about the vmstat counters, not
> > > > the pages on the per-cpu lists. The issue being dealt with is that the page
> > > > allocator grants a page going below the min watermark because NR_FREE_PAGES
> > > > can be inaccurate. The patch aims to fix that but taking greater care
> > > > with NR_FREE_PAGES when memory is low.
> > > 
> > > Your goal is to protect _min_ pages which is reserved. Right?
> > > I thought your final goal is to protect the livelock problem. 
> > > Hmm.. Sorry for the noise. :(
> > > 
> > 
> > Emm, it's the same thing. If the min watermark is not properly
> > preserved, the system is in danger of being live-locked.
> 
> Totally right. 
> Maybe I am sleeping.
> 
> Let's add follwing as comment about livelock.
> 

Sure!

> "If NR_FREE_PAGES is much higher than number of real free page in buddy,
> the VM can allocate pages below min watermark(At worst, buddy is zero). 
> Although VM kills some victim for freeing memory, it can't do it if the 
> exit path requires new page since buddy have zero page. It can result in
> livelock."
> 

Thanks

> At least, it help to not hurt you in future by me who is fool. 
> 

The patch leader now reads as

Ordinarily watermark checks are based on the vmstat NR_FREE_PAGES as it is
cheaper than scanning a number of lists. To avoid synchronization overhead,
counter deltas are maintained on a per-cpu basis and drained both periodically
and when the delta is above a threshold. On large CPU systems, the difference
between the estimated and real value of NR_FREE_PAGES can be very high.
If NR_FREE_PAGES is much higher than number of real free page in buddy, the VM
can allocate pages below min watermark, at worst reducing the real number of
pages to zero. Even if the OOM killer kills some victim for freeing memory, it
may not free memory if the exit path requires a new page resulting in livelock.

This patch introduces zone_nr_free_pages() to take a slightly more accurate
estimate of NR_FREE_PAGES while kswapd is awake. The estimate is not perfect
and may result in cache line bounces but is expected to be lighter than the
IPI calls necessary to continually drain the per-cpu counters while kswapd
is awake.

Is that better?

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>