Free memory starvation in a zone?

Free memory starvation in a zone?

[Jeremy Linton]

In kreclaimd() there is a nice loop that looks like

 for(i = 0; i < MAX_NR_ZONES; i++) {
    zone_t *zone = pgdat->node_zones + i;
    if (!zone->size)
        continue;

    while (zone->free_pages < zone->pages_low) {
        struct page * page;
        page = reclaim_page(zone);
        if (!page)
            break;
        __free_page(page);
    }
}

I was playing around with the page age algorithm when i noticed that it
appears that the machine will get into a state where the inner loop _NEVER_
exits the current zone because applications running in that zone are eating
the memory as fast as it is being freed up. I imaging that this could be
causing some pretty serious problems since the other zone's pages will only
get recleaimed during a page alloc. Maybe there should be a max number of
pages that can be reclaimed out of any given zone to force this loop to
break? Something like 5 or 10% of the zone?

Any comments?


BTW: I started playing with the page age system when I noticed that it
wasn't very evenly distributed. All the pages in a  tend to fall into one or
two age groups pretty close to PAGE_AGE_START with a significant number of
them often below PAGE_AGE_START.

Re: Free memory starvation in a zone?

[Marcelo Tosatti]

On Fri, 3 Aug 2001, Jeremy Linton wrote:

> In kreclaimd() there is a nice loop that looks like
> 
>  for(i = 0; i < MAX_NR_ZONES; i++) {
>     zone_t *zone = pgdat->node_zones + i;
>     if (!zone->size)
>         continue;
> 
>     while (zone->free_pages < zone->pages_low) {
>         struct page * page;
>         page = reclaim_page(zone);
>         if (!page)
>             break;
>         __free_page(page);
>     }
> }
> 
> I was playing around with the page age algorithm when i noticed that it
> appears that the machine will get into a state where the inner loop _NEVER_
> exits the current zone because applications running in that zone are eating
> the memory as fast as it is being freed up.

Normal allocations are going to block giving a chance for kreclaimd to run
(and remember, the loop and the freeing routines are really fast).

Are you sure you're seeing kreclaimd looping too much here ?

I've never seen that, and if I did I would get really really
concerned: we rely on kreclaimd to avoid atomic allocations from failing
in a fragile way. 

Re: Free memory starvation in a zone?

[Mike Galbraith]

On Fri, 3 Aug 2001, Marcelo Tosatti wrote:

> On Fri, 3 Aug 2001, Jeremy Linton wrote:
>
> > In kreclaimd() there is a nice loop that looks like
> >
> >  for(i = 0; i < MAX_NR_ZONES; i++) {
> >     zone_t *zone = pgdat->node_zones + i;
> >     if (!zone->size)
> >         continue;
> >
> >     while (zone->free_pages < zone->pages_low) {
> >         struct page * page;
> >         page = reclaim_page(zone);
> >         if (!page)
> >             break;
> >         __free_page(page);
> >     }
> > }
> >
> > I was playing around with the page age algorithm when i noticed that it
> > appears that the machine will get into a state where the inner loop _NEVER_
> > exits the current zone because applications running in that zone are eating
> > the memory as fast as it is being freed up.
>
> Normal allocations are going to block giving a chance for kreclaimd to run
> (and remember, the loop and the freeing routines are really fast).
>
> Are you sure you're seeing kreclaimd looping too much here ?
>
> I've never seen that, and if I did I would get really really
> concerned: we rely on kreclaimd to avoid atomic allocations from failing
> in a fragile way.

Snippet from one of Dirk's logs.

  PID  PPID USER     PRI  SIZE SWAP  RSS SHARE   D STAT %CPU %MEM   TIME COMMA
    3     1 root      20     0    0    0     0   0 RW   58.8  0.0   2:41 kswapd
 1494  1421 novatest  15 2009M 640M 1.3G 51476  0M R N  40.8 34.5   6:26 ceqsim
 1751  1747 root      14  1048    4 1044   824  55 R    28.0  0.0   0:02 top
    4     1 root      14     0    0    0     0   0 SW   27.1  0.0   1:06 krecla

	-Mike

Re: Free memory starvation in a zone?

[Marcelo Tosatti]

On Sat, 4 Aug 2001, Mike Galbraith wrote:

> On Fri, 3 Aug 2001, Marcelo Tosatti wrote:
> 
> > On Fri, 3 Aug 2001, Jeremy Linton wrote:
> >
> > > In kreclaimd() there is a nice loop that looks like
> > >
> > >  for(i = 0; i < MAX_NR_ZONES; i++) {
> > >     zone_t *zone = pgdat->node_zones + i;
> > >     if (!zone->size)
> > >         continue;
> > >
> > >     while (zone->free_pages < zone->pages_low) {
> > >         struct page * page;
> > >         page = reclaim_page(zone);
> > >         if (!page)
> > >             break;
> > >         __free_page(page);
> > >     }
> > > }
> > >
> > > I was playing around with the page age algorithm when i noticed that it
> > > appears that the machine will get into a state where the inner loop _NEVER_
> > > exits the current zone because applications running in that zone are eating
> > > the memory as fast as it is being freed up.
> >
> > Normal allocations are going to block giving a chance for kreclaimd to run
> > (and remember, the loop and the freeing routines are really fast).
> >
> > Are you sure you're seeing kreclaimd looping too much here ?
> >
> > I've never seen that, and if I did I would get really really
> > concerned: we rely on kreclaimd to avoid atomic allocations from failing
> > in a fragile way.
> 
> Snippet from one of Dirk's logs.
> 
>   PID  PPID USER     PRI  SIZE SWAP  RSS SHARE   D STAT %CPU %MEM   TIME COMMA
>     3     1 root      20     0    0    0     0   0 RW   58.8  0.0   2:41 kswapd
>  1494  1421 novatest  15 2009M 640M 1.3G 51476  0M R N  40.8 34.5   6:26 ceqsim
>  1751  1747 root      14  1048    4 1044   824  55 R    28.0  0.0   0:02 top
>     4     1 root      14     0    0    0     0   0 SW   27.1  0.0   1:06 krecla

The problem is if we have kreclaimd not able to satisfy all zones needs. 

CPU usage does not mean zone freemem starvation.

Re: Free memory starvation in a zone?

[Rik van Riel]

On Sat, 4 Aug 2001, Mike Galbraith wrote:

> > Are you sure you're seeing kreclaimd looping too much here ?
>
> Snippet from one of Dirk's logs.
>
>   PID  PPID USER     PRI  SIZE SWAP  RSS SHARE   D STAT %CPU %MEM   TIME COMMA
>     3     1 root      20     0    0    0     0   0 RW   58.8  0.0   2:41 kswapd
>  1494  1421 novatest  15 2009M 640M 1.3G 51476  0M R N  40.8 34.5   6:26 ceqsim
>  1751  1747 root      14  1048    4 1044   824  55 R    28.0  0.0   0:02 top
>     4     1 root      14     0    0    0     0   0 SW   27.1  0.0   1:06 krecla

I'm pretty sure this is because kreclaimd is woken up from
__alloc_pages() all the time and cannot find anything useful
to do ...

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Free memory starvation in a zone?

[Mike Galbraith]

On Sat, 4 Aug 2001, Rik van Riel wrote:

> On Sat, 4 Aug 2001, Mike Galbraith wrote:
>
> > > Are you sure you're seeing kreclaimd looping too much here ?
> >
> > Snippet from one of Dirk's logs.
> >
> >   PID  PPID USER     PRI  SIZE SWAP  RSS SHARE   D STAT %CPU %MEM   TIME COMMA
> >     3     1 root      20     0    0    0     0   0 RW   58.8  0.0   2:41 kswapd
> >  1494  1421 novatest  15 2009M 640M 1.3G 51476  0M R N  40.8 34.5   6:26 ceqsim
> >  1751  1747 root      14  1048    4 1044   824  55 R    28.0  0.0   0:02 top
> >     4     1 root      14     0    0    0     0   0 SW   27.1  0.0   1:06 krecla
>
> I'm pretty sure this is because kreclaimd is woken up from
> __alloc_pages() all the time and cannot find anything useful
> to do ...

Yes, it could be bouncing (eg) referenced pages instead of looping.

In any case though, seems it would it be better to take locks, evaluate
once and free in batch.  With the possibility to bounce off of locks,
it does look like looping is possible (if unlikely).  Just picking a
quantity and freeing it instead of reevaluating constantly would close
off that possibility without batch freeing.

	-Mike