out_of_memory called to often

out_of_memory called to often

[Jason Li]

Hi,

We are running an embedded linux (2.4.19) with no swap
sapce. And we haven't applied any rmap yet.

Currely when the system runs low (reached the
pages_low water-mark), kswapd kicks in. But soon after
kswapd kicks in, the out_of_memory is called due to
swapd not being able to reclaim enough pages from the
page cache from each run of it. I don't understand why
sometimes swapd can't reclaim the SWAP_CLUSTER_MAX
number of pages.

We know there are still some number of cache pages
that can be reclaimed if we run the following c
program:

int main() {
 int i, j;
 char *tmp[10000]

 for (i=0; i<NUMBER_OF_PAGES_LEFT_BEFORE_pages_low;
i++) tmp[i]=malloc(4k);
 for (i=0; i<NUMBER_OF_PAGES_LEFT_BEFORE_pages_low;
i++) free(tmp[i]);
}
To solve this problem, I am trying to make a
conditional call to the out_of_memory() -- only if the
pages_min is reached, then the out_of_memory() can be
called.

If we can solve the swapd livelock problem, is this a
feasible solution? If you have a better idea, can you
please share with us?

Any input will be greatly appreciated.

Thanks very much in advance! Please include my email
address in your reply.

-Jason



__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
http://webhosting.yahoo.com

Re: out_of_memory called to often

[Ross Biro]

Jason Li wrote:

>Hi,
>
>We are running an embedded linux (2.4.19) with no swap
>sapce. And we haven't applied any rmap yet.
>
I believe you may be seeing the same issue I brought up in a previous 
email.  Have you tried what I suggested? I've appended the email to this 
message with out the test code.

    Ross


I've verified this in 2.4.21-pre5 by code inspection and can trigger the 
problem on 2.4.18.  It appears to have been fixed in 2.5.

The folowing code vmscan.c assumes that there is available swap space.

       /*
        * this is the non-racy check for busy page.
        */
       if (!page->mapping || !is_page_cache_freeable(page)) {
           spin_unlock(&pagecache_lock);
           UnlockPage(page);
page_mapped:
           if (--max_mapped >= 0)
               continue;

           /*
            * Alert! We've found too many mapped pages on the
            * inactive list, so we start swapping out now!
            */
           spin_unlock(&pagemap_lru_lock);
           swap_out(priority, gfp_mask, classzone);
           return nr_pages;
       }


If there is no swap space, then unfreeable pages are left on the 
inactive queue and the vmtree is walked rather than going through the 
rest of the inactive queue.  I believe something like
       /*
        * this is the non-racy check for busy page.
        */
       if (!page->mapping || !is_page_cache_freeable(page)) {
           spin_unlock(&pagecache_lock);
           UnlockPage(page);
page_mapped:
                       /* If we don't have any swap space left, there
                          is no reason to worry about pages that do
                          not have swap associated with them, there
                          is nothing we can do about it. */
                       if (!page->mapping && !swap_avail()) {
                               /* Let's make the page active since we
                                  cannot swap it out.  It get's it off
                                  the inactive list. */
                               spin_unlock(&pagemap_lru_lock);
                               activate_page(page);
                               ClearPageReferenced(page);
                               spin_lock(&pagemap_lru_lock);
                               continue;
                       }
           if (--max_mapped >= 0)
               continue;

           /*
            * Alert! We've found too many mapped pages on the
            * inactive list, so we start swapping out now!
            */
           spin_unlock(&pagemap_lru_lock);
           swap_out(priority, gfp_mask, classzone);
           return nr_pages;
       }

will work better when there is no swap space available.  If this change 
is made, it may also be necessary to limit refill_inactive to prevent it 
from using too much cpu.  This bug can be triggered with the attached 
code and the correct parameters.  In particular on a 3 gigabyte machine 
with no swap,

for i in $(seq 0 9); do dd if=/dev/zero of=file$i bs=1024k count=512; done
killmm 1032735283 2 9

Usually causes an out of memory error when there is hundreds of 
megabytes of cache.


   Ross