Re: OOM behavior in constrained memory situations

[Andi Kleen <ak@suse.de>]

On Monday 06 February 2006 22:10, Andrew Morton wrote:

> Do we really want to kill the application?  A more convetional response
> would be to return NULL from the page allocator and let that trickle back.

Yes that is what it's supposed to be doing.
 
> The hugepage thing is special, because it's a pagefault, not a syscall.

At least remnants from my old 80% hack to avoid this (huge_page_needed)
seem to be still there in mainline:

fs/hugetlbfs/inode.c:hugetlbfs_file_mmap

   bytes = huge_pages_needed(mapping, vma);
   if (!is_hugepage_mem_enough(bytes))
          return -ENOMEM;


So something must be broken if this doesn't work. Or did you allocate
the pages in some other way? 

>From taking a quick look at ipc/shm.c it might be missing an equivalent 
check when allocating a huge page segment.

-Andi