[PATCH] BUG #3054 madvise doesn't fail for exceding RSS limit.

[Matt <matthew.keenan@ntlworld.com>]

Patch that fixes bug #3054. Compiles, tests ok.

--- linux-2.6.11.10/mm/madvise.c 2005-05-12 15:58:30.000000000 +0100
+++ linux/mm/madvise.c  2005-06-04 23:25:03.000000000 +0100
@@ -70,6 +70,17 @@
                end = vma->vm_end;
        end = ((end - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;

+       /*
+        * This doesn't account for pages that may already be mapped
+        * due to readahead, but since this is merely a hint to the
+        * kernel no real harm should be done, it will just make things
+        * run a little slower. Sometimes less is more! More than enough
+        * code for this minor corner case.
+        */
+       if (((max_sane_readahead(end-start) << 
PAGE_SHIFT)+current->mm->rss)>
+           current->signal->rlim[RLIMIT_RSS].rlim_cur)
+               return -ENOMEM;
+
        force_page_cache_readahead(file->f_mapping,
                        file, start, max_sane_readahead(end - start));
        return 0;