Re: [PATCH] BUG #3054 madvise doesn't fail for exceding RSS limit.

[Matt <matthew.keenan@ntlworld.com>]

sorry that error return should be -EIO not -ENOMEM, sorry! new patch 
attached.

--- linux-2.6.11.10/mm/madvise.c 2005-04-12 15:58:30.000000000 +0100
+++ linux/mm/madvise.c  2005-06-04 23:25:03.000000000 +0100
@@ -70,6 +70,17 @@
                end = vma->vm_end;
        end = ((end - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;

+       /*
+        * This doesn't account for pages that may already be mapped
+        * due to readahead, but since this is merely a hint to the
+        * kernel no real harm should be done, it will just make things
+        * run a little slower. Sometimes less is more! More than enough
+        * code for this minor corner case.
+        */
+       if (((max_sane_readahead(end-start) << 
PAGE_SHIFT)+current->mm->rss)>
+           current->signal->rlim[RLIMIT_RSS].rlim_cur)
+               return -EIO;
+
        force_page_cache_readahead(file->f_mapping,
                        file, start, max_sane_readahead(end - start));
        return 0;


Matt wrote:

> Patch that fixes bug #3054. Compiles, tests ok.
>
> --- linux-2.6.11.10/mm/madvise.c 2005-05-12 15:58:30.000000000 +0100
> +++ linux/mm/madvise.c  2005-06-04 23:25:03.000000000 +0100
> @@ -70,6 +70,17 @@
>                end = vma->vm_end;
>        end = ((end - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
>
> +       /*
> +        * This doesn't account for pages that may already be mapped
> +        * due to readahead, but since this is merely a hint to the
> +        * kernel no real harm should be done, it will just make things
> +        * run a little slower. Sometimes less is more! More than enough
> +        * code for this minor corner case.
> +        */
> +       if (((max_sane_readahead(end-start) << 
> PAGE_SHIFT)+current->mm->rss)>
> +           current->signal->rlim[RLIMIT_RSS].rlim_cur)
> +               return -ENOMEM;
> +
>        force_page_cache_readahead(file->f_mapping,
>                        file, start, max_sane_readahead(end - start));
>        return 0;
>
>