Re: Bug in 2.4.26 in mm/filemap.c when using RLIMIT_RSS

[Marcelo Tosatti <marcelo.tosatti@cyclades.com>]

On Wed, Jan 26, 2005 at 12:07:50PM +0100, Ake wrote:
> Use of rlim[RLIMIT_RSS] in mm/filemap.c is wrong.
> It is passed down to kernel as a number of bytes but is being used as a
> number of pages.
> 
> There is also a misinformative comment in fs/proc/array.c
> in proc_pid_stat where it says
> mm ? mm->rss : 0, /* you might want to shift this left 3 */
> the number 3 should probably be PAGE_SHIFT-10.

Amazing that this has never been noticed before - I bet not many people use RSS 
limits with madvise().

This transform the rlimit in pages before the comparison, can you please test
it.

--- a/mm/filemap.c.orig	2004-11-17 09:54:22.000000000 -0200
+++ b/mm/filemap.c	2005-01-26 15:21:10.614842296 -0200
@@ -2609,6 +2609,9 @@
 	error = -EIO;
 	rlim_rss = current->rlim ?  current->rlim[RLIMIT_RSS].rlim_cur :
 				LONG_MAX; /* default: see resource.h */
+
+	rlim_rss = (rlim_rss & PAGE_MASK) >> PAGE_SHIFT;
+
 	if ((vma->vm_mm->rss + (end - start)) > rlim_rss)
 		return error;