[PATCH] fix madvise vma merging

[PATCH] fix madvise vma merging

[Hugh Dickins]

Better late than never, I've at last reviewed the madvise vma merging
going into 2.6.13.  Remove a pointless check and fix two little bugs -
a simple test (with /proc/<pid>/maps hacked to show ReadHints) showed
both mismerges in practice: though being madvise, neither was disastrous.

1. Correct placement of the success label in madvise_behavior: as in
   mprotect_fixup and mlock_fixup, it is necessary to update vm_flags
   when vma_merge succeeds (to handle the exceptional Case 8 noted in
   the comments above vma_merge itself).

2. Correct initial value of prev when starting part way into a vma: as
   in sys_mprotect and do_mlock, it needs to be set to vma in this case
   (vma_merge handles only that minimum of cases shown in its comments).

3. If find_vma_prev sets prev, then the vma it returns is prev->vm_next,
   so it's pointless to make that same assignment again in sys_madvise.

Signed-off-by: Hugh Dickins <hugh@veritas.com>

--- 2.6.13-rc5-git3/mm/madvise.c	2005-08-02 12:07:23.000000000 +0100
+++ linux/mm/madvise.c	2005-08-05 18:06:47.000000000 +0100
@@ -37,7 +37,7 @@ static long madvise_behavior(struct vm_a
 
 	if (new_flags == vma->vm_flags) {
 		*prev = vma;
-		goto success;
+		goto out;
 	}
 
 	pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);
@@ -62,6 +62,7 @@ static long madvise_behavior(struct vm_a
 			goto out;
 	}
 
+success:
 	/*
 	 * vm_flags is protected by the mmap_sem held in write mode.
 	 */
@@ -70,7 +71,6 @@ static long madvise_behavior(struct vm_a
 out:
 	if (error == -ENOMEM)
 		error = -EAGAIN;
-success:
 	return error;
 }
 
@@ -237,8 +237,9 @@ asmlinkage long sys_madvise(unsigned lon
 	 * - different from the way of handling in mlock etc.
 	 */
 	vma = find_vma_prev(current->mm, start, &prev);
-	if (!vma && prev)
-		vma = prev->vm_next;
+	if (vma && start > vma->vm_start)
+		prev = vma;
+
 	for (;;) {
 		/* Still start < end. */
 		error = -ENOMEM;

Re: [PATCH] fix madvise vma merging

[Prasanna Meda]

Hugh Dickins wrote:

> Better late than never, I've at last reviewed the madvise vma merging
> going into 2.6.13.  Remove a pointless check and fix two little bugs -
> a simple test (with /proc/<pid>/maps hacked to show ReadHints) showed
> both mismerges in practice: though being madvise, neither was disastrous.
>
> 1. Correct placement of the success label in madvise_behavior: as in
>    mprotect_fixup and mlock_fixup, it is necessary to update vm_flags
>    when vma_merge succeeds (to handle the exceptional Case 8 noted in
>    the comments above vma_merge itself).
>
> 2. Correct initial value of prev when starting part way into a vma: as
>    in sys_mprotect and do_mlock, it needs to be set to vma in this case
>    (vma_merge handles only that minimum of cases shown in its comments).
>
> 3. If find_vma_prev sets prev, then the vma it returns is prev->vm_next,
>    so it's pointless to make that same assignment again in sys_madvise.

Acknowledge corrections 1 and  3 readily.  Treated vma_merge
as block box that can handle all cases.  Motivation for pointless
case 3 is to skip holes and did not notice that has been covered.  Thanks for
corrections.

>  (vma_merge handles only that minimum of cases shown in its comments).
>

Correction 2 is tricky.  Sometimes it merges similar to case 3,
misses a needed split,  where after the fix  we can get case 4
merge. If that is what you are saying, we are in agreement.  Otherwise,
can you explain the real problem?


Thanks,
Prasanna.

Re: [PATCH] fix madvise vma merging

[Hugh Dickins]

On Fri, 5 Aug 2005, Prasanna Meda wrote:
> Hugh Dickins wrote:
> 
> > 2. Correct initial value of prev when starting part way into a vma: as
> >    in sys_mprotect and do_mlock, it needs to be set to vma in this case
> >    (vma_merge handles only that minimum of cases shown in its comments).
> 
> Acknowledge corrections 1 and  3 readily.  Treated vma_merge
> as block box that can handle all cases.  Motivation for pointless
> case 3 is to skip holes and did not notice that has been covered.
> Thanks for corrections.

And thanks for the confirmations.

> Correction 2 is tricky.  Sometimes it merges similar to case 3,
> misses a needed split,  where after the fix  we can get case 4
> merge. If that is what you are saying, we are in agreement.
> Otherwise, can you explain the real problem?

I probably am saying what you are saying there,
but it's hard for me to understand it that way.

Missing out the "start > vma->vm_start" adjustment of prev introduces
additional (but redundant: non-canonical) cases not considered at all
by vma_merge, now entered with a "prev" which is remote and surely
irrelevant to merging.  "misses a needed split", yes, I saw that;
indeed my test ended up taking the "cases 3, 8" path, when, given
the right prev, it should have been handled as a "case 4".

mmap(0x80000000, 0x3000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, fd, 0);
madvise(0x80000000, 0x2000, MADV_RANDOM);
madvise(0x80001000, 0x1000, MADV_NORMAL);

ended up (if I'm remembering it aright) as one single VM_RAND_READ vma,
when it should have been a one page VM_RAND_READ vma followed by a two
page normal vma.  Merged into one because of the incorrect prev, and
VM_RAND_READ rather than normal because of the misplaced success label.

Hugh

Re: [PATCH] fix madvise vma merging

[Prasanna Meda]

Hugh Dickins wrote:

> On Fri, 5 Aug 2005, Prasanna Meda wrote:
> > Hugh Dickins wrote:
> >
> > > 2. Correct initial value of prev when starting part way into a vma: as
> > >    in sys_mprotect and do_mlock, it needs to be set to vma in this case
> > >    (vma_merge handles only that minimum of cases shown in its comments).
> >
> > Acknowledge corrections 1 and  3 readily.  Treated vma_merge
> > as block box that can handle all cases.  Motivation for pointless
> > case 3 is to skip holes and did not notice that has been covered.
> > Thanks for corrections.
>
> And thanks for the confirmations.
>
> > Correction 2 is tricky.  Sometimes it merges similar to case 3,
> > misses a needed split,  where after the fix  we can get case 4
> > merge. If that is what you are saying, we are in agreement.
> > Otherwise, can you explain the real problem?
>
> I probably am saying what you are saying there,
> but it's hard for me to understand it that way.
>
> Missing out the "start > vma->vm_start" adjustment of prev introduces
> additional (but redundant: non-canonical) cases not considered at all
> by vma_merge, now entered with a "prev" which is remote and surely
> irrelevant to merging.  "misses a needed split", yes, I saw that;
> indeed my test ended up taking the "cases 3, 8" path, when, given
> the right prev, it should have been handled as a "case 4".
>

Ok,   we both are on the same page.  Your obseravtions are same.
Thanks a lot for the  code review and finding corner cases.

-Prasanna.