mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

Hi all,

I've ended up deleting the log file by mistake, but this bug does seem to be important
so I'd rather not wait before the same issue is triggered again.

The call chain is:

	mlock (mm/mlock.c:745)
		__mm_populate (mm/mlock.c:700)
			__mlock_vma_pages_range (mm/mlock.c:229)
				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));

It seems to be a rather simple trace triggered from userspace. The only recent patch
in the area (that I've noticed) was "mm/mlock: prepare params outside critical region".
I've reverted it and trying to testing without it.


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:
> Hi all,
> 
> I've ended up deleting the log file by mistake, but this bug does seem to be important
> so I'd rather not wait before the same issue is triggered again.
> 
> The call chain is:
> 
> 	mlock (mm/mlock.c:745)
> 		__mm_populate (mm/mlock.c:700)
> 			__mlock_vma_pages_range (mm/mlock.c:229)
> 				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));

So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
wrong as we call it without the lock held:

	up_write(&current->mm->mmap_sem);
	if (!error)
		error = __mm_populate(start, len, 0);
	return error;
}

> 
> It seems to be a rather simple trace triggered from userspace. The only recent patch
> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region".
> I've reverted it and trying to testing without it.

Odd, this patch should definitely *not* cause this. In any case every
operation removed from the critical region is local to the function:

	lock_limit = rlimit(RLIMIT_MEMLOCK);
	lock_limit >>= PAGE_SHIFT;
	locked = len >> PAGE_SHIFT;

	down_write(&current->mm->mmap_sem);

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/11/2014 04:07 PM, Davidlohr Bueso wrote:
> On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:
>> Hi all,
>>
>> I've ended up deleting the log file by mistake, but this bug does seem to be important
>> so I'd rather not wait before the same issue is triggered again.
>>
>> The call chain is:
>>
>> 	mlock (mm/mlock.c:745)
>> 		__mm_populate (mm/mlock.c:700)
>> 			__mlock_vma_pages_range (mm/mlock.c:229)
>> 				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
>
> So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
> wrong as we call it without the lock held:
>
> 	up_write(&current->mm->mmap_sem);
> 	if (!error)
> 		error = __mm_populate(start, len, 0);
> 	return error;
> }
>
>>
>> It seems to be a rather simple trace triggered from userspace. The only recent patch
>> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region".
>> I've reverted it and trying to testing without it.
>
> Odd, this patch should definitely *not* cause this. In any case every
> operation removed from the critical region is local to the function:
>
> 	lock_limit = rlimit(RLIMIT_MEMLOCK);
> 	lock_limit >>= PAGE_SHIFT;
> 	locked = len >> PAGE_SHIFT;
>
> 	down_write(&current->mm->mmap_sem);

Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this
code last - do you still remember what's going on here?" :).

It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time
I've seen it and it's probably been there for a while (that BUG_ON is there from 2009).


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 16:12 -0400, Sasha Levin wrote:
> On 03/11/2014 04:07 PM, Davidlohr Bueso wrote:
> > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:
> >> Hi all,
> >>
> >> I've ended up deleting the log file by mistake, but this bug does seem to be important
> >> so I'd rather not wait before the same issue is triggered again.
> >>
> >> The call chain is:
> >>
> >> 	mlock (mm/mlock.c:745)
> >> 		__mm_populate (mm/mlock.c:700)
> >> 			__mlock_vma_pages_range (mm/mlock.c:229)
> >> 				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
> >
> > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
> > wrong as we call it without the lock held:
> >
> > 	up_write(&current->mm->mmap_sem);
> > 	if (!error)
> > 		error = __mm_populate(start, len, 0);
> > 	return error;
> > }
> >
> >>
> >> It seems to be a rather simple trace triggered from userspace. The only recent patch
> >> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region".
> >> I've reverted it and trying to testing without it.
> >
> > Odd, this patch should definitely *not* cause this. In any case every
> > operation removed from the critical region is local to the function:
> >
> > 	lock_limit = rlimit(RLIMIT_MEMLOCK);
> > 	lock_limit >>= PAGE_SHIFT;
> > 	locked = len >> PAGE_SHIFT;
> >
> > 	down_write(&current->mm->mmap_sem);
> 
> Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this
> code last - do you still remember what's going on here?" :).

How frequently do you trigger this issue? Could you verify if it still
occurs by reverting my patch?

> It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time
> I've seen it and it's probably been there for a while (that BUG_ON is there from 2009).

Actually that VM_BUG_ON is correct, because we do in fact take the
mmap_sem (for reading) inside __mm_populate(), which in return calls
__mlock_vma_pages_range() with the lock held. Now, the lock is taken
within the for loop, which does the hole "if (!locked) down_read()"
dance, but it's just making sure that we take the lock upon the first
iteration. So besides doing the locking outside of the loop, which is
just a cleanup, I don't really see how it could be triggered.

Thanks,
Davidlohr

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Andrew Morton]

On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso <davidlohr@hp.com> wrote:

> On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:
> > Hi all,
> > 
> > I've ended up deleting the log file by mistake, but this bug does seem to be important
> > so I'd rather not wait before the same issue is triggered again.
> > 
> > The call chain is:
> > 
> > 	mlock (mm/mlock.c:745)
> > 		__mm_populate (mm/mlock.c:700)
> > 			__mlock_vma_pages_range (mm/mlock.c:229)
> > 				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
> 
> So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
> wrong as we call it without the lock held:
> 
> 	up_write(&current->mm->mmap_sem);
> 	if (!error)
> 		error = __mm_populate(start, len, 0);
> 	return error;
> }

__mm_populate() pretty clearly calls __mlock_vma_pages_range() under
down_read(mm->mmap_sem).

I worry about what happens if __get_user_pages decides to do

				if (ret & VM_FAULT_RETRY) {
					if (nonblocking)
						*nonblocking = 0;
					return i;
				}

uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll
forget to undo mmap_sem.  That won't explain this result, but it's a
potential problem.


All I can think is that find_vma() went and returned a vma from a
different mm, which would be odd.  How about I toss this in there?

--- a/mm/vmacache.c~a
+++ a/mm/vmacache.c
@@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str
 	for (i = 0; i < VMACACHE_SIZE; i++) {
 		struct vm_area_struct *vma = current->vmacache[i];
 
-		if (vma && vma->vm_start <= addr && vma->vm_end > addr)
+		if (vma && vma->vm_start <= addr && vma->vm_end > addr) {
+			BUG_ON(vma->vm_mm != mm);
 			return vma;
+		}
 	}
 
 	return NULL;
_

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/11/2014 04:30 PM, Andrew Morton wrote:
> I worry about what happens if __get_user_pages decides to do
>
> 				if (ret & VM_FAULT_RETRY) {
> 					if (nonblocking)
> 						*nonblocking = 0;
> 					return i;
> 				}
>
> uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll
> forget to undo mmap_sem.  That won't explain this result, but it's a
> potential problem.

That's actually seems right because if 'ret & VM_FAULT_RETRY' is true it means that
lock_page_or_retry() was supposed to release mmap_sem for us.


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 13:30 -0700, Andrew Morton wrote:
> On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso <davidlohr@hp.com> wrote:
> 
> > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:
> > > Hi all,
> > > 
> > > I've ended up deleting the log file by mistake, but this bug does seem to be important
> > > so I'd rather not wait before the same issue is triggered again.
> > > 
> > > The call chain is:
> > > 
> > > 	mlock (mm/mlock.c:745)
> > > 		__mm_populate (mm/mlock.c:700)
> > > 			__mlock_vma_pages_range (mm/mlock.c:229)
> > > 				VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
> > 
> > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
> > wrong as we call it without the lock held:
> > 
> > 	up_write(&current->mm->mmap_sem);
> > 	if (!error)
> > 		error = __mm_populate(start, len, 0);
> > 	return error;
> > }
> 
> __mm_populate() pretty clearly calls __mlock_vma_pages_range() under
> down_read(mm->mmap_sem).
> 
> I worry about what happens if __get_user_pages decides to do
> 
> 				if (ret & VM_FAULT_RETRY) {
> 					if (nonblocking)
> 						*nonblocking = 0;
> 					return i;
> 				}
> 
> uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll
> forget to undo mmap_sem.  That won't explain this result, but it's a
> potential problem.
> 
> 
> All I can think is that find_vma() went and returned a vma from a
> different mm, which would be odd.  How about I toss this in there?

... and we know that there is a bug (https://lkml.org/lkml/2014/3/9/201)
with stale caches going on. We seem to be missing an invalidation and/or
flush somewhere.

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/11/2014 04:30 PM, Andrew Morton wrote:
> All I can think is that find_vma() went and returned a vma from a
> different mm, which would be odd.  How about I toss this in there?
>
> --- a/mm/vmacache.c~a
> +++ a/mm/vmacache.c
> @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str
>   	for (i = 0; i < VMACACHE_SIZE; i++) {
>   		struct vm_area_struct *vma = current->vmacache[i];
>
> -		if (vma && vma->vm_start <= addr && vma->vm_end > addr)
> +		if (vma && vma->vm_start <= addr && vma->vm_end > addr) {
> +			BUG_ON(vma->vm_mm != mm);
>   			return vma;
> +		}
>   	}
>
>   	return NULL;

Bingo! With the above patch:

[  243.565794] kernel BUG at mm/vmacache.c:76!
[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  243.568048] Dumping ftrace buffer:
[  243.568740]    (ftrace buffer empty)
[  243.569481] Modules linked in:
[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
[  243.571505] task: ffff8800b8698000 ti: ffff8800b8694000 task.ti: ffff8800b8694000
[  243.571505] RIP: 0010:[<ffffffff81299396>]  [<ffffffff81299396>] vmacache_find+0x86/0xb0
[  243.571505] RSP: 0000:ffff8800b8695da8  EFLAGS: 00010287
[  243.571505] RAX: ffff88042890e400 RBX: 0000000002d6cb18 RCX: 0000000000000002
[  243.571505] RDX: 0000000000000002 RSI: 0000000002d6cb18 RDI: ffff8800b86a8000
[  243.571505] RBP: ffff8800b8695da8 R08: ffff8800b8698000 R09: 0000000000000000
[  243.571505] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8800b86a8000
[  243.571505] R13: 00000000000014f1 R14: ffff8800b86a80a8 R15: ffff8800b86a8000
[  243.582665] FS:  00007f74df081700(0000) GS:ffff880b2b800000(0000) knlGS:0000000000000000
[  243.582665] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  243.582665] CR2: 0000000002d6cb18 CR3: 00000000b8682000 CR4: 00000000000006a0
[  243.582665] Stack:
[  243.582665]  ffff8800b8695dd8 ffffffff812a7620 ffffffff844abd82 0000000002d6cb18
[  243.582665]  0000000002d6cb18 00000000000000a9 ffff8800b8695ef8 ffffffff844abdd6
[  243.582665]  ffff8800b8698000 ffff8800b8698000 0000000000000002 ffff8800b8698000
[  243.582665] Call Trace:
[  243.582665]  [<ffffffff812a7620>] find_vma+0x20/0x90
[  243.582665]  [<ffffffff844abd82>] ? __do_page_fault+0x302/0x5d0
[  243.582665]  [<ffffffff844abdd6>] __do_page_fault+0x356/0x5d0
[  243.582665]  [<ffffffff8118ab46>] ? vtime_account_user+0x96/0xb0
[  243.582665]  [<ffffffff844ac4d2>] ? preempt_count_sub+0xe2/0x120
[  243.582665]  [<ffffffff81269567>] ? context_tracking_user_exit+0x187/0x1d0
[  243.582665]  [<ffffffff844ac115>] do_page_fault+0x45/0x70
[  243.582665]  [<ffffffff844ab3c6>] do_async_page_fault+0x36/0x100
[  243.582665]  [<ffffffff844a7f58>] async_page_fault+0x28/0x30
[  243.582665] Code: 00 eb 44 66 90 31 d2 49 89 c0 48 63 ca 49 8b 84 c8 b8 07 00 00 48 85 c0 74 23 48 39 30 77 1e 48 3b 70 08 73 18 48 3b 78 40 74 1c <0f> 0b 0f 1f 84 00 00 00 00 00 eb fe 66 0f 1f 44 00 00 ff c2 83
[  243.582665] RIP  [<ffffffff81299396>] vmacache_find+0x86/0xb0
[  243.582665]  RSP <ffff8800b8695da8>


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote:
> On 03/11/2014 04:30 PM, Andrew Morton wrote:
> > All I can think is that find_vma() went and returned a vma from a
> > different mm, which would be odd.  How about I toss this in there?
> >
> > --- a/mm/vmacache.c~a
> > +++ a/mm/vmacache.c
> > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str
> >   	for (i = 0; i < VMACACHE_SIZE; i++) {
> >   		struct vm_area_struct *vma = current->vmacache[i];
> >
> > -		if (vma && vma->vm_start <= addr && vma->vm_end > addr)
> > +		if (vma && vma->vm_start <= addr && vma->vm_end > addr) {
> > +			BUG_ON(vma->vm_mm != mm);
> >   			return vma;
> > +		}
> >   	}
> >
> >   	return NULL;
> 
> Bingo! With the above patch:
> 
> [  243.565794] kernel BUG at mm/vmacache.c:76!
> [  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> [  243.568048] Dumping ftrace buffer:
> [  243.568740]    (ftrace buffer empty)
> [  243.569481] Modules linked in:
> [  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143

and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
the root cause it the same as Fengguang's report.

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Dave Jones]

On Tue, Mar 11, 2014 at 01:47:24PM -0700, Davidlohr Bueso wrote:
 > On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote:
 > > On 03/11/2014 04:30 PM, Andrew Morton wrote:
 > > > All I can think is that find_vma() went and returned a vma from a
 > > > different mm, which would be odd.  How about I toss this in there?
 > > >
 > > > --- a/mm/vmacache.c~a
 > > > +++ a/mm/vmacache.c
 > > > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str
 > > >   	for (i = 0; i < VMACACHE_SIZE; i++) {
 > > >   		struct vm_area_struct *vma = current->vmacache[i];
 > > >
 > > > -		if (vma && vma->vm_start <= addr && vma->vm_end > addr)
 > > > +		if (vma && vma->vm_start <= addr && vma->vm_end > addr) {
 > > > +			BUG_ON(vma->vm_mm != mm);
 > > >   			return vma;
 > > > +		}
 > > >   	}
 > > 
 > > [  243.565794] kernel BUG at mm/vmacache.c:76!
 > 
 > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
 > the root cause it the same as Fengguang's report.

btw, I added a (ugly) script to trinity.git this afternoon:
https://github.com/kernelslacker/trinity/commit/b6cfe63ea5b205a34bc6c5698e5b766dfcddad1d
which should make reproducing VM related bugs happen a lot faster.

While chasing one particular issue, that script has hit 2-3 others in my
testing so far today (All already known).

It would be really good if people other than me & Sasha started running stuff like this occasionally
to try and get some of the more annoying things fixed up faster. Though it's only really
useful with debug kernels, and I know from past experience that asking people to try running things
with debug options enabled is like pulling teeth..

	Dave

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
>> Bingo! With the above patch:
>> >
>> >[  243.565794] kernel BUG at mm/vmacache.c:76!
>> >[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>> >[  243.568048] Dumping ftrace buffer:
>> >[  243.568740]    (ftrace buffer empty)
>> >[  243.569481] Modules linked in:
>> >[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
> and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
> the root cause it the same as Fengguang's report.

The BUG still happens without DEBUG_PAGEALLOC.


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote:
> On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
> >> Bingo! With the above patch:
> >> >
> >> >[  243.565794] kernel BUG at mm/vmacache.c:76!
> >> >[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> >> >[  243.568048] Dumping ftrace buffer:
> >> >[  243.568740]    (ftrace buffer empty)
> >> >[  243.569481] Modules linked in:
> >> >[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
> > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
> > the root cause it the same as Fengguang's report.
> 
> The BUG still happens without DEBUG_PAGEALLOC.

Any idea what trinity itself is doing?

Could you add the following, I just want to make sure the bug isn't
being caused by an overflow:

diff --git a/mm/vmacache.c b/mm/vmacache.c
index add3162..abb85cc 100644
--- a/mm/vmacache.c
+++ b/mm/vmacache.c
@@ -17,6 +17,8 @@ void vmacache_flush_all(struct mm_struct *mm)
 {
        struct task_struct *g, *p;
 
+       WARN_ON_ONCE(1);
+
        rcu_read_lock();
        for_each_process_thread(g, p) {
                /*


Thanks.

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/11/2014 05:45 PM, Davidlohr Bueso wrote:
> On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote:
>> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
>>>> > >>Bingo! With the above patch:
>>>>> > >> >
>>>>> > >> >[  243.565794] kernel BUG at mm/vmacache.c:76!
>>>>> > >> >[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>>>>> > >> >[  243.568048] Dumping ftrace buffer:
>>>>> > >> >[  243.568740]    (ftrace buffer empty)
>>>>> > >> >[  243.569481] Modules linked in:
>>>>> > >> >[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
>>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
>>> > >the root cause it the same as Fengguang's report.
>> >
>> >The BUG still happens without DEBUG_PAGEALLOC.
> Any idea what trinity itself is doing?
>
> Could you add the following, I just want to make sure the bug isn't
> being caused by an overflow:

Not hitting that WARN.


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Davidlohr Bueso]

On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote:
> On 03/11/2014 05:45 PM, Davidlohr Bueso wrote:
> > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote:
> >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
> >>>> > >>Bingo! With the above patch:
> >>>>> > >> >
> >>>>> > >> >[  243.565794] kernel BUG at mm/vmacache.c:76!
> >>>>> > >> >[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> >>>>> > >> >[  243.568048] Dumping ftrace buffer:
> >>>>> > >> >[  243.568740]    (ftrace buffer empty)
> >>>>> > >> >[  243.569481] Modules linked in:
> >>>>> > >> >[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
> >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
> >>> > >the root cause it the same as Fengguang's report.
> >> >
> >> >The BUG still happens without DEBUG_PAGEALLOC.
> > Any idea what trinity itself is doing?
> >
> > Could you add the following, I just want to make sure the bug isn't
> > being caused by an overflow:
> 
> Not hitting that WARN.

Sasha, could you please try the following patch:
https://lkml.org/lkml/2014/3/13/312

thanks.

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Hugh Dickins]

On Thu, 13 Mar 2014, Davidlohr Bueso wrote:
> On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote:
> > On 03/11/2014 05:45 PM, Davidlohr Bueso wrote:
> > > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote:
> > >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
> > >>>> > >>Bingo! With the above patch:
> > >>>>> > >> >
> > >>>>> > >> >[  243.565794] kernel BUG at mm/vmacache.c:76!
> > >>>>> > >> >[  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> > >>>>> > >> >[  243.568048] Dumping ftrace buffer:
> > >>>>> > >> >[  243.568740]    (ftrace buffer empty)
> > >>>>> > >> >[  243.569481] Modules linked in:
> > >>>>> > >> >[  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
> > >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
> > >>> > >the root cause it the same as Fengguang's report.
> > >> >
> > >> >The BUG still happens without DEBUG_PAGEALLOC.
> > > Any idea what trinity itself is doing?
> > >
> > > Could you add the following, I just want to make sure the bug isn't
> > > being caused by an overflow:
> > 
> > Not hitting that WARN.
> 
> Sasha, could you please try the following patch:
> https://lkml.org/lkml/2014/3/13/312

I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM
on mmotm: Oleg's patch (buildable version below) fixes it for me.

Hugh

--- mmotm/mm/vmacache.c	2014-03-12 18:39:38.008011317 -0700
+++ linux/mm/vmacache.c	2014-03-13 12:21:11.592030813 -0700
@@ -31,15 +31,20 @@ void vmacache_flush_all(struct mm_struct
 	rcu_read_unlock();
 }
 
+static bool vmacache_valid_mm(struct mm_struct *mm)
+{
+	return current->mm == mm && !(current->flags & PF_KTHREAD);
+}
+
 void vmacache_update(unsigned long addr, struct vm_area_struct *newvma)
 {
-	int idx = VMACACHE_HASH(addr);
-	current->vmacache[idx] = newvma;
+	if (vmacache_valid_mm(newvma->vm_mm))
+		current->vmacache[VMACACHE_HASH(addr)] = newvma;
 }
 
 static bool vmacache_valid(struct mm_struct *mm)
 {
-	struct task_struct *curr = current;
+	struct task_struct *curr;
 
 	/*
 	 * This task may be accessing a foreign mm via (for example)
@@ -47,9 +52,10 @@ static bool vmacache_valid(struct mm_str
 	 * task's vmacache pertains to a different mm (ie, its own).  There is
 	 * nothing we can do here.
 	 */
-	if (mm != curr->mm)
-		return false;
+	if (!vmacache_valid_mm(mm))
+ 		return false;
 
+	curr = current;
 	if (mm->vmacache_seqnum != curr->vmacache_seqnum) {
 		/*
 		 * First attempt will always be invalid, initialize

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Sasha Levin]

On 03/13/2014 04:57 PM, Hugh Dickins wrote:
> On Thu, 13 Mar 2014, Davidlohr Bueso wrote:
>> On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote:
>>> On 03/11/2014 05:45 PM, Davidlohr Bueso wrote:
>>>> On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote:
>>>>>> On 03/11/2014 04:47 PM, Davidlohr Bueso wrote:
>>>>>>>>>> Bingo! With the above patch:
>>>>>>>>>>>>
>>>>>>>>>>>> [  243.565794] kernel BUG at mm/vmacache.c:76!
>>>>>>>>>>>> [  243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
>>>>>>>>>>>> [  243.568048] Dumping ftrace buffer:
>>>>>>>>>>>> [  243.568740]    (ftrace buffer empty)
>>>>>>>>>>>> [  243.569481] Modules linked in:
>>>>>>>>>>>> [  243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G        W    3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143
>>>>>>>> and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect
>>>>>>>> the root cause it the same as Fengguang's report.
>>>>>>
>>>>>> The BUG still happens without DEBUG_PAGEALLOC.
>>>> Any idea what trinity itself is doing?
>>>>
>>>> Could you add the following, I just want to make sure the bug isn't
>>>> being caused by an overflow:
>>>
>>> Not hitting that WARN.
>>
>> Sasha, could you please try the following patch:
>> https://lkml.org/lkml/2014/3/13/312
>
> I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM
> on mmotm: Oleg's patch (buildable version below) fixes it for me.

Sorry for the delay, some patch in the last -next broke boot and I had to spend
a while waiting for the bisect before I could test this patch.

The patch fixes the vmacache issues I've been seeing.


Thanks,
Sasha

Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range

[Michel Lespinasse]

On Tue, Mar 11, 2014 at 1:07 PM, Davidlohr Bueso <davidlohr@hp.com> wrote:
> On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote:

>> I've ended up deleting the log file by mistake, but this bug does seem to be important
>> so I'd rather not wait before the same issue is triggered again.
>>
>> The call chain is:
>>
>>       mlock (mm/mlock.c:745)
>>               __mm_populate (mm/mlock.c:700)
>>                       __mlock_vma_pages_range (mm/mlock.c:229)
>>                               VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
>
> So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems
> wrong as we call it without the lock held:

Not related to the bug, but please note that __mm_populate() is public
and has other call sites outside of mlock.c - namely, it is called
during mmap with MAP_POPULATE.

-- 
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.