From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-f175.google.com (mail-yk0-f175.google.com [209.85.160.175]) by kanga.kvack.org (Postfix) with ESMTP id 8F7AB6B0037 for ; Tue, 11 Mar 2014 15:40:00 -0400 (EDT) Received: by mail-yk0-f175.google.com with SMTP id 131so24361574ykp.6 for ; Tue, 11 Mar 2014 12:40:00 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id z48si37935598yhb.86.2014.03.11.12.39.59 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 12:39:59 -0700 (PDT) Message-ID: <531F6689.60307@oracle.com> Date: Tue, 11 Mar 2014 15:39:53 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: "linux-mm@kvack.org" Cc: Andrew Morton , Davidlohr Bueso , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Hi all, I've ended up deleting the log file by mistake, but this bug does seem to be important so I'd rather not wait before the same issue is triggered again. The call chain is: mlock (mm/mlock.c:745) __mm_populate (mm/mlock.c:700) __mlock_vma_pages_range (mm/mlock.c:229) VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); It seems to be a rather simple trace triggered from userspace. The only recent patch in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". I've reverted it and trying to testing without it. Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-f170.google.com (mail-ob0-f170.google.com [209.85.214.170]) by kanga.kvack.org (Postfix) with ESMTP id 0E2826B0036 for ; Tue, 11 Mar 2014 16:07:45 -0400 (EDT) Received: by mail-ob0-f170.google.com with SMTP id uz6so8981484obc.1 for ; Tue, 11 Mar 2014 13:07:44 -0700 (PDT) Received: from g2t2353.austin.hp.com (g2t2353.austin.hp.com. [15.217.128.52]) by mx.google.com with ESMTPS id m4si22413754oel.126.2014.03.11.13.07.44 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:07:44 -0700 (PDT) Message-ID: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Tue, 11 Mar 2014 13:07:33 -0700 In-Reply-To: <531F6689.60307@oracle.com> References: <531F6689.60307@oracle.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin Cc: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > Hi all, > > I've ended up deleting the log file by mistake, but this bug does seem to be important > so I'd rather not wait before the same issue is triggered again. > > The call chain is: > > mlock (mm/mlock.c:745) > __mm_populate (mm/mlock.c:700) > __mlock_vma_pages_range (mm/mlock.c:229) > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems wrong as we call it without the lock held: up_write(¤t->mm->mmap_sem); if (!error) error = __mm_populate(start, len, 0); return error; } > > It seems to be a rather simple trace triggered from userspace. The only recent patch > in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". > I've reverted it and trying to testing without it. Odd, this patch should definitely *not* cause this. In any case every operation removed from the critical region is local to the function: lock_limit = rlimit(RLIMIT_MEMLOCK); lock_limit >>= PAGE_SHIFT; locked = len >> PAGE_SHIFT; down_write(¤t->mm->mmap_sem); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f43.google.com (mail-yh0-f43.google.com [209.85.213.43]) by kanga.kvack.org (Postfix) with ESMTP id 878D96B0035 for ; Tue, 11 Mar 2014 16:12:57 -0400 (EDT) Received: by mail-yh0-f43.google.com with SMTP id b6so9203597yha.30 for ; Tue, 11 Mar 2014 13:12:57 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id t49si38141945yhd.9.2014.03.11.13.12.56 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:12:57 -0700 (PDT) Message-ID: <531F6E43.40901@oracle.com> Date: Tue, 11 Mar 2014 16:12:51 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/11/2014 04:07 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: >> Hi all, >> >> I've ended up deleting the log file by mistake, but this bug does seem to be important >> so I'd rather not wait before the same issue is triggered again. >> >> The call chain is: >> >> mlock (mm/mlock.c:745) >> __mm_populate (mm/mlock.c:700) >> __mlock_vma_pages_range (mm/mlock.c:229) >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: > > up_write(¤t->mm->mmap_sem); > if (!error) > error = __mm_populate(start, len, 0); > return error; > } > >> >> It seems to be a rather simple trace triggered from userspace. The only recent patch >> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". >> I've reverted it and trying to testing without it. > > Odd, this patch should definitely *not* cause this. In any case every > operation removed from the critical region is local to the function: > > lock_limit = rlimit(RLIMIT_MEMLOCK); > lock_limit >>= PAGE_SHIFT; > locked = len >> PAGE_SHIFT; > > down_write(¤t->mm->mmap_sem); Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this code last - do you still remember what's going on here?" :). It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time I've seen it and it's probably been there for a while (that BUG_ON is there from 2009). Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f41.google.com (mail-oa0-f41.google.com [209.85.219.41]) by kanga.kvack.org (Postfix) with ESMTP id ED5F26B0035 for ; Tue, 11 Mar 2014 16:21:38 -0400 (EDT) Received: by mail-oa0-f41.google.com with SMTP id j17so9290290oag.0 for ; Tue, 11 Mar 2014 13:21:38 -0700 (PDT) Received: from g2t2352.austin.hp.com (g2t2352.austin.hp.com. [15.217.128.51]) by mx.google.com with ESMTPS id 2si6603759oep.54.2014.03.11.13.21.38 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:21:38 -0700 (PDT) Message-ID: <1394569297.2786.36.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Tue, 11 Mar 2014 13:21:37 -0700 In-Reply-To: <531F6E43.40901@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <531F6E43.40901@oracle.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin Cc: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 16:12 -0400, Sasha Levin wrote: > On 03/11/2014 04:07 PM, Davidlohr Bueso wrote: > > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > >> Hi all, > >> > >> I've ended up deleting the log file by mistake, but this bug does seem to be important > >> so I'd rather not wait before the same issue is triggered again. > >> > >> The call chain is: > >> > >> mlock (mm/mlock.c:745) > >> __mm_populate (mm/mlock.c:700) > >> __mlock_vma_pages_range (mm/mlock.c:229) > >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > > wrong as we call it without the lock held: > > > > up_write(¤t->mm->mmap_sem); > > if (!error) > > error = __mm_populate(start, len, 0); > > return error; > > } > > > >> > >> It seems to be a rather simple trace triggered from userspace. The only recent patch > >> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". > >> I've reverted it and trying to testing without it. > > > > Odd, this patch should definitely *not* cause this. In any case every > > operation removed from the critical region is local to the function: > > > > lock_limit = rlimit(RLIMIT_MEMLOCK); > > lock_limit >>= PAGE_SHIFT; > > locked = len >> PAGE_SHIFT; > > > > down_write(¤t->mm->mmap_sem); > > Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this > code last - do you still remember what's going on here?" :). How frequently do you trigger this issue? Could you verify if it still occurs by reverting my patch? > It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time > I've seen it and it's probably been there for a while (that BUG_ON is there from 2009). Actually that VM_BUG_ON is correct, because we do in fact take the mmap_sem (for reading) inside __mm_populate(), which in return calls __mlock_vma_pages_range() with the lock held. Now, the lock is taken within the for loop, which does the hole "if (!locked) down_read()" dance, but it's just making sure that we take the lock upon the first iteration. So besides doing the locking outside of the loop, which is just a cleanup, I don't really see how it could be triggered. Thanks, Davidlohr -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) by kanga.kvack.org (Postfix) with ESMTP id 9FB2C6B0035 for ; Tue, 11 Mar 2014 16:30:54 -0400 (EDT) Received: by mail-pd0-f175.google.com with SMTP id x10so58806pdj.6 for ; Tue, 11 Mar 2014 13:30:54 -0700 (PDT) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org. [140.211.169.12]) by mx.google.com with ESMTP id tu7si106233pac.193.2014.03.11.13.30.52 for ; Tue, 11 Mar 2014 13:30:53 -0700 (PDT) Date: Tue, 11 Mar 2014 13:30:51 -0700 From: Andrew Morton Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Message-Id: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Sasha Levin , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > > Hi all, > > > > I've ended up deleting the log file by mistake, but this bug does seem to be important > > so I'd rather not wait before the same issue is triggered again. > > > > The call chain is: > > > > mlock (mm/mlock.c:745) > > __mm_populate (mm/mlock.c:700) > > __mlock_vma_pages_range (mm/mlock.c:229) > > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: > > up_write(¤t->mm->mmap_sem); > if (!error) > error = __mm_populate(start, len, 0); > return error; > } __mm_populate() pretty clearly calls __mlock_vma_pages_range() under down_read(mm->mmap_sem). I worry about what happens if __get_user_pages decides to do if (ret & VM_FAULT_RETRY) { if (nonblocking) *nonblocking = 0; return i; } uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll forget to undo mmap_sem. That won't explain this result, but it's a potential problem. All I can think is that find_vma() went and returned a vma from a different mm, which would be odd. How about I toss this in there? --- a/mm/vmacache.c~a +++ a/mm/vmacache.c @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str for (i = 0; i < VMACACHE_SIZE; i++) { struct vm_area_struct *vma = current->vmacache[i]; - if (vma && vma->vm_start <= addr && vma->vm_end > addr) + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { + BUG_ON(vma->vm_mm != mm); return vma; + } } return NULL; _ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-f173.google.com (mail-yk0-f173.google.com [209.85.160.173]) by kanga.kvack.org (Postfix) with ESMTP id 363D36B0035 for ; Tue, 11 Mar 2014 16:36:01 -0400 (EDT) Received: by mail-yk0-f173.google.com with SMTP id 10so24494691ykt.4 for ; Tue, 11 Mar 2014 13:36:01 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id q45si38200426yhb.95.2014.03.11.13.36.00 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:36:00 -0700 (PDT) Message-ID: <531F73AB.5060402@oracle.com> Date: Tue, 11 Mar 2014 16:35:55 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , Davidlohr Bueso Cc: "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/11/2014 04:30 PM, Andrew Morton wrote: > I worry about what happens if __get_user_pages decides to do > > if (ret & VM_FAULT_RETRY) { > if (nonblocking) > *nonblocking = 0; > return i; > } > > uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll > forget to undo mmap_sem. That won't explain this result, but it's a > potential problem. That's actually seems right because if 'ret & VM_FAULT_RETRY' is true it means that lock_page_or_retry() was supposed to release mmap_sem for us. Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f48.google.com (mail-oa0-f48.google.com [209.85.219.48]) by kanga.kvack.org (Postfix) with ESMTP id 6A9386B0035 for ; Tue, 11 Mar 2014 16:42:46 -0400 (EDT) Received: by mail-oa0-f48.google.com with SMTP id m1so9216266oag.35 for ; Tue, 11 Mar 2014 13:42:46 -0700 (PDT) Received: from g5t1625.atlanta.hp.com (g5t1625.atlanta.hp.com. [15.192.137.8]) by mx.google.com with ESMTPS id tm2si22508885oeb.81.2014.03.11.13.42.45 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:42:45 -0700 (PDT) Message-ID: <1394570564.2786.40.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Tue, 11 Mar 2014 13:42:44 -0700 In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Sasha Levin , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 13:30 -0700, Andrew Morton wrote: > On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso wrote: > > > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > > > Hi all, > > > > > > I've ended up deleting the log file by mistake, but this bug does seem to be important > > > so I'd rather not wait before the same issue is triggered again. > > > > > > The call chain is: > > > > > > mlock (mm/mlock.c:745) > > > __mm_populate (mm/mlock.c:700) > > > __mlock_vma_pages_range (mm/mlock.c:229) > > > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > > wrong as we call it without the lock held: > > > > up_write(¤t->mm->mmap_sem); > > if (!error) > > error = __mm_populate(start, len, 0); > > return error; > > } > > __mm_populate() pretty clearly calls __mlock_vma_pages_range() under > down_read(mm->mmap_sem). > > I worry about what happens if __get_user_pages decides to do > > if (ret & VM_FAULT_RETRY) { > if (nonblocking) > *nonblocking = 0; > return i; > } > > uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll > forget to undo mmap_sem. That won't explain this result, but it's a > potential problem. > > > All I can think is that find_vma() went and returned a vma from a > different mm, which would be odd. How about I toss this in there? ... and we know that there is a bug (https://lkml.org/lkml/2014/3/9/201) with stale caches going on. We seem to be missing an invalidation and/or flush somewhere. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f48.google.com (mail-yh0-f48.google.com [209.85.213.48]) by kanga.kvack.org (Postfix) with ESMTP id 57BA56B0035 for ; Tue, 11 Mar 2014 16:45:21 -0400 (EDT) Received: by mail-yh0-f48.google.com with SMTP id z6so9247570yhz.35 for ; Tue, 11 Mar 2014 13:45:21 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id f64si38242493yhq.104.2014.03.11.13.45.20 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:45:20 -0700 (PDT) Message-ID: <531F75D1.3060909@oracle.com> Date: Tue, 11 Mar 2014 16:45:05 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , Davidlohr Bueso Cc: "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/11/2014 04:30 PM, Andrew Morton wrote: > All I can think is that find_vma() went and returned a vma from a > different mm, which would be odd. How about I toss this in there? > > --- a/mm/vmacache.c~a > +++ a/mm/vmacache.c > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > for (i = 0; i < VMACACHE_SIZE; i++) { > struct vm_area_struct *vma = current->vmacache[i]; > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > + BUG_ON(vma->vm_mm != mm); > return vma; > + } > } > > return NULL; Bingo! With the above patch: [ 243.565794] kernel BUG at mm/vmacache.c:76! [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 243.568048] Dumping ftrace buffer: [ 243.568740] (ftrace buffer empty) [ 243.569481] Modules linked in: [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 [ 243.571505] task: ffff8800b8698000 ti: ffff8800b8694000 task.ti: ffff8800b8694000 [ 243.571505] RIP: 0010:[] [] vmacache_find+0x86/0xb0 [ 243.571505] RSP: 0000:ffff8800b8695da8 EFLAGS: 00010287 [ 243.571505] RAX: ffff88042890e400 RBX: 0000000002d6cb18 RCX: 0000000000000002 [ 243.571505] RDX: 0000000000000002 RSI: 0000000002d6cb18 RDI: ffff8800b86a8000 [ 243.571505] RBP: ffff8800b8695da8 R08: ffff8800b8698000 R09: 0000000000000000 [ 243.571505] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8800b86a8000 [ 243.571505] R13: 00000000000014f1 R14: ffff8800b86a80a8 R15: ffff8800b86a8000 [ 243.582665] FS: 00007f74df081700(0000) GS:ffff880b2b800000(0000) knlGS:0000000000000000 [ 243.582665] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 243.582665] CR2: 0000000002d6cb18 CR3: 00000000b8682000 CR4: 00000000000006a0 [ 243.582665] Stack: [ 243.582665] ffff8800b8695dd8 ffffffff812a7620 ffffffff844abd82 0000000002d6cb18 [ 243.582665] 0000000002d6cb18 00000000000000a9 ffff8800b8695ef8 ffffffff844abdd6 [ 243.582665] ffff8800b8698000 ffff8800b8698000 0000000000000002 ffff8800b8698000 [ 243.582665] Call Trace: [ 243.582665] [] find_vma+0x20/0x90 [ 243.582665] [] ? __do_page_fault+0x302/0x5d0 [ 243.582665] [] __do_page_fault+0x356/0x5d0 [ 243.582665] [] ? vtime_account_user+0x96/0xb0 [ 243.582665] [] ? preempt_count_sub+0xe2/0x120 [ 243.582665] [] ? context_tracking_user_exit+0x187/0x1d0 [ 243.582665] [] do_page_fault+0x45/0x70 [ 243.582665] [] do_async_page_fault+0x36/0x100 [ 243.582665] [] async_page_fault+0x28/0x30 [ 243.582665] Code: 00 eb 44 66 90 31 d2 49 89 c0 48 63 ca 49 8b 84 c8 b8 07 00 00 48 85 c0 74 23 48 39 30 77 1e 48 3b 70 08 73 18 48 3b 78 40 74 1c <0f> 0b 0f 1f 84 00 00 00 00 00 eb fe 66 0f 1f 44 00 00 ff c2 83 [ 243.582665] RIP [] vmacache_find+0x86/0xb0 [ 243.582665] RSP Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f41.google.com (mail-oa0-f41.google.com [209.85.219.41]) by kanga.kvack.org (Postfix) with ESMTP id 477C16B0035 for ; Tue, 11 Mar 2014 16:47:26 -0400 (EDT) Received: by mail-oa0-f41.google.com with SMTP id j17so9289562oag.28 for ; Tue, 11 Mar 2014 13:47:25 -0700 (PDT) Received: from g5t1626.atlanta.hp.com (g5t1626.atlanta.hp.com. [15.192.137.9]) by mx.google.com with ESMTPS id uv2si25523900obb.19.2014.03.11.13.47.25 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 13:47:25 -0700 (PDT) Message-ID: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Tue, 11 Mar 2014 13:47:24 -0700 In-Reply-To: <531F75D1.3060909@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote: > On 03/11/2014 04:30 PM, Andrew Morton wrote: > > All I can think is that find_vma() went and returned a vma from a > > different mm, which would be odd. How about I toss this in there? > > > > --- a/mm/vmacache.c~a > > +++ a/mm/vmacache.c > > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > > for (i = 0; i < VMACACHE_SIZE; i++) { > > struct vm_area_struct *vma = current->vmacache[i]; > > > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > > + BUG_ON(vma->vm_mm != mm); > > return vma; > > + } > > } > > > > return NULL; > > Bingo! With the above patch: > > [ 243.565794] kernel BUG at mm/vmacache.c:76! > [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > [ 243.568048] Dumping ftrace buffer: > [ 243.568740] (ftrace buffer empty) > [ 243.569481] Modules linked in: > [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect the root cause it the same as Fengguang's report. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f46.google.com (mail-yh0-f46.google.com [209.85.213.46]) by kanga.kvack.org (Postfix) with ESMTP id 61E0E6B0035 for ; Tue, 11 Mar 2014 17:02:55 -0400 (EDT) Received: by mail-yh0-f46.google.com with SMTP id b6so6261072yha.5 for ; Tue, 11 Mar 2014 14:02:55 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id v46si38327526yhh.87.2014.03.11.14.02.54 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 14:02:54 -0700 (PDT) Message-ID: <531F79F7.5090201@oracle.com> Date: Tue, 11 Mar 2014 17:02:47 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >> Bingo! With the above patch: >> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >> >[ 243.568048] Dumping ftrace buffer: >> >[ 243.568740] (ftrace buffer empty) >> >[ 243.569481] Modules linked in: >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > the root cause it the same as Fengguang's report. The BUG still happens without DEBUG_PAGEALLOC. Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f41.google.com (mail-pb0-f41.google.com [209.85.160.41]) by kanga.kvack.org (Postfix) with ESMTP id 282B66B0035 for ; Tue, 11 Mar 2014 17:45:27 -0400 (EDT) Received: by mail-pb0-f41.google.com with SMTP id jt11so136119pbb.14 for ; Tue, 11 Mar 2014 14:45:26 -0700 (PDT) Received: from g4t3427.houston.hp.com (g4t3427.houston.hp.com. [15.201.208.55]) by mx.google.com with ESMTPS id sh5si252884pbc.80.2014.03.11.14.45.25 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 14:45:26 -0700 (PDT) Message-ID: <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Tue, 11 Mar 2014 14:45:23 -0700 In-Reply-To: <531F79F7.5090201@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > >> Bingo! With the above patch: > >> > > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > >> >[ 243.568048] Dumping ftrace buffer: > >> >[ 243.568740] (ftrace buffer empty) > >> >[ 243.569481] Modules linked in: > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > > the root cause it the same as Fengguang's report. > > The BUG still happens without DEBUG_PAGEALLOC. Any idea what trinity itself is doing? Could you add the following, I just want to make sure the bug isn't being caused by an overflow: diff --git a/mm/vmacache.c b/mm/vmacache.c index add3162..abb85cc 100644 --- a/mm/vmacache.c +++ b/mm/vmacache.c @@ -17,6 +17,8 @@ void vmacache_flush_all(struct mm_struct *mm) { struct task_struct *g, *p; + WARN_ON_ONCE(1); + rcu_read_lock(); for_each_process_thread(g, p) { /* Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qc0-f180.google.com (mail-qc0-f180.google.com [209.85.216.180]) by kanga.kvack.org (Postfix) with ESMTP id CDFB66B0035 for ; Tue, 11 Mar 2014 17:59:48 -0400 (EDT) Received: by mail-qc0-f180.google.com with SMTP id x3so10501059qcv.11 for ; Tue, 11 Mar 2014 14:59:48 -0700 (PDT) Received: from mail-qa0-x230.google.com (mail-qa0-x230.google.com [2607:f8b0:400d:c00::230]) by mx.google.com with ESMTPS id y69si11988545qgd.62.2014.03.11.14.59.48 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 11 Mar 2014 14:59:48 -0700 (PDT) Received: by mail-qa0-f48.google.com with SMTP id m5so9016763qaj.7 for ; Tue, 11 Mar 2014 14:59:48 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Date: Tue, 11 Mar 2014 14:59:47 -0700 Message-ID: Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Michel Lespinasse Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Sasha Levin , "linux-mm@kvack.org" , Andrew Morton , Rik van Riel , Vlastimil Babka , LKML On Tue, Mar 11, 2014 at 1:07 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: >> I've ended up deleting the log file by mistake, but this bug does seem to be important >> so I'd rather not wait before the same issue is triggered again. >> >> The call chain is: >> >> mlock (mm/mlock.c:745) >> __mm_populate (mm/mlock.c:700) >> __mlock_vma_pages_range (mm/mlock.c:229) >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: Not related to the bug, but please note that __mm_populate() is public and has other call sites outside of mlock.c - namely, it is called during mmap with MAP_POPULATE. -- Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ea0-f182.google.com (mail-ea0-f182.google.com [209.85.215.182]) by kanga.kvack.org (Postfix) with ESMTP id 6E71B6B0035 for ; Tue, 11 Mar 2014 18:07:33 -0400 (EDT) Received: by mail-ea0-f182.google.com with SMTP id b10so4629077eae.27 for ; Tue, 11 Mar 2014 15:07:32 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTP id p44si44039749eeu.5.2014.03.11.15.07.30 for ; Tue, 11 Mar 2014 15:07:31 -0700 (PDT) Date: Tue, 11 Mar 2014 16:57:01 -0400 From: Dave Jones Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Message-ID: <20140311205701.GA16658@redhat.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Sasha Levin , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, Mar 11, 2014 at 01:47:24PM -0700, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote: > > On 03/11/2014 04:30 PM, Andrew Morton wrote: > > > All I can think is that find_vma() went and returned a vma from a > > > different mm, which would be odd. How about I toss this in there? > > > > > > --- a/mm/vmacache.c~a > > > +++ a/mm/vmacache.c > > > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > > > for (i = 0; i < VMACACHE_SIZE; i++) { > > > struct vm_area_struct *vma = current->vmacache[i]; > > > > > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > > > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > > > + BUG_ON(vma->vm_mm != mm); > > > return vma; > > > + } > > > } > > > > [ 243.565794] kernel BUG at mm/vmacache.c:76! > > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > the root cause it the same as Fengguang's report. btw, I added a (ugly) script to trinity.git this afternoon: https://github.com/kernelslacker/trinity/commit/b6cfe63ea5b205a34bc6c5698e5b766dfcddad1d which should make reproducing VM related bugs happen a lot faster. While chasing one particular issue, that script has hit 2-3 others in my testing so far today (All already known). It would be really good if people other than me & Sasha started running stuff like this occasionally to try and get some of the more annoying things fixed up faster. Though it's only really useful with debug kernels, and I know from past experience that asking people to try running things with debug options enabled is like pulling teeth.. Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f46.google.com (mail-yh0-f46.google.com [209.85.213.46]) by kanga.kvack.org (Postfix) with ESMTP id B8F966B0035 for ; Tue, 11 Mar 2014 18:27:46 -0400 (EDT) Received: by mail-yh0-f46.google.com with SMTP id b6so6385063yha.33 for ; Tue, 11 Mar 2014 15:27:46 -0700 (PDT) Received: from userp1040.oracle.com (userp1040.oracle.com. [156.151.31.81]) by mx.google.com with ESMTPS id c1si38611897yhc.73.2014.03.11.15.20.49 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 11 Mar 2014 15:21:19 -0700 (PDT) Message-ID: <531F8C3A.1040502@oracle.com> Date: Tue, 11 Mar 2014 18:20:42 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >>>> > >>Bingo! With the above patch: >>>>> > >> > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >>>>> > >> >[ 243.568048] Dumping ftrace buffer: >>>>> > >> >[ 243.568740] (ftrace buffer empty) >>>>> > >> >[ 243.569481] Modules linked in: >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect >>> > >the root cause it the same as Fengguang's report. >> > >> >The BUG still happens without DEBUG_PAGEALLOC. > Any idea what trinity itself is doing? > > Could you add the following, I just want to make sure the bug isn't > being caused by an overflow: Not hitting that WARN. Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f49.google.com (mail-pb0-f49.google.com [209.85.160.49]) by kanga.kvack.org (Postfix) with ESMTP id D49FD6B0035 for ; Thu, 13 Mar 2014 15:00:08 -0400 (EDT) Received: by mail-pb0-f49.google.com with SMTP id jt11so1511818pbb.36 for ; Thu, 13 Mar 2014 12:00:08 -0700 (PDT) Received: from g4t3425.houston.hp.com (g4t3425.houston.hp.com. [15.201.208.53]) by mx.google.com with ESMTPS id a3si2050268pay.307.2014.03.13.12.00.07 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 13 Mar 2014 12:00:07 -0700 (PDT) Message-ID: <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso Date: Thu, 13 Mar 2014 12:00:02 -0700 In-Reply-To: <531F8C3A.1040502@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: > On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > >>>> > >>Bingo! With the above patch: > >>>>> > >> > > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > >>>>> > >> >[ 243.568048] Dumping ftrace buffer: > >>>>> > >> >[ 243.568740] (ftrace buffer empty) > >>>>> > >> >[ 243.569481] Modules linked in: > >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > >>> > >the root cause it the same as Fengguang's report. > >> > > >> >The BUG still happens without DEBUG_PAGEALLOC. > > Any idea what trinity itself is doing? > > > > Could you add the following, I just want to make sure the bug isn't > > being caused by an overflow: > > Not hitting that WARN. Sasha, could you please try the following patch: https://lkml.org/lkml/2014/3/13/312 thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by kanga.kvack.org (Postfix) with ESMTP id D205D6B0035 for ; Thu, 13 Mar 2014 16:58:58 -0400 (EDT) Received: by mail-pa0-f42.google.com with SMTP id fb1so1675963pad.1 for ; Thu, 13 Mar 2014 13:58:58 -0700 (PDT) Received: from mail-pd0-x22f.google.com (mail-pd0-x22f.google.com [2607:f8b0:400e:c02::22f]) by mx.google.com with ESMTPS id po10si2213944pab.384.2014.03.13.13.58.57 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 13 Mar 2014 13:58:57 -0700 (PDT) Received: by mail-pd0-f175.google.com with SMTP id x10so1584808pdj.34 for ; Thu, 13 Mar 2014 13:58:57 -0700 (PDT) Date: Thu, 13 Mar 2014 13:57:59 -0700 (PDT) From: Hugh Dickins Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range In-Reply-To: <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> Message-ID: References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso Cc: Oleg Nesterov , Sasha Levin , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On Thu, 13 Mar 2014, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: > > On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > > > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > > >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > > >>>> > >>Bingo! With the above patch: > > >>>>> > >> > > > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > > >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > > >>>>> > >> >[ 243.568048] Dumping ftrace buffer: > > >>>>> > >> >[ 243.568740] (ftrace buffer empty) > > >>>>> > >> >[ 243.569481] Modules linked in: > > >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > > >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > > >>> > >the root cause it the same as Fengguang's report. > > >> > > > >> >The BUG still happens without DEBUG_PAGEALLOC. > > > Any idea what trinity itself is doing? > > > > > > Could you add the following, I just want to make sure the bug isn't > > > being caused by an overflow: > > > > Not hitting that WARN. > > Sasha, could you please try the following patch: > https://lkml.org/lkml/2014/3/13/312 I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM on mmotm: Oleg's patch (buildable version below) fixes it for me. Hugh --- mmotm/mm/vmacache.c 2014-03-12 18:39:38.008011317 -0700 +++ linux/mm/vmacache.c 2014-03-13 12:21:11.592030813 -0700 @@ -31,15 +31,20 @@ void vmacache_flush_all(struct mm_struct rcu_read_unlock(); } +static bool vmacache_valid_mm(struct mm_struct *mm) +{ + return current->mm == mm && !(current->flags & PF_KTHREAD); +} + void vmacache_update(unsigned long addr, struct vm_area_struct *newvma) { - int idx = VMACACHE_HASH(addr); - current->vmacache[idx] = newvma; + if (vmacache_valid_mm(newvma->vm_mm)) + current->vmacache[VMACACHE_HASH(addr)] = newvma; } static bool vmacache_valid(struct mm_struct *mm) { - struct task_struct *curr = current; + struct task_struct *curr; /* * This task may be accessing a foreign mm via (for example) @@ -47,9 +52,10 @@ static bool vmacache_valid(struct mm_str * task's vmacache pertains to a different mm (ie, its own). There is * nothing we can do here. */ - if (mm != curr->mm) - return false; + if (!vmacache_valid_mm(mm)) + return false; + curr = current; if (mm->vmacache_seqnum != curr->vmacache_seqnum) { /* * First attempt will always be invalid, initialize -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f178.google.com (mail-ie0-f178.google.com [209.85.223.178]) by kanga.kvack.org (Postfix) with ESMTP id 7DD8C6B0082 for ; Fri, 14 Mar 2014 12:14:52 -0400 (EDT) Received: by mail-ie0-f178.google.com with SMTP id lx4so2764501iec.37 for ; Fri, 14 Mar 2014 09:14:52 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id k7si2003187igd.4.2014.03.14.09.14.51 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Fri, 14 Mar 2014 09:14:51 -0700 (PDT) Message-ID: <53232AF3.3040300@oracle.com> Date: Fri, 14 Mar 2014 12:14:43 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Hugh Dickins , Davidlohr Bueso Cc: Oleg Nesterov , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML On 03/13/2014 04:57 PM, Hugh Dickins wrote: > On Thu, 13 Mar 2014, Davidlohr Bueso wrote: >> On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: >>> On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: >>>> On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: >>>>>> On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >>>>>>>>>> Bingo! With the above patch: >>>>>>>>>>>> >>>>>>>>>>>> [ 243.565794] kernel BUG at mm/vmacache.c:76! >>>>>>>>>>>> [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >>>>>>>>>>>> [ 243.568048] Dumping ftrace buffer: >>>>>>>>>>>> [ 243.568740] (ftrace buffer empty) >>>>>>>>>>>> [ 243.569481] Modules linked in: >>>>>>>>>>>> [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 >>>>>>>> and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect >>>>>>>> the root cause it the same as Fengguang's report. >>>>>> >>>>>> The BUG still happens without DEBUG_PAGEALLOC. >>>> Any idea what trinity itself is doing? >>>> >>>> Could you add the following, I just want to make sure the bug isn't >>>> being caused by an overflow: >>> >>> Not hitting that WARN. >> >> Sasha, could you please try the following patch: >> https://lkml.org/lkml/2014/3/13/312 > > I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM > on mmotm: Oleg's patch (buildable version below) fixes it for me. Sorry for the delay, some patch in the last -next broke boot and I had to spend a while waiting for the bisect before I could test this patch. The patch fixes the vmacache issues I've been seeing. Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755301AbaCKTkI (ORCPT ); Tue, 11 Mar 2014 15:40:08 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:28251 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753782AbaCKTkH (ORCPT ); Tue, 11 Mar 2014 15:40:07 -0400 Message-ID: <531F6689.60307@oracle.com> Date: Tue, 11 Mar 2014 15:39:53 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: "linux-mm@kvack.org" CC: Andrew Morton , Davidlohr Bueso , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, I've ended up deleting the log file by mistake, but this bug does seem to be important so I'd rather not wait before the same issue is triggered again. The call chain is: mlock (mm/mlock.c:745) __mm_populate (mm/mlock.c:700) __mlock_vma_pages_range (mm/mlock.c:229) VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); It seems to be a rather simple trace triggered from userspace. The only recent patch in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". I've reverted it and trying to testing without it. Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755587AbaCKUHt (ORCPT ); Tue, 11 Mar 2014 16:07:49 -0400 Received: from g2t2353.austin.hp.com ([15.217.128.52]:23438 "EHLO g2t2353.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755257AbaCKUHo (ORCPT ); Tue, 11 Mar 2014 16:07:44 -0400 Message-ID: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Sasha Levin Cc: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Tue, 11 Mar 2014 13:07:33 -0700 In-Reply-To: <531F6689.60307@oracle.com> References: <531F6689.60307@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > Hi all, > > I've ended up deleting the log file by mistake, but this bug does seem to be important > so I'd rather not wait before the same issue is triggered again. > > The call chain is: > > mlock (mm/mlock.c:745) > __mm_populate (mm/mlock.c:700) > __mlock_vma_pages_range (mm/mlock.c:229) > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems wrong as we call it without the lock held: up_write(¤t->mm->mmap_sem); if (!error) error = __mm_populate(start, len, 0); return error; } > > It seems to be a rather simple trace triggered from userspace. The only recent patch > in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". > I've reverted it and trying to testing without it. Odd, this patch should definitely *not* cause this. In any case every operation removed from the critical region is local to the function: lock_limit = rlimit(RLIMIT_MEMLOCK); lock_limit >>= PAGE_SHIFT; locked = len >> PAGE_SHIFT; down_write(¤t->mm->mmap_sem); From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755621AbaCKUNG (ORCPT ); Tue, 11 Mar 2014 16:13:06 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:47607 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755531AbaCKUNE (ORCPT ); Tue, 11 Mar 2014 16:13:04 -0400 Message-ID: <531F6E43.40901@oracle.com> Date: Tue, 11 Mar 2014 16:12:51 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Davidlohr Bueso CC: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/11/2014 04:07 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: >> Hi all, >> >> I've ended up deleting the log file by mistake, but this bug does seem to be important >> so I'd rather not wait before the same issue is triggered again. >> >> The call chain is: >> >> mlock (mm/mlock.c:745) >> __mm_populate (mm/mlock.c:700) >> __mlock_vma_pages_range (mm/mlock.c:229) >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: > > up_write(¤t->mm->mmap_sem); > if (!error) > error = __mm_populate(start, len, 0); > return error; > } > >> >> It seems to be a rather simple trace triggered from userspace. The only recent patch >> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". >> I've reverted it and trying to testing without it. > > Odd, this patch should definitely *not* cause this. In any case every > operation removed from the critical region is local to the function: > > lock_limit = rlimit(RLIMIT_MEMLOCK); > lock_limit >>= PAGE_SHIFT; > locked = len >> PAGE_SHIFT; > > down_write(¤t->mm->mmap_sem); Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this code last - do you still remember what's going on here?" :). It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time I've seen it and it's probably been there for a while (that BUG_ON is there from 2009). Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755754AbaCKUVk (ORCPT ); Tue, 11 Mar 2014 16:21:40 -0400 Received: from g2t2352.austin.hp.com ([15.217.128.51]:28202 "EHLO g2t2352.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755362AbaCKUVi (ORCPT ); Tue, 11 Mar 2014 16:21:38 -0400 Message-ID: <1394569297.2786.36.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Sasha Levin Cc: "linux-mm@kvack.org" , Andrew Morton , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Tue, 11 Mar 2014 13:21:37 -0700 In-Reply-To: <531F6E43.40901@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <531F6E43.40901@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 16:12 -0400, Sasha Levin wrote: > On 03/11/2014 04:07 PM, Davidlohr Bueso wrote: > > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > >> Hi all, > >> > >> I've ended up deleting the log file by mistake, but this bug does seem to be important > >> so I'd rather not wait before the same issue is triggered again. > >> > >> The call chain is: > >> > >> mlock (mm/mlock.c:745) > >> __mm_populate (mm/mlock.c:700) > >> __mlock_vma_pages_range (mm/mlock.c:229) > >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > > wrong as we call it without the lock held: > > > > up_write(¤t->mm->mmap_sem); > > if (!error) > > error = __mm_populate(start, len, 0); > > return error; > > } > > > >> > >> It seems to be a rather simple trace triggered from userspace. The only recent patch > >> in the area (that I've noticed) was "mm/mlock: prepare params outside critical region". > >> I've reverted it and trying to testing without it. > > > > Odd, this patch should definitely *not* cause this. In any case every > > operation removed from the critical region is local to the function: > > > > lock_limit = rlimit(RLIMIT_MEMLOCK); > > lock_limit >>= PAGE_SHIFT; > > locked = len >> PAGE_SHIFT; > > > > down_write(¤t->mm->mmap_sem); > > Yeah, this patch doesn't look like it's causing it, I guess it was more of a "you touched this > code last - do you still remember what's going on here?" :). How frequently do you trigger this issue? Could you verify if it still occurs by reverting my patch? > It's semi-odd because it seems like an obvious issue to hit with trinity but it's the first time > I've seen it and it's probably been there for a while (that BUG_ON is there from 2009). Actually that VM_BUG_ON is correct, because we do in fact take the mmap_sem (for reading) inside __mm_populate(), which in return calls __mlock_vma_pages_range() with the lock held. Now, the lock is taken within the for loop, which does the hole "if (!locked) down_read()" dance, but it's just making sure that we take the lock upon the first iteration. So besides doing the locking outside of the loop, which is just a cleanup, I don't really see how it could be triggered. Thanks, Davidlohr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755668AbaCKUax (ORCPT ); Tue, 11 Mar 2014 16:30:53 -0400 Received: from mail.linuxfoundation.org ([140.211.169.12]:44864 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755422AbaCKUaw (ORCPT ); Tue, 11 Mar 2014 16:30:52 -0400 Date: Tue, 11 Mar 2014 13:30:51 -0700 From: Andrew Morton To: Davidlohr Bueso Cc: Sasha Levin , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Message-Id: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> X-Mailer: Sylpheed 3.2.0beta5 (GTK+ 2.24.10; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > > Hi all, > > > > I've ended up deleting the log file by mistake, but this bug does seem to be important > > so I'd rather not wait before the same issue is triggered again. > > > > The call chain is: > > > > mlock (mm/mlock.c:745) > > __mm_populate (mm/mlock.c:700) > > __mlock_vma_pages_range (mm/mlock.c:229) > > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: > > up_write(¤t->mm->mmap_sem); > if (!error) > error = __mm_populate(start, len, 0); > return error; > } __mm_populate() pretty clearly calls __mlock_vma_pages_range() under down_read(mm->mmap_sem). I worry about what happens if __get_user_pages decides to do if (ret & VM_FAULT_RETRY) { if (nonblocking) *nonblocking = 0; return i; } uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll forget to undo mmap_sem. That won't explain this result, but it's a potential problem. All I can think is that find_vma() went and returned a vma from a different mm, which would be odd. How about I toss this in there? --- a/mm/vmacache.c~a +++ a/mm/vmacache.c @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str for (i = 0; i < VMACACHE_SIZE; i++) { struct vm_area_struct *vma = current->vmacache[i]; - if (vma && vma->vm_start <= addr && vma->vm_end > addr) + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { + BUG_ON(vma->vm_mm != mm); return vma; + } } return NULL; _ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755608AbaCKUgK (ORCPT ); Tue, 11 Mar 2014 16:36:10 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:23792 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755336AbaCKUgI (ORCPT ); Tue, 11 Mar 2014 16:36:08 -0400 Message-ID: <531F73AB.5060402@oracle.com> Date: Tue, 11 Mar 2014 16:35:55 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Andrew Morton , Davidlohr Bueso CC: "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/11/2014 04:30 PM, Andrew Morton wrote: > I worry about what happens if __get_user_pages decides to do > > if (ret & VM_FAULT_RETRY) { > if (nonblocking) > *nonblocking = 0; > return i; > } > > uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll > forget to undo mmap_sem. That won't explain this result, but it's a > potential problem. That's actually seems right because if 'ret & VM_FAULT_RETRY' is true it means that lock_page_or_retry() was supposed to release mmap_sem for us. Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755718AbaCKUmr (ORCPT ); Tue, 11 Mar 2014 16:42:47 -0400 Received: from g5t1625.atlanta.hp.com ([15.192.137.8]:33082 "EHLO g5t1625.atlanta.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755213AbaCKUmq (ORCPT ); Tue, 11 Mar 2014 16:42:46 -0400 Message-ID: <1394570564.2786.40.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Andrew Morton Cc: Sasha Levin , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Tue, 11 Mar 2014 13:42:44 -0700 In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 13:30 -0700, Andrew Morton wrote: > On Tue, 11 Mar 2014 13:07:33 -0700 Davidlohr Bueso wrote: > > > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: > > > Hi all, > > > > > > I've ended up deleting the log file by mistake, but this bug does seem to be important > > > so I'd rather not wait before the same issue is triggered again. > > > > > > The call chain is: > > > > > > mlock (mm/mlock.c:745) > > > __mm_populate (mm/mlock.c:700) > > > __mlock_vma_pages_range (mm/mlock.c:229) > > > VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > > wrong as we call it without the lock held: > > > > up_write(¤t->mm->mmap_sem); > > if (!error) > > error = __mm_populate(start, len, 0); > > return error; > > } > > __mm_populate() pretty clearly calls __mlock_vma_pages_range() under > down_read(mm->mmap_sem). > > I worry about what happens if __get_user_pages decides to do > > if (ret & VM_FAULT_RETRY) { > if (nonblocking) > *nonblocking = 0; > return i; > } > > uh-oh, that just cleared __mm_populate()'s `locked' variable and we'll > forget to undo mmap_sem. That won't explain this result, but it's a > potential problem. > > > All I can think is that find_vma() went and returned a vma from a > different mm, which would be odd. How about I toss this in there? ... and we know that there is a bug (https://lkml.org/lkml/2014/3/9/201) with stale caches going on. We seem to be missing an invalidation and/or flush somewhere. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755767AbaCKUp0 (ORCPT ); Tue, 11 Mar 2014 16:45:26 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:28322 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754963AbaCKUpZ (ORCPT ); Tue, 11 Mar 2014 16:45:25 -0400 Message-ID: <531F75D1.3060909@oracle.com> Date: Tue, 11 Mar 2014 16:45:05 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Andrew Morton , Davidlohr Bueso CC: "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> In-Reply-To: <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet22.oracle.com [156.151.31.94] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/11/2014 04:30 PM, Andrew Morton wrote: > All I can think is that find_vma() went and returned a vma from a > different mm, which would be odd. How about I toss this in there? > > --- a/mm/vmacache.c~a > +++ a/mm/vmacache.c > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > for (i = 0; i < VMACACHE_SIZE; i++) { > struct vm_area_struct *vma = current->vmacache[i]; > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > + BUG_ON(vma->vm_mm != mm); > return vma; > + } > } > > return NULL; Bingo! With the above patch: [ 243.565794] kernel BUG at mm/vmacache.c:76! [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 243.568048] Dumping ftrace buffer: [ 243.568740] (ftrace buffer empty) [ 243.569481] Modules linked in: [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 [ 243.571505] task: ffff8800b8698000 ti: ffff8800b8694000 task.ti: ffff8800b8694000 [ 243.571505] RIP: 0010:[] [] vmacache_find+0x86/0xb0 [ 243.571505] RSP: 0000:ffff8800b8695da8 EFLAGS: 00010287 [ 243.571505] RAX: ffff88042890e400 RBX: 0000000002d6cb18 RCX: 0000000000000002 [ 243.571505] RDX: 0000000000000002 RSI: 0000000002d6cb18 RDI: ffff8800b86a8000 [ 243.571505] RBP: ffff8800b8695da8 R08: ffff8800b8698000 R09: 0000000000000000 [ 243.571505] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8800b86a8000 [ 243.571505] R13: 00000000000014f1 R14: ffff8800b86a80a8 R15: ffff8800b86a8000 [ 243.582665] FS: 00007f74df081700(0000) GS:ffff880b2b800000(0000) knlGS:0000000000000000 [ 243.582665] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 243.582665] CR2: 0000000002d6cb18 CR3: 00000000b8682000 CR4: 00000000000006a0 [ 243.582665] Stack: [ 243.582665] ffff8800b8695dd8 ffffffff812a7620 ffffffff844abd82 0000000002d6cb18 [ 243.582665] 0000000002d6cb18 00000000000000a9 ffff8800b8695ef8 ffffffff844abdd6 [ 243.582665] ffff8800b8698000 ffff8800b8698000 0000000000000002 ffff8800b8698000 [ 243.582665] Call Trace: [ 243.582665] [] find_vma+0x20/0x90 [ 243.582665] [] ? __do_page_fault+0x302/0x5d0 [ 243.582665] [] __do_page_fault+0x356/0x5d0 [ 243.582665] [] ? vtime_account_user+0x96/0xb0 [ 243.582665] [] ? preempt_count_sub+0xe2/0x120 [ 243.582665] [] ? context_tracking_user_exit+0x187/0x1d0 [ 243.582665] [] do_page_fault+0x45/0x70 [ 243.582665] [] do_async_page_fault+0x36/0x100 [ 243.582665] [] async_page_fault+0x28/0x30 [ 243.582665] Code: 00 eb 44 66 90 31 d2 49 89 c0 48 63 ca 49 8b 84 c8 b8 07 00 00 48 85 c0 74 23 48 39 30 77 1e 48 3b 70 08 73 18 48 3b 78 40 74 1c <0f> 0b 0f 1f 84 00 00 00 00 00 eb fe 66 0f 1f 44 00 00 ff c2 83 [ 243.582665] RIP [] vmacache_find+0x86/0xb0 [ 243.582665] RSP Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755569AbaCKUr1 (ORCPT ); Tue, 11 Mar 2014 16:47:27 -0400 Received: from g5t1626.atlanta.hp.com ([15.192.137.9]:40671 "EHLO g5t1626.atlanta.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755336AbaCKUrZ (ORCPT ); Tue, 11 Mar 2014 16:47:25 -0400 Message-ID: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Tue, 11 Mar 2014 13:47:24 -0700 In-Reply-To: <531F75D1.3060909@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote: > On 03/11/2014 04:30 PM, Andrew Morton wrote: > > All I can think is that find_vma() went and returned a vma from a > > different mm, which would be odd. How about I toss this in there? > > > > --- a/mm/vmacache.c~a > > +++ a/mm/vmacache.c > > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > > for (i = 0; i < VMACACHE_SIZE; i++) { > > struct vm_area_struct *vma = current->vmacache[i]; > > > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > > + BUG_ON(vma->vm_mm != mm); > > return vma; > > + } > > } > > > > return NULL; > > Bingo! With the above patch: > > [ 243.565794] kernel BUG at mm/vmacache.c:76! > [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > [ 243.568048] Dumping ftrace buffer: > [ 243.568740] (ftrace buffer empty) > [ 243.569481] Modules linked in: > [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect the root cause it the same as Fengguang's report. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755444AbaCKVDB (ORCPT ); Tue, 11 Mar 2014 17:03:01 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:37515 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754358AbaCKVC7 (ORCPT ); Tue, 11 Mar 2014 17:02:59 -0400 Message-ID: <531F79F7.5090201@oracle.com> Date: Tue, 11 Mar 2014 17:02:47 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Davidlohr Bueso CC: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >> Bingo! With the above patch: >> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >> >[ 243.568048] Dumping ftrace buffer: >> >[ 243.568740] (ftrace buffer empty) >> >[ 243.569481] Modules linked in: >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > the root cause it the same as Fengguang's report. The BUG still happens without DEBUG_PAGEALLOC. Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755331AbaCKVp1 (ORCPT ); Tue, 11 Mar 2014 17:45:27 -0400 Received: from g4t3427.houston.hp.com ([15.201.208.55]:37868 "EHLO g4t3427.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751412AbaCKVpZ (ORCPT ); Tue, 11 Mar 2014 17:45:25 -0400 Message-ID: <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Tue, 11 Mar 2014 14:45:23 -0700 In-Reply-To: <531F79F7.5090201@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > >> Bingo! With the above patch: > >> > > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > >> >[ 243.568048] Dumping ftrace buffer: > >> >[ 243.568740] (ftrace buffer empty) > >> >[ 243.569481] Modules linked in: > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > > the root cause it the same as Fengguang's report. > > The BUG still happens without DEBUG_PAGEALLOC. Any idea what trinity itself is doing? Could you add the following, I just want to make sure the bug isn't being caused by an overflow: diff --git a/mm/vmacache.c b/mm/vmacache.c index add3162..abb85cc 100644 --- a/mm/vmacache.c +++ b/mm/vmacache.c @@ -17,6 +17,8 @@ void vmacache_flush_all(struct mm_struct *mm) { struct task_struct *g, *p; + WARN_ON_ONCE(1); + rcu_read_lock(); for_each_process_thread(g, p) { /* Thanks. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755880AbaCKV7x (ORCPT ); Tue, 11 Mar 2014 17:59:53 -0400 Received: from mail-qc0-f178.google.com ([209.85.216.178]:53027 "EHLO mail-qc0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753560AbaCKV7s (ORCPT ); Tue, 11 Mar 2014 17:59:48 -0400 MIME-Version: 1.0 In-Reply-To: <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> Date: Tue, 11 Mar 2014 14:59:47 -0700 Message-ID: Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Michel Lespinasse To: Davidlohr Bueso Cc: Sasha Levin , "linux-mm@kvack.org" , Andrew Morton , Rik van Riel , Vlastimil Babka , LKML Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 11, 2014 at 1:07 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 15:39 -0400, Sasha Levin wrote: >> I've ended up deleting the log file by mistake, but this bug does seem to be important >> so I'd rather not wait before the same issue is triggered again. >> >> The call chain is: >> >> mlock (mm/mlock.c:745) >> __mm_populate (mm/mlock.c:700) >> __mlock_vma_pages_range (mm/mlock.c:229) >> VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem)); > > So __mm_populate() is only called by mlock(2) and this VM_BUG_ON seems > wrong as we call it without the lock held: Not related to the bug, but please note that __mm_populate() is public and has other call sites outside of mlock.c - namely, it is called during mmap with MAP_POPULATE. -- Michel "Walken" Lespinasse A program is never fully debugged until the last user dies. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755824AbaCKWHg (ORCPT ); Tue, 11 Mar 2014 18:07:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34780 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753718AbaCKWHe (ORCPT ); Tue, 11 Mar 2014 18:07:34 -0400 Date: Tue, 11 Mar 2014 16:57:01 -0400 From: Dave Jones To: Davidlohr Bueso Cc: Sasha Levin , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range Message-ID: <20140311205701.GA16658@redhat.com> Mail-Followup-To: Dave Jones , Davidlohr Bueso , Sasha Levin , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 11, 2014 at 01:47:24PM -0700, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 16:45 -0400, Sasha Levin wrote: > > On 03/11/2014 04:30 PM, Andrew Morton wrote: > > > All I can think is that find_vma() went and returned a vma from a > > > different mm, which would be odd. How about I toss this in there? > > > > > > --- a/mm/vmacache.c~a > > > +++ a/mm/vmacache.c > > > @@ -72,8 +72,10 @@ struct vm_area_struct *vmacache_find(str > > > for (i = 0; i < VMACACHE_SIZE; i++) { > > > struct vm_area_struct *vma = current->vmacache[i]; > > > > > > - if (vma && vma->vm_start <= addr && vma->vm_end > addr) > > > + if (vma && vma->vm_start <= addr && vma->vm_end > addr) { > > > + BUG_ON(vma->vm_mm != mm); > > > return vma; > > > + } > > > } > > > > [ 243.565794] kernel BUG at mm/vmacache.c:76! > > and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > the root cause it the same as Fengguang's report. btw, I added a (ugly) script to trinity.git this afternoon: https://github.com/kernelslacker/trinity/commit/b6cfe63ea5b205a34bc6c5698e5b766dfcddad1d which should make reproducing VM related bugs happen a lot faster. While chasing one particular issue, that script has hit 2-3 others in my testing so far today (All already known). It would be really good if people other than me & Sasha started running stuff like this occasionally to try and get some of the more annoying things fixed up faster. Though it's only really useful with debug kernels, and I know from past experience that asking people to try running things with debug options enabled is like pulling teeth.. Dave From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755932AbaCKWV2 (ORCPT ); Tue, 11 Mar 2014 18:21:28 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:46598 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755563AbaCKWVZ (ORCPT ); Tue, 11 Mar 2014 18:21:25 -0400 Message-ID: <531F8C3A.1040502@oracle.com> Date: Tue, 11 Mar 2014 18:20:42 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Davidlohr Bueso CC: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet22.oracle.com [156.151.31.94] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >>>> > >>Bingo! With the above patch: >>>>> > >> > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >>>>> > >> >[ 243.568048] Dumping ftrace buffer: >>>>> > >> >[ 243.568740] (ftrace buffer empty) >>>>> > >> >[ 243.569481] Modules linked in: >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect >>> > >the root cause it the same as Fengguang's report. >> > >> >The BUG still happens without DEBUG_PAGEALLOC. > Any idea what trinity itself is doing? > > Could you add the following, I just want to make sure the bug isn't > being caused by an overflow: Not hitting that WARN. Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754880AbaCMTAJ (ORCPT ); Thu, 13 Mar 2014 15:00:09 -0400 Received: from g4t3425.houston.hp.com ([15.201.208.53]:20613 "EHLO g4t3425.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753626AbaCMTAH (ORCPT ); Thu, 13 Mar 2014 15:00:07 -0400 Message-ID: <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range From: Davidlohr Bueso To: Sasha Levin Cc: Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Date: Thu, 13 Mar 2014 12:00:02 -0700 In-Reply-To: <531F8C3A.1040502@oracle.com> References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: > On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > >>>> > >>Bingo! With the above patch: > >>>>> > >> > > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > >>>>> > >> >[ 243.568048] Dumping ftrace buffer: > >>>>> > >> >[ 243.568740] (ftrace buffer empty) > >>>>> > >> >[ 243.569481] Modules linked in: > >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > >>> > >the root cause it the same as Fengguang's report. > >> > > >> >The BUG still happens without DEBUG_PAGEALLOC. > > Any idea what trinity itself is doing? > > > > Could you add the following, I just want to make sure the bug isn't > > being caused by an overflow: > > Not hitting that WARN. Sasha, could you please try the following patch: https://lkml.org/lkml/2014/3/13/312 thanks. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755306AbaCMU67 (ORCPT ); Thu, 13 Mar 2014 16:58:59 -0400 Received: from mail-pd0-f178.google.com ([209.85.192.178]:52301 "EHLO mail-pd0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754145AbaCMU65 (ORCPT ); Thu, 13 Mar 2014 16:58:57 -0400 Date: Thu, 13 Mar 2014 13:57:59 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Davidlohr Bueso cc: Oleg Nesterov , Sasha Levin , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range In-Reply-To: <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> Message-ID: References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 13 Mar 2014, Davidlohr Bueso wrote: > On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: > > On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: > > > On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: > > >> >On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: > > >>>> > >>Bingo! With the above patch: > > >>>>> > >> > > > >>>>> > >> >[ 243.565794] kernel BUG at mm/vmacache.c:76! > > >>>>> > >> >[ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > > >>>>> > >> >[ 243.568048] Dumping ftrace buffer: > > >>>>> > >> >[ 243.568740] (ftrace buffer empty) > > >>>>> > >> >[ 243.569481] Modules linked in: > > >>>>> > >> >[ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 > > >>> > >and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect > > >>> > >the root cause it the same as Fengguang's report. > > >> > > > >> >The BUG still happens without DEBUG_PAGEALLOC. > > > Any idea what trinity itself is doing? > > > > > > Could you add the following, I just want to make sure the bug isn't > > > being caused by an overflow: > > > > Not hitting that WARN. > > Sasha, could you please try the following patch: > https://lkml.org/lkml/2014/3/13/312 I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM on mmotm: Oleg's patch (buildable version below) fixes it for me. Hugh --- mmotm/mm/vmacache.c 2014-03-12 18:39:38.008011317 -0700 +++ linux/mm/vmacache.c 2014-03-13 12:21:11.592030813 -0700 @@ -31,15 +31,20 @@ void vmacache_flush_all(struct mm_struct rcu_read_unlock(); } +static bool vmacache_valid_mm(struct mm_struct *mm) +{ + return current->mm == mm && !(current->flags & PF_KTHREAD); +} + void vmacache_update(unsigned long addr, struct vm_area_struct *newvma) { - int idx = VMACACHE_HASH(addr); - current->vmacache[idx] = newvma; + if (vmacache_valid_mm(newvma->vm_mm)) + current->vmacache[VMACACHE_HASH(addr)] = newvma; } static bool vmacache_valid(struct mm_struct *mm) { - struct task_struct *curr = current; + struct task_struct *curr; /* * This task may be accessing a foreign mm via (for example) @@ -47,9 +52,10 @@ static bool vmacache_valid(struct mm_str * task's vmacache pertains to a different mm (ie, its own). There is * nothing we can do here. */ - if (mm != curr->mm) - return false; + if (!vmacache_valid_mm(mm)) + return false; + curr = current; if (mm->vmacache_seqnum != curr->vmacache_seqnum) { /* * First attempt will always be invalid, initialize From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755908AbaCNQO6 (ORCPT ); Fri, 14 Mar 2014 12:14:58 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:31562 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755076AbaCNQO4 (ORCPT ); Fri, 14 Mar 2014 12:14:56 -0400 Message-ID: <53232AF3.3040300@oracle.com> Date: Fri, 14 Mar 2014 12:14:43 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Hugh Dickins , Davidlohr Bueso CC: Oleg Nesterov , Andrew Morton , "linux-mm@kvack.org" , Michel Lespinasse , Rik van Riel , Vlastimil Babka , LKML Subject: Re: mm: mmap_sem lock assertion failure in __mlock_vma_pages_range References: <531F6689.60307@oracle.com> <1394568453.2786.28.camel@buesod1.americas.hpqcorp.net> <20140311133051.bf5ca716ef189746ebcff431@linux-foundation.org> <531F75D1.3060909@oracle.com> <1394570844.2786.42.camel@buesod1.americas.hpqcorp.net> <531F79F7.5090201@oracle.com> <1394574323.2786.45.camel@buesod1.americas.hpqcorp.net> <531F8C3A.1040502@oracle.com> <1394737202.2452.8.camel@buesod1.americas.hpqcorp.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/13/2014 04:57 PM, Hugh Dickins wrote: > On Thu, 13 Mar 2014, Davidlohr Bueso wrote: >> On Tue, 2014-03-11 at 18:20 -0400, Sasha Levin wrote: >>> On 03/11/2014 05:45 PM, Davidlohr Bueso wrote: >>>> On Tue, 2014-03-11 at 17:02 -0400, Sasha Levin wrote: >>>>>> On 03/11/2014 04:47 PM, Davidlohr Bueso wrote: >>>>>>>>>> Bingo! With the above patch: >>>>>>>>>>>> >>>>>>>>>>>> [ 243.565794] kernel BUG at mm/vmacache.c:76! >>>>>>>>>>>> [ 243.566720] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >>>>>>>>>>>> [ 243.568048] Dumping ftrace buffer: >>>>>>>>>>>> [ 243.568740] (ftrace buffer empty) >>>>>>>>>>>> [ 243.569481] Modules linked in: >>>>>>>>>>>> [ 243.570203] CPU: 10 PID: 10073 Comm: trinity-c332 Tainted: G W 3.14.0-rc5-next-20140307-sasha-00010-g1f812cb-dirty #143 >>>>>>>> and this is also part of the DEBUG_PAGEALLOC + trinity combo! I suspect >>>>>>>> the root cause it the same as Fengguang's report. >>>>>> >>>>>> The BUG still happens without DEBUG_PAGEALLOC. >>>> Any idea what trinity itself is doing? >>>> >>>> Could you add the following, I just want to make sure the bug isn't >>>> being caused by an overflow: >>> >>> Not hitting that WARN. >> >> Sasha, could you please try the following patch: >> https://lkml.org/lkml/2014/3/13/312 > > I was getting the "kernel BUG at mm/vmacache.c:76!" running KSM > on mmotm: Oleg's patch (buildable version below) fixes it for me. Sorry for the delay, some patch in the last -next broke boot and I had to spend a while waiting for the bisect before I could test this patch. The patch fixes the vmacache issues I've been seeing. Thanks, Sasha