From: Oleg Nesterov <oleg@redhat.com>
To: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Sasha Levin <sasha.levin@oracle.com>,
Hugh Dickins <hughd@google.com>,
"Kirill A. Shutemov" <kirill@shutemov.name>,
Andrey Konovalov <andreyknvl@google.com>,
Rik van Riel <riel@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Dmitry Vyukov <dvyukov@google.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>,
Vlastimil Babka <vbabka@suse.cz>
Subject: Re: Multiple potential races on vma->vm_flags
Date: Fri, 25 Sep 2015 14:41:51 +0200 [thread overview]
Message-ID: <20150925124151.GA5384@redhat.com> (raw)
In-Reply-To: <CAPAsAGx660uSk=WbpWmZR9FpSFXmp3G9yXxRXu65gozu3qT63g@mail.gmail.com>
On 09/24, Andrey Ryabinin wrote:
>
> 2015-09-24 20:26 GMT+03:00 Oleg Nesterov <oleg@redhat.com>:
> > On 09/24, Sasha Levin wrote:
> >>
> >> void unmap_vmas(struct mmu_gather *tlb,
> >> struct vm_area_struct *vma, unsigned long start_addr,
> >> unsigned long end_addr)
> >> {
> >> struct mm_struct *mm = vma->vm_mm;
> >>
> >> mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
> >> for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
> >> unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
> >> mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
> >> }
> >
> > And I do not see any dereference at this line,
> >
>
> I noticed, that addr2line sometimes doesn't work reliably on
> compiler-instrumented code.
> I've seen couple times that it points to the next line of code.
Yes, I know that we can't trust it. That is why I think (at least in
this particular case) function+offset would be more helpful. And we
need more asm probably.
> >> >> 0: 08 80 3c 02 00 0f or %al,0xf00023c(%rax)
> >> >> 6: 85 22 test %esp,(%rdx)
> >> >> 8: 01 00 add %eax,(%rax)
> >> >> a: 00 48 8b add %cl,-0x75(%rax)
> >> >> d: 43 rex.XB
> >> >> e: 40 rex
> >> >> f: 48 8d b8 c8 04 00 00 lea 0x4c8(%rax),%rdi
> >> >> 16: 48 89 45 d0 mov %rax,-0x30(%rbp)
> >> >> 1a: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> >> >> 21: fc ff df
> >> >> 24: 48 89 fa mov %rdi,%rdx
> >> >> 27: 48 c1 ea 03 shr $0x3,%rdx
> >> >> 2b:* 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction
> >> >> 2f: 0f 85 ee 00 00 00 jne 0x123
> >> >> 35: 48 8b 45 d0 mov -0x30(%rbp),%rax
> >> >> 39: 48 83 b8 c8 04 00 00 cmpq $0x0,0x4c8(%rax)
> >> >> 40: 00
> >> >
> >> > And I do not see anything similar in "objdump -d". So could you at least
> >> > show mm/memory.c:1337 in your tree?
> >> >
> >> > Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
> >> > like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
> >> > (I can't check, my gcc is very old). Or what?
> >>
> >> This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
> >> kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.
> >
> > and thus this asm can't help, right?
> >
>
> I think it can.
>
> > So how can we figure out where exactly the kernel hits NULL ? And what
> > exactly it tries to dereference?
>
> So we tried to dereference 0x4c8. That 0x4c8 is probably offset in some struct.
> The only big struct here is mm_struct.
> So I think that we tried to derefernce null mm, and this asm:
> > cmpq $0x0,0x4c8(%rax)
>
> is likely from inlined mm_has_notifiers():
> static inline int mm_has_notifiers(struct mm_struct *mm)
> {
> return unlikely(mm->mmu_notifier_mm);
> }
Looks reasonable... Thanks.
I was going to say that this is impossible because the caller should have
crashed if ->mm == NULL. But unmap_vmas() uses mm = vma->vm_mm, so it looks
like this vma or mm->mmap was corrupted...
Oleg.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Oleg Nesterov <oleg@redhat.com>
To: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Sasha Levin <sasha.levin@oracle.com>,
Hugh Dickins <hughd@google.com>,
"Kirill A. Shutemov" <kirill@shutemov.name>,
Andrey Konovalov <andreyknvl@google.com>,
Rik van Riel <riel@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Dmitry Vyukov <dvyukov@google.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>,
Vlastimil Babka <vbabka@suse.cz>
Subject: Re: Multiple potential races on vma->vm_flags
Date: Fri, 25 Sep 2015 14:41:51 +0200 [thread overview]
Message-ID: <20150925124151.GA5384@redhat.com> (raw)
In-Reply-To: <CAPAsAGx660uSk=WbpWmZR9FpSFXmp3G9yXxRXu65gozu3qT63g@mail.gmail.com>
On 09/24, Andrey Ryabinin wrote:
>
> 2015-09-24 20:26 GMT+03:00 Oleg Nesterov <oleg@redhat.com>:
> > On 09/24, Sasha Levin wrote:
> >>
> >> void unmap_vmas(struct mmu_gather *tlb,
> >> struct vm_area_struct *vma, unsigned long start_addr,
> >> unsigned long end_addr)
> >> {
> >> struct mm_struct *mm = vma->vm_mm;
> >>
> >> mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
> >> for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
> >> unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
> >> mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
> >> }
> >
> > And I do not see any dereference at this line,
> >
>
> I noticed, that addr2line sometimes doesn't work reliably on
> compiler-instrumented code.
> I've seen couple times that it points to the next line of code.
Yes, I know that we can't trust it. That is why I think (at least in
this particular case) function+offset would be more helpful. And we
need more asm probably.
> >> >> 0: 08 80 3c 02 00 0f or %al,0xf00023c(%rax)
> >> >> 6: 85 22 test %esp,(%rdx)
> >> >> 8: 01 00 add %eax,(%rax)
> >> >> a: 00 48 8b add %cl,-0x75(%rax)
> >> >> d: 43 rex.XB
> >> >> e: 40 rex
> >> >> f: 48 8d b8 c8 04 00 00 lea 0x4c8(%rax),%rdi
> >> >> 16: 48 89 45 d0 mov %rax,-0x30(%rbp)
> >> >> 1a: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> >> >> 21: fc ff df
> >> >> 24: 48 89 fa mov %rdi,%rdx
> >> >> 27: 48 c1 ea 03 shr $0x3,%rdx
> >> >> 2b:* 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction
> >> >> 2f: 0f 85 ee 00 00 00 jne 0x123
> >> >> 35: 48 8b 45 d0 mov -0x30(%rbp),%rax
> >> >> 39: 48 83 b8 c8 04 00 00 cmpq $0x0,0x4c8(%rax)
> >> >> 40: 00
> >> >
> >> > And I do not see anything similar in "objdump -d". So could you at least
> >> > show mm/memory.c:1337 in your tree?
> >> >
> >> > Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
> >> > like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
> >> > (I can't check, my gcc is very old). Or what?
> >>
> >> This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
> >> kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.
> >
> > and thus this asm can't help, right?
> >
>
> I think it can.
>
> > So how can we figure out where exactly the kernel hits NULL ? And what
> > exactly it tries to dereference?
>
> So we tried to dereference 0x4c8. That 0x4c8 is probably offset in some struct.
> The only big struct here is mm_struct.
> So I think that we tried to derefernce null mm, and this asm:
> > cmpq $0x0,0x4c8(%rax)
>
> is likely from inlined mm_has_notifiers():
> static inline int mm_has_notifiers(struct mm_struct *mm)
> {
> return unlikely(mm->mmu_notifier_mm);
> }
Looks reasonable... Thanks.
I was going to say that this is impossible because the caller should have
crashed if ->mm == NULL. But unmap_vmas() uses mm = vma->vm_mm, so it looks
like this vma or mm->mmap was corrupted...
Oleg.
next prev parent reply other threads:[~2015-09-25 12:44 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAAeHK+z8o96YeRF-fQXmoApOKXa0b9pWsQHDeP=5GC_hMTuoDg@mail.gmail.com>
[not found] ` <55EC9221.4040603@oracle.com>
2015-09-07 11:40 ` Multiple potential races on vma->vm_flags Kirill A. Shutemov
2015-09-07 11:40 ` Kirill A. Shutemov
2015-09-09 15:27 ` Vlastimil Babka
2015-09-09 15:27 ` Vlastimil Babka
2015-09-09 16:01 ` Kirill A. Shutemov
2015-09-09 16:01 ` Kirill A. Shutemov
2015-09-10 0:58 ` Sasha Levin
2015-09-10 0:58 ` Sasha Levin
2015-09-10 8:36 ` Kirill A. Shutemov
2015-09-10 8:36 ` Kirill A. Shutemov
2015-09-10 13:27 ` Andrey Konovalov
2015-09-10 13:27 ` Andrey Konovalov
2015-09-11 10:39 ` Kirill A. Shutemov
2015-09-11 10:39 ` Kirill A. Shutemov
2015-09-11 15:29 ` Vlastimil Babka
2015-09-11 15:29 ` Vlastimil Babka
2015-09-11 16:08 ` Vlastimil Babka
2015-09-11 16:08 ` Vlastimil Babka
2015-09-12 1:27 ` Hugh Dickins
2015-09-12 1:27 ` Hugh Dickins
2015-09-14 10:16 ` Kirill A. Shutemov
2015-09-14 10:16 ` Kirill A. Shutemov
2015-09-15 17:36 ` Sasha Levin
2015-09-15 17:36 ` Sasha Levin
2015-09-15 19:01 ` Kirill A. Shutemov
2015-09-15 19:01 ` Kirill A. Shutemov
2015-09-22 16:47 ` Andrey Konovalov
2015-09-22 16:47 ` Andrey Konovalov
2015-09-22 18:54 ` Hugh Dickins
2015-09-22 18:54 ` Hugh Dickins
2015-09-22 19:45 ` Andrey Konovalov
2015-09-22 19:45 ` Andrey Konovalov
2015-09-23 1:39 ` Hugh Dickins
2015-09-23 1:39 ` Hugh Dickins
2015-09-23 11:46 ` Kirill A. Shutemov
2015-09-23 11:46 ` Kirill A. Shutemov
2015-09-23 22:58 ` Davidlohr Bueso
2015-09-23 22:58 ` Davidlohr Bueso
2015-09-23 13:08 ` Andrey Konovalov
2015-09-23 13:08 ` Andrey Konovalov
2015-09-24 0:42 ` Sasha Levin
2015-09-24 0:42 ` Sasha Levin
2015-09-25 19:33 ` Kirill A. Shutemov
2015-09-25 19:33 ` Kirill A. Shutemov
2015-10-13 22:38 ` Hugh Dickins
2015-10-13 22:38 ` Hugh Dickins
2015-10-13 22:33 ` Hugh Dickins
2015-10-13 22:33 ` Hugh Dickins
2015-10-15 16:58 ` Andrey Konovalov
2015-10-15 16:58 ` Andrey Konovalov
2015-09-23 21:30 ` Sasha Levin
2015-09-23 21:30 ` Sasha Levin
2015-09-25 14:26 ` Oleg Nesterov
2015-09-25 14:26 ` Oleg Nesterov
2015-09-24 13:11 ` Oleg Nesterov
2015-09-24 13:11 ` Oleg Nesterov
2015-09-24 16:27 ` Sasha Levin
2015-09-24 16:27 ` Sasha Levin
2015-09-24 17:26 ` Oleg Nesterov
2015-09-24 17:26 ` Oleg Nesterov
2015-09-24 18:52 ` Andrey Ryabinin
2015-09-24 18:52 ` Andrey Ryabinin
2015-09-24 19:01 ` Sasha Levin
2015-09-24 19:01 ` Sasha Levin
2015-09-25 12:41 ` Oleg Nesterov [this message]
2015-09-25 12:41 ` Oleg Nesterov
2015-09-23 15:34 ` Oleg Nesterov
2015-09-23 15:34 ` Oleg Nesterov
2015-09-23 15:38 ` Oleg Nesterov
2015-09-23 15:38 ` Oleg Nesterov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150925124151.GA5384@redhat.com \
--to=oleg@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=andreyknvl@google.com \
--cc=dvyukov@google.com \
--cc=hughd@google.com \
--cc=kirill@shutemov.name \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=riel@redhat.com \
--cc=ryabinin.a.a@gmail.com \
--cc=sasha.levin@oracle.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.