2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Sonny Rao]

(Apologies if this is a re-post)

Hi, I was running 2.6.19 and running some benchmarks using
libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug: 

kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

6:mon> e
cpu 0x6: Vector: 700 (Program Check) at [c0000002d92c7a10]
    pc: c000000000030170: .huge_pte_offset+0xa8/0xd8
    lr: c000000000031024: .hash_huge_page+0x50/0x42c
    sp: c0000002d92c7c90
   msr: 8000000000021032
  current = 0xc0000002da2a19a0
  paca    = 0xc0000000005b3f80
    pid   = 24695, comm = 
kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!
6:mon>
6:mon> t
[link register   ] c000000000031024 .hash_huge_page+0x50/0x42c
[c0000002d92c7c90] c0000002d92c7d50 (unreliable)
[c0000002d92c7d50] c00000000002cf80 .hash_page+0x214/0x3bc
[c0000002d92c7e30] c000000000004834 .do_hash_page+0x34/0x40
--- Exception: 301 (Data Access) at 000000001009b0cc
SP (f33bc830) is in userspace
6:mon> r

the bug comes from this code:

static inline pte_t *hugepd_page(hugepd_t hpd)
{
        BUG_ON(!(hpd.pd & HUGEPD_OK));
        return (pte_t *)(hpd.pd & ~HUGEPD_OK);
}


I'm trying to reproduce on 2.6.20-rc3 now, but was wondering if anyone
has seen this before. 

Sonny

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Adam Litke]

On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> (Apologies if this is a re-post)
> 
> Hi, I was running 2.6.19 and running some benchmarks using
> libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:

Is this triggered by a libhugetlbfs test case?  If so, which one?

> 
> kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!
> 
> 6:mon> e
> cpu 0x6: Vector: 700 (Program Check) at [c0000002d92c7a10]
>     pc: c000000000030170: .huge_pte_offset+0xa8/0xd8
>     lr: c000000000031024: .hash_huge_page+0x50/0x42c
>     sp: c0000002d92c7c90
>    msr: 8000000000021032
>   current = 0xc0000002da2a19a0
>   paca    = 0xc0000000005b3f80
>     pid   = 24695, comm =
> kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!
> 6:mon>
> 6:mon> t
> [link register   ] c000000000031024 .hash_huge_page+0x50/0x42c
> [c0000002d92c7c90] c0000002d92c7d50 (unreliable)
> [c0000002d92c7d50] c00000000002cf80 .hash_page+0x214/0x3bc
> [c0000002d92c7e30] c000000000004834 .do_hash_page+0x34/0x40
> --- Exception: 301 (Data Access) at 000000001009b0cc
> SP (f33bc830) is in userspace
> 6:mon> r
> 
> the bug comes from this code:
> 
> static inline pte_t *hugepd_page(hugepd_t hpd)
> {
>         BUG_ON(!(hpd.pd & HUGEPD_OK));
>         return (pte_t *)(hpd.pd & ~HUGEPD_OK);
> }
> 
> 
> I'm trying to reproduce on 2.6.20-rc3 now, but was wondering if anyone
> has seen this before.
> 
> Sonny
> 
> 
-- 
Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Sonny Rao]

On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > (Apologies if this is a re-post)
> > 
> > Hi, I was running 2.6.19 and running some benchmarks using
> > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> 
> Is this triggered by a libhugetlbfs test case?  If so, which one?

Nope, this was an application/benchmark, but I'm running the testsuite
on the machine now to see if I can trigger it.

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Nishanth Aravamudan]

On 12.01.2007 [15:30:17 -0500], Sonny Rao wrote:
> On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > (Apologies if this is a re-post)
> > >
> > > Hi, I was running 2.6.19 and running some benchmarks using
> > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> >
> > Is this triggered by a libhugetlbfs test case?  If so, which one?
> 
> Nope, this was an application/benchmark, but I'm running the testsuite
> on the machine now to see if I can trigger it.

Any idea what the application/benchmark was trying to do? Does it happen
to be open-source? :)

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Sonny Rao]

On Fri, Jan 12, 2007 at 01:11:59PM -0800, Nishanth Aravamudan wrote:
> On 12.01.2007 [15:30:17 -0500], Sonny Rao wrote:
> > On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > > (Apologies if this is a re-post)
> > > >
> > > > Hi, I was running 2.6.19 and running some benchmarks using
> > > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> > >
> > > Is this triggered by a libhugetlbfs test case?  If so, which one?
> > 
> > Nope, this was an application/benchmark, but I'm running the testsuite
> > on the machine now to see if I can trigger it.
> 
> Any idea what the application/benchmark was trying to do? Does it happen
> to be open-source? :)

Unfortunately not, I haven't had any time to analyze what's going on.
It doesn't seem to happen all the time either, so it may take a while
before I know anything more.

Sonny

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Sonny Rao]

On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > (Apologies if this is a re-post)
> > 
> > Hi, I was running 2.6.19 and running some benchmarks using
> > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> 
> Is this triggered by a libhugetlbfs test case?  If so, which one?

Ok so the testsuite all passed except for "slbpacaflush" which said
"PASS (inconclusive)" ... not sure if that is expected or not. 

Re: 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Nishanth Aravamudan]

On 12.01.2007 [15:42:50 -0500], Sonny Rao wrote:
> On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > (Apologies if this is a re-post)
> > >
> > > Hi, I was running 2.6.19 and running some benchmarks using
> > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> >
> > Is this triggered by a libhugetlbfs test case?  If so, which one?
> 
> Ok so the testsuite all passed except for "slbpacaflush" which said
> "PASS (inconclusive)" ... not sure if that is expected or not.

Should be fine.

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

Re: [Libhugetlbfs-devel] 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[David Gibson]

On Fri, Jan 12, 2007 at 03:42:50PM -0500, Sonny Rao wrote:
> On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > (Apologies if this is a re-post)
> > > 
> > > Hi, I was running 2.6.19 and running some benchmarks using
> > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> > 
> > Is this triggered by a libhugetlbfs test case?  If so, which one?
> 
> Ok so the testsuite all passed except for "slbpacaflush" which said
> "PASS (inconclusive)" ... not sure if that is expected or not. 

I used "PASS (inconclusive)" to mean: you're probably ok, but the bug
in question is non-deterministically triggered, so maybe we just got
lucky.

This testcase attempts to trigger a bunch of times (50?), but the
conditions are sufficiently dicey that a false PASS is still a
realistic possibility (I've seen it happen, but not often).  Some
other tests (e.g. alloc-instantiate-race) are technically
non-deterministic too, but I've managed to device trigger conditions
which are reliable in practice, those tests report plain PASS.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Libhugetlbfs-devel] 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Sonny Rao]

On Sat, Jan 13, 2007 at 09:43:48AM +1100, David Gibson wrote:
> On Fri, Jan 12, 2007 at 03:42:50PM -0500, Sonny Rao wrote:
> > On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > > (Apologies if this is a re-post)
> > > > 
> > > > Hi, I was running 2.6.19 and running some benchmarks using
> > > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> > > 
> > > Is this triggered by a libhugetlbfs test case?  If so, which one?
> > 
> > Ok so the testsuite all passed except for "slbpacaflush" which said
> > "PASS (inconclusive)" ... not sure if that is expected or not. 
> 
> I used "PASS (inconclusive)" to mean: you're probably ok, but the bug
> in question is non-deterministically triggered, so maybe we just got
> lucky.
> 
> This testcase attempts to trigger a bunch of times (50?), but the
> conditions are sufficiently dicey that a false PASS is still a
> realistic possibility (I've seen it happen, but not often).  Some
> other tests (e.g. alloc-instantiate-race) are technically
> non-deterministic too, but I've managed to device trigger conditions
> which are reliable in practice, those tests report plain PASS.


Ok, I have figured out what is happening.. here we go

I have a 32bit process and I make sure ulimit -s is set to unlimited
beforehand.  When libhugetlbfs sets up the text and data sections it
temporarily maps a hugepage at 0xe0000000 and tears it down after
copying the contents in.  Then later the stack grows to the point that
it runs into that segment.

The problem is that we never clear out the bits in
mm->context.low_htlb_areas once they're set... so the arch-specific
code thinks it is handling a huge page while the generic code thinks
we're instantiating a regular page.

Specifically, follow_page() in mm/memory.c unconditionally calls
arch-specific follow_huge_page() to determine if it's a huge page.  We
look at the bits in context.low_htlb_areas and determine that it is a
huge page, even though the VM thinks it's a stack page resulting in
confusion and dead kernels.  The basic problem seems to be that we
never cleared out that bit when we unmapped the file, and I've even
hit this problem in other ways (with gdb debugging the process and
trying to touch the area in question, get a NULL ptep and die);

I have a testcase which will demonstrate the fail on 2.6.19 and
2.6.20-rc4 using 64k or 4k pages below.

You must set ulimit -s unlimited before running the case to cause the
fail.. I tried setting it using programatically using setrlimit(3) but
that didn't reproduce the fail for some reason...

Messy. I'll leave it to you guys to figure out what to do.

Sonny

uncesessarily complex testcase source below:

/* ulimit -s unlimited */
/* gcc -m32 -O2 -Wall -B /usr/local/share/libhugetlbfs -Wl,--hugetlbfs-link=BDT  */
static char buf[1024 * 1024 * 16 * 10];

/* outwit the optimizer, since we were foolish enough to turn on optimization */
/* Without all the "buf" gunk, GCC was smart enough to emit a branch to self */
/* and no stack frames */
int recurse_forever(int n) 
{
	char buf[256] = { 0xff, };
	int ret = n * recurse_forever(n+1); 
	return ret + buf[n % 256];
}

int main(int argc, char *argv[])
{
	int i = 0;
	for( i = 0; i< 1024 * 1024 * 16 * 10; i++) {
		buf[i] = 0xff;
	}
	return recurse_forever(1);

}

Re: [Libhugetlbfs-devel] 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[David Gibson]

On Tue, Jan 23, 2007 at 12:10:40AM -0500, Sonny Rao wrote:
> On Sat, Jan 13, 2007 at 09:43:48AM +1100, David Gibson wrote:
> > On Fri, Jan 12, 2007 at 03:42:50PM -0500, Sonny Rao wrote:
> > > On Fri, Jan 12, 2007 at 02:08:30PM -0600, Adam Litke wrote:
> > > > On Fri, 2007-01-12 at 14:57 -0500, Sonny Rao wrote:
> > > > > (Apologies if this is a re-post)
> > > > > 
> > > > > Hi, I was running 2.6.19 and running some benchmarks using
> > > > > libhugetlbfs (1.0.1) and I can fairly reliably trigger this bug:
> > > > 
> > > > Is this triggered by a libhugetlbfs test case?  If so, which one?
> > > 
> > > Ok so the testsuite all passed except for "slbpacaflush" which said
> > > "PASS (inconclusive)" ... not sure if that is expected or not. 
> > 
> > I used "PASS (inconclusive)" to mean: you're probably ok, but the bug
> > in question is non-deterministically triggered, so maybe we just got
> > lucky.
> > 
> > This testcase attempts to trigger a bunch of times (50?), but the
> > conditions are sufficiently dicey that a false PASS is still a
> > realistic possibility (I've seen it happen, but not often).  Some
> > other tests (e.g. alloc-instantiate-race) are technically
> > non-deterministic too, but I've managed to device trigger conditions
> > which are reliable in practice, those tests report plain PASS.
> 
> 
> Ok, I have figured out what is happening.. here we go
> 
> I have a 32bit process and I make sure ulimit -s is set to unlimited
> beforehand.  When libhugetlbfs sets up the text and data sections it
> temporarily maps a hugepage at 0xe0000000 and tears it down after
> copying the contents in.  Then later the stack grows to the point that
> it runs into that segment.

Ah.  I have this vague memory of thinking at some point "I really
should check the stack grow-down logic for hugepage interactions".
Guess I should have done that.

> The problem is that we never clear out the bits in
> mm->context.low_htlb_areas once they're set... so the arch-specific
> code thinks it is handling a huge page while the generic code thinks
> we're instantiating a regular page.

Well... there's three parts to this problem.  First, and most
critically, the stack-growing logic doesn't do a check for a hugepage
region and fail if it runs into one (SEGV or SIGBUS, I guess).  We
have to look after this one in case we get a similar case where the
hugepage hasn't been unmapped.

Second, there's the fact that we never demote hugepage segments back
to normal pages.  That was a deliberate decision to keep things
simple, incidentally, not simply an oversight.  I guess it would help
in this case and shouldn't be that hard.  It would mean a find_vma()
on each unmap to see if the region is now clear, but that's probably
not too bad.  Plus a bunch of on_each_cpu()ed slbies as when we open a
new hugepage segment.  Oh.. and making sure we get rid of any empty
hugepage directories, which might be a bit fiddly.

Third, there's the question of whether our heuristics for which
hugepage segment to open need tweaking (the current ones try to open
the highest unoccupied segment, so will always start with 0xe0000000
segment unless the stack is at a strange address).  I suspect so:
anywhere else is further from the stack, but closer to other things
and a 256M stack is rather rarer than a large heap or set of mmap()s.

> Specifically, follow_page() in mm/memory.c unconditionally calls
> arch-specific follow_huge_page() to determine if it's a huge page.  We
> look at the bits in context.low_htlb_areas and determine that it is a
> huge page, even though the VM thinks it's a stack page resulting in
> confusion and dead kernels.  The basic problem seems to be that we
> never cleared out that bit when we unmapped the file, and I've even
> hit this problem in other ways (with gdb debugging the process and
> trying to touch the area in question, get a NULL ptep and die);
> 
> I have a testcase which will demonstrate the fail on 2.6.19 and
> 2.6.20-rc4 using 64k or 4k pages below.
> 
> You must set ulimit -s unlimited before running the case to cause the
> fail.. I tried setting it using programatically using setrlimit(3) but
> that didn't reproduce the fail for some reason...
> 
> Messy. I'll leave it to you guys to figure out what to do.
> 
> Sonny
> 
> uncesessarily complex testcase source below:
> 
> /* ulimit -s unlimited */
> /* gcc -m32 -O2 -Wall -B /usr/local/share/libhugetlbfs -Wl,--hugetlbfs-link=BDT  */
> static char buf[1024 * 1024 * 16 * 10];
> 
> /* outwit the optimizer, since we were foolish enough to turn on optimization */
> /* Without all the "buf" gunk, GCC was smart enough to emit a branch to self */
> /* and no stack frames */
> int recurse_forever(int n) 
> {
> 	char buf[256] = { 0xff, };
> 	int ret = n * recurse_forever(n+1); 
> 	return ret + buf[n % 256];
> }
> 
> int main(int argc, char *argv[])
> {
> 	int i = 0;
> 	for( i = 0; i< 1024 * 1024 * 16 * 10; i++) {
> 		buf[i] = 0xff;
> 	}
> 	return recurse_forever(1);
> 
> }
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Libhugetlbfs-devel] 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[Adam Litke]

On Tue, 2007-01-23 at 17:18 +1100, David Gibson wrote:
> Second, there's the fact that we never demote hugepage segments back
> to normal pages.  That was a deliberate decision to keep things
> simple, incidentally, not simply an oversight.  I guess it would help
> in this case and shouldn't be that hard.  It would mean a find_vma()
> on each unmap to see if the region is now clear, but that's probably
> not too bad.  Plus a bunch of on_each_cpu()ed slbies as when we open a
> new hugepage segment.  Oh.. and making sure we get rid of any empty
> hugepage directories, which might be a bit fiddly.

Could we also try lazy conversion of huge segments to normal ones?  When
is_hugepage_only_range() detects overlapping hugepage ranges, it could
attempt to "close" those ranges for huge pages first.  Then the heavy
lifting only needs to happen when a small page mapping needs the space.

-- 
Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center

Re: [Libhugetlbfs-devel] 2.6.19: kernel BUG in hugepd_page at arch/powerpc/mm/hugetlbpage.c:58!

[David Gibson]

On Tue, Jan 23, 2007 at 10:20:03AM -0600, Adam Litke wrote:
> On Tue, 2007-01-23 at 17:18 +1100, David Gibson wrote:
> > Second, there's the fact that we never demote hugepage segments back
> > to normal pages.  That was a deliberate decision to keep things
> > simple, incidentally, not simply an oversight.  I guess it would help
> > in this case and shouldn't be that hard.  It would mean a find_vma()
> > on each unmap to see if the region is now clear, but that's probably
> > not too bad.  Plus a bunch of on_each_cpu()ed slbies as when we open a
> > new hugepage segment.  Oh.. and making sure we get rid of any empty
> > hugepage directories, which might be a bit fiddly.
> 
> Could we also try lazy conversion of huge segments to normal ones?  When
> is_hugepage_only_range() detects overlapping hugepage ranges, it could
> attempt to "close" those ranges for huge pages first.  Then the heavy
> lifting only needs to happen when a small page mapping needs the space.

We could, but I think it's both easier and less operations to do the
check on unmap.  The lifting isn't that heavy.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson