Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Sachin Sant]

While executing hugetlb tests against today's Next tree on
a Power 6 box came across following OOPS.

 ------------[ cut here ]------------
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=1024 NUMA pSeries
Modules linked in: ipv6 fuse loop dm_mod ehea sg sd_mod crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
NIP: c00000000003e794 LR: c00000000003e9ec CTR: 000000000000bba4
REGS: c00000006a72b5d0 TRAP: 0700   Not tainted  (2.6.31-rc4-autotest-next-20090729-5-ppc64)
MSR: 8000000000029032 <EE,ME,CE,IR,DR>  CR: 22044444  XER: 00000001
TASK = c000000069c00180[1115] 'readback' THREAD: c00000006a728000 CPU: 2
GPR00: 0000000000000001 c00000006a72b850 c000000000a93190 c00000006f6f04d0
GPR04: c00000006b810001 0000000000000008 0000000000000000 0000040000000000
GPR08: c00000006ece0ca8 c00000006a137ff8 c00000006ece0ca8 0000000000000018
GPR12: 0000000042000448 c000000000b72800 00000000ffffffff ffffffffffffffff
GPR16: 00000000477555d0 0000000000000000 0000000000000001 000003ffffffffff
GPR20: 0000040000000000 0000000000000000 0000010000000000 0000040000000000
GPR24: c00000006f6f04d0 000003ffffffffff 0000000000000007 c00000006cdc28d0
GPR28: 0000040000000000 000003fff0000000 c00000006a137ff8 0000040000000000
NIP [c00000000003e794] .free_hugepte_range+0x44/0x68
LR [c00000000003e9ec] .hugetlb_free_pgd_range+0x234/0x374
Call Trace:
[c00000006a72b850] [0000175c08000393] 0x175c08000393 (unreliable)
[c00000006a72b8c0] [c00000000003e9ec] .hugetlb_free_pgd_range+0x234/0x374
[c00000006a72b9b0] [c00000000013742c] .free_pgtables+0x90/0x140
[c00000006a72ba60] [c0000000001393c4] .exit_mmap+0x12c/0x1b8
[c00000006a72bb10] [c00000000008d460] .mmput+0x54/0x14c
[c00000006a72bba0] [c000000000092428] .exit_mm+0x17c/0x1a0
[c00000006a72bc50] [c00000000009481c] .do_exit+0x204/0x774
[c00000006a72bd30] [c000000000094e40] .do_group_exit+0xb4/0xe8
[c00000006a72bdc0] [c000000000094e88] .SyS_exit_group+0x14/0x28
[c00000006a72be30] [c0000000000085b4] syscall_exit+0x0/0x40
Instruction dump:
68800001 780007e0 0b000000 38a50001 38000000 7ca507b4 f8090000 38000001
2f850007 9003000c 7c101026 5400f7fe <0b000000> 78840724 7ca42378 4bff8ed5

next-20090728 worked fine. Last commit that changed
arch/powerpc/mm/hugetlbpage.c was cb7f3f2d92d1b26c13e30e639b6ee4a78e9a3afa

powerpc: Add memory management headers for new 64-bit BookE

I will try reverting that commit and check if that helps.

Thanks
-Sachin


-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Sachin Sant]

Sachin Sant wrote:
> next-20090728 worked fine. Last commit that changed
> arch/powerpc/mm/hugetlbpage.c was 
> cb7f3f2d92d1b26c13e30e639b6ee4a78e9a3afa
>
> powerpc: Add memory management headers for new 64-bit BookE
>
> I will try reverting that commit and check if that helps.
Hi Ben,

Reverting the above patch helped. The tests ran fine against the
patched kernel. But ofcourse that's not the solution :-)

Here is some data from xmon that might help find the reason for
the failure. This is with today's next.

: ------------[ cut here ]------------
cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
    pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
    lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
    sp: c0000000389237e0
   msr: 8000000000029032
  current = 0xc00000003b1d7780
  paca    = 0xc000000001002400
    pid   = 2839, comm = readback
kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
enter ? for help
[c000000038923880] c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
[c000000038923970] c000000000165a48 .free_pgtables+0xa0/0x154
[c000000038923a30] c000000000167f78 .exit_mmap+0x13c/0x1cc
[c000000038923ae0] c0000000000997ec .mmput+0x68/0x14c
[c000000038923b70] c00000000009f1d4 .exit_mm+0x190/0x1b8
[c000000038923c20] c0000000000a16e8 .do_exit+0x214/0x784
[c000000038923d00] c0000000000a1d1c .do_group_exit+0xc4/0xf8
[c000000038923da0] c0000000000a1d7c .SyS_exit_group+0x2c/0x48
[c000000038923e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 000000000fe15038
SP (ffb8e030) is in userspace
0:mon> e
cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
    pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
    lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
    sp: c0000000389237e0
   msr: 8000000000029032
  current = 0xc00000003b1d7780
  paca    = 0xc000000001002400
    pid   = 2839, comm = readback
kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
0:mon> r
R00 = 0000000000000001   R16 = 0000000000000000
R01 = c0000000389237e0   R17 = 0000000000000001
R02 = c000000000f165a8   R18 = 000000003fffffff
R03 = c0000000014504d0   R19 = 0000000000000000
R04 = c000000039390001   R20 = 0000000000000000
R05 = 0000000000000007   R21 = 0000010000000000
R06 = 0000000000000000   R22 = 0000000040000000
R07 = 0000000040000000   R23 = c0000000014504d0
R08 = c00000003d708188   R24 = 000000003fffffff
R09 = c00000003eb40000   R25 = 0000000000000007
R10 = c00000003d708188   R26 = c00000003ebd41b8
R11 = 0000000000000018   R27 = c0000000014504d0
R12 = 0000000040000448   R28 = c00000003eb40018
R13 = c000000001002400   R29 = 0000000000000008
R14 = 00000000ffffffff   R30 = 0000000040000000
R15 = 00000000ffffffff   R31 = c0000000389237e0
pc  = c0000000000486d4 .free_hugepte_range+0x68/0xa0
lr  = c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
msr = 8000000000029032   cr  = 20042444
ctr = 800000000000b6f4   xer = 0000000000000001   trap =  700
0:mon> 

Line 36 of arch/powerpc/include/asm/pgalloc.h corresponds to

BUG_ON(cachenum > PGF_CACHENUM_MASK);

May be something to do with number of elements in huge_pgtable_cache_name ??

Thanks
-Sachin

-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Benjamin Herrenschmidt]

On Thu, 2009-07-30 at 17:55 +0530, Sachin Sant wrote:
> Sachin Sant wrote:
> > next-20090728 worked fine. Last commit that changed
> > arch/powerpc/mm/hugetlbpage.c was 
> > cb7f3f2d92d1b26c13e30e639b6ee4a78e9a3afa
> >
> > powerpc: Add memory management headers for new 64-bit BookE
> >
> > I will try reverting that commit and check if that helps.
> Hi Ben,
> 
> Reverting the above patch helped. The tests ran fine against the
> patched kernel. But ofcourse that's not the solution :-)
> 
> Here is some data from xmon that might help find the reason for
> the failure. This is with today's next.

Thanks. I'll have a look next week. I think when I changed the indices
I may have forgotten to update something.

Cheers,
Ben.

> : ------------[ cut here ]------------
> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
>     sp: c0000000389237e0
>    msr: 8000000000029032
>   current = 0xc00000003b1d7780
>   paca    = 0xc000000001002400
>     pid   = 2839, comm = readback
> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
> enter ? for help
> [c000000038923880] c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
> [c000000038923970] c000000000165a48 .free_pgtables+0xa0/0x154
> [c000000038923a30] c000000000167f78 .exit_mmap+0x13c/0x1cc
> [c000000038923ae0] c0000000000997ec .mmput+0x68/0x14c
> [c000000038923b70] c00000000009f1d4 .exit_mm+0x190/0x1b8
> [c000000038923c20] c0000000000a16e8 .do_exit+0x214/0x784
> [c000000038923d00] c0000000000a1d1c .do_group_exit+0xc4/0xf8
> [c000000038923da0] c0000000000a1d7c .SyS_exit_group+0x2c/0x48
> [c000000038923e30] c0000000000085b4 syscall_exit+0x0/0x40
> --- Exception: c01 (System Call) at 000000000fe15038
> SP (ffb8e030) is in userspace
> 0:mon> e
> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
>     sp: c0000000389237e0
>    msr: 8000000000029032
>   current = 0xc00000003b1d7780
>   paca    = 0xc000000001002400
>     pid   = 2839, comm = readback
> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
> 0:mon> r
> R00 = 0000000000000001   R16 = 0000000000000000
> R01 = c0000000389237e0   R17 = 0000000000000001
> R02 = c000000000f165a8   R18 = 000000003fffffff
> R03 = c0000000014504d0   R19 = 0000000000000000
> R04 = c000000039390001   R20 = 0000000000000000
> R05 = 0000000000000007   R21 = 0000010000000000
> R06 = 0000000000000000   R22 = 0000000040000000
> R07 = 0000000040000000   R23 = c0000000014504d0
> R08 = c00000003d708188   R24 = 000000003fffffff
> R09 = c00000003eb40000   R25 = 0000000000000007
> R10 = c00000003d708188   R26 = c00000003ebd41b8
> R11 = 0000000000000018   R27 = c0000000014504d0
> R12 = 0000000040000448   R28 = c00000003eb40018
> R13 = c000000001002400   R29 = 0000000000000008
> R14 = 00000000ffffffff   R30 = 0000000040000000
> R15 = 00000000ffffffff   R31 = c0000000389237e0
> pc  = c0000000000486d4 .free_hugepte_range+0x68/0xa0
> lr  = c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
> msr = 8000000000029032   cr  = 20042444
> ctr = 800000000000b6f4   xer = 0000000000000001   trap =  700
> 0:mon> 
> 
> Line 36 of arch/powerpc/include/asm/pgalloc.h corresponds to
> 
> BUG_ON(cachenum > PGF_CACHENUM_MASK);
> 
> May be something to do with number of elements in huge_pgtable_cache_name ??
> 
> Thanks
> -Sachin
> 

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Sachin Sant]

Benjamin Herrenschmidt wrote:
> Thanks. I'll have a look next week. I think when I changed the indices
> I may have forgotten to update something.
>   
Ben,

I can recreate this issue with today's next.
Let me know if i can help in any way to fix this issue.

Thanks
-Sachin

>> : ------------[ cut here ]------------
>> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
>>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
>>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
>>     sp: c0000000389237e0
>>    msr: 8000000000029032
>>   current = 0xc00000003b1d7780
>>   paca    = 0xc000000001002400
>>     pid   = 2839, comm = readback
>> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
>> enter ? for help
>> [c000000038923880] c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
>> [c000000038923970] c000000000165a48 .free_pgtables+0xa0/0x154
>> [c000000038923a30] c000000000167f78 .exit_mmap+0x13c/0x1cc
>> [c000000038923ae0] c0000000000997ec .mmput+0x68/0x14c
>> [c000000038923b70] c00000000009f1d4 .exit_mm+0x190/0x1b8
>> [c000000038923c20] c0000000000a16e8 .do_exit+0x214/0x784
>> [c000000038923d00] c0000000000a1d1c .do_group_exit+0xc4/0xf8
>> [c000000038923da0] c0000000000a1d7c .SyS_exit_group+0x2c/0x48
>> [c000000038923e30] c0000000000085b4 syscall_exit+0x0/0x40
>> --- Exception: c01 (System Call) at 000000000fe15038
>> SP (ffb8e030) is in userspace
>> 0:mon> e
>> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
>>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
>>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
>>     sp: c0000000389237e0
>>    msr: 8000000000029032
>>   current = 0xc00000003b1d7780
>>   paca    = 0xc000000001002400
>>     pid   = 2839, comm = readback
>> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
>> 0:mon> r
>> R00 = 0000000000000001   R16 = 0000000000000000
>> R01 = c0000000389237e0   R17 = 0000000000000001
>> R02 = c000000000f165a8   R18 = 000000003fffffff
>> R03 = c0000000014504d0   R19 = 0000000000000000
>> R04 = c000000039390001   R20 = 0000000000000000
>> R05 = 0000000000000007   R21 = 0000010000000000
>> R06 = 0000000000000000   R22 = 0000000040000000
>> R07 = 0000000040000000   R23 = c0000000014504d0
>> R08 = c00000003d708188   R24 = 000000003fffffff
>> R09 = c00000003eb40000   R25 = 0000000000000007
>> R10 = c00000003d708188   R26 = c00000003ebd41b8
>> R11 = 0000000000000018   R27 = c0000000014504d0
>> R12 = 0000000040000448   R28 = c00000003eb40018
>> R13 = c000000001002400   R29 = 0000000000000008
>> R14 = 00000000ffffffff   R30 = 0000000040000000
>> R15 = 00000000ffffffff   R31 = c0000000389237e0
>> pc  = c0000000000486d4 .free_hugepte_range+0x68/0xa0
>> lr  = c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
>> msr = 8000000000029032   cr  = 20042444
>> ctr = 800000000000b6f4   xer = 0000000000000001   trap =  700
>> 0:mon> 
>>
>> Line 36 of arch/powerpc/include/asm/pgalloc.h corresponds to
>>
>> BUG_ON(cachenum > PGF_CACHENUM_MASK);
>>
>> May be something to do with number of elements in huge_pgtable_cache_name ??
>>
>> Thanks
>> -Sachin
-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Kumar Gala]

On Jul 29, 2009, at 10:04 AM, Sachin Sant wrote:

> While executing hugetlb tests against today's Next tree on
> a Power 6 box came across following OOPS.

out of interest what tests are you running for hugetlb?

- k

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Sachin Sant]

Kumar Gala wrote:
>
> On Jul 29, 2009, at 10:04 AM, Sachin Sant wrote:
>
>> While executing hugetlb tests against today's Next tree on
>> a Power 6 box came across following OOPS.
>
> out of interest what tests are you running for hugetlb?
The one maintained at : http://libhugetlbfs.ozlabs.org/ which points
to the sourceforge libhugetlbfs project.

Latest release can be downloaded from sourceforge using
http://sourceforge.net/projects/libhugetlbfs/files/

I am using version 2.5

Thanks
-Sachin



-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Benjamin Herrenschmidt]

On Wed, 2009-08-05 at 16:13 +0530, Sachin Sant wrote:
> Benjamin Herrenschmidt wrote:
> > Thanks. I'll have a look next week. I think when I changed the indices
> > I may have forgotten to update something.
> >   
> Ben,
> 
> I can recreate this issue with today's next.
> Let me know if i can help in any way to fix this issue.

Does this patch fixes it ?

[PATCH] powerpc/mm: Fix encoding of page table cache numbers

The mask used to encode the page table cache number in the
batch when freeing page tables was too small for the new
possible values of MMU page sizes. This increases it along
with a comment explaining the constraints.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
 arch/powerpc/include/asm/pgalloc.h |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h
index 34b0806..f2e812d 100644
--- a/arch/powerpc/include/asm/pgalloc.h
+++ b/arch/powerpc/include/asm/pgalloc.h
@@ -28,7 +28,12 @@ typedef struct pgtable_free {
 	unsigned long val;
 } pgtable_free_t;
 
-#define PGF_CACHENUM_MASK	0x7
+/* This needs to be big enough to allow for MMU_PAGE_COUNT + 2 to be stored
+ * and small enough to fit in the low bits of any naturally aligned page
+ * table cache entry. Arbitrarily set to 0x1f, that should give us some
+ * room to grow
+ */
+#define PGF_CACHENUM_MASK	0x1f
 
 static inline pgtable_free_t pgtable_free_cache(void *p, int cachenum,
 						unsigned long mask)
-- 
1.6.0.4


> Thanks
> -Sachin
> 
> >> : ------------[ cut here ]------------
> >> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
> >>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
> >>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
> >>     sp: c0000000389237e0
> >>    msr: 8000000000029032
> >>   current = 0xc00000003b1d7780
> >>   paca    = 0xc000000001002400
> >>     pid   = 2839, comm = readback
> >> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
> >> enter ? for help
> >> [c000000038923880] c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
> >> [c000000038923970] c000000000165a48 .free_pgtables+0xa0/0x154
> >> [c000000038923a30] c000000000167f78 .exit_mmap+0x13c/0x1cc
> >> [c000000038923ae0] c0000000000997ec .mmput+0x68/0x14c
> >> [c000000038923b70] c00000000009f1d4 .exit_mm+0x190/0x1b8
> >> [c000000038923c20] c0000000000a16e8 .do_exit+0x214/0x784
> >> [c000000038923d00] c0000000000a1d1c .do_group_exit+0xc4/0xf8
> >> [c000000038923da0] c0000000000a1d7c .SyS_exit_group+0x2c/0x48
> >> [c000000038923e30] c0000000000085b4 syscall_exit+0x0/0x40
> >> --- Exception: c01 (System Call) at 000000000fe15038
> >> SP (ffb8e030) is in userspace
> >> 0:mon> e
> >> cpu 0x0: Vector: 700 (Program Check) at [c000000038923560]
> >>     pc: c0000000000486d4: .free_hugepte_range+0x68/0xa0
> >>     lr: c000000000048954: .hugetlb_free_pgd_range+0x248/0x38c
> >>     sp: c0000000389237e0
> >>    msr: 8000000000029032
> >>   current = 0xc00000003b1d7780
> >>   paca    = 0xc000000001002400
> >>     pid   = 2839, comm = readback
> >> kernel BUG at /home/linux-2.6.31-rc4/arch/powerpc/include/asm/pgalloc.h:36!
> >> 0:mon> r
> >> R00 = 0000000000000001   R16 = 0000000000000000
> >> R01 = c0000000389237e0   R17 = 0000000000000001
> >> R02 = c000000000f165a8   R18 = 000000003fffffff
> >> R03 = c0000000014504d0   R19 = 0000000000000000
> >> R04 = c000000039390001   R20 = 0000000000000000
> >> R05 = 0000000000000007   R21 = 0000010000000000
> >> R06 = 0000000000000000   R22 = 0000000040000000
> >> R07 = 0000000040000000   R23 = c0000000014504d0
> >> R08 = c00000003d708188   R24 = 000000003fffffff
> >> R09 = c00000003eb40000   R25 = 0000000000000007
> >> R10 = c00000003d708188   R26 = c00000003ebd41b8
> >> R11 = 0000000000000018   R27 = c0000000014504d0
> >> R12 = 0000000040000448   R28 = c00000003eb40018
> >> R13 = c000000001002400   R29 = 0000000000000008
> >> R14 = 00000000ffffffff   R30 = 0000000040000000
> >> R15 = 00000000ffffffff   R31 = c0000000389237e0
> >> pc  = c0000000000486d4 .free_hugepte_range+0x68/0xa0
> >> lr  = c000000000048954 .hugetlb_free_pgd_range+0x248/0x38c
> >> msr = 8000000000029032   cr  = 20042444
> >> ctr = 800000000000b6f4   xer = 0000000000000001   trap =  700
> >> 0:mon> 
> >>
> >> Line 36 of arch/powerpc/include/asm/pgalloc.h corresponds to
> >>
> >> BUG_ON(cachenum > PGF_CACHENUM_MASK);
> >>
> >> May be something to do with number of elements in huge_pgtable_cache_name ??
> >>
> >> Thanks
> >> -Sachin

Re: Next July 29 : Hugetlb test failure (OOPS free_hugepte_range)

[Sachin Sant]

Benjamin Herrenschmidt wrote:
> Does this patch fixes it ?
>
> [PATCH] powerpc/mm: Fix encoding of page table cache numbers
>
> The mask used to encode the page table cache number in the
> batch when freeing page tables was too small for the new
> possible values of MMU page sizes. This increases it along
> with a comment explaining the constraints.
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
>   
Yes this patch fixed the issue for me. Thanks Ben.

Tested-by: Sachin Sant <sachinp@in.ibm.com>

Regards
-Sachin

-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------