Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

Hi,

I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu.
It only happens if I configure a page size of 16k or 64k; 4k page size is fine.

A similar problem was recently fixed for ppc. It turned out to be a problem in ppc
specific memory management code, so that fix won't help here.

Has anyone else seen this before ? Any idea where to start looking for the problem ?

Thanks,
Guenter

--------------------------------------

[instrumentation]
pcpu_get_vm_areas: VMALLOC_START: 0xc000000000000000 VMALLOC_END: 0xc0007fff00000000 PTRS_PER_PGD: 2048 PTRS_PER_PMD: 2048 PTRS_PER_PTE: 2048 PAGE_SIZE: 0x4000
emulate_load_store_insn: Attempted access to supervisor address space: opcode=0x2c addr=0xc0007ffefffc0000
[/instrumentation]

Unhandled kernel unaligned access[#1]:
Cpu 2
$ 0   : 0000000000000000 0000000014001fe0 a8000002fe067b68 0000000000000000
$ 4   : ffffffff80116f10 0000000014001fe3 000000000000003f c0007ffefffc0000
$ 8   : ffffffff807e9998 a8000002fe716228 0000000000000000 fffffffffe000000
$12   : 0000000014001fe1 000000001000001e 6db6db6db6db6db7 0000000000000001
$16   : fffffffffc85ffc0 a8000002fe067b40 ffffffff80810000 0000000000000000
$20   : a8000002fe7161e0 ffffffff80810000 ffffffff80730000 ffffffff80810000
$24   : 0000000000000000 ffffffff8f1bbcdc
$28   : a8000002fe064000 a8000002fe067b10 a8000002fe2e53c0 ffffffff801004e0
Hi    : 000000000023d1ad
Lo    : 00000000000bf08f
epc   : ffffffff80116f38 do_ade+0x248/0x468
    Not tainted
ra    : ffffffff801004e0 ret_from_exception+0x0/0x20
Status: 14001fe3    KX SX UX KERNEL EXL IE
Cause : 00808014
BadVA : c0007ffefffc0000
PrId  : 05041100 (SiByte SB1A)
Modules linked in:
Process swapper (pid: 1, threadinfo=a8000002fe064000, task=a8000002fe063658, tls=0000000000000000)
Stack : 0000000000000100 0000000000000000 ffffffff807e9998 ffffffff80810000
        0000000000000000 ffffffff801004e0 0000000000000000 0000000014001fe0
        c0007ffefffc0000 a800000003a5c060 c0007ffefffc0040 0000000000000000
        0000000000001000 0000000000000000 ffffffff807e9998 a8000002fe716228
        0000000000000000 fffffffffe000000 0000000000000000 c0007ffefffc1000
        6db6db6db6db6db7 0000000000000001 0000000000000000 ffffffff807e9998
        ffffffff80810000 0000000000000000 a8000002fe7161e0 ffffffff80810000
        ffffffff80730000 ffffffff80810000 0000000000000000 ffffffff8f1bbcdc
        6db6db6db6db6db7 a8000002fe7161e0 a8000002fe064000 a8000002fe067c70
        a8000002fe2e53c0 ffffffff801f0d04 0000000014001fe3 000000000023d1ad
        ...
Call Trace:
[<ffffffff80116f38>] do_ade+0x248/0x468
[<ffffffff801004e0>] ret_from_exception+0x0/0x20
[<ffffffff8010585c>] __bzero+0x38/0x11c
[<ffffffff801f0d04>] pcpu_alloc+0x9b4/0xba0
[<ffffffff804b1460>] snmp_mib_init+0x40/0x80
[<ffffffff804fa6e8>] ipv6_add_dev+0x170/0x3c0
[<ffffffff80797178>] addrconf_init+0x44/0x17c
[<ffffffff8079705c>] inet6_init+0x29c/0x368
[<ffffffff8010fa88>] do_one_initcall+0x38/0x198
[<ffffffff80778350>] kernel_init+0x1ec/0x26c
[<ffffffff80112780>] kernel_thread_helper+0x10/0x18


Code: 000210f8  0222102d  dc430000 <b0e30000> b4e30007  24030000  1060ffe7  00000000  00000000
Disabling lock debugging due to kernel taint
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 5 seconds..

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Thu, Jan 28, 2010 at 07:55:14AM -0800, Guenter Roeck wrote:
> 
> I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu.
> It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
> 
> A similar problem was recently fixed for ppc. It turned out to be a problem in ppc
> specific memory management code, so that fix won't help here.
> 
> Has anyone else seen this before ? Any idea where to start looking for the problem ?

Supposedly this was working for SB1.  I suggest you find an older kernel
version that works for your with 16k pages then use git bisect to find
the problem.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 08:24:07AM -0500, Ralf Baechle wrote:
> On Thu, Jan 28, 2010 at 07:55:14AM -0800, Guenter Roeck wrote:
> > 
> > I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu.
> > It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
> > 
> > A similar problem was recently fixed for ppc. It turned out to be a problem in ppc
> > specific memory management code, so that fix won't help here.
> > 
> > Has anyone else seen this before ? Any idea where to start looking for the problem ?
> 
> Supposedly this was working for SB1.  I suggest you find an older kernel
> version that works for your with 16k pages then use git bisect to find
> the problem.
> 
It used to work with 2.6.27.

However, bisect won't work, because the code in question (per cpu memory allocation) was
completely rewritten since then.

The new percpu code tries to allocate memory just below VMALLOC_END. This works on sb1 for
a page size of 4k, but not for a page size of 16k and 64k. The value of VMALLOC_END depends
on the page size.

ppc had a similar problem. It had nothing to do with the new percpu memory allocation code,
but with memory alocation close to VMALLOC_END. In other words, it was a day-one bug which
was never noticed because allocating memory in that address space is highly unlikely.

I suspect the same is the case here. I could write some code for 2.6.27, to test the same
memory allocation there, but I am quite sure the problem is going to show up there as well.

So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 
and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.

Thanks,
Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[David Daney]

Guenter Roeck wrote:
> On Fri, Jan 29, 2010 at 08:24:07AM -0500, Ralf Baechle wrote:
>> On Thu, Jan 28, 2010 at 07:55:14AM -0800, Guenter Roeck wrote:
>>> I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu.
>>> It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
>>>
>>> A similar problem was recently fixed for ppc. It turned out to be a problem in ppc
>>> specific memory management code, so that fix won't help here.
>>>
>>> Has anyone else seen this before ? Any idea where to start looking for the problem ?
>> Supposedly this was working for SB1.  I suggest you find an older kernel
>> version that works for your with 16k pages then use git bisect to find
>> the problem.
>>
> It used to work with 2.6.27.
> 
> However, bisect won't work, because the code in question (per cpu memory allocation) was
> completely rewritten since then.
> 
> The new percpu code tries to allocate memory just below VMALLOC_END. This works on sb1 for
> a page size of 4k, but not for a page size of 16k and 64k. The value of VMALLOC_END depends
> on the page size.
> 
> ppc had a similar problem. It had nothing to do with the new percpu memory allocation code,
> but with memory alocation close to VMALLOC_END. In other words, it was a day-one bug which
> was never noticed because allocating memory in that address space is highly unlikely.
> 
> I suspect the same is the case here. I could write some code for 2.6.27, to test the same
> memory allocation there, but I am quite sure the problem is going to show up there as well.
> 
> So first question would be: Has anyone successfully loaded a 64 bit mips kernel with 2.6.32 
> and a page size of 16k or 64k ? This would at least help me reducing the problem to sb1.
> 

Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 
2.6.33-rc*.  I have not seen any crashes that can not be easily explained.

David Daney.

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:

> >So first question would be: Has anyone successfully loaded a 64
> >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> >would at least help me reducing the problem to sb1.
> 
> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> 2.6.33-rc*.  I have not seen any crashes that can not be easily
> explained.

I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
Note, I was testing with a non-16K capable userland so ok means userland is
reached.

Either way, that's good enought to look into things.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> 
> > >So first question would be: Has anyone successfully loaded a 64
> > >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> > >would at least help me reducing the problem to sb1.
> > 
> > Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> > 2.6.33-rc*.  I have not seen any crashes that can not be easily
> > explained.
> 
> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> Note, I was testing with a non-16K capable userland so ok means userland is
> reached.
> 
Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do
with the problem, though. IPv6 enabled just means that the percpu code needs to
allocate more memory. This memory allocation then crashes.

> Either way, that's good enought to look into things.
> 
Did you see the problem on a bcm1250/1480 or with some other mips core ?

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 12:11:46PM -0500, David Daney wrote:
> Guenter Roeck wrote:
> > On Fri, Jan 29, 2010 at 08:24:07AM -0500, Ralf Baechle wrote:
> >> On Thu, Jan 28, 2010 at 07:55:14AM -0800, Guenter Roeck wrote:
> >>> I get the following kernel crash when running a 2.6.32.6 kernel on a bcm1480 cpu.
> >>> It only happens if I configure a page size of 16k or 64k; 4k page size is fine.
> >>>
...
> 
> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and 
> 2.6.33-rc*.  I have not seen any crashes that can not be easily explained.
> 
Do you have IPV6 enabled ? If not, you won't see the crash. I forgot to mention that earlier.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> 
> > >So first question would be: Has anyone successfully loaded a 64
> > >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> > >would at least help me reducing the problem to sb1.
> > 
> > Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> > 2.6.33-rc*.  I have not seen any crashes that can not be easily
> > explained.
> 
> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> Note, I was testing with a non-16K capable userland so ok means userland is
> reached.
> 
> Either way, that's good enought to look into things.
> 
16k page size works for me with the patch below. Not that I have any idea why;
this was just a blind test.

It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
available virtual memory per page size may contradict with the definition
of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
but the comments indicate that a system with 16k pages should have _less_
virtual memory available than a system with 4k pages because it only uses
a 2 level page table.

Guenter

---------------

git diff arch/mips/include/asm/pgtable-64.h
diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
index 9cd5089..bd61030 100644
--- a/arch/mips/include/asm/pgtable-64.h
+++ b/arch/mips/include/asm/pgtable-64.h
@@ -110,7 +110,7 @@
 #define VMALLOC_START          MAP_BASE
 #define VMALLOC_END    \
        (VMALLOC_START + \
-        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
+        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
 #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
        VMALLOC_START != CKSSEG
 /* Load modules into 32bit-compatible segment. */

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[David Daney]

Guenter Roeck wrote:
> On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
>> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
>>
>>>> So first question would be: Has anyone successfully loaded a 64
>>>> bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
>>>> would at least help me reducing the problem to sb1.
>>> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
>>> 2.6.33-rc*.  I have not seen any crashes that can not be easily
>>> explained.
>> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
>> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
>> Note, I was testing with a non-16K capable userland so ok means userland is
>> reached.
>>
>> Either way, that's good enought to look into things.
>>
> 16k page size works for me with the patch below. Not that I have any idea why;
> this was just a blind test.
> 
> It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
> available virtual memory per page size may contradict with the definition
> of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
> but the comments indicate that a system with 16k pages should have _less_
> virtual memory available than a system with 4k pages because it only uses
> a 2 level page table.
> 
> Guenter
> 
> ---------------
> 
> git diff arch/mips/include/asm/pgtable-64.h
> diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
> index 9cd5089..bd61030 100644
> --- a/arch/mips/include/asm/pgtable-64.h
> +++ b/arch/mips/include/asm/pgtable-64.h
> @@ -110,7 +110,7 @@
>  #define VMALLOC_START          MAP_BASE
>  #define VMALLOC_END    \
>         (VMALLOC_START + \
> -        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
> +        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
>  #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
>         VMALLOC_START != CKSSEG
>  /* Load modules into 32bit-compatible segment. */

Although it may fix it, I think something along the lines of this:

In asm/mach-generic/spaces.h:
#define MAX_VIRTUAL_SIZE (1 << some number)

In asm/mach-sibyte/paces.h:
#define MAX_VIRTUAL_SIZE (1 << some other umber)

In arch/mips/include/asm/pgtable-64.h

#define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * 
PAGE_SIZE)

#define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) 
- (1UL << 32))

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 01:56:32PM -0500, David Daney wrote:
> Guenter Roeck wrote:
> > On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> >> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> >>
> >>>> So first question would be: Has anyone successfully loaded a 64
> >>>> bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> >>>> would at least help me reducing the problem to sb1.
> >>> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> >>> 2.6.33-rc*.  I have not seen any crashes that can not be easily
> >>> explained.
> >> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> >> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> >> Note, I was testing with a non-16K capable userland so ok means userland is
> >> reached.
> >>
> >> Either way, that's good enought to look into things.
> >>
> > 16k page size works for me with the patch below. Not that I have any idea why;
> > this was just a blind test.
> > 
> > It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
> > available virtual memory per page size may contradict with the definition
> > of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
> > but the comments indicate that a system with 16k pages should have _less_
> > virtual memory available than a system with 4k pages because it only uses
> > a 2 level page table.
> > 
> > Guenter
> > 
> > ---------------
> > 
> > git diff arch/mips/include/asm/pgtable-64.h
> > diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
> > index 9cd5089..bd61030 100644
> > --- a/arch/mips/include/asm/pgtable-64.h
> > +++ b/arch/mips/include/asm/pgtable-64.h
> > @@ -110,7 +110,7 @@
> >  #define VMALLOC_START          MAP_BASE
> >  #define VMALLOC_END    \
> >         (VMALLOC_START + \
> > -        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
> > +        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
> >  #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
> >         VMALLOC_START != CKSSEG
> >  /* Load modules into 32bit-compatible segment. */
> 
> Although it may fix it, I think something along the lines of this:
> 
> In asm/mach-generic/spaces.h:
> #define MAX_VIRTUAL_SIZE (1 << some number)
> 
> In asm/mach-sibyte/paces.h:
> #define MAX_VIRTUAL_SIZE (1 << some other umber)
> 
> In arch/mips/include/asm/pgtable-64.h
> 
> #define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * 
> PAGE_SIZE)
> 
> #define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) 
> - (1UL << 32))

Something like that. My patch wasn't supposed to be a proposal for a fix,
just a guide to help finding the underlying problem.
It may require some factor related to the page size.
Someone who knows mips and its memory management scheme should have
a look, otherwise it would just be a trial-end-error fix.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[David Daney]

Guenter Roeck wrote:
> On Fri, Jan 29, 2010 at 01:56:32PM -0500, David Daney wrote:
>> Guenter Roeck wrote:
>>> On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
>>>> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
>>>>
>>>>>> So first question would be: Has anyone successfully loaded a 64
>>>>>> bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
>>>>>> would at least help me reducing the problem to sb1.
>>>>> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
>>>>> 2.6.33-rc*.  I have not seen any crashes that can not be easily
>>>>> explained.
>>>> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
>>>> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
>>>> Note, I was testing with a non-16K capable userland so ok means userland is
>>>> reached.
>>>>
>>>> Either way, that's good enought to look into things.
>>>>
>>> 16k page size works for me with the patch below. Not that I have any idea why;
>>> this was just a blind test.
>>>
>>> It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
>>> available virtual memory per page size may contradict with the definition
>>> of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
>>> but the comments indicate that a system with 16k pages should have _less_
>>> virtual memory available than a system with 4k pages because it only uses
>>> a 2 level page table.
>>>
>>> Guenter
>>>
>>> ---------------
>>>
>>> git diff arch/mips/include/asm/pgtable-64.h
>>> diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
>>> index 9cd5089..bd61030 100644
>>> --- a/arch/mips/include/asm/pgtable-64.h
>>> +++ b/arch/mips/include/asm/pgtable-64.h
>>> @@ -110,7 +110,7 @@
>>>  #define VMALLOC_START          MAP_BASE
>>>  #define VMALLOC_END    \
>>>         (VMALLOC_START + \
>>> -        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
>>> +        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
>>>  #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
>>>         VMALLOC_START != CKSSEG
>>>  /* Load modules into 32bit-compatible segment. */
>> Although it may fix it, I think something along the lines of this:
>>
>> In asm/mach-generic/spaces.h:
>> #define MAX_VIRTUAL_SIZE (1 << some number)
>>
>> In asm/mach-sibyte/paces.h:
>> #define MAX_VIRTUAL_SIZE (1 << some other umber)
>>
>> In arch/mips/include/asm/pgtable-64.h
>>
>> #define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * 
>> PAGE_SIZE)
>>
>> #define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) 
>> - (1UL << 32))
> 
> Something like that. My patch wasn't supposed to be a proposal for a fix,
> just a guide to help finding the underlying problem.
> It may require some factor related to the page size.
> Someone who knows mips and its memory management scheme should have
> a look, otherwise it would just be a trial-end-error fix.
> 

I suspect you are hitting a maximum valid address bits limit and getting 
the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
limit seems to be the solution.  I don't have the manual for the sibyte, 
so I don't know what the limit is.  The architecture specification 
doesn't state a fixed limit, although it tells what should happen when 
the limit is reached.

David Daney

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 02:28:49PM -0500, David Daney wrote:
...
> 
> I suspect you are hitting a maximum valid address bits limit and getting 
> the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
> limit seems to be the solution.  I don't have the manual for the sibyte, 
> so I don't know what the limit is.  The architecture specification 
> doesn't state a fixed limit, although it tells what should happen when 
> the limit is reached.
> 
You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.

I'll see if I can dig up a sibyte manual.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 02:28:49PM -0500, David Daney wrote:
> Guenter Roeck wrote:
> > On Fri, Jan 29, 2010 at 01:56:32PM -0500, David Daney wrote:
> >> Guenter Roeck wrote:
> >>> On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> >>>> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> >>>>
> >>>>>> So first question would be: Has anyone successfully loaded a 64
> >>>>>> bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> >>>>>> would at least help me reducing the problem to sb1.
> >>>>> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> >>>>> 2.6.33-rc*.  I have not seen any crashes that can not be easily
> >>>>> explained.
> >>>> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> >>>> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> >>>> Note, I was testing with a non-16K capable userland so ok means userland is
> >>>> reached.
> >>>>
> >>>> Either way, that's good enought to look into things.
> >>>>
> >>> 16k page size works for me with the patch below. Not that I have any idea why;
> >>> this was just a blind test.
> >>>
> >>> It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
> >>> available virtual memory per page size may contradict with the definition
> >>> of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
> >>> but the comments indicate that a system with 16k pages should have _less_
> >>> virtual memory available than a system with 4k pages because it only uses
> >>> a 2 level page table.
> >>>
> >>> Guenter
> >>>
> >>> ---------------
> >>>
> >>> git diff arch/mips/include/asm/pgtable-64.h
> >>> diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
> >>> index 9cd5089..bd61030 100644
> >>> --- a/arch/mips/include/asm/pgtable-64.h
> >>> +++ b/arch/mips/include/asm/pgtable-64.h
> >>> @@ -110,7 +110,7 @@
> >>>  #define VMALLOC_START          MAP_BASE
> >>>  #define VMALLOC_END    \
> >>>         (VMALLOC_START + \
> >>> -        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
> >>> +        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
> >>>  #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
> >>>         VMALLOC_START != CKSSEG
> >>>  /* Load modules into 32bit-compatible segment. */
> >> Although it may fix it, I think something along the lines of this:
> >>
> >> In asm/mach-generic/spaces.h:
> >> #define MAX_VIRTUAL_SIZE (1 << some number)
> >>
> >> In asm/mach-sibyte/paces.h:
> >> #define MAX_VIRTUAL_SIZE (1 << some other umber)
> >>
> >> In arch/mips/include/asm/pgtable-64.h
> >>
> >> #define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * 
> >> PAGE_SIZE)
> >>
> >> #define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) 
> >> - (1UL << 32))
> > 
> > Something like that. My patch wasn't supposed to be a proposal for a fix,
> > just a guide to help finding the underlying problem.
> > It may require some factor related to the page size.
> > Someone who knows mips and its memory management scheme should have
> > a look, otherwise it would just be a trial-end-error fix.
> > 
> 
> I suspect you are hitting a maximum valid address bits limit and getting 
> the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
> limit seems to be the solution.  I don't have the manual for the sibyte, 
> so I don't know what the limit is.  The architecture specification 
> doesn't state a fixed limit, although it tells what should happen when 
> the limit is reached.
> 
To follow up on this - does Octeon have a fixed VM size limit ?
And when you run your tests with larger page sizes, do you have IPV6 enabled ?

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[David Daney]

Guenter Roeck wrote:
> On Fri, Jan 29, 2010 at 02:28:49PM -0500, David Daney wrote:
>> Guenter Roeck wrote:
>>> On Fri, Jan 29, 2010 at 01:56:32PM -0500, David Daney wrote:
>>>> Guenter Roeck wrote:
>>>>> On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
>>>>>> On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
>>>>>>
>>>>>>>> So first question would be: Has anyone successfully loaded a 64
>>>>>>>> bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
>>>>>>>> would at least help me reducing the problem to sb1.
>>>>>>> Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
>>>>>>> 2.6.33-rc*.  I have not seen any crashes that can not be easily
>>>>>>> explained.
>>>>>> I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
>>>>>> 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
>>>>>> Note, I was testing with a non-16K capable userland so ok means userland is
>>>>>> reached.
>>>>>>
>>>>>> Either way, that's good enought to look into things.
>>>>>>
>>>>> 16k page size works for me with the patch below. Not that I have any idea why;
>>>>> this was just a blind test.
>>>>>
>>>>> It seems to me that the notes in arch/mips/include/asm/pgtable-64.h regarding
>>>>> available virtual memory per page size may contradict with the definition
>>>>> of VMALLOC_END. VMALLOC_END-VMALLOC_START increases as the page size increases,
>>>>> but the comments indicate that a system with 16k pages should have _less_
>>>>> virtual memory available than a system with 4k pages because it only uses
>>>>> a 2 level page table.
>>>>>
>>>>> Guenter
>>>>>
>>>>> ---------------
>>>>>
>>>>> git diff arch/mips/include/asm/pgtable-64.h
>>>>> diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
>>>>> index 9cd5089..bd61030 100644
>>>>> --- a/arch/mips/include/asm/pgtable-64.h
>>>>> +++ b/arch/mips/include/asm/pgtable-64.h
>>>>> @@ -110,7 +110,7 @@
>>>>>  #define VMALLOC_START          MAP_BASE
>>>>>  #define VMALLOC_END    \
>>>>>         (VMALLOC_START + \
>>>>> -        PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE - (1UL << 32))
>>>>> +        (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE / 16) - (1UL << 32))
>>>>>  #if defined(CONFIG_MODULES) && defined(KBUILD_64BIT_SYM32) && \
>>>>>         VMALLOC_START != CKSSEG
>>>>>  /* Load modules into 32bit-compatible segment. */
>>>> Although it may fix it, I think something along the lines of this:
>>>>
>>>> In asm/mach-generic/spaces.h:
>>>> #define MAX_VIRTUAL_SIZE (1 << some number)
>>>>
>>>> In asm/mach-sibyte/paces.h:
>>>> #define MAX_VIRTUAL_SIZE (1 << some other umber)
>>>>
>>>> In arch/mips/include/asm/pgtable-64.h
>>>>
>>>> #define VIRTUAL_SIZE (PTRS_PER_PGD * PTRS_PER_PMD * PTRS_PER_PTE * 
>>>> PAGE_SIZE)
>>>>
>>>> #define VMALLOC_END (VMALLOC_START + min(MAX_VIRTUAL_SIZE, VIRTUAL_SIZE) 
>>>> - (1UL << 32))
>>> Something like that. My patch wasn't supposed to be a proposal for a fix,
>>> just a guide to help finding the underlying problem.
>>> It may require some factor related to the page size.
>>> Someone who knows mips and its memory management scheme should have
>>> a look, otherwise it would just be a trial-end-error fix.
>>>
>> I suspect you are hitting a maximum valid address bits limit and getting 
>> the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
>> limit seems to be the solution.  I don't have the manual for the sibyte, 
>> so I don't know what the limit is.  The architecture specification 
>> doesn't state a fixed limit, although it tells what should happen when 
>> the limit is reached.
>>
> To follow up on this - does Octeon have a fixed VM size limit ?

Yes.  But it is larger than SB1's 44 bits.  For Octeon it is 49 bits.


> And when you run your tests with larger page sizes, do you have IPV6 enabled ?

Yes.

But at 16K pages the VM size is 47 bits, so it is under the limit.

At 64K pages I am using 2-level page tables for 42 bits of VM space 
which is again under the limit.

On SB1 and 3-level tables, you will always exceed the limit for both 16K 
(47 bits) and 64K (55 bits) page sizes.

David Daney

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 03:23:21PM -0500, David Daney wrote:
...
> > To follow up on this - does Octeon have a fixed VM size limit ?
> 
> Yes.  But it is larger than SB1's 44 bits.  For Octeon it is 49 bits.
> 
> 
> > And when you run your tests with larger page sizes, do you have IPV6 enabled ?
> 
> Yes.
> 
> But at 16K pages the VM size is 47 bits, so it is under the limit.
> 
> At 64K pages I am using 2-level page tables for 42 bits of VM space 
> which is again under the limit.
> 
> On SB1 and 3-level tables, you will always exceed the limit for both 16K 
> (47 bits) and 64K (55 bits) page sizes.
> 
Ok, based on this info I'll send out a proposed patch shortly.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Fri, Jan 29, 2010 at 10:21:19AM -0800, Guenter Roeck wrote:

> On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> > On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> > 
> > > >So first question would be: Has anyone successfully loaded a 64
> > > >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> > > >would at least help me reducing the problem to sb1.
> > > 
> > > Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> > > 2.6.33-rc*.  I have not seen any crashes that can not be easily
> > > explained.
> > 
> > I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> > 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> > Note, I was testing with a non-16K capable userland so ok means userland is
> > reached.
> > 
> Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do
> with the problem, though. IPv6 enabled just means that the percpu code needs to
> allocate more memory. This memory allocation then crashes.
> 
> > Either way, that's good enought to look into things.
> > 
> Did you see the problem on a bcm1250/1480 or with some other mips core ?

That was on R10000.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Fri, Jan 29, 2010 at 09:05:41PM -0500, Ralf Baechle wrote:
> On Fri, Jan 29, 2010 at 10:21:19AM -0800, Guenter Roeck wrote:
> 
> > On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> > > On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> > > 
> > > > >So first question would be: Has anyone successfully loaded a 64
> > > > >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> > > > >would at least help me reducing the problem to sb1.
> > > > 
> > > > Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> > > > 2.6.33-rc*.  I have not seen any crashes that can not be easily
> > > > explained.
> > > 
> > > I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> > > 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> > > Note, I was testing with a non-16K capable userland so ok means userland is
> > > reached.
> > > 
> > Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do
> > with the problem, though. IPv6 enabled just means that the percpu code needs to
> > allocate more memory. This memory allocation then crashes.
> > 
> > > Either way, that's good enought to look into things.
> > > 
> > Did you see the problem on a bcm1250/1480 or with some other mips core ?
> 
> That was on R10000.
> 
According to Wikipedia, R10k has a 44 bit virtual memory size limit.

So I think we have two options, assuming we go with the approach I used 
in the patch I sent out yesterday. We could either set the default to 44 bit
and override it with larger values for processors like the Octeon, or go with
a large default (eg with the 60 bit I proposed in the patch) and override it
with smaller values as needed (ie pretty much for all CPUs).

Seems to me if we would have fewer exceptions if we set the default to 44 bit.
Also, it would probably be better if the number of bits is too low for a given CPU,
since it would not result in a crash. So I would prefer to use 44 bit as default.

Thoughts ?

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Sat, Jan 30, 2010 at 04:34:45PM -0500, Guenter Roeck wrote:
> On Fri, Jan 29, 2010 at 09:05:41PM -0500, Ralf Baechle wrote:
> > On Fri, Jan 29, 2010 at 10:21:19AM -0800, Guenter Roeck wrote:
> > 
> > > On Fri, Jan 29, 2010 at 01:06:20PM -0500, Ralf Baechle wrote:
> > > > On Fri, Jan 29, 2010 at 09:11:46AM -0800, David Daney wrote:
> > > > 
> > > > > >So first question would be: Has anyone successfully loaded a 64
> > > > > >bit mips kernel with 2.6.32 and a page size of 16k or 64k ? This
> > > > > >would at least help me reducing the problem to sb1.
> > > > > 
> > > > > Yes, I routinely run with both 64K and 16K page sizes on 2.6.32 and
> > > > > 2.6.33-rc*.  I have not seen any crashes that can not be easily
> > > > > explained.
> > > > 
> > > > I can reproduce it with today's 14b7baff3eb4b1b46a592630e6f85ded9264798a.
> > > > 4K page size works ok, 16K without IPv6 works ok and 16K with IPv6 crashes.
> > > > Note, I was testing with a non-16K capable userland so ok means userland is
> > > > reached.
> > > > 
> > > Yes, I forgot to mention that IPv6 needs to be enabled. That has nothing to do
> > > with the problem, though. IPv6 enabled just means that the percpu code needs to
> > > allocate more memory. This memory allocation then crashes.
> > > 
> > > > Either way, that's good enought to look into things.
> > > > 
> > > Did you see the problem on a bcm1250/1480 or with some other mips core ?
> > 
> > That was on R10000.
> > 
> According to Wikipedia, R10k has a 44 bit virtual memory size limit.
> 
> So I think we have two options, assuming we go with the approach I used 
> in the patch I sent out yesterday. We could either set the default to 44 bit
> and override it with larger values for processors like the Octeon, or go with
> a large default (eg with the 60 bit I proposed in the patch) and override it
> with smaller values as needed (ie pretty much for all CPUs).
> 
> Seems to me if we would have fewer exceptions if we set the default to 44 bit.
> Also, it would probably be better if the number of bits is too low for a given CPU,
> since it would not result in a crash. So I would prefer to use 44 bit as default.
> 
> Thoughts ?
> 
As a followup, this is what I found so far for the various mips64 cores in respect to
virtual memory size.

Core		Virtual memory size
R10000		44 bit
SB1		44 bit
Octeon		49 bit
MIPS 20Kc	40 bit
MIPS 5Kc/5Kf	42 bit
Loongson 2E/2F	40 bit

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Maciej W. Rozycki]

On Fri, 29 Jan 2010, Guenter Roeck wrote:

> > I suspect you are hitting a maximum valid address bits limit and getting 
> > the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
> > limit seems to be the solution.  I don't have the manual for the sibyte, 
> > so I don't know what the limit is.  The architecture specification 
> > doesn't state a fixed limit, although it tells what should happen when 
> > the limit is reached.
> > 
> You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.

 The size of the address space can be probed via CP0 registers (for MIPS 
architecture processors that is).  No need to add any CPU dependencies 
(except from legacy 64-bit MIPS processors perhaps).

  Maciej

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Sun, Jan 31, 2010 at 04:10:10AM -0500, Maciej W. Rozycki wrote:
> On Fri, 29 Jan 2010, Guenter Roeck wrote:
> 
> > > I suspect you are hitting a maximum valid address bits limit and getting 
> > > the Address Exception.  Limiting VMALLOC_END so that you don't hit the 
> > > limit seems to be the solution.  I don't have the manual for the sibyte, 
> > > so I don't know what the limit is.  The architecture specification 
> > > doesn't state a fixed limit, although it tells what should happen when 
> > > the limit is reached.
> > > 
> > You mean there might be a CPU-specific limit ? I hope not - that would be quite messy.
> 
>  The size of the address space can be probed via CP0 registers (for MIPS 
> architecture processors that is).  No need to add any CPU dependencies 
> (except from legacy 64-bit MIPS processors perhaps).
> 
That would help. Do you happen to know which CP0 register(s) to look for ? 
I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Sun, Jan 31, 2010 at 08:55:03AM -0800, Guenter Roeck wrote:

> >  The size of the address space can be probed via CP0 registers (for MIPS 
> > architecture processors that is).  No need to add any CPU dependencies 
> > (except from legacy 64-bit MIPS processors perhaps).
> > 
> That would help. Do you happen to know which CP0 register(s) to look for ? 
> I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.

Write a value with all bits set to c0_entryhi, then read it back again.
The set bits in the VPN2 bitfield will indicate the size of the virtual
address range supported.  The MIPS64 documentation also calls this value
SEGBITS.  The nice thing about this probe is that it is supported for
all 64-bit MIPS processors except the R8000 which has an entirely different
TLB scheme anyway.

Similarly it is possible to probe the physical address range in either
c0_entrylo0 or c0_entrylo1.  This is also of interest on 32-bit processors.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Maciej W. Rozycki]

On Mon, 1 Feb 2010, Ralf Baechle wrote:

> Write a value with all bits set to c0_entryhi, then read it back again.
> The set bits in the VPN2 bitfield will indicate the size of the virtual
> address range supported.  The MIPS64 documentation also calls this value
> SEGBITS.  The nice thing about this probe is that it is supported for
> all 64-bit MIPS processors except the R8000 which has an entirely different
> TLB scheme anyway.
> 
> Similarly it is possible to probe the physical address range in either
> c0_entrylo0 or c0_entrylo1.  This is also of interest on 32-bit processors.

 Indeed -- IIRC the architecture spec calls this value PABITS.  I wasn't 
sure about the legacy processors -- if that works with them too, then it's 
even better.

  Maciej

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

Ralf Baechle DL5RB wrote:
> 
> On Sun, Jan 31, 2010 at 08:55:03AM -0800, Guenter Roeck wrote:
> 
>> >  The size of the address space can be probed via CP0 registers (for
>> MIPS 
>> > architecture processors that is).  No need to add any CPU dependencies 
>> > (except from legacy 64-bit MIPS processors perhaps).
>> > 
>> That would help. Do you happen to know which CP0 register(s) to look for
>> ? 
>> I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.
> 
> Write a value with all bits set to c0_entryhi, then read it back again.
> The set bits in the VPN2 bitfield will indicate the size of the virtual
> address range supported.  The MIPS64 documentation also calls this value
> SEGBITS.  The nice thing about this probe is that it is supported for
> all 64-bit MIPS processors except the R8000 which has an entirely
> different
> TLB scheme anyway.
> 
> Similarly it is possible to probe the physical address range in either
> c0_entrylo0 or c0_entrylo1.  This is also of interest on 32-bit
> processors.
> 
>   Ralf
> 
> 
> 

Ok, I'll try that and submit a new set of patches if it works.

Guenter

-- 
View this message in context: http://old.nabble.com/Kernel-crash-in-2.6.32.6---bcm1480-with-16k-page-size-tp27358179p27405624.html
Sent from the linux-mips main mailing list archive at Nabble.com.

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Mon, Feb 01, 2010 at 02:50:27PM +0000, Maciej W. Rozycki wrote:

> > Write a value with all bits set to c0_entryhi, then read it back again.
> > The set bits in the VPN2 bitfield will indicate the size of the virtual
> > address range supported.  The MIPS64 documentation also calls this value
> > SEGBITS.  The nice thing about this probe is that it is supported for
> > all 64-bit MIPS processors except the R8000 which has an entirely different
> > TLB scheme anyway.
> > 
> > Similarly it is possible to probe the physical address range in either
> > c0_entrylo0 or c0_entrylo1.  This is also of interest on 32-bit processors.
> 
>  Indeed -- IIRC the architecture spec calls this value PABITS.  I wasn't 
> sure about the legacy processors -- if that works with them too, then it's 
> even better.

The probing method was undocumented until MIPS64 but if you look at the
format of the EntryLo register it's always been possible.  The R10000
needs special treatment though - it has the UC (Uncache Attribute) field
in the bits 62..63 of EntryLo; this field needs to be ignored.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Sun, 2010-01-31 at 21:18 -0500, Ralf Baechle wrote:
> On Sun, Jan 31, 2010 at 08:55:03AM -0800, Guenter Roeck wrote:
> 
> > >  The size of the address space can be probed via CP0 registers (for MIPS 
> > > architecture processors that is).  No need to add any CPU dependencies 
> > > (except from legacy 64-bit MIPS processors perhaps).
> > > 
> > That would help. Do you happen to know which CP0 register(s) to look for ? 
> > I browsed through the MIPS 5K and 20Kc manuals, but didn't find it.
> 
> Write a value with all bits set to c0_entryhi, then read it back again.
> The set bits in the VPN2 bitfield will indicate the size of the virtual
> address range supported.  The MIPS64 documentation also calls this value
> SEGBITS.  The nice thing about this probe is that it is supported for
> all 64-bit MIPS processors except the R8000 which has an entirely different
> TLB scheme anyway.

Are you sure that this doesn't work for the R8000 ? From the user's
manual (section 2.1.9, EntryHi) it looks like it should work.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Ralf Baechle]

On Mon, Feb 01, 2010 at 12:21:17PM -0800, Guenter Roeck wrote:

> > Write a value with all bits set to c0_entryhi, then read it back again.
> > The set bits in the VPN2 bitfield will indicate the size of the virtual
> > address range supported.  The MIPS64 documentation also calls this value
> > SEGBITS.  The nice thing about this probe is that it is supported for
> > all 64-bit MIPS processors except the R8000 which has an entirely different
> > TLB scheme anyway.
> 
> Are you sure that this doesn't work for the R8000 ? From the user's
> manual (section 2.1.9, EntryHi) it looks like it should work.

The probe itself will work if it's carefully written not to get fooled by
difference in bits 12..18 but that's easy) but still the R8000 has a fairly
different TLB architecture which Linux doesn't support - the machines are
rare and none of the active developers has one.

  Ralf

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Guenter Roeck]

On Mon, 2010-02-01 at 15:49 -0500, Ralf Baechle wrote:
> On Mon, Feb 01, 2010 at 12:21:17PM -0800, Guenter Roeck wrote:
> 
> > > Write a value with all bits set to c0_entryhi, then read it back again.
> > > The set bits in the VPN2 bitfield will indicate the size of the virtual
> > > address range supported.  The MIPS64 documentation also calls this value
> > > SEGBITS.  The nice thing about this probe is that it is supported for
> > > all 64-bit MIPS processors except the R8000 which has an entirely different
> > > TLB scheme anyway.
> > 
> > Are you sure that this doesn't work for the R8000 ? From the user's
> > manual (section 2.1.9, EntryHi) it looks like it should work.
> 
> The probe itself will work if it's carefully written not to get fooled by
> difference in bits 12..18 but that's easy) but still the R8000 has a fairly
> different TLB architecture which Linux doesn't support - the machines are
> rare and none of the active developers has one.
> 
Looks like my code should work. I'll send out a patch in a couple of
minutes.

Guenter

Re: Kernel crash in 2.6.32.6 / bcm1480 with 16k page size

[Maciej W. Rozycki]

On Mon, 1 Feb 2010, Ralf Baechle wrote:

> >  Indeed -- IIRC the architecture spec calls this value PABITS.  I wasn't 
> > sure about the legacy processors -- if that works with them too, then it's 
> > even better.
> 
> The probing method was undocumented until MIPS64 but if you look at the
> format of the EntryLo register it's always been possible.  The R10000

 Frankly, I would be concerned about some reserved bits left floating 
rather than hardwired to any particular logical state, but the number of 
chips to check is limited.  At least for some of them documentation is 
explicit the value reported is fixed and we can take it as an indication 
it is likely actually the case.

> needs special treatment though - it has the UC (Uncache Attribute) field
> in the bits 62..63 of EntryLo; this field needs to be ignored.

 Well, I think masking out everything beyond what would become the bit #63 
of the physical address unconditionally is a safe and reasonable way to 
make the approach automagically work for the R10k too.

  Maciej