[Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Yang Shi]

Hi experts,

I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I
use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64
bit kernel is used. During kernel loading module got bus error, see
below log:

root@bcm1250a-1:/root> insmod firmware_class.ko
DBE physical address: 0080e87020
Data bus error, epc == ffffffff801053e0, ra == ffffffff801725d0
Oops[#1]:
Cpu 0
$ 0 : 0000000000000000 0000000014001fe0 ffffffff8051d308 0000000000000000
$ 4 : a800000080e97000 0000000000000000 0000000000000000 0000000000000002
$ 8 : ffffffff80515868 0000000000000000 0000000000000001 a80000000fb0c030
$12 : 0000000000000010 ffffffff804ab728 a80000000f81ba00 0000000000000001
$16 : 0000000000000001 ffffffffc0001950 0000000000000000 c000000000005000
$20 : 0000000000000000 a80000000faabd90 c000000000008d18 c000000000007e98
$24 : 0000000000000000 0000000010010598
$28 : a80000000faa8000 a80000000faabd50 ffffffffc0001968 ffffffff801725d0
Hi : ffffffffffffffff
Lo : fffffffffffffff0
epc : ffffffff801053e0 memcpy+0x0/0x4
Not tainted
ra : ffffffff801725d0 load_module+0x1240/0x1c30
Status: 14001fe3 KX SX UX KERNEL EXL IE
Cause : 0080801c
PrId : 01040102 (SiByte SB1)
Modules linked in:
Process insmod (pid: 1799, threadinfo=a80000000faa8000,
task=a80000000f898800, tls=00000000100a7480)
Stack : 0000000000000000 ffffffff80484190 ffffffffc0001738 ffffffff00000003
0000000000000000 0000000000000000 0000000000000000 0000000200000000
0000017f00000180 c000000000008d58 ffffffff803ec728 0000000000001cf8
ffffffff80515868 a80000000f8d14a0 0000000000001b60 0000000000000000
0000000000000029 000000000000002d 000000000000003b 000000000000003a
c00000000000c008 a80000000f92aa20 c000000000007bba c000000000008a98
ffffffffc0004d88 ffffffff803ec718 ffffffff803ec718 ffffffff801df1f8
00000000000076ce 00000000100a0ff8 00000000100a1008 ffffffff804b0000
000000007f9d0f11 0000000000000003 0000000000000000 00000000100d0bc0
0000000000000000 ffffffff80173048 00000000100a0ff8 00000000000076ce
...
Call Trace:
[<ffffffff801053e0>] memcpy+0x0/0x4
[<ffffffff801725d0>] load_module+0x1240/0x1c30
[<ffffffff80173048>] SyS_init_module+0x88/0x230
[<ffffffff80102ec4>] handle_sysn32+0x44/0x9c


Code: 03e00008 00000000 00000000 <0080102d> cca00000 cc810000 2cca0008
30890007 cca00020
Disabling lock debugging due to kernel taint
Cache error exception on CPU 0:
Cache error exception on CPU 0:
c0_errorepc == 800000f0
c0_errctl == 40000000 dcache
c0_cerr_d == 88801000 fill/wb, multi-err data-DBE
c0_cerr_dpa == 0080e87000
Dcache index 0x1000 [Bank 2 Set 0x00] LRU > 1 2 0 3 > MRU
0 [PA 00004ad000] [state COH-SHD (0f)] raw tags: 1E724000-00000000004AC400
00-0000000000000000 9E-FFFFFFFFFFFFFED4 9E-FFFFFFFFFFFFFED4
4E-0000000000020002
1 [PA 000050f000] [state COH-SHD (0f)] raw tags: 1E724000-000000000050E400
EC-3E343C0A30207570 A7-203A202020302024 00-3030303030303030
00-3030303030303030
2 [PA 000f93d000] [state COH-SHD (0f)] raw tags: 1E724000-000000000F93C000
00-0000000000000000 00-FFFFFFFFFFFFFFFF 00-FFFFFFFFFFFFFFFF
00-FFFFFFFFFFFFFFFF
3 [PA 000f8db000] [state COH-E-C (16)] raw tags: 2C724000-000000000F8DA000
84-FFFFFFFF80504000 84-FFFFFFFF80504000 84-FFFFFFFF80504000
84-FFFFFFFF80504000
...didn't see indicated dcache problem
Bus watcher error counters: 00000000 00030000

Last recorded signature:
Request 00 from 0, answered by 7 with Dcode 4
Kernel panic - not syncing: unhandled cache error
Rebooting in 5 seconds..

It seems bus error occurs when copying module since the memcpy is called
by percpu_modcopy. I recalled that SB1250 prior C0 version has a TLB M3
workaround which caused kernel module loading fail, however, it seems
Ralf committed a patch to solve this issue. See git log:

commit 3d45285dd1ff4d4a1361b95e2d6508579a4402b5
Author: Ralf Baechle <ralf@linux-mips.org>
Date: Tue Mar 23 17:56:38 2010 +0100

MIPS: Sibyte: Fix M3 TLB exception handler workaround.

The M3 workaround needs to cmpare the region and VPN2 fields only.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

It seems this patch can't solve the issue, or this is another different bug?

Thanks,
Yang

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Ralf Baechle]

On Fri, May 14, 2010 at 06:29:07PM +0800, Yang Shi wrote:

> I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I
> use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64
> bit kernel is used. During kernel loading module got bus error, see
> below log:

Whops.  Fixes which were supposed to handle exactly this problem went
upstream for 2.6.34-rc3 and were tested successfully by others on their
systems.

I wonder if in arch/mips/sibyte/sb1250/setup.c you can instrument
the function sb1250_m3_workaround_needed() and print the values of
soc_type, soc_pass and the retun value of that function.  Then let's take
it from there.

  Ralf

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Yang Shi]

Ralf Baechle 写道:
> On Fri, May 14, 2010 at 06:29:07PM +0800, Yang Shi wrote:
>
>   
>> I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I
>> use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64
>> bit kernel is used. During kernel loading module got bus error, see
>> below log:
>>     
>
> Whops.  Fixes which were supposed to handle exactly this problem went
> upstream for 2.6.34-rc3 and were tested successfully by others on their
> systems.
>
> I wonder if in arch/mips/sibyte/sb1250/setup.c you can instrument
> the function sb1250_m3_workaround_needed() and print the values of
> soc_type, soc_pass and the retun value of that function.  Then let's take
> it from there.
>   

See below log:

Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)

And, soc_typs is 0x0 and soc_pass is 0x11, sb1250_m3_workaround_needed 
should return 1. So, tlb refill handler should go the m3 workaround code 
path.

Thanks,
Yang

>   Ralf
>
>   

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Yang Shi]

Yang Shi 写道:
> Ralf Baechle 写道:
>   
>> On Fri, May 14, 2010 at 06:29:07PM +0800, Yang Shi wrote:
>>
>>   
>>     
>>> I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I
>>> use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64
>>> bit kernel is used. During kernel loading module got bus error, see
>>> below log:
>>>     
>>>       
>> Whops.  Fixes which were supposed to handle exactly this problem went
>> upstream for 2.6.34-rc3 and were tested successfully by others on their
>> systems.
>>
>> I wonder if in arch/mips/sibyte/sb1250/setup.c you can instrument
>> the function sb1250_m3_workaround_needed() and print the values of
>> soc_type, soc_pass and the retun value of that function.  Then let's take
>> it from there.
>>   
>>     
>
> See below log:
>
> Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
>
> And, soc_typs is 0x0 and soc_pass is 0x11, sb1250_m3_workaround_needed 
> should return 1. So, tlb refill handler should go the m3 workaround code 
> path.
>   

It seems CPU_PREFETCH caused this issue. See commit:

commit 6b4caed2ebff4ee232f227d62eb3180d0b558a31
Author: Ralf Baechle <ralf@linux-mips.org>
Date:   Wed Jan 28 17:48:40 2009 +0000

    MIPS: IP27: Switch from DMA_IP27 to DMA_COHERENT
   
    commit 0d356eaa6316cbb3e89b4607de20b2f2d0ceda25 from linux-mips
   
    The special IP27 DMA code selected by DMA_IP27 has been removed a while
    ago turning DMA_IP27 into almost a nop.  Also fixup the broken logic of
    its last users memcpy.S and memcpy-inatomic.s.
   
    Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

If undef CPU_PREFETCH for SB1250, module can be loaded correctly.

Thanks,
Yang

> Thanks,
> Yang
>
>   
>>   Ralf
>>
>>   
>>     
>
>
>
>   

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Yang Shi]

Yang Shi 写道:
> Yang Shi 写道:
>   
>> Ralf Baechle 写道:
>>   
>>     
>>> On Fri, May 14, 2010 at 06:29:07PM +0800, Yang Shi wrote:
>>>
>>>   
>>>     
>>>       
>>>> I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I
>>>> use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64
>>>> bit kernel is used. During kernel loading module got bus error, see
>>>> below log:
>>>>     
>>>>       
>>>>         
>>> Whops.  Fixes which were supposed to handle exactly this problem went
>>> upstream for 2.6.34-rc3 and were tested successfully by others on their
>>> systems.
>>>
>>> I wonder if in arch/mips/sibyte/sb1250/setup.c you can instrument
>>> the function sb1250_m3_workaround_needed() and print the values of
>>> soc_type, soc_pass and the retun value of that function.  Then let's take
>>> it from there.
>>>   
>>>     
>>>       
>> See below log:
>>
>> Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
>>
>> And, soc_typs is 0x0 and soc_pass is 0x11, sb1250_m3_workaround_needed 
>> should return 1. So, tlb refill handler should go the m3 workaround code 
>> path.
>>   
>>     
>
> It seems CPU_PREFETCH caused this issue. See commit:
>
> commit 6b4caed2ebff4ee232f227d62eb3180d0b558a31
> Author: Ralf Baechle <ralf@linux-mips.org>
> Date:   Wed Jan 28 17:48:40 2009 +0000
>
>     MIPS: IP27: Switch from DMA_IP27 to DMA_COHERENT
>    
>     commit 0d356eaa6316cbb3e89b4607de20b2f2d0ceda25 from linux-mips
>    
>     The special IP27 DMA code selected by DMA_IP27 has been removed a while
>     ago turning DMA_IP27 into almost a nop.  Also fixup the broken logic of
>     its last users memcpy.S and memcpy-inatomic.s.
>    
>     Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
>   

This patch did below fix:

-#if !defined(CONFIG_DMA_COHERENT) || !defined(CONFIG_DMA_IP27)
+#ifdef CONFIG_DMA_NONCOHERENT
 #undef CONFIG_CPU_HAS_PREFETCH

Before the fix, CONFIG_DMA_IP27 is undefined for all of boards except 
IP27, so CONFIG_CPU_HAS_PREFETCH is undefined always.

Thanks,
Yang

> If undef CPU_PREFETCH for SB1250, module can be loaded correctly.
>
> Thanks,
> Yang
>
>   
>> Thanks,
>> Yang
>>
>>   
>>     
>>>   Ralf
>>>
>>>   
>>>     
>>>       
>>
>>   
>>     
>
>
>   

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Ralf Baechle]

On Tue, May 18, 2010 at 11:23:39AM +0800, Yang Shi wrote:

> It seems CPU_PREFETCH caused this issue. See commit:
> 
> commit 6b4caed2ebff4ee232f227d62eb3180d0b558a31

Is this a commit ID of a Wind River tree?  The commit ID for the kernel.org
kernel tree is 634286f127bef8799cd04799d3e1d5471e8fd91c, for the lmo tree
e3bf818d95cb372bfe78696957586d9afff0b405.

> Author: Ralf Baechle <ralf@linux-mips.org>
> Date:   Wed Jan 28 17:48:40 2009 +0000
> 
>    MIPS: IP27: Switch from DMA_IP27 to DMA_COHERENT
>    commit 0d356eaa6316cbb3e89b4607de20b2f2d0ceda25 from linux-mips
>    The special IP27 DMA code selected by DMA_IP27 has been removed a while
>    ago turning DMA_IP27 into almost a nop.  Also fixup the broken logic of
>    its last users memcpy.S and memcpy-inatomic.s.
>    Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
> 
> If undef CPU_PREFETCH for SB1250, module can be loaded correctly.

[...]

> This patch did below fix:
>
> -#if !defined(CONFIG_DMA_COHERENT) || !defined(CONFIG_DMA_IP27)
> +#ifdef CONFIG_DMA_NONCOHERENT
> #undef CONFIG_CPU_HAS_PREFETCH
>
> Before the fix, CONFIG_DMA_IP27 is undefined for all of boards except
> IP27, so CONFIG_CPU_HAS_PREFETCH is undefined always.

That's an old issue striking here then.  Memcpy will prefetch beyond the
end of the source or destination area, if the use of prefetch is enabled.

On non-coherent systems this is a problem because these prefetches might
bring back data that was just flushed from the cache back into the cache
resulting in corrupted DMA transfers.

Some systems that do very tight checking of address but do not differenciate
between actual loads / stores and prefetch accesses may also also throw
address error exceptions when prefetching from non-decoded address ranges.
That would typically be just beyond the end of a RAM range.  For those
systems the solution is to either disable prefetching or to not use the
last page of every physically contiguous area of memory.

I'm still looking at why we this wasn't triggered earlier.  Something else
must have changed to suddenly trigger this issue - and your system seems
to be the only system affected.

  Ralf

Re: [Bug report] Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

[Yang Shi]

Ralf Baechle 写道:
> On Tue, May 18, 2010 at 11:23:39AM +0800, Yang Shi wrote:
>
>   
>> It seems CPU_PREFETCH caused this issue. See commit:
>>
>> commit 6b4caed2ebff4ee232f227d62eb3180d0b558a31
>>     
>
> Is this a commit ID of a Wind River tree?  The commit ID for the kernel.org
> kernel tree is 634286f127bef8799cd04799d3e1d5471e8fd91c, for the lmo tree
> e3bf818d95cb372bfe78696957586d9afff0b405.
>   

Yes, that commit is cherry-picked from mainline commit 
634286f127bef8799cd04799d3e1d5471e8fd91c. If I revert this commit, 
module loading works well.

>   
>> Author: Ralf Baechle <ralf@linux-mips.org>
>> Date:   Wed Jan 28 17:48:40 2009 +0000
>>
>>    MIPS: IP27: Switch from DMA_IP27 to DMA_COHERENT
>>    commit 0d356eaa6316cbb3e89b4607de20b2f2d0ceda25 from linux-mips
>>    The special IP27 DMA code selected by DMA_IP27 has been removed a while
>>    ago turning DMA_IP27 into almost a nop.  Also fixup the broken logic of
>>    its last users memcpy.S and memcpy-inatomic.s.
>>    Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
>>
>> If undef CPU_PREFETCH for SB1250, module can be loaded correctly.
>>     
>
> [...]
>
>   
>> This patch did below fix:
>>
>> -#if !defined(CONFIG_DMA_COHERENT) || !defined(CONFIG_DMA_IP27)
>> +#ifdef CONFIG_DMA_NONCOHERENT
>> #undef CONFIG_CPU_HAS_PREFETCH
>>
>> Before the fix, CONFIG_DMA_IP27 is undefined for all of boards except
>> IP27, so CONFIG_CPU_HAS_PREFETCH is undefined always.
>>     
>
> That's an old issue striking here then.  Memcpy will prefetch beyond the
> end of the source or destination area, if the use of prefetch is enabled.
>
> On non-coherent systems this is a problem because these prefetches might
> bring back data that was just flushed from the cache back into the cache
> resulting in corrupted DMA transfers.
>
> Some systems that do very tight checking of address but do not differenciate
> between actual loads / stores and prefetch accesses may also also throw
> address error exceptions when prefetching from non-decoded address ranges.
> That would typically be just beyond the end of a RAM range.  For those
>   
> systems the solution is to either disable prefetching or to not use the
> last page of every physically contiguous area of memory.
>   
> I'm still looking at why we this wasn't triggered earlier.  Something else
> must have changed to suddenly trigger this issue - and your system seems
> to be the only system affected.
>   

As I said above, before 634286f127bef8799cd04799d3e1d5471e8fd91c 
checking in, actually prefetch should be *undefined* for all of boards 
except IP27, so maybe no one noticed this issue.

Seems yes, just Sibyte is affected :(

Thanks,
Yang

>   Ralf
>
>