Problem with TLB mcheck!

Problem with TLB mcheck!

[art]

Hello linux community!
Please CC to me personaly, because I'am not subsribed yet.

I am relatively new Linux kernel (just work with some network adapter
drivers) and my english is poor, so excuse me if something wrong.
I work with Infineon ADM5120 chip (MIPS32 processor, little endian,
using tlb), setting embedded Linux on it.
Almost everything is work now (I port some drivers). But there is big
problem ( sg16lan is my network driver module):

[4294714.623000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.627000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.628000] Got mcheck at c0096604
[4294714.628000] Cpu 0
[4294714.628000] $ 0   : 00000000 10008400 00000000 00000000
[4294714.628000] $ 4   : c0098384 c00aa26c 00000001 00000001
[4294714.628000] $ 8   : 81261fe0 00008400 00000000 80318000
[4294714.628000] $12   : 0000000c 00 00000078 00000000
[4294714.628000] $16   : 000000d1 0000aea8 8038a220 8038a000
[4294714.628000] $20   : 80323420 80030000 00000002 8038a220
[4294714.628000] $24   : 00000000 802c0000
[4294714.628000] $28   : 81260000 81261e60 00003fde c0096604
[4294714.628000] Hi    : 00000280
[4294714.628000] Lo    : 00000230
[4294714.628000] epc   : c0096604 store_download+0x424/0x4fc [sg16lan]     Not tainted
[4294714.628000] ra    : c0096604 store_download+0x424/0x4fc [sg16lan]
[4294714.628000] Status: 10208403    KERNEL EXL IE
[4294714.628000] Cause : 00800060
[4294714.628000] PrId  : 0001800b
[4294714.628000]
[4294714.628000] Index:  4 pgmask=4kb va=c0094000 asid=b2
[4294714.628000]                        [pa=003d4000 c=3 d=1 v=1 g=1]
[4294714.628000]                        [pa=003d5000 c=3 d=1 v=1 g=1]
[4294714.628000]
[4294714.628000] Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.
[4294714.628000]

Panic occures when I work with big memory arrays (about 100Kb), when load firmware in the insmod-ed
driver. BUT! if driver is staticaly compiled in kernel all is ok.
I have try loading with two variants:
1. request_firmware call (it allocates memory with valloc)
2. ioctl call (it allocate memory with kalloc).
The same thing (if driver insmoded - panic, if staticaly compiled -
all work correctly).

IMHO this is Memory Management subsystem problem, because ported
driver is work fine on x86 processor and can not be reason of the
problem. But I can be wrong!
Where can I look and whom ask the questions??

I woud be thankful for any help.
Thanks anyway, Artem

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, art wrote:

> Hello linux community!
> Please CC to me personaly, because I'am not subsribed yet.
> 
> I am relatively new Linux kernel (just work with some network adapter
> drivers) and my english is poor, so excuse me if something wrong.
> I work with Infineon ADM5120 chip (MIPS32 processor, little endian,
> using tlb), setting embedded Linux on it.
> Almost everything is work now (I port some drivers). But there is big
> problem ( sg16lan is my network driver module):
> 
> [4294714.623000] sg16lan.c: shdsl_interrupt wake up dsl0
> [4294714.627000] sg16lan.c: shdsl_interrupt wake up dsl0
> [4294714.628000] Got mcheck at c0096604
> [4294714.628000] Cpu 0
> [4294714.628000] $ 0   : 00000000 10008400 00000000 00000000
> [4294714.628000] $ 4   : c0098384 c00aa26c 00000001 00000001
> [4294714.628000] $ 8   : 81261fe0 00008400 00000000 80318000
> [4294714.628000] $12   : 0000000c 00 00000078 00000000
> [4294714.628000] $16   : 000000d1 0000aea8 8038a220 8038a000
> [4294714.628000] $20   : 80323420 80030000 00000002 8038a220
> [4294714.628000] $24   : 00000000 802c0000
> [4294714.628000] $28   : 81260000 81261e60 00003fde c0096604
> [4294714.628000] Hi    : 00000280
> [4294714.628000] Lo    : 00000230
> [4294714.628000] epc   : c0096604 store_download+0x424/0x4fc [sg16lan]     Not tainted
> [4294714.628000] ra    : c0096604 store_download+0x424/0x4fc [sg16lan]
> [4294714.628000] Status: 10208403    KERNEL EXL IE
> [4294714.628000] Cause : 00800060
> [4294714.628000] PrId  : 0001800b
> [4294714.628000]
> [4294714.628000] Index:  4 pgmask=4kb va=c0094000 asid=b2
> [4294714.628000]                        [pa=003d4000 c=3 d=1 v=1 g=1]
> [4294714.628000]                        [pa=003d5000 c=3 d=1 v=1 g=1]
> [4294714.628000]
> [4294714.628000] Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.
> [4294714.628000]
> 
> Panic occures when I work with big memory arrays (about 100Kb), when load firmware in the insmod-ed
> driver. BUT! if driver is staticaly compiled in kernel all is ok.
> I have try loading with two variants:
> 1. request_firmware call (it allocates memory with valloc)
> 2. ioctl call (it allocate memory with kalloc).
> The same thing (if driver insmoded - panic, if staticaly compiled -
> all work correctly).
> 
> IMHO this is Memory Management subsystem problem, because ported
> driver is work fine on x86 processor and can not be reason of the
> problem. But I can be wrong!
> Where can I look and whom ask the questions??
> 
> I woud be thankful for any help.
> Thanks anyway, Artem

 The "linux-mips" mailing list (as cc-ed in this response) is a better 
place to ask such questions.

 You haven't included some important information, such as the version 
number of your Linux kernel and where you got your sources from.  Without 
that bit all I can say is there is something wrong with handling of the 
TLB.

 Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* 
registers in show_regs() if the cause is a machine check?  I think it 
would be useful and these registers shouldn't have been corrupted since 
the triggering tlbw* instruction.  A call to show_code() could be useful 
too, to determine which kind of TLB exception has been taken originally.  

  Maciej

Re: Problem with TLB mcheck!

[Ralf Baechle]

On Wed, May 24, 2006 at 01:45:26PM +0100, Maciej W. Rozycki wrote:

j  The "linux-mips" mailing list (as cc-ed in this response) is a better 
> place to ask such questions.
> 
>  You haven't included some important information, such as the version 
> number of your Linux kernel and where you got your sources from.  Without 
> that bit all I can say is there is something wrong with handling of the 
> TLB.
> 
>  Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* 
> registers in show_regs() if the cause is a machine check?  I think it 
> would be useful and these registers shouldn't have been corrupted since 
> the triggering tlbw* instruction.  A call to show_code() could be useful 
> too, to determine which kind of TLB exception has been taken originally.  

Depends on when exactly a CPU will raise the machine check.  On some cores
the information in registers is totally useless if not even missloading.
But generally a good idea, patch below.

  Ralf

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 35cb08d..44a30e6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -819,8 +819,19 @@ asmlinkage void do_watch(struct pt_regs 
 
 asmlinkage void do_mcheck(struct pt_regs *regs)
 {
+	const int field = 2 * sizeof(unsigned long);
+
 	show_regs(regs);
+
+	printk("Hi    : %0*lx\n", field, regs->hi);
+	printk("Pagemask: %0*x\n", read_c0_pagemask());
+	printk("EntryHi : %0*lx\n", field, read_c0_entryhi());
+	printk("EntryLo0: %0*lx\n", field, read_c0_entrylo0());
+	printk("EntryLo1: %0*lx\n", field, read_c0_entrylo1());
+	printk("\n");
 	dump_tlb_all();
+	show_code((unsigned int *) regs->cp0_epc);
+
 	/*
 	 * Some chips may have other causes of machine check (e.g. SB1
 	 * graduation timer)

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, Ralf Baechle wrote:

> Depends on when exactly a CPU will raise the machine check.  On some cores
> the information in registers is totally useless if not even missloading.

 We have got PRId to filter out these.  Though rev. 2 of the architecture 
limits conditions when to raise the exception so it may eventually be a 
non-issue.

> But generally a good idea, patch below.

 Except Index would be a bit more useful than HI. ;-)

  Maciej

Re: Problem with TLB mcheck!

[Ralf Baechle]

On Wed, May 24, 2006 at 04:11:12PM +0100, Maciej W. Rozycki wrote:

> > Depends on when exactly a CPU will raise the machine check.  On some cores
> > the information in registers is totally useless if not even missloading.
> 
>  We have got PRId to filter out these.  Though rev. 2 of the architecture 
> limits conditions when to raise the exception so it may eventually be a 
> non-issue.

Doesn't really help, the exception is asynchronous by definition, so the
CPU can be far away by the time it's struck be the lightning bolt.
Machine check is just a _bad_ place to be.

> > But generally a good idea, patch below.
> 
>  Except Index would be a bit more useful than HI. ;-)

Index may not matter at all in case of a TLBWR.  But yes, will include
index in the patch.

  Ralf

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, Ralf Baechle wrote:

> >  We have got PRId to filter out these.  Though rev. 2 of the architecture 
> > limits conditions when to raise the exception so it may eventually be a 
> > non-issue.
> 
> Doesn't really help, the exception is asynchronous by definition, so the
> CPU can be far away by the time it's struck be the lightning bolt.
> Machine check is just a _bad_ place to be.

 It does help -- while it is asynchronous indeed, TLB writes are far rarer 
than reads and happen in well defined places and a machine check will 
happen within limited time after such a write attempt, at the very worst.  
With the 4Kc the machine check looks synchronous.

  Maciej

Problem with TLB mcheck!

[art]

Sorry for resending, but I find out that I wasn't subscribed!
I use 2.6.16 kernel from linux-mips.org (from tarrball).
When I execute:
# cat /bin/busybox > box
Got mcheck at 800f9b68
Cpu 0
$ 0   : 00000000 10008400 c00216bc c00316bc
$ 4   : c00116bc 00004000 00000005 00008000
$ 8   : c00356bc c003d6bc c00016bc 00010000
$12   : 0000000f 00008000 00007fff 00007fff
$16   : 00000003 8024da80 00000008 c0000000
$20   : 00000000 803d7000 8118c000 80240000
$24   : 0000000f 00000000
$28   : 81200000 81201b80 00001000 800f9c0c
Hi    : 0000000b
Lo    : 5555555b
epc   : 800f9b68 zlib_deflateInit2_+0x184/0x200     Not tainted
ra    : 800f9c0c zlib_deflateInit_+0x28/0x34
Status: 10208403    KERNEL EXL IE
Cause : 00800060
PrId  : 0001800b
Hi    : 0000000b
Pagemask: ffffffff
EntryHi : c0000096
EntryLo0: 0000a05f
EntryLo1: 0000a09f

Index:  3 pgmask=4kb va=00468000 asid=96
                        [pa=00378000 c=3 d=0 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index:  6 pgmask=4kb va=7fa0a000 asid=96
                        [pa=012f7000 c=3 d=1 v=1 g=0]
                        [pa=01244000 c=3 d=1 v=1 g=0]

Index:  7 pgmask=4kb va=004f0000 asid=96
                        [pa=012f4000 c=3 d=1 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index: 10 pgmask=4kb va=0046e000 asid=96
                        [pa=00310000 c=3 d=0 v=1 g=0]
                        [pa=00311000 c=3 d=0 v=1 g=0]

Index: 11 pgmask=4kb va=0046a000 asid=96
                        [pa=0030c000 c=3 d=0 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index: 15 pgmask=4kb va=0048c000 asid=96
                        [pa=0032e000 c=3 d=0 v=1 g=0]
                        [pa=0032f000 c=3 d=0 v=1 g=0]


Code: 00684021  00055880  ae33001c <ae640038> a272001d  ae740018 
ae6e002c  ae6f004c  ae660050
Kernel panic - not syncing: Caught Machine Check exception - caused
by multiple matching entries in the TLB.

This thig happens, as I noted, when work with relatively big amounts
of memory ( more than 10-60Kb). busybox for example 700Kb


-- 
Best regards,
 art                            mailto:art@sigrand.ru

RE: Problem with TLB mcheck!

[Zhan, Rongkai]

[-- Attachment #1: Type: text/plain, Size: 2994 bytes --]

Hi,

 

Please see: http://www.linux-mips.org/archives/linux-mips/2005-04/msg00026.html

 

Best Regards,

Mark. Zhan

---------------------------------------

Wind River System, Inc. China Beijing Office

     Tel: +86 010 64398185/86/87 Ext. 16

Mobile: +86 139 1097 7353

    Fax: +86 64398189

E-Mail: rongkai.zhan@windriver.com

---------------------------------------

-----Original Message-----
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of art
Sent: Wednesday, May 31, 2006 11:42 AM
To: linux-mips@linux-mips.org
Subject: Problem with TLB mcheck!

 

Sorry for resending, but I find out that I wasn't subscribed!

I use 2.6.16 kernel from linux-mips.org (from tarrball).

When I execute:

# cat /bin/busybox > box

Got mcheck at 800f9b68

Cpu 0

$ 0   : 00000000 10008400 c00216bc c00316bc

$ 4   : c00116bc 00004000 00000005 00008000

$ 8   : c00356bc c003d6bc c00016bc 00010000

$12   : 0000000f 00008000 00007fff 00007fff

$16   : 00000003 8024da80 00000008 c0000000

$20   : 00000000 803d7000 8118c000 80240000

$24   : 0000000f 00000000

$28   : 81200000 81201b80 00001000 800f9c0c

Hi    : 0000000b

Lo    : 5555555b

epc   : 800f9b68 zlib_deflateInit2_+0x184/0x200     Not tainted

ra    : 800f9c0c zlib_deflateInit_+0x28/0x34

Status: 10208403    KERNEL EXL IE

Cause : 00800060

PrId  : 0001800b

Hi    : 0000000b

Pagemask: ffffffff

EntryHi : c0000096

EntryLo0: 0000a05f

EntryLo1: 0000a09f

 

Index:  3 pgmask=4kb va=00468000 asid=96

                        [pa=00378000 c=3 d=0 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index:  6 pgmask=4kb va=7fa0a000 asid=96

                        [pa=012f7000 c=3 d=1 v=1 g=0]

                        [pa=01244000 c=3 d=1 v=1 g=0]

 

Index:  7 pgmask=4kb va=004f0000 asid=96

                        [pa=012f4000 c=3 d=1 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index: 10 pgmask=4kb va=0046e000 asid=96

                        [pa=00310000 c=3 d=0 v=1 g=0]

                        [pa=00311000 c=3 d=0 v=1 g=0]

 

Index: 11 pgmask=4kb va=0046a000 asid=96

                        [pa=0030c000 c=3 d=0 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index: 15 pgmask=4kb va=0048c000 asid=96

                        [pa=0032e000 c=3 d=0 v=1 g=0]

                        [pa=0032f000 c=3 d=0 v=1 g=0]

 

 

Code: 00684021  00055880  ae33001c <ae640038> a272001d  ae740018 

ae6e002c  ae6f004c  ae660050

Kernel panic - not syncing: Caught Machine Check exception - caused

by multiple matching entries in the TLB.

 

This thig happens, as I noted, when work with relatively big amounts

of memory ( more than 10-60Kb). busybox for example 700Kb

 

 

-- 

Best regards,

 art                            mailto:art@sigrand.ru

 

 

 


[-- Attachment #2: Type: text/html, Size: 22392 bytes --]

RE: Problem with TLB mcheck!

[Zhan, Rongkai]

[-- Attachment #1: Type: text/plain, Size: 2994 bytes --]

Hi,

 

Please see: http://www.linux-mips.org/archives/linux-mips/2005-04/msg00026.html

 

Best Regards,

Mark. Zhan

---------------------------------------

Wind River System, Inc. China Beijing Office

     Tel: +86 010 64398185/86/87 Ext. 16

Mobile: +86 139 1097 7353

    Fax: +86 64398189

E-Mail: rongkai.zhan@windriver.com

---------------------------------------

-----Original Message-----
From: linux-mips-bounce@linux-mips.org [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of art
Sent: Wednesday, May 31, 2006 11:42 AM
To: linux-mips@linux-mips.org
Subject: Problem with TLB mcheck!

 

Sorry for resending, but I find out that I wasn't subscribed!

I use 2.6.16 kernel from linux-mips.org (from tarrball).

When I execute:

# cat /bin/busybox > box

Got mcheck at 800f9b68

Cpu 0

$ 0   : 00000000 10008400 c00216bc c00316bc

$ 4   : c00116bc 00004000 00000005 00008000

$ 8   : c00356bc c003d6bc c00016bc 00010000

$12   : 0000000f 00008000 00007fff 00007fff

$16   : 00000003 8024da80 00000008 c0000000

$20   : 00000000 803d7000 8118c000 80240000

$24   : 0000000f 00000000

$28   : 81200000 81201b80 00001000 800f9c0c

Hi    : 0000000b

Lo    : 5555555b

epc   : 800f9b68 zlib_deflateInit2_+0x184/0x200     Not tainted

ra    : 800f9c0c zlib_deflateInit_+0x28/0x34

Status: 10208403    KERNEL EXL IE

Cause : 00800060

PrId  : 0001800b

Hi    : 0000000b

Pagemask: ffffffff

EntryHi : c0000096

EntryLo0: 0000a05f

EntryLo1: 0000a09f

 

Index:  3 pgmask=4kb va=00468000 asid=96

                        [pa=00378000 c=3 d=0 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index:  6 pgmask=4kb va=7fa0a000 asid=96

                        [pa=012f7000 c=3 d=1 v=1 g=0]

                        [pa=01244000 c=3 d=1 v=1 g=0]

 

Index:  7 pgmask=4kb va=004f0000 asid=96

                        [pa=012f4000 c=3 d=1 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index: 10 pgmask=4kb va=0046e000 asid=96

                        [pa=00310000 c=3 d=0 v=1 g=0]

                        [pa=00311000 c=3 d=0 v=1 g=0]

 

Index: 11 pgmask=4kb va=0046a000 asid=96

                        [pa=0030c000 c=3 d=0 v=1 g=0]

                        [pa=00000000 c=0 d=0 v=0 g=0]

 

Index: 15 pgmask=4kb va=0048c000 asid=96

                        [pa=0032e000 c=3 d=0 v=1 g=0]

                        [pa=0032f000 c=3 d=0 v=1 g=0]

 

 

Code: 00684021  00055880  ae33001c <ae640038> a272001d  ae740018 

ae6e002c  ae6f004c  ae660050

Kernel panic - not syncing: Caught Machine Check exception - caused

by multiple matching entries in the TLB.

 

This thig happens, as I noted, when work with relatively big amounts

of memory ( more than 10-60Kb). busybox for example 700Kb

 

 

-- 

Best regards,

 art                            mailto:art@sigrand.ru

 

 

 


[-- Attachment #2: Type: text/html, Size: 22392 bytes --]

Re: Problem with TLB mcheck!

[art]

Hello , All

I think I'm on wright way:
I change /arch/mips/mm/tlbex.c:

--- /home/artpol/router/buildroot/kernels/linux-2.6.16-old/arch/mips/mm/tlbex.c 2006-03-20 11:35:39.000000000 +0000
+++ tlbex.c     2006-05-31 16:57:58.000000000 +0000
@@ -742,7 +742,7 @@
        }
 #endif

-       memcpy((void *)CAC_BASE, tlb_handler, 0x80);
+       memcpy((void *)ebase, tlb_handler, 0x80);
 }

 /*
@@ -838,6 +838,7 @@
                break;

        case CPU_R4300:
+       case CPU_4KC:
        case CPU_5KC:
        case CPU_TX49XX:
        case CPU_AU1000:
@@ -852,13 +853,12 @@

        case CPU_R10000:
        case CPU_R12000:
-       case CPU_4KC:
        case CPU_SB1:
        case CPU_SB1A:
        case CPU_4KSC:
        case CPU_20KC:
        case CPU_25KF:
-               tlbw(p);
+               tlbw(p);
                break;

        case CPU_NEVADA:
@@ -1260,7 +1260,7 @@
        }
 #endif

-       memcpy((void *)CAC_BASE, final_handler, 0x100);
+       memcpy((void *)ebase, final_handler, 0x100);
}

And it seem than no problem now (`cat /bin/busybox > box` work as much
as need!).
I think this is wright way, but not all - I'am not guru in memory
subsystem and can miss something! So wait for your advices!



-- 
Best regards,
 art                          mailto:art@sigrand.ru

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 31 May 2006, art wrote:

> @@ -1260,7 +1260,7 @@
>         }
>  #endif
> 
> -       memcpy((void *)CAC_BASE, final_handler, 0x100);
> +       memcpy((void *)ebase, final_handler, 0x100);
> }
> 
> And it seem than no problem now (`cat /bin/busybox > box` work as much
> as need!).
> I think this is wright way, but not all - I'am not guru in memory
> subsystem and can miss something! So wait for your advices!

 That change has been in the tree since Apr 16th -- there is a reason you 
should always try the latest revision before reporting a problem.

  Maciej