Problem with TLB mcheck!

Problem with TLB mcheck!

[art]

Hello linux community!
Please CC to me personaly, because I'am not subsribed yet.

I am relatively new Linux kernel (just work with some network adapter
drivers) and my english is poor, so excuse me if something wrong.
I work with Infineon ADM5120 chip (MIPS32 processor, little endian,
using tlb), setting embedded Linux on it.
Almost everything is work now (I port some drivers). But there is big
problem ( sg16lan is my network driver module):

[4294714.623000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.627000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.628000] Got mcheck at c0096604
[4294714.628000] Cpu 0
[4294714.628000] $ 0   : 00000000 10008400 00000000 00000000
[4294714.628000] $ 4   : c0098384 c00aa26c 00000001 00000001
[4294714.628000] $ 8   : 81261fe0 00008400 00000000 80318000
[4294714.628000] $12   : 0000000c 00 00000078 00000000
[4294714.628000] $16   : 000000d1 0000aea8 8038a220 8038a000
[4294714.628000] $20   : 80323420 80030000 00000002 8038a220
[4294714.628000] $24   : 00000000 802c0000
[4294714.628000] $28   : 81260000 81261e60 00003fde c0096604
[4294714.628000] Hi    : 00000280
[4294714.628000] Lo    : 00000230
[4294714.628000] epc   : c0096604 store_download+0x424/0x4fc [sg16lan]     Not tainted
[4294714.628000] ra    : c0096604 store_download+0x424/0x4fc [sg16lan]
[4294714.628000] Status: 10208403    KERNEL EXL IE
[4294714.628000] Cause : 00800060
[4294714.628000] PrId  : 0001800b
[4294714.628000]
[4294714.628000] Index:  4 pgmask=4kb va=c0094000 asid=b2
[4294714.628000]                        [pa=003d4000 c=3 d=1 v=1 g=1]
[4294714.628000]                        [pa=003d5000 c=3 d=1 v=1 g=1]
[4294714.628000]
[4294714.628000] Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.
[4294714.628000]

Panic occures when I work with big memory arrays (about 100Kb), when load firmware in the insmod-ed
driver. BUT! if driver is staticaly compiled in kernel all is ok.
I have try loading with two variants:
1. request_firmware call (it allocates memory with valloc)
2. ioctl call (it allocate memory with kalloc).
The same thing (if driver insmoded - panic, if staticaly compiled -
all work correctly).

IMHO this is Memory Management subsystem problem, because ported
driver is work fine on x86 processor and can not be reason of the
problem. But I can be wrong!
Where can I look and whom ask the questions??

I woud be thankful for any help.
Thanks anyway, Artem

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, art wrote:

> Hello linux community!
> Please CC to me personaly, because I'am not subsribed yet.
> 
> I am relatively new Linux kernel (just work with some network adapter
> drivers) and my english is poor, so excuse me if something wrong.
> I work with Infineon ADM5120 chip (MIPS32 processor, little endian,
> using tlb), setting embedded Linux on it.
> Almost everything is work now (I port some drivers). But there is big
> problem ( sg16lan is my network driver module):
> 
> [4294714.623000] sg16lan.c: shdsl_interrupt wake up dsl0
> [4294714.627000] sg16lan.c: shdsl_interrupt wake up dsl0
> [4294714.628000] Got mcheck at c0096604
> [4294714.628000] Cpu 0
> [4294714.628000] $ 0   : 00000000 10008400 00000000 00000000
> [4294714.628000] $ 4   : c0098384 c00aa26c 00000001 00000001
> [4294714.628000] $ 8   : 81261fe0 00008400 00000000 80318000
> [4294714.628000] $12   : 0000000c 00 00000078 00000000
> [4294714.628000] $16   : 000000d1 0000aea8 8038a220 8038a000
> [4294714.628000] $20   : 80323420 80030000 00000002 8038a220
> [4294714.628000] $24   : 00000000 802c0000
> [4294714.628000] $28   : 81260000 81261e60 00003fde c0096604
> [4294714.628000] Hi    : 00000280
> [4294714.628000] Lo    : 00000230
> [4294714.628000] epc   : c0096604 store_download+0x424/0x4fc [sg16lan]     Not tainted
> [4294714.628000] ra    : c0096604 store_download+0x424/0x4fc [sg16lan]
> [4294714.628000] Status: 10208403    KERNEL EXL IE
> [4294714.628000] Cause : 00800060
> [4294714.628000] PrId  : 0001800b
> [4294714.628000]
> [4294714.628000] Index:  4 pgmask=4kb va=c0094000 asid=b2
> [4294714.628000]                        [pa=003d4000 c=3 d=1 v=1 g=1]
> [4294714.628000]                        [pa=003d5000 c=3 d=1 v=1 g=1]
> [4294714.628000]
> [4294714.628000] Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.
> [4294714.628000]
> 
> Panic occures when I work with big memory arrays (about 100Kb), when load firmware in the insmod-ed
> driver. BUT! if driver is staticaly compiled in kernel all is ok.
> I have try loading with two variants:
> 1. request_firmware call (it allocates memory with valloc)
> 2. ioctl call (it allocate memory with kalloc).
> The same thing (if driver insmoded - panic, if staticaly compiled -
> all work correctly).
> 
> IMHO this is Memory Management subsystem problem, because ported
> driver is work fine on x86 processor and can not be reason of the
> problem. But I can be wrong!
> Where can I look and whom ask the questions??
> 
> I woud be thankful for any help.
> Thanks anyway, Artem

 The "linux-mips" mailing list (as cc-ed in this response) is a better 
place to ask such questions.

 You haven't included some important information, such as the version 
number of your Linux kernel and where you got your sources from.  Without 
that bit all I can say is there is something wrong with handling of the 
TLB.

 Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* 
registers in show_regs() if the cause is a machine check?  I think it 
would be useful and these registers shouldn't have been corrupted since 
the triggering tlbw* instruction.  A call to show_code() could be useful 
too, to determine which kind of TLB exception has been taken originally.  

  Maciej

Re: Problem with TLB mcheck!

[Ralf Baechle]

On Wed, May 24, 2006 at 01:45:26PM +0100, Maciej W. Rozycki wrote:

j  The "linux-mips" mailing list (as cc-ed in this response) is a better 
> place to ask such questions.
> 
>  You haven't included some important information, such as the version 
> number of your Linux kernel and where you got your sources from.  Without 
> that bit all I can say is there is something wrong with handling of the 
> TLB.
> 
>  Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* 
> registers in show_regs() if the cause is a machine check?  I think it 
> would be useful and these registers shouldn't have been corrupted since 
> the triggering tlbw* instruction.  A call to show_code() could be useful 
> too, to determine which kind of TLB exception has been taken originally.  

Depends on when exactly a CPU will raise the machine check.  On some cores
the information in registers is totally useless if not even missloading.
But generally a good idea, patch below.

  Ralf

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 35cb08d..44a30e6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -819,8 +819,19 @@ asmlinkage void do_watch(struct pt_regs 
 
 asmlinkage void do_mcheck(struct pt_regs *regs)
 {
+	const int field = 2 * sizeof(unsigned long);
+
 	show_regs(regs);
+
+	printk("Hi    : %0*lx\n", field, regs->hi);
+	printk("Pagemask: %0*x\n", read_c0_pagemask());
+	printk("EntryHi : %0*lx\n", field, read_c0_entryhi());
+	printk("EntryLo0: %0*lx\n", field, read_c0_entrylo0());
+	printk("EntryLo1: %0*lx\n", field, read_c0_entrylo1());
+	printk("\n");
 	dump_tlb_all();
+	show_code((unsigned int *) regs->cp0_epc);
+
 	/*
 	 * Some chips may have other causes of machine check (e.g. SB1
 	 * graduation timer)

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, Ralf Baechle wrote:

> Depends on when exactly a CPU will raise the machine check.  On some cores
> the information in registers is totally useless if not even missloading.

 We have got PRId to filter out these.  Though rev. 2 of the architecture 
limits conditions when to raise the exception so it may eventually be a 
non-issue.

> But generally a good idea, patch below.

 Except Index would be a bit more useful than HI. ;-)

  Maciej

Re: Problem with TLB mcheck!

[Ralf Baechle]

On Wed, May 24, 2006 at 04:11:12PM +0100, Maciej W. Rozycki wrote:

> > Depends on when exactly a CPU will raise the machine check.  On some cores
> > the information in registers is totally useless if not even missloading.
> 
>  We have got PRId to filter out these.  Though rev. 2 of the architecture 
> limits conditions when to raise the exception so it may eventually be a 
> non-issue.

Doesn't really help, the exception is asynchronous by definition, so the
CPU can be far away by the time it's struck be the lightning bolt.
Machine check is just a _bad_ place to be.

> > But generally a good idea, patch below.
> 
>  Except Index would be a bit more useful than HI. ;-)

Index may not matter at all in case of a TLBWR.  But yes, will include
index in the patch.

  Ralf

Re: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 24 May 2006, Ralf Baechle wrote:

> >  We have got PRId to filter out these.  Though rev. 2 of the architecture 
> > limits conditions when to raise the exception so it may eventually be a 
> > non-issue.
> 
> Doesn't really help, the exception is asynchronous by definition, so the
> CPU can be far away by the time it's struck be the lightning bolt.
> Machine check is just a _bad_ place to be.

 It does help -- while it is asynchronous indeed, TLB writes are far rarer 
than reads and happen in well defined places and a machine check will 
happen within limited time after such a write attempt, at the very worst.  
With the 4Kc the machine check looks synchronous.

  Maciej

Re[2]: Problem with TLB mcheck!

[art]

Hello Maciej, thanks for answer! when it is at first time it is wery
important :)!

Wednesday, May 24, 2006, 7:45:26 PM, you wrote:

MWR>  The "linux-mips" mailing list (as cc-ed in this response) is a better
MWR> place to ask such questions.

MWR>  You haven't included some important information, such as the version 
MWR> number of your Linux kernel and where you got your sources from.  Without 
MWR> that bit all I can say is there is something wrong with handling of the 
MWR> TLB.

MWR>  Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* 
MWR> registers in show_regs() if the cause is a machine check?  I think it 
MWR> would be useful and these registers shouldn't have been corrupted since 
MWR> the triggering tlbw* instruction.  A call to show_code() could be useful 
MWR> too, to determine which kind of TLB exception has been taken originally.  

MWR>   Maciej

I know about linux-mips, but I have not get answer from that for two
days! Maybe I write not complete message.
I will try Ralf Baechles diff and send the result a bit later.
Information about kernel sources (how can I forgot this!!):
   Version: linux-2.6.12-rc1-mipscvs-20050403 (this is tar-s full
   name). So I think this kernel is from linux-mips cvs repositary.
   It was downloaded from http://www.student.tue.nl/Q/t.f.a.wilms/adm5120/

Here is mcheck with respect to your patch:
   
[4294701.476000] Got mcheck at 8011c818
[4294701.476000] Hi    : 00000000
[4294701.476000] Pagemask: ffffffff
[4294701.476000] EntryHi : c008a013
[4294701.476000] EntryLo0: 0000f25f
[4294701.476000] EntryLo1: 0000ff9f
[4294701.476000]
[4294701.476000] Cpu 0
[4294701.476000] $ 0   : 00000000 10008400 00000000 802b17ac
[4294701.476000] $ 4   : 802b13ac 00000400 c008ac00 80353d6c
[4294701.476000] $ 8   : 10008400 1000001f 80393000 fffffff8
[4294701.476000] $12   : 802b0000 802b0000 00000001 ffffffff
[4294701.476000] $16   : 00000400 802b17ab 80353d6c 00000020
[4294701.476000] $20   : 80353e38 81144800 00000400 802b13ac
[4294701.476000] $24   : 00000006 0048ce30
[4294701.476000] $28   : 80352000 80353c70 7faf5d98 8011ce94
[4294701.476000] Hi    : 00000000
[4294701.476000] Lo    : 000c3500
[4294701.476000] epc   : 8011c818 vsnprintf+0x54/0x6bc     Not tainted
[4294701.476000] ra    : 8011ce94 vscnprintf+0x14/0x30
[4294701.476000] Status: 10208402    KERNEL EXL
[4294701.476000] Cause : 00800060
[4294701.476000] PrId  : 0001800b
[4294701.476000]
[4294701.476000] Index:  0 pgmask=4kb va=0050a000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=00379000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  1 pgmask=4kb va=7faf4000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=0031b000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  2 pgmask=4kb va=0048a000 asid=13
[4294701.476000]                        [pa=011bc000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011bd000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  5 pgmask=4kb va=00492000 asid=13
[4294701.476000]                        [pa=011c4000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011c5000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  6 pgmask=4kb va=004ee000 asid=13
[4294701.476000]                        [pa=0028a000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011f7000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  7 pgmask=4kb va=0048c000 asid=13
[4294701.476000]                        [pa=011be000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011bf000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  8 pgmask=4kb va=004f0000 asid=13
[4294701.476000]                        [pa=00322000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000] Index:  9 pgmask=4kb va=00480000 asid=13
[4294701.476000]                        [pa=011b0000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011b1000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 10 pgmask=4kb va=00488000 asid=13
[4294701.476000]                        [pa=011b8000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011b9000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 11 pgmask=4kb va=7faf6000 asid=13
[4294701.476000]                        [pa=00357000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000] Index: 12 pgmask=4kb va=00478000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=011a9000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 13 pgmask=4kb va=c008c000 asid=13
[4294701.476000]                        [pa=003ff000 c=3 d=1 v=1 g=1]
[4294701.476000]                        [pa=01200000 c=3 d=1 v=1 g=1]
[4294701.476000]
[4294701.476000] Index: 14 pgmask=4kb va=004aa000 asid=13
[4294701.476000]                        [pa=00286000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=00287000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 15 pgmask=4kb va=00508000 asid=13
[4294701.476000]                        [pa=0031c000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000]
[4294701.476000] Code: 0222102b  14400034  00000000 <80c60000> 14c00014  02e08021  0230102b  10400021  00001821
[4294701.476000] Caught Machine Check exception - caused by multiple matching entries in the TLB


-- 
Best regards,
 art                            mailto:art@sigrand.ru

Re[2]: Problem with TLB mcheck!

[art]

Hello Maciej,

Friday, May 26, 2006, 7:37:19 PM, you wrote:

MWR> On Fri, 26 May 2006, art wrote:

>> Hello Maciej, thanks for answer! when it is at first time it is wery
>> important :)!

MWR>  No problem.

>> Information about kernel sources (how can I forgot this!!):
>>    Version: linux-2.6.12-rc1-mipscvs-20050403 (this is tar-s full
>>    name). So I think this kernel is from linux-mips cvs repositary.
>>    It was downloaded from http://www.student.tue.nl/Q/t.f.a.wilms/adm5120/

MWR>  Please retry with the head of the GIT tree at: "http://linux-mips.org/" 
MWR> -- it's more than a year worth of fixes!  The Ralf's patch has already 
MWR> been included there.

MWR>   Maciej

I have try 2.6.16 kernel from linux-mips.org (from tarrball). You recomend newest kernel, but I get stable,
thinking it would be more robust.But have same situation :(. When i execute:
# cat /bin/busybox > box
Got mcheck at 800f9b68
Cpu 0
$ 0   : 00000000 10008400 c00216bc c00316bc
$ 4   : c00116bc 00004000 00000005 00008000
$ 8   : c00356bc c003d6bc c00016bc 00010000
$12   : 0000000f 00008000 00007fff 00007fff
$16   : 00000003 8024da80 00000008 c0000000
$20   : 00000000 803d7000 8118c000 80240000
$24   : 0000000f 00000000
$28   : 81200000 81201b80 00001000 800f9c0c
Hi    : 0000000b
Lo    : 5555555b
epc   : 800f9b68 zlib_deflateInit2_+0x184/0x200     Not tainted
ra    : 800f9c0c zlib_deflateInit_+0x28/0x34
Status: 10208403    KERNEL EXL IE
Cause : 00800060
PrId  : 0001800b
Hi    : 0000000b
Pagemask: ffffffff
EntryHi : c0000096
EntryLo0: 0000a05f
EntryLo1: 0000a09f

Index:  3 pgmask=4kb va=00468000 asid=96
                        [pa=00378000 c=3 d=0 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index:  6 pgmask=4kb va=7fa0a000 asid=96
                        [pa=012f7000 c=3 d=1 v=1 g=0]
                        [pa=01244000 c=3 d=1 v=1 g=0]

Index:  7 pgmask=4kb va=004f0000 asid=96
                        [pa=012f4000 c=3 d=1 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index: 10 pgmask=4kb va=0046e000 asid=96
                        [pa=00310000 c=3 d=0 v=1 g=0]
                        [pa=00311000 c=3 d=0 v=1 g=0]

Index: 11 pgmask=4kb va=0046a000 asid=96
                        [pa=0030c000 c=3 d=0 v=1 g=0]
                        [pa=00000000 c=0 d=0 v=0 g=0]

Index: 15 pgmask=4kb va=0048c000 asid=96
                        [pa=0032e000 c=3 d=0 v=1 g=0]
                        [pa=0032f000 c=3 d=0 v=1 g=0]


Code: 00684021  00055880  ae33001c <ae640038> a272001d  ae740018  ae6e002c  ae6f004c  ae660050
Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.


-- 
Best regards,
 art                            mailto:art@sigrand.ru

Re[2]: Problem with TLB mcheck!

[art]

Hello Mark!

Wednesday, May 31, 2006, 11:18:26 AM, you wrote:

ZR> Hi,

 

ZR> Please see: http://www.linux-mips.org/archives/linux-mips/2005-04/msg00026.html

 

ZR> Best Regards,

ZR> Mark. Zhan

Thanks for advice! But problem steel present :(. BUT! there is
progress! After pathing I execute cut /bin/busybox about 10-12 times
before system crash, and before patching it occurs after 1-2 times!

Re[2]: Problem with TLB mcheck!

[Maciej W. Rozycki]

On Wed, 31 May 2006, art wrote:

> ZR> Please see: http://www.linux-mips.org/archives/linux-mips/2005-04/msg00026.html
> 
>  
> 
> ZR> Best Regards,
> 
> ZR> Mark. Zhan
> 
> Thanks for advice! But problem steel present :(. BUT! there is
> progress! After pathing I execute cut /bin/busybox about 10-12 times
> before system crash, and before patching it occurs after 1-2 times!

 The patch is supposed to be irrelevant for the revision of the 4Kc you 
have.  Anyway, as I wrote, please retry with the head of the tree (which 
is 2.6.17-rc5) and then I can think about it.  Whether it's overall stable 
or not is irrelevant -- once the issue is resolved you can either wait for 
a release that is considered stable enough or retrofit the fix into a 
release of your choice.

  Maciej