Malta crashes on the latest 2.4 kernel

Malta crashes on the latest 2.4 kernel

[Jun Sun]

See the crash scene.  Anybody knows the cause?  It is strange to see the 
reserved exception.

Jun


FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
pcnet32.c:v1.27a 10.02.2002 tsbogend@alpha.franken.de
pcnet32: PCnet/FAST III 79C973 at 0x1200, 00 d0 a0 00 01 e7 assigned IRQ 10.
eth0: registered as PCnet/FAST III 79C973
pcnet32: 1 cards_found.
SCSI subsystem driver Revision: 1.00
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 4096)
Sending BOOTP requests . OK
IP-Config: Got BOOTP answer from 10.0.0.75, my address is 10.0.18.6
IP-Config: Complete:
       device=eth0, addr=10.0.18.6, mask=255.255.0.0, gw=255.255.255.255,
      host=10.0.18.6, domain=, nis-domain=(none),
      bootserver=10.0.0.75, rootserver=10.0.0.75, rootpath=/opt/mvl-installs/mvl2
.1/hardhat/devkit/mips/fp_le/target
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Looking up port of RPC 100003/2 on 10.0.0.75
Looking up port of RPC 100005/1 on 10.0.0.75
VFS: Mounted root (nfs filesystem).
Freeing prom memory: 956kb freed
Freeing unused kernel memory: 84k freed
Algorithmics/MIPS FPU Emulator v1.5
INIT: version 2.78 booting
$0 : 00000000 80000000 80002000 00000006 00000006 0040c000 0040c000 00000001
$8 : 80000000 00000034 8004bde8 8004bbf0 8004bde8 00000080 8004baf8 00000001
$16: 00800000 800071c0 1000fc01 8004c008 0000b000 8004c008 00800000 0040b000
$24: 00000000 00000080                   8004a000 8004ba78 0000000b 8011db6c
Hi : ffffe4f4
Lo : 00000904
epc  : 8010d528    Not tainted
Status: 1020fc02
Cause : 00800060
Kernel panic: Caught reserved exception - should not happen.


==========================================================ffffffff8010d490: 
     92240080        lbu     $a0,128($s1)
ffffffff8010d494:       3c088000        lui     $t0,0x8000
ffffffff8010d498:       00a41025        or      $v0,$a1,$a0
ffffffff8010d49c:       40825000        mtc0    $v0,$10
ffffffff8010d4a0:       24a52000        addiu   $a1,$a1,8192
         ...
ffffffff8010d4bc:       42000008        tlbp
         ...
ffffffff8010d4d8:       40070000        mfc0    $a3,$0
ffffffff8010d4dc:       00000000        nop
ffffffff8010d4e0:       00e01021        move    $v0,$a3
ffffffff8010d4e4:       40801000        mtc0    $zero,$2
ffffffff8010d4e8:       00000000        nop
ffffffff8010d4ec:       40801800        mtc0    $zero,$3
ffffffff8010d4f0:       00000000        nop
ffffffff8010d4f4:       40885000        mtc0    $t0,$10
ffffffff8010d4f8:       04400013        bltz    $v0,ffffffff8010d548 <local_flus
h_tlb_range+0x150>
ffffffff8010d4fc:       00a6102b        sltu    $v0,$a1,$a2
         ...
ffffffff8010d518:       00071340        sll     $v0,$a3,0xd
ffffffff8010d51c:       3c018000        lui     $at,0x8000
ffffffff8010d520:       00221021        addu    $v0,$at,$v0
ffffffff8010d524:       40825000        mtc0    $v0,$10
ffffffff8010d528:       42000002        tlbwi
         ...
ffffffff8010d544:       00a6102b        sltu    $v0,$a1,$a2
ffffffff8010d548:       1440ffd4        bnez    $v0,ffffffff8010d49c <local_flus
h_tlb_range+0xa4>
ffffffff8010d54c:       00a41025        or      $v0,$a1,$a0
ffffffff8010d550:       40835000        mtc0    $v1,$10
ffffffff8010d554:       08043569        j       ffffffff8010d5a4 <local_flush_tl
b_range+0x1ac>
ffffffff8010d558:       00000000        nop
ffffffff8010d55c:       3c108025        lui     $s0,0x8025
ffffffff8010d560:       8e105910        lw      $s0,22800($s0)
ffffffff8010d564:       26100001        addiu   $s0,$s0,1
ffffffff8010d568:       320200ff        andi    $v0,$s0,0xff
ffffffff8010d56c:       14400005        bnez    $v0,ffffffff8010d584 <local_flus
h_tlb_range+0x18c>
ffffffff8010d570:       00000000        nop

Re: Malta crashes on the latest 2.4 kernel

[H. J. Lu]

On Wed, Jul 10, 2002 at 04:12:51PM -0700, Jun Sun wrote:
> 
> See the crash scene.  Anybody knows the cause?  It is strange to see the 
> reserved exception.
> 

The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.


H.J.

Re: Malta crashes on the latest 2.4 kernel

[Jun Sun]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

H. J. Lu wrote:

> On Wed, Jul 10, 2002 at 04:12:51PM -0700, Jun Sun wrote:
> 
>>See the crash scene.  Anybody knows the cause?  It is strange to see the 
>>reserved exception.
>>
>>
> 
> The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.
> 


Ralf,

Apparently this is a well-known problem.  So *please* apply this patch or give 
  some constructive advice if the patch cannot be applied as it is.

Jun


[-- Attachment #2: junk --]
[-- Type: text/plain, Size: 341 bytes --]

diff -Nru arch/mips/mm/tlb-r4k.c.orig arch/mips/mm/tlb-r4k.c
--- arch/mips/mm/tlb-r4k.c.orig	Thu Jun 27 14:12:40 2002
+++ arch/mips/mm/tlb-r4k.c	Wed Jul 10 18:06:34 2002
@@ -125,6 +125,7 @@
 				BARRIER;
 				/* Make sure all entries differ. */
 				set_entryhi(KSEG0+idx*0x2000);
+				BARRIER;
 				tlb_write_indexed();
 				BARRIER;
 			}

Re: Malta crashes on the latest 2.4 kernel

[Ralf Baechle]

On Wed, Jul 10, 2002 at 04:49:00PM -0700, H. J. Lu wrote:

> > See the crash scene.  Anybody knows the cause?  It is strange to see the 
> > reserved exception.
> > 
> 
> The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.

Jun's patch certainly is correct for some MIPS32 CPUs; others may get
away without this one.  Previous experience with pipeline hazards on MIPS
processors has shown that at times hazards may change even between minor
revisions of a CPU; documentation isn't always trustworthy about such
minor details of the pipeline.

  Ralf

Re: Malta crashes on the latest 2.4 kernel

[Kevin D. Kissell]

> On Wed, Jul 10, 2002 at 04:49:00PM -0700, H. J. Lu wrote:
> 
> > > See the crash scene.  Anybody knows the cause?  It is strange to see the 
> > > reserved exception.
> > > 
> > 
> > The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.
> 
> Jun's patch certainly is correct for some MIPS32 CPUs; others may get
> away without this one.  Previous experience with pipeline hazards on MIPS
> processors has shown that at times hazards may change even between minor
> revisions of a CPU; documentation isn't always trustworthy about such
> minor details of the pipeline.

Excuse me, but I've seen this statement used by others in
the past as an excuse for doing something silly or not doing
something reasonable, and it generally hasn't washed.
In what specific cases have the CP0 pipeline hazards 
changed between minor revisions of any production
MIPS CPU?  The *documentation* may have been
corrected, but these hazards are fairly fundamental
artifacts of the pipeline microarchitecture of a given
processor.

The CP0 hazard between a write of EntryHi
and a subsequent TLBWI instruction is flagged
in the MIPS32 spec and noted as being "typically" 
2 cycles.  I'm not going to spend the time going
through my full set of users manuals, but a representative
sampling shows this hazard as being specified for
every R4xxx and R5xxx CPU I checked.  The fact
that a given CPU *may* get away with it is no
excuse for not protecting common code.

I note that Ralf has, in fact, applied the fix to the
OSS CVS repository.  I also note that "BARRIER"
is still defined to be a string of 6 nops.  I would argue
(again) that those really, really ought to be ssnops,
and that if they *were* ssnops, one could probably
have fewer of them.

            Regards,

            Kevin K.

Re: Malta crashes on the latest 2.4 kernel

[Kevin D. Kissell]

> On Wed, Jul 10, 2002 at 04:49:00PM -0700, H. J. Lu wrote:
> 
> > > See the crash scene.  Anybody knows the cause?  It is strange to see the 
> > > reserved exception.
> > > 
> > 
> > The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.
> 
> Jun's patch certainly is correct for some MIPS32 CPUs; others may get
> away without this one.  Previous experience with pipeline hazards on MIPS
> processors has shown that at times hazards may change even between minor
> revisions of a CPU; documentation isn't always trustworthy about such
> minor details of the pipeline.

Excuse me, but I've seen this statement used by others in
the past as an excuse for doing something silly or not doing
something reasonable, and it generally hasn't washed.
In what specific cases have the CP0 pipeline hazards 
changed between minor revisions of any production
MIPS CPU?  The *documentation* may have been
corrected, but these hazards are fairly fundamental
artifacts of the pipeline microarchitecture of a given
processor.

The CP0 hazard between a write of EntryHi
and a subsequent TLBWI instruction is flagged
in the MIPS32 spec and noted as being "typically" 
2 cycles.  I'm not going to spend the time going
through my full set of users manuals, but a representative
sampling shows this hazard as being specified for
every R4xxx and R5xxx CPU I checked.  The fact
that a given CPU *may* get away with it is no
excuse for not protecting common code.

I note that Ralf has, in fact, applied the fix to the
OSS CVS repository.  I also note that "BARRIER"
is still defined to be a string of 6 nops.  I would argue
(again) that those really, really ought to be ssnops,
and that if they *were* ssnops, one could probably
have fewer of them.

            Regards,

            Kevin K.

Re: Malta crashes on the latest 2.4 kernel

[Geert Uytterhoeven]

On Thu, 11 Jul 2002, Kevin D. Kissell wrote:
> I note that Ralf has, in fact, applied the fix to the
> OSS CVS repository.  I also note that "BARRIER"
> is still defined to be a string of 6 nops.  I would argue
> (again) that those really, really ought to be ssnops,
> and that if they *were* ssnops, one could probably
> have fewer of them.

Sorry for being ignorant, but what's the difference between nop and ssnop?

I see that SSNOP is defined to be `sll zero,zero,1' in <asm/asm.h>, but that
doesn't give me any clue.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Malta crashes on the latest 2.4 kernel

[Carsten Langgaard]

Sounds like a problem I so a couple of months ago, I thought the fix was already in
the CVS.

Here is my previous mail:

There seems to be a hazard problem in the local_flush_tlb_range function
in tlb-r4k.c, which the patch below will fix.
It could hit anyone, but it probably only a problem on CPUs, which
doesn't allow matching entries in the TLB.

/Carsten

Index: arch/mips/mm/tlb-r4k.c
===================================================================
RCS file: /cvs/linux/arch/mips/mm/tlb-r4k.c,v
retrieving revision 1.6.2.3
diff -u -r1.6.2.3 tlb-r4k.c
--- arch/mips/mm/tlb-r4k.c      2002/01/18 03:16:24     1.6.2.3
+++ arch/mips/mm/tlb-r4k.c      2002/05/17 11:36:58
@@ -119,12 +119,11 @@
                                idx = get_index();
                                set_entrylo0(0);
                                set_entrylo1(0);
-                               set_entryhi(KSEG0);
                                if (idx < 0)
                                        continue;
-                               BARRIER;
                                /* Make sure all entries differ. */
                                set_entryhi(KSEG0+idx*0x2000);
+                               BARRIER;
                                tlb_write_indexed();
                                BARRIER;
                        }


Jun Sun wrote:

> See the crash scene.  Anybody knows the cause?  It is strange to see the
> reserved exception.
>
> Jun
>
> FDC 0 is a post-1991 82077
> RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
> pcnet32.c:v1.27a 10.02.2002 tsbogend@alpha.franken.de
> pcnet32: PCnet/FAST III 79C973 at 0x1200, 00 d0 a0 00 01 e7 assigned IRQ 10.
> eth0: registered as PCnet/FAST III 79C973
> pcnet32: 1 cards_found.
> SCSI subsystem driver Revision: 1.00
> NET4: Linux TCP/IP 1.0 for NET4.0
> IP Protocols: ICMP, UDP, TCP
> IP: routing cache hash table of 512 buckets, 4Kbytes
> TCP: Hash tables configured (established 4096 bind 4096)
> Sending BOOTP requests . OK
> IP-Config: Got BOOTP answer from 10.0.0.75, my address is 10.0.18.6
> IP-Config: Complete:
>        device=eth0, addr=10.0.18.6, mask=255.255.0.0, gw=255.255.255.255,
>       host=10.0.18.6, domain=, nis-domain=(none),
>       bootserver=10.0.0.75, rootserver=10.0.0.75, rootpath=/opt/mvl-installs/mvl2
> .1/hardhat/devkit/mips/fp_le/target
> NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
> Looking up port of RPC 100003/2 on 10.0.0.75
> Looking up port of RPC 100005/1 on 10.0.0.75
> VFS: Mounted root (nfs filesystem).
> Freeing prom memory: 956kb freed
> Freeing unused kernel memory: 84k freed
> Algorithmics/MIPS FPU Emulator v1.5
> INIT: version 2.78 booting
> $0 : 00000000 80000000 80002000 00000006 00000006 0040c000 0040c000 00000001
> $8 : 80000000 00000034 8004bde8 8004bbf0 8004bde8 00000080 8004baf8 00000001
> $16: 00800000 800071c0 1000fc01 8004c008 0000b000 8004c008 00800000 0040b000
> $24: 00000000 00000080                   8004a000 8004ba78 0000000b 8011db6c
> Hi : ffffe4f4
> Lo : 00000904
> epc  : 8010d528    Not tainted
> Status: 1020fc02
> Cause : 00800060
> Kernel panic: Caught reserved exception - should not happen.
>
> ==========================================================ffffffff8010d490:
>      92240080        lbu     $a0,128($s1)
> ffffffff8010d494:       3c088000        lui     $t0,0x8000
> ffffffff8010d498:       00a41025        or      $v0,$a1,$a0
> ffffffff8010d49c:       40825000        mtc0    $v0,$10
> ffffffff8010d4a0:       24a52000        addiu   $a1,$a1,8192
>          ...
> ffffffff8010d4bc:       42000008        tlbp
>          ...
> ffffffff8010d4d8:       40070000        mfc0    $a3,$0
> ffffffff8010d4dc:       00000000        nop
> ffffffff8010d4e0:       00e01021        move    $v0,$a3
> ffffffff8010d4e4:       40801000        mtc0    $zero,$2
> ffffffff8010d4e8:       00000000        nop
> ffffffff8010d4ec:       40801800        mtc0    $zero,$3
> ffffffff8010d4f0:       00000000        nop
> ffffffff8010d4f4:       40885000        mtc0    $t0,$10
> ffffffff8010d4f8:       04400013        bltz    $v0,ffffffff8010d548 <local_flus
> h_tlb_range+0x150>
> ffffffff8010d4fc:       00a6102b        sltu    $v0,$a1,$a2
>          ...
> ffffffff8010d518:       00071340        sll     $v0,$a3,0xd
> ffffffff8010d51c:       3c018000        lui     $at,0x8000
> ffffffff8010d520:       00221021        addu    $v0,$at,$v0
> ffffffff8010d524:       40825000        mtc0    $v0,$10
> ffffffff8010d528:       42000002        tlbwi
>          ...
> ffffffff8010d544:       00a6102b        sltu    $v0,$a1,$a2
> ffffffff8010d548:       1440ffd4        bnez    $v0,ffffffff8010d49c <local_flus
> h_tlb_range+0xa4>
> ffffffff8010d54c:       00a41025        or      $v0,$a1,$a0
> ffffffff8010d550:       40835000        mtc0    $v1,$10
> ffffffff8010d554:       08043569        j       ffffffff8010d5a4 <local_flush_tl
> b_range+0x1ac>
> ffffffff8010d558:       00000000        nop
> ffffffff8010d55c:       3c108025        lui     $s0,0x8025
> ffffffff8010d560:       8e105910        lw      $s0,22800($s0)
> ffffffff8010d564:       26100001        addiu   $s0,$s0,1
> ffffffff8010d568:       320200ff        andi    $v0,$s0,0xff
> ffffffff8010d56c:       14400005        bnez    $v0,ffffffff8010d584 <local_flus
> h_tlb_range+0x18c>
> ffffffff8010d570:       00000000        nop

--
_    _ ____  ___   Carsten Langgaard   Mailto:carstenl@mips.com
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com

Re: Malta crashes on the latest 2.4 kernel

[Carsten Langgaard]

Ralf Baechle wrote:

> On Wed, Jul 10, 2002 at 04:49:00PM -0700, H. J. Lu wrote:
>
> > > See the crash scene.  Anybody knows the cause?  It is strange to see the
> > > reserved exception.
> > >
> >
> > The 2.4 kernel checked out around Jul  4 09:28 PDT works fine on Malta.
>
> Jun's patch certainly is correct for some MIPS32 CPUs; others may get
> away without this one.  Previous experience with pipeline hazards on MIPS
> processors has shown that at times hazards may change even between minor
> revisions of a CPU; documentation isn't always trustworthy about such
> minor details of the pipeline.

I actually discovered the hazard problem on a 4Kc, based on some older RTL, but
the hazard some how disappear on a newer revision.
So you are absolutely right.

>
>   Ralf

--
_    _ ____  ___   Carsten Langgaard   Mailto:carstenl@mips.com
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com

Re: Malta crashes on the latest 2.4 kernel

[Kevin D. Kissell]

> On Thu, 11 Jul 2002, Kevin D. Kissell wrote:
> > I note that Ralf has, in fact, applied the fix to the
> > OSS CVS repository.  I also note that "BARRIER"
> > is still defined to be a string of 6 nops.  I would argue
> > (again) that those really, really ought to be ssnops,
> > and that if they *were* ssnops, one could probably
> > have fewer of them.
> 
> Sorry for being ignorant, but what's the difference between nop and ssnop?
> 
> I see that SSNOP is defined to be `sll zero,zero,1' in <asm/asm.h>, but that
> doesn't give me any clue.

SSNOPs are "super-scalar NOPs", which were first
invented (but not documented at the time) for the 
R8000, which was the first superscalar MIPS
implementation.  They wanted to be able to absorb
the standard "overhead" NOPS associated with
unfilled branch delay slots, etc, in the dual-issue
mechanism, but still have some means to handle
CP0 hazards such that it could be assured to force
a 1 cycle stall per instruction.  While it wasn't officially
a part of the architecture until the late 1990's, the
convention was carried forward by the R5xxx
and R1xxxx families. There have been rumours of 
superscalar MIPS processors that do not enforce 
single-issue on SSNOPs, but I don't know of any 
offhand, and the MIPS32/MIPS64 specs formalize 
the definition.

            Regards,

            Kevin K.

Re: Malta crashes on the latest 2.4 kernel

[Ralf Baechle]

On Thu, Jul 11, 2002 at 08:56:17AM +0200, Geert Uytterhoeven wrote:

> On Thu, 11 Jul 2002, Kevin D. Kissell wrote:
> > I note that Ralf has, in fact, applied the fix to the
> > OSS CVS repository.  I also note that "BARRIER"
> > is still defined to be a string of 6 nops.  I would argue
> > (again) that those really, really ought to be ssnops,
> > and that if they *were* ssnops, one could probably
> > have fewer of them.
> 
> Sorry for being ignorant, but what's the difference between nop and ssnop?
> 
> I see that SSNOP is defined to be `sll zero,zero,1' in <asm/asm.h>, but that
> doesn't give me any clue.

Ssnop is a superscalar nop.  It's instruction encoding is the same as of
sll, zero, zero, 1.  Unlike a normal nop a ssnop is guaranteed to single
issue even on superscalar implementations.

  Ralf

Re: Malta crashes on the latest 2.4 kernel

[Ralf Baechle]

On Thu, Jul 11, 2002 at 08:19:55AM +0200, Kevin D. Kissell wrote:

> Excuse me, but I've seen this statement used by others in
> the past as an excuse for doing something silly or not doing
> something reasonable, and it generally hasn't washed.
> In what specific cases have the CP0 pipeline hazards 
> changed between minor revisions of any production
> MIPS CPU?  The *documentation* may have been
> corrected, but these hazards are fairly fundamental
> artifacts of the pipeline microarchitecture of a given
> processor.

Ancient TLB exception handler code was assuming out of order execution of
the instruction stream in cp0 based on the documentation in appendix H
of the R4400 manual, version 2.  I wrote that code for a R4400 version 5.0
and it was running fine on R4000 3.0 but somebody found it to break on
R4000 version 2.2.  At least that are the details as I remember them.  I
don't blame MIPS (well, probably SGI at that time ...) for not documenting
these details perfectly right for each and every R4[04]00 implementation.
The code broken was written extremly aggressivly and eventually had to be
changed anyway for the sake of other processors.

> The CP0 hazard between a write of EntryHi
> and a subsequent TLBWI instruction is flagged
> in the MIPS32 spec and noted as being "typically" 
> 2 cycles.  I'm not going to spend the time going
> through my full set of users manuals, but a representative
> sampling shows this hazard as being specified for
> every R4xxx and R5xxx CPU I checked.  The fact
> that a given CPU *may* get away with it is no
> excuse for not protecting common code.

No argument about this one.  We definately were lucky.

> I note that Ralf has, in fact, applied the fix to the
> OSS CVS repository.  I also note that "BARRIER"
> is still defined to be a string of 6 nops.  I would argue
> (again) that those really, really ought to be ssnops,
> and that if they *were* ssnops, one could probably
> have fewer of them.

I've applied it because I think the whole update_mmu_cache implementation
is ready for a reimplementation anyway.  On the performance this isn't
going to have measurable impact anyway as update_mmu_cache is only being
called once per page fault.

BARRIER is defined as 6 nops since it was written somewhen during the summer
'96.  By that time Linux didn't yet support any processor that was featuring
ssnop, so 6 nops certainly were too paranoid.  These days you're certainly
right, ssnops are the way to go, especially because they don't have any
negative impact on pre-ssnop implementations.

  Ralf