MIPS32 patches breaking DecStation

MIPS32 patches breaking DecStation

[Florian Lohoff]

Hi,
i found this snippet from arch/mips/kernel/head.S breaking DecStations:

@@ -68,9 +76,9 @@
        mtc0    k0, CP0_ENTRYLO0                # load it
        srl     k1, k1, 6                       # convert to entrylo1
        mtc0    k1, CP0_ENTRYLO1                # load it
-       b       1f
-        tlbwr                                  # write random tlb entry
-1:     
+       nop                                     # Need 2 cycles between mtc0
+       nop                                     #  and tlbwr (CP0 hazard).
+       tlbwr                                   # write random tlb entry
        nop
        eret                                    # return from trap
        END(except_vec0_r4000)


>From the Documentation and how i understand it this is perfectly
valid as the mtc0 instruction causes a cp0 hazard within the next 2 instruction
thought the delay of 2 cycles would be ok.

This is probably related to the Decstations having REALLY old R4000 silicion
revisions - Probably one should look through the erratas for these

flo@repeat:~$ cat /proc/cpuinfo 
cpu			: MIPS
cpu model		: R4000SC V3.0
system type		: Digital DECstation 5000/1xx

OK - I just had a look at the errata - This IS a workaround to a 
Mips R4000SC 2.0, 3.0 errata - I restored the original code back.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

MIPS32 patches breaking DecStation

[Florian Lohoff]

Hi,
i found this snippet from arch/mips/kernel/head.S breaking DecStations:

@@ -68,9 +76,9 @@
        mtc0    k0, CP0_ENTRYLO0                # load it
        srl     k1, k1, 6                       # convert to entrylo1
        mtc0    k1, CP0_ENTRYLO1                # load it
-       b       1f
-        tlbwr                                  # write random tlb entry
-1:     
+       nop                                     # Need 2 cycles between mtc0
+       nop                                     #  and tlbwr (CP0 hazard).
+       tlbwr                                   # write random tlb entry
        nop
        eret                                    # return from trap
        END(except_vec0_r4000)


From the Documentation and how i understand it this is perfectly
valid as the mtc0 instruction causes a cp0 hazard within the next 2 instruction
thought the delay of 2 cycles would be ok.

This is probably related to the Decstations having REALLY old R4000 silicion
revisions - Probably one should look through the erratas for these

flo@repeat:~$ cat /proc/cpuinfo 
cpu			: MIPS
cpu model		: R4000SC V3.0
system type		: Digital DECstation 5000/1xx

OK - I just had a look at the errata - This IS a workaround to a 
Mips R4000SC 2.0, 3.0 errata - I restored the original code back.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

Florian,

Could you do me a huge favor and try a build that
uses 3 or 4 nops instead of the branch to the instruction
after the delay slot?   There was a reason that I eliminated
the branch construct from the MIPS internal Linux source
base - it's a hack that works perfectly on R4000's, but
it's pretty much a coincidence that it does so.  Yes,
the code fragment in question is R4K-specific, but
we really need to migrate towards the use of consistent
mechanisms that work across the full range of MIPS
CPUs.  Ideally, *all* CP0 hazards should some day be 
padded out with "ssnops" (sll $0,$0,1, if I recall), which 
force a 1 cycle delay per instruction even on superscalar 
MIPS CPUs.

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

Florian,

Could you do me a huge favor and try a build that
uses 3 or 4 nops instead of the branch to the instruction
after the delay slot?   There was a reason that I eliminated
the branch construct from the MIPS internal Linux source
base - it's a hack that works perfectly on R4000's, but
it's pretty much a coincidence that it does so.  Yes,
the code fragment in question is R4K-specific, but
we really need to migrate towards the use of consistent
mechanisms that work across the full range of MIPS
CPUs.  Ideally, *all* CP0 hazards should some day be 
padded out with "ssnops" (sll $0,$0,1, if I recall), which 
force a 1 cycle delay per instruction even on superscalar 
MIPS CPUs.

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Florian Lohoff]

On Tue, Jan 09, 2001 at 01:34:47AM +0100, Kevin D. Kissell wrote:
> Florian,
> 
> Could you do me a huge favor and try a build that
> uses 3 or 4 nops instead of the branch to the instruction
> after the delay slot?   There was a reason that I eliminated

No problem - Done - doesnt work

> the branch construct from the MIPS internal Linux source
> base - it's a hack that works perfectly on R4000's, but
> it's pretty much a coincidence that it does so.  Yes,
> the code fragment in question is R4K-specific, but
> we really need to migrate towards the use of consistent
> mechanisms that work across the full range of MIPS
> CPUs.  Ideally, *all* CP0 hazards should some day be 
> padded out with "ssnops" (sll $0,$0,1, if I recall), which 
> force a 1 cycle delay per instruction even on superscalar 
> MIPS CPUs.

It immediatly after starting init goes crazy with "Illegal instruction"
and dies like this:

[ ... a couple hundret Illegal instruction ... ]
[init:1] Illegal instruction 8f9983d4 at 0fb68df8 ra=0fb68a20
[init:1] Illegal instruction 8fbc0018 at 0fb68e08 ra=0fb68a20
[init:1] Illegal instruction 02402021 at 0fb68e10 ra=0fb68a20
[init:1] Illegal instruction 00000000 at 0fb68e18 ra=0fb68a20
Kernel panic: Attempted to kill init!
BUG IN DYNAMIC LINKER ld.so: dl-minimal.c: 67: malloc: Assertion `n <= _dl_page!

It seems we need a R4000 2.0/3.0 special except_vec_r4000 like we already
have for the R4600 and some other kinds of mips CPUs.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: MIPS32 patches breaking DecStation

[Harald Koerfgen]

On 09-Jan-01 Florian Lohoff wrote:
> On Tue, Jan 09, 2001 at 01:34:47AM +0100, Kevin D. Kissell wrote:
>> Florian,
>> 
>> Could you do me a huge favor and try a build that
>> uses 3 or 4 nops instead of the branch to the instruction
>> after the delay slot?   There was a reason that I eliminated
> 
> No problem - Done - doesnt work

Same here on my /260 (R4400SC V4.0). Neither inserting four "sll $0,$0,1" nor
four "nop" seem to work. The branch, on the other hand, does.

-- 
Regards,
Harald

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 06:11:00PM +0100, Harald Koerfgen wrote:

> > No problem - Done - doesnt work
> 
> Same here on my /260 (R4400SC V4.0). Neither inserting four "sll $0,$0,1" nor
> four "nop" seem to work. The branch, on the other hand, does.

Note the ssnops only make sense on superscalar CPUs, so not on the R4000.
Also note that the branch is equivalent to three nops.  One for the
branch instruction itself, the two more for instructions in the pipeline
that get killed.  On the R4600 / R500 where the hazard is only a single
instruction the branch is equivalent to only a single nop.  So while
unobvious the branch is a rather neat idea.

  Ralf

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Same here on my /260 (R4400SC V4.0). Neither inserting four "sll
$0,$0,1" nor
> > four "nop" seem to work. The branch, on the other hand, does.

Then it's apparently more than just a garden-variety CP0 hazard.

> Note the ssnops only make sense on superscalar CPUs, so not on the R4000.

My point is that an SSNOP should cause a 1 cycle stall on *any* MIPS
implementation to date, superscalar or not.  It's not documented that
way for the R10000, but if I recall correctly, it works there too.  If one
wants to standardize on a single mechanism, that's the one to use.
That's all I'm saying.

> Also note that the branch is equivalent to three nops.  One for the
> branch instruction itself, the two more for instructions in the pipeline
> that get killed.  On the R4600 / R500 where the hazard is only a single
> instruction the branch is equivalent to only a single nop.  So while
> unobvious the branch is a rather neat idea.

Yes, it's cute, but it relies on accidents of implementation to work,
and could easily fail on other CPUs otherwise compatible with
the R4000.  In principle, such a branch might incur no delay at
all on an advanced 64-bit processor.  By all means, use it for
the specific cases of the CPUs on which it is known to work,
but it should not be used in "default" MIPS CP0 handlers.

            Regards,

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Same here on my /260 (R4400SC V4.0). Neither inserting four "sll
$0,$0,1" nor
> > four "nop" seem to work. The branch, on the other hand, does.

Then it's apparently more than just a garden-variety CP0 hazard.

> Note the ssnops only make sense on superscalar CPUs, so not on the R4000.

My point is that an SSNOP should cause a 1 cycle stall on *any* MIPS
implementation to date, superscalar or not.  It's not documented that
way for the R10000, but if I recall correctly, it works there too.  If one
wants to standardize on a single mechanism, that's the one to use.
That's all I'm saying.

> Also note that the branch is equivalent to three nops.  One for the
> branch instruction itself, the two more for instructions in the pipeline
> that get killed.  On the R4600 / R500 where the hazard is only a single
> instruction the branch is equivalent to only a single nop.  So while
> unobvious the branch is a rather neat idea.

Yes, it's cute, but it relies on accidents of implementation to work,
and could easily fail on other CPUs otherwise compatible with
the R4000.  In principle, such a branch might incur no delay at
all on an advanced 64-bit processor.  By all means, use it for
the specific cases of the CPUs on which it is known to work,
but it should not be used in "default" MIPS CP0 handlers.

            Regards,

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 08:30:05PM +0100, Kevin D. Kissell wrote:

> My point is that an SSNOP should cause a 1 cycle stall on *any* MIPS
> implementation to date, superscalar or not.  It's not documented that
> way for the R10000, but if I recall correctly, it works there too.  If one
> wants to standardize on a single mechanism, that's the one to use.
> That's all I'm saying.

I agree on that - except that I haven't seen the various various flavours
of nops documented anywhere except in IRIX's <sys/asm.h> ...

> > Also note that the branch is equivalent to three nops.  One for the
> > branch instruction itself, the two more for instructions in the pipeline
> > that get killed.  On the R4600 / R500 where the hazard is only a single
> > instruction the branch is equivalent to only a single nop.  So while
> > unobvious the branch is a rather neat idea.
> 
> Yes, it's cute, but it relies on accidents of implementation to work,
> and could easily fail on other CPUs otherwise compatible with
> the R4000.  In principle, such a branch might incur no delay at
> all on an advanced 64-bit processor.  By all means, use it for
> the specific cases of the CPUs on which it is known to work,
> but it should not be used in "default" MIPS CP0 handlers.

This behaviour of the R4000 branch is documented in the R4000 manual's
description of the pipeline.

  Ralf

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 08:30:05PM +0100, Kevin D. Kissell wrote:

> My point is that an SSNOP should cause a 1 cycle stall on *any* MIPS
> implementation to date, superscalar or not.  It's not documented that
> way for the R10000, but if I recall correctly, it works there too.  If one
> wants to standardize on a single mechanism, that's the one to use.
> That's all I'm saying.

I agree on that - except that I haven't seen the various various flavours
of nops documented anywhere except in IRIX's <sys/asm.h> ...

> > Also note that the branch is equivalent to three nops.  One for the
> > branch instruction itself, the two more for instructions in the pipeline
> > that get killed.  On the R4600 / R500 where the hazard is only a single
> > instruction the branch is equivalent to only a single nop.  So while
> > unobvious the branch is a rather neat idea.
> 
> Yes, it's cute, but it relies on accidents of implementation to work,
> and could easily fail on other CPUs otherwise compatible with
> the R4000.  In principle, such a branch might incur no delay at
> all on an advanced 64-bit processor.  By all means, use it for
> the specific cases of the CPUs on which it is known to work,
> but it should not be used in "default" MIPS CP0 handlers.

This behaviour of the R4000 branch is documented in the R4000 manual's
description of the pipeline.

  Ralf

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Yes, it's cute, but it relies on accidents of implementation to work,
> > and could easily fail on other CPUs otherwise compatible with
> > the R4000.  In principle, such a branch might incur no delay at
> > all on an advanced 64-bit processor.  By all means, use it for
> > the specific cases of the CPUs on which it is known to work,
> > but it should not be used in "default" MIPS CP0 handlers.
> 
> This behaviour of the R4000 branch is documented in the R4000 manual's
> description of the pipeline.

Yes, yes, like I said, use it whenever you see an R4000 PrID
if you like.  Just don't use it as the handler installed for a PrID
not otherwise known to the kernel.

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Yes, it's cute, but it relies on accidents of implementation to work,
> > and could easily fail on other CPUs otherwise compatible with
> > the R4000.  In principle, such a branch might incur no delay at
> > all on an advanced 64-bit processor.  By all means, use it for
> > the specific cases of the CPUs on which it is known to work,
> > but it should not be used in "default" MIPS CP0 handlers.
> 
> This behaviour of the R4000 branch is documented in the R4000 manual's
> description of the pipeline.

Yes, yes, like I said, use it whenever you see an R4000 PrID
if you like.  Just don't use it as the handler installed for a PrID
not otherwise known to the kernel.

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 01:34:47AM +0100, Kevin D. Kissell wrote:

> uses 3 or 4 nops instead of the branch to the instruction
> after the delay slot?   There was a reason that I eliminated
> the branch construct from the MIPS internal Linux source
> base - it's a hack that works perfectly on R4000's, but
> it's pretty much a coincidence that it does so.  Yes,
> the code fragment in question is R4K-specific, but
> we really need to migrate towards the use of consistent
> mechanisms that work across the full range of MIPS
> CPUs.  Ideally, *all* CP0 hazards should some day be 
> padded out with "ssnops" (sll $0,$0,1, if I recall), which 
> force a 1 cycle delay per instruction even on superscalar 
> MIPS CPUs.

While we could come up with a common TLB fault handler I really don't want
to.  For many applications this TLB fault handler is extremly performance
sensitive; every single cycle directly translates into application
performance.  Seems like we'll need some more TLB fault handler.  And a
complete TLB fault handler rewrite would be a good ide anyway, sigh ...

  Ralf

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 01:34:47AM +0100, Kevin D. Kissell wrote:

> uses 3 or 4 nops instead of the branch to the instruction
> after the delay slot?   There was a reason that I eliminated
> the branch construct from the MIPS internal Linux source
> base - it's a hack that works perfectly on R4000's, but
> it's pretty much a coincidence that it does so.  Yes,
> the code fragment in question is R4K-specific, but
> we really need to migrate towards the use of consistent
> mechanisms that work across the full range of MIPS
> CPUs.  Ideally, *all* CP0 hazards should some day be 
> padded out with "ssnops" (sll $0,$0,1, if I recall), which 
> force a 1 cycle delay per instruction even on superscalar 
> MIPS CPUs.

While we could come up with a common TLB fault handler I really don't want
to.  For many applications this TLB fault handler is extremly performance
sensitive; every single cycle directly translates into application
performance.  Seems like we'll need some more TLB fault handler.  And a
complete TLB fault handler rewrite would be a good ide anyway, sigh ...

  Ralf

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Yes, the code fragment in question is R4K-specific, but
> > we really need to migrate towards the use of consistent
> > mechanisms that work across the full range of MIPS
> > CPUs.  Ideally, *all* CP0 hazards should some day be
> > padded out with "ssnops" (sll $0,$0,1, if I recall), which
> > force a 1 cycle delay per instruction even on superscalar
> > MIPS CPUs.
>
> While we could come up with a common TLB fault handler I really don't want
> to.  For many applications this TLB fault handler is extremly performance
> sensitive; every single cycle directly translates into application
> performance.  Seems like we'll need some more TLB fault handler.  And a
> complete TLB fault handler rewrite would be a good ide anyway, sigh ...

Sorry if I wasn't clear.  I am not suggesting a "one size
fits all" TLB handler - though as the old SGI hardware
gets retired and the newer, more standardized MIPS32
and MIPS64 parts become the principal targets, we
may see a greater convergence.  I am simply suggesting
that, even if there are differences in policy necessitated
by different CPU implementations, they should use the
same mechanisms.  If all CP0 hazards are avoided by
using ssnops, for example, we could evolve from writing
a new handler for every R4K variant to having a routine
that generates a handler with the right pipeline hazard
management, based on a set of paramters for the CPU.
And, while Ralf and I sometimes disagree on the importance
of this, in my own opinion, being consistent in these small
details helps avoid errors when a systems programmer
new to the architecture and/or the OS needs to work on
the system.

You may say that I'm a dreamer, but I'm not the only one.  ;-)

            Regards,

            Kevin K.

Re: MIPS32 patches breaking DecStation

[Kevin D. Kissell]

> > Yes, the code fragment in question is R4K-specific, but
> > we really need to migrate towards the use of consistent
> > mechanisms that work across the full range of MIPS
> > CPUs.  Ideally, *all* CP0 hazards should some day be
> > padded out with "ssnops" (sll $0,$0,1, if I recall), which
> > force a 1 cycle delay per instruction even on superscalar
> > MIPS CPUs.
>
> While we could come up with a common TLB fault handler I really don't want
> to.  For many applications this TLB fault handler is extremly performance
> sensitive; every single cycle directly translates into application
> performance.  Seems like we'll need some more TLB fault handler.  And a
> complete TLB fault handler rewrite would be a good ide anyway, sigh ...

Sorry if I wasn't clear.  I am not suggesting a "one size
fits all" TLB handler - though as the old SGI hardware
gets retired and the newer, more standardized MIPS32
and MIPS64 parts become the principal targets, we
may see a greater convergence.  I am simply suggesting
that, even if there are differences in policy necessitated
by different CPU implementations, they should use the
same mechanisms.  If all CP0 hazards are avoided by
using ssnops, for example, we could evolve from writing
a new handler for every R4K variant to having a routine
that generates a handler with the right pipeline hazard
management, based on a set of paramters for the CPU.
And, while Ralf and I sometimes disagree on the importance
of this, in my own opinion, being consistent in these small
details helps avoid errors when a systems programmer
new to the architecture and/or the OS needs to work on
the system.

You may say that I'm a dreamer, but I'm not the only one.  ;-)

            Regards,

            Kevin K.

RE: MIPS32 patches breaking DecStation

[Harald Koerfgen]

On 08-Jan-01 Florian Lohoff wrote:
> OK - I just had a look at the errata - This IS a workaround to a 
> Mips R4000SC 2.0, 3.0 errata - I restored the original code back.

Where have you found those errata?

-- 
Regards,
Harald

Re: MIPS32 patches breaking DecStation

[Ralf Baechle]

On Tue, Jan 09, 2001 at 05:43:08PM +0100, Harald Koerfgen wrote:

> On 08-Jan-01 Florian Lohoff wrote:
> > OK - I just had a look at the errata - This IS a workaround to a 
> > Mips R4000SC 2.0, 3.0 errata - I restored the original code back.
> 
> Where have you found those errata?

Go to www.mips.com, follow the publications link.  Yes, MIPS be praised
for having errata and docs for rotten old silicon online!

  Ralf