dcbz works on 862 everywhere!

dcbz works on 862 everywhere!

[Joakim Tjernlund]

Hi list

I have been researching why I can't use dcbz and friends on 8xx CPU.
Till Strauman found out long ago that the dcxx instuctions did not update
the DAR register and that was the key I needed.

After a lot of testing/reading I found a 2 instruction workaround
that lets me use dcbz in copy_tofrom_user().
I can also used dcbz in a user space app without problems. I even
removed the TLB change bit hack by Tom Rini and everything works.

By copying the MD_EPN register to the DAR register in the DTLB Miss handler
just before MD_RPN is written everything just works. The only explanation that
makes sense to me is that the MD_RPN has a undocumented dependency to the DAR
register, the 20 msb of MD_EPN and DAR needs to be the same before MD_RPN is written.

Does this make any sense to anyone?
I can prepare a patch if somebody is interested to try it on their 8xx CPU.

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Does this make any sense to anyone?
> I can prepare a patch if somebody is interested to try it on their 8xx CPU.
>
>  Jocke

I forgot:
Why is the Guarded flag set in DTLB Miss handler for the pte?
Won't performance degrade?

 Jocke

PS.
  Here is the patch if anyone wants to test. Please report your results.
  Don't forget to enable dcbz in arch/ppc/kernel/misc.S and arch/ppc/lib/string.S
  for 8xx.

Index: arch/ppc/kernel/head_8xx.S
===================================================================
RCS file: /home/cvsadmin/cvsroot/kernel/linuxppc/arch/ppc/kernel/head_8xx.S,v
retrieving revision 1.1.1.1
diff -u -r1.1.1.1 head_8xx.S
--- arch/ppc/kernel/head_8xx.S	1 Nov 2002 13:44:02 -0000	1.1.1.1
+++ arch/ppc/kernel/head_8xx.S	24 Mar 2003 10:46:22 -0000
@@ -407,6 +407,13 @@
 	rlwinm.	r20, r21,0,0,19	/* Extract page descriptor page address */
 	beq	2f		/* If zero, don't try to find a pte */

+	/* For some reason the 20 msb of MD_EPN and DAR must be the same when
+	 * MD_RPN is written. The dcxx instructions don't set DAR when they
+	 * cause a DTLB Miss so copy them from MD_EPN.
+	 */
+	mfspr	r20, MD_EPN
+	mtspr	DAR, r20
+
 	/* We have a pte table, so load fetch the pte from the table.
 	 */
 	tophys(r21, r21)
@@ -463,7 +470,18 @@
 #endif
 	rfi

-2:	mfspr	r20, M_TW	/* Restore registers */
+2:
+	/* Copy 20 msb from EPN to DAR since the dcxx instuctions fails
+	 * update the DAR when they cause a DTLB Miss.
+	 */
+	mfspr	r21, MD_EPN
+	rlwinm	r21, r21, 0, 0, 19
+	mfspr	r20, DAR
+	rlwinm	r20, r20, 0, 20, 31
+	or	r20, r20, r21
+	mtspr	DAR, r20
+
+	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
 	lwz	r21, 4(r0)


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Hi list
>
> I have been researching why I can't use dcbz and friends on 8xx CPU.
> Till Strauman found out long ago that the dcxx instuctions did not update
> the DAR register and that was the key I needed.
>
> After a lot of testing/reading I found a 2 instruction workaround
> that lets me use dcbz in copy_tofrom_user().
> I can also used dcbz in a user space app without problems. I even
> removed the TLB change bit hack by Tom Rini and everything works.
>
> By copying the MD_EPN register to the DAR register in the DTLB Miss handler
> just before MD_RPN is written everything just works. The only explanation that
> makes sense to me is that the MD_RPN has a undocumented dependency to the DAR
> register, the 20 msb of MD_EPN and DAR needs to be the same before MD_RPN is written.

hmm, this is not quite correct. I can copy MD_EPN to DAR after the write to MD_RPN
and it still works. Seems like DAR must be correct when rfi is executed. I wonder
why?

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> > Hi list
> >
> > I have been researching why I can't use dcbz and friends on 8xx CPU.
> > Till Strauman found out long ago that the dcxx instuctions did not update
> > the DAR register and that was the key I needed.
> >
> > After a lot of testing/reading I found a 2 instruction workaround
> > that lets me use dcbz in copy_tofrom_user().
> > I can also used dcbz in a user space app without problems. I even
> > removed the TLB change bit hack by Tom Rini and everything works.
> >
> > By copying the MD_EPN register to the DAR register in the DTLB Miss handler
> > just before MD_RPN is written everything just works. The only explanation that
> > makes sense to me is that the MD_RPN has a undocumented dependency to the DAR
> > register, the 20 msb of MD_EPN and DAR needs to be the same before MD_RPN is written.
>
> hmm, this is not quite correct. I can copy MD_EPN to DAR after the write to MD_RPN
> and it still works. Seems like DAR must be correct when rfi is executed. I wonder
> why?

hmm, can it be that when you return from the DTLB Miss, you sometimes get an DTLB error
immediately and that DTLB Error "inherits" the DAR from the DTLB Miss?

I removed the "copy DAR to MD_EPN" part in the DTLB Error handler and my systems
runs fine.

   Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Till Straumann]

Joakim Tjernlund wrote:

>>>Hi list
>>>
-- snip --

>>>  I even
>>>removed the TLB change bit hack by Tom Rini and everything works.
>>>
-- snip --

>
>I removed the "copy DAR to MD_EPN" part in the DTLB Error handler and my systems
>runs fine.
>

How exactly do you know that "everything works fine" ???

Please note that

    a) you may have bug-free hardware.
    b) you may have a valid TLB entry at the time 'dcbz' is executed.
    c) you may end up with a bogus mapping but not notice it.

You really should first confirm that your hardware suffers from the bug by
running a _simple_ test program in a _simple_ environment (NOT linux).

Then, you should confirm that your workaround solution is correct by
testing it in the same simple environment.

Just because you are able to use 'dcbz' on linux repeatedly without
crashing does'nt prove that your hardware is buggy but linux handles
it correctly.

-- Till

PS: please CC me on any replies; I'm not subscribed to this list

>
>   Jocke
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> >>>Hi list
> >>>
> -- snip --
>
> >>>  I even
> >>>removed the TLB change bit hack by Tom Rini and everything works.
> >>>
> -- snip --
>
> >
> >I removed the "copy DAR to MD_EPN" part in the DTLB Error handler and my systems
> >runs fine.
> >
>
> How exactly do you know that "everything works fine" ???
>
> Please note that
>
>     a) you may have bug-free hardware.
>     b) you may have a valid TLB entry at the time 'dcbz' is executed.
>     c) you may end up with a bogus mapping but not notice it.

Every time I tried to use dcbz on user space it failed HARD directly.
Whenever I tried to "undo" any hacks that were there to compensate for
buggy dcxx instructions, the kernel would not boot.

Now I can boot and run my app and even run a user space app which clears
memory with dcbz. I can remove HWWRITE bit setting on kernel space address and
remove the copying of DAR to MD_EPN in the DTLB Error handler.

I think I am on to something here, don't you?
>
> You really should first confirm that your hardware suffers from the bug by
> running a _simple_ test program in a _simple_ environment (NOT linux).

I don't have a simple environment. I have ppcboot on my board which doesn't
use the MMU at all.

>
> Then, you should confirm that your workaround solution is correct by
> testing it in the same simple environment.
>
> Just because you are able to use 'dcbz' on linux repeatedly without
> crashing does'nt prove that your hardware is buggy but linux handles
> it correctly.

Yes, I know that and that's why I write to this list to seek input from
others that know more than I do and have a simple test environment(hint, hint :-)

>
> -- Till
>
> PS: please CC me on any replies; I'm not subscribed to this list
>
> >
> >   Jocke
> >

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Tom Rini]

On Mon, Mar 24, 2003 at 11:19:29AM +0100, Joakim Tjernlund wrote:

>
> Hi list
>
> I have been researching why I can't use dcbz and friends on 8xx CPU.
> Till Strauman found out long ago that the dcxx instuctions did not update
> the DAR register and that was the key I needed.
>
> After a lot of testing/reading I found a 2 instruction workaround
> that lets me use dcbz in copy_tofrom_user().
> I can also used dcbz in a user space app without problems. I even
> removed the TLB change bit hack by Tom Rini and everything works.
>
> By copying the MD_EPN register to the DAR register in the DTLB Miss handler
> just before MD_RPN is written everything just works. The only explanation that
> makes sense to me is that the MD_RPN has a undocumented dependency to the DAR
> register, the 20 msb of MD_EPN and DAR needs to be the same before MD_RPN is written.
>
> Does this make any sense to anyone?
> I can prepare a patch if somebody is interested to try it on their 8xx CPU.

Can you post a patch which does everything (remove the TLB change bit,
etc)?  Thanks.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Can you post a patch which does everything (remove the TLB change bit,
> etc)?  Thanks.

Sure, but I don't want to restore the old TLB change bit. I removed it as a test only to see if
my system could take it. The kernel will do a lot DTLB errors if it is restored.

I will removed the Guarded flag though since I don't see why it's there. Speak up
now if you want to have it left.

     Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Tom Rini]

On Mon, Mar 24, 2003 at 09:57:43PM +0100, Joakim Tjernlund wrote:
> > Can you post a patch which does everything (remove the TLB change bit,
> > etc)?  Thanks.
>
> Sure, but I don't want to restore the old TLB change bit. I removed it as a test only to see if
> my system could take it. The kernel will do a lot DTLB errors if it is restored.
>
> I will removed the Guarded flag though since I don't see why it's there. Speak up
> now if you want to have it left.

Well, if I understand what you think you've got working correctly, what
you've got replaces the current workaround for a similar (related?)
issue.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

> I will removed the Guarded flag though since I don't see why it's there. Speak up
> now if you want to have it left.

Everything is there for a reason, including setting the guarded flag if
the PTE indicates this should be done.  Since it doesn't seem to be clear
from messages I've sent in the past, the guarded flag is used to ensure
the proper order of operation on I/O spaces.  Any ioremap will request
this feature.

Thanks.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> > I will removed the Guarded flag though since I don't see why it's there. Speak up
> > now if you want to have it left.
>
> Everything is there for a reason, including setting the guarded flag if
> the PTE indicates this should be done.  Since it doesn't seem to be clear
> from messages I've sent in the past, the guarded flag is used to ensure
> the proper order of operation on I/O spaces.  Any ioremap will request
> this feature.

I got the impression that the Guarded flag was ALWAYS set. Reading
the comment again I think I missunderstood(my assembly is rusty), right?

  Jocke

> -- Dan
>

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> On Mon, Mar 24, 2003 at 09:57:43PM +0100, Joakim Tjernlund wrote:
> > > Can you post a patch which does everything (remove the TLB change bit,
> > > etc)?  Thanks.
> >
> > Sure, but I don't want to restore the old TLB change bit. I removed it as a test only to see if
> > my system could take it. The kernel will do a lot DTLB errors if it is restored.
> >
> > I will removed the Guarded flag though since I don't see why it's there. Speak up
> > now if you want to have it left.
>
> Well, if I understand what you think you've got working correctly, what
> you've got replaces the current workaround for a similar (related?)
> issue.

I think it makes the "copy DAR to MD_EPN" part in the DTLB error handler needless.
Anyway it works with and without that part.

  Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Can you post a patch which does everything (remove the TLB change bit,
> etc)?  Thanks.
>
> --
> Tom Rini

OK, here it goes. I had do this blind since I am still  on 2.4.20. Let me know
if it does not compile.

 Jocke
===== arch/ppc/kernel/head_8xx.S 1.41 vs edited =====
--- 1.41/arch/ppc/kernel/head_8xx.S Thu Feb 27 20:40:15 2003
+++ edited/arch/ppc/kernel/head_8xx.S Mon Mar 24 23:31:27 2003
@@ -451,6 +451,13 @@
 #endif
  mtspr MD_RPN, r20 /* Update TLB entry */

+ /* The 20 msb of MD_EPN and DAR must be the same when rfi is
+  * executed. The dcxx instructions don't set DAR when they
+  * cause a DTLB Miss so copy them from MD_EPN.
+  */
+ mfspr r20, MD_EPN
+ mtspr DAR, r20
+
  mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
@@ -460,7 +467,18 @@
 #endif
  rfi

-2: mfspr r20, M_TW /* Restore registers */
+2:
+ /* Copy 20 msb from EPN to DAR since the dcxx instuctions fails
+  * update the DAR when they cause a DTLB Miss.
+  */
+ mfspr r21, MD_EPN
+ rlwinm r21, r21, 0, 0, 19
+ mfspr r20, DAR
+ rlwinm r20, r20, 0, 20, 31
+ or r20, r20, r21
+ mtspr DAR, r20
+
+ mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
  lwz r21, 4(r0)
@@ -504,6 +522,9 @@
  andis. r21, r20, 0x0200 /* If set, indicates store op */
  beq 2f

+#if 0
+ /* Maybe this should stay? Time will tell */
+
  /* The EA of a data TLB miss is automatically stored in the MD_EPN
   * register.  The EA of a data TLB error is automatically stored in
   * the DAR, but not the MD_EPN register.  We must copy the 20 most
@@ -531,7 +552,7 @@
  lwz r3, 12(r0)
 #endif
  mtspr MD_EPN, r21
-
+#endif
  mfspr r20, M_TWB /* Get level 1 table entry address */

  /* If we are faulting a kernel address, we have to use the
===== arch/ppc/kernel/misc.S 1.80 vs edited =====
--- 1.80/arch/ppc/kernel/misc.S Thu Mar  6 06:36:05 2003
+++ edited/arch/ppc/kernel/misc.S Mon Mar 24 23:33:53 2003
@@ -664,7 +664,7 @@
 _GLOBAL(clear_page)
  li r0,4096/L1_CACHE_LINE_SIZE
  mtctr r0
-#ifdef CONFIG_8xx
+#ifdef CONFIG_8xx_CPU6
  li r4, 0
 1: stw r4, 0(r3)
  stw r4, 4(r3)
@@ -698,7 +698,7 @@
  addi r4,r4,-4
  li r5,4

-#ifndef CONFIG_8xx
+#ifndef CONFIG_8xx_CPU6
 #if MAX_COPY_PREFETCH > 1
  li r0,MAX_COPY_PREFETCH
  li r11,4
@@ -706,7 +706,7 @@
 11: dcbt r11,r4
  addi r11,r11,L1_CACHE_LINE_SIZE
  bdnz 11b
-#else /* MAX_L1_COPY_PREFETCH == 1 */
+#elif !defined(CONFIG_8xx) /* MAX_L1_COPY_PREFETCH == 1 */
  dcbt r5,r4
  li r11,L1_CACHE_LINE_SIZE+4
 #endif /* MAX_L1_COPY_PREFETCH */
@@ -715,8 +715,10 @@
  li r0,4096/L1_CACHE_LINE_SIZE
  mtctr r0
 1:
+#ifndef CONFIG_8xx_CPU6
 #ifndef CONFIG_8xx
- dcbt r11,r4
+ dcbt r11,r4 /* Makes 8xx slower */
+#endif
  dcbz r5,r3
 #endif
  COPY_16_BYTES
===== arch/ppc/lib/string.S 1.20 vs edited =====
--- 1.20/arch/ppc/lib/string.S Thu Feb 27 20:40:16 2003
+++ edited/arch/ppc/lib/string.S Mon Mar 24 23:34:20 2003
@@ -151,7 +151,7 @@
  bdnz 4b
 3: mtctr r9
  li r7,4
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx_CPU6)
 10: dcbz r7,r6
 #else
 10: stw r4, 4(r6)
@@ -253,7 +253,7 @@
  mtctr r0
  beq 63f
 53:
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx_CPU6)
  dcbz r11,r6
 #endif
  COPY_16_BYTES
@@ -427,7 +427,7 @@
  li r11,4
  beq 63f

-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx_CPU6)
  /* Here we decide how far ahead to prefetch the source */
 #if MAX_COPY_PREFETCH > 1
  /* Heuristically, for large transfers we prefetch
@@ -442,7 +442,7 @@
 112: dcbt r3,r4
  addi r3,r3,CACHELINE_BYTES
  bdnz 112b
-#else /* MAX_COPY_PREFETCH == 1 */
+#elif !defined(CONFIG_8xx) /* MAX_COPY_PREFETCH == 1 */
  li r3,CACHELINE_BYTES + 4
  dcbt r11,r4
 #endif /* MAX_COPY_PREFETCH */
@@ -450,8 +450,10 @@

  mtctr r0
 53:
+#if !defined(CONFIG_8xx_CPU6)
 #if !defined(CONFIG_8xx)
- dcbt r3,r4
+ dcbt r3,r4 /* Makes 8xx slower */
+#endif
 54: dcbz r11,r6
 #endif
 /* had to move these to keep extable in order */
@@ -461,7 +463,7 @@
  .long 71b,101f
  .long 72b,102f
  .long 73b,103f
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx_CPU6)
  .long 54b,105f
 #endif
  .text


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

>
> I think it makes the "copy DAR to MD_EPN" part in the DTLB error handler needless.
> Anyway it works with and without that part.

No, you need to leave that there as well.  Read the comment.....

If it appears to "work" for you, what happens is you have a DTLB Error
immediately following a DTLB miss (the common case lots of the time), so
the MD_EPN just happens to contain the proper information.  When you generate
the case where a DTLB Error does not match a previous DTLB miss, you will
end up tracking down the wrong PTEs.  Even in this case it is likely to
appear things work correctly, but they do not.  There is one case with the dcbi
where there is no proper operation of the code, so it was necessary to solve
the problem another way.

This is exactly why people are telling you to carefully generate test cases
the prove an error exists 100% of the time, and that your modifications
properly correct the condition 100% of the time.  When you run something
like Linux you get pathological cases that do not exercise all conditions
with a particular configurations, or you could be generating errors that
are not precisely tested as failure conditions.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

> OK, here it goes.

No, this isn't correct......


> + /* The 20 msb of MD_EPN and DAR must be the same when rfi is
> +  * executed. The dcxx instructions don't set DAR when they
> +  * cause a DTLB Miss so copy them from MD_EPN.

No, don't be screwing around with these registers.



> -#ifdef CONFIG_8xx
> +#ifdef CONFIG_8xx_CPU6

What is this all about?  The CPU6 errata is a clearly defined
problem on well known silicon revisions.  Don't be using it where
it isn't intended.

Please try again..........thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> > OK, here it goes.
>
> No, this isn't correct......
>
>
> > + /* The 20 msb of MD_EPN and DAR must be the same when rfi is
> > +  * executed. The dcxx instructions don't set DAR when they
> > +  * cause a DTLB Miss so copy them from MD_EPN.
>
> No, don't be screwing around with these registers.

Did you read my mails? This is the key.
What harm can it do? This is much less intrusive than the
"copy DAR to MD_EPN" part in the TLB Error handler.

>
>
>
> > -#ifdef CONFIG_8xx
> > +#ifdef CONFIG_8xx_CPU6
>
> What is this all about?  The CPU6 errata is a clearly defined
> problem on well known silicon revisions.  Don't be using it where
> it isn't intended.

OK, I have removed all CONFIG_8xx_CPU6 and CONFIG_8xx
I restored the "copy DAR to MD_EPN"  in the TLB Error hanler as well.

>
> Please try again..........thanks.


Here it is:
===== arch/ppc/kernel/head_8xx.S 1.41 vs edited =====
--- 1.41/arch/ppc/kernel/head_8xx.S Thu Feb 27 20:40:15 2003
+++ edited/arch/ppc/kernel/head_8xx.S Tue Mar 25 00:43:39 2003
@@ -451,6 +451,13 @@
 #endif
  mtspr MD_RPN, r20 /* Update TLB entry */

+ /* The 20 msb of MD_EPN and DAR must be the same when rfi is
+  * executed. The dcxx instructions don't set DAR when they
+  * cause a DTLB Miss so copy them from MD_EPN.
+  */
+ mfspr r20, MD_EPN
+ mtspr DAR, r20
+
  mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
@@ -460,7 +467,18 @@
 #endif
  rfi

-2: mfspr r20, M_TW /* Restore registers */
+2:
+ /* Copy 20 msb from EPN to DAR since the dcxx instuctions fails
+  * update the DAR when they cause a DTLB Miss.
+  */
+ mfspr r21, MD_EPN
+ rlwinm r21, r21, 0, 0, 19
+ mfspr r20, DAR
+ rlwinm r20, r20, 0, 20, 31
+ or r20, r20, r21
+ mtspr DAR, r20
+
+ mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
  lwz r21, 4(r0)
===== arch/ppc/kernel/misc.S 1.80 vs edited =====
--- 1.80/arch/ppc/kernel/misc.S Thu Mar  6 06:36:05 2003
+++ edited/arch/ppc/kernel/misc.S Tue Mar 25 00:41:16 2003
@@ -664,15 +664,7 @@
 _GLOBAL(clear_page)
  li r0,4096/L1_CACHE_LINE_SIZE
  mtctr r0
-#ifdef CONFIG_8xx
- li r4, 0
-1: stw r4, 0(r3)
- stw r4, 4(r3)
- stw r4, 8(r3)
- stw r4, 12(r3)
-#else
 1: dcbz 0,r3
-#endif
  addi r3,r3,L1_CACHE_LINE_SIZE
  bdnz 1b
  blr
@@ -698,7 +690,6 @@
  addi r4,r4,-4
  li r5,4

-#ifndef CONFIG_8xx
 #if MAX_COPY_PREFETCH > 1
  li r0,MAX_COPY_PREFETCH
  li r11,4
@@ -706,7 +697,7 @@
 11: dcbt r11,r4
  addi r11,r11,L1_CACHE_LINE_SIZE
  bdnz 11b
-#else /* MAX_L1_COPY_PREFETCH == 1 */
+#elif !defined(CONFIG_8xx) /* MAX_L1_COPY_PREFETCH == 1 */
  dcbt r5,r4
  li r11,L1_CACHE_LINE_SIZE+4
 #endif /* MAX_L1_COPY_PREFETCH */
@@ -716,9 +707,10 @@
  mtctr r0
 1:
 #ifndef CONFIG_8xx
- dcbt r11,r4
- dcbz r5,r3
+ dcbt r11,r4 /* Makes 8xx slower */
 #endif
+ dcbz r5,r3
+
  COPY_16_BYTES
 #if L1_CACHE_LINE_SIZE >= 32
  COPY_16_BYTES
===== arch/ppc/lib/string.S 1.20 vs edited =====
--- 1.20/arch/ppc/lib/string.S Thu Feb 27 20:40:16 2003
+++ edited/arch/ppc/lib/string.S Tue Mar 25 00:42:56 2003
@@ -151,14 +151,7 @@
  bdnz 4b
 3: mtctr r9
  li r7,4
-#if !defined(CONFIG_8xx)
 10: dcbz r7,r6
-#else
-10: stw r4, 4(r6)
- stw r4, 8(r6)
- stw r4, 12(r6)
- stw r4, 16(r6)
-#endif
  addi r6,r6,CACHELINE_BYTES
  bdnz 10b
  clrlwi r5,r8,32-LG_CACHELINE_BYTES
@@ -253,9 +246,8 @@
  mtctr r0
  beq 63f
 53:
-#if !defined(CONFIG_8xx)
  dcbz r11,r6
-#endif
+
  COPY_16_BYTES
 #if L1_CACHE_LINE_SIZE >= 32
  COPY_16_BYTES
@@ -427,7 +419,6 @@
  li r11,4
  beq 63f

-#if !defined(CONFIG_8xx)
  /* Here we decide how far ahead to prefetch the source */
 #if MAX_COPY_PREFETCH > 1
  /* Heuristically, for large transfers we prefetch
@@ -442,7 +433,7 @@
 112: dcbt r3,r4
  addi r3,r3,CACHELINE_BYTES
  bdnz 112b
-#else /* MAX_COPY_PREFETCH == 1 */
+#elif !defined(CONFIG_8xx) /* MAX_COPY_PREFETCH == 1 */
  li r3,CACHELINE_BYTES + 4
  dcbt r11,r4
 #endif /* MAX_COPY_PREFETCH */
@@ -451,9 +442,9 @@
  mtctr r0
 53:
 #if !defined(CONFIG_8xx)
- dcbt r3,r4
-54: dcbz r11,r6
+ dcbt r3,r4 /* Makes 8xx slower */
 #endif
+54: dcbz r11,r6
 /* had to move these to keep extable in order */
  .section __ex_table,"a"
  .align 2
@@ -461,9 +452,7 @@
  .long 71b,101f
  .long 72b,102f
  .long 73b,103f
-#if !defined(CONFIG_8xx)
  .long 54b,105f
-#endif
  .text
 /* the main body of the cacheline loop */
  COPY_16_BYTES_WITHEX(0)


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> >
> > I think it makes the "copy DAR to MD_EPN" part in the DTLB error handler needless.
> > Anyway it works with and without that part.
>
> No, you need to leave that there as well.  Read the comment.....
>
> If it appears to "work" for you, what happens is you have a DTLB Error
> immediately following a DTLB miss (the common case lots of the time), so
> the MD_EPN just happens to contain the proper information.  When you generate
> the case where a DTLB Error does not match a previous DTLB miss, you will
> end up tracking down the wrong PTEs.  Even in this case it is likely to
> appear things work correctly, but they do not.  There is one case with the dcbi
> where there is no proper operation of the code, so it was necessary to solve
> the problem another way.

What case is that(when dcbi does not work)? Give me an example so I can test
and see what happens, with and without my patch.

> This is exactly why people are telling you to carefully generate test cases
> the prove an error exists 100% of the time, and that your modifications
> properly correct the condition 100% of the time.  When you run something
> like Linux you get pathological cases that do not exercise all conditions
> with a particular configurations, or you could be generating errors that
> are not precisely tested as failure conditions.

I know all that and thats why post to the list with questions, ideas  and  patches to try out.
If you had read my mails you would had understood that too.
Your yelling at me for trying isn't very helpful.

    Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:


> Did you read my mails?

I did.  From this I gathered that you don't understand why some
of this code is there in the first place, you removed code that
would have guaranteed to cause an error if you would have run
the proper test, and you can't explain why the changes you made
cause the results you see.  Not something that instills much
confidence.

All of these MMU and cache control registers have effects on
one another.  Writing one register can have the side effect of
updating another.  The DAR must have valid addresses in order
to properly rfi, we know some cache operations may not set this
properly, and we also know that in these cases the MD_EPN is also
not set properly.  The reason it "works" for you is you found the
most likely case where you took a DTLB miss, followed by a DTLB Error
to the same page (or at least to valid pages).  If the miss/error
sequence was to the same page, it will likely work correctly.  If the
miss/error sequence is not to the same pages, it could appear
to "work" but chances are the dcbz was not executed properly and
you didn't notice the side effects of that.  If you happen to
find the case where you generate the DTLB miss/error/error with
intervening TLB exceptions for other pages (like when swapping)
and the instructions involved are cache instructions (like the
dcbi in the comment example), you will end up with a big mess.
It's really hard to generate all of these test cases, and you
can't always use Linux to do so, but eventually Linux will stumble
across them.

For a long time, we "solved" these problems on 8xx by not properly
implementing copy-on-write pages and always marking pages dirty.
It reduced the likelyhood of the problem, reduced the MMU overhead,
but cost lots more in memory and processing time during forks.
This still may not be a bad configuration option for selected
applications.

Like I said from the beginning, the 100% guaranteed working solution
is to not use dcbz/dcbt on the 8xx.  Anything else will eventually
stumble across the error conditions.  If I'm developing a product
that uses Linux (or any other OS that will dynamically update the MMU
and take DTLB exceptions) I won't take the risk of failure using
these instructions.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

After reading this I guess I need to further clarify what happens.....

Dan Malek wrote:

> ... The reason it "works" for you is you found the
> most likely case where you took a DTLB miss, followed by a DTLB Error
> to the same page...

What happens in these cases (which occurs like almost all of the time),
is there is "leftover" information in the control/status registers from
previous exceptions that causes the fault processing to take place
properly.  In some error conditions, the leftover information isn't
exactly correct, causing the page fault handler to look up wrong
information or the dcbz/dcbt to do their thing on the wrong address.
If you are not running a specific test case and precisely testing
for the results of the operation, you may not notice this or at some
time in the future notice erratic system behavior that is impossible
to explain.  The "normal" instructions have exactly one well documented
failure case, in the case of DTLBError, the MD_EPN sometimes doesn't
get the proper status, but we know the DAR does, so we can copy the
bits we need from the DAR to the EPN to properly process the fault.
The cache instructions have never been normal instructions.  In some
versions of silicon they won't generate a fault, some will allow cache
operations to uncached spaces, some will fault but not provide the
proper status in any MMU register.  Sometimes they work, it all depends
upon the state of the cache line and TLB.  The cases most likely to
fail are the ones that are the hardest to create.

I've spent way too much time working on this on many versions of
silicon with the assistance of people from Motorola.  It isn't a trivial
problem to detect, quantify and document so someone else can repeat
the exercise.  Through the first four major versions of 860 silicon
(and countless minor versions) the wrong behavior I discussed was
fixed in the next release, only to have something else unexpected appear.
Since I didn't want a career in cache instruction debugging (and I had
products to get to market), the solution was to simply not use these
instructions.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
>
> > Did you read my mails?
>
> I did.  From this I gathered that you don't understand why some
> of this code is there in the first place, you removed code that

No I don't understand this code fully. I don't think anybody but you understand
this code since it contains undocumented workarounds.
I asked the list before I removed it. Had you objected before I got to the
patch I had left it alone. It was just a test.

> would have guaranteed to cause an error if you would have run
> the proper test, and you can't explain why the changes you made
> cause the results you see.

But I can explain(didn't I do that earlier?). We agree that dcxx will
cause DTLB Misses without setting DAR, right?. By copying the MD_EPN to
DAR before rfi, DAR will have address whose EPN is correct but with a
semirandom page offset(the last bits in MD_EPN that doesn't referrer to
the EPN). This must be better than just leaving dar as is, especially if
rfi needs a valid addess to operate properly as you write below.

The second part:
+ /* Copy 20 msb from EPN to DAR since the dcxx instuctions fails
+  * update the DAR when they cause a DTLB Miss.
+  */
+ mfspr r21, MD_EPN
+ rlwinm r21, r21, 0, 0, 19
+ mfspr r20, DAR
+ rlwinm r20, r20, 0, 20, 31
+ or r20, r20, r21
+ mtspr DAR, r20

Just replaces the EPN part to generate a valid address within the page. The lower bits
are untouched. This is a nop if DAR already contains the right address, otherwise DAR
will at least contain a valid page address.

This could be move to before error test if you want play safe(the 2 instructions before
rfi should be removed in this case)

> Not something that instills much confidence.

After the above explanation I hope that will improve somewhat.

>
> All of these MMU and cache control registers have effects on
> one another.  Writing one register can have the side effect of
> updating another.  The DAR must have valid addresses in order
> to properly rfi, we know some cache operations may not set this
> properly, and we also know that in these cases the MD_EPN is also
> not set properly.
If MD_EPN is not set propely in the DTLB Miss, you are lost regardless of
my patch.

[SNIP]
> For a long time, we "solved" these problems on 8xx by not properly
> implementing copy-on-write pages and always marking pages dirty.
> It reduced the likelyhood of the problem, reduced the MMU overhead,
> but cost lots more in memory and processing time during forks.
> This still may not be a bad configuration option for selected
> applications.

Is it easy to patch/hard code? I would like to test the stability of my system.

> Like I said from the beginning, the 100% guaranteed working solution
> is to not use dcbz/dcbt on the 8xx.  Anything else will eventually
> stumble across the error conditions.  If I'm developing a product
> that uses Linux (or any other OS that will dynamically update the MMU
> and take DTLB exceptions) I won't take the risk of failure using
> these instructions.

I still that the patch to head_8xx.S has merit on its own. It will workaround
the case where DAR has an invalid value in the DTLB Miss handler.

Still, it would be nice if you could give the patch a go in your lab(with recent silicon).

Time to sleep for me
                                    Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> After reading this I guess I need to further clarify what happens.....
>
> Dan Malek wrote:
>
> > ... The reason it "works" for you is you found the
> > most likely case where you took a DTLB miss, followed by a DTLB Error
> > to the same page...
>
> What happens in these cases (which occurs like almost all of the time),
> is there is "leftover" information in the control/status registers from
> previous exceptions that causes the fault processing to take place
> properly.  In some error conditions, the leftover information isn't
> exactly correct, causing the page fault handler to look up wrong
> information or the dcbz/dcbt to do their thing on the wrong address.
> If you are not running a specific test case and precisely testing
> for the results of the operation, you may not notice this or at some
> time in the future notice erratic system behavior that is impossible
> to explain.  The "normal" instructions have exactly one well documented
> failure case, in the case of DTLBError, the MD_EPN sometimes doesn't
> get the proper status, but we know the DAR does, so we can copy the
> bits we need from the DAR to the EPN to properly process the fault.
> The cache instructions have never been normal instructions.  In some
> versions of silicon they won't generate a fault, some will allow cache
> operations to uncached spaces, some will fault but not provide the
> proper status in any MMU register.  Sometimes they work, it all depends
> upon the state of the cache line and TLB.  The cases most likely to
> fail are the ones that are the hardest to create.

So maybe there still is silicon bugs which is visible in the DTLB Error handler.
Is there a problem with MD_EPN in DTLB Miss context?
We do know that the dcxx instructions (including those that are used by
xxx_dcache_range functions) does not update the DAR register. You say that
the rfi depends on DAR(I would like to know in what way, I can't find any info
about that) so should we not try to make sure that DAR contains a valid address
by copying the EPN from MD_EPN?

I have another idea: If you get a DTLB Error direcly after DTLB Miss for the
same page, DAR will NOT be updated(at least in some cases) and you "inherit"
the DAR from the preceeding DTLB Miss. I have tried to verify that but I have
not been able to construct a test case that will prove this. Any ideas on that one?

Either way the patch makes it possible to use dcbz in user space, I can remove
the "copy DAR to MD_EPN" part in the DTLB Error handler, restore the Change bit in
the pte for kernel space and still have a usable system, wheras doing any one of
these modifications without my patch resulted in hard lookups during boot.

That suggests to me that the DTLB Miss patch does some good.
Can you present a case where will be worse off with the DTLB Miss part of
my patch applied?

> I've spent way too much time working on this on many versions of
> silicon with the assistance of people from Motorola.  It isn't a trivial
> problem to detect, quantify and document so someone else can repeat
> the exercise.  Through the first four major versions of 860 silicon
> (and countless minor versions) the wrong behavior I discussed was
> fixed in the next release, only to have something else unexpected appear.
> Since I didn't want a career in cache instruction debugging (and I had
> products to get to market), the solution was to simply not use these
> instructions.

I do understand your position, but please consider that the current impl.
may still have bugs or can be improved upon so please don't dismiss any
attempt to make the 8xx CPUs perform better under linux.

Again, you don't have to enable the use of dcbz(yet :-). The DTLB Miss modification
should IMHO go in regardless, unless you find a case were it does not work.

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> > For a long time, we "solved" these problems on 8xx by not properly
> > implementing copy-on-write pages and always marking pages dirty.
> > It reduced the likelyhood of the problem, reduced the MMU overhead,
> > but cost lots more in memory and processing time during forks.
> > This still may not be a bad configuration option for selected
> > applications.
>
> Is it easy to patch/hard code? I would like to test the stability of my system.

I am still intrested in trying copy-on-write but I can't figure out
how. Is it a big job? Any pointers?

  Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Till Straumann]

Hi guys.

I revitalized my old test environment on a
mpc860t (XPC860TZP50B5 / 3J21M to be precise).

I found that 'dcbz' (while failing to set DAR)
indeed sets MD_EPN correctly. Hence, Jocke's fix
(copy EPN[0:19]->DAR) would handle that.

My older idea (fixing up MD_EPN and DAR based
on the faulting instruction opcode and the involved
GPR contents) should work even if we have neither
a valid MD_EPN nor DAR.

-- Till.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Hi guys.

Thanks Till, you are the first one who actually tested
my fix.

> I revitalized my old test environment on a
> mpc860t (XPC860TZP50B5 / 3J21M to be precise).

hmm, B5 rev is pretty buggy. Good that it works on
this silicon as well.

>
> I found that 'dcbz' (while failing to set DAR)
> indeed sets MD_EPN correctly. Hence, Jocke's fix
> (copy EPN[0:19]->DAR) would handle that.

Did you try any of the other dcxx instructions as well?

> My older idea (fixing up MD_EPN and DAR based
> on the faulting instruction opcode and the involved
> GPR contents) should work even if we have neither
> a valid MD_EPN nor DAR.

Yes, I tried to do that. I never got past how to convert the
virtual address in SRR0 to physical so I could read the instruction
and decode it.

>
> -- Till.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Till Straumann wrote:

> I found that 'dcbz' (while failing to set DAR)
> indeed sets MD_EPN correctly. Hence, Jocke's fix
> (copy EPN[0:19]->DAR) would handle that.


After sleeping for a couple of days and consuming large
amounts of medicine to cure a cold, I think I understand
why copying these bits around seems to "fix" problems.

It's all related to the sequence of TLB miss/error exceptions
that I had been describing all along.  The first thing that
is going to most likely happen is you will get a TLB miss to
load a PTE  into the TLB.  It will be marked valid but not
dirty (not writable).  Immediately upon performing the rfi
you will get a TLB Error to handle the dirty PTE update.
By copying the bits from MD_EPN to the DAR in the miss handler,
the Error handler will have at least a 4K boundary aligned DAR
and it will execute correctly to update the dirty state.  At
this point, it will appear to "work" properly (even though
it is likely the dcbz didn't execute) because the system will
at least keep running (for a while).

If you have a situation where you get a TLB Error without
a matching TLB miss (very rare, but they can happen as the
result of swapping, copy on write, certain other page table
updates), then you are hosed.  The DAR will contain some information
from a previous exception, we will likely end up on a "hung"
system continually taking TLB Error exceptions because we
can't fix them properly.  This is basically what happens
without the bit copying "fix".


> My older idea (fixing up MD_EPN and DAR based
> on the faulting instruction opcode and the involved
> GPR contents) should work even if we have neither
> a valid MD_EPN nor DAR.

All of the TLB exception handlers must have minimal instructions.
The ones in Linux are too big already.  The very little you would
gain from making a dcbz/dcbt work correctly would be lost many,
many, many times over in a more complex TLB exception handler.

Copying bits from MD_EPN to DAR doesn't set the DAR "correctly",
it only gives you the page boundary.  This is going to further
confuse debuggers or signal handlers if you actually have an
addressing bug that is detected by one of these instructions.

The only update I would like to see to TLB exception handlers is
the removal of code due to streamlining of the page table organization.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

> I am still intrested in trying copy-on-write but I can't figure out
> how. Is it a big job? Any pointers?

The 8xx properly implements copy-on-write now.  I don't remember when
I updated this, maybe somewhere in the mid 2.3.xxx timeframe.  All of
the processor specific MM functions have to be changed, it isn't
something that is likely going to be a configuration option. due to
all of the common #define, macros, and page table functions.  It
does remove three instructions and a branch from the TLBmiss handler :-)


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Till Straumann wrote:
>
> > I found that 'dcbz' (while failing to set DAR)
> > indeed sets MD_EPN correctly. Hence, Jocke's fix
> > (copy EPN[0:19]->DAR) would handle that.
>
>
> After sleeping for a couple of days and consuming large
> amounts of medicine to cure a cold, I think I understand
> why copying these bits around seems to "fix" problems.

Sleeping is also good, I wish I got some more of that too :-)

>
> It's all related to the sequence of TLB miss/error exceptions
> that I had been describing all along.  The first thing that
> is going to most likely happen is you will get a TLB miss to
> load a PTE  into the TLB.  It will be marked valid but not
> dirty (not writable).  Immediately upon performing the rfi
> you will get a TLB Error to handle the dirty PTE update.
> By copying the bits from MD_EPN to the DAR in the miss handler,
> the Error handler will have at least a 4K boundary aligned DAR
> and it will execute correctly to update the dirty state.

Yes, that's the conclusion I came to too.

> At this point, it will appear to "work" properly (even though
> it is likely the dcbz didn't execute)

hmm, why would not dcbz execute properly? Surely the instruction is restarted
when the DTLB Miss/Error handler return?

> because the system will
> at least keep running (for a while).
>
> If you have a situation where you get a TLB Error without
> a matching TLB miss (very rare, but they can happen as the
> result of swapping, copy on write, certain other page table
> updates), then you are hosed.  The DAR will contain some information
> from a previous exception, we will likely end up on a "hung"
> system continually taking TLB Error exceptions because we
> can't fix them properly.  This is basically what happens
> without the bit copying "fix".

In a earlier mail you said the there was a case where the "normal" instructions
would not set MD_EPN in the DTLB Error handler and that it was documented.
What case is this and where can I find this documentation?

What this boils down to is that you don't known what register, DAR or MD_EPN to
trust in the DTLB Error handler, right?

Maybe Till Straumann's suggestion to set the DAR register to a known "bad" address
when leaving an exception. Then test for this bad address in the DTLB error handler
to decide what register to trust?

Note that the other dcxx instructions(according to Till) does not set DAR either
so if a dcbi ever causes a
 1) DTLB Miss followed by a DTLB Error
 2) DTLB Miss taking the DataAccess path
 3) DTLB Error taking the DataAccess path*
 4) DTLB Error not the DataAccess path*
* = I assume that these does not update DAR either, can anyone confirm this?
then you are also hosed. My patch will at least fix 1 and 2.

  Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> > I am still intrested in trying copy-on-write but I can't figure out
> > how. Is it a big job? Any pointers?
>
> The 8xx properly implements copy-on-write now.  I don't remember when
> I updated this, maybe somewhere in the mid 2.3.xxx timeframe.  All of
> the processor specific MM functions have to be changed, it isn't
> something that is likely going to be a configuration option. due to
> all of the common #define, macros, and page table functions.  It
> does remove three instructions and a branch from the TLBmiss handler :-)

Ahh, that explanins why I had such a hard time trying to find
where it didn't do copy-on-write :-)

 Thanks
         Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Till Straumann]

Dan Malek wrote:

> Till Straumann wrote:
>
>> I found that 'dcbz' (while failing to set DAR)
>> indeed sets MD_EPN correctly. Hence, Jocke's fix
>> (copy EPN[0:19]->DAR) would handle that.
>
>
>
> After sleeping for a couple of days and consuming large
> amounts of medicine to cure a cold, I think I understand
> why copying these bits around seems to "fix" problems.
>
> It's all related to the sequence of TLB miss/error exceptions
> that I had been describing all along.  The first thing that
> is going to most likely happen is you will get a TLB miss to
> load a PTE  into the TLB.  It will be marked valid but not
> dirty (not writable).  Immediately upon performing the rfi
> you will get a TLB Error to handle the dirty PTE update.
> By copying the bits from MD_EPN to the DAR in the miss handler,
> the Error handler will have at least a 4K boundary aligned DAR
> and it will execute correctly to update the dirty state.  At
> this point, it will appear to "work" properly (even though
> it is likely the dcbz didn't execute) because the system will
> at least keep running (for a while).

I'm not sure whether to agree with this analysis. I thought
you said that TLB Error correctly set DAR? Hence you would
get

 - TLB Miss, copy EPN to DAR (DAR is semi-correct, i.e. to
    a page boundary).
 - RFI, dcbz is restarted

--- here, task switching could occur ---

 - TLB Error because the page was not writable (but, according
   to your earlier mail, DAR should be correct now).
 - copy DAR-> EPN, lookup PTE, replace TLB
 - RFI, dcbz is restarted and eventually executed.

Under the premise that TLB Miss always sets EPN correctly
and TLB Error always sets DAR correctly, this should work
even if a task switch (or IRQ) sneaks in.

>
> If you have a situation where you get a TLB Error without
> a matching TLB miss (very rare, but they can happen as the
> result of swapping, copy on write, certain other page table
> updates), then you are hosed.

Again: I thought you said that TLB Error _does_ set DAR?

> The DAR will contain some information
> from a previous exception, we will likely end up on a "hung"
> system continually taking TLB Error exceptions because we
> can't fix them properly.  This is basically what happens
> without the bit copying "fix".
>
>
>> My older idea (fixing up MD_EPN and DAR based
>> on the faulting instruction opcode and the involved
>> GPR contents) should work even if we have neither
>> a valid MD_EPN nor DAR.
>
>
> All of the TLB exception handlers must have minimal instructions.
> The ones in Linux are too big already.  The very little you would
> gain from making a dcbz/dcbt work correctly would be lost many,
> many, many times over in a more complex TLB exception handler.

 From a performance standpoint, perhaps (although I believe that
working around the problem is more of a 'size' than execution path
length increase). But you gain the huge advantage that you don't
have to carefully screen any software you intend to port for cache
instructions.

>
> Copying bits from MD_EPN to DAR doesn't set the DAR "correctly",
> it only gives you the page boundary.  This is going to further
> confuse debuggers or signal handlers if you actually have an
> addressing bug that is detected by one of these instructions.

The 'fixup' workaround would provide a correct DAR, however.

-- Till

>
>
> The only update I would like to see to TLB exception handlers is
> the removal of code due to streamlining of the page table organization.
>
> Thanks.
>
>
>     -- Dan
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

> hmm, why would not dcbz execute properly? Surely the instruction is restarted
> when the DTLB Miss/Error handler return?

Write a test to verify it.  I know there were cases in the past when
it didn't work.  You would notice it when you expected to have some
zero initialized spaces that weren't.

> In a earlier mail you said the there was a case where the "normal" instructions
> would not set MD_EPN in the DTLB Error handler and that it was documented.
> What case is this and where can I find this documentation?

Well, it's more of an interpretation of the documentation and discussions
with Motorola engineers.

> What this boils down to is that you don't known what register, DAR or MD_EPN to
> trust in the DTLB Error handler, right?

The registers we are discussing belong to two distinct functional areas.
The Mx_xxx registers are all part of the mmu functional unit and can serve
to be hardware assist to the software TLB management.  The DAR, DSISR, SRRx
and so on are part of the PowerPC core, set during exceptions and used by
fault handlers to determine the course of action for fault repair.  The
latter function is PowerPC generic, the former is unique to the 8xx
implementation.

I do not know why DAR is sometimes not set properly during some types
cache operation exceptions.  In general, cache operation exceptions
have been unpredictable in various ways across all of the different
silicon revisions.  Sometimes a dcbz to uncached space was ignored,
sometimes it generated an alignment exception (which is the correct
behavior), sometimes it generated tlb error exceptions, for example.

In all cases except some cache instructions, the DAR is properly set
for tlb error exceptions.  It has to be or the system won't function
properly, so there isn't any question as to whether we can trust it.

For TLB miss exceptions, the Mx_xxx hardware assist registers work
properly, as documented, to minimize the number of instructions needed
to determine what PTE to load.  The TLB miss exception has exactly
one function, to as quickly as possible locate the PTE from the table
hierarchy and stuff the bits into the MMU.  That's it and it should
really be implemented in five lines of code.

The interpretation of the value of the Mx_xxx registers during TLB Error
exceptions has always been under discussion.  Sometimes you can read they
should work like the TLB miss case, but since we aren't processing a
TLB miss exception, perhaps they aren't set.  Perhaps when it appears to
work it is the result of residual information from a previous TLB miss,
other times we know it has been set as the result of the exception.  In
any case, we know if we copy the bits from the DAR into the MD_EPN, we
can then use the hardware assist of these registers to perform the PTE
lookup just like the TLB miss function.

We optimize the update of the usage and dirty indicators in the PTE
right in the TLB Error exception function to more closely emulate "real"
PPC processors.  This way, we can use the same fault processing functions.
If we didn't wish to do this, we could simply call an 8xx specific version
of do_fault, in which case it doesn't matter what is in the Mx_xxx registers
because we would do a software lookup using DAR.  It wouldn't be as efficient
and we couldn't use a common PPC function, but we wouldn't have to question
the workings of the silicon in this case.

So, we can always trust DAR, it's just that MD_EPN is left to your interpretation
of the documentation and depends upon the implementation of the functions.

> Maybe Till Straumann's suggestion to set the DAR register to a known "bad" address
> when leaving an exception. Then test for this bad address in the DTLB error handler
> to decide what register to trust?

Not necessary.

By not "correctly" tracking usage and dirty pages, as we did in the past,
it caused us to never generate DTLB Error exceptions unless it was truly
an access to a non-recoverable bad address or if the page was swapped out.
I would just set 'used' on every instruction page and 'used + dirty' on
every data page allocation.  A static embedded system would just converge on
this point anyway, but in more dynamic systems it cost more real memory.

Of course, the 8xx has custom cache management/control functions that
have always seemed to work properly.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Till Straumann]

Jocke.

There's of course a trivial test to check Dan's suspicion
that it might just be a lucky sequence of TLBMiss-TLBError
exceptions helping you out.

Just load some bogus values (a la 0xdeadbeef) into
MD_EPN and DAR shortly prior to returning from the
TLBMiss exception (but after walking the tables, of
course). Thereby you make sure that any 'leftover'
values seen by a subsequent TLBError are invalidated.

-- Till

Dan Malek wrote:

> Till Straumann wrote:
>
>> I found that 'dcbz' (while failing to set DAR)
>> indeed sets MD_EPN correctly. Hence, Jocke's fix
>> (copy EPN[0:19]->DAR) would handle that.
>
>
>
> After sleeping for a couple of days and consuming large
> amounts of medicine to cure a cold, I think I understand
> why copying these bits around seems to "fix" problems.
>
> It's all related to the sequence of TLB miss/error exceptions
> that I had been describing all along.  The first thing that
> is going to most likely happen is you will get a TLB miss to
> load a PTE  into the TLB.  It will be marked valid but not
> dirty (not writable).  Immediately upon performing the rfi
> you will get a TLB Error to handle the dirty PTE update.
> By copying the bits from MD_EPN to the DAR in the miss handler,
> the Error handler will have at least a 4K boundary aligned DAR
> and it will execute correctly to update the dirty state.  At
> this point, it will appear to "work" properly (even though
> it is likely the dcbz didn't execute) because the system will
> at least keep running (for a while).
>
> If you have a situation where you get a TLB Error without
> a matching TLB miss (very rare, but they can happen as the
> result of swapping, copy on write, certain other page table
> updates), then you are hosed.  The DAR will contain some information
> from a previous exception, we will likely end up on a "hung"
> system continually taking TLB Error exceptions because we
> can't fix them properly.  This is basically what happens
> without the bit copying "fix".
>
>
>> My older idea (fixing up MD_EPN and DAR based
>> on the faulting instruction opcode and the involved
>> GPR contents) should work even if we have neither
>> a valid MD_EPN nor DAR.
>
>
> All of the TLB exception handlers must have minimal instructions.
> The ones in Linux are too big already.  The very little you would
> gain from making a dcbz/dcbt work correctly would be lost many,
> many, many times over in a more complex TLB exception handler.
>
> Copying bits from MD_EPN to DAR doesn't set the DAR "correctly",
> it only gives you the page boundary.  This is going to further
> confuse debuggers or signal handlers if you actually have an
> addressing bug that is detected by one of these instructions.
>
> The only update I would like to see to TLB exception handlers is
> the removal of code due to streamlining of the page table organization.
>
> Thanks.
>
>
>     -- Dan
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Till Straumann wrote:

> - TLB Miss, copy EPN to DAR (DAR is semi-correct, i.e. to
>    a page boundary).
> - RFI, dcbz is restarted

The EPN has never been copied to DAR.  Some bits of DAR are copied
to EPN so we can use the hardware assist in the TLB Error exception.

>
> --- here, task switching could occur ---

No, it can't because the TLB Miss rfi is likely to immediately result in
a TLB Error.  The TLB Miss doesn't return through a path that causes
a context switch.

> Under the premise that TLB Miss always sets EPN correctly
> and TLB Error always sets DAR correctly, this should work
> even if a task switch (or IRQ) sneaks in.

The current implementation relies on TLB Miss always setting EPN
(and other hardware assist registers) correctly, and TLB Error
relies on DAR being set correctly.  Everything works fine until
you get a TLB Error on a dcbz/dcbt that doesn't properly update DAR.
In this case EPN isn't set correctly either.

> Again: I thought you said that TLB Error _does_ set DAR?

It does, except in the case of some cache instruction exceptions.

> ... But you gain the huge advantage that you don't
> have to carefully screen any software you intend to port for cache
> instructions.

The instructions that don't work properly are dcbz/dcbt.  We have
corrected this in the kernel and all GNU libraries long ago.  There
are other reasons to need a unique 8xx (and 4xx) set of libraries
other than the use of these cache instructions.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Till Straumann]

Here are the results of some tests I ran
on my mpc860t (XPC860TZP50B5 / 3J21M).

I use mpc8bug on a FADS board to load motorola's
'init860.S' example. The example was slightly
modified to include (1-4 use differend EAs, of
course).

  1) a 1:1 TLB to DRAM, R/W access (already present
     in the vanilla code)

  2) a TLB to map DRAM r/w, caching inhibited

  3) a TLB to map DRAM r/o (behavior is the same
     if for a 'real RO' mapping and for a RW
     mapping with the 'CHANGE' bit cleared).

  4) a TLB to map DRAM r/w but with the VALID
     bit clear.

  5) a 'main' routine which executes the
     instructions under test through one
     of the aforementioned TLB mappings.

Here's what I found (I generally load bogus
values into MD_EPN/DAR prior to executing any
of the cache instructions under test)

A) All of dcbf, dcbi, dcbst, dcbz (dcbt and dcbtst
    should not and do not) raise a TLBMiss exception
    with MD_EPN set correctly but failing to set DAR
    (i.e. DAR is left unmodified) when trying to
    access an address not in any of the TLBs.

B) All of dcbf, dcbi, dcbst, dcbz raise a TLBError
    exception when accessing mapping 4) [as they should].

    Unlike regular loads/stores (non-cache instructions),
    DAR is NOT SET, however (i.e. left unmodified).
    MD_EPN is set to 0x00000000 which does NOT point to
    the faulting EA.

C) dcbi and dcbz behave like B) when accessing
    through 3) [RO mapping]. Note that dcbf and
    dcbst are treated like 'loads' and hence do
    not raise a TLBError due to protection violation
    (in accordance with the docs).

D) dcbz through a cache inhibited mapping (2)) does
    not generate an alignment exception but
    transparently clears the memory (either is allowed
    according to the PPC specs).

Note: dcbt/dcbtst were included in all of the
tests but they never raise an exception which is
the correct behavior.

CONCLUSION:

  - the only correct workaround for TLBError
    is the one I suggested earlier: TLBError
    handler has to inspect the faulting opcode
    and fixup DAR and MD_EPN based on the GPR
    values if the faulting instruction is any
    of dcbf, dcbi, dcbst or dcbz.
    Performance of this solution could be
    improved (eliminate opcode-check in the
    vast majority of the cases) by storing
    a 'tag' value in DAR.

  - the aforementioned workaround also could handle
    TLBMiss - although performance is more of an issue
    there.
    Alternatives are
       a) Jocke's workaround: copy MD_EPN -> DAR
          (disadvantage: DAR truncated to page
          boundary).
       b) like a) but do this only if the faulting
          instruction is any dcbxx (disadvantage:
          performance loss).
       c) merge significant bits of MD_EPN into
          DAR. Thus, only cache instructions suffer
          from an incorrect page offset in DAR.
          At the same time, it's cheap.

Based on the analysis, I'd suggest to implement
the 'fixup' workaround for TLBError and alternative
c) for TLBMiss

-- Till

PS. due to unclear re-distribution terms of init860.S,
I refrain from appending it to this message. Send me
email for details about the test software.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Till Straumann]

My earlier message was not completely
correct. The TLBError exception raised
by any of the cache instructions
(dcbf, dcbst, dcbi, dcbz) does actually
leave the EPN bits in MD_EPN unmodified,
clears the reserved bits and sets CASID.

Here are the (corrected) results of some tests I ran
on my mpc860t (XPC860TZP50B5 / 3J21M).

I use mpc8bug on a FADS board to load motorola's
'init860.S' example. The example was slightly
modified to include (1-4 use differend EAs, of
course).

   1) a 1:1 TLB to DRAM, R/W access (already present
      in the vanilla code)

   2) a TLB to map DRAM r/w, caching inhibited

   3) a TLB to map DRAM r/o (behavior is the same
      if for a 'real RO' mapping and for a RW
      mapping with the 'CHANGE' bit cleared).

   4) a TLB to map DRAM r/w but with the VALID
      bit clear.

   5) a 'main' routine which executes the
      instructions under test through one
      of the aforementioned TLB mappings.

Here's what I found (I generally load bogus
values into MD_EPN/DAR prior to executing any
of the cache instructions under test)

A) All of dcbf, dcbi, dcbst, dcbz (dcbt and dcbtst
     should not and do not) raise a TLBMiss exception
     with MD_EPN set correctly but failing to set DAR
     (i.e. DAR is left unmodified) when trying to
     access an address not in any of the TLBs.

B) All of dcbf, dcbi, dcbst, dcbz raise a TLBError
     exception when accessing mapping 4) [as they should].

     Unlike regular loads/stores (non-cache instructions),
     NEITHER DAR not MD_EPN are set, however. DAR is left
     unmodified MD_EPN has the CASID and reserved bits set/reset
     but leaves the EPN bits unmodified.

C) dcbi and dcbz behave like B) when accessing
     through 3) [RO mapping]. Note that dcbf and
     dcbst are treated like 'loads' and hence do
     not raise a TLBError due to protection violation
     (in accordance with the docs).

D) dcbz through a cache inhibited mapping (2)) does
     not generate an alignment exception but
     transparently clears the memory (either is allowed
     according to the PPC specs).

Note: dcbt/dcbtst were included in all of the
tests but they never raise an exception which is
the correct behavior.

CONCLUSION:

   - the only correct workaround for TLBError
     is the one I suggested earlier: TLBError
     handler has to inspect the faulting opcode
     and fixup DAR and MD_EPN based on the GPR
     values if the faulting instruction is any
     of dcbf, dcbi, dcbst or dcbz.
     Performance of this solution could be
     improved (eliminate opcode-check in the
     vast majority of the cases) by storing
     a 'tag' value in DAR.

   - the aforementioned workaround also could handle
     TLBMiss - although performance is more of an issue
     there.
     Alternatives are
        a) Jocke's workaround: copy MD_EPN -> DAR
           (disadvantage: DAR truncated to page
           boundary).
        b) like a) but do this only if the faulting
           instruction is any dcbxx (disadvantage:
           performance loss).
        c) merge significant bits of MD_EPN into
           DAR. Thus, only cache instructions suffer
           from an incorrect page offset in DAR.
           At the same time, it's cheap.

Based on the analysis, I'd suggest to implement
the 'fixup' workaround for TLBError and alternative
c) for TLBMiss

-- Till

PS. due to unclear re-distribution terms of init860.S,
I refrain from appending it to this message. Send me
email for details about the test software.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

Great job Till!

I have forwarded this to Motorola. Primarly I want to know if never silicon
behaves the same way.

> Here are the (corrected) results of some tests I ran
> on my mpc860t (XPC860TZP50B5 / 3J21M).
>
> I use mpc8bug on a FADS board to load motorola's
> 'init860.S' example. The example was slightly
> modified to include (1-4 use differend EAs, of
> course).
>
>    1) a 1:1 TLB to DRAM, R/W access (already present
>       in the vanilla code)
>
>    2) a TLB to map DRAM r/w, caching inhibited
>
>    3) a TLB to map DRAM r/o (behavior is the same
>       if for a 'real RO' mapping and for a RW
>       mapping with the 'CHANGE' bit cleared).
>
>    4) a TLB to map DRAM r/w but with the VALID
>       bit clear.

Level1 or Level2 valid bit? Does it matter which one?

>
>    5) a 'main' routine which executes the
>       instructions under test through one
>       of the aforementioned TLB mappings.

hmm, I wonder if the Guarded bit would change anything.

>
> Here's what I found (I generally load bogus
> values into MD_EPN/DAR prior to executing any
> of the cache instructions under test)
>
> A) All of dcbf, dcbi, dcbst, dcbz (dcbt and dcbtst
>      should not and do not) raise a TLBMiss exception
>      with MD_EPN set correctly but failing to set DAR
>      (i.e. DAR is left unmodified) when trying to
>      access an address not in any of the TLBs.
>
> B) All of dcbf, dcbi, dcbst, dcbz raise a TLBError
>      exception when accessing mapping 4) [as they should].
>
>      Unlike regular loads/stores (non-cache instructions),
>      NEITHER DAR not MD_EPN are set, however. DAR is left
>      unmodified MD_EPN has the CASID and reserved bits set/reset
>      but leaves the EPN bits unmodified.
>
> C) dcbi and dcbz behave like B) when accessing
>      through 3) [RO mapping]. Note that dcbf and
>      dcbst are treated like 'loads' and hence do
>      not raise a TLBError due to protection violation
>      (in accordance with the docs).
>
> D) dcbz through a cache inhibited mapping (2)) does
>      not generate an alignment exception but
>      transparently clears the memory (either is allowed
>      according to the PPC specs).
>
> Note: dcbt/dcbtst were included in all of the
> tests but they never raise an exception which is
> the correct behavior.
>
> CONCLUSION:
>
>    - the only correct workaround for TLBError
>      is the one I suggested earlier: TLBError
>      handler has to inspect the faulting opcode
>      and fixup DAR and MD_EPN based on the GPR
>      values if the faulting instruction is any
>      of dcbf, dcbi, dcbst or dcbz.
>      Performance of this solution could be
>      improved (eliminate opcode-check in the
>      vast majority of the cases) by storing
>      a 'tag' value in DAR.

Yes, but I can't even figure out how to read the instruction. Address translation
is off so how do I figure out the real address?

hmm, what does mfspr r20, M_TWB return when MD_EPN contain old EPN? The old EPN in
MD_EPN or does M_TWB know real address that generated the fault?

>
>    - the aforementioned workaround also could handle
>      TLBMiss - although performance is more of an issue
>      there.
>      Alternatives are
>         a) Jocke's workaround: copy MD_EPN -> DAR
>            (disadvantage: DAR truncated to page
>            boundary).
>         b) like a) but do this only if the faulting
>            instruction is any dcbxx (disadvantage:
>            performance loss).
>         c) merge significant bits of MD_EPN into
>            DAR. Thus, only cache instructions suffer
>            from an incorrect page offset in DAR.
>            At the same time, it's cheap.
>
> Based on the analysis, I'd suggest to implement
> the 'fixup' workaround for TLBError and alternative
> c) for TLBMiss

The TLB Miss workaround only needs to be in the slow path(DataAccess path) and
yes, it should be alternative c).


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> > hmm, why would not dcbz execute properly? Surely the instruction is restarted
> > when the DTLB Miss/Error handler return?
>
> Write a test to verify it.  I know there were cases in the past when
> it didn't work.  You would notice it when you expected to have some
> zero initialized spaces that weren't.

I think Till's superb analysis show that this is not a problem.

>
> > In a earlier mail you said the there was a case where the "normal" instructions
> > would not set MD_EPN in the DTLB Error handler and that it was documented.
> > What case is this and where can I find this documentation?
>
> Well, it's more of an interpretation of the documentation and discussions
> with Motorola engineers.

So this case is mere speculation and has never been proven?

[SNIP]
> For TLB miss exceptions, the Mx_xxx hardware assist registers work
> properly, as documented, to minimize the number of instructions needed
> to determine what PTE to load.  The TLB miss exception has exactly
> one function, to as quickly as possible locate the PTE from the table
> hierarchy and stuff the bits into the MMU.  That's it and it should
> really be implemented in five lines of code.

Five lines of code, that I would like to see :-) Seriously, why do the
DTLB handlers have two tables, one for kernel space and one for the rest?

Can the DataAccess path be removed in the DTLB Miss handler?

[SNIP]
> > Maybe Till Straumann's suggestion to set the DAR register to a known "bad" address
> > when leaving an exception. Then test for this bad address in the DTLB error handler
> > to decide what register to trust?
>
> Not necessary.
>
> By not "correctly" tracking usage and dirty pages, as we did in the past,
> it caused us to never generate DTLB Error exceptions unless it was truly
> an access to a non-recoverable bad address or if the page was swapped out.
> I would just set 'used' on every instruction page and 'used + dirty' on
> every data page allocation.  A static embedded system would just converge on
> this point anyway, but in more dynamic systems it cost more real memory.

hmm, don't you loose copy-on-write by doing so?

Would the suggested instruction decoding in the DLTB Error handler
be too ugly for your liking?
Since dcbi also causes DTLB Errors with undefined values in DAR and MD_EPN it
might be a good idea.

 Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> The current implementation relies on TLB Miss always setting EPN
> (and other hardware assist registers) correctly, and TLB Error
> relies on DAR being set correctly.  Everything works fine until
> you get a TLB Error on a dcbz/dcbt that doesn't properly update DAR.
> In this case EPN isn't set correctly either.

I just want to point out(again) that TLB Miss caused by most of the
dcxx(dcbf, dcbi, dcbst, dcbz) instructions that end up in the slow path(DataAccess)
don't work with the current impl. since DAR isn't set. The code fragment below
will fix that(from my earlier patch). This won't affect the fast path at all
since it all of it can be in the slow path.

+	/* Copy 20 msb from EPN to DAR since the dcxx instructions fails
+	 * update the DAR when they cause a DTLB Miss.
+	 */
+	mfspr	r21, MD_EPN
+	rlwinm	r21, r21, 0, 0, 19
+	mfspr	r20, DAR
+	rlwinm	r20, r20, 0, 20, 31
+	or	r20, r20, r21
+	mtspr	DAR, r20

It is hard to end up in slow path for kernel space addresses, but
it's not impossible I guess. Comments?

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

> CONCLUSION:
>
>    - the only correct workaround for TLBError
>      is the one I suggested earlier: TLBError
>      handler has to inspect the faulting opcode
>      and fixup DAR and MD_EPN based on the GPR
>      values if the faulting instruction is any
>      of dcbf, dcbi, dcbst or dcbz.
>      Performance of this solution could be
>      improved (eliminate opcode-check in the
>      vast majority of the cases) by storing
>      a 'tag' value in DAR.

Hi again

I have been hacking on dcxx address decoder. Since assembler isn't my cup of tea,
I used C mixed with asm statements. The resulting assembler isn't too bad either IMHO.

To load the instruction into r21 I used:
	mfspr	r20,SRR0
	andis.	r21, r20, 0x8000
	beq	56f

	tophys(r21, r20)
	lwz r21,0(r21)
56:
This only works for kernel space addresses. I can't figure out how to get to user space as well.
I can live without user space anyway.

I am still thinking about the 'tag'. Since MD_EPN isn't set as well as DAR I thinking about
storing a tag in MD_EPN instead. It's less intrusive. Maybe it is enough to look at the
valid bit in MD_EPN?

What do you think so far?
Oh, this should go into the DTLB Error handler.

     Jocke
======================= dcxx address decoder =========================

#define	RA(inst)	(((inst) & 0x001F0000) >> 16)
#define	RA_MASK	        0x001F0000
#define	RB(inst)	(((inst) & 0x0000F800) >> 11)
#define	RB_MASK	        0x0000F800

/* Compile with ppc_8xx-gcc -S -O2 -mregnames dcbz.c -fcall-used-r20 -fcall-used-r21
 * to see the resulting assembler */
/* Assumes dcxx instruction in reg 21 when called */
decode_dcxx_and_sum(void)
{
  register unsigned long r21 asm("r21"); /* make it live in reg 21 */
  register unsigned long r20 asm("r20"); /* make it live in reg 20 */

  asm("mfctr %0\n\t"
      "stw %0, 16(0)":: "r" (r20)); /* save ctr reg on stack */
  r20 = RB(r21) * 8; /* offset into jump table for reg RB */
  asm("addi %0, %0, 100f@l":: "r" (r20)); /* add start of table */
  asm("mtctr %0":: "r" (r20)); /* load ctr with jump address */
  r20 = 0; /* sum starts at zero */
  asm("bctr"::); /* jump into table */
  /* Below is the jump table. */
  asm("100:\n\t"
      "add %0, %0, 0\n\t"
      "b 99f\n\t"
      "add %0, %0, 1\n\t"
      "b 99f\n\t"
      "add %0, %0, 2\n\t"
      "b 99f\n\t"
      "add %0, %0, 3\n\t"
      "b 99f\n\t"
      "add %0, %0, 4\n\t"
      "b 99f\n\t"
      "add %0, %0, 5\n\t"
      "b 99f\n\t"
      "add %0, %0, 6\n\t"
      "b 99f\n\t"
      "add %0, %0, 7\n\t"
      "b 99f\n\t"
      "add %0, %0, 8\n\t"
      "b 99f\n\t"
      "add %0, %0, 9\n\t"
      "b 99f\n\t"
      "add %0, %0, 10\n\t"
      "b 99f\n\t"
      "add %0, %0, 11\n\t"
      "b 99f\n\t"
      "add %0, %0, 12\n\t"
      "b 99f\n\t"
      "99:\n\t"
      : : "r" (r20));
  r21 = RA(r21) * 8; /* offset into jump table for reg RA */
  if(r21){ /* if reg zero, don't add it */
    asm("addi %0, %0, 100b@l":: "r" (r21)); /* add start of table */
    asm("mtctr %0":: "r" (r21)); /* load ctr with jump address */
    r21 &= ~RA_MASK; /* make sure we don't execute this mre than once */
    asm("bctr":: "r" (r21)); /* jump into table */
  }
  asm("mtdar %0": : "r" (r20)); /* save sum to DAR */
  asm("lwz %0, 16(0)\n\t"
      "mtctr %0" :: "r" (r21)); /* restore ctr reg from stack */
}

This is the resulting assembler:

	.file	"dcbz.c"
gcc2_compiled.:
	.section	".text"
	.align 2
	.globl decode_dcxx_and_sum
	.type	 decode_dcxx_and_sum,@function
decode_dcxx_and_sum:
	mfctr %r20
	stw %r20, 16(0)
	rlwinm %r20,%r21,24,24,28
	addi %r20, %r20, 100f@l
	mtctr %r20
	li %r20,0
	bctr
	100:
	add %r20, %r20, 0
	b 99f
	add %r20, %r20, 1
	b 99f
	add %r20, %r20, 2
	b 99f
	add %r20, %r20, 3
	b 99f
	add %r20, %r20, 4
	b 99f
	add %r20, %r20, 5
	b 99f
	add %r20, %r20, 6
	b 99f
	add %r20, %r20, 7
	b 99f
	add %r20, %r20, 8
	b 99f
	add %r20, %r20, 9
	b 99f
	add %r20, %r20, 10
	b 99f
	add %r20, %r20, 11
	b 99f
	add %r20, %r20, 12
	b 99f
	99:

	rlwinm. %r21,%r21,19,24,28
	bc 12,2,.L3
	addi %r21, %r21, 100b@l
	mtctr %r21
	rlwinm %r21,%r21,0,16,10
	bctr
.L3:
	mtdar %r20
	lwz %r21, 16(0)
	mtctr %r21
	blr
.Lfe1:
	.size	 decode_dcxx_and_sum,.Lfe1-decode_dcxx_and_sum
	.ident	"GCC: (GNU) 2.95.3 20010315 (release/MontaVista)"


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> > The current implementation relies on TLB Miss always setting EPN
> > (and other hardware assist registers) correctly, and TLB Error
> > relies on DAR being set correctly.  Everything works fine until
> > you get a TLB Error on a dcbz/dcbt that doesn't properly update DAR.
> > In this case EPN isn't set correctly either.
>
> I just want to point out(again) that TLB Miss caused by most of the
> dcxx(dcbf, dcbi, dcbst, dcbz) instructions that end up in the slow path(DataAccess)
> don't work with the current impl. since DAR isn't set. The code fragment below
> will fix that(from my earlier patch). This won't affect the fast path at all
> since it all of it can be in the slow path.
>
> +	/* Copy 20 msb from EPN to DAR since the dcxx instructions fails
> +	 * update the DAR when they cause a DTLB Miss.
> +	 */
> +	mfspr	r21, MD_EPN
> +	rlwinm	r21, r21, 0, 0, 19
> +	mfspr	r20, DAR
> +	rlwinm	r20, r20, 0, 20, 31
> +	or	r20, r20, r21
> +	mtspr	DAR, r20
>
> It is hard to end up in slow path for kernel space addresses, but
> it's not impossible I guess. Comments?

hmm, no response from the maintainer(s). You don't agree?

Did another test. I tagged DAR with a bad address(0xdead0000) in
the TLB Miss handler(after the mtspr MD_RPN, r20) shortly before
rfi.

Now the kernel won't boot. It turns out it's consistent_alloc that is
doing a invalidate_dcache_range(). According to a comment, memory is allocated
from the vmalloc memory space. Doing a invalidate_dcache_range() on this memory
causes a DTLB error with the change bit cleared and since DAR isn't set by
dcbi, linux hangs.

Clearly, the DTLB exeception procedure still is buggy.

   Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: dcbz works on 862 everywhere!

[Dan Malek]

Joakim Tjernlund wrote:

> hmm, no response from the maintainer(s). You don't agree?

It's interesting to watch these hacks, but I can't justify
complicating a general purpose function with more bus cycles by
emulating a functional problem.  By not using these instructions
we have a working system that costs just a few more cycles during
the memory copy/zero operations.  If we had _working_ dcbz
instructions, it would be a gain to use them, but from a system
perspective it is going to cost more to "fix up" these than
the code that already exists.

As I said in the past, I'm sensitive to the code in the TLB exception
processing.  So do something to remove code and streamline the process
and I'm really interested.  Do something to add more code and it's
going to get placed pretty low in my pile of things to do.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> > hmm, no response from the maintainer(s). You don't agree?
>
> It's interesting to watch these hacks, but I can't justify
> complicating a general purpose function with more bus cycles by
> emulating a functional problem.  By not using these instructions
> we have a working system that costs just a few more cycles during
> the memory copy/zero operations.  If we had _working_ dcbz
> instructions, it would be a gain to use them, but from a system
> perspective it is going to cost more to "fix up" these than
> the code that already exists.

Look a little harder. I am not talking just about dcbz, dcbi has
the same problem. As it stands today, there are real bugs w.r.t dcbi.

consistent_alloc() causes DTLB errors with a incorrect address in DAR.
Just force DAR to an invalid address in the DTLB Miss handler before
you leave it and see if you kernel boots.

> As I said in the past, I'm sensitive to the code in the TLB exception
> processing.  So do something to remove code and streamline the process
> and I'm really interested.  Do something to add more code and it's
> going to get placed pretty low in my pile of things to do.

Yes I know, but you are not reading what am writing. For instance,
the change below does not add to the fast path of the DTLB Miss handler and
it fixes a real problem:
> I just want to point out(again) that TLB Miss caused by most of the
> dcxx(dcbf, dcbi, dcbst, dcbz) instructions that end up in the slow path(DataAccess)
> don't work with the current impl. since DAR isn't set. The code fragment below
> will fix that(from my earlier patch). This won't affect the fast path at all
> since it all of it can be in the slow path.
>
> +	/* Copy 20 msb from EPN to DAR since the dcxx instructions fails
> +	 * update the DAR when they cause a DTLB Miss.
> +	 */
> +	mfspr	r21, MD_EPN
> +	rlwinm	r21, r21, 0, 0, 19
> +	mfspr	r20, DAR
> +	rlwinm	r20, r20, 0, 20, 31
> +	or	r20, r20, r21
> +	mtspr	DAR, r20

Go back and think dcbi instead of dcbz.

     Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

> > CONCLUSION:
> >
> >    - the only correct workaround for TLBError
> >      is the one I suggested earlier: TLBError
> >      handler has to inspect the faulting opcode
> >      and fixup DAR and MD_EPN based on the GPR
> >      values if the faulting instruction is any
> >      of dcbf, dcbi, dcbst or dcbz.
> >      Performance of this solution could be
> >      improved (eliminate opcode-check in the
> >      vast majority of the cases) by storing
> >      a 'tag' value in DAR.
>
> Hi again
>
> I have been hacking on dcxx address decoder. Since assembler isn't my cup of tea,
> I used C mixed with asm statements. The resulting assembler isn't too bad either IMHO.
>
> To load the instruction into r21 I used:
> 	mfspr	r20,SRR0
> 	andis.	r21, r20, 0x8000
> 	beq	56f
>
> 	tophys(r21, r20)
> 	lwz r21,0(r21)
> 56:
> This only works for kernel space addresses. I can't figure out how to get to user space as well.
> I can live without user space anyway.
>
> I am still thinking about the 'tag'. Since MD_EPN isn't set as well as DAR I thinking about
> storing a tag in MD_EPN instead. It's less intrusive. Maybe it is enough to look at the
> valid bit in MD_EPN?
>
> What do you think so far?
> Oh, this should go into the DTLB Error handler.

Me again :-)

I have completed and tested my workaround for the dcbx instructions. The workaround
handles ALL dcbx instructions, ANY register combination and works both on
kernel space and user space addresses.

During my testing I noticed that memory allocated with
consistent_alloc() or kmalloc() causes TLB Errors while vmalloc does not.
If this is true(confirmation wanted) it means that the current impl. is fragile.
Both dcbi and dcbz does not update DAR when a TLB error happens, instead the
the previous setting of DAR is used.

I also did some benchmarking using copy_page(dcbz enabled) and memcpy to
memory allocated with kmalloc and/or vmalloc. copy_page is about 30% faster
than memcpy even with the workaround applied.

There is one concern left. I tag DAR with a "bad address" just before
an exception is finished. In the TLB Error handler, check if DAR contains
the "bad address" and if it does then the workaround is executed.

I need find all exceptions where DAR is modified. Currently I tag DAR in
STD_EXCEPTON(), DataAccess, Alignment, DataStore and DataTLBError. Have I
missed any exception?

I also need to find a good value for the "bad address". Currently I use
0xdead0000 and that's probably not the best value.
Tagging with this value is a two instruction operation:
   lis r20, 0xdead
   mtspr DAR, r20

The test in the TLB Error handler look like this:
  mfspr r21, DAR
  lis   r20, 0xdead
  cmpw	cr0, r20, r21
  beq-  <workaround address>

I can not see any reason NOT to add this to the BK tree(after some minor modifications
mentioned above and a little cleanup). It fixes a real problem
with dcbi and as a bonus you can use dcbz as well since it has the same problem
that dcbi has and the fix is generic for all dcbx instructions. Dan's argument,
"It's interesting to watch these hacks, but I can't justify
 complicating a general purpose function with more bus cycles by
 emulating a functional problem.  By not using these instructions
 we have a working system that costs just a few more cycles during
 the memory copy/zero operations.  If we had _working_ dcbz
 instructions, it would be a gain to use them, but from a system
 perspective it is going to cost more to "fix up" these than
 the code that already exists", is not valid. This is not only about
optimization, but also about correctness of existing use of dcbi.

Patch against 2.4.20 devel available on request until I have cleaned it up a little.

   Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Till Straumann]

Joakim Tjernlund wrote:

Why not tag MD_EPN with an unused CASID?

-- Till


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> Why not tag MD_EPN with an unused CASID?

That won't work, will it? Your test report said that the CASID
bits were set/reset by dcbi and dcbz? Did I missunderstand something?

Also, the tagging procedure will require more than 2 instructions, I think.

I tried to use the VALID bit in MD_EPN, but that does not work.

One thing I noticed is that I can remove M_CASID and M_EVALID copy in the DTLB Error
handler without any visible problems. Why are they needed?

 Jocke
>
> -- Till


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

Hi again

I am wonder if it is possible to reduce the number of instructions in:

  mfspr r20, DAR
  lis   r21, 0xdead
  cmpw  cr0, r20, r21
  beq-  100f

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

>
> Patch against 2.4.20 devel available on request until I have cleaned it up a little.
>
>    Jocke

Here is the patch, against 2.4.20 devel but I think it will apply
against BK HEAD. Comments welcome :-)

 Jocke
Index: arch/ppc/kernel/head_8xx.S
===================================================================
RCS file: /home/cvsadmin/cvsroot/kernel/linuxppc/arch/ppc/kernel/head_8xx.S,v
retrieving revision 1.1
diff -u -r1.1 head_8xx.S
--- arch/ppc/kernel/head_8xx.S	1 Nov 2002 13:44:02 -0000	1.1
+++ arch/ppc/kernel/head_8xx.S	8 Apr 2003 17:21:08 -0000
@@ -34,6 +34,24 @@
 #include <asm/ppc_asm.h>
 #include "ppc_defs.h"

+/* These macros are used to tag DAR with a known value so that the
+ * DataTLBError can recognize a buggy dcbx instruction and workaround
+ * the problem.
+ */
+#define TAG_VAL 0xdead
+#define TAG_DAR_R20 	\
+	lis	r20, TAG_VAL;\
+	mtspr	DAR, r20;
+
+/* Macro to make the code more readable. */
+#ifdef CONFIG_8xx_CPU6
+  #define DO_8xx_CPU6(val, reg) \
+	li	reg, val; \
+	stw	reg, 12(r0); \
+	lwz	reg, 12(r0);
+#else
+  #define DO_8xx_CPU6(val, reg)
+#endif
 	.text
 	.globl	_stext
 _stext:
@@ -163,6 +181,7 @@
 	. = n;					\
 label:						\
 	EXCEPTION_PROLOG;			\
+	TAG_DAR_R20;				\
 	addi	r3,r1,STACK_FRAME_OVERHEAD;	\
 	li	r20,MSR_KERNEL;			\
 	FINISH_EXCEPTION(hdlr)
@@ -185,6 +204,7 @@
 	mr	r5,r20
 	mfspr	r4,DAR
 	stw	r4,_DAR(r21)
+	TAG_DAR_R20
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	li	r20,MSR_KERNEL
 	rlwimi	r20,r23,0,16,16		/* copy EE bit from saved MSR */
@@ -223,6 +243,7 @@
 	EXCEPTION_PROLOG
 	mfspr	r4,DAR
 	stw	r4,_DAR(r21)
+	TAG_DAR_R20
 	mfspr	r5,DSISR
 	stw	r5,_DSISR(r21)
 	addi	r3,r1,STACK_FRAME_OVERHEAD
@@ -454,6 +475,7 @@
 #endif
 	mtspr	MD_RPN, r20	/* Update TLB entry */

+	TAG_DAR_R20
 	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
@@ -463,7 +485,17 @@
 #endif
 	rfi

-2:	mfspr	r20, M_TW	/* Restore registers */
+2:
+	/* Copy 20 msb from MD_EPN to DAR since the dcxx instructions fails
+	 * to update DAR when they cause a DTLB Miss.
+	 */
+	mfspr	r21, MD_EPN
+	rlwinm	r21, r21, 0, 0, 19
+	mfspr	r20, DAR
+	rlwinm	r20, r20, 0, 20, 31
+	or	r20, r20, r21
+	mtspr	DAR, r20
+	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
 	lwz	r21, 4(r0)
@@ -501,10 +533,18 @@
 	stw	r20, 0(r0)
 	stw	r21, 4(r0)

+	/* If DAR contains TAG_VAL implies a buggy dcbx instruction
+	 * than did not set DAR.
+	 */
+	mfspr	r20, DAR
+	lis	r21, TAG_VAL
+	cmpw	cr0, r20, r21
+	beq-	100f	/* Branch if TAG_VAL to dcbx workaround procedure */
+101: /* return from dcbx instruction bug workaround, r20 holds value of DAR */
 	/* First, make sure this was a store operation.
 	*/
-	mfspr	r20, DSISR
-	andis.	r21, r20, 0x0200	/* If set, indicates store op */
+	mfspr	r21, DSISR
+	andis.	r21, r21, 0x0200	/* If set, indicates store op */
 	beq	2f

 	/* The EA of a data TLB miss is automatically stored in the MD_EPN
@@ -523,7 +563,7 @@
 	 * are initialized in mapin_ram().  This will avoid the problem,
 	 * assuming we only use the dcbi instruction on kernel addresses.
 	 */
-	mfspr	r20, DAR
+	/* DAR is in r20 already */
 	rlwinm	r21, r20, 0, 0, 19
 	ori	r21, r21, MD_EVALID
 	mfspr	r20, M_CASID
@@ -588,6 +628,7 @@
 #endif
 	mtspr	MD_RPN, r20	/* Update TLB entry */

+	TAG_DAR_R20
 	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
@@ -624,6 +665,146 @@
 	STD_EXCEPTION(0x1f00, Trap_1f, UnknownException)

 	. = 0x2000
+/* This is the workaround procedure to calculate the data EA for a buggy dcbx instruction
+ * by decoding the registers used by the dcbx instruction and adding them.
+ * DAR is set to the calculated address and r20 also holds the EA on exit.
+ */
+139:/* fetch instruction from userspace memory */
+	DO_8xx_CPU6(0x3780, r3)
+	mtspr	MD_EPN, r20	/* Have to use MD_EPN for walk, MI_EPN can't */
+	mfspr	r21, M_TWB	/* Get level 1 table entry address */
+	lwz	r21, 0(r21)	/* Get the level 1 entry */
+	tophys  (r21, r21)
+	DO_8xx_CPU6(0x3b80, r3)
+	mtspr	MD_TWC, r21	/* Load pte table base address */
+	mfspr	r21, MD_TWC	/* ....and get the pte address */
+	lwz	r21, 0(r21)	/* Get the pte */
+	/* concat physical page address and page offset */
+	rlwinm	r21, r21, 0, 0, 19
+	rlwinm	r20, r20, 0, 20, 31
+	or	r21, r20, r21
+	b	140f
+100:/* Entry point for dcbx workaround. */
+    /* fetch instruction from memory. */
+	mfspr	r20,SRR0
+	andis.	r21, r20, 0x8000
+	beq-	139b		/* Branch if user space address */
+	tophys  (r21, r20)
+140:	lwz	r21,0(r21)
+#if DEBUG_DCBX_INSTRUCTIONS
+/* Check if it really is a dcbx instruction */
+	rlwinm r20,r21,0,21,30
+	cmpwi cr0,r20,940	/* Is dcbi? */
+	bc 12,2,142f
+	cmpwi cr0,r20,2028	/* Is dcbz? */
+	bc 12,2,142f
+	cmpwi cr0,r20,492	/* Is dcbtst? */
+	bc 12,2,142f
+	cmpwi cr0,r20,556	/* Is dcbt? */
+	bc 12,2,142f
+	cmpwi cr0,r20,108	/* Is dcbst? */
+	bc 12,2,142f
+	cmpwi cr0,r20,172	/* Is dcbf? */
+	bc 12,2,142f
+
+141:	b 141b /* Stop here if no dcbx instruction */
+142: /* continue, it was a dcbx instruction. */
+#endif
+
+#ifdef CONFIG_8xx_CPU6
+	lwz	r3, 8(r0)		/* restore r3 from memory */
+#endif
+	mfctr	r20
+	mtdar	r20			/* save ctr reg in DAR */
+	rlwinm	r20, r21, 24, 24, 28	/* offset into jump table for reg RB */
+	addi	r20, r20, 150f@l	/* add start of table */
+	mtctr	r20			/* load ctr with jump address */
+	xor	r20, r20, r20		/* sum starts at zero */
+	bctr				/* jump into table */
+150:
+	add	r20, r20, r0
+	b	151f
+	add	r20, r20, r1
+	b	151f
+	add	r20, r20, r2
+	b	151f
+	add	r20, r20, r3
+	b	151f
+	add	r20, r20, r4
+	b	151f
+	add	r20, r20, r5
+	b	151f
+	add	r20, r20, r6
+	b	151f
+	add	r20, r20, r7
+	b	151f
+	add	r20, r20, r8
+	b	151f
+	add	r20, r20, r9
+	b	151f
+	add	r20, r20, r10
+	b	151f
+	add	r20, r20, r11
+	b	151f
+	add	r20, r20, r12
+	b	151f
+	add	r20, r20, r13
+	b	151f
+	add	r20, r20, r14
+	b	151f
+	add	r20, r20, r15
+	b	151f
+	add	r20, r20, r16
+	b	151f
+	add	r20, r20, r17
+	b	151f
+	add	r20, r20, r18
+	b	151f
+	add	r20, r20, r19
+	b	151f
+	mtctr	r21	/* reg 20 needs special handling */
+	b	154f
+	mtctr	r21	/* reg 21 needs special handling */
+	b	153f
+	add	r20, r20, r22
+	b	151f
+	add	r20, r20, r23
+	b	151f
+	add	r20, r20, r24
+	b	151f
+	add	r20, r20, r25
+	b	151f
+	add	r20, r20, r25
+	b	151f
+	add	r20, r20, r27
+	b	151f
+	add	r20, r20, r28
+	b	151f
+	add	r20, r20, r29
+	b	151f
+	add	r20, r20, r30
+	b	151f
+	add	r20, r20, r31
+151:
+	rlwinm. r21,r21,19,24,28	/* offset into jump table for reg RA */
+	bc	12,2,152f		/* if reg RA is zero, don't add it */
+	addi	r21, r21, 150b@l	/* add start of table */
+	mtctr	r21			/* load ctr with jump address */
+	rlwinm	r21,r21,0,16,10		/* make sure we don't execute this more than once */
+	bctr				/* jump into table */
+152:
+	mfdar	r21
+	mtctr	r21			/* restore ctr reg from DAR */
+	mtdar	r20			/* save fault EA to DAR */
+	b	101b			/* Go back to normal TLB handling */
+
+	/* special handling for r20,r21 since these are modified already */
+153:	lwz	r21, 4(r0)	/* load r21 from memory */
+	b	155f
+154:	mfspr	r21, M_TW	/* load r20 from M_TW */
+155:	add	r20, r20, r21	/* add it */
+	mfctr	r21		/* restore r21 */
+	b	151b

 /*
  * This code finishes saving the registers to the exception frame


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Tom Rini]

On Tue, Apr 08, 2003 at 07:25:37PM +0200, Joakim Tjernlund wrote:
> >
> > Patch against 2.4.20 devel available on request until I have cleaned it up a little.
> >
> >    Jocke
>
> Here is the patch, against 2.4.20 devel but I think it will apply
> against BK HEAD. Comments welcome :-)

Can you let me know if this applies cleanly to the current linux-2.4
tree (Or, 2.4.21-pre7 tarball)?  If not, can you generate a patch
against this?  Thanks.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Dan Malek]

Till Straumann wrote:

> Why not tag MD_EPN with an unused CASID?

There isn't such a thing.  The ASIDs are used for MMU optimizations.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

>
> On Tue, Apr 08, 2003 at 07:25:37PM +0200, Joakim Tjernlund wrote:
> > >
> > > Patch against 2.4.20 devel available on request until I have cleaned it up a little.
> > >
> > >    Jocke
> >
> > Here is the patch, against 2.4.20 devel but I think it will apply
> > against BK HEAD. Comments welcome :-)
>
> Can you let me know if this applies cleanly to the current linux-2.4
> tree (Or, 2.4.21-pre7 tarball)?  If not, can you generate a patch
> against this?  Thanks.
>
> --
> Tom Rini

pulled the latest linuxppc_2_4, patched it and inspected the result.
Current patch will apply with some offset and looks good.

I cannot run standard 2.4 without major patching, so I can't be sure
that it works.

Anyhow here is a version against linuxppc_2_4:

===== arch/ppc/kernel/head_8xx.S 1.33 vs edited =====
--- 1.33/arch/ppc/kernel/head_8xx.S Thu Mar 27 10:05:55 2003
+++ edited/arch/ppc/kernel/head_8xx.S Tue Apr  8 22:49:49 2003
@@ -31,6 +31,24 @@
 #include <asm/ppc_asm.h>
 #include "ppc_defs.h"

+/* These macros are used to tag DAR with a known value so that the
+ * DataTLBError can recognize a buggy dcbx instruction and workaround
+ * the problem.
+ */
+#define TAG_VAL 0xdead
+#define TAG_DAR_R20  \
+ lis r20, TAG_VAL;\
+ mtspr DAR, r20;
+
+/* Macro to make the code more readable. */
+#ifdef CONFIG_8xx_CPU6
+  #define DO_8xx_CPU6(val, reg) \
+ li reg, val; \
+ stw reg, 12(r0); \
+ lwz reg, 12(r0);
+#else
+  #define DO_8xx_CPU6(val, reg)
+#endif
  .text
  .globl _stext
 _stext:
@@ -166,6 +184,7 @@
  . = n;     \
 label:      \
  EXCEPTION_PROLOG;   \
+ TAG_DAR_R20;    \
  addi r3,r1,STACK_FRAME_OVERHEAD; \
  li r20,MSR_KERNEL;   \
  FINISH_EXCEPTION(hdlr)
@@ -188,6 +207,7 @@
  mr r5,r20
  mfspr r4,DAR
  stw r4,_DAR(r21)
+ TAG_DAR_R20
  addi r3,r1,STACK_FRAME_OVERHEAD
  li r20,MSR_KERNEL
  rlwimi r20,r23,0,16,16  /* copy EE bit from saved MSR */
@@ -226,6 +246,7 @@
  EXCEPTION_PROLOG
  mfspr r4,DAR
  stw r4,_DAR(r21)
+ TAG_DAR_R20
  mfspr r5,DSISR
  stw r5,_DSISR(r21)
  addi r3,r1,STACK_FRAME_OVERHEAD
@@ -461,6 +482,7 @@
 #endif
  mtspr MD_RPN, r20 /* Update TLB entry */

+ TAG_DAR_R20
  mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
@@ -470,7 +492,17 @@
 #endif
  rfi

-2: mfspr r20, M_TW /* Restore registers */
+2:
+ /* Copy 20 msb from MD_EPN to DAR since the dcxx instructions fails
+  * to update DAR when they cause a DTLB Miss.
+  */
+ mfspr r21, MD_EPN
+ rlwinm r21, r21, 0, 0, 19
+ mfspr r20, DAR
+ rlwinm r20, r20, 0, 20, 31
+ or r20, r20, r21
+ mtspr DAR, r20
+ mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
  lwz r21, 4(r0)
@@ -508,10 +540,18 @@
  stw r20, 0(r0)
  stw r21, 4(r0)

+ /* If DAR contains TAG_VAL implies a buggy dcbx instruction
+  * than did not set DAR.
+  */
+ mfspr r20, DAR
+ lis r21, TAG_VAL
+ cmpw cr0, r20, r21
+ beq- 100f /* Branch if TAG_VAL to dcbx workaround procedure */
+101: /* return from dcbx instruction bug workaround, r20 holds value of DAR */
  /* First, make sure this was a store operation.
  */
- mfspr r20, DSISR
- andis. r21, r20, 0x0200 /* If set, indicates store op */
+ mfspr r21, DSISR
+ andis. r21, r21, 0x0200 /* If set, indicates store op */
  beq 2f

  /* The EA of a data TLB miss is automatically stored in the MD_EPN
@@ -530,7 +570,7 @@
   * are initialized in mapin_ram().  This will avoid the problem,
   * assuming we only use the dcbi instruction on kernel addresses.
   */
- mfspr r20, DAR
+ /* DAR is in r20 already */
  rlwinm r21, r20, 0, 0, 19
  ori r21, r21, MD_EVALID
  mfspr r20, M_CASID
@@ -597,6 +637,7 @@
 #endif
  mtspr MD_RPN, r20 /* Update TLB entry */

+ TAG_DAR_R20
  mfspr r20, M_TW /* Restore registers */
  lwz r21, 0(r0)
  mtcr r21
@@ -633,6 +674,146 @@
  STD_EXCEPTION(0x1f00, Trap_1f, UnknownException)

  . = 0x2000
+/* This is the workaround procedure to calculate the data EA for a buggy dcbx instruction
+ * by decoding the registers used by the dcbx instruction and adding them.
+ * DAR is set to the calculated address and r20 also holds the EA on exit.
+ */
+139:/* fetch instruction from userspace memory */
+ DO_8xx_CPU6(0x3780, r3)
+ mtspr MD_EPN, r20 /* Have to use MD_EPN for walk, MI_EPN can't */
+ mfspr r21, M_TWB /* Get level 1 table entry address */
+ lwz r21, 0(r21) /* Get the level 1 entry */
+ tophys  (r21, r21)
+ DO_8xx_CPU6(0x3b80, r3)
+ mtspr MD_TWC, r21 /* Load pte table base address */
+ mfspr r21, MD_TWC /* ....and get the pte address */
+ lwz r21, 0(r21) /* Get the pte */
+ /* concat physical page address and page offset */
+ rlwinm r21, r21, 0, 0, 19
+ rlwinm r20, r20, 0, 20, 31
+ or r21, r20, r21
+ b 140f
+100:/* Entry point for dcbx workaround. */
+    /* fetch instruction from memory. */
+ mfspr r20,SRR0
+ andis. r21, r20, 0x8000
+ beq- 139b  /* Branch if user space address */
+ tophys  (r21, r20)
+140: lwz r21,0(r21)
+#if DEBUG_DCBX_INSTRUCTIONS
+/* Check if it really is a dcbx instruction */
+ rlwinm r20,r21,0,21,30
+ cmpwi cr0,r20,940 /* Is dcbi? */
+ bc 12,2,142f
+ cmpwi cr0,r20,2028 /* Is dcbz? */
+ bc 12,2,142f
+ cmpwi cr0,r20,492 /* Is dcbtst? */
+ bc 12,2,142f
+ cmpwi cr0,r20,556 /* Is dcbt? */
+ bc 12,2,142f
+ cmpwi cr0,r20,108 /* Is dcbst? */
+ bc 12,2,142f
+ cmpwi cr0,r20,172 /* Is dcbf? */
+ bc 12,2,142f
+
+141: b 141b /* Stop here if no dcbx instruction */
+142: /* continue, it was a dcbx instruction. */
+#endif
+
+#ifdef CONFIG_8xx_CPU6
+ lwz r3, 8(r0)  /* restore r3 from memory */
+#endif
+ mfctr r20
+ mtdar r20   /* save ctr reg in DAR */
+ rlwinm r20, r21, 24, 24, 28 /* offset into jump table for reg RB */
+ addi r20, r20, 150f@l /* add start of table */
+ mtctr r20   /* load ctr with jump address */
+ xor r20, r20, r20  /* sum starts at zero */
+ bctr    /* jump into table */
+150:
+ add r20, r20, r0
+ b 151f
+ add r20, r20, r1
+ b 151f
+ add r20, r20, r2
+ b 151f
+ add r20, r20, r3
+ b 151f
+ add r20, r20, r4
+ b 151f
+ add r20, r20, r5
+ b 151f
+ add r20, r20, r6
+ b 151f
+ add r20, r20, r7
+ b 151f
+ add r20, r20, r8
+ b 151f
+ add r20, r20, r9
+ b 151f
+ add r20, r20, r10
+ b 151f
+ add r20, r20, r11
+ b 151f
+ add r20, r20, r12
+ b 151f
+ add r20, r20, r13
+ b 151f
+ add r20, r20, r14
+ b 151f
+ add r20, r20, r15
+ b 151f
+ add r20, r20, r16
+ b 151f
+ add r20, r20, r17
+ b 151f
+ add r20, r20, r18
+ b 151f
+ add r20, r20, r19
+ b 151f
+ mtctr r21 /* reg 20 needs special handling */
+ b 154f
+ mtctr r21 /* reg 21 needs special handling */
+ b 153f
+ add r20, r20, r22
+ b 151f
+ add r20, r20, r23
+ b 151f
+ add r20, r20, r24
+ b 151f
+ add r20, r20, r25
+ b 151f
+ add r20, r20, r25
+ b 151f
+ add r20, r20, r27
+ b 151f
+ add r20, r20, r28
+ b 151f
+ add r20, r20, r29
+ b 151f
+ add r20, r20, r30
+ b 151f
+ add r20, r20, r31
+151:
+ rlwinm. r21,r21,19,24,28 /* offset into jump table for reg RA */
+ bc 12,2,152f  /* if reg RA is zero, don't add it */
+ addi r21, r21, 150b@l /* add start of table */
+ mtctr r21   /* load ctr with jump address */
+ rlwinm r21,r21,0,16,10  /* make sure we don't execute this more than once */
+ bctr    /* jump into table */
+152:
+ mfdar r21
+ mtctr r21   /* restore ctr reg from DAR */
+ mtdar r20   /* save fault EA to DAR */
+ b 101b   /* Go back to normal TLB handling */
+
+ /* special handling for r20,r21 since these are modified already */
+153: lwz r21, 4(r0) /* load r21 from memory */
+ b 155f
+154: mfspr r21, M_TW /* load r20 from M_TW */
+155: add r20, r20, r21 /* add it */
+ mfctr r21  /* restore r21 */
+ b 151b

 /*
  * This code finishes saving the registers to the exception frame


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: dcbz works on 862 everywhere!

[Joakim Tjernlund]

> As I said in the past, I'm sensitive to the code in the TLB exception
> processing.  So do something to remove code and streamline the process
> and I'm really interested.  Do something to add more code and it's
> going to get placed pretty low in my pile of things to do.
>
> Thanks.

If you make swapper_pg_dir 64KB aligned, you can removed
ori r21, r21, swapper_pg_dir@l instructions in
DataStoreTLBMiss, DataTLBError and InstructionTLBMiss

I did it with the following, ugly, hack in vmliunx.lds:
 swapper_pg_dir = ALIGN(0x10000);
 . = . + 4096;

I guess there is a better way to this.

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: mpc8xx DCBZ (&friends) hw bug. Tests, analysis + conclusions.

[Joakim Tjernlund]

> Me again :-)
>
> I have completed and tested my workaround for the dcbx instructions. The workaround
> handles ALL dcbx instructions, ANY register combination and works both on
> kernel space and user space addresses.
>
> I also did some benchmarking using copy_page(dcbz enabled) and memcpy to
> memory allocated with kmalloc and/or vmalloc. copy_page is about 30% faster
> than memcpy even with the workaround applied.

Here I go again :)

I have been running this patch on some 20-30 custom MPC860/862 boards in our test lab
since I posted this message and it is stable.

I made some changes since then:
 - Made it configurable, #define CONFIG_8xx_DCBxFIXED to enable it.
 - Tagging in the fast path in the DLBMiss handler is just one(1) instruction.
 - Test and branch if TAG present is two instructions in the DTLB Error handler.
 - Enabled the use of the dcbz instruction in copy_tofrom_user(), cacheable_memzero(),
   cacheable_memcpy(), clear_page() and copy_page()

Feedback most welcome!
Patch against linuxppc_2_4_devel follows.

 Jocke

--- a/arch/ppc/kernel/head_8xx.S	Tue Apr 29 00:45:35 2003
+++ b/arch/ppc/kernel/head_8xx.S	Fri May  9 14:16:44 2003
@@ -31,6 +31,27 @@
 #include <asm/ppc_asm.h>
 #include "ppc_defs.h"

+#ifdef CONFIG_8xx_DCBxFIXED
+/* These macros are used to tag DAR with a known value so that the
+ * DataTLBError can recognize a buggy dcbx instruction and workaround
+ * the problem.
+ */
+	#define TAG_VAL 0x00f0
+	#define TAG_DAR_R20 	\
+		li	r20, TAG_VAL;\
+		mtspr	DAR, r20;
+#else
+	#define TAG_DAR_R20
+#endif
+/* Macro to make the code more readable. */
+#ifdef CONFIG_8xx_CPU6
+  #define DO_8xx_CPU6(val, reg) \
+	li	reg, val; \
+	stw	reg, 12(r0); \
+	lwz	reg, 12(r0);
+#else
+  #define DO_8xx_CPU6(val, reg)
+#endif
 	.text
 	.globl	_stext
 _stext:
@@ -166,6 +187,7 @@
 	. = n;					\
 label:						\
 	EXCEPTION_PROLOG;			\
+	TAG_DAR_R20;				\
 	addi	r3,r1,STACK_FRAME_OVERHEAD;	\
 	li	r20,MSR_KERNEL;			\
 	FINISH_EXCEPTION(hdlr)
@@ -188,6 +210,7 @@
 	mr	r5,r20
 	mfspr	r4,DAR
 	stw	r4,_DAR(r21)
+	TAG_DAR_R20
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	li	r20,MSR_KERNEL
 	rlwimi	r20,r23,0,16,16		/* copy EE bit from saved MSR */
@@ -226,6 +249,7 @@
 	EXCEPTION_PROLOG
 	mfspr	r4,DAR
 	stw	r4,_DAR(r21)
+	TAG_DAR_R20
 	mfspr	r5,DSISR
 	stw	r5,_DSISR(r21)
 	addi	r3,r1,STACK_FRAME_OVERHEAD
@@ -457,6 +481,13 @@
 #endif
 	mtspr	MD_RPN, r20	/* Update TLB entry */

+#ifdef CONFIG_8xx_DCBxFIXED
+#if TAG_VAL == 0x00f0 /* Save 1 instr. by reusing the val loaded in r21 above */
+	mtspr	DAR, r21
+#else
+	TAG_DAR_R20
+#endif
+#endif
 	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
@@ -466,7 +497,17 @@
 #endif
 	rfi

-2:	mfspr	r20, M_TW	/* Restore registers */
+2:
+#ifdef CONFIG_8xx_DCBxFIXED
+	/* Copy 20 msb from MD_EPN to DAR since the dcxx instructions fails
+	 * to update DAR when they cause a DTLB Miss.
+	 */
+	mfspr	r21, MD_EPN
+	mfspr	r20, DAR
+	rlwimi	r20, r21, 0, 0, 19
+	mtspr	DAR, r20
+#endif
+	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
 	lwz	r21, 4(r0)
@@ -504,10 +545,19 @@
 	stw	r20, 0(r0)
 	stw	r21, 4(r0)

+	mfspr	r20, DAR
+#ifdef  CONFIG_8xx_DCBxFIXED
+	/* If DAR contains TAG_VAL implies a buggy dcbx instruction
+	 * that did not set DAR.
+	 */
+	cmplwi	cr0, r20, TAG_VAL
+	beq-	100f	/* Branch if TAG_VAL to dcbx workaround procedure */
+101:	/* return from dcbx instruction bug workaround, r20 holds value of DAR */
 	/* First, make sure this was a store operation.
 	*/
-	mfspr	r20, DSISR
-	andis.	r21, r20, 0x0200	/* If set, indicates store op */
+#endif
+	mfspr	r21, DSISR
+	andis.	r21, r21, 0x0200	/* If set, indicates store op */
 	beq	2f

 	/* The EA of a data TLB miss is automatically stored in the MD_EPN
@@ -526,7 +576,7 @@
 	 * are initialized in mapin_ram().  This will avoid the problem,
 	 * assuming we only use the dcbi instruction on kernel addresses.
 	 */
-	mfspr	r20, DAR
+	/* DAR is in r20 already */
 	rlwinm	r21, r20, 0, 0, 19
 	ori	r21, r21, MD_EVALID
 	mfspr	r20, M_CASID
@@ -591,6 +641,13 @@
 #endif
 	mtspr	MD_RPN, r20	/* Update TLB entry */

+#ifdef CONFIG_8xx_DCBxFIXED
+#if TAG_VAL == 0x00f0 /* Save 1 instr. by reusing the val loaded in r21 above */
+	mtspr	DAR, r21
+#else
+	TAG_DAR_R20
+#endif
+#endif
 	mfspr	r20, M_TW	/* Restore registers */
 	lwz	r21, 0(r0)
 	mtcr	r21
@@ -628,6 +685,149 @@

 	. = 0x2000

+#ifdef CONFIG_8xx_DCBxFIXED
+/* This is the workaround procedure to calculate the data EA for a buggy dcbx instruction
+ * by decoding the registers used by the dcbx instruction and adding them.
+ * DAR is set to the calculated address and r20 also holds the EA on exit.
+ */
+139:	/* fetch instruction from userspace memory */
+	DO_8xx_CPU6(0x3780, r3)
+	mtspr	MD_EPN, r20
+	mfspr	r21, M_TWB	/* Get level 1 table entry address */
+	lwz	r21, 0(r21)	/* Get the level 1 entry */
+	tophys  (r21, r21)
+	DO_8xx_CPU6(0x3b80, r3)
+	mtspr	MD_TWC, r21	/* Load pte table base address */
+	mfspr	r21, MD_TWC	/* ....and get the pte address */
+	lwz	r21, 0(r21)	/* Get the pte */
+	/* concat physical page address(r21) and page offset(r20) */
+	rlwimi	r21, r20, 0, 20, 31
+	b	140f
+100:	/* Entry point for dcbx workaround. */
+	/* fetch instruction from memory. */
+	mfspr	r20,SRR0
+	andis.	r21, r20, 0x8000
+	beq-	139b		/* Branch if user space address */
+	tophys  (r21, r20)
+140:	lwz	r21,0(r21)
+
+/* Check if it really is a dcbx instruction */
+	rlwinm	r20, r21, 0, 21, 30
+	cmpwi	cr0, r20, 2028	/* Is dcbz? */
+	beq+	142f
+	cmpwi	cr0, r20, 940	/* Is dcbi? */
+	beq+	142f
+	cmpwi	cr0, r20, 556	/* Is dcbt? */
+	beq+	142f
+	cmpwi	cr0, r20, 172	/* Is dcbf? */
+	beq+	142f
+#ifdef DEBUG_DCBX_INSTRUCTIONS
+	cmpwi	cr0, r20, 108	/* Is dcbst? Should never cause a DTLB Miss/Error */
+	beq+	142f
+	cmpwi	cr0, r20, 492	/* Is dcbtst? Should never cause a DTLB Miss/Error */
+	beq+	142f
+
+141:	b 141b /* Stop here if no dcbx instruction */
+#endif
+	mfspr	r20, DAR	/* r20 must hold DAR at exit */
+	b 101b			/* None of the above, go back to normal TLB processing */
+142:	/* continue, it was a dcbx instruction. */
+
+#ifdef CONFIG_8xx_CPU6
+	lwz	r3, 8(r0)		/* restore r3 from memory */
+#endif
+	mfctr	r20
+	mtdar	r20			/* save ctr reg in DAR */
+	rlwinm	r20, r21, 24, 24, 28	/* offset into jump table for reg RB */
+	addi	r20, r20, 150f@l	/* add start of table */
+	mtctr	r20			/* load ctr with jump address */
+	xor	r20, r20, r20		/* sum starts at zero */
+	bctr				/* jump into table */
+150:
+	add	r20, r20, r0
+	b	151f
+	add	r20, r20, r1
+	b	151f
+	add	r20, r20, r2
+	b	151f
+	add	r20, r20, r3
+	b	151f
+	add	r20, r20, r4
+	b	151f
+	add	r20, r20, r5
+	b	151f
+	add	r20, r20, r6
+	b	151f
+	add	r20, r20, r7
+	b	151f
+	add	r20, r20, r8
+	b	151f
+	add	r20, r20, r9
+	b	151f
+	add	r20, r20, r10
+	b	151f
+	add	r20, r20, r11
+	b	151f
+	add	r20, r20, r12
+	b	151f
+	add	r20, r20, r13
+	b	151f
+	add	r20, r20, r14
+	b	151f
+	add	r20, r20, r15
+	b	151f
+	add	r20, r20, r16
+	b	151f
+	add	r20, r20, r17
+	b	151f
+	add	r20, r20, r18
+	b	151f
+	add	r20, r20, r19
+	b	151f
+	mtctr	r21	/* reg 20 needs special handling */
+	b	154f
+	mtctr	r21	/* reg 21 needs special handling */
+	b	153f
+	add	r20, r20, r22
+	b	151f
+	add	r20, r20, r23
+	b	151f
+	add	r20, r20, r24
+	b	151f
+	add	r20, r20, r25
+	b	151f
+	add	r20, r20, r25
+	b	151f
+	add	r20, r20, r27
+	b	151f
+	add	r20, r20, r28
+	b	151f
+	add	r20, r20, r29
+	b	151f
+	add	r20, r20, r30
+	b	151f
+	add	r20, r20, r31
+151:
+	rlwinm. r21,r21,19,24,28	/* offset into jump table for reg RA */
+	beq	152f			/* if reg RA is zero, don't add it */
+	addi	r21, r21, 150b@l	/* add start of table */
+	mtctr	r21			/* load ctr with jump address */
+	rlwinm	r21,r21,0,16,10		/* make sure we don't execute this more than once */
+	bctr				/* jump into table */
+152:
+	mfdar	r21
+	mtctr	r21			/* restore ctr reg from DAR */
+	mtdar	r20			/* save fault EA to DAR */
+	b	101b			/* Go back to normal TLB handling */
+
+	/* special handling for r20,r21 since these are modified already */
+153:	lwz	r21, 4(r0)	/* load r21 from memory */
+	b	155f
+154:	mfspr	r21, M_TW	/* load r20 from M_TW */
+155:	add	r20, r20, r21	/* add it */
+	mfctr	r21		/* restore r21 */
+	b	151b
+#endif
 /*
  * This code finishes saving the registers to the exception frame
  * and jumps to the appropriate handler for the exception, turning
--- a/arch/ppc/lib/string.S	Tue Apr 29 00:45:35 2003
+++ b/arch/ppc/lib/string.S	Fri May  9 14:17:07 2003
@@ -151,7 +151,7 @@
 	bdnz	4b
 3:	mtctr	r9
 	li	r7,4
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx) || defined(CONFIG_8xx_DCBxFIXED)
 10:	dcbz	r7,r6
 #else
 10:	stw	r4, 4(r6)
@@ -253,7 +253,7 @@
 	mtctr	r0
 	beq	63f
 53:
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx) || defined(CONFIG_8xx_DCBxFIXED)
 	dcbz	r11,r6
 #endif
 	COPY_16_BYTES
@@ -452,6 +452,8 @@
 53:
 #if !defined(CONFIG_8xx)
 	dcbt	r3,r4
+#endif
+#if !defined(CONFIG_8xx) || defined(CONFIG_8xx_DCBxFIXED)
 54:	dcbz	r11,r6
 #endif
 /* had to move these to keep extable in order */
@@ -461,7 +463,7 @@
 	.long	71b,101f
 	.long	72b,102f
 	.long	73b,103f
-#if !defined(CONFIG_8xx)
+#if !defined(CONFIG_8xx) || defined(CONFIG_8xx_DCBxFIXED)
 	.long	54b,105f
 #endif
 	.text
--- a/arch/ppc/kernel/misc.S	Tue Apr 29 00:45:35 2003
+++ b/arch/ppc/kernel/misc.S	Fri May  9 14:16:23 2003
@@ -657,7 +657,7 @@
 _GLOBAL(clear_page)
 	li	r0,4096/L1_CACHE_LINE_SIZE
 	mtctr	r0
-#ifdef CONFIG_8xx
+#if defined(CONFIG_8xx) && !defined(CONFIG_8xx_DCBxFIXED)
 	li	r4, 0
 1:	stw	r4, 0(r3)
 	stw	r4, 4(r3)
@@ -710,6 +710,8 @@
 1:
 #ifndef CONFIG_8xx
 	dcbt	r11,r4
+#endif
+#if !defined(CONFIG_8xx) || defined(CONFIG_8xx_DCBxFIXED)
 	dcbz	r5,r3
 #endif
 	COPY_16_BYTES


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/