How to fix 8xx dcbst bug?

How to fix 8xx dcbst bug?

[Marcelo Tosatti]

Hi Dan,

So, restarting this conversation...

On Tue, Apr 05, 2005 at 11:58:17AM -0400, Dan Malek wrote:
> 
> On Apr 4, 2005, at 3:17 PM, Marcelo Tosatti wrote:
> 
> >Problem is that the "dcbst" instruction will, _sometimes_ (the 
> >failure/success rate is about 1/4
> >with my test application) fault as a _write_ operation on the data.
> 
> Oh, geeze .... It's all coming back to me now ....
> 
> The 8xx cache operations don't always operate as defined in the PEM.
> There are likely to be some archive discussions within the Freescale
> knowledge data base that describe the different behaviors I've seen
> with the chip variants and revisions.  I can't find any of those e-mail
> discussions, so I'll try to recall from memory.
> 
> The PEM cache instructions are all implemented in a microcode that
> uses the 8xx unique cache control SPRs.  Depending upon the state
> of the cache and MMU, it seems in some cases the EA translation is
> subject to a "normal" protection match instead of a load operation 
> match.
> 
> The behavior of these operations isn't consistent across all of the 8xx
> processor revisions, especially with early silicon if people are still
> using those.	During conversations with Freescale engineers, it seems
> the only guaranteed operation was to use the 8xx unique SPRs, but
> I think I only did that in 8xx specific functions.
>
> We have way too much code in the TLB exception handlers already,
> so let's just try a tlbia of the EA in the update_mmu_cache, with an 
> #ifdef
> for the 8xx.	

>  We may want to make the dcbxxx instructions 
> some
> kind of macro, so on 8xx we can include such operations in otherwise
> "standard" software.

Now that I think of it, userspace dcbst users should not be an issue
because the intermediate invalid TLB entry is not visible to applications.

Userspace sees only: not present pte, or valid present pte.

Well, at least the entry which has been causing problems,
created by DataStoreTLBMiss.

Is it safe to assume that dcbst usage on userspace is restricted
to valid TLBs? Since MMU SPRs are restricted to supervisor 
protection, I think so.

So, if that is true, can you please merge the update_mmu_cache() fix 
for the dcbst misbehaviour previously discussed ? 

Thanks!!

RE: How to fix 8xx dcbst bug?

[Joakim Tjernlund]

> 
> 
> Hi Dan,
> 
> So, restarting this conversation...
> 
> On Tue, Apr 05, 2005 at 11:58:17AM -0400, Dan Malek wrote:
> > 
> > On Apr 4, 2005, at 3:17 PM, Marcelo Tosatti wrote:
> > 
> > >Problem is that the "dcbst" instruction will, _sometimes_ (the 
> > >failure/success rate is about 1/4
> > >with my test application) fault as a _write_ operation on the data.
> > 
> > Oh, geeze .... It's all coming back to me now ....
> > 
> > The 8xx cache operations don't always operate as defined in the PEM.
> > There are likely to be some archive discussions within the Freescale
> > knowledge data base that describe the different behaviors I've seen
> > with the chip variants and revisions.  I can't find any of those e-mail
> > discussions, so I'll try to recall from memory.
> > 
> > The PEM cache instructions are all implemented in a microcode that
> > uses the 8xx unique cache control SPRs.  Depending upon the state
> > of the cache and MMU, it seems in some cases the EA translation is
> > subject to a "normal" protection match instead of a load operation 
> > match.
> > 
> > The behavior of these operations isn't consistent across all of the 8xx
> > processor revisions, especially with early silicon if people are still
> > using those.	During conversations with Freescale engineers, it seems
> > the only guaranteed operation was to use the 8xx unique SPRs, but
> > I think I only did that in 8xx specific functions.
> >
> > We have way too much code in the TLB exception handlers already,
> > so let's just try a tlbia of the EA in the update_mmu_cache, with an 
> > #ifdef
> > for the 8xx.	
> 
> >  We may want to make the dcbxxx instructions 
> > some
> > kind of macro, so on 8xx we can include such operations in otherwise
> > "standard" software.
> 
> Now that I think of it, userspace dcbst users should not be an issue
> because the intermediate invalid TLB entry is not visible to applications.
> 
> Userspace sees only: not present pte, or valid present pte.
> 
> Well, at least the entry which has been causing problems,
> created by DataStoreTLBMiss.
> 
> Is it safe to assume that dcbst usage on userspace is restricted
> to valid TLBs? Since MMU SPRs are restricted to supervisor 
> protection, I think so.

Not sure what you mean here. Currently all dcbX instr. in user space have to be on valid\populated pte's
since otherwise it will SEGV. If you write your app so that any dcbX will only cause a plain DTLB you are
safe, just look at ld.so. This should not be a requirement but for 8xx it is currently and I think 8xx gets
away with it because nobody is using swap on 8xx.

   Jocke

> 
> So, if that is true, can you please merge the update_mmu_cache() fix 
> for the dcbst misbehaviour previously discussed ? 
> 
> Thanks!!

Re: How to fix 8xx dcbst bug?

[Marcelo Tosatti]

On Sat, May 07, 2005 at 08:24:01PM +0200, Joakim Tjernlund wrote:
> > 
> > 
> > Hi Dan,
> > 
> > So, restarting this conversation...
> > 
> > On Tue, Apr 05, 2005 at 11:58:17AM -0400, Dan Malek wrote:
> > > 
> > > On Apr 4, 2005, at 3:17 PM, Marcelo Tosatti wrote:
> > > 
> > > >Problem is that the "dcbst" instruction will, _sometimes_ (the 
> > > >failure/success rate is about 1/4
> > > >with my test application) fault as a _write_ operation on the data.
> > > 
> > > Oh, geeze .... It's all coming back to me now ....
> > > 
> > > The 8xx cache operations don't always operate as defined in the PEM.
> > > There are likely to be some archive discussions within the Freescale
> > > knowledge data base that describe the different behaviors I've seen
> > > with the chip variants and revisions.  I can't find any of those e-mail
> > > discussions, so I'll try to recall from memory.
> > > 
> > > The PEM cache instructions are all implemented in a microcode that
> > > uses the 8xx unique cache control SPRs.  Depending upon the state
> > > of the cache and MMU, it seems in some cases the EA translation is
> > > subject to a "normal" protection match instead of a load operation 
> > > match.
> > > 
> > > The behavior of these operations isn't consistent across all of the 8xx
> > > processor revisions, especially with early silicon if people are still
> > > using those.	During conversations with Freescale engineers, it seems
> > > the only guaranteed operation was to use the 8xx unique SPRs, but
> > > I think I only did that in 8xx specific functions.
> > >
> > > We have way too much code in the TLB exception handlers already,
> > > so let's just try a tlbia of the EA in the update_mmu_cache, with an 
> > > #ifdef
> > > for the 8xx.	
> > 
> > >  We may want to make the dcbxxx instructions 
> > > some
> > > kind of macro, so on 8xx we can include such operations in otherwise
> > > "standard" software.
> > 
> > Now that I think of it, userspace dcbst users should not be an issue
> > because the intermediate invalid TLB entry is not visible to applications.
> > 
> > Userspace sees only: not present pte, or valid present pte.
> > 
> > Well, at least the entry which has been causing problems,
> > created by DataStoreTLBMiss.
> > 
> > Is it safe to assume that dcbst usage on userspace is restricted
> > to valid TLBs? Since MMU SPRs are restricted to supervisor 
> > protection, I think so.
> 
> Not sure what you mean here. Currently all dcbX instr. in user space 
> have to be on valid\populated pte's since otherwise it will SEGV. 

OK. The BUG in update_mmu_cache() is triggered because dcbX happens 
on populated but invalid page mapping.

> If you write your app so that any dcbX will only cause a plain DTLB you are
> safe, just look at ld.so. This should not be a requirement but for 8xx it is currently and I think 8xx gets
> away with it because nobody is using swap on 8xx.

Re: How to fix 8xx dcbst bug?

[Wolfgang Denk]

In message <JPEALJAFNGDDLOPNDIEECEAKDDAA.joakim.tjernlund@lumentis.se> you wrote:
>
...
> safe, just look at ld.so. This should not be a requirement but for 8xx it is currently and I think 8xx gets
> away with it because nobody is using swap on 8xx.

I understand that the "not using swap" refers to 2.6 kernel only, right?

Because on 2.4, we use swap on several systems quite heavily. I  have
to admit that we had to patch pgtable.h to get it work.

[In case anybody needs it: it's this patchset in our CVS tree:

	PatchSet 80
	Date: 2003/08/24 14:58:59
	Author: wd
	Branch: HEAD
	Tag: (none)
	Log:
	Fix swap problem as observed on IVMS systems with IDE harddisk.
	
	Members:
	        include/asm-ppc/pgtable.h:1.6->1.7
]

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Single tasking: Just Say No.

RE: How to fix 8xx dcbst bug?

[Joakim Tjernlund]

> 
> In message <JPEALJAFNGDDLOPNDIEECEAKDDAA.joakim.tjernlund@lumentis.se> you wrote:
> >
> ...
> > safe, just look at ld.so. This should not be a requirement but for 8xx it is currently and I think 8xx gets
> > away with it because nobody is using swap on 8xx.
> 
> I understand that the "not using swap" refers to 2.6 kernel only, right?
> 
> Because on 2.4, we use swap on several systems quite heavily. I  have
> to admit that we had to patch pgtable.h to get it work.

I was once told by a senior kernel hacker(Dan?) that using swap could make an already
populated pte invalid. This was for 2.4 kernels.

ldso uses dcbst and icbi heavly during relocation but always after doing a store to the
address in question. That will make sure that if dcbst/icbi causes a TLB Miss, it will be
a simple/regular TLB miss not requiring any special handling i fault.c

Perhaps using dcbst/icbi in this way will not make the pte go away between the store and the
dcbst/icbi instructions when also using swap. Someone more exprienced will have to answer that.

 Jocke

Re: How to fix 8xx dcbst bug?

[Dan Malek]

On May 7, 2005, at 6:47 PM, Joakim Tjernlund wrote:

> I was once told by a senior kernel hacker(Dan?) that using swap could 
> make an already
> populated pte invalid. This was for 2.4 kernels.

The original problem was I didn't pay attention to the swap PTE when
constructing the originals for the 8xx.  The swap PTE has to be 
indicated
as such in one of the lowest 2 bits of the PTE, as the rest of the PTE 
is
used for the swap block number.  What would happen is there would
be the occasion where some control bit was set in the PTE within the
swap block number, and upon restoring from disk the wrong block
would be brought into memory (or the fact it was swapped was lost).

Thanks.


	-- Dan

Re: How to fix 8xx dcbst bug?

[Dan Malek]

On May 6, 2005, at 11:45 AM, Marcelo Tosatti wrote:

>
> Hi Dan,
>
> So, restarting this conversation...

One of the things I don't want to lose sight of during
all of this is the real performance problem in 2.6.
Your test results show there is something that needs
attention, regardless of using pinned entries.  We
need to continue some of this investigation, it
affects all processors.

Thanks.

	-- Dan

Re: How to fix 8xx dcbst bug?

[Marcelo Tosatti]

On Sat, May 07, 2005 at 09:10:17PM -0400, Dan Malek wrote:
> 
> On May 6, 2005, at 11:45 AM, Marcelo Tosatti wrote:
> 
> >
> >Hi Dan,
> >
> >So, restarting this conversation...
> 
> One of the things I don't want to lose sight of during
> all of this is the real performance problem in 2.6.
> Your test results show there is something that needs
> attention, regardless of using pinned entries.  We
> need to continue some of this investigation, it
> affects all processors.

True.

Follows some useful data: 

"itlb-content-before" and "itlb-content-after" are dumps of TLB cache
content before and after "sys_read()", for both v2.4 and v2.6.

The "diff" output shows which TLB's have been faulted in:

[marcelo@logos itlb]$ diff -u 24-itlb-content-before.txt 24-itlb-content-after.txt  
| grep SPR | grep 816 | grep "+"
+SPR  816 : 0x0ffe800f    268337167
+SPR  816 : 0x0ffeb00f    268349455
+SPR  816 : 0xc009e01f  -1073094625
+SPR  816 : 0xc009d01f  -1073098721
+SPR  816 : 0xc000301f  -1073729505
+SPR  816 : 0xc009c01f  -1073102817

[marcelo@logos itlb]$ diff -u 24-itlb-content-before.txt 24-itlb-content-after.txt  
| grep SPR | grep 818 | grep "+"  | wc -l
6

Now v2.6:

[marcelo@logos itlb]$ diff -u 26-itlb-before.txt 26-itlb-after.txt  
| grep 816 | grep SPR | grep "+"
+SPR  816 : 0x0feda16f    267231599
+SPR  816 : 0xc004b17f  -1073434241
+SPR  816 : 0xc004a17f  -1073438337
+SPR  816 : 0x0ff7e16f    267903343
+SPR  816 : 0x1001016f    268501359
+SPR  816 : 0xc000217f  -1073733249
+SPR  816 : 0xc001617f  -1073651329
+SPR  816 : 0xc002e17f  -1073553025
+SPR  816 : 0xc010e17f  -1072635521
+SPR  816 : 0xc002d17f  -1073557121
+SPR  816 : 0xc010d17f  -1072639617
+SPR  816 : 0xc000c17f  -1073692289
+SPR  816 : 0xc000317f  -1073729153

[marcelo@logos itlb]$ diff -u 26-itlb-before.txt 26-itlb-after.txt  
| grep 816 | grep SPR | grep "+" | wc -l
13

So, for sys_read() v2.6 i-cache translation footprint is about 100% 
more than v2.4.  

I suspect that actual cache footprint is higher, too.

Re: How to fix 8xx dcbst bug?

[Dan Malek]

On May 7, 2005, at 5:47 PM, Marcelo Tosatti wrote:

> So, for sys_read() v2.6 i-cache translation footprint is about 100%
> more than v2.4.

It could actually be something as "simple" as the organization
of the functions in the image.  Would you also send along the
portion of the System.map that covers these kernel address
ranges?  There is only one additional user space TLB, which
is probably the C library, all of the rest are kernel ones.

For example, what functions are in the range of 0xc009c00,
0xc009d000, 0xc009e000, and in the 0xc0003000
pages for your 2.4 kernel?

Thanks.

	-- Dan