Disable cache on 74xx

Disable cache on 74xx

[Brian Waite]

Hi all,
	I am trying to hunt down a memory controller configuration problem and I have
been asked to disable all caching so as to remove it from the equation. I can
easl disable L2 but when I start ucking with the WIMG bits to set cache
inhibit, The kernel panics with stack overflows. Does anyone know where or
what I have to set to disable caching?

Thanks
Brian

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Benjamin Herrenschmidt]

On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> Hi all,
> 	I am trying to hunt down a memory controller configuration problem and I have
> been asked to disable all caching so as to remove it from the equation. I can
> easl disable L2 but when I start ucking with the WIMG bits to set cache
> inhibit, The kernel panics with stack overflows. Does anyone know where or
> what I have to set to disable caching?

Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Gary Thomas]

On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
>
> On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> > Hi all,
> > 	I am trying to hunt down a memory controller configuration problem and I have
> > been asked to disable all caching so as to remove it from the equation. I can
> > easl disable L2 but when I start ucking with the WIMG bits to set cache
> > inhibit, The kernel panics with stack overflows. Does anyone know where or
> > what I have to set to disable caching?
>
> Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?

There are other problems with this.  With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.

... totally disabling the data CACHE with Linux is non-trivial.

--
------------------------------------------------------------
Gary Thomas                 |
MLB Associates              |  Consulting for the
+1 (970) 229-1963           |    Embedded world
http://www.mlbassoc.com/    |
email: <gary@mlbassoc.com>  |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Brian Waite]

According to The User's Manual, the data cache instructions become no-ops if
the data cache is disabled.

Thanks
Brian
On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
> On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
> > On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> > > Hi all,
> > > 	I am trying to hunt down a memory controller configuration problem and
> > > I have been asked to disable all caching so as to remove it from the
> > > equation. I can easl disable L2 but when I start ucking with the WIMG
> > > bits to set cache inhibit, The kernel panics with stack overflows. Does
> > > anyone know where or what I have to set to disable caching?
> >
> > Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
>
> There are other problems with this.  With the caches disabled
> (via HID0) all data cache instructions (like DCBF) will fail.
> There are some of these in the kernel itself, but beware that
> GLIBC has it's own set.
>
> ... totally disabling the data CACHE with Linux is non-trivial.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Gary Thomas]

On Thu, 2003-02-20 at 06:55, Brian Waite wrote:
> According to The User's Manual, the data cache instructions become no-ops if
> the data cache is disabled.
>

There are "User Manuals" and then there is "Real Life".  My
experience has been that trying to execute these instructions
with the cache disabled was treated like an invalid instruction
(hence causing a trap).  Maybe this isn't always true with all
PowerPC processors, but I was just pointing out that you might
run into trouble with it.

> Thanks
> Brian
> On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
> > On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
> > > On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> > > > Hi all,
> > > > 	I am trying to hunt down a memory controller configuration problem and
> > > > I have been asked to disable all caching so as to remove it from the
> > > > equation. I can easl disable L2 but when I start ucking with the WIMG
> > > > bits to set cache inhibit, The kernel panics with stack overflows. Does
> > > > anyone know where or what I have to set to disable caching?
> > >
> > > Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
> >
> > There are other problems with this.  With the caches disabled
> > (via HID0) all data cache instructions (like DCBF) will fail.
> > There are some of these in the kernel itself, but beware that
> > GLIBC has it's own set.
> >
> > ... totally disabling the data CACHE with Linux is non-trivial.
--
------------------------------------------------------------
Gary Thomas                 |
MLB Associates              |  Consulting for the
+1 (970) 229-1963           |    Embedded world
http://www.mlbassoc.com/    |
email: <gary@mlbassoc.com>  |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Geert Uytterhoeven]

On 20 Feb 2003, Gary Thomas wrote:
> On Thu, 2003-02-20 at 06:55, Brian Waite wrote:
> > According to The User's Manual, the data cache instructions become no-ops if
> > the data cache is disabled.
>
> There are "User Manuals" and then there is "Real Life".  My
> experience has been that trying to execute these instructions
> with the cache disabled was treated like an invalid instruction
> (hence causing a trap).  Maybe this isn't always true with all
> PowerPC processors, but I was just pointing out that you might
> run into trouble with it.

There may be a different behavior for `disabling the data cache globally' and
`using e.g. dcbf on uncached memory' (with the data cache enabled globally).

> > On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
> > > On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
> > > > On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> > > > > Hi all,
> > > > > 	I am trying to hunt down a memory controller configuration problem and
> > > > > I have been asked to disable all caching so as to remove it from the
> > > > > equation. I can easl disable L2 but when I start ucking with the WIMG
> > > > > bits to set cache inhibit, The kernel panics with stack overflows. Does
> > > > > anyone know where or what I have to set to disable caching?
> > > >
> > > > Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
> > >
> > > There are other problems with this.  With the caches disabled
> > > (via HID0) all data cache instructions (like DCBF) will fail.
> > > There are some of these in the kernel itself, but beware that
> > > GLIBC has it's own set.
> > >
> > > ... totally disabling the data CACHE with Linux is non-trivial.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Brian Waite]

Gary,
	Thanks for the clarification. I appreciate the real life experiences. This is
exactly why I said "in the User Manual". I was hoping you would come back
with the exact answer you provided. :) I'll keep my eyes open.

Thanks
Brian
On Thursday 20 February 2003 9:01 am, Gary Thomas wrote:
> On Thu, 2003-02-20 at 06:55, Brian Waite wrote:
> > According to The User's Manual, the data cache instructions become no-ops
> > if the data cache is disabled.
>
> There are "User Manuals" and then there is "Real Life".  My
> experience has been that trying to execute these instructions
> with the cache disabled was treated like an invalid instruction
> (hence causing a trap).  Maybe this isn't always true with all
> PowerPC processors, but I was just pointing out that you might
> run into trouble with it.
>
> > Thanks
> > Brian
> >
> > On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
> > > On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
> > > > On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
> > > > > Hi all,
> > > > > 	I am trying to hunt down a memory controller configuration problem
> > > > > and I have been asked to disable all caching so as to remove it
> > > > > from the equation. I can easl disable L2 but when I start ucking
> > > > > with the WIMG bits to set cache inhibit, The kernel panics with
> > > > > stack overflows. Does anyone know where or what I have to set to
> > > > > disable caching?
> > > >
> > > > Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
> > >
> > > There are other problems with this.  With the caches disabled
> > > (via HID0) all data cache instructions (like DCBF) will fail.
> > > There are some of these in the kernel itself, but beware that
> > > GLIBC has it's own set.
> > >
> > > ... totally disabling the data CACHE with Linux is non-trivial.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Dan Malek]

Geert Uytterhoeven wrote:

> There may be a different behavior for `disabling the data cache globally' and
> `using e.g. dcbf on uncached memory' (with the data cache enabled globally).

I decided to take a diversion and read some processor manuals.  :-)

The behavior of the cache instructions on areas that are not cacheable
for any reason depends upon the cache instruction used.  Some have
no effect, some have unpredictable behavior, some cause alignment/access
traps.  It all depends upon how they would translate/access the cache
and the operation they would perform on a cache line.  We use all
instructions in Linux that would trigger any of the behavior. :-)

The bottom line appears to be you really don't want to execute cache
instructions on areas that are not cached.  This includes global disable
or any method of marking pages uncached.

For Brian, I'm also curious why you think running Linux with disabled caches
will assist in debugging memory controller problems?  If you are looking for
such fine control of the bus cycles for debugging it seems a simple
memory diagnostic used in conjuction with hardware debug tools is a
better approach.

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Brian Waite]

Well to preface, some people around here think disabling cache is a good idea.
I report to one of those :). What I am trying to debug is why my custom board
hangs. Basically,  what I see is when a PCI interface starts reading/writing
SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
serviced and using a BDI 2000 JTAG interface I see the processors stalled on
some memory access instruciton. The memory controller (gt64260) seems to be
doing something very wrong and it is thought to be a cache coherency issue.
So to prove this, I was asked to try running without any sort of cache.
Hopefully between Gary's information and yours, Dan I can convince people
that it is futile.

Thanks
Brian

On Thursday 20 February 2003 10:23 am, Dan Malek wrote:
> Geert Uytterhoeven wrote:
> > There may be a different behavior for `disabling the data cache globally'
> > and `using e.g. dcbf on uncached memory' (with the data cache enabled
> > globally).
>
> I decided to take a diversion and read some processor manuals.  :-)
>
> The behavior of the cache instructions on areas that are not cacheable
> for any reason depends upon the cache instruction used.  Some have
> no effect, some have unpredictable behavior, some cause alignment/access
> traps.  It all depends upon how they would translate/access the cache
> and the operation they would perform on a cache line.  We use all
> instructions in Linux that would trigger any of the behavior. :-)
>
> The bottom line appears to be you really don't want to execute cache
> instructions on areas that are not cached.  This includes global disable
> or any method of marking pages uncached.
>
> For Brian, I'm also curious why you think running Linux with disabled
> caches will assist in debugging memory controller problems?  If you are
> looking for such fine control of the bus cycles for debugging it seems a
> simple memory diagnostic used in conjuction with hardware debug tools is a
> better approach.
>
> Thanks.
>
>
> 	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Gary Thomas]

On Thu, 2003-02-20 at 08:47, Brian Waite wrote:
>
> Well to preface, some people around here think disabling cache is a good idea.
> I report to one of those :). What I am trying to debug is why my custom board
> hangs. Basically,  what I see is when a PCI interface starts reading/writing
> SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
> serviced and using a BDI 2000 JTAG interface I see the processors stalled on
> some memory access instruciton. The memory controller (gt64260) seems to be
> doing something very wrong and it is thought to be a cache coherency issue.
> So to prove this, I was asked to try running without any sort of cache.
> Hopefully between Gary's information and yours, Dan I can convince people
> that it is futile.
>

I agree with Dan that you should try and reduce the scope of your test.
Try to [re]create the failure with as little as possible, hopefully
without Linux in the way, etc.  (I'm recommending that you write a
separate test program)  In this way, your test can have complete control
over the environment and doing things like verifying cache issues, etc,
should be simpler.

> Thanks
> Brian
>
> On Thursday 20 February 2003 10:23 am, Dan Malek wrote:
> > Geert Uytterhoeven wrote:
> > > There may be a different behavior for `disabling the data cache globally'
> > > and `using e.g. dcbf on uncached memory' (with the data cache enabled
> > > globally).
> >
> > I decided to take a diversion and read some processor manuals.  :-)
> >
> > The behavior of the cache instructions on areas that are not cacheable
> > for any reason depends upon the cache instruction used.  Some have
> > no effect, some have unpredictable behavior, some cause alignment/access
> > traps.  It all depends upon how they would translate/access the cache
> > and the operation they would perform on a cache line.  We use all
> > instructions in Linux that would trigger any of the behavior. :-)
> >
> > The bottom line appears to be you really don't want to execute cache
> > instructions on areas that are not cached.  This includes global disable
> > or any method of marking pages uncached.
> >
> > For Brian, I'm also curious why you think running Linux with disabled
> > caches will assist in debugging memory controller problems?  If you are
> > looking for such fine control of the bus cycles for debugging it seems a
> > simple memory diagnostic used in conjuction with hardware debug tools is a
> > better approach.
> >
> > Thanks.
> >
> >
> > 	-- Dan
>
>
--
------------------------------------------------------------
Gary Thomas                 |
MLB Associates              |  Consulting for the
+1 (970) 229-1963           |    Embedded world
http://www.mlbassoc.com/    |
email: <gary@mlbassoc.com>  |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Benjamin Herrenschmidt]

On Thu, 2003-02-20 at 16:47, Brian Waite wrote:
> Well to preface, some people around here think disabling cache is a good idea.
> I report to one of those :). What I am trying to debug is why my custom board
> hangs. Basically,  what I see is when a PCI interface starts reading/writing
> SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
> serviced and using a BDI 2000 JTAG interface I see the processors stalled on
> some memory access instruciton. The memory controller (gt64260) seems to be
> doing something very wrong and it is thought to be a cache coherency issue.
> So to prove this, I was asked to try running without any sort of cache.
> Hopefully between Gary's information and yours, Dan I can convince people
> that it is futile.

It could be the gt doing something wrong, it could also be the
CPU, 745x have known lockup conditions, though at least some of
them have workarounds enabled in the kernel.

Look closely at the CPU errata.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Dan Malek]

Brian Waite wrote:

 > .... The memory controller (gt64260) seems to be
> doing something very wrong and it is thought to be a cache coherency issue.

The problem is cache coherency is central to the operation of processor
bus.  One possibility, especially in the example you described, is the gt
is generating incorrect cache coherency control cycles on the bus.  Disabling
the processor cache isn't going to affect this.

> So to prove this, I was asked to try running without any sort of cache.

Make sure you disable it in the memory controller as well.

> Hopefully between Gary's information and yours, Dan I can convince people
> that it is futile.

Linux is a very poor diagnostic tool.  When I run Linux and discover this
type of problem, I reduce it down into a non-Linux diagnostic that can
generate the same type of bus cycles.  That way you can just flash it into
rom and let some hardware engineer run it over and over without having to
boot up something as complex as Linux and sorting through millions of
bus cycles for the failure mode.  With these types of problems you have
to determine exactly what causes the failure.  Since the processor hangs
so completely it should be easy to see the condition of the failure.  If
it is a memory controller timing problem, you should see exactly what
failed and it should be clear how to program the controller to correct it.

Of course, like Ben said, check the errata.

Good Luck.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Disable cache on 74xx

[Mark A. Greer]

Brian,

I know you are aware of this but I *always* experience cache coherency
problems with the GT64260A when Sysclk/Tclk are anything above 100MHz
and I believe that your board is clocked over that.  Try a 100MHz
crystal and see if the problem goes away.

Mark
--
Dan Malek wrote:

>
> Brian Waite wrote:
>
> > .... The memory controller (gt64260) seems to be
>
>> doing something very wrong and it is thought to be a cache coherency
>> issue.
>
>
> The problem is cache coherency is central to the operation of processor
> bus.  One possibility, especially in the example you described, is the gt
> is generating incorrect cache coherency control cycles on the bus.
> Disabling
> the processor cache isn't going to affect this.
>
>> So to prove this, I was asked to try running without any sort of cache.
>
>
> Make sure you disable it in the memory controller as well.
>
>> Hopefully between Gary's information and yours, Dan I can convince
>> people
>> that it is futile.
>
>
> Linux is a very poor diagnostic tool.  When I run Linux and discover this
> type of problem, I reduce it down into a non-Linux diagnostic that can
> generate the same type of bus cycles.  That way you can just flash it
> into
> rom and let some hardware engineer run it over and over without having to
> boot up something as complex as Linux and sorting through millions of
> bus cycles for the failure mode.  With these types of problems you have
> to determine exactly what causes the failure.  Since the processor hangs
> so completely it should be easy to see the condition of the failure.  If
> it is a memory controller timing problem, you should see exactly what
> failed and it should be clear how to program the controller to correct
> it.
>
> Of course, like Ben said, check the errata.
>
> Good Luck.
>
>
>     -- Dan
>
>
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/