Strange, strange occurence

Strange, strange occurence

[S C]

Well I'm hoping it isn't so strange to some of you folks and you'll be able 
to tell what's going on :)

Here's my problem:

Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, when I 
download the kernel via the debug ethernet port it runs fine. If I download 
the kernel via the Tx4938 inbuilt ethernet controller, it crashes!

Memory checksumming and a quick manual memory dump inspection reveals that 
the kernel download went perfectly ok, and the image is completely and 
correctly downloaded to RAM.

The crash is occuring inside the function r4k_flush_icache_range().

I tried 'flush -i' and 'flush -d' on YAMON after the download but before the 
'go', but that didn't help. I also tried completely disabling caches and 
loading/running uncached, but it gave the same error.

Now, the final twist! Using an ICE, I set a breakpoint at the 
r4k_flush_icache_range function. Then I loaded the kernel as usual, ran it 
with the ICE, stepped through a few instructions inside the 
r4k_flush_icache_range function and then did a 'cont'. The kernel now booted 
fine!

If I don't set the breakpoint inside that function though, and just try to 
run with the ICE the same
error (Inst fetch/Load error) occurs.

I'm at a loss trying to figure out what's going on. I suspect it has 
something to do with caches perhaps (duh!), but have no clue what!  Anybody 
out there face a similar kind of a situation before?

Thanks in advance for any help offered.

Regards,
-Steve

_________________________________________________________________
MSN 9 Dial-up Internet Access helps fight spam and pop-ups – now 2 months 
FREE! http://join.msn.click-url.com/go/onm00200361ave/direct/01/

Re: Strange, strange occurence

[Niels Sterrenburg]

Hi Steve,

When you set a breakpoint with the ice and step through,
for each step the ICE set's a breakpoint on the next instruction:
e.g.:
- modifies next instruction in memory (via CPU !!!)
  with a breakpoint instruction,
- and update the cache so that the breakpoint instruction is really in
memory and that i-cache is invalid.
(or something like that)

So indead you have a cache problem which is explanable solved/disapeared
by steppig through with the ICE.

Maybe a stupid question from me but why flushing an icache ?:
r4k_flush_icache_range

regards,

Niels

> Well I'm hoping it isn't so strange to some of you folks and you'll be
> able
> to tell what's going on :)
>
> Here's my problem:
>
> Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, when
> I
> download the kernel via the debug ethernet port it runs fine. If I
> download
> the kernel via the Tx4938 inbuilt ethernet controller, it crashes!
>
> Memory checksumming and a quick manual memory dump inspection reveals that
> the kernel download went perfectly ok, and the image is completely and
> correctly downloaded to RAM.
>
> The crash is occuring inside the function r4k_flush_icache_range().
>
> I tried 'flush -i' and 'flush -d' on YAMON after the download but before
> the
> 'go', but that didn't help. I also tried completely disabling caches and
> loading/running uncached, but it gave the same error.
>
> Now, the final twist! Using an ICE, I set a breakpoint at the
> r4k_flush_icache_range function. Then I loaded the kernel as usual, ran it
> with the ICE, stepped through a few instructions inside the
> r4k_flush_icache_range function and then did a 'cont'. The kernel now
> booted
> fine!
>
> If I don't set the breakpoint inside that function though, and just try to
> run with the ICE the same
> error (Inst fetch/Load error) occurs.
>
> I'm at a loss trying to figure out what's going on. I suspect it has
> something to do with caches perhaps (duh!), but have no clue what!
> Anybody
> out there face a similar kind of a situation before?
>
> Thanks in advance for any help offered.
>
> Regards,
> -Steve
>
> _________________________________________________________________
> MSN 9 Dial-up Internet Access helps fight spam and pop-ups – now 2 months
> FREE! http://join.msn.click-url.com/go/onm00200361ave/direct/01/
>
>
>

Re: Strange, strange occurence

[Ralf Baechle]

On Fri, Jul 09, 2004 at 06:50:00PM +0000, S C wrote:

> Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, when I 
> download the kernel via the debug ethernet port it runs fine. If I download 
> the kernel via the Tx4938 inbuilt ethernet controller, it crashes!

If you're using a Montavista kernel you should report to Montavista.  We
don't have the source so any comment here is speculation.

> The crash is occuring inside the function r4k_flush_icache_range().
> 
> I tried 'flush -i' and 'flush -d' on YAMON after the download but before 
> the 'go', but that didn't help. I also tried completely disabling caches 
> and loading/running uncached, but it gave the same error.
> 
> Now, the final twist! Using an ICE, I set a breakpoint at the 
> r4k_flush_icache_range function. Then I loaded the kernel as usual, ran it 
> with the ICE, stepped through a few instructions inside the 
> r4k_flush_icache_range function and then did a 'cont'. The kernel now 
> booted fine!

As already pointed out by the other poster Niels Sterrenburg using a
debugger unavoidably changes the state of the system to be debugged.

For at least some of the TX49xx processors there is a problem under certain
circumstances if a flush of an I-cache line flushes that cache instruction
itself.  Make sure you're not getting hit by that one.

  Ralf

Re: Strange, strange occurence

[Kevin D. Kissell]

[snip]
> > The crash is occuring inside the function r4k_flush_icache_range().
> > 
> > I tried 'flush -i' and 'flush -d' on YAMON after the download but before 
> > the 'go', but that didn't help. I also tried completely disabling caches 
> > and loading/running uncached, but it gave the same error.
> > 
> > Now, the final twist! Using an ICE, I set a breakpoint at the 
> > r4k_flush_icache_range function. Then I loaded the kernel as usual, ran it 
> > with the ICE, stepped through a few instructions inside the 
> > r4k_flush_icache_range function and then did a 'cont'. The kernel now 
> > booted fine!
> 
> As already pointed out by the other poster Niels Sterrenburg using a
> debugger unavoidably changes the state of the system to be debugged.
> 
> For at least some of the TX49xx processors there is a problem under certain
> circumstances if a flush of an I-cache line flushes that cache instruction
> itself.  Make sure you're not getting hit by that one.

It's not just the TX49xx series.  While many MIPS-compatible processors 
do handle the special case of flushing the active CACHE instruction itself, 
not all of them do, and the MIPS32 spec calls it out as having an "UNPREDICTABLE"
result.

A truly safe and general I-cache flush routine should itself run uncached,
but a cursory glance at the linux-mips.org sources makes me think
that we do not take that precaution by default - the flush_icache_range
pointer looks to be set to the address of r4k_flush_icache_range()
function, rather than its (uncacheable) alias in kseg1.  Is this something
that's fixed in a linker script, or are we just living dangerously?

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Kevin D. Kissell]

[snip]
> > The crash is occuring inside the function r4k_flush_icache_range().
> > 
> > I tried 'flush -i' and 'flush -d' on YAMON after the download but before 
> > the 'go', but that didn't help. I also tried completely disabling caches 
> > and loading/running uncached, but it gave the same error.
> > 
> > Now, the final twist! Using an ICE, I set a breakpoint at the 
> > r4k_flush_icache_range function. Then I loaded the kernel as usual, ran it 
> > with the ICE, stepped through a few instructions inside the 
> > r4k_flush_icache_range function and then did a 'cont'. The kernel now 
> > booted fine!
> 
> As already pointed out by the other poster Niels Sterrenburg using a
> debugger unavoidably changes the state of the system to be debugged.
> 
> For at least some of the TX49xx processors there is a problem under certain
> circumstances if a flush of an I-cache line flushes that cache instruction
> itself.  Make sure you're not getting hit by that one.

It's not just the TX49xx series.  While many MIPS-compatible processors 
do handle the special case of flushing the active CACHE instruction itself, 
not all of them do, and the MIPS32 spec calls it out as having an "UNPREDICTABLE"
result.

A truly safe and general I-cache flush routine should itself run uncached,
but a cursory glance at the linux-mips.org sources makes me think
that we do not take that precaution by default - the flush_icache_range
pointer looks to be set to the address of r4k_flush_icache_range()
function, rather than its (uncacheable) alias in kseg1.  Is this something
that's fixed in a linker script, or are we just living dangerously?

            Regards,

            Kevin K.

Re: Strange, strange occurence

[S C]

Hi Niels, Ralf, Kevin

Thanks for responding to my query! For the moment, I have fixed the 
symptoms, but still don't know what the problem is (it could be what you 
guys suspect it to be). The r4k_flush_icache_range() was being called from 
inside r4k_tlb_init() and the range being specified was KSEG0 to KSEG0 + 
0x80, i.e. the TLB handler. I commented out the call to 
r4k_flush_icache_range and now the kernel seems to boot normally! But I 
suspect I have not fixed a problem, merely sidestepped it.

So now maybe I should repeat the question Niels asked: why was that call 
necessary?

Thanks again,
-Steve.




>From: Ralf Baechle <ralf@linux-mips.org>
>To: S C <theansweriz42@hotmail.com>
>CC: linux-mips@linux-mips.org
>Subject: Re: Strange, strange occurence
>Date: Sat, 10 Jul 2004 12:04:12 +0200
>
>On Fri, Jul 09, 2004 at 06:50:00PM +0000, S C wrote:
>
> > Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, 
>when I
> > download the kernel via the debug ethernet port it runs fine. If I 
>download
> > the kernel via the Tx4938 inbuilt ethernet controller, it crashes!
>
>If you're using a Montavista kernel you should report to Montavista.  We
>don't have the source so any comment here is speculation.
>
> > The crash is occuring inside the function r4k_flush_icache_range().
> >
> > I tried 'flush -i' and 'flush -d' on YAMON after the download but before
> > the 'go', but that didn't help. I also tried completely disabling caches
> > and loading/running uncached, but it gave the same error.
> >
> > Now, the final twist! Using an ICE, I set a breakpoint at the
> > r4k_flush_icache_range function. Then I loaded the kernel as usual, ran 
>it
> > with the ICE, stepped through a few instructions inside the
> > r4k_flush_icache_range function and then did a 'cont'. The kernel now
> > booted fine!
>
>As already pointed out by the other poster Niels Sterrenburg using a
>debugger unavoidably changes the state of the system to be debugged.
>
>For at least some of the TX49xx processors there is a problem under certain
>circumstances if a flush of an I-cache line flushes that cache instruction
>itself.  Make sure you're not getting hit by that one.
>
>   Ralf

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar – get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: Strange, strange occurence

[S C]

And thinking about it a little more, I might as well clarfy my understanding 
while we're on the topic.

Here's a code snippet from r4k_tlb_init() in arch/mips/mm/c-r4k.c

memcpy((void *)KSEG0, &except_vec0_r4000, 0x80);
flush_icache_range(KSEG0, KSEG0 + 0x80);

So my understanding is that the TLB exception handler is being copied to the 
right memory location, and just in case some other TLB exception handler 
(YAMON's presumably) is residing in I-cache, to flush ( hit invalidate) it. 
Is this correct?

Shouldn't there be code to writeback_invalidate the exception handler from 
the data cache ? Presumably the memcpy causes the TLB handler to lodge 
itself in the D cache till it is moved to RAM (either explicitly or when 
some other lines replace the cache lines where the handler is), right?

Thanks in advance for helping me understand the issue here.

Regards,
-Steve.





>From: Ralf Baechle <ralf@linux-mips.org>
>To: S C <theansweriz42@hotmail.com>
>CC: linux-mips@linux-mips.org
>Subject: Re: Strange, strange occurence
>Date: Sat, 10 Jul 2004 12:04:12 +0200
>
>On Fri, Jul 09, 2004 at 06:50:00PM +0000, S C wrote:
>
> > Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, 
>when I
> > download the kernel via the debug ethernet port it runs fine. If I 
>download
> > the kernel via the Tx4938 inbuilt ethernet controller, it crashes!
>
>If you're using a Montavista kernel you should report to Montavista.  We
>don't have the source so any comment here is speculation.
>
> > The crash is occuring inside the function r4k_flush_icache_range().
> >
> > I tried 'flush -i' and 'flush -d' on YAMON after the download but before
> > the 'go', but that didn't help. I also tried completely disabling caches
> > and loading/running uncached, but it gave the same error.
> >
> > Now, the final twist! Using an ICE, I set a breakpoint at the
> > r4k_flush_icache_range function. Then I loaded the kernel as usual, ran 
>it
> > with the ICE, stepped through a few instructions inside the
> > r4k_flush_icache_range function and then did a 'cont'. The kernel now
> > booted fine!
>
>As already pointed out by the other poster Niels Sterrenburg using a
>debugger unavoidably changes the state of the system to be debugged.
>
>For at least some of the TX49xx processors there is a problem under certain
>circumstances if a flush of an I-cache line flushes that cache instruction
>itself.  Make sure you're not getting hit by that one.
>
>   Ralf

_________________________________________________________________
Check out the latest news, polls and tools in the MSN 2004 Election Guide! 
http://special.msn.com/msn/election2004.armx

Re: Strange, strange occurence

[Kevin D. Kissell]

> And thinking about it a little more, I might as well clarfy my understanding 
> while we're on the topic.
> 
> Here's a code snippet from r4k_tlb_init() in arch/mips/mm/c-r4k.c
> 
> memcpy((void *)KSEG0, &except_vec0_r4000, 0x80);
> flush_icache_range(KSEG0, KSEG0 + 0x80);
> 
> So my understanding is that the TLB exception handler is being copied to the 
> right memory location, and just in case some other TLB exception handler 
> (YAMON's presumably) is residing in I-cache, to flush ( hit invalidate) it. 
> Is this correct?
> 
> Shouldn't there be code to writeback_invalidate the exception handler from 
> the data cache ? Presumably the memcpy causes the TLB handler to lodge 
> itself in the D cache till it is moved to RAM (either explicitly or when 
> some other lines replace the cache lines where the handler is), right?
> 
> Thanks in advance for helping me understand the issue here.

Your intuition is correct, and the code in r4k_tlb_init() does look scary.
But at least in the linux-mips CVS tree, flush_icache_range() tests to see
if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
is set, and if it isn't, it pushes the specified range out of the Dcache before
flushing the Icache.  I would speculate that either your c-r4k.c is out of
sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
set.

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Kevin D. Kissell]

> And thinking about it a little more, I might as well clarfy my understanding 
> while we're on the topic.
> 
> Here's a code snippet from r4k_tlb_init() in arch/mips/mm/c-r4k.c
> 
> memcpy((void *)KSEG0, &except_vec0_r4000, 0x80);
> flush_icache_range(KSEG0, KSEG0 + 0x80);
> 
> So my understanding is that the TLB exception handler is being copied to the 
> right memory location, and just in case some other TLB exception handler 
> (YAMON's presumably) is residing in I-cache, to flush ( hit invalidate) it. 
> Is this correct?
> 
> Shouldn't there be code to writeback_invalidate the exception handler from 
> the data cache ? Presumably the memcpy causes the TLB handler to lodge 
> itself in the D cache till it is moved to RAM (either explicitly or when 
> some other lines replace the cache lines where the handler is), right?
> 
> Thanks in advance for helping me understand the issue here.

Your intuition is correct, and the code in r4k_tlb_init() does look scary.
But at least in the linux-mips CVS tree, flush_icache_range() tests to see
if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
is set, and if it isn't, it pushes the specified range out of the Dcache before
flushing the Icache.  I would speculate that either your c-r4k.c is out of
sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
set.

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Kevin D. Kissell]

> Your intuition is correct, and the code in r4k_tlb_init() does look scary.
> But at least in the linux-mips CVS tree, flush_icache_range() tests to see
> if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
> is set, and if it isn't, it pushes the specified range out of the Dcache before
> flushing the Icache.  I would speculate that either your c-r4k.c is out of
> sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
> set.

Hmm.  On closer examination, there *is* a bug in the current r4k_flush_icache_range(),
in that it computes its cache flush loop for the I-cache based on the D-cache line size.
Those line sizes are *usually* the same.  By any chance are they different for the
TX49 family?  If the icache line is longer than the dcache line, there should be no
functional problem, just some wasted cycles.  But if the dcache line were, say, 
twice the length of the Icache line, only half of the icache lines would be invalidated...

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Kevin D. Kissell]

> Your intuition is correct, and the code in r4k_tlb_init() does look scary.
> But at least in the linux-mips CVS tree, flush_icache_range() tests to see
> if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
> is set, and if it isn't, it pushes the specified range out of the Dcache before
> flushing the Icache.  I would speculate that either your c-r4k.c is out of
> sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
> set.

Hmm.  On closer examination, there *is* a bug in the current r4k_flush_icache_range(),
in that it computes its cache flush loop for the I-cache based on the D-cache line size.
Those line sizes are *usually* the same.  By any chance are they different for the
TX49 family?  If the icache line is longer than the dcache line, there should be no
functional problem, just some wasted cycles.  But if the dcache line were, say, 
twice the length of the Icache line, only half of the icache lines would be invalidated...

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Ralf Baechle]

On Mon, Jul 12, 2004 at 09:23:20PM +0000, S C wrote:

> And thinking about it a little more, I might as well clarfy my 
> understanding while we're on the topic.
> 
> Here's a code snippet from r4k_tlb_init() in arch/mips/mm/c-r4k.c
> 
> memcpy((void *)KSEG0, &except_vec0_r4000, 0x80);
> flush_icache_range(KSEG0, KSEG0 + 0x80);
> 
> So my understanding is that the TLB exception handler is being copied to 
> the right memory location, and just in case some other TLB exception 
> handler (YAMON's presumably) is residing in I-cache, to flush ( hit 
> invalidate) it. Is this correct?
> 
> Shouldn't there be code to writeback_invalidate the exception handler from 
> the data cache ?

flush_icache_range() does that where necessary.

> Presumably the memcpy causes the TLB handler to lodge 
> itself in the D cache till it is moved to RAM

Correct.  All MIPS processors [1] do their primary I-cache refills from
second or third level cache or memory - whatever hits first.  If code
was changed a flush of the data cache is required so the I-cache can
actually fetch the new data and because old stale code might still be
in the I-cache an I-cache flush is also required.

> (either explicitly or when 
> some other lines replace the cache lines where the handler is), right?
> 
> Thanks in advance for helping me understand the issue here.

  Ralf

[1] Exceptions are the R4000/R4400 SC and MC versions in a split S-cache
    configuration where the primary I-cache is refilled from the secondary
    I-cache which mean this flush has to flush the secondary data cache
    back to memory also.  Linux doesn't support this configuration because
    no known system uses it iow it's only a theoretically possible config.

    The second exception are the AMD MIPS32 processors where the I-cache
    is snooping the D-cache and therefore the D-cache flush can is
    unnecessary.

    The third exception are R1x000 in SMP configurations where I-caches
    snoop remote stores so coherency doesn't need any maintenance in
    software at all.  Only the I-cache of the CPU that did modify the code
    needs flushing.

Re: Strange, strange occurence

[S C]

Hi Kevin and Ralf,

Thanks for your inputs and suggestions! In the case of the Tx49 family, the 
primary I and D cache lines are both the same size (8 words), so the problem 
you mention below will not arise.

  I didn't think about the meaning of cpu_has_ic_fills_f_dc before writing 
my previous mail, and I see now that my intuition (and your explanation 
helps) was correct.

  For the moment, the problem is fixed. But I'm going to try and get to the 
bottom of this when I have the time.

  Thanks,
-Steve.


>From: "Kevin D. Kissell" <KevinK@mips.com>
>To: "Kevin D. Kissell" <KevinK@mips.com>, "S C" 
><theansweriz42@hotmail.com>,        "Ralf Baechle" <ralf@linux-mips.org>
>CC: <linux-mips@linux-mips.org>
>Subject: Re: Strange, strange occurence
>Date: Tue, 13 Jul 2004 00:25:37 +0200
>
> > Your intuition is correct, and the code in r4k_tlb_init() does look 
>scary.
> > But at least in the linux-mips CVS tree, flush_icache_range() tests to 
>see
> > if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
> > is set, and if it isn't, it pushes the specified range out of the Dcache 
>before
> > flushing the Icache.  I would speculate that either your c-r4k.c is out 
>of
> > sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
> > set.
>
>Hmm.  On closer examination, there *is* a bug in the current 
>r4k_flush_icache_range(),
>in that it computes its cache flush loop for the I-cache based on the 
>D-cache line size.
>Those line sizes are *usually* the same.  By any chance are they different 
>for the
>TX49 family?  If the icache line is longer than the dcache line, there 
>should be no
>functional problem, just some wasted cycles.  But if the dcache line were, 
>say,
>twice the length of the Icache line, only half of the icache lines would be 
>invalidated...
>
>             Regards,
>
>             Kevin K.
>

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar – get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: Strange, strange occurence

[Ralf Baechle]

On Mon, Jul 12, 2004 at 11:48:31PM +0200, Kevin D. Kissell wrote:

> Your intuition is correct, and the code in r4k_tlb_init() does look scary.

Not scarry at all.  flush_icache_range() has to do whatever is needed to
maintain I-cache coherency for the range passed in as the argument.  And
I don't think we should really have to deal with all the complicated
details of cache maintenance in a function like r4k_tlb_init().

> But at least in the linux-mips CVS tree, flush_icache_range() tests to see
> if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)

Right.  Cpu_has_ic_fills_f_dc is only non-zero for the AMD processors
where the I-cache is refilled from the D-cache.  For typical kernel
configurations The definition of cpu_has_ic_fills_f_dc is a constant so
the compiler can optimize this further.

  Ralf

Re: Strange, strange occurence

[Ralf Baechle]

On Tue, Jul 13, 2004 at 12:25:37AM +0200, Kevin D. Kissell wrote:

> Hmm.  On closer examination, there *is* a bug in the current r4k_flush_icache_range(),
> in that it computes its cache flush loop for the I-cache based on the D-cache line size.
> Those line sizes are *usually* the same.  By any chance are they different for the
> TX49 family?  If the icache line is longer than the dcache line, there should be no
> functional problem, just some wasted cycles.  But if the dcache line were, say, 
> twice the length of the Icache line, only half of the icache lines would be invalidated...

Whoops.  Fixing ...

  Ralf

Re: Strange, strange occurence

[Ralf Baechle]

On Mon, Jul 12, 2004 at 05:16:31PM +0200, Kevin D. Kissell wrote:

> A truly safe and general I-cache flush routine should itself run uncached,
> but a cursory glance at the linux-mips.org sources makes me think
> that we do not take that precaution by default - the flush_icache_range
> pointer looks to be set to the address of r4k_flush_icache_range()
> function, rather than its (uncacheable) alias in kseg1.  Is this something
> that's fixed in a linker script, or are we just living dangerously?

That's a new restriction in MIPS32 v2.0 and you're right, we're not trying
to deal with it yet except for the TX49xx.

  Ralf

Re: Strange, strange occurence

[Kevin D. Kissell]

> > A truly safe and general I-cache flush routine should itself run uncached,
> > but a cursory glance at the linux-mips.org sources makes me think
> > that we do not take that precaution by default - the flush_icache_range
> > pointer looks to be set to the address of r4k_flush_icache_range()
> > function, rather than its (uncacheable) alias in kseg1.  Is this something
> > that's fixed in a linker script, or are we just living dangerously?
> 
> That's a new restriction in MIPS32 v2.0 and you're right, we're not trying
> to deal with it yet except for the TX49xx.

I'm pretty sure that restriction is not new to MIPS32 v2.0.  In any
case, there are pre-MIPS32/MIPS64 chips in current mass production
and use, under Linux among other OSes, which specify in their user
manuals that one should not invalidate the Icache line from which one
is currently executing.  The clause about unpredictable behavior in
that case went into the MIPS32 spec because it was known that such
parts existed, and we wanted to make it as easy as possible for such 
designs to be made compliant

Invalidating the entire Icache with a routine executing out of the Icache
is a Bad Idea, and will almost certainly cause problems some of the time
on some MIPS processors.  Reasonable people could disagree on whether
we want to handle that in the generic code, or create a variant icache flush 
routine which gets plugged in only for those parts that really need it.

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Kevin D. Kissell]

> > A truly safe and general I-cache flush routine should itself run uncached,
> > but a cursory glance at the linux-mips.org sources makes me think
> > that we do not take that precaution by default - the flush_icache_range
> > pointer looks to be set to the address of r4k_flush_icache_range()
> > function, rather than its (uncacheable) alias in kseg1.  Is this something
> > that's fixed in a linker script, or are we just living dangerously?
> 
> That's a new restriction in MIPS32 v2.0 and you're right, we're not trying
> to deal with it yet except for the TX49xx.

I'm pretty sure that restriction is not new to MIPS32 v2.0.  In any
case, there are pre-MIPS32/MIPS64 chips in current mass production
and use, under Linux among other OSes, which specify in their user
manuals that one should not invalidate the Icache line from which one
is currently executing.  The clause about unpredictable behavior in
that case went into the MIPS32 spec because it was known that such
parts existed, and we wanted to make it as easy as possible for such 
designs to be made compliant

Invalidating the entire Icache with a routine executing out of the Icache
is a Bad Idea, and will almost certainly cause problems some of the time
on some MIPS processors.  Reasonable people could disagree on whether
we want to handle that in the generic code, or create a variant icache flush 
routine which gets plugged in only for those parts that really need it.

            Regards,

            Kevin K.

Re: Strange, strange occurence

[Maciej W. Rozycki]

On Tue, 13 Jul 2004, Kevin D. Kissell wrote:

> > That's a new restriction in MIPS32 v2.0 and you're right, we're not trying
> > to deal with it yet except for the TX49xx.
> 
> I'm pretty sure that restriction is not new to MIPS32 v2.0.  In any

 The restriction was apparently added in revision 1.00 of the MIPS32
architecture vol.II document.  I don't have that revision -- I have
versions 0.95 and 2.00 only -- without looking at the revision history in
2.00 I'd expect the original MIPS32 spec not to enforce such a
restriction, either.

> case, there are pre-MIPS32/MIPS64 chips in current mass production
> and use, under Linux among other OSes, which specify in their user
> manuals that one should not invalidate the Icache line from which one
> is currently executing.  The clause about unpredictable behavior in
> that case went into the MIPS32 spec because it was known that such
> parts existed, and we wanted to make it as easy as possible for such 
> designs to be made compliant

 Ugh, although I can imagine valid arguments for such a decision.

> Invalidating the entire Icache with a routine executing out of the Icache
> is a Bad Idea, and will almost certainly cause problems some of the time
> on some MIPS processors.  Reasonable people could disagree on whether
> we want to handle that in the generic code, or create a variant icache flush 
> routine which gets plugged in only for those parts that really need it.

 As executing code from an uncached space is terribly slow, there are at
least two points of optimization:

1. The Icache invalidation handler should run cached on processors known
to handle it gracefully.

2. For others, as you suggest, it should attempt to figure out whether its
code may invalidate itself and run uncached then, perhaps for the
problematic lines only.

  Maciej

Re: Strange, strange occurence

[Dominic Sweetman]

Ralf Baechle (ralf@linux-mips.org) writes:

> > A truly safe and general I-cache flush routine should itself run
> > uncached... 

It depends what you mean by general, and uncached is not the only
option.  The spec says:

 "The operation of the instruction is UNPREDICTABLE if the cache line
  that contains the CACHE instruction is the target of an
  invalidate..." 

If you use hit-type cache operations in a kernel routine, then you're
safe.  I can't envisage any circumstance in which Linux would try to
invalidate kernel mainline code locations from the I-cache (well, you
might be doing something fabulous with debugging the kernel, but
that's not normal and you'd hardly expect to be able to support such
an activity with standard cache management calls).

So this problem can only arise on index-type I-cache invalidation.  I
claim that a running kernel on a MIPS CPU should only use index-type
invalidation when it is necessary to invalidate the entire I-cache.
(If you use index-type operations for a range which doesn't resolve to
"the whole cache" then that should be fixed).

That implies that a MIPS32-paranoid "invalidate-whole-I-cache" routine
should:

1. Identify which indexes might alias to cache lines 
   containing the routines's own 'cache invalidate' instruction(s),
   and thus hit the problem.  There won't be that many of them.

2. Arrange to skip those indexes when zapping the cache, then do
   something weird to invalidate that handful of lines.  You could 
   do that by running uncached, but you could also do it just by using
   some auxiliary routine which is known to be more than a cache line
   but much less than a whole I-cache span distant, so can't possibly
   alias to the same thing...

This is fiddly, but not terribly difficult and should have a
negligible performance impact.

Does that make sense?  Am I now, having named the solution,
responsible for figuring out a patch (yeuch, I never wanted to be a
kernel programmer again...).

--
Dominic Sweetman
MIPS Technologies

RE: Strange, strange occurence

[Michael Uhler]

Dom's proposed solution is probably the right thing to do.  We've got some
code in MIPS that does exactly this, and I've suggested that we convert this
to provide to the Linux community.

/gmu

---
Michael Uhler, Chief Technology Officer
MIPS Technologies, Inc.   Email: uhler@mips.com
1225 Charleston Road      Voice:  (650)567-5025   FAX:   (650)567-5225
Mountain View, CA 94043   Mobile: (650)868-6870   Admin: (650)567-5085


> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org 
> [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of 
> Dominic Sweetman
> Sent: Wednesday, July 14, 2004 9:35 AM
> To: Ralf Baechle
> Cc: Kevin D. Kissell; S C; linux-mips@linux-mips.org
> Subject: Re: Strange, strange occurence
> 
> 
> 
> Ralf Baechle (ralf@linux-mips.org) writes:
> 
> > > A truly safe and general I-cache flush routine should itself run 
> > > uncached...
> 
> It depends what you mean by general, and uncached is not the 
> only option.  The spec says:
> 
>  "The operation of the instruction is UNPREDICTABLE if the cache line
>   that contains the CACHE instruction is the target of an
>   invalidate..." 
> 
> If you use hit-type cache operations in a kernel routine, 
> then you're safe.  I can't envisage any circumstance in which 
> Linux would try to invalidate kernel mainline code locations 
> from the I-cache (well, you might be doing something fabulous 
> with debugging the kernel, but that's not normal and you'd 
> hardly expect to be able to support such an activity with 
> standard cache management calls).
> 
> So this problem can only arise on index-type I-cache 
> invalidation.  I claim that a running kernel on a MIPS CPU 
> should only use index-type invalidation when it is necessary 
> to invalidate the entire I-cache. (If you use index-type 
> operations for a range which doesn't resolve to "the whole 
> cache" then that should be fixed).
> 
> That implies that a MIPS32-paranoid "invalidate-whole-I-cache" routine
> should:
> 
> 1. Identify which indexes might alias to cache lines 
>    containing the routines's own 'cache invalidate' instruction(s),
>    and thus hit the problem.  There won't be that many of them.
> 
> 2. Arrange to skip those indexes when zapping the cache, then do
>    something weird to invalidate that handful of lines.  You could 
>    do that by running uncached, but you could also do it just by using
>    some auxiliary routine which is known to be more than a cache line
>    but much less than a whole I-cache span distant, so can't possibly
>    alias to the same thing...
> 
> This is fiddly, but not terribly difficult and should have a 
> negligible performance impact.
> 
> Does that make sense?  Am I now, having named the solution, 
> responsible for figuring out a patch (yeuch, I never wanted 
> to be a kernel programmer again...).
> 
> --
> Dominic Sweetman
> MIPS Technologies
> 
> 

RE: Strange, strange occurence

[Michael Uhler]

Dom's proposed solution is probably the right thing to do.  We've got some
code in MIPS that does exactly this, and I've suggested that we convert this
to provide to the Linux community.

/gmu

---
Michael Uhler, Chief Technology Officer
MIPS Technologies, Inc.   Email: uhler@mips.com
1225 Charleston Road      Voice:  (650)567-5025   FAX:   (650)567-5225
Mountain View, CA 94043   Mobile: (650)868-6870   Admin: (650)567-5085


> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org 
> [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of 
> Dominic Sweetman
> Sent: Wednesday, July 14, 2004 9:35 AM
> To: Ralf Baechle
> Cc: Kevin D. Kissell; S C; linux-mips@linux-mips.org
> Subject: Re: Strange, strange occurence
> 
> 
> 
> Ralf Baechle (ralf@linux-mips.org) writes:
> 
> > > A truly safe and general I-cache flush routine should itself run 
> > > uncached...
> 
> It depends what you mean by general, and uncached is not the 
> only option.  The spec says:
> 
>  "The operation of the instruction is UNPREDICTABLE if the cache line
>   that contains the CACHE instruction is the target of an
>   invalidate..." 
> 
> If you use hit-type cache operations in a kernel routine, 
> then you're safe.  I can't envisage any circumstance in which 
> Linux would try to invalidate kernel mainline code locations 
> from the I-cache (well, you might be doing something fabulous 
> with debugging the kernel, but that's not normal and you'd 
> hardly expect to be able to support such an activity with 
> standard cache management calls).
> 
> So this problem can only arise on index-type I-cache 
> invalidation.  I claim that a running kernel on a MIPS CPU 
> should only use index-type invalidation when it is necessary 
> to invalidate the entire I-cache. (If you use index-type 
> operations for a range which doesn't resolve to "the whole 
> cache" then that should be fixed).
> 
> That implies that a MIPS32-paranoid "invalidate-whole-I-cache" routine
> should:
> 
> 1. Identify which indexes might alias to cache lines 
>    containing the routines's own 'cache invalidate' instruction(s),
>    and thus hit the problem.  There won't be that many of them.
> 
> 2. Arrange to skip those indexes when zapping the cache, then do
>    something weird to invalidate that handful of lines.  You could 
>    do that by running uncached, but you could also do it just by using
>    some auxiliary routine which is known to be more than a cache line
>    but much less than a whole I-cache span distant, so can't possibly
>    alias to the same thing...
> 
> This is fiddly, but not terribly difficult and should have a 
> negligible performance impact.
> 
> Does that make sense?  Am I now, having named the solution, 
> responsible for figuring out a patch (yeuch, I never wanted 
> to be a kernel programmer again...).
> 
> --
> Dominic Sweetman
> MIPS Technologies
> 
> 

Re: Strange, strange occurence

[Atsushi Nemoto]

>>>>> On Wed, 14 Jul 2004 17:35:19 +0100, Dominic Sweetman <dom@mips.com> said:
dom> 2. Arrange to skip those indexes when zapping the cache, then do
dom> something weird to invalidate that handful of lines.  You could
dom> do that by running uncached, but you could also do it just by
dom> using some auxiliary routine which is known to be more than a
dom> cache line but much less than a whole I-cache span distant, so
dom> can't possibly alias to the same thing...

dom> This is fiddly, but not terribly difficult and should have a
dom> negligible performance impact.

Yes.  The cache routines for TX49XX surely do it (2 phase
invalidating).  Please look at tx49_blast_icache32() in c-r4k.c.

---
Atsushi Nemoto

Re: Strange, strange occurence

[Ralf Baechle]

On Wed, Jul 14, 2004 at 05:35:19PM +0100, Dominic Sweetman wrote:

> If you use hit-type cache operations in a kernel routine, then you're
> safe.  I can't envisage any circumstance in which Linux would try to
> invalidate kernel mainline code locations from the I-cache (well, you
> might be doing something fabulous with debugging the kernel, but
> that's not normal and you'd hardly expect to be able to support such
> an activity with standard cache management calls).
> 
> So this problem can only arise on index-type I-cache invalidation.  I
> claim that a running kernel on a MIPS CPU should only use index-type
> invalidation when it is necessary to invalidate the entire I-cache.
> (If you use index-type operations for a range which doesn't resolve to
> "the whole cache" then that should be fixed).
> 
> That implies that a MIPS32-paranoid "invalidate-whole-I-cache" routine
> should:
> 
> 1. Identify which indexes might alias to cache lines 
>    containing the routines's own 'cache invalidate' instruction(s),
>    and thus hit the problem.  There won't be that many of them.
> 
> 2. Arrange to skip those indexes when zapping the cache, then do
>    something weird to invalidate that handful of lines.  You could 
>    do that by running uncached, but you could also do it just by using
>    some auxiliary routine which is known to be more than a cache line
>    but much less than a whole I-cache span distant, so can't possibly
>    alias to the same thing...
> 
> This is fiddly, but not terribly difficult and should have a
> negligible performance impact.
> 
> Does that make sense?  Am I now, having named the solution,
> responsible for figuring out a patch (yeuch, I never wanted to be a
> kernel programmer again...).

You don't have to :-)  What became a architectural restriction for MIPS32
did already show up earlier as an erratum for the TX49/H2 core.  This is
the solution which we currently have in the kernel code:

#define JUMP_TO_ALIGN(order) \
	__asm__ __volatile__( \
		"b\t1f\n\t" \
		".align\t" #order "\n\t" \
		"1:\n\t" \
		)
#define CACHE32_UNROLL32_ALIGN	JUMP_TO_ALIGN(10) /* 32 * 32 = 1024 */
#define CACHE32_UNROLL32_ALIGN2	JUMP_TO_ALIGN(11)

static inline void mips32_blast_icache32(void)
{
	unsigned long start = INDEX_BASE;
	unsigned long end = start + current_cpu_data.icache.waysize;
	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
	unsigned long ws_end = current_cpu_data.icache.ways <<
	                       current_cpu_data.icache.waybit;
	unsigned long ws, addr;

	CACHE32_UNROLL32_ALIGN2;
	/* I'm in even chunk.  blast odd chunks */
	for (ws = 0; ws < ws_end; ws += ws_inc) 
		for (addr = start + 0x400; addr < end; addr += 0x400 * 2) 
			cache32_unroll32(addr|ws,Index_Invalidate_I);
	CACHE32_UNROLL32_ALIGN;
	/* I'm in odd chunk.  blast even chunks */
	for (ws = 0; ws < ws_end; ws += ws_inc) 
		for (addr = start; addr < end; addr += 0x400 * 2) 
			cache32_unroll32(addr|ws,Index_Invalidate_I);
}

All it takes is using this for all MIPS32 / MIPS64 or maybe even all
processors and some tuning of constants to make this suitable for
all possible I-cache configurations.

  Ralf

Re: Strange, strange occurence

[Ralf Baechle]

On Wed, Jul 14, 2004 at 05:35:19PM +0100, Dominic Sweetman wrote:

> This is fiddly, but not terribly difficult and should have a
> negligible performance impact.
> 
> Does that make sense?  Am I now, having named the solution,
> responsible for figuring out a patch (yeuch, I never wanted to be a
> kernel programmer again...).

No and yes - but here is a simpler solution.  Below patch solves the
problem and adds just 32 bytes of code but removes the special case for
TX49/H2 entirely.

  Ralf

 arch/mips/mm/c-r4k.c        |   59 -------------------------------------------- include/asm-mips/r4kcache.h |   18 ++++++++-----
 include/asm-mips/war.h      |   14 ----------
 3 files changed, 13 insertions(+), 78 deletions(-)


Index: arch/mips/mm/c-r4k.c
===================================================================
RCS file: /home/cvs/linux/arch/mips/mm/c-r4k.c,v
retrieving revision 1.88
diff -u -r1.88 c-r4k.c
--- arch/mips/mm/c-r4k.c	16 Jul 2004 12:06:13 -0000	1.88
+++ arch/mips/mm/c-r4k.c	16 Jul 2004 12:17:05 -0000
@@ -96,16 +96,6 @@
 		r4k_blast_dcache = blast_dcache32;
 }
 
-/* force code alignment (used for TX49XX_ICACHE_INDEX_INV_WAR) */
-#define JUMP_TO_ALIGN(order) \
-	__asm__ __volatile__( \
-		"b\t1f\n\t" \
-		".align\t" #order "\n\t" \
-		"1:\n\t" \
-		)
-#define CACHE32_UNROLL32_ALIGN	JUMP_TO_ALIGN(10) /* 32 * 32 = 1024 */
-#define CACHE32_UNROLL32_ALIGN2	JUMP_TO_ALIGN(11)
-
 static inline void blast_r4600_v1_icache32(void)
 {
 	unsigned long flags;
@@ -115,27 +105,6 @@
 	local_irq_restore(flags);
 }
 
-static inline void tx49_blast_icache32(void)
-{
-	unsigned long start = INDEX_BASE;
-	unsigned long end = start + current_cpu_data.icache.waysize;
-	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
-	unsigned long ws_end = current_cpu_data.icache.ways <<
-	                       current_cpu_data.icache.waybit;
-	unsigned long ws, addr;
-
-	CACHE32_UNROLL32_ALIGN2;
-	/* I'm in even chunk.  blast odd chunks */
-	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start + 0x400; addr < end; addr += 0x400 * 2) 
-			cache32_unroll32(addr|ws,Index_Invalidate_I);
-	CACHE32_UNROLL32_ALIGN;
-	/* I'm in odd chunk.  blast even chunks */
-	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start; addr < end; addr += 0x400 * 2) 
-			cache32_unroll32(addr|ws,Index_Invalidate_I);
-}
-
 static inline void blast_icache32_r4600_v1_page_indexed(unsigned long page)
 {
 	unsigned long flags;
@@ -145,27 +114,6 @@
 	local_irq_restore(flags);
 }
 
-static inline void tx49_blast_icache32_page_indexed(unsigned long page)
-{
-	unsigned long start = page;
-	unsigned long end = start + PAGE_SIZE;
-	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
-	unsigned long ws_end = current_cpu_data.icache.ways <<
-	                       current_cpu_data.icache.waybit;
-	unsigned long ws, addr;
-
-	CACHE32_UNROLL32_ALIGN2;
-	/* I'm in even chunk.  blast odd chunks */
-	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start + 0x400; addr < end; addr += 0x400 * 2) 
-			cache32_unroll32(addr|ws,Index_Invalidate_I);
-	CACHE32_UNROLL32_ALIGN;
-	/* I'm in odd chunk.  blast even chunks */
-	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start; addr < end; addr += 0x400 * 2) 
-			cache32_unroll32(addr|ws,Index_Invalidate_I);
-}
-
 static void (* r4k_blast_icache_page)(unsigned long addr);
 
 static inline void r4k_blast_icache_page_setup(void)
@@ -190,10 +138,7 @@
 	if (ic_lsize == 16)
 		r4k_blast_icache_page_indexed = blast_icache16_page_indexed;
 	else if (ic_lsize == 32) {
-		if (TX49XX_ICACHE_INDEX_INV_WAR)
-			r4k_blast_icache_page_indexed =
-				tx49_blast_icache32_page_indexed;
-		else if (R4600_V1_INDEX_ICACHEOP_WAR && cpu_is_r4600_v1_x())
+		if (R4600_V1_INDEX_ICACHEOP_WAR && cpu_is_r4600_v1_x())
 			r4k_blast_icache_page_indexed =
 				blast_icache32_r4600_v1_page_indexed;
 		else
@@ -214,8 +159,6 @@
 	else if (ic_lsize == 32) {
 		if (R4600_V1_INDEX_ICACHEOP_WAR && cpu_is_r4600_v1_x())
 			r4k_blast_icache = blast_r4600_v1_icache32;
-		else if (TX49XX_ICACHE_INDEX_INV_WAR)
-			r4k_blast_icache = tx49_blast_icache32;
 		else
 			r4k_blast_icache = blast_icache32;
 	} else if (ic_lsize == 64)
Index: include/asm-mips/r4kcache.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/r4kcache.h,v
retrieving revision 1.22
diff -u -r1.22 r4kcache.h
--- include/asm-mips/r4kcache.h	5 Jan 2004 01:56:01 -0000	1.22
+++ include/asm-mips/r4kcache.h	16 Jul 2004 12:17:05 -0000
@@ -192,7 +192,7 @@
 
 static inline void blast_icache16(void)
 {
-	unsigned long start = INDEX_BASE;
+	unsigned long start = (unsigned long) &&body + 0x100;
 	unsigned long end = start + current_cpu_data.icache.waysize;
 	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
 	unsigned long ws_end = current_cpu_data.icache.ways <<
@@ -200,8 +200,10 @@
 	unsigned long ws, addr;
 
 	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start; addr < end; addr += 0x200) 
+		for (addr = start; addr < end; addr += 0x200) {
+body:
 			cache16_unroll32(addr|ws,Index_Invalidate_I);
+		}
 }
 
 static inline void blast_icache16_page(unsigned long page)
@@ -335,7 +337,7 @@
 
 static inline void blast_icache32(void)
 {
-	unsigned long start = INDEX_BASE;
+	unsigned long start = (unsigned long) &&body + 0x200;
 	unsigned long end = start + current_cpu_data.icache.waysize;
 	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
 	unsigned long ws_end = current_cpu_data.icache.ways <<
@@ -343,8 +345,10 @@
 	unsigned long ws, addr;
 
 	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start; addr < end; addr += 0x400) 
+		for (addr = start; addr < end; addr += 0x400)  {
+body:
 			cache32_unroll32(addr|ws,Index_Invalidate_I);
+		}
 }
 
 static inline void blast_icache32_page(unsigned long page)
@@ -439,7 +443,7 @@
 
 static inline void blast_icache64(void)
 {
-	unsigned long start = INDEX_BASE;
+	unsigned long start = (unsigned long) &&body + 0x400;
 	unsigned long end = start + current_cpu_data.icache.waysize;
 	unsigned long ws_inc = 1UL << current_cpu_data.icache.waybit;
 	unsigned long ws_end = current_cpu_data.icache.ways <<
@@ -447,8 +451,10 @@
 	unsigned long ws, addr;
 
 	for (ws = 0; ws < ws_end; ws += ws_inc) 
-		for (addr = start; addr < end; addr += 0x800) 
+		for (addr = start; addr < end; addr += 0x800) {
+body:
 			cache64_unroll32(addr|ws,Index_Invalidate_I);
+		}
 }
 
 static inline void blast_icache64_page(unsigned long page)
Index: include/asm-mips/war.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/war.h,v
retrieving revision 1.18
diff -u -r1.18 war.h
--- include/asm-mips/war.h	5 Mar 2004 02:47:19 -0000	1.18
+++ include/asm-mips/war.h	16 Jul 2004 12:17:05 -0000
@@ -158,17 +158,6 @@
 #endif
 
 /*
- * From TX49/H2 manual: "If the instruction (i.e. CACHE) is issued for
- * the line which this instruction itself exists, the following
- * operation is not guaranteed."
- *
- * Workaround: do two phase flushing for Index_Invalidate_I
- */
-#ifdef CONFIG_CPU_TX49XX
-#define TX49XX_ICACHE_INDEX_INV_WAR 1
-#endif
-
-/*
  * On the RM9000 there is a problem which makes the CreateDirtyExclusive
  * cache operation unusable on SMP systems.
  */
@@ -203,9 +192,6 @@
 #ifndef MIPS_CACHE_SYNC_WAR
 #define MIPS_CACHE_SYNC_WAR		0
 #endif
-#ifndef TX49XX_ICACHE_INDEX_INV_WAR
-#define TX49XX_ICACHE_INDEX_INV_WAR	0
-#endif
 #ifndef RM9000_CDEX_SMP_WAR
 #define RM9000_CDEX_SMP_WAR		0
 #endif

Re: Strange, strange occurence

[Atsushi Nemoto]

>>>>> On Fri, 16 Jul 2004 14:24:09 +0200, Ralf Baechle <ralf@linux-mips.org> said:

ralf> No and yes - but here is a simpler solution.  Below patch solves
ralf> the problem and adds just 32 bytes of code but removes the
ralf> special case for TX49/H2 entirely.

Hmm... Does this patch really solves the problem?

Here is a disassemble list of blast_icache32 (gcc 3.3.3).  The
comments on CACHE instructions are indices of cachelines which the
instruction invalidates.  (for 4 way 16KB cache --- 4KB waysize).

80039ca0 <blast_icache32>:
80039ca0:       3c048038        lui     a0,0x8038
80039ca4:       8c849e2c        lw      a0,-25044(a0)
80039ca8:       3c028038        lui     v0,0x8038
80039cac:       94429e22        lhu     v0,-25054(v0)
80039cb0:       3c038038        lui     v1,0x8038
80039cb4:       8c639e28        lw      v1,-25048(v1)
80039cb8:       00824004        sllv    t0,v0,a0
80039cbc:       24020001        li      v0,1
80039cc0:       3c018004        lui     at,0x8004
80039cc4:       24219ef4        addiu   at,at,-24844	# 80039ef4
80039cc8:       00231821        addu    v1,at,v1
80039ccc:       00823804        sllv    a3,v0,a0
80039cd0:       11000031        beqz    t0,80039d98 <blast_icache32+0xf8>
80039cd4:       00002821        move    a1,zero
80039cd8:       3c028004        lui     v0,0x8004
80039cdc:       24429ef4        addiu   v0,v0,-24844	# 80039ef4
80039ce0:       0043302b        sltu    a2,v0,v1
80039ce4:       3c048004        lui     a0,0x8004
80039ce8:       24849ef4        addiu   a0,a0,-24844	# 80039ef4
80039cec:       50c00027        beqzl   a2,80039d8c <blast_icache32+0xec>
80039cf0:       00a72821        addu    a1,a1,a3
80039cf4:       00851025        or      v0,a0,a1
80039cf8:       bc400000        cache   0x0,0(v0)	# ee0 2e0 6e0 ae0
80039cfc:       bc400020        cache   0x0,32(v0)	# f00 300 700 b00
80039d00:       bc400040        cache   0x0,64(v0)
80039d04:       bc400060        cache   0x0,96(v0)
80039d08:       bc400080        cache   0x0,128(v0)
80039d0c:       bc4000a0        cache   0x0,160(v0)
80039d10:       bc4000c0        cache   0x0,192(v0)
80039d14:       bc4000e0        cache   0x0,224(v0)
80039d18:       bc400100        cache   0x0,256(v0)
80039d1c:       bc400120        cache   0x0,288(v0)	# 000 400 800 c00
80039d20:       bc400140        cache   0x0,320(v0)
80039d24:       bc400160        cache   0x0,352(v0)
80039d28:       bc400180        cache   0x0,384(v0)
80039d2c:       bc4001a0        cache   0x0,416(v0)
80039d30:       bc4001c0        cache   0x0,448(v0)
80039d34:       bc4001e0        cache   0x0,480(v0)
80039d38:       bc400200        cache   0x0,512(v0)
80039d3c:       bc400220        cache   0x0,544(v0)	# 100 500 900 d00
80039d40:       bc400240        cache   0x0,576(v0)	# 120 520 920 d20
80039d44:       bc400260        cache   0x0,608(v0)	# 140 540 940 d40 !!!
80039d48:       bc400280        cache   0x0,640(v0)
80039d4c:       bc4002a0        cache   0x0,672(v0)
80039d50:       bc4002c0        cache   0x0,704(v0)
80039d54:       bc4002e0        cache   0x0,736(v0)
80039d58:       bc400300        cache   0x0,768(v0)
80039d5c:       bc400320        cache   0x0,800(v0)	# 200 600 a00 e00
80039d60:       bc400340        cache   0x0,832(v0)
80039d64:       bc400360        cache   0x0,864(v0)
80039d68:       bc400380        cache   0x0,896(v0)
80039d6c:       bc4003a0        cache   0x0,928(v0)
80039d70:       bc4003c0        cache   0x0,960(v0)
80039d74:       bc4003e0        cache   0x0,992(v0)
80039d78:       24840400        addiu   a0,a0,1024
80039d7c:       0083102b        sltu    v0,a0,v1
80039d80:       1440ffdd        bnez    v0,80039cf8 <blast_icache32+0x58>
80039d84:       00851025        or      v0,a0,a1
80039d88:       00a72821        addu    a1,a1,a3
80039d8c:       00a8102b        sltu    v0,a1,t0
80039d90:       1440ffd4        bnez    v0,80039ce4 <blast_icache32+0x44>
80039d94:       00000000        nop
80039d98:       03e00008        jr      ra
80039d9c:       00000000        nop


The target address of the first CACHE instruction is 0x80039ef4.  It
invalidates index 0xee0 line.  The CACHE instruction on 0x80039d44
invalidates index 0xd40 line which may contains the
instruction... Ouch!!!

I suppose "two-phase invalidating" must be required to solve this
problem.

---
Atsushi Nemoto

Re: Strange, strange occurence

[G H]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

Is the suggested patch below going to be added to CVS or is considered too experimental to add ?
 
>No and yes - but here is a simpler solution.  Below patch solves the
>problem and adds just 32 bytes of code but removes the special case for
>TX49/H2 entirely.

>  Ralf

> arch/mips/mm/c-r4k.c        |   59 
>-------------------------------------------- include/asm-mips/r4kcache.h |   18 
>++++++++-----
> include/asm-mips/war.h      |   14 ----------
> 3 files changed, 13 insertions(+), 78 deletions(-)
>
>
>Index: arch/mips/mm/c-r4k.c
>===================================================================
>RCS file: /home/cvs/linux/arch/mips/mm/c-r4k.c,v
>retrieving revision 1.88
>diff -u -r1.88 c-r4k.c
>--- arch/mips/mm/c-r4k.c        16 Jul 2004 12:06:13 -0000      1.88
>+++ arch/mips/mm/c-r4k.c        16 Jul 2004 12:17:05 -0000
>@@ -96,16 +96,6 @@
>                r4k_blast_dcache = blast_dcache32;
> }


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[-- Attachment #2: Type: text/html, Size: 1521 bytes --]

Re: Strange, strange occurence

[Ralf Baechle]

On Fri, Jul 30, 2004 at 02:06:39PM -0700, G H wrote:

> Is the suggested patch below going to be added to CVS or is considered too experimental to add ?

No, that patch is simply wrong.

I'll implement a new one but being on a trip through the Silicon Valley
right now that's a little hard to do from here.

  Ralf