[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Rafal Jaworowski]

Hi,

Currently do_bootelf() disables data cache just before passing control to the
entry point:

        /*
         * QNX images require the data cache is disabled.
         * Data cache is already flushed, so just turn it off.
         */
        if (dcache_status ())
                dcache_disable ();

With this piece of code present I'm seeing strange effects (corrupt data
access, hangs) in standalone apps running from the ELF envelope. A pure binary
made of this same ELF works without problems. Contrary to the comment above,
there's no quarantee in do_bootelf() code flow that all data cache has been
flushed and it seems there needs to be flush_data_cache() executed before the
disable.

1. Does it really hold true that QNX requires d-cache disabled upon passing
control to it?

2. If so, this is a custom QNX thing that belongs to do_bootm_qnxelf() and
should be handled there and not at the common ELF handling level.

Any objections to pushing this down to do_bootm_qnxelf()? I can provide a
patch, but cannot test it with any QNX system. Is there anyone with such setup
that would give this a try?

Rafal

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Stefan Roese]

Hi Rafal,

On Wednesday 02 January 2008, Rafal Jaworowski wrote:
> Currently do_bootelf() disables data cache just before passing control to
> the entry point:
>
>         /*
>          * QNX images require the data cache is disabled.
>          * Data cache is already flushed, so just turn it off.
>          */
>         if (dcache_status ())
>                 dcache_disable ();
>
> With this piece of code present I'm seeing strange effects (corrupt data
> access, hangs) in standalone apps running from the ELF envelope.

What platform are you testing on? PPC? If yes, which one?

> A pure 
> binary made of this same ELF works without problems. Contrary to the
> comment above, there's no quarantee in do_bootelf() code flow that all data
> cache has been flushed and it seems there needs to be flush_data_cache()
> executed before the disable.

dcache_disable() should flush the dcache itself before disabling it. At least 
this is the implementation I have seen on 405 and IIRC other PPC platforms.

> 1. Does it really hold true that QNX requires d-cache disabled upon passing
> control to it?
>
> 2. If so, this is a custom QNX thing that belongs to do_bootm_qnxelf() and
> should be handled there and not at the common ELF handling level.

Hmmm. I think that calling applications and especially OS'es is more safe with 
caches disabled then with caches enabled.

> Any objections to pushing this down to do_bootm_qnxelf()? I can provide a
> patch, but cannot test it with any QNX system. Is there anyone with such
> setup that would give this a try?

It's possible that other OS'es rely on caches being disabled too. I'm not sure 
here, but VxWorks could be one of those. Does anybody know for sure?

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Rafal Jaworowski]

Hi Stefan,

you wrote:
>> Currently do_bootelf() disables data cache just before passing control to
>> the entry point:
>>
>>         /*
>>          * QNX images require the data cache is disabled.
>>          * Data cache is already flushed, so just turn it off.
>>          */
>>         if (dcache_status ())
>>                 dcache_disable ();
>>
>> With this piece of code present I'm seeing strange effects (corrupt data
>> access, hangs) in standalone apps running from the ELF envelope.
> 
> What platform are you testing on? PPC? If yes, which one?
> 

Yes, 85xx.

>> A pure 
>> binary made of this same ELF works without problems. Contrary to the
>> comment above, there's no quarantee in do_bootelf() code flow that all data
>> cache has been flushed and it seems there needs to be flush_data_cache()
>> executed before the disable.
> 
> dcache_disable() should flush the dcache itself before disabling it. At least 
> this is the implementation I have seen on 405 and IIRC other PPC platforms.
> 

Indeed, in certain cases it does actually (e.g. 86xx), but it's not uniform
unfortunatelly. In my case (85xx) dcache_disable() does not flush the cache
before disabling.

>> 1. Does it really hold true that QNX requires d-cache disabled upon passing
>> control to it?
>>
>> 2. If so, this is a custom QNX thing that belongs to do_bootm_qnxelf() and
>> should be handled there and not at the common ELF handling level.
> 
> Hmmm. I think that calling applications and especially OS'es is more safe with 
> caches disabled then with caches enabled.
> 

Yeah, after a second thought I tend to agree. Maybe the way to go is doing
data cache flush from within dcache_disable() properly i.e. bring it in for
arch variations that don't do it currently like 85xx... Actually to confirm my
observations I tested a working patch that flushes d-cache at cache_disable()
just like 86xx and it works for me, this is: my problems disappear. Do you
think this is a better option?

kind regards,
Rafal

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Stefan Roese]

On Thursday 03 January 2008, Rafal Jaworowski wrote:
> >> 1. Does it really hold true that QNX requires d-cache disabled upon
> >> passing control to it?
> >>
> >> 2. If so, this is a custom QNX thing that belongs to do_bootm_qnxelf()
> >> and should be handled there and not at the common ELF handling level.
> >
> > Hmmm. I think that calling applications and especially OS'es is more safe
> > with caches disabled then with caches enabled.
>
> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
> data cache flush from within dcache_disable() properly i.e. bring it in for
> arch variations that don't do it currently like 85xx... Actually to confirm
> my observations I tested a working patch that flushes d-cache at
> cache_disable() just like 86xx and it works for me, this is: my problems
> disappear. Do you think this is a better option?

Yes, I think this is the way to go. Please provide a patch and send it to the 
85xx maintainer. Best would be if you could check the other ARCH's (at least 
PPC) for this dcache_disable() behaviour too.

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Wolfgang Denk]

Dear Rafal,

in message <477CB941.2020001@semihalf.com> you wrote:
> 
> Indeed, in certain cases it does actually (e.g. 86xx), but it's not uniform
> unfortunatelly. In my case (85xx) dcache_disable() does not flush the cache
> before disabling.

I consider this a bug in the 85xx code, then. The bug should be fixed,
and not worked around it.

> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
> data cache flush from within dcache_disable() properly i.e. bring it in for
> arch variations that don't do it currently like 85xx... Actually to confirm my
> observations I tested a working patch that flushes d-cache at cache_disable()
> just like 86xx and it works for me, this is: my problems disappear. Do you
> think this is a better option?

Yes, indeed.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The human race has one really effective weapon, and that is laughter.
                                                         - Mark Twain

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Rafal Jaworowski]

Stefan Roese wrote:
>> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
>> data cache flush from within dcache_disable() properly i.e. bring it in for
>> arch variations that don't do it currently like 85xx... Actually to confirm
>> my observations I tested a working patch that flushes d-cache at
>> cache_disable() just like 86xx and it works for me, this is: my problems
>> disappear. Do you think this is a better option?
> 
> Yes, I think this is the way to go. Please provide a patch and send it to the 
> 85xx maintainer. Best would be if you could check the other ARCH's (at least 
> PPC) for this dcache_disable() behaviour too.
> 

I checked, and only 7xx/74xx, 86xx and 4xx do it properly. For all other PPC
variants cache_disable() does not flush d-cache before disabling it... So the
problem is quite widespread. I'll try to come up with something generic, but
am not sure if it'll fit every other variant.

kind regards,
Rafal

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Josh Boyer]

On Sat, 05 Jan 2008 23:18:10 +0100
Rafal Jaworowski <raj@semihalf.com> wrote:

> Stefan Roese wrote:
> >> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
> >> data cache flush from within dcache_disable() properly i.e. bring it in for
> >> arch variations that don't do it currently like 85xx... Actually to confirm
> >> my observations I tested a working patch that flushes d-cache at
> >> cache_disable() just like 86xx and it works for me, this is: my problems
> >> disappear. Do you think this is a better option?
> > 
> > Yes, I think this is the way to go. Please provide a patch and send it to the 
> > 85xx maintainer. Best would be if you could check the other ARCH's (at least 
> > PPC) for this dcache_disable() behaviour too.
> > 
> 
> I checked, and only 7xx/74xx, 86xx and 4xx do it properly. For all other PPC
> variants cache_disable() does not flush d-cache before disabling it... So the
> problem is quite widespread. I'll try to come up with something generic, but
> am not sure if it'll fit every other variant.

Out of curiosity, how do you disable dcache on 44x?  The only way I
know how to turn the cache off on 440 is to set the Guarded bit in the
TLB entries.  There is no cache disable bit.

(And from what I see, dcache_disable, icache_disable, and icache_enable
all just simply return without doing anything on 440).

josh

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Rafal Jaworowski]

Josh Boyer wrote:
>>>> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
>>>> data cache flush from within dcache_disable() properly i.e. bring it in for
>>>> arch variations that don't do it currently like 85xx... Actually to confirm
>>>> my observations I tested a working patch that flushes d-cache at
>>>> cache_disable() just like 86xx and it works for me, this is: my problems
>>>> disappear. Do you think this is a better option?
>>> Yes, I think this is the way to go. Please provide a patch and send it to the 
>>> 85xx maintainer. Best would be if you could check the other ARCH's (at least 
>>> PPC) for this dcache_disable() behaviour too.
>>>
>> I checked, and only 7xx/74xx, 86xx and 4xx do it properly. For all other PPC
>> variants cache_disable() does not flush d-cache before disabling it... So the
>> problem is quite widespread. I'll try to come up with something generic, but
>> am not sure if it'll fit every other variant.
> 
> Out of curiosity, how do you disable dcache on 44x?  The only way I
> know how to turn the cache off on 440 is to set the Guarded bit in the
> TLB entries.  There is no cache disable bit.
> 

I believe this is how "disabling" is achieved currently in U-Boot: the DRAM
region is TLB-mapped with the G bit set. But I was rather referring to
introducing a d-cache flush as a first step of dcache_disable() in cases it is
not there, and not re-working the disable sequence itself. 440 is special
about this and we probably have to live with d-cache being either always
present or "disabled" via G bit.

> (And from what I see, dcache_disable, icache_disable, and icache_enable
> all just simply return without doing anything on 440).
> 

I mistyped: it is 40x that does flushing before disabling d-cache; on 440, as
you say, there are stubs for these ops.

kind regards,
Rafal

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Matthias Fuchs]

Hi,

there is a simple approach to disable dcaching on 440 used for the 440 usb code (cpu/ppc4xx/usb.c).
The code is used to disable dcaching before starting the USB subsystem. It updates the tlb entry.
But it is not complete working. Disbling works, but reenabling (via 'usb stop') does not work.
But with CONFIG_4xx_DCACHE enabled this at least allows you to use USB but with disabled 
dcache from that moment on.

Perhaps it is possible to move the change_tlb() call into the dcache_disable when it is 
working sometime. 

Matthias

On Sunday 06 January 2008 15:11, Rafal Jaworowski wrote:
> Josh Boyer wrote:
> >>>> Yeah, after a second thought I tend to agree. Maybe the way to go is doing
> >>>> data cache flush from within dcache_disable() properly i.e. bring it in for
> >>>> arch variations that don't do it currently like 85xx... Actually to confirm
> >>>> my observations I tested a working patch that flushes d-cache at
> >>>> cache_disable() just like 86xx and it works for me, this is: my problems
> >>>> disappear. Do you think this is a better option?
> >>> Yes, I think this is the way to go. Please provide a patch and send it to the 
> >>> 85xx maintainer. Best would be if you could check the other ARCH's (at least 
> >>> PPC) for this dcache_disable() behaviour too.
> >>>
> >> I checked, and only 7xx/74xx, 86xx and 4xx do it properly. For all other PPC
> >> variants cache_disable() does not flush d-cache before disabling it... So the
> >> problem is quite widespread. I'll try to come up with something generic, but
> >> am not sure if it'll fit every other variant.
> > 
> > Out of curiosity, how do you disable dcache on 44x?  The only way I
> > know how to turn the cache off on 440 is to set the Guarded bit in the
> > TLB entries.  There is no cache disable bit.
> > 
> 
> I believe this is how "disabling" is achieved currently in U-Boot: the DRAM
> region is TLB-mapped with the G bit set. But I was rather referring to
> introducing a d-cache flush as a first step of dcache_disable() in cases it is
> not there, and not re-working the disable sequence itself. 440 is special
> about this and we probably have to live with d-cache being either always
> present or "disabled" via G bit.
> 
> > (And from what I see, dcache_disable, icache_disable, and icache_enable
> > all just simply return without doing anything on 440).
> > 
> 
> I mistyped: it is 40x that does flushing before disabling d-cache; on 440, as
> you say, there are stubs for these ops.
> 
> kind regards,
> Rafal

[U-Boot-Users] disabling d-cache in 'bootelf' for QNX

[Stefan Roese]

Hi Matthias,

On Monday 07 January 2008, Matthias Fuchs wrote:
> there is a simple approach to disable dcaching on 440 used for the 440 usb
> code (cpu/ppc4xx/usb.c). The code is used to disable dcaching before
> starting the USB subsystem. It updates the tlb entry. But it is not
> complete working. Disbling works, but reenabling (via 'usb stop') does not
> work.

Right. It would really be great if somebody could take a look at this issue. 
Since this is the last showstopper to enable cache (I & D) on 440 platforms.

> But with CONFIG_4xx_DCACHE enabled this at least allows you to use 
> USB but with disabled dcache from that moment on.
>
> Perhaps it is possible to move the change_tlb() call into the
> dcache_disable when it is working sometime.

Yes, this is the plan. When it is finally working completely, the "normal" 
cache functions should be supported for 440 too.

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================