allocating non-cacheable regions

allocating non-cacheable regions

[Steve Rossi]

How do I allocate a region of memory that is flagged as being
non-cacheable?
Does it have to be done at allocation or is it possible to mark a region
non-cacheable
after its been allocated?

Thanks,
Steve

--
-------------------------------------------------------
Steven K. Rossi                     srossi@ccrl.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Dan Malek]

Steve Rossi wrote:
>
> How do I allocate a region of memory that is flagged as being
> non-cacheable?

I am assuming you intend to do this on the 8xx processor as it
doesn't make sense on others.  It may also not make sense on the 8xx
as it appears from testing that caching memory and push/invalidate
during I/O is better......

For an example, look at the commproc.c or enet.c driver in the
arch/ppc/8xx_io directory.  When pages are allocated, the PTEs are
tracked down and marked non-cache.

> Does it have to be done at allocation or is it possible to mark a region
> non-cacheable
> after its been allocated?

I haven't found a way to get the kernel to allocate non-cache regions
of real memory.  It would be nice if there was something along the
lines of DMA attribute, or perhaps I am just not looking in the right
place (and I am sure someone will point this out :-).

All I/O space is uncached and guarded.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

MPC860 enet driver ...

[dony]

Hello,
        I want to write a bootloader which will do the following steps:
       1.  download the kernel via TFTP protocol onto RAM address 0x400000.
       2. go to 0x400000 and run the kernel.
 The receive()/send() is based on /usr/src/linux/arch/ppc/8xx_io/enet.c, using
polling mode instead of interrupt mode.
However , It can only download 96 packets(512bytes/packet,so
96*512/1024=48k)successfully. If the kernel file is larger than 48K, then it
will stop here when it downloads 48k. That is to say, it can only download
files which is smaller than 48K.
Maybe I should modify some parameters for MPC860 in enet.c?
How can I solve this?
 Thank you very much.
dony


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Steve Rossi]

I'm back on this ... with another question. Following the examples in
commproc.c and enet.c - does setting the _PAGE_NO_CACHE flag
for a page table entry also invalidate any cached data for that page?
I am observing a write-though behavior when I write to addresses in
the _PAGE_NO_CACHE page but when I read from addresses marked
as _PAGE_NO_CACHE it appears to be retrieving data from the cache
not from memory. Is this expected behavior? How can I make it so that
reads as well as writes to a particular page bypass the cache?

Thanks,
Steve

Dan Malek wrote:

> Steve Rossi wrote:
> >
> > How do I allocate a region of memory that is flagged as being
> > non-cacheable?
>
> For an example, look at the commproc.c or enet.c driver in the
> arch/ppc/8xx_io directory.  When pages are allocated, the PTEs are
> tracked down and marked non-cache.
>

--
-------------------------------------------------------
Steven K. Rossi                     srossi@ccrl.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Dan Malek]

Steve Rossi wrote:

> ....does setting the _PAGE_NO_CACHE flag
> for a page table entry also invalidate any cached data for that page?

No.  It only ensures uncached behavior when the TLB is loaded.

> I am observing a write-though behavior when I write to addresses in
> the _PAGE_NO_CACHE page but when I read from addresses marked
> as _PAGE_NO_CACHE it appears to be retrieving data from the cache
> not from memory. Is this expected behavior?

That's a rhetorical, question, right :-)?  If not....of course that
isn't correct.

> .... How can I make it so that
> reads as well as writes to a particular page bypass the cache?

You need to invalidate the data cache for this address and the TLB
entry for this address when you set the flag.  Those drivers in your
example get away with it because the caches/TLB have been invalidated
and not yet touched at this point.  You also need to ensure you are not
multiple mapping the same physical address....


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Tom Roberts]

Dan Malek wrote:
> You need to invalidate the data cache for this address and the TLB
> entry for this address when you set the flag.

That is probably his problem.


> You also need to ensure you are not
> multiple mapping the same physical address....

No, that is not necessary, as long as your program is aware of the
multiple mapping. In certain cases (com buffers shared by two non-
snooping CPUs) this is quite useful -- use the uncached addrs for the
queue pointers and the cached addrs for the data (being careful to
manually flush the cache before reading and after writing).

You do need to ensure you are not mapping the same _virtual_ address
(i.e. the effective address, in the language of Motorola's manuals),
but the kernel routines will probably take care of that for you....


Tom Roberts	tjroberts@lucent.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Steve Rossi]

Dan Malek wrote:

>
> > .... How can I make it so that
> > reads as well as writes to a particular page bypass the cache?
>
> You need to invalidate the data cache for this address and the TLB
> entry for this address when you set the flag.  Those drivers in your
> example get away with it because the caches/TLB have been invalidated
> and not yet touched at this point.  You also need to ensure you are not
> multiple mapping the same physical address....
>

I'm invalidating the TLB entry via a call to flush_tlb_page as in the examples
that you pointed me to - which simply maps to tlbia on 8xx. But I haven't yet
figured out how to invalidate the data cache for the entire page. I'm trying
invalidate_dcache_range() - but I'm having really strange problems with it. If
I compile the driver into the kernel with a call to invalidate_dcache_range(),
the kernel won't boot. (It'll get as far as "Uncompressing Linux ..." and
it'll hang). The same happens if I export invalidate_dcache_range in
ppc_ksyms.c in an attempt to load the driver as a module. I haven't looked
into what is going on here - but I do know that it is being caused by the
invalidate_dcache_range being present. Am I totally off base here? Should
I even be using invalidate_dcache_range or is there another method for
invalidating the data cache? A pointer to an example would be sufficient if
anyone has one.

Thanks,
Steve

--
-------------------------------------------------------
Steven K. Rossi                     srossi@ccrl.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Steve Rossi]

Well I found the solution - I needed to added an  extern
invalidate_dcache_range() declaration to include/asm/cache.h then the kernel will
compile with invalidate_dcache_range exported in ppc_ksyms.c (contrary to what
I reported, my problem with compiling the device driver as a module was that the
kernel wouldn't compile with invalidate_dcache_range exported - not that it
wouldn't boot. I just wasn't remembering correctly when I wrote that.) Now I can
compile the driver as a module with invalidate_dcache_range and it works!

My problem with the kernel not booting was actually related to compiling the
driver into the kernel. I have to investigate this further because it used to
work.

Steve


Steve Rossi wrote:

> Dan Malek wrote:
>
> >
> > > .... How can I make it so that
> > > reads as well as writes to a particular page bypass the cache?
> >
> > You need to invalidate the data cache for this address and the TLB
> > entry for this address when you set the flag.  Those drivers in your
> > example get away with it because the caches/TLB have been invalidated
> > and not yet touched at this point.  You also need to ensure you are not
> > multiple mapping the same physical address....
> >
>
> I'm invalidating the TLB entry via a call to flush_tlb_page as in the examples
> that you pointed me to - which simply maps to tlbia on 8xx. But I haven't yet
> figured out how to invalidate the data cache for the entire page. I'm trying
> invalidate_dcache_range() - but I'm having really strange problems with it. If
> I compile the driver into the kernel with a call to invalidate_dcache_range(),
> the kernel won't boot. (It'll get as far as "Uncompressing Linux ..." and
> it'll hang). The same happens if I export invalidate_dcache_range in
> ppc_ksyms.c in an attempt to load the driver as a module. I haven't looked
> into what is going on here - but I do know that it is being caused by the
> invalidate_dcache_range being present. Am I totally off base here? Should
> I even be using invalidate_dcache_range or is there another method for
> invalidating the data cache? A pointer to an example would be sufficient if
> anyone has one.
>
> Thanks,
> Steve
>
> --
> -------------------------------------------------------
> Steven K. Rossi                     srossi@ccrl.mot.com
> Staff Engineer
> Multimedia Communications Research Laboratory
> Motorola Labs
> -------------------------------------------------------
>

--
-------------------------------------------------------
Steven K. Rossi                     srossi@ccrl.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: allocating non-cacheable regions

[Steve Rossi]

well it sort of works. I think I'm running into some strange compiler problems.
There are four scenarios that I'm working with. They are:

compiling driver as a module using x86 cross-compiler - doesn't work
compiling driver as a module using native compiler - works

compiling driver into the kernel using x86 cross-compiler - doesn't work
compiling driver into the kernel using native compiler - sorta works

when I say doesn't work, I mean that invalidate_dcache_range is having no effect -
the powerpc is always getting bad data when it reads memory that was written by the
device - in this case a SCSI controller
when I say sorta works, I mean that what I've observed is the first chunk of data
returned by the scsi controller for the first query command seems to be correct or
partially correct, but subsequent data is not
when I say works - that means it completely works, I can mount the filesystem on the
scsi disk and read and write files with no problem

when I compile on x86 it never works (monta vista cdk, gcc version 2.95.2 19991024
(release)) whether I compile it as a module or into the kernel. compiler
bug? something to do with the fact that invalidate_dcache_range is an assembly
routine? any thoughts on this?

when I compile on ppc (g4 running yellow dog linux cs 1.2, gcc version 2.95.2
19991024 (release/franzo)) I can only get it to work as a module. this is ok, but
not ideal. I would like to be able to compile the driver into the kernel. Does
anyone have any ideas as to why it would be different with respect to regions marked
as no-cache when compiled as a module versus when compiled into the kernel?

will somone verify that my assumptions are correct - the driver that I'm working
with allocates memory for its own data structures - both those used only by the
driver and those shared between the driver and the device - using get_free_pages().
I'm taking the simple approach - attempting to make all of that memory that the
driver allocates non-cacheable - that is I want it to always access external memory
on both write and reads to these addresses and never use the cache.  So for each
call to get_free_pages (which is always called with order = 0 so only returns a
pointer to one page) I track down the pte, mark it as _PAGE_NO_CACHE then flush the
tlb using flush_tlb_page, then invalidate the cache using invalidate_dcache_range
passing the address returned by get_free_pages as the start and that address +
PAGE_SIZE as the end. My assumption is that this is sufficient to ensure that all
further accesses to the addresses within this page will bypass the cache. This
assumption seems to hold for the case of writing to these addresses - as the
_PAGE_NO_CACHE flag appears to force it to writethrough. But I'm a little leary as
to whether I need to invalidate the cache each time before I read these addresses.
I would like to not have to do that. If what I'm doing isn't sufficient to ensure
that reads will never come from the cache, is there something else I can do to
ensure that?

regarding multiple mapping the same physical address - I don't believe that's
happening - or at least the driver isn't doing anything that would cause that to
happen as far as I can tell.

Thanks,
Steve



Steve Rossi wrote:

> Well I found the solution - I needed to added an  extern
> invalidate_dcache_range() declaration to include/asm/cache.h then the kernel will
> compile with invalidate_dcache_range exported in ppc_ksyms.c (contrary to what
> I reported, my problem with compiling the device driver as a module was that the
> kernel wouldn't compile with invalidate_dcache_range exported - not that it
> wouldn't boot. I just wasn't remembering correctly when I wrote that.) Now I can
> compile the driver as a module with invalidate_dcache_range and it works!

--
-------------------------------------------------------
Steven K. Rossi                     srossi@ccrl.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/