Re: (allocating non-cachable memory) (or More on the i82596)

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

Hrm... well, that's not the answer that I wanted, but it does explain a
lot :-/

It sounds like you know a lot more about what's going on here, Paul, but
since it sounds like we're in the same boat, I'm more than willing to help
in whatever way I can (especially since I have nothing better to do until
I get the ethernet controller working, and that just isn't happening
without this fixed :)

And thanks for the heads up that setting PAGE_NO_CACHE doesn't work- I'd
just tried disabling caches entirely and was about to go in and start
mucking with lower level things- starting with a BAT entry :)

Back to the debugger...
--Gus

On Thu, 28 Jun 2001, Paul White wrote:

>
> Justin,
>
> >From my recent experience, this does not work.  We have a device on our
> PCI bus which is DMAing to SDRAM, and so far I can't get anything to
> work without cache coherency support in the system controller.
>
> If you find anything out, please let me know.  I may end up adding a new
> memory queue for non-cacheable memory, and just use another BAT entry
> for some pre-defined amount of memory for a non-cacheable pool.  A new
> flag to kmalloc() could then be used to get non-cacheable memory.  Does
> anyone know if anything like this already exists, or if theres a much
> easier way around this??  I should hopefully get cache coherency
> working, but we have been unable to so far.  If I end up doing the work,
> I"ll post the patches here.
>
> Btw..  I attempted to disable cache, as well as simply set the RAM bat's
> to PAGE_NO_CACHE, however this does not work because the PowerPC will
> throw alignment exceptions whenever a cache instruction is called on
> either a non-cached memory region, or if caching is disabled.  Just
> wanted to give you a heads up on this.
>
> Paul W.
>
>
> At 04:52 AM 6/28/2001 -0400, Justin (Gus) Hurwitz wrote:
> >
> >Did yo ever get a reply to this? That sounds like just what I want to do
> >(well, it sounds just like what our vxworks code does to do what I want to
> >do, which I guess is good enough).
> >
> >TIA,
> >--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Justin,

I was able to get Data cache disabled today, after playing around with it
for a while.

There were several things I had to do.  Now, I don't recall what CPU you
are using, so I'll just tell you what I found with mine.

First off, I'm using the 7410, which has 32 byte cache lines.  Note that the
8xx and such have 16 byte cache lines.  In my manual, under "Enabling
and Disabling Data Cache", it states that the dcbz instruction will cause an
alignment exception, and that all others are nop'd.  So the first thing I did,
was search for dcbz throughout my code, and I found two files that had
several matches:

arch/ppc/kernel/misc.S

Line 380:
Inside clear_page.  Here, I commented out the dcbz line, and added more
of the stw lines to clear 32 bytes (it was set to only do 16, as on the 8xx
its
only 16 bytes):
//#ifdef CONFIG_8xx
        li      r4, 0
1:      stw     r4, 0(r3)
        stw     r4, 4(r3)
        stw     r4, 8(r3)
        stw     r4, 12(r3)
        stw     r4, 16(r3)
        stw     r4, 20(r3)
        stw     r4, 24(r3)
        stw     r4, 28(r3)
//#else
//1:    dcbz    0,r3
//#endif

Line 426:
Inside copy_page.  I left this one alone, as this one is called after Linux is
running normally and the exception handler catches it.  Note that the
exception handler detects it was just a dcbz w/ data cache disabled, and
returns normally.  Doesn't kernel panic.

arch/ppc/lib/string.S

Line 170:
Inside cacheable_memzero, this one is called from MMU_init() where
exceptions aren't enabled yet, so this one must be commented out,
and changed to store 0's in its place, as such:
        li      r7,4
//#if !defined(CONFIG_8xx)
//10:   dcbz    r7,r6
//#else
10:     stw     r4, 0(r6)
        stw     r4, 4(r6)
        stw     r4, 8(r6)
        stw     r4, 12(r6)
        stw     r4, 16(r6)
        stw     r4, 20(r6)
        stw     r4, 24(r6)
        stw     r4, 28(r6)
//#endif

Line 279:
Inside cacheable_memcpy.  This one is also called after Linux is running
normally, so I left this one alone as the exception handler catches it.

Line 483:
Inside copy_tofrom_user.  This one, I just needed to comment out:
//#if !defined(CONFIG_8xx)
//      dcbt    r3,r4
//      dcbz    r11,r6
//#endif


After doing this, I then edited arch/ppc/mm/init.c
Changed the RAM_PAGE define to:

#define RAM_PAGE (_PAGE_RW | _PAGE_NO_CACHE )

Then, in setbat() where it says "make IBAT same as DBAT":
               } else {
                        /* make IBAT same as DBAT */
                        bat[0] = bat[1];
                }

I changed this to:
               } else {
                        /* make IBAT same as DBAT */
                        bat[0] = bat[1];
                        bat[0].word[1] &= ~(_PAGE_NO_CACHE);
                }

This allowed Instruction cache to work still.

After this, I then wanted to be really sure data cache was disabled, so I
edited
arch/ppc/kernel/head.S:

Inside enable_caches, I changed the line:
	ori	r11,r11,HID0_DCE|HID0_ICE

to:
	ori	r11,r11,HID0_ICE

This just never enabled data cache, and everything worked fine (but slowly!).

Hope this information helps...  Eventually I'll work up a patch with
something like
CONFIG_NO_DCACHE, and maybe even CONFIG_WRITE_THRU on the cache
and e-mail them out.

Paul W.



At 06:43 AM 6/28/2001 -0400, Justin (Gus) Hurwitz wrote:
>
>Hrm... well, that's not the answer that I wanted, but it does explain a
>lot :-/
>
>It sounds like you know a lot more about what's going on here, Paul, but
>since it sounds like we're in the same boat, I'm more than willing to help
>in whatever way I can (especially since I have nothing better to do until
>I get the ethernet controller working, and that just isn't happening
>without this fixed :)
>
>And thanks for the heads up that setting PAGE_NO_CACHE doesn't work- I'd
>just tried disabling caches entirely and was about to go in and start
>mucking with lower level things- starting with a BAT entry :)
>
>Back to the debugger...
>--Gus
>
>On Thu, 28 Jun 2001, Paul White wrote:
>
>>
>> Justin,
>>
>> >From my recent experience, this does not work.  We have a device on our
>> PCI bus which is DMAing to SDRAM, and so far I can't get anything to
>> work without cache coherency support in the system controller.
>>
>> If you find anything out, please let me know.  I may end up adding a new
>> memory queue for non-cacheable memory, and just use another BAT entry
>> for some pre-defined amount of memory for a non-cacheable pool.  A new
>> flag to kmalloc() could then be used to get non-cacheable memory.  Does
>> anyone know if anything like this already exists, or if theres a much
>> easier way around this??  I should hopefully get cache coherency
>> working, but we have been unable to so far.  If I end up doing the work,
>> I"ll post the patches here.
>>
>> Btw..  I attempted to disable cache, as well as simply set the RAM bat's
>> to PAGE_NO_CACHE, however this does not work because the PowerPC will
>> throw alignment exceptions whenever a cache instruction is called on
>> either a non-cached memory region, or if caching is disabled.  Just
>> wanted to give you a heads up on this.
>>
>> Paul W.
>>
>>
>> At 04:52 AM 6/28/2001 -0400, Justin (Gus) Hurwitz wrote:
>> >
>> >Did yo ever get a reply to this? That sounds like just what I want to do
>> >(well, it sounds just like what our vxworks code does to do what I want to
>> >do, which I guess is good enough).
>> >
>> >TIA,
>> >--Gus
>
>
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

On Thu, 28 Jun 2001, Paul White wrote:

>
> Justin,
>
> I was able to get Data cache disabled today, after playing around with it
> for a while.
>
> There were several things I had to do.  Now, I don't recall what CPU you
> are using, so I'll just tell you what I found with mine.
>
> First off, I'm using the 7410, which has 32 byte cache lines.  Note that the
> 8xx and such have 16 byte cache lines.  In my manual, under "Enabling
> and Disabling Data Cache", it states that the dcbz instruction will cause an
> alignment exception, and that all others are nop'd.  So the first thing I did,
> was search for dcbz throughout my code, and I found two files that had
> several matches:

I went through and made the same set of changes that you did- I am on a
603e, so I would expect the L1 cache behaviour to be about the same.
Indeed, the kernel does boot properly, up until id decompresses the
ramdisk. I am getting a crc error decompressing the ramdisk, however-
presumably because of a problem with the caches. I'd like to set the MMU
to caching inhibited to tighten up the processor's load and store
ordering. I think that I only need to set this in the pte entries- the
errors are only occuring after the MMU has been brought up. Irony of
ironies, I know how to do this with the BATs, but not in the PTEs. It
looks like there is a field in the PTE struct (in asm/mmu.h), but I can't
find where that structure is initialised. It looks like, from my
meandering through the kernel's memory management code that this might wee
be a good step towards allowing chunks of non cachable memory to be
allocated.

Thoughts?

--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Justin,

I seen in a previous e-mail that you are using a 4xx proccessor.  I'm assuming
that this processor only has 16 bytes cache lines.  When you went through the
code I sent you, did you change it to only zero out 16 bytes instead of 32?

Also, Make sure in your CPU book that dcbz is the only instruction that
can't be done when data cache is disabled.

You also may want to check arch/ppc/kernel/align.c to make sure it still
simply "returns" from an alignment excepetion caused by a dcbz.  My
tree (base line 2.4.4) has it.  Otherwise, you may want to grep your
code (in the arch/ppc/*, and in include/asm/*) for any other occurences
of dcbz.  Any at bootup time must be replaced with assembly instructions
to zero out the memory locations,  but ones that will happen during runtime
should be OK if your alignment handler ignores them.

Again, be sure to check your CPU manual.  Maybe the 4xx CPU handles
the data cache instructions differently from the 7410.

Also..  Make absolutely sure you made the right changes in the
arch/ppc/mm/init.c file as well.

SInce its crapping out uncompressing the ramdisk, I am assuming
that the copy_pages or copy_tofrom_user stuff isn't working right
(Maybe you still have it zero'ing out 32 bytes instead of 16?)

Paul W.


At 05:16 AM 6/29/2001 -0400, Justin (Gus) Hurwitz wrote:
>On Thu, 28 Jun 2001, Paul White wrote:
>
>>
>> Justin,
>>
>> I was able to get Data cache disabled today, after playing around with it
>> for a while.
>>
>> There were several things I had to do.  Now, I don't recall what CPU you
>> are using, so I'll just tell you what I found with mine.
>>
>> First off, I'm using the 7410, which has 32 byte cache lines.  Note that
the
>> 8xx and such have 16 byte cache lines.  In my manual, under "Enabling
>> and Disabling Data Cache", it states that the dcbz instruction will
cause an
>> alignment exception, and that all others are nop'd.  So the first thing
I did,
>> was search for dcbz throughout my code, and I found two files that had
>> several matches:
>
>I went through and made the same set of changes that you did- I am on a
>603e, so I would expect the L1 cache behaviour to be about the same.
>Indeed, the kernel does boot properly, up until id decompresses the
>ramdisk. I am getting a crc error decompressing the ramdisk, however-
>presumably because of a problem with the caches. I'd like to set the MMU
>to caching inhibited to tighten up the processor's load and store
>ordering. I think that I only need to set this in the pte entries- the
>errors are only occuring after the MMU has been brought up. Irony of
>ironies, I know how to do this with the BATs, but not in the PTEs. It
>looks like there is a field in the PTE struct (in asm/mmu.h), but I can't
>find where that structure is initialised. It looks like, from my
>meandering through the kernel's memory management code that this might wee
>be a good step towards allowing chunks of non cachable memory to be
>allocated.
>
>Thoughts?
>
>--Gus
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

All true- except that I'm on a 603e :) It was someone else that was on the
4xx. I'm fairly sure that the 603e has 32 byte lines.

I think that I've hunted down all of the dcbz instructions, but I should
go and check align.c. I've not had a chance to look through the book to
see if there are other non-usable functions- I want to say that everything
else is a nop on the 603e, but I'll double check when I get the chance.

Fingers crossed,
--Gus

On Fri, 29 Jun 2001, Paul White wrote:

> Justin,
>
> I seen in a previous e-mail that you are using a 4xx proccessor.  I'm assuming
> that this processor only has 16 bytes cache lines.  When you went through the
> code I sent you, did you change it to only zero out 16 bytes instead of 32?
>
> Also, Make sure in your CPU book that dcbz is the only instruction that
> can't be done when data cache is disabled.
>
> You also may want to check arch/ppc/kernel/align.c to make sure it still
> simply "returns" from an alignment excepetion caused by a dcbz.  My
> tree (base line 2.4.4) has it.  Otherwise, you may want to grep your
> code (in the arch/ppc/*, and in include/asm/*) for any other occurences
> of dcbz.  Any at bootup time must be replaced with assembly instructions
> to zero out the memory locations,  but ones that will happen during runtime
> should be OK if your alignment handler ignores them.
>
> Again, be sure to check your CPU manual.  Maybe the 4xx CPU handles
> the data cache instructions differently from the 7410.
>
> Also..  Make absolutely sure you made the right changes in the
> arch/ppc/mm/init.c file as well.
>
> SInce its crapping out uncompressing the ramdisk, I am assuming
> that the copy_pages or copy_tofrom_user stuff isn't working right
> (Maybe you still have it zero'ing out 32 bytes instead of 16?)
>
> Paul W.
>
>
> At 05:16 AM 6/29/2001 -0400, Justin (Gus) Hurwitz wrote:
> >On Thu, 28 Jun 2001, Paul White wrote:
> >
> >>
> >> Justin,
> >>
> >> I was able to get Data cache disabled today, after playing around with it
> >> for a while.
> >>
> >> There were several things I had to do.  Now, I don't recall what CPU you
> >> are using, so I'll just tell you what I found with mine.
> >>
> >> First off, I'm using the 7410, which has 32 byte cache lines.  Note that
> the
> >> 8xx and such have 16 byte cache lines.  In my manual, under "Enabling
> >> and Disabling Data Cache", it states that the dcbz instruction will
> cause an
> >> alignment exception, and that all others are nop'd.  So the first thing
> I did,
> >> was search for dcbz throughout my code, and I found two files that had
> >> several matches:
> >
> >I went through and made the same set of changes that you did- I am on a
> >603e, so I would expect the L1 cache behaviour to be about the same.
> >Indeed, the kernel does boot properly, up until id decompresses the
> >ramdisk. I am getting a crc error decompressing the ramdisk, however-
> >presumably because of a problem with the caches. I'd like to set the MMU
> >to caching inhibited to tighten up the processor's load and store
> >ordering. I think that I only need to set this in the pte entries- the
> >errors are only occuring after the MMU has been brought up. Irony of
> >ironies, I know how to do this with the BATs, but not in the PTEs. It
> >looks like there is a field in the PTE struct (in asm/mmu.h), but I can't
> >find where that structure is initialised. It looks like, from my
> >meandering through the kernel's memory management code that this might wee
> >be a good step towards allowing chunks of non cachable memory to be
> >allocated.
> >
> >Thoughts?
> >
> >--Gus
> >


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

"Justin (Gus) Hurwitz" wrote:

> I think that I've hunted down all of the dcbz instructions, but I should
> go and check align.c.

There is no reason to do this.  The behavior of these instruction on the
603e aren't any different than other processors.  If you know you are
operating on I/O space, use the memset_io/memcpy_fromio()/memcpy_toio()
and not the usual memory copy/set functions.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Justin,

>From my recent experience, this does not work.   We have a device on our
PCI bus which is
DMAing to SDRAM, and so far I can't get anything to work without cache
coherency
support in the system controller.

If you find anything out, please let me know.  I may end up adding a new
memory queue
for non-cacheable memory, and just use another BAT entry for some
pre-defined amount
of memory for a non-cacheable pool.  A new flag to kmalloc() could then be
used to
get non-cacheable memory.  Does anyone know if anything like this already
exists, or
if theres a much easier way around this??  I should hopefully get cache
coherency
working, but we have been unable to so far.  If I end up doing the work,
I"ll post the patches
here.

Btw..  I attempted to disable cache, as well as simply set the RAM bat's to
PAGE_NO_CACHE,
however this does not work because the PowerPC will throw alignment
exceptions whenever a
cache instruction is called on either a non-cached memory region, or if
caching is disabled.
Just wanted to give you a heads up on this.

Paul W.


At 04:52 AM 6/28/2001 -0400, Justin (Gus) Hurwitz wrote:
>
>Did yo ever get a reply to this? That sounds like just what I want to do
>(well, it sounds just like what our vxworks code does to do what I want to
>do, which I guess is good enough).
>
>TIA,
>--Gus
>
>On Tue, 26 Jun 2001 jtm@smoothsmoothie.com wrote:
>
>>
>> I am looking for a solution to a similar problem. I am writing a
>> driver for an FCC on the 8260 in transparent mode, and need
>> to allocate buffer memory. The buffer memory will get filled
>> via DMA, and therefore must not be cached. Will the following work
>> correctly?
>>
>> 	vaddr = kmalloc(BUF_SIZE, GFP_DMA);
>> 	paddr = __pa(vaddr);
>>
>> Thanks.
>> --
>> Jay Monkman	    The truth knocks on the door and you say "Go away, I'm
>> monkman@jump.net    looking for the truth," and so it goes away. Puzzling.
>> 		     - from _Zen_and_the_Art_of_Motorcycle_Maintenance_
>
>
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

(allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

I've been looking at the code for the i82596 ethernet controller a bit
more and think that (thank god) it provides a better base for my board's
driver than I initially though.

But...

Our board is 603e based, with non functional hardware snooping. This is
not too much a problem for this chip- it supports writing to a block of
shared memory marked as non-cachable in lieu of using snooping. In fact,
our vxworks port for the board uses a whopping half meg memory region for
this purpose (seems a bit large to me, but I wasn't involved with that
port). What's more, the driver in the kernel seems to work using a shared
memory region as well. This is architecturally good. Unfortunately, I am
unable to figure out how to mark an area of memory as non-cachable.

The existing driver was written for 68k hardware, and therefore uses calls
that are non-existant in the ppc port:

#ifdef __mc68000__
        cache_push(virt_to_phys((void *)(dev->mem_start)), 4096);
        cache_clear(virt_to_phys((void *)(dev->mem_start)), 4096);
        kernel_set_cachemode((void *)(dev->mem_start), 4096,  IOMAP_NOCACHE_SER);
#endif

Therefore, my concise question is most likely: what are the equivalent
functions for a 603e based board? (I assume that the two cache_ functions
are comprable to flush_instruction_cache, but am probably wrong about
that, so I blatantly make the error hear so that someone will correct my
ignorance :)

Thanks,
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[jtm]

I am looking for a solution to a similar problem. I am writing a
driver for an FCC on the 8260 in transparent mode, and need
to allocate buffer memory. The buffer memory will get filled
via DMA, and therefore must not be cached. Will the following work
correctly?

	vaddr = kmalloc(BUF_SIZE, GFP_DMA);
	paddr = __pa(vaddr);

Thanks.
--
Jay Monkman	    The truth knocks on the door and you say "Go away, I'm
monkman@jump.net    looking for the truth," and so it goes away. Puzzling.
		     - from _Zen_and_the_Art_of_Motorcycle_Maintenance_

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

Did yo ever get a reply to this? That sounds like just what I want to do
(well, it sounds just like what our vxworks code does to do what I want to
do, which I guess is good enough).

TIA,
--Gus

On Tue, 26 Jun 2001 jtm@smoothsmoothie.com wrote:

>
> I am looking for a solution to a similar problem. I am writing a
> driver for an FCC on the 8260 in transparent mode, and need
> to allocate buffer memory. The buffer memory will get filled
> via DMA, and therefore must not be cached. Will the following work
> correctly?
>
> 	vaddr = kmalloc(BUF_SIZE, GFP_DMA);
> 	paddr = __pa(vaddr);
>
> Thanks.
> --
> Jay Monkman	    The truth knocks on the door and you say "Go away, I'm
> monkman@jump.net    looking for the truth," and so it goes away. Puzzling.
> 		     - from _Zen_and_the_Art_of_Motorcycle_Maintenance_


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

jtm@smoothsmoothie.com wrote:

> .... The buffer memory will get filled
> via DMA, and therefore must not be cached.

Huh????  The 8260 is cache coherent, you don't need to do that.

For processors that are not cache coherent (4xx and 8xx), there
are standard 'consistent_alloc()' functions available.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

On Fri, 29 Jun 2001, Dan Malek wrote:

>
> jtm@smoothsmoothie.com wrote:
>
> > .... The buffer memory will get filled
> > via DMA, and therefore must not be cached.
>
> Huh????  The 8260 is cache coherent, you don't need to do that.
>
> For processors that are not cache coherent (4xx and 8xx), there
> are standard 'consistent_alloc()' functions available.
>

Are those functions in the code for the 6xx proceesors? I have a 603e with
a broken memory controller- it does everything (or so I am assured) except
support atomic functions (ie, no memory coherency). If they are not in the
codo, are they at least portable to the 6xx?

Thanks,
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

On Fri, 29 Jun 2001, Dan Malek wrote:

>
> jtm@smoothsmoothie.com wrote:
>
> > .... The buffer memory will get filled
> > via DMA, and therefore must not be cached.
>
> Huh????  The 8260 is cache coherent, you don't need to do that.
>
> For processors that are not cache coherent (4xx and 8xx), there
> are standard 'consistent_alloc()' functions available.

Dan, Wwhere are the consistent_alloc functions? I just did a grep of the
entire source tree (it's about 2 weeks old 2_4_devel) and was unable to
find any consistent_* functions except for in the sparc and arm
architectures. There were pci_consistent_ functions under ppc, but I
assume that that requires a pci bus, which I do not have.

Thanks,
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

"Justin (Gus) Hurwitz" wrote:

> Dan, Wwhere are the consistent_alloc functions?

They are supposed to be in arch/ppc/mm/ioremap.c, but there has been
lots of work done there recently.  I haven't worked in the tree
recently,
so I'll take a look.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[jtm]

On Fri, Jun 29, 2001 at 03:39:57AM -0400, Dan Malek wrote:
>
> jtm@smoothsmoothie.com wrote:
>
> > .... The buffer memory will get filled
> > via DMA, and therefore must not be cached.
>
> Huh????  The 8260 is cache coherent, you don't need to do that.

OK. I've had more experience with the 860, and just assumed the
8260 had the same no cache requirement on DMA. Thanks.

--
Jay Monkman	    The truth knocks on the door and you say "Go away, I'm
monkman@jump.net    looking for the truth," and so it goes away. Puzzling.
		     - from _Zen_and_the_Art_of_Motorcycle_Maintenance_

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Jay,

Basically, on any CPU, DMA requires either no cache, or at least a cache
coherent
system.  Cache Coherency is a two-part thing, the CPU must support it, the
memory (system) controller.  If the memory controller doesn't generate the
proper
cache coherency snoop cycles, the CPU still has no idea to flush or invalidate
cache lines.

Paul W.



At 11:30 AM 6/29/2001 -0500, jtm@smoothsmoothie.com wrote:
>
>On Fri, Jun 29, 2001 at 03:39:57AM -0400, Dan Malek wrote:
>>
>> jtm@smoothsmoothie.com wrote:
>>
>> > .... The buffer memory will get filled
>> > via DMA, and therefore must not be cached.
>>
>> Huh????  The 8260 is cache coherent, you don't need to do that.
>
>OK. I've had more experience with the 860, and just assumed the
>8260 had the same no cache requirement on DMA. Thanks.
>
>--
>Jay Monkman	    The truth knocks on the door and you say "Go away, I'm
>monkman@jump.net    looking for the truth," and so it goes away. Puzzling.
>		     - from _Zen_and_the_Art_of_Motorcycle_Maintenance_
>
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Dan,

It doesn't just matter if the CPU is cache coherent or not.  If you have a
System Controller, which controls
the memory, and the DMA'ing device is off of that System Controller, then
the controller must support
Snooping, to generate CleanBlock,FlushBlock, or KillBlock transactions on
the CPU's bus.

The CPU supporting cache coherency just means it supports at least one of
these snoop transactions.
Someone still needs to generate these transactions.

Paul W.


At 03:39 AM 6/29/2001 -0400, Dan Malek wrote:
>
>jtm@smoothsmoothie.com wrote:
>
>> .... The buffer memory will get filled
>> via DMA, and therefore must not be cached.
>
>Huh????  The 8260 is cache coherent, you don't need to do that.
>
>For processors that are not cache coherent (4xx and 8xx), there
>are standard 'consistent_alloc()' functions available.
>
>
>	-- Dan
>
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Btw...Where are these "consistent_alloc()" functions?  I've not seen theses
anywhere.

I also can't imagine how they would possibly work on any CPU using BATs.
Since the BAT is setup as cacheable space, you can't just take a "chunk"
of it and make it non-cacheable.  Not possible.

Now, yes, for the 4xx and 8xx proccessors that don't use BATs, I would
assume that when you do one of these "non-cache" allocs, it would
simply allocate you a batch of 4k pages, and set them as Caching
Inhibited.

Is there something I'm missing here?

Paul W.

At 03:39 AM 6/29/2001 -0400, Dan Malek wrote:
>
>jtm@smoothsmoothie.com wrote:
>
>> .... The buffer memory will get filled
>> via DMA, and therefore must not be cached.
>
>Huh????  The 8260 is cache coherent, you don't need to do that.
>
>For processors that are not cache coherent (4xx and 8xx), there
>are standard 'consistent_alloc()' functions available.
>
>
>	-- Dan
>
>
-------------------------------------------------------------
Network Robots, Inc.         /--\               Paul H. White
                           / ____ \
4695 Chabot Dr. #200      / ( oo ) \       Software Developer
Pleasanton, CA 94588     /   ----   \             Engineering
(925) 924-8600           \   ----   /          (925) 580-4883
                          \  V--V  /
                           \      /
                             \--/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Matt Porter]

On Fri, Jun 29, 2001 at 10:48:45AM -0700, Paul White wrote:
>
> Btw...Where are these "consistent_alloc()" functions?  I've not seen theses
> anywhere.
>
> I also can't imagine how they would possibly work on any CPU using BATs.
> Since the BAT is setup as cacheable space, you can't just take a "chunk"
> of it and make it non-cacheable.  Not possible.
>
> Now, yes, for the 4xx and 8xx proccessors that don't use BATs, I would
> assume that when you do one of these "non-cache" allocs, it would
> simply allocate you a batch of 4k pages, and set them as Caching
> Inhibited.
>
> Is there something I'm missing here?

If you are on broken 6xx/7xx/82xx implementation then you can use the
"nobats" kernel argument to map memory with only PTEs.  Then the
consistent_alloc() stuff that Dan has brought over from arch/arm
will work fine.  The implementation is simply to get_free_pages(),
virt_to_phys(), ioremap(), and finally throw away the original
cacheable mappings.

Hrm, I see this stuff Dan's been talking about isn't in linuxppc_devel
(bad Dan).  If we all nag him perhaps it will get there faster. :)

In the meantime, look at the arch/arm's implementation to get an
idea of how it optimizes software coherency with the knowledge of
how you are using the memory region.

--
Matt Porter
MontaVista Software, Inc.
mporter@mvista.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

Matt Porter wrote:

> Hrm, I see this stuff Dan's been talking about isn't in linuxppc_devel
> (bad Dan).  If we all nag him perhaps it will get there faster. :)

Nag Tom :-).  I'll check with him to see what has happened to all of
these changes.  I'll get it in there.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Justin (Gus) Hurwitz]

On Fri, 29 Jun 2001, Matt Porter wrote:

> If you are on broken 6xx/7xx/82xx implementation then you can use the
> "nobats" kernel argument to map memory with only PTEs.  Then the
> consistent_alloc() stuff that Dan has brought over from arch/arm
> will work fine.  The implementation is simply to get_free_pages(),
> virt_to_phys(), ioremap(), and finally throw away the original
> cacheable mappings.
>
> Hrm, I see this stuff Dan's been talking about isn't in linuxppc_devel
> (bad Dan).  If we all nag him perhaps it will get there faster. :)
>
> In the meantime, look at the arch/arm's implementation to get an
> idea of how it optimizes software coherency with the knowledge of
> how you are using the memory region.

I've been poking around in the arch/arm code, impatiently contemplating
bringing the code over the my 2_4_devel tree on my own until it's been
done by someone more powerful. I looks like the big this done in the
consistent_alloc function is the calling of SetPageReserved for each page
allocated- this is a macro that sets the PG_reserved bit for the page,
which looks like it is a linux kernel thing, and not a hardware thig. Am I
missing something here, or will this actually prevent caching of those
pages? Or is the real magic done in the ioremap call, and the PG_reserved
bit tells the kernel not to touch the page (though it can still access the
data)? I presume the latter is the case, but I'd rather be sure before I
muck with this and add make my debugging job even harder.

Thanks
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

"Justin (Gus) Hurwitz" wrote:


> .....Am I
> missing something here,....

Yeah.

I'll have this upgraded shortly.

	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Dan Malek]

Paul White wrote:

> Is there something I'm missing here?

Yeah...you can run 'nobats' as a start option and get the memory
mapped as pages (or fix your system so cache coherency works :-).


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: (allocating non-cachable memory) (or More on the i82596)

[Paul White]

Dan,

hehe..  I went the harder way, and worked around the dcbz instructions
to allow me to disable data cache all together.

Btw, we also fixed our Cache Coherency problem on Friday.  It happened
to be a bug in our system controller chip.  They found it and gave us
a work-around.

Thanks,

Paul W.

On Sun, 1 Jul 2001, Dan Malek wrote:

> Paul White wrote:
>
> > Is there something I'm missing here?
>
> Yeah...you can run 'nobats' as a start option and get the memory
> mapped as pages (or fix your system so cache coherency works :-).
>
>
> 	-- Dan
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/