Non-cacheable memory

Non-cacheable memory

[Justin (Gus) Hurwitz]

Several weeks ago I was attempting to allocate a region of non-cacheable
memory (required for my 603e based board's ethernet chip) and was told the
work was underway to get support for that into the kernel. I've not
checked up on that progress for a while now, so I thought I'd give it
another try:

Does the current 2_4_devel tree have support for allocating regions of
non-cachable memory? And, if so, what is the proper (or preferred) way of
doing this?

It sure would be nice if I could turn the board's L1 cache back on :)

Thanks,
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Tom Roberts]

"Justin (Gus) Hurwitz" wrote:
> Does the current 2_4_devel tree have support for allocating regions of
> non-cachable memory? And, if so, what is the proper (or preferred) way of
> doing this?

What I did in 2.2.15 was to edit the main startup routine to call my driver
directly so it could "eat up" a region of memory before Linux started its
memory management. I then used a BAT to reference it uncached. Yes, this
is ugly, and requires care in making sure the BAT effective address is
otherwise unused by the kernel, but it works. This is on a MPC7400, and
other PowerPC CPUs might not have BATs available....

Note also that my driver uses unchached accesses only for buffer pointers.
For the buffer data I used cached references, with cache-flushes
appropriately sprinkled throughout the code -- the MAJOR bandwidth gain
was important.

Note that if your hardware does bus snooping, you might not need uncached
memory references or explicit flushes to read DMA data. But embedded and
non-SMP systems are unlikely to do it.


Tom Roberts	tjroberts@lucent.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Daris A Nevil]

Gus, Tom,

Here is the procedure I used to allocate pages and mark them as non-cached in
the 2.2.5 kernel (SNMC's QSLinux):

    mem_addr = __get_free_page(GFP_KERNEL);
    pte = va_to_pe(&init_task, mem_addr);
    pte_val(*pte) |= _PAGE_NO_CACHE;
    flush_tlb_page(current->mm_mmap, mem_addr);

I used these pages for SCC buffer descriptors, so they were never returned to
the system. I'm not sure if this is the preferred way, but it certainly works.
I have not tried this in the 2.4 kernel yet.  The functions __get_free_page()
and flush_tlb_page() do exist in 2.4.4, but I could not locate va_to_pe()  (I
think that was a macro in 2.2.5).

You may also want to look at the file linux/asm/pgalloc.h, esp. the functions
get_pgd_slow() and get_pgd_fast().

Regards,
Daris Nevil


Tom Roberts wrote:

> "Justin (Gus) Hurwitz" wrote:
> > Does the current 2_4_devel tree have support for allocating regions of
> > non-cachable memory? And, if so, what is the proper (or preferred) way of
> > doing this?
>
> What I did in 2.2.15 was to edit the main startup routine to call my driver
> directly so it could "eat up" a region of memory before Linux started its
> memory management. I then used a BAT to reference it uncached. Yes, this
> is ugly, and requires care in making sure the BAT effective address is
> otherwise unused by the kernel, but it works. This is on a MPC7400, and
> other PowerPC CPUs might not have BATs available....

--
Daris A Nevil
Simple Network Magic Corporation
401 Kentucky Lane
McKinney, TX 75069
214-793-7757
dnevil@snmc.com
www.snmc.com
quickstack.com
qslinux.org


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Justin (Gus) Hurwitz]

On Tue, 7 Aug 2001, Daris A Nevil wrote:

> Gus, Tom,
>
> Here is the procedure I used to allocate pages and mark them as non-cached in
> the 2.2.5 kernel (SNMC's QSLinux):
>
>     mem_addr = __get_free_page(GFP_KERNEL);
>     pte = va_to_pe(&init_task, mem_addr);
>     pte_val(*pte) |= _PAGE_NO_CACHE;
>     flush_tlb_page(current->mm_mmap, mem_addr);
>
> I used these pages for SCC buffer descriptors, so they were never returned to
> the system. I'm not sure if this is the preferred way, but it certainly works.
> I have not tried this in the 2.4 kernel yet.  The functions __get_free_page()
> and flush_tlb_page() do exist in 2.4.4, but I could not locate va_to_pe()  (I
> think that was a macro in 2.2.5).
>
> You may also want to look at the file linux/asm/pgalloc.h, esp. the functions
> get_pgd_slow() and get_pgd_fast().

Indeed, that does look quite like what I would expect to do. But, trust my
luck, it didn't work. I'm on a 2.4 kernel (a few weeks old from the devel
tree (2.4.7-pre6?). Here's how my code looks:

        dev->mem_start = (int)__get_free_pages(GFP_KERNEL, 0);
#ifndef CONFIG_NO_DCACHE
        {
        /* This is where code goes to set the above allocated memory as
        *  non-cacheable. These lines DO NOT WORK right now. */

        pte_t *pte = va_to_pte(dev->mem_start);
        pte_val(*pte) |= _PAGE_NO_CACHE;
        flush_tlb_page(init_mm.mmap, dev->mem_start);
        }
#endif


Notes: CONFIG_NO_DCACHE is an option I added which disables the L1 cache
(no cache means no need for uncached memory :) Also, va_to_pte is in
arch/ppc/mm/fault.c, and is ifdef'd with a CONFIG_8xx. I altered that
ifdef to also include the function for my board.

I might also need to set the addresses 0x10710000 and 0x10718000 to be
non-cacheable (for MMIO with the chip). Would those addresses be in the
page table? And if not, thoughts on how to un-cache those areas (which are
each only a few bytes)?

Thanks,
--Gus


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Peter Desnoyers]

Justin (Gus) Hurwitz wrote:

> On Tue, 7 Aug 2001, Daris A Nevil wrote:
>>
>>Here is the procedure I used to allocate pages and mark them as non-cached in
>>the 2.2.5 kernel (SNMC's QSLinux):
>>
>>    mem_addr = __get_free_page(GFP_KERNEL);
>>    pte = va_to_pe(&init_task, mem_addr);
>>    pte_val(*pte) |= _PAGE_NO_CACHE;
>>    flush_tlb_page(current->mm_mmap, mem_addr);
>>
>>I used these pages for SCC buffer descriptors, so they were never returned to
>>the system. I'm not sure if this is the preferred way, but it certainly works.
>>I have not tried this in the 2.4 kernel yet.  The functions __get_free_page()
>>and flush_tlb_page() do exist in 2.4.4, but I could not locate va_to_pe()  (I
>>think that was a macro in 2.2.5).


In 2.4 you can use consistent_alloc() - this allocates memory and makes
it non-cachable.  (well, it makes it consistent w.r.t. PCI DMA, but for
860s it's the same thing.)  It returns a virtual address plus a DMA
address in an arg ptr - throw away the DMA address, as that's
PCI-specific, and run the virtual address through virt_to_phys().

--
.....................................................................
  Peter Desnoyers            (781) 457-1165   pdesnoyers@chinook.com
  Chinook Communications     (617) 661-1979   pjd@fred.cambridge.ma.us
  100 Hayden Ave, Lexington MA 02421


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Dan Malek]

Peter Desnoyers wrote:

> ..... It returns a virtual address plus a DMA
> address in an arg ptr - throw away the DMA address, as that's
> PCI-specific, and run the virtual address through virt_to_phys().

Ummm...damn, that's a tricky one.  On 8xx, the intention is to use
the DMA address for stuffing into the CPM.  The virt_to_* functions
are on their way out (on all platforms), and everyone should use the
DMA address returned by the consistent_alloc function.  I guess if you
have a 8xx/PCI that doesn't map PCI masters to system memory at zero,
you are kind of hosed at the moment........


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Peter Desnoyers]

Dan Malek wrote:
>
> Peter Desnoyers wrote:
>
> > ..... It returns a virtual address plus a DMA
> > address in an arg ptr - throw away the DMA address, as that's
> > PCI-specific, and run the virtual address through virt_to_phys().
>
> Ummm...damn, that's a tricky one.  On 8xx, the intention is to use
> the DMA address for stuffing into the CPM.

Yow.  So the MBX860 setup, for instance, with PCI_DRAM_OFFSET=2gig, just
won't work.

This seems like another case of either too little abstraction, or too
much.  An embedded PPC system with PCI basically has two buses, with
different DMA addressing, and there's no way to tell consistent_alloc(),
etc. about that.

> The virt_to_* functions
> are on their way out (on all platforms), and everyone should use the
> DMA address returned by the consistent_alloc function.

Jeez, you look the other way for a little while :-)  The last time I was
doing Linux driver development was in 1999, and virt_to_* was still
getting phased in, it seemed...

If you wouldn't mind, could you give me a pointer to the discussion of
this phase-out?

> I guess if you
> have a 8xx/PCI that doesn't map PCI masters to system memory at zero,
> you are kind of hosed at the moment........

I wonder if the proper thing to do is to enhance consistent_alloc to
take an argument indicating the bus type?  There are more elegant ways
of doing things, but this wouldn't involve changes to a lot of code.

--
.....................................................................
 Peter Desnoyers            (781) 457-1165   pdesnoyers@chinook.com
 Chinook Communications     (617) 661-1979   pjd@fred.cambridge.ma.us
 100 Hayden Ave, Lexington MA 02421

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Non-cacheable memory

[Dan Malek]

Peter Desnoyers wrote:

> This seems like another case of either too little abstraction, or too
> much.  An embedded PPC system with PCI basically has two buses, with
> different DMA addressing, and there's no way to tell consistent_alloc(),
> etc. about that.

Well, if you are using PCI devices, you should be calling the pci_*
versions of these functions.  Those include a pci_dev structure, so
you can get the information you need to do this correctly, and seems
like the solution (more discussion later in the message).

> If you wouldn't mind, could you give me a pointer to the discussion of
> this phase-out?

I hear it from other folks on ppc-dev or something.  There are too many
lists for me to read and find this information first hand :-).


> I wonder if the proper thing to do is to enhance consistent_alloc to
> take an argument indicating the bus type?

Yep.  Like I said above, you should be using the pci_* versions of
these functions.  We can extract information from the pci_dev and pass
it along to consistent_alloc.  The local, internal peripherals can just
pass a NULL or zero or something, whatever this parameter is supposed to
mean.  Or, perhaps the pci_* functions fix up the information returned
from consistent_alloc.

> ....  There are more elegant ways
> of doing things, but this wouldn't involve changes to a lot of code.

This seems elegant enough to me :-).


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/