mapping uncached memory

mapping uncached memory

[Budhee Jamaich]

hello,

we are looking for ways to map memory as uncached,
so remote devices reading/writing to this memory will see consistent data.


these are the alternatives we currently see:


1. clean cache after every write to the memory (/invalidate the cache
before every read)

this can be done using dmac_clean/inv_range and outer_clean/inv_range
just like in dma_cache_maint.
btw - what's the difference between the dmac_* and the outer_*
functions ? why both are needed ?

disadvantage: this will be needed to be called before/after every memory access





2. use pgprot_noncached when assigning vma->vm_page_prot in our
driver's mmap method


will that really work ? all memory accesses will be uncached ?








3. use dma_alloc_coherent in some way

cons: documentation says we still need to use cache clean/inv
operations, so this might not really be of any advantage. but if that
is so, what's the idea behind coherent pools of memory ? how does it
work at all ?




what do you say ?


thank you all very much in advance
budhee

mapping uncached memory

[Russell King - ARM Linux]

On Wed, Mar 17, 2010 at 01:17:45AM +0200, Budhee Jamaich wrote:
> these are the alternatives we currently see:
> 
> 
> 1. clean cache after every write to the memory (/invalidate the cache
> before every read)
> 
> this can be done using dmac_clean/inv_range and outer_clean/inv_range
> just like in dma_cache_maint.
> btw - what's the difference between the dmac_* and the outer_*
> functions ? why both are needed ?

You don't need to know; they're the wrong interface to use.

See the DMA API instead; that is the official interface.  What you're
looking at above is an implementation detail of that interface which is
not intended for direct use by anything other than the DMA API.  Use of
this implementation detail will lead to breakage.

> 2. use pgprot_noncached when assigning vma->vm_page_prot in our
> driver's mmap method
> 
> 
> will that really work ? all memory accesses will be uncached ?

Not on ARMv6 and above - it has become illegal to remap memory with
differing type attributes.

> 3. use dma_alloc_coherent in some way
> 
> cons: documentation says we still need to use cache clean/inv
> operations, so this might not really be of any advantage. but if that
> is so, what's the idea behind coherent pools of memory ? how does it
> work at all ?

Please provide a pointer to that documentation.

Note that the whole "DMA to user mapped pages" issue is a _big_ can of
worms and doesn't appear to have any sane and non-expensive solutions.

mapping uncached memory

[Jamie Lokier]

Russell King - ARM Linux wrote:
> Note that the whole "DMA to user mapped pages" issue is a _big_ can of
> worms and doesn't appear to have any sane and non-expensive solutions.

I'm still worried about whether this means certain programs,
especially well-known open source databases, are broken on some ARMs
when they use O_DIRECT file access.

Fyi, O_DIRECT is flag to open(), mostly used by databases and programs
like QEMU, and sometimes from scripts using "dd oflag=direct" and "dd
iflag=direct".

It tells the kernel that reads and writes are to bypass the page
cache.  That causes DMA to happen between the userspace buffers given
to read() and write(), and the disk interface.

Unfortunately, this isn't some obscure specialised driver interface :-(

-- Jamie

mapping uncached memory

[Budhee Jamaich]

On Wed, Mar 17, 2010 at 1:54 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
>> 3. use dma_alloc_coherent in some way
>>
>> cons: documentation says we still need to use cache clean/inv

> Please provide a pointer to that documentation.


from Documentation/DMA-API.txt:

"void *
dma_alloc_coherent(struct device *dev, size_t size,
                             dma_addr_t *dma_handle, gfp_t flag)

Consistent memory is memory for which a write by either the device or
the processor can immediately be read by the processor or device
without having to worry about caching effects.  (You may however need
to make sure to flush the processor's write buffers before telling
devices to read that memory.)"



That last sentence - what does it really say ? That I still need to
manually clean/invalidate the caches myself ?



By reading consistent_init() I couldn't find any magic that makes the
memory uncached. Can you please tell me how the uncached property of
the memory is assured ?


thank you
budhee

mapping uncached memory

[Russell King - ARM Linux]

On Wed, Mar 17, 2010 at 10:02:09AM +0200, Budhee Jamaich wrote:
> On Wed, Mar 17, 2010 at 1:54 AM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> >> 3. use dma_alloc_coherent in some way
> >>
> >> cons: documentation says we still need to use cache clean/inv
> 
> > Please provide a pointer to that documentation.
> 
> 
> from Documentation/DMA-API.txt:
> 
> "void *
> dma_alloc_coherent(struct device *dev, size_t size,
>                              dma_addr_t *dma_handle, gfp_t flag)
> 
> Consistent memory is memory for which a write by either the device or
> the processor can immediately be read by the processor or device
> without having to worry about caching effects.  (You may however need
> to make sure to flush the processor's write buffers before telling
> devices to read that memory.)"
> 
> 
> 
> That last sentence - what does it really say ? That I still need to
> manually clean/invalidate the caches myself ?

No - if it did, it would contradict the previous sentence.  What it's
referring to is that on weakly ordered CPUs, you may need barriers.

mapping uncached memory

[Budhee Jamaich]

On Wed, Mar 17, 2010 at 10:15 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> No - if it did, it would contradict the previous sentence. ?What it's
> referring to is that on weakly ordered CPUs, you may need barriers.
>

thank you so much.

can you please just give me a hint so i can understand the underlying
mechanism that makes  dma_alloc_coherent work ?
how does the hardware know not to cache access to these addresses ?

again thank you so much

mapping uncached memory

[Colin Cross]

On Wed, Mar 17, 2010 at 1:15 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Mar 17, 2010 at 10:02:09AM +0200, Budhee Jamaich wrote:
>> On Wed, Mar 17, 2010 at 1:54 AM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>> from Documentation/DMA-API.txt:
>>
>> "void *
>> dma_alloc_coherent(struct device *dev, size_t size,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?dma_addr_t *dma_handle, gfp_t flag)
>>
>> Consistent memory is memory for which a write by either the device or
>> the processor can immediately be read by the processor or device
>> without having to worry about caching effects. ?(You may however need
>> to make sure to flush the processor's write buffers before telling
>> devices to read that memory.)"
>>
>>
>>
>> That last sentence - what does it really say ? That I still need to
>> manually clean/invalidate the caches myself ?
>
> No - if it did, it would contradict the previous sentence. ?What it's
> referring to is that on weakly ordered CPUs, you may need barriers.

Tegra2 needs a wmb() and an L2 cache sync on dma_alloc_coherent memory
before handing it to the device.  Currently, I am using Catalin's
patches for machine-specific wmb() implementations to do both
operations using a wmb() and keep the L2 details out of the drivers.
Is that the correct use of those patches?  Is there any other way to
handle dma_alloc_coherent memory on ARMv7?

mapping uncached memory

[Catalin Marinas]

On Wed, 2010-03-17 at 21:26 +0000, Colin Cross wrote:
> On Wed, Mar 17, 2010 at 1:15 AM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Wed, Mar 17, 2010 at 10:02:09AM +0200, Budhee Jamaich wrote:
> >> On Wed, Mar 17, 2010 at 1:54 AM, Russell King - ARM Linux
> >> <linux@arm.linux.org.uk> wrote:
> >> from Documentation/DMA-API.txt:
> >>
> >> "void *
> >> dma_alloc_coherent(struct device *dev, size_t size,
> >>                              dma_addr_t *dma_handle, gfp_t flag)
> >>
> >> Consistent memory is memory for which a write by either the device or
> >> the processor can immediately be read by the processor or device
> >> without having to worry about caching effects.  (You may however need
> >> to make sure to flush the processor's write buffers before telling
> >> devices to read that memory.)"
> >>
> >>
> >>
> >> That last sentence - what does it really say ? That I still need to
> >> manually clean/invalidate the caches myself ?
> >
> > No - if it did, it would contradict the previous sentence.  What it's
> > referring to is that on weakly ordered CPUs, you may need barriers.
> 
> Tegra2 needs a wmb() and an L2 cache sync on dma_alloc_coherent memory
> before handing it to the device.  

RealView boards with an L2 cache needs such thing as well.

> Currently, I am using Catalin's
> patches for machine-specific wmb() implementations to do both
> operations using a wmb() and keep the L2 details out of the drivers.
> Is that the correct use of those patches?  

Yes, the driver should only invoke a barrier rather than the L2 cache
specific functions.

> Is there any other way to
> handle dma_alloc_coherent memory on ARMv7?

I'm not aware of a different way. You could use streaming DMA API via
the dma_map_*() etc functions but this is cached memory and the
corresponding functions perform the flushing.

-- 
Catalin

mapping uncached memory

[Budhee Jamaich]

On Wed, Mar 17, 2010 at 1:54 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Mar 17, 2010 at 01:17:45AM +0200, Budhee Jamaich wrote:
>> these are the alternatives we currently see:
>>
>> 2. use pgprot_noncached when assigning vma->vm_page_prot in our
>> driver's mmap method
>>
>>
>> will that really work ? all memory accesses will be uncached ?
>
> Not on ARMv6 and above - it has become illegal to remap memory with
> differing type attributes.

what about using pgprot_dmacoherent in our driver to mark memory as
uncached ? will this work ?

thank you
budhee