non barrier versions of dma_map functions

non barrier versions of dma_map functions

[adharmap at codeaurora.org]

We have a situation where we need to dma map multiple cached buffers for a
single dma transaction.

The current DMA api suggests the use of dma_map_single for cache
consistency. On ARMv7 it performs the necessary cache-operations and calls
data sync barrier instruction (DSB). In our case we would be executing
multiple DSB instruction before starting the dma operation - we need
memory to be consistent only after we map the last buffer.

I am thinking we could define "no barrier" version's of all the mapping
functions and then a barrier function that results in DSB before the dma
is started.

Requesting alternative ideas or code design to get the desired nonbarrier
versions of the mapping functions.

Abhijeet Dharmapurikar

non barrier versions of dma_map functions

[Russell King - ARM Linux]

On Mon, Dec 07, 2009 at 11:37:21AM -0800, adharmap at codeaurora.org wrote:
> We have a situation where we need to dma map multiple cached buffers for a
> single dma transaction.
> 
> The current DMA api suggests the use of dma_map_single for cache
> consistency. On ARMv7 it performs the necessary cache-operations and calls
> data sync barrier instruction (DSB). In our case we would be executing
> multiple DSB instruction before starting the dma operation - we need
> memory to be consistent only after we map the last buffer.

Is it a problem and do you have numbers to illustrate why it is a
problem, or is this just theory?

I suspect that the DSB doesn't figure in the bigger scheme of things
(such as running the cache maintainence operations.)  Moreover, there's
bigger issues (such as the DMA cache maintainence on ARMv7 actually
*being* correct - by invalidating after the DMA has completed) to be
getting on with at the moment that things as you're suggesting aren't
worth worrying about.

non barrier versions of dma_map functions

[Abhijeet Dharmapurikar]

Russell King - ARM Linux wrote:
> On Mon, Dec 07, 2009 at 11:37:21AM -0800, adharmap at codeaurora.org wrote:
>> We have a situation where we need to dma map multiple cached buffers for a
>> single dma transaction.
>>
>> The current DMA api suggests the use of dma_map_single for cache
>> consistency. On ARMv7 it performs the necessary cache-operations and calls
>> data sync barrier instruction (DSB). In our case we would be executing
>> multiple DSB instruction before starting the dma operation - we need
>> memory to be consistent only after we map the last buffer.
> 
> Is it a problem and do you have numbers to illustrate why it is a
> problem, or is this just theory?


Here are numbers from a test ran on ARMv7 based device
It kmallocs N buffers of size 'size', dirties their cache by writing
to them and calls dma_map_single that calls the arch specific clean
operations with and without dsb. In "without dsb" case a dsb is executed
after the last buffer is mapped. The time is in microseconds

size	N	map_single	map_single w/o dsb	delta
128	16	8		5			60%
512	16	9		6			50%
512	32	15		8			88%
512	48	20		11			82%
512	64	27		14			93%
64	4	4		3			33%
64	8	4		3			33%
64	16	7		4			75%
64	32	12		4			200%
64	48	17		6			183%
64	64	21		7			200%
1024	16	9		7			29%

These buffer sizes and N are very close to real world sizes the
framebuffer driver handles. Cases where N is large happen the most
often.

Clearly,we could benefit from the nobarrier versions of the cache
operations and we could use them in scatter gather mappings as well.

Abhijeet

non barrier versions of dma_map functions

[Catalin Marinas]

On Thu, 2009-12-10 at 00:32 +0000, Abhijeet Dharmapurikar wrote:
> Russell King - ARM Linux wrote:
> > On Mon, Dec 07, 2009 at 11:37:21AM -0800, adharmap at codeaurora.org wrote:
> >> We have a situation where we need to dma map multiple cached buffers for a
> >> single dma transaction.
> >>
> >> The current DMA api suggests the use of dma_map_single for cache
> >> consistency. On ARMv7 it performs the necessary cache-operations and calls
> >> data sync barrier instruction (DSB). In our case we would be executing
> >> multiple DSB instruction before starting the dma operation - we need
> >> memory to be consistent only after we map the last buffer.
> >
> > Is it a problem and do you have numbers to illustrate why it is a
> > problem, or is this just theory?
> 
> 
> Here are numbers from a test ran on ARMv7 based device
> It kmallocs N buffers of size 'size', dirties their cache by writing
> to them and calls dma_map_single that calls the arch specific clean
> operations with and without dsb. In "without dsb" case a dsb is executed
> after the last buffer is mapped. The time is in microseconds

Interesting results but I have some additional questions:

What is the direction given to dma_map_single()?

If the direction is TO_DEVICE, has the buffer been dirtied before
calling dma_map_single()? The DSB overhead could be smaller compared to
the actual cache flushing.

-- 
Catalin

non barrier versions of dma_map functions

[Abhijeet Dharmapurikar]

Catalin Marinas wrote:
> On Thu, 2009-12-10 at 00:32 +0000, Abhijeet Dharmapurikar wrote:
>> Russell King - ARM Linux wrote:
>>> On Mon, Dec 07, 2009 at 11:37:21AM -0800, adharmap at codeaurora.org wrote:
>>>> We have a situation where we need to dma map multiple cached buffers for a
>>>> single dma transaction.
>>>>
>>>> The current DMA api suggests the use of dma_map_single for cache
>>>> consistency. On ARMv7 it performs the necessary cache-operations and calls
>>>> data sync barrier instruction (DSB). In our case we would be executing
>>>> multiple DSB instruction before starting the dma operation - we need
>>>> memory to be consistent only after we map the last buffer.
>>> Is it a problem and do you have numbers to illustrate why it is a
>>> problem, or is this just theory?
>>
>> Here are numbers from a test ran on ARMv7 based device
>> It kmallocs N buffers of size 'size', dirties their cache by writing
>> to them and calls dma_map_single that calls the arch specific clean
>> operations with and without dsb. In "without dsb" case a dsb is executed
>> after the last buffer is mapped. The time is in microseconds
> 
> Interesting results but I have some additional questions:
> 
> What is the direction given to dma_map_single()?
> 
DMA_TO_DEVICE
> If the direction is TO_DEVICE, has the buffer been dirtied before
> calling dma_map_single()? The DSB overhead could be smaller compared to
> the actual cache flushing.
Yes the buffers were dirtied before calling 
dma_map_single(...DMA_TO_DEVICE);
> 

non barrier versions of dma_map functions

[Russell King - ARM Linux]

On Thu, Dec 10, 2009 at 10:16:49AM -0800, Abhijeet Dharmapurikar wrote:
> Catalin Marinas wrote:
>> What is the direction given to dma_map_single()?
>>
> DMA_TO_DEVICE
>> If the direction is TO_DEVICE, has the buffer been dirtied before
>> calling dma_map_single()? The DSB overhead could be smaller compared to
>> the actual cache flushing.
> Yes the buffers were dirtied before calling  
> dma_map_single(...DMA_TO_DEVICE);

Well, I don't think we can extend the API like this without cross-platform
agreement.

non barrier versions of dma_map functions

[Abhijeet Dharmapurikar]

Hello  All,

   This is a request for extending the DMA api for efficient handling of 
multiple buffers or scatter gather mapping/unmapping operations.

   I am based on an ARMv7 device and we have a situation where we need 
to dma map multiple cached buffers for a single dma transaction.

   The current DMA api suggests the use of dma_map_single/ 
dma_unmap_single for cache consistency. On ARMv7 it performs the 
necessary cache-operations and calls data sync barrier instruction 
(DSB). In our case we would be executing multiple DSB instructions
before starting the dma operation - we need memory to be consistent
only after we map the last buffer.

   I am thinking we could define "no barrier" version's of all the 
mapping/unmapping functions and then a barrier function that results
in DSB before the dma is started.

   Here are numbers from a test ran on my board.

It kmallocs N buffers of size 'size', dirties their cache by writing
to them and calls dma_map_single that calls the arch specific clean
operations with and without DSB. In "without DSB" case a dsb is executed
after the last buffer is mapped. The time is in microseconds

size    N    map_single    map_single w/o DSB    delta
128    16    8             5                     60%
512    16    9             6                     50%
512    32    15            8                     88%
512    48    20            11                    82%
512    64    27            14                    93%
64     4     4             3                     33%
64     8     4             3                     33%
64     16    7             4                     75%
64     32    12            4                     200%
64     48    17            6                     183%
64     64    21            7                     200%
1024   16    9             7                     29%

These buffer sizes and N are very close to real world sizes the
framebuffer driver handles. Cases where N is large happen the most
often.

Clearly,we could benefit from the nobarrier versions of the cache
operations and we could use them in scatter gather mappings as well.

Since this kind of API change will affect all the platforms, I was 
directed by the arm-linux community to take this up on the linux kernel 
mailing list.

For architectures that don't need a barrier for the completions of
cache operations we can simply call the existing
dma_map_signle/dma_unmap_single.

Requesting alternative ideas or code design to get the desired 
nonbarrier versions of the mapping functions.

Thanks,
Abhijeet Dharmapurikar