Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

hi ,everybody:

       I encountered this problem when porting my ethernet driver from
linux-2.6.28 to linux-2.6.33-arm1.

       In the linux-2.6.28, I used the dma_alloc_coherent to share the
informations between cpu and ethernet device.The program flow in the
function ndo_start_xmit:
               1,preparing the struct sk_buff->data for device internal DMA to
read,using the dma_map_single function.
               2,update the information in the memory which is allocated by
dma_alloc_coherent to tell the device DMA the data is readable.
               3,write the device register to inform the device DMA
to read the data.
               4,cpu captures the interrupt of reading completion
form the device.

       In the linux-2.6.28,the whole program flow soon complete in order.
       But when the driver was ported to the linux-2.6.33-arm1,the problem
came:
                I found when the cpu executed from step 1-3, the
device DMA alarm the
step 2 was not completed.As a result,the step 4 was not triggered.This
is different form linux-2.6.28.

               After some tests, I think the problem comes from the
dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
memory allocated by dma_alloc_coherent is written,the data entry into
the ddr much slower than the same process in linux-2.6.28.Therefore,when
cpu has executed the step 3, the step 2 has not yet completed.So the DMA
can't get the correct information ,and the step 4 will never be reached.

       I don't know the reason until now.I guess the memory allocated by the
dma_alloc_coherent may be cached....if not , why it is so slowly?

 Thanks!

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Ben Dooks]

On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
> hi ,everybody:
> 
>        I encountered this problem when porting my ethernet driver from
> linux-2.6.28 to linux-2.6.33-arm1.
> 
>        In the linux-2.6.28, I used the dma_alloc_coherent to share the
> informations between cpu and ethernet device.The program flow in the
> function ndo_start_xmit:
>                1,preparing the struct sk_buff->data for device internal DMA to
> read,using the dma_map_single function.
>                2,update the information in the memory which is allocated by
> dma_alloc_coherent to tell the device DMA the data is readable.
>                3,write the device register to inform the device DMA
> to read the data.
>                4,cpu captures the interrupt of reading completion
> form the device.
>
>        In the linux-2.6.28,the whole program flow soon complete in order.
>        But when the driver was ported to the linux-2.6.33-arm1,the problem
> came:
>                 I found when the cpu executed from step 1-3, the
> device DMA alarm the
> step 2 was not completed.As a result,the step 4 was not triggered.This
> is different form linux-2.6.28.
> 
>                After some tests, I think the problem comes from the
> dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
> memory allocated by dma_alloc_coherent is written,the data entry into
> the ddr much slower than the same process in linux-2.6.28.Therefore,when
> cpu has executed the step 3, the step 2 has not yet completed.So the DMA
> can't get the correct information ,and the step 4 will never be reached.
> 
>        I don't know the reason until now.I guess the memory allocated by the
> dma_alloc_coherent may be cached....if not , why it is so slowly?

The dma_alloc_coherent coherent should return uncachable and un-bufferable
memory, otherwise you meed explicit flushing commands when changing between
the HW and CPU ownership.

-- 
Ben

Q:      What's a light-year?
A:      One-third less calories than a regular year.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Russell King - ARM Linux]

On Thu, Jul 15, 2010 at 10:31:07AM +0100, Ben Dooks wrote:
> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
> >        I don't know the reason until now.I guess the memory allocated by the
> > dma_alloc_coherent may be cached....if not , why it is so slowly?
> 
> The dma_alloc_coherent coherent should return uncachable and un-bufferable
> memory, otherwise you meed explicit flushing commands when changing between
> the HW and CPU ownership.

It's a result of the weakly ordered memory model - writes to memory
are now delayed and can be re-ordered, but not across the appropriate
barrier instructions.

Lack of barrier instructions makes that these writes can arrive in
unexpected orders, which will cause exactly these kinds of problems.

Catalin has been tracking down where these barriers need to be added,
so we're getting these issues solved - but we can only find them
through people using the kernel, reporting problems like this, and
fixing the missing places.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

I have explicitly added the barrier() in my driver, but it did't work.

I will go on testing until solve the problem. I need more advice.

Thanks!

On Thu, Jul 15, 2010 at 5:38 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 15, 2010 at 10:31:07AM +0100, Ben Dooks wrote:
>> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
>> > ? ? ? ?I don't know the reason until now.I guess the memory allocated by the
>> > dma_alloc_coherent may be cached....if not , why it is so slowly?
>>
>> The dma_alloc_coherent coherent should return uncachable and un-bufferable
>> memory, otherwise you meed explicit flushing commands when changing between
>> the HW and CPU ownership.
>
> It's a result of the weakly ordered memory model - writes to memory
> are now delayed and can be re-ordered, but not across the appropriate
> barrier instructions.
>
> Lack of barrier instructions makes that these writes can arrive in
> unexpected orders, which will cause exactly these kinds of problems.
>
> Catalin has been tracking down where these barriers need to be added,
> so we're getting these issues solved - but we can only find them
> through people using the kernel, reporting problems like this, and
> fixing the missing places.
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Russell King - ARM Linux]

On Thu, Jul 15, 2010 at 05:56:41PM +0800, David Yang wrote:
> I have explicitly added the barrier() in my driver, but it did't work.

This imples that you've tried your own solution, which is completely
different from Catalin's patches.

barrier() is just a compiler barrier, nothing more.  It won't have any
effect on external memory ordering.  It's not surprising this doesn't
have much effect.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

Ok,I see .

I will try the patch you offered.

And I will give the results later.

Thanks!

On Thu, Jul 15, 2010 at 5:59 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 15, 2010 at 05:56:41PM +0800, David Yang wrote:
>> I have explicitly added the barrier() in my driver, but it did't work.
>
> This imples that you've tried your own solution, which is completely
> different from Catalin's patches.
>
> barrier() is just a compiler barrier, nothing more. ?It won't have any
> effect on external memory ordering. ?It's not surprising this doesn't
> have much effect.
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Shilimkar, Santosh]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> kernel-bounces at lists.infradead.org] On Behalf Of David Yang
> Sent: Thursday, July 15, 2010 3:27 PM
> To: Russell King - ARM Linux
> Cc: linux-arm-kernel at lists.infradead.org; Ben Dooks
> Subject: Re: Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with
> RealView platform,board PBX-A9 and armv7 instructions.
> 
> I have explicitly added the barrier() in my driver, but it did't work.
> 
> I will go on testing until solve the problem. I need more advice.
>
The 'barrier()' is just a compiler barrier. You should be using "mb" "rmb" 
or "wmb"  depending on the usage.

Regards,
Santosh

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Russell King - ARM Linux]

On Thu, Jul 15, 2010 at 03:34:58PM +0530, Shilimkar, Santosh wrote:
> > -----Original Message-----
> > From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> > kernel-bounces at lists.infradead.org] On Behalf Of David Yang
> > Sent: Thursday, July 15, 2010 3:27 PM
> > To: Russell King - ARM Linux
> > Cc: linux-arm-kernel at lists.infradead.org; Ben Dooks
> > Subject: Re: Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with
> > RealView platform,board PBX-A9 and armv7 instructions.
> > 
> > I have explicitly added the barrier() in my driver, but it did't work.
> > 
> > I will go on testing until solve the problem. I need more advice.
> >
> The 'barrier()' is just a compiler barrier. You should be using "mb" "rmb" 
> or "wmb"  depending on the usage.

It'd be useful if the patches which are destined for mainline get some
independent testing by people who have experienced the problem, rather
than creating their own solutions.

We can then have much more confidence in the patch which may eventually
be applied is solving problems people are seeing - and more importantly
identify any defficiencies in the patch so we don't have to repeatedly
patch the same area multiple times.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

patch-6220/1   can't be  patched on kernel version linux-2.6.33-arm1.

the file arch/arm/mm/cache-l2x0.c  may be too old for the patch.


On Thu, Jul 15, 2010 at 6:09 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 15, 2010 at 03:34:58PM +0530, Shilimkar, Santosh wrote:
>> > -----Original Message-----
>> > From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
>> > kernel-bounces at lists.infradead.org] On Behalf Of David Yang
>> > Sent: Thursday, July 15, 2010 3:27 PM
>> > To: Russell King - ARM Linux
>> > Cc: linux-arm-kernel at lists.infradead.org; Ben Dooks
>> > Subject: Re: Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with
>> > RealView platform,board PBX-A9 and armv7 instructions.
>> >
>> > I have explicitly added the barrier() in my driver, but it did't work.
>> >
>> > I will go on testing until solve the problem. I need more advice.
>> >
>> The 'barrier()' is just a compiler barrier. You should be using "mb" "rmb"
>> or "wmb" ?depending on the usage.
>
> It'd be useful if the patches which are destined for mainline get some
> independent testing by people who have experienced the problem, rather
> than creating their own solutions.
>
> We can then have much more confidence in the patch which may eventually
> be applied is solving problems people are seeing - and more importantly
> identify any defficiencies in the patch so we don't have to repeatedly
> patch the same area multiple times.
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Russell King - ARM Linux]

Catalin, can you help with this please?

Thanks.

On Thu, Jul 15, 2010 at 07:16:24PM +0800, David Yang wrote:
> patch-6220/1   can't be  patched on kernel version linux-2.6.33-arm1.
> 
> the file arch/arm/mm/cache-l2x0.c  may be too old for the patch.
> 
> 
> On Thu, Jul 15, 2010 at 6:09 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Thu, Jul 15, 2010 at 03:34:58PM +0530, Shilimkar, Santosh wrote:
> >> > -----Original Message-----
> >> > From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> >> > kernel-bounces at lists.infradead.org] On Behalf Of David Yang
> >> > Sent: Thursday, July 15, 2010 3:27 PM
> >> > To: Russell King - ARM Linux
> >> > Cc: linux-arm-kernel at lists.infradead.org; Ben Dooks
> >> > Subject: Re: Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with
> >> > RealView platform,board PBX-A9 and armv7 instructions.
> >> >
> >> > I have explicitly added the barrier() in my driver, but it did't work.
> >> >
> >> > I will go on testing until solve the problem. I need more advice.
> >> >
> >> The 'barrier()' is just a compiler barrier. You should be using "mb" "rmb"
> >> or "wmb" ?depending on the usage.
> >
> > It'd be useful if the patches which are destined for mainline get some
> > independent testing by people who have experienced the problem, rather
> > than creating their own solutions.
> >
> > We can then have much more confidence in the patch which may eventually
> > be applied is solving problems people are seeing - and more importantly
> > identify any defficiencies in the patch so we don't have to repeatedly
> > patch the same area multiple times.
> >

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 11:09 +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 15, 2010 at 03:34:58PM +0530, Shilimkar, Santosh wrote:
> > > -----Original Message-----
> > > From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> > > kernel-bounces at lists.infradead.org] On Behalf Of David Yang
> > > Sent: Thursday, July 15, 2010 3:27 PM
> > > To: Russell King - ARM Linux
> > > Cc: linux-arm-kernel at lists.infradead.org; Ben Dooks
> > > Subject: Re: Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with
> > > RealView platform,board PBX-A9 and armv7 instructions.
> > >
> > > I have explicitly added the barrier() in my driver, but it did't work.
> > >
> > > I will go on testing until solve the problem. I need more advice.
> > >
> > The 'barrier()' is just a compiler barrier. You should be using "mb" "rmb"
> > or "wmb"  depending on the usage.
> 
> It'd be useful if the patches which are destined for mainline get some
> independent testing by people who have experienced the problem, rather
> than creating their own solutions.

In case I forgot to mention, the people from Trident Micro tested my
three "ordered" accessors patches and it solved the problem for them
(unfortunately the discussion was in private e-mails).

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

I think so ?but it looks like that it did not return uncacheble and
unbufferable memory.

what flushing command could I use in driver for  testing?

On Thu, Jul 15, 2010 at 5:31 PM, Ben Dooks <ben-linux@fluff.org> wrote:
> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
>> hi ,everybody:
>>
>> ? ? ? ?I encountered this problem when porting my ethernet driver from
>> linux-2.6.28 to linux-2.6.33-arm1.
>>
>> ? ? ? ?In the linux-2.6.28, I used the dma_alloc_coherent to share the
>> informations between cpu and ethernet device.The program flow in the
>> function ndo_start_xmit:
>> ? ? ? ? ? ? ? ?1,preparing the struct sk_buff->data for device internal DMA to
>> read,using the dma_map_single function.
>> ? ? ? ? ? ? ? ?2,update the information in the memory which is allocated by
>> dma_alloc_coherent to tell the device DMA the data is readable.
>> ? ? ? ? ? ? ? ?3,write the device register to inform the device DMA
>> to read the data.
>> ? ? ? ? ? ? ? ?4,cpu captures the interrupt of reading completion
>> form the device.
>>
>> ? ? ? ?In the linux-2.6.28,the whole program flow soon complete in order.
>> ? ? ? ?But when the driver was ported to the linux-2.6.33-arm1,the problem
>> came:
>> ? ? ? ? ? ? ? ? I found when the cpu executed from step 1-3, the
>> device DMA alarm the
>> step 2 was not completed.As a result,the step 4 was not triggered.This
>> is different form linux-2.6.28.
>>
>> ? ? ? ? ? ? ? ?After some tests, I think the problem comes from the
>> dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
>> memory allocated by dma_alloc_coherent is written,the data entry into
>> the ddr much slower than the same process in linux-2.6.28.Therefore,when
>> cpu has executed the step 3, the step 2 has not yet completed.So the DMA
>> can't get the correct information ,and the step 4 will never be reached.
>>
>> ? ? ? ?I don't know the reason until now.I guess the memory allocated by the
>> dma_alloc_coherent may be cached....if not , why it is so slowly?
>
> The dma_alloc_coherent coherent should return uncachable and un-bufferable
> memory, otherwise you meed explicit flushing commands when changing between
> the HW and CPU ownership.
>
> --
> Ben
>
> Q: ? ? ?What's a light-year?
> A: ? ? ?One-third less calories than a regular year.
>
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 10:31 +0100, Ben Dooks wrote:
> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
> >        I encountered this problem when porting my ethernet driver from
> > linux-2.6.28 to linux-2.6.33-arm1.
> >
> >        In the linux-2.6.28, I used the dma_alloc_coherent to share the
> > informations between cpu and ethernet device.The program flow in the
> > function ndo_start_xmit:
> >                1,preparing the struct sk_buff->data for device internal DMA to
> > read,using the dma_map_single function.
> >                2,update the information in the memory which is allocated by
> > dma_alloc_coherent to tell the device DMA the data is readable.
> >                3,write the device register to inform the device DMA
> > to read the data.
> >                4,cpu captures the interrupt of reading completion
> > form the device.
> >
> >        In the linux-2.6.28,the whole program flow soon complete in order.
> >        But when the driver was ported to the linux-2.6.33-arm1,the problem
> > came:
> >                 I found when the cpu executed from step 1-3, the
> > device DMA alarm the
> > step 2 was not completed.As a result,the step 4 was not triggered.This
> > is different form linux-2.6.28.
> >
> >                After some tests, I think the problem comes from the
> > dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
> > memory allocated by dma_alloc_coherent is written,the data entry into
> > the ddr much slower than the same process in linux-2.6.28.Therefore,when
> > cpu has executed the step 3, the step 2 has not yet completed.So the DMA
> > can't get the correct information ,and the step 4 will never be reached.
> >
> >        I don't know the reason until now.I guess the memory allocated by the
> > dma_alloc_coherent may be cached....if not , why it is so slowly?
> 
> The dma_alloc_coherent coherent should return uncachable and un-bufferable
> memory, otherwise you meed explicit flushing commands when changing between
> the HW and CPU ownership.

When CONFIG_DMA_MEM_BUFFERABLE is enabled (ARMv6 and ARMv7), the
coherent DMA buffers are uncached but "buffered". That's because Normal
Non-cacheable memory is always buffered. While the Device accesses are
also buffered, the two accesses aren't guaranteed to be ordered with
each-other and you would need to use a barrier - wmb() usually.

If you have an outer cache (L2), in some hardware configurations (I
would say most), the DSB doesn't get through to the L2 cache but it
buffers Normal Non-cacheable accesses. This is solved by doing an L2
cache sync in the wmb() macro.

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Russell King - ARM Linux]

On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
>        But when the driver was ported to the linux-2.6.33-arm1,the problem
> came:
>                 I found when the cpu executed from step 1-3, the
> device DMA alarm the
> step 2 was not completed.As a result,the step 4 was not triggered.This
> is different form linux-2.6.28.

Can you try patches 6219/1 6220/1 and 6221/1 from the patch system?
They can be viewed/downloaded via:

http://www.arm.linux.org.uk/developer/patches/view.php?id=6219/1
http://www.arm.linux.org.uk/developer/patches/view.php?id=6220/1
http://www.arm.linux.org.uk/developer/patches/view.php?id=6221/1

Thanks.

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 10:35 +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
> >        But when the driver was ported to the linux-2.6.33-arm1,the problem
> > came:
> >                 I found when the cpu executed from step 1-3, the
> > device DMA alarm the
> > step 2 was not completed.As a result,the step 4 was not triggered.This
> > is different form linux-2.6.28.
> 
> Can you try patches 6219/1 6220/1 and 6221/1 from the patch system?
> They can be viewed/downloaded via:
> 
> http://www.arm.linux.org.uk/developer/patches/view.php?id=6219/1
> http://www.arm.linux.org.uk/developer/patches/view.php?id=6220/1
> http://www.arm.linux.org.uk/developer/patches/view.php?id=6221/1

I posted an updated set of patches here to avoid the do...while()
construct causing compilation errors in some drivers:

http://thread.gmane.org/gmane.linux.ports.arm.kernel/84948

If you are ok with them, I'll push them to the patch system (and discard
the previous ones).

Thanks.

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

hi,Catalin

the patch v3 2/3 can not be patched to my  arch/arm/mm/cache-l2x0.c
from linux-2.6.33-arm1

should I modify it manually?


On Thu, Jul 15, 2010 at 8:21 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, 2010-07-15 at 10:35 +0100, Russell King - ARM Linux wrote:
>> On Thu, Jul 15, 2010 at 05:13:57PM +0800, David Yang wrote:
>> > ? ? ? ?But when the driver was ported to the linux-2.6.33-arm1,the problem
>> > came:
>> > ? ? ? ? ? ? ? ? I found when the cpu executed from step 1-3, the
>> > device DMA alarm the
>> > step 2 was not completed.As a result,the step 4 was not triggered.This
>> > is different form linux-2.6.28.
>>
>> Can you try patches 6219/1 6220/1 and 6221/1 from the patch system?
>> They can be viewed/downloaded via:
>>
>> http://www.arm.linux.org.uk/developer/patches/view.php?id=6219/1
>> http://www.arm.linux.org.uk/developer/patches/view.php?id=6220/1
>> http://www.arm.linux.org.uk/developer/patches/view.php?id=6221/1
>
> I posted an updated set of patches here to avoid the do...while()
> construct causing compilation errors in some drivers:
>
> http://thread.gmane.org/gmane.linux.ports.arm.kernel/84948
>
> If you are ok with them, I'll push them to the patch system (and discard
> the previous ones).
>
> Thanks.
>
> --
> Catalin
>
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 13:52 +0100, David Yang wrote:
> the patch v3 2/3 can not be patched to my  arch/arm/mm/cache-l2x0.c
> from linux-2.6.33-arm1
> 
> should I modify it manually?

It's just a simple replacement - s/writel/writel_relaxed/

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

I have passed the test,when using wmb() and without PATCH 0/3;

But,I have a  doubt : if I use the wmb(), then the l2x0.c 's patch is
not needed?


I will test the PATCH v3 0/3  tomorrow, without using wmb();



Thanks!

On Thu, Jul 15, 2010 at 8:56 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, 2010-07-15 at 13:52 +0100, David Yang wrote:
>> the patch v3 2/3 can not be patched to my ?arch/arm/mm/cache-l2x0.c
>> from linux-2.6.33-arm1
>>
>> should I modify it manually?
>
> It's just a simple replacement - s/writel/writel_relaxed/
>
> --
> Catalin
>
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 14:10 +0100, David Yang wrote:
> I have passed the test,when using wmb() and without PATCH 0/3;

Great, so it's a memory ordering issue.

> But,I have a  doubt : if I use the wmb(), then the l2x0.c 's patch is
> not needed?

Correct, if you add the wmb(), none of the three ordered accessors
patches are needed. But we'll probably push them anyway, in which case
the wmb() becomes superfluous.

> I will test the PATCH v3 0/3  tomorrow, without using wmb();

Thanks.

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

I have a question: in linux-2.6.28, there is no barriers in the IO accessors.

So,how does it prevent the reorder of the io  instructions?

On Thu, Jul 15, 2010 at 9:17 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, 2010-07-15 at 14:10 +0100, David Yang wrote:
>> I have passed the test,when using wmb() and without PATCH 0/3;
>
> Great, so it's a memory ordering issue.
>
>> But,I have a ?doubt : if I use the wmb(), then the l2x0.c 's patch is
>> not needed?
>
> Correct, if you add the wmb(), none of the three ordered accessors
> patches are needed. But we'll probably push them anyway, in which case
> the wmb() becomes superfluous.
>
>> I will test the PATCH v3 0/3 ?tomorrow, without using wmb();
>
> Thanks.
>
> --
> Catalin
>
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Sat, 2010-07-17 at 08:05 +0100, David Yang wrote:
> I have a question: in linux-2.6.28, there is no barriers in the IO
> accessors.

Because in 2.6.28 we used Strongly Ordered memory for the coherent DMA
buffers and on many CPUs SO accesses are ordered relatively to Device
accesses. But that even this isn't guaranteed by the ARM ARM.

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

On Mon, Jul 19, 2010 at 8:29 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Sat, 2010-07-17 at 08:05 +0100, David Yang wrote:
>> I have a question: in linux-2.6.28, there is no barriers in the IO
>> accessors.
>
> Because in 2.6.28 we used Strongly Ordered memory for the coherent DMA
> buffers and on many CPUs SO accesses are ordered relatively to Device
> accesses. But that even this isn't guaranteed by the ARM ARM.
>

Yes, I know that.

But I mean that ,in linux-2.6.28,whether the order between the IO
access will be re-ordered at the hardware level? And whether we should
use the wmb() to guarantee the order? Here we just talk about the IO
register access,without the normal memory or coherent DMA buffers.

Thank you!

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Tue, 2010-07-20 at 04:17 +0100, David Yang wrote:
> On Mon, Jul 19, 2010 at 8:29 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Sat, 2010-07-17 at 08:05 +0100, David Yang wrote:
> >> I have a question: in linux-2.6.28, there is no barriers in the IO
> >> accessors.
> >
> > Because in 2.6.28 we used Strongly Ordered memory for the coherent DMA
> > buffers and on many CPUs SO accesses are ordered relatively to Device
> > accesses. But that even this isn't guaranteed by the ARM ARM.
> >
> 
> Yes, I know that.
> 
> But I mean that ,in linux-2.6.28,whether the order between the IO
> access will be re-ordered at the hardware level? And whether we should
> use the wmb() to guarantee the order? Here we just talk about the IO
> register access,without the normal memory or coherent DMA buffers.

IO accesses (via Device memory mapping) are guaranteed by the hardware
to be ordered (i.e. they reach the device in the specified program
order).

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

eh...is it right that I need not use  wmb()  between two writel()?

On Tue, Jul 20, 2010 at 4:53 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Tue, 2010-07-20 at 04:17 +0100, David Yang wrote:
>> On Mon, Jul 19, 2010 at 8:29 PM, Catalin Marinas
>> <catalin.marinas@arm.com> wrote:
>> > On Sat, 2010-07-17 at 08:05 +0100, David Yang wrote:
>> >> I have a question: in linux-2.6.28, there is no barriers in the IO
>> >> accessors.
>> >
>> > Because in 2.6.28 we used Strongly Ordered memory for the coherent DMA
>> > buffers and on many CPUs SO accesses are ordered relatively to Device
>> > accesses. But that even this isn't guaranteed by the ARM ARM.
>> >
>>
>> Yes, I know that.
>>
>> But I mean that ,in linux-2.6.28,whether the order between the IO
>> access will be re-ordered at the hardware level? And whether we should
>> use the wmb() to guarantee the order? Here we just talk about the IO
>> register access,without the normal memory or coherent DMA buffers.
>
> IO accesses (via Device memory mapping) are guaranteed by the hardware
> to be ordered (i.e. they reach the device in the specified program
> order).
>

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , withRealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Tue, 2010-07-20 at 17:22 +0800, David Yang wrote:
> eh...is it right that I need not use  wmb()  between two writel()?

It is right.

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[Catalin Marinas]

On Thu, 2010-07-15 at 17:13 +0800, David Yang wrote:
>        I encountered this problem when porting my ethernet driver from
> linux-2.6.28 to linux-2.6.33-arm1.
> 
>        In the linux-2.6.28, I used the dma_alloc_coherent to share the
> informations between cpu and ethernet device.The program flow in the
> function ndo_start_xmit:
>                1,preparing the struct sk_buff->data for device internal DMA to
> read,using the dma_map_single function.
>                2,update the information in the memory which is allocated by
> dma_alloc_coherent to tell the device DMA the data is readable.
>                3,write the device register to inform the device DMA
> to read the data.

You need a wmb() before the writel() which tells the device to start the
DMA transfer. We'd like to make this the default (in the write* macros)
and there are patches for review on the list.

Without the barrier, the card may transfer garbage since the date
written in step (2) may have not reached the RAM.

>                4,cpu captures the interrupt of reading completion
> form the device.
[...]
>                After some tests, I think the problem comes from the
> dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
> memory allocated by dma_alloc_coherent is written,the data entry into
> the ddr much slower than the same process in linux-2.6.28.Therefore,when
> cpu has executed the step 3, the step 2 has not yet completed.So the DMA
> can't get the correct information ,and the step 4 will never be reached.

Correct.

Note that I haven't fixed this issue in the 2.6.33-arm1 kernel. My
opinion for some time was that drivers should take care of memory
ordering and use the correct barriers. But after discussion on LKML, I
eventually changed my mind (and we now have a small drop in performance
for PIO drivers on ARMv6/v7).

Could you use a newer kernel? I can backport the I/O accessors patches
to 2.6.33-arm1 but only after Russell is ok with the latest version (to
avoid additional work on my side).

-- 
Catalin

Problem with dma_alloc_coherent at linux-2.6.33-arm1 , with RealView platform,board PBX-A9 and armv7 instructions.

[David Yang]

I have to use the linux-2.6.33-arm1 ,because I will use the android .

And ARM only offer the android patch for 2.6.33.

On Thu, Jul 15, 2010 at 8:25 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, 2010-07-15 at 17:13 +0800, David Yang wrote:
>> ? ? ? ?I encountered this problem when porting my ethernet driver from
>> linux-2.6.28 to linux-2.6.33-arm1.
>>
>> ? ? ? ?In the linux-2.6.28, I used the dma_alloc_coherent to share the
>> informations between cpu and ethernet device.The program flow in the
>> function ndo_start_xmit:
>> ? ? ? ? ? ? ? ?1,preparing the struct sk_buff->data for device internal DMA to
>> read,using the dma_map_single function.
>> ? ? ? ? ? ? ? ?2,update the information in the memory which is allocated by
>> dma_alloc_coherent to tell the device DMA the data is readable.
>> ? ? ? ? ? ? ? ?3,write the device register to inform the device DMA
>> to read the data.
>
> You need a wmb() before the writel() which tells the device to start the
> DMA transfer. We'd like to make this the default (in the write* macros)
> and there are patches for review on the list.
>
> Without the barrier, the card may transfer garbage since the date
> written in step (2) may have not reached the RAM.
>
>> ? ? ? ? ? ? ? ?4,cpu captures the interrupt of reading completion
>> form the device.
> [...]
>> ? ? ? ? ? ? ? ?After some tests, I think the problem comes from the
>> dma_alloc_coherent.It looks like ,in the linux-2.6.33-arm1, when the
>> memory allocated by dma_alloc_coherent is written,the data entry into
>> the ddr much slower than the same process in linux-2.6.28.Therefore,when
>> cpu has executed the step 3, the step 2 has not yet completed.So the DMA
>> can't get the correct information ,and the step 4 will never be reached.
>
> Correct.
>
> Note that I haven't fixed this issue in the 2.6.33-arm1 kernel. My
> opinion for some time was that drivers should take care of memory
> ordering and use the correct barriers. But after discussion on LKML, I
> eventually changed my mind (and we now have a small drop in performance
> for PIO drivers on ARMv6/v7).
>
> Could you use a newer kernel? I can backport the I/O accessors patches
> to 2.6.33-arm1 but only after Russell is ok with the latest version (to
> avoid additional work on my side).
>
> --
> Catalin
>
>