atomic_add function on memory allocated with dma_alloc_coherent hangs

atomic_add function on memory allocated with dma_alloc_coherent hangs

[Riisisulg]

Hello. I'm using AMD Alchemy db1550 with au1550, it has 2.6.15 kernel
patched with linux-2.6.14.5-mips-1.patch, i have compiled cross compiler
for the platform (but not checked it for correctness)

since db1550 does not have USB2.0 but instead has two PCI slots, i
installed NEC's PCI to USB 2.0 host controller board to it. I was able
to load ehci-hcd driver with success but after the usb mass storage
device was inserted the system hanged. 
After debugging a while i found out that function atomic_add did not
return. I run few tests where atomic add asm code was alerted and got
confidence that sc instruction did not never succeed so the cycle was
repeating forever.
72                 unsigned long temp;
73 
74                 __asm__ __volatile__(
75                 "1:     ll      %0, %1          # atomic_add         
  \n"
76                 "       addu    %0, %2                               
  \n"
77                 "       sc      %0, %1                               
  \n"
78                 "       beqz    %0, 1b                               
  \n"
79                 : "=&r" (temp), "=m" (v->counter)
80                 : "Ir" (i), "m" (v->counter));
At this point it seems like atomic_add problem but that code is old for
ages. Next i started to make tests where atomic_add was used to get the
cpu hanged. 

It appears that the atomic_add function will not work when certain
memory allocation scheme is used. If the atomic variable is allocated
from stack, with kmalloc or with dma_allock_noncoherent it will work
fine, but when dma_allock_coherent or memory taken form memory pool
atomic_add will fail (i think that all ll sc instructions can fail but i
tested only with atomic_add)

To easy the testing i wrote driver that anybody should be able to
compile and run. For testing i used also an x86 machine and driver
worked (Test function is in module's init functioin)

Test driver available at
http://frogman.gf.ttu.ee/kernel_test.tar.bz2

to view the contents of the driver
http://frogman.gf.ttu.ee/kernel_test/

Since i have only one MIPS based system i dont know wether this problem
is specefic to amd alchemy 1550 or the entire amd alchemy family or some
other MIPS32 based system or is it just my cross compiler or else that
made mistake somewhere.

On irc i found one guy who did use my test driver on his MIPS and the
driver crashed (Of course probability is that the driver itself has bug
on it, but that driver worked fine on x86)
By saying driver works i mean kernel was able to load it without
"crashing" (looping infinitely).

jyr

Re: atomic_add function on memory allocated with dma_alloc_coherent hangs

[Ralf Baechle]

On Tue, Jan 17, 2006 at 11:40:53AM +0200, Riisisulg wrote:

> Hello. I'm using AMD Alchemy db1550 with au1550, it has 2.6.15 kernel
> patched with linux-2.6.14.5-mips-1.patch, i have compiled cross compiler
> for the platform (but not checked it for correctness)
> 
> since db1550 does not have USB2.0 but instead has two PCI slots, i
> installed NEC's PCI to USB 2.0 host controller board to it. I was able
> to load ehci-hcd driver with success but after the usb mass storage
> device was inserted the system hanged. 
> After debugging a while i found out that function atomic_add did not
> return. I run few tests where atomic add asm code was alerted and got
> confidence that sc instruction did not never succeed so the cycle was
> repeating forever.

The operation of ll/sc is undefined on uncached memory; non-coherent
MIPS systems - that is most embedded MIPS systems - will return non
cache-coherent memory for dma_alloc_coherent.  Yes, the naming sucks,
dma_alloc_{non}coherent do the opposite of what their name is.

  Ralf

RE: Re: atomic_add function on memory allocated with dma_alloc_coherent hangs

[Riisisulg]

> On Tue, Jan 17, 2006 at 11:40:53AM +0200, Riisisulg wrote:
> 
> > Hello. I'm using AMD Alchemy db1550 with au1550, it has 2.6.15
kernel
> > patched with linux-2.6.14.5-mips-1.patch, i have compiled cross
compiler
> > for the platform (but not checked it for correctness)
> > 
> > since db1550 does not have USB2.0 but instead has two PCI slots, i
> > installed NEC's PCI to USB 2.0 host controller board to it. I was
able
> > to load ehci-hcd driver with success but after the usb mass storage
> > device was inserted the system hanged. 
> > After debugging a while i found out that function atomic_add did
not
> > return. I run few tests where atomic add asm code was alerted and
got
> > confidence that sc instruction did not never succeed so the cycle
was
> > repeating forever.
> 
> The operation of ll/sc is undefined on uncached memory; non-coherent
> MIPS systems - that is most embedded MIPS systems - will return non
> cache-coherent memory for dma_alloc_coherent.  Yes, the naming sucks,
> dma_alloc_{non}coherent do the opposite of what their name is.
> 
>   Ralf

Thank you for quick answer so i will make some conclusions here

The problem come out by using ehci-hcd, that diver uses it's own dma
pool where it allocates structures of type ehci_qh.
struct ehci_qh has field of kref of type struct kref

in ehci-mem.c file there are functions qh_get(struct ehci_qh *qh),
qh_put(struct ehci_qh *qh) that use functions kref_get and kref_put.

Since ehci_qh structures are allocated from dma pool and both kref_get
and kref_put are using atomic functions that use ll sc then on
non-coherent MIPS system both qh_get and qh_put will hang the system.
The simplest workaround is to replace kref_get and kref_put in
ehci-mem.c or not to use dma pool. I have not cheked other kernel
drivers that use atomic functions on uncached memory.

END, i hope.