DMA descriptor alignment

DMA descriptor alignment

[Larry Finger]

For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor
rings must be aligned on an 8K boundary and must fit within an 8K page. On the x86_64 architecture
where the page size is 4K, I was getting addresses like 0x67AF000 when using dma_alloc_coherent
calls. From the description of the dma_pool_create and dma_pool_allocate routines, I thought they
would fix my problems; however, even with a dma_pool_create(name, dev, 8192, 8192, 8192) call, I'm
still getting 4K rather than 8K alignment, which results in DMA errors.

Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of
what it takes to get this kind of alignment?

Thanks,

Larry

Re: DMA descriptor alignment

[Andi Kleen]

Larry Finger <Larry.Finger@lwfinger.net> writes:

> For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor
> rings must be aligned on an 8K boundary and must fit within an 8K page. On the x86_64 architecture
> where the page size is 4K, I was getting addresses like 0x67AF000 when using dma_alloc_coherent
> calls.

Normally x86-64 dma_alloc_coherent calls the buddy allocator which gives
you always naturally aligned blocks. But there is a fallback calling
into swiotlb and swiotlb uses best fit allocation which only guarantees
single page alignment. That is probably what you're seeing.

My dma zone rework would remove that fallback case and should make it work.

> From the description of the dma_pool_create and dma_pool_allocate routines, I thought they
> would fix my problems; however, even with a dma_pool_create(name, dev, 8192, 8192, 8192) call, I'm
> still getting 4K rather than 8K alignment, which results in DMA errors.

They cannot give you more alignment than the underlying allocator.

> Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of
> what it takes to get this kind of alignment?

My suggestion as a short term workaround would be to first allocate 
8K using dma_alloc_coherent and if that has the wrong alignment get 16K 
and align yourself. When the driver is loaded later that might be unreliable,
but near boot or with enough free memory using order 2 should usually work.
 
With the dma zone rework that could be removed later.

-Andi

Re: DMA descriptor alignment

[Larry Finger]

Andi Kleen wrote:
> Larry Finger <Larry.Finger@lwfinger.net> writes:
> 
>> For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor
>> rings must be aligned on an 8K boundary and must fit within an 8K page. On the x86_64 architecture
>> where the page size is 4K, I was getting addresses like 0x67AF000 when using dma_alloc_coherent
>> calls.
> 
> Normally x86-64 dma_alloc_coherent calls the buddy allocator which gives
> you always naturally aligned blocks. But there is a fallback calling
> into swiotlb and swiotlb uses best fit allocation which only guarantees
> single page alignment. That is probably what you're seeing.
> 
> My dma zone rework would remove that fallback case and should make it work.
> 
>> From the description of the dma_pool_create and dma_pool_allocate routines, I thought they
>> would fix my problems; however, even with a dma_pool_create(name, dev, 8192, 8192, 8192) call, I'm
>> still getting 4K rather than 8K alignment, which results in DMA errors.
> 
> They cannot give you more alignment than the underlying allocator.
> 
>> Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of
>> what it takes to get this kind of alignment?
> 
> My suggestion as a short term workaround would be to first allocate 
> 8K using dma_alloc_coherent and if that has the wrong alignment get 16K 
> and align yourself. When the driver is loaded later that might be unreliable,
> but near boot or with enough free memory using order 2 should usually work.
>  
> With the dma zone rework that could be removed later.

I will be most interested in the dma zone rework; however, I now have the descriptors properly
allocated merely by asking for 8K, just as I had thought it should work. I'm not quite sure what was
wrong before, but I added a test to make certain that the alignment is OK just in case the problem
comes back.

Thanks for your response,

Larry