Default atomic_pool_size value & changing it

Default atomic_pool_size value & changing it

[Rafał Miłecki]

Hi,

I've hit problem with allocating enough coherent DMA memory. Obviously
there is a limit of this memory as pointed by:
[ 0.283667] DMA: preallocated 256 KiB pool for atomic coherent allocations

In my case I was experimenting with a coherent memory used for DMA
descriptors. They are some small structs addressing SKBs and
controlling them:
struct bgmac_dma_desc {
        __le32 ctl0;
        __le32 ctl1;
        __le32 addr_low;
        __le32 addr_high;
} __packed;
Having 5-6 rings 512 entries each requires 48 KiB. There are devices
with 3 Ethernet bgmac devices and 2 wireless devices, 48 KiB * 5 gets
me dangerously close to the limit. I don't see this problem with
upstream default drivers yet, but it may become an issue at some
point.

Raspberry Pi was dealing a similar problem:
https://github.com/raspberrypi/linux/issues/170

Does it make any sense to increase default atomic_pool_size value?

I see there is init_dma_coherent_pool_size function but it seems to be
dead code (can we drop it?).
I can change that value using coherent_pool param, but is that sane to
put it in bootargs of upstream DTS files?
Any advise on this?

-- 
Rafa?

Default atomic_pool_size value & changing it

[Russell King - ARM Linux]

On Tue, Nov 15, 2016 at 10:15:51PM +0100, Rafa? Mi?ecki wrote:
> I've hit problem with allocating enough coherent DMA memory. Obviously
> there is a limit of this memory as pointed by:
> [ 0.283667] DMA: preallocated 256 KiB pool for atomic coherent allocations

This is for _atomic_ coherent allocations, not for normal coherent
allocations.  Normal coherent allocations come from the CMA memory.
Make sure you have CMA enabled, and you have enough of it.

Boot with cma=xxxMB with xxx set appropriately.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Default atomic_pool_size value & changing it

[Rafał Miłecki]

On 15 November 2016 at 22:37, Russell King - ARM Linux
<linux@armlinux.org.uk> wrote:
> On Tue, Nov 15, 2016 at 10:15:51PM +0100, Rafa? Mi?ecki wrote:
>> I've hit problem with allocating enough coherent DMA memory. Obviously
>> there is a limit of this memory as pointed by:
>> [ 0.283667] DMA: preallocated 256 KiB pool for atomic coherent allocations
>
> This is for _atomic_ coherent allocations, not for normal coherent
> allocations.  Normal coherent allocations come from the CMA memory.
> Make sure you have CMA enabled, and you have enough of it.
>
> Boot with cma=xxxMB with xxx set appropriately.

Oh, that definitely sounds like worth trying
# CONFIG_CMA is not set
thanks for pointing it to me!

-- 
Rafa?

Default atomic_pool_size value & changing it

[Rafał Miłecki]

On 15 November 2016 at 22:43, Rafa? Mi?ecki <zajec5@gmail.com> wrote:
> On 15 November 2016 at 22:37, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
>> On Tue, Nov 15, 2016 at 10:15:51PM +0100, Rafa? Mi?ecki wrote:
>>> I've hit problem with allocating enough coherent DMA memory. Obviously
>>> there is a limit of this memory as pointed by:
>>> [ 0.283667] DMA: preallocated 256 KiB pool for atomic coherent allocations
>>
>> This is for _atomic_ coherent allocations, not for normal coherent
>> allocations.  Normal coherent allocations come from the CMA memory.
>> Make sure you have CMA enabled, and you have enough of it.
>>
>> Boot with cma=xxxMB with xxx set appropriately.
>
> Oh, that definitely sounds like worth trying
> # CONFIG_CMA is not set
> thanks for pointing it to me!

For my quick testing I silly used:
cpu_addr[i] = dma_zalloc_coherent(dev, size, &dma_addr[i], GFP_ATOMIC);
which obviously shouldn't be done in non-sleep mode and shouldn't need
GFP_ATOMIC.

I got it now, thanks again.

-- 
Rafa?