Re: ath11k swiotlb buffer is full (on IMX8M with 4GiB DRAM)

[Baochen Qiang <quic_bqiang@quicinc.com>]

On 11/23/2024 8:43 AM, Tim Harvey wrote:
> On Thu, Nov 21, 2024 at 9:51 PM Baochen Qiang <quic_bqiang@quicinc.com> wrote:
>>
>>
>>
>> On 11/22/2024 5:50 AM, Tim Harvey wrote:
>>> On Tue, Nov 19, 2024 at 6:32 PM Baochen Qiang <quic_bqiang@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/20/2024 4:16 AM, Tim Harvey wrote:
>>>>> Greetings,
>>>>>
>>>>> I've got an ath11k card that is failing to init on an IMX8MM system
>>>>> with 4GB of DRAM:
>>>>> [    7.551582] ath11k_pci 0000:01:00.0: BAR 0 [mem
>>>>> 0x18000000-0x181fffff 64bit]: assigned
>>>>> [    7.551713] ath11k_pci 0000:01:00.0: enabling device (0000 -> 0002)
>>>>> [    7.552401] ath11k_pci 0000:01:00.0: MSI vectors: 16
>>>>> [    7.552440] ath11k_pci 0000:01:00.0: qcn9074 hw1.0
>>>>> [    7.887186] mhi mhi0: Loaded FW: ath11k/QCN9074/hw1.0/amss.bin,
>>>>> sha256: 5ee1b7b204541b5f99984f21d694ececaec08fbce1b520ffe6fe740b02a4afd7
>>>>> [    8.435964] ath11k_pci 0000:01:00.0: chip_id 0x0 chip_family 0x0
>>>>> board_id 0xff soc_id 0xffffffff
>>>>> [    8.435991] ath11k_pci 0000:01:00.0: fw_version 0x270206d0
>>>>> fw_build_timestamp 2022-08-04 12:48 fw_build_id
>>>>> WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
>>>>> [    8.441700] ath11k_pci 0000:01:00.0: Loaded FW:
>>>>> ath11k/QCN9074/hw1.0/board-2.bin, sha256:
>>>>> dbf0ca14aa1229eccd48f26f1026901b9718b143bd30b51b8ea67c84ba6207f1
>>>>> [    9.753764] ath11k_pci 0000:01:00.0: Loaded FW:
>>>>> ath11k/QCN9074/hw1.0/m3.bin, sha256:
>>>>> b6d957f335073a15a8de809398e1506f0200a08747eaf7189c843cf519ffc1de
>>>>> [    9.789791] ath11k_pci 0000:01:00.0: swiotlb buffer is full (sz:
>>>>> 1048583 bytes), total 32768 (slots), used 2528 (slots)
>>>>> [    9.789853] ath11k_pci 0000:01:00.0: failed to set up tcl_comp ring (0) :-12
>>>>> [    9.790238] ath11k_pci 0000:01:00.0: failed to init DP: -12
>>>>> root@noble-venice:~# cat /proc/cmdline
>>>>> console=ttymxc1,115200 earlycon=ec_imx6q,0x30890000,115200
>>>>> root=PARTUUID=5cdde84f-01 rootwait net.ifnames=0 cma=196M
>>>>>
>>>>> The IMX8MM's DRAM base is at 1GB so anything above 3GB hits the 32bit
>>>>> address boundary. If I pass in a mem=3096M the device registers just
>>>>> fine.
>>>> yeah ... that parameter makes kernel alloc memory below 32bit boundary, thus swiotlb is not necessary.
>>>
>>> Hi Baochen,
>>>
>>> Yes, that makes sense as I step through the code. On IMX8M with DRAM
>>> 3GB or less dma_capable(...) is true so swiotlb bounce buffers are not
>>> needed.
>>>
>>>>
>>>>>
>>>>> I found this to be the case with modern kernels however I found
>>>>> differing behavior with older kernels:
>>>>> - 6.6 and 6.1 the device registers with 4GB DRAM but crashes on client connect
>>>>> - 5.15 devices registers with 4GB DRAM and appears to work just fine
>>>> are you using Linus' tree or the stable tree?
>>>>
>>>
>>> For 6.6 I tested stable.
>> can you try Linus's tree ? as I know the stable tree is possible to miss some important fix.
>>
>>>
>>> This likely has something to do with commit dbd73acb22d8 ("wifi:
>>> ath11k: enable 36 bit mask for stream DMA") but it would seem to me
>>> that patch was trying to avoid the entire 32bit DMA limitation. Maybe
>>> that patch sets the ath11k device DMA mask to 36 bits but maybe the
>>> IMX8M PCI DMA is only capable of 32bits?
>> that patch is making situation better, not worse. that said, it helps to avoid swiotlb in
>> ath11k DMA, rather than to get it involved.
>>
> 
> Yes, that patch would be an improvement on systems capable of
> addressing 64bit memory but not on the IMX8M which is seemingly
> capable of only 32bit DMA over PCI.
> 
>>>
>>>>>
>>>>> Could anyone explain what is going on here? Obviously there have been
>>>>> changes at some point to start using swiotlb which I believe was all
>>>>> about avoiding 32bit DMA limitations but I'm not clear how I should be
>>>>> configuring this for IMX8MM with 4GB DRAM. Maybe my kernel IOMMU
>>>>> configuration is incorrect somehow?
>>>> there are quite some options associated with IOMMU, not sure which one might be causing this. But basically you may check:
>>>>
>>>> CONFIG_IOMMU_IOVA
>>>> CONFIG_IOMMU_API
>>>> CONFIG_IOMMU_SUPPORT
>>>> CONFIG_IOMMU_DMA=y
>>>>
>>>
>>> These are enabled which I believe appropriate for IMX8M. If I want to
>>> utilize the full 4GB DRAM on IMX then I must use IOMMU and swiotlb
>>> which would mean a performance hit due to copying mem to/from bounce
>>> buffers not to mention the fact that I can't figure out how to
>>> configure the system to avoid the 'swiotlb swiotlb buffer is full'
>>> issue.
> 
> My statement regarding needing an IOMMU above is wrong; apparently the
> IMX8M SoC's don't have an IOMMU but the fact I have it enabled in the
> kernel should be a don't-care. If I understand swiotlb correctly, if I
> did have an IOMMU then it would be used instead of swiotlb.
> 
>>>
>>> Enabling CONFIG_SWIOTLB_DYNAMIC does not help nor does increasing the
>>> number of slots - it has something to do with the number/size of DMA
>>> buffers that ath11k is asking for:
>> yeah, ath11k asks for fixed size DMA buffer regardless of that config.
>>
>>> # dmesg | grep swiotlb_tbl_map_single
>>> [    5.237731] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16384 (slots=32768/    32)
>>> [    5.247519] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16416 (slots=32768/    64)
>>> [    5.261794] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16448 (slots=32768/    96)
>>> [    5.275114] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16480 (slots=32768/   128)
>>> [    5.287757] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16512 (slots=32768/   160)
>>> [    5.299688] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16544 (slots=32768/   192)
>>> [    5.312482] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16576 (slots=32768/   224)
>>> [    5.324493] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16608 (slots=32768/   256)
>>> [    5.337001] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16640 (slots=32768/   288)
>>> [    5.346754] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16672 (slots=32768/   320)
>>> [    5.356571] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16704 (slots=32768/   352)
>>> [    5.366372] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16736 (slots=32768/   384)
>>> [    5.376164] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16768 (slots=32768/   416)
>>> [    5.385944] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16800 (slots=32768/   448)
>>> [    5.395712] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16832 (slots=32768/   480)
>>> [    5.408270] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16864 (slots=32768/   512)
>>> [    5.419768] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16896 (slots=32768/   544)
>>> [    5.430966] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16928 (slots=32768/   576)
>>> [    5.442368] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16960 (slots=32768/   608)
>>> [    5.452422] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 16992 (slots=32768/   640)
>>> [    5.463507] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17024 (slots=32768/   672)
>>> [    5.473536] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17056 (slots=32768/   704)
>>> [    5.485661] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17088 (slots=32768/   736)
>>> [    5.495404] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17120 (slots=32768/   768)
>>> [    5.509626] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17152 (slots=32768/   800)
>>> [    5.519353] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17184 (slots=32768/   832)
>>> [    5.529077] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17216 (slots=32768/   864)
>>> [    5.538799] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17248 (slots=32768/   896)
>>> [    5.548517] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17280 (slots=32768/   928)
>>> [    5.558238] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17312 (slots=32768/   960)
>>> [    5.567965] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index= 17344 (slots=32768/   992)
>>> [    5.578943] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=     0 (slots=32768/   992)
>>> [    5.578964] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8192 (slots=32768/   993)
>>> [    5.599793] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=    32 (slots=32768/   992)
>>> [    5.599861] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  68B index=  8193 (slots=32768/   993)
>>> [    5.609589] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=    64 (slots=32768/   993)
>>> [    5.628921] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=    96 (slots=32768/   992)
>>> [    5.638703] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  68B index= 17376 (slots=32768/   993)
>>> [    5.649602] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   128 (slots=32768/   992)
>>> [    5.659389] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   160 (slots=32768/   992)
>>> [    5.674038] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  96B index= 17377 (slots=32768/   993)
>>> [    5.685016] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   192 (slots=32768/   992)
>>> [    5.694819] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   224 (slots=32768/   992)
>>> [    5.694831] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index= 17378 (slots=32768/   993)
>>> [    5.714194] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  40B index= 17379 (slots=32768/   994)
>>> [    5.725089] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   256 (slots=32768/   992)
>>> [    5.753507] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17380 (slots=32768/   996)
>>> [    5.764668] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   288 (slots=32768/   992)
>>> [    5.774456] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   320 (slots=32768/   992)
>>> [    5.774620] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17384 (slots=32768/   996)
>>> [    5.795091] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   352 (slots=32768/   992)
>>> [    5.795241] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17388 (slots=32768/   996)
>>> [    5.815724] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   384 (slots=32768/   992)
>>> [    5.815884] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17392 (slots=32768/   996)
>>> [    5.836357] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   416 (slots=32768/   992)
>>> [    5.836368] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8194 (slots=32768/   993)
>>> [    5.855856] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17396 (slots=32768/   997)
>>> [    5.866818] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   448 (slots=32768/   992)
>>> [    5.866978] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17400 (slots=32768/   996)
>>> [    5.887451] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   480 (slots=32768/   992)
>>> [    5.897231] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   512 (slots=32768/   992)
>>> [    5.897389] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17404 (slots=32768/   996)
>>> [    5.917866] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   544 (slots=32768/   992)
>>> [    5.918026] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17408 (slots=32768/   996)
>>> [    5.938489] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   576 (slots=32768/   992)
>>> [    5.938642] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17412 (slots=32768/   996)
>>> [    5.959121] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   608 (slots=32768/   992)
>>> [    5.959135] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8195 (slots=32768/   993)
>>> [    5.978619] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17416 (slots=32768/   997)
>>> [    5.989588] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   640 (slots=32768/   992)
>>> [    5.989738] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17420 (slots=32768/   996)
>>> [    6.010215] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   672 (slots=32768/   992)
>>> [    6.020001] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   704 (slots=32768/   992)
>>> [    6.020158] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17424 (slots=32768/   996)
>>> [    6.040643] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   736 (slots=32768/   992)
>>> [    6.040798] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17428 (slots=32768/   996)
>>> [    6.061287] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   768 (slots=32768/   992)
>>> [    6.061437] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17432 (slots=32768/   996)
>>> [    6.081918] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   800 (slots=32768/   992)
>>> [    6.081929] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8196 (slots=32768/   993)
>>> [    6.101409] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17436 (slots=32768/   997)
>>> [    6.112375] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   832 (slots=32768/   992)
>>> [    6.112528] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17440 (slots=32768/   996)
>>> [    6.133004] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   864 (slots=32768/   992)
>>> [    6.142785] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   896 (slots=32768/   992)
>>> [    6.142949] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17444 (slots=32768/   996)
>>> [    6.163426] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   928 (slots=32768/   992)
>>> [    6.163576] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17448 (slots=32768/   996)
>>> [    6.184058] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   960 (slots=32768/   992)
>>> [    6.184208] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17452 (slots=32768/   996)
>>> [    6.204691] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=   992 (slots=32768/   992)
>>> [    6.204704] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8197 (slots=32768/   993)
>>> [    6.224183] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17456 (slots=32768/   997)
>>> [    6.235148] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1024 (slots=32768/   992)
>>> [    6.235308] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 6224B index= 17460 (slots=32768/   996)
>>> [    6.255777] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1056 (slots=32768/   992)
>>> [    6.265552] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1088 (slots=32768/   992)
>>> [    6.265633] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 2128B index= 17464 (slots=32768/   994)
>>> [    6.286142] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1120 (slots=32768/   992)
>>> [    6.286182] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  72B index= 17466 (slots=32768/   993)
>>> [    7.574489] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1152 (slots=32768/   992)
>>> [    7.584645] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  60B index= 17467 (slots=32768/   993)
>>> [    7.595593] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1184 (slots=32768/   992)
>>> [    7.595608] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  8198 (slots=32768/   993)
>>> [    7.605359] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1216 (slots=32768/   993)
>>> [    7.624703] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 452B index=  1248 (slots=32768/   993)
>>> [    7.635603] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1280 (slots=32768/   992)
>>> [    7.645344] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>>  52B index=  1312 (slots=32768/   993)
>>> [    7.656247] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1314 (slots=32768/   992)
>>> [    7.683567] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single size=
>>> 65535B index=  1346 (slots=32768/   992)
>>> [    7.696095] ath11k_pci 0000:01:00.0: swiotlb_tbl_map_single
>>> size=1048583B index=    -1 (slots=32768/   992)
>>>
>>> I'm still trying to understand the swiotlb allocation to see if there
>>> is some configuration change I should be making.
>>
>> I suspect you hit the same issue mentioned here:
>>
>> https://lore.kernel.org/all/CAOMZO5A7+nxACoBPY0k8cOpVQByZtEV_N1489MK5wETHF_RXWA@mail.gmail.com/
>>
>> so can you check if below commit present in your kernel, and if not could you pick it up
>> and try again?
>>
>> commit 14cebf689a78 ("swiotlb: Reinstate page-alignment for mappings >= PAGE_SIZE")
ignore this request since it should be no related to your issue :(

>>
> 
> I bisected the 'swiotlb buffer is full' issue back to commit
> aaf244141ed7 ("wifi: ath11k: fix IOMMU errors on buffer rings") which
> looks to me to be a legitimate fix and if I revert it swiotlb is now
> happy and the driver registers but I get the crash on client connect
> that I was seeing in 6.6 so that commit fixes an issue, but causes
> swiotlb to not be fulfilled.
not really ... that commit is not the cause to your issue. you don;t see the 'swiotlb
full' error after revert it simply because dma_map_single() is NOT called then.


> 
> The issue seems to be that the swiotlb memory buffer allocator is
> getting too fragmented to be useful with what ath11k is now asking for
> (a lot of 2K and 64K buffers and then finally a 1048583B buffer which
> fails due to the fragmentation of the swiotlb buffer.
no, the direct cause to 'swiotlb full' error is that kernel does not allow a swiotlb map
request larger than 256kb [1]:

'A single allocation from swiotlb is limited to IO_TLB_SIZE * IO_TLB_SEGSIZE bytes, which
is 256 KiB with current definitions'

while here ath11k is requesting a buffer of 1048583 bytes.


howevr the question is that why swiotlb is involved here: for streamed DMA operation
ath11k is capable of addressing 64GB memory (with 36bit DMA mask), in your case this
covers whole system memory. the most possible reason I can think of is that swiotlb is
forcebly enabled in your kernel (with swiotlb=force?) such that each DMA buffer would be
bounced by swiotlb regardless of its physical address.



[1] Documentation/core-api/swiotlb.rst

> 
> I'm guessing that this has gone unnoticed for a while because there
> are maybe not a lot of systems out there that require swiotlb with
> ath11k (either no IOMMU or more memory than DMA can address) and my
> guess is that if you test ath11k with swiotlb=force you will easily
> see this 'swiotlb buffer is full' issue on other systems.
> 
> I'm not that knowledgeable about ath11k but I do know that ath10 and
> ath12k do not have this issue with swiotlb. Debugging a bit shows that
> there are a lot of large DMA buffers being requested by ath11k and I'm
> wondering if that could be reduced or optimized somehow.
> 
>>
>>>
>>> To avoid using swiotlb is there some way to limit the memory region
>>> used for DMA operations to below 32bit boundary yet still allow the
>>> memory above 32bit to be useful in the system for userspace maybe?
>> if you are using dma_alloc_coherent() I'm afraid there is no way for that. the API
>> internally ignores any zone flags passed with the 'gfp' argument. see
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/dma/mapping.c#n615
>>
> 
> is DMA_RESTRICTED_POOL a solution for me?
i don;t think this help since this is used in coherent DMA?

> 
> Best Regards,
> 
> Tim
>