ath12k driver on Debian 12 - High ping latency and MLO support query on 6.11rc7

ath12k driver on Debian 12 - High ping latency and MLO support query on 6.11rc7

[Benoit Masson]

Hello ath12k developers,

I'm testing the ath12k driver on Debian 12 with a custom 6.11rc7
kernel for an M2 QCNCM865 card. While generally stable, I'm
encountering two issues:

1. High ping latency/jitter:
- ath12k: 3.5ms to 55ms (40ms jitter)
- Intel AX211 (same setup): ~4ms LAN, ~6ms internet (stable)
- AP: TP-Link BE800 (6GHz band) - Tested with various OFDMA, MIMO, and
MLO settings

2. MLO support:
- When connected via MLO, performance mirrors 6GHz-only connection
- Doesn't appear to utilize all three bands

Questions:
1. Are there any recent patches addressing the latency issue for
kernel 6.11rc7?
2. Is MLO fully supported by Linux/Qualcomm for this hardware? I'm
willing to test any proposed solutions or gather additional data if
needed.

Thank you for your ongoing work on the driver. Best regards,
Benoit



lshw output:
           *-network
                description: Ethernet interface
                product: Qualcomm Technologies, Inc
                vendor: Qualcomm Technologies, Inc
                physical id: 0
                bus info: pci@0000:02:00.0
                logical name: wlp2s0
                version: 01
                serial: 4c:82:a9:XX:YY:ZZ
                width: 64 bits
                clock: 33MHz
                capabilities: bus_master cap_list ethernet physical
                configuration: broadcast=yes driver=ath12k_pci
driverversion=6.11.0-rc7 firmware=N/A ip=192.168.1.169 latency=0
link=yes multicast=yes
                resources: irq:97 memory:dc600000-dc7fffff

lspci output
02:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi-Fi
7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
Subsystem: Foxconn International, Inc. WCN785x Wi-Fi 7(802.11be)
320MHz 2x2 [FastConnect 7800]
Flags: bus master, fast devsel, latency 0, IRQ 97, IOMMU group 14
Memory at dc600000 (64-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=16/32 Maskable+ 64bit-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [158] Transaction Processing Hints
Capabilities: [1e4] Latency Tolerance Reporting
Capabilities: [1ec] L1 PM Substates
Kernel driver in use: ath12k_pci
Kernel modules: ath12k

Re: ath12k driver on Debian 12 - High ping latency and MLO support query on 6.11rc7

[Kalle Valo]

Benoit Masson <benoitm@perenite.com> writes:

> Hello ath12k developers,
>
> I'm testing the ath12k driver on Debian 12 with a custom 6.11rc7
> kernel for an M2 QCNCM865 card.

You didn't provide any dmesg out but I assume this has a WCN7850
chipset. Please provide output from 'dmesg | grep ath12k' so that we
know what firmware etc you are using.

> While generally stable, I'm encountering two issues:
>
> 1. High ping latency/jitter:
> - ath12k: 3.5ms to 55ms (40ms jitter)

Is this from network to ath12k or from ath12k to network? I recommend
testing both.

Usually Power Save Mode causes latency, have you tried disabling that?

> 2. MLO support:
> - When connected via MLO, performance mirrors 6GHz-only connection
> - Doesn't appear to utilize all three bands
>
> Questions:
> 1. Are there any recent patches addressing the latency issue for
> kernel 6.11rc7?

At least I can't think of any fix right now.

> 2. Is MLO fully supported by Linux/Qualcomm for this hardware? I'm
> willing to test any proposed solutions or gather additional data if
> needed.

MLO is not yet supported. Once there is something to test we will
announce it in this ath12k list.

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Fwd: ath12k driver on Debian 12 - High ping latency and MLO support query on 6.11rc7

[Benoit Masson]

Hi Kalle thanks a million for answering and taking the time!

On Mon, Sep 16, 2024 at 10:28 AM Kalle Valo <kvalo@kernel.org> wrote:
>
> Benoit Masson <benoitm@perenite.com> writes:
>
> > Hello ath12k developers,
> >
> > I'm testing the ath12k driver on Debian 12 with a custom 6.11rc7
> > kernel for an M2 QCNCM865 card.
>
> You didn't provide any dmesg out but I assume this has a WCN7850
> chipset. Please provide output from 'dmesg | grep ath12k' so that we
> know what firmware etc you are using.

Sure here is the dmesg message:
dmesg | grep ath12k
[   49.185903] ath12k_pci 0000:02:00.0: BAR 0 [mem
0xdc600000-0xdc7fffff 64bit]: assigned
[   49.185920] ath12k_pci 0000:02:00.0: enabling device (0000 -> 0002)
[   49.186714] ath12k_pci 0000:02:00.0: MSI vectors: 16
[   49.186718] ath12k_pci 0000:02:00.0: Hardware name: wcn7850 hw2.0
[   49.747963] ath12k_pci 0000:02:00.0: chip_id 0x2 chip_family 0x4
board_id 0xff soc_id 0x40170200
[   49.747977] ath12k_pci 0000:02:00.0: fw_version 0x100301e1
fw_build_timestamp 2023-12-06 04:05 fw_build_id
QC_IMAGE_VERSION_STRING=WLAN.HMT.1.0.c5-00481-QCAHMTSWPL_V1.0_V2.0_SILICONZ-3
[   50.188095] ath12k_pci 0000:02:00.0 wlp2s0: renamed from wlan0
[16865.398653] ath12k_pci 0000:02:00.0: chip_id 0x2 chip_family 0x4
board_id 0xff soc_id 0x40170200
[16865.398658] ath12k_pci 0000:02:00.0: fw_version 0x100301e1
fw_build_timestamp 2023-12-06 04:05 fw_build_id
QC_IMAGE_VERSION_STRING=WLAN.HMT.1.0.c5-00481-QCAHMTSWPL_V1.0_V2.0_SILICONZ-3
[20300.513950] ath12k_pci 0000:02:00.0: failed to send
HAL_REO_CMD_FLUSH_CACHE cmd, tid 15 (-105)
[20300.513965] ath12k_pci 0000:02:00.0: failed to send
HAL_REO_CMD_FLUSH_CACHE cmd, tid 16 (-105)
[28269.690504] ath12k_pci 0000:02:00.0: Spurious quick kickout for STA
3a:23:51:73:23:26
[28278.156974] ath12k_pci 0000:02:00.0: failed to send
HAL_REO_CMD_FLUSH_CACHE, tid 15 (-105)
[29715.843960] ath12k_pci 0000:02:00.0: failed to send
HAL_REO_CMD_FLUSH_CACHE, tid 15 (-105)
[29715.843974] ath12k_pci 0000:02:00.0: failed to send
HAL_REO_CMD_FLUSH_CACHE, tid 15 (-105)
[45013.847353] ath12k_pci 0000:02:00.0: chip_id 0x2 chip_family 0x4
board_id 0xff soc_id 0x40170200
[45013.847355] ath12k_pci 0000:02:00.0: fw_version 0x100301e1
fw_build_timestamp 2023-12-06 04:05 fw_build_id
QC_IMAGE_VERSION_STRING=WLAN.HMT.1.0.c5-00481-QCAHMTSWPL_V1.0_V2.0_SILICONZ-3

I also grep the interface name in case and only found this error:
# dmesg | grep wlp2s0
[27660.154844] wlp2s0: AP bug: VHT capa missing from AssocResp
[27660.154845] wlp2s0: AP bug: VHT operation missing from AssocResp


> > While generally stable, I'm encountering two issues:
> >
> > 1. High ping latency/jitter:
> > - ath12k: 3.5ms to 55ms (40ms jitter)
>
> Is this from network to ath12k or from ath12k to network? I recommend
> testing both.

Below the test both way;
1) From station to access point:
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=13.7 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=15.5 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=3.70 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=12.2 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=23.4 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=12.8 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=13.2 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=12.0 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=11.9 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=23.3 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=12.9 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=12.2 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=6.48 ms

2) from access point to station:
ping 192.168.1.169
PING 192.168.1.169 (192.168.1.169): 56 data bytes
64 bytes from 192.168.1.169: seq=0 ttl=64 time=5.582 ms
64 bytes from 192.168.1.169: seq=1 ttl=64 time=4.469 ms
64 bytes from 192.168.1.169: seq=2 ttl=64 time=357.588 ms
64 bytes from 192.168.1.169: seq=3 ttl=64 time=398.582 ms
64 bytes from 192.168.1.169: seq=4 ttl=64 time=419.028 ms
64 bytes from 192.168.1.169: seq=5 ttl=64 time=449.935 ms
64 bytes from 192.168.1.169: seq=6 ttl=64 time=21.677 ms
64 bytes from 192.168.1.169: seq=7 ttl=64 time=3.836 ms
64 bytes from 192.168.1.169: seq=8 ttl=64 time=370.563 ms
64 bytes from 192.168.1.169: seq=9 ttl=64 time=17.126 ms
64 bytes from 192.168.1.169: seq=10 ttl=64 time=3.833 ms
64 bytes from 192.168.1.169: seq=11 ttl=64 time=58.386 ms
64 bytes from 192.168.1.169: seq=12 ttl=64 time=81.823 ms
64 bytes from 192.168.1.169: seq=13 ttl=64 time=107.806 ms
64 bytes from 192.168.1.169: seq=14 ttl=64 time=170.339 ms
64 bytes from 192.168.1.169: seq=15 ttl=64 time=52.075 ms
64 bytes from 192.168.1.169: seq=16 ttl=64 time=7.189 ms

> Usually Power Save Mode causes latency, have you tried disabling that?

Indeed after disabling power_save with command below
# iw dev wlp2s0 set power_save off
# iw dev wlp2s0 get power_save
Power save: off

then new ping are much better:
3) From station to access point power_save off:
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=4.25 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=4.08 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=4.65 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=5.37 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=9.99 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=4.43 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=3.33 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=4.79 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=37.9 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=14.1 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=3.83 ms

4) from access point to station power_save off:
ping 192.168.1.169
PING 192.168.1.169 (192.168.1.169): 56 data bytes
64 bytes from 192.168.1.169: seq=0 ttl=64 time=16.917 ms
64 bytes from 192.168.1.169: seq=1 ttl=64 time=5.478 ms
64 bytes from 192.168.1.169: seq=2 ttl=64 time=3.910 ms
64 bytes from 192.168.1.169: seq=3 ttl=64 time=3.938 ms
64 bytes from 192.168.1.169: seq=4 ttl=64 time=8.326 ms
64 bytes from 192.168.1.169: seq=5 ttl=64 time=3.745 ms
64 bytes from 192.168.1.169: seq=6 ttl=64 time=3.909 ms
64 bytes from 192.168.1.169: seq=7 ttl=64 time=8.720 ms
64 bytes from 192.168.1.169: seq=8 ttl=64 time=11.626 ms
64 bytes from 192.168.1.169: seq=9 ttl=64 time=6.859 ms
64 bytes from 192.168.1.169: seq=10 ttl=64 time=4.573 ms
64 bytes from 192.168.1.169: seq=11 ttl=64 time=3.864 ms
64 bytes from 192.168.1.169: seq=12 ttl=64 time=3.757 ms

Which made me look at Tp link access point settings and remember there
is a 'TWT' option described as : "Target Wake Time allows 802.11ax
routers and clients to negotiate their periods to transmit and receive
data packets. Clients only wake up at TWT sessions and remain in sleep
mode for the rest of the time, which significantly extend their
battery life."

Disabling it seems to have a big impact on ping latency even with
power_save is on, in the end disabling TWT and power_save give very
nice ping both ways:

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=2.88 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=3.42 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=3.46 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=2.99 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=3.59 ms
PING 192.168.1.169 (192.168.1.169): 56 data bytes
64 bytes from 192.168.1.169: seq=99 ttl=64 time=4.957 ms
64 bytes from 192.168.1.169: seq=100 ttl=64 time=3.602 ms
64 bytes from 192.168.1.169: seq=101 ttl=64 time=2.924 ms
64 bytes from 192.168.1.169: seq=102 ttl=64 time=3.093 ms
64 bytes from 192.168.1.169: seq=103 ttl=64 time=3.767 ms

Thanks a million for having me to look in this direction ! Any thoughs
on power_saving settings for ath12k and best way to make those value
kept after boot if any in firmware or config of the driver or module
parameters ? Indeed that would be great to have TWT activated for
phone and tablet battery saving but ath12k on desktop to ignore it.


> > 2. MLO support:
> > - When connected via MLO, performance mirrors 6GHz-only connection
> > - Doesn't appear to utilize all three bands
> >
> > Questions:
> > 1. Are there any recent patches addressing the latency issue for
> > kernel 6.11rc7?
>
> At least I can't think of any fix right now.

Thanks for lettings me know, any though about the firmware binary are
there beta firmware somewhere that would make sense to test or is the
linux-firmware git main branch the more recent one ?

> > 2. Is MLO fully supported by Linux/Qualcomm for this hardware? I'm
> > willing to test any proposed solutions or gather additional data if
> > needed.
>
> MLO is not yet supported. Once there is something to test we will
> announce it in this ath12k list.

Perfect I'll be reading the mailing list, if you need tester able to
compile a kernel and module don't hesitate. or need tester with wifi7
access point (I only own TP-Link BE800 but is it in beta firmware mode
with terminal access so I can also do some test the device)

Have a great day!
Benoit
> --
> https://patchwork.kernel.org/project/linux-wireless/list/
>
> https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: Fwd: ath12k driver on Debian 12 - High ping latency and MLO support query on 6.11rc7

[Kalle Valo]

Benoit Masson <benoitm@perenite.com> writes:

> Hi Kalle thanks a million for answering and taking the time!
>
> On Mon, Sep 16, 2024 at 10:28 AM Kalle Valo <kvalo@kernel.org> wrote:
>>
>> > While generally stable, I'm encountering two issues:
>> >
>> > 1. High ping latency/jitter:
>> > - ath12k: 3.5ms to 55ms (40ms jitter)
>>
>> Is this from network to ath12k or from ath12k to network? I recommend
>> testing both.
>
> Below the test both way;
> 1) From station to access point:
> ping 192.168.1.1
> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=13.7 ms
> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=15.5 ms
> 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=3.70 ms
> 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=12.2 ms
> 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=23.4 ms
> 64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=12.8 ms
> 64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=13.2 ms
> 64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=12.0 ms
> 64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=11.9 ms
> 64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=23.3 ms
> 64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=12.9 ms
> 64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=12.2 ms
> 64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=6.48 ms
>
> 2) from access point to station:
> ping 192.168.1.169
> PING 192.168.1.169 (192.168.1.169): 56 data bytes
> 64 bytes from 192.168.1.169: seq=0 ttl=64 time=5.582 ms
> 64 bytes from 192.168.1.169: seq=1 ttl=64 time=4.469 ms
> 64 bytes from 192.168.1.169: seq=2 ttl=64 time=357.588 ms
> 64 bytes from 192.168.1.169: seq=3 ttl=64 time=398.582 ms
> 64 bytes from 192.168.1.169: seq=4 ttl=64 time=419.028 ms
> 64 bytes from 192.168.1.169: seq=5 ttl=64 time=449.935 ms
> 64 bytes from 192.168.1.169: seq=6 ttl=64 time=21.677 ms
> 64 bytes from 192.168.1.169: seq=7 ttl=64 time=3.836 ms
> 64 bytes from 192.168.1.169: seq=8 ttl=64 time=370.563 ms
> 64 bytes from 192.168.1.169: seq=9 ttl=64 time=17.126 ms
> 64 bytes from 192.168.1.169: seq=10 ttl=64 time=3.833 ms
> 64 bytes from 192.168.1.169: seq=11 ttl=64 time=58.386 ms
> 64 bytes from 192.168.1.169: seq=12 ttl=64 time=81.823 ms
> 64 bytes from 192.168.1.169: seq=13 ttl=64 time=107.806 ms
> 64 bytes from 192.168.1.169: seq=14 ttl=64 time=170.339 ms
> 64 bytes from 192.168.1.169: seq=15 ttl=64 time=52.075 ms
> 64 bytes from 192.168.1.169: seq=16 ttl=64 time=7.189 ms

Yeah, this look like delay caused by IEEE 802.11 Power Save Mode. But of
course this is just a guess still, not a full analysis :)

> Thanks a million for having me to look in this direction ! Any thoughs
> on power_saving settings for ath12k and best way to make those value
> kept after boot if any in firmware or config of the driver or module
> parameters ? Indeed that would be great to have TWT activated for
> phone and tablet battery saving but ath12k on desktop to ignore it.

Do you use Network Manager? I don't use it but there seems to be a
wifi.powersave setting:

https://askubuntu.com/questions/1386217/wifi-power-management-keeps-turning-on

>> > 2. MLO support:
>> > - When connected via MLO, performance mirrors 6GHz-only connection
>> > - Doesn't appear to utilize all three bands
>> >
>> > Questions:
>> > 1. Are there any recent patches addressing the latency issue for
>> > kernel 6.11rc7?
>>
>> At least I can't think of any fix right now.
>
> Thanks for lettings me know, any though about the firmware binary are
> there beta firmware somewhere that would make sense to test or is the
> linux-firmware git main branch the more recent one ?

If you are feeling brave, the latest firmware images are here:

https://git.codelinaro.org/clo/ath-firmware/ath12k-firmware

And the latest driver is here:

https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/

You can update the firmware and driver independently, there should not
be any dependencies.

>> > 2. Is MLO fully supported by Linux/Qualcomm for this hardware? I'm
>> > willing to test any proposed solutions or gather additional data if
>> > needed.
>>
>> MLO is not yet supported. Once there is something to test we will
>> announce it in this ath12k list.
>
> Perfect I'll be reading the mailing list, if you need tester able to
> compile a kernel and module don't hesitate. or need tester with wifi7
> access point (I only own TP-Link BE800 but is it in beta firmware mode
> with terminal access so I can also do some test the device)

Great, thanks. We will be sending an announcement once there's something
to test, but we are not there yet (even though we just created
ath12k-mlo branch).

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches