bizarre network problem on kernel 6.6.4

bizarre network problem on kernel 6.6.4

[Fourhundred Thecat]

Hello,

I have the most bizarre network problem on my laptop that I have ever
seen, and I don't even know where to start:

This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci

Previously I was using kernel 6.6.1 and wireless network was working fine.

Now I compiled kernel 6.6.4 with identical config, and when I boot it,
wireless network does not work:

- WPA supplicant authenticated and associated successfully
- ifconfig shows interface wlan0 is configured properly and has IP
address 10.1.4.8
- route is configured properly:

  default via 10.1.4.1   wlan0 onlink
  10.1.4.0/24            wlan0 proto kernel src 10.1.4.8

but I cannot ping even my Gataway: 10.1.4.1

bizarrely, when I start tcpdump on my gateway, I see arp requests coming
from my laptop asking for some strange public ips, such as 208.163.36.0

when I interrupt ping, and start again, then arp asks for other strange IPs.

But in every case, I see only these arp requests, not the pings coming
from my laptop.

But when I ping the other way (from gateway to my laptop), I do see the
incoming pings on my laptop (but no response pings)

Also, with "arp-scan -l" on my laptop, I see other devices on my
wireless subnet:

  10.1.4.1    00:0d:b9:53:ea:3c    PC Engines GmbH
  10.1.4.2    b0:7f:b9:50:5b:21    NETGEAR

any idea what is going on?

Re: bizarre network problem on kernel 6.6.4

[Kalle Valo]

Fourhundred Thecat <400thecat@ik.me> writes:

> I have the most bizarre network problem on my laptop that I have ever
> seen, and I don't even know where to start:
>
> This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci
>
> Previously I was using kernel 6.6.1 and wireless network was working fine.
>
> Now I compiled kernel 6.6.4 with identical config, and when I boot it,
> wireless network does not work:

Did you verify that booting back to 6.6.1 fixes the problem? That would
rule out that this isn't due to some external reasons (user space or
network)

If you can verify that 6.6.1 always works and 6.6.4 always fails I would
just try to git bisect the issue. It shouldn't take too long.

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: bizarre network problem on kernel 6.6.4

[Fourhundred Thecat]

On 05/12/2023 13.11, Kalle Valo wrote:
> Fourhundred Thecat <400thecat@ik.me> writes:
> 
>> I have the most bizarre network problem on my laptop that I have ever
>> seen, and I don't even know where to start:
>>
>> This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci
>>
>> Previously I was using kernel 6.6.1 and wireless network was working fine.
>>
>> Now I compiled kernel 6.6.4 with identical config, and when I boot it,
>> wireless network does not work:
> 
> Did you verify that booting back to 6.6.1 fixes the problem? That would
> rule out that this isn't due to some external reasons (user space or
> network)
> 
> If you can verify that 6.6.1 always works and 6.6.4 always fails I would
> just try to git bisect the issue. It shouldn't take too long.

yes. I verified multiple times that 6.6.1 works, and 6.6.4 has this problem.

I am not really familiar with git bisect, and I am also not a kernel
developer.

But I tried checking out linux kernel git repository, switched to tag
6.6. and looked inside drivers/net/wireless/ath/ath11k trying to spot
any change. But git log shows last change on 2023-08-30, whereas kernel
6.6.4 was released few days ago

Re: bizarre network problem on kernel 6.6.4

[Kalle Valo]

Fourhundred Thecat <400thecat@ik.me> writes:

> On 05/12/2023 13.11, Kalle Valo wrote:
>> Fourhundred Thecat <400thecat@ik.me> writes:
>> 
>>> I have the most bizarre network problem on my laptop that I have ever
>>> seen, and I don't even know where to start:
>>>
>>> This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci
>>>
>>> Previously I was using kernel 6.6.1 and wireless network was working fine.
>>>
>>> Now I compiled kernel 6.6.4 with identical config, and when I boot it,
>>> wireless network does not work:
>> 
>> Did you verify that booting back to 6.6.1 fixes the problem? That would
>> rule out that this isn't due to some external reasons (user space or
>> network)
>> 
>> If you can verify that 6.6.1 always works and 6.6.4 always fails I would
>> just try to git bisect the issue. It shouldn't take too long.
>
> yes. I verified multiple times that 6.6.1 works, and 6.6.4 has this problem.
>
> I am not really familiar with git bisect, and I am also not a kernel
> developer.

git bisect is a very useful tool, I recommend learning it as it makes so
much easier to find the culprit for regressions. You don't have to be a
kernel developer, it's enough that you know how to compile and install a
kernel.

Here's one doc I found with google, hopefully it helps:

https://wiki.gentoo.org/wiki/Kernel_git-bisect

Here's one from kernel documentation:

https://docs.kernel.org/admin-guide/bug-bisect.html

> But I tried checking out linux kernel git repository, switched to tag
> 6.6. and looked inside drivers/net/wireless/ath/ath11k trying to spot
> any change. But git log shows last change on 2023-08-30, whereas kernel
> 6.6.4 was released few days ago

I assume you were looking at Torvalds' tree only. The release 6.6.4
means that you are using a stable release which are in linux-stable
tree, the gentoo doc above should help more.

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: bizarre network problem on kernel 6.6.4

[Jeff Johnson]

On 12/5/2023 9:04 AM, Kalle Valo wrote:
> Fourhundred Thecat <400thecat@ik.me> writes:
> 
>> On 05/12/2023 13.11, Kalle Valo wrote:
>>> Fourhundred Thecat <400thecat@ik.me> writes:
>>>
>>>> I have the most bizarre network problem on my laptop that I have ever
>>>> seen, and I don't even know where to start:
>>>>
>>>> This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci
>>>>
>>>> Previously I was using kernel 6.6.1 and wireless network was working fine.
>>>>
>>>> Now I compiled kernel 6.6.4 with identical config, and when I boot it,
>>>> wireless network does not work:
>>>
>>> Did you verify that booting back to 6.6.1 fixes the problem? That would
>>> rule out that this isn't due to some external reasons (user space or
>>> network)
>>>
>>> If you can verify that 6.6.1 always works and 6.6.4 always fails I would
>>> just try to git bisect the issue. It shouldn't take too long.
>>
>> yes. I verified multiple times that 6.6.1 works, and 6.6.4 has this problem.
>>
>> I am not really familiar with git bisect, and I am also not a kernel
>> developer.
> 
> git bisect is a very useful tool, I recommend learning it as it makes so
> much easier to find the culprit for regressions. You don't have to be a
> kernel developer, it's enough that you know how to compile and install a
> kernel.
> 
> Here's one doc I found with google, hopefully it helps:
> 
> https://wiki.gentoo.org/wiki/Kernel_git-bisect
> 
> Here's one from kernel documentation:
> 
> https://docs.kernel.org/admin-guide/bug-bisect.html
> 
>> But I tried checking out linux kernel git repository, switched to tag
>> 6.6. and looked inside drivers/net/wireless/ath/ath11k trying to spot
>> any change. But git log shows last change on 2023-08-30, whereas kernel
>> 6.6.4 was released few days ago
> 
> I assume you were looking at Torvalds' tree only. The release 6.6.4
> means that you are using a stable release which are in linux-stable
> tree, the gentoo doc above should help more.

FWIW it seems that the only changes to ath11k have been trivial:

% git log --oneline v6.6.1..v6.6.4 -- drivers/net/wireless/ath/ath11k/
e83246ecd3b1 wifi: ath11k: fix gtk offload status event locking
69cede2a5a5f wifi: ath11k: fix htt pktlog locking
21ebb0aba580 wifi: ath11k: fix dfs radar event locking
c5b914528e55 wifi: ath11k: fix temperature event locking
2d88afdac295 wifi: ath11k: fix Tx power value during active CAC
d412d0ef300f wifi: ath11k: fix boot failure with one MSI vector
% git remote -v
origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
(fetch)
origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
(push)

Suggest you follow the instructions at the following link and create a
kernel bug report:
https://wireless.wiki.kernel.org/en/users/drivers/ath11k/bugreport

Meanwhile I'll build and load 6.6.4 on my laptop.

/jeff

Re: bizarre network problem on kernel 6.6.4

[Fourhundred Thecat]

On 06/12/2023 20.15, Jeff Johnson wrote:
> On 12/5/2023 9:04 AM, Kalle Valo wrote:
>> Fourhundred Thecat <400thecat@ik.me> writes:
>>
>>> On 05/12/2023 13.11, Kalle Valo wrote:
>>>> Fourhundred Thecat <400thecat@ik.me> writes:
>>>>
>>>>> I have the most bizarre network problem on my laptop that I have ever
>>>>> seen, and I don't even know where to start:
>>>>>
>>>>> This is lenovo T14s Gen 4, using qualcomm driver ath11k_pci
>>>>>
>>>>> Previously I was using kernel 6.6.1 and wireless network was working fine.
>>>>>
>>>>> Now I compiled kernel 6.6.4 with identical config, and when I boot it,
>>>>> wireless network does not work:
>>>>
>>>> Did you verify that booting back to 6.6.1 fixes the problem? That would
>>>> rule out that this isn't due to some external reasons (user space or
>>>> network)
>>>>
>>>> If you can verify that 6.6.1 always works and 6.6.4 always fails I would
>>>> just try to git bisect the issue. It shouldn't take too long.
>>>
>>> yes. I verified multiple times that 6.6.1 works, and 6.6.4 has this problem.
>>>
>>> I am not really familiar with git bisect, and I am also not a kernel
>>> developer.
>>
>> git bisect is a very useful tool, I recommend learning it as it makes so
>> much easier to find the culprit for regressions. You don't have to be a
>> kernel developer, it's enough that you know how to compile and install a
>> kernel.
>>
>> Here's one doc I found with google, hopefully it helps:
>>
>> https://wiki.gentoo.org/wiki/Kernel_git-bisect
>>
>> Here's one from kernel documentation:
>>
>> https://docs.kernel.org/admin-guide/bug-bisect.html
>>
>>> But I tried checking out linux kernel git repository, switched to tag
>>> 6.6. and looked inside drivers/net/wireless/ath/ath11k trying to spot
>>> any change. But git log shows last change on 2023-08-30, whereas kernel
>>> 6.6.4 was released few days ago
>>
>> I assume you were looking at Torvalds' tree only. The release 6.6.4
>> means that you are using a stable release which are in linux-stable
>> tree, the gentoo doc above should help more.
> 
> FWIW it seems that the only changes to ath11k have been trivial:
> 
> % git log --oneline v6.6.1..v6.6.4 -- drivers/net/wireless/ath/ath11k/
> e83246ecd3b1 wifi: ath11k: fix gtk offload status event locking
> 69cede2a5a5f wifi: ath11k: fix htt pktlog locking
> 21ebb0aba580 wifi: ath11k: fix dfs radar event locking
> c5b914528e55 wifi: ath11k: fix temperature event locking
> 2d88afdac295 wifi: ath11k: fix Tx power value during active CAC
> d412d0ef300f wifi: ath11k: fix boot failure with one MSI vector
> % git remote -v
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> (fetch)
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> (push)
> 
> Suggest you follow the instructions at the following link and create a
> kernel bug report:
> https://wireless.wiki.kernel.org/en/users/drivers/ath11k/bugreport
> 
> Meanwhile I'll build and load 6.6.4 on my laptop.

I have just tried new kernel 6.6.5 and everything works fine !!!

so whatever was the cause, seems to have been fixed between 6.6.4 and 6.6.5

any idea what was the issue?

Re: bizarre network problem on kernel 6.6.4

[Kalle Valo]

Fourhundred Thecat <400thecat@ik.me> writes:

>>>> But I tried checking out linux kernel git repository, switched to tag
>>>> 6.6. and looked inside drivers/net/wireless/ath/ath11k trying to spot
>>>> any change. But git log shows last change on 2023-08-30, whereas kernel
>>>> 6.6.4 was released few days ago
>>>
>>> I assume you were looking at Torvalds' tree only. The release 6.6.4
>>> means that you are using a stable release which are in linux-stable
>>> tree, the gentoo doc above should help more.
>> 
>> FWIW it seems that the only changes to ath11k have been trivial:
>> 
>> % git log --oneline v6.6.1..v6.6.4 -- drivers/net/wireless/ath/ath11k/
>> e83246ecd3b1 wifi: ath11k: fix gtk offload status event locking
>> 69cede2a5a5f wifi: ath11k: fix htt pktlog locking
>> 21ebb0aba580 wifi: ath11k: fix dfs radar event locking
>> c5b914528e55 wifi: ath11k: fix temperature event locking
>> 2d88afdac295 wifi: ath11k: fix Tx power value during active CAC
>> d412d0ef300f wifi: ath11k: fix boot failure with one MSI vector
>> % git remote -v
>> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> (fetch)
>> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> (push)
>> 
>> Suggest you follow the instructions at the following link and create a
>> kernel bug report:
>> https://wireless.wiki.kernel.org/en/users/drivers/ath11k/bugreport
>> 
>> Meanwhile I'll build and load 6.6.4 on my laptop.
>
> I have just tried new kernel 6.6.5 and everything works fine !!!
>
> so whatever was the cause, seems to have been fixed between 6.6.4 and 6.6.5
>
> any idea what was the issue?

Heh, for once something which is easy to answer :) The only change in
6.6.5 was this commit:

Revert "wifi: cfg80211: fix CQM for non-range use"

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.6.y&id=9099d0682462e8664c8cbbe8989de65e50f62b17

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches