ath11k-qca crashing

ath11k-qca crashing

[André Eberle]

Hello,

I have been using the ath11k wireless driver (qca version ie. QSDK) in
a IPQ807x router, but I am facing crashes. Ethernet and WAN work fine.
I can connect to WiFi, and receive an address from DHCP. I can even
send/receive a number of packets via the wifi nic, but if I try to,
say, send/recv a larger number of packets at once (like visiting a
webpage), the kernel crashes. I have been digging into the NSS and
ath11k drivers trying to isolate the problem, but so far my debugging
attempts have failed. I have so far failed to set up kgdb on qsdk
also, so no help on that front.

What is the best way to debug the ath11k drivers for such problems? Or
at least try to determine if the crashes are from the driver, or
elsewhere in the NSS system?

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k-qca crashing

[Bhagavathi Perumal S]

On 2021-04-15 14:58, Bhagavathi Perumal S wrote:
> Hello,
> 
> I have been using the ath11k wireless driver (qca version ie. QSDK) in
> a IPQ807x router, but I am facing crashes. Ethernet and WAN work fine.
> 
> 
> I can connect to WiFi, and receive an address from DHCP. I can even
> send/receive a number of packets via the wifi nic, but if I try to,
> say, send/recv a larger number of packets at once (like visiting a
> webpage), the kernel crashes. I have been digging into the NSS and
> ath11k drivers trying to isolate the problem, but so far my debugging
> attempts have failed. I have so far failed to set up kgdb on qsdk
> also, so no help on that front.
> 
> What is the best way to debug the ath11k drivers for such problems? Or
> at least try to determine if the crashes are from the driver, or
> elsewhere in the NSS system?

Serial logs and crash dump will help to debug the kernel crashes. Please 
share dmesg logs and serial logs from power on and please share call 
stack details during crash if any.

> 
> --
> 
> ath11k mailing list
> 
> ath11k@lists.infradead.org
> 
> http://lists.infradead.org/mailman/listinfo/ath11k

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k-qca crashing

[André Eberle]

I have attached the dmesg. It is the UART dump essentially, so it
includes some pre-boot info, maybe it is useful:

https://drive.google.com/file/d/1u5_C7q6yK9qRyq4TJYQpkZeU-1eQmRwR/view?usp=sharing

There is no stack trace dump during the crash, unfortunately. As you
can see the system suddenly reboots (sometimes it just freezes).

I have added the logread from openwrt also. This one has trace-level
messages from dmesg, so maybe it is more helpful. I enabled maximum
debug level for ath11 on this one, hope it helps:

echo 0xffffffff > /sys/module/ath11k/parameters/debug_mask

https://drive.google.com/file/d/1sasD99QVf2UqF7lOQ_pD5JhI43Ie3DO6/view?usp=sharing

On Thu, Apr 15, 2021 at 6:48 AM Bhagavathi Perumal S
<bperumal@codeaurora.org> wrote:
>
> On 2021-04-15 14:58, Bhagavathi Perumal S wrote:
> > Hello,
> >
> > I have been using the ath11k wireless driver (qca version ie. QSDK) in
> > a IPQ807x router, but I am facing crashes. Ethernet and WAN work fine.
> >
> >
> > I can connect to WiFi, and receive an address from DHCP. I can even
> > send/receive a number of packets via the wifi nic, but if I try to,
> > say, send/recv a larger number of packets at once (like visiting a
> > webpage), the kernel crashes. I have been digging into the NSS and
> > ath11k drivers trying to isolate the problem, but so far my debugging
> > attempts have failed. I have so far failed to set up kgdb on qsdk
> > also, so no help on that front.
> >
> > What is the best way to debug the ath11k drivers for such problems? Or
> > at least try to determine if the crashes are from the driver, or
> > elsewhere in the NSS system?
>
> Serial logs and crash dump will help to debug the kernel crashes. Please
> share dmesg logs and serial logs from power on and please share call
> stack details during crash if any.
>
> >
> > --
> >
> > ath11k mailing list
> >
> > ath11k@lists.infradead.org
> >
> > http://lists.infradead.org/mailman/listinfo/ath11k

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k-qca crashing

[André Eberle]

After some more debugging, I figured there is some problem on TX.
Right now, adding small delays to the ath11k_nss_tx function in nss.c
is preventing the crash from happening. Could someone point me to
where race conditions/other issues could be happening on the TX stack?

On Thu, Apr 15, 2021 at 3:01 PM André Eberle <andre.eberle@gmail.com> wrote:
>
> I have attached the dmesg. It is the UART dump essentially, so it
> includes some pre-boot info, maybe it is useful:
>
> https://drive.google.com/file/d/1u5_C7q6yK9qRyq4TJYQpkZeU-1eQmRwR/view?usp=sharing
>
> There is no stack trace dump during the crash, unfortunately. As you
> can see the system suddenly reboots (sometimes it just freezes).
>
> I have added the logread from openwrt also. This one has trace-level
> messages from dmesg, so maybe it is more helpful. I enabled maximum
> debug level for ath11 on this one, hope it helps:
>
> echo 0xffffffff > /sys/module/ath11k/parameters/debug_mask
>
> https://drive.google.com/file/d/1sasD99QVf2UqF7lOQ_pD5JhI43Ie3DO6/view?usp=sharing
>
> On Thu, Apr 15, 2021 at 6:48 AM Bhagavathi Perumal S
> <bperumal@codeaurora.org> wrote:
> >
> > On 2021-04-15 14:58, Bhagavathi Perumal S wrote:
> > > Hello,
> > >
> > > I have been using the ath11k wireless driver (qca version ie. QSDK) in
> > > a IPQ807x router, but I am facing crashes. Ethernet and WAN work fine.
> > >
> > >
> > > I can connect to WiFi, and receive an address from DHCP. I can even
> > > send/receive a number of packets via the wifi nic, but if I try to,
> > > say, send/recv a larger number of packets at once (like visiting a
> > > webpage), the kernel crashes. I have been digging into the NSS and
> > > ath11k drivers trying to isolate the problem, but so far my debugging
> > > attempts have failed. I have so far failed to set up kgdb on qsdk
> > > also, so no help on that front.
> > >
> > > What is the best way to debug the ath11k drivers for such problems? Or
> > > at least try to determine if the crashes are from the driver, or
> > > elsewhere in the NSS system?
> >
> > Serial logs and crash dump will help to debug the kernel crashes. Please
> > share dmesg logs and serial logs from power on and please share call
> > stack details during crash if any.
> >
> > >
> > > --
> > >
> > > ath11k mailing list
> > >
> > > ath11k@lists.infradead.org
> > >
> > > http://lists.infradead.org/mailman/listinfo/ath11k

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k

Re: ath11k-qca crashing

[Bhagavathi Perumal S]

On 2021-04-18 00:45, André Eberle wrote:
> After some more debugging, I figured there is some problem on TX.
> Right now, adding small delays to the ath11k_nss_tx function in nss.c
> is preventing the crash from happening. Could someone point me to
> where race conditions/other issues could be happening on the TX stack?
> 

Upstream ath11k driver is not using the api ath11k_nss_tx(), I think you 
are using some private ath11k driver, can you please check whether the 
ath11k driver is completed enabled and fully tested for the platform 
"IPQ807x/AP-AC04"?

> On Thu, Apr 15, 2021 at 3:01 PM André Eberle <andre.eberle@gmail.com> 
> wrote:
>> 
>> I have attached the dmesg. It is the UART dump essentially, so it
>> includes some pre-boot info, maybe it is useful:
>> 
>> https://drive.google.com/file/d/1u5_C7q6yK9qRyq4TJYQpkZeU-1eQmRwR/view?usp=sharing
>> 
>> There is no stack trace dump during the crash, unfortunately. As you
>> can see the system suddenly reboots (sometimes it just freezes).
>> 
>> I have added the logread from openwrt also. This one has trace-level
>> messages from dmesg, so maybe it is more helpful. I enabled maximum
>> debug level for ath11 on this one, hope it helps:
>> 
>> echo 0xffffffff > /sys/module/ath11k/parameters/debug_mask
>> 
>> https://drive.google.com/file/d/1sasD99QVf2UqF7lOQ_pD5JhI43Ie3DO6/view?usp=sharing
>> 
>> On Thu, Apr 15, 2021 at 6:48 AM Bhagavathi Perumal S
>> <bperumal@codeaurora.org> wrote:
>> >
>> > On 2021-04-15 14:58, Bhagavathi Perumal S wrote:
>> > > Hello,
>> > >
>> > > I have been using the ath11k wireless driver (qca version ie. QSDK) in
>> > > a IPQ807x router, but I am facing crashes. Ethernet and WAN work fine.
>> > >
>> > >
>> > > I can connect to WiFi, and receive an address from DHCP. I can even
>> > > send/receive a number of packets via the wifi nic, but if I try to,
>> > > say, send/recv a larger number of packets at once (like visiting a
>> > > webpage), the kernel crashes. I have been digging into the NSS and
>> > > ath11k drivers trying to isolate the problem, but so far my debugging
>> > > attempts have failed. I have so far failed to set up kgdb on qsdk
>> > > also, so no help on that front.
>> > >
>> > > What is the best way to debug the ath11k drivers for such problems? Or
>> > > at least try to determine if the crashes are from the driver, or
>> > > elsewhere in the NSS system?
>> >
>> > Serial logs and crash dump will help to debug the kernel crashes. Please
>> > share dmesg logs and serial logs from power on and please share call
>> > stack details during crash if any.
>> >
>> > >
>> > > --
>> > >
>> > > ath11k mailing list
>> > >
>> > > ath11k@lists.infradead.org
>> > >
>> > > http://lists.infradead.org/mailman/listinfo/ath11k

-- 
ath11k mailing list
ath11k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath11k