Unable to read firmware registers on crash?

Unable to read firmware registers on crash?

[Ben Greear]

I am trying to debug a case where firmware occasionally crashes and
the driver cannot read any crash dump to debug problem further.

Any idea what might be the problem and how I might could read info
from the firmware (or hack firmware to deliver the crash info in some
other means...maybe though a previously reserved piece of memory on
the host??)

[147334.397148] ath10k: firmware crashed! (uuid ff405224-b2a4-493d-b619-19ad8152d190)
[147334.404808] ath10k: hardware name qca988x hw2.0 version 0x4100016c
[147334.411111] ath10k: firmware version: 10.1.467-ct-community-full-013
[147334.429603] ath10k: failed to read diag value at 0x1300804: -16
[147334.435647] ath10k: failed to read FW dump area address: -16


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Michal Kazior]

On 1 February 2015 at 18:46, Ben Greear <greearb@candelatech.com> wrote:
> I am trying to debug a case where firmware occasionally crashes and
> the driver cannot read any crash dump to debug problem further.
>
> Any idea what might be the problem and how I might could read info
> from the firmware (or hack firmware to deliver the crash info in some
> other means...maybe though a previously reserved piece of memory on
> the host??)
>
> [147334.397148] ath10k: firmware crashed! (uuid
> ff405224-b2a4-493d-b619-19ad8152d190)
> [147334.404808] ath10k: hardware name qca988x hw2.0 version 0x4100016c
> [147334.411111] ath10k: firmware version: 10.1.467-ct-community-full-013
> [147334.429603] ath10k: failed to read diag value at 0x1300804: -16
> [147334.435647] ath10k: failed to read FW dump area address: -16

I see this rarely. Mostly when the device goes bonkers at which point
warm reset doesn't work anymore and I'm forced to either risk cold
reset causing platform lock up or re-inject the card in the express
card slot.

I haven't played with this so I don't know if DMA is still possible
(it might not). MMIO should be operational and there should be some
scratch registers chilling around... ;-)


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Ben Greear]

On 02/02/2015 04:11 AM, Michal Kazior wrote:
> On 1 February 2015 at 18:46, Ben Greear <greearb@candelatech.com> wrote:
>> I am trying to debug a case where firmware occasionally crashes and
>> the driver cannot read any crash dump to debug problem further.
>>
>> Any idea what might be the problem and how I might could read info
>> from the firmware (or hack firmware to deliver the crash info in some
>> other means...maybe though a previously reserved piece of memory on
>> the host??)
>>
>> [147334.397148] ath10k: firmware crashed! (uuid
>> ff405224-b2a4-493d-b619-19ad8152d190)
>> [147334.404808] ath10k: hardware name qca988x hw2.0 version 0x4100016c
>> [147334.411111] ath10k: firmware version: 10.1.467-ct-community-full-013
>> [147334.429603] ath10k: failed to read diag value at 0x1300804: -16
>> [147334.435647] ath10k: failed to read FW dump area address: -16
> 
> I see this rarely. Mostly when the device goes bonkers at which point
> warm reset doesn't work anymore and I'm forced to either risk cold
> reset causing platform lock up or re-inject the card in the express
> card slot.
> 
> I haven't played with this so I don't know if DMA is still possible
> (it might not). MMIO should be operational and there should be some
> scratch registers chilling around... ;-)

To normally read the crash dump, we do this over a CE pipe?

So, if IRQ handlers are thoroughly busted on the target, then
that could be reason why we cannot read the crash dump?

If I were to use MMIO, what sort of things could I get
access to?  Just registers you think?  I might could tweak
the firmware assert routine to scribble the crash register
contents into a specific place, perhaps a bit at a time
so that the host could read it if normal crash dump read
fails?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Michal Kazior]

On 2 February 2015 at 18:03, Ben Greear <greearb@candelatech.com> wrote:
> On 02/02/2015 04:11 AM, Michal Kazior wrote:
>> On 1 February 2015 at 18:46, Ben Greear <greearb@candelatech.com> wrote:
>>> I am trying to debug a case where firmware occasionally crashes and
>>> the driver cannot read any crash dump to debug problem further.
>>>
>>> Any idea what might be the problem and how I might could read info
>>> from the firmware (or hack firmware to deliver the crash info in some
>>> other means...maybe though a previously reserved piece of memory on
>>> the host??)
>>>
>>> [147334.397148] ath10k: firmware crashed! (uuid
>>> ff405224-b2a4-493d-b619-19ad8152d190)
>>> [147334.404808] ath10k: hardware name qca988x hw2.0 version 0x4100016c
>>> [147334.411111] ath10k: firmware version: 10.1.467-ct-community-full-013
>>> [147334.429603] ath10k: failed to read diag value at 0x1300804: -16
>>> [147334.435647] ath10k: failed to read FW dump area address: -16
>>
>> I see this rarely. Mostly when the device goes bonkers at which point
>> warm reset doesn't work anymore and I'm forced to either risk cold
>> reset causing platform lock up or re-inject the card in the express
>> card slot.
>>
>> I haven't played with this so I don't know if DMA is still possible
>> (it might not). MMIO should be operational and there should be some
>> scratch registers chilling around... ;-)
>
> To normally read the crash dump, we do this over a CE pipe?

Correct. The CE7, so called diagnostic window, is used.


> So, if IRQ handlers are thoroughly busted on the target, then
> that could be reason why we cannot read the crash dump?

Incorrect. From what I understand the diagnostic window CE has
built-in logic for fetching target RAM memory chunks as per each
request. This means even if target program has masked IRQs and is
running in a while (1) {} host is still able to access its memory.

If you're unable to read the dump it means that CE has crashed
(whatever that _actually_ means) and thus the built-in logic for
diagnostic windows goes down as well. I never got around to recover CE
crash alone without resorting to the risky target cold reset.


> If I were to use MMIO, what sort of things could I get
> access to?  Just registers you think?  I might could tweak
> the firmware assert routine to scribble the crash register
> contents into a specific place, perhaps a bit at a time
> so that the host could read it if normal crash dump read
> fails?

I was thinking that instead of doing a while (1) {} loop towards the
end of the assert routine you could instead put a busy loop and
introduce a simple interaction within it via a bunch of MMIO registers
that are software writable (e.g. scratch registers) and aren't used by
the MAC related hw. With that you could have a fallback way to
"stream" the crash dump data by putting each data word in a register
at a time and host ACKing each one.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Ben Greear]

On 02/02/2015 10:31 PM, Michal Kazior wrote:
> On 2 February 2015 at 18:03, Ben Greear <greearb@candelatech.com> wrote:
>> On 02/02/2015 04:11 AM, Michal Kazior wrote:
>>> On 1 February 2015 at 18:46, Ben Greear <greearb@candelatech.com> wrote:
>>>> I am trying to debug a case where firmware occasionally crashes and
>>>> the driver cannot read any crash dump to debug problem further.
>>>>
>>>> Any idea what might be the problem and how I might could read info
>>>> from the firmware (or hack firmware to deliver the crash info in some
>>>> other means...maybe though a previously reserved piece of memory on
>>>> the host??)
>>>>
>>>> [147334.397148] ath10k: firmware crashed! (uuid
>>>> ff405224-b2a4-493d-b619-19ad8152d190)
>>>> [147334.404808] ath10k: hardware name qca988x hw2.0 version 0x4100016c
>>>> [147334.411111] ath10k: firmware version: 10.1.467-ct-community-full-013
>>>> [147334.429603] ath10k: failed to read diag value at 0x1300804: -16
>>>> [147334.435647] ath10k: failed to read FW dump area address: -16
>>>
>>> I see this rarely. Mostly when the device goes bonkers at which point
>>> warm reset doesn't work anymore and I'm forced to either risk cold
>>> reset causing platform lock up or re-inject the card in the express
>>> card slot.
>>>
>>> I haven't played with this so I don't know if DMA is still possible
>>> (it might not). MMIO should be operational and there should be some
>>> scratch registers chilling around... ;-)
>>
>> To normally read the crash dump, we do this over a CE pipe?
>
> Correct. The CE7, so called diagnostic window, is used.
>
>
>> So, if IRQ handlers are thoroughly busted on the target, then
>> that could be reason why we cannot read the crash dump?
>
> Incorrect. From what I understand the diagnostic window CE has
> built-in logic for fetching target RAM memory chunks as per each
> request. This means even if target program has masked IRQs and is
> running in a while (1) {} host is still able to access its memory.
>
> If you're unable to read the dump it means that CE has crashed
> (whatever that _actually_ means) and thus the built-in logic for
> diagnostic windows goes down as well. I never got around to recover CE
> crash alone without resorting to the risky target cold reset.

Ok, that makes sense...I was curious how dump could be read with the while(1) assert loop going...

>> If I were to use MMIO, what sort of things could I get
>> access to?  Just registers you think?  I might could tweak
>> the firmware assert routine to scribble the crash register
>> contents into a specific place, perhaps a bit at a time
>> so that the host could read it if normal crash dump read
>> fails?
>
> I was thinking that instead of doing a while (1) {} loop towards the
> end of the assert routine you could instead put a busy loop and
> introduce a simple interaction within it via a bunch of MMIO registers
> that are software writable (e.g. scratch registers) and aren't used by
> the MAC related hw. With that you could have a fallback way to
> "stream" the crash dump data by putting each data word in a register
> at a time and host ACKing each one.

That sounds reasonable.  Do you know of any example code that accesses
a target register or two with the MMIO logic on the host?  I think firmware has some examples of how
to fiddle with registers, so I think I can make a stab at that part....

Thanks,
Ben

>
>
> Michał
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Michal Kazior]

On 3 February 2015 at 14:57, Ben Greear <greearb@candelatech.com> wrote:
> On 02/02/2015 10:31 PM, Michal Kazior wrote:
>> On 2 February 2015 at 18:03, Ben Greear <greearb@candelatech.com> wrote:
[...]
>>> If I were to use MMIO, what sort of things could I get
>>> access to?  Just registers you think?  I might could tweak
>>> the firmware assert routine to scribble the crash register
>>> contents into a specific place, perhaps a bit at a time
>>> so that the host could read it if normal crash dump read
>>> fails?
>>
>> I was thinking that instead of doing a while (1) {} loop towards the
>> end of the assert routine you could instead put a busy loop and
>> introduce a simple interaction within it via a bunch of MMIO registers
>> that are software writable (e.g. scratch registers) and aren't used by
>> the MAC related hw. With that you could have a fallback way to
>> "stream" the crash dump data by putting each data word in a register
>> at a time and host ACKing each one.
>
> That sounds reasonable.  Do you know of any example code that accesses
> a target register or two with the MMIO logic on the host?  I think firmware
> has some examples of how
> to fiddle with registers, so I think I can make a stab at that part....

From what I understand the MMIO space is shared between host and
target program so there's no need for any special handling. You can
look for FW_IND_EVENT_PENDING and FW_INDICATOR_ADDRESS usage both in
firmware and ath10k - it's already used to notify host about a crash.
It should be pretty straightforward. The indicator address itself
seems to be a scratch register already and there are a few more so you
should be able to design a ping-pong protocol to pass data.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Ben Greear]

On 02/03/2015 06:26 AM, Michal Kazior wrote:
> On 3 February 2015 at 14:57, Ben Greear <greearb@candelatech.com> wrote:

>> That sounds reasonable.  Do you know of any example code that accesses
>> a target register or two with the MMIO logic on the host?  I think firmware
>> has some examples of how
>> to fiddle with registers, so I think I can make a stab at that part....
>
>  From what I understand the MMIO space is shared between host and
> target program so there's no need for any special handling. You can
> look for FW_IND_EVENT_PENDING and FW_INDICATOR_ADDRESS usage both in
> firmware and ath10k - it's already used to notify host about a crash.
> It should be pretty straightforward. The indicator address itself
> seems to be a scratch register already and there are a few more so you
> should be able to design a ping-pong protocol to pass data.

While implementing this, I ran into a question:

 From what I can tell, the ath10k_pci_read32 and ath10k_pci_write32
are treated like they return and write data in host-byte-order.

But, iowrite32 and ioread32 are defined to be pci-byte-order which
is little-endian?
http://comments.gmane.org/gmane.linux.kernel/1042371
(or is that bad info?)

Do we need to be converting values to little-endian before
writing them?

And upon, read, do we need to convert them to host order?

I guess we must not since it seems things mostly work now, but I
am not sure why that is...

Thanks,
Ben



>
>
> Michał
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Ben Greear]

Well, it seems to work at least when I crash it nicely with the
debugfs firmware-assert hooks.  Tested on x86...if I have endian
issues, things might go poorly on other platforms.

Here is patch I am using in ath10k...  I added similar logic to the
CT firmware.  Now, will have to see if we can reproduce the crash where
CE transport totally fails...

http://dmz2.candelatech.com/git/gitweb.cgi?p=linux-3.17.dev.y/.git;a=commitdiff;h=52333b889f0c818d2766d0d514ac31e5c53ce0ab;hp=997d36105fe9803bf7be0bbb6dba61710c3e8232

Comments welcome on the patch, though it has little to no chance of ever
making it upstream...

Thanks,
Ben


On 02/03/2015 07:45 PM, Ben Greear wrote:
> On 02/03/2015 06:26 AM, Michal Kazior wrote:
>> On 3 February 2015 at 14:57, Ben Greear <greearb@candelatech.com> wrote:
>
>>> That sounds reasonable.  Do you know of any example code that accesses
>>> a target register or two with the MMIO logic on the host?  I think firmware
>>> has some examples of how
>>> to fiddle with registers, so I think I can make a stab at that part....
>>
>>  From what I understand the MMIO space is shared between host and
>> target program so there's no need for any special handling. You can
>> look for FW_IND_EVENT_PENDING and FW_INDICATOR_ADDRESS usage both in
>> firmware and ath10k - it's already used to notify host about a crash.
>> It should be pretty straightforward. The indicator address itself
>> seems to be a scratch register already and there are a few more so you
>> should be able to design a ping-pong protocol to pass data.
>
> While implementing this, I ran into a question:
>
>  From what I can tell, the ath10k_pci_read32 and ath10k_pci_write32
> are treated like they return and write data in host-byte-order.
>
> But, iowrite32 and ioread32 are defined to be pci-byte-order which
> is little-endian?
> http://comments.gmane.org/gmane.linux.kernel/1042371
> (or is that bad info?)
>
> Do we need to be converting values to little-endian before
> writing them?
>
> And upon, read, do we need to convert them to host order?
>
> I guess we must not since it seems things mostly work now, but I
> am not sure why that is...
>
> Thanks,
> Ben
>
>
>
>>
>>
>> Michał
>>
>
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Unable to read firmware registers on crash?

[Michal Kazior]

On 4 February 2015 at 05:28, Ben Greear <greearb@candelatech.com> wrote:
> Well, it seems to work at least when I crash it nicely with the
> debugfs firmware-assert hooks.  Tested on x86...if I have endian
> issues, things might go poorly on other platforms.
>
> Here is patch I am using in ath10k...  I added similar logic to the
> CT firmware.  Now, will have to see if we can reproduce the crash where
> CE transport totally fails...

If you have a system capable of IOMMU you can force CE to access a
non-mapped memory. You could try inserting a bogus CE item address in
one of its rings.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k