hard lockup problem

hard lockup problem

[Lin Ming]

Hi list,

I encounter a problem that cpu stuck with irq disabled, which is known
as hard lockup.
I know there is NMI hard lockup detector for x86, which can dump the
back trace of the hard lockup.

Is there any similar feature for MIPS?

Thanks,
Lin Ming

Re: hard lockup problem

[Lin Ming]

On Thu, Apr 18, 2013 at 3:13 PM, Lin Ming <minggr@gmail.com> wrote:
> Hi list,
>
> I encounter a problem that cpu stuck with irq disabled, which is known
> as hard lockup.
> I know there is NMI hard lockup detector for x86, which can dump the
> back trace of the hard lockup.
>
> Is there any similar feature for MIPS?

And it's MIPS32 4K cpu running 2.6.30 kernel.

>
> Thanks,
> Lin Ming

Re: hard lockup problem

[Ralf Baechle]

On Thu, Apr 18, 2013 at 03:13:55PM +0800, Lin Ming wrote:

> I encounter a problem that cpu stuck with irq disabled, which is known
> as hard lockup.
> I know there is NMI hard lockup detector for x86, which can dump the
> back trace of the hard lockup.
> 
> Is there any similar feature for MIPS?

No, there isn't, unfortunately.

This is because on MIPS an NMI is very different from for example x86.
An NMI goes straight to a firmware address and most firmware implementations
don't provided a suitable hook for an OS to gain control back from an NMI.

Generally on MIPS NMIs are used to signal catastrophic problems, things
like a machine check exception but external to the CPU.

One of the notable exceptions is Octeon where (see the Octeon watchdog
driver) an OS can regain control after an NMI.  Malta and SGI IP27 also
have somewhat useful NMIs.

  Ralf

Re: hard lockup problem

[Lin Ming]

On Thu, Apr 18, 2013 at 8:44 AM, Ralf Baechle <ralf@linux-mips.org> wrote:
> On Thu, Apr 18, 2013 at 03:13:55PM +0800, Lin Ming wrote:
>
>> I encounter a problem that cpu stuck with irq disabled, which is known
>> as hard lockup.
>> I know there is NMI hard lockup detector for x86, which can dump the
>> back trace of the hard lockup.
>>
>> Is there any similar feature for MIPS?
>
> No, there isn't, unfortunately.
>
> This is because on MIPS an NMI is very different from for example x86.
> An NMI goes straight to a firmware address and most firmware implementations
> don't provided a suitable hook for an OS to gain control back from an NMI.
>
> Generally on MIPS NMIs are used to signal catastrophic problems, things
> like a machine check exception but external to the CPU.
>
> One of the notable exceptions is Octeon where (see the Octeon watchdog
> driver) an OS can regain control after an NMI.  Malta and SGI IP27 also
> have somewhat useful NMIs.
>
>   Ralf

Re: hard lockup problem

[Lin Ming]

On Thu, Apr 18, 2013 at 8:44 AM, Ralf Baechle <ralf@linux-mips.org> wrote:
> On Thu, Apr 18, 2013 at 03:13:55PM +0800, Lin Ming wrote:
>
>> I encounter a problem that cpu stuck with irq disabled, which is known
>> as hard lockup.
>> I know there is NMI hard lockup detector for x86, which can dump the
>> back trace of the hard lockup.
>>
>> Is there any similar feature for MIPS?
>
> No, there isn't, unfortunately.
>
> This is because on MIPS an NMI is very different from for example x86.
> An NMI goes straight to a firmware address and most firmware implementations
> don't provided a suitable hook for an OS to gain control back from an NMI.
>
> Generally on MIPS NMIs are used to signal catastrophic problems, things
> like a machine check exception but external to the CPU.
>
> One of the notable exceptions is Octeon where (see the Octeon watchdog
> driver) an OS can regain control after an NMI.  Malta and SGI IP27 also
> have somewhat useful NMIs.

Thanks for the explanation.

Since no NMI available, what I am trying to do is:

- reserve some boot memory
- write a ftrace tracer, save ftrace functions to the reserved memory
- when hard lockup happens, hardware watch dog will reboot the board
- get the functions from the reserved memory, see what it was doing
when hard lockup happened.

Or any other hints to debug such kind of hard lockup on MIPS?

Thanks,
Lin Ming

>
>   Ralf