PowerPC, P2020RDB, application debug when the application is in tight loop, Sysrq

PowerPC, P2020RDB, application debug when the application is in tight loop, Sysrq

[saikrishna gajula]

[-- Attachment #1: Type: text/plain, Size: 4135 bytes --]

HI All,

             I am new to this group. I am working on Freescale P2020
platform running linux 2.6.21.   I am looking for debug mechanism/utility,
when a multi threaded application running on linux , appears to be hung (
running in a tight loop,deadlock)  while not able to access the board
through serial/SSH/Telnet.

   I was looking at Magic sysrq option in linux to generate the stack,
register dump when the application is hung. I am able to dump the call
trace in normal working conditions. But i can't use  echo  t >
/proc/sysrq-trigger and debug when the application hung.

 I am using below piece of code(drivers/serial/8250.c) on P2020RDB to debug
the application where in , in hung situation, when i press 'y'  followed by
't'  on serial console it should go to sysrq handler, and dump the call
trace, but it is not happening.(simply board hung)

{
           if(sysrq_enable_flag)
                  handle_sysrq(ch, up->port.info->tty);

        sysrq_enable_flag = 0;

        if(ch == 'y')
            sysrq_enable_flag = 1;
}

It would be helpful if you provide any hint on the issue, or any other way
to debug the application in hang situations.

Thanks,
Sai


On Tue, Apr 16, 2013 at 9:24 PM, Anshuman Khandual <
khandual@linux.vnet.ibm.com> wrote:

>                 Branch History Rolling Buffer (BHRB) is a new PMU feaure
> in IBM
> POWER8 processor which records the branch instructions inside the execution
> pipeline. This patchset enables the basic functionality of the feature
> through
> generic perf branch stack sampling framework.
>
> Sample output
> -------------
> $./perf record -b top
> $./perf report
>
> Overhead  Command  Source Shared Object                           Source
> Symbol  Target Shared Object                        Target Symbol
> # ........  .......  ....................
>  ......................................  ....................
>  ...................................
> #
>
>      7.82%      top  libc-2.11.2.so        [k] _IO_vfscanf
>           libc-2.11.2.so        [k] _IO_vfscanf
>      6.17%      top  libc-2.11.2.so        [k] _IO_vfscanf
>           [unknown]             [k] 00000000
>      2.37%      top  [unknown]             [k] 0xf7aafb30
>          [unknown]             [k] 00000000
>      1.80%      top  [unknown]             [k] 0x0fe07978
>          libc-2.11.2.so        [k] _IO_vfscanf
>      1.60%      top  libc-2.11.2.so        [k] _IO_vfscanf
>           [kernel.kallsyms]     [k] .do_task_stat
>      1.20%      top  [kernel.kallsyms]     [k] .do_task_stat
>         [kernel.kallsyms]     [k] .do_task_stat
>      1.02%      top  libc-2.11.2.so        [k] vfprintf
>          libc-2.11.2.so        [k] vfprintf
>      0.92%      top  top                   [k] _init
>         [unknown]             [k] 0x0fe037f4
>
> Changes in V2
> --------------
> - Added copyright messages to the newly created files
> - Modified couple of commit messages
>
> Anshuman Khandual (5):
>   powerpc, perf: Add new BHRB related instructions on POWER8
>   powerpc, perf: Add basic assembly code to read BHRB entries on POWER8
>   powerpc, perf: Add new BHRB related generic functions, data and flags
>   powerpc, perf: Define BHRB generic functions, data and flags for POWER8
>   powerpc, perf: Enable branch stack sampling framework support with BHRB
>
>  arch/powerpc/include/asm/perf_event_server.h |  6 ++
>  arch/powerpc/include/asm/ppc-opcode.h        |  7 ++
>  arch/powerpc/perf/Makefile                   |  2 +-
>  arch/powerpc/perf/bhrb.S                     | 44 +++++++++++++
>  arch/powerpc/perf/core-book3s.c              | 96
> +++++++++++++++++++++++++++-
>  arch/powerpc/perf/perf_event_bhrb.c          | 85 ++++++++++++++++++++++++
>  arch/powerpc/perf/power8-pmu.c               | 57 ++++++++++++++++-
>  7 files changed, 292 insertions(+), 5 deletions(-)
>  create mode 100644 arch/powerpc/perf/bhrb.S
>  create mode 100644 arch/powerpc/perf/perf_event_bhrb.c
>
> --
> 1.7.11.7
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>

[-- Attachment #2: Type: text/html, Size: 5464 bytes --]

Re: PowerPC, P2020RDB, application debug when the application is in tight loop, Sysrq

[Scott Wood]

On 04/17/2013 12:04:10 AM, saikrishna gajula wrote:
> HI All,
>=20
>              I am new to this group. I am working on Freescale P2020
> platform running linux 2.6.21.   I am looking for debug =20
> mechanism/utility,
> when a multi threaded application running on linux , appears to be =20
> hung (
> running in a tight loop,deadlock)  while not able to access the board
> through serial/SSH/Telnet.
>=20
>    I was looking at Magic sysrq option in linux to generate the stack,
> register dump when the application is hung. I am able to dump the call
> trace in normal working conditions. But i can't use  echo  t >
> /proc/sysrq-trigger and debug when the application hung.
>=20
>  I am using below piece of code(drivers/serial/8250.c) on P2020RDB to =20
> debug
> the application where in , in hung situation, when i press 'y'  =20
> followed by
> 't'  on serial console it should go to sysrq handler, and dump the =20
> call
> trace, but it is not happening.(simply board hung)
>=20
> {
>            if(sysrq_enable_flag)
>                   handle_sysrq(ch, up->port.info->tty);
>=20
>         sysrq_enable_flag =3D 0;
>=20
>         if(ch =3D=3D 'y')
>             sysrq_enable_flag =3D 1;
> }
>=20
> It would be helpful if you provide any hint on the issue, or any =20
> other way
> to debug the application in hang situations.

There's an erratum regarding breaks in Freescale's serial port =20
implementation that has been worked around in more recent kernels.  =20
2.6.21 is six years old.  If you update to a kernel that has the =20
workaround, or backport the workaround yourself to your old kernel, you =20
should be able to use a break instead of hacking in a check for 'y'.

Other than that, either the kernel is hung too badly to respond to =20
serial input at all (in which case use JTAG to debug) or there's =20
something wrong with your 'y' hack (sysrq_enable_flag doesn't even =20
appear in current kernels so I can't see if this is the case without =20
digging through old code).

In any case, upgrade to a newer kernel -- maybe you're hitting a bug =20
that has been fixed since then.

-Scott=

Re: PowerPC, P2020RDB, application debug when the application is in tight loop, Sysrq

[saikrishna gajula]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

Hi Scott,

           Thanks for the information. "sysrq_enable_flag" is custom code
introduced as part of  'y' hack implementation. I will check with the
latest kernels to make use of "sysrq"  feature.

Regrads,
Sai


On Thu, May 2, 2013 at 1:24 AM, Scott Wood <scottwood@freescale.com> wrote:

> On 04/17/2013 12:04:10 AM, saikrishna gajula wrote:
>
>> HI All,
>>
>>              I am new to this group. I am working on Freescale P2020
>> platform running linux 2.6.21.   I am looking for debug mechanism/utility,
>> when a multi threaded application running on linux , appears to be hung (
>> running in a tight loop,deadlock)  while not able to access the board
>> through serial/SSH/Telnet.
>>
>>    I was looking at Magic sysrq option in linux to generate the stack,
>> register dump when the application is hung. I am able to dump the call
>> trace in normal working conditions. But i can't use  echo  t >
>> /proc/sysrq-trigger and debug when the application hung.
>>
>>  I am using below piece of code(drivers/serial/8250.c) on P2020RDB to
>> debug
>> the application where in , in hung situation, when i press 'y'  followed
>> by
>> 't'  on serial console it should go to sysrq handler, and dump the call
>> trace, but it is not happening.(simply board hung)
>>
>> {
>>            if(sysrq_enable_flag)
>>                   handle_sysrq(ch, up->port.info->tty);
>>
>>         sysrq_enable_flag = 0;
>>
>>         if(ch == 'y')
>>             sysrq_enable_flag = 1;
>> }
>>
>> It would be helpful if you provide any hint on the issue, or any other way
>> to debug the application in hang situations.
>>
>
> There's an erratum regarding breaks in Freescale's serial port
> implementation that has been worked around in more recent kernels.  2.6.21
> is six years old.  If you update to a kernel that has the workaround, or
> backport the workaround yourself to your old kernel, you should be able to
> use a break instead of hacking in a check for 'y'.
>
> Other than that, either the kernel is hung too badly to respond to serial
> input at all (in which case use JTAG to debug) or there's something wrong
> with your 'y' hack (sysrq_enable_flag doesn't even appear in current
> kernels so I can't see if this is the case without digging through old
> code).
>
> In any case, upgrade to a newer kernel -- maybe you're hitting a bug that
> has been fixed since then.
>
> -Scott
>

[-- Attachment #2: Type: text/html, Size: 3119 bytes --]