kgdb errors with serial console

kgdb errors with serial console

[Elvis Dowson]

Hi,
        I'm getting the following errors when attempting to use kgdb with the serial console. Any idea how I can resolve this issue? 

The correct serial parameters when I use kermit with the target board is 

set flow-control off
set carrier-watch none
set speed 115200

and it connects correctly.

I'm not sure if the gdb configuration commands for the serial port are sufficient, and if there is a parameter that I am missing here. The host is connected to the target using an FTDI USB to serial converter, so it appears as /dev/ttyUSB0 on the host and as ttyS2 on the target.

# su
# cd /tool/patches/android-rowboat-froyo-2.2-patchwork/kernel
# arm-angstrom-linux-gnueabi-gdb
(gdb) file vmlinux
Reading symbols from /tool/patches/android-rowboat-froyo-2.2-patchwork/kernel/vmlinux...done.

(gdb) set remotebaud 115200
(gdb) set remoteflow 0
(gdb) set debug remote 1
(gdb) set debug serial 1
(gdb) target remote /dev/ttyUSB0

Remote debugging using /dev/ttyUSB0
Sending packet: $qSupported#37...[
r +]Ack
[$][q][S][u][p][p][o][r][t][e][d][#][3][7]Packet received: qSupported
Packet qSupported (supported-packets) is supported
warning: unrecognized item "qSupported" in "qSupported" response
Sending packet: $Hg0#df...[+]Ack
[$][H][g][0][#][d][f]Packet received: Hg0
Sending packet: $?#3f...[+]Ack
[$][?][#][3][f]Packet received: ?
Sending packet: $Hc-1#09...[+]Ack
[$][H][c][-][1][#][0][9]Packet received: Hc-1
Sending packet: $qC#b4...[+]Ack
[$][q][C][#][b][4]Packet received: qC
Sending packet: $qAttached#8f...[+]Ack
[$][q][A][t][t][a][c][h][e][d][#][8][f]Packet received: qAttached
Packet qAttached (query-attached) is supported
Sending packet: $qOffsets#4b...[+]Ack
[$][q][O][f][f][s][\r][\r][\n][e][t][s][#][4][b]Bad checksum, sentsum=0x4b, csum=0x6f, buf=qOffs\r\r\nets
[-][<Timeout: 2 seconds>]Timed out.
[-][<Timeout: 2 seconds>]Timed out.
Ignoring packet error, continuing...
Malformed response to offset query, qOffs
ets
(gdb) 


Best regards,

Elvis Dowson

Re: kgdb errors with serial console

[Jason Wessel]

On 10/07/2010 04:47 PM, Elvis Dowson wrote:
> Hi,
>         I'm getting the following errors when attempting to use kgdb with the serial console. Any idea how I can resolve this issue? 
>
> The correct serial parameters when I use kermit with the target board is 
>
> set flow-control off
> set carrier-watch none
> set speed 115200
>
> and it connects correctly.
>
> I'm not sure if the gdb configuration commands for the serial port are sufficient, and if there is a parameter that I am missing here. The host is connected to the target using an FTDI USB to serial converter, so it appears as /dev/ttyUSB0 on the host and as ttyS2 on the target.
>
> # su
> # cd /tool/patches/android-rowboat-froyo-2.2-patchwork/kernel
> # arm-angstrom-linux-gnueabi-gdb
> (gdb) file vmlinux
> Reading symbols from /tool/patches/android-rowboat-froyo-2.2-patchwork/kernel/vmlinux...done.
>
> (gdb) set remotebaud 115200
> (gdb) set remoteflow 0
> (gdb) set debug remote 1
> (gdb) set debug serial 1
>   
You don't want to use this one, it just tends to pollute the logs, "set
debug remote 1" should be enough.

> (gdb) target remote /dev/ttyUSB0
>
>   


Looks to me like you did not activate the kernel debugger before using
gdb to enter, because the gdb you are using is not going to send the
"sysrq g" sequence.


What you would want to do here is to "echo g > /proc/sysrq-trigger" with
your terminal program, disconnect and then try connecting the debugger
to the kernel.   Another option is to use the agent-proxy so you can
have a console connection and a debugger connection.

It would probably also be good to test if the debugger is working at all
on your serial port.

Configure the debugger with:
# echo ttyS2 > /sys/module/kgdboc/parameters/kgdboc
kgdb: Registered I/O driver kgdboc.
# echo g > /proc/sysrq-trigger
SysRq : DEBUG

And now to exit debugger you must blindly and perfectly type
$D#44+

Nothing will be echoed because at this stage the kernel serial polling
driver would just be collecting characters.

After typing that the kernel should return back to the running state and
print something like:
+$OK#9a#


Jason.





> Remote debugging using /dev/ttyUSB0
> Sending packet: $qSupported#37...[
> r +]Ack
> [$][q][S][u][p][p][o][r][t][e][d][#][3][7]Packet received: qSupported
> Packet qSupported (supported-packets) is supported
> warning: unrecognized item "qSupported" in "qSupported" response
> Sending packet: $Hg0#df...[+]Ack
> [$][H][g][0][#][d][f]Packet received: Hg0
> Sending packet: $?#3f...[+]Ack
> [$][?][#][3][f]Packet received: ?
> Sending packet: $Hc-1#09...[+]Ack
> [$][H][c][-][1][#][0][9]Packet received: Hc-1
> Sending packet: $qC#b4...[+]Ack
> [$][q][C][#][b][4]Packet received: qC
> Sending packet: $qAttached#8f...[+]Ack
> [$][q][A][t][t][a][c][h][e][d][#][8][f]Packet received: qAttached
> Packet qAttached (query-attached) is supported
> Sending packet: $qOffsets#4b...[+]Ack
> [$][q][O][f][f][s][\r][\r][\n][e][t][s][#][4][b]Bad checksum, sentsum=0x4b, csum=0x6f, buf=qOffs\r\r\nets
> [-][<Timeout: 2 seconds>]Timed out.
> [-][<Timeout: 2 seconds>]Timed out.
> Ignoring packet error, continuing...
> Malformed response to offset query, qOffs
> ets
> (gdb) 
>
>
> Best regards,
>
> Elvis Dowson
>
>
>   

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,

On Oct 8, 2010, at 2:37 AM, Jason Wessel wrote:
> 
> Looks to me like you did not activate the kernel debugger before using
> gdb to enter, because the gdb you are using is not going to send the
> "sysrq g" sequence.

Will it not be activated and wait with the following boot args?

mmcargs=setenv bootargs console=${console} vram=12M omapfb.vram=0:4M omapfb.mode=dvi:${dvimode} omapdss.def_disp=${defaultdisplay} root=/dev/mmcblk0p2 rw rootfstype=ext3 rootwait kgdb=${console} kgdboc=${console} kgdbwait

After specific the following commands in .gdbinit, I am able to get properly formatted messages in the Eclipse IDE

set remotebaud 115200
set remoteflow 0
set debug remote 1

Best regards,

Elvis

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,

On Oct 8, 2010, at 2:37 AM, Jason Wessel wrote:

> It would probably also be good to test if the debugger is working at all
> on your serial port.
> 
> Configure the debugger with:
> # echo ttyS2 > /sys/module/kgdboc/parameters/kgdboc
> kgdb: Registered I/O driver kgdboc.
> # echo g > /proc/sysrq-trigger
> SysRq : DEBUG
> 
> And now to exit debugger you must blindly and perfectly type
> $D#44+
> 
> Nothing will be echoed because at this stage the kernel serial polling
> driver would just be collecting characters.
> 
> After typing that the kernel should return back to the running state and
> print something like:
> +$OK#9a#

Cool, I tried all the commands that you suggested and it worked perfectly. I got the same outputs. 

So, if I understand correctly, I boot into the kernel without the kgbdwait option, and then manually launch the kgdb and then connect from Eclipse using gdb.

The system waits at the root login prompt. 

Just so that I can see some quick results, can you suggest which file I could set a break point into, and monitor the registers and variable states at this stage, i.e. immediately after the root login?

Best regards,

Elvis

Re: kgdb errors with serial console

[Jason Wessel]

On 10/07/2010 06:07 PM, Elvis Dowson wrote:
> On Oct 8, 2010, at 2:37 AM, Jason Wessel wrote:
>
>   
>> It would probably also be good to test if the debugger is working at all
>> on your serial port.
>>
>> Configure the debugger with:
>> # echo ttyS2 > /sys/module/kgdboc/parameters/kgdboc
>> kgdb: Registered I/O driver kgdboc.
>> # echo g > /proc/sysrq-trigger
>> SysRq : DEBUG
>>
>> And now to exit debugger you must blindly and perfectly type
>> $D#44+
>>
>> Nothing will be echoed because at this stage the kernel serial polling
>> driver would just be collecting characters.
>>
>> After typing that the kernel should return back to the running state and
>> print something like:
>> +$OK#9a#
>>     
>
> Cool, I tried all the commands that you suggested and it worked perfectly. I got the same outputs. 
>
> So, if I understand correctly, I boot into the kernel without the kgbdwait option, and then manually launch the kgdb and then connect from Eclipse using gdb.
>
> The system waits at the root login prompt. 
>   

Probably just set a break point at sys_sync.

Of course if you are willing to try something a bit more complex you can
probably make use of being able to interactively break in with the
eclipse gdb plugin, assuming your serial port is connected to a linux
host, and it appeared that way based on your description.

On your host you could set up the following:


git clone git://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
cd agent-proxy
make
./agent-proxy 4440^4441 0 /dev/ttyUSB0,115200 &

telnet localhost 4440

Now in your gdbinit file you replace the target remote with

target remote localhost:4441

The agent proxy should take care of sending the sysrq g for you, and the
telnet window will provide you a look at the target console.

Jason.

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,
                      I'm getting eclipse target time outs, I've attached link to the video here : http://drop.io/kgdbwe01
 
On Oct 8, 2010, at 3:44 AM, Jason Wessel wrote:
> 
> Of course if you are willing to try something a bit more complex you can
> probably make use of being able to interactively break in with the
> eclipse gdb plugin, assuming your serial port is connected to a linux
> host, and it appeared that way based on your description.
> 
> On your host you could set up the following:
> 
> git clone git://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
> cd agent-proxy
> make
> ./agent-proxy 4440^4441 0 /dev/ttyUSB0,115200 &
> 
> telnet localhost 4440
> 
> Now in your gdbinit file you replace the target remote with
> 
> target remote localhost:4441
> 
> The agent proxy should take care of sending the sysrq g for you, and the
> telnet window will provide you a look at the target console.

The agent-proxy suggestion was a lifesaver!!  It makes it so much easier to debug the target board, and you can run it as a normal user when connecting to /dev/ttyUSB0. 

I made the required changes to .gdbinit, and modified the Eclipse debugger settings to TCP/IP, with localhost and port 4441. 

I also then go to the telnet terminal session, and then type 

# echo g > /proc/sysrq-trigger
SysRq : DEBUG
Entering KGDB

and then launched eclipse debugger. It loads the vmlinux file, and a bunch of junk characters keep appearing continuously on the telnet session. 

After about a minute or so, Eclipse terminates the session saying that it timed out. You can see this in the video. I edited the video, in between to trim off the long wait, with the characters scrolling in the terminal session, to the point where it crashes.

Best regards,

Elvis Dowson

Re: kgdb errors with serial console

[Jason Wessel]

On 10/08/2010 05:41 AM, Elvis Dowson wrote:
> Hi Jason,
>                       I'm getting eclipse target time outs, I've attached link to the video here : http://drop.io/kgdbwe01
>   

Because you are using a serial link you certainly want to close any
related to extra data you don't need.   Not knowing a whole lot about
what the Eclipse plugin gdb does, it is hard to know if this was an
eclipse timeout or not.

If you add the following to your .gdbinit it will at least tell you what
happened at the end.

set remotelogfile /tmp/gdb_serial_log.txt

It would be good to know if eclipse timed out and it just detached from
the target, or if some command crashed gdb.  It is purely a guess, but
it is probably eclipse requesting lots of information and simply timing out.



> and then launched eclipse debugger. It loads the vmlinux file, and a bunch of junk characters keep appearing continuously on the telnet session. 
>
>   

The "junk" you speak of is simply the gdb serial traffic, it is answered
in the FAQ


http://git.kernel.org/?p=utils/kernel/kgdb/agent-proxy.git;a=blob;f=README.TXT
> After about a minute or so, Eclipse terminates the session saying that it timed out. You can see this in the video. I edited the video, in between to trim off the long wait, with the characters scrolling in the terminal session, to the point where it crashes.
>
>   

I would not expect the terminal session to crash.  If an oops is
printed, it means there is an internal kgdb problem.

Jason.

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,
                     I've reproduced the log file below. I don't know how to interpret this, do you see anything wrong in the log fie?

On Oct 8, 2010, at 5:17 PM, Jason Wessel wrote:
> 
> If you add the following to your .gdbinit it will at least tell you what
> happened at the end.
> 
> set remotelogfile /tmp/gdb_serial_log.txt

w +$qSupported#37
r +$#00
w +$Hg0#df
r +$OK#9a
w +$?#3f
r +$S05#b8
w +$Hc-1#09
r +$OK#9a
w +$qC#b4
r +$QC00000000fffffffe#43
w +$qAttached#8f
r +$#00
w +$qOffsets#4b
r +$#00
w +$g#67
r +$140000009301008001000000844ec7c0acf16fc067000000010000000700000000e823cf00e823cf930100209301004028de58c0b0de58c0749409c0849409c00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000093010040#aa
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099480,4#9e
r +$5ff07ff5#69
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099480,4#9e
r +$5ff07ff5#69
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$qSymbol::#5b
r +$#00
w +
c cd /tool/patches/android-rowboat-froyo-2.2-patchwork/kernel
c info proc
End of log


> It would be good to know if eclipse timed out and it just detached from
> the target, or if some command crashed gdb.  It is purely a guess, but
> it is probably eclipse requesting lots of information and simply timing out.
> 
> 
> I would not expect the terminal session to crash.  If an oops is
> printed, it means there is an internal kgdb problem.

I don't think kgdb crashed. It seems to be working reliably on the target side. If I restart the debug session, it receives the sysreq + g command and starts the debugger. 

Best regards,

Elvis Dowson

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,
                  There was a setting in Eclipse that allowed me to increase the timeout, located under Windows > Preferences > C/C++ > Debug

I increase the settings to 120000(ms) for the debugger time out and 60000(ms) for the launcher. 

So, it didn't time out, but it kept repeating the same sequence over and over again, I haven't yet set a break-point. 

$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001de3#f1+$04d04d02#f2+$60509fe5#04+$047089e5#d6+$60509fe5#04+$047089e5#d6+$60509fe5#04+$60509fe5#04+$60509fe5#04+$44d04de2#2b+$04001d

and then after sometime, stops with the timeout error. This is with kernel linux-omap-2.6.32. 

Here is the log file for this session:


w +$qSupported#37
r +$#00
w +$Hg0#df
r +$OK#9a
w +$?#3f
r +$S05#b8
w +$Hc-1#09
r +$OK#9a
w +$qC#b4
r +$QC00000000fffffffe#43
w +$qAttached#8f
r +$#00
w +$qOffsets#4b
r +$#00
w +$g#67
r +$140000009301008001000000844ec7c0acf16fc067000000010000000700000000e823cf00e823cf930100209301004020de58c0a8de58c0749409c0849409c00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000093010040#a9
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099480,4#9e
r +$5ff07ff5#69
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099480,4#9e
r +$5ff07ff5#69
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$mc0099484,4#a2
r +$ffdeffe7#fd
w +$qSymbol::#5b
r +$#00
w +
c info proc
End of log

Best regards,

Elvis

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,
                    Some good news. I switch the GDB command set from mi to mi1, and it seems to be working now. 

What I did is added the kgdbwait command to the bootargs as follows (not sure if it was necessary, but will try once again without it)

setenv mmcargs 'setenv bootargs console=${console} vram=12M omapfb.vram=0:4M omapfb.mode=dvi:${dvimode} omapdss.def_disp=${defaultdisplay} root=/dev/mmcblk0p2 rw rootfstype=ext3 rootwait kgdb=${console} kgdboc=${console} kgdbwait'

Then I launched the debugger, and it did something, and resumed normally. 

Then I manually went to the console and activated kgdb by typing 

echo g > /proc/sysrq-trigger

This caused the debugger on the Eclipse side to process something, and then it stopped in file kgdb.c at the call to arch_kgdb_breakpoint(), in the following method:

void kgdb_breakpoint(void)
{
	atomic_set(&kgdb_setting_breakpoint, 1);
	wmb(); /* Sync point before breakpoint */
	arch_kgdb_breakpoint();
	wmb(); /* Sync point after breakpoint */
	atomic_set(&kgdb_setting_breakpoint, 0);
}

I can now clearly see all the registers and variables in Eclipse. 

Question 02: sys_sync doesn't seem to be called in the ARM arch files, it only exists for x86 and sparc architectures.

Is there any other kernel file I can set a break-point into, to be able to check if everything is working ok?

For example, if I type a normal cd or ls command? 

Best regards,

Elvis

Re: kgdb errors with serial console

[Elvis Dowson]

Hi Jason,
                   Perhaps the debugger is timing out because it doesn't get a break point to debug a specific event? 

Could you tell me the order in which I should try it? 

I think earlier on, when I used the kgdbwait bootarg, it ran and initialized eclipse debugger. Then when I entered the 

echo g > /proc/sysrq-trigger

it just activated the debugger correctly because the kgdb_breakpoint method was called in file kgdb.c

So, what should I do to test this? Do I write a small file, and add a call to kgdb_breakpoint in that example file, and then when I execute it, the debugger would get activated? 

Is there any other test that I can run, without writing my own example ?

Best regards,

Elvis