booting up: blocking indefinitely on kgdb?

booting up: blocking indefinitely on kgdb?

[Peter Teoh]

Today, both my system (2.6.32.-rc4 from linus git tree and linux-next)
bootup blocked indefinitely on:

kgdb: Registered I/O driver kgdbts.

while booting up.   The expected line:

kgdb: Unregistered I/O driver kgdbts, debugger disabled.

never comes up.

My bootup menu.lst:

title Fedora (2.6.26-rc4-next-20080530)
	root (hd1,7)
	kernel /boot/vmlinuz-2.6.26-rc4-next-20080530 ro
root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
	initrd /boot/initrd-2.6.26-rc4-next-20080530.img
title Fedora (2.6.32-rc4)
	root (hd1,7)
	kernel /boot/vmlinuz-2.6.32-rc4 ro
root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
	initrd /boot/initrd-2.6.32-rc4.img

and kgdb-related option:

CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_TESTS_ON_BOOT=y
CONFIG_KGDB_TESTS_BOOT_STRING="y"

The same 2.6.32-rc4 image have bootup previously before without any
problem.   So what could be the potential cause of this permanent
wait?

-- 
Regards,
Peter Teoh

Re: booting up: blocking indefinitely on kgdb?

[Peter Teoh]

sorry....now I reboot it is ok.   I don't know why. sorry about that.

On Sat, Oct 17, 2009 at 1:43 AM, Peter Teoh <htmldeveloper@gmail.com> wrote:
> Today, both my system (2.6.32.-rc4 from linus git tree and linux-next)
> bootup blocked indefinitely on:
>
> kgdb: Registered I/O driver kgdbts.
>
> while booting up.   The expected line:
>
> kgdb: Unregistered I/O driver kgdbts, debugger disabled.
>
> never comes up.
>
> My bootup menu.lst:
>
> title Fedora (2.6.26-rc4-next-20080530)
>        root (hd1,7)
>        kernel /boot/vmlinuz-2.6.26-rc4-next-20080530 ro
> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>        initrd /boot/initrd-2.6.26-rc4-next-20080530.img
> title Fedora (2.6.32-rc4)
>        root (hd1,7)
>        kernel /boot/vmlinuz-2.6.32-rc4 ro
> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>        initrd /boot/initrd-2.6.32-rc4.img
>
> and kgdb-related option:
>
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_KGDB=y
> CONFIG_KGDB_SERIAL_CONSOLE=y
> CONFIG_KGDB_TESTS=y
> CONFIG_KGDB_TESTS_ON_BOOT=y
> CONFIG_KGDB_TESTS_BOOT_STRING="y"
>
> The same 2.6.32-rc4 image have bootup previously before without any
> problem.   So what could be the potential cause of this permanent
> wait?
>
> --
> Regards,
> Peter Teoh
>



-- 
Regards,
Peter Teoh

Re: booting up: blocking indefinitely on kgdb?

[Jason Wessel]

Peter Teoh wrote:
> sorry....now I reboot it is ok.   I don't know why. sorry about that.
>
>   

This is actually a real problem.  It is a race condition, and there are
actually two separate problems.

1) When a processor kernel thread is put into the single step state,
kgdb expects it to hit the single trap on the same processor the single
step request was made on.

On an SMP system a process or kernel thread can migrate to another
processor after kgdb resumes.  This will result in a hard hang in the
cpu roundup part of kgdb.

2) Schedule lock contention can cause a hard hang.

On an SMP system kgdb for the x86 architecture single steps by running
only a single core.  This is quite problematic if you have the schedule
a lock held by a cpu which is in busy wait.  The system will deadlock on
the single step operation from kgdb.  This problem is easily observed by
doing on a 2 processor system by doing:

   while [ 1 ] ; do find /  2> /dev/null > /dev/null; done &
   while [ 1 ] ; do date > /dev/null ; done &
   echo V1 > /sys/module/kgdbts/parameters/kgdbts

For the first problem, I have a fix which is in the linux-next branch
and will I will send a merge request to Linus to get it into the
mainline tree.

For the second problem, I am going to merge a change to release all the
processors to run, at the expense of missing a breakpoint.   It is
possible to change the behavior of this dynamically, for someone who
might care about this behavior, until a longer term approach is
implemented.  I have an experimental patch which implements the longer
term approach of using displaced stepping. 

The experimental patch uses kprobes to manage software breakpoints.  The
kprobe allows the breakpoint to remain planted while stepping around it
by using out of line instruction execution, where you emulate the
original instruction using memory elsewhere, followed by another trap
instruction.

Thanks,
Jason.

> On Sat, Oct 17, 2009 at 1:43 AM, Peter Teoh <htmldeveloper@gmail.com> wrote:
>   
>> Today, both my system (2.6.32.-rc4 from linus git tree and linux-next)
>> bootup blocked indefinitely on:
>>
>> kgdb: Registered I/O driver kgdbts.
>>
>> while booting up.   The expected line:
>>
>> kgdb: Unregistered I/O driver kgdbts, debugger disabled.
>>
>> never comes up.
>>
>> My bootup menu.lst:
>>
>> title Fedora (2.6.26-rc4-next-20080530)
>>        root (hd1,7)
>>        kernel /boot/vmlinuz-2.6.26-rc4-next-20080530 ro
>> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>>        initrd /boot/initrd-2.6.26-rc4-next-20080530.img
>> title Fedora (2.6.32-rc4)
>>        root (hd1,7)
>>        kernel /boot/vmlinuz-2.6.32-rc4 ro
>> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>>        initrd /boot/initrd-2.6.32-rc4.img
>>
>> and kgdb-related option:
>>
>> CONFIG_HAVE_ARCH_KGDB=y
>> CONFIG_KGDB=y
>> CONFIG_KGDB_SERIAL_CONSOLE=y
>> CONFIG_KGDB_TESTS=y
>> CONFIG_KGDB_TESTS_ON_BOOT=y
>> CONFIG_KGDB_TESTS_BOOT_STRING="y"
>>
>> The same 2.6.32-rc4 image have bootup previously before without any
>> problem.   So what could be the potential cause of this permanent
>> wait?
>>
>> --
>> Regards,
>> Peter Teoh
>>
>>     
>
>
>
>   

Re: booting up: blocking indefinitely on kgdb?

[Peter Teoh]

thank you for the explanation.

On Mon, Oct 19, 2009 at 9:24 AM, Jason Wessel
<jason.wessel@windriver.com> wrote:
> Peter Teoh wrote:
>> sorry....now I reboot it is ok.   I don't know why. sorry about that.
>>
>>
>
> This is actually a real problem.  It is a race condition, and there are
> actually two separate problems.
>
> 1) When a processor kernel thread is put into the single step state,
> kgdb expects it to hit the single trap on the same processor the single
> step request was made on.
>

sorry for being irrelevant....can i ask this:   even if the present
CPU is in single step mode, all other CPU can be fully running and
executing all the time, correct?   kgdb is not designed to handle more
than one CPU in single step mode, right?   if wrong, then i supposed
there must be a way to switch among processor, which i don't know how.
  not sure if the same concept pertained to kdb?

> On an SMP system a process or kernel thread can migrate to another
> processor after kgdb resumes.  This will result in a hard hang in the
> cpu roundup part of kgdb.

not sure if it is ok if i can know more about the reason for the hard
hang (in slightly more detail).   The reason is because i am trying to
understand if this same problem does exists in any other parts of the
kernel?   eg, kdb? or anywhere in the suspend-resume cycle?  or
perhaps it can be generalized into a smatch or sparse rules for
standard error pattern recognition?  or perhaps inlined into the
kernel source some kind of dynamic test to test/identify the problem?

>
> 2) Schedule lock contention can cause a hard hang.
>
> On an SMP system kgdb for the x86 architecture single steps by running
> only a single core.  This is quite problematic if you have the schedule
> a lock held by a cpu which is in busy wait.  The system will deadlock on
> the single step operation from kgdb.  This problem is easily observed by
> doing on a 2 processor system by doing:
>
>   while [ 1 ] ; do find /  2> /dev/null > /dev/null; done &
>   while [ 1 ] ; do date > /dev/null ; done &
>   echo V1 > /sys/module/kgdbts/parameters/kgdbts
>
> For the first problem, I have a fix which is in the linux-next branch
> and will I will send a merge request to Linus to get it into the
> mainline tree.
>
> For the second problem, I am going to merge a change to release all the
> processors to run, at the expense of missing a breakpoint.   It is
> possible to change the behavior of this dynamically, for someone who
> might care about this behavior, until a longer term approach is
> implemented.  I have an experimental patch which implements the longer
> term approach of using displaced stepping.
>
> The experimental patch uses kprobes to manage software breakpoints.  The
> kprobe allows the breakpoint to remain planted while stepping around it
> by using out of line instruction execution, where you emulate the
> original instruction using memory elsewhere, followed by another trap
> instruction.
>
> Thanks,

thank you for the verbose explanation.......appreciate very much.
let me take some time to understand......

> Jason.
>
>> On Sat, Oct 17, 2009 at 1:43 AM, Peter Teoh <htmldeveloper@gmail.com> wrote:
>>
>>> Today, both my system (2.6.32.-rc4 from linus git tree and linux-next)
>>> bootup blocked indefinitely on:
>>>
>>> kgdb: Registered I/O driver kgdbts.
>>>
>>> while booting up.   The expected line:
>>>
>>> kgdb: Unregistered I/O driver kgdbts, debugger disabled.
>>>
>>> never comes up.
>>>
>>> My bootup menu.lst:
>>>
>>> title Fedora (2.6.26-rc4-next-20080530)
>>>        root (hd1,7)
>>>        kernel /boot/vmlinuz-2.6.26-rc4-next-20080530 ro
>>> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>>>        initrd /boot/initrd-2.6.26-rc4-next-20080530.img
>>> title Fedora (2.6.32-rc4)
>>>        root (hd1,7)
>>>        kernel /boot/vmlinuz-2.6.32-rc4 ro
>>> root=UUID=d10fe8db-e7d4-4b42-b265-0109a3f3eedf
>>>        initrd /boot/initrd-2.6.32-rc4.img
>>>
>>> and kgdb-related option:
>>>
>>> CONFIG_HAVE_ARCH_KGDB=y
>>> CONFIG_KGDB=y
>>> CONFIG_KGDB_SERIAL_CONSOLE=y
>>> CONFIG_KGDB_TESTS=y
>>> CONFIG_KGDB_TESTS_ON_BOOT=y
>>> CONFIG_KGDB_TESTS_BOOT_STRING="y"
>>>
>>> The same 2.6.32-rc4 image have bootup previously before without any
>>> problem.   So what could be the potential cause of this permanent
>>> wait?
>>>
>>> --
>>> Regards,
>>> Peter Teoh
>>>
>>>
>>
>>
>>
>>
>
>



-- 
Regards,
Peter Teoh

Re: booting up: blocking indefinitely on kgdb?

[Jason Wessel]

Peter Teoh wrote:
> On Mon, Oct 19, 2009 at 9:24 AM, Jason Wessel
> <jason.wessel@windriver.com> wrote:
>   
>> This is actually a real problem.  It is a race condition, and there are
>> actually two separate problems.
>>
>> 1) When a processor kernel thread is put into the single step state,
>> kgdb expects it to hit the single trap on the same processor the single
>> step request was made on.
>>
>>     
>
> sorry for being irrelevant....can i ask this:   even if the present
> CPU is in single step mode, all other CPU can be fully running and
> executing all the time, correct?  

It is not quite that simple.  The single step mode is a kernel task state.

When kgdb does a single step on the x86 architecture, the HW single step
bit is set in the active kernel task on CPU 2 for instance.   Then kgdb
starts just that CPU.   If an interrupt occurs or any kind of
preemption, is when the problem case arises.  This task may get
scheduled onto a different CPU at a later point, and dead lock ensues.


>  kgdb is not designed to handle more
> than one CPU in single step mode, right?   if wrong, then i supposed
> there must be a way to switch among processor, which i don't know how.
>   not sure if the same concept pertained to kdb?
>
>   

Kgdb will not single step more that one task at a time.  In kdb it has
the capability of switching CPUs, and in the kgdb+kdb merge branch I
implemented that functionality as well.  Either way it it still can only
single step one kernel thread at a time.

>> On an SMP system a process or kernel thread can migrate to another
>> processor after kgdb resumes.  This will result in a hard hang in the
>> cpu roundup part of kgdb.
>>     
>
> not sure if it is ok if i can know more about the reason for the hard
> hang (in slightly more detail).   The reason is because i am trying to
> understand if this same problem does exists in any other parts of the
> kernel?   eg, kdb? or anywhere in the suspend-resume cycle?  or
> perhaps it can be generalized into a smatch or sparse rules for
> standard error pattern recognition?  or perhaps inlined into the
> kernel source some kind of dynamic test to test/identify the problem?
>
>   

This particular problem does not exist anywhere else in the kernel.  It
is unique to the way kgdb deals with stopping and starting the system. 

In kernel/kgdb.c the key is in anything that touches the variable
"kgdb_cpu_doing_single_step".  It is up to each architecture that makes
use of kgdb to set/unset this variable.  The x86 arch sets it, and what
it does is not allow the other CPUs to run when single stepping.  If we
remove the set on the x86 arch, then you end up with the task migration
issue, so I was proposing putting in the fix to both issues, until a
displaced solution with kprobes or another implementation is completed.

You trade one problem for another of course with allowing the CPU's to run.

The original problem was a "hard hang".   The new problem is the
possibility of a missed break point.   For instance if you set a
breakpoint in a chunk of common code that can execute in parallel on two
different CPUs.  The breakpoint gets removed, the single step HW flag is
set, and if another CPU or task runs through that chunk of code, the
break point is missed.  My preference is to trade the hard hang away for
the time being.

Jason.