Accessing Thread Information in kernel crash dumps with ddd+gdb

Accessing Thread Information in kernel crash dumps with ddd+gdb

[Piet Delaney]

Hey Guys:

I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel
as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed
with the visibility of local variables on the threads/tasks not currently
running on CPUs. Both crash, and the gdb macros that you guys wrote,
show the most important stuff but I'd prefer to be able to see everything
with gdb/ddd as I can with kgdb; including all local variables and formal
parameters at each stack frame.

A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
and $sp from the saved information in the U-block to view a process.
I wish gdb would allow be to run your macros, btt for example, and extract
the stackp from task.thread.esp assign it temporally to $sp for the 
current task,
do the backtrace command and see everything. Changing $sp and $fp for a 
while
like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to 
browse the
stack formals and locals would be nice. Just doing a 'set write on' 
isn't sufficient,
gdb wants a process and I can't see to satisfy it with simply setting 
the current
thread.

I was wondering if any of you guys have been thinking of anything like this
and had and hacks or ideas on how to see the locals and formals for all 
tasks.

One thought I had was a minor hack of the kexec code to do something 
like your gdb macros
and walk thru the task list and then append a ELF Notes, like done by 
crash_save_this_cpu(),
for each task. I have no idea if gdb has a limit on the number of 
elf_prstatus structures
that can be provided. I suppose I'd leave it a KEXEC config variable to 
enable this, as
some would argue that it's not as save as simply saving the regs for the 
active CPUs.
This would leave 'info threads' with gdb similar to 'ps' with crash and 
virtually identical
to the experience with kgdb.

-piet

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Accessing Thread Information in kernel crash dumps with ddd+gdb

[Piet Delaney]

[-- Attachment #1.1: Type: text/plain, Size: 2162 bytes --]

Ingo,

It appears you already did a


      thread-aware coredump

back in Oct 2002:

        http://lwn.net/Articles/12035/

Your patch appears to dump the registers for all task.
What happen with your idea? 

-piet


   

> Hey Guys:
>
> I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel
> as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed
> with the visibility of local variables on the threads/tasks not currently
> running on CPUs. Both crash, and the gdb macros that you guys wrote,
> show the most important stuff but I'd prefer to be able to see everything
> with gdb/ddd as I can with kgdb; including all local variables and formal
> parameters at each stack frame.
>
> A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
> and $sp from the saved information in the U-block to view a process.
> I wish gdb would allow be to run your macros, btt for example, and 
> extract
> the stackp from task.thread.esp assign it temporally to $sp for the 
> current task,
> do the backtrace command and see everything. Changing $sp and $fp for 
> a while
> like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to 
> browse the
> stack formals and locals would be nice. Just doing a 'set write on' 
> isn't sufficient,
> gdb wants a process and I can't see to satisfy it with simply setting 
> the current
> thread.
>
> I was wondering if any of you guys have been thinking of anything like 
> this
> and had and hacks or ideas on how to see the locals and formals for 
> all tasks.
>
> One thought I had was a minor hack of the kexec code to do something 
> like your gdb macros
> and walk thru the task list and then append a ELF Notes, like done by 
> crash_save_this_cpu(),
> for each task. I have no idea if gdb has a limit on the number of 
> elf_prstatus structures
> that can be provided. I suppose I'd leave it a KEXEC config variable 
> to enable this, as
> some would argue that it's not as save as simply saving the regs for 
> the active CPUs.
> This would leave 'info threads' with gdb similar to 'ps' with crash 
> and virtually identical
> to the experience with kgdb.
>
> -piet
>


[-- Attachment #1.2: Type: text/html, Size: 2807 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Accessing Thread Information in kernel crash dumps with ddd+gdb

[Vivek Goyal]

On Thu, Apr 17, 2008 at 05:16:55PM -0700, Piet Delaney wrote:
> Hey Guys:
> 
> I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel
> as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed
> with the visibility of local variables on the threads/tasks not currently
> running on CPUs. Both crash, and the gdb macros that you guys wrote,
> show the most important stuff but I'd prefer to be able to see everything
> with gdb/ddd as I can with kgdb; including all local variables and formal
> parameters at each stack frame.
> 
> A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
> and $sp from the saved information in the U-block to view a process.
> I wish gdb would allow be to run your macros, btt for example, and extract
> the stackp from task.thread.esp assign it temporally to $sp for the 
> current task,
> do the backtrace command and see everything. Changing $sp and $fp for a 
> while
> like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to 
> browse the
> stack formals and locals would be nice. Just doing a 'set write on' 
> isn't sufficient,
> gdb wants a process and I can't see to satisfy it with simply setting 
> the current
> thread.
> 
> I was wondering if any of you guys have been thinking of anything like this
> and had and hacks or ideas on how to see the locals and formals for all 
> tasks.
> 
> One thought I had was a minor hack of the kexec code to do something 
> like your gdb macros
> and walk thru the task list and then append a ELF Notes, like done by 
> crash_save_this_cpu(),
> for each task. I have no idea if gdb has a limit on the number of 
> elf_prstatus structures
> that can be provided. I suppose I'd leave it a KEXEC config variable to 
> enable this, as
> some would argue that it's not as save as simply saving the regs for the 
> active CPUs.
> This would leave 'info threads' with gdb similar to 'ps' with crash and 
> virtually identical
> to the experience with kgdb.

IIUC, you are suggesting that we create elf notes even for non-active
tasks in vmcore. We should not be doing that.

- It is not safe to traverse through task list after system has crashed.
- We reserve the memory for elf notes at system boot. At that time we
  have no idea how many task system will have at the time of crash.

I think following can be a way forward for your requirement.

- Either gdb should provide a SunOS kind of facility where one can 
  provide stack pointer and switch the task context. ( I don't know
  if there is already a way to do that).

- Or one can write a user space tool, which parses original vmcore,
  walks through task list, prepare elf notes for all the tasks and emit
  a new vmcore which is fetched to gdb.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Accessing Thread Information in kernel crash dumps with ddd+gdb

[Piet Delaney]

[-- Attachment #1.1: Type: text/plain, Size: 4561 bytes --]

Vivek Goyal wrote:

>On Thu, Apr 17, 2008 at 05:16:55PM -0700, Piet Delaney wrote:
>  
>
>>Hey Guys:
>>
>>I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel
>>as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed
>>with the visibility of local variables on the threads/tasks not currently
>>running on CPUs. Both crash, and the gdb macros that you guys wrote,
>>show the most important stuff but I'd prefer to be able to see everything
>>with gdb/ddd as I can with kgdb; including all local variables and formal
>>parameters at each stack frame.
>>
>>A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
>>and $sp from the saved information in the U-block to view a process.
>>I wish gdb would allow be to run your macros, btt for example, and extract
>>the stackp from task.thread.esp assign it temporally to $sp for the 
>>current task,
>>do the backtrace command and see everything. Changing $sp and $fp for a 
>>while
>>like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to 
>>browse the
>>stack formals and locals would be nice. Just doing a 'set write on' 
>>isn't sufficient,
>>gdb wants a process and I can't see to satisfy it with simply setting 
>>the current
>>thread.
>>
>>I was wondering if any of you guys have been thinking of anything like this
>>and had and hacks or ideas on how to see the locals and formals for all 
>>tasks.
>>
>>One thought I had was a minor hack of the kexec code to do something 
>>like your gdb macros
>>and walk thru the task list and then append a ELF Notes, like done by 
>>crash_save_this_cpu(),
>>for each task. I have no idea if gdb has a limit on the number of 
>>elf_prstatus structures
>>that can be provided. I suppose I'd leave it a KEXEC config variable to 
>>enable this, as
>>some would argue that it's not as save as simply saving the regs for the 
>>active CPUs.
>>This would leave 'info threads' with gdb similar to 'ps' with crash and 
>>virtually identical
>>to the experience with kgdb.
>>    
>>
>
>IIUC, you are suggesting that we create elf notes even for non-active
>tasks in vmcore.
>
Yep.

> We should not be doing that.
>
>- It is not safe to traverse through task list after system has crashed.
>  
>
I agree it's not 100% safe, but for many developers it's a risk
worth taking. For example we make a lot of changes in the
TCP/IP stack for implementing a proxy for filtering network
traffic. Virtually all bugs are ones we make in the networking
code and having a precise view of each stack would be helpful.
It's extremely unlikely that the task structures have been whacked
in our case; and likely for many other developers.

>- We reserve the memory for elf notes at system boot. At that time we
>  have no idea how many task system will have at the time of crash.
>  
>
How about reserving memory with each task structure for the
ELF notes? With Solaris I reserved a page for each CPU to be
able to store the register windows during a stack overflow so
I could map it in during the stack overflow. It's not unreasonable
to allocate the space with the task; especially if it's a kernel
config option. If no one used it we could remove it and go for
an alternate approach.

>I think following can be a way forward for your requirement.
>
>- Either gdb should provide a SunOS kind of facility where one can 
>  provide stack pointer and switch the task context. ( I don't know
>  if there is already a way to do that).
>  
>
I'll talk on the gdb mailing list about it. Perhaps we could
initially get it in as a developer maintance function to allow
the registers to be changed for a thread in a core dump.

>- Or one can write a user space tool, which parses original vmcore,
>  walks through task list, prepare elf notes for all the tasks and emit
>  a new vmcore which is fetched to gdb.
>  
>
Sounds difficult. I was talking to Dave Anderson about having
crash provide task info to his embedded gdb process but he
wasn't supportive of that approach.

Having a peace of code that has to walk thru the task list and
modify the elf notes seems a lot harder than just having the kernel
do it. More code that has to be maintained.

Isn't there precedence with other kernels like freebsd and Solaris
having the core dumps provide task information? If it was conditional
we could see just how many developers find the risk of causing a problem
during the crash dump a issue.

Looks to me like the size of the ELF notes is quite small and wouldn't
be a significant burden to add to the task structure.

-piet

>Thanks
>Vivek
>  
>


[-- Attachment #1.2: Type: text/html, Size: 5548 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Accessing Thread Information in kernel crash dumps with ddd+gdb

[Piet Delaney]

[-- Attachment #1.1: Type: text/plain, Size: 3354 bytes --]

Vivek Goyal wrote:

>On Thu, Apr 17, 2008 at 05:16:55PM -0700, Piet Delaney wrote:
>  
>

I was thinking over lunch that the largest risk to whacking
the task list might be a stack overflows. Perhaps a list of crash
notes could be maintained with the allocation/freeing of the
task structures. With the notes being so small it might not
be too bad just to allocate an array based on some configurable
maximum. Being able to dump say up to 1000 task notes would
be more that sufficient for our systems. The wasted memory
of allocating more memory than actually needed would be a
small cost to ensure well debugged kernels.

-piet

>>Hey Guys:
>>
>>I've been using kgdb for a while with our 2.6.12 and now 2.6.16 kernel
>>as well as kdump/kexec with our 2.6.16 kernel. I'm a bit disappointed
>>with the visibility of local variables on the threads/tasks not currently
>>running on CPUs. Both crash, and the gdb macros that you guys wrote,
>>show the most important stuff but I'd prefer to be able to see everything
>>with gdb/ddd as I can with kgdb; including all local variables and formal
>>parameters at each stack frame.
>>
>>A long time ago I used gdb on SunOS 4.1.4 and use to simply set $fp
>>and $sp from the saved information in the U-block to view a process.
>>I wish gdb would allow be to run your macros, btt for example, and extract
>>the stackp from task.thread.esp assign it temporally to $sp for the 
>>current task,
>>do the backtrace command and see everything. Changing $sp and $fp for a 
>>while
>>like I use to do with gdb on SunOS 4.1.4 and then using ddd+gdb to 
>>browse the
>>stack formals and locals would be nice. Just doing a 'set write on' 
>>isn't sufficient,
>>gdb wants a process and I can't see to satisfy it with simply setting 
>>the current
>>thread.
>>
>>I was wondering if any of you guys have been thinking of anything like this
>>and had and hacks or ideas on how to see the locals and formals for all 
>>tasks.
>>
>>One thought I had was a minor hack of the kexec code to do something 
>>like your gdb macros
>>and walk thru the task list and then append a ELF Notes, like done by 
>>crash_save_this_cpu(),
>>for each task. I have no idea if gdb has a limit on the number of 
>>elf_prstatus structures
>>that can be provided. I suppose I'd leave it a KEXEC config variable to 
>>enable this, as
>>some would argue that it's not as save as simply saving the regs for the 
>>active CPUs.
>>This would leave 'info threads' with gdb similar to 'ps' with crash and 
>>virtually identical
>>to the experience with kgdb.
>>    
>>
>
>IIUC, you are suggesting that we create elf notes even for non-active
>tasks in vmcore. We should not be doing that.
>
>- It is not safe to traverse through task list after system has crashed.
>- We reserve the memory for elf notes at system boot. At that time we
>  have no idea how many task system will have at the time of crash.
>
>I think following can be a way forward for your requirement.
>
>- Either gdb should provide a SunOS kind of facility where one can 
>  provide stack pointer and switch the task context. ( I don't know
>  if there is already a way to do that).
>
>- Or one can write a user space tool, which parses original vmcore,
>  walks through task list, prepare elf notes for all the tasks and emit
>  a new vmcore which is fetched to gdb.
>
>Thanks
>Vivek
>  
>


[-- Attachment #1.2: Type: text/html, Size: 3805 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Accessing Thread Information in kernel crash dumps with ddd+gdb

[Eric W. Biederman]

Piet Delaney <pdelaney@bluelane.com> writes:

> I agree it's not 100% safe, but for many developers it's a risk
> worth taking. For example we make a lot of changes in the
> TCP/IP stack for implementing a proxy for filtering network
> traffic. Virtually all bugs are ones we make in the networking
> code and having a precise view of each stack would be helpful.
> It's extremely unlikely that the task structures have been whacked
> in our case; and likely for many other developers.

Kernel crash dumps in development?
Relying on debuggers in kernel development?

There is something that smells horribly like the debug it until
it works version of code development instead of stopping and
understanding what the code does.

> How about reserving memory with each task structure for the
> ELF notes? 

Doesn't fly we need to know where the memory is before the kernel
crashes.  You have just required the task list walker you are
trying to avoid be present to gather up your ELF notes.  If you
need the walker anyway there is no point in generating the notes
a priori.

> With Solaris I reserved a page for each CPU to be
> able to store the register windows during a stack overflow so
> I could map it in during the stack overflow. It's not unreasonable
> to allocate the space with the task; especially if it's a kernel
> config option. If no one used it we could remove it and go for
> an alternate approach.

In some sense we have already been where you are suggesting.  Having
a nice crash dump facility in the kernel proved to fail to capture
crash dumps in real world failure scenarios.

>     I think following can be a way forward for your requirement.
>
>     - Either gdb should provide a SunOS kind of facility where one can
>       provide stack pointer and switch the task context. ( I don't know
>       if there is already a way to do that).
>
>
>
> I'll talk on the gdb mailing list about it. Perhaps we could
> initially get it in as a developer maintance function to allow
> the registers to be changed for a thread in a core dump.
>
>
>     - Or one can write a user space tool, which parses original vmcore,
>       walks through task list, prepare elf notes for all the tasks and emit
>       a new vmcore which is fetched to gdb.
>
>
>
> Sounds difficult. I was talking to Dave Anderson about having
> crash provide task info to his embedded gdb process but he
> wasn't supportive of that approach.
>
> Having a peace of code that has to walk thru the task list and
> modify the elf notes seems a lot harder than just having the kernel
> do it. More code that has to be maintained. 

Totally Wrong.

It is critical that the kexec on panic code path remains as small
as possible to maintain reliability.

Doing just about anything elsewhere is less work then on that code
path because we need to assume just about any part of the kernel
is busted and wrong.

Further there is no significant benefit (except possibly the code being
more visible to kernel developers) to doing the work in the kernel.
If that is a real concern placing a debug utility in the kernel source
that finds the task list and walks it should provide the same benefit.

Debuggers do have perfect information about the structure layouts,
as normal dwarf2 debugging information can record it.

So forget the notion of doing this on the kexec on panic code path
and see if you can make one of one of Vivek's ideas fly.  If nothing
is busted walking the task list is just a stupid linked list walk
easy, and not at all fatal if you get it wrong in user space, and
easy to fix.  A mysterious failure that takes weeks of work if you
are not perfectly paranoid in the kexec on panic code path.


Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec