Re: [PATCH] x86 NUMA panic compile error

[Ingo Molnar <mingo@elte.hu>]

* Andi Kleen <ak@suse.de> wrote:

> > Nevertheless for hard-to-debug bugs i prefer if they can be reproduced 
> > and debugged on 32-bit too, because x86_64 debugging is still quite a 
> > PITA and wastes alot of time: for example it has no support for exact 
> > kernel stacktraces.
> 
> Hopefully soon.

i've already implemented it for FRAME_POINTERS (i really needed it to 
not go insane when looking at lock validator output).

As you suggested a few weeks ago the real solution would be a dwarf 
parser. Maybe ia64's could be taken? Will post the patch for the 
FRAME_POINTERS solution soon. Sample output:

     [<ffffffff8020af4c>] __show_trace+0x3a/0x66
     [<ffffffff8020b36b>] show_trace+0x17/0x1a
     [<ffffffff80207e36>] show_regs+0x36/0x3c
     [<ffffffff80207e75>] smp_show_regs+0x39/0x52
     [<ffffffff8021559e>] smp_nmi_callback+0x6a/0x85
     [<ffffffff802163f2>] do_nmi+0x69/0x91
     [<ffffffff80601dca>] nmi+0x7e/0x85
     [<ffffffffffffffff>] 0xffffffffffffffff
     [<ffffffff80601ad0>] _spin_unlock_irqrestore+0x3e/0x47
     [<ffffffff8024424c>] prepare_to_wait+0x63/0x6d
     [<ffffffff8022ff0c>] do_syslog+0xf1/0x3ca
     [<ffffffff802ba3ed>] kmsg_read+0x3a/0x46
     [<ffffffff8027db72>] vfs_read+0xe6/0x191
     [<ffffffff8027e79d>] sys_read+0x44/0x82
     [<ffffffff80209b11>] system_call+0x7d/0x83
     [<ffffffffffffffff>] 0xffffffffffffffff

(and it works fine across irq/exception stacks too.)

> I think i386 only gained it very recently, so it can't be _that_ big a 
> problem.

i certainly used exact backtraces on i386 for many many years. Not sure 
whether those patches were all upstream though. It's also the 
combination of effects that makes the difference between i386 and x86_64 
so striking.

furthermore, the kernel's debugging infrastructure improved 
significantly, and we get more and more stackdumps to interpret [instead 
of hard to debug corruptions, etc.].

> The real issue is too deeply nested code like the callback hell we 
> have in some subsystems. Better would be to eliminate that. 2.4 was 
> much nicer in this regard and there has been quite a lot of 
> unnecessary complications in this area when the kernel went to 2.6.

i have no problems with interpreting occasional non-exact backtraces, 
but it is certainly non-obvious, and when you are looking at backtraces 
en masse, the unnecessary repetitive task can get really distracting and 
frustrating.

Exact backtraces on the other hand almost immediately create a unique 
and reliable "visual fingerprint", and if you have looked at enough of 
them, you almost recognize them just from looking at the shape of them. 
It's a completely different 'experience'. (and userspace developers will 
laugh out loud at us now i suspect ...)

> > Also, the printout of the backtrace is butt-ugly and as un-ergonomic 
> > to the human eye as it gets - who came up with that "two-maybe-one 
> > function entries per-line" nonsense? [Whoever did it he never had to 
> > look at (and make sense of) hundreds of stacktraces in a row.]
> 
> The original goal was to make it fit as much as possible on the screen 
> when you don't have a serial/net/fireconsole. But arguably it's less 
> and less useful because the kernel has gotten so huge that most 
> backtraces are very long and scroll away anyways.

yeah.

	Ingo