frame unwinder patches

frame unwinder patches

[Bernd Schubert]

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, 
which gave beautiful stack traces. Unfortunately these frame unwinder patches
seem to have been dropped :(

Anyone still maintaining updated patches?

Here is an example of a horrible stack trace, confusing my eyes

[  855.504734] Call Trace:
[  855.504734]  [<ffffffff80495af2>] ? scsi_request_fn+0x252/0x370
[  855.504734]  [<ffffffff8058b068>] io_schedule+0x28/0x40
[  855.504734]  [<ffffffff8038b876>] get_request_wait+0x96/0x110
[  855.504734]  [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40
[  855.504734]  [<ffffffff8038bca8>] blk_get_request+0x38/0x80
[  855.504734]  [<ffffffff804950a3>] scsi_execute+0x43/0x140
[  855.504734]  [<ffffffffa00250dc>] :scsi_transport_spi:spi_execute+0x7c/0xd0
[  855.504734]  [<ffffffff8038accb>] ? blk_put_request+0x4b/0x60
[  855.504734]  [<ffffffff8049515f>] ? scsi_execute+0xff/0x140
[  855.504734]  [<ffffffffa0025b86>] :scsi_transport_spi:spi_dv_device_echo_buffer+0x1c6/0x290
[  855.504734]  [<ffffffffa00349f0>] ? :mptbase:mpt_timer_expired+0x0/0x70
[  855.504734]  [<ffffffffa0024dc3>] :scsi_transport_spi:spi_dv_retrain+0x63/0x240
[  855.504734]  [<ffffffffa00259c0>] ? :scsi_transport_spi:spi_dv_device_echo_buffer+0x0/0x290
[  855.504734]  [<ffffffffa00255f7>] :scsi_transport_spi:spi_dv_device+0x3b7/0x720
[  855.504734]  [<ffffffffa005017b>] :mptspi:mptspi_dv_device+0x6b/0x1a0
[  855.504734]  [<ffffffffa00502ce>] ? :mptspi:mptspi_dv_renegotiate_work+0x1e/0x100
[  855.504734]  [<ffffffffa005038f>] :mptspi:mptspi_dv_renegotiate_work+0xdf/0x100
[  855.504734]  [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100
[  855.504734]  [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100
[  855.504734]  [<ffffffff80245b36>] run_workqueue+0x176/0x220
[  855.504734]  [<ffffffff80246786>] worker_thread+0xa6/0x110
[  855.504734]  [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40
[  855.504734]  [<ffffffff802466e0>] ? worker_thread+0x0/0x110
[  855.504734]  [<ffffffff8024990d>] kthread+0x4d/0x80
[  855.504734]  [<ffffffff8020c5e8>] child_rip+0xa/0x12
[  855.504734]  [<ffffffff8020bcff>] ? restore_args+0x0/0x30
[  855.504734]  [<ffffffff802498c0>] ? kthread+0x0/0x80
[  855.504734]  [<ffffffff8020c5de>] ? child_rip+0x0/0x12
[  855.504734]

Probably all the "?" lines shouldn't be printed.


Thanks,
Bernd

Re: frame unwinder patches

[Arjan van de Ven]

On Thu, 04 Sep 2008 18:46:52 +0200
Bernd Schubert <bs@q-leap.de> wrote:

> Hello,
> 
> up to 2.6.22 the -mm series had these nice frame unwinder patches, 
> which gave beautiful stack traces. Unfortunately these frame unwinder
> patches seem to have been dropped :(
> 
> Anyone still maintaining updated patches?
> 
> Probably all the "?" lines shouldn't be printed.

printing the ? lines is the result of a compromise between getting the
most reliable information out of a stack trace, and getting a useful
data for the case where not everything is going as well as it should.

not printing the ? would be trivial (for some backtraces we do that),
but so far hasn't been seen as desirable because you lose information
in those hard-to-debug cases when you need anything you can get from
the backtrace.


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: frame unwinder patches

[Andi Kleen]

Bernd Schubert <bs@q-leap.de> writes:

> Hello,
>
> up to 2.6.22 the -mm series had these nice frame unwinder patches, 
> which gave beautiful stack traces. Unfortunately these frame unwinder patches
> seem to have been dropped :(
>
> Anyone still maintaining updated patches?

The SUSE kernel rpm should have an uptodate set

(ftp.suse.com/pub/projects/kernel/HEAD ) 

I agree that they were much better than the current unwinder,
also they didn't require slowing down the kernel with 
frame pointers.

-Andi 

Re: frame unwinder patches

[Bernd Schubert]

On Thursday 04 September 2008 19:29:36 Arjan van de Ven wrote:
> On Thu, 04 Sep 2008 18:46:52 +0200
>
> Bernd Schubert <bs@q-leap.de> wrote:
> > Hello,
> >
> > up to 2.6.22 the -mm series had these nice frame unwinder patches,
> > which gave beautiful stack traces. Unfortunately these frame unwinder
> > patches seem to have been dropped :(
> >
> > Anyone still maintaining updated patches?
> >
> > Probably all the "?" lines shouldn't be printed.
>
> printing the ? lines is the result of a compromise between getting the
> most reliable information out of a stack trace, and getting a useful
> data for the case where not everything is going as well as it should.
>
> not printing the ? would be trivial (for some backtraces we do that),
> but so far hasn't been seen as desirable because you lose information
> in those hard-to-debug cases when you need anything you can get from
> the backtrace.

I see your point, but in most cases it is rather annoying.


Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: frame unwinder patches

[Bernd Schubert]

On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
> Bernd Schubert <bs@q-leap.de> writes:
> > Hello,
> >
> > up to 2.6.22 the -mm series had these nice frame unwinder patches,
> > which gave beautiful stack traces. Unfortunately these frame unwinder
> > patches seem to have been dropped :(
> >
> > Anyone still maintaining updated patches?
>
> The SUSE kernel rpm should have an uptodate set
>
> (ftp.suse.com/pub/projects/kernel/HEAD )
>

Thanks Andi, found it! If someone else needs it, the correct path is 
ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD. 


> I agree that they were much better than the current unwinder,
> also they didn't require slowing down the kernel with
> frame pointers.

What it the reason this patch isn't mainline? It provides better traces, 
doesn't slow down the kernel (which is important in our HPC envirement), so 
what is the disadvantage?


Thanks again,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: frame unwinder patches

[Andi Kleen]

On Fri, Sep 05, 2008 at 11:24:16AM +0200, Bernd Schubert wrote:
> On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
> > Bernd Schubert <bs@q-leap.de> writes:
> > > Hello,
> > >
> > > up to 2.6.22 the -mm series had these nice frame unwinder patches,
> > > which gave beautiful stack traces. Unfortunately these frame unwinder
> > > patches seem to have been dropped :(
> > >
> > > Anyone still maintaining updated patches?
> >
> > The SUSE kernel rpm should have an uptodate set
> >
> > (ftp.suse.com/pub/projects/kernel/HEAD )
> >
> 
> Thanks Andi, found it! If someone else needs it, the correct path is 
> ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD. 

Thanks for the correction.

> > I agree that they were much better than the current unwinder,
> > also they didn't require slowing down the kernel with
> > frame pointers.
> 
> What it the reason this patch isn't mainline? It provides better traces, 
> doesn't slow down the kernel (which is important in our HPC envirement), so 
> what is the disadvantage?

The initial version that was merged had a few teething problems because
it ran into tool chain bugs (which were quickly resolved). Unfortunately 
it lead to a few people developing irrational fears of dwarf2 in the 
process and it was reverted, annoying the original contributor.
Right now someone would just need to retry the submission I guess.

-Andi

-- 
ak@linux.intel.com

Re: frame unwinder patches

[Arjan van de Ven]

On Fri, 5 Sep 2008 11:19:21 +0200
Bernd Schubert <bs@q-leap.de> wrote:

> > not printing the ? would be trivial (for some backtraces we do
> > that), but so far hasn't been seen as desirable because you lose
> > information in those hard-to-debug cases when you need anything you
> > can get from the backtrace.
> 
> I see your point, but in most cases it is rather annoying.

... if you see backtraces regularly there's something wrong ;-)

(and if you really care it's 1 line of code to turn it off)


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: frame unwinder patches

[Bernd Schubert]

On Friday 05 September 2008 15:33:17 Arjan van de Ven wrote:
> On Fri, 5 Sep 2008 11:19:21 +0200
>
> Bernd Schubert <bs@q-leap.de> wrote:
> > > not printing the ? would be trivial (for some backtraces we do
> > > that), but so far hasn't been seen as desirable because you lose
> > > information in those hard-to-debug cases when you need anything you
> > > can get from the backtrace.
> >
> > I see your point, but in most cases it is rather annoying.
>
> ... if you see backtraces regularly there's something wrong ;-)

It always depends on what you are doing. Presently I'm fixing the 
reset-handler of the mpt fusion driver and since I'm not 
$hardcore_kernel_hacker_with_super_scsi_knowlegde, but only know C and a 
little bit about the kernel API I insert dump_stack() and printks all over 
the place into the code to understand what is going on.

Well, this reset part is mostly done, now this problem: 
http://marc.info/?l=linux-scsi&m=118039573814323&w=2

So yes, there is something wrong ;) Unfortunately so far nobody cared about 
these bugs we are hitting regularily.

>
> (and if you really care it's 1 line of code to turn it off)

It is not only this, I think the dwarf2 stack unwinder patches provide by far 
better traces than the in-kernel unwinder. At least ever since I applied 
these patches to our kernels, I was able to read the stack dumps...


Cheers,
Bernd


-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: frame unwinder patches

[Arjan van de Ven]

On Fri, 5 Sep 2008 15:52:47 +0200
Bernd Schubert <bs@q-leap.de> wrote:
> >
> > (and if you really care it's 1 line of code to turn it off)
> 
> It is not only this, I think the dwarf2 stack unwinder patches
> provide by far better traces than the in-kernel unwinder. At least
> ever since I applied these patches to our kernels, I was able to read
> the stack dumps...
> 

they really wouldn't be different than the ones you get if you remove
the "?" lines.


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: frame unwinder patches

[Bernd Schubert]

On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:
> On Fri, 5 Sep 2008 15:52:47 +0200
>
> Bernd Schubert <bs@q-leap.de> wrote:
> > > (and if you really care it's 1 line of code to turn it off)
> >
> > It is not only this, I think the dwarf2 stack unwinder patches
> > provide by far better traces than the in-kernel unwinder. At least
> > ever since I applied these patches to our kernels, I was able to read
> > the stack dumps...
>
> they really wouldn't be different than the ones you get if you remove
> the "?" lines.

Well may be, but then there is still the performace degrading, so I don't want 
to have it enabled on our production kernels. I admit I never measured what 
is the difference between of CONFIG_FRAME_POINTER=y and =n, but the fact the 
help text says there is a difference already makes me want to disable it 
(especially, since we have to provide benchmarks before we can sell a 
system).


Cheers,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Re: frame unwinder patches

[Arjan van de Ven]

On Fri, 5 Sep 2008 16:48:52 +0200
Bernd Schubert <bs@q-leap.de> wrote:

> On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:
> > On Fri, 5 Sep 2008 15:52:47 +0200
> >
> > Bernd Schubert <bs@q-leap.de> wrote:
> > > > (and if you really care it's 1 line of code to turn it off)
> > >
> > > It is not only this, I think the dwarf2 stack unwinder patches
> > > provide by far better traces than the in-kernel unwinder. At least
> > > ever since I applied these patches to our kernels, I was able to
> > > read the stack dumps...
> >
> > they really wouldn't be different than the ones you get if you
> > remove the "?" lines.
> 
> Well may be, but then there is still the performace degrading, so I
> don't want to have it enabled on our production kernels. I admit I
> never measured what is the difference between of
> CONFIG_FRAME_POINTER=y and =n, but the fact the help text says there
> is a difference already makes me want to disable it (especially,
> since we have to provide benchmarks before we can sell a system).

to be honest, on 64 bit the overhead is quite small (the extra
instructions it adds are optimized for by the modern cpus that you use
in the systems you're selling); on 32 bit the overhead is.. well a
little bigger but not THAT much. yes it loses a register for the
compiler to use, but no it's not a general purpose register, and with
the register renaming that today's cpus do, I'd be surprised if you
could see anything significant.



-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: frame unwinder patches

[Andi Kleen]

> to be honest, on 64 bit the overhead is quite small (the extra
> instructions it adds are optimized for by the modern cpus that you use

The pipeline dependency stalls yes, the icache/decode overhead no.
Also CONFIG_FRAME_POINTER currently enables -fno-sibling-calls
which generates significantly worse code for a lot of common
kernel constructs.

> in the systems you're selling); on 32 bit the overhead is.. well a
> little bigger but not THAT much. yes it loses a register for the
> compiler to use, but no it's not a general purpose register, and with

%rbp is a general purpose register.  In fact it's even better
than a general purpose register because it has often shorter 
encoding than the other registers, but is as versatile.

> the register renaming that today's cpus do, I'd be surprised if you
> could see anything significant.

Even on modern CPUs you can measure it in macro benchmarks, at 
least that was the state last time that was investigated.
On older CPUs without the magic hardware it was even more
significant. Also there are even new CPUs like Atom which don't
have the magic hardware.

-Andi