Re: [PATCH v4 0/3] Compile-time stack frame pointer validation

Ingo Molnar
2015-05-21T10:27:18Z

* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> On Wed, May 20, 2015 at 09:59:18AM -0700, Linus Torvalds wrote:
> > On Wed, May 20, 2015 at 9:25 AM, Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > > On Wed, May 20, 2015 at 09:03:37AM -0700, Andy Lutomirski wrote:
> > >>
> > >> I've never quite understood what the '?' means.
> > >
> > > It basically means "here's a function address we found on the
> > > stack, which may or may not have been called." It's needed
> > > because stack walking isn't currently 100% reliable.
> >
> > It is often quite interesting and helpful, because it shows stale
> > data on the stack, giving clues about what happened just before.
> >
> > Now, I'd like gcc to generally be better about not wasting so much
> > stack frame, so in that sense I'd like to see fewer '?" entries
> > just from a code quality standpoint, but when debugging those
> > things, the downside of "noise" is often cancelled by the upside
> > of "ahh, it happens after calling X".
> >
> > So the "perfect stack frames" is actually not as great a thing as
> > some people want to make it seem.
>
> Ok, I can see how looking at stale stack data could be useful for
> some of the really tough problems.

And note that the tough problems are actually the ones where we need
that information the most. So any stack backtrace printing method must
be biased towards helping the difficult scenarios - not the trivial
crashes. That is one of the reasons why we are always printing the
question marks.

> But right now, the meaning of '?' is ambiguous. It could be stale
> data, or it could be part of a frame for the current stack which was
> skipped due to missing frame pointers or an exception.

Yes, of course. That's not a big problem as the actual symbolic
information will tell us a lot, which allows us to reconstruct the
real call chain, plus allows us to see any 'recent execution activity'
that might be on the stack as stale entries.

> If we can somehow make the stack unwinder reliable, then it would at
> least allow us to remove the ambiguity of the '?' entries. And it
> would reduce the "noise" for the majority of issues where we don't
> care about stale stack data, and can simply ignore it.

Yes, but note the above consideration - the probability distribution
of kernel bugs tends to have a _very_ long tail, with bugs that
sometimes take years to trigger and fix. Kernel developers upstream
and at distros tend to spend a disproportionately large amount of time
staring at difficult to decode bugs.

For that reason it is far more important to still stay maintainable
with those kinds of difficult bugs, than to make the resolution of
trivial, unambiguous crashes a tiny bit easier by printing fewer
'distractions'...

Also, note that the '?' entries have another role: they cross-check
the unwinder.

If you think we'll be able to do a perfect unwinder then think again:
debug info _will_ be messed up periodically, either by us or by
tooling, because right now no kernel code or other functionality
relies on perfect unwinding.

So this is not like C++ exception handling where broken unwinding will
break the code. This is something that is literally only visible in
kernel logs currently, as a slight anomaly.

So any x86 stack unwinder code must be fundamentally based on the idea
and expectation that stack unwinding is always going to be somewhat
imperfect and somewhat statistical.

Thanks,

Ingo