Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

[Ingo Molnar <mingo@elte.hu>]

* Frank Ch. Eigler <fche@redhat.com> wrote:

> Hi -
> 
> On Sat, Mar 21, 2009 at 04:45:01PM +0100, Ingo Molnar wrote:
> > [...]
> > To me personally there are two big direct usability issues with 
> > SystemTap:
> > 
> >  1) It relies on DEBUG_INFO for any reasonable level of utility.
> >     Yes, it will limp along otherwise as well, but most of the
> >     actual novel capabilities depend on debuginfo. Which is an
> >     acceptable constraint for enterprise usage where kernels are
> >     switched every few months and having a debuginfo package is not
> >     a big issue. Not acceptable for upstream kernel development. 
> 
> In my own limited kernel-building experience, I find the debuginfo 
> data conveniently and instantly available after every "make".  Can 
> you elaborate how is it harder for you to incidentally make it 
> than for someone to download it?

Four reasons:

1)

I have CONFIG_DEBUG_INFO turned off in 99.9% of my kernel builds, 
because it slows down the kernel build times significantly:

  without:   4343.31 user 416.39 system 6:09.97 elapsed 1286%CPU 
  with:      4871.07 user 501.90 system 7:43.22 elapsed 1159 %CPU 

( x86 allyesconfig. On an obscenely overpowered Nehalem box
  with 12 GB of RAM. )

2)

When the kernel build becomes IO-bound, for example when i build 
over a distcc cluster (which is how i generally build my kernels) - 
or when others with less RAM build a debuginfo kernel, the ratio 
becomes even worse:

  without:   870.36 user 292.79 system 3:32.10 elapsed  548% CPU
  with:      929.65 user 384.55 system 8:28.70 elapsed  258% CPU

3)

Another metric. Here's an x86 defconfig (i.e. fairly regular config 
- not allyesconfig) build's size:

  with:     1645 MB
  without:   211 MB

Try to build 1.6 GB of dirty data on ext3 and run into an fsync() in 
your editor ... you'll sit there twiddling thumbs for a minute or 
more.

4)

Or yet another metric - Linux distro package overhead. Try 
installing a debuginfo package:

 # yum install kernel-debuginfo

 ==========================================
  Package                  Arch    Version
 ==========================================
 Installing:
  kernel-debuginfo         x86_64  2.6.29-0.258.rc8.git2.fc11   
 rawhide-debuginfo  294 M
 Installing for dependencies:
  kernel-debuginfo-common  x86_64  2.6.29-0.258.rc8.git2.fc11   
 rawhide-debuginfo   35 M

 Total download size: 329 M

That size of a _compressed_ debuginfo kernel package is obscene. We 
can fit 4 years of full Linux kernel Git history into that size - 
60,000+ commits, full metadata and full 20 million lines of code 
flux included!

Uncompressed it blows up to gigabytes of on-disk data.

And this download has to be repeated for _every_ minor kernel 
upgrade.

So when i come into a situation where i could use some debugging 
help ... i'd have to rebuild the kernel with DEBUG_INFO=y and i'll 
always notice when i have a debuginfo kernel because it's 
inconvenient.

The solution?)

Dunno - but i definitely think we should think bigger:

The fundamental disconnect i believe seems to come from the fact 
that most user-space projects are relatively small, so debuginfo 
bloat is a secondary issue there.

But for a project with the size of the kernel, even for moderate 
builds (not allyesconfig), it's a _much_ bigger deal. This has been 
known for a long time and the situation has become worse over the 
last two years, not better. (last time i checked the debuginfo 
package overhead it was below 150 MB)

A few random ideas:

Instead of trying to cache 2+GB of debuginfo for a 50 MB kernel 
source repo (+50 MB of genuine .o output) - just to be able to debug 
one or two source files [which is the typical scope of a debugging 
session], why not build debuginfo on the fly, when a debugging 
session requires it? Rarely do we need debuginfo for more than a 
fraction of the whole kernel.

( Yes, it needs a few smarts like knowing the SHA1 of the source
  code module that a particular kernel portion got built with, to 
  make sure the debuginfo is fresh and relevant - but nothing major. )

I mean, lets _use_ the fact that we have source code available, more 
intelligently. It takes zero time to build detailed debuginfo for a 
portion of a tree.

If 'download debuginfo' can be replaced with: 'have a recent Git 
repository of the distro kernel source', we'll have a _much_ more 
efficient use of resources all around.

	Ingo