Re: DTrace-like analysis possible with future Linux kernels?

Robert Milkowski
2004-08-29T10:29:26Z

On Sat, 28 Aug 2004, Alan Cox wrote:

> On Llu, 2004-08-23 at 20:48, Robert Milkowski wrote:
>> Solaris runs on x86 platform, and runs quite well.
>> And guess what - DTrace runs on x86 like a charm.
>
> Larger x86 boxes. I can't seem to find PDA's with Solaris or phones
> with Solaris or $70 wireless routers with Solaris.

Yeah, I can agree with you that tools like DTrace aren't very usefull on
PDA or on phone. :)

>> I must admit I don't know OProfile.
>> But can you profile already running application without interuption
>
> Yes
>
>> What about getting structure contents, function arguments and returns,
>> etc... all on the fly.
>
> ptrace. Actually there are folks who want to take ptrace a bit further
> for some things - at least one vendor posted some proposals which when
> recast into ptrace extensions look good.
>
>> I think you missed the point.
>
> Nope
>
>> Sure, you can make your own module on Linux, load it and trace whatever
>> you want. But:
>
> Why do that, why not use the existing functionality that the kernel
> provides built on the stuff Intel AMD and friends stuck in the CPU. I'm
> not claiming our debugging tools are as good as dtrace but most of it
> (especially with kprobes patches installed) is essentially a UI design
> issue.

ok, so maybe real example.

Let's say you want aggregate by user stack if given thread number of
specified process is taken off cpu by scheduler and is in SLEEP state and
is off cpu for more then one second. You want output evey 10s. With DTrace
it's relly simple. Ususally if I want quick answer I'm gonna write like
this:

bash-2.05b# dtrace -n sched:::off-cpu'/curlwpsinfo->pr_state == SSLEEP &&
pid == 18819 && tid == 3/{self->t1 = timestamp;}'
-n sched:::on-cpu'/self->t1 && (timestamp - self->t1)
> 1000000000/{@[ustack()]=count();self->t1=0;}'
-n tick-10s'{printa(@);}'

But this is not so readable so let put it in a more readable form (script)

#!/usr/sbin/dtace -s

sched:::off-cpu
/curlwpsinfo->pr_state == SSLEEP && pid == 18819 && tid == 3/
{
{self->t1 = timestamp;
}

sched:::on-cpu
/self->t1 && (timestamp - self->t1) > 1000000000/
{
@[ustack()]=count();
self->t1 = 0;
}

tick-10s
{
printa(@);
}

Here is what you get:

25 34812 :tick-10s

libc.so.1`__pollsys+0x4
libc.so.1`poll+0x88
wpserver`wpio_loop_pool+0x9c
libc.so.1`_lwp_start
26

Which means you get 26 times the same stack.

Or maybe another example which shows how one can easly learn something
about apps behaviour and about system.

Let's say you see a lot of fspgin's in vmstat and want to know which
applications are cousing it. And to complicate things you have some
applications running from inted like daemon (so fork() + exec() every
request) and of course you want aggregate as a whole for such application.

So, with DTrace:

bash-2.05b# dtrace -n vminfo:::fspgin'{@[execname]=sum(arg0);}'
dtrace: description 'vminfo:::fspgin' matched 1 probe
^C

Application-A 282
Application-B 304
Application-C 335
zsched 1200
bash-2.05b#

Well, now yo can go further and see Application-C in more detail.
But instead let's say you don't know why zsched is cousing fspgins.
So lets' learn why.

bash-2.05b# dtrace -n vminfo:::fspgin'/execname == "zsched"/{@[stack()]=count();}'
dtrace: description 'vminfo:::fspgin' matched 1 probe
^C

genunix`pageio_setup+0x1f8
nfs`nfs3_readahead+0xc0
nfs`nfs_async_start+0x2c8
unix`thread_start+0x4
405
bash-2.05b#

Well, it's doing nfs3 read aheads.
You can disable read aheads for nfs3 and see if it disappears - and it
does. And all in a production without stopping anything.

Simple, easy and safe.

Now how much work and knowledge would it be needed to get the same results
with KProbes, oprofile, ... and how much time will it take?

And how easy would it be with KProbes to panick kernel?
All examples above I've just did on a production server.

DTrace lets you correlate data from kernel level and application level
which is really usuefull. It gives sys admins the power to answer to see
what is really happening in system and why.

And these are simple cases which do not show all DTrace features.
Real fun starts with more complicated examples :)

Of course you can do SOME things with DProbes or Oprofile that you
could with DTrace but usually it
will take MUCH more time with them then with DTrace. And there are things
you can do
with DTrace which you can't with DProbes and others in a reasonable time
period.

btw: about UI - it's really important. DTrace agreagations for example
save a lot of time in analyzing data. Especially when usually you
get huge amount of data to analyze from a production systems. With
'D' language you get most of data aggregation done during
collectioning data. I know there's Perl, Awk, etc. but it takes time.

btw2: you mention ptrace, AFAIK it would have more performance impact then
DTrace/KProbes technologies. Second, I'm not sure if it's still the
case, but there were (are?) some problems using ptrace on threaded
applications. And still all you see is application level - no
correlation between kernel and app (for example you want to see
if given code path in application is cousing fspgins or xcalls or
something else...)

btw3: and Oprofile, Kprobes, ptrace are separate tools and you have
then corelate data which will often be not possible.

--
Robert Milkowski
milek@rudy.mif.pg.gda.pl