Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[Soeren Sandmann]

* What is it?
--------------------------

Sysprof is a sampling system-wide CPU profiler for Linux.

Sysprof uses a Linux kernel module to profile the entire system, not
just an individual application. 

Of course, sysprof handles threads and shared libraries, and
applications do not have to be recompiled or instrumented. In fact
they don't even have to be restarted. 

Just insert the kernel module and start sysprof.


* Features:
----------------------

    - Profiles all running processes, not just a single application

    - Call graph support showing time spent in each branch of the call tree

    - Has a simple graphical interface

    - Profiles can be loaded and saved

    - Easy to use: Just insert the kernel module and start sysprof

    - Supports Fedora debuginfo packages


* Sysprof 1.0
--------------------------

This is the first release of sysprof with a 1.x version number. I am
very excited about that.

New features since Sysprof 0.91:

- support for x86-64, thanks to Mike Frysinger
- better reporting of time spent in the kernel

Please report bugs as well as success or failure stories to 

    sandmann@daimi au dk


* Where can I get it?
--------------------------------------------

Home page:

       http://www.daimi.au.dk/~sandmann/sysprof/

Source code:

       http://www.daimi.au.dk/~sandmann/sysprof/sysprof-1.0.tar.gz

Sysprof requires

        - Linux 2.6.11 or later, compiled with profiling support
        - GTK+ 2.6
	- libglade 2.5

It is known to work out the box on Fedora Core 4.


Søren

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[John Levon]

On Sat, Sep 17, 2005 at 11:05:04PM +0200, Soeren Sandmann wrote:

> Sysprof is a sampling system-wide CPU profiler for Linux.

I didn't really understand you last time, so I may as well ask again in
public: what is the point in you re-implementing OProfile?

regards,
john

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[bert hubert]

On Sat, Sep 17, 2005 at 11:05:04PM +0200, Soeren Sandmann wrote:
> * What is it?
> --------------------------
> 
> Sysprof is a sampling system-wide CPU profiler for Linux.
> 
> Sysprof uses a Linux kernel module to profile the entire system, not
> just an individual application. 

How is this different from oprofile? 

Looks like you did an exact, but less capable, reimplementation.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[Soeren Sandmann]

bert hubert <bert.hubert@netherlabs.nl> writes:

> How is this different from oprofile? 
> 
> Looks like you did an exact, but less capable, reimplementation.

The kernel module is indeed less capable by itself, but the userspace
tools are a lot easier to use and show you significantly more
details. Oprofile may have improved since then, but when I started
sysprof:

- oprofile didn't work on anything but smp kernels

- oprofile could not produce callgraph information without not only
  recompiling my kernel, but also patching the kernel

- oprofile had a user interface that I simply didn't understand. As I
  remember it, I had to know about performance counters and how to
  translate those into binary masks

Contrast with sysprof, where you

        - insert the module
        - hit the start button
        - do the thing you want to profile
        - hit the profile button

and you get data presented in a way that is just a whole lot more
useful than the flat text files generated by oprofile.



Søren

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[John Levon]

On Sun, Sep 18, 2005 at 12:05:15AM +0200, Soeren Sandmann wrote:

> details. Oprofile may have improved since then, but when I started
> sysprof:
> 
> - oprofile didn't work on anything but smp kernels
> 
> - oprofile could not produce callgraph information without not only
>   recompiling my kernel, but also patching the kernel
> 
> - oprofile had a user interface that I simply didn't understand. As I
>   remember it, I had to know about performance counters and how to
>   translate those into binary masks

This is what you said first time I asked you, none of which explain why
you're continuing this duplication of effort, since *none* of the above
are true any more (and the first /never/ was).

> and you get data presented in a way that is just a whole lot more
> useful than the flat text files generated by oprofile.

Building a GUI around OProfile would have been welcome, but you've
chosen to re-implement the entire stack...

regards,
john

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[Soeren Sandmann]

John Levon <levon@movementarian.org> writes:

> > and you get data presented in a way that is just a whole lot more
> > useful than the flat text files generated by oprofile.
> 
> Building a GUI around OProfile would have been welcome, but you've
> chosen to re-implement the entire stack...

The only thing I have reimplemented is a kernel module. If you remove
all the #if 0'd lines in sysprof-module.c you are left with 296 lines
of code. So it's hardly a reimplementation of the entire stack.

And in fact I don't care all that much about the kernel module. The
things I really care about, the gui and the analysis code, do not
really have an equivalent in oprofile, and those two things are the
meat of the sysprof package:

      [ssp@localhost sysprof]$ cat *.[ch] | wc -l
      6706

My motive is not to duplicate oprofile - I basically don't care about
the kernel level mechanism as long as it can produce stack traces that
the GUI can interprete and analyse. In fact, one of the first times I
wrote about sysprof publicly [1], I said:

      It seems to me that since oprofile probably reports more and
      better data than my kernel module, we should try and get the
      graphical presentation from sysprof to present oprofile data.

and I still think so, but it's a fairly substantial amount of work to
get rid of 296 lines of code. 


Soren



[1] http://www.redhat.com/archives/fedora-desktop-list/2004-May/msg00041.html

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[Anton Blanchard]

 
Hi,

> My motive is not to duplicate oprofile - I basically don't care about
> the kernel level mechanism as long as it can produce stack traces that
> the GUI can interprete and analyse. In fact, one of the first times I
> wrote about sysprof publicly [1], I said:
> 
>       It seems to me that since oprofile probably reports more and
>       better data than my kernel module, we should try and get the
>       graphical presentation from sysprof to present oprofile data.
> 
> and I still think so, but it's a fairly substantial amount of work to
> get rid of 296 lines of code. 

Looking at your kernel module it should be pretty easy to plug on top of
oprofile callgraph data. All the hard work of programming hardware
performance counters will then come for free. Cross platform support
will also come for free.

Anton

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[Tomasz Torcz]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Sat, Sep 17, 2005 at 11:20:15PM +0100, John Levon wrote:
> > and you get data presented in a way that is just a whole lot more
> > useful than the flat text files generated by oprofile.
> 
> Building a GUI around OProfile would have been welcome, but you've
> chosen to re-implement the entire stack...

 Sysprof seems to profile userspace simultanous, while oprofile is
bounded to kernel calls.
 
-- 
Tomasz Torcz                                                       72->|   80->|
zdzichu@irc.-nie.spam-.pl                                          72->|   80->|


[-- Attachment #2: Type: application/pgp-signature, Size: 229 bytes --]

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[John Levon]

On Sun, Sep 18, 2005 at 01:28:10AM +0200, Tomasz Torcz wrote:

> > Building a GUI around OProfile would have been welcome, but you've
> > chosen to re-implement the entire stack...
> 
>  Sysprof seems to profile userspace simultanous, while oprofile is
> bounded to kernel calls.

I don't know how you got this idea, but it's completely wrong.

john

Re: Announce: Sysprof 1.0 -- a sampling, systemwide Linux profiler

[bert hubert]

> - oprofile had a user interface that I simply didn't understand. As I
>   remember it, I had to know about performance counters and how to
>   translate those into binary masks

This all hasn't been true for a very long while.

> Contrast with sysprof, where you
>         - insert the module

modprobe oprofile

>         - hit the start button

opcontrol --start --vmlinux=/boot/vmlinux

>         - do the thing you want to profile
>         - hit the profile button

opcontrol --stop
opreport -l

> and you get data presented in a way that is just a whole lot more
> useful than the flat text files generated by oprofile.

Not *that* may be very true, and your GUI indeed looks very very promising.
Elsewhere you note regarding removing your sysprof.o code:

> and I still think so, but it's a fairly substantial amount of work to
> get rid of 296 lines of code.

You'll find this work orders of magnitude easier than convincing people that
we need yet another profiling system in the kernel.

So I urge you, keep the cool gui but adapt it to oprofile. I'm sure John
will be more than willing to work with you to make sure oprofile
supports/will support everything you need.

Ok?

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services