Re: KVM usability

Re: KVM usability

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> Here's our experience with tools/perf/. Hosting the project in the kernel 
> proper helped its quality immensely:
> 
>  - It's much easier to synchronize new features on the kernel side and on the
>    user-space side. The two go hand in hand - they are often implemented in
>    the same patch.

Just look at an example from today, a perf+KVM feature patch posted by Yanmin 
Zhang:

  http://www.mail-archive.com/kvm@vger.kernel.org/msg29770.html

That single patch implements the following "perf kvm" commands:

  perf kvm top
  perf kvm record
  perf kvm report
  perf kvm diff

Both the kernel-space and the user-space changes are in that single patch.

Anyone who'd like to try it out can apply it and get an updated kernel plus 
updated tooling and can start profiling KVM guests straight away. You just 
check out the kernel, apply the patch and that's it - you can go. It doesnt 
get any more convenient than that to do development.

Such kind of a unified repository is a powerful concept, and we make use of 
those aspects of tools/perf/ every day. You could only pry it out of our cold, 
dead fingers ;-)

Btw., this is one of the things that FreeBSD does right - and i believe it is 
one of the technical concepts behind Apple's success as well. Apple, with a 
tenth's of Linux's effective R&D budget can consistently out-develop Linux. I 
think that's in part due to there not being a strict chinese wall between the 
Apple kernel, libraries and applications - it's one coherent project where 
everyone is well-connected to each piece, with no artificial project-cultural 
boundaries and barriers. People can and do move between those areas of the 
larger "Apple" project to achieve their goals - regardless of how many 
components need touching for a given area of interest.

IMHO we should learn from that - while we are good in many areas there's 
always aspects of Linux that can be improved. But i digress.

Thanks,

	Ingo

Re: KVM usability

[Avi Kivity]

On 03/02/2010 12:30 PM, Ingo Molnar wrote:
> * Ingo Molnar<mingo@elte.hu>  wrote:
>
>    
>> Here's our experience with tools/perf/. Hosting the project in the kernel
>> proper helped its quality immensely:
>>
>>   - It's much easier to synchronize new features on the kernel side and on the
>>     user-space side. The two go hand in hand - they are often implemented in
>>     the same patch.
>>      
> Just look at an example from today, a perf+KVM feature patch posted by Yanmin
> Zhang:
>
>    http://www.mail-archive.com/kvm@vger.kernel.org/msg29770.html
>
> That single patch implements the following "perf kvm" commands:
>
>    perf kvm top
>    perf kvm record
>    perf kvm report
>    perf kvm diff
>
> Both the kernel-space and the user-space changes are in that single patch.
>
> Anyone who'd like to try it out can apply it and get an updated kernel plus
> updated tooling and can start profiling KVM guests straight away. You just
> check out the kernel, apply the patch and that's it - you can go. It doesnt
> get any more convenient than that to do development.
>
> Such kind of a unified repository is a powerful concept, and we make use of
> those aspects of tools/perf/ every day. You could only pry it out of our cold,
> dead fingers ;-)
>    

perf really is wonderful, but to be really competitive, and usable to 
more developers, it needs to be in a graphical environment.  I want 
'perf report' output to start out collapsed and drill down by clicking 
on a tree widget.  Clicking on a function name opens its definition.  
'perf annotate' should display annotations on my editor window, not in a 
pager.  I should be able to check events on a list, not using 'perf list'.

Is something like that suitable for tools/perf/?  I think you'll find 
the intersection of kernel developers and GUI developers to be fairly small.

> Btw., this is one of the things that FreeBSD does right - and i believe it is
> one of the technical concepts behind Apple's success as well. Apple, with a
> tenth's of Linux's effective R&D budget can consistently out-develop Linux. I
> think that's in part due to there not being a strict chinese wall between the
> Apple kernel, libraries and applications - it's one coherent project where
> everyone is well-connected to each piece, with no artificial project-cultural
> boundaries and barriers. People can and do move between those areas of the
> larger "Apple" project to achieve their goals - regardless of how many
> components need touching for a given area of interest.
>
> IMHO we should learn from that - while we are good in many areas there's
> always aspects of Linux that can be improved. But i digress.
>    

Folding everything into the kernel tree is one way to approach it; IMO 
it is completely unreasonable.  The kernel is a very small part of a 
complete system.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Pekka Enberg]

Hi Avi,

(slightly off-topic)

On Sun, Mar 7, 2010 at 11:35 AM, Avi Kivity <avi@redhat.com> wrote:
> perf really is wonderful, but to be really competitive, and usable to more
> developers, it needs to be in a graphical environment.  I want 'perf report'
> output to start out collapsed and drill down by clicking on a tree widget.
>  Clicking on a function name opens its definition.  'perf annotate' should
> display annotations on my editor window, not in a pager.  I should be able
> to check events on a list, not using 'perf list'.

People keep bringing this up but I don't quite agree. Mac OS X has
"shark" which is pretty much what you describe above. However, having
used both, I prefer perf's simple UI for two reasons: it's much easier
to automate perf commands and text-based reports are superior for
sharing results (and keeping track of results when doing performance
optimizations).

That said, AFAICT, it should be pretty simple to implement a
shark-like UI with GTK as current perf code is pretty good fit for
that. I've pondered about doing that myself but quite frankly, I don't
see any big gains in that.

                        Pekka

Re: KVM usability

[Avi Kivity]

On 03/07/2010 11:56 AM, Pekka Enberg wrote:
> Hi Avi,
>
> (slightly off-topic)
>
> On Sun, Mar 7, 2010 at 11:35 AM, Avi Kivity<avi@redhat.com>  wrote:
>    
>> perf really is wonderful, but to be really competitive, and usable to more
>> developers, it needs to be in a graphical environment.  I want 'perf report'
>> output to start out collapsed and drill down by clicking on a tree widget.
>>   Clicking on a function name opens its definition.  'perf annotate' should
>> display annotations on my editor window, not in a pager.  I should be able
>> to check events on a list, not using 'perf list'.
>>      
> People keep bringing this up but I don't quite agree. Mac OS X has
> "shark" which is pretty much what you describe above. However, having
> used both, I prefer perf's simple UI for two reasons: it's much easier
> to automate perf commands and text-based reports are superior for
> sharing results (and keeping track of results when doing performance
> optimizations).
>    

Yes, you (and me as well).  But most people are quite unlike new and 
me.  There's a reason GUIs dominate today, and there are even a few 
kernel developers that don't use mutt for reading email.

Even for command-line-happy people, GUIs still have an advantage in that 
it is much easier to discover features by exploring the UI vs. reading 
manual pages.

> That said, AFAICT, it should be pretty simple to implement a
> shark-like UI with GTK as current perf code is pretty good fit for
> that. I've pondered about doing that myself but quite frankly, I don't
> see any big gains in that.
>    

Because you are only interested in your own itches (which is perfectly 
legitimate, but will keep perf's userbase down).

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Luca Barbieri]

> perf really is wonderful, but to be really competitive, and usable to more
> developers, it needs to be in a graphical environment.  I want 'perf report'
> output to start out collapsed and drill down by clicking on a tree widget.
>  Clicking on a function name opens its definition.  'perf annotate' should
> display annotations on my editor window, not in a pager.  I should be able
> to check events on a list, not using 'perf list'.
>
> Is something like that suitable for tools/perf/?  I think you'll find the
> intersection of kernel developers and GUI developers to be fairly small.

The latest versions of Gnome Sysprof use perf and provide a GTK+ tree
interface for the profiling output.

However, they are not configurable at all and don't support anything
but call graph profiling, unless they added more features very
recently.
It would be nice to extend sysprof into a more capable tool, and one
that can read perf output files and do so when launched from the
command line.

Re: KVM usability

[Arnaldo Carvalho de Melo]

Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
> perf really is wonderful, but to be really competitive, and usable to  
> more developers, it needs to be in a graphical environment.  I want  
> 'perf report' output to start out collapsed and drill down by clicking  
> on a tree widget.  Clicking on a function name opens its definition.   
> 'perf annotate' should display annotations on my editor window, not in a  
> pager.  I should be able to check events on a list, not using 'perf 
> list'.

Do you really think that more kernel developers would use perf more
frequently if it had some GUI?

I plan to work on a ncurses tool combining aspects of the existing perf
tools, integrating them more, like you suggest above, but even having
worked on a pygtk tool that is close to the kernel [1], I'm unsure if
doing it using gtk or QT would be something that would entice more
developers to use it.

- Arnaldo

[1] http://www.osadl.org/Single-View.111+M52212cb1379.0.html

Re: KVM usability

[Avi Kivity]

On 03/07/2010 08:01 PM, Arnaldo Carvalho de Melo wrote:
> Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
>    
>> perf really is wonderful, but to be really competitive, and usable to
>> more developers, it needs to be in a graphical environment.  I want
>> 'perf report' output to start out collapsed and drill down by clicking
>> on a tree widget.  Clicking on a function name opens its definition.
>> 'perf annotate' should display annotations on my editor window, not in a
>> pager.  I should be able to check events on a list, not using 'perf
>> list'.
>>      
> Do you really think that more kernel developers would use perf more
> frequently if it had some GUI?
>    

Not much.  Is perf's target kernel developers exclusively?  Who are we 
writing this kernel for?

No wonder everything is benchmarked using kbuild.

> I plan to work on a ncurses tool combining aspects of the existing perf
> tools, integrating them more, like you suggest above, but even having
> worked on a pygtk tool that is close to the kernel [1], I'm unsure if
> doing it using gtk or QT would be something that would entice more
> developers to use it.
>    

Even for kernel developers there are advantages in a GUI, namely that 
features are easily discovered, the amount of information is easily 
controlled, and in that you can interact (not redo everything from 
scratch every time you want to change something).  The difference 
between a curses based tool and a true GUI are minimal for this audience.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Avi Kivity]

On 03/07/2010 05:14 PM, Luca Barbieri wrote:
>> perf really is wonderful, but to be really competitive, and usable to more
>> developers, it needs to be in a graphical environment.  I want 'perf report'
>> output to start out collapsed and drill down by clicking on a tree widget.
>>   Clicking on a function name opens its definition.  'perf annotate' should
>> display annotations on my editor window, not in a pager.  I should be able
>> to check events on a list, not using 'perf list'.
>>
>> Is something like that suitable for tools/perf/?  I think you'll find the
>> intersection of kernel developers and GUI developers to be fairly small.
>>      
> The latest versions of Gnome Sysprof use perf and provide a GTK+ tree
> interface for the profiling output.
>
> However, they are not configurable at all and don't support anything
> but call graph profiling, unless they added more features very
> recently.
> It would be nice to extend sysprof into a more capable tool, and one
> that can read perf output files and do so when launched from the
> command line.
>    

Looks like a step in the right direction.  I don't think this belong in 
tools/, though.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Ingo Molnar]

* Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> That said, AFAICT, it should be pretty simple to implement a shark-like UI 
> with GTK as current perf code is pretty good fit for that. I've pondered 
> about doing that myself but quite frankly, I don't see any big gains in 
> that.

There's a perf events based GUI: sysprof.

It's not integrated into tools/perf/ (it's using the syscall) and we'd welcome 
it integrated as 'perf view' or 'perf gui' - just like we'd welcome a new 
from-scratch GUI frontend as well.

The latter would have the advantage of growing gradually and naturally - so if 
you feel like implementing it, please do not hesitate ;-) I had a good hard 
look at the perf GUI situation a few months ago and GTK looks like the best 
fit.

Even if it's simple initially it would clearly show us the exact demand of GUI 
versus TUI. I.e. if you are right and it doesnt matter, it will stay simple, 
if Avi is right and it matters then people will use it as a base and improve 
it. A win-win scenario.

Thanks,

	Ingo

Re: KVM usability

[Arnaldo Carvalho de Melo]

Em Sun, Mar 07, 2010 at 08:15:40PM +0200, Avi Kivity escreveu:
> On 03/07/2010 08:01 PM, Arnaldo Carvalho de Melo wrote:
> >Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
> >>perf really is wonderful, but to be really competitive, and usable to
> >>more developers, it needs to be in a graphical environment.  I want
> >>'perf report' output to start out collapsed and drill down by clicking
> >>on a tree widget.  Clicking on a function name opens its definition.
> >>'perf annotate' should display annotations on my editor window, not in a
> >>pager.  I should be able to check events on a list, not using 'perf
> >>list'.
> >Do you really think that more kernel developers would use perf more
> >frequently if it had some GUI?
> 
> Not much.  Is perf's target kernel developers exclusively?  Who are
> we writing this kernel for?

No, we aren't writing this tool only for kernel developers exclusively,
but that wasn't my question, it was badly formulated, sorry, I shouldn't
have included "kernel" in it :-\
 
> No wonder everything is benchmarked using kbuild.
> 
> >I plan to work on a ncurses tool combining aspects of the existing perf
> >tools, integrating them more, like you suggest above, but even having
> >worked on a pygtk tool that is close to the kernel [1], I'm unsure if
> >doing it using gtk or QT would be something that would entice more
> >developers to use it.
> 
> Even for kernel developers there are advantages in a GUI, namely
> that features are easily discovered, the amount of information is
> easily controlled, and in that you can interact (not redo everything
> from scratch every time you want to change something).  The
> difference between a curses based tool and a true GUI are minimal
> for this audience.

Ok, I agree with you about easier discoverability of features, path
shortened from report to annotate to starting the editor right at the
line where some event of interest happened, will try to keep the
routines not much coupled with ncurses, but definetely ncurses will be
the first step.

- Arnaldo

Re: KVM usability

[Avi Kivity]

On 03/07/2010 08:53 PM, Arnaldo Carvalho de Melo wrote:
>>> Do you really think that more kernel developers would use perf more
>>> frequently if it had some GUI?
>>>        
>> Not much.  Is perf's target kernel developers exclusively?  Who are
>> we writing this kernel for?
>>      
> No, we aren't writing this tool only for kernel developers exclusively,
> but that wasn't my question, it was badly formulated, sorry, I shouldn't
> have included "kernel" in it :-\
>    

In this case, I will reformulate my answer.  Very much.

>> Even for kernel developers there are advantages in a GUI, namely
>> that features are easily discovered, the amount of information is
>> easily controlled, and in that you can interact (not redo everything
>> from scratch every time you want to change something).  The
>> difference between a curses based tool and a true GUI are minimal
>> for this audience.
>>      
> Ok, I agree with you about easier discoverability of features, path
> shortened from report to annotate to starting the editor right at the
> line where some event of interest happened,

Another path is browse some function, start profiling, see perf data 
fill up in the margin.  Or, jump to callers.  etc.  You need an 
integrated browser for that (or an emacs perf mode).

> will try to keep the
> routines not much coupled with ncurses, but definetely ncurses will be
> the first step.
>    

Great.  ncurses is certainly much easier to experiment with and will 
likely provide useful experience.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.