Perfctr-Xen framework for permonace analysis

Perfctr-Xen framework for permonace analysis

[Ruslan Nikolaev]

Hi

I want to make an announcement about new perfomance monitoring framework.

Perfctr-Xen framework that enables per-thread performance analysis in Xen. Current version is capable of properly virtualizing counters in both paravirtualized and HVM modes. It is based on perfctr (which is a library and kernel module for non-virtualized guests), ported to Xen, and extended to work properly in virtualized environment. Both accumulative and interrupt modes counting (profiling) are supported.

The advantage of Perfctr-Xen is that it does not require specific HVM extensions which are needed for vpmu driver, can work in paravirtualized mode, and it also quite universal: works with many common tools such as PAPI, HPCToolkit, TAU PerfExplorer. It supports proper per-domain and per-thread virtualization. It is light-weight, supports wide range of CPUs, does not require save-and-restore for accumulative mode of counting (it uses counter offsetting), avoids expensive hypercalls and counter re-programming in certain circumstances (when threads are counting the same type of events). In addition, some techniques are employed to account for the overhead caused by the framework itself. This makes measurements quite accurate.

Perfctr-Xen consists of series of patches that need to be applied to Xen, Linux, perfctr. There are available at:
http://people.cs.vt.edu/~rnikola/

The code is available under LGPL. It would be great to discuss if and how it can be integrated into Xen.

The publication regarding Perfctr-Xen is at:
http://portal.acm.org/citation.cfm?id=1952687

Thanks,
Ruslan Nikolaev

Re: Perfctr-Xen framework for permonace analysis

[Keir Fraser]

On 12/05/2011 20:36, "Ruslan Nikolaev" <nruslan_devel@yahoo.com> wrote:

> Hi
> 
> I want to make an announcement about new perfomance monitoring framework.
> 
> Perfctr-Xen framework that enables per-thread performance analysis in Xen.
> Current version is capable of properly virtualizing counters in both
> paravirtualized and HVM modes. It is based on perfctr (which is a library and
> kernel module for non-virtualized guests), ported to Xen, and extended to work
> properly in virtualized environment. Both accumulative and interrupt modes
> counting (profiling) are supported.
> 
> The advantage of Perfctr-Xen is that it does not require specific HVM
> extensions which are needed for vpmu driver, can work in paravirtualized mode,
> and it also quite universal: works with many common tools such as PAPI,
> HPCToolkit, TAU PerfExplorer. It supports proper per-domain and per-thread
> virtualization. It is light-weight, supports wide range of CPUs, does not
> require save-and-restore for accumulative mode of counting (it uses counter
> offsetting), avoids expensive hypercalls and counter re-programming in certain
> circumstances (when threads are counting the same type of events). In
> addition, some techniques are employed to account for the overhead caused by
> the framework itself. This makes measurements quite accurate.
> 
> Perfctr-Xen consists of series of patches that need to be applied to Xen,
> Linux, perfctr. There are available at:
> http://people.cs.vt.edu/~rnikola/
> 
> The code is available under LGPL. It would be great to discuss if and how it
> can be integrated into Xen.

Could it reasonably replace the oprofile stuff we have already? I wouldn't
want yet another perfctr subsystem/interface unless it supplants an existing
one. We need a revolving door policy here I think.

 -- Keir

> The publication regarding Perfctr-Xen is at:
> http://portal.acm.org/citation.cfm?id=1952687
> 
> Thanks,
> Ruslan Nikolaev
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Perfctr-Xen framework for permonace analysis

[Ruslan Nikolaev]

Well, at least it can peacefully coexist with oprofile. AFAIK, XenOProf has some limitations (system wide monitoring only), and works only with OProfile. Many people would also prefer to use PAPI/HPCToolkit/PerfExplorer. HPCToolkit itself is quite sophisticated framework with elaborated functionality. It is quite comparable to OProfile. PerfExplorer is also a very interesting project. PAPI is used by developers to do wide range of measurements which are not necessarily possible with OProfile. The key feature of Perfctr-Xen is that it allows to do all levels of performance monitoring, it does not restrict to profiling.

This framework and xenoprof provide orthogonal solutions which both can be used in different circumstances.

Thanks,
Ruslan Nikolaev

--- On Fri, 5/13/11, Keir Fraser <keir.xen@gmail.com> wrote:

> From: Keir Fraser <keir.xen@gmail.com>
> Subject: Re: [Xen-devel] Perfctr-Xen framework for permonace analysis
> To: "Ruslan Nikolaev" <nruslan_devel@yahoo.com>, xen-devel@lists.xensource.com
> Date: Friday, May 13, 2011, 12:09 AM
> On 12/05/2011 20:36, "Ruslan
> Nikolaev" <nruslan_devel@yahoo.com>
> wrote:
> 
> > Hi
> > 
> > I want to make an announcement about new perfomance
> monitoring framework.
> > 
> > Perfctr-Xen framework that enables per-thread
> performance analysis in Xen.
> > Current version is capable of properly virtualizing
> counters in both
> > paravirtualized and HVM modes. It is based on perfctr
> (which is a library and
> > kernel module for non-virtualized guests), ported to
> Xen, and extended to work
> > properly in virtualized environment. Both accumulative
> and interrupt modes
> > counting (profiling) are supported.
> > 
> > The advantage of Perfctr-Xen is that it does not
> require specific HVM
> > extensions which are needed for vpmu driver, can work
> in paravirtualized mode,
> > and it also quite universal: works with many common
> tools such as PAPI,
> > HPCToolkit, TAU PerfExplorer. It supports proper
> per-domain and per-thread
> > virtualization. It is light-weight, supports wide
> range of CPUs, does not
> > require save-and-restore for accumulative mode of
> counting (it uses counter
> > offsetting), avoids expensive hypercalls and counter
> re-programming in certain
> > circumstances (when threads are counting the same type
> of events). In
> > addition, some techniques are employed to account for
> the overhead caused by
> > the framework itself. This makes measurements quite
> accurate.
> > 
> > Perfctr-Xen consists of series of patches that need to
> be applied to Xen,
> > Linux, perfctr. There are available at:
> > http://people.cs.vt.edu/~rnikola/
> > 
> > The code is available under LGPL. It would be great to
> discuss if and how it
> > can be integrated into Xen.
> 
> Could it reasonably replace the oprofile stuff we have
> already? I wouldn't
> want yet another perfctr subsystem/interface unless it
> supplants an existing
> one. We need a revolving door policy here I think.
> 
>  -- Keir
> 
> > The publication regarding Perfctr-Xen is at:
> > http://portal.acm.org/citation.cfm?id=1952687
> > 
> > Thanks,
> > Ruslan Nikolaev
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Perfctr-Xen framework for permonace analysis

[George Dunlap]

On Thu, May 12, 2011 at 9:35 PM, Ruslan Nikolaev
<nruslan_devel@yahoo.com> wrote:
> Well, at least it can peacefully coexist with oprofile. AFAIK, XenOProf has some limitations (system wide monitoring only), and works only with OProfile. Many people would also prefer to use PAPI/HPCToolkit/PerfExplorer. HPCToolkit itself is quite sophisticated framework with elaborated functionality. It is quite comparable to OProfile. PerfExplorer is also a very interesting project. PAPI is used by developers to do wide range of measurements which are not necessarily possible with OProfile. The key feature of Perfctr-Xen is that it allows to do all levels of performance monitoring, it does not restrict to profiling.

xenoprof enables the normal Linux oprofile, which can be used on
processes, as well as on domains and on Xen itself.  I don't remember
if oprofile allows collection of performance counters on individual
processes though; I suspect not.

I'm a bit torn -- if people are used to using oprofile under Linux, it
may be advantageous to keep that, rather than replacing it outright
with a new tool.  But as you said about the scheduler, Keir, oprofile
gets little enough love as it is...

 -George

>
> This framework and xenoprof provide orthogonal solutions which both can be used in different circumstances.
>
> Thanks,
> Ruslan Nikolaev
>
> --- On Fri, 5/13/11, Keir Fraser <keir.xen@gmail.com> wrote:
>
>> From: Keir Fraser <keir.xen@gmail.com>
>> Subject: Re: [Xen-devel] Perfctr-Xen framework for permonace analysis
>> To: "Ruslan Nikolaev" <nruslan_devel@yahoo.com>, xen-devel@lists.xensource.com
>> Date: Friday, May 13, 2011, 12:09 AM
>> On 12/05/2011 20:36, "Ruslan
>> Nikolaev" <nruslan_devel@yahoo.com>
>> wrote:
>>
>> > Hi
>> >
>> > I want to make an announcement about new perfomance
>> monitoring framework.
>> >
>> > Perfctr-Xen framework that enables per-thread
>> performance analysis in Xen.
>> > Current version is capable of properly virtualizing
>> counters in both
>> > paravirtualized and HVM modes. It is based on perfctr
>> (which is a library and
>> > kernel module for non-virtualized guests), ported to
>> Xen, and extended to work
>> > properly in virtualized environment. Both accumulative
>> and interrupt modes
>> > counting (profiling) are supported.
>> >
>> > The advantage of Perfctr-Xen is that it does not
>> require specific HVM
>> > extensions which are needed for vpmu driver, can work
>> in paravirtualized mode,
>> > and it also quite universal: works with many common
>> tools such as PAPI,
>> > HPCToolkit, TAU PerfExplorer. It supports proper
>> per-domain and per-thread
>> > virtualization. It is light-weight, supports wide
>> range of CPUs, does not
>> > require save-and-restore for accumulative mode of
>> counting (it uses counter
>> > offsetting), avoids expensive hypercalls and counter
>> re-programming in certain
>> > circumstances (when threads are counting the same type
>> of events). In
>> > addition, some techniques are employed to account for
>> the overhead caused by
>> > the framework itself. This makes measurements quite
>> accurate.
>> >
>> > Perfctr-Xen consists of series of patches that need to
>> be applied to Xen,
>> > Linux, perfctr. There are available at:
>> > http://people.cs.vt.edu/~rnikola/
>> >
>> > The code is available under LGPL. It would be great to
>> discuss if and how it
>> > can be integrated into Xen.
>>
>> Could it reasonably replace the oprofile stuff we have
>> already? I wouldn't
>> want yet another perfctr subsystem/interface unless it
>> supplants an existing
>> one. We need a revolving door policy here I think.
>>
>>  -- Keir
>>
>> > The publication regarding Perfctr-Xen is at:
>> > http://portal.acm.org/citation.cfm?id=1952687
>> >
>> > Thanks,
>> > Ruslan Nikolaev
>> >
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xensource.com
>> > http://lists.xensource.com/xen-devel
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Perfctr-Xen framework for permonace analysis

[George Dunlap]

On Thu, May 12, 2011 at 8:36 PM, Ruslan Nikolaev
<nruslan_devel@yahoo.com> wrote:
> Hi
>
> I want to make an announcement about new perfomance monitoring framework.
>
> Perfctr-Xen framework that enables per-thread performance analysis in Xen. Current version is capable of properly virtualizing counters in both paravirtualized and HVM modes. It is based on perfctr (which is a library and kernel module for non-virtualized guests), ported to Xen, and extended to work properly in virtualized environment. Both accumulative and interrupt modes counting (profiling) are supported.
>
> The advantage of Perfctr-Xen is that it does not require specific HVM extensions which are needed for vpmu driver, can work in paravirtualized mode, and it also quite universal: works with many common tools such as PAPI, HPCToolkit, TAU PerfExplorer. It supports proper per-domain and per-thread virtualization. It is light-weight, supports wide range of CPUs, does not require save-and-restore for accumulative mode of counting (it uses counter offsetting), avoids expensive hypercalls and counter re-programming in certain circumstances (when threads are counting the same type of events). In addition, some techniques are employed to account for the overhead caused by the framework itself. This makes measurements quite accurate.
>
> Perfctr-Xen consists of series of patches that need to be applied to Xen, Linux, perfctr. There are available at:
> http://people.cs.vt.edu/~rnikola/
>
> The code is available under LGPL. It would be great to discuss if and how it can be integrated into Xen.
>
> The publication regarding Perfctr-Xen is at:
> http://portal.acm.org/citation.cfm?id=1952687

Do you have this paper, or perhaps a pre-print, posted somewhere not
behind a paywall?  Or a different document describing the library in
detail?  Most of us don't have subscriptions to ACM...

 -George

Re: Perfctr-Xen framework for permonace analysis

[Ruslan Nikolaev]

--- On Fri, 5/13/11, George Dunlap <George.Dunlap@eu.citrix.com> wrote:

> From: George Dunlap <George.Dunlap@eu.citrix.com>
> Subject: Re: [Xen-devel] Perfctr-Xen framework for permonace analysis
> To: "Ruslan Nikolaev" <nruslan_devel@yahoo.com>
> Cc: xen-devel@lists.xensource.com
> Date: Friday, May 13, 2011, 5:59 PM
> On Thu, May 12, 2011 at 8:36 PM,
> Ruslan Nikolaev
> <nruslan_devel@yahoo.com>
> wrote:
> > Hi
> >
> > I want to make an announcement about new perfomance
> monitoring framework.
> >
> > Perfctr-Xen framework that enables per-thread
> performance analysis in Xen. Current version is capable of
> properly virtualizing counters in both paravirtualized and
> HVM modes. It is based on perfctr (which is a library and
> kernel module for non-virtualized guests), ported to Xen,
> and extended to work properly in virtualized environment.
> Both accumulative and interrupt modes counting (profiling)
> are supported.
> >
> > The advantage of Perfctr-Xen is that it does not
> require specific HVM extensions which are needed for vpmu
> driver, can work in paravirtualized mode, and it also quite
> universal: works with many common tools such as PAPI,
> HPCToolkit, TAU PerfExplorer. It supports proper per-domain
> and per-thread virtualization. It is light-weight, supports
> wide range of CPUs, does not require save-and-restore for
> accumulative mode of counting (it uses counter offsetting),
> avoids expensive hypercalls and counter re-programming in
> certain circumstances (when threads are counting the same
> type of events). In addition, some techniques are employed
> to account for the overhead caused by the framework itself.
> This makes measurements quite accurate.
> >
> > Perfctr-Xen consists of series of patches that need to
> be applied to Xen, Linux, perfctr. There are available at:
> > http://people.cs.vt.edu/~rnikola/
> >
> > The code is available under LGPL. It would be great to
> discuss if and how it can be integrated into Xen.
> >
> > The publication regarding Perfctr-Xen is at:
> > http://portal.acm.org/citation.cfm?id=1952687
> 
> Do you have this paper, or perhaps a pre-print, posted
> somewhere not
> behind a paywall?  Or a different document describing
> the library in
> detail?  Most of us don't have subscriptions to
> ACM...

Ok. I will post a link.

Ruslan.

> 
>  -George
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Perfctr-Xen framework for permonace analysis

[Ruslan Nikolaev]

The author's paper is available on my website:

http://people.cs.vt.edu/~rnikola/?page_id=159

Please, let me know what you think.

Thanks,
Ruslan Nikolaev

--- On Fri, 5/13/11, George Dunlap <George.Dunlap@eu.citrix.com> wrote:

> From: George Dunlap <George.Dunlap@eu.citrix.com>
> Subject: Re: [Xen-devel] Perfctr-Xen framework for permonace analysis
> To: "Ruslan Nikolaev" <nruslan_devel@yahoo.com>
> Cc: xen-devel@lists.xensource.com
> Date: Friday, May 13, 2011, 5:59 PM
> On Thu, May 12, 2011 at 8:36 PM,
> Ruslan Nikolaev
> <nruslan_devel@yahoo.com>
> wrote:
> > Hi
> >
> > I want to make an announcement about new perfomance
> monitoring framework.
> >
> > Perfctr-Xen framework that enables per-thread
> performance analysis in Xen. Current version is capable of
> properly virtualizing counters in both paravirtualized and
> HVM modes. It is based on perfctr (which is a library and
> kernel module for non-virtualized guests), ported to Xen,
> and extended to work properly in virtualized environment.
> Both accumulative and interrupt modes counting (profiling)
> are supported.
> >
> > The advantage of Perfctr-Xen is that it does not
> require specific HVM extensions which are needed for vpmu
> driver, can work in paravirtualized mode, and it also quite
> universal: works with many common tools such as PAPI,
> HPCToolkit, TAU PerfExplorer. It supports proper per-domain
> and per-thread virtualization. It is light-weight, supports
> wide range of CPUs, does not require save-and-restore for
> accumulative mode of counting (it uses counter offsetting),
> avoids expensive hypercalls and counter re-programming in
> certain circumstances (when threads are counting the same
> type of events). In addition, some techniques are employed
> to account for the overhead caused by the framework itself.
> This makes measurements quite accurate.
> >
> > Perfctr-Xen consists of series of patches that need to
> be applied to Xen, Linux, perfctr. There are available at:
> > http://people.cs.vt.edu/~rnikola/
> >
> > The code is available under LGPL. It would be great to
> discuss if and how it can be integrated into Xen.
> >
> > The publication regarding Perfctr-Xen is at:
> > http://portal.acm.org/citation.cfm?id=1952687
> 
> Do you have this paper, or perhaps a pre-print, posted
> somewhere not
> behind a paywall?  Or a different document describing
> the library in
> detail?  Most of us don't have subscriptions to
> ACM...
> 
>  -George
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Perfctr-Xen framework for permonace analysis

[Konrad Rzeszutek Wilk]

On Thu, May 12, 2011 at 12:36:07PM -0700, Ruslan Nikolaev wrote:
> Hi
> 
> I want to make an announcement about new perfomance monitoring framework.
> 
> Perfctr-Xen framework that enables per-thread performance analysis in Xen. Current version is capable of properly virtualizing counters in both paravirtualized and HVM modes. It is based on perfctr (which is a library and kernel module for non-virtualized guests), ported to Xen, and extended to work properly in virtualized environment. Both accumulative and interrupt modes counting (profiling) are supported.
> 
> The advantage of Perfctr-Xen is that it does not require specific HVM extensions which are needed for vpmu driver, can work in paravirtualized mode, and it also quite universal: works with many common tools such as PAPI, HPCToolkit, TAU PerfExplorer. It supports proper per-domain and per-thread virtualization. It is light-weight, supports wide range of CPUs, does not require save-and-restore for accumulative mode of counting (it uses counter offsetting), avoids expensive hypercalls and counter re-programming in certain circumstances (when threads are counting the same type of events). In addition, some techniques are employed to account for the overhead caused by the framework itself. This makes measurements quite accurate.
> 
> Perfctr-Xen consists of series of patches that need to be applied to Xen, Linux, perfctr. There are available at:
> http://people.cs.vt.edu/~rnikola/
> 
> The code is available under LGPL. It would be great to discuss if and how it can be integrated into Xen.


So I read your guys paper and found it interesting. But I am bit confused about the baremetal
state of perf monitoring tool. Which one is in the upstream kernel (3.0) that runs when you
use 'perftop' and like?

> 
> The publication regarding Perfctr-Xen is at:
> http://portal.acm.org/citation.cfm?id=1952687
> 
> Thanks,
> Ruslan Nikolaev
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel