Measure performance of hypercalls

Measure performance of hypercalls

[Tommy Huang]

[-- Attachment #1.1: Type: text/plain, Size: 303 bytes --]

Hi all,

I am wondering how to measure the performance of hypercalls.
For example, how many X86 instructions are executed in Xen when you issue a
hypercall.
Also, how to get the frequency of hypercalls which is that how many
hypercalls have been issued during a period of time?
Thanks in advance.

- Nb

[-- Attachment #1.2: Type: text/html, Size: 350 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Measure performance of hypercalls

[George Dunlap]

Getting the frequency of hypercalls can be done using xentrace.  See
xen/include/public/trace.h for the appropriate mask and pertinent
trace records.

If you want to find the performance of hypercalls, you'll have to add
some instrumentation.  You can extend xentrace to find out the number
of *cycles* executed in a hypercall by taking a trace (with a
timestamp) at hypercall entry and exit.  We do this already for
vmenter/vmexit.

 -George

On Wed, Oct 15, 2008 at 3:12 AM, Tommy Huang <nbp0204@gmail.com> wrote:
> Hi all,
>
> I am wondering how to measure the performance of hypercalls.
> For example, how many X86 instructions are executed in Xen when you issue a
> hypercall.
> Also, how to get the frequency of hypercalls which is that how many
> hypercalls have been issued during a period of time?
> Thanks in advance.
>
> - Nb
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

RE: Measure performance of hypercalls

[Dan Magenheimer]

> If you want to find the performance of hypercalls, you'll have to add
> some instrumentation.  You can extend xentrace to find out the number
> of *cycles* executed in a hypercall by taking a trace (with a
> timestamp) at hypercall entry and exit.  We do this already for
> vmenter/vmexit.

Hmmm... lots of interesting things can happen between a hypercall
entry and exit, eg. domain switch, interrupts, timer queues, etc.
So just taking a pair of timestamps is not likely to have the
desired result.

I too am interested in a similar measurement (for a specific hypercall),
so if there IS a way to do this with reasonable accuracy, please
correct me.

> On Wed, Oct 15, 2008 at 3:12 AM, Tommy Huang 
> <nbp0204@gmail.com> wrote:
> > Hi all,
> >
> > I am wondering how to measure the performance of hypercalls.
> > For example, how many X86 instructions are executed in Xen 
> when you issue a
> > hypercall.
> > Also, how to get the frequency of hypercalls which is that how many
> > hypercalls have been issued during a period of time?
> > Thanks in advance.
> >
> > - Nb
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Measure performance of hypercalls

[George Dunlap]

On Wed, Oct 15, 2008 at 3:00 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:
> Hmmm... lots of interesting things can happen between a hypercall
> entry and exit, eg. domain switch, interrupts, timer queues, etc.
> So just taking a pair of timestamps is not likely to have the
> desired result.

Domain switch you can factor in, if you trace runstate changes as well
(albeit a bit tricky).  Long-term averages are probably what you
really want (since a one-off isn't going to affect performance too
much).  So if you filter out hypercalls interrupted by a context
switch, and do an average over the rest, the interrupts, timer queues,
&c should more or less disappear in the noise (unless they're a
significant source of overhead, in which case they need some attention
anyway).

I have a tool that will do a bunch of this stuff for vmexit/enter
analysis.  If there were hypercall enter/exit timestamp traces, it
could be modified to do the same for hypercalls.  Then we might think
about fun things like analyzing hypercall continuations and what not.
:-)

 -George

>
> I too am interested in a similar measurement (for a specific hypercall),
> so if there IS a way to do this with reasonable accuracy, please
> correct me.
>
>> On Wed, Oct 15, 2008 at 3:12 AM, Tommy Huang
>> <nbp0204@gmail.com> wrote:
>> > Hi all,
>> >
>> > I am wondering how to measure the performance of hypercalls.
>> > For example, how many X86 instructions are executed in Xen
>> when you issue a
>> > hypercall.
>> > Also, how to get the frequency of hypercalls which is that how many
>> > hypercalls have been issued during a period of time?
>> > Thanks in advance.
>> >
>> > - Nb
>> >
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xensource.com
>> > http://lists.xensource.com/xen-devel
>> >
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Measure performance of hypercalls

[George Dunlap]

There are a number of other things that happen inside of Xen besides
hypercalls:
* Access to the guest pagetables are emulated: i.e., the guest has read
but not write permission. When the guest writes to pagetables, it faults
to the hypervisor, and emulates the instruction there.
* Most traps, even if destined for the guest (i.e., math processor
exceptions, debug exceptions, and so on).  For 64-bit xen (I believe)
system calls also go through to the hypervisor before being relayed to
the guest.  Normal guest page faults also go through they hypervisor
(because it needs to check to see if the fault requires an emulation).
* Emulated privileged instructions.  It turns out to be not very
expensive, and require a lot less changes to the guest kernels, to
simply emulate certain privileged operations.

If what you want is a statistical proportion, something like xenoprofile
might be closer to what you want.  Xenoprofile uses statistical
sampling: it sets a non-maskable interrupt to happen every X processor
cycles.  By taking a large number of "samples", you can get a good idea
how much time is spent in different parts of the system on average.

Unfortunately, it seems to be rather suffering from bitrot at the
moment.  But if you're using a 32-bit kernel on a 32-bit hypervisor, I
think you might be able to get the information you want.  (32-on-64 has
some issues due to truncating of 64-bit addresses and the fact that the
oprofile kernel portion discards some key information provided by Xen.)

Another option would be (if you're up for a bit of a challenge) adding
calls to take traces at the hypervisor entry points in entry.S.  If you
manage that, I might be able to help you get statistical information
using my analysis tool.

Peace,
 -George

Tommy Huang wrote:
> Hi George,
> 
> Thanks for this tutorial. It is very helpful to me.
> I will try it few hours later.
> Actually, my purpose is to test how much time does the CPU spends on the
> hypervisor versus other parts(OS and applications).
> So, what I am trying to do is to know:
> 1. Are hypercalls the only way that the domain traps to the hypervisor?
> Is there any other way that domain can trap to the hypervisor? If yes,
> is it frequent?
> 2. Other ways that need the hypervisor intervention...like domain
> switch, device interrupts.... I think those things don't need hypercalls
> but the hypervisor still needs to handle them. Is that right?
> 
> Any thought or hint is very appreciated. Thanks in advance.
> 
> Regards,
> - Tommy
> 
> 2008/10/16 George Dunlap <George.Dunlap@eu.citrix.com
> <mailto:George.Dunlap@eu.citrix.com>>
> 
>     Hey Tommy,
> 
>     No worries.  I didn't realize that the documentation for xentrace was so
>     sparse.
> 
>     A quick summary:
> 
>     * Take a trace
>     # xentrace -D -e all /tmp/test.trace
> 
>     * Look at the trace file using xentrace_format
>     # cat /tmp/test.trace | xentrace_format formats
> 
>     The 'formats' file can be found in tools/xentrace/formats.  Run
>     xentrace_format for a brief description of what the format will look
>     like.
> 
>     If you do that, you'll see hypercall traces.
> 
>     If you want to refine the trace, you can change the event mask using the
>     -e parameter to xentrace.  Look in xen/include/public/trace.h.  For
>     example, if you only want to trace hypercalls and runstate events, you
>     could do:
> 
>     # xentrace -D -e 0x22f000 {filename}
> 
>     NB that the records in the file are in order *within* a physical cpu,
>     but not necessarily across physical cpus.  The way tracing works is that
>     inside Xen, each processor gets its own trace buffer that it writes to.
>      In dom0, the xentrace process periodically copies all the data from
>     each buffer to disk, one processor at a time.
> 
>     Anyway, give it a go, and let me know if you get stuck. :-)
> 
>      -George
> 
>     Tommy Huang wrote:
>     >
>     > Hi George,
>     >
>     > Sorry for bothering you.
>     > Do you have any tutorial about xen trace or some helpful example
>     that I
>     > can refer to?
>     > I did some search but I can hardly find any xentrace examples or
>     tutorials.
>     > I appreciate it if you can give me a few instructions about how to
>     play
>     > with that.
>     > Anyway, thanks for telling me those information. I'll try to learn how
>     > to modify.
>     > Thanks in advance.
>     >
>     > Regards,
>     > - Tommy
>     >
>     > 2008/10/15 George Dunlap <George.Dunlap@eu.citrix.com
>     <mailto:George.Dunlap@eu.citrix.com>
>     > <mailto:George.Dunlap@eu.citrix.com
>     <mailto:George.Dunlap@eu.citrix.com>>>
>     >
>     >     Getting the frequency of hypercalls can be done using
>     xentrace.  See
>     >     xen/include/public/trace.h for the appropriate mask and pertinent
>     >     trace records.
>     >
>     >     If you want to find the performance of hypercalls, you'll have
>     to add
>     >     some instrumentation.  You can extend xentrace to find out the
>     number
>     >     of *cycles* executed in a hypercall by taking a trace (with a
>     >     timestamp) at hypercall entry and exit.  We do this already for
>     >     vmenter/vmexit.
>     >
>     >      -George
>     >
>     >     On Wed, Oct 15, 2008 at 3:12 AM, Tommy Huang
>     <nbp0204@gmail.com <mailto:nbp0204@gmail.com>
>     >     <mailto:nbp0204@gmail.com <mailto:nbp0204@gmail.com>>> wrote:
>     >     > Hi all,
>     >     >
>     >     > I am wondering how to measure the performance of hypercalls.
>     >     > For example, how many X86 instructions are executed in Xen when
>     >     you issue a
>     >     > hypercall.
>     >     > Also, how to get the frequency of hypercalls which is that
>     how many
>     >     > hypercalls have been issued during a period of time?
>     >     > Thanks in advance.
>     >     >
>     >     > - Nb
>     >     >
>     >     > _______________________________________________
>     >     > Xen-devel mailing list
>     >     > Xen-devel@lists.xensource.com
>     <mailto:Xen-devel@lists.xensource.com>
>     <mailto:Xen-devel@lists.xensource.com
>     <mailto:Xen-devel@lists.xensource.com>>
>     >     > http://lists.xensource.com/xen-devel
>     >     >
>     >     >
>     >
>     >
> 
>