RE: Proposal for Xen support of performance monitoring anddebug hardware

RE: Proposal for Xen support of performance monitoring anddebug hardware

[Ian Pratt]

> Rik van Riel pointed me at the Santos's patch for oprofile support. 
> There are some differences between the two approaches. The 
> Xen oprofile 
> support by HP pretty much just supports oprofile and was 
> designed to get 
> some information about what was going on in the Xen hypervisor. It 
> doesn't provide access to the other performance monitoring (or 
> debugging) hardware.

The extended xen-oprofile is a useful tool for getting a global view of
what's going on in the whole machine. 

I think what you're proposing is for enabling a guest to get a better
idea of what's going on while it is running, which is also useful.

You can't really mix the two modes of operation on the same system, at
least not in the general case.
 
> > I can certainly see some merit in having fine grained access control
> > over MSRs, though for the case of perf counter registers I wander
> > whether we'd be better off with some higher-level interface?
> 
> I was aiming for minimal support low-level, trying to follow the 
> existing Xen approach of not coding too much knowledge about 
> the system 
> in Xen. Make the MSR registers visible and make sure that a guest OS 
> cannot clobber other guest OSs. The guests OS decide how to use the 
> performance monitoring hw.  The hypervisor needs a list of which 
> registers are in which class, but the hypervisor doesn't need to know 
> the details of what the registers do.

It seems sensible to incorporate a simple mechanism for enabling
fine-grained access control to msr's, and also code to save/restore msr
values that have been changed when context switching between guests. 

One issue for guests that are using this mechanism is that I don't
believe it's possible to selectively count events in ring 1 vs ring 0.
Hence it will also count events in Xen during hypercalls and interrupt
handling. In some cases this will be what's wanted, in others, not. I
guess we could slectively save/restore counter values when
entering/leaving Xen, but that's slow and ugly.

Ian

Re: Proposal for Xen support of performance monitoring anddebug hardware

[William Cohen]

Ian Pratt wrote:
>>Rik van Riel pointed me at the Santos's patch for oprofile support. 
>>There are some differences between the two approaches. The 
>>Xen oprofile 
>>support by HP pretty much just supports oprofile and was 
>>designed to get 
>>some information about what was going on in the Xen hypervisor. It 
>>doesn't provide access to the other performance monitoring (or 
>>debugging) hardware.
> 
> 
> The extended xen-oprofile is a useful tool for getting a global view of
> what's going on in the whole machine. 
> 
> I think what you're proposing is for enabling a guest to get a better
> idea of what's going on while it is running, which is also useful.
> 
> You can't really mix the two modes of operation on the same system, at
> least not in the general case.

Right, mixing the different modes at the same time is not going to work, 
but it would be nice to make the mechanism flexible enough so that the 
same mechanism can be both for local and system-wide profiling.

>>>I can certainly see some merit in having fine grained access control
>>>over MSRs, though for the case of perf counter registers I wander
>>>whether we'd be better off with some higher-level interface?
>>
>>I was aiming for minimal support low-level, trying to follow the 
>>existing Xen approach of not coding too much knowledge about 
>>the system 
>>in Xen. Make the MSR registers visible and make sure that a guest OS 
>>cannot clobber other guest OSs. The guests OS decide how to use the 
>>performance monitoring hw.  The hypervisor needs a list of which 
>>registers are in which class, but the hypervisor doesn't need to know 
>>the details of what the registers do.
> 
> 
> It seems sensible to incorporate a simple mechanism for enabling
> fine-grained access control to msr's, and also code to save/restore msr
> values that have been changed when context switching between guests. 
> 
> One issue for guests that are using this mechanism is that I don't
> believe it's possible to selectively count events in ring 1 vs ring 0.
> Hence it will also count events in Xen during hypercalls and interrupt
> handling. In some cases this will be what's wanted, in others, not. I
> guess we could slectively save/restore counter values when
> entering/leaving Xen, but that's slow and ugly.

Yes, the selectivity is ring != 0 or ring ==0. There isn't the finer 
grain to separate the guest OS from the normal user space.

A place that things might get odd if the domain-0 handles things on the 
behalf of guest domain. The sampling in local mode would not cross 
domains. Thus, a user guest domain might not know that it is having a 
lot of overhead in domain-0. However, in these cases people are probably 
better off doing system-wide sampling.

-Will

RE: Proposal for Xen support of performance monitoring anddebug hardware

[Ian Pratt]

 
> I have been working on a proposal to add Xen support for 
> performance monitoring and debugging hardware. The goal of 
> this would be enable OProfile, perfmon, and perfctr to work 
> on Xen. The proposal is still pretty preliminary, but I would 
> like comments on the current version.

William, have you seen the patches from Jose Renato Santos for multi VM
oprofile support? We're planning on getting these checked in to the xen
repo, after a little reworking.

It's somewhat orthogonal to your msr protection scheme, but you should
be aware of it.

I can certainly see some merit in having fine grained access control
over MSRs, though for the case of perf counter registers I wander
whether we'd be better off with some higher-level interface?

What other msr's do you anticipate your scheme being used to provide
restricted access to for selected VMs?

Best,
Ian

Re: Proposal for Xen support of performance monitoring anddebug hardware

[William Cohen]

Ian Pratt wrote:
>  
> 
>>I have been working on a proposal to add Xen support for 
>>performance monitoring and debugging hardware. The goal of 
>>this would be enable OProfile, perfmon, and perfctr to work 
>>on Xen. The proposal is still pretty preliminary, but I would 
>>like comments on the current version.
> 
> 
> William, have you seen the patches from Jose Renato Santos for multi VM
> oprofile support? We're planning on getting these checked in to the xen
> repo, after a little reworking.
> 
> It's somewhat orthogonal to your msr protection scheme, but you should
> be aware of it.

Rik van Riel pointed me at the Santos's patch for oprofile support. 
There are some differences between the two approaches. The Xen oprofile 
support by HP pretty much just supports oprofile and was designed to get 
some information about what was going on in the Xen hypervisor. It 
doesn't provide access to the other performance monitoring (or 
debugging) hardware.

> I can certainly see some merit in having fine grained access control
> over MSRs, though for the case of perf counter registers I wander
> whether we'd be better off with some higher-level interface?

I was aiming for minimal support low-level, trying to follow the 
existing Xen approach of not coding too much knowledge about the system 
in Xen. Make the MSR registers visible and make sure that a guest OS 
cannot clobber other guest OSs. The guests OS decide how to use the 
performance monitoring hw.  The hypervisor needs a list of which 
registers are in which class, but the hypervisor doesn't need to know 
the details of what the registers do.

There is significant variations in the precise events and contraints on 
the combinations of events allowed in many of the performance monitoring 
systems. OProfile has files for each architecture to map events to the 
counter setup.  There are a lot of variations in the events available on 
a processor; OProfile doesn't hide those differences. The University of 
Tennessee knoxville PAPI has abstraction to hide some of these 
differences with generic events, e.g. cache miss event.  ppc64 (aix) and 
ia64 (perfmon) have libraries to do the complicated constraints testing 
to determine whether events can be done at the same time. However, these 
mapping operations are handled in user-space, not in the kernel.

I am not sure that that should be pushed into the hypervisor. I suspect 
that someone will complain that the high-level interface doesn't handle 
some particular instrumentation mode of the performance monitoring 
hardware. Adding it to Xen will require rebuilding xen and the guest OS 
and rebooting the entire machime. The low-level interface makes the 
guest OS responsible and only it would need to be recompiled, and only 
rebuild and reboot the guest OS.

> What other msr's do you anticipate your scheme being used to provide
> restricted access to for selected VMs?

The sampling used by OProfile would naturally be something high on the 
list of things to use. It would also be nice to be able to do the 
stopwatch counting provided by perfctr and perfmon.

The PPC64, IA64 and Pentium 4 they have precise event sampling. I would 
like to be able access those through the hypervisor.

-Will