Re: Proposal for Xen support of performance monitoring anddebug hardware

[William Cohen <wcohen@redhat.com>]

Ian Pratt wrote:
>  
> 
>>I have been working on a proposal to add Xen support for 
>>performance monitoring and debugging hardware. The goal of 
>>this would be enable OProfile, perfmon, and perfctr to work 
>>on Xen. The proposal is still pretty preliminary, but I would 
>>like comments on the current version.
> 
> 
> William, have you seen the patches from Jose Renato Santos for multi VM
> oprofile support? We're planning on getting these checked in to the xen
> repo, after a little reworking.
> 
> It's somewhat orthogonal to your msr protection scheme, but you should
> be aware of it.

Rik van Riel pointed me at the Santos's patch for oprofile support. 
There are some differences between the two approaches. The Xen oprofile 
support by HP pretty much just supports oprofile and was designed to get 
some information about what was going on in the Xen hypervisor. It 
doesn't provide access to the other performance monitoring (or 
debugging) hardware.

> I can certainly see some merit in having fine grained access control
> over MSRs, though for the case of perf counter registers I wander
> whether we'd be better off with some higher-level interface?

I was aiming for minimal support low-level, trying to follow the 
existing Xen approach of not coding too much knowledge about the system 
in Xen. Make the MSR registers visible and make sure that a guest OS 
cannot clobber other guest OSs. The guests OS decide how to use the 
performance monitoring hw.  The hypervisor needs a list of which 
registers are in which class, but the hypervisor doesn't need to know 
the details of what the registers do.

There is significant variations in the precise events and contraints on 
the combinations of events allowed in many of the performance monitoring 
systems. OProfile has files for each architecture to map events to the 
counter setup.  There are a lot of variations in the events available on 
a processor; OProfile doesn't hide those differences. The University of 
Tennessee knoxville PAPI has abstraction to hide some of these 
differences with generic events, e.g. cache miss event.  ppc64 (aix) and 
ia64 (perfmon) have libraries to do the complicated constraints testing 
to determine whether events can be done at the same time. However, these 
mapping operations are handled in user-space, not in the kernel.

I am not sure that that should be pushed into the hypervisor. I suspect 
that someone will complain that the high-level interface doesn't handle 
some particular instrumentation mode of the performance monitoring 
hardware. Adding it to Xen will require rebuilding xen and the guest OS 
and rebooting the entire machime. The low-level interface makes the 
guest OS responsible and only it would need to be recompiled, and only 
rebuild and reboot the guest OS.

> What other msr's do you anticipate your scheme being used to provide
> restricted access to for selected VMs?

The sampling used by OProfile would naturally be something high on the 
list of things to use. It would also be nice to be able to do the 
stopwatch counting provided by perfctr and perfmon.

The PPC64, IA64 and Pentium 4 they have precise event sampling. I would 
like to be able access those through the hypervisor.

-Will