cpu utilization monitoring

cpu utilization monitoring

[Karl Rister]

Currently in the vcpu_runstate_info structure we have fields that contain the 
time spent when the vcpu is in one of four states.  When trying to determine 
the physical utilization that a vcpu is requiring of the system from libxc we 
make a domctl operation call to the XEN_DOMCTL_getvcpuinfo command.  When 
processing this command we pull the value for the RUNSTATE_running field and 
attribute it to the processor that the vcpu is currently running on.

I believe that with the new credit scheduler these values are wrong as it is 
possible that the vcpu has time in RUNSTATE_running that was not actually 
spent on the currently running processor.  Is there a separate path through 
the libraries and into Xen that retrieves this information in a more correct 
manner?  Given that a vcpu can migrate among several physical processors in 
the time between samples it seems that a finer grained accounting mechanism 
is needed to correctly report physical utilization of the hardware resources.



-- 
Karl Rister
IBM Linux Performance Team
kmr@us.ibm.com

Re: cpu utilization monitoring

[Keir Fraser]

The hypercall reports the point-of-view of the timeline of that VCPU. The
fact that its timeline may involve running on various different physical
CPUs should not matter, except for cache-warming and other context-switching
costs. The runstate_info interface absolutely does not state that the
reported run time only includes time spent running on the currently assigned
physical CPU.

 -- Keir

On 11/9/06 8:44 pm, "Karl Rister" <kmr@us.ibm.com> wrote:

> Currently in the vcpu_runstate_info structure we have fields that contain the
> time spent when the vcpu is in one of four states.  When trying to determine
> the physical utilization that a vcpu is requiring of the system from libxc we
> make a domctl operation call to the XEN_DOMCTL_getvcpuinfo command.  When
> processing this command we pull the value for the RUNSTATE_running field and
> attribute it to the processor that the vcpu is currently running on.
> 
> I believe that with the new credit scheduler these values are wrong as it is
> possible that the vcpu has time in RUNSTATE_running that was not actually
> spent on the currently running processor.  Is there a separate path through
> the libraries and into Xen that retrieves this information in a more correct
> manner?  Given that a vcpu can migrate among several physical processors in
> the time between samples it seems that a finer grained accounting mechanism
> is needed to correctly report physical utilization of the hardware resources.
> 
> 

Re: cpu utilization monitoring

[Karl Rister]

Is there a more appropriate interface for collecting this information then?  I 
would like to be able to sample on an some interval (say 3 or 5 seconds) and 
be able to see the amount of time a vcpu spent running on the individual 
physical processors during that time.

On Monday 11 September 2006 2:56 pm, Keir Fraser wrote:
> The hypercall reports the point-of-view of the timeline of that VCPU. The
> fact that its timeline may involve running on various different physical
> CPUs should not matter, except for cache-warming and other
> context-switching costs. The runstate_info interface absolutely does not
> state that the reported run time only includes time spent running on the
> currently assigned physical CPU.
>
>  -- Keir
>
> On 11/9/06 8:44 pm, "Karl Rister" <kmr@us.ibm.com> wrote:
> > Currently in the vcpu_runstate_info structure we have fields that contain
> > the time spent when the vcpu is in one of four states.  When trying to
> > determine the physical utilization that a vcpu is requiring of the system
> > from libxc we make a domctl operation call to the XEN_DOMCTL_getvcpuinfo
> > command.  When processing this command we pull the value for the
> > RUNSTATE_running field and attribute it to the processor that the vcpu is
> > currently running on.
> >
> > I believe that with the new credit scheduler these values are wrong as it
> > is possible that the vcpu has time in RUNSTATE_running that was not
> > actually spent on the currently running processor.  Is there a separate
> > path through the libraries and into Xen that retrieves this information
> > in a more correct manner?  Given that a vcpu can migrate among several
> > physical processors in the time between samples it seems that a finer
> > grained accounting mechanism is needed to correctly report physical
> > utilization of the hardware resources.

-- 
Karl Rister
IBM Linux Performance Team
kmr@us.ibm.com

Re: cpu utilization monitoring

[Keir Fraser]

If you turn on trace buffers and use xentrace/xenbaked, I think you get a
trace event whenever a VCPU is scheduled. From this you can work out how
long a VCPU spent running on each physical CPU. This is a bit heavyweight
but okay for e.g., debugging the scheduler and making sure it isn't
ping-ponging things too quickly. I'm not sure why you'd care about the stats
across individual physical CPUs during normal execution?

 -- Keir

On 11/9/06 10:48 pm, "Karl Rister" <kmr@us.ibm.com> wrote:

> Is there a more appropriate interface for collecting this information then?  I
> would like to be able to sample on an some interval (say 3 or 5 seconds) and
> be able to see the amount of time a vcpu spent running on the individual
> physical processors during that time.
> 
> On Monday 11 September 2006 2:56 pm, Keir Fraser wrote:
>> The hypercall reports the point-of-view of the timeline of that VCPU. The
>> fact that its timeline may involve running on various different physical
>> CPUs should not matter, except for cache-warming and other
>> context-switching costs. The runstate_info interface absolutely does not
>> state that the reported run time only includes time spent running on the
>> currently assigned physical CPU.
>> 
>>  -- Keir
>> 
>> On 11/9/06 8:44 pm, "Karl Rister" <kmr@us.ibm.com> wrote:
>>> Currently in the vcpu_runstate_info structure we have fields that contain
>>> the time spent when the vcpu is in one of four states.  When trying to
>>> determine the physical utilization that a vcpu is requiring of the system
>>> from libxc we make a domctl operation call to the XEN_DOMCTL_getvcpuinfo
>>> command.  When processing this command we pull the value for the
>>> RUNSTATE_running field and attribute it to the processor that the vcpu is
>>> currently running on.
>>> 
>>> I believe that with the new credit scheduler these values are wrong as it
>>> is possible that the vcpu has time in RUNSTATE_running that was not
>>> actually spent on the currently running processor.  Is there a separate
>>> path through the libraries and into Xen that retrieves this information
>>> in a more correct manner?  Given that a vcpu can migrate among several
>>> physical processors in the time between samples it seems that a finer
>>> grained accounting mechanism is needed to correctly report physical
>>> utilization of the hardware resources.