CPU Utilization

CPU Utilization

[Dave Thompson (davetho)]

I have a question concerning the CPU utilization number provided by
the 'xm top' command in xen 3.0.  I've noticed that if I run a TCP
performance tool on domain 0 which simply receives and echos back
a TCP stream, the output of 'xm top' during the test does not match
up with the output of 'top' on domain 0 in terms of CPU utilization
(nor with the CPU utilization computed by the test program).

In this example, 'xm top' shows a CPU utilization for domain 0 of
around 65% during the test while 'top' at the same time indicates
that the domain is around 70% idle (the test tool also indicates
that the CPU utilization averages about 30% during the test).

Output from 'xm top':
xentop - 12:44:29   Xen 3.0.0
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 2094904k total, 559792k used, 1535112k free    CPUs: 2 @ 3800MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) SSID
  Domain-0 -----r        273   66.6     519192   24.8   no limit
n/a     2    8 17120030 17489822    0
VCPUs(sec):   0:        273s

Output from 'top':
Tasks:  69 total,   2 running,  67 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0% us,  7.3% sy,  0.0% ni, 73.3% id,  0.0% wa,  3.3% hi,
15.0% si
Mem:    505556k total,   497812k used,     7744k free,   109624k buffers
Swap:  2104472k total,        0k used,  2104472k free,   212904k cached

Any idea why these displays disagree?

Thanks,
Dave Thompson

Re: CPU Utilization

[Rob Gardner]

Dave Thompson (davetho) wrote:

>I have a question concerning the CPU utilization number provided by
>the 'xm top' command in xen 3.0.  I've noticed that if I run a TCP
>performance tool on domain 0 which simply receives and echos back
>a TCP stream, the output of 'xm top' during the test does not match
>up with the output of 'top' on domain 0 in terms of CPU utilization
>(nor with the CPU utilization computed by the test program).
>
>...
>
>Any idea why these displays disagree?
>  
>

Top shows cpu usage for user processes in dom0, while xm top shows cpu 
usage for the domain, which includes cpu time spent in the dom0 kernel. 
Perhaps the thing you're running incurs lots of kernel cpu time, but not 
user time?

Rob

CPU utilization

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Team,
I am getting very high values of cpu usage and very less CPU idle time. Following is the output of /proc/stat hwere cpu idle time is 167 which is less than a 1% of total cpu time. Please look at your system and paste your output.
root@tiogapass:~# cat /proc/stat cpu 16893302 999635 6823650 167 0 0 56950 0 0 0

Does anyone know of this issue or am I missing any kernel configuration?

Regards
-Vijay

[-- Attachment #2: Type: text/html, Size: 2351 bytes --]

RE: CPU Utilization

[Dave Thompson (davetho)]

> -----Original Message-----
> From: Andrew Theurer [mailto:habanero@us.ibm.com] 
> Sent: Monday, December 12, 2005 9:24 PM
> To: Dave Thompson (davetho)
> Cc: Anthony Liguori; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] CPU Utilization
> 
> >
> >But what else is running?  In this case I only have dom0 configured,
> >there is no domU.  The only other possibility would be the hypervisor
> >and I hope the hypervisor is not accounting for the other 30%.
> >  
> 
> If xend is started, you may have the software bridge running 
> which can use as much as 10% cpu.

But I would think that the bridge activity should be showing up
in the top CPU summary as well.  It is running on domain 0 after all.
I know one person suggested that kernel activity is not represented
in the top CPU util output.  But I don't see how that can be right.
If so, where else is that time accounted for?  It seems to be all
there (in the sy, hi, and si values).

> Also, I don't see soft ints in that top output.  
> That could also be another ~7% cpu.

Soft interrupt time is accounted for in the si field (15%) of the
summary.  I believe that is where most (if not all) of the TCP
processing is performed. Here is the top CPU summary display again:

Cpu(s):  1.0% us,  7.3% sy,  0.0% ni, 73.3% id,  0.0% wa,  3.3% hi,
15.0% si

> Also xen is doing some work, receiving the real interrupts
> and generating virtual interrupts to dom0, so with all this,
> it is possible that you are using another 30% unseen 
> in top.

But aren't the hypervisor calls actually still being accounted for
by the domain since clock ticks are not lost but made up for in the
timer_interrupt() function of arch/xen/i386/kiernel/time.c?  The
only issue is really when a domain is preempted by another domain
by the xen scheduler and this is actually a problem in the other
direction.  The swapped out domain will still account for the
time in whichever time bucket it was using when the domain was
preempted (so the same time is accounted for by both domains).
Basically the aggregated CPU time for all domains on a CPU could
add greater than 100% because of this.  If the domain is
re-scheduled because of a SCHEDOP_block in the idle loop, the time
will be properly accounted for as idle time.

However, none of this really matters for my case since I am
only running domain 0, there is no guest domain.  I just want
a good explanation why 'xm top' is reporting 30% more CPU utilization
than top in this case.

> Best way to confirm this would be to use xenoprofile.

Xenoprof is great for seeing which kernel functions are taking
the majority of time but does it really help with CPU utilization?
It counts (in the default case) unhalted clock cycles and in the
xen idle loop the processor is halted (to save power) so the
clock cycles are not accounted for.  Is this right or am I
missing something.

> -Andrew
 

Thanks,
Dave

Re: CPU Utilization

[Andrew Theurer]

Dave Thompson (davetho) wrote:

>>-----Original Message-----
>>From: Andrew Theurer [mailto:habanero@us.ibm.com] 
>>Sent: Monday, December 12, 2005 9:24 PM
>>To: Dave Thompson (davetho)
>>Cc: Anthony Liguori; xen-devel@lists.xensource.com
>>Subject: Re: [Xen-devel] CPU Utilization
>>
>>
>>>But what else is running?  In this case I only have dom0 configured,
>>>there is no domU.  The only other possibility would be the hypervisor
>>>and I hope the hypervisor is not accounting for the other 30%.
>>> 
>>>
>>If xend is started, you may have the software bridge running 
>>which can use as much as 10% cpu.
>>
>
>But I would think that the bridge activity should be showing up
>in the top CPU summary as well.  It is running on domain 0 after all.
>I know one person suggested that kernel activity is not represented
>in the top CPU util output.  But I don't see how that can be right.
>If so, where else is that time accounted for?  It seems to be all
>there (in the sy, hi, and si values).
>
>
>>Also, I don't see soft ints in that top output.  
>>That could also be another ~7% cpu.
>>
>
>Soft interrupt time is accounted for in the si field (15%) of the
>summary.  I believe that is where most (if not all) of the TCP
>processing is performed. Here is the top CPU summary display again:
>
>Cpu(s):  1.0% us,  7.3% sy,  0.0% ni, 73.3% id,  0.0% wa,  3.3% hi,
>15.0% si
>
>
Sorry, I overlooked the si.

>>Also xen is doing some work, receiving the real interrupts
>>and generating virtual interrupts to dom0, so with all this,
>>it is possible that you are using another 30% unseen 
>>in top.
>>
>
>But aren't the hypervisor calls actually still being accounted for
>by the domain since clock ticks are not lost but made up for in the
>timer_interrupt() function of arch/xen/i386/kiernel/time.c?  The
>only issue is really when a domain is preempted by another domain
>by the xen scheduler and this is actually a problem in the other
>direction.  The swapped out domain will still account for the
>time in whichever time bucket it was using when the domain was
>preempted (so the same time is accounted for by both domains).
>Basically the aggregated CPU time for all domains on a CPU could
>add greater than 100% because of this.  If the domain is
>re-scheduled because of a SCHEDOP_block in the idle loop, the time
>will be properly accounted for as idle time.
>
I wonder if this is working under all situations.  This problem seems 
familiar.  Before the kernel accounted for si and hi properly, we had a 
very similar situation with this type of workload: lots of cpu time 
unaccounted for because the interrupt processing happend mostly when the 
system was idle, and the timer tick did not account for this properly.  
I wonder if we have a similar problem in xen/linux.  If lost ticks are 
"queued up" but accounted for just one type of mode, then I think we 
could be way off in some sitations like this.

>
>However, none of this really matters for my case since I am
>only running domain 0, there is no guest domain.  I just want
>a good explanation why 'xm top' is reporting 30% more CPU utilization
>than top in this case.
>
>
>>Best way to confirm this would be to use xenoprofile.
>>
>
>Xenoprof is great for seeing which kernel functions are taking
>the majority of time but does it really help with CPU utilization?
>It counts (in the default case) unhalted clock cycles and in the
>xen idle loop the processor is halted (to save power) so the
>clock cycles are not accounted for.  Is this right or am I
>missing something.
>
I guess I was hoping to find a smoking gun in xen :).  The only other 
thing I think we could do is count the number of total samples we got 
over x seconds and compare this with the number of samples we would get 
in the same time period on a 100% busy system.  We should then be able 
to figure out how much % time the cpu was halted.

-Andrew

RE: CPU Utilization

[Dave Thompson (davetho)]

 

> -----Original Message-----
> From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
> Sent: Monday, December 12, 2005 5:01 PM
> To: Dave Thompson (davetho)
> Cc: Rob Gardner; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] CPU Utilization
> 
> top isn't smart enough to realize that for a portion of the running 
> time, a domain has been pre-empted out and is not running.
> 
> top will then charge whatever was running at pre-emption time for the 
> time that other domain running.

But what else is running?  In this case I only have dom0 configured,
there is no domU.  The only other possibility would be the hypervisor
and I hope the hypervisor is not accounting for the other 30%.

> Regards,
> 
> Anthony Liguori
> 

Re: CPU Utilization

[Andrew Theurer]

Dave Thompson (davetho) wrote:

> 
>
>  
>
>>-----Original Message-----
>>From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
>>Sent: Monday, December 12, 2005 5:01 PM
>>To: Dave Thompson (davetho)
>>Cc: Rob Gardner; xen-devel@lists.xensource.com
>>Subject: Re: [Xen-devel] CPU Utilization
>>
>>top isn't smart enough to realize that for a portion of the running 
>>time, a domain has been pre-empted out and is not running.
>>
>>top will then charge whatever was running at pre-emption time for the 
>>time that other domain running.
>>    
>>
>
>But what else is running?  In this case I only have dom0 configured,
>there is no domU.  The only other possibility would be the hypervisor
>and I hope the hypervisor is not accounting for the other 30%.
>  
>
If xend is started, you may have the software bridge running which can 
use as much as 10% cpu. Also, I don't see soft ints in that top output.  
That could also be another ~7% cpu.  Also xen is doing some work, 
receiving the real interrupts and generating virtual interrupts to dom0, 
so with all this, it is possible that you are using another 30% unseen 
in top.  Best way to confirm this would be to use xenoprofile.

-Andrew

>  
>

RE: CPU Utilization

[Dave Thompson (davetho)]

 

> -----Original Message-----
> From: Rob Gardner [mailto:rob.gardner@hp.com] 
> Sent: Monday, December 12, 2005 1:23 PM
> To: Dave Thompson (davetho)
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] CPU Utilization
> 
> Dave Thompson (davetho) wrote:
> 
> >I have a question concerning the CPU utilization number provided by
> >the 'xm top' command in xen 3.0.  I've noticed that if I run a TCP
> >performance tool on domain 0 which simply receives and echos back
> >a TCP stream, the output of 'xm top' during the test does not match
> >up with the output of 'top' on domain 0 in terms of CPU utilization
> >(nor with the CPU utilization computed by the test program).
> >
> >...
> >
> >Any idea why these displays disagree?
> >  
> >
> 
> Top shows cpu usage for user processes in dom0, while xm top 
> shows cpu usage for the domain, which includes cpu time spent
> in the dom0 kernel.

Doesn't top's CPU summary line show both?  I thought us=user space,
sy=system, ni=nice, id=idle, hi=hwintr, si=softintr.  So in my
example 1% in user space, 7.3% in system level, 3.3% in h/w intr,
and 15% in soft intr.  Total of 26.6% utilized of which 25.6% is
in kernel.
 
> Perhaps the thing you're running incurs lots of kernel cpu 
> time, but not 
> user time?
> 
> Rob
> 

Re: CPU Utilization

[Rob Gardner]

[-- Attachment #1.1: Type: text/plain, Size: 687 bytes --]

Dave Thompson (davetho) wrote:

>
>Doesn't top's CPU summary line show both?  I thought us=user space,
>sy=system, ni=nice, id=idle, hi=hwintr, si=softintr.  So in my
>example 1% in user space, 7.3% in system level, 3.3% in h/w intr,
>and 15% in soft intr.  Total of 26.6% utilized of which 25.6% is
>in kernel.
> 
>  
>

I think that 'top' shows system time spent on behalf of user processes. 
Dom0 has stuff going on in the kernel that has nothing to do with user 
processes. For instance, when you generate lan traffic to/from another 
domain, it causes the dom0 kernel to use a bunch of cpu time, and this 
may not show up using 'top'. That's my guess about what is happening.

Rob


[-- Attachment #1.2: Type: text/html, Size: 1053 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: CPU Utilization

[Anthony Liguori]

top isn't smart enough to realize that for a portion of the running 
time, a domain has been pre-empted out and is not running.

top will then charge whatever was running at pre-emption time for the 
time that other domain running.

Regards,

Anthony Liguori

Dave Thompson (davetho) wrote:

> 
>
>  
>
>>-----Original Message-----
>>From: Rob Gardner [mailto:rob.gardner@hp.com] 
>>Sent: Monday, December 12, 2005 1:23 PM
>>To: Dave Thompson (davetho)
>>Cc: xen-devel@lists.xensource.com
>>Subject: Re: [Xen-devel] CPU Utilization
>>
>>Dave Thompson (davetho) wrote:
>>
>>    
>>
>>>I have a question concerning the CPU utilization number provided by
>>>the 'xm top' command in xen 3.0.  I've noticed that if I run a TCP
>>>performance tool on domain 0 which simply receives and echos back
>>>a TCP stream, the output of 'xm top' during the test does not match
>>>up with the output of 'top' on domain 0 in terms of CPU utilization
>>>(nor with the CPU utilization computed by the test program).
>>>
>>>...
>>>
>>>Any idea why these displays disagree?
>>> 
>>>
>>>      
>>>
>>Top shows cpu usage for user processes in dom0, while xm top 
>>shows cpu usage for the domain, which includes cpu time spent
>>in the dom0 kernel.
>>    
>>
>
>Doesn't top's CPU summary line show both?  I thought us=user space,
>sy=system, ni=nice, id=idle, hi=hwintr, si=softintr.  So in my
>example 1% in user space, 7.3% in system level, 3.3% in h/w intr,
>and 15% in soft intr.  Total of 26.6% utilized of which 25.6% is
>in kernel.
> 
>  
>
>>Perhaps the thing you're running incurs lots of kernel cpu 
>>time, but not 
>>user time?
>>
>>Rob
>>
>>    
>>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: CPU Utilization

[Rik van Riel]

On Mon, 12 Dec 2005, Anthony Liguori wrote:

> top isn't smart enough to realize that for a portion of the running 
> time, a domain has been pre-empted out and is not running.
> 
> top will then charge whatever was running at pre-emption time for the 
> time that other domain running.

Actually, the kernel does this charging of time to the wrong
entity.  I am working on a patch to account the preempted time
as "cpu steal" time (like s390 Linux does).

Here is the (really simple) hypervisor bit of the patch. The
kernel bits need some more work ;)


Index: xen-3.0/xen/common/domain.c
===================================================================
--- xen-3.0.orig/xen/common/domain.c
+++ xen-3.0/xen/common/domain.c
@@ -431,8 +431,25 @@ long do_vcpu_op(int cmd, int vcpuid, voi
     case VCPUOP_is_up:
         rc = !test_bit(_VCPUF_down, &v->vcpu_flags);
         break;
+
+    case VCPUOP_cpu_time:
+	{
+	    dom0_getvcpuinfo_t vi;
+	    vi.online = !test_bit(_VCPUF_down, &v->vcpu_flags);
+	    vi.blocked = test_bit(_VCPUF_blocked, &v->vcpu_flags);
+	    vi.running  = test_bit(_VCPUF_running, &v->vcpu_flags);
+	    vi.cpu_time = v->cpu_time;
+	    vi.cpu = v->processor;
+	    vi.cpumap = v->cpumap;
+	    rc = 0;
+
+	    if ( copy_to_user(arg, &vi, sizeof(dom0_getvcpuinfo_t)) )
+		rc = -EFAULT;
+	    break;
+	}
     }
 
+
     return rc;
 }
 
Index: xen-3.0/xen/include/public/vcpu.h
===================================================================
--- xen-3.0.orig/xen/include/public/vcpu.h
+++ xen-3.0/xen/include/public/vcpu.h
@@ -51,6 +51,14 @@
 /* Returns 1 if the given VCPU is up. */
 #define VCPUOP_is_up                3
 
+/*
+ * Get information on how much CPU time this VCPU has used, etc...
+ *
+ * @extra_arg == pointer to an empty dom0_getvcpuinfo_t, the "OUT" variables
+ *               of which filled in with scheduler info.
+ */
+#define VCPUOP_cpu_time             4
+
 #endif /* __XEN_PUBLIC_VCPU_H__ */
 
 /*

CPU Utilization.

[Preethi M]

[-- Attachment #1.1: Type: text/plain, Size: 645 bytes --]

Hi,
 I have a basic question. I would like to know the best way to find out the
CPU utilization in Xen. For example if i have a setup with Dom 0 and Dom 1
and some partitions are exported to Dom 1. When i perform IOs in Domain 1,
since the actual disk device is under the control of Domain 0, the backend
driver will be responsible for the actual IO operation completion. So it's
obvious some amount of CPU will be utilized in Domain 0. In this scenario, i
would like to know what is the best way to identify the consolidated CPU
utilization both in Domain 0 and Domian 1. Is there any tool available which
does this?
 Thanks,
Pree.

[-- Attachment #1.2: Type: text/html, Size: 711 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: CPU Utilization.

[Rob Gardner]

Preethi M wrote: 

> I have a basic question. I would like to know the best way to find out 
> the CPU utilization in Xen. For example if i have a setup with Dom 0 
> and Dom 1 and some partitions are exported to Dom 1. When i perform 
> IOs in Domain 1, since the actual disk device is under the control of 
> Domain 0, the backend driver will be responsible for the actual IO 
> operation completion. So it's obvious some amount of CPU will be 
> utilized in Domain 0. In this scenario, i would like to know what is 
> the best way to identify the consolidated CPU utilization both in 
> Domain 0 and Domian 1. Is there any tool available which does this?


Pree,

If you check back in the xen-devel archives, you'll find around late 
August I sent out a package called xenmon, which provides fairly 
fine-grained reporting of various metrics, including cpu utilization for 
each domain, and cumulative data over 1 second and 10 seconds. This tool 
can show some of what you are interested in. I am planning another 
release of xenmon "real soon now" that incorporates some improvements 
and bug fixes.

Rob Gardner

CPU utilization

[Brian Hays]

A question about CPU utilization...

If I have a domU that is under heavy processing load will it affect
the performance of other domU's or dom0? ...or is processing divided
in such a way that one domU will have no effect on others no matter
what load it is under?

Thank you,
Brian

Re: CPU utilization

[Keir Fraser]

The default BVT scheduler is weighted fair share with extensions to 
better support I/O. If you add extra domains then the existing ones 
will get proportionally less CPU, assuming all domains are CPU bound.

More directly answering your question: most sane CPU schedulers are 
work conserving, meaning that if there is work to be done then they 
will schedule it in preference to idling. This means that if, say, 
domain0 is the only domain with work to do then it will get 100% of the 
CPU, even if its 'fair share' is, say, 50%. Now, however strict you are 
in scheduling a runnable domU onto the same processor it is clearly 
going to have some negative impact on domain0's CPU allocation compared 
with dom0 getting all the CPU.

  -- Keir

On 1 May 2005, at 14:51, Brian Hays wrote:

> A question about CPU utilization...
>
> If I have a domU that is under heavy processing load will it affect
> the performance of other domU's or dom0? ...or is processing divided
> in such a way that one domU will have no effect on others no matter
> what load it is under?
>
> Thank you,
> Brian
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel