Snmp cpuRawIdle showing double

Snmp cpuRawIdle showing double

[Nathan March]

Posted this to xen-users back in Oct
(http://www.gossamer-threads.com/lists/xen/users/186768), but just
confirmed that I'm still seeing this behavior on 4.1.

Polling the UCE-SNMP-MIB::ssCpuRawIdle counter results in a total that's
double the actual number of cpu's on the system, so for example:

nms ~ # snmpwalk -v 2c -c 'XXX' xen1 1.3.6.1.4.1.2021.11.53 && sleep 60
&& snmpwalk -v 2c -c 'XXX' xen1 1.3.6.1.4.1.2021.11.53
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 1313274
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 1324862

(1324862 - 1313274) / 6000 * 100 = 193%

This is polling a dom0 restricted to 1 core using "dom0_max_vcpus=1
dom0_vcpus_pin=true", both proc/cpuinfo and top show only 1 core.

I also see the issue on the domU's, the domU's are running a stock
kernel which does not exhibit this behavior on physical hardware.

This is using stock xen 4.1 with the xen/stable-2.6.32.x kernel, using
net-snmp 5.4.3-r6. Anyone have any insight here or info on how I might
be able to debug this further? I'm not even sure where to look beyond this.

Thanks!

- Nathan

-- 
Nathan March <nathan@gt.net>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806

Re: Snmp cpuRawIdle showing double

[Nathan March]

On 4/5/2011 2:10 PM, Nathan March wrote:
> Polling the UCE-SNMP-MIB::ssCpuRawIdle counter results in a total that's
> double the actual number of cpu's on the system, so for example:
>
> nms ~ # snmpwalk -v 2c -c 'XXX' xen1 1.3.6.1.4.1.2021.11.53 && sleep 60
> && snmpwalk -v 2c -c 'XXX' xen1 1.3.6.1.4.1.2021.11.53
> UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 1313274
> UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 1324862
>
> (1324862 - 1313274) / 6000 * 100 = 193%

Dug into the net-snmp source code to figure out where it pulls that data
from, here's a test case:

x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60 &&
y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && echo -e
"X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)

xen1 ~ # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60
&& y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && echo -e
"X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)    
X:15718848
Y:15730453
IDLE: 193.400

>From the man page, it says the user_hz is 1/100th:

              cpu  3357 0 4313 1362393
                     The amount of time, measured in units of USER_HZ
(1/100ths of a second on most architectures, use sysconf(_SC_CLK_TCK) to
obtain the right value), that the system spent in user mode, user mode
                     with low priority (nice), system mode, and the idle
task, respectively.  The last value should be USER_HZ times the second
entry in the uptime pseudo-file.

and sysconf agrees:

xen1 ~ # perl -e 'use POSIX; print POSIX::sysconf(&POSIX::_SC_CLK_TCK);'
100

Anyone have any ideas?

This works as expected on a very outdated dom0 I have, 2.6.21 and xen
3.2.1. It seems to be broken on at least xen 4.0 (2.6.31.13) and above.

- Nathan

Re: Snmp cpuRawIdle showing double

[Nathan March]

On 4/6/2011 10:51 AM, Nathan March wrote:
> Dug into the net-snmp source code to figure out where it pulls that data
> from, here's a test case:
>
> x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60 &&
> y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && echo -e
> "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
>
> xen1 ~ # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60
> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && echo -e
> "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)    
> X:15718848
> Y:15730453
> IDLE: 193.400
>

Anyone?

Could I ask some people to try running this script on their non-loaded
dom0 and see what it returns?

x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60 &&
y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && echo -e
"X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)

The Idle output should close to 100% per cpu in the dom0.

- Nathan