[Bug 55061] New: cpu stats reported in cgroup file is erroneous

[Bug 55061] New: cpu stats reported in cgroup file is erroneous

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55061

           Summary: cpu stats reported in cgroup file is erroneous
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.5.3-1.fc17.i686.PAE
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: roopagovind@yahoo.com
        Regression: No


While testing the virsh cpu-stats functionality, I found that the stats
reported in the command output is incorrect, specifically the reported user
time and system time.

Please find a sample output.

virsh cpu-stats fedora-18 --total --start 0 --count 4
CPU0:
    cpu_time            15.321903938 seconds
CPU1:
    cpu_time            11.410812819 seconds
CPU2:
    cpu_time             2.948269295 seconds
CPU3:
    cpu_time             1.724287630 seconds
Total:
    cpu_time            31.405273682 seconds
    user_time            1.310000000 seconds
    system_time         12.860000000 seconds


As you can see here, the user time is less than system_time which is incorrect.
As an experiment, I tried running a while 1 loop inside this guest and observed
that stats for a period of time was not changing appropriately or rather not
the expected way.

While the while 1 loop was running inside the guest, the user_time continued to
be less than system_time which is incorrect. More over the guest is not running
any workload expect this while 1 loop.

So definitely there seems to be a bug in the cgroup accounting of cpu stats.
Please fix this cgroup bug.

I've verified this against different distro versions and flavors and hit the
same issue across all these test environments.

--roopa

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55061] cpu stats reported in cgroup file is erroneous

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55061


Prasad Gajanan Joshi. <prasadjoshi124@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |prasadjoshi124@gmail.com




--- Comment #1 from Prasad Gajanan Joshi. <prasadjoshi124@gmail.com>  2013-03-23 11:22:01 ---
user_time and system_time reported by virsh are from the point of view of the
domain on the host machine. Assuming it is QEMU, the stats reported are for the
QEMU process which is created when VM is started.

A while 1 loop inside VM would only change the vcpu_time and cpu_time.
Therefore, user_time or system_time would not be modified at all. cgroups
inside the VM is proper place to look for the stats you are looking for.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55061] cpu stats reported in cgroup file is erroneous

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55061





--- Comment #2 from Roopa Govind <roopagovind@yahoo.com>  2013-03-26 17:24:51 ---
(In reply to comment #1)
> user_time and system_time reported by virsh are from the point of view of the
> domain on the host machine. Assuming it is QEMU, the stats reported are for the
> QEMU process which is created when VM is started.
> 

Agree, it is reported under cpuacct.usage file under the /<guest-name>/ cgroup.

> A while 1 loop inside VM would only change the vcpu_time and cpu_time.
> Therefore, user_time or system_time would not be modified at all. cgroups
> inside the VM is proper place to look for the stats you are looking for.

I beg to disagree here. I performed this test.

For my guest my 
1. # of vcpus = # of host cpus
2. Used cpu load generator (https://github.com/beloglazov/cpu-load-generator)
inside the guest to generate 90% load for 600 seconds.

#python cpu-load-generator.py -n 4 600 test.data

test.data has values 90 and 80 (for generating 90% and 80% load)

3. changed the cpu shares for my cgroup /<guest-name> to 4096 and to my rest of
the system to 200 using cgset

4. Now ran a sar utility on the host with following options

sar -u ALL 1 600 

and i could see that my %guest utilization percentage shot up to ~90% and ~80%
respectively and %idle was showing ~10% and ~20% while the cpu-load generator
was running inside the guest. 

As soon as the cpu-load-generator stopped the cpu utilization drastically
reduced to 95% idle and 5% utilization.

When I observed the cgroup cpuacct.usage file for my guest under /<guest-name>,
i found that user time < system time.


Now i repeated steps 1-4 with a twist. Before start of my cpu-load-generator in 
my guest, i cleared the cpuacct.usage file to 0 value using the cgset -r
cpuacct.usage=0 /<guest-name>

and reran the whole steps. Now I could find that my user time was drastically
improving compared to system time in cpuacct.usage file. So it does report it
correctly if I set the values to 0 before starting my test.

Let me know if you like to see some screenshots, i can provide you the same.

Thank You for looking into this.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55061] cpu stats reported in cgroup file is erroneous

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55061


Roopa Govind <roopagovind@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Platform|All                         |i386




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.