[PATCH 0/1] cputime: Make the reported utime+stime correspond to the actual runtime.

[Fredrik Markstrom <fredrik.markstrom@gmail.com>]

We've hunted a bug that manifests itself in top showing abnormal (>100%)
loads on single threads, sometimes top can show a couple of thousand percent. 

We've found the cause of this behavior in cputime_adjust() where the kernel
tries to split the actual runtime into user and system time based on the number
of times that thread was found executing in system or user mode respectively.

The actual problem is a piece of code that *independently* updates the reported
utime and stime to force it to be monotonically increasing.

A small application tailored to trigger this case causes top to output the
following  (the load should always be ~20%). The application can be found at:

https://gist.github.com/frma71/706410acbd31556dbbfa

The cause of this is the data read from /proc/<pid>/stat as demonstrated by
the awk-script further down in this email.

The application to reproduce it is not very robust and will sometimes not
reliably reproduce it, on my arndale (2*CA15) it does at least nine
times out of ten. Anyway, the code should explain the problem pretty clearly.
 
I will follow up with my proposed fix (tested with next-20150610) in a minute or so.

# top -d 1 -H -b -p $(pidof loadbug) |grep loadbug
3100 root 20 0 1596 964 700 S 22 0.0 0:00.47 loadbug
3100 root 20 0 1596 964 700 R 21 0.0 0:00.69 loadbug
3100 root 20 0 1596 964 700 S 22 0.0 0:00.92 loadbug
3100 root 20 0 1596 964 700 S 21 0.0 0:01.14 loadbug
3100 root 20 0 1596 964 700 R 21 0.0 0:01.36 loadbug
3100 root 20 0 1596 964 700 S 22 0.0 0:01.59 loadbug
3100 root 20 0 1596 964 700 S 21 0.0 0:01.81 loadbug
3100 root 20 0 1596 964 700 S 22 0.0 0:02.04 loadbug
3100 root 20 0 1596 964 700 S 204 0.0 0:04.13 loadbug <==
3100 root 20 0 1596 964 700 R 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 S 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 S 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 S 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 S 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 R 0 0.0 0:04.13 loadbug
3100 root 20 0 1596 964 700 S 0 0.0 0:04.13 loadbug 
3100 root 20 0 1596 964 700 S 209 0.0 0:06.28 loadbug

% while true ; do cat /proc/$(pidof loadbug)/task/*/stat ; sleep 2 ; done | awk '{print $14 " " $15 " Delta=" $14+$15-L ;L=$14+$15}'

0 33 Delta=-103
0 77 Delta=44
0 121 Delta=44
0 165 Delta=44
0 208 Delta=43
254 208 Delta=254  <==
254 208 Delta=0
254 208 Delta=0
254 208 Delta=0
254 208 Delta=0
494 208 Delta=240
494 208 Delta=0
494 208 Delta=0
494 208 Delta=0
494 208 Delta=0
717 208 Delta=223
717 208 Delta=0

Fredrik Markstrom (1):
  cputime: Make the reported utime+stime correspond to the actual
    runtime.

 kernel/sched/cputime.c | 46 +++++++++++++++++++---------------------------
 1 file changed, 19 insertions(+), 27 deletions(-)

-- 
1.9.1