How to measue the total CPU specific load (including real time tasks)?

How to measue the total CPU specific load (including real time tasks)?

[Koehrer Mathias (ETAS/ESW5)]

Hi all,

we use the RT_PREEMPT kernel fairly long now and it is working really fine.
However, I have one question that is related to measure the available "CPU headroom" on all available CPU cores.
We are using an Intel Core i7 PC (32 and 64bit, Debian distribution, vanilla kernel + rt_preempt patch).
The question is the same for kernel 3.2.x and 3.18.x.

On the PC a couple of real time threads are running. They are hard bound to a particular CPU core 
(via pthread_setaffinity_np or sched_setaffinity).
As we are using the boot parameter  "isolcpus=1-31", all processes run per default on CPU core 0.
Now, I want to measure the total CPU specific load.
How can this be done easily?

When I use "top" in thread-mode I see the relative CPU load for particular threads/processes.
However the "idle" value in the "top" summary does not fit. With some experiments I saw the idle value for a CPU core to 
be nearly 100% even if on the very same CPU core a real time thread was consuming about 40% CPU time.
I looked into /proc/stat to take the idle tick value but this value seems not to consider the real time threads.


The only approach I found so far is to parse all /proc/PID/stat files periodically and to extract the 
fields utime, stime, cutime, cstime, task_cpu. These values can be used to compute the CPU specific load.
However, the effort to do so is really high.

Is there any /proc file (or something similar) I can use to get the total CPU specific load (including real time threads)?
Or do I have to enable a particular kernel configuration to get this value?

Thanks for any feedback on this question.

Regards

Mathias

Re: How to measue the total CPU specific load (including real time tasks)?

[Joe Korty]

On Tue, Jul 07, 2015 at 08:18:08AM -0400, Koehrer Mathias (ETAS/ESW5) wrote:
> Hi all,
> 
> we use the RT_PREEMPT kernel fairly long now and it is working really fine.
> However, I have one question that is related to measure the available "CPU
> headroom" on all available CPU cores.  We are using an Intel Core i7 PC
> (32 and 64bit, Debian distribution, vanilla kernel + rt_preempt patch).
> The question is the same for kernel 3.2.x and 3.18.x.
> 
> On the PC a couple of real time threads are running. They are hard bound
> to a particular CPU core (via pthread_setaffinity_np or sched_setaffinity).
> As we are using the boot parameter  "isolcpus=1-31", all processes run per
> default on CPU core 0.  Now, I want to measure the total CPU specific load.
> How can this be done easily?
> 
> When I use "top" in thread-mode I see the relative CPU load for particular
> threads/processes.  However the "idle" value in the "top" summary does
> not fit. With some experiments I saw the idle value for a CPU core to
> be nearly 100% even if on the very same CPU core a real time thread was
> consuming about 40% CPU time.  I looked into /proc/stat to take the idle
> tick value but this value seems not to consider the real time threads.
> 
> 
> The only approach I found so far is to parse all /proc/PID/stat files
> periodically and to extract the fields utime, stime, cutime, cstime,
> task_cpu. These values can be used to compute the CPU specific load.
> However, the effort to do so is really high.
> 
> Is there any /proc file (or something similar) I can use to get the total
> CPU specific load (including real time threads)?  Or do I have to enable
> a particular kernel configuration to get this value?
> 
> Thanks for any feedback on this question.
> 
> Regards
> Mathias


I think that if you turn on CONFIG_VIRT_CPU_ACCOUNTING_GEN,
you will get per-cpu high resolution idle time accounting
as a side effect.  (I'm looking at the 3.18 kernel, YMMV
for earlier kernels).

Regards,
Joe

RE: How to measue the total CPU specific load (including real time tasks)?

[Koehrer Mathias (ETAS/ESW5)]

Hi Joe,

> > we use the RT_PREEMPT kernel fairly long now and it is working really fine.
> > However, I have one question that is related to measure the available "CPU
> > headroom" on all available CPU cores.  We are using an Intel Core i7 PC
> > (32 and 64bit, Debian distribution, vanilla kernel + rt_preempt patch).
> > The question is the same for kernel 3.2.x and 3.18.x.
> >
> > On the PC a couple of real time threads are running. They are hard bound
> > to a particular CPU core (via pthread_setaffinity_np or sched_setaffinity).
> > As we are using the boot parameter  "isolcpus=1-31", all processes run per
> > default on CPU core 0.  Now, I want to measure the total CPU specific load.
> > How can this be done easily?
> >
> > When I use "top" in thread-mode I see the relative CPU load for particular
> > threads/processes.  However the "idle" value in the "top" summary does
> > not fit. With some experiments I saw the idle value for a CPU core to
> > be nearly 100% even if on the very same CPU core a real time thread was
> > consuming about 40% CPU time.  I looked into /proc/stat to take the idle
> > tick value but this value seems not to consider the real time threads.
> >
> >
> > The only approach I found so far is to parse all /proc/PID/stat files
> > periodically and to extract the fields utime, stime, cutime, cstime,
> > task_cpu. These values can be used to compute the CPU specific load.
> > However, the effort to do so is really high.
> >
> > Is there any /proc file (or something similar) I can use to get the total
> > CPU specific load (including real time threads)?  Or do I have to enable
> > a particular kernel configuration to get this value?
> >
> > Thanks for any feedback on this question.
> >
> > Regards
> > Mathias
> 
> 
> I think that if you turn on CONFIG_VIRT_CPU_ACCOUNTING_GEN,
> you will get per-cpu high resolution idle time accounting
> as a side effect.  (I'm looking at the 3.18 kernel, YMMV
> for earlier kernels).
> 
> Regards,
> Joe
the x86 (32bit) does not support HAVE_VIRT_CPU_ACCOUNTING_GEN. Only the 64bit version does...
And I need an approach that works for 32 and 64 bit x86.
In the meantime  I have found a patch from Carsten Emde that is used on the OSADL QA farm:
https://www.osadl.org/monitoring/patches/r5s1/sched-add-per-cpu-load-measurement.patch

This patch does exactly what I need...
One minor note on this patch:
There is an issue with writing to /proc/idleruntime/resetall.
The following patch fixes that...

Index: linux-3.18.13-rt10/kernel/sched/cpu_idleruntime.c
===================================================================
--- linux-3.18.13-rt10.orig/kernel/sched/cpu_idleruntime.c
+++ linux-3.18.13-rt10/kernel/sched/cpu_idleruntime.c
@@ -82,7 +82,6 @@ static ssize_t idleruntime_resetall(stru
 
 static const struct file_operations idleruntime_resetall_fops = {
 	.write = idleruntime_resetall,
-	.release = single_release,
 };
 
 static int idleruntime_open(struct inode *inode, struct file *file)


Regards

Mathias