[PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[Mike Chan]

v2:
Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring,
which is based off of linus's tree. Thomas it might be easier to merge our
patches if you take these patches and put them in a series ontop of your
original patches (presuming there are no objections to your patch).

This patch series introduces cpu frequency and power tracking for cpuacct
cgroups. A similar patch set was discussed a while back and it was concluded
that due to varying architectures (ppc, x86 with overboot) you cannot account
for frequencies and their power consumption generically in sched.c, thus we
have platform specific hooks the cpuacct can call into (if available).

This patch series is not 3 instead of 4. I have left out the power
implementation for OMAP due to implementation conflicts in linux-next.

Mike Chan (3):
  scheduler: cpuacct: Enable platform hooks to track cpuusage for CPU
    frequencies
  scheduler: cpuacct: Enable platform callbacks for cpuacct power
    tracking
  omap: cpu: Implement callbacks for cpu frequency tracking in cpuacct

 Documentation/cgroups/cpuacct.txt |    7 ++++
 arch/arm/plat-omap/cpu-omap.c     |   67 +++++++++++++++++++++++++++++++++++-
 include/linux/cpuacct.h           |   43 +++++++++++++++++++++++
 kernel/cgroup_cpuaccount.c        |   69 +++++++++++++++++++++++++++++++++++++
 4 files changed, 185 insertions(+), 1 deletions(-)
 create mode 100644 include/linux/cpuacct.h

 Signed-off-by: Mike Chan <mike@android.com>

[PATCH v2 1/3] scheduler: cpuacct: Enable platform hooks to track cpuusage for CPU frequencies

[Mike Chan]

V2:
- Rebased off of Thomas Renninger's cgroup_cpuacct refactoring.
- Renamed function struct to cpuacct_chrage_calls  from cpuacct_cpufreq_calls

Introduce new platform callback hooks for cpuacct for tracking CPU frequencies

Not all platforms / architectures have a set CPU_FREQ_TABLE defined
for CPU transition speeds. In order to track time spent in at various
CPU frequencies, we enable platform callbacks from cpuacct for this accounting.

Architectures that support overclock boosting, or don't have pre-defined
frequency tables can implement their own bucketing system that makes sense
given their cpufreq scaling abilities.

New file:
cpuacct.cpufreq reports the CPU time (in nanoseconds) spent at each CPU
frequency.

Signed-off-by: Mike Chan <mike@android.com>
---
 Documentation/cgroups/cpuacct.txt |    4 +++
 include/linux/cpuacct.h           |   41 +++++++++++++++++++++++++++++++
 kernel/cgroup_cpuaccount.c        |   49 +++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/cpuacct.h

diff --git a/Documentation/cgroups/cpuacct.txt b/Documentation/cgroups/cpuacct.txt
index 8b93094..600d2d0 100644
--- a/Documentation/cgroups/cpuacct.txt
+++ b/Documentation/cgroups/cpuacct.txt
@@ -40,6 +40,10 @@ system: Time spent by tasks of the cgroup in kernel mode.
 
 user and system are in USER_HZ unit.
 
+cpuacct.cpufreq file gives CPU time (in nanoseconds) spent at each CPU
+frequency. Platform hooks must be implemented inorder to properly track
+time at each CPU frequency.
+
 cpuacct controller uses percpu_counter interface to collect user and
 system times. This has two side effects:
 
diff --git a/include/linux/cpuacct.h b/include/linux/cpuacct.h
new file mode 100644
index 0000000..6205d29
--- /dev/null
+++ b/include/linux/cpuacct.h
@@ -0,0 +1,41 @@
+/* include/linux/cpuacct.h
+ *
+ * Copyright (C) 2010 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef _CPUACCT_H_
+#define _CPUACCT_H_
+
+#ifdef CONFIG_CGROUP_CPUACCT
+
+#include <linux/cgroup.h>
+
+/*
+ * Platform specific CPU frequency hooks for cpuacct. These functions are
+ * called from the scheduler.
+ */
+struct cpuacct_charge_calls {
+	/*
+	 * Platforms can take advantage of this data and use
+	 * per-cpu allocations if necessary.
+	 */
+	void (*init) (void **cpuacct_data);
+	void (*charge) (void *cpuacct_data,  u64 cputime, unsigned int cpu);
+	void (*show) (void *cpuacct_data, struct cgroup_map_cb *cb);
+};
+
+int cpuacct_charge_register(struct cpuacct_charge_calls *fn);
+
+#endif /* CONFIG_CGROUPS */
+
+#endif /* _CPUACCT_H_ */
diff --git a/kernel/cgroup_cpuaccount.c b/kernel/cgroup_cpuaccount.c
index 0ad356a..11799a7 100644
--- a/kernel/cgroup_cpuaccount.c
+++ b/kernel/cgroup_cpuaccount.c
@@ -7,6 +7,7 @@
 #include <linux/slab.h>
 #include <linux/seq_file.h>
 #include <linux/err.h>
+#include <linux/cpuacct.h>
 
 #include <asm/cputime.h>
 
@@ -26,8 +27,30 @@ struct cpuacct {
 	u64 __percpu *cpuusage;
 	struct percpu_counter cpustat[CPUACCT_STAT_NSTATS];
 	struct cpuacct *parent;
+	struct cpuacct_charge_calls *cpufreq_fn;
+	void *cpuacct_data;
 };
 
+static struct cpuacct *cpuacct_root;
+
+/* Default calls for cpufreq accounting */
+static struct cpuacct_charge_calls *cpuacct_cpufreq;
+int cpuacct_charge_register(struct cpuacct_charge_calls *fn)
+{
+	cpuacct_cpufreq = fn;
+
+	/*
+	 * Root node is created before platform can register callbacks,
+	 * initalize here.
+	 */
+	if (cpuacct_root && fn) {
+		cpuacct_root->cpufreq_fn = fn;
+		if (fn->init)
+			fn->init(&cpuacct_root->cpuacct_data);
+	}
+	return 0;
+}
+
 struct cgroup_subsys cpuacct_subsys;
 
 /* return cpu accounting group corresponding to this container */
@@ -62,8 +85,16 @@ static struct cgroup_subsys_state *cpuacct_create(
 		if (percpu_counter_init(&ca->cpustat[i], 0))
 			goto out_free_counters;
 
+	ca->cpufreq_fn = cpuacct_cpufreq;
+
+	/* If available, have platform code initalize cpu frequency data */
+	if (ca->cpufreq_fn && ca->cpufreq_fn->init)
+		ca->cpufreq_fn->init(&ca->cpuacct_data);
+
 	if (cgrp->parent)
 		ca->parent = cgroup_ca(cgrp->parent);
+	else
+		cpuacct_root = ca;
 
 	return &ca->css;
 
@@ -191,6 +222,16 @@ static int cpuacct_stats_show(struct cgroup *cgrp, struct cftype *cft,
 	return 0;
 }
 
+static int cpuacct_cpufreq_show(struct cgroup *cgrp, struct cftype *cft,
+		struct cgroup_map_cb *cb)
+{
+	struct cpuacct *ca = cgroup_ca(cgrp);
+	if (ca->cpufreq_fn && ca->cpufreq_fn->show)
+		ca->cpufreq_fn->show(ca->cpuacct_data, cb);
+
+	return 0;
+}
+
 static struct cftype files[] = {
 	{
 		.name = "usage",
@@ -205,6 +246,10 @@ static struct cftype files[] = {
 		.name = "stat",
 		.read_map = cpuacct_stats_show,
 	},
+	{
+		.name =  "cpufreq",
+		.read_map = cpuacct_cpufreq_show,
+	},
 };
 
 static int cpuacct_populate(struct cgroup_subsys *ss, struct cgroup *cgrp)
@@ -234,6 +279,10 @@ void cpuacct_charge(struct task_struct *tsk, u64 cputime)
 	for (; ca; ca = ca->parent) {
 		u64 *cpuusage = per_cpu_ptr(ca->cpuusage, cpu);
 		*cpuusage += cputime;
+
+		/* Call back into platform code to account for CPU speeds */
+		if (ca->cpufreq_fn && ca->cpufreq_fn->charge)
+			ca->cpufreq_fn->charge(ca->cpuacct_data, cputime, cpu);
 	}
 
 	rcu_read_unlock();
-- 
1.7.0.1

[PATCH v2 2/3] scheduler: cpuacct: Enable platform callbacks for cpuacct power tracking

[Mike Chan]

V2:
- Rebased off Thomass Renninger's cgroup_cpuacct refactoring

Platform must register cpu power function that return power in
milliWatt seconds.

New file:
cpuacct.power reports the power consumed in milliWatt seconds

Signed-off-by: Mike Chan <mike@android.com>
---
 Documentation/cgroups/cpuacct.txt |    3 +++
 include/linux/cpuacct.h           |    4 +++-
 kernel/cgroup_cpuaccount.c        |   24 ++++++++++++++++++++++--
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/Documentation/cgroups/cpuacct.txt b/Documentation/cgroups/cpuacct.txt
index 600d2d0..84e471b 100644
--- a/Documentation/cgroups/cpuacct.txt
+++ b/Documentation/cgroups/cpuacct.txt
@@ -44,6 +44,9 @@ cpuacct.cpufreq file gives CPU time (in nanoseconds) spent at each CPU
 frequency. Platform hooks must be implemented inorder to properly track
 time at each CPU frequency.
 
+cpuacct.power file gives CPU power consumed (in milliWatt seconds). Platform
+must provide and implement power callback functions.
+
 cpuacct controller uses percpu_counter interface to collect user and
 system times. This has two side effects:
 
diff --git a/include/linux/cpuacct.h b/include/linux/cpuacct.h
index 6205d29..c17a634 100644
--- a/include/linux/cpuacct.h
+++ b/include/linux/cpuacct.h
@@ -31,7 +31,9 @@ struct cpuacct_charge_calls {
 	 */
 	void (*init) (void **cpuacct_data);
 	void (*charge) (void *cpuacct_data,  u64 cputime, unsigned int cpu);
-	void (*show) (void *cpuacct_data, struct cgroup_map_cb *cb);
+	void (*cpufreq_show) (void *cpuacct_data, struct cgroup_map_cb *cb);
+	/* Returns power consumed in milliWatt seconds */
+	u64 (*power_usage) (void *cpuacct_data);
 };
 
 int cpuacct_charge_register(struct cpuacct_charge_calls *fn);
diff --git a/kernel/cgroup_cpuaccount.c b/kernel/cgroup_cpuaccount.c
index 11799a7..d9bf889 100644
--- a/kernel/cgroup_cpuaccount.c
+++ b/kernel/cgroup_cpuaccount.c
@@ -226,12 +226,28 @@ static int cpuacct_cpufreq_show(struct cgroup *cgrp, struct cftype *cft,
 		struct cgroup_map_cb *cb)
 {
 	struct cpuacct *ca = cgroup_ca(cgrp);
-	if (ca->cpufreq_fn && ca->cpufreq_fn->show)
-		ca->cpufreq_fn->show(ca->cpuacct_data, cb);
+	if (ca->cpufreq_fn && ca->cpufreq_fn->cpufreq_show)
+		ca->cpufreq_fn->cpufreq_show(ca->cpuacct_data, cb);
 
 	return 0;
 }
 
+/* return total cpu power usage (milliWatt second) of a group */
+static u64 cpuacct_powerusage_read(struct cgroup *cgrp, struct cftype *cft)
+{
+	int i;
+	struct cpuacct *ca = cgroup_ca(cgrp);
+	u64 totalpower = 0;
+
+	if (ca->cpufreq_fn && ca->cpufreq_fn->power_usage)
+		for_each_present_cpu(i) {
+			totalpower += ca->cpufreq_fn->power_usage(
+					ca->cpuacct_data);
+		}
+
+	return totalpower;
+}
+
 static struct cftype files[] = {
 	{
 		.name = "usage",
@@ -250,6 +266,10 @@ static struct cftype files[] = {
 		.name =  "cpufreq",
 		.read_map = cpuacct_cpufreq_show,
 	},
+	{
+		.name = "power",
+		.read_u64 = cpuacct_powerusage_read
+	},
 };
 
 static int cpuacct_populate(struct cgroup_subsys *ss, struct cgroup *cgrp)
-- 
1.7.0.1

[PATCH v2 3/3] omap: cpu: Implement callbacks for cpu frequency tracking in cpuacct

[Mike Chan]

Implement OMAP platform specific scheduler callbacks for tracking
cpu frequencies per cpuacct cgroup.

Signed-off-by: Mike Chan <mike@android.com>
---
 arch/arm/plat-omap/cpu-omap.c |   67 ++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 66 insertions(+), 1 deletions(-)

diff --git a/arch/arm/plat-omap/cpu-omap.c b/arch/arm/plat-omap/cpu-omap.c
index 6d3d333..176417a 100644
--- a/arch/arm/plat-omap/cpu-omap.c
+++ b/arch/arm/plat-omap/cpu-omap.c
@@ -21,6 +21,8 @@
 #include <linux/err.h>
 #include <linux/clk.h>
 #include <linux/io.h>
+#include <linux/slab.h>
+#include <linux/cpuacct.h>
 
 #include <mach/hardware.h>
 #include <plat/clock.h>
@@ -38,6 +40,10 @@ static struct cpufreq_frequency_table *freq_table;
 
 static struct clk *mpu_clk;
 
+#ifdef CONFIG_CGROUP_CPUACCT
+static int freq_index;
+#endif
+
 /* TODO: Add support for SDRAM timing changes */
 
 int omap_verify_speed(struct cpufreq_policy *policy)
@@ -96,6 +102,11 @@ static int omap_target(struct cpufreq_policy *policy,
 	       freqs.old, freqs.new);
 #endif
 	ret = clk_set_rate(mpu_clk, freqs.new * 1000);
+#ifdef CONFIG_CGROUP_CPUACCT
+	/* Update freq_index before cpufreq transition post notification. */
+	cpufreq_frequency_table_target(policy, freq_table, freqs.new,
+			CPUFREQ_RELATION_L, &freq_index);
+#endif
 	cpufreq_notify_transition(&freqs, CPUFREQ_POSTCHANGE);
 
 	return ret;
@@ -125,7 +136,14 @@ static int __init omap_cpu_init(struct cpufreq_policy *policy)
 		policy->cpuinfo.max_freq = clk_round_rate(mpu_clk,
 							VERY_HI_RATE) / 1000;
 	}
-
+#ifdef CONFIG_CGROUP_CPUACCT
+	/*
+	 * Update freq_index, since we are using the extact frequency
+	 * we can use any relation.
+	 */
+	cpufreq_frequency_table_target(policy, freq_table, policy->cur,
+			CPUFREQ_RELATION_L, &freq_index);
+#endif
 	/* FIXME: what's the actual transition time? */
 	policy->cpuinfo.transition_latency = 300 * 1000;
 
@@ -169,3 +187,50 @@ arch_initcall(omap_cpufreq_init);
  * cpufreq_frequency_table_put_attr()
  */
 
+#ifdef CONFIG_CGROUP_CPUACCT
+/*
+ * Omap platform calls for cpuacct frequency accounting.
+ */
+
+static void omap_cpuacct_freq_init(void **cpuacct_data)
+{
+	/* MAX_VDD1_OPP is gone, define a table size to fit available values */
+	*cpuacct_data = kzalloc(sizeof(u64) * 8, GFP_KERNEL);
+}
+
+/* Called with rcu_read_lock() held. */
+static void omap_cpuacct_freq_charge(void *cpuacct_data, u64 cputime, unsigned int cpu)
+{
+	u64 *cpuacct_freq = cpuacct_data;
+	if (freq_index < 0)
+		return;
+
+	cpuacct_freq[freq_index] += cputime;
+}
+
+static void omap_cpuacct_freq_show(void *cpuacct_data, struct cgroup_map_cb *cb)
+{
+	int i;
+	char buf[32];
+	u64 *cpuacct_freq = cpuacct_data;
+	for (i = 0; freq_table[i].frequency != CPUFREQ_TABLE_END; i++) {
+		snprintf(buf, sizeof(buf), "%u", freq_table[i].frequency);
+		cb->fill(cb, buf, cpuacct_freq[i]);
+	}
+}
+
+static struct cpuacct_charge_calls omap_cpuacct_cpufreq = {
+	.init = omap_cpuacct_freq_init,
+	.charge = omap_cpuacct_freq_charge,
+	.cpufreq_show = omap_cpuacct_freq_show,
+};
+
+static int __init omap_cpuacct_init(void)
+{
+	freq_index = -1;
+	cpuacct_charge_register(&omap_cpuacct_cpufreq);
+	return 0;
+}
+
+early_initcall(omap_cpuacct_init);
+#endif
-- 
1.7.0.1

Re: [PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[Thomas Renninger]

Hi Mike,

On Thursday 20 May 2010 08:42:21 pm Mike Chan wrote:
> v2:
> Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring,
thanks.
A general comment:
I don't know much about the cgroup stuff.
I am also not sure how exactly power can be measured on this arch based on
frequency accounting (there also were some threads I was not aware of?)
A signed-off-by or reviewed-by from someone who is more involved in this omap
stuff would probably not that bad.

Still I Iike this interface and I could imagine others hook into it as well, 
for whatever has to be cpu cgroup accounted.

My two cents...,

     Thomas

Re: [PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[Mike Chan]

On Thu, May 20, 2010 at 2:01 PM, Thomas Renninger <trenn@suse.de> wrote:
> Hi Mike,
>
> On Thursday 20 May 2010 08:42:21 pm Mike Chan wrote:
>> v2:
>> Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring,
> thanks.
> A general comment:
> I don't know much about the cgroup stuff.

Perhaps Paul Menage or Balbir Singh can look and sign off on the API's?

> I am also not sure how exactly power can be measured on this arch based on

If you know how much time was spent at each frequency executing code,
you can calculate how much power was consumed if the platform (with
hooks) provide power numbers (in milliWatts) for the power at
frequency X.

I did some initial testing on Motorola Droid comparing to a power
meter and I got within 2% variance.

> frequency accounting (there also were some threads I was not aware of?)
> A signed-off-by or reviewed-by from someone who is more involved in this omap
> stuff would probably not that bad.
>

OMAP was the closest with mainline support I could provide an example
how to use these hooks.

I'm hoping for some blessing from some people on the linux-omap list
for that. However can we possibly just stack the first two patches to
get the API in? This will make it easier to fixup the omap hooks if
they don't get in.

-- Mike

> Still I Iike this interface and I could imagine others hook into it as well,
> for whatever has to be cpu cgroup accounted.
>
> My two cents...,
>
>     Thomas
> --
> To unsubscribe from this list: send the line "unsubscribe cpufreq" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[Kevin Hilman]

Mike Chan <mike@android.com> writes:

> On Thu, May 20, 2010 at 2:01 PM, Thomas Renninger <trenn@suse.de> wrote:
>> Hi Mike,
>>
>> On Thursday 20 May 2010 08:42:21 pm Mike Chan wrote:
>>> v2:
>>> Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring,
>> thanks.
>> A general comment:
>> I don't know much about the cgroup stuff.
>
> Perhaps Paul Menage or Balbir Singh can look and sign off on the API's?
>
>> I am also not sure how exactly power can be measured on this arch based on
>
> If you know how much time was spent at each frequency executing code,
> you can calculate how much power was consumed if the platform (with
> hooks) provide power numbers (in milliWatts) for the power at
> frequency X.
>
> I did some initial testing on Motorola Droid comparing to a power
> meter and I got within 2% variance.
>
>> frequency accounting (there also were some threads I was not aware of?)
>> A signed-off-by or reviewed-by from someone who is more involved in this omap
>> stuff would probably not that bad.
>>
>
> OMAP was the closest with mainline support I could provide an example
> how to use these hooks.
>
> I'm hoping for some blessing from some people on the linux-omap list
> for that. However can we possibly just stack the first two patches to
> get the API in? This will make it easier to fixup the omap hooks if
> they don't get in.

This looks like a great enhancement to me.

Speaking for OMAP PM... I'd suggest getting the generic stuff upstream
(or into -next) soon and then work out the OMAP specifics after.

Since the OMAP OPP layer is going through some churn (but stabilizing
and will be submitted for 2.6.36), I'd suggest we queue the
OMAP-specific parts of this along with the OPP layer changes.

Kevin

Re: [PATCH v2 0/3] Enable CPU frequency and power tracking in cpuacct cgroup

[Mike Chan]

On Fri, May 21, 2010 at 10:05 AM, Kevin Hilman
<khilman@deeprootsystems.com> wrote:
> Mike Chan <mike@android.com> writes:
>
>> On Thu, May 20, 2010 at 2:01 PM, Thomas Renninger <trenn@suse.de> wrote:
>>> Hi Mike,
>>>
>>> On Thursday 20 May 2010 08:42:21 pm Mike Chan wrote:
>>>> v2:
>>>> Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring,
>>> thanks.
>>> A general comment:
>>> I don't know much about the cgroup stuff.
>>
>> Perhaps Paul Menage or Balbir Singh can look and sign off on the API's?
>>
>>> I am also not sure how exactly power can be measured on this arch based on
>>
>> If you know how much time was spent at each frequency executing code,
>> you can calculate how much power was consumed if the platform (with
>> hooks) provide power numbers (in milliWatts) for the power at
>> frequency X.
>>
>> I did some initial testing on Motorola Droid comparing to a power
>> meter and I got within 2% variance.
>>
>>> frequency accounting (there also were some threads I was not aware of?)
>>> A signed-off-by or reviewed-by from someone who is more involved in this omap
>>> stuff would probably not that bad.
>>>
>>
>> OMAP was the closest with mainline support I could provide an example
>> how to use these hooks.
>>
>> I'm hoping for some blessing from some people on the linux-omap list
>> for that. However can we possibly just stack the first two patches to
>> get the API in? This will make it easier to fixup the omap hooks if
>> they don't get in.
>
> This looks like a great enhancement to me.
>
> Speaking for OMAP PM... I'd suggest getting the generic stuff upstream
> (or into -next) soon and then work out the OMAP specifics after.
>
> Since the OMAP OPP layer is going through some churn (but stabilizing
> and will be submitted for 2.6.36), I'd suggest we queue the
> OMAP-specific parts of this along with the OPP layer changes.
>

So it looks like there is no objections to this API and I'm OK with
dropping the omap hooks for now until things are settled in 2.6.36. So
are things good with Thomas' re-factoring patch for cpuacct as well as
the first 2 patches?

-- Mike

> Kevin
>