[PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[riel-H+wXaHxf7aLQT0dZR+AlfA]

Ensure that cpus specified with the isolcpus= boot commandline
option stay outside of the load balancing in the kernel scheduler.

Operations like load balancing can introduce unwanted latencies,
which is exactly what the isolcpus= commandline is there to prevent.

Previously, simply creating a new cpuset, without even touching the
cpuset.cpus field inside the new cpuset, would undo the effects of
isolcpus=, by creating a scheduler domain spanning the whole system,
and setting up load balancing inside that domain. The cpuset root
cpuset.cpus file is read-only, so there was not even a way to undo
that effect.

This does not impact the majority of cpusets users, since isolcpus=
is a fairly specialized feature used for realtime purposes.

This version fixes the UP compilation issue, in the same way done
for the other cpumasks.

[PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[riel]

Ensure that cpus specified with the isolcpus= boot commandline
option stay outside of the load balancing in the kernel scheduler.

Operations like load balancing can introduce unwanted latencies,
which is exactly what the isolcpus= commandline is there to prevent.

Previously, simply creating a new cpuset, without even touching the
cpuset.cpus field inside the new cpuset, would undo the effects of
isolcpus=, by creating a scheduler domain spanning the whole system,
and setting up load balancing inside that domain. The cpuset root
cpuset.cpus file is read-only, so there was not even a way to undo
that effect.

This does not impact the majority of cpusets users, since isolcpus=
is a fairly specialized feature used for realtime purposes.

This version fixes the UP compilation issue, in the same way done
for the other cpumasks.

[PATCH 1/4] sched,isolcpu: make cpu_isolated_map visible outside scheduler

[riel]

From: Rik van Riel <riel@redhat.com>

Needed by the next patch. Also makes cpu_isolated_map present
when compiled without SMP and/or with CONFIG_NR_CPUS=1, like
the other cpu masks.

At some point we may want to clean things up so cpumasks do
not exist in UP kernels. Maybe something for the CONFIG_TINY
crowd.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: cgroups@vger.kernel.org
Signed-off-by: Rik van Riel <riel@redhat.com>
---
 include/linux/sched.h | 2 ++
 kernel/sched/core.c   | 6 +++---
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6d77432e14ff..ca365d79480c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -329,6 +329,8 @@ extern asmlinkage void schedule_tail(struct task_struct *prev);
 extern void init_idle(struct task_struct *idle, int cpu);
 extern void init_idle_bootup_task(struct task_struct *idle);
 
+extern cpumask_var_t cpu_isolated_map;
+
 extern int runqueue_is_locked(int cpu);
 
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f0f831e8a345..b578bb23410b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -306,6 +306,9 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 
+/* cpus with isolated domains */
+cpumask_var_t cpu_isolated_map;
+
 /*
  * this_rq_lock - lock this runqueue and disable interrupts.
  */
@@ -5811,9 +5814,6 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
 	update_top_cache_domain(cpu);
 }
 
-/* cpus with isolated domains */
-static cpumask_var_t cpu_isolated_map;
-
 /* Setup the mask of cpus configured for isolated domains */
 static int __init isolated_cpu_setup(char *str)
 {
-- 
2.1.0

[PATCH 2/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[riel]

From: Rik van Riel <riel@redhat.com>

Ensure that cpus specified with the isolcpus= boot commandline
option stay outside of the load balancing in the kernel scheduler.

Operations like load balancing can introduce unwanted latencies,
which is exactly what the isolcpus= commandline is there to prevent.

Previously, simply creating a new cpuset, without even touching the
cpuset.cpus field inside the new cpuset, would undo the effects of
isolcpus=, by creating a scheduler domain spanning the whole system,
and setting up load balancing inside that domain. The cpuset root
cpuset.cpus file is read-only, so there was not even a way to undo
that effect.

This does not impact the majority of cpusets users, since isolcpus=
is a fairly specialized feature used for realtime purposes.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: cgroups@vger.kernel.org
Signed-off-by: Rik van Riel <riel@redhat.com>
Tested-by: David Rientjes <rientjes@google.com>
---
 kernel/cpuset.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index 1d1fe9361d29..b544e5229d99 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -625,6 +625,7 @@ static int generate_sched_domains(cpumask_var_t **domains,
 	int csn;		/* how many cpuset ptrs in csa so far */
 	int i, j, k;		/* indices for partition finding loops */
 	cpumask_var_t *doms;	/* resulting partition; i.e. sched domains */
+	cpumask_var_t non_isolated_cpus;  /* load balanced CPUs */
 	struct sched_domain_attr *dattr;  /* attributes for custom domains */
 	int ndoms = 0;		/* number of sched domains in result */
 	int nslot;		/* next empty doms[] struct cpumask slot */
@@ -634,6 +635,10 @@ static int generate_sched_domains(cpumask_var_t **domains,
 	dattr = NULL;
 	csa = NULL;
 
+	if (!alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL))
+		goto done;
+	cpumask_andnot(non_isolated_cpus, cpu_possible_mask, cpu_isolated_map);
+
 	/* Special case for the 99% of systems with one, full, sched domain */
 	if (is_sched_load_balance(&top_cpuset)) {
 		ndoms = 1;
@@ -646,7 +651,8 @@ static int generate_sched_domains(cpumask_var_t **domains,
 			*dattr = SD_ATTR_INIT;
 			update_domain_attr_tree(dattr, &top_cpuset);
 		}
-		cpumask_copy(doms[0], top_cpuset.effective_cpus);
+		cpumask_and(doms[0], top_cpuset.effective_cpus,
+				     non_isolated_cpus);
 
 		goto done;
 	}
@@ -669,7 +675,8 @@ static int generate_sched_domains(cpumask_var_t **domains,
 		 * the corresponding sched domain.
 		 */
 		if (!cpumask_empty(cp->cpus_allowed) &&
-		    !is_sched_load_balance(cp))
+		    !(is_sched_load_balance(cp) &&
+		      cpumask_intersects(cp->cpus_allowed, non_isolated_cpus)))
 			continue;
 
 		if (is_sched_load_balance(cp))
@@ -751,6 +758,7 @@ static int generate_sched_domains(cpumask_var_t **domains,
 
 			if (apn == b->pn) {
 				cpumask_or(dp, dp, b->effective_cpus);
+				cpumask_and(dp, dp, non_isolated_cpus);
 				if (dattr)
 					update_domain_attr_tree(dattr + nslot, b);
 
@@ -763,6 +771,7 @@ static int generate_sched_domains(cpumask_var_t **domains,
 	BUG_ON(nslot != ndoms);
 
 done:
+	free_cpumask_var(non_isolated_cpus);
 	kfree(csa);
 
 	/*
-- 
2.1.0

[PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[riel]

From: Rik van Riel <riel@redhat.com>

The previous patch makes it so the code skips over isolcpus when
building scheduler load balancing domains. This makes it hard to
see for a user which of the CPUs in a cpuset are participating in
load balancing, and which ones are isolated cpus.

Add a cpuset.isolcpus file with info on which cpus in a cpuset are
isolated CPUs.

This file is read-only for now. In the future we could extend things
so isolcpus can be changed at run time, for the root (system wide)
cpuset only.

Acked-by: David Rientjes <rientjes@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: cgroups@vger.kernel.org
Signed-off-by: Rik van Riel <riel@redhat.com>
---
 kernel/cpuset.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index b544e5229d99..5462e1ca90bd 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1563,6 +1563,7 @@ typedef enum {
 	FILE_MEMORY_PRESSURE,
 	FILE_SPREAD_PAGE,
 	FILE_SPREAD_SLAB,
+	FILE_ISOLCPUS,
 } cpuset_filetype_t;
 
 static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype *cft,
@@ -1704,6 +1705,16 @@ static ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 	return retval ?: nbytes;
 }
 
+/* protected by the lock in cpuset_common_seq_show */
+static cpumask_var_t print_isolated_cpus;
+
+static void cpuset_seq_print_isolcpus(struct seq_file *sf, struct cpuset *cs)
+{
+	cpumask_and(print_isolated_cpus, cs->cpus_allowed, cpu_isolated_map);
+
+	seq_printf(sf, "%*pbl\n", cpumask_pr_args(print_isolated_cpus));
+}
+
 /*
  * These ascii lists should be read in a single call, by using a user
  * buffer large enough to hold the entire map.  If read in smaller
@@ -1733,6 +1744,9 @@ static int cpuset_common_seq_show(struct seq_file *sf, void *v)
 	case FILE_EFFECTIVE_MEMLIST:
 		seq_printf(sf, "%*pbl\n", nodemask_pr_args(&cs->effective_mems));
 		break;
+	case FILE_ISOLCPUS:
+		cpuset_seq_print_isolcpus(sf, cs);
+		break;
 	default:
 		ret = -EINVAL;
 	}
@@ -1893,6 +1907,12 @@ static struct cftype files[] = {
 		.private = FILE_MEMORY_PRESSURE_ENABLED,
 	},
 
+	{
+		.name = "isolcpus",
+		.seq_show = cpuset_common_seq_show,
+		.private = FILE_ISOLCPUS,
+	},
+
 	{ }	/* terminate */
 };
 
@@ -2070,6 +2090,8 @@ int __init cpuset_init(void)
 		BUG();
 	if (!alloc_cpumask_var(&top_cpuset.effective_cpus, GFP_KERNEL))
 		BUG();
+	if (!alloc_cpumask_var(&print_isolated_cpus, GFP_KERNEL))
+		BUG();
 
 	cpumask_setall(top_cpuset.cpus_allowed);
 	nodes_setall(top_cpuset.mems_allowed);
-- 
2.1.0

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Tejun Heo]

On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
> From: Rik van Riel <riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> 
> The previous patch makes it so the code skips over isolcpus when
> building scheduler load balancing domains. This makes it hard to
> see for a user which of the CPUs in a cpuset are participating in
> load balancing, and which ones are isolated cpus.
> 
> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
> isolated CPUs.
> 
> This file is read-only for now. In the future we could extend things
> so isolcpus can be changed at run time, for the root (system wide)
> cpuset only.

Didn't Li say that this is trivially computable from userland?  I'm
not sure this knob actually belongs to cpuset.

Thanks.

-- 
tejun

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Tejun Heo]

On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
> From: Rik van Riel <riel@redhat.com>
> 
> The previous patch makes it so the code skips over isolcpus when
> building scheduler load balancing domains. This makes it hard to
> see for a user which of the CPUs in a cpuset are participating in
> load balancing, and which ones are isolated cpus.
> 
> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
> isolated CPUs.
> 
> This file is read-only for now. In the future we could extend things
> so isolcpus can be changed at run time, for the root (system wide)
> cpuset only.

Didn't Li say that this is trivially computable from userland?  I'm
not sure this knob actually belongs to cpuset.

Thanks.

-- 
tejun

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 12:47 PM, Tejun Heo wrote:
> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
>> From: Rik van Riel <riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>
>> The previous patch makes it so the code skips over isolcpus when
>> building scheduler load balancing domains. This makes it hard to
>> see for a user which of the CPUs in a cpuset are participating in
>> load balancing, and which ones are isolated cpus.
>>
>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>> isolated CPUs.
>>
>> This file is read-only for now. In the future we could extend things
>> so isolcpus can be changed at run time, for the root (system wide)
>> cpuset only.
> 
> Didn't Li say that this is trivially computable from userland?  I'm
> not sure this knob actually belongs to cpuset.

I don't know whether the information to compute this is
always visible from userland.  I am happy to drop this
patch if Li prefers things that way, though.

Li?

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 12:47 PM, Tejun Heo wrote:
> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>> From: Rik van Riel <riel@redhat.com>
>>
>> The previous patch makes it so the code skips over isolcpus when
>> building scheduler load balancing domains. This makes it hard to
>> see for a user which of the CPUs in a cpuset are participating in
>> load balancing, and which ones are isolated cpus.
>>
>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>> isolated CPUs.
>>
>> This file is read-only for now. In the future we could extend things
>> so isolcpus can be changed at run time, for the root (system wide)
>> cpuset only.
> 
> Didn't Li say that this is trivially computable from userland?  I'm
> not sure this knob actually belongs to cpuset.

I don't know whether the information to compute this is
always visible from userland.  I am happy to drop this
patch if Li prefers things that way, though.

Li?

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Zefan Li]

On 2015/3/19 7:40, Rik van Riel wrote:
> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>>> From: Rik van Riel <riel@redhat.com>
>>>
>>> The previous patch makes it so the code skips over isolcpus when
>>> building scheduler load balancing domains. This makes it hard to
>>> see for a user which of the CPUs in a cpuset are participating in
>>> load balancing, and which ones are isolated cpus.
>>>
>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>> isolated CPUs.
>>>
>>> This file is read-only for now. In the future we could extend things
>>> so isolcpus can be changed at run time, for the root (system wide)
>>> cpuset only.
>>
>> Didn't Li say that this is trivially computable from userland?  I'm
>> not sure this knob actually belongs to cpuset.
> 
> I don't know whether the information to compute this is
> always visible from userland.  I am happy to drop this
> patch if Li prefers things that way, though.
> 

What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
visible in containers, unless specially configured not so.

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Zefan Li]

On 2015/3/19 7:40, Rik van Riel wrote:
> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>>> From: Rik van Riel <riel@redhat.com>
>>>
>>> The previous patch makes it so the code skips over isolcpus when
>>> building scheduler load balancing domains. This makes it hard to
>>> see for a user which of the CPUs in a cpuset are participating in
>>> load balancing, and which ones are isolated cpus.
>>>
>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>> isolated CPUs.
>>>
>>> This file is read-only for now. In the future we could extend things
>>> so isolcpus can be changed at run time, for the root (system wide)
>>> cpuset only.
>>
>> Didn't Li say that this is trivially computable from userland?  I'm
>> not sure this knob actually belongs to cpuset.
> 
> I don't know whether the information to compute this is
> always visible from userland.  I am happy to drop this
> patch if Li prefers things that way, though.
> 

What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
visible in containers, unless specially configured not so.

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 09:50 PM, Zefan Li wrote:
> On 2015/3/19 7:40, Rik van Riel wrote:
>> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
>>>> From: Rik van Riel <riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>>>
>>>> The previous patch makes it so the code skips over isolcpus when
>>>> building scheduler load balancing domains. This makes it hard to
>>>> see for a user which of the CPUs in a cpuset are participating in
>>>> load balancing, and which ones are isolated cpus.
>>>>
>>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>>> isolated CPUs.
>>>>
>>>> This file is read-only for now. In the future we could extend things
>>>> so isolcpus can be changed at run time, for the root (system wide)
>>>> cpuset only.
>>>
>>> Didn't Li say that this is trivially computable from userland?  I'm
>>> not sure this knob actually belongs to cpuset.
>>
>> I don't know whether the information to compute this is
>> always visible from userland.  I am happy to drop this
>> patch if Li prefers things that way, though.
>>
> 
> What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
> visible in containers, unless specially configured not so.

OK, are you willing to take patches 1, 2, and the first hunk of patch
4 now? I can submit a patch to add /sys/devices/system/cpu/isolated
on Friday, to the appropriate maintainer.

(taking tomorrow off to go hiking on the last full day of winter)

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 09:50 PM, Zefan Li wrote:
> On 2015/3/19 7:40, Rik van Riel wrote:
>> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>>>> From: Rik van Riel <riel@redhat.com>
>>>>
>>>> The previous patch makes it so the code skips over isolcpus when
>>>> building scheduler load balancing domains. This makes it hard to
>>>> see for a user which of the CPUs in a cpuset are participating in
>>>> load balancing, and which ones are isolated cpus.
>>>>
>>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>>> isolated CPUs.
>>>>
>>>> This file is read-only for now. In the future we could extend things
>>>> so isolcpus can be changed at run time, for the root (system wide)
>>>> cpuset only.
>>>
>>> Didn't Li say that this is trivially computable from userland?  I'm
>>> not sure this knob actually belongs to cpuset.
>>
>> I don't know whether the information to compute this is
>> always visible from userland.  I am happy to drop this
>> patch if Li prefers things that way, though.
>>
> 
> What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
> visible in containers, unless specially configured not so.

OK, are you willing to take patches 1, 2, and the first hunk of patch
4 now? I can submit a patch to add /sys/devices/system/cpu/isolated
on Friday, to the appropriate maintainer.

(taking tomorrow off to go hiking on the last full day of winter)

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Zefan Li]

On 2015/3/19 9:54, Rik van Riel wrote:
> On 03/18/2015 09:50 PM, Zefan Li wrote:
>> On 2015/3/19 7:40, Rik van Riel wrote:
>>> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>>>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
>>>>> From: Rik van Riel <riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>>>>
>>>>> The previous patch makes it so the code skips over isolcpus when
>>>>> building scheduler load balancing domains. This makes it hard to
>>>>> see for a user which of the CPUs in a cpuset are participating in
>>>>> load balancing, and which ones are isolated cpus.
>>>>>
>>>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>>>> isolated CPUs.
>>>>>
>>>>> This file is read-only for now. In the future we could extend things
>>>>> so isolcpus can be changed at run time, for the root (system wide)
>>>>> cpuset only.
>>>>
>>>> Didn't Li say that this is trivially computable from userland?  I'm
>>>> not sure this knob actually belongs to cpuset.
>>>
>>> I don't know whether the information to compute this is
>>> always visible from userland.  I am happy to drop this
>>> patch if Li prefers things that way, though.
>>>
>>
>> What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
>> visible in containers, unless specially configured not so.
> 
> OK, are you willing to take patches 1, 2, and the first hunk of patch
> 4 now? I can submit a patch to add /sys/devices/system/cpu/isolated
> on Friday, to the appropriate maintainer.
> 

Other patches look good to me.

> (taking tomorrow off to go hiking on the last full day of winter)
> 

Have a good day!

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Zefan Li]

On 2015/3/19 9:54, Rik van Riel wrote:
> On 03/18/2015 09:50 PM, Zefan Li wrote:
>> On 2015/3/19 7:40, Rik van Riel wrote:
>>> On 03/18/2015 12:47 PM, Tejun Heo wrote:
>>>> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>>>>> From: Rik van Riel <riel@redhat.com>
>>>>>
>>>>> The previous patch makes it so the code skips over isolcpus when
>>>>> building scheduler load balancing domains. This makes it hard to
>>>>> see for a user which of the CPUs in a cpuset are participating in
>>>>> load balancing, and which ones are isolated cpus.
>>>>>
>>>>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>>>>> isolated CPUs.
>>>>>
>>>>> This file is read-only for now. In the future we could extend things
>>>>> so isolcpus can be changed at run time, for the root (system wide)
>>>>> cpuset only.
>>>>
>>>> Didn't Li say that this is trivially computable from userland?  I'm
>>>> not sure this knob actually belongs to cpuset.
>>>
>>> I don't know whether the information to compute this is
>>> always visible from userland.  I am happy to drop this
>>> patch if Li prefers things that way, though.
>>>
>>
>> What I proposed is adding /sys/devices/system/cpu/isolated. Sysfs is
>> visible in containers, unless specially configured not so.
> 
> OK, are you willing to take patches 1, 2, and the first hunk of patch
> 4 now? I can submit a patch to add /sys/devices/system/cpu/isolated
> on Friday, to the appropriate maintainer.
> 

Other patches look good to me.

> (taking tomorrow off to go hiking on the last full day of winter)
> 

Have a good day!

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 12:47 PM, Tejun Heo wrote:
> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
>> From: Rik van Riel <riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>
>> The previous patch makes it so the code skips over isolcpus when
>> building scheduler load balancing domains. This makes it hard to
>> see for a user which of the CPUs in a cpuset are participating in
>> load balancing, and which ones are isolated cpus.
>>
>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>> isolated CPUs.
>>
>> This file is read-only for now. In the future we could extend things
>> so isolcpus can be changed at run time, for the root (system wide)
>> cpuset only.
> 
> Didn't Li say that this is trivially computable from userland?  I'm
> not sure this knob actually belongs to cpuset.

Going back through my email, it looks like Li
previously acked this patch.

That explains why I'm confused about your comment :)

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Rik van Riel]

On 03/18/2015 12:47 PM, Tejun Heo wrote:
> On Mon, Mar 09, 2015 at 12:12:09PM -0400, riel@redhat.com wrote:
>> From: Rik van Riel <riel@redhat.com>
>>
>> The previous patch makes it so the code skips over isolcpus when
>> building scheduler load balancing domains. This makes it hard to
>> see for a user which of the CPUs in a cpuset are participating in
>> load balancing, and which ones are isolated cpus.
>>
>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are
>> isolated CPUs.
>>
>> This file is read-only for now. In the future we could extend things
>> so isolcpus can be changed at run time, for the root (system wide)
>> cpuset only.
> 
> Didn't Li say that this is trivially computable from userland?  I'm
> not sure this knob actually belongs to cpuset.

Going back through my email, it looks like Li
previously acked this patch.

That explains why I'm confused about your comment :)

-- 
All rights reversed

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Tejun Heo]

Hello,

On Wed, Mar 18, 2015 at 09:45:45PM -0400, Rik van Riel wrote:
> Going back through my email, it looks like Li
> previously acked this patch.

Ooh, the patch didn't have acked-by.  Sorry about the confusion.

> That explains why I'm confused about your comment :)

I still don't think cpuset is the right place to put this knob.  This
is a system-wide boot paramter, isn't it?  There gotta be a place
better suited for this, be that a sysctl knob or a file under sys.

Thanks.

-- 
tejun

Re: [PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[Tejun Heo]

Hello,

On Wed, Mar 18, 2015 at 09:45:45PM -0400, Rik van Riel wrote:
> Going back through my email, it looks like Li
> previously acked this patch.

Ooh, the patch didn't have acked-by.  Sorry about the confusion.

> That explains why I'm confused about your comment :)

I still don't think cpuset is the right place to put this knob.  This
is a system-wide boot paramter, isn't it?  There gotta be a place
better suited for this, be that a sysctl knob or a file under sys.

Thanks.

-- 
tejun

[PATCH 4/4] cpuset,isolcpus: document relationship between cpusets & isolcpus

[riel]

From: Rik van Riel <riel@redhat.com>

Document the subtly changed relationship between cpusets and isolcpus.
Turns out the old documentation did not match the code...

Signed-off-by: Rik van Riel <riel@redhat.com>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
---
 Documentation/cgroups/cpusets.txt | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/Documentation/cgroups/cpusets.txt b/Documentation/cgroups/cpusets.txt
index f2235a162529..fdf7dff3f607 100644
--- a/Documentation/cgroups/cpusets.txt
+++ b/Documentation/cgroups/cpusets.txt
@@ -392,8 +392,10 @@ Put simply, it costs less to balance between two smaller sched domains
 than one big one, but doing so means that overloads in one of the
 two domains won't be load balanced to the other one.
 
-By default, there is one sched domain covering all CPUs, except those
-marked isolated using the kernel boot time "isolcpus=" argument.
+By default, there is one sched domain covering all CPUs, including those
+marked isolated using the kernel boot time "isolcpus=" argument. However,
+the isolated CPUs will not participate in load balancing, and will not
+have tasks running on them unless explicitly assigned.
 
 This default load balancing across all CPUs is not well suited for
 the following two situations:
@@ -465,6 +467,10 @@ such partially load balanced cpusets, as they may be artificially
 constrained to some subset of the CPUs allowed to them, for lack of
 load balancing to the other CPUs.
 
+CPUs in "cpuset.isolcpus" were excluded from load balancing by the
+isolcpus= kernel boot option, and will never be load balanced regardless
+of the value of "cpuset.sched_load_balance" in any cpuset.
+
 1.7.1 sched_load_balance implementation details.
 ------------------------------------------------
 
-- 
2.1.0

Re: [PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[Rik van Riel]

On 03/09/2015 12:12 PM, riel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org wrote:
> Ensure that cpus specified with the isolcpus= boot commandline
> option stay outside of the load balancing in the kernel scheduler.
> 
> Operations like load balancing can introduce unwanted latencies,
> which is exactly what the isolcpus= commandline is there to prevent.
> 
> Previously, simply creating a new cpuset, without even touching the
> cpuset.cpus field inside the new cpuset, would undo the effects of
> isolcpus=, by creating a scheduler domain spanning the whole system,
> and setting up load balancing inside that domain. The cpuset root
> cpuset.cpus file is read-only, so there was not even a way to undo
> that effect.
> 
> This does not impact the majority of cpusets users, since isolcpus=
> is a fairly specialized feature used for realtime purposes.
> 
> This version fixes the UP compilation issue, in the same way done
> for the other cpumasks.

Ping?

Tejun? Li?

Re: [PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[Rik van Riel]

On 03/09/2015 12:12 PM, riel@redhat.com wrote:
> Ensure that cpus specified with the isolcpus= boot commandline
> option stay outside of the load balancing in the kernel scheduler.
> 
> Operations like load balancing can introduce unwanted latencies,
> which is exactly what the isolcpus= commandline is there to prevent.
> 
> Previously, simply creating a new cpuset, without even touching the
> cpuset.cpus field inside the new cpuset, would undo the effects of
> isolcpus=, by creating a scheduler domain spanning the whole system,
> and setting up load balancing inside that domain. The cpuset root
> cpuset.cpus file is read-only, so there was not even a way to undo
> that effect.
> 
> This does not impact the majority of cpusets users, since isolcpus=
> is a fairly specialized feature used for realtime purposes.
> 
> This version fixes the UP compilation issue, in the same way done
> for the other cpumasks.

Ping?

Tejun? Li?

Re: [PATCH v4 RESEND 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[Tejun Heo]

On Mon, Mar 09, 2015 at 12:12:06PM -0400, riel@redhat.com wrote:
> Ensure that cpus specified with the isolcpus= boot commandline
> option stay outside of the load balancing in the kernel scheduler.
> 
> Operations like load balancing can introduce unwanted latencies,
> which is exactly what the isolcpus= commandline is there to prevent.
> 
> Previously, simply creating a new cpuset, without even touching the
> cpuset.cpus field inside the new cpuset, would undo the effects of
> isolcpus=, by creating a scheduler domain spanning the whole system,
> and setting up load balancing inside that domain. The cpuset root
> cpuset.cpus file is read-only, so there was not even a way to undo
> that effect.
> 
> This does not impact the majority of cpusets users, since isolcpus=
> is a fairly specialized feature used for realtime purposes.
> 
> This version fixes the UP compilation issue, in the same way done
> for the other cpumasks.

Applied 1, 2, and 4 to cgroup/for-4.1.  I collected the acks from the
previous threads.  Can you please collect them when posting new
revisions in the future?

Thanks.

-- 
tejun

[PATCH v4 0/4] cpusets,isolcpus: exclude isolcpus from load balancing in cpusets

[riel]

Ensure that cpus specified with the isolcpus= boot commandline
option stay outside of the load balancing in the kernel scheduler.

Operations like load balancing can introduce unwanted latencies,
which is exactly what the isolcpus= commandline is there to prevent.

Previously, simply creating a new cpuset, without even touching the
cpuset.cpus field inside the new cpuset, would undo the effects of
isolcpus=, by creating a scheduler domain spanning the whole system,
and setting up load balancing inside that domain. The cpuset root
cpuset.cpus file is read-only, so there was not even a way to undo
that effect.

This does not impact the majority of cpusets users, since isolcpus=
is a fairly specialized feature used for realtime purposes.

This version fixes the UP compilation issue, in the same way done
for the other cpumasks.

[PATCH 3/4] cpusets,isolcpus: add file to show isolated cpus in cpuset

[riel]

From: Rik van Riel <riel@redhat.com>

The previous patch makes it so the code skips over isolcpus when
building scheduler load balancing domains. This makes it hard to
see for a user which of the CPUs in a cpuset are participating in
load balancing, and which ones are isolated cpus.

Add a cpuset.isolcpus file with info on which cpus in a cpuset are
isolated CPUs.

This file is read-only for now. In the future we could extend things
so isolcpus can be changed at run time, for the root (system wide)
cpuset only.

Acked-by: David Rientjes <rientjes@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: cgroups@vger.kernel.org
Signed-off-by: Rik van Riel <riel@redhat.com>
---
 kernel/cpuset.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index b544e5229d99..5462e1ca90bd 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1563,6 +1563,7 @@ typedef enum {
 	FILE_MEMORY_PRESSURE,
 	FILE_SPREAD_PAGE,
 	FILE_SPREAD_SLAB,
+	FILE_ISOLCPUS,
 } cpuset_filetype_t;
 
 static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype *cft,
@@ -1704,6 +1705,16 @@ static ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 	return retval ?: nbytes;
 }
 
+/* protected by the lock in cpuset_common_seq_show */
+static cpumask_var_t print_isolated_cpus;
+
+static void cpuset_seq_print_isolcpus(struct seq_file *sf, struct cpuset *cs)
+{
+	cpumask_and(print_isolated_cpus, cs->cpus_allowed, cpu_isolated_map);
+
+	seq_printf(sf, "%*pbl\n", cpumask_pr_args(print_isolated_cpus));
+}
+
 /*
  * These ascii lists should be read in a single call, by using a user
  * buffer large enough to hold the entire map.  If read in smaller
@@ -1733,6 +1744,9 @@ static int cpuset_common_seq_show(struct seq_file *sf, void *v)
 	case FILE_EFFECTIVE_MEMLIST:
 		seq_printf(sf, "%*pbl\n", nodemask_pr_args(&cs->effective_mems));
 		break;
+	case FILE_ISOLCPUS:
+		cpuset_seq_print_isolcpus(sf, cs);
+		break;
 	default:
 		ret = -EINVAL;
 	}
@@ -1893,6 +1907,12 @@ static struct cftype files[] = {
 		.private = FILE_MEMORY_PRESSURE_ENABLED,
 	},
 
+	{
+		.name = "isolcpus",
+		.seq_show = cpuset_common_seq_show,
+		.private = FILE_ISOLCPUS,
+	},
+
 	{ }	/* terminate */
 };
 
@@ -2070,6 +2090,8 @@ int __init cpuset_init(void)
 		BUG();
 	if (!alloc_cpumask_var(&top_cpuset.effective_cpus, GFP_KERNEL))
 		BUG();
+	if (!alloc_cpumask_var(&print_isolated_cpus, GFP_KERNEL))
+		BUG();
 
 	cpumask_setall(top_cpuset.cpus_allowed);
 	nodes_setall(top_cpuset.mems_allowed);
-- 
2.1.0