[PATCH 0/3] resend, cpuset/hotplug fixes

[PATCH 0/3] resend, cpuset/hotplug fixes

[Oleg Nesterov]

The 1st patch is preparation. 2-3 fix different problems, the 3rd one
depends on 2nd.

These patches change the code which I don't really understand, please
review.


As for the 3rd patch, it replaces

	cpu_hotplug-dont-affect-current-tasks-affinity.patch

in -mm tree. Imho the new patch is more simple and clean, but of course
this is subjective and I am biased.

Oleg.

[PATCH 0/3] resend, cpuset/hotplug fixes

[Oleg Nesterov]

(my apologies to those who see this twice).


The 1st patch is preparation. 2-3 fix different problems, the 3rd one
depends on 2nd.

These patches change the code which I don't really understand, please
review.


As for the 3rd patch, it replaces

	cpu_hotplug-dont-affect-current-tasks-affinity.patch

in -mm tree. Imho the new patch is more simple and clean, but of course
this is subjective and I am biased.

Oleg.

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Rafael J. Wysocki]

Hi Oleg,

On Thursday 10 September 2009, Oleg Nesterov wrote:
> (my apologies to those who see this twice).
> 
> 
> The 1st patch is preparation. 2-3 fix different problems, the 3rd one
> depends on 2nd.
> 
> These patches change the code which I don't really understand, please
> review.
> 
> 
> As for the 3rd patch, it replaces
> 
> 	cpu_hotplug-dont-affect-current-tasks-affinity.patch
> 
> in -mm tree. Imho the new patch is more simple and clean, but of course
> this is subjective and I am biased.

The patches look good to me FWIW.

Thanks a lot for taking care of this!

Rafael

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Peter Zijlstra]

On Thu, 2009-09-10 at 21:21 +0200, Oleg Nesterov wrote:
> (my apologies to those who see this twice).
> 
> 
> The 1st patch is preparation. 2-3 fix different problems, the 3rd one
> depends on 2nd.
> 
> These patches change the code which I don't really understand, please
> review.
> 
> 
> As for the 3rd patch, it replaces
> 
> 	cpu_hotplug-dont-affect-current-tasks-affinity.patch
> 
> in -mm tree. Imho the new patch is more simple and clean, but of course
> this is subjective and I am biased.

Look good to me.

Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>

Ingo, will you stick them in -tip?

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Lai Jiangshan]

Peter Zijlstra wrote:
> On Thu, 2009-09-10 at 21:21 +0200, Oleg Nesterov wrote:
>> (my apologies to those who see this twice).
>>
>>
>> The 1st patch is preparation. 2-3 fix different problems, the 3rd one
>> depends on 2nd.
>>
>> These patches change the code which I don't really understand, please
>> review.
>>
>>
>> As for the 3rd patch, it replaces
>>
>> 	cpu_hotplug-dont-affect-current-tasks-affinity.patch
>>
>> in -mm tree. Imho the new patch is more simple and clean, but of course
>> this is subjective and I am biased.
> 
> Look good to me.
> 
> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> 
> Ingo, will you stick them in -tip?
> 
>

Sorry. I was taken ill for weeks and forgot to follow these discussions.
Especially I should say sorry to Oleg.

I have different concept. cpuset_cpus_allowed() is not called at atomic
context nor non-preemptable context nor other critical context.
So it should be allowed to use mutexs. That's what I think.

There is a bug when migration_call() requires a mutex
before migration has been finished when cpu offline as Oleg described.

Bug this bug is only happened when cpu offline. cpu offline is rare and
is slowpath. I think we should fix cpu offline and ensure it requires
the mutex safely.

Oleg's patch moves all dirty things into CPUSET subsystem and makes
cpuset_cpus_allowed() does not require any mutex and increases CPUSET's
coupling. I don't feel it's good.

Anyway, Oleg's patch works good.

> > cpuset_cpus_allowed() is not only  used for CPU offline.
> > >
> > > sched_setaffinity() also uses it.
> 
> Sure. And it must take get_online_cpus() to avoid the races with hotplug.

Oleg hasn't answered that
"is it safe when pdflush() calls cpuset_cpus_allowed()?".
A patch may be needed to ensure pdflush() calls cpuset_cpus_allowed() safely.

One other minor thing:
Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because
spinlock in RT is also mutex. Likely I'm wrong.

- Lai

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Peter Zijlstra]

On Fri, 2009-09-11 at 15:15 +0800, Lai Jiangshan wrote:
> One other minor thing:
> Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because
> spinlock in RT is also mutex. Likely I'm wrong.

But they have priority-inheritance, hence you cannot create a deadlock
through preemption. If the kstopmachine thread blocks on a mutex, the
owner gets boosted to kstopmachine's prio and runs to completion, after
that kstopmachine continues.

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Peter Zijlstra]

On Fri, 2009-09-11 at 15:15 +0800, Lai Jiangshan wrote:
> 
> I have different concept. cpuset_cpus_allowed() is not called at atomic
> context nor non-preemptable context nor other critical context.
> So it should be allowed to use mutexs. That's what I think.

It hardly does any work at all, so why bother with a mutex?

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Lai Jiangshan]

Peter Zijlstra wrote:
> On Fri, 2009-09-11 at 15:15 +0800, Lai Jiangshan wrote:
>> One other minor thing:
>> Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because
>> spinlock in RT is also mutex. Likely I'm wrong.
> 
> But they have priority-inheritance, hence you cannot create a deadlock
> through preemption. If the kstopmachine thread blocks on a mutex, the
> owner gets boosted to kstopmachine's prio and runs to completion, after
> that kstopmachine continues.
> 

The deadlock is because the owner is at the dead cpu, It's not because
the owner's prio is low. priority-inheritance can't help it.

I think we need to use a raw spinlock.

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Peter Zijlstra]

On Fri, 2009-09-11 at 15:53 +0800, Lai Jiangshan wrote:
> Peter Zijlstra wrote:
> > On Fri, 2009-09-11 at 15:15 +0800, Lai Jiangshan wrote:
> >> One other minor thing:
> >> Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because
> >> spinlock in RT is also mutex. Likely I'm wrong.
> > 
> > But they have priority-inheritance, hence you cannot create a deadlock
> > through preemption. If the kstopmachine thread blocks on a mutex, the
> > owner gets boosted to kstopmachine's prio and runs to completion, after
> > that kstopmachine continues.
> > 
> 
> The deadlock is because the owner is at the dead cpu, It's not because
> the owner's prio is low. priority-inheritance can't help it.
> 
> I think we need to use a raw spinlock.

Not in mainline you don't.

Re: [PATCH 0/3] resend, cpuset/hotplug fixes

[Oleg Nesterov]

On 09/11, Lai Jiangshan wrote:
>
> I have different concept. cpuset_cpus_allowed() is not called at atomic
> context nor non-preemptable context nor other critical context.
> So it should be allowed to use mutexs. That's what I think.

Well, it is called from non-preemptable context: move_task_off_dead_cpu().
That is why before this patch we had cpuset_cpus_allowed_lock(). And this
imho adds unneeded complications.

And I can't understand why sched_setaffinity() path should take the
global mutex instead of per-cpuset spinlock.

> There is a bug when migration_call() requires a mutex
> before migration has been finished when cpu offline as Oleg described.
>
> Bug this bug is only happened when cpu offline. cpu offline is rare and
> is slowpath. I think we should fix cpu offline and ensure it requires
> the mutex safely.

This is subjective, but I can't agree. I think we should fix cpusets
instead. We should try to avoid the dependencies between different
subsystems as much as possible.

> Oleg's patch moves all dirty things into CPUSET subsystem and makes
> cpuset_cpus_allowed() does not require any mutex and increases CPUSET's
> coupling. I don't feel it's good.

Again, subjective... But I can't understand "increases CPUSET's coupling".

>From my pov, this patch cleanups and simplifies the code. This was the
main motivation, the bugfix is just the "side effect". I don't understand
how this subtle cpuset_lock() can make things better. I can't understand
why we need the global lock to calc cpus_allowed.

> > > cpuset_cpus_allowed() is not only  used for CPU offline.
> > > >
> > > > sched_setaffinity() also uses it.
> >
> > Sure. And it must take get_online_cpus() to avoid the races with hotplug.
>
> Oleg hasn't answered that
> "is it safe when pdflush() calls cpuset_cpus_allowed()?".

Because I didn't see such a question ;) perhaps I missed it previously...

> A patch may be needed to ensure pdflush() calls cpuset_cpus_allowed() safely.

What is wrong with pdflush()->cpuset_cpus_allowed() ? Why this is not safe?

This

	cpuset_cpus_allowed(current, cpus_allowed);
	set_cpus_allowed_ptr(current, cpus_allowed);

looks equally racy, with or without the patch. But this is a bit off-topic,
mm/pdflush.c has gone away.

> One other minor thing:
> Oleg's patch may introduce a trouble in PREEEMPT_RT tree, because
> spinlock in RT is also mutex. Likely I'm wrong.

Yes, probably -rt need raw_lock (as you pointed out).

Oleg.