Re: [RFC] Thread Migration Preemption

[Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>]

* Oleg Nesterov (oleg@tv-sign.ru) wrote:
> Mathieu Desnoyers wrote:
> >
> > This patch adds the ability to protect critical sections from migration to
> > another CPU without disabling preemption.
> >
> > Typical use:
> >
> > migration_disable();
> > local_inc(&__get_cpu_var(&my_local_t_var));
> > migration_enable();
> >
> > --- linux-2.6-lttng.orig/kernel/sched.c	2007-07-05 16:28:15.000000000 -0400
> > +++ linux-2.6-lttng/kernel/sched.c	2007-07-05 16:53:24.000000000 -0400
> > @@ -1996,6 +1996,7 @@
> >  	 * 1) running (obviously), or
> >  	 * 2) cannot be migrated to this CPU due to cpus_allowed, or
> >  	 * 3) are cache-hot on their current CPU.
> > +	 * 4) migration preemption is non 0 for this non running task.
> >  	 */
> >  	if (!cpu_isset(this_cpu, p->cpus_allowed))
> >  		return 0;
> > @@ -2003,6 +2004,8 @@
> >
> >  	if (task_running(rq, p))
> >  		return 0;
> > +	if (task_thread_info(p)->migration_count)
> > +		return 0;
> 

Hi Oleg,

> Question: This means that the task could be preempted, but can't sleep, yes?
> 

No exactly. This means that the task can be preempted, and can therefore
sleep, but we do not allow this sleeping task to be migrated to a
different CPU.

> Because try_to_wake_up() can change ->cpu. Shouldn't might_sleep() check
> ->migration_count then? Or we can change try_to_wake_up().
> 

If we look at try_to_wake_up, it seems to take the new_cpu from
task_cpu(), which comes from the thread info ->cpu, updated by
set_task_cpu(). The *_move_tasks seems to be responsible for calling
can_migrate_task() to see if, in a deferred way, the migration_thread
can use set_task_cpu() to change the task's cpu.

move_task_off_dead_cpu() does not seem to respect this though: it could
potentially move away a process while it is in a critical section by
forcing a __migrate_task. I guess I should put a test in __migrate_task
also so move_task_off_dead_cpu() can try to repeat the migration.

> What if the task does copy_process() under migration_disable() ? Child
> gets a copy of ->migration_count.
> 

Right. Neighter does copy_process set the preemption count. It seems to
be dealt by sched_fork() when the thread is first scheduled in to make
sure that the preempt count is 1. I guess is would be safer to set the
migration count to 0 there too. (or set it to one if we ever decide to
make it the kernel default).

> Also, cpu_down() still can migrate this task to another CPU.
> 

Yes, just noted this earlier. I first thought it was using
can_migrate_task(), but it does not seem so.

Thanks,

Mathieu

> Oleg.
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68