Re: [RFC] Thread Migration Preemption - v4

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

On Sat, 2007-07-14 at 14:42 -0400, Mathieu Desnoyers wrote:

> Note: (or we could say FIXME)
> Is we ever want to check migration pending in assembly code, we will have to
> make sure we test the right thread flag bits on each architectures. Care should
> also be taken to check that the thread flags used won't trigger false positives
> in non selective asm thread flag checks.
> 
> FIXME (HOTPLUG) :
> 
> > >     /* Affinity changed (again). */
> > >     if (!cpu_isset(dest_cpu, p->cpus_allowed))
> > >             goto out;
> > >  
> > >     on_rq = p->se.on_rq;
> > > +#ifdef CONFIG_PREEMPT
> > > +   if (!on_rq && task_thread_info(p)->migrate_count)
> > > +           goto out;
> > > +#endif
> > 
> > This means that move_task_off_dead_cpu() will spin until the task will be
> > scheduled
> > on the dead CPU. Given that we hold tasklist_lock and irqs are disabled, this
> > may
> > never happen.
> >
> 
> Yes. My idea to fix this issue is the following:
> 
> If a thread has non zero migrate_count, we should still move it to a
> different CPU upon hotplug cpu removal, even if this thread resists
> migration. Care should be taken to send _all_ such threads to the _same_
> CPU so they don't race for the per-cpu ressources. Does it make sense ?
> 
> We would have to keep the CPU affinity of the threads running on the
> wrong CPU until they end their migrate disabled section, so that we can
> put them back on their original CPU if it goes back online, otherwise we
> could end up with concurrent per-cpu variables accesses.
> 
> (I'll wait for reply before coding a solution for this CPU HOTPLUG
> related problem)

What would, aside from technical issues, be the problem with making
migration_disable() delay CPU_DOWN until migration_enable()?