Re: Scheduler questions

[William Lee Irwin III <wli@holomorphy.com>]

On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> I'd like to understand which task structure elements are protected by
> which locks (as far as scheduling is concerned). Is there somewhere a
> paper summarizing the mutual exclusion rules ?
> Let's take some code e.g. from the 2.6.5 kernel:
> set_cpus_allowed(task_t *p, cpumask_t new_mask):
> 
> 	rq = task_rq_lock(p, &flags)
> 	__set_cpus_allowed(p, new_mask, &req):
> 
> 		p->cpus_allowed = new_mask
> 		/*
> 		 * If the task is not on a runqueue (and not running), then
> 		 * it is sufficient to simply update the task's cpu field.
> 		 */
> 		if (!p->array && !task_running(rq, p))
> 			set_task_cpu(p, any_online_cpu(p->cpus_allowed))
> 		/* ... */
> 
> 	task_rq_unlock(rq, &flags)

Okay.

On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> Apparently, the "p->cpus_allowed" and the "p->thread_info->cpu" fields
> are protected by the "(&per_cpu(runqueues, (p->thread_info->cpu)))->lock".
> Which are the other task structure elements protected by the same lock ?
> Let's take an example:

Yes. ->cpus_allowed is protected by rq->lock. Do not look at the midget
behind the curtain for what protects ->cpus_allowed of a sleeping task.


On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> - I've got a sleeping task that ran on the CPU #3 previously
> - I want to set its CPU mask equal to {1, 2}
> - I take the lock of the run queue #3
> - I do set the CPU mask of the task
> - It's not running (BTW when can happen that "p->array" is NULL and the
>   task is still running ?)
> - I do set the task's CPU e.g. equal to 2. As a consequence, the task
>   falls out of the protection provided by the lock of the run queue #3.
> - Someone else deciding to play with the same task, s/he takes the
>   lock of the run queue #2 !!!
> - Me, I have not arrived yet to the unlock. There are stuffs to do before.
> - Both of us think to have the exclusive access right to the task...
> Can someone explain me, please, why I have to take a run queue lock
> to protect a not running task, and why we do not use "proc_lock"
> instead ?

You looked at the midget behind the curtain. Anyway, rq->lock is just
reused based on the sleeping task's cpu affinity.

->array == NULL while a task is running should not happen apart from
a dequeueing in progress. ->array should be protected by
per_cpu(runqueues, task_cpu(task)).lock.

Scheduler-private fields of tasks are considered to be under the
protection of per_cpu(runqueues, task_cpu(task)).lock until someone
succeeds in a double runqueue lock acquisition and moves the task
between two cpus, or as an optimization, to reset task->thread_info->cpu
under its current rq->lock without holding the target's lock if it's not
queued anywhere.

There is some retry logic in task_rq_lock() to make sure the cpu and
the runqueue are consistent that resolves the remainder of the cases.


-- wli