Scheduler questions

Scheduler questions

[Zoltan Menyhart]

I'd like to understand which task structure elements are protected by
which locks (as far as scheduling is concerned). Is there somewhere a
paper summarizing the mutual exclusion rules ?

Let's take some code e.g. from the 2.6.5 kernel:

set_cpus_allowed(task_t *p, cpumask_t new_mask):

	rq = task_rq_lock(p, &flags)
	__set_cpus_allowed(p, new_mask, &req):

		p->cpus_allowed = new_mask
		/*
		 * If the task is not on a runqueue (and not running), then
		 * it is sufficient to simply update the task's cpu field.
		 */
		if (!p->array && !task_running(rq, p))
			set_task_cpu(p, any_online_cpu(p->cpus_allowed))
		/* ... */

	task_rq_unlock(rq, &flags)

Apparently, the "p->cpus_allowed" and the "p->thread_info->cpu" fields
are protected by the "(&per_cpu(runqueues, (p->thread_info->cpu)))->lock".

Which are the other task structure elements protected by the same lock ?

Let's take an example:

- I've got a sleeping task that ran on the CPU #3 previously
- I want to set its CPU mask equal to {1, 2}
- I take the lock of the run queue #3
- I do set the CPU mask of the task
- It's not running (BTW when can happen that "p->array" is NULL and the
  task is still running ?)
- I do set the task's CPU e.g. equal to 2. As a consequence, the task
  falls out of the protection provided by the lock of the run queue #3.
- Someone else deciding to play with the same task, s/he takes the
  lock of the run queue #2 !!!
- Me, I have not arrived yet to the unlock. There are stuffs to do before.
- Both of us think to have the exclusive access right to the task...

Can someone explain me, please, why I have to take a run queue lock
to protect a not running task, and why we do not use "proc_lock"
instead ?

Thanks,

Zoltán Menyhárt

VM validation question

[Zoltan Menyhart]

I do "some" modification to the virtual memory subsystem.
How can I make sure that the compatibility is kept ?
Have you got some VM validation packages, stress tests, etc. ?
Could you please, indicate how I can pick up to these tests ?

Thanks,


Zoltán Menyhárt

Re: VM validation question

[William Lee Irwin III]

On Wed, Jun 16, 2004 at 11:36:52AM +0200, Zoltan Menyhart wrote:
> I do "some" modification to the virtual memory subsystem.
> How can I make sure that the compatibility is kept ?
> Have you got some VM validation packages, stress tests, etc. ?
> Could you please, indicate how I can pick up to these tests ?

There are few, if any ABI compatibility test suites. By and large,
important breakages are almost immediately observable, and the less
important ones can be fixed up afterward, so I'd encourage you to
proceed with testing by means of targeted testcases when a specific
area is in doubt, and general benchmarks and stress tests for the rest.


-- wli

Re: Scheduler questions

[William Lee Irwin III]

On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> I'd like to understand which task structure elements are protected by
> which locks (as far as scheduling is concerned). Is there somewhere a
> paper summarizing the mutual exclusion rules ?
> Let's take some code e.g. from the 2.6.5 kernel:
> set_cpus_allowed(task_t *p, cpumask_t new_mask):
> 
> 	rq = task_rq_lock(p, &flags)
> 	__set_cpus_allowed(p, new_mask, &req):
> 
> 		p->cpus_allowed = new_mask
> 		/*
> 		 * If the task is not on a runqueue (and not running), then
> 		 * it is sufficient to simply update the task's cpu field.
> 		 */
> 		if (!p->array && !task_running(rq, p))
> 			set_task_cpu(p, any_online_cpu(p->cpus_allowed))
> 		/* ... */
> 
> 	task_rq_unlock(rq, &flags)

Okay.

On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> Apparently, the "p->cpus_allowed" and the "p->thread_info->cpu" fields
> are protected by the "(&per_cpu(runqueues, (p->thread_info->cpu)))->lock".
> Which are the other task structure elements protected by the same lock ?
> Let's take an example:

Yes. ->cpus_allowed is protected by rq->lock. Do not look at the midget
behind the curtain for what protects ->cpus_allowed of a sleeping task.


On Fri, Jun 04, 2004 at 05:10:22PM +0200, Zoltan Menyhart wrote:
> - I've got a sleeping task that ran on the CPU #3 previously
> - I want to set its CPU mask equal to {1, 2}
> - I take the lock of the run queue #3
> - I do set the CPU mask of the task
> - It's not running (BTW when can happen that "p->array" is NULL and the
>   task is still running ?)
> - I do set the task's CPU e.g. equal to 2. As a consequence, the task
>   falls out of the protection provided by the lock of the run queue #3.
> - Someone else deciding to play with the same task, s/he takes the
>   lock of the run queue #2 !!!
> - Me, I have not arrived yet to the unlock. There are stuffs to do before.
> - Both of us think to have the exclusive access right to the task...
> Can someone explain me, please, why I have to take a run queue lock
> to protect a not running task, and why we do not use "proc_lock"
> instead ?

You looked at the midget behind the curtain. Anyway, rq->lock is just
reused based on the sleeping task's cpu affinity.

->array == NULL while a task is running should not happen apart from
a dequeueing in progress. ->array should be protected by
per_cpu(runqueues, task_cpu(task)).lock.

Scheduler-private fields of tasks are considered to be under the
protection of per_cpu(runqueues, task_cpu(task)).lock until someone
succeeds in a double runqueue lock acquisition and moves the task
between two cpus, or as an optimization, to reset task->thread_info->cpu
under its current rq->lock without holding the target's lock if it's not
queued anywhere.

There is some retry logic in task_rq_lock() to make sure the cpu and
the runqueue are consistent that resolves the remainder of the cases.


-- wli