[PATCH -rt 0/5] making SLUB -rt friendly

[PATCH -rt 0/5] making SLUB -rt friendly

[Peter Zijlstra]

Hi,

This is an attempt at converting SLUB to -rt. It seems to work reasonably
well. But I haven't really pushed it _very_ _very_ hard yet.

Enjoy


--

[PATCH -rt 1/5] workqueue: queue_work_cpu

[Peter Zijlstra]

[-- Attachment #1: queue_work_cpu.patch --]
[-- Type: text/plain, Size: 1615 bytes --]

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 include/linux/workqueue.h |    1 +
 kernel/workqueue.c        |   13 +++++++++++++
 2 files changed, 14 insertions(+)

Index: linux-2.6/include/linux/workqueue.h
===================================================================
--- linux-2.6.orig/include/linux/workqueue.h
+++ linux-2.6/include/linux/workqueue.h
@@ -131,6 +131,7 @@ extern void set_workqueue_prio(struct wo
 extern void destroy_workqueue(struct workqueue_struct *wq);
 
 extern int FASTCALL(queue_work(struct workqueue_struct *wq, struct work_struct *work));
+extern int FASTCALL(queue_work_cpu(struct workqueue_struct *wq, struct work_struct *work, int cpu));
 extern int FASTCALL(queue_delayed_work(struct workqueue_struct *wq,
 			struct delayed_work *work, unsigned long delay));
 extern int queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
Index: linux-2.6/kernel/workqueue.c
===================================================================
--- linux-2.6.orig/kernel/workqueue.c
+++ linux-2.6/kernel/workqueue.c
@@ -176,6 +176,19 @@ int fastcall queue_work(struct workqueue
 }
 EXPORT_SYMBOL_GPL(queue_work);
 
+int fastcall queue_work_cpu(struct workqueue_struct *wq, struct work_struct *work, int cpu)
+{
+	int ret = 0;
+
+	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
+		BUG_ON(!list_empty(&work->entry));
+		__queue_work(wq_per_cpu(wq, cpu), work);
+		ret = 1;
+	}
+	return ret;
+}
+EXPORT_SYMBOL_GPL(queue_work_cpu);
+
 void delayed_work_timer_fn(unsigned long __data)
 {
 	struct delayed_work *dwork = (struct delayed_work *)__data;

--

Re: [PATCH -rt 1/5] workqueue: queue_work_cpu

[Oleg Nesterov]

On 07/14, Peter Zijlstra wrote:
>
> +int fastcall queue_work_cpu(struct workqueue_struct *wq, struct work_struct *work, int cpu)
> +{
> +	int ret = 0;
> +
> +	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
> +		BUG_ON(!list_empty(&work->entry));
> +		__queue_work(wq_per_cpu(wq, cpu), work);
> +		ret = 1;
> +	}
> +	return ret;
> +}

Minor nit, we already have queue_delayed_work_on(), perhaps it should be named
queue_work_on().

Oleg.

Re: [PATCH -rt 1/5] workqueue: queue_work_cpu

[Peter Zijlstra]

On Sat, 2007-07-14 at 21:14 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:
> >
> > +int fastcall queue_work_cpu(struct workqueue_struct *wq, struct work_struct *work, int cpu)
> > +{
> > +	int ret = 0;
> > +
> > +	if (!test_and_set_bit(WORK_STRUCT_PENDING, work_data_bits(work))) {
> > +		BUG_ON(!list_empty(&work->entry));
> > +		__queue_work(wq_per_cpu(wq, cpu), work);
> > +		ret = 1;
> > +	}
> > +	return ret;
> > +}
> 
> Minor nit, we already have queue_delayed_work_on(), perhaps it should be named
> queue_work_on().

Yes, that makes sense. Shall do.

[PATCH -rt 2/5] Thread Migration Preemption - v2

[Peter Zijlstra]

[-- Attachment #1: thread-migration.patch --]
[-- Type: text/plain, Size: 48685 bytes --]

From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

This patch adds the ability to protect critical sections from migration to
another CPU without disabling preemption.

This will be useful to minimize the amount of preemption disabling for the -rt
patch. It will help leveraging improvements brought by the local_t types in
asm/local.h (see Documentation/local_ops.txt). Note that the updates done to
variables protected by migrate_disable must be either atomic or protected from
concurrent updates done by other threads.

Typical use:

migrate_disable();
local_inc(&__get_cpu_var(&my_local_t_var));
migrate_enable();

Which will increment the variable atomically wrt the local CPU.

Changes:

Do a check_migrate() upon migrate_enable() to answer to the migration
thread waiting for us to exit the migration disabled critical section. Use a
NEED_MIGRATE thread flag for this.

Note: (or we could say FIXME)
Is we ever want to check migration pending in assembly code, we will have to
make sure we test the right thread flag bits on each architectures. Care should
also be taken to check that the thread flags used won't trigger false positives
in non selective asm thread flag checks.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 include/asm-alpha/thread_info.h     |    3 +
 include/asm-arm/thread_info.h       |    5 +
 include/asm-arm26/thread_info.h     |    5 +
 include/asm-avr32/thread_info.h     |    4 +
 include/asm-blackfin/thread_info.h  |    4 +
 include/asm-cris/thread_info.h      |    4 +
 include/asm-frv/thread_info.h       |    4 +
 include/asm-h8300/thread_info.h     |    4 +
 include/asm-i386/thread_info.h      |    6 +-
 include/asm-ia64/thread_info.h      |    4 +
 include/asm-m32r/thread_info.h      |    4 +
 include/asm-m68k/thread_info.h      |    2 
 include/asm-m68knommu/thread_info.h |    3 +
 include/asm-mips/thread_info.h      |    4 +
 include/asm-parisc/thread_info.h    |    4 +
 include/asm-powerpc/thread_info.h   |    4 +
 include/asm-s390/thread_info.h      |    4 +
 include/asm-sh/thread_info.h        |    4 +
 include/asm-sh64/thread_info.h      |    4 +
 include/asm-sparc/thread_info.h     |    4 +
 include/asm-sparc64/thread_info.h   |    5 +
 include/asm-um/thread_info.h        |    4 +
 include/asm-v850/thread_info.h      |    4 +
 include/asm-x86_64/thread_info.h    |    4 +
 include/asm-xtensa/thread_info.h    |    4 +
 include/linux/preempt.h             |   44 +++++++++++++++++
 kernel/sched.c                      |   91 +++++++++++++++++++++++++++++++++---
 lib/smp_processor_id.c              |    6 +-
 28 files changed, 230 insertions(+), 12 deletions(-)

Index: linux-2.6/include/asm-i386/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-i386/thread_info.h
+++ linux-2.6/include/asm-i386/thread_info.h
@@ -31,8 +31,7 @@ struct thread_info {
 	unsigned long		status;		/* thread-synchronous flags */
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
-
-
+	int			migrate_count;/* 0: can migrate, <0: BUG */
 	mm_segment_t		addr_limit;	/* thread address space:
 					 	   0-0xBFFFFFFF for user-thead
 						   0-0xFFFFFFFF for kernel-thread
@@ -74,6 +73,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
@@ -134,6 +134,7 @@ static inline struct thread_info *curren
 #define TIF_SECCOMP		8	/* secure computing */
 #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal() */
 #define TIF_NEED_RESCHED_DELAYED 10	/* reschedule on return to userspace */
+#define TIF_NEED_MIGRATE	11	/* migration necessary */
 #define TIF_MEMDIE		16
 #define TIF_DEBUG		17	/* uses debug registers */
 #define TIF_IO_BITMAP		18	/* uses I/O bitmap */
@@ -151,6 +152,7 @@ static inline struct thread_info *curren
 #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
 #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_DEBUG		(1<<TIF_DEBUG)
 #define _TIF_IO_BITMAP		(1<<TIF_IO_BITMAP)
 #define _TIF_FREEZE		(1<<TIF_FREEZE)
Index: linux-2.6/include/linux/preempt.h
===================================================================
--- linux-2.6.orig/include/linux/preempt.h
+++ linux-2.6/include/linux/preempt.h
@@ -15,6 +15,8 @@
   extern void notrace sub_preempt_count(unsigned int val);
   extern void notrace mask_preempt_count(unsigned int mask);
   extern void notrace unmask_preempt_count(unsigned int mask);
+  extern void fastcall add_migrate_count(int val);
+  extern void fastcall sub_migrate_count(int val);
 #else
 # define add_preempt_count(val)	do { preempt_count() += (val); } while (0)
 # define sub_preempt_count(val)	do { preempt_count() -= (val); } while (0)
@@ -22,6 +24,8 @@
 		do { preempt_count() |= (mask); } while (0)
 # define unmask_preempt_count(mask) \
 		do { preempt_count() &= ~(mask); } while (0)
+# define add_migrate_count(val) do { migrate_count() += (val); } while (0)
+# define sub_migrate_count(val) do { migrate_count() -= (val); } while (0)
 #endif
 
 #ifdef CONFIG_CRITICAL_TIMING
@@ -35,7 +39,12 @@
 #define inc_preempt_count() add_preempt_count(1)
 #define dec_preempt_count() sub_preempt_count(1)
 
-#define preempt_count()		(current_thread_info()->preempt_count)
+#define preempt_count() (current_thread_info()->preempt_count)
+
+#define inc_migrate_count() add_migrate_count(1)
+#define dec_migrate_count() sub_migrate_count(1)
+
+#define migrate_count() (current_thread_info()->migrate_count)
 
 #ifdef CONFIG_PREEMPT
 
@@ -80,6 +89,36 @@ do { \
 	preempt_check_resched(); \
 } while (0)
 
+#define migrate_disable() \
+do { \
+	inc_migrate_count(); \
+	barrier(); \
+} while (0)
+
+#define migrate_enable_no_check() \
+do { \
+	barrier(); \
+	dec_migrate_count(); \
+} while (0)
+
+#ifdef CONFIG_SMP
+extern void do_check_migrate(void);
+#define check_migrate() \
+do { \
+	if (unlikely(test_thread_flag(TIF_NEED_MIGRATE))) \
+		do_check_migrate(); \
+} while (0)
+#else
+#define check_migrate()
+#endif
+
+#define migrate_enable() \
+do { \
+	migrate_enable_no_check(); \
+	barrier(); \
+	check_migrate(); \
+} while (0)
+
 #else
 
 #define preempt_disable()		do { } while (0)
@@ -91,6 +130,9 @@ do { \
 
 #define preempt_schedule_irq()		do { } while (0)
 
+#define migrate_disable()		do { } while (0)
+#define migrate_enable()		do { } while (0)
+
 #endif
 
 #endif /* __LINUX_PREEMPT_H */
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c
+++ linux-2.6/kernel/sched.c
@@ -1175,7 +1175,8 @@ migrate_task(struct task_struct *p, int 
 	 * If the task is not on a runqueue (and not running), then
 	 * it is sufficient to simply update the task's cpu field.
 	 */
-	if (!p->se.on_rq && !task_running(rq, p)) {
+	if (!p->se.on_rq && !task_running(rq, p)
+			&& !task_thread_info(p)->migrate_count) {
 		set_task_cpu(p, dest_cpu);
 		return 0;
 	}
@@ -1506,6 +1507,29 @@ static void balance_rt_tasks(struct rq *
 
 #endif
 
+void fastcall add_migrate_count(int val)
+{
+	/*
+	 * Underflow?
+	 */
+	if (DEBUG_LOCKS_WARN_ON((migrate_count() < 0)))
+		return;
+	migrate_count() += val;
+}
+EXPORT_SYMBOL(add_migrate_count);
+
+void fastcall sub_migrate_count(int val)
+{
+	/*
+	 * Underflow?
+	 */
+	if (DEBUG_LOCKS_WARN_ON(val > migrate_count()))
+		return;
+
+	migrate_count() -= val;
+}
+EXPORT_SYMBOL(sub_migrate_count);
+
 /*
  * find_idlest_cpu - find the idlest cpu among the cpus in group.
  */
@@ -1696,7 +1720,10 @@ try_to_wake_up(struct task_struct *p, un
 #ifdef CONFIG_SMP
 	if (unlikely(task_running(rq, p)))
 		goto out_activate;
-
+#ifdef CONFIG_PREEMPT
+	if (task_thread_info(p)->migrate_count)
+		goto out_activate;
+#endif
 	new_cpu = cpu;
 
 	schedstat_inc(rq, ttwu_cnt);
@@ -1970,6 +1997,7 @@ void sched_fork(struct task_struct *p, i
 #ifdef CONFIG_PREEMPT
 	/* Want to start with kernel preemption disabled. */
 	task_thread_info(p)->preempt_count = 1;
+	task_thread_info(p)->migrate_count = 0;
 #endif
 	put_cpu();
 }
@@ -2435,6 +2463,10 @@ static void sched_migrate_task(struct ta
 	if (!cpu_isset(dest_cpu, p->cpus_allowed)
 	    || unlikely(cpu_is_offline(dest_cpu)))
 		goto out;
+#ifdef CONFIG_PREEMPT
+	if (task_thread_info(p)->migrate_count)
+		goto out;
+#endif
 
 	/* force the process onto the specified CPU */
 	if (migrate_task(p, dest_cpu, &req)) {
@@ -2496,6 +2528,7 @@ int can_migrate_task(struct task_struct 
 	 * 1) running (obviously), or
 	 * 2) cannot be migrated to this CPU due to cpus_allowed, or
 	 * 3) are cache-hot on their current CPU.
+	 * 4) migration preemption is non 0 for this non running task.
 	 */
 	if (!cpu_isset(this_cpu, p->cpus_allowed))
 		return 0;
@@ -2503,6 +2536,10 @@ int can_migrate_task(struct task_struct 
 
 	if (task_running(rq, p))
 		return 0;
+#ifdef CONFIG_PREEMPT
+	if (task_thread_info(p)->migrate_count)
+		return 0;
+#endif
 
 	/*
 	 * Aggressive migration if too many balance attempts have failed:
@@ -3822,6 +3859,27 @@ EXPORT_SYMBOL(schedule);
 
 #ifdef CONFIG_PREEMPT
 
+#ifdef CONFIG_SMP
+/*
+ * Wake up the migration thread to deal with our pending migration.
+ * If we have been moved since, we will just wake up the migration thread from
+ * the wrong CPU, which does not hurt anyone. (that's why we use
+ * raw_smp_processor_id()).
+ */
+void __sched do_check_migrate(void)
+{
+	struct rq *rq;
+
+	if (migrate_count())
+		return;
+
+	clear_thread_flag(TIF_NEED_MIGRATE);
+	rq = cpu_rq(raw_smp_processor_id());
+	wake_up_process(rq->migration_thread);
+}
+EXPORT_SYMBOL(do_check_migrate);
+#endif
+
 /*
  * Global flag to turn preemption off on a CONFIG_PREEMPT kernel:
  */
@@ -5478,7 +5536,7 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed);
  * So we race with normal scheduler movements, but that's OK, as long
  * as the task is no longer on this CPU.
  *
- * Returns non-zero if task was successfully migrated.
+ * Returns non-zero if task is on dest_cpu when this function ends.
  */
 static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)
 {
@@ -5500,13 +5558,19 @@ static int __migrate_task(struct task_st
 
 	double_rq_lock(rq_src, rq_dest);
 	/* Already moved. */
-	if (task_cpu(p) != src_cpu)
+	if (task_cpu(p) != src_cpu) {
+		ret = 1;
 		goto out;
+	}
 	/* Affinity changed (again). */
 	if (!cpu_isset(dest_cpu, p->cpus_allowed))
 		goto out;
 
 	on_rq = p->se.on_rq;
+#ifdef CONFIG_PREEMPT
+	if (!on_rq && task_thread_info(p)->migrate_count)
+		goto out;
+#endif
 	if (on_rq)
 		deactivate_task(rq_src, p, 0);
 
@@ -5532,6 +5596,7 @@ static int migration_thread(void *data)
 {
 	int cpu = (long)data;
 	struct rq *rq;
+	int migrated;
 
 	rq = cpu_rq(cpu);
 	BUG_ON(rq->migration_thread != current);
@@ -5567,10 +5632,22 @@ static int migration_thread(void *data)
 		list_del_init(head->next);
 
 		spin_unlock(&rq->lock);
-		__migrate_task(req->task, cpu, req->dest_cpu);
+		migrated = __migrate_task(req->task, cpu, req->dest_cpu);
 		local_irq_enable();
-
-		complete(&req->done);
+		if (!migrated) {
+			/*
+			 * If the process has not been migrated, let it run
+			 * until it reaches a migration_check() so it can
+			 * wake us up.
+			 */
+			spin_lock_irq(&rq->lock);
+			head = &rq->migration_queue;
+			list_add(&req->list, head);
+			set_tsk_thread_flag(req->task, TIF_NEED_MIGRATE);
+			spin_unlock_irq(&rq->lock);
+			wake_up_process(req->task);
+		} else
+			complete(&req->done);
 	}
 	__set_current_state(TASK_RUNNING);
 	return 0;
Index: linux-2.6/include/asm-alpha/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-alpha/thread_info.h
+++ linux-2.6/include/asm-alpha/thread_info.h
@@ -21,6 +21,7 @@ struct thread_info {
 	mm_segment_t		addr_limit;	/* thread address space */
 	unsigned		cpu;		/* current CPU */
 	int			preempt_count; /* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	int bpt_nsaved;
 	unsigned long bpt_addr[2];		/* breakpoint handling  */
@@ -77,6 +78,7 @@ register struct thread_info *__current_t
 #define TIF_UAC_SIGBUS		8
 #define TIF_MEMDIE		9
 #define TIF_RESTORE_SIGMASK	10	/* restore signal mask in do_signal */
+#define TIF_NEED_MIGRATE	11	/* migration necessary */
 
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
 #define _TIF_NOTIFY_RESUME	(1<<TIF_NOTIFY_RESUME)
@@ -84,6 +86,7 @@ register struct thread_info *__current_t
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 /* Work to do on interrupt/exception return.  */
 #define _TIF_WORK_MASK		(_TIF_NOTIFY_RESUME	\
Index: linux-2.6/include/asm-arm/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-arm/thread_info.h
+++ linux-2.6/include/asm-arm/thread_info.h
@@ -51,6 +51,7 @@ struct cpu_context_save {
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	int			preempt_count;	/* 0 => preemptable, <0 => bug */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	mm_segment_t		addr_limit;	/* address limit */
 	struct task_struct	*task;		/* main task structure */
 	struct exec_domain	*exec_domain;	/* execution domain */
@@ -72,6 +73,7 @@ struct thread_info {
 	.exec_domain	= &default_exec_domain,				\
 	.flags		= 0,						\
 	.preempt_count	= 1,						\
+	.migrate_count = 0,						\
 	.addr_limit	= KERNEL_DS,					\
 	.cpu_domain	= domain_val(DOMAIN_USER, DOMAIN_MANAGER) |	\
 			  domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) |	\
@@ -138,6 +140,7 @@ extern void iwmmxt_task_switch(struct th
  *  TIF_NOTIFY_RESUME	- resumption notification requested
  *  TIF_SIGPENDING	- signal pending
  *  TIF_NEED_RESCHED	- rescheduling necessary
+ *  TIF_NEED_MIGRATE	- migration necessary
  *  TIF_USEDFPU		- FPU was used by this task this quantum (SMP)
  *  TIF_POLLING_NRFLAG	- true if poll_idle() is polling TIF_NEED_RESCHED
  */
@@ -145,6 +148,7 @@ extern void iwmmxt_task_switch(struct th
 #define TIF_SIGPENDING		1
 #define TIF_NEED_RESCHED	2
 #define TIF_NEED_RESCHED_DELAYED 3
+#define TIF_NEED_MIGRATE	4
 #define TIF_SYSCALL_TRACE	8
 #define TIF_POLLING_NRFLAG	16
 #define TIF_USING_IWMMXT	17
@@ -155,6 +159,7 @@ extern void iwmmxt_task_switch(struct th
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
 #define _TIF_USING_IWMMXT	(1 << TIF_USING_IWMMXT)
Index: linux-2.6/include/asm-arm26/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-arm26/thread_info.h
+++ linux-2.6/include/asm-arm26/thread_info.h
@@ -45,6 +45,7 @@ struct cpu_context_save {
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	int			preempt_count;	/* 0 => preemptable, <0 => bug */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	mm_segment_t		addr_limit;	/* address limit */
 	struct task_struct	*task;		/* main task structure */
 	struct exec_domain      *exec_domain;   /* execution domain */
@@ -60,6 +61,7 @@ struct thread_info {
 	.exec_domain	&default_exec_domain,	\
 	.flags		0,			\
 	.preempt_count	0,			\
+	.migrate_count 0,			\
 	.addr_limit	KERNEL_DS,		\
 	.restart_block  = {                             \
 		.fn     = do_no_restart_syscall,        \
@@ -113,12 +115,14 @@ extern void free_thread_info(struct thre
  *  TIF_NOTIFY_RESUME	- resumption notification requested
  *  TIF_SIGPENDING	- signal pending
  *  TIF_NEED_RESCHED	- rescheduling necessary
+ *  TIF_NEED_MIGRATE	- migration necessary
  *  TIF_USEDFPU		- FPU was used by this task this quantum (SMP)
  *  TIF_POLLING_NRFLAG	- true if poll_idle() is polling TIF_NEED_RESCHED
  */
 #define TIF_NOTIFY_RESUME	0
 #define TIF_SIGPENDING		1
 #define TIF_NEED_RESCHED	2
+#define TIF_NEED_MIGRATE	3
 #define TIF_SYSCALL_TRACE	8
 #define TIF_USED_FPU		16
 #define TIF_POLLING_NRFLAG	17
@@ -127,6 +131,7 @@ extern void free_thread_info(struct thre
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_USED_FPU		(1 << TIF_USED_FPU)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
Index: linux-2.6/include/asm-avr32/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-avr32/thread_info.h
+++ linux-2.6/include/asm-avr32/thread_info.h
@@ -25,6 +25,7 @@ struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	__u32			cpu;
 	__s32			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	struct restart_block	restart_block;
 	__u8			supervisor_stack[0];
 };
@@ -36,6 +37,7 @@ struct thread_info {
 	.flags		= 0,						\
 	.cpu		= 0,						\
 	.preempt_count	= 1,						\
+	.migrate_count = 0,						\
 	.restart_block	= {						\
 		.fn	= do_no_restart_syscall				\
 	}								\
@@ -84,6 +86,7 @@ static inline struct thread_info *curren
 #define TIF_MEMDIE		7
 #define TIF_RESTORE_SIGMASK	8	/* restore signal mask in do_signal */
 #define TIF_CPU_GOING_TO_SLEEP	9	/* CPU is entering sleep 0 mode */
+#define TIF_NEED_MIGRATE      10       /* migration necessary */
 #define TIF_USERSPACE		31      /* true if FS sets userspace */
 
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
@@ -96,6 +99,7 @@ static inline struct thread_info *curren
 #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
 #define _TIF_CPU_GOING_TO_SLEEP (1 << TIF_CPU_GOING_TO_SLEEP)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 
 /* XXX: These two masks must never span more than 16 bits! */
 /* work to do on interrupt/exception return */
Index: linux-2.6/include/asm-blackfin/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-blackfin/thread_info.h
+++ linux-2.6/include/asm-blackfin/thread_info.h
@@ -54,6 +54,7 @@ struct thread_info {
 	unsigned long flags;	/* low level flags */
 	int cpu;		/* cpu we're on */
 	int preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int migrate_count;	/* 0: can migrate, <0 => BUG */
 	mm_segment_t addr_limit;	/* address limit */
 	struct restart_block restart_block;
 	struct l1_scratch_task_info l1_task_info;
@@ -69,6 +70,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count  = 1,                    \
+	.migrate_count = 0,			\
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
 	},					\
@@ -126,6 +128,7 @@ static inline struct thread_info *curren
 #define TIF_MEMDIE              5
 #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
 #define TIF_FREEZE              7       /* is freezing for suspend */
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -135,6 +138,7 @@ static inline struct thread_info *curren
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
 #define _TIF_FREEZE             (1<<TIF_FREEZE)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 
Index: linux-2.6/include/asm-cris/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-cris/thread_info.h
+++ linux-2.6/include/asm-cris/thread_info.h
@@ -32,6 +32,7 @@ struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	/* thread address space:
 					 	   0-0xBFFFFFFF for user-thead
@@ -58,6 +59,7 @@ struct thread_info {
 	.flags		= 0,				\
 	.cpu		= 0,				\
 	.preempt_count	= 1,				\
+	.migrate_count = 0,				\
 	.addr_limit	= KERNEL_DS,			\
 	.restart_block = {				\
 		       .fn = do_no_restart_syscall,	\
@@ -82,6 +84,7 @@ struct thread_info {
 #define TIF_NOTIFY_RESUME	1	/* resumption notification requested */
 #define TIF_SIGPENDING		2	/* signal pending */
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
+#define TIF_NEED_MIGRATE	4	/* migration necessary */
 #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 #define TIF_MEMDIE		17
 
@@ -90,6 +93,7 @@ struct thread_info {
 #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
Index: linux-2.6/include/asm-frv/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-frv/thread_info.h
+++ linux-2.6/include/asm-frv/thread_info.h
@@ -36,6 +36,7 @@ struct thread_info {
 	unsigned long		status;		/* thread-synchronous flags */
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	/* thread address space:
 					 	   0-0xBFFFFFFF for user-thead
@@ -68,6 +69,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
@@ -114,6 +116,7 @@ register struct thread_info *__current_t
 #define TIF_SINGLESTEP		4	/* restore singlestep on return to user mode */
 #define TIF_IRET		5	/* return with iret */
 #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
+#define TIF_NEED_MIGRATE	7	/* migration necessary */
 #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 #define TIF_MEMDIE		17	/* OOM killer killed process */
 #define TIF_FREEZE		18	/* freezing for suspend */
@@ -127,6 +130,7 @@ register struct thread_info *__current_t
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
 #define _TIF_FREEZE		(1 << TIF_FREEZE)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
Index: linux-2.6/include/asm-h8300/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-h8300/thread_info.h
+++ linux-2.6/include/asm-h8300/thread_info.h
@@ -24,6 +24,7 @@ struct thread_info {
 	unsigned long	   flags;		/* low level flags */
 	int		   cpu;			/* cpu we're on */
 	int		   preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	struct restart_block restart_block;
 };
 
@@ -37,6 +38,7 @@ struct thread_info {
 	.flags =	0,			\
 	.cpu =		0,			\
 	.preempt_count = 1,			\
+	.migrate_count = 0,			\
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
 	},					\
@@ -92,6 +94,7 @@ static inline struct thread_info *curren
 #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
 					   TIF_NEED_RESCHED */
 #define TIF_MEMDIE		5
+#define TIF_NEED_MIGRATE	6	/* migration necessary */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -99,6 +102,7 @@ static inline struct thread_info *curren
 #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 
Index: linux-2.6/include/asm-ia64/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-ia64/thread_info.h
+++ linux-2.6/include/asm-ia64/thread_info.h
@@ -30,6 +30,7 @@ struct thread_info {
 	__u32 status;			/* Thread synchronous flags */
 	mm_segment_t addr_limit;	/* user-level address space limit */
 	int preempt_count;		/* 0=premptable, <0=BUG; will also serve as bh-counter */
+	int migrate_count;		/* 0: can migrate, <0 => BUG */
 	struct restart_block restart_block;
 };
 
@@ -43,6 +44,7 @@ struct thread_info {
 	.cpu		= 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.preempt_count	= 0,			\
+	.migrate_count = 0,			\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
 	},					\
@@ -86,6 +88,7 @@ struct thread_info {
 #define TIF_SYSCALL_AUDIT	4	/* syscall auditing active */
 #define TIF_SINGLESTEP		5	/* restore singlestep on return to user mode */
 #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
+#define TIF_NEED_MIGRATE	7	/* migration necessary */
 #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 #define TIF_MEMDIE		17
 #define TIF_MCA_INIT		18	/* this task is processing MCA or INIT */
@@ -99,6 +102,7 @@ struct thread_info {
 #define _TIF_SYSCALL_TRACEAUDIT	(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP)
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
Index: linux-2.6/include/asm-m32r/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-m32r/thread_info.h
+++ linux-2.6/include/asm-m32r/thread_info.h
@@ -29,6 +29,7 @@ struct thread_info {
 	unsigned long		status;		/* thread-synchronous flags */
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	/* thread address space:
 					 	   0-0xBFFFFFFF for user-thread
@@ -69,6 +70,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
@@ -151,6 +153,7 @@ static inline unsigned int get_thread_fa
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
 #define TIF_SINGLESTEP		4	/* restore singlestep on return to user mode */
 #define TIF_IRET		5	/* return with iret */
+#define TIF_NEED_MIGRATE	6	/* migration necessary */
 #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 					/* 31..28 fault code */
 #define TIF_MEMDIE		17
@@ -162,6 +165,7 @@ static inline unsigned int get_thread_fa
 #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
 #define _TIF_IRET		(1<<TIF_IRET)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
Index: linux-2.6/include/asm-m68k/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-m68k/thread_info.h
+++ linux-2.6/include/asm-m68k/thread_info.h
@@ -9,6 +9,7 @@ struct thread_info {
 	unsigned long		flags;
 	struct exec_domain	*exec_domain;	/* execution domain */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	__u32 cpu; /* should always be 0 on m68k */
 	struct restart_block    restart_block;
 };
@@ -58,5 +59,6 @@ struct thread_info {
 #define TIF_DELAYED_TRACE	14	/* single step a syscall */
 #define TIF_SYSCALL_TRACE	15	/* syscall trace active */
 #define TIF_MEMDIE		16
+#define TIF_NEED_MIGRATE	17	/* migration necessary */
 
 #endif	/* _ASM_M68K_THREAD_INFO_H */
Index: linux-2.6/include/asm-m68knommu/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-m68knommu/thread_info.h
+++ linux-2.6/include/asm-m68knommu/thread_info.h
@@ -37,6 +37,7 @@ struct thread_info {
 	unsigned long	   flags;		/* low level flags */
 	int		   cpu;			/* cpu we're on */
 	int		   preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int		   migrate_count;	/* 0: can migrate, <0 => BUG */
 	struct restart_block restart_block;
 };
 
@@ -89,6 +90,7 @@ static inline struct thread_info *curren
 #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
 					   TIF_NEED_RESCHED */
 #define TIF_MEMDIE		5
+#define TIF_NEED_MIGRATE	6	/* migration necessary */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -96,6 +98,7 @@ static inline struct thread_info *curren
 #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
 
Index: linux-2.6/include/asm-mips/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-mips/thread_info.h
+++ linux-2.6/include/asm-mips/thread_info.h
@@ -28,6 +28,7 @@ struct thread_info {
 	unsigned long		tp_value;	/* thread pointer */
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	/* thread address space:
 						   0-0xBFFFFFFF for user-thead
@@ -49,6 +50,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
@@ -115,6 +117,7 @@ register struct thread_info *__current_t
 #define TIF_SYSCALL_AUDIT	4	/* syscall auditing active */
 #define TIF_SECCOMP		5	/* secure computing */
 #define TIF_NEED_RESCHED_DELAYED 6	/* reschedule on return to userspace */
+#define TIF_NEED_MIGRATE	7	/* migration necessary */
 #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal() */
 #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
 #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling TIF_NEED_RESCHED */
@@ -129,6 +132,7 @@ register struct thread_info *__current_t
 #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
 #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
 #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
 #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
Index: linux-2.6/include/asm-parisc/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-parisc/thread_info.h
+++ linux-2.6/include/asm-parisc/thread_info.h
@@ -13,6 +13,7 @@ struct thread_info {
 	mm_segment_t addr_limit;	/* user-level address space limit */
 	__u32 cpu;			/* current CPU */
 	int preempt_count;		/* 0=premptable, <0=BUG; will also serve as bh-counter */
+	int migrate_count;		/* 0: can migrate, <0 => BUG */
 	struct restart_block restart_block;
 };
 
@@ -24,6 +25,7 @@ struct thread_info {
 	.cpu		= 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
   	.restart_block	= {			\
 		.fn = do_no_restart_syscall	\
 	}					\
@@ -63,6 +65,7 @@ struct thread_info {
 #define TIF_32BIT               5       /* 32 bit binary */
 #define TIF_MEMDIE		6
 #define TIF_RESTORE_SIGMASK	7	/* restore saved signal mask */
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
@@ -71,6 +74,7 @@ struct thread_info {
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
 #define _TIF_32BIT		(1 << TIF_32BIT)
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 
 #define _TIF_USER_WORK_MASK     (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
                                  _TIF_NEED_RESCHED | _TIF_RESTORE_SIGMASK)
Index: linux-2.6/include/asm-powerpc/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-powerpc/thread_info.h
+++ linux-2.6/include/asm-powerpc/thread_info.h
@@ -35,6 +35,7 @@ struct thread_info {
 	int		cpu;			/* cpu we're on */
 	int		preempt_count;		/* 0 => preemptable,
 						   <0 => BUG */
+	int		migrate_count;	/* 0: can migrate, <0 => BUG */
 	struct restart_block restart_block;
 	unsigned long	local_flags;		/* private flags for thread */
 
@@ -53,6 +54,7 @@ struct thread_info {
 	.exec_domain =	&default_exec_domain,	\
 	.cpu =		0,			\
 	.preempt_count = 1,			\
+	.migrate_count = 0,			\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
 	},					\
@@ -124,6 +126,7 @@ static inline struct thread_info *curren
 #define TIF_NOERROR		14	/* Force successful syscall return */
 #define TIF_RESTORE_SIGMASK	15	/* Restore signal mask in do_signal */
 #define TIF_FREEZE		16	/* Freezing for suspend */
+#define TIF_NEED_MIGRATE	17	/* migration necessary */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -143,6 +146,7 @@ static inline struct thread_info *curren
 #define _TIF_FREEZE		(1<<TIF_FREEZE)
 #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
 
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_SYSCALL_T_OR_A	(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP)
 
 #define _TIF_USER_WORK_MASK	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
Index: linux-2.6/include/asm-s390/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-s390/thread_info.h
+++ linux-2.6/include/asm-s390/thread_info.h
@@ -51,6 +51,7 @@ struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	unsigned int		cpu;		/* current CPU */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	struct restart_block	restart_block;
 };
 
@@ -64,6 +65,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
 	},					\
@@ -96,6 +98,7 @@ static inline struct thread_info *curren
 #define TIF_SYSCALL_AUDIT	5	/* syscall auditing active */
 #define TIF_SINGLE_STEP		6	/* deliver sigtrap on return to user */
 #define TIF_MCCK_PENDING	7	/* machine check handling is pending */
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
 #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling 
 					   TIF_NEED_RESCHED */
@@ -110,6 +113,7 @@ static inline struct thread_info *curren
 #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
 #define _TIF_SINGLE_STEP	(1<<TIF_SINGLE_STEP)
 #define _TIF_MCCK_PENDING	(1<<TIF_MCCK_PENDING)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 #define _TIF_31BIT		(1<<TIF_31BIT)
Index: linux-2.6/include/asm-sh/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-sh/thread_info.h
+++ linux-2.6/include/asm-sh/thread_info.h
@@ -21,6 +21,7 @@ struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	__u32			cpu;
 	int			preempt_count; /* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	mm_segment_t		addr_limit;	/* thread address space */
 	struct restart_block	restart_block;
 	unsigned long		previous_sp;	/* sp of previous stack in case
@@ -58,6 +59,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
@@ -113,6 +115,7 @@ static inline struct thread_info *curren
 #define TIF_RESTORE_SIGMASK	4	/* restore signal mask in do_signal() */
 #define TIF_SINGLESTEP		5	/* singlestepping active */
 #define TIF_NEED_RESCHED_DELAYED	6	/* reschedule on return to userspace */
+#define TIF_NEED_MIGRATE	7	/* migration necessary */
 #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
 #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 #define TIF_MEMDIE		18
@@ -125,6 +128,7 @@ static inline struct thread_info *curren
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
 #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
 #define _TIF_NEED_RESCHED_DELAYED	(1<<TIF_NEED_RESCHED_DELAYED)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 #define _TIF_FREEZE		(1<<TIF_FREEZE)
Index: linux-2.6/include/asm-sh64/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-sh64/thread_info.h
+++ linux-2.6/include/asm-sh64/thread_info.h
@@ -23,6 +23,7 @@ struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	/* Put the 4 32-bit fields together to make asm offsetting easier. */
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	__u16			cpu;
 
 	mm_segment_t		addr_limit;
@@ -41,6 +42,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit     = KERNEL_DS,            \
 	.restart_block	= {			\
 		.fn = do_no_restart_syscall,	\
@@ -79,12 +81,14 @@ static inline struct thread_info *curren
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
 #define TIF_MEMDIE		4
 #define TIF_RESTORE_SIGMASK	5	/* Restore signal mask in do_signal */
+#define TIF_NEED_MIGRATE	6	/* migration necessary */
 
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 
 #endif /* __KERNEL__ */
 
Index: linux-2.6/include/asm-sparc/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-sparc/thread_info.h
+++ linux-2.6/include/asm-sparc/thread_info.h
@@ -33,6 +33,7 @@ struct thread_info {
 	int			cpu;		/* cpu we're on */
 	int			preempt_count;	/* 0 => preemptable,
 						   <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	int			softirq_count;
 	int			hardirq_count;
 
@@ -65,6 +66,7 @@ struct thread_info {
 	.flags		=	0,			\
 	.cpu		=	0,			\
 	.preempt_count	=	1,			\
+	.migrate_count =	0,			\
 	.restart_block	= {				\
 		.fn	=	do_no_restart_syscall,	\
 	},						\
@@ -132,6 +134,7 @@ BTFIXUPDEF_CALL(void, free_thread_info, 
 #define TIF_SIGPENDING		2	/* signal pending */
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
 #define TIF_RESTORE_SIGMASK	4	/* restore signal mask in do_signal() */
+#define TIF_NEED_MIGRATE	5	/* migration necessary */
 #define TIF_USEDFPU		8	/* FPU was used by this task
 					 * this quantum (SMP) */
 #define TIF_POLLING_NRFLAG	9	/* true if poll_idle() is polling
@@ -143,6 +146,7 @@ BTFIXUPDEF_CALL(void, free_thread_info, 
 #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 
Index: linux-2.6/include/asm-sparc64/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-sparc64/thread_info.h
+++ linux-2.6/include/asm-sparc64/thread_info.h
@@ -47,6 +47,7 @@ struct thread_info {
 	struct pt_regs		*kregs;
 	struct exec_domain	*exec_domain;
 	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	__u8			new_child;
 	__u8			syscall_noerror;
 	__u16			cpu;
@@ -137,6 +138,7 @@ struct thread_info {
 	.flags		= ((unsigned long)ASI_P) << TI_FLAG_CURRENT_DS_SHIFT,	\
 	.exec_domain	=	&default_exec_domain,	\
 	.preempt_count	=	1,			\
+	.migrate_count =	0,			\
 	.restart_block	= {				\
 		.fn	=	do_no_restart_syscall,	\
 	},						\
@@ -225,7 +227,7 @@ register struct thread_info *current_thr
 #define TIF_UNALIGNED		5	/* allowed to do unaligned accesses */
 #define TIF_NEWSIGNALS		6	/* wants new-style signals */
 #define TIF_32BIT		7	/* 32-bit binary */
-/* flag bit 8 is available */
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 #define TIF_SECCOMP		9	/* secure computing */
 #define TIF_SYSCALL_AUDIT	10	/* syscall auditing active */
 /* flag bit 11 is available */
@@ -244,6 +246,7 @@ register struct thread_info *current_thr
 #define _TIF_UNALIGNED		(1<<TIF_UNALIGNED)
 #define _TIF_NEWSIGNALS		(1<<TIF_NEWSIGNALS)
 #define _TIF_32BIT		(1<<TIF_32BIT)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
 #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
Index: linux-2.6/include/asm-um/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-um/thread_info.h
+++ linux-2.6/include/asm-um/thread_info.h
@@ -18,6 +18,7 @@ struct thread_info {
 	__u32			cpu;		/* current CPU */
 	int			preempt_count;  /* 0 => preemptable,
 						   <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	mm_segment_t		addr_limit;	/* thread address space:
 					 	   0-0xBFFFFFFF for user
 						   0-0xFFFFFFFF for kernel */
@@ -32,6 +33,7 @@ struct thread_info {
 	.flags =		0,		\
 	.cpu =		0,			\
 	.preempt_count =	1,		\
+	.migrate_count =	0,		\
 	.addr_limit =	KERNEL_DS,		\
 	.restart_block =  {			\
 		.fn =  do_no_restart_syscall,	\
@@ -71,6 +73,7 @@ static inline struct thread_info *curren
 #define TIF_MEMDIE	 	5
 #define TIF_SYSCALL_AUDIT	6
 #define TIF_RESTORE_SIGMASK	7
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 
 #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
@@ -79,5 +82,6 @@ static inline struct thread_info *curren
 #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
 #define _TIF_SYSCALL_AUDIT	(1 << TIF_SYSCALL_AUDIT)
 #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
+#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
 
 #endif
Index: linux-2.6/include/asm-v850/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-v850/thread_info.h
+++ linux-2.6/include/asm-v850/thread_info.h
@@ -32,6 +32,7 @@ struct thread_info {
 	int			cpu;		/* cpu we're on */
 	int			preempt_count;	/* 0 => preemptable,
 						   <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 	struct restart_block	restart_block;
 };
 
@@ -42,6 +43,7 @@ struct thread_info {
 	.flags =	0,						      \
 	.cpu =		0,						      \
 	.preempt_count = 1,						      \
+	.migrate_count = 0,						      \
 	.restart_block = {						      \
 		.fn = do_no_restart_syscall,				      \
 	},								      \
@@ -83,6 +85,7 @@ struct thread_info {
 #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
 					   TIF_NEED_RESCHED */
 #define TIF_MEMDIE		5
+#define TIF_NEED_MIGRATE	6	/* migration necessary */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -90,6 +93,7 @@ struct thread_info {
 #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 
 
 /* Size of kernel stack for each process.  */
Index: linux-2.6/include/asm-x86_64/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-x86_64/thread_info.h
+++ linux-2.6/include/asm-x86_64/thread_info.h
@@ -30,6 +30,7 @@ struct thread_info {
 	__u32			status;		/* thread synchronous flags */
 	__u32			cpu;		/* current CPU */
 	int 			preempt_count;	/* 0 => preemptable, <0 => BUG */
+	int			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	
 	struct restart_block    restart_block;
@@ -48,6 +49,7 @@ struct thread_info {
 	.flags	       = 0,			\
 	.cpu	       = 0,			\
 	.preempt_count = 1,			\
+	.migrate_count = 0,			\
 	.addr_limit     = KERNEL_DS,		\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
@@ -116,6 +118,7 @@ static inline struct thread_info *stack_
 #define TIF_SYSCALL_AUDIT	7	/* syscall auditing active */
 #define TIF_SECCOMP		8	/* secure computing */
 #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal */
+#define TIF_NEED_MIGRATE	11	/* migration necessary */
 /* 16 free */
 #define TIF_IA32		17	/* 32bit process */ 
 #define TIF_FORK		18	/* ret_from_fork */
@@ -135,6 +138,7 @@ static inline struct thread_info *stack_
 #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
 #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_IA32		(1<<TIF_IA32)
 #define _TIF_FORK		(1<<TIF_FORK)
 #define _TIF_ABI_PENDING	(1<<TIF_ABI_PENDING)
Index: linux-2.6/include/asm-xtensa/thread_info.h
===================================================================
--- linux-2.6.orig/include/asm-xtensa/thread_info.h
+++ linux-2.6/include/asm-xtensa/thread_info.h
@@ -34,6 +34,7 @@ struct thread_info {
 	unsigned long		status;		/* thread-synchronous flags */
 	__u32			cpu;		/* current CPU */
 	__s32			preempt_count;	/* 0 => preemptable,< 0 => BUG*/
+	__s32			migrate_count;/* 0: can migrate, <0 => BUG */
 
 	mm_segment_t		addr_limit;	/* thread address space */
 	struct restart_block    restart_block;
@@ -72,6 +73,7 @@ struct thread_info {
 	.flags		= 0,			\
 	.cpu		= 0,			\
 	.preempt_count	= 1,			\
+	.migrate_count = 0,			\
 	.addr_limit	= KERNEL_DS,		\
 	.restart_block = {			\
 		.fn = do_no_restart_syscall,	\
@@ -117,6 +119,7 @@ static inline struct thread_info *curren
 #define TIF_IRET		5	/* return with iret */
 #define TIF_MEMDIE		6
 #define TIF_RESTORE_SIGMASK	7	/* restore signal mask in do_signal() */
+#define TIF_NEED_MIGRATE	8	/* migration necessary */
 #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
 
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -125,6 +128,7 @@ static inline struct thread_info *curren
 #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
 #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
 #define _TIF_IRET		(1<<TIF_IRET)
+#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
 #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
 #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
 
Index: linux-2.6/lib/smp_processor_id.c
===================================================================
--- linux-2.6.orig/lib/smp_processor_id.c
+++ linux-2.6/lib/smp_processor_id.c
@@ -10,12 +10,16 @@
 unsigned int notrace debug_smp_processor_id(void)
 {
 	unsigned long preempt_count = preempt_count();
+	unsigned long migrate_count = migrate_count();
 	int this_cpu = raw_smp_processor_id();
 	cpumask_t this_mask;
 
 	if (likely(preempt_count))
 		goto out;
 
+	if (likely(migrate_count))
+		goto out;
+
 	if (irqs_disabled())
 		goto out;
 
@@ -42,7 +46,7 @@ unsigned int notrace debug_smp_processor
 	if (!printk_ratelimit())
 		goto out_enable;
 
-	printk(KERN_ERR "BUG: using smp_processor_id() in preemptible [%08x] code: %s/%d\n", preempt_count()-1, current->comm, current->pid);
+	printk(KERN_ERR "BUG: using smp_processor_id() in preemptible [%08x] migration [%08x] code: %s/%d\n", preempt_count()-1, migrate_count(), current->comm, current->pid);
 	print_symbol("caller is %s\n", (long)__builtin_return_address(0));
 	dump_stack();
 

--

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Mathieu Desnoyers]

Oh, testers for this patch will be welcome.. it runs fine on my system,
but I have not tested the CPU hotplug corner cases.

Mathieu

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> 
> This patch adds the ability to protect critical sections from migration to
> another CPU without disabling preemption.
> 
> This will be useful to minimize the amount of preemption disabling for the -rt
> patch. It will help leveraging improvements brought by the local_t types in
> asm/local.h (see Documentation/local_ops.txt). Note that the updates done to
> variables protected by migrate_disable must be either atomic or protected from
> concurrent updates done by other threads.
> 
> Typical use:
> 
> migrate_disable();
> local_inc(&__get_cpu_var(&my_local_t_var));
> migrate_enable();
> 
> Which will increment the variable atomically wrt the local CPU.
> 
> Changes:
> 
> Do a check_migrate() upon migrate_enable() to answer to the migration
> thread waiting for us to exit the migration disabled critical section. Use a
> NEED_MIGRATE thread flag for this.
> 
> Note: (or we could say FIXME)
> Is we ever want to check migration pending in assembly code, we will have to
> make sure we test the right thread flag bits on each architectures. Care should
> also be taken to check that the thread flags used won't trigger false positives
> in non selective asm thread flag checks.
> 
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
>  include/asm-alpha/thread_info.h     |    3 +
>  include/asm-arm/thread_info.h       |    5 +
>  include/asm-arm26/thread_info.h     |    5 +
>  include/asm-avr32/thread_info.h     |    4 +
>  include/asm-blackfin/thread_info.h  |    4 +
>  include/asm-cris/thread_info.h      |    4 +
>  include/asm-frv/thread_info.h       |    4 +
>  include/asm-h8300/thread_info.h     |    4 +
>  include/asm-i386/thread_info.h      |    6 +-
>  include/asm-ia64/thread_info.h      |    4 +
>  include/asm-m32r/thread_info.h      |    4 +
>  include/asm-m68k/thread_info.h      |    2 
>  include/asm-m68knommu/thread_info.h |    3 +
>  include/asm-mips/thread_info.h      |    4 +
>  include/asm-parisc/thread_info.h    |    4 +
>  include/asm-powerpc/thread_info.h   |    4 +
>  include/asm-s390/thread_info.h      |    4 +
>  include/asm-sh/thread_info.h        |    4 +
>  include/asm-sh64/thread_info.h      |    4 +
>  include/asm-sparc/thread_info.h     |    4 +
>  include/asm-sparc64/thread_info.h   |    5 +
>  include/asm-um/thread_info.h        |    4 +
>  include/asm-v850/thread_info.h      |    4 +
>  include/asm-x86_64/thread_info.h    |    4 +
>  include/asm-xtensa/thread_info.h    |    4 +
>  include/linux/preempt.h             |   44 +++++++++++++++++
>  kernel/sched.c                      |   91 +++++++++++++++++++++++++++++++++---
>  lib/smp_processor_id.c              |    6 +-
>  28 files changed, 230 insertions(+), 12 deletions(-)
> 
> Index: linux-2.6/include/asm-i386/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-i386/thread_info.h
> +++ linux-2.6/include/asm-i386/thread_info.h
> @@ -31,8 +31,7 @@ struct thread_info {
>  	unsigned long		status;		/* thread-synchronous flags */
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> -
> -
> +	int			migrate_count;/* 0: can migrate, <0: BUG */
>  	mm_segment_t		addr_limit;	/* thread address space:
>  					 	   0-0xBFFFFFFF for user-thead
>  						   0-0xFFFFFFFF for kernel-thread
> @@ -74,6 +73,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -134,6 +134,7 @@ static inline struct thread_info *curren
>  #define TIF_SECCOMP		8	/* secure computing */
>  #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal() */
>  #define TIF_NEED_RESCHED_DELAYED 10	/* reschedule on return to userspace */
> +#define TIF_NEED_MIGRATE	11	/* migration necessary */
>  #define TIF_MEMDIE		16
>  #define TIF_DEBUG		17	/* uses debug registers */
>  #define TIF_IO_BITMAP		18	/* uses I/O bitmap */
> @@ -151,6 +152,7 @@ static inline struct thread_info *curren
>  #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
>  #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_DEBUG		(1<<TIF_DEBUG)
>  #define _TIF_IO_BITMAP		(1<<TIF_IO_BITMAP)
>  #define _TIF_FREEZE		(1<<TIF_FREEZE)
> Index: linux-2.6/include/linux/preempt.h
> ===================================================================
> --- linux-2.6.orig/include/linux/preempt.h
> +++ linux-2.6/include/linux/preempt.h
> @@ -15,6 +15,8 @@
>    extern void notrace sub_preempt_count(unsigned int val);
>    extern void notrace mask_preempt_count(unsigned int mask);
>    extern void notrace unmask_preempt_count(unsigned int mask);
> +  extern void fastcall add_migrate_count(int val);
> +  extern void fastcall sub_migrate_count(int val);
>  #else
>  # define add_preempt_count(val)	do { preempt_count() += (val); } while (0)
>  # define sub_preempt_count(val)	do { preempt_count() -= (val); } while (0)
> @@ -22,6 +24,8 @@
>  		do { preempt_count() |= (mask); } while (0)
>  # define unmask_preempt_count(mask) \
>  		do { preempt_count() &= ~(mask); } while (0)
> +# define add_migrate_count(val) do { migrate_count() += (val); } while (0)
> +# define sub_migrate_count(val) do { migrate_count() -= (val); } while (0)
>  #endif
>  
>  #ifdef CONFIG_CRITICAL_TIMING
> @@ -35,7 +39,12 @@
>  #define inc_preempt_count() add_preempt_count(1)
>  #define dec_preempt_count() sub_preempt_count(1)
>  
> -#define preempt_count()		(current_thread_info()->preempt_count)
> +#define preempt_count() (current_thread_info()->preempt_count)
> +
> +#define inc_migrate_count() add_migrate_count(1)
> +#define dec_migrate_count() sub_migrate_count(1)
> +
> +#define migrate_count() (current_thread_info()->migrate_count)
>  
>  #ifdef CONFIG_PREEMPT
>  
> @@ -80,6 +89,36 @@ do { \
>  	preempt_check_resched(); \
>  } while (0)
>  
> +#define migrate_disable() \
> +do { \
> +	inc_migrate_count(); \
> +	barrier(); \
> +} while (0)
> +
> +#define migrate_enable_no_check() \
> +do { \
> +	barrier(); \
> +	dec_migrate_count(); \
> +} while (0)
> +
> +#ifdef CONFIG_SMP
> +extern void do_check_migrate(void);
> +#define check_migrate() \
> +do { \
> +	if (unlikely(test_thread_flag(TIF_NEED_MIGRATE))) \
> +		do_check_migrate(); \
> +} while (0)
> +#else
> +#define check_migrate()
> +#endif
> +
> +#define migrate_enable() \
> +do { \
> +	migrate_enable_no_check(); \
> +	barrier(); \
> +	check_migrate(); \
> +} while (0)
> +
>  #else
>  
>  #define preempt_disable()		do { } while (0)
> @@ -91,6 +130,9 @@ do { \
>  
>  #define preempt_schedule_irq()		do { } while (0)
>  
> +#define migrate_disable()		do { } while (0)
> +#define migrate_enable()		do { } while (0)
> +
>  #endif
>  
>  #endif /* __LINUX_PREEMPT_H */
> Index: linux-2.6/kernel/sched.c
> ===================================================================
> --- linux-2.6.orig/kernel/sched.c
> +++ linux-2.6/kernel/sched.c
> @@ -1175,7 +1175,8 @@ migrate_task(struct task_struct *p, int 
>  	 * If the task is not on a runqueue (and not running), then
>  	 * it is sufficient to simply update the task's cpu field.
>  	 */
> -	if (!p->se.on_rq && !task_running(rq, p)) {
> +	if (!p->se.on_rq && !task_running(rq, p)
> +			&& !task_thread_info(p)->migrate_count) {
>  		set_task_cpu(p, dest_cpu);
>  		return 0;
>  	}
> @@ -1506,6 +1507,29 @@ static void balance_rt_tasks(struct rq *
>  
>  #endif
>  
> +void fastcall add_migrate_count(int val)
> +{
> +	/*
> +	 * Underflow?
> +	 */
> +	if (DEBUG_LOCKS_WARN_ON((migrate_count() < 0)))
> +		return;
> +	migrate_count() += val;
> +}
> +EXPORT_SYMBOL(add_migrate_count);
> +
> +void fastcall sub_migrate_count(int val)
> +{
> +	/*
> +	 * Underflow?
> +	 */
> +	if (DEBUG_LOCKS_WARN_ON(val > migrate_count()))
> +		return;
> +
> +	migrate_count() -= val;
> +}
> +EXPORT_SYMBOL(sub_migrate_count);
> +
>  /*
>   * find_idlest_cpu - find the idlest cpu among the cpus in group.
>   */
> @@ -1696,7 +1720,10 @@ try_to_wake_up(struct task_struct *p, un
>  #ifdef CONFIG_SMP
>  	if (unlikely(task_running(rq, p)))
>  		goto out_activate;
> -
> +#ifdef CONFIG_PREEMPT
> +	if (task_thread_info(p)->migrate_count)
> +		goto out_activate;
> +#endif
>  	new_cpu = cpu;
>  
>  	schedstat_inc(rq, ttwu_cnt);
> @@ -1970,6 +1997,7 @@ void sched_fork(struct task_struct *p, i
>  #ifdef CONFIG_PREEMPT
>  	/* Want to start with kernel preemption disabled. */
>  	task_thread_info(p)->preempt_count = 1;
> +	task_thread_info(p)->migrate_count = 0;
>  #endif
>  	put_cpu();
>  }
> @@ -2435,6 +2463,10 @@ static void sched_migrate_task(struct ta
>  	if (!cpu_isset(dest_cpu, p->cpus_allowed)
>  	    || unlikely(cpu_is_offline(dest_cpu)))
>  		goto out;
> +#ifdef CONFIG_PREEMPT
> +	if (task_thread_info(p)->migrate_count)
> +		goto out;
> +#endif
>  
>  	/* force the process onto the specified CPU */
>  	if (migrate_task(p, dest_cpu, &req)) {
> @@ -2496,6 +2528,7 @@ int can_migrate_task(struct task_struct 
>  	 * 1) running (obviously), or
>  	 * 2) cannot be migrated to this CPU due to cpus_allowed, or
>  	 * 3) are cache-hot on their current CPU.
> +	 * 4) migration preemption is non 0 for this non running task.
>  	 */
>  	if (!cpu_isset(this_cpu, p->cpus_allowed))
>  		return 0;
> @@ -2503,6 +2536,10 @@ int can_migrate_task(struct task_struct 
>  
>  	if (task_running(rq, p))
>  		return 0;
> +#ifdef CONFIG_PREEMPT
> +	if (task_thread_info(p)->migrate_count)
> +		return 0;
> +#endif
>  
>  	/*
>  	 * Aggressive migration if too many balance attempts have failed:
> @@ -3822,6 +3859,27 @@ EXPORT_SYMBOL(schedule);
>  
>  #ifdef CONFIG_PREEMPT
>  
> +#ifdef CONFIG_SMP
> +/*
> + * Wake up the migration thread to deal with our pending migration.
> + * If we have been moved since, we will just wake up the migration thread from
> + * the wrong CPU, which does not hurt anyone. (that's why we use
> + * raw_smp_processor_id()).
> + */
> +void __sched do_check_migrate(void)
> +{
> +	struct rq *rq;
> +
> +	if (migrate_count())
> +		return;
> +
> +	clear_thread_flag(TIF_NEED_MIGRATE);
> +	rq = cpu_rq(raw_smp_processor_id());
> +	wake_up_process(rq->migration_thread);
> +}
> +EXPORT_SYMBOL(do_check_migrate);
> +#endif
> +
>  /*
>   * Global flag to turn preemption off on a CONFIG_PREEMPT kernel:
>   */
> @@ -5478,7 +5536,7 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed);
>   * So we race with normal scheduler movements, but that's OK, as long
>   * as the task is no longer on this CPU.
>   *
> - * Returns non-zero if task was successfully migrated.
> + * Returns non-zero if task is on dest_cpu when this function ends.
>   */
>  static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)
>  {
> @@ -5500,13 +5558,19 @@ static int __migrate_task(struct task_st
>  
>  	double_rq_lock(rq_src, rq_dest);
>  	/* Already moved. */
> -	if (task_cpu(p) != src_cpu)
> +	if (task_cpu(p) != src_cpu) {
> +		ret = 1;
>  		goto out;
> +	}
>  	/* Affinity changed (again). */
>  	if (!cpu_isset(dest_cpu, p->cpus_allowed))
>  		goto out;
>  
>  	on_rq = p->se.on_rq;
> +#ifdef CONFIG_PREEMPT
> +	if (!on_rq && task_thread_info(p)->migrate_count)
> +		goto out;
> +#endif
>  	if (on_rq)
>  		deactivate_task(rq_src, p, 0);
>  
> @@ -5532,6 +5596,7 @@ static int migration_thread(void *data)
>  {
>  	int cpu = (long)data;
>  	struct rq *rq;
> +	int migrated;
>  
>  	rq = cpu_rq(cpu);
>  	BUG_ON(rq->migration_thread != current);
> @@ -5567,10 +5632,22 @@ static int migration_thread(void *data)
>  		list_del_init(head->next);
>  
>  		spin_unlock(&rq->lock);
> -		__migrate_task(req->task, cpu, req->dest_cpu);
> +		migrated = __migrate_task(req->task, cpu, req->dest_cpu);
>  		local_irq_enable();
> -
> -		complete(&req->done);
> +		if (!migrated) {
> +			/*
> +			 * If the process has not been migrated, let it run
> +			 * until it reaches a migration_check() so it can
> +			 * wake us up.
> +			 */
> +			spin_lock_irq(&rq->lock);
> +			head = &rq->migration_queue;
> +			list_add(&req->list, head);
> +			set_tsk_thread_flag(req->task, TIF_NEED_MIGRATE);
> +			spin_unlock_irq(&rq->lock);
> +			wake_up_process(req->task);
> +		} else
> +			complete(&req->done);
>  	}
>  	__set_current_state(TASK_RUNNING);
>  	return 0;
> Index: linux-2.6/include/asm-alpha/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-alpha/thread_info.h
> +++ linux-2.6/include/asm-alpha/thread_info.h
> @@ -21,6 +21,7 @@ struct thread_info {
>  	mm_segment_t		addr_limit;	/* thread address space */
>  	unsigned		cpu;		/* current CPU */
>  	int			preempt_count; /* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	int bpt_nsaved;
>  	unsigned long bpt_addr[2];		/* breakpoint handling  */
> @@ -77,6 +78,7 @@ register struct thread_info *__current_t
>  #define TIF_UAC_SIGBUS		8
>  #define TIF_MEMDIE		9
>  #define TIF_RESTORE_SIGMASK	10	/* restore signal mask in do_signal */
> +#define TIF_NEED_MIGRATE	11	/* migration necessary */
>  
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
>  #define _TIF_NOTIFY_RESUME	(1<<TIF_NOTIFY_RESUME)
> @@ -84,6 +86,7 @@ register struct thread_info *__current_t
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  /* Work to do on interrupt/exception return.  */
>  #define _TIF_WORK_MASK		(_TIF_NOTIFY_RESUME	\
> Index: linux-2.6/include/asm-arm/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-arm/thread_info.h
> +++ linux-2.6/include/asm-arm/thread_info.h
> @@ -51,6 +51,7 @@ struct cpu_context_save {
>  struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	int			preempt_count;	/* 0 => preemptable, <0 => bug */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	mm_segment_t		addr_limit;	/* address limit */
>  	struct task_struct	*task;		/* main task structure */
>  	struct exec_domain	*exec_domain;	/* execution domain */
> @@ -72,6 +73,7 @@ struct thread_info {
>  	.exec_domain	= &default_exec_domain,				\
>  	.flags		= 0,						\
>  	.preempt_count	= 1,						\
> +	.migrate_count = 0,						\
>  	.addr_limit	= KERNEL_DS,					\
>  	.cpu_domain	= domain_val(DOMAIN_USER, DOMAIN_MANAGER) |	\
>  			  domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) |	\
> @@ -138,6 +140,7 @@ extern void iwmmxt_task_switch(struct th
>   *  TIF_NOTIFY_RESUME	- resumption notification requested
>   *  TIF_SIGPENDING	- signal pending
>   *  TIF_NEED_RESCHED	- rescheduling necessary
> + *  TIF_NEED_MIGRATE	- migration necessary
>   *  TIF_USEDFPU		- FPU was used by this task this quantum (SMP)
>   *  TIF_POLLING_NRFLAG	- true if poll_idle() is polling TIF_NEED_RESCHED
>   */
> @@ -145,6 +148,7 @@ extern void iwmmxt_task_switch(struct th
>  #define TIF_SIGPENDING		1
>  #define TIF_NEED_RESCHED	2
>  #define TIF_NEED_RESCHED_DELAYED 3
> +#define TIF_NEED_MIGRATE	4
>  #define TIF_SYSCALL_TRACE	8
>  #define TIF_POLLING_NRFLAG	16
>  #define TIF_USING_IWMMXT	17
> @@ -155,6 +159,7 @@ extern void iwmmxt_task_switch(struct th
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
>  #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
>  #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
>  #define _TIF_USING_IWMMXT	(1 << TIF_USING_IWMMXT)
> Index: linux-2.6/include/asm-arm26/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-arm26/thread_info.h
> +++ linux-2.6/include/asm-arm26/thread_info.h
> @@ -45,6 +45,7 @@ struct cpu_context_save {
>  struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	int			preempt_count;	/* 0 => preemptable, <0 => bug */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	mm_segment_t		addr_limit;	/* address limit */
>  	struct task_struct	*task;		/* main task structure */
>  	struct exec_domain      *exec_domain;   /* execution domain */
> @@ -60,6 +61,7 @@ struct thread_info {
>  	.exec_domain	&default_exec_domain,	\
>  	.flags		0,			\
>  	.preempt_count	0,			\
> +	.migrate_count 0,			\
>  	.addr_limit	KERNEL_DS,		\
>  	.restart_block  = {                             \
>  		.fn     = do_no_restart_syscall,        \
> @@ -113,12 +115,14 @@ extern void free_thread_info(struct thre
>   *  TIF_NOTIFY_RESUME	- resumption notification requested
>   *  TIF_SIGPENDING	- signal pending
>   *  TIF_NEED_RESCHED	- rescheduling necessary
> + *  TIF_NEED_MIGRATE	- migration necessary
>   *  TIF_USEDFPU		- FPU was used by this task this quantum (SMP)
>   *  TIF_POLLING_NRFLAG	- true if poll_idle() is polling TIF_NEED_RESCHED
>   */
>  #define TIF_NOTIFY_RESUME	0
>  #define TIF_SIGPENDING		1
>  #define TIF_NEED_RESCHED	2
> +#define TIF_NEED_MIGRATE	3
>  #define TIF_SYSCALL_TRACE	8
>  #define TIF_USED_FPU		16
>  #define TIF_POLLING_NRFLAG	17
> @@ -127,6 +131,7 @@ extern void free_thread_info(struct thre
>  #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
>  #define _TIF_USED_FPU		(1 << TIF_USED_FPU)
>  #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
> Index: linux-2.6/include/asm-avr32/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-avr32/thread_info.h
> +++ linux-2.6/include/asm-avr32/thread_info.h
> @@ -25,6 +25,7 @@ struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	__u32			cpu;
>  	__s32			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	struct restart_block	restart_block;
>  	__u8			supervisor_stack[0];
>  };
> @@ -36,6 +37,7 @@ struct thread_info {
>  	.flags		= 0,						\
>  	.cpu		= 0,						\
>  	.preempt_count	= 1,						\
> +	.migrate_count = 0,						\
>  	.restart_block	= {						\
>  		.fn	= do_no_restart_syscall				\
>  	}								\
> @@ -84,6 +86,7 @@ static inline struct thread_info *curren
>  #define TIF_MEMDIE		7
>  #define TIF_RESTORE_SIGMASK	8	/* restore signal mask in do_signal */
>  #define TIF_CPU_GOING_TO_SLEEP	9	/* CPU is entering sleep 0 mode */
> +#define TIF_NEED_MIGRATE      10       /* migration necessary */
>  #define TIF_USERSPACE		31      /* true if FS sets userspace */
>  
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
> @@ -96,6 +99,7 @@ static inline struct thread_info *curren
>  #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
>  #define _TIF_CPU_GOING_TO_SLEEP (1 << TIF_CPU_GOING_TO_SLEEP)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  
>  /* XXX: These two masks must never span more than 16 bits! */
>  /* work to do on interrupt/exception return */
> Index: linux-2.6/include/asm-blackfin/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-blackfin/thread_info.h
> +++ linux-2.6/include/asm-blackfin/thread_info.h
> @@ -54,6 +54,7 @@ struct thread_info {
>  	unsigned long flags;	/* low level flags */
>  	int cpu;		/* cpu we're on */
>  	int preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int migrate_count;	/* 0: can migrate, <0 => BUG */
>  	mm_segment_t addr_limit;	/* address limit */
>  	struct restart_block restart_block;
>  	struct l1_scratch_task_info l1_task_info;
> @@ -69,6 +70,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count  = 1,                    \
> +	.migrate_count = 0,			\
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
>  	},					\
> @@ -126,6 +128,7 @@ static inline struct thread_info *curren
>  #define TIF_MEMDIE              5
>  #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
>  #define TIF_FREEZE              7       /* is freezing for suspend */
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  
>  /* as above, but as bit values */
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -135,6 +138,7 @@ static inline struct thread_info *curren
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
>  #define _TIF_FREEZE             (1<<TIF_FREEZE)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  
> Index: linux-2.6/include/asm-cris/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-cris/thread_info.h
> +++ linux-2.6/include/asm-cris/thread_info.h
> @@ -32,6 +32,7 @@ struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	/* thread address space:
>  					 	   0-0xBFFFFFFF for user-thead
> @@ -58,6 +59,7 @@ struct thread_info {
>  	.flags		= 0,				\
>  	.cpu		= 0,				\
>  	.preempt_count	= 1,				\
> +	.migrate_count = 0,				\
>  	.addr_limit	= KERNEL_DS,			\
>  	.restart_block = {				\
>  		       .fn = do_no_restart_syscall,	\
> @@ -82,6 +84,7 @@ struct thread_info {
>  #define TIF_NOTIFY_RESUME	1	/* resumption notification requested */
>  #define TIF_SIGPENDING		2	/* signal pending */
>  #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
> +#define TIF_NEED_MIGRATE	4	/* migration necessary */
>  #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		17
>  
> @@ -90,6 +93,7 @@ struct thread_info {
>  #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
> Index: linux-2.6/include/asm-frv/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-frv/thread_info.h
> +++ linux-2.6/include/asm-frv/thread_info.h
> @@ -36,6 +36,7 @@ struct thread_info {
>  	unsigned long		status;		/* thread-synchronous flags */
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	/* thread address space:
>  					 	   0-0xBFFFFFFF for user-thead
> @@ -68,6 +69,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -114,6 +116,7 @@ register struct thread_info *__current_t
>  #define TIF_SINGLESTEP		4	/* restore singlestep on return to user mode */
>  #define TIF_IRET		5	/* return with iret */
>  #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
> +#define TIF_NEED_MIGRATE	7	/* migration necessary */
>  #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		17	/* OOM killer killed process */
>  #define TIF_FREEZE		18	/* freezing for suspend */
> @@ -127,6 +130,7 @@ register struct thread_info *__current_t
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
>  #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
>  #define _TIF_FREEZE		(1 << TIF_FREEZE)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
> Index: linux-2.6/include/asm-h8300/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-h8300/thread_info.h
> +++ linux-2.6/include/asm-h8300/thread_info.h
> @@ -24,6 +24,7 @@ struct thread_info {
>  	unsigned long	   flags;		/* low level flags */
>  	int		   cpu;			/* cpu we're on */
>  	int		   preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	struct restart_block restart_block;
>  };
>  
> @@ -37,6 +38,7 @@ struct thread_info {
>  	.flags =	0,			\
>  	.cpu =		0,			\
>  	.preempt_count = 1,			\
> +	.migrate_count = 0,			\
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
>  	},					\
> @@ -92,6 +94,7 @@ static inline struct thread_info *curren
>  #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
>  					   TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		5
> +#define TIF_NEED_MIGRATE	6	/* migration necessary */
>  
>  /* as above, but as bit values */
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -99,6 +102,7 @@ static inline struct thread_info *curren
>  #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  
> Index: linux-2.6/include/asm-ia64/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-ia64/thread_info.h
> +++ linux-2.6/include/asm-ia64/thread_info.h
> @@ -30,6 +30,7 @@ struct thread_info {
>  	__u32 status;			/* Thread synchronous flags */
>  	mm_segment_t addr_limit;	/* user-level address space limit */
>  	int preempt_count;		/* 0=premptable, <0=BUG; will also serve as bh-counter */
> +	int migrate_count;		/* 0: can migrate, <0 => BUG */
>  	struct restart_block restart_block;
>  };
>  
> @@ -43,6 +44,7 @@ struct thread_info {
>  	.cpu		= 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.preempt_count	= 0,			\
> +	.migrate_count = 0,			\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
>  	},					\
> @@ -86,6 +88,7 @@ struct thread_info {
>  #define TIF_SYSCALL_AUDIT	4	/* syscall auditing active */
>  #define TIF_SINGLESTEP		5	/* restore singlestep on return to user mode */
>  #define TIF_RESTORE_SIGMASK	6	/* restore signal mask in do_signal() */
> +#define TIF_NEED_MIGRATE	7	/* migration necessary */
>  #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		17
>  #define TIF_MCA_INIT		18	/* this task is processing MCA or INIT */
> @@ -99,6 +102,7 @@ struct thread_info {
>  #define _TIF_SYSCALL_TRACEAUDIT	(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP)
>  #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
> Index: linux-2.6/include/asm-m32r/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-m32r/thread_info.h
> +++ linux-2.6/include/asm-m32r/thread_info.h
> @@ -29,6 +29,7 @@ struct thread_info {
>  	unsigned long		status;		/* thread-synchronous flags */
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	/* thread address space:
>  					 	   0-0xBFFFFFFF for user-thread
> @@ -69,6 +70,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -151,6 +153,7 @@ static inline unsigned int get_thread_fa
>  #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
>  #define TIF_SINGLESTEP		4	/* restore singlestep on return to user mode */
>  #define TIF_IRET		5	/* return with iret */
> +#define TIF_NEED_MIGRATE	6	/* migration necessary */
>  #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  					/* 31..28 fault code */
>  #define TIF_MEMDIE		17
> @@ -162,6 +165,7 @@ static inline unsigned int get_thread_fa
>  #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
>  #define _TIF_IRET		(1<<TIF_IRET)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  #define _TIF_ALLWORK_MASK	0x0000FFFF	/* work to do on any return to u-space */
> Index: linux-2.6/include/asm-m68k/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-m68k/thread_info.h
> +++ linux-2.6/include/asm-m68k/thread_info.h
> @@ -9,6 +9,7 @@ struct thread_info {
>  	unsigned long		flags;
>  	struct exec_domain	*exec_domain;	/* execution domain */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	__u32 cpu; /* should always be 0 on m68k */
>  	struct restart_block    restart_block;
>  };
> @@ -58,5 +59,6 @@ struct thread_info {
>  #define TIF_DELAYED_TRACE	14	/* single step a syscall */
>  #define TIF_SYSCALL_TRACE	15	/* syscall trace active */
>  #define TIF_MEMDIE		16
> +#define TIF_NEED_MIGRATE	17	/* migration necessary */
>  
>  #endif	/* _ASM_M68K_THREAD_INFO_H */
> Index: linux-2.6/include/asm-m68knommu/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-m68knommu/thread_info.h
> +++ linux-2.6/include/asm-m68knommu/thread_info.h
> @@ -37,6 +37,7 @@ struct thread_info {
>  	unsigned long	   flags;		/* low level flags */
>  	int		   cpu;			/* cpu we're on */
>  	int		   preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int		   migrate_count;	/* 0: can migrate, <0 => BUG */
>  	struct restart_block restart_block;
>  };
>  
> @@ -89,6 +90,7 @@ static inline struct thread_info *curren
>  #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
>  					   TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		5
> +#define TIF_NEED_MIGRATE	6	/* migration necessary */
>  
>  /* as above, but as bit values */
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -96,6 +98,7 @@ static inline struct thread_info *curren
>  #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  #define _TIF_WORK_MASK		0x0000FFFE	/* work to do on interrupt/exception return */
>  
> Index: linux-2.6/include/asm-mips/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-mips/thread_info.h
> +++ linux-2.6/include/asm-mips/thread_info.h
> @@ -28,6 +28,7 @@ struct thread_info {
>  	unsigned long		tp_value;	/* thread pointer */
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	/* thread address space:
>  						   0-0xBFFFFFFF for user-thead
> @@ -49,6 +50,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -115,6 +117,7 @@ register struct thread_info *__current_t
>  #define TIF_SYSCALL_AUDIT	4	/* syscall auditing active */
>  #define TIF_SECCOMP		5	/* secure computing */
>  #define TIF_NEED_RESCHED_DELAYED 6	/* reschedule on return to userspace */
> +#define TIF_NEED_MIGRATE	7	/* migration necessary */
>  #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal() */
>  #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
>  #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling TIF_NEED_RESCHED */
> @@ -129,6 +132,7 @@ register struct thread_info *__current_t
>  #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
>  #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
>  #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
>  #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> Index: linux-2.6/include/asm-parisc/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-parisc/thread_info.h
> +++ linux-2.6/include/asm-parisc/thread_info.h
> @@ -13,6 +13,7 @@ struct thread_info {
>  	mm_segment_t addr_limit;	/* user-level address space limit */
>  	__u32 cpu;			/* current CPU */
>  	int preempt_count;		/* 0=premptable, <0=BUG; will also serve as bh-counter */
> +	int migrate_count;		/* 0: can migrate, <0 => BUG */
>  	struct restart_block restart_block;
>  };
>  
> @@ -24,6 +25,7 @@ struct thread_info {
>  	.cpu		= 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>    	.restart_block	= {			\
>  		.fn = do_no_restart_syscall	\
>  	}					\
> @@ -63,6 +65,7 @@ struct thread_info {
>  #define TIF_32BIT               5       /* 32 bit binary */
>  #define TIF_MEMDIE		6
>  #define TIF_RESTORE_SIGMASK	7	/* restore saved signal mask */
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
>  #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
> @@ -71,6 +74,7 @@ struct thread_info {
>  #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
>  #define _TIF_32BIT		(1 << TIF_32BIT)
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  
>  #define _TIF_USER_WORK_MASK     (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
>                                   _TIF_NEED_RESCHED | _TIF_RESTORE_SIGMASK)
> Index: linux-2.6/include/asm-powerpc/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-powerpc/thread_info.h
> +++ linux-2.6/include/asm-powerpc/thread_info.h
> @@ -35,6 +35,7 @@ struct thread_info {
>  	int		cpu;			/* cpu we're on */
>  	int		preempt_count;		/* 0 => preemptable,
>  						   <0 => BUG */
> +	int		migrate_count;	/* 0: can migrate, <0 => BUG */
>  	struct restart_block restart_block;
>  	unsigned long	local_flags;		/* private flags for thread */
>  
> @@ -53,6 +54,7 @@ struct thread_info {
>  	.exec_domain =	&default_exec_domain,	\
>  	.cpu =		0,			\
>  	.preempt_count = 1,			\
> +	.migrate_count = 0,			\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
>  	},					\
> @@ -124,6 +126,7 @@ static inline struct thread_info *curren
>  #define TIF_NOERROR		14	/* Force successful syscall return */
>  #define TIF_RESTORE_SIGMASK	15	/* Restore signal mask in do_signal */
>  #define TIF_FREEZE		16	/* Freezing for suspend */
> +#define TIF_NEED_MIGRATE	17	/* migration necessary */
>  
>  /* as above, but as bit values */
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -143,6 +146,7 @@ static inline struct thread_info *curren
>  #define _TIF_FREEZE		(1<<TIF_FREEZE)
>  #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
>  
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_SYSCALL_T_OR_A	(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP)
>  
>  #define _TIF_USER_WORK_MASK	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
> Index: linux-2.6/include/asm-s390/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-s390/thread_info.h
> +++ linux-2.6/include/asm-s390/thread_info.h
> @@ -51,6 +51,7 @@ struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	unsigned int		cpu;		/* current CPU */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	struct restart_block	restart_block;
>  };
>  
> @@ -64,6 +65,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
>  	},					\
> @@ -96,6 +98,7 @@ static inline struct thread_info *curren
>  #define TIF_SYSCALL_AUDIT	5	/* syscall auditing active */
>  #define TIF_SINGLE_STEP		6	/* deliver sigtrap on return to user */
>  #define TIF_MCCK_PENDING	7	/* machine check handling is pending */
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
>  #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling 
>  					   TIF_NEED_RESCHED */
> @@ -110,6 +113,7 @@ static inline struct thread_info *curren
>  #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
>  #define _TIF_SINGLE_STEP	(1<<TIF_SINGLE_STEP)
>  #define _TIF_MCCK_PENDING	(1<<TIF_MCCK_PENDING)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  #define _TIF_31BIT		(1<<TIF_31BIT)
> Index: linux-2.6/include/asm-sh/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-sh/thread_info.h
> +++ linux-2.6/include/asm-sh/thread_info.h
> @@ -21,6 +21,7 @@ struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	__u32			cpu;
>  	int			preempt_count; /* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	mm_segment_t		addr_limit;	/* thread address space */
>  	struct restart_block	restart_block;
>  	unsigned long		previous_sp;	/* sp of previous stack in case
> @@ -58,6 +59,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -113,6 +115,7 @@ static inline struct thread_info *curren
>  #define TIF_RESTORE_SIGMASK	4	/* restore signal mask in do_signal() */
>  #define TIF_SINGLESTEP		5	/* singlestepping active */
>  #define TIF_NEED_RESCHED_DELAYED	6	/* reschedule on return to userspace */
> +#define TIF_NEED_MIGRATE	7	/* migration necessary */
>  #define TIF_USEDFPU		16	/* FPU was used by this task this quantum (SMP) */
>  #define TIF_POLLING_NRFLAG	17	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		18
> @@ -125,6 +128,7 @@ static inline struct thread_info *curren
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
>  #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
>  #define _TIF_NEED_RESCHED_DELAYED	(1<<TIF_NEED_RESCHED_DELAYED)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  #define _TIF_FREEZE		(1<<TIF_FREEZE)
> Index: linux-2.6/include/asm-sh64/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-sh64/thread_info.h
> +++ linux-2.6/include/asm-sh64/thread_info.h
> @@ -23,6 +23,7 @@ struct thread_info {
>  	unsigned long		flags;		/* low level flags */
>  	/* Put the 4 32-bit fields together to make asm offsetting easier. */
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	__u16			cpu;
>  
>  	mm_segment_t		addr_limit;
> @@ -41,6 +42,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit     = KERNEL_DS,            \
>  	.restart_block	= {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -79,12 +81,14 @@ static inline struct thread_info *curren
>  #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
>  #define TIF_MEMDIE		4
>  #define TIF_RESTORE_SIGMASK	5	/* Restore signal mask in do_signal */
> +#define TIF_NEED_MIGRATE	6	/* migration necessary */
>  
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
>  #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  
>  #endif /* __KERNEL__ */
>  
> Index: linux-2.6/include/asm-sparc/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-sparc/thread_info.h
> +++ linux-2.6/include/asm-sparc/thread_info.h
> @@ -33,6 +33,7 @@ struct thread_info {
>  	int			cpu;		/* cpu we're on */
>  	int			preempt_count;	/* 0 => preemptable,
>  						   <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	int			softirq_count;
>  	int			hardirq_count;
>  
> @@ -65,6 +66,7 @@ struct thread_info {
>  	.flags		=	0,			\
>  	.cpu		=	0,			\
>  	.preempt_count	=	1,			\
> +	.migrate_count =	0,			\
>  	.restart_block	= {				\
>  		.fn	=	do_no_restart_syscall,	\
>  	},						\
> @@ -132,6 +134,7 @@ BTFIXUPDEF_CALL(void, free_thread_info, 
>  #define TIF_SIGPENDING		2	/* signal pending */
>  #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
>  #define TIF_RESTORE_SIGMASK	4	/* restore signal mask in do_signal() */
> +#define TIF_NEED_MIGRATE	5	/* migration necessary */
>  #define TIF_USEDFPU		8	/* FPU was used by this task
>  					 * this quantum (SMP) */
>  #define TIF_POLLING_NRFLAG	9	/* true if poll_idle() is polling
> @@ -143,6 +146,7 @@ BTFIXUPDEF_CALL(void, free_thread_info, 
>  #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_USEDFPU		(1<<TIF_USEDFPU)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  
> Index: linux-2.6/include/asm-sparc64/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-sparc64/thread_info.h
> +++ linux-2.6/include/asm-sparc64/thread_info.h
> @@ -47,6 +47,7 @@ struct thread_info {
>  	struct pt_regs		*kregs;
>  	struct exec_domain	*exec_domain;
>  	int			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	__u8			new_child;
>  	__u8			syscall_noerror;
>  	__u16			cpu;
> @@ -137,6 +138,7 @@ struct thread_info {
>  	.flags		= ((unsigned long)ASI_P) << TI_FLAG_CURRENT_DS_SHIFT,	\
>  	.exec_domain	=	&default_exec_domain,	\
>  	.preempt_count	=	1,			\
> +	.migrate_count =	0,			\
>  	.restart_block	= {				\
>  		.fn	=	do_no_restart_syscall,	\
>  	},						\
> @@ -225,7 +227,7 @@ register struct thread_info *current_thr
>  #define TIF_UNALIGNED		5	/* allowed to do unaligned accesses */
>  #define TIF_NEWSIGNALS		6	/* wants new-style signals */
>  #define TIF_32BIT		7	/* 32-bit binary */
> -/* flag bit 8 is available */
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  #define TIF_SECCOMP		9	/* secure computing */
>  #define TIF_SYSCALL_AUDIT	10	/* syscall auditing active */
>  /* flag bit 11 is available */
> @@ -244,6 +246,7 @@ register struct thread_info *current_thr
>  #define _TIF_UNALIGNED		(1<<TIF_UNALIGNED)
>  #define _TIF_NEWSIGNALS		(1<<TIF_NEWSIGNALS)
>  #define _TIF_32BIT		(1<<TIF_32BIT)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
>  #define _TIF_SYSCALL_AUDIT	(1<<TIF_SYSCALL_AUDIT)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
> Index: linux-2.6/include/asm-um/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-um/thread_info.h
> +++ linux-2.6/include/asm-um/thread_info.h
> @@ -18,6 +18,7 @@ struct thread_info {
>  	__u32			cpu;		/* current CPU */
>  	int			preempt_count;  /* 0 => preemptable,
>  						   <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	mm_segment_t		addr_limit;	/* thread address space:
>  					 	   0-0xBFFFFFFF for user
>  						   0-0xFFFFFFFF for kernel */
> @@ -32,6 +33,7 @@ struct thread_info {
>  	.flags =		0,		\
>  	.cpu =		0,			\
>  	.preempt_count =	1,		\
> +	.migrate_count =	0,		\
>  	.addr_limit =	KERNEL_DS,		\
>  	.restart_block =  {			\
>  		.fn =  do_no_restart_syscall,	\
> @@ -71,6 +73,7 @@ static inline struct thread_info *curren
>  #define TIF_MEMDIE	 	5
>  #define TIF_SYSCALL_AUDIT	6
>  #define TIF_RESTORE_SIGMASK	7
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  
>  #define _TIF_SYSCALL_TRACE	(1 << TIF_SYSCALL_TRACE)
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
> @@ -79,5 +82,6 @@ static inline struct thread_info *curren
>  #define _TIF_MEMDIE		(1 << TIF_MEMDIE)
>  #define _TIF_SYSCALL_AUDIT	(1 << TIF_SYSCALL_AUDIT)
>  #define _TIF_RESTORE_SIGMASK	(1 << TIF_RESTORE_SIGMASK)
> +#define _TIF_NEED_MIGRATE	(1 << TIF_NEED_MIGRATE)
>  
>  #endif
> Index: linux-2.6/include/asm-v850/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-v850/thread_info.h
> +++ linux-2.6/include/asm-v850/thread_info.h
> @@ -32,6 +32,7 @@ struct thread_info {
>  	int			cpu;		/* cpu we're on */
>  	int			preempt_count;	/* 0 => preemptable,
>  						   <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  	struct restart_block	restart_block;
>  };
>  
> @@ -42,6 +43,7 @@ struct thread_info {
>  	.flags =	0,						      \
>  	.cpu =		0,						      \
>  	.preempt_count = 1,						      \
> +	.migrate_count = 0,						      \
>  	.restart_block = {						      \
>  		.fn = do_no_restart_syscall,				      \
>  	},								      \
> @@ -83,6 +85,7 @@ struct thread_info {
>  #define TIF_POLLING_NRFLAG	4	/* true if poll_idle() is polling
>  					   TIF_NEED_RESCHED */
>  #define TIF_MEMDIE		5
> +#define TIF_NEED_MIGRATE	6	/* migration necessary */
>  
>  /* as above, but as bit values */
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -90,6 +93,7 @@ struct thread_info {
>  #define _TIF_SIGPENDING		(1<<TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  
>  
>  /* Size of kernel stack for each process.  */
> Index: linux-2.6/include/asm-x86_64/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-x86_64/thread_info.h
> +++ linux-2.6/include/asm-x86_64/thread_info.h
> @@ -30,6 +30,7 @@ struct thread_info {
>  	__u32			status;		/* thread synchronous flags */
>  	__u32			cpu;		/* current CPU */
>  	int 			preempt_count;	/* 0 => preemptable, <0 => BUG */
> +	int			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	
>  	struct restart_block    restart_block;
> @@ -48,6 +49,7 @@ struct thread_info {
>  	.flags	       = 0,			\
>  	.cpu	       = 0,			\
>  	.preempt_count = 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit     = KERNEL_DS,		\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -116,6 +118,7 @@ static inline struct thread_info *stack_
>  #define TIF_SYSCALL_AUDIT	7	/* syscall auditing active */
>  #define TIF_SECCOMP		8	/* secure computing */
>  #define TIF_RESTORE_SIGMASK	9	/* restore signal mask in do_signal */
> +#define TIF_NEED_MIGRATE	11	/* migration necessary */
>  /* 16 free */
>  #define TIF_IA32		17	/* 32bit process */ 
>  #define TIF_FORK		18	/* ret_from_fork */
> @@ -135,6 +138,7 @@ static inline struct thread_info *stack_
>  #define _TIF_SECCOMP		(1<<TIF_SECCOMP)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
>  #define _TIF_NEED_RESCHED_DELAYED (1<<TIF_NEED_RESCHED_DELAYED)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_IA32		(1<<TIF_IA32)
>  #define _TIF_FORK		(1<<TIF_FORK)
>  #define _TIF_ABI_PENDING	(1<<TIF_ABI_PENDING)
> Index: linux-2.6/include/asm-xtensa/thread_info.h
> ===================================================================
> --- linux-2.6.orig/include/asm-xtensa/thread_info.h
> +++ linux-2.6/include/asm-xtensa/thread_info.h
> @@ -34,6 +34,7 @@ struct thread_info {
>  	unsigned long		status;		/* thread-synchronous flags */
>  	__u32			cpu;		/* current CPU */
>  	__s32			preempt_count;	/* 0 => preemptable,< 0 => BUG*/
> +	__s32			migrate_count;/* 0: can migrate, <0 => BUG */
>  
>  	mm_segment_t		addr_limit;	/* thread address space */
>  	struct restart_block    restart_block;
> @@ -72,6 +73,7 @@ struct thread_info {
>  	.flags		= 0,			\
>  	.cpu		= 0,			\
>  	.preempt_count	= 1,			\
> +	.migrate_count = 0,			\
>  	.addr_limit	= KERNEL_DS,		\
>  	.restart_block = {			\
>  		.fn = do_no_restart_syscall,	\
> @@ -117,6 +119,7 @@ static inline struct thread_info *curren
>  #define TIF_IRET		5	/* return with iret */
>  #define TIF_MEMDIE		6
>  #define TIF_RESTORE_SIGMASK	7	/* restore signal mask in do_signal() */
> +#define TIF_NEED_MIGRATE	8	/* migration necessary */
>  #define TIF_POLLING_NRFLAG	16	/* true if poll_idle() is polling TIF_NEED_RESCHED */
>  
>  #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
> @@ -125,6 +128,7 @@ static inline struct thread_info *curren
>  #define _TIF_NEED_RESCHED	(1<<TIF_NEED_RESCHED)
>  #define _TIF_SINGLESTEP		(1<<TIF_SINGLESTEP)
>  #define _TIF_IRET		(1<<TIF_IRET)
> +#define _TIF_NEED_MIGRATE	(1<<TIF_NEED_MIGRATE)
>  #define _TIF_POLLING_NRFLAG	(1<<TIF_POLLING_NRFLAG)
>  #define _TIF_RESTORE_SIGMASK	(1<<TIF_RESTORE_SIGMASK)
>  
> Index: linux-2.6/lib/smp_processor_id.c
> ===================================================================
> --- linux-2.6.orig/lib/smp_processor_id.c
> +++ linux-2.6/lib/smp_processor_id.c
> @@ -10,12 +10,16 @@
>  unsigned int notrace debug_smp_processor_id(void)
>  {
>  	unsigned long preempt_count = preempt_count();
> +	unsigned long migrate_count = migrate_count();
>  	int this_cpu = raw_smp_processor_id();
>  	cpumask_t this_mask;
>  
>  	if (likely(preempt_count))
>  		goto out;
>  
> +	if (likely(migrate_count))
> +		goto out;
> +
>  	if (irqs_disabled())
>  		goto out;
>  
> @@ -42,7 +46,7 @@ unsigned int notrace debug_smp_processor
>  	if (!printk_ratelimit())
>  		goto out_enable;
>  
> -	printk(KERN_ERR "BUG: using smp_processor_id() in preemptible [%08x] code: %s/%d\n", preempt_count()-1, current->comm, current->pid);
> +	printk(KERN_ERR "BUG: using smp_processor_id() in preemptible [%08x] migration [%08x] code: %s/%d\n", preempt_count()-1, migrate_count(), current->comm, current->pid);
>  	print_symbol("caller is %s\n", (long)__builtin_return_address(0));
>  	dump_stack();
>  
> 
> --
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Oleg Nesterov]

On 07/14, Peter Zijlstra wrote:
>
> From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> 
> This patch adds the ability to protect critical sections from migration to
> another CPU without disabling preemption.
> 
> This will be useful to minimize the amount of preemption disabling for the -rt
> patch. It will help leveraging improvements brought by the local_t types in
> asm/local.h (see Documentation/local_ops.txt). Note that the updates done to
> variables protected by migrate_disable must be either atomic or protected from
> concurrent updates done by other threads.
> 
> Typical use:
> 
> migrate_disable();
> local_inc(&__get_cpu_var(&my_local_t_var));
> migrate_enable();
> 
> Which will increment the variable atomically wrt the local CPU.

I still think this patch is buggy. Perhaps I am wrong. Please look at

	http://marc.info/?l=linux-kernel&m=118417177818825

there was no any reply.

Oleg.

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Peter Zijlstra]

On Sat, 2007-07-14 at 21:16 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:
> >
> > From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> > 
> > This patch adds the ability to protect critical sections from migration to
> > another CPU without disabling preemption.
> > 
> > This will be useful to minimize the amount of preemption disabling for the -rt
> > patch. It will help leveraging improvements brought by the local_t types in
> > asm/local.h (see Documentation/local_ops.txt). Note that the updates done to
> > variables protected by migrate_disable must be either atomic or protected from
> > concurrent updates done by other threads.
> > 
> > Typical use:
> > 
> > migrate_disable();
> > local_inc(&__get_cpu_var(&my_local_t_var));
> > migrate_enable();
> > 
> > Which will increment the variable atomically wrt the local CPU.
> 
> I still think this patch is buggy. Perhaps I am wrong. Please look at
> 
> 	http://marc.info/?l=linux-kernel&m=118417177818825
> 
> there was no any reply.

Ah, yes. That would be quite nasty. I assumed the patch worked for the
non-hotplug case. But you suggest even regular wakeups could go wrong.
That would need fixing indeed.

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Peter Zijlstra]

On Sat, 2007-07-14 at 21:16 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:
> >
> > From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> > 
> > This patch adds the ability to protect critical sections from migration to
> > another CPU without disabling preemption.
> > 
> > This will be useful to minimize the amount of preemption disabling for the -rt
> > patch. It will help leveraging improvements brought by the local_t types in
> > asm/local.h (see Documentation/local_ops.txt). Note that the updates done to
> > variables protected by migrate_disable must be either atomic or protected from
> > concurrent updates done by other threads.
> > 
> > Typical use:
> > 
> > migrate_disable();
> > local_inc(&__get_cpu_var(&my_local_t_var));
> > migrate_enable();
> > 
> > Which will increment the variable atomically wrt the local CPU.
> 
> I still think this patch is buggy. Perhaps I am wrong. Please look at
> 
> 	http://marc.info/?l=linux-kernel&m=118417177818825
> 
> there was no any reply.

Looking at it, I tend to agree that the migration_thread thing is indeed
quite broken. Not quite sure on how to solve it nicely though, one would
need to stick the migration request structure somewhere in task_struct
or thread_info, and not back on the migration_queue. But given that
people already complained about the adding ov migration_counter,. I'm
not daring to propose such things.

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Peter Zijlstra]

How about somethign like this?

---

Avoid busy looping on unmigratable tasks by pushing the migration requests
onto a delayed_migration_queue, which we try on each wakeup.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 kernel/sched.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c
+++ linux-2.6/kernel/sched.c
@@ -288,6 +288,7 @@ struct rq {
 
 	struct task_struct *migration_thread;
 	struct list_head migration_queue;
+	struct list_head delayed_migration_queue;
 #endif
 
 #ifdef CONFIG_SCHEDSTATS
@@ -5623,6 +5624,11 @@ static int migration_thread(void *data)
 		head = &rq->migration_queue;
 
 		if (list_empty(head)) {
+			/*
+			 * we got a wakeup, give the delayed list another shot.
+			 */
+			if (current->state != TASK_INTERRUPTIBLE)
+				list_splice(&rq->delayed_migration_queue, head);
 			spin_unlock_irq(&rq->lock);
 			schedule();
 			set_current_state(TASK_INTERRUPTIBLE);
@@ -5641,8 +5647,7 @@ static int migration_thread(void *data)
 			 * wake us up.
 			 */
 			spin_lock_irq(&rq->lock);
-			head = &rq->migration_queue;
-			list_add(&req->list, head);
+			list_add(&req->list, &rq->delayed_migration_queue);
 			set_tsk_thread_flag(req->task, TIF_NEED_MIGRATE);
 			spin_unlock_irq(&rq->lock);
 			wake_up_process(req->task);
@@ -7006,6 +7011,7 @@ void __init sched_init(void)
 		rq->cpu = i;
 		rq->migration_thread = NULL;
 		INIT_LIST_HEAD(&rq->migration_queue);
+		INIT_LIST_HEAD(&rq->delayed_migration_queue);
 #endif
 		atomic_set(&rq->nr_iowait, 0);
 

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> How about somethign like this?
> 

Interesting approach.. then the process in its migrate disable critical
section will wake us up when it ends, but we are still available for
other migrations.

I'll try to merge this with what I have.

> ---
> 
> Avoid busy looping on unmigratable tasks by pushing the migration requests
> onto a delayed_migration_queue, which we try on each wakeup.
> 
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
>  kernel/sched.c |   10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/kernel/sched.c
> ===================================================================
> --- linux-2.6.orig/kernel/sched.c
> +++ linux-2.6/kernel/sched.c
> @@ -288,6 +288,7 @@ struct rq {
>  
>  	struct task_struct *migration_thread;
>  	struct list_head migration_queue;
> +	struct list_head delayed_migration_queue;
>  #endif
>  
>  #ifdef CONFIG_SCHEDSTATS
> @@ -5623,6 +5624,11 @@ static int migration_thread(void *data)
>  		head = &rq->migration_queue;
>  
>  		if (list_empty(head)) {
> +			/*
> +			 * we got a wakeup, give the delayed list another shot.
> +			 */
> +			if (current->state != TASK_INTERRUPTIBLE)
> +				list_splice(&rq->delayed_migration_queue, head);
>  			spin_unlock_irq(&rq->lock);
>  			schedule();
>  			set_current_state(TASK_INTERRUPTIBLE);
> @@ -5641,8 +5647,7 @@ static int migration_thread(void *data)
>  			 * wake us up.
>  			 */
>  			spin_lock_irq(&rq->lock);
> -			head = &rq->migration_queue;
> -			list_add(&req->list, head);
> +			list_add(&req->list, &rq->delayed_migration_queue);
>  			set_tsk_thread_flag(req->task, TIF_NEED_MIGRATE);
>  			spin_unlock_irq(&rq->lock);
>  			wake_up_process(req->task);
> @@ -7006,6 +7011,7 @@ void __init sched_init(void)
>  		rq->cpu = i;
>  		rq->migration_thread = NULL;
>  		INIT_LIST_HEAD(&rq->migration_queue);
> +		INIT_LIST_HEAD(&rq->delayed_migration_queue);
>  #endif
>  		atomic_set(&rq->nr_iowait, 0);
>  
> 
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Oleg Nesterov]

On 07/14, Peter Zijlstra wrote:

> --- linux-2.6.orig/kernel/sched.c
> +++ linux-2.6/kernel/sched.c
> @@ -288,6 +288,7 @@ struct rq {
>  
>  	struct task_struct *migration_thread;
>  	struct list_head migration_queue;
> +	struct list_head delayed_migration_queue;
>  #endif

I think this could be simplified a bit.

delayed_migration_queue can be a local variable in migration_thread(), then

>  #ifdef CONFIG_SCHEDSTATS
> @@ -5623,6 +5624,11 @@ static int migration_thread(void *data)
>  		head = &rq->migration_queue;
>  
>  		if (list_empty(head)) {
> +			/*
> +			 * we got a wakeup, give the delayed list another shot.
> +			 */
> +			if (current->state != TASK_INTERRUPTIBLE)
> +				list_splice(&rq->delayed_migration_queue, head);

No need to check TASK_INTERRUPTIBLE, just do list_splice() unconditionally and
call schedule().

Oleg.

Re: [PATCH -rt 2/5] Thread Migration Preemption - v2

[Peter Zijlstra]

On Sun, 2007-07-15 at 00:48 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:
> 
> > --- linux-2.6.orig/kernel/sched.c
> > +++ linux-2.6/kernel/sched.c
> > @@ -288,6 +288,7 @@ struct rq {
> >  
> >  	struct task_struct *migration_thread;
> >  	struct list_head migration_queue;
> > +	struct list_head delayed_migration_queue;
> >  #endif
> 
> I think this could be simplified a bit.
> 
> delayed_migration_queue can be a local variable in migration_thread(), then
> 
> >  #ifdef CONFIG_SCHEDSTATS
> > @@ -5623,6 +5624,11 @@ static int migration_thread(void *data)
> >  		head = &rq->migration_queue;
> >  
> >  		if (list_empty(head)) {
> > +			/*
> > +			 * we got a wakeup, give the delayed list another shot.
> > +			 */
> > +			if (current->state != TASK_INTERRUPTIBLE)
> > +				list_splice(&rq->delayed_migration_queue, head);
> 
> No need to check TASK_INTERRUPTIBLE, just do list_splice() unconditionally and
> call schedule().

Yes, I got my head in a twist on that one. You are quite right!

[PATCH -rt 3/5] asm/local.h cmpxchg

[Peter Zijlstra]

[-- Attachment #1: local.patch --]
[-- Type: text/plain, Size: 4307 bytes --]

From: Christoph Lameter <clameter@sgi.com>

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 include/asm-generic/local.h |   25 +++++++++++++++++++++++--
 include/asm-i386/local.h    |   13 +++++++++++++
 include/asm-x86_64/local.h  |   16 ++++++++++++++++
 3 files changed, 52 insertions(+), 2 deletions(-)

Index: linux-2.6.22-rc6-mm1/include/asm-generic/local.h
===================================================================
--- linux-2.6.22-rc6-mm1.orig/include/asm-generic/local.h	2007-07-12 19:44:18.000000000 -0700
+++ linux-2.6.22-rc6-mm1/include/asm-generic/local.h	2007-07-12 19:44:57.000000000 -0700
@@ -46,13 +46,34 @@ typedef struct
 #define local_add_unless(l, a, u) atomic_long_add_unless((&(l)->a), (a), (u))
 #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a)
 
-/* Non-atomic variants, ie. preemption disabled and won't be touched
- * in interrupt, etc.  Some archs can optimize this case well. */
+/*
+ * Establish a state necessary for __local_xx functions to work.
+ */
+#define __local_begin(flags)	local_irq_disable(flags)
+
+static inline void __local_end(unsigned long flags)
+{
+	local_irq_restore(flags);
+}
+
+/*
+ * Non-atomic variants, ie. within local_begin() / local_end() or
+ * preempt_disable / enable() and won't be touched in interrupt, etc.
+ * Some archs can optimize this case well.
+ */
 #define __local_inc(l)		local_set((l), local_read(l) + 1)
 #define __local_dec(l)		local_set((l), local_read(l) - 1)
 #define __local_add(i,l)	local_set((l), local_read(l) + (i))
 #define __local_sub(i,l)	local_set((l), local_read(l) - (i))
 
+#define __local_cmpxchg((v), (o), (n)) (*(v) = (n), (o))
+#define __local_xchg((v), (n)) 						\
+({									\
+	__typeof(v) x = *(v);						\
+	*(v) = (n);							\
+	x;								\
+)}
+
 /* Use these for per-cpu local_t variables: on some archs they are
  * much more efficient than these naive implementations.  Note they take
  * a variable (eg. mystruct.foo), not an address.
Index: linux-2.6.22-rc6-mm1/include/asm-x86_64/local.h
===================================================================
--- linux-2.6.22-rc6-mm1.orig/include/asm-x86_64/local.h	2007-07-12 19:44:18.000000000 -0700
+++ linux-2.6.22-rc6-mm1/include/asm-x86_64/local.h	2007-07-12 19:44:57.000000000 -0700
@@ -9,6 +9,7 @@ typedef struct
 	atomic_long_t a;
 } local_t;
 
+
 #define LOCAL_INIT(i)	{ ATOMIC_LONG_INIT(i) }
 
 #define local_read(l)	atomic_long_read(&(l)->a)
@@ -181,11 +182,26 @@ static __inline__ long local_sub_return(
 
 /* On x86-64 these are better than the atomic variants on SMP kernels
    because they dont use a lock prefix. */
+
+#define __local_begin(__flags)		\
+{					\
+	(__flags) = 0;			\
+	preempt_disable();		\
+}
+
+static inline void __local_end(unsigned long flags) {
+	preempt_enable();
+}
+
 #define __local_inc(l)		local_inc(l)
 #define __local_dec(l)		local_dec(l)
 #define __local_add(i,l)	local_add((i),(l))
 #define __local_sub(i,l)	local_sub((i),(l))
 
+#define __local_cmpxchg		cmpxchg_local
+#define __local_xchg		xchg
+
+
 /* Use these for per-cpu local_t variables: on some archs they are
  * much more efficient than these naive implementations.  Note they take
  * a variable, not an address.
Index: linux-2.6.22-rc6-mm1/include/asm-i386/local.h
===================================================================
--- linux-2.6.22-rc6-mm1.orig/include/asm-i386/local.h	2007-07-12 19:53:00.000000000 -0700
+++ linux-2.6.22-rc6-mm1/include/asm-i386/local.h	2007-07-12 19:55:22.000000000 -0700
@@ -194,12 +194,25 @@ static __inline__ long local_sub_return(
 })
 #define local_inc_not_zero(l) local_add_unless((l), 1, 0)
 
+#define __local_begin(__flags)					\
+{								\
+	(__flags) = 0;						\
+	preempt_disable();					\
+}
+
+static inline void __local_end(unsigned long flags) {
+	preempt_enable();
+}
+
 /* On x86, these are no better than the atomic variants. */
 #define __local_inc(l)		local_inc(l)
 #define __local_dec(l)		local_dec(l)
 #define __local_add(i,l)	local_add((i),(l))
 #define __local_sub(i,l)	local_sub((i),(l))
 
+#define __local_cmpxchg		cmpxchg_local
+#define __local_xchg		xchg
+
 /* Use these for per-cpu local_t variables: on some archs they are
  * much more efficient than these naive implementations.  Note they take
  * a variable, not an address.

--

Re: [PATCH -rt 3/5] asm/local.h cmpxchg

[Daniel Walker]

On Sat, 2007-07-14 at 19:57 +0200, Peter Zijlstra wrote:
> ===================================================================
> ---
> linux-2.6.22-rc6-mm1.orig/include/asm-generic/local.h       2007-07-12
> 19:44:18.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/include/asm-generic/local.h    2007-07-12
> 19:44:57.000000000 -0700
> @@ -46,13 +46,34 @@ typedef struct
>  #define local_add_unless(l, a, u) atomic_long_add_unless((&(l)->a),
> (a), (u))
>  #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a)
>  
> -/* Non-atomic variants, ie. preemption disabled and won't be touched
> - * in interrupt, etc.  Some archs can optimize this case well. */
> +/*
> + * Establish a state necessary for __local_xx functions to work.
> + */
> +#define __local_begin(flags)   local_irq_disable(flags) 

Did you mean local_irq_save(flags)? local_irq_disable() doesn't take any
arguments.

Daniel

Re: [PATCH -rt 3/5] asm/local.h cmpxchg

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> From: Christoph Lameter <clameter@sgi.com>
> 
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
>  include/asm-generic/local.h |   25 +++++++++++++++++++++++--
>  include/asm-i386/local.h    |   13 +++++++++++++
>  include/asm-x86_64/local.h  |   16 ++++++++++++++++
>  3 files changed, 52 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6.22-rc6-mm1/include/asm-generic/local.h
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/include/asm-generic/local.h	2007-07-12 19:44:18.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/include/asm-generic/local.h	2007-07-12 19:44:57.000000000 -0700
> @@ -46,13 +46,34 @@ typedef struct
>  #define local_add_unless(l, a, u) atomic_long_add_unless((&(l)->a), (a), (u))
>  #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a)
>  
> -/* Non-atomic variants, ie. preemption disabled and won't be touched
> - * in interrupt, etc.  Some archs can optimize this case well. */
> +/*
> + * Establish a state necessary for __local_xx functions to work.
> + */
> +#define __local_begin(flags)	local_irq_disable(flags)
> +
> +static inline void __local_end(unsigned long flags)
> +{
> +	local_irq_restore(flags);
> +}
> +

Wouldn't it be cheaper to use preempt_disable/enable instead of irq
disable/enable in asm-generic to protect the __local accesses since they
are not supposed to be touched by interrupt context ?

I also think that __local_begin/end() should be changed into
local_begin() and local_end(). It makes sense to use this around all
local_t variable accesses.

> +/*
> + * Non-atomic variants, ie. within local_begin() / local_end() or
> + * preempt_disable / enable() and won't be touched in interrupt, etc.
> + * Some archs can optimize this case well.
> + */
>  #define __local_inc(l)		local_set((l), local_read(l) + 1)
>  #define __local_dec(l)		local_set((l), local_read(l) - 1)
>  #define __local_add(i,l)	local_set((l), local_read(l) + (i))
>  #define __local_sub(i,l)	local_set((l), local_read(l) - (i))
>  
> +#define __local_cmpxchg((v), (o), (n)) (*(v) = (n), (o))

Shouldn't this look like a while loop instead ? Where is the
comparison ? It should use local_set and local_read...

The proper way to do this would be to take all architectures that only
define a atomic_cmpxchg and atomic_xchg and turn that into a cmpxchg and
xchg. Then, the same could be done for architectures which only have a
local_cmpxchg, but no cmpxchg_local.

Then, cmpxchg_local could be used to touch a variable in an atomic wrt
cpu fashion without wrapping the variable in a local_t.

All the local_*() functions should only touch local_t types.

If local_begin()/local_end() is defined as preempt disable/enable, then
it's ok to use this to protect __local_*() accesses. However, if you
turn this into a migrate_disable/enable(), then the protection against
other threads on the same CPU is not insured.

> +#define __local_xchg((v), (n)) 						\
> +({									\
> +	__typeof(v) x = *(v);						\
> +	*(v) = (n);							\
> +	x;								\
> +)}
> +
>  /* Use these for per-cpu local_t variables: on some archs they are
>   * much more efficient than these naive implementations.  Note they take
>   * a variable (eg. mystruct.foo), not an address.
> Index: linux-2.6.22-rc6-mm1/include/asm-x86_64/local.h
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/include/asm-x86_64/local.h	2007-07-12 19:44:18.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/include/asm-x86_64/local.h	2007-07-12 19:44:57.000000000 -0700
> @@ -9,6 +9,7 @@ typedef struct
>  	atomic_long_t a;
>  } local_t;
>  
> +

Empty line ?

>  #define LOCAL_INIT(i)	{ ATOMIC_LONG_INIT(i) }
>  
>  #define local_read(l)	atomic_long_read(&(l)->a)
> @@ -181,11 +182,26 @@ static __inline__ long local_sub_return(
>  
>  /* On x86-64 these are better than the atomic variants on SMP kernels
>     because they dont use a lock prefix. */
> +
> +#define __local_begin(__flags)		\
> +{					\
> +	(__flags) = 0;			\

What are these flags doing ? And why are they not set back in
__local_end ? I'm a bit curious :)

> +	preempt_disable();		\
> +}
> +
> +static inline void __local_end(unsigned long flags) {
> +	preempt_enable();
> +}
> +

Same here : __local_begin/end() -> local_begin/end().

>  #define __local_inc(l)		local_inc(l)
>  #define __local_dec(l)		local_dec(l)
>  #define __local_add(i,l)	local_add((i),(l))
>  #define __local_sub(i,l)	local_sub((i),(l))
>  
> +#define __local_cmpxchg		cmpxchg_local
> +#define __local_xchg		xchg
> +

local_*() should only touch local_t vars.

> +
>  /* Use these for per-cpu local_t variables: on some archs they are
>   * much more efficient than these naive implementations.  Note they take
>   * a variable, not an address.
> Index: linux-2.6.22-rc6-mm1/include/asm-i386/local.h
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/include/asm-i386/local.h	2007-07-12 19:53:00.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/include/asm-i386/local.h	2007-07-12 19:55:22.000000000 -0700
> @@ -194,12 +194,25 @@ static __inline__ long local_sub_return(
>  })
>  #define local_inc_not_zero(l) local_add_unless((l), 1, 0)
>  
> +#define __local_begin(__flags)					\
> +{								\
> +	(__flags) = 0;						\
> +	preempt_disable();					\
> +}
> +
> +static inline void __local_end(unsigned long flags) {
> +	preempt_enable();
> +}

__local_begin/end -> local_begin/end.

> +
>  /* On x86, these are no better than the atomic variants. */
>  #define __local_inc(l)		local_inc(l)
>  #define __local_dec(l)		local_dec(l)
>  #define __local_add(i,l)	local_add((i),(l))
>  #define __local_sub(i,l)	local_sub((i),(l))
>  
> +#define __local_cmpxchg		cmpxchg_local
> +#define __local_xchg		xchg
> +

local_*() should only touch local_t vars.

Mathieu

>  /* Use these for per-cpu local_t variables: on some archs they are
>   * much more efficient than these naive implementations.  Note they take
>   * a variable, not an address.
> 
> --
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH -rt 3/5] asm/local.h cmpxchg

[Peter Zijlstra]

On Sat, 2007-07-14 at 13:14 -0400, Mathieu Desnoyers wrote:
> * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > From: Christoph Lameter <clameter@sgi.com>
> > 
> > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > ---
> >  include/asm-generic/local.h |   25 +++++++++++++++++++++++--
> >  include/asm-i386/local.h    |   13 +++++++++++++
> >  include/asm-x86_64/local.h  |   16 ++++++++++++++++
> >  3 files changed, 52 insertions(+), 2 deletions(-)
> > 
> > Index: linux-2.6.22-rc6-mm1/include/asm-generic/local.h
> > ===================================================================
> > --- linux-2.6.22-rc6-mm1.orig/include/asm-generic/local.h	2007-07-12 19:44:18.000000000 -0700
> > +++ linux-2.6.22-rc6-mm1/include/asm-generic/local.h	2007-07-12 19:44:57.000000000 -0700
> > @@ -46,13 +46,34 @@ typedef struct
> >  #define local_add_unless(l, a, u) atomic_long_add_unless((&(l)->a), (a), (u))
> >  #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a)
> >  
> > -/* Non-atomic variants, ie. preemption disabled and won't be touched
> > - * in interrupt, etc.  Some archs can optimize this case well. */
> > +/*
> > + * Establish a state necessary for __local_xx functions to work.
> > + */
> > +#define __local_begin(flags)	local_irq_disable(flags)
> > +
> > +static inline void __local_end(unsigned long flags)
> > +{
> > +	local_irq_restore(flags);
> > +}
> > +
> 
> Wouldn't it be cheaper to use preempt_disable/enable instead of irq
> disable/enable in asm-generic to protect the __local accesses since they
> are not supposed to be touched by interrupt context ?
> 
> I also think that __local_begin/end() should be changed into
> local_begin() and local_end(). It makes sense to use this around all
> local_t variable accesses.
> 
> > +/*
> > + * Non-atomic variants, ie. within local_begin() / local_end() or
> > + * preempt_disable / enable() and won't be touched in interrupt, etc.
> > + * Some archs can optimize this case well.
> > + */
> >  #define __local_inc(l)		local_set((l), local_read(l) + 1)
> >  #define __local_dec(l)		local_set((l), local_read(l) - 1)
> >  #define __local_add(i,l)	local_set((l), local_read(l) + (i))
> >  #define __local_sub(i,l)	local_set((l), local_read(l) - (i))
> >  
> > +#define __local_cmpxchg((v), (o), (n)) (*(v) = (n), (o))
> 
> Shouldn't this look like a while loop instead ? Where is the
> comparison ? It should use local_set and local_read...

Yeah, needs more thought.

> The proper way to do this would be to take all architectures that only
> define a atomic_cmpxchg and atomic_xchg and turn that into a cmpxchg and
> xchg. Then, the same could be done for architectures which only have a
> local_cmpxchg, but no cmpxchg_local.
> 
> Then, cmpxchg_local could be used to touch a variable in an atomic wrt
> cpu fashion without wrapping the variable in a local_t.
> 
> All the local_*() functions should only touch local_t types.
> 
> If local_begin()/local_end() is defined as preempt disable/enable, then
> it's ok to use this to protect __local_*() accesses. However, if you
> turn this into a migrate_disable/enable(), then the protection against
> other threads on the same CPU is not insured.

Yeah, the generic stuff is rather broken. For -rt begin/end should map
to migrate_disable/enable, and the primitives could be done using
preempt_disable/enable.

something like so:

static inline void local_begin(flags)
{
	migrate_disable();
}

static inline void local_end(flags)
{
	migrate_enable();
}

#define __local_cmpxchg(__p, __o, __n)		\
({	typeof(__o) o;				\
	preempt_disable();			\
	o = *(__p);				\
	if (o == (__o))				\
		*(__p) = __n;			\
	preempt_enable();			\
	o; })


Obviously that only works for -rt.

Re: [PATCH -rt 3/5] asm/local.h cmpxchg

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> On Sat, 2007-07-14 at 13:14 -0400, Mathieu Desnoyers wrote:
> > * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > > From: Christoph Lameter <clameter@sgi.com>
> > > 
> > > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > ---
> > >  include/asm-generic/local.h |   25 +++++++++++++++++++++++--
> > >  include/asm-i386/local.h    |   13 +++++++++++++
> > >  include/asm-x86_64/local.h  |   16 ++++++++++++++++
> > >  3 files changed, 52 insertions(+), 2 deletions(-)
> > > 
> > > Index: linux-2.6.22-rc6-mm1/include/asm-generic/local.h
> > > ===================================================================
> > > --- linux-2.6.22-rc6-mm1.orig/include/asm-generic/local.h	2007-07-12 19:44:18.000000000 -0700
> > > +++ linux-2.6.22-rc6-mm1/include/asm-generic/local.h	2007-07-12 19:44:57.000000000 -0700
> > > @@ -46,13 +46,34 @@ typedef struct
> > >  #define local_add_unless(l, a, u) atomic_long_add_unless((&(l)->a), (a), (u))
> > >  #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a)
> > >  
> > > -/* Non-atomic variants, ie. preemption disabled and won't be touched
> > > - * in interrupt, etc.  Some archs can optimize this case well. */
> > > +/*
> > > + * Establish a state necessary for __local_xx functions to work.
> > > + */
> > > +#define __local_begin(flags)	local_irq_disable(flags)
> > > +
> > > +static inline void __local_end(unsigned long flags)
> > > +{
> > > +	local_irq_restore(flags);
> > > +}
> > > +
> > 
> > Wouldn't it be cheaper to use preempt_disable/enable instead of irq
> > disable/enable in asm-generic to protect the __local accesses since they
> > are not supposed to be touched by interrupt context ?
> > 
> > I also think that __local_begin/end() should be changed into
> > local_begin() and local_end(). It makes sense to use this around all
> > local_t variable accesses.
> > 
> > > +/*
> > > + * Non-atomic variants, ie. within local_begin() / local_end() or
> > > + * preempt_disable / enable() and won't be touched in interrupt, etc.
> > > + * Some archs can optimize this case well.
> > > + */
> > >  #define __local_inc(l)		local_set((l), local_read(l) + 1)
> > >  #define __local_dec(l)		local_set((l), local_read(l) - 1)
> > >  #define __local_add(i,l)	local_set((l), local_read(l) + (i))
> > >  #define __local_sub(i,l)	local_set((l), local_read(l) - (i))
> > >  
> > > +#define __local_cmpxchg((v), (o), (n)) (*(v) = (n), (o))
> > 
> > Shouldn't this look like a while loop instead ? Where is the
> > comparison ? It should use local_set and local_read...
> 
> Yeah, needs more thought.
> 
> > The proper way to do this would be to take all architectures that only
> > define a atomic_cmpxchg and atomic_xchg and turn that into a cmpxchg and
> > xchg. Then, the same could be done for architectures which only have a
> > local_cmpxchg, but no cmpxchg_local.
> > 
> > Then, cmpxchg_local could be used to touch a variable in an atomic wrt
> > cpu fashion without wrapping the variable in a local_t.
> > 
> > All the local_*() functions should only touch local_t types.
> > 
> > If local_begin()/local_end() is defined as preempt disable/enable, then
> > it's ok to use this to protect __local_*() accesses. However, if you
> > turn this into a migrate_disable/enable(), then the protection against
> > other threads on the same CPU is not insured.
> 
> Yeah, the generic stuff is rather broken. For -rt begin/end should map
> to migrate_disable/enable, and the primitives could be done using
> preempt_disable/enable.
> 
> something like so:
> 
> static inline void local_begin(flags)
> {
> 	migrate_disable();
> }
> 
> static inline void local_end(flags)
> {
> 	migrate_enable();
> }
> 
> #define __local_cmpxchg(__p, __o, __n)		\
> ({	typeof(__o) o;				\
> 	preempt_disable();			\
> 	o = *(__p);				\
> 	if (o == (__o))				\
> 		*(__p) = __n;			\
> 	preempt_enable();			\
> 	o; })
> 

Nitpicking:

This should be named __cmpxchg_local(), since it does not apply to
local_t variables.

Also, it would not hurt to have __local_begin and __local_end around
__cmpxchg_local to offer an interface similar to local_begin and
local_end. Those two would map to preempt_disable/enable.

> 
> Obviously that only works for -rt.
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH -rt 4/5] use migrate_disable for __local_begin

[Peter Zijlstra]

[-- Attachment #1: local-no_migrate.patch --]
[-- Type: text/plain, Size: 1381 bytes --]

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 include/asm-i386/local.h   |    7 ++++---
 include/asm-x86_64/local.h |    7 ++++---
 2 files changed, 8 insertions(+), 6 deletions(-)

Index: linux-2.6/include/asm-i386/local.h
===================================================================
--- linux-2.6.orig/include/asm-i386/local.h
+++ linux-2.6/include/asm-i386/local.h
@@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
 #define __local_begin(__flags)					\
 {								\
 	(__flags) = 0;						\
-	preempt_disable();					\
+	migrate_disable();					\
 }
 
-static inline void __local_end(unsigned long flags) {
-	preempt_enable();
+static inline void __local_end(unsigned long flags)
+{
+	migrate_enable();
 }
 
 /* On x86, these are no better than the atomic variants. */
Index: linux-2.6/include/asm-x86_64/local.h
===================================================================
--- linux-2.6.orig/include/asm-x86_64/local.h
+++ linux-2.6/include/asm-x86_64/local.h
@@ -186,11 +186,12 @@ static __inline__ long local_sub_return(
 #define __local_begin(__flags)		\
 {					\
 	(__flags) = 0;			\
-	preempt_disable();		\
+	migrate_disable();		\
 }
 
-static inline void __local_end(unsigned long flags) {
-	preempt_enable();
+static inline void __local_end(unsigned long flags)
+{
+	migrate_enable();
 }
 
 #define __local_inc(l)		local_inc(l)

--

Re: [PATCH -rt 4/5] use migrate_disable for __local_begin

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
>  include/asm-i386/local.h   |    7 ++++---
>  include/asm-x86_64/local.h |    7 ++++---
>  2 files changed, 8 insertions(+), 6 deletions(-)
> 
> Index: linux-2.6/include/asm-i386/local.h
> ===================================================================
> --- linux-2.6.orig/include/asm-i386/local.h
> +++ linux-2.6/include/asm-i386/local.h
> @@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
>  #define __local_begin(__flags)					\
>  {								\
>  	(__flags) = 0;						\
> -	preempt_disable();					\
> +	migrate_disable();					\

Brrrr. That's wrong. Your non atomic __local*() updates only makes sense
when preempt_disable/enable() protects them from concurrent threads on
the same CPU, which is not the case of migrate_disable/enable(). This is
why I suggest that you use local_begin/end() mapped to
migrate_disable/enable() for normal local variables, and, if you really
want a __local_begin/end(), then it should be mapped to
preempt_disable/enable() and should state that it provides no protection
against interrupts.

Mathieu

>  }
>  
> -static inline void __local_end(unsigned long flags) {
> -	preempt_enable();
> +static inline void __local_end(unsigned long flags)
> +{
> +	migrate_enable();
>  }
>  
>  /* On x86, these are no better than the atomic variants. */
> Index: linux-2.6/include/asm-x86_64/local.h
> ===================================================================
> --- linux-2.6.orig/include/asm-x86_64/local.h
> +++ linux-2.6/include/asm-x86_64/local.h
> @@ -186,11 +186,12 @@ static __inline__ long local_sub_return(
>  #define __local_begin(__flags)		\
>  {					\
>  	(__flags) = 0;			\
> -	preempt_disable();		\
> +	migrate_disable();		\
>  }
>  
> -static inline void __local_end(unsigned long flags) {
> -	preempt_enable();
> +static inline void __local_end(unsigned long flags)
> +{
> +	migrate_enable();
>  }
>  
>  #define __local_inc(l)		local_inc(l)
> 
> --
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH -rt 4/5] use migrate_disable for __local_begin

[Peter Zijlstra]

On Sat, 2007-07-14 at 13:16 -0400, Mathieu Desnoyers wrote:
> * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > ---
> >  include/asm-i386/local.h   |    7 ++++---
> >  include/asm-x86_64/local.h |    7 ++++---
> >  2 files changed, 8 insertions(+), 6 deletions(-)
> > 
> > Index: linux-2.6/include/asm-i386/local.h
> > ===================================================================
> > --- linux-2.6.orig/include/asm-i386/local.h
> > +++ linux-2.6/include/asm-i386/local.h
> > @@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
> >  #define __local_begin(__flags)					\
> >  {								\
> >  	(__flags) = 0;						\
> > -	preempt_disable();					\
> > +	migrate_disable();					\
> 
> Brrrr. That's wrong. Your non atomic __local*() updates only makes sense
> when preempt_disable/enable() protects them from concurrent threads on
> the same CPU, which is not the case of migrate_disable/enable(). This is
> why I suggest that you use local_begin/end() mapped to
> migrate_disable/enable() for normal local variables, and, if you really
> want a __local_begin/end(), then it should be mapped to
> preempt_disable/enable() and should state that it provides no protection
> against interrupts.

Sure, but on -rt it does suffice, this part of the patch is rather WIP.

Re: [PATCH -rt 4/5] use migrate_disable for __local_begin

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> On Sat, 2007-07-14 at 13:16 -0400, Mathieu Desnoyers wrote:
> > * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > ---
> > >  include/asm-i386/local.h   |    7 ++++---
> > >  include/asm-x86_64/local.h |    7 ++++---
> > >  2 files changed, 8 insertions(+), 6 deletions(-)
> > > 
> > > Index: linux-2.6/include/asm-i386/local.h
> > > ===================================================================
> > > --- linux-2.6.orig/include/asm-i386/local.h
> > > +++ linux-2.6/include/asm-i386/local.h
> > > @@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
> > >  #define __local_begin(__flags)					\
> > >  {								\
> > >  	(__flags) = 0;						\
> > > -	preempt_disable();					\
> > > +	migrate_disable();					\
> > 
> > Brrrr. That's wrong. Your non atomic __local*() updates only makes sense
> > when preempt_disable/enable() protects them from concurrent threads on
> > the same CPU, which is not the case of migrate_disable/enable(). This is
> > why I suggest that you use local_begin/end() mapped to
> > migrate_disable/enable() for normal local variables, and, if you really
> > want a __local_begin/end(), then it should be mapped to
> > preempt_disable/enable() and should state that it provides no protection
> > against interrupts.
> 
> Sure, but on -rt it does suffice, this part of the patch is rather WIP.
> 
> 

Hrm, how can it suffice, I wonder ? migrate_disable() does not protect
against other threads on the same CPU, so you could suffer from
concurrent updates to the same variables. How is it different in -rt ?

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH -rt 4/5] use migrate_disable for __local_begin

[Peter Zijlstra]

On Sat, 2007-07-14 at 14:35 -0400, Mathieu Desnoyers wrote:
> * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > On Sat, 2007-07-14 at 13:16 -0400, Mathieu Desnoyers wrote:
> > > * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > > > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > > ---
> > > >  include/asm-i386/local.h   |    7 ++++---
> > > >  include/asm-x86_64/local.h |    7 ++++---
> > > >  2 files changed, 8 insertions(+), 6 deletions(-)
> > > > 
> > > > Index: linux-2.6/include/asm-i386/local.h
> > > > ===================================================================
> > > > --- linux-2.6.orig/include/asm-i386/local.h
> > > > +++ linux-2.6/include/asm-i386/local.h
> > > > @@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
> > > >  #define __local_begin(__flags)					\
> > > >  {								\
> > > >  	(__flags) = 0;						\
> > > > -	preempt_disable();					\
> > > > +	migrate_disable();					\
> > > 
> > > Brrrr. That's wrong. Your non atomic __local*() updates only makes sense
> > > when preempt_disable/enable() protects them from concurrent threads on
> > > the same CPU, which is not the case of migrate_disable/enable(). This is
> > > why I suggest that you use local_begin/end() mapped to
> > > migrate_disable/enable() for normal local variables, and, if you really
> > > want a __local_begin/end(), then it should be mapped to
> > > preempt_disable/enable() and should state that it provides no protection
> > > against interrupts.
> > 
> > Sure, but on -rt it does suffice, this part of the patch is rather WIP.
> > 
> > 
> 
> Hrm, how can it suffice, I wonder ? migrate_disable() does not protect
> against other threads on the same CPU, so you could suffer from
> concurrent updates to the same variables. How is it different in -rt ?

I thought the idea was that all these local_* operation were atomic wrt
to the local cpu.

The only difference with -rt is that we generally don't care about
interrupts.

Anyway, I'm dropping all this local stuff, and just hard code it right
into slub.c

Re: [PATCH -rt 4/5] use migrate_disable for __local_begin

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> On Sat, 2007-07-14 at 14:35 -0400, Mathieu Desnoyers wrote:
> > * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > > On Sat, 2007-07-14 at 13:16 -0400, Mathieu Desnoyers wrote:
> > > > * Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> > > > > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > > > ---
> > > > >  include/asm-i386/local.h   |    7 ++++---
> > > > >  include/asm-x86_64/local.h |    7 ++++---
> > > > >  2 files changed, 8 insertions(+), 6 deletions(-)
> > > > > 
> > > > > Index: linux-2.6/include/asm-i386/local.h
> > > > > ===================================================================
> > > > > --- linux-2.6.orig/include/asm-i386/local.h
> > > > > +++ linux-2.6/include/asm-i386/local.h
> > > > > @@ -197,11 +197,12 @@ static __inline__ long local_sub_return(
> > > > >  #define __local_begin(__flags)					\
> > > > >  {								\
> > > > >  	(__flags) = 0;						\
> > > > > -	preempt_disable();					\
> > > > > +	migrate_disable();					\
> > > > 
> > > > Brrrr. That's wrong. Your non atomic __local*() updates only makes sense
> > > > when preempt_disable/enable() protects them from concurrent threads on
> > > > the same CPU, which is not the case of migrate_disable/enable(). This is
> > > > why I suggest that you use local_begin/end() mapped to
> > > > migrate_disable/enable() for normal local variables, and, if you really
> > > > want a __local_begin/end(), then it should be mapped to
> > > > preempt_disable/enable() and should state that it provides no protection
> > > > against interrupts.
> > > 
> > > Sure, but on -rt it does suffice, this part of the patch is rather WIP.
> > > 
> > > 
> > 
> > Hrm, how can it suffice, I wonder ? migrate_disable() does not protect
> > against other threads on the same CPU, so you could suffer from
> > concurrent updates to the same variables. How is it different in -rt ?
> 
> I thought the idea was that all these local_* operation were atomic wrt
> to the local cpu.
> 

Yes, they are. But your __local_*() variants are not. They require a
full preempt_disable because they do not execute atomically.

So having local_begin/end() -> migrate_disable/enable() is ok.
And __local_begin/end() -> preempt_disable/enable() is ok.

I just want to make the naming and usage consistent with the protection
offered.

Mathieu

> The only difference with -rt is that we generally don't care about
> interrupts.
> 
> Anyway, I'm dropping all this local stuff, and just hard code it right
> into slub.c
> 
> 
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH -rt 5/5] slub: -rt port

[Peter Zijlstra]

[-- Attachment #1: slub-rt.patch --]
[-- Type: text/plain, Size: 11696 bytes --]

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 init/Kconfig |    1 
 mm/slub.c    |  260 ++++++++++++++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 214 insertions(+), 47 deletions(-)

Index: linux-2.6/mm/slub.c
===================================================================
--- linux-2.6.orig/mm/slub.c
+++ linux-2.6/mm/slub.c
@@ -20,6 +20,7 @@
 #include <linux/mempolicy.h>
 #include <linux/ctype.h>
 #include <linux/kallsyms.h>
+#include <linux/pagemap.h>
 
 /*
  * Lock order:
@@ -99,6 +100,8 @@
  * 			the fast path and disables lockless freelists.
  */
 
+#ifndef CONFIG_PREEMPT_RT
+
 #define FROZEN (1 << PG_active)
 
 #ifdef CONFIG_SLUB_DEBUG
@@ -137,6 +140,46 @@ static inline void ClearSlabDebug(struct
 	page->flags &= ~SLABDEBUG;
 }
 
+#else /* CONFIG_PREEMPT_RT */
+/*
+ * when the allocator is preemptible these operations might be concurrent with
+ * lock_page(), and hence need atomic ops.
+ */
+
+#define PG_frozen		PG_active
+#define PG_debug		PG_error
+
+static inline int SlabFrozen(struct page *page)
+{
+	return test_bit(PG_frozen, &page->flags);
+}
+
+static inline void SetSlabFrozen(struct page *page)
+{
+	set_bit(PG_frozen, &page->flags);
+}
+
+static inline void ClearSlabFrozen(struct page *page)
+{
+	clear_bit(PG_frozen, &page->flags);
+}
+
+static inline int SlabDebug(struct page *page)
+{
+	return test_bit(PG_debug, &page->flags);
+}
+
+static inline void SetSlabDebug(struct page *page)
+{
+	set_bit(PG_debug, &page->flags);
+}
+
+static inline void ClearSlabDebug(struct page *page)
+{
+	clear_bit(PG_debug, &page->flags);
+}
+#endif
+
 /*
  * Issues still to be resolved:
  *
@@ -1021,7 +1064,7 @@ static struct page *new_slab(struct kmem
 	BUG_ON(flags & ~(GFP_DMA | GFP_LEVEL_MASK));
 
 	if (flags & __GFP_WAIT)
-		local_irq_enable();
+		local_irq_enable_nort();
 
 	page = allocate_slab(s, flags & GFP_LEVEL_MASK, node);
 	if (!page)
@@ -1057,7 +1100,7 @@ static struct page *new_slab(struct kmem
 	page->inuse = 0;
 out:
 	if (flags & __GFP_WAIT)
-		local_irq_disable();
+		local_irq_disable_nort();
 	return page;
 }
 
@@ -1117,6 +1160,7 @@ static void discard_slab(struct kmem_cac
 /*
  * Per slab locking using the pagelock
  */
+#ifndef CONFIG_PREEMPT_RT
 static __always_inline void slab_lock(struct page *page)
 {
 	bit_spin_lock(PG_locked, &page->flags);
@@ -1134,6 +1178,22 @@ static __always_inline int slab_trylock(
 	rc = bit_spin_trylock(PG_locked, &page->flags);
 	return rc;
 }
+#else
+static __always_inline void slab_lock(struct page *page)
+{
+	lock_page(page);
+}
+
+static __always_inline void slab_unlock(struct page *page)
+{
+	unlock_page(page);
+}
+
+static __always_inline int slab_trylock(struct page *page)
+{
+	return !TestSetPageLocked(page);
+}
+#endif
 
 /*
  * Management of partially allocated slabs
@@ -1154,8 +1214,7 @@ static void add_partial(struct kmem_cach
 	spin_unlock(&n->list_lock);
 }
 
-static void remove_partial(struct kmem_cache *s,
-						struct page *page)
+static void remove_partial(struct kmem_cache *s, struct page *page)
 {
 	struct kmem_cache_node *n = get_node(s, page_to_nid(page));
 
@@ -1282,6 +1341,7 @@ static void unfreeze_slab(struct kmem_ca
 {
 	struct kmem_cache_node *n = get_node(s, page_to_nid(page));
 
+	BUG_ON(!SlabFrozen(page));
 	ClearSlabFrozen(page);
 	if (page->inuse) {
 
@@ -1310,29 +1370,52 @@ static void unfreeze_slab(struct kmem_ca
 	}
 }
 
+static void **get_lockless_object(struct page *page)
+{
+	void **object;
+
+again:
+	object = page->lockless_freelist;
+	if (object && __local_cmpxchg(&page->lockless_freelist,
+				object, object[page->offset]) != object)
+		goto again;
+
+	return object;
+}
+
 /*
  * Remove the cpu slab
  */
 static void deactivate_slab(struct kmem_cache *s, struct page *page, int cpu)
 {
 	/*
+	 * take away the slab page before merging the lockless free list into
+	 * the regular free list to ensure that no new entries are put on the
+	 * lockless list between the merge and removal.
+	 */
+	BUG_ON(page != s->cpu_slab[cpu]);
+	s->cpu_slab[cpu] = NULL;
+	barrier();
+
+	/*
 	 * Merge cpu freelist into freelist. Typically we get here
 	 * because both freelists are empty. So this is unlikely
 	 * to occur.
 	 */
-	while (unlikely(page->lockless_freelist)) {
+	for (;;) {
 		void **object;
 
 		/* Retrieve object from cpu_freelist */
-		object = page->lockless_freelist;
-		page->lockless_freelist = page->lockless_freelist[page->offset];
+		object = get_lockless_object(page);
+		if (likely(!object))
+			break;
 
 		/* And put onto the regular freelist */
 		object[page->offset] = page->freelist;
 		page->freelist = object;
 		page->inuse--;
 	}
-	s->cpu_slab[cpu] = NULL;
+
 	unfreeze_slab(s, page);
 }
 
@@ -1354,6 +1437,55 @@ static void __flush_cpu_slab(struct kmem
 		flush_slab(s, page, cpu);
 }
 
+#ifdef CONFIG_PREEMPT_RT
+struct slab_work_struct {
+	struct work_struct work;
+	struct kmem_cache *s;
+};
+
+static struct workqueue_struct *flush_slab_workqueue;
+static DEFINE_PER_CPU(struct slab_work_struct, slab_works);
+static DEFINE_MUTEX(flush_slab_mutex); /* XXX kill this */
+
+static int __init flush_cpu_slab_init(void)
+{
+	flush_slab_workqueue = create_workqueue("slub_flushd");
+	if (!flush_slab_workqueue)
+		panic("Failed to create slub_flushd\n");
+
+	return 0;
+}
+
+core_initcall(flush_cpu_slab_init);
+
+static void flush_cpu_slab_wq(struct work_struct *work)
+{
+	struct slab_work_struct *sw;
+	int cpu = smp_processor_id();
+
+	sw = container_of(work, struct slab_work_struct, work);
+	__flush_cpu_slab(sw->s, cpu);
+}
+
+static void flush_all(struct kmem_cache *s)
+{
+	int cpu;
+	struct workqueue_struct *wq = flush_slab_workqueue;
+
+	mutex_lock(&flush_slab_mutex);
+	for_each_online_cpu(cpu) {
+		struct slab_work_struct *sw = &per_cpu(slab_works, cpu);
+
+		INIT_WORK(&sw->work, flush_cpu_slab_wq);
+		sw->s = s;
+		queue_work_cpu(wq, &sw->work, cpu);
+	}
+	flush_workqueue(wq);
+	mutex_unlock(&flush_slab_mutex);
+}
+
+#else
+
 static void flush_cpu_slab(void *d)
 {
 	struct kmem_cache *s = d;
@@ -1374,6 +1506,7 @@ static void flush_all(struct kmem_cache 
 	local_irq_restore(flags);
 #endif
 }
+#endif
 
 /*
  * Slow path. The lockless freelist is empty or we need to perform
@@ -1396,13 +1529,24 @@ static void *__slab_alloc(struct kmem_ca
 		gfp_t gfpflags, int node, void *addr, struct page *page)
 {
 	void **object;
+	unsigned long flags;
 	int cpu = smp_processor_id();
 
+	local_irq_save_nort(flags);
+
+again:
 	if (!page)
 		goto new_slab;
 
 	slab_lock(page);
-	if (unlikely(node != -1 && page_to_nid(page) != node))
+	if (!SlabFrozen(page) || page != s->cpu_slab[cpu]) {
+		slab_unlock(page);
+		page = s->cpu_slab[cpu];
+		goto again;
+	}
+
+	if (unlikely((node != -1 && page_to_nid(page) != node) ||
+			page->lockless_freelist))  /* validate the need for this check */
 		goto another_slab;
 load_freelist:
 	object = page->freelist;
@@ -1415,7 +1559,9 @@ load_freelist:
 	page->lockless_freelist = object[page->offset];
 	page->inuse = s->objects;
 	page->freelist = NULL;
+out:
 	slab_unlock(page);
+	local_irq_restore_nort(flags);
 	return object;
 
 another_slab:
@@ -1424,40 +1570,42 @@ another_slab:
 new_slab:
 	page = get_partial(s, gfpflags, node);
 	if (page) {
-		s->cpu_slab[cpu] = page;
+		struct page *cur_page;
+
+		cur_page = __local_cmpxchg(&s->cpu_slab[cpu], NULL, page);
+		if (cur_page) {
+			/*
+			 * Someone else populated the cpu_slab while we got
+			 * preempted. We want the current one since its cache
+			 * hot
+			 */
+			unfreeze_slab(s, page);
+			page = cur_page;
+			goto again;
+		}
 		goto load_freelist;
 	}
 
 	page = new_slab(s, gfpflags, node);
 	if (page) {
-		cpu = smp_processor_id();
-		if (s->cpu_slab[cpu]) {
+		struct page *cur_page;
+
+		slab_lock(page);
+		SetSlabFrozen(page);
+		cur_page = __local_cmpxchg(&s->cpu_slab[cpu], NULL, page);
+		if (cur_page) {
 			/*
-			 * Someone else populated the cpu_slab while we
-			 * enabled interrupts, or we have gotten scheduled
-			 * on another cpu. The page may not be on the
-			 * requested node even if __GFP_THISNODE was
-			 * specified. So we need to recheck.
+			 * Someone else populated the cpu_slab while we got
+			 * preempted. We want the current one since its cache
+			 * hot
 			 */
-			if (node == -1 ||
-				page_to_nid(s->cpu_slab[cpu]) == node) {
-				/*
-				 * Current cpuslab is acceptable and we
-				 * want the current one since its cache hot
-				 */
-				discard_slab(s, page);
-				page = s->cpu_slab[cpu];
-				slab_lock(page);
-				goto load_freelist;
-			}
-			/* New slab does not fit our expectations */
-			flush_slab(s, s->cpu_slab[cpu], cpu);
+			unfreeze_slab(s, page);
+			page = cur_page;
+			goto again;
 		}
-		slab_lock(page);
-		SetSlabFrozen(page);
-		s->cpu_slab[cpu] = page;
 		goto load_freelist;
 	}
+	local_irq_restore_nort(flags);
 	return NULL;
 debug:
 	object = page->freelist;
@@ -1466,8 +1614,7 @@ debug:
 
 	page->inuse++;
 	page->freelist = object[page->offset];
-	slab_unlock(page);
-	return object;
+	goto out;
 }
 
 /*
@@ -1487,18 +1634,20 @@ static void __always_inline *slab_alloc(
 	void **object;
 	unsigned long flags;
 
-	local_irq_save(flags);
+	__local_begin(flags);
 	page = s->cpu_slab[smp_processor_id()];
 	if (unlikely(!page || !page->lockless_freelist ||
-			(node != -1 && page_to_nid(page) != node)))
+			(node != -1 && page_to_nid(page) != node))) {
 
+do_alloc:
 		object = __slab_alloc(s, gfpflags, node, addr, page);
 
-	else {
-		object = page->lockless_freelist;
-		page->lockless_freelist = object[page->offset];
+	} else {
+		object = get_lockless_object(page);
+		if (unlikely(!object))
+			goto do_alloc;
 	}
-	local_irq_restore(flags);
+	__local_end(flags);
 	return object;
 }
 
@@ -1529,7 +1678,9 @@ static void __slab_free(struct kmem_cach
 {
 	void *prior;
 	void **object = (void *)x;
+	unsigned long flags;
 
+	local_irq_save_nort(flags);
 	slab_lock(page);
 
 	if (unlikely(SlabDebug(page)))
@@ -1555,6 +1706,7 @@ checks_ok:
 
 out_unlock:
 	slab_unlock(page);
+	local_irq_restore_nort(flags);
 	return;
 
 slab_empty:
@@ -1566,6 +1718,7 @@ slab_empty:
 
 	slab_unlock(page);
 	discard_slab(s, page);
+	local_irq_restore_nort(flags);
 	return;
 
 debug:
@@ -1591,15 +1744,30 @@ static void __always_inline slab_free(st
 	void **object = (void *)x;
 	unsigned long flags;
 
-	local_irq_save(flags);
+	__local_begin(flags);
+	/*
+	 * We have to either take slab_lock(page) or disable preemption while
+	 * trying to add to the lockless freelist because we have to guarantee
+	 * page == s->cpu_slab[cpu] during the operation.
+	 *
+	 * fix this by allowing non active slabs to have a lockless_freelist?
+	 * cannot do since Christoph is about to pull lockless_freelist from
+	 * the struct page.
+	 *
+	 * preempt_disable() seems cheapest for these few instructions vs the
+	 * atomic ops involved with slab_lock()
+	 */
+	preempt_disable();
 	if (likely(page == s->cpu_slab[smp_processor_id()] &&
-						!SlabDebug(page))) {
+				!SlabDebug(page))) {
 		object[page->offset] = page->lockless_freelist;
 		page->lockless_freelist = object;
-	} else
+		preempt_enable();
+	} else {
+		preempt_enable();
 		__slab_free(s, page, x, addr);
-
-	local_irq_restore(flags);
+	}
+	__local_end(flags);
 }
 
 void kmem_cache_free(struct kmem_cache *s, void *x)
Index: linux-2.6/init/Kconfig
===================================================================
--- linux-2.6.orig/init/Kconfig
+++ linux-2.6/init/Kconfig
@@ -578,7 +578,6 @@ config SLAB
 
 config SLUB
 	bool "SLUB (Unqueued Allocator)"
-	depends on !PREEMPT_RT
 	help
 	   SLUB is a slab allocator that minimizes cache line usage
 	   instead of managing queues of cached objects (SLAB approach).

--

Re: [PATCH -rt 5/5] slub: -rt port

[Oleg Nesterov]

On 07/14, Peter Zijlstra wrote:
>
> +static void flush_all(struct kmem_cache *s)
> +{
> +	int cpu;
> +	struct workqueue_struct *wq = flush_slab_workqueue;
> +
> +	mutex_lock(&flush_slab_mutex);
> +	for_each_online_cpu(cpu) {
> +		struct slab_work_struct *sw = &per_cpu(slab_works, cpu);
> +
> +		INIT_WORK(&sw->work, flush_cpu_slab_wq);
> +		sw->s = s;
> +		queue_work_cpu(wq, &sw->work, cpu);
> +	}
> +	flush_workqueue(wq);
> +	mutex_unlock(&flush_slab_mutex);
> +}

I suspect this is not cpu-hotplug safe. flush_slab_mutex doesn't protect
from cpu_down(). This means that slab_work_struct could be scheduled on
the already dead CPU. flush_workqueue(wq) will hang in that case.

Oleg.

Re: [PATCH -rt 5/5] slub: -rt port

[Peter Zijlstra]

On Sat, 2007-07-14 at 21:39 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:
> >
> > +static void flush_all(struct kmem_cache *s)
> > +{
> > +	int cpu;
> > +	struct workqueue_struct *wq = flush_slab_workqueue;
> > +
> > +	mutex_lock(&flush_slab_mutex);
> > +	for_each_online_cpu(cpu) {
> > +		struct slab_work_struct *sw = &per_cpu(slab_works, cpu);
> > +
> > +		INIT_WORK(&sw->work, flush_cpu_slab_wq);
> > +		sw->s = s;
> > +		queue_work_cpu(wq, &sw->work, cpu);
> > +	}
> > +	flush_workqueue(wq);
> > +	mutex_unlock(&flush_slab_mutex);
> > +}
> 
> I suspect this is not cpu-hotplug safe. flush_slab_mutex doesn't protect
> from cpu_down(). This means that slab_work_struct could be scheduled on
> the already dead CPU. flush_workqueue(wq) will hang in that case.

Yeah, the function I copied this from: schedule_on_each_cpu() has a
comment to that effect.

Any ideas on how to solve this?

Re: [PATCH -rt 5/5] slub: -rt port

[Oleg Nesterov]

On 07/14, Peter Zijlstra wrote:
>
> On Sat, 2007-07-14 at 21:39 +0400, Oleg Nesterov wrote:
> > On 07/14, Peter Zijlstra wrote:
> > >
> > > +static void flush_all(struct kmem_cache *s)
> > > +{
> > > +	int cpu;
> > > +	struct workqueue_struct *wq = flush_slab_workqueue;
> > > +
> > > +	mutex_lock(&flush_slab_mutex);
> > > +	for_each_online_cpu(cpu) {
> > > +		struct slab_work_struct *sw = &per_cpu(slab_works, cpu);
> > > +
> > > +		INIT_WORK(&sw->work, flush_cpu_slab_wq);
> > > +		sw->s = s;
> > > +		queue_work_cpu(wq, &sw->work, cpu);
> > > +	}
> > > +	flush_workqueue(wq);
> > > +	mutex_unlock(&flush_slab_mutex);
> > > +}
> > 
> > I suspect this is not cpu-hotplug safe. flush_slab_mutex doesn't protect
> > from cpu_down(). This means that slab_work_struct could be scheduled on
> > the already dead CPU. flush_workqueue(wq) will hang in that case.
> 
> Yeah, the function I copied this from: schedule_on_each_cpu() has a
> comment to that effect.

Just in case, schedule_on_each_cpu() ptotects cpu_online_map with
preempt_disable(), its problem is quite different.

> Any ideas on how to solve this?

Perhaps slab_cpuup_callback() can take flush_slab_mutex too, in that
case cpu_online_map will be stable under flush_slab_mutex.

Oleg.

Re: [PATCH -rt 5/5] slub: -rt port

[Peter Zijlstra]

On Sat, 2007-07-14 at 23:38 +0400, Oleg Nesterov wrote:
> On 07/14, Peter Zijlstra wrote:

> > Yeah, the function I copied this from: schedule_on_each_cpu() has a
> > comment to that effect.
> 
> Just in case, schedule_on_each_cpu() ptotects cpu_online_map with
> preempt_disable(), its problem is quite different.


I was schedule_on_each_cpu_wq() I copied, and that does not disable
preemption. Also, I'm failing to see how schedule_on_each_cpu() could
work with -rt, since __queue_work() takes a regular spinlock.

> > Any ideas on how to solve this?
> 
> Perhaps slab_cpuup_callback() can take flush_slab_mutex too, in that
> case cpu_online_map will be stable under flush_slab_mutex.

Right, I'll give that a try.