[ANNOUNCE] 3.14.2-rt3

[ANNOUNCE] 3.14.2-rt3

[Sebastian Andrzej Siewior]

Dear RT folks!

I'm pleased to announce the v3.14.2-rt3 patch set.

Changes since v3.14.2-rt2
- rwsem readers are now not allowed to nest. A patch rom Steven Rostedt.
- a few bugs were fixed in the hotplug code which were made during the
  v3.14 port. Fixed by Mike Galbraith.
- Mike Galbraith sent a patch which might fix lazy preempt on x86_64.
  Patch applied and my machine still explodes therefore lazy preempt
  remains off on x86_64.
- Mike Galbraith sent a few patches to get cpu hoplug to work. This
  includes lg_global_trylock_relax().
- A few push downs of migrate_disable() (where we call migrate_disable()
  after the rt_mutex_trylock()) have been reverted. It seems hotplug is
  not too happy about this. A patch by Steven Rostedt and and Mike
  Galbraith
- There was a complaint about a backrace from run_local_timers() in UP
  mode because a spin_try_lock() failed. _This_ particular case was not
  an error. This optimization was for FULL_NO_HZ which is pointless on
  UP because there is no spare CPU. Therefore, this optimization is
  disabled in UP mode and the backtrace is gone. Reported by Stanislav
  Meduna.
- block-mq notifier uses now a spinlock and runs during CPU_POST_DEAD
  instead at CPU_DEAD time. lockdep complained about the sleeping
  ctx->lock within the rawlock (blk_mq_cpu_notify_lock) and CPU_DEAD
  runs with irqs off.

Known issues:

      - bcache is disabled.

      - lazy preempt on x86_64 leads to a crash with some load.

      - CPU hotplug works in general. Steven's test script however
        deadlocks usually on the second invocation.

The delta patch against v3.14.2-rt2 is appended below and can be found
here:
   https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/incr/patch-3.14.2-rt2-rt3.patch.xz

The RT patch against 3.14.2 can be found here:

   https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patch-3.14.2-rt3.patch.xz

The split quilt queue is available at:

   https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patches-3.14.2-rt3.tar.xz

Sebastian

diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h
index 752fe56..1e649c4 100644
--- a/arch/x86/include/asm/preempt.h
+++ b/arch/x86/include/asm/preempt.h
@@ -94,7 +94,11 @@ static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	if (____preempt_count_dec_and_test())
 		return true;
+#ifdef CONFIG_PREEMPT_LAZY
 	return test_thread_flag(TIF_NEED_RESCHED_LAZY);
+#else
+	return false;
+#endif
 }
 
 /*
@@ -102,8 +106,12 @@ static __always_inline bool __preempt_count_dec_and_test(void)
  */
 static __always_inline bool should_resched(void)
 {
+#ifdef CONFIG_PREEMPT_LAZY
 	return unlikely(!__this_cpu_read_4(__preempt_count) || \
 			test_thread_flag(TIF_NEED_RESCHED_LAZY));
+#else
+	return unlikely(!__this_cpu_read_4(__preempt_count));
+#endif
 }
 
 #ifdef CONFIG_PREEMPT
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 7c8b356..5701b50 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -72,4 +72,5 @@ void common(void) {
 
 	BLANK();
 	DEFINE(PTREGS_SIZE, sizeof(struct pt_regs));
+	DEFINE(_PREEMPT_ENABLED, PREEMPT_ENABLED);
 }
diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
index fd2d976..6157ed6 100644
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -365,19 +365,22 @@ ENTRY(resume_kernel)
 need_resched:
 	# preempt count == 0 + NEED_RS set?
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
+	jnz restore_all
+#else
 	jz test_int_off
 
 	# atleast preempt count == 0 ?
-	cmpl $_TIF_NEED_RESCHED,PER_CPU_VAR(__preempt_count)
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
 	jne restore_all
 
 	cmpl $0,TI_preempt_lazy_count(%ebp)	# non-zero preempt_lazy_count ?
 	jnz restore_all
 
-	testl $_TIF_NEED_RESCHED_LAZY, %ecx
+	testl $_TIF_NEED_RESCHED_LAZY, TI_flags(%ebp)
 	jz restore_all
-
 test_int_off:
+#endif
 	testl $X86_EFLAGS_IF,PT_EFLAGS(%esp)	# interrupts off (exception path) ?
 	jz restore_all
 	call preempt_schedule_irq
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index b650b43..d893814 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -658,8 +658,8 @@ GLOBAL(system_call_after_swapgs)
 	/* Handle reschedules */
 	/* edx:	work, edi: workmask */
 sysret_careful:
-	bt $TIF_NEED_RESCHED,%edx
-	jnc sysret_signal
+	testl $_TIF_NEED_RESCHED_MASK,%edx
+	jz sysret_signal
 	TRACE_IRQS_ON
 	ENABLE_INTERRUPTS(CLBR_NONE)
 	pushq_cfi %rdi
@@ -771,8 +771,8 @@ GLOBAL(int_with_check)
 	/* First do a reschedule test. */
 	/* edx:	work, edi: workmask */
 int_careful:
-	bt $TIF_NEED_RESCHED,%edx
-	jnc  int_very_careful
+	testl $_TIF_NEED_RESCHED_MASK,%edx
+	jz  int_very_careful
 	TRACE_IRQS_ON
 	ENABLE_INTERRUPTS(CLBR_NONE)
 	pushq_cfi %rdi
@@ -1071,8 +1071,8 @@ ENTRY(native_iret)
 	/* edi: workmask, edx: work */
 retint_careful:
 	CFI_RESTORE_STATE
-	bt    $TIF_NEED_RESCHED,%edx
-	jnc   retint_signal
+	testl $_TIF_NEED_RESCHED_MASK,%edx
+	jz   retint_signal
 	TRACE_IRQS_ON
 	ENABLE_INTERRUPTS(CLBR_NONE)
 	pushq_cfi %rdi
@@ -1104,7 +1104,22 @@ ENTRY(native_iret)
 	/* rcx:	 threadinfo. interrupts off. */
 ENTRY(retint_kernel)
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
 	jnz  retint_restore_args
+#else
+	jz  check_int_off
+
+	# atleast preempt count == 0 ?
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
+	jnz retint_restore_args
+
+	cmpl $0, TI_preempt_lazy_count(%rcx)
+	jnz retint_restore_args
+
+	bt $TIF_NEED_RESCHED_LAZY,TI_flags(%rcx)
+	jnc  retint_restore_args
+check_int_off:
+#endif
 	bt   $9,EFLAGS-ARGOFFSET(%rsp)	/* interrupts off? */
 	jnc  retint_restore_args
 	call preempt_schedule_irq
@@ -1540,7 +1555,7 @@ ENTRY(paranoid_exit)
 	movq %rsp,%rdi			/* &pt_regs */
 	call sync_regs
 	movq %rax,%rsp			/* switch stack for scheduling */
-	testl $_TIF_NEED_RESCHED,%ebx
+	testl $_TIF_NEED_RESCHED_MASK,%ebx
 	jnz paranoid_schedule
 	movl %ebx,%edx			/* arg3: thread flags */
 	TRACE_IRQS_ON
diff --git a/block/blk-mq-cpu.c b/block/blk-mq-cpu.c
index 136ef86..37acc3a 100644
--- a/block/blk-mq-cpu.c
+++ b/block/blk-mq-cpu.c
@@ -11,7 +11,7 @@
 #include "blk-mq.h"
 
 static LIST_HEAD(blk_mq_cpu_notify_list);
-static DEFINE_RAW_SPINLOCK(blk_mq_cpu_notify_lock);
+static DEFINE_SPINLOCK(blk_mq_cpu_notify_lock);
 
 static int blk_mq_main_cpu_notify(struct notifier_block *self,
 				  unsigned long action, void *hcpu)
@@ -19,12 +19,15 @@ static int blk_mq_main_cpu_notify(struct notifier_block *self,
 	unsigned int cpu = (unsigned long) hcpu;
 	struct blk_mq_cpu_notifier *notify;
 
-	raw_spin_lock(&blk_mq_cpu_notify_lock);
+	if (action != CPU_POST_DEAD && action != CPU_POST_DEAD)
+		return NOTIFY_OK;
+
+	spin_lock(&blk_mq_cpu_notify_lock);
 
 	list_for_each_entry(notify, &blk_mq_cpu_notify_list, list)
 		notify->notify(notify->data, action, cpu);
 
-	raw_spin_unlock(&blk_mq_cpu_notify_lock);
+	spin_unlock(&blk_mq_cpu_notify_lock);
 	return NOTIFY_OK;
 }
 
@@ -32,16 +35,16 @@ void blk_mq_register_cpu_notifier(struct blk_mq_cpu_notifier *notifier)
 {
 	BUG_ON(!notifier->notify);
 
-	raw_spin_lock(&blk_mq_cpu_notify_lock);
+	spin_lock(&blk_mq_cpu_notify_lock);
 	list_add_tail(&notifier->list, &blk_mq_cpu_notify_list);
-	raw_spin_unlock(&blk_mq_cpu_notify_lock);
+	spin_unlock(&blk_mq_cpu_notify_lock);
 }
 
 void blk_mq_unregister_cpu_notifier(struct blk_mq_cpu_notifier *notifier)
 {
-	raw_spin_lock(&blk_mq_cpu_notify_lock);
+	spin_lock(&blk_mq_cpu_notify_lock);
 	list_del(&notifier->list);
-	raw_spin_unlock(&blk_mq_cpu_notify_lock);
+	spin_unlock(&blk_mq_cpu_notify_lock);
 }
 
 void blk_mq_init_cpu_notifier(struct blk_mq_cpu_notifier *notifier,
diff --git a/block/blk-mq.c b/block/blk-mq.c
index a5f25f9..5fb26f7 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -48,9 +48,14 @@ static struct blk_mq_ctx *blk_mq_get_ctx(struct request_queue *q)
 	return __blk_mq_get_ctx(q, get_cpu_light());
 }
 
-static void blk_mq_put_ctx(struct blk_mq_ctx *ctx)
+static void __blk_mq_put_ctx(struct blk_mq_ctx *ctx)
 {
 	spin_unlock(&ctx->cpu_lock);
+}
+
+static void blk_mq_put_ctx(struct blk_mq_ctx *ctx)
+{
+	__blk_mq_put_ctx(ctx);
 	put_cpu_light();
 }
 
@@ -966,7 +971,7 @@ static void blk_mq_hctx_notify(void *data, unsigned long action,
 	struct blk_mq_ctx *ctx;
 	LIST_HEAD(tmp);
 
-	if (action != CPU_DEAD && action != CPU_DEAD_FROZEN)
+	if (action != CPU_POST_DEAD && action != CPU_POST_DEAD)
 		return;
 
 	/*
@@ -980,6 +985,7 @@ static void blk_mq_hctx_notify(void *data, unsigned long action,
 		clear_bit(ctx->index_hw, hctx->ctx_map);
 	}
 	spin_unlock(&ctx->lock);
+	__blk_mq_put_ctx(ctx);
 
 	if (list_empty(&tmp))
 		return;
diff --git a/include/linux/lglock.h b/include/linux/lglock.h
index 2b2204e..534b16e 100644
--- a/include/linux/lglock.h
+++ b/include/linux/lglock.h
@@ -74,4 +74,10 @@ void lg_local_unlock_cpu(struct lglock *lg, int cpu);
 void lg_global_lock(struct lglock *lg);
 void lg_global_unlock(struct lglock *lg);
 
+#ifndef CONFIG_PREEMPT_RT_FULL
+#define lg_global_trylock_relax(name)	lg_global_lock(name)
+#else
+void lg_global_trylock_relax(struct lglock *lg);
+#endif
+
 #endif
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 116af6a..5b2cdf4 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -126,8 +126,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	if (unlikely(__preempt_count_dec_and_test() || \
-				test_thread_flag(TIF_NEED_RESCHED_LAZY))) \
+	if (unlikely(__preempt_count_dec_and_test())) \
 		__preempt_schedule_context(); \
 } while (0)
 #else
diff --git a/include/linux/rwsem_rt.h b/include/linux/rwsem_rt.h
index 924c2d2..0065b08 100644
--- a/include/linux/rwsem_rt.h
+++ b/include/linux/rwsem_rt.h
@@ -20,7 +20,6 @@
 
 struct rw_semaphore {
 	struct rt_mutex		lock;
-	int			read_depth;
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 	struct lockdep_map	dep_map;
 #endif
diff --git a/include/linux/spinlock_rt.h b/include/linux/spinlock_rt.h
index ac6f08b..c0d1367 100644
--- a/include/linux/spinlock_rt.h
+++ b/include/linux/spinlock_rt.h
@@ -35,6 +35,7 @@ extern int atomic_dec_and_spin_lock(atomic_t *atomic, spinlock_t *lock);
  */
 extern void __lockfunc __rt_spin_lock(struct rt_mutex *lock);
 extern void __lockfunc __rt_spin_unlock(struct rt_mutex *lock);
+extern int __lockfunc __rt_spin_trylock(struct rt_mutex *lock);
 
 #define spin_lock(lock)				\
 	do {					\
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 041fada..ce00329 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -649,7 +649,7 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen)
 		/* CPU didn't die: tell everyone.  Can't complain. */
 		smpboot_unpark_threads(cpu);
 		cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
-		goto out_cancel;
+		goto out_release;
 	}
 	BUG_ON(cpu_online(cpu));
 
diff --git a/kernel/locking/lglock.c b/kernel/locking/lglock.c
index f2356df..9397974 100644
--- a/kernel/locking/lglock.c
+++ b/kernel/locking/lglock.c
@@ -105,3 +105,28 @@ void lg_global_unlock(struct lglock *lg)
 	preempt_enable_nort();
 }
 EXPORT_SYMBOL(lg_global_unlock);
+
+#ifdef CONFIG_PREEMPT_RT_FULL
+/*
+ * HACK: If you use this, you get to keep the pieces.
+ * Used in queue_stop_cpus_work() when stop machinery
+ * is called from inactive CPU, so we can't schedule.
+ */
+# define lg_do_trylock_relax(l)			\
+	do {					\
+		while (!__rt_spin_trylock(l))	\
+			cpu_relax();		\
+	} while (0)
+
+void lg_global_trylock_relax(struct lglock *lg)
+{
+	int i;
+
+	lock_acquire_exclusive(&lg->lock_dep_map, 0, 0, NULL, _RET_IP_);
+	for_each_possible_cpu(i) {
+		lg_lock_ptr *lock;
+		lock = per_cpu_ptr(lg->lock, i);
+		lg_do_trylock_relax(lock);
+	}
+}
+#endif
diff --git a/kernel/locking/rt.c b/kernel/locking/rt.c
index 5d17727..055a3df 100644
--- a/kernel/locking/rt.c
+++ b/kernel/locking/rt.c
@@ -180,12 +180,14 @@ EXPORT_SYMBOL(_mutex_unlock);
  */
 int __lockfunc rt_write_trylock(rwlock_t *rwlock)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	int ret;
 
-	if (ret) {
+	migrate_disable();
+	ret = rt_mutex_trylock(&rwlock->lock);
+	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-		migrate_disable();
-	}
+	else
+		migrate_enable();
 
 	return ret;
 }
@@ -212,11 +214,13 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
 	 * write locked.
 	 */
 	if (rt_mutex_owner(lock) != current) {
+		migrate_disable();
 		ret = rt_mutex_trylock(lock);
-		if (ret) {
+		if (ret)
 			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-			migrate_disable();
-		}
+		else
+			migrate_enable();
+
 	} else if (!rwlock->read_depth) {
 		ret = 0;
 	}
@@ -240,13 +244,14 @@ void __lockfunc rt_read_lock(rwlock_t *rwlock)
 {
 	struct rt_mutex *lock = &rwlock->lock;
 
+
 	/*
 	 * recursive read locks succeed when current owns the lock
 	 */
 	if (rt_mutex_owner(lock) != current) {
-		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
-		__rt_spin_lock(lock);
 		migrate_disable();
+		rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
+		__rt_spin_lock(lock);
 	}
 	rwlock->read_depth++;
 }
@@ -316,10 +321,8 @@ EXPORT_SYMBOL(rt_up_write);
 
 void  rt_up_read(struct rw_semaphore *rwsem)
 {
-	if (--rwsem->read_depth == 0) {
-		rwsem_release(&rwsem->dep_map, 1, _RET_IP_);
-		rt_mutex_unlock(&rwsem->lock);
-	}
+	rwsem_release(&rwsem->dep_map, 1, _RET_IP_);
+	rt_mutex_unlock(&rwsem->lock);
 }
 EXPORT_SYMBOL(rt_up_read);
 
@@ -330,7 +333,6 @@ EXPORT_SYMBOL(rt_up_read);
 void  rt_downgrade_write(struct rw_semaphore *rwsem)
 {
 	BUG_ON(rt_mutex_owner(&rwsem->lock) != current);
-	rwsem->read_depth = 1;
 }
 EXPORT_SYMBOL(rt_downgrade_write);
 
@@ -367,37 +369,20 @@ void rt_down_write_nested_lock(struct rw_semaphore *rwsem,
 
 int  rt_down_read_trylock(struct rw_semaphore *rwsem)
 {
-	struct rt_mutex *lock = &rwsem->lock;
-	int ret = 1;
-
-	/*
-	 * recursive read locks succeed when current owns the rwsem,
-	 * but not when read_depth == 0 which means that the rwsem is
-	 * write locked.
-	 */
-	if (rt_mutex_owner(lock) != current) {
-		ret = rt_mutex_trylock(&rwsem->lock);
-		if (ret)
-			rwsem_acquire(&rwsem->dep_map, 0, 1, _RET_IP_);
-	} else if (!rwsem->read_depth) {
-		ret = 0;
-	}
+	int ret;
 
+	ret = rt_mutex_trylock(&rwsem->lock);
 	if (ret)
-		rwsem->read_depth++;
+		rwsem_acquire(&rwsem->dep_map, 0, 1, _RET_IP_);
+
 	return ret;
 }
 EXPORT_SYMBOL(rt_down_read_trylock);
 
 static void __rt_down_read(struct rw_semaphore *rwsem, int subclass)
 {
-	struct rt_mutex *lock = &rwsem->lock;
-
-	if (rt_mutex_owner(lock) != current) {
-		rwsem_acquire(&rwsem->dep_map, subclass, 0, _RET_IP_);
-		rt_mutex_lock(&rwsem->lock);
-	}
-	rwsem->read_depth++;
+	rwsem_acquire(&rwsem->dep_map, subclass, 0, _RET_IP_);
+	rt_mutex_lock(&rwsem->lock);
 }
 
 void  rt_down_read(struct rw_semaphore *rwsem)
@@ -422,7 +407,6 @@ void  __rt_rwsem_init(struct rw_semaphore *rwsem, const char *name,
 	debug_check_no_locks_freed((void *)rwsem, sizeof(*rwsem));
 	lockdep_init_map(&rwsem->dep_map, name, key, 0);
 #endif
-	rwsem->read_depth = 0;
 	rwsem->lock.save_state = 0;
 }
 EXPORT_SYMBOL(__rt_rwsem_init);
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 42f4f28..5c5cc76 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1001,6 +1001,11 @@ void __lockfunc rt_spin_unlock_wait(spinlock_t *lock)
 }
 EXPORT_SYMBOL(rt_spin_unlock_wait);
 
+int __lockfunc __rt_spin_trylock(struct rt_mutex *lock)
+{
+	return rt_mutex_trylock(lock);
+}
+
 int __lockfunc rt_spin_trylock(spinlock_t *lock)
 {
 	int ret = rt_mutex_trylock(&lock->lock);
@@ -1045,12 +1050,12 @@ int atomic_dec_and_spin_lock(atomic_t *atomic, spinlock_t *lock)
 	/* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
 	if (atomic_add_unless(atomic, -1, 1))
 		return 0;
+	migrate_disable();
 	rt_spin_lock(lock);
-	if (atomic_dec_and_test(atomic)){
-		migrate_disable();
+	if (atomic_dec_and_test(atomic))
 		return 1;
-	}
 	rt_spin_unlock(lock);
+	migrate_enable();
 	return 0;
 }
 EXPORT_SYMBOL(atomic_dec_and_spin_lock);
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index aaae9f1..bcbae9c 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -266,7 +266,7 @@ int stop_two_cpus(unsigned int cpu1, unsigned int cpu2, cpu_stop_fn_t fn, void *
 	struct irq_cpu_stop_queue_work_info call_args;
 	struct multi_stop_data msdata;
 
-	preempt_disable();
+	preempt_disable_nort();
 	msdata = (struct multi_stop_data){
 		.fn = fn,
 		.data = arg,
@@ -299,7 +299,7 @@ int stop_two_cpus(unsigned int cpu1, unsigned int cpu2, cpu_stop_fn_t fn, void *
 	 * This relies on the stopper workqueues to be FIFO.
 	 */
 	if (!cpu_active(cpu1) || !cpu_active(cpu2)) {
-		preempt_enable();
+		preempt_enable_nort();
 		return -ENOENT;
 	}
 
@@ -313,7 +313,7 @@ int stop_two_cpus(unsigned int cpu1, unsigned int cpu2, cpu_stop_fn_t fn, void *
 				 &irq_cpu_stop_queue_work,
 				 &call_args, 1);
 	lg_local_unlock(&stop_cpus_lock);
-	preempt_enable();
+	preempt_enable_nort();
 
 	wait_for_stop_done(&done);
 
@@ -346,7 +346,7 @@ static DEFINE_PER_CPU(struct cpu_stop_work, stop_cpus_work);
 
 static void queue_stop_cpus_work(const struct cpumask *cpumask,
 				 cpu_stop_fn_t fn, void *arg,
-				 struct cpu_stop_done *done)
+				 struct cpu_stop_done *done, bool inactive)
 {
 	struct cpu_stop_work *work;
 	unsigned int cpu;
@@ -360,11 +360,13 @@ static void queue_stop_cpus_work(const struct cpumask *cpumask,
 	}
 
 	/*
-	 * Disable preemption while queueing to avoid getting
-	 * preempted by a stopper which might wait for other stoppers
-	 * to enter @fn which can lead to deadlock.
+	 * Make sure that all work is queued on all cpus before
+	 * any of the cpus can execute it.
 	 */
-	lg_global_lock(&stop_cpus_lock);
+	if (!inactive)
+		lg_global_lock(&stop_cpus_lock);
+	else
+		lg_global_trylock_relax(&stop_cpus_lock);
 	for_each_cpu(cpu, cpumask)
 		cpu_stop_queue_work(cpu, &per_cpu(stop_cpus_work, cpu));
 	lg_global_unlock(&stop_cpus_lock);
@@ -376,7 +378,7 @@ static int __stop_cpus(const struct cpumask *cpumask,
 	struct cpu_stop_done done;
 
 	cpu_stop_init_done(&done, cpumask_weight(cpumask));
-	queue_stop_cpus_work(cpumask, fn, arg, &done);
+	queue_stop_cpus_work(cpumask, fn, arg, &done, false);
 	wait_for_stop_done(&done);
 	return done.executed ? done.ret : -ENOENT;
 }
@@ -572,6 +574,8 @@ static int __init cpu_stop_init(void)
 		INIT_LIST_HEAD(&stopper->works);
 	}
 
+	lg_lock_init(&stop_cpus_lock, "stop_cpus_lock");
+
 	BUG_ON(smpboot_register_percpu_thread(&cpu_stop_threads));
 	stop_machine_initialized = true;
 	return 0;
@@ -667,11 +671,11 @@ int stop_machine_from_inactive_cpu(int (*fn)(void *), void *data,
 	set_state(&msdata, MULTI_STOP_PREPARE);
 	cpu_stop_init_done(&done, num_active_cpus());
 	queue_stop_cpus_work(cpu_active_mask, multi_cpu_stop, &msdata,
-			     &done);
+			     &done, true);
 	ret = multi_cpu_stop(&msdata);
 
 	/* Busy wait for completion. */
-	while (!atomic_read(&done.nr_todo))
+	while (atomic_read(&done.nr_todo))
 		cpu_relax();
 
 	mutex_unlock(&stop_cpus_mutex);
diff --git a/kernel/timer.c b/kernel/timer.c
index 54596b5..8750875 100644
--- a/kernel/timer.c
+++ b/kernel/timer.c
@@ -1461,6 +1461,19 @@ void run_local_timers(void)
 	 * the timer softirq.
 	 */
 #ifdef CONFIG_PREEMPT_RT_FULL
+
+#ifndef CONFIG_SMP
+	/*
+	 * The spin_do_trylock() later may fail as the lock may be hold before
+	 * the interrupt arrived. The spin-lock debugging code will raise a
+	 * warning if the try_lock fails on UP. Since this is only an
+	 * optimization for the FULL_NO_HZ case (not to run the timer softirq on
+	 * an nohz_full CPU) we don't really care and shedule the softirq.
+	 */
+	raise_softirq(TIMER_SOFTIRQ);
+	return;
+#endif
+
 	/* On RT, irq work runs from softirq */
 	if (irq_work_needs_cpu()) {
 		raise_softirq(TIMER_SOFTIRQ);
diff --git a/localversion-rt b/localversion-rt
index c3054d0..1445cd6 100644
--- a/localversion-rt
+++ b/localversion-rt
@@ -1 +1 @@
--rt2
+-rt3

Re: [ANNOUNCE] 3.14.2-rt3

[Mike Galbraith]

On Sat, 2014-05-03 at 19:01 +0200, Sebastian Andrzej Siewior wrote:

> - Mike Galbraith sent a patch which might fix lazy preempt on x86_64.
>   Patch applied and my machine still explodes therefore lazy preempt
>   remains off on x86_64.

Can you send me your config offline?  I'll try to scrape up some cycles
to take another peek.  It seems to work fine on my boxen, so there must
be a config dependent bug somewhere.

> - Mike Galbraith sent a few patches to get cpu hoplug to work. This
>   includes lg_global_trylock_relax().

(applies frozen shark repellent.. liberally)

-Mike

[patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Mike Galbraith]

rt-rw-lockdep-annotations.patch missed an rwlock_acquire_read()

Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
---
 kernel/locking/rt.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

--- a/kernel/locking/rt.c
+++ b/kernel/locking/rt.c
@@ -244,13 +244,12 @@ void __lockfunc rt_read_lock(rwlock_t *r
 {
 	struct rt_mutex *lock = &rwlock->lock;
 

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Steven Rostedt]

On Wed, 07 May 2014 09:54:07 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> 
> rt-rw-lockdep-annotations.patch missed an rwlock_acquire_read()
> 
> Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> ---
>  kernel/locking/rt.c |    3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> --- a/kernel/locking/rt.c
> +++ b/kernel/locking/rt.c
> @@ -244,13 +244,12 @@ void __lockfunc rt_read_lock(rwlock_t *r
>  {
>  	struct rt_mutex *lock = &rwlock->lock;
>  
> -
>  	/*
>  	 * recursive read locks succeed when current owns the lock
>  	 */
>  	if (rt_mutex_owner(lock) != current) {
>  		migrate_disable();
> -		rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
> +		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
>  		__rt_spin_lock(lock);

This looks like it reverse applies?

-- Steve

>  	}
>  	rwlock->read_depth++;
> 

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Mike Galbraith]

On Wed, 2014-05-07 at 09:48 -0400, Steven Rostedt wrote: 
> On Wed, 07 May 2014 09:54:07 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > 
> > rt-rw-lockdep-annotations.patch missed an rwlock_acquire_read()
> > 
> > Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> > ---
> >  kernel/locking/rt.c |    3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > --- a/kernel/locking/rt.c
> > +++ b/kernel/locking/rt.c
> > @@ -244,13 +244,12 @@ void __lockfunc rt_read_lock(rwlock_t *r
> >  {
> >  	struct rt_mutex *lock = &rwlock->lock;
> >  
> > -
> >  	/*
> >  	 * recursive read locks succeed when current owns the lock
> >  	 */
> >  	if (rt_mutex_owner(lock) != current) {
> >  		migrate_disable();
> > -		rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
> > +		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
> >  		__rt_spin_lock(lock);
> 
> This looks like it reverse applies?


Nope, I thought Thomas's whacked the one he did on purpose.  

-Mike

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Mike Galbraith]

On Wed, 2014-05-07 at 16:01 +0200, Mike Galbraith wrote: 
> On Wed, 2014-05-07 at 09:48 -0400, Steven Rostedt wrote: 
> > On Wed, 07 May 2014 09:54:07 +0200
> > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > 
> > > 
> > > rt-rw-lockdep-annotations.patch missed an rwlock_acquire_read()
> > > 
> > > Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> > > ---
> > >  kernel/locking/rt.c |    3 +--
> > >  1 file changed, 1 insertion(+), 2 deletions(-)
> > > 
> > > --- a/kernel/locking/rt.c
> > > +++ b/kernel/locking/rt.c
> > > @@ -244,13 +244,12 @@ void __lockfunc rt_read_lock(rwlock_t *r
> > >  {
> > >  	struct rt_mutex *lock = &rwlock->lock;
> > >  
> > > -
> > >  	/*
> > >  	 * recursive read locks succeed when current owns the lock
> > >  	 */
> > >  	if (rt_mutex_owner(lock) != current) {
> > >  		migrate_disable();
> > > -		rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
> > > +		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
> > >  		__rt_spin_lock(lock);
> > 
> > This looks like it reverse applies?
> 
> 
> Nope, I thought Thomas's whacked the one he did on purpose.  

I should learn to speak lockdep.  How about this instead?

locking/rt: fix rt_read_trylock() lockdep annotation.

rt-rw-lockdep-annotations.patch dropped a rwlock_acquire_read

Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
---
 kernel/locking/rt.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

--- a/kernel/locking/rt.c
+++ b/kernel/locking/rt.c
@@ -217,7 +217,7 @@ int __lockfunc rt_read_trylock(rwlock_t
 		migrate_disable();
 		ret = rt_mutex_trylock(lock);
 		if (ret)
-			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
+			rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
 		else
 			migrate_enable();
 
@@ -244,7 +244,6 @@ void __lockfunc rt_read_lock(rwlock_t *r
 {
 	struct rt_mutex *lock = &rwlock->lock;
 

Re: [ANNOUNCE] 3.14.2-rt3

[Joakim Hernberg]

[-- Attachment #1: Type: text/plain, Size: 40997 bytes --]

On Sat, 3 May 2014 19:01:11 +0200
Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:

> I'm pleased to announce the v3.14.2-rt3 patch set.

Hello,

My machine had a hicup :)  Don't know if the sr0 stuff belongs to it,
but this is what I found in the buffer afterwards.

Kernel config attached:

[89649.850386] sr 4:0:0:0: [sr0] Device not ready
[89649.850394] sr 4:0:0:0: [sr0]  
[89649.850397] Result: hostbyte=0x00 driverbyte=0x08
[89649.850401] sr 4:0:0:0: [sr0]  
[89649.850404] Sense Key : 0x2 [current] 
[89649.850410] sr 4:0:0:0: [sr0]  
[89649.850412] ASC=0x3a ASCQ=0x1
[89649.850416] sr 4:0:0:0: [sr0] CDB: 
[89649.850418] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[89649.850432] end_request: I/O error, dev sr0, sector 0
[89649.850438] Buffer I/O error on device sr0, logical block 0
[89649.850442] Buffer I/O error on device sr0, logical block 1
[89649.850446] Buffer I/O error on device sr0, logical block 2
[89649.850449] Buffer I/O error on device sr0, logical block 3
[89649.850453] Buffer I/O error on device sr0, logical block 4
[89649.850456] Buffer I/O error on device sr0, logical block 5
[89649.850460] Buffer I/O error on device sr0, logical block 6
[89649.850464] Buffer I/O error on device sr0, logical block 7
[89649.850482] Buffer I/O error on device sr0, logical block 8
[89649.850486] Buffer I/O error on device sr0, logical block 9
[89649.850505] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850534] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850557] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850570] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850583] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850602] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850611] sr 4:0:0:0: [sr0] unaligned transfer
[89649.850618] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895322] sr 4:0:0:0: [sr0] Device not ready
[89649.895329] sr 4:0:0:0: [sr0]  
[89649.895654] Result: hostbyte=0x00 driverbyte=0x08
[89649.895659] sr 4:0:0:0: [sr0]  
[89649.895662] Sense Key : 0x2 [current] 
[89649.895667] sr 4:0:0:0: [sr0]  
[89649.895670] ASC=0x3a ASCQ=0x1
[89649.895675] sr 4:0:0:0: [sr0] CDB: 
[89649.895678] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[89649.895688] end_request: I/O error, dev sr0, sector 0
[89649.895735] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895779] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895806] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895832] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895865] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895886] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895901] sr 4:0:0:0: [sr0] unaligned transfer
[89649.895916] sr 4:0:0:0: [sr0] unaligned transfer
[89724.648852] INFO: rcu_preempt self-detected stall on CPU { 1}  (t=60000 jiffies g=12214731 c=12214730 q=0)
[89724.648853] sending NMI to all CPUs:
[89724.648856] NMI backtrace for cpu 2
[89724.648859] CPU: 2 PID: 50 Comm: ksoftirqd/2 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.648859] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.648861] task: ffff8803fd470000 ti: ffff8803fd478000 task.ti: ffff8803fd478000
[89724.648865] RIP: 0010:[<ffffffff810aaf05>]  [<ffffffff810aaf05>] update_blocked_averages+0x95/0x6a0
[89724.648866] RSP: 0018:ffff8803fd479cd0  EFLAGS: 00000006
[89724.648867] RAX: 0000000000000002 RBX: ffff8803fb7cca00 RCX: ffff88040f405000
[89724.648868] RDX: ffffffff81c8d240 RSI: ffff88040f9659c0 RDI: ffff8803fb7cca00
[89724.648868] RBP: ffff8803fd479d30 R08: 0000000000000002 R09: ffff8803fd55cc00
[89724.648869] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803fb7cfe00
[89724.648870] R13: 000000000000b40e R14: ffff88040f965940 R15: ffff88040f9659c0
[89724.648871] FS:  0000000000000000(0000) GS:ffff88040f900000(0000) knlGS:0000000000000000
[89724.648871] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.648872] CR2: 00007f50cc3ad000 CR3: 000000038f630000 CR4: 00000000000407e0
[89724.648873] Stack:
[89724.648875]  0000000000000306 0000000000001fde ffff88040f9659c0 0000000000000246
[89724.648876]  ffff88040f9661d0 ffff88040f965a80 00000000904bfafe 0000000000000002
[89724.648878]  0000000105556270 0000000000000007 ffff88040f965940 000000000420a140
[89724.648878] Call Trace:
[89724.648882]  [<ffffffff810b25dc>] rebalance_domains+0x4c/0x2c0
[89724.648884]  [<ffffffff810b288f>] run_rebalance_domains+0x3f/0x190
[89724.648886]  [<ffffffff810a7f5e>] ? __vtime_account_system+0x2e/0x40
[89724.648889]  [<ffffffff81070926>] do_current_softirqs+0x1f6/0x3f0
[89724.648891]  [<ffffffff81070d8d>] run_ksoftirqd+0x2d/0x50
[89724.648893]  [<ffffffff8109971d>] smpboot_thread_fn+0x15d/0x2a0
[89724.648896]  [<ffffffff815157ba>] ? schedule+0x2a/0x90
[89724.648898]  [<ffffffff810995c0>] ? SyS_setgroups+0x150/0x150
[89724.648900]  [<ffffffff8109168c>] kthread+0xdc/0x100
[89724.648901]  [<ffffffff810915b0>] ? flush_kthread_worker+0xe0/0xe0
[89724.648904]  [<ffffffff81520cbc>] ret_from_fork+0x7c/0xb0
[89724.648906]  [<ffffffff810915b0>] ? flush_kthread_worker+0xe0/0xe0
[89724.648918] Code: c7 4c 8d b8 40 ff ff ff 0f 84 f8 01 00 00 0f 1f 84 00 00 00 00 00 49 8b 97 d0 00 00 00 49 63 86 38 09 00 00 48 8b 8a 80 00 00 00 <48> 8b 92 88 00 00 00 48 8b 1c c1 4c 8b 24 c2 66 66 66 66 90 be 
[89724.648919] NMI backtrace for cpu 5
[89724.648921] CPU: 5 PID: 71 Comm: ksoftirqd/5 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.648921] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.648922] task: ffff8803fd5899f0 ti: ffff8803fd592000 task.ti: ffff8803fd592000
[89724.648925] RIP: 0010:[<ffffffff81070da8>]  [<ffffffff81070da8>] run_ksoftirqd+0x48/0x50
[89724.648926] RSP: 0000:ffff8803fd593e68  EFLAGS: 00000296
[89724.648927] RAX: ffff8803fd5899f0 RBX: 0000000000000005 RCX: ffffffff81a503c0
[89724.648928] RDX: 000000000000ed00 RSI: 0000000000000000 RDI: 0000000000000046
[89724.648928] RBP: ffff8803fd593e70 R08: 0000000000000000 R09: ffff88040fae5940
[89724.648929] R10: 0000000000000807 R11: 0000000000000020 R12: ffff88040f401960
[89724.648930] R13: ffffffff81a3d700 R14: ffff8803fd5899f0 R15: ffff8803fd5899f0
[89724.648931] FS:  0000000000000000(0000) GS:ffff88040fa80000(0000) knlGS:0000000000000000
[89724.648931] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.648932] CR2: 00007f1652eb4000 CR3: 00000003fc072000 CR4: 00000000000407e0
[89724.648933] Stack:
[89724.648934]  ffff8803fd5899f0 ffff8803fd593ec0 ffffffff8109971d 0000000000000000
[89724.648936]  ffffffff815157ba 0000000000000001 ffff8803fdb553c0 ffff88040f401960
[89724.648937]  ffffffff810995c0 0000000000000000 0000000000000000 ffff8803fd593f48
[89724.648938] Call Trace:
[89724.648940]  [<ffffffff8109971d>] smpboot_thread_fn+0x15d/0x2a0
[89724.648942]  [<ffffffff815157ba>] ? schedule+0x2a/0x90
[89724.648944]  [<ffffffff810995c0>] ? SyS_setgroups+0x150/0x150
[89724.648945]  [<ffffffff8109168c>] kthread+0xdc/0x100
[89724.648947]  [<ffffffff810915b0>] ? flush_kthread_worker+0xe0/0xe0
[89724.648949]  [<ffffffff81520cbc>] ret_from_fork+0x7c/0xb0
[89724.648950]  [<ffffffff810915b0>] ? flush_kthread_worker+0xe0/0xe0
[89724.648963] Code: 80 e8 19 00 00 01 bf 01 00 00 00 e8 a3 f9 ff ff 65 48 8b 04 25 40 c8 00 00 83 a8 e8 19 00 00 01 89 df e8 4c 78 06 00 fb 66 66 90 <66> 66 90 5b 5d c3 66 90 66 66 66 66 90 55 48 89 e5 41 57 41 56 
[89724.648964] NMI backtrace for cpu 7
[89724.648966] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.648966] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.648968] task: ffff8803fdb799f0 ti: ffff8803fdb80000 task.ti: ffff8803fdb80000
[89724.648971] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89724.648972] RSP: 0018:ffff8803fdb81e28  EFLAGS: 00000046
[89724.648973] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89724.648973] RDX: 0000000000000000 RSI: ffff8803fdb81fd8 RDI: 0000000000000007
[89724.648974] RBP: ffff8803fdb81e58 R08: 0000000000593616 R09: 0000000034d860f9
[89724.648975] R10: 000116aba098c9bd R11: 0000000000000293 R12: 0000000000000001
[89724.648975] R13: 0000000000000001 R14: 0000000000000002 R15: ffff8803fdb81fd8
[89724.648976] FS:  0000000000000000(0000) GS:ffff88040fb80000(0000) knlGS:0000000000000000
[89724.648977] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.648978] CR2: 00000000ed60a000 CR3: 000000038f630000 CR4: 00000000000407e0
[89724.648978] Stack:
[89724.648980]  00000007fdb81e58 32a7a8fb32a7a8da ffff88040fbecd00 ffffffff81a8ba40
[89724.648982]  000051a7f69eebbd 0000000000000002 ffff8803fdb81e88 ffffffff813e03ca
[89724.648983]  0000000000000002 ffff88040fbecd00 0000000000000000 ffffffff81a8ba40
[89724.648983] Call Trace:
[89724.648987]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89724.648989]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89724.648991]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89724.648994]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89724.648997]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89724.649010] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89724.649011] NMI backtrace for cpu 6
[89724.649012] CPU: 6 PID: 1015 Comm: ksysguard Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.649012] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.649013] task: ffff8803d3728000 ti: ffff8803d3726000 task.ti: ffff8803d3726000
[89724.649029] RIP: 0033:[<00007f50caf4d74e>]  [<00007f50caf4d74e>] 0x7f50caf4d74e
[89724.649029] RSP: 002b:00007ffff8927c10  EFLAGS: 00000202
[89724.649030] RAX: 0000000000000000 RBX: 00000000025f88c0 RCX: 0000000000000002
[89724.649030] RDX: 0000000000000000 RSI: 00000000028a3638 RDI: 00000000025f88c0
[89724.649031] RBP: 0000000002607fb0 R08: 0000000000000002 R09: 0000000000000052
[89724.649031] R10: 00000000000002a7 R11: 000000000263fde0 R12: 0000000000000000
[89724.649031] R13: 000000000263c390 R14: 0000000000000008 R15: 0000000002503080
[89724.649032] FS:  00007f50cc393780(0000) GS:ffff88040fb00000(0000) knlGS:0000000000000000
[89724.649033] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.649033] CR2: 00000000ed60a000 CR3: 00000003d365f000 CR4: 00000000000407e0
 
[89724.649034] NMI backtrace for cpu 3
[89724.649035] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.649035] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.649036] task: ffff8803fdb699f0 ti: ffff8803fdb70000 task.ti: ffff8803fdb70000
[89724.649038] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89724.649038] RSP: 0018:ffff8803fdb71e28  EFLAGS: 00000046
[89724.649039] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[89724.649039] RDX: 0000000000000000 RSI: ffff8803fdb71fd8 RDI: 0000000000000003
[89724.649040] RBP: ffff8803fdb71e58 R08: 0000000000258b7a R09: 0000000034d860f9
[89724.649041] R10: 000116aba0ccd674 R11: 0000000000000000 R12: 0000000000000000
[89724.649041] R13: 0000000000000001 R14: 0000000000000001 R15: ffff8803fdb71fd8
[89724.649042] FS:  0000000000000000(0000) GS:ffff88040f980000(0000) knlGS:0000000000000000
[89724.649043] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.649044] CR2: 00007fa8ea61d000 CR3: 000000026b955000 CR4: 00000000000407e0
[89724.649044] Stack:
[89724.649046]  00000003fdb71e58 24a3ff0124a3fee0 ffff88040f9ecd00 ffffffff81a8ba40
[89724.649047]  000051a7f69f087f 0000000000000001 ffff8803fdb71e88 ffffffff813e03ca
[89724.649047]  0000000000000001 ffff88040f9ecd00 0000000000000000 ffffffff81a8ba40
[89724.649048] Call Trace:
[89724.649049]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89724.649050]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89724.649051]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89724.649053]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89724.649054]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89724.649061] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89724.649062] NMI backtrace for cpu 1
[89724.649063] CPU: 1 PID: 28438 Comm: Ableton Live 9  Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.649064] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.649065] task: ffff880116290000 ti: ffff88026033e000 task.ti: ffff88026033e000
[89724.649069] RIP: 0010:[<ffffffff812aa610>]  [<ffffffff812aa610>] __const_udelay+0x0/0x30
[89724.649070] RSP: 0000:ffff88040f883da8  EFLAGS: 00200046
[89724.649071] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000002
[89724.649071] RDX: 0000000000000c00 RSI: 0000000000000080 RDI: 0000000000418958
[89724.649072] RBP: ffff88040f883db8 R08: 0000000000000002 R09: 0000000000000028
[89724.649073] R10: 0000000000000003 R11: 000000000001de30 R12: ffffffff81a51440
[89724.649073] R13: ffffffff81ad07f0 R14: ffffffff81a51440 R15: 0000000000000001
[89724.649075] FS:  0000000080b60000(0063) GS:ffff88040f880000(006b) knlGS:000000000b03fb40
[89724.649075] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[89724.649076] CR2: 00007f50ae7ee000 CR3: 000000032517e000 CR4: 00000000000407e0
[89724.649077] Stack:
[89724.649078]  ffffffff8104b315 ffff88040f88ed00 ffff88040f883e10 ffffffff810d9f4e
[89724.649080]  0000000000000001 ffff88040f883de0 ffffffff8107118e 0000000000000000
[89724.649081]  ffff880116290000 0000000000000001 0000000000000001 ffff88040f88e7c0
[89724.649082] Call Trace:
[89724.649083]  <IRQ> 
[89724.649085]  [<ffffffff8104b315>] ? arch_trigger_all_cpu_backtrace+0x75/0x90
[89724.649088]  [<ffffffff810d9f4e>] rcu_check_callbacks+0x50e/0x790
[89724.649090]  [<ffffffff8107118e>] ? raise_softirq_irqoff+0xe/0x40
[89724.649092]  [<ffffffff8107b042>] update_process_times+0x42/0x70
[89724.649095]  [<ffffffff810e5bb5>] tick_sched_handle.isra.16+0x25/0x60
[89724.649097]  [<ffffffff810e5c34>] tick_sched_timer+0x44/0x70
[89724.649099]  [<ffffffff81094a91>] __run_hrtimer+0x81/0x270
[89724.649100]  [<ffffffff810e5bf0>] ? tick_sched_handle.isra.16+0x60/0x60
[89724.649102]  [<ffffffff81095f3c>] hrtimer_interrupt+0x38c/0x3f0
[89724.649105]  [<ffffffff810495aa>] local_apic_timer_interrupt+0x3a/0x60
[89724.649107]  [<ffffffff81522daf>] smp_apic_timer_interrupt+0x3f/0x50
[89724.649109]  [<ffffffff815218dd>] apic_timer_interrupt+0x6d/0x80
[89724.649110]  <EOI> 
[89724.649123] Code: 48 ff c8 5d c3 66 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 ff 15 69 da 7d 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 8d 04 bd 00 00 00 00 65 48 8b 14 25 60 50 06 00 48 69 d2 
[89724.649123] NMI backtrace for cpu 4
[89724.649125] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.649126] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.649127] task: ffff8803fdb6b3e0 ti: ffff8803fdb72000 task.ti: ffff8803fdb72000
[89724.649129] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89724.649130] RSP: 0018:ffff8803fdb73e28  EFLAGS: 00000046
[89724.649131] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89724.649131] RDX: 0000000000000000 RSI: ffff8803fdb73fd8 RDI: 0000000000000004
[89724.649132] RBP: ffff8803fdb73e58 R08: 0000000000580210 R09: 0000000034d860f9
[89724.649133] R10: 000116aba098c9bd R11: 0000000000000293 R12: 0000000000000001
[89724.649133] R13: 0000000000000001 R14: 0000000000000002 R15: ffff8803fdb73fd8
[89724.649134] FS:  0000000000000000(0000) GS:ffff88040fa00000(0000) knlGS:0000000000000000
[89724.649135] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.649136] CR2: 00001c2f75b3c000 CR3: 000000038f630000 CR4: 00000000000407e0
[89724.649136] Stack:
[89724.649138]  00000004fdb73e58 2828648928286468 ffff88040fa6cd00 ffffffff81a8ba40
[89724.649140]  000051a7f69e9170 0000000000000002 ffff8803fdb73e88 ffffffff813e03ca
[89724.649141]  0000000000000002 ffff88040fa6cd00 0000000000000000 ffffffff81a8ba40
[89724.649141] Call Trace:
[89724.649144]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89724.649146]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89724.649148]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89724.649150]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89724.649152]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89724.649165] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89724.649165] NMI backtrace for cpu 0
[89724.649167] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89724.649167] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89724.649168] task: ffffffff81a16500 ti: ffffffff81a00000 task.ti: ffffffff81a00000
[89724.649171] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89724.649171] RSP: 0018:ffffffff81a01e28  EFLAGS: 00000046
[89724.649172] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89724.649173] RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: 0000000000000000
[89724.649173] RBP: ffffffff81a01e58 R08: 0000000000256e34 R09: 0000000034d860f9
[89724.649174] R10: 000116aba0ccd674 R11: 0000000000000293 R12: 0000000000000001
[89724.649175] R13: 0000000000000001 R14: 0000000000000002 R15: ffffffff81a01fd8
[89724.649175] FS:  0000000000000000(0000) GS:ffff88040f800000(0000) knlGS:0000000000000000
[89724.649176] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89724.649176] CR2: 00007f50cc3ad000 CR3: 000000038f630000 CR4: 00000000000407f0
[89724.649177] Stack:
[89724.649178]  0000000081a01e58 181d9c45181d9c24 ffff88040f86cd00 ffffffff81a8ba40
[89724.649179]  000051a7f69effeb 0000000000000002 ffffffff81a01e88 ffffffff813e03ca
[89724.649179]  0000000000000002 ffff88040f86cd00 0000000000000000 ffffffff81a8ba40
[89724.649180] Call Trace:
[89724.649181]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89724.649182]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89724.649184]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89724.649185]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89724.649187]  [<ffffffff81508664>] rest_init+0x84/0x90
[89724.649189]  [<ffffffff81b41ea2>] start_kernel+0x444/0x465
[89724.649190]  [<ffffffff81b41120>] ? early_idt_handlers+0x120/0x120
[89724.649191]  [<ffffffff81b414d7>] x86_64_start_reservations+0x2a/0x2c
[89724.649192]  [<ffffffff81b41626>] x86_64_start_kernel+0x14d/0x170
[89724.649199] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89904.537490] INFO: rcu_preempt self-detected stall on CPU { 1}  (t=240004 jiffies g=12214731 c=12214730 q=0)
[89904.537491] sending NMI to all CPUs:
[89904.537496] NMI backtrace for cpu 0
[89904.537499] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537500] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537502] task: ffffffff81a16500 ti: ffffffff81a00000 task.ti: ffffffff81a00000
[89904.537508] RIP: 0010:[<ffffffff810a7931>]  [<ffffffff810a7931>] sched_clock_idle_sleep_event+0x1/0x20
[89904.537509] RSP: 0018:ffffffff81a01e98  EFLAGS: 00000002
[89904.537510] RAX: 000051d1dfb228c7 RBX: ffff88040f80f5c0 RCX: 0000000000000018
[89904.537511] RDX: 000000006dda18c7 RSI: 0000000000015f68 RDI: 00000ae1befa96e0
[89904.537512] RBP: ffffffff81a01ec8 R08: 0000000000251e96 R09: 0000000034d860f9
[89904.537513] R10: 0001173a973d35e7 R11: 0000000000000293 R12: 0000000000000000
[89904.537514] R13: 000051d1dfb228c7 R14: ffffffff81bdd2c0 R15: ffffffff81a01fd8
[89904.537516] FS:  0000000000000000(0000) GS:ffff88040f800000(0000) knlGS:0000000000000000
[89904.537517] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537518] CR2: 00007f50cc3ad000 CR3: 000000038f630000 CR4: 00000000000407f0
[89904.537519] Stack:
[89904.537522]  ffffffff81a01ec8 ffffffff810e59fa 0000000000000002 ffffffff81ad07e0
[89904.537525]  ffffffff81a01fd8 ffffffff81bdd2c0 ffffffff81a01ed8 ffffffff810e5b50
[89904.537527]  ffffffff81a01f28 ffffffff810cddb5 ffffffff81a01fd8 ffffffff81a01fd8
[89904.537528] Call Trace:
[89904.537533]  [<ffffffff810e59fa>] ? __tick_nohz_idle_enter+0x3a/0x150
[89904.537536]  [<ffffffff810e5b50>] tick_nohz_idle_enter+0x40/0x80
[89904.537539]  [<ffffffff810cddb5>] cpu_startup_entry+0x95/0x2e0
[89904.537543]  [<ffffffff81508664>] rest_init+0x84/0x90
[89904.537547]  [<ffffffff81b41ea2>] start_kernel+0x444/0x465
[89904.537549]  [<ffffffff81b41120>] ? early_idt_handlers+0x120/0x120
[89904.537552]  [<ffffffff81b414d7>] x86_64_start_reservations+0x2a/0x2c
[89904.537554]  [<ffffffff81b41626>] x86_64_start_kernel+0x14d/0x170
[89904.537573] Code: 48 89 d0 41 5c 5d c3 0f 1f 00 e8 cb 7f f7 ff 5b 41 5c 5d c3 66 0f 1f 44 00 00 e8 fb 8f f7 ff 48 89 d0 eb eb 31 c0 eb e7 66 90 55 <65> 8b 3c 25 34 b0 00 00 48 89 e5 e8 1f ff ff ff 5d c3 66 66 66 
[89904.537575] NMI backtrace for cpu 2
[89904.537578] CPU: 2 PID: 1008 Comm: konsole Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537579] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537580] task: ffff8803f64d33e0 ti: ffff8803d34f6000 task.ti: ffff8803d34f6000
[89904.537600] RIP: 0033:[<00007f50ca3429be>]  [<00007f50ca3429be>] 0x7f50ca3429be
[89904.537601] RSP: 002b:00007ffff8926430  EFLAGS: 00000246
[89904.537602] RAX: 0000000000000004 RBX: 0000000000ff00ff RCX: 00000000ffc3c3c3
[89904.537603] RDX: 00000000000000ff RSI: 00007f50af21c0d4 RDI: 00000000ffffffff
[89904.537604] RBP: 0000000000000eac R08: 000000000287d448 R09: 0000000000ff0000
[89904.537605] R10: 00000000c23dc23d R11: 0000000000ff00ff R12: 0000000000000006
[89904.537606] R13: 0000000000000003 R14: 00000000000000ff R15: 0000000000000008
[89904.537608] FS:  00007f50cc393780(0000) GS:ffff88040f900000(0000) knlGS:0000000000000000
[89904.537609] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537610] CR2: 00007f1652eb4000 CR3: 00000003e1dfd000 CR4: 00000000000407e0
 
[89904.537612] NMI backtrace for cpu 1
[89904.537614] CPU: 1 PID: 28438 Comm: Ableton Live 9  Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537615] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537616] task: ffff880116290000 ti: ffff88026033e000 task.ti: ffff88026033e000
[89904.537622] RIP: 0010:[<ffffffff812aa622>]  [<ffffffff812aa622>] __const_udelay+0x12/0x30
[89904.537623] RSP: 0000:ffff88040f883da0  EFLAGS: 00200046
[89904.537624] RAX: 0000000001062560 RBX: 0000000000002710 RCX: 0000000000000002
[89904.537625] RDX: 0000000000340c34 RSI: 0000000000000080 RDI: 0000000000418958
[89904.537626] RBP: ffff88040f883db8 R08: 0000000000000002 R09: 0000000000000028
[89904.537627] R10: 0000000000000003 R11: 0000000000021fcc R12: ffffffff81a51440
[89904.537628] R13: ffffffff81ad07f0 R14: ffffffff81a51440 R15: 0000000000000001
[89904.537630] FS:  0000000080b60000(0063) GS:ffff88040f880000(006b) knlGS:000000000b03fb40
[89904.537631] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[89904.537633] CR2: 00007f50ae7ee000 CR3: 000000032517e000 CR4: 00000000000407e0
[89904.537633] Stack:
[89904.537636]  ffff88040f883db8 ffffffff8104b315 ffff88040f88ed00 ffff88040f883e10
[89904.537638]  ffffffff810d9f4e 0000000000000001 ffff88040f883de0 ffffffff8107118e
[89904.537641]  0000000000000000 ffff880116290000 0000000000000001 0000000000000001
[89904.537641] Call Trace:
[89904.537643]  <IRQ> 
[89904.537646]  [<ffffffff8104b315>] ? arch_trigger_all_cpu_backtrace+0x75/0x90
[89904.537650]  [<ffffffff810d9f4e>] rcu_check_callbacks+0x50e/0x790
[89904.537654]  [<ffffffff8107118e>] ? raise_softirq_irqoff+0xe/0x40
[89904.537657]  [<ffffffff8107b042>] update_process_times+0x42/0x70
[89904.537660]  [<ffffffff810e5bb5>] tick_sched_handle.isra.16+0x25/0x60
[89904.537662]  [<ffffffff810e5c34>] tick_sched_timer+0x44/0x70
[89904.537665]  [<ffffffff81094a91>] __run_hrtimer+0x81/0x270
[89904.537668]  [<ffffffff810e5bf0>] ? tick_sched_handle.isra.16+0x60/0x60
[89904.537670]  [<ffffffff81095f3c>] hrtimer_interrupt+0x38c/0x3f0
[89904.537675]  [<ffffffff810495aa>] local_apic_timer_interrupt+0x3a/0x60
[89904.537678]  [<ffffffff81522daf>] smp_apic_timer_interrupt+0x3f/0x50
[89904.537682]  [<ffffffff815218dd>] apic_timer_interrupt+0x6d/0x80
[89904.537684]  <EOI> 
[89904.537702] Code: 89 e5 ff 15 69 da 7d 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8d 04 bd 00 00 00 00 65 48 8b 14 25 60 50 06 00 <48> 69 d2 fa 00 00 00 48 89 e5 f7 e2 48 8d 7a 01 e8 b9 ff ff ff 
[89904.537703] NMI backtrace for cpu 7
[89904.537706] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537707] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537708] task: ffff8803fdb799f0 ti: ffff8803fdb80000 task.ti: ffff8803fdb80000
[89904.537713] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89904.537714] RSP: 0018:ffff8803fdb81e28  EFLAGS: 00000046
[89904.537715] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89904.537716] RDX: 0000000000000000 RSI: ffff8803fdb81fd8 RDI: 0000000000000007
[89904.537717] RBP: ffff8803fdb81e58 R08: 000000000058909e R09: 0000000034d860f9
[89904.537719] R10: 0001173a97092930 R11: 0000000000000293 R12: 0000000000000001
[89904.537720] R13: 0000000000000001 R14: 0000000000000002 R15: ffff8803fdb81fd8
[89904.537721] FS:  0000000000000000(0000) GS:ffff88040fb80000(0000) knlGS:0000000000000000
[89904.537723] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537724] CR2: 00007fa8ea61d000 CR3: 000000038f630000 CR4: 00000000000407e0
[89904.537724] Stack:
[89904.537728]  00000007fdb81e58 32a7a8fb32a7a8da ffff88040fbecd00 ffffffff81a8ba40
[89904.537730]  000051d1dfb1fb6f 0000000000000002 ffff8803fdb81e88 ffffffff813e03ca
[89904.537732]  0000000000000002 ffff88040fbecd00 0000000000000000 ffffffff81a8ba40
[89904.537733] Call Trace:
[89904.537738]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89904.537740]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89904.537744]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89904.537747]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89904.537750]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89904.537770] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89904.537771] NMI backtrace for cpu 4
[89904.537773] CPU: 4 PID: 1015 Comm: ksysguard Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537774] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537775] task: ffff8803d3728000 ti: ffff8803d3726000 task.ti: ffff8803d3726000
[89904.537784] RIP: 0033:[<00007f50ca2c03dd>]  [<00007f50ca2c03dd>] 0x7f50ca2c03dd
[89904.537784] RSP: 002b:00007ffff892b380  EFLAGS: 00000202
[89904.537785] RAX: 0000000000000040 RBX: 00007ffff892b530 RCX: 00007ffff892b570
[89904.537786] RDX: 00007ffff892b530 RSI: 00007ffff892b550 RDI: 00007ffff892b530
[89904.537786] RBP: 00000000028f8d60 R08: 00000000ffad17f8 R09: 000000000271bbe0
[89904.537787] R10: 0000000000000000 R11: 000000000000012c R12: 00007ffff892b550
[89904.537788] R13: 00007ffff892b570 R14: 00007ffff892b960 R15: 0000000000000017
[89904.537789] FS:  00007f50cc393780(0000) GS:ffff88040fa00000(0000) knlGS:0000000000000000
[89904.537789] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537790] CR2: 00000000ed60a000 CR3: 00000003d365f000 CR4: 00000000000407e0
 
[89904.537792] NMI backtrace for cpu 6
[89904.537794] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537795] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537796] task: ffff8803fdb78000 ti: ffff8803fdb76000 task.ti: ffff8803fdb76000
[89904.537801] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89904.537802] RSP: 0018:ffff8803fdb77e28  EFLAGS: 00000046
[89904.537803] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89904.537804] RDX: 0000000000000000 RSI: ffff8803fdb77fd8 RDI: 0000000000000006
[89904.537805] RBP: ffff8803fdb77e58 R08: 000000000056e0c8 R09: 0000000034d860f9
[89904.537806] R10: 0001173a97092930 R11: 0000000000000293 R12: 0000000000000001
[89904.537807] R13: 0000000000000001 R14: 0000000000000002 R15: ffff8803fdb77fd8
[89904.537809] FS:  0000000000000000(0000) GS:ffff88040fb00000(0000) knlGS:0000000000000000
[89904.537810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537811] CR2: 00007f50cc3ad000 CR3: 000000038f630000 CR4: 00000000000407e0
[89904.537812] Stack:
[89904.537815]  00000006fdb77e58 2f29da8b2f29da6a ffff88040fb6cd00 ffffffff81a8ba40
[89904.537817]  000051d1dfb17cd6 0000000000000002 ffff8803fdb77e88 ffffffff813e03ca
[89904.537820]  0000000000000002 ffff88040fb6cd00 0000000000000000 ffffffff81a8ba40
[89904.537820] Call Trace:
[89904.537824]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89904.537827]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89904.537830]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89904.537833]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89904.537836]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89904.537854] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89904.537855] NMI backtrace for cpu 3
[89904.537857] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537858] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537858] task: ffff8803fdb699f0 ti: ffff8803fdb70000 task.ti: ffff8803fdb70000
[89904.537861] RIP: 0010:[<ffffffff81314de9>]  [<ffffffff81314de9>] intel_idle+0xe9/0x180
[89904.537862] RSP: 0018:ffff8803fdb71e28  EFLAGS: 00000046
[89904.537862] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[89904.537863] RDX: 0000000000000000 RSI: ffff8803fdb71fd8 RDI: 0000000000000003
[89904.537864] RBP: ffff8803fdb71e58 R08: 000000000057466b R09: 0000000034d860f9
[89904.537864] R10: 0001173a97092930 R11: 0000000000000293 R12: 0000000000000001
[89904.537865] R13: 0000000000000001 R14: 0000000000000002 R15: ffff8803fdb71fd8
[89904.537866] FS:  0000000000000000(0000) GS:ffff88040f980000(0000) knlGS:0000000000000000
[89904.537867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537867] CR2: 000008fe86073000 CR3: 000000038f630000 CR4: 00000000000407e0
[89904.537868] Stack:
[89904.537869]  00000003fdb71e58 24a3ff0124a3fee0 ffff88040f9ecd00 ffffffff81a8ba40
[89904.537871]  000051d1dfb19aa2 0000000000000002 ffff8803fdb71e88 ffffffff813e03ca
[89904.537872]  0000000000000002 ffff88040f9ecd00 0000000000000000 ffffffff81a8ba40
[89904.537873] Call Trace:
[89904.537875]  [<ffffffff813e03ca>] cpuidle_enter_state+0x3a/0xc0
[89904.537878]  [<ffffffff813e0507>] cpuidle_idle_call+0xb7/0x250
[89904.537881]  [<ffffffff8102142e>] arch_cpu_idle+0xe/0x30
[89904.537883]  [<ffffffff810cdf79>] cpu_startup_entry+0x259/0x2e0
[89904.537885]  [<ffffffff81047572>] start_secondary+0x212/0x2d0
[89904.537896] Code: 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 65 48 8b 04 25 30 c8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 30 c8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 
[89904.537897] NMI backtrace for cpu 5
[89904.537899] CPU: 5 PID: 28534 Comm: chromium Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89904.537900] Hardware name: System manufacturer System Product Name/P8Z68-V PRO GEN3, BIOS 3603 11/09/2012
[89904.537902] task: ffff880023984dd0 ti: ffff880071208000 task.ti: ffff880071208000
[89904.537909] RIP: 0033:[<00007f05dafb25fe>]  [<00007f05dafb25fe>] 0x7f05dafb25fe
[89904.537910] RSP: 002b:00007fff08efaa90  EFLAGS: 00000202
[89904.537911] RAX: 0000000000000001 RBX: 00000d73f2566f41 RCX: 0000000000000018
[89904.537912] RDX: 0000000000008a02 RSI: 0000026c78b04611 RDI: 00000d73f2500000
[89904.537913] RBP: 00007fff08efaaf0 R08: 00007f05c7e96a00 R09: 00007fff08efabb0
[89904.537914] R10: 00000d73f2500000 R11: 00000d73f2566f41 R12: 00007f05c7e6b000
[89904.537915] R13: 00002bbffdb0eb01 R14: 00007fff08efae88 R15: 0000000000000000
[89904.537916] FS:  00007f05d9495940(0000) GS:ffff88040fa80000(0000) knlGS:0000000000000000
[89904.537918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[89904.537919] CR2: 000027f5e1189000 CR3: 000000004a5cb000 CR4: 00000000000407e0
 
[89929.940648] sr 4:0:0:0: [sr0] Device not ready
[89929.940651] sr 4:0:0:0: [sr0]  
[89929.940653] Result: hostbyte=0x00 driverbyte=0x08
[89929.940654] sr 4:0:0:0: [sr0]  
[89929.940655] Sense Key : 0x2 [current] 
[89929.940657] sr 4:0:0:0: [sr0]  
[89929.940658] ASC=0x3a ASCQ=0x1
[89929.940660] sr 4:0:0:0: [sr0] CDB: 
[89929.940661] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[89929.940666] end_request: I/O error, dev sr0, sector 0
[89929.940669] quiet_error: 54 callbacks suppressed
[89929.940669] Buffer I/O error on device sr0, logical block 0
[89929.940672] Buffer I/O error on device sr0, logical block 1
[89929.940673] Buffer I/O error on device sr0, logical block 2
[89929.940675] Buffer I/O error on device sr0, logical block 3
[89929.940676] Buffer I/O error on device sr0, logical block 4
[89929.940677] Buffer I/O error on device sr0, logical block 5
[89929.940679] Buffer I/O error on device sr0, logical block 6
[89929.940680] Buffer I/O error on device sr0, logical block 7
[89929.940683] Buffer I/O error on device sr0, logical block 8
[89929.940684] Buffer I/O error on device sr0, logical block 9
[89929.940697] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940712] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940720] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940724] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940737] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940743] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940749] sr 4:0:0:0: [sr0] unaligned transfer
[89929.940757] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987630] sr 4:0:0:0: [sr0] Device not ready
[89929.987637] sr 4:0:0:0: [sr0]  
[89929.987640] Result: hostbyte=0x00 driverbyte=0x08
[89929.987645] sr 4:0:0:0: [sr0]  
[89929.987648] Sense Key : 0x2 [current] 
[89929.987653] sr 4:0:0:0: [sr0]  
[89929.987654] ASC=0x3a ASCQ=0x1
[89929.987657] sr 4:0:0:0: [sr0] CDB: 
[89929.987658] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[89929.987667] end_request: I/O error, dev sr0, sector 0
[89929.987711] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987754] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987768] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987781] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987805] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987817] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987828] sr 4:0:0:0: [sr0] unaligned transfer
[89929.987839] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032855] sr 4:0:0:0: [sr0] Device not ready
[89930.032859] sr 4:0:0:0: [sr0]  
[89930.032861] Result: hostbyte=0x00 driverbyte=0x08
[89930.032862] sr 4:0:0:0: [sr0]  
[89930.032864] Sense Key : 0x2 [current] 
[89930.032867] sr 4:0:0:0: [sr0]  
[89930.032868] ASC=0x3a ASCQ=0x1
[89930.032870] sr 4:0:0:0: [sr0] CDB: 
[89930.032871] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[89930.032879] end_request: I/O error, dev sr0, sector 0
[89930.032934] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032947] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032966] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032974] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032983] sr 4:0:0:0: [sr0] unaligned transfer
[89930.032993] sr 4:0:0:0: [sr0] unaligned transfer
[89930.033000] sr 4:0:0:0: [sr0] unaligned transfer
[89930.033016] sr 4:0:0:0: [sr0] unaligned transfer
[89943.406650] INFO: task nacl_helper:28497 blocked for more than 120 seconds.
[89943.406654]       Tainted: G        W    3.14.2-rt3-1-rt-314 #1
[89943.406655] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[89943.406656] nacl_helper     D ffff88001a3b4dd0     0 28497  28493 0x00000002
[89943.406660]  ffff88015bcfbb38 0000000000000002 0000000000065940 ffff88015bcfbfd8
[89943.406661]  0000000000065940 ffff8803fdb699f0 ffff880374383c00 ffff8803b0260530
[89943.406662]  ffff88015bcfbb78 ffffffff81515339 0000000000065940 ffff88015bcfbfd8
[89943.406664] Call Trace:
[89943.406670]  [<ffffffff81515339>] ? __schedule+0x3b9/0x810
[89943.406674]  [<ffffffff811560f0>] ? free_hot_cold_page+0x1d0/0x2f0
[89943.406675]  [<ffffffff815157ba>] schedule+0x2a/0x90
[89943.406677]  [<ffffffff81514949>] schedule_timeout+0x209/0x2b0
[89943.406680]  [<ffffffff812033cd>] ? fsnotify_clear_marks_by_inode+0xbd/0x160
[89943.406682]  [<ffffffff815162ef>] wait_for_common+0xff/0x170
[89943.406685]  [<ffffffff810d77a0>] ? call_rcu_sched+0x20/0x20
[89943.406686]  [<ffffffff8151637d>] wait_for_completion+0x1d/0x20
[89943.406687]  [<ffffffff810d4e24>] wait_rcu_gp+0x54/0x70
[89943.406688]  [<ffffffff810d4db0>] ? __rcu_read_unlock+0x70/0x70
[89943.406690]  [<ffffffff810d9489>] synchronize_rcu+0x29/0x40
[89943.406692]  [<ffffffff81096a3d>] switch_task_namespaces+0x2d/0x40
[89943.406693]  [<ffffffff81096a60>] exit_task_namespaces+0x10/0x20
[89943.406695]  [<ffffffff8106dae1>] do_exit+0x3a1/0xb30
[89943.406697]  [<ffffffff8106e300>] do_group_exit+0x50/0xe0
[89943.406699]  [<ffffffff8107f688>] get_signal_to_deliver+0x288/0x710
[89943.406703]  [<ffffffff81016557>] do_signal+0x57/0x6c0
[89943.406704]  [<ffffffff810d96a3>] ? rcu_eqs_enter+0x93/0xa0
[89943.406705]  [<ffffffff8110914c>] ? acct_account_cputime+0x1c/0x20
[89943.406707]  [<ffffffff810a7d52>] ? account_user_time+0x92/0xb0
[89943.406709]  [<ffffffff81016c70>] do_notify_resume+0xb0/0x110
[89943.406711]  [<ffffffff81520fa4>] int_signal+0x12/0x17
[89971.807100] Clocksource tsc unstable (delta = 299966057930 ns)
[89971.808750] Switched to clocksource hpet
[90485.778300] sr 4:0:0:0: [sr0] Device not ready
[90485.778309] sr 4:0:0:0: [sr0]  
[90485.778332] Result: hostbyte=0x00 driverbyte=0x08
[90485.778351] sr 4:0:0:0: [sr0]  
[90485.778353] Sense Key : 0x2 [current] 
[90485.778358] sr 4:0:0:0: [sr0]  
[90485.778360] ASC=0x3a ASCQ=0x1
[90485.778364] sr 4:0:0:0: [sr0] CDB: 
[90485.778366] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[90485.778396] end_request: I/O error, dev sr0, sector 0
[90485.778401] quiet_error: 86 callbacks suppressed
[90485.778404] Buffer I/O error on device sr0, logical block 0
[90485.778428] Buffer I/O error on device sr0, logical block 1
[90485.778431] Buffer I/O error on device sr0, logical block 2
[90485.778434] Buffer I/O error on device sr0, logical block 3
[90485.778436] Buffer I/O error on device sr0, logical block 4
[90485.778438] Buffer I/O error on device sr0, logical block 5
[90485.778441] Buffer I/O error on device sr0, logical block 6
[90485.778443] Buffer I/O error on device sr0, logical block 7
[90485.778447] Buffer I/O error on device sr0, logical block 8
[90485.778450] Buffer I/O error on device sr0, logical block 9
[90485.778499] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778527] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778545] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778573] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778601] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778615] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778625] sr 4:0:0:0: [sr0] unaligned transfer
[90485.778645] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856225] sr 4:0:0:0: [sr0] Device not ready
[90485.856230] sr 4:0:0:0: [sr0]  
[90485.856245] Result: hostbyte=0x00 driverbyte=0x08
[90485.856247] sr 4:0:0:0: [sr0]  
[90485.856249] Sense Key : 0x2 [current] 
[90485.856253] sr 4:0:0:0: [sr0]  
[90485.856255] ASC=0x3a ASCQ=0x1
[90485.856258] sr 4:0:0:0: [sr0] CDB: 
[90485.856259] cdb[0]=0x28: 28 00 00 00 00 00 00 00 08 00
[90485.856268] end_request: I/O error, dev sr0, sector 0
[90485.856332] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856368] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856420] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856441] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856472] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856517] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856540] sr 4:0:0:0: [sr0] unaligned transfer
[90485.856556] sr 4:0:0:0: [sr0] unaligned transfer
[91788.235522] hrtimer: interrupt took 28426 ns
jack@tor /home/jack $ 

-- 

   Joakim

[-- Attachment #2: config.x86_64 --]
[-- Type: application/octet-stream, Size: 36915 bytes --]

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Sebastian Andrzej Siewior]

* Mike Galbraith | 2014-05-07 16:20:15 [+0200]:

>> > This looks like it reverse applies?
>> Nope, I thought Thomas's whacked the one he did on purpose.  
>I should learn to speak lockdep.  How about this instead?
>
>locking/rt: fix rt_read_trylock() lockdep annotation.
>
>rt-rw-lockdep-annotations.patch dropped a rwlock_acquire_read
>
>Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
>---
> kernel/locking/rt.c |    3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
>--- a/kernel/locking/rt.c
>+++ b/kernel/locking/rt.c
>@@ -217,7 +217,7 @@ int __lockfunc rt_read_trylock(rwlock_t
> 		migrate_disable();
> 		ret = rt_mutex_trylock(lock);
> 		if (ret)
>-			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
>+			rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);

Think we should drop rwlock_acquire_read() and use rwlock_acquire()
instead (like the previous patch) because on -RT there is no difference
between a read and a write lock. Or is there more to it?

> 		else
> 			migrate_enable();
> 

Sebastian

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Mike Galbraith]

On Fri, 2014-05-09 at 15:27 +0200, Sebastian Andrzej Siewior wrote: 
> * Mike Galbraith | 2014-05-07 16:20:15 [+0200]:
> 
> >> > This looks like it reverse applies?
> >> Nope, I thought Thomas's whacked the one he did on purpose.  
> >I should learn to speak lockdep.  How about this instead?
> >
> >locking/rt: fix rt_read_trylock() lockdep annotation.
> >
> >rt-rw-lockdep-annotations.patch dropped a rwlock_acquire_read
> >
> >Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> >---
> > kernel/locking/rt.c |    3 +--
> > 1 file changed, 1 insertion(+), 2 deletions(-)
> >
> >--- a/kernel/locking/rt.c
> >+++ b/kernel/locking/rt.c
> >@@ -217,7 +217,7 @@ int __lockfunc rt_read_trylock(rwlock_t
> > 		migrate_disable();
> > 		ret = rt_mutex_trylock(lock);
> > 		if (ret)
> >-			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
> >+			rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
> 
> Think we should drop rwlock_acquire_read() and use rwlock_acquire()
> instead (like the previous patch) because on -RT there is no difference
> between a read and a write lock. Or is there more to it?

No, I had done the right thing in the first patch, as confirmed by what
landed in 12-rt.  The lock ain't "shared" is why I whacked it.  You saw
what happened to my confidence in that logic when Steven spoke.  Eek :)

-Mike

Re: [patchlet] locking/rt: fix rt_read_lock() lockdep annotation.

[Steven Rostedt]

On Fri, 09 May 2014 16:18:00 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:


> No, I had done the right thing in the first patch, as confirmed by what
> landed in 12-rt.  The lock ain't "shared" is why I whacked it.  You saw
> what happened to my confidence in that logic when Steven spoke.  Eek :)

I was confused because the patch didn't apply to my tree :-)

-- Steve