preempt rcu bug on s390

preempt rcu bug on s390

[Heiko Carstens]

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle
and not advancing anymore. However there is the init process which is
waitung for synchronize_rcu() to complete (lcrash output):

STACK TRACE FOR TASK: 0xf84d968 (swapper)

 STACK:
 0 schedule+842 [0x36c956]
 1 schedule_timeout+172 [0x36d0e4]
 2 wait_for_common+204 [0x36c398]
 3 synchronize_rcu+76 [0x567bc]
 4 netlink_change_ngroups+150 [0x2b4302]
 5 genl_register_mc_group+256 [0x2b6174]
 6 genl_init+188 [0x534e44]
 7 kernel_init+444 [0x518334]
 8 kernel_thread_starter+6 [0x192a6]

If I change the code so that timer ticks won't be disabled everything
runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

Re: preempt rcu bug on s390

[Paul E. McKenney]

On Sat, Feb 09, 2008 at 12:34:35PM +0100, Heiko Carstens wrote:
> Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
> gets stuck when running with more than one cpu.
> When booting with four cpus I get all four cpus caught withing cpu_idle
> and not advancing anymore. However there is the init process which is
> waitung for synchronize_rcu() to complete (lcrash output):
> 
> STACK TRACE FOR TASK: 0xf84d968 (swapper)
> 
>  STACK:
>  0 schedule+842 [0x36c956]
>  1 schedule_timeout+172 [0x36d0e4]
>  2 wait_for_common+204 [0x36c398]
>  3 synchronize_rcu+76 [0x567bc]
>  4 netlink_change_ngroups+150 [0x2b4302]
>  5 genl_register_mc_group+256 [0x2b6174]
>  6 genl_init+188 [0x534e44]
>  7 kernel_init+444 [0x518334]
>  8 kernel_thread_starter+6 [0x192a6]
> 
> If I change the code so that timer ticks won't be disabled everything
> runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
> thing for the rcu preemptible case.
> 
> Kernel version is git head of today.
> 
> Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

If not, could you please check it out?

						Thanx, Paul

Re: preempt rcu bug on s390

[Heiko Carstens]

On Sat, Feb 09, 2008 at 06:07:11AM -0800, Paul E. McKenney wrote:
> On Sat, Feb 09, 2008 at 12:34:35PM +0100, Heiko Carstens wrote:
> > Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
> > gets stuck when running with more than one cpu.
> > When booting with four cpus I get all four cpus caught withing cpu_idle
> > and not advancing anymore. However there is the init process which is
> > waitung for synchronize_rcu() to complete (lcrash output):
> > 
> > STACK TRACE FOR TASK: 0xf84d968 (swapper)
> > 
> >  STACK:
> >  0 schedule+842 [0x36c956]
> >  1 schedule_timeout+172 [0x36d0e4]
> >  2 wait_for_common+204 [0x36c398]
> >  3 synchronize_rcu+76 [0x567bc]
> >  4 netlink_change_ngroups+150 [0x2b4302]
> >  5 genl_register_mc_group+256 [0x2b6174]
> >  6 genl_init+188 [0x534e44]
> >  7 kernel_init+444 [0x518334]
> >  8 kernel_thread_starter+6 [0x192a6]
> > 
> > If I change the code so that timer ticks won't be disabled everything
> > runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
> > thing for the rcu preemptible case.
> > 
> > Kernel version is git head of today.
> > 
> > Any ideas?
> 
> Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?
> 
> If not, could you please check it out?

It's not applied, however it doesn't change anything. Also the patch
is tied to the dynticks implementation which is differently from
s390's nohz implementation.
I had to add the patch below so it would make at least some sense.
But it doesn't fix the problem.

---
 arch/s390/kernel/time.c |    2 ++
 include/linux/hardirq.h |    2 +-
 kernel/rcupreempt.c     |    2 +-
 3 files changed, 4 insertions(+), 2 deletions(-)

Index: linux-2.6/kernel/rcupreempt.c
===================================================================
--- linux-2.6.orig/kernel/rcupreempt.c
+++ linux-2.6/kernel/rcupreempt.c
@@ -413,7 +413,7 @@ static void __rcu_advance_callbacks(stru
 	}
 }
 
-#ifdef CONFIG_NO_HZ
+#if defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ)
 
 DEFINE_PER_CPU(long, dynticks_progress_counter) = 1;
 static DEFINE_PER_CPU(long, rcu_dyntick_snapshot);
Index: linux-2.6/arch/s390/kernel/time.c
===================================================================
--- linux-2.6.orig/arch/s390/kernel/time.c
+++ linux-2.6/arch/s390/kernel/time.c
@@ -200,6 +200,7 @@ static void stop_hz_timer(void)
 		if (timer >= jiffies_timer_cc)
 			todval = timer;
 	}
+	rcu_enter_nohz();
 	set_clock_comparator(todval);
 }
 
@@ -213,6 +214,7 @@ static void start_hz_timer(void)
 
 	if (!cpu_isset(smp_processor_id(), nohz_cpu_mask))
 		return;
+	rcu_exit_nohz();
 	account_ticks(get_clock());
 	set_clock_comparator(S390_lowcore.jiffy_timer + CPU_DEVIATION);
 	cpu_clear(smp_processor_id(), nohz_cpu_mask);
Index: linux-2.6/include/linux/hardirq.h
===================================================================
--- linux-2.6.orig/include/linux/hardirq.h
+++ linux-2.6/include/linux/hardirq.h
@@ -109,7 +109,7 @@ static inline void account_system_vtime(
 }
 #endif
 
-#if defined(CONFIG_PREEMPT_RCU) && defined(CONFIG_NO_HZ)
+#if defined(CONFIG_PREEMPT_RCU) && (defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ))
 extern void rcu_irq_enter(void);
 extern void rcu_irq_exit(void);
 #else

Re: preempt rcu bug on s390

[Paul E. McKenney]

On Sat, Feb 09, 2008 at 06:14:51PM +0100, Heiko Carstens wrote:
> On Sat, Feb 09, 2008 at 06:07:11AM -0800, Paul E. McKenney wrote:
> > On Sat, Feb 09, 2008 at 12:34:35PM +0100, Heiko Carstens wrote:
> > > Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
> > > gets stuck when running with more than one cpu.
> > > When booting with four cpus I get all four cpus caught withing cpu_idle
> > > and not advancing anymore. However there is the init process which is
> > > waitung for synchronize_rcu() to complete (lcrash output):
> > > 
> > > STACK TRACE FOR TASK: 0xf84d968 (swapper)
> > > 
> > >  STACK:
> > >  0 schedule+842 [0x36c956]
> > >  1 schedule_timeout+172 [0x36d0e4]
> > >  2 wait_for_common+204 [0x36c398]
> > >  3 synchronize_rcu+76 [0x567bc]
> > >  4 netlink_change_ngroups+150 [0x2b4302]
> > >  5 genl_register_mc_group+256 [0x2b6174]
> > >  6 genl_init+188 [0x534e44]
> > >  7 kernel_init+444 [0x518334]
> > >  8 kernel_thread_starter+6 [0x192a6]
> > > 
> > > If I change the code so that timer ticks won't be disabled everything
> > > runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
> > > thing for the rcu preemptible case.
> > > 
> > > Kernel version is git head of today.
> > > 
> > > Any ideas?
> > 
> > Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?
> > 
> > If not, could you please check it out?
> 
> It's not applied, however it doesn't change anything. Also the patch
> is tied to the dynticks implementation which is differently from
> s390's nohz implementation.
> I had to add the patch below so it would make at least some sense.
> But it doesn't fix the problem.

OK, I was afraid of that.  ;-)

Does s390 start out in nohz mode?  The reason I ask is that it feels like
an off-by-one error for the dynticks_progress_counter.

							Thanx, Paul

> ---
>  arch/s390/kernel/time.c |    2 ++
>  include/linux/hardirq.h |    2 +-
>  kernel/rcupreempt.c     |    2 +-
>  3 files changed, 4 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/kernel/rcupreempt.c
> ===================================================================
> --- linux-2.6.orig/kernel/rcupreempt.c
> +++ linux-2.6/kernel/rcupreempt.c
> @@ -413,7 +413,7 @@ static void __rcu_advance_callbacks(stru
>  	}
>  }
> 
> -#ifdef CONFIG_NO_HZ
> +#if defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ)
> 
>  DEFINE_PER_CPU(long, dynticks_progress_counter) = 1;
>  static DEFINE_PER_CPU(long, rcu_dyntick_snapshot);
> Index: linux-2.6/arch/s390/kernel/time.c
> ===================================================================
> --- linux-2.6.orig/arch/s390/kernel/time.c
> +++ linux-2.6/arch/s390/kernel/time.c
> @@ -200,6 +200,7 @@ static void stop_hz_timer(void)
>  		if (timer >= jiffies_timer_cc)
>  			todval = timer;
>  	}
> +	rcu_enter_nohz();
>  	set_clock_comparator(todval);
>  }
> 
> @@ -213,6 +214,7 @@ static void start_hz_timer(void)
> 
>  	if (!cpu_isset(smp_processor_id(), nohz_cpu_mask))
>  		return;
> +	rcu_exit_nohz();
>  	account_ticks(get_clock());
>  	set_clock_comparator(S390_lowcore.jiffy_timer + CPU_DEVIATION);
>  	cpu_clear(smp_processor_id(), nohz_cpu_mask);
> Index: linux-2.6/include/linux/hardirq.h
> ===================================================================
> --- linux-2.6.orig/include/linux/hardirq.h
> +++ linux-2.6/include/linux/hardirq.h
> @@ -109,7 +109,7 @@ static inline void account_system_vtime(
>  }
>  #endif
> 
> -#if defined(CONFIG_PREEMPT_RCU) && defined(CONFIG_NO_HZ)
> +#if defined(CONFIG_PREEMPT_RCU) && (defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ))
>  extern void rcu_irq_enter(void);
>  extern void rcu_irq_exit(void);
>  #else

Re: preempt rcu bug on s390

[Heiko Carstens]

> > > > Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
> > > > gets stuck when running with more than one cpu.
> > > > When booting with four cpus I get all four cpus caught withing cpu_idle
> > > > and not advancing anymore. However there is the init process which is
> > > > waitung for synchronize_rcu() to complete (lcrash output):
> > > > 
> > > > If I change the code so that timer ticks won't be disabled everything
> > > > runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
> > > > thing for the rcu preemptible case.
> > > > 
> > > > Kernel version is git head of today.
> > > > 
> > > > Any ideas?
> > > 
> > > Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?
> > > 
> > > If not, could you please check it out?
> > 
> > It's not applied, however it doesn't change anything. Also the patch
> > is tied to the dynticks implementation which is differently from
> > s390's nohz implementation.
> > I had to add the patch below so it would make at least some sense.
> > But it doesn't fix the problem.
> 
> OK, I was afraid of that.  ;-)
> 
> Does s390 start out in nohz mode?  The reason I ask is that it feels like
> an off-by-one error for the dynticks_progress_counter.

Actually I forgot to add a few ifdefs to make the code do something :)
That just reveals that we have a conflict with the dynticks implementation
and s390's nohz that shows up in what rcu_irq_enter/exit assume.
I didn't patch s390 and common code so it will work, but I think the
patch you mentionened will fix the problem I reported.
So I guess we should either convert s390 to use the generic dynticks
implementation or disable preemptible rcu on s390 until we converted
our code.

Thanks for helping debugging this!

Re: preempt rcu bug on s390

[Paul E. McKenney]

On Sun, Feb 10, 2008 at 02:01:50PM +0100, Heiko Carstens wrote:
> > > > > Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always
> > > > > gets stuck when running with more than one cpu.
> > > > > When booting with four cpus I get all four cpus caught withing cpu_idle
> > > > > and not advancing anymore. However there is the init process which is
> > > > > waitung for synchronize_rcu() to complete (lcrash output):
> > > > > 
> > > > > If I change the code so that timer ticks won't be disabled everything
> > > > > runs fine. So my guess is that rcu_needs_cpu() doesn't do the right
> > > > > thing for the rcu preemptible case.
> > > > > 
> > > > > Kernel version is git head of today.
> > > > > 
> > > > > Any ideas?
> > > > 
> > > > Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?
> > > > 
> > > > If not, could you please check it out?
> > > 
> > > It's not applied, however it doesn't change anything. Also the patch
> > > is tied to the dynticks implementation which is differently from
> > > s390's nohz implementation.
> > > I had to add the patch below so it would make at least some sense.
> > > But it doesn't fix the problem.
> > 
> > OK, I was afraid of that.  ;-)
> > 
> > Does s390 start out in nohz mode?  The reason I ask is that it feels like
> > an off-by-one error for the dynticks_progress_counter.
> 
> Actually I forgot to add a few ifdefs to make the code do something :)
> That just reveals that we have a conflict with the dynticks implementation
> and s390's nohz that shows up in what rcu_irq_enter/exit assume.
> I didn't patch s390 and common code so it will work, but I think the
> patch you mentionened will fix the problem I reported.
> So I guess we should either convert s390 to use the generic dynticks
> implementation or disable preemptible rcu on s390 until we converted
> our code.

Sounds good to me!!!  (Especially converting s390 to generic algorithm.)

I believe that the generic implementation will do what you need, but
I am sure you will let me know of any problems that arise.

> Thanks for helping debugging this!

Thank you for tracking it down!

							Thanx, Paul

Re: preempt rcu bug on s390

[Steven Rostedt]

Heiko Carstens wrote:

>> Does s390 start out in nohz mode?  The reason I ask is that it feels like
>> an off-by-one error for the dynticks_progress_counter.
> 
> Actually I forgot to add a few ifdefs to make the code do something :)
> That just reveals that we have a conflict with the dynticks implementation
> and s390's nohz that shows up in what rcu_irq_enter/exit assume.
> I didn't patch s390 and common code so it will work, but I think the
> patch you mentionened will fix the problem I reported.
> So I guess we should either convert s390 to use the generic dynticks
> implementation or disable preemptible rcu on s390 until we converted
> our code.
> 
> Thanks for helping debugging this!

Heiko, thanks for reporting this.

This patch still didn't make it into -rc1, and it really should. Because 
without this patch, PREEMPT_RCU and NO_HZ together is broken, on all boxes.

The patch is in Ingo's sched-devel git tree, as 
9460545f81ea48b07dbb20456a8ede776d8ebc1b (last I checked) and titled:

     rcu: add support for dynamic ticks and preempt rcu


-- Steve