[PATCH 0/6] ppc time fixes

[PATCH 0/6] ppc time fixes

[Anton Blanchard]

Here is a roll up of various time fixes I have pending.

Anton

[PATCH 1/6] powerpc/time: Handle wrapping of decrementer

[Anton Blanchard]

When re-enabling interrupts we have code to handle edge sensitive
decrementers by resetting the decrementer to 1 whenever it is negative.
If interrupts were disabled long enough that the decrementer wrapped to
positive we do nothing. This means interrupts can be delayed for a long
time until it finally goes negative again.

While we hope interrupts are never be disabled long enough for the
decrementer to go positive, we have a very good test team that can
drive any kernel into the ground. The softlockup data we get back
from these fails could be seconds in the future, completely missing
the cause of the lockup.

We already keep track of the timebase of the next event so use that
to work out if we should trigger a decrementer exception.

Signed-off-by: Anton Blanchard <anton@samba.org>
Cc: stable@kernel.org
---

- I'd like to get this into -stable if possible since the reports are
  coming in on 3.0 and 3.1. It needs some solid review though.

- This has the unfortunate side effect of forcing arch_local_irq_restore
  to create a stack frame, but I will try to optimise that in a subsequent
  patch.

Index: linux-build/arch/powerpc/include/asm/time.h
===================================================================
--- linux-build.orig/arch/powerpc/include/asm/time.h	2011-11-17 10:04:16.543137415 +1100
+++ linux-build/arch/powerpc/include/asm/time.h	2011-11-17 13:19:47.831758211 +1100
@@ -219,5 +219,7 @@ DECLARE_PER_CPU(struct cpu_usage, cpu_us
 extern void secondary_cpu_time_init(void);
 extern void iSeries_time_init_early(void);
 
+extern void decrementer_check_overflow(void);
+
 #endif /* __KERNEL__ */
 #endif /* __POWERPC_TIME_H */
Index: linux-build/arch/powerpc/kernel/irq.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/irq.c	2011-11-17 10:04:16.551137554 +1100
+++ linux-build/arch/powerpc/kernel/irq.c	2011-11-17 14:23:10.834514143 +1100
@@ -164,16 +164,13 @@ notrace void arch_local_irq_restore(unsi
 	 */
 	local_paca->hard_enabled = en;
 
-#ifndef CONFIG_BOOKE
-	/* On server, re-trigger the decrementer if it went negative since
-	 * some processors only trigger on edge transitions of the sign bit.
-	 *
-	 * BookE has a level sensitive decrementer (latches in TSR) so we
-	 * don't need that
+	/*
+	 * Trigger the decrementer if we have a pending event. Some processors
+	 * only trigger on edge transitions of the sign bit. We might also
+	 * have disabled interrupts long enough that the decrementer wrapped
+	 * to positive.
 	 */
-	if ((int)mfspr(SPRN_DEC) < 0)
-		mtspr(SPRN_DEC, 1);
-#endif /* CONFIG_BOOKE */
+	decrementer_check_overflow();
 
 	/*
 	 * Force the delivery of pending soft-disabled interrupts on PS3.
Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 10:04:16.559137691 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 13:19:49.415785997 +1100
@@ -889,6 +889,15 @@ static void __init clocksource_init(void
 	       clock->name, clock->mult, clock->shift);
 }
 
+void decrementer_check_overflow(void)
+{
+	u64 now = get_tb_or_rtc();
+	struct decrementer_clock *decrementer = &__get_cpu_var(decrementers);
+
+	if (now >= decrementer->next_tb)
+		set_dec(1);
+}
+
 static int decrementer_set_next_event(unsigned long evt,
 				      struct clock_event_device *dev)
 {

Re: [PATCH 1/6] powerpc/time: Handle wrapping of decrementer

[Scott Wood]

On 11/24/2011 12:07 AM, Anton Blanchard wrote:
> Index: linux-build/arch/powerpc/kernel/irq.c
> ===================================================================
> --- linux-build.orig/arch/powerpc/kernel/irq.c	2011-11-17 10:04:16.551137554 +1100
> +++ linux-build/arch/powerpc/kernel/irq.c	2011-11-17 14:23:10.834514143 +1100
> @@ -164,16 +164,13 @@ notrace void arch_local_irq_restore(unsi
>  	 */
>  	local_paca->hard_enabled = en;
>  
> -#ifndef CONFIG_BOOKE
> -	/* On server, re-trigger the decrementer if it went negative since
> -	 * some processors only trigger on edge transitions of the sign bit.
> -	 *
> -	 * BookE has a level sensitive decrementer (latches in TSR) so we
> -	 * don't need that
> +	/*
> +	 * Trigger the decrementer if we have a pending event. Some processors
> +	 * only trigger on edge transitions of the sign bit. We might also
> +	 * have disabled interrupts long enough that the decrementer wrapped
> +	 * to positive.
>  	 */
> -	if ((int)mfspr(SPRN_DEC) < 0)
> -		mtspr(SPRN_DEC, 1);
> -#endif /* CONFIG_BOOKE */
> +	decrementer_check_overflow();

Where did the #ifndef CONFIG_BOOKE go?  BookE doesn't need this; the
interrupt will continue asserting until software clears TSR[DIS].

-Scott

Re: [PATCH 1/6] powerpc/time: Handle wrapping of decrementer

[Benjamin Herrenschmidt]

On Mon, 2011-11-28 at 11:44 -0600, Scott Wood wrote:
> > -#ifndef CONFIG_BOOKE
> > -     /* On server, re-trigger the decrementer if it went negative since
> > -      * some processors only trigger on edge transitions of the sign bit.
> > -      *
> > -      * BookE has a level sensitive decrementer (latches in TSR) so we
> > -      * don't need that
> > +     /*
> > +      * Trigger the decrementer if we have a pending event. Some processors
> > +      * only trigger on edge transitions of the sign bit. We might also
> > +      * have disabled interrupts long enough that the decrementer wrapped
> > +      * to positive.
> >        */
> > -     if ((int)mfspr(SPRN_DEC) < 0)
> > -             mtspr(SPRN_DEC, 1);
> > -#endif /* CONFIG_BOOKE */
> > +     decrementer_check_overflow();
> 
> Where did the #ifndef CONFIG_BOOKE go?  BookE doesn't need this; the
> interrupt will continue asserting until software clears TSR[DIS].

Ooops, I didnt notice Anton was removing it. Please send me a followup
patch to make decrementer_check_overflow() an empty inline on BookE.

Cheers,
Ben.

[PATCH 2/6] powerpc/time: Use clockevents_calc_mult_shift

[Anton Blanchard]

We can use clockevents_calc_mult_shift instead of doing all
the work ourselves.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 10:11:49.035001617 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 10:11:51.175038860 +1100
@@ -112,8 +112,6 @@ static void decrementer_set_mode(enum cl
 static struct clock_event_device decrementer_clockevent = {
        .name           = "decrementer",
        .rating         = 200,
-       .shift          = 0,	/* To be filled in */
-       .mult           = 0,	/* To be filled in */
        .irq            = 0,
        .set_next_event = decrementer_set_next_event,
        .set_mode       = decrementer_set_mode,
@@ -913,31 +911,6 @@ static void decrementer_set_mode(enum cl
 		decrementer_set_next_event(DECREMENTER_MAX, dev);
 }
 
-static inline uint64_t div_sc64(unsigned long ticks, unsigned long nsec,
-				int shift)
-{
-	uint64_t tmp = ((uint64_t)ticks) << shift;
-
-	do_div(tmp, nsec);
-	return tmp;
-}
-
-static void __init setup_clockevent_multiplier(unsigned long hz)
-{
-	u64 mult, shift = 32;
-
-	while (1) {
-		mult = div_sc64(hz, NSEC_PER_SEC, shift);
-		if (mult && (mult >> 32UL) == 0UL)
-			break;
-
-		shift--;
-	}
-
-	decrementer_clockevent.shift = shift;
-	decrementer_clockevent.mult = mult;
-}
-
 static void register_decrementer_clockevent(int cpu)
 {
 	struct clock_event_device *dec = &per_cpu(decrementers, cpu).event;
@@ -955,7 +928,8 @@ static void __init init_decrementer_cloc
 {
 	int cpu = smp_processor_id();
 
-	setup_clockevent_multiplier(ppc_tb_freq);
+	clockevents_calc_mult_shift(&decrementer_clockevent, ppc_tb_freq, 4);
+
 	decrementer_clockevent.max_delta_ns =
 		clockevent_delta2ns(DECREMENTER_MAX, &decrementer_clockevent);
 	decrementer_clockevent.min_delta_ns =

Re: [PATCH 2/6] powerpc/time: Use clockevents_calc_mult_shift

[Kumar Gala]

On Nov 24, 2011, at 12:07 AM, Anton Blanchard wrote:

> We can use clockevents_calc_mult_shift instead of doing all
> the work ourselves.
>=20
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
>=20
> Index: linux-build/arch/powerpc/kernel/time.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 =
10:11:49.035001617 +1100
> +++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 =
10:11:51.175038860 +1100
> @@ -112,8 +112,6 @@ static void decrementer_set_mode(enum cl
> static struct clock_event_device decrementer_clockevent =3D {
>        .name           =3D "decrementer",
>        .rating         =3D 200,
> -       .shift          =3D 0,	/* To be filled in */
> -       .mult           =3D 0,	/* To be filled in */
>        .irq            =3D 0,
>        .set_next_event =3D decrementer_set_next_event,
>        .set_mode       =3D decrementer_set_mode,
> @@ -913,31 +911,6 @@ static void decrementer_set_mode(enum cl
> 		decrementer_set_next_event(DECREMENTER_MAX, dev);
> }
>=20
> -static inline uint64_t div_sc64(unsigned long ticks, unsigned long =
nsec,
> -				int shift)
> -{
> -	uint64_t tmp =3D ((uint64_t)ticks) << shift;
> -
> -	do_div(tmp, nsec);
> -	return tmp;
> -}
> -
> -static void __init setup_clockevent_multiplier(unsigned long hz)
> -{
> -	u64 mult, shift =3D 32;
> -
> -	while (1) {
> -		mult =3D div_sc64(hz, NSEC_PER_SEC, shift);
> -		if (mult && (mult >> 32UL) =3D=3D 0UL)
> -			break;
> -
> -		shift--;
> -	}
> -
> -	decrementer_clockevent.shift =3D shift;
> -	decrementer_clockevent.mult =3D mult;
> -}
> -
> static void register_decrementer_clockevent(int cpu)
> {
> 	struct clock_event_device *dec =3D &per_cpu(decrementers, =
cpu).event;
> @@ -955,7 +928,8 @@ static void __init init_decrementer_cloc
> {
> 	int cpu =3D smp_processor_id();
>=20
> -	setup_clockevent_multiplier(ppc_tb_freq);
> +	clockevents_calc_mult_shift(&decrementer_clockevent, =
ppc_tb_freq, 4);
> +

Where's this magic 4 come from?

> 	decrementer_clockevent.max_delta_ns =3D
> 		clockevent_delta2ns(DECREMENTER_MAX, =
&decrementer_clockevent);
> 	decrementer_clockevent.min_delta_ns =3D
>=20
>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [PATCH 2/6] powerpc/time: Use clockevents_calc_mult_shift

[Anton Blanchard]

Hi Kumar,

> > static void register_decrementer_clockevent(int cpu)
> > {
> > 	struct clock_event_device *dec = &per_cpu(decrementers,
> > cpu).event; @@ -955,7 +928,8 @@ static void __init
> > init_decrementer_cloc {
> > 	int cpu = smp_processor_id();
> > 
> > -	setup_clockevent_multiplier(ppc_tb_freq);
> > +	clockevents_calc_mult_shift(&decrementer_clockevent,
> > ppc_tb_freq, 4); +
> 
> Where's this magic 4 come from?

No better reason than that's what most other users do. We weren't
placing any limits on the shift/multiply in the old loop I don't think
we need to, so we could probably just use 1 instead.

Anton

[PATCH 3/6] powerpc/time: Use clocksource_register_hz

[Anton Blanchard]

Use clocksource_register_hz which calculates the shift/mult
factors for us. Also remove the shift = 22 assumption in
vsyscall_update - thanks to Paul Mackerras and John Stultz for
catching that.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 10:11:51.175038860 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 10:11:55.547114957 +1100
@@ -86,8 +86,6 @@ static struct clocksource clocksource_rt
 	.rating       = 400,
 	.flags        = CLOCK_SOURCE_IS_CONTINUOUS,
 	.mask         = CLOCKSOURCE_MASK(64),
-	.shift        = 22,
-	.mult         = 0,	/* To be filled in */
 	.read         = rtc_read,
 };
 
@@ -97,8 +95,6 @@ static struct clocksource clocksource_ti
 	.rating       = 400,
 	.flags        = CLOCK_SOURCE_IS_CONTINUOUS,
 	.mask         = CLOCKSOURCE_MASK(64),
-	.shift        = 22,
-	.mult         = 0,	/* To be filled in */
 	.read         = timebase_read,
 };
 
@@ -822,9 +818,8 @@ void update_vsyscall(struct timespec *wa
 	++vdso_data->tb_update_count;
 	smp_mb();
 
-	/* XXX this assumes clock->shift == 22 */
-	/* 4611686018 ~= 2^(20+64-22) / 1e9 */
-	new_tb_to_xs = (u64) mult * 4611686018ULL;
+	/* 19342813113834067 ~= 2^(20+64) / 1e9 */
+	new_tb_to_xs = (u64) mult * (19342813113834067ULL >> clock->shift);
 	new_stamp_xsec = (u64) wall_time->tv_nsec * XSEC_PER_SEC;
 	do_div(new_stamp_xsec, 1000000000);
 	new_stamp_xsec += (u64) wall_time->tv_sec * XSEC_PER_SEC;
@@ -875,9 +870,7 @@ static void __init clocksource_init(void
 	else
 		clock = &clocksource_timebase;
 
-	clock->mult = clocksource_hz2mult(tb_ticks_per_sec, clock->shift);
-
-	if (clocksource_register(clock)) {
+	if (clocksource_register_hz(clock, tb_ticks_per_sec)) {
 		printk(KERN_ERR "clocksource: %s is already registered\n",
 		       clock->name);
 		return;

Re: [PATCH 3/6] powerpc/time: Use clocksource_register_hz

[john stultz]

On Thu, 2011-11-24 at 17:07 +1100, Anton Blanchard wrote:
> plain text document attachment (clock3)
> Use clocksource_register_hz which calculates the shift/mult
> factors for us. Also remove the shift = 22 assumption in
> vsyscall_update - thanks to Paul Mackerras and John Stultz for
> catching that.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
> 
> Index: linux-build/arch/powerpc/kernel/time.c
> ===================================================================
> --- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 10:11:51.175038860 +1100
> +++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 10:11:55.547114957 +1100
> @@ -86,8 +86,6 @@ static struct clocksource clocksource_rt
>  	.rating       = 400,
>  	.flags        = CLOCK_SOURCE_IS_CONTINUOUS,
>  	.mask         = CLOCKSOURCE_MASK(64),
> -	.shift        = 22,
> -	.mult         = 0,	/* To be filled in */
>  	.read         = rtc_read,
>  };
> 
> @@ -97,8 +95,6 @@ static struct clocksource clocksource_ti
>  	.rating       = 400,
>  	.flags        = CLOCK_SOURCE_IS_CONTINUOUS,
>  	.mask         = CLOCKSOURCE_MASK(64),
> -	.shift        = 22,
> -	.mult         = 0,	/* To be filled in */
>  	.read         = timebase_read,
>  };
> 
> @@ -822,9 +818,8 @@ void update_vsyscall(struct timespec *wa
>  	++vdso_data->tb_update_count;
>  	smp_mb();
> 
> -	/* XXX this assumes clock->shift == 22 */
> -	/* 4611686018 ~= 2^(20+64-22) / 1e9 */
> -	new_tb_to_xs = (u64) mult * 4611686018ULL;
> +	/* 19342813113834067 ~= 2^(20+64) / 1e9 */
> +	new_tb_to_xs = (u64) mult * (19342813113834067ULL >> clock->shift);

I never verified the math on this, but assuming Paul had it right, this
patch looks good!

Acked-by: John Stultz <johnstul@us.ibm.com>

Thanks again!
-john

[PATCH 4/6] powerpc/time: Remove unnecessary sanity check of decrementer expiration

[Anton Blanchard]

The clockevents code uses max_delta_ns to avoid calling a
clockevent with too large a value.

Remove the redundant version of this in the timer_interrupt
code.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 10:12:28.891690395 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 10:13:54.241143987 +1100
@@ -572,7 +572,6 @@ void timer_interrupt(struct pt_regs * re
 	struct pt_regs *old_regs;
 	struct decrementer_clock *decrementer =  &__get_cpu_var(decrementers);
 	struct clock_event_device *evt = &decrementer->event;
-	u64 now;
 
 	/* Ensure a positive value is written to the decrementer, or else
 	 * some CPUs will continue to take decrementer exceptions.
@@ -607,16 +606,9 @@ void timer_interrupt(struct pt_regs * re
 		get_lppaca()->int_dword.fields.decr_int = 0;
 #endif
 
-	now = get_tb_or_rtc();
-	if (now >= decrementer->next_tb) {
-		decrementer->next_tb = ~(u64)0;
-		if (evt->event_handler)
-			evt->event_handler(evt);
-	} else {
-		now = decrementer->next_tb - now;
-		if (now <= DECREMENTER_MAX)
-			set_dec((int)now);
-	}
+	decrementer->next_tb = ~(u64)0;
+	if (evt->event_handler)
+		evt->event_handler(evt);
 
 #ifdef CONFIG_PPC_ISERIES
 	if (firmware_has_feature(FW_FEATURE_ISERIES) && hvlpevent_is_pending())

[PATCH 5/6] powerpc/time: Fix some style issues

[Anton Blanchard]

Fix some formatting issues and use the DECREMENTER_MAX
define instead of 0x7fffffff.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-24 16:49:25.853715846 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-24 16:51:04.651432999 +1100
@@ -106,12 +106,12 @@ static void decrementer_set_mode(enum cl
 				 struct clock_event_device *dev);
 
 static struct clock_event_device decrementer_clockevent = {
-       .name           = "decrementer",
-       .rating         = 200,
-       .irq            = 0,
-       .set_next_event = decrementer_set_next_event,
-       .set_mode       = decrementer_set_mode,
-       .features       = CLOCK_EVT_FEAT_ONESHOT,
+	.name           = "decrementer",
+	.rating         = 200,
+	.irq            = 0,
+	.set_next_event = decrementer_set_next_event,
+	.set_mode       = decrementer_set_mode,
+	.features       = CLOCK_EVT_FEAT_ONESHOT,
 };
 
 struct decrementer_clock {
@@ -435,7 +435,7 @@ EXPORT_SYMBOL(profile_pc);
 /* 
  * This function recalibrates the timebase based on the 49-bit time-of-day
  * value in the Titan chip.  The Titan is much more accurate than the value
- * returned by the service processor for the timebase frequency.  
+ * returned by the service processor for the timebase frequency.
  */
 
 static int __init iSeries_tb_recal(void)
@@ -636,9 +636,9 @@ static void generic_suspend_disable_irqs
 	 * with suspending.
 	 */
 
-	set_dec(0x7fffffff);
+	set_dec(DECREMENTER_MAX);
 	local_irq_disable();
-	set_dec(0x7fffffff);
+	set_dec(DECREMENTER_MAX);
 }
 
 static void generic_suspend_enable_irqs(void)
@@ -982,10 +982,10 @@ void __init time_init(void)
 	boot_tb = get_tb_or_rtc();
 
 	/* If platform provided a timezone (pmac), we correct the time */
-        if (timezone_offset) {
+	if (timezone_offset) {
 		sys_tz.tz_minuteswest = -timezone_offset / 60;
 		sys_tz.tz_dsttime = 0;
-        }
+	}
 
 	vdso_data->tb_update_count = 0;
 	vdso_data->tb_ticks_per_sec = tb_ticks_per_sec;

[PATCH 6/6] powerpc/time: Optimise decrementer_check_overflow

[Anton Blanchard]

decrementer_check_overflow is called from arch_local_irq_restore so
we want to make it as light weight as possible. As such, turn
decrementer_check_overflow into an inline function.

To avoid a circular mess of includes, separate out the two components
of struct decrementer_clock and keep the struct clock_event_device
part local to time.c.

The fast path improves from:

arch_local_irq_restore
     0:       mflr    r0
     4:       std     r0,16(r1)
     8:       stdu    r1,-112(r1)
     c:       stb     r3,578(r13)
    10:       cmpdi   cr7,r3,0
    14:       beq-    cr7,24 <.arch_local_irq_restore+0x24>
...
    24:       addi    r1,r1,112
    28:       ld      r0,16(r1)
    2c:       mtlr    r0
    30:       blr

to:

arch_local_irq_restore
    0:       std     r30,-16(r1)
    4:       ld      r30,0(r2)
    8:       stb     r3,578(r13)
    c:       cmpdi   cr7,r3,0
   10:       beq-    cr7,6c <.arch_local_irq_restore+0x6c>
...
   6c:       ld      r30,-16(r1)
   70:       blr

Unfortunately we still setup a local TOC (due to -mminimal-toc). Yet
another sign we should be moving to -mcmodel=medium.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux-build/arch/powerpc/kernel/time.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/time.c	2011-11-17 13:07:26.638774436 +1100
+++ linux-build/arch/powerpc/kernel/time.c	2011-11-17 13:13:22.409001783 +1100
@@ -114,12 +114,8 @@ static struct clock_event_device decreme
 	.features	= CLOCK_EVT_FEAT_ONESHOT,
 };
 
-struct decrementer_clock {
-	struct clock_event_device event;
-	u64 next_tb;
-};
-
-static DEFINE_PER_CPU(struct decrementer_clock, decrementers);
+DEFINE_PER_CPU(u64, decrementers_next_tb);
+static DEFINE_PER_CPU(struct clock_event_device, decrementers);
 
 #ifdef CONFIG_PPC_ISERIES
 static unsigned long __initdata iSeries_recal_titan;
@@ -570,8 +566,8 @@ void arch_irq_work_raise(void)
 void timer_interrupt(struct pt_regs * regs)
 {
 	struct pt_regs *old_regs;
-	struct decrementer_clock *decrementer =  &__get_cpu_var(decrementers);
-	struct clock_event_device *evt = &decrementer->event;
+	u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
+	struct clock_event_device *evt = &__get_cpu_var(decrementers);
 
 	/* Ensure a positive value is written to the decrementer, or else
 	 * some CPUs will continue to take decrementer exceptions.
@@ -606,7 +602,7 @@ void timer_interrupt(struct pt_regs * re
 		get_lppaca()->int_dword.fields.decr_int = 0;
 #endif
 
-	decrementer->next_tb = ~(u64)0;
+	*next_tb = ~(u64)0;
 	if (evt->event_handler)
 		evt->event_handler(evt);
 
@@ -872,19 +868,10 @@ static void __init clocksource_init(void
 	       clock->name, clock->mult, clock->shift);
 }
 
-void decrementer_check_overflow(void)
-{
-	u64 now = get_tb_or_rtc();
-	struct decrementer_clock *decrementer = &__get_cpu_var(decrementers);
-
-	if (now >= decrementer->next_tb)
-		set_dec(1);
-}
-
 static int decrementer_set_next_event(unsigned long evt,
 				      struct clock_event_device *dev)
 {
-	__get_cpu_var(decrementers).next_tb = get_tb_or_rtc() + evt;
+	__get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
 	set_dec(evt);
 	return 0;
 }
@@ -898,7 +885,7 @@ static void decrementer_set_mode(enum cl
 
 static void register_decrementer_clockevent(int cpu)
 {
-	struct clock_event_device *dec = &per_cpu(decrementers, cpu).event;
+	struct clock_event_device *dec = &per_cpu(decrementers, cpu);
 
 	*dec = decrementer_clockevent;
 	dec->cpumask = cpumask_of(cpu);
Index: linux-build/arch/powerpc/include/asm/time.h
===================================================================
--- linux-build.orig/arch/powerpc/include/asm/time.h	2011-11-17 13:07:30.438840750 +1100
+++ linux-build/arch/powerpc/include/asm/time.h	2011-11-17 13:13:05.492705459 +1100
@@ -219,7 +219,7 @@ DECLARE_PER_CPU(struct cpu_usage, cpu_us
 extern void secondary_cpu_time_init(void);
 extern void iSeries_time_init_early(void);
 
-extern void decrementer_check_overflow(void);
+DECLARE_PER_CPU(u64, decrementers_next_tb);
 
 #endif /* __KERNEL__ */
 #endif /* __POWERPC_TIME_H */
Index: linux-build/arch/powerpc/kernel/irq.c
===================================================================
--- linux-build.orig/arch/powerpc/kernel/irq.c	2011-11-17 13:07:26.662774855 +1100
+++ linux-build/arch/powerpc/kernel/irq.c	2011-11-17 13:15:02.038747391 +1100
@@ -115,6 +115,15 @@ static inline notrace void set_soft_enab
 	: : "r" (enable), "i" (offsetof(struct paca_struct, soft_enabled)));
 }
 
+static inline notrace void decrementer_check_overflow(void)
+{
+	u64 now = get_tb_or_rtc();
+	u64 *next_tb = &__get_cpu_var(decrementers_next_tb);
+
+	if (now >= *next_tb)
+		set_dec(1);
+}
+
 notrace void arch_local_irq_restore(unsigned long en)
 {
 	/*