[PATCH 1/9] xen/smp: Fix leakage of timer interrupt line for every CPU online/offline.

[PATCH 1/9] xen/smp: Fix leakage of timer interrupt line for every CPU online/offline.

[Konrad Rzeszutek Wilk]

In the PVHVM path when we do CPU online/offline path we would
leak the timer%d IRQ line everytime we do a offline event. The
online path (xen_hvm_setup_cpu_clockevents via
x86_cpuinit.setup_percpu_clockev) would allocate a new interrupt
line for the timer%d.

But we would still use the old interrupt line leading to:

kernel BUG at /home/konrad/ssd/konrad/linux/kernel/hrtimer.c:1261!
invalid opcode: 0000 [#1] SMP
RIP: 0010:[<ffffffff810b9e21>]  [<ffffffff810b9e21>] hrtimer_interrupt+0x261/0x270
.. snip..
 <IRQ>
 [<ffffffff810445ef>] xen_timer_interrupt+0x2f/0x1b0
 [<ffffffff81104825>] ? stop_machine_cpu_stop+0xb5/0xf0
 [<ffffffff8111434c>] handle_irq_event_percpu+0x7c/0x240
 [<ffffffff811175b9>] handle_percpu_irq+0x49/0x70
 [<ffffffff813a74a3>] __xen_evtchn_do_upcall+0x1c3/0x2f0
 [<ffffffff813a760a>] xen_evtchn_do_upcall+0x2a/0x40
 [<ffffffff8167c26d>] xen_hvm_callback_vector+0x6d/0x80
 <EOI>
 [<ffffffff81666d01>] ? start_secondary+0x193/0x1a8
 [<ffffffff81666cfd>] ? start_secondary+0x18f/0x1a8

There is also the oddity (timer1) in the /proc/interrupts after
offlining CPU1:

  64:       1121          0  xen-percpu-virq      timer0
  78:          0          0  xen-percpu-virq      timer1
  84:          0       2483  xen-percpu-virq      timer2

This patch fixes it.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: stable@vger.kernel.org
---
 arch/x86/xen/smp.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 09ea61d..f80e69c 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -662,6 +662,7 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	unbind_from_irqhandler(per_cpu(xen_debug_irq, cpu), NULL);
 	unbind_from_irqhandler(per_cpu(xen_callfuncsingle_irq, cpu), NULL);
 	unbind_from_irqhandler(per_cpu(xen_irq_work, cpu), NULL);
+	xen_teardown_timer(cpu);
 	native_cpu_die(cpu);
 }
 
-- 
1.8.1.4

[PATCH 2/9] xen/smp/spinlock: Fix leakage of the spinlock interrupt line for every CPU online/offline

[Konrad Rzeszutek Wilk]

While we don't use the spinlock interrupt line (see for details
commit f10cd522c5fbfec9ae3cc01967868c9c2401ed23 -
xen: disable PV spinlocks on HVM) - we should still do the proper
init / deinit sequence. We did not do that correctly and for the
CPU init for PVHVM guest we would allocate an interrupt line - but
failed to deallocate the old interrupt line.

This resulted in leakage of an irq_desc but more importantly this splat
as we online an offlined CPU:

genirq: Flags mismatch irq 71. 0002cc20 (spinlock1) vs. 0002cc20 (spinlock1)
Pid: 2542, comm: init.late Not tainted 3.9.0-rc6upstream #1
Call Trace:
 [<ffffffff811156de>] __setup_irq+0x23e/0x4a0
 [<ffffffff81194191>] ? kmem_cache_alloc_trace+0x221/0x250
 [<ffffffff811161bb>] request_threaded_irq+0xfb/0x160
 [<ffffffff8104c6f0>] ? xen_spin_trylock+0x20/0x20
 [<ffffffff813a8423>] bind_ipi_to_irqhandler+0xa3/0x160
 [<ffffffff81303758>] ? kasprintf+0x38/0x40
 [<ffffffff8104c6f0>] ? xen_spin_trylock+0x20/0x20
 [<ffffffff810cad35>] ? update_max_interval+0x15/0x40
 [<ffffffff816605db>] xen_init_lock_cpu+0x3c/0x78
 [<ffffffff81660029>] xen_hvm_cpu_notify+0x29/0x33
 [<ffffffff81676bdd>] notifier_call_chain+0x4d/0x70
 [<ffffffff810bb2a9>] __raw_notifier_call_chain+0x9/0x10
 [<ffffffff8109402b>] __cpu_notify+0x1b/0x30
 [<ffffffff8166834a>] _cpu_up+0xa0/0x14b
 [<ffffffff816684ce>] cpu_up+0xd9/0xec
 [<ffffffff8165f754>] store_online+0x94/0xd0
 [<ffffffff8141d15b>] dev_attr_store+0x1b/0x20
 [<ffffffff81218f44>] sysfs_write_file+0xf4/0x170
 [<ffffffff811a2864>] vfs_write+0xb4/0x130
 [<ffffffff811a302a>] sys_write+0x5a/0xa0
 [<ffffffff8167ada9>] system_call_fastpath+0x16/0x1b
cpu 1 spinlock event irq -16
smpboot: Booting Node 0 Processor 1 APIC 0x2

And if one looks at the /proc/interrupts right after
offlining (CPU1):

  70:          0          0  xen-percpu-ipi       spinlock0
  71:          0          0  xen-percpu-ipi       spinlock1
  77:          0          0  xen-percpu-ipi       spinlock2

There is the oddity of the 'spinlock1' still being present.

CC: stable@vger.kernel.org
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/smp.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index f80e69c..22c800a 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -662,6 +662,7 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	unbind_from_irqhandler(per_cpu(xen_debug_irq, cpu), NULL);
 	unbind_from_irqhandler(per_cpu(xen_callfuncsingle_irq, cpu), NULL);
 	unbind_from_irqhandler(per_cpu(xen_irq_work, cpu), NULL);
+	xen_uninit_lock_cpu(cpu);
 	xen_teardown_timer(cpu);
 	native_cpu_die(cpu);
 }
-- 
1.8.1.4

[PATCH 3/9] xen/time: Fix kasprintf splat when allocating timer%d IRQ line.

[Konrad Rzeszutek Wilk]

When we online the CPU, we get this splat:

smpboot: Booting Node 0 Processor 1 APIC 0x2
installing Xen timer for CPU 1
BUG: sleeping function called from invalid context at /home/konrad/ssd/konrad/linux/mm/slab.c:3179
in_atomic(): 1, irqs_disabled(): 0, pid: 0, name: swapper/1
Pid: 0, comm: swapper/1 Not tainted 3.9.0-rc6upstream-00001-g3884fad #1
Call Trace:
 [<ffffffff810c1fea>] __might_sleep+0xda/0x100
 [<ffffffff81194617>] __kmalloc_track_caller+0x1e7/0x2c0
 [<ffffffff81303758>] ? kasprintf+0x38/0x40
 [<ffffffff813036eb>] kvasprintf+0x5b/0x90
 [<ffffffff81303758>] kasprintf+0x38/0x40
 [<ffffffff81044510>] xen_setup_timer+0x30/0xb0
 [<ffffffff810445af>] xen_hvm_setup_cpu_clockevents+0x1f/0x30
 [<ffffffff81666d0a>] start_secondary+0x19c/0x1a8

The solution to that is use kasprintf in the CPU hotplug path
that 'online's the CPU. That is, do it in in xen_hvm_cpu_notify,
and remove the call to in xen_hvm_setup_cpu_clockevents.

Unfortunatly the later is not a good idea as the bootup path
does not use xen_hvm_cpu_notify so we would end up never allocating
timer%d interrupt lines when booting. As such add the check for
atomic() to continue.

CC: stable@vger.kernel.org
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/enlighten.c | 5 ++++-
 arch/x86/xen/time.c      | 6 +++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 47d3243..ddbd54a 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1641,8 +1641,11 @@ static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
 	switch (action) {
 	case CPU_UP_PREPARE:
 		xen_vcpu_setup(cpu);
-		if (xen_have_vector_callback)
+		if (xen_have_vector_callback) {
 			xen_init_lock_cpu(cpu);
+			if (xen_feature(XENFEAT_hvm_safe_pvclock))
+				xen_setup_timer(cpu);
+		}
 		break;
 	default:
 		break;
diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 0296a95..054cc01 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -497,7 +497,11 @@ static void xen_hvm_setup_cpu_clockevents(void)
 {
 	int cpu = smp_processor_id();
 	xen_setup_runstate_info(cpu);
-	xen_setup_timer(cpu);
+	/*
+	 * xen_setup_timer(cpu) - snprintf is bad in atomic context. Hence
+	 * doing it xen_hvm_cpu_notify (which gets called by smp_init during
+	 * early bootup and also during CPU hotplug events).
+	 */
 	xen_setup_cpu_clockevents();
 }
 
-- 
1.8.1.4

Re: [PATCH 1/9] xen/smp: Fix leakage of timer interrupt line for every CPU online/offline.

[Stefano Stabellini]

On Tue, 16 Apr 2013, Konrad Rzeszutek Wilk wrote:
> In the PVHVM path when we do CPU online/offline path we would
> leak the timer%d IRQ line everytime we do a offline event. The
> online path (xen_hvm_setup_cpu_clockevents via
> x86_cpuinit.setup_percpu_clockev) would allocate a new interrupt
> line for the timer%d.
> 
> But we would still use the old interrupt line leading to:
> 
> kernel BUG at /home/konrad/ssd/konrad/linux/kernel/hrtimer.c:1261!
> invalid opcode: 0000 [#1] SMP
> RIP: 0010:[<ffffffff810b9e21>]  [<ffffffff810b9e21>] hrtimer_interrupt+0x261/0x270
> .. snip..
>  <IRQ>
>  [<ffffffff810445ef>] xen_timer_interrupt+0x2f/0x1b0
>  [<ffffffff81104825>] ? stop_machine_cpu_stop+0xb5/0xf0
>  [<ffffffff8111434c>] handle_irq_event_percpu+0x7c/0x240
>  [<ffffffff811175b9>] handle_percpu_irq+0x49/0x70
>  [<ffffffff813a74a3>] __xen_evtchn_do_upcall+0x1c3/0x2f0
>  [<ffffffff813a760a>] xen_evtchn_do_upcall+0x2a/0x40
>  [<ffffffff8167c26d>] xen_hvm_callback_vector+0x6d/0x80
>  <EOI>
>  [<ffffffff81666d01>] ? start_secondary+0x193/0x1a8
>  [<ffffffff81666cfd>] ? start_secondary+0x18f/0x1a8
> 
> There is also the oddity (timer1) in the /proc/interrupts after
> offlining CPU1:
> 
>   64:       1121          0  xen-percpu-virq      timer0
>   78:          0          0  xen-percpu-virq      timer1
>   84:          0       2483  xen-percpu-virq      timer2
> 
> This patch fixes it.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> CC: stable@vger.kernel.org

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>


>  arch/x86/xen/smp.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
> index 09ea61d..f80e69c 100644
> --- a/arch/x86/xen/smp.c
> +++ b/arch/x86/xen/smp.c
> @@ -662,6 +662,7 @@ static void xen_hvm_cpu_die(unsigned int cpu)
>  	unbind_from_irqhandler(per_cpu(xen_debug_irq, cpu), NULL);
>  	unbind_from_irqhandler(per_cpu(xen_callfuncsingle_irq, cpu), NULL);
>  	unbind_from_irqhandler(per_cpu(xen_irq_work, cpu), NULL);
> +	xen_teardown_timer(cpu);
>  	native_cpu_die(cpu);
>  }
>  
> -- 
> 1.8.1.4
> 

Re: [PATCH 2/9] xen/smp/spinlock: Fix leakage of the spinlock interrupt line for every CPU online/offline

[Stefano Stabellini]

On Tue, 16 Apr 2013, Konrad Rzeszutek Wilk wrote:
> While we don't use the spinlock interrupt line (see for details
> commit f10cd522c5fbfec9ae3cc01967868c9c2401ed23 -
> xen: disable PV spinlocks on HVM) - we should still do the proper
> init / deinit sequence. We did not do that correctly and for the
> CPU init for PVHVM guest we would allocate an interrupt line - but
> failed to deallocate the old interrupt line.
> 
> This resulted in leakage of an irq_desc but more importantly this splat
> as we online an offlined CPU:
> 
> genirq: Flags mismatch irq 71. 0002cc20 (spinlock1) vs. 0002cc20 (spinlock1)
> Pid: 2542, comm: init.late Not tainted 3.9.0-rc6upstream #1
> Call Trace:
>  [<ffffffff811156de>] __setup_irq+0x23e/0x4a0
>  [<ffffffff81194191>] ? kmem_cache_alloc_trace+0x221/0x250
>  [<ffffffff811161bb>] request_threaded_irq+0xfb/0x160
>  [<ffffffff8104c6f0>] ? xen_spin_trylock+0x20/0x20
>  [<ffffffff813a8423>] bind_ipi_to_irqhandler+0xa3/0x160
>  [<ffffffff81303758>] ? kasprintf+0x38/0x40
>  [<ffffffff8104c6f0>] ? xen_spin_trylock+0x20/0x20
>  [<ffffffff810cad35>] ? update_max_interval+0x15/0x40
>  [<ffffffff816605db>] xen_init_lock_cpu+0x3c/0x78
>  [<ffffffff81660029>] xen_hvm_cpu_notify+0x29/0x33
>  [<ffffffff81676bdd>] notifier_call_chain+0x4d/0x70
>  [<ffffffff810bb2a9>] __raw_notifier_call_chain+0x9/0x10
>  [<ffffffff8109402b>] __cpu_notify+0x1b/0x30
>  [<ffffffff8166834a>] _cpu_up+0xa0/0x14b
>  [<ffffffff816684ce>] cpu_up+0xd9/0xec
>  [<ffffffff8165f754>] store_online+0x94/0xd0
>  [<ffffffff8141d15b>] dev_attr_store+0x1b/0x20
>  [<ffffffff81218f44>] sysfs_write_file+0xf4/0x170
>  [<ffffffff811a2864>] vfs_write+0xb4/0x130
>  [<ffffffff811a302a>] sys_write+0x5a/0xa0
>  [<ffffffff8167ada9>] system_call_fastpath+0x16/0x1b
> cpu 1 spinlock event irq -16
> smpboot: Booting Node 0 Processor 1 APIC 0x2
> 
> And if one looks at the /proc/interrupts right after
> offlining (CPU1):
> 
>   70:          0          0  xen-percpu-ipi       spinlock0
>   71:          0          0  xen-percpu-ipi       spinlock1
>   77:          0          0  xen-percpu-ipi       spinlock2
> 
> There is the oddity of the 'spinlock1' still being present.
> 
> CC: stable@vger.kernel.org
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>


>  arch/x86/xen/smp.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
> index f80e69c..22c800a 100644
> --- a/arch/x86/xen/smp.c
> +++ b/arch/x86/xen/smp.c
> @@ -662,6 +662,7 @@ static void xen_hvm_cpu_die(unsigned int cpu)
>  	unbind_from_irqhandler(per_cpu(xen_debug_irq, cpu), NULL);
>  	unbind_from_irqhandler(per_cpu(xen_callfuncsingle_irq, cpu), NULL);
>  	unbind_from_irqhandler(per_cpu(xen_irq_work, cpu), NULL);
> +	xen_uninit_lock_cpu(cpu);
>  	xen_teardown_timer(cpu);
>  	native_cpu_die(cpu);
>  }
> -- 
> 1.8.1.4
> 

Re: [PATCH 3/9] xen/time: Fix kasprintf splat when allocating timer%d IRQ line.

[Stefano Stabellini]

On Tue, 16 Apr 2013, Konrad Rzeszutek Wilk wrote:
> When we online the CPU, we get this splat:
> 
> smpboot: Booting Node 0 Processor 1 APIC 0x2
> installing Xen timer for CPU 1
> BUG: sleeping function called from invalid context at /home/konrad/ssd/konrad/linux/mm/slab.c:3179
> in_atomic(): 1, irqs_disabled(): 0, pid: 0, name: swapper/1
> Pid: 0, comm: swapper/1 Not tainted 3.9.0-rc6upstream-00001-g3884fad #1
> Call Trace:
>  [<ffffffff810c1fea>] __might_sleep+0xda/0x100
>  [<ffffffff81194617>] __kmalloc_track_caller+0x1e7/0x2c0
>  [<ffffffff81303758>] ? kasprintf+0x38/0x40
>  [<ffffffff813036eb>] kvasprintf+0x5b/0x90
>  [<ffffffff81303758>] kasprintf+0x38/0x40
>  [<ffffffff81044510>] xen_setup_timer+0x30/0xb0
>  [<ffffffff810445af>] xen_hvm_setup_cpu_clockevents+0x1f/0x30
>  [<ffffffff81666d0a>] start_secondary+0x19c/0x1a8
> 
> The solution to that is use kasprintf in the CPU hotplug path
> that 'online's the CPU. That is, do it in in xen_hvm_cpu_notify,
> and remove the call to in xen_hvm_setup_cpu_clockevents.
> 
> Unfortunatly the later is not a good idea as the bootup path
> does not use xen_hvm_cpu_notify so we would end up never allocating
> timer%d interrupt lines when booting. As such add the check for
> atomic() to continue.

This last is not reflected in the code.

Also, is it actually OK to move xen_setup_timer out of
xen_hvm_setup_cpu_clockevents?

xen_setup_cpu_clockevents registers xen_clock_events as clocksource and
xen_clock_events is setup by xen_setup_timer so we need to make sure
that the call order remains the same.


> CC: stable@vger.kernel.org
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  arch/x86/xen/enlighten.c | 5 ++++-
>  arch/x86/xen/time.c      | 6 +++++-
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 47d3243..ddbd54a 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1641,8 +1641,11 @@ static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
>  	switch (action) {
>  	case CPU_UP_PREPARE:
>  		xen_vcpu_setup(cpu);
> -		if (xen_have_vector_callback)
> +		if (xen_have_vector_callback) {
>  			xen_init_lock_cpu(cpu);
> +			if (xen_feature(XENFEAT_hvm_safe_pvclock))
> +				xen_setup_timer(cpu);
> +		}
>  		break;
>  	default:
>  		break;
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 0296a95..054cc01 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -497,7 +497,11 @@ static void xen_hvm_setup_cpu_clockevents(void)
>  {
>  	int cpu = smp_processor_id();
>  	xen_setup_runstate_info(cpu);
> -	xen_setup_timer(cpu);
> +	/*
> +	 * xen_setup_timer(cpu) - snprintf is bad in atomic context. Hence
> +	 * doing it xen_hvm_cpu_notify (which gets called by smp_init during
> +	 * early bootup and also during CPU hotplug events).
> +	 */
>  	xen_setup_cpu_clockevents();
>  }
>  
> -- 
> 1.8.1.4
> 

Re: [PATCH 3/9] xen/time: Fix kasprintf splat when allocating timer%d IRQ line.

[Konrad Rzeszutek Wilk]

On Fri, Apr 26, 2013 at 05:11:35PM +0100, Stefano Stabellini wrote:
> On Tue, 16 Apr 2013, Konrad Rzeszutek Wilk wrote:
> > When we online the CPU, we get this splat:
> > 
> > smpboot: Booting Node 0 Processor 1 APIC 0x2
> > installing Xen timer for CPU 1
> > BUG: sleeping function called from invalid context at /home/konrad/ssd/konrad/linux/mm/slab.c:3179
> > in_atomic(): 1, irqs_disabled(): 0, pid: 0, name: swapper/1
> > Pid: 0, comm: swapper/1 Not tainted 3.9.0-rc6upstream-00001-g3884fad #1
> > Call Trace:
> >  [<ffffffff810c1fea>] __might_sleep+0xda/0x100
> >  [<ffffffff81194617>] __kmalloc_track_caller+0x1e7/0x2c0
> >  [<ffffffff81303758>] ? kasprintf+0x38/0x40
> >  [<ffffffff813036eb>] kvasprintf+0x5b/0x90
> >  [<ffffffff81303758>] kasprintf+0x38/0x40
> >  [<ffffffff81044510>] xen_setup_timer+0x30/0xb0
> >  [<ffffffff810445af>] xen_hvm_setup_cpu_clockevents+0x1f/0x30
> >  [<ffffffff81666d0a>] start_secondary+0x19c/0x1a8
> > 
> > The solution to that is use kasprintf in the CPU hotplug path
> > that 'online's the CPU. That is, do it in in xen_hvm_cpu_notify,
> > and remove the call to in xen_hvm_setup_cpu_clockevents.
> > 
> > Unfortunatly the later is not a good idea as the bootup path
> > does not use xen_hvm_cpu_notify so we would end up never allocating
> > timer%d interrupt lines when booting. As such add the check for
> > atomic() to continue.
> 
> This last is not reflected in the code.

I found out that it was not needed.
> 
> Also, is it actually OK to move xen_setup_timer out of
> xen_hvm_setup_cpu_clockevents?

Yes. It ends up being called earlier - in the notifier.
> 
> xen_setup_cpu_clockevents registers xen_clock_events as clocksource and
> xen_clock_events is setup by xen_setup_timer so we need to make sure
> that the call order remains the same.

The order is still the same.
> 
> 
> > CC: stable@vger.kernel.org
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > ---
> >  arch/x86/xen/enlighten.c | 5 ++++-
> >  arch/x86/xen/time.c      | 6 +++++-
> >  2 files changed, 9 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> > index 47d3243..ddbd54a 100644
> > --- a/arch/x86/xen/enlighten.c
> > +++ b/arch/x86/xen/enlighten.c
> > @@ -1641,8 +1641,11 @@ static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
> >  	switch (action) {
> >  	case CPU_UP_PREPARE:
> >  		xen_vcpu_setup(cpu);
> > -		if (xen_have_vector_callback)
> > +		if (xen_have_vector_callback) {
> >  			xen_init_lock_cpu(cpu);
> > +			if (xen_feature(XENFEAT_hvm_safe_pvclock))
> > +				xen_setup_timer(cpu);
> > +		}
> >  		break;
> >  	default:
> >  		break;
> > diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> > index 0296a95..054cc01 100644
> > --- a/arch/x86/xen/time.c
> > +++ b/arch/x86/xen/time.c
> > @@ -497,7 +497,11 @@ static void xen_hvm_setup_cpu_clockevents(void)
> >  {
> >  	int cpu = smp_processor_id();
> >  	xen_setup_runstate_info(cpu);
> > -	xen_setup_timer(cpu);
> > +	/*
> > +	 * xen_setup_timer(cpu) - snprintf is bad in atomic context. Hence
> > +	 * doing it xen_hvm_cpu_notify (which gets called by smp_init during
> > +	 * early bootup and also during CPU hotplug events).
> > +	 */
> >  	xen_setup_cpu_clockevents();
> >  }
> >  
> > -- 
> > 1.8.1.4
> >