[PATCH v2] oprofile: add SMP barriers for hrtimer hotplug code

[PATCH v2] oprofile: add SMP barriers for hrtimer hotplug code

[Will Deacon]

OProfile uses a CPU notifier to start and stop any hrtimers when CPUs change
between ONLINE and DEAD. A static int ctr_running is used to keep track of
the counter state.

This can lead to problems where writes to the state variable are re-ordered
with repect to reads of the variable occurring on other CPUs, meaning that
__oprofile_hrtimer_start may read ctr_running as 0 and not initialise the
hrtimer. Potential deadlock can occur in __oprofile_hrtimer_stop because
lock_hrtimer_base will poll until timer->base != NULL, which will never
happen.

This patch adds an smp_mb() before initialising the hrtimers to ensure that
ctr_running mirrors the correct counter state.

Cc: Santosh Shilimkar <santosh.shilimkar@ti.com>
Cc: Robert Richter <robert.richter@amd.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---

Changes since v1: Removed redundant barrier from stop() code.

 drivers/oprofile/timer_int.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/oprofile/timer_int.c b/drivers/oprofile/timer_int.c
index 0107251..a3a94ea 100644
--- a/drivers/oprofile/timer_int.c
+++ b/drivers/oprofile/timer_int.c
@@ -48,6 +48,7 @@ static int oprofile_hrtimer_start(void)
 {
 	get_online_cpus();
 	ctr_running = 1;
+	smp_mb();
 	on_each_cpu(__oprofile_hrtimer_start, NULL, 1);
 	put_online_cpus();
 	return 0;
-- 
1.7.0.4

RE: [PATCH v2] oprofile: add SMP barriers for hrtimer hotplug code

[Santosh Shilimkar]

> -----Original Message-----
> From: Will Deacon [mailto:will.deacon@arm.com]
> Sent: Monday, January 31, 2011 11:23 PM
> To: linux-kernel@vger.kernel.org; oprofile-list@lists.sf.net
> Cc: Will Deacon; Santosh Shilimkar; Robert Richter
> Subject: [PATCH v2] oprofile: add SMP barriers for hrtimer hotplug
> code
>
> OProfile uses a CPU notifier to start and stop any hrtimers when
> CPUs change
> between ONLINE and DEAD. A static int ctr_running is used to keep
> track of
> the counter state.
>
> This can lead to problems where writes to the state variable are re-
> ordered
> with repect to reads of the variable occurring on other CPUs,
> meaning that
> __oprofile_hrtimer_start may read ctr_running as 0 and not
> initialise the
> hrtimer. Potential deadlock can occur in __oprofile_hrtimer_stop
> because
> lock_hrtimer_base will poll until timer->base != NULL, which will
> never
> happen.
>
> This patch adds an smp_mb() before initialising the hrtimers to
> ensure that
> ctr_running mirrors the correct counter state.
>
> Cc: Santosh Shilimkar <santosh.shilimkar@ti.com>
> Cc: Robert Richter <robert.richter@amd.com>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
>
Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>

> Changes since v1: Removed redundant barrier from stop() code.
>
>  drivers/oprofile/timer_int.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/oprofile/timer_int.c
> b/drivers/oprofile/timer_int.c
> index 0107251..a3a94ea 100644
> --- a/drivers/oprofile/timer_int.c
> +++ b/drivers/oprofile/timer_int.c
> @@ -48,6 +48,7 @@ static int oprofile_hrtimer_start(void)
>  {
>  	get_online_cpus();
>  	ctr_running = 1;
> +	smp_mb();
>  	on_each_cpu(__oprofile_hrtimer_start, NULL, 1);
>  	put_online_cpus();
>  	return 0;
> --
> 1.7.0.4
>

Re: [PATCH v2] oprofile: add SMP barriers for hrtimer hotplug code

[Ming Lei]

Hi,

2011/2/1 Santosh Shilimkar <santosh.shilimkar@ti.com>:
>> -----Original Message-----
>> From: Will Deacon [mailto:will.deacon@arm.com]
>> diff --git a/drivers/oprofile/timer_int.c
>> b/drivers/oprofile/timer_int.c
>> index 0107251..a3a94ea 100644
>> --- a/drivers/oprofile/timer_int.c
>> +++ b/drivers/oprofile/timer_int.c
>> @@ -48,6 +48,7 @@ static int oprofile_hrtimer_start(void)
>>  {
>>       get_online_cpus();
>>       ctr_running = 1;
>> +     smp_mb();
>>       on_each_cpu(__oprofile_hrtimer_start, NULL, 1);

Just a little question:

Before calling __oprofile_hrtimer_start in other CPUs,
there have been some smp_mb executed already, such as called by
smp_call_function_many in current CPU, and called by
generic_smp_call_function_interrupt in the func-calling CPU,
so are these smp_mb enough for correct order of access
for 'ctr_running' ?

>>       put_online_cpus();
>>       return 0;

thanks,
-- 
Lei Ming

Re: [PATCH v2] oprofile: add SMP barriers for hrtimer hotplug code

[Will Deacon]

On Tue, 2011-02-01 at 08:03 +0000, Ming Lei wrote:
> Hi,
> 
Hello Lei Ming,

> 2011/2/1 Santosh Shilimkar <santosh.shilimkar@ti.com>:
> >> -----Original Message-----
> >> From: Will Deacon [mailto:will.deacon@arm.com]
> >> diff --git a/drivers/oprofile/timer_int.c
> >> b/drivers/oprofile/timer_int.c
> >> index 0107251..a3a94ea 100644
> >> --- a/drivers/oprofile/timer_int.c
> >> +++ b/drivers/oprofile/timer_int.c
> >> @@ -48,6 +48,7 @@ static int oprofile_hrtimer_start(void)
> >>  {
> >>       get_online_cpus();
> >>       ctr_running = 1;
> >> +     smp_mb();
> >>       on_each_cpu(__oprofile_hrtimer_start, NULL, 1);
> 
> Just a little question:
> 
> Before calling __oprofile_hrtimer_start in other CPUs,
> there have been some smp_mb executed already, such as called by
> smp_call_function_many in current CPU, and called by
> generic_smp_call_function_interrupt in the func-calling CPU,
> so are these smp_mb enough for correct order of access
> for 'ctr_running' ?

I didn't dive into the guts of the cross-calling implementation, but it
appears that you're correct. There's also the csd_{un}lock functions
which have smp_mb() in them too, so I think we have plenty of barrier
goodness as it is.

I'll trash the patch.

Cheers,

Will