Re: [PATCH v5 UPDATEDv3 3/3] CPU hotplug, smp: Flush any pending IPI callbacks before CPU offline

["Srivatsa S. Bhat" <srivatsa.bhat@linux.vnet.ibm.com>]

On 05/20/2014 03:12 PM, Peter Zijlstra wrote:
> On Tue, May 20, 2014 at 01:52:41AM +0530, Srivatsa S. Bhat wrote:
>> From: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
>> [PATCH v5 UPDATEDv3 3/3] CPU hotplug, smp: Flush any pending IPI callbacks before CPU offline
>>
>> During CPU offline, in the stop-machine loop, we use 2 separate stages to
>> disable interrupts, to ensure that the CPU going offline doesn't get any new
>> IPIs from the other CPUs after it has gone offline.
>>
>> However, an IPI sent much earlier might arrive late on the target CPU
>> (possibly _after_ the CPU has gone offline) due to hardware latencies,
>> and due to this, the smp-call-function callbacks queued on the outgoing
>> CPU might not get noticed (and hence not executed) at all.
>>
>> This is somewhat theoretical, but in any case, it makes sense to explicitly
>> loop through the call_single_queue and flush any pending callbacks before the
>> CPU goes completely offline. So, flush the queued smp-call-function callbacks
>> in the MULTI_STOP_DISABLE_IRQ_ACTIVE stage, after disabling interrupts on the
>> active CPU. This can be trivially achieved by invoking the
>> generic_smp_call_function_single_interrupt() function itself (and since the
>> outgoing CPU is still online at this point, we won't trigger the "IPI to offline
>> CPU" warning in this function; so we are safe to call it here).
>>
>> This way, we would have handled all the queued callbacks before going offline,
>> and also, no new IPIs can be sent by the other CPUs to the outgoing CPU at that
>> point, because they will all be executing the stop-machine code with interrupts
>> disabled.
>>
>> Suggested-by: Frederic Weisbecker <fweisbec@gmail.com>
>> Reviewed-by: Tejun Heo <tj@kernel.org>
>> Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
>> ---
>>
>>  include/linux/smp.h   |    2 ++
>>  kernel/smp.c          |   17 ++++++++++++++---
>>  kernel/stop_machine.c |   11 +++++++++++
>>  3 files changed, 27 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/smp.h b/include/linux/smp.h
>> index 633f5ed..e6b090d 100644
>> --- a/include/linux/smp.h
>> +++ b/include/linux/smp.h
>> @@ -151,6 +151,8 @@ smp_call_function_any(const struct cpumask *mask, smp_call_func_t func,
>>  
>>  static inline void kick_all_cpus_sync(void) {  }
>>  
>> +static inline void generic_smp_call_function_single_interrupt(void) { }
>> +
>>  #endif /* !SMP */
>>  
>>  /*
>> diff --git a/kernel/smp.c b/kernel/smp.c
>> index 306f818..b765167 100644
>> --- a/kernel/smp.c
>> +++ b/kernel/smp.c
>> @@ -177,9 +177,18 @@ static int generic_exec_single(int cpu, struct call_single_data *csd,
>>  	return 0;
>>  }
>>  
>> -/*
>> - * Invoked by arch to handle an IPI for call function single. Must be
>> - * called from the arch with interrupts disabled.
>> +/**
>> + * generic_smp_call_function_single_interrupt - Execute SMP IPI callbacks
>> + *
>> + * Invoked by arch to handle an IPI for call function single.
>> + *
>> + * This is also invoked by a CPU about to go offline, to flush any pending
>> + * smp-call-function callbacks queued on this CPU (including those for which
>> + * the source CPU's IPIs might not have been received on this CPU yet).
>> + * This ensures that all pending IPI callbacks are run before the CPU goes
>> + * completely offline.
>> + *
>> + * Must be called with interrupts disabled.
>>   */
>>  void generic_smp_call_function_single_interrupt(void)
>>  {
>> @@ -187,6 +196,8 @@ void generic_smp_call_function_single_interrupt(void)
>>  	struct call_single_data *csd, *csd_next;
>>  	static bool warned;
>>  
>> +	WARN_ON(!irqs_disabled());
>> +
>>  	entry = llist_del_all(&__get_cpu_var(call_single_queue));
>>  	entry = llist_reverse_order(entry);
>>  
>> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
>> index 288f7fe..4069c13 100644
>> --- a/kernel/stop_machine.c
>> +++ b/kernel/stop_machine.c
>> @@ -21,6 +21,7 @@
>>  #include <linux/smpboot.h>
>>  #include <linux/atomic.h>
>>  #include <linux/lglock.h>
>> +#include <linux/smp.h>
>>  
>>  /*
>>   * Structure to determine completion condition and record errors.  May
>> @@ -223,6 +224,16 @@ static int multi_cpu_stop(void *data)
>>  				if (is_active) {
>>  					local_irq_disable();
>>  					hard_irq_disable();
>> +
>> +					/*
>> +					 * IPIs (from the inactive CPUs) might
>> +					 * arrive late due to hardware latencies.
>> +					 * So flush out any pending IPI callbacks
>> +					 * explicitly, to ensure that the outgoing
>> +					 * CPU doesn't go offline with work still
>> +					 * pending (during CPU hotplug).
>> +					 */
>> +					generic_smp_call_function_single_interrupt();
>>  				}
>>  				break;
>>  			case MULTI_STOP_RUN:
> 
> The multi_cpu_stop() path isn't exclusive to hotplug, so your changelog
> is wrong or the patch is.
> 

Yes, I know that multi_cpu_stop() isn't exclusive to hotplug. That's why
I have explicitly referred to CPU hotplug in the comment as well as the
changelog.

But I totally agree that code-wise this is not the best way to do it since
this affects (although harmlessly) usecases other than hotplug as well.

Do you have any other suggestions?

> Furthermore, you yourself have worked on trying to remove stop machine
> from hotplug, that too should've been a big hint this is the wrong place
> for this.
> 

I know :-( But for lack of any other solution (other than getting rid of
stop-machine from the hotplug path), I went with this rather poor fix, as
a stop-gap arrangement until we implement the long-term solution (stop-
machine-free CPU hotplug).

While we are on the topic of stop-machine removal, I'll try to revive that
patchset again and hope that this time it will get more reviews and probably
get a chance to go upstream. That should solve a _lot_ of problems with
hotplug!

Regards,
Srivatsa S. Bhat