[RFC][PATCH 8/9] smp: Make smp_call_function_single_async() safer

Peter Zijlstra
2020-07-22T15:33:33Z

Make smp_call_function_single_async() a safer and more convenient
interface by using an atomic op for setting CSD_FLAG_LOCK. This allows
safe concurrent use of this function as would be expected by the
-EBUSY return value.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/smp.c | 28 ++++++++++++++++++++--------
1 file changed, 20 insertions(+), 8 deletions(-)

--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -297,6 +297,13 @@ static void flush_smp_call_function_queu
void *info = csd->info;

csd_unlock(csd);
+ /*
+ * Ensures any LOAD of func() will be after
+ * the above UNLOCK, this guarantees that
+ * func() will observe any state prior
+ * to _async() returning -EBUSY.
+ */
+ smp_mb();
func(info);
} else if (type == CSD_TYPE_IRQ_WORK) {
irq_work_single(csd);
@@ -397,16 +404,18 @@ EXPORT_SYMBOL(smp_call_function_single);
* can thus be done from contexts with disabled interrupts.
*
* The caller passes his own pre-allocated data structure
- * (ie: embedded in an object) and is responsible for synchronizing it
- * such that the IPIs performed on the @csd are strictly serialized.
+ * and is responsible for it's life-time, it must not be re-used
+ * until csd->node.u_flags == 0.
*
* If the function is called with one csd which has not yet been
* processed by previous call to smp_call_function_single_async(), the
* function will return immediately with -EBUSY showing that the csd
* object is still in progress.
*
- * NOTE: Be careful, there is unfortunately no current debugging facility to
- * validate the correctness of this serialization.
+ * When -EBUSY is returned, any invocation of csd->func() is guaranteed to see
+ * the state prior to this call.
+ *
+ * Also, consider using irq_work_queue_remote() if at all possible.
*/
int smp_call_function_single_async(int cpu, call_single_data_t *csd)
{
@@ -414,13 +423,16 @@ int smp_call_function_single_async(int c

preempt_disable();

- if (csd->node.u_flags & CSD_FLAG_LOCK) {
+ /*
+ * We still need RELEASE like semantics, even when the cmpxchg() fails.
+ * Pairs with the smp_mb() in flush_smp_call_function_queue().
+ */
+ smp_mb__before_atomic();
+ if (cmpxchg_relaxed(&csd->node.u_flags, 0, CSD_FLAG_LOCK) != 0) {
err = -EBUSY;
goto out;
}
-
- csd->node.u_flags = CSD_FLAG_LOCK;
- smp_wmb();
+ /* ctrl-dep orders later stores */

err = generic_exec_single(cpu, csd);