Re: Preventing IPI sending races in arch code

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Nov 25, 2013 at 04:22:18PM +0530, Vineet Gupta wrote:
> Hi,
> 
> I've been looking into cleaning up bitrot in ARC SMP support. Unlike some other
> arches/platforms, we don't have per-msg-type IRQ, so the actual msg (say cross
> function call) corresponding to IPI needs to be encoded in a per-cpu word (1 bit
> per msg type) before kicking the IPI.
> 
> The current code (indicative below) is completely bonkers as it calls set_bit w/o
> any protection whatsoever, clearly racy in case of multiple senders, where
> receiver could end up NOT seeing one of the writes.
> 
> ipi_send_msg(cpu, msg-type)
> {
>    struct ipi_data *ipi_data = &per_cpu(ipi_data, cpu);
> 
>    local_irq_save();
>    set_bit(msg-type, &ipi_data->bits)
>    plat_smp_ops.ipi_send(cpumask)
>    local_irq_restore();
> }
> 
> Adding a spinlock here would serialize the sending part, but I still see issue in
> receiver. Upon receipt of First IPI, the msg holding word will be atomically
> exchanged with 0, so 2nd IPI will not see any msg in the word. Augmenting with an
> atomic counter would only help detect the issue - but I don't see how it will help
> elide the issue.
> 
> Does that mean w/o proper hardware assist (i.e. IRQ providing the msg id
> indication), the race, however small, remains ?

You can use cmpxchg to set the bit, and in case the previous value
wasn't 0 not send a second IPI.