Re: [PATCH v2 2/3] block: make BlockBackend->disable_request_queuing atomic

[Paolo Bonzini <pbonzini@redhat.com>]

On 3/9/23 13:31, Stefan Hajnoczi wrote:
> On Thu, Mar 09, 2023 at 10:07:40AM +0100, Paolo Bonzini wrote:
>> On 3/7/23 22:04, Stefan Hajnoczi wrote:
>>> This field is accessed by multiple threads without a lock. Use explicit
>>> qatomic_read()/qatomic_set() calls. There is no need for acquire/release
>>> because blk_set_disable_request_queuing() doesn't provide any
>>> guarantees (it helps that it's used at BlockBackend creation time and
>>> not when there is I/O in flight).
>>
>> This in turn means itdoes not need to be atomic - atomics are only needed if
>> there are concurrent writes.  No big deal; I am now resurrecting the series
>> from the time I had noticed the queued_requests thread-safety problem, so
>> this will be simplified in 8.1.  For now your version is okay, thanks for
>> fixing it!
> 
> I was under the impression that variables accessed by multiple threads
> outside a lock or similar primitive need memory_order_relaxed both as
> documentation and to tell the compiler that they should indeed be atomic
> (but without ordering guarantees).

Atomic accesses are needed to avoid data races.  Data races are 
concurrent accesses, of which at least one is a non-atomic write.  (A is 
concurrent with B is you can't be sure that A happens before B or vice 
versa; this intuitively is the "lock or similar primitive" that you 
mentioned.  Happens-before changes from one execution to the other, but 
it is enough to somehow prove there _is_ an ordering; for example, given 
two accesses that are done while a mutex is taken, one will always 
happen before the other).

In this case all writes to disable_request_queuing happen not just 
outside I/O, but even *before* the first I/O.  No writes that are 
concurrent with reads => no need to use atomic for reads.

For example the stdin global variable is accessed from multiple threads 
and you would never use atomics to read the pointer.  Just don't write 
to it and there won't be data races.

> I think memory_order_relaxed also tells the compiler to do a bit more,
> like to generate just a single store to the variable for each occurrence
> in the code ("speculative" and "out-of-thin air" stores).

The correspondence is not necessarily 1:1, some optimizations are 
possible; for example this:

   qatomic_set(&x, 0);
   a = qatomic_read(&x);
   qatomic_set(&x, a + 1);

can be changed to

   a = 0;
   qatomic_set(&x, 1);

(because it is safe to assume that no other thread sees the state where 
x==0).  Or the first read here:

   a = qatomic_read(&x);
   a = qatomic_read(&x);

can be optimized out, unlike Linux's READ_ONCE().

I have no idea if compilers actually perform these optimizations, but if 
they do they are neither frequent (maybe in languages that inline a lot 
more, but not in QEMU) nor problematic.  Even though there's some 
freedom in removing/consolidating code, it's true as you said that 
speculation is a no-no!

> It's the documentation part that's most interesting in this case. Do we
> not want to identify variables that are accessed outside a lock and
> therefore require some thought?

I think it depends, see the stdin example before.  As the API is now, I 
agree that using qatomic_read() is the right thing to do.  In principle 
the flag could bounce back and forth many times. :/

With a better API, the balance may tilt on the side of not using 
atomics.  We'll see when I post the patch. :)

Paolo