Re: XDP socket rings, and LKMM litmus tests

Paul E. McKenney
2021-03-04T00:14:42Z

On Wed, Mar 03, 2021 at 03:22:46PM -0500, Alan Stern wrote:
> On Wed, Mar 03, 2021 at 09:40:22AM -0800, Paul E. McKenney wrote:
> > On Wed, Mar 03, 2021 at 12:12:21PM -0500, Alan Stern wrote:
>
> > > Local variables absolutely should be treated just like CPU registers, if
> > > possible. In fact, the compiler has the option of keeping local
> > > variables stored in registers.
> > >
> > > (Of course, things may get complicated if anyone writes a litmus test
> > > that uses a pointer to a local variable, Especially if the pointer
> > > could hold the address of a local variable in one execution and a
> > > shared variable in another! Or if the pointer is itself a shared
> > > variable and is dereferenced in another thread!)
> >
> > Good point! I did miss this complication. ;-)
>
> I suspect it wouldn't be so bad if herd7 disallowed taking addresses of
> local variables.
>
> > As you say, when its address is taken, the "local" variable needs to be
> > treated as is it were shared. There are exceptions where the pointed-to
> > local is still used only by its process. Are any of these exceptions
> > problematic?
>
> Easiest just to rule out the whole can of worms.

Good point, given that a global can be used instead of a local for
any case where an address must be taken.

> > > But even if local variables are treated as non-shared storage locations,
> > > we should still handle this correctly. Part of the problem seems to lie
> > > in the definition of the to-r dependency relation; the relevant portion
> > > is:
> > >
> > > (dep ; [Marked] ; rfi)
> > >
> > > Here dep is the control dependency from the READ_ONCE to the
> > > local-variable store, and the rfi refers to the following load of the
> > > local variable. The problem is that the store to the local variable
> > > doesn't go in the Marked class, because it is notated as a plain C
> > > assignment. (And likewise for the following load.)
> > >
> > > Should we change the model to make loads from and stores to local
> > > variables always count as Marked?
> >
> > As long as the initial (possibly unmarked) load would be properly
> > complained about.
>
> Sorry, I don't understand what you mean.

I was thinking in terms of something like this in one of the processes:

p = gp; // Unmarked!
r1 = p;
q = r1; // Implicitly marked now?
if (q)
WRITE_ONCE(x, 1); // ctrl dep from gp???

> > And I cannot immediately think of a situation where
> > this approach would break that would not result in a data race being
> > flagged. Or is this yet another failure of my imagination?
>
> By definition, an access to a local variable cannot participate in a
> data race because all such accesses are confined to a single thread.

True, but its value might have come from a load from a shared variable.

> However, there are other aspects to consider, in particular, the
> ordering relations on local-variable accesses. But if, as Luc says,
> local variables are treated just like registers then perhaps the issue
> doesn't arise.

Here is hoping!

> > > What should have happened if the local variable were instead a shared
> > > variable which the other thread didn't access at all? It seems like a
> > > weak point of the memory model that it treats these two things
> > > differently.
> >
> > But is this really any different than the situation where a global
> > variable is only accessed by a single thread?
>
> Indeed; it is the _same_ situation. Which leads to some interesting
> questions, such as: What does READ_ONCE(r) mean when r is a local
> variable? Should it be allowed at all? In what way is it different
> from a plain read of r?
>
> One difference is that the LKMM doesn't allow dependencies to originate
> from a plain load. Of course, when you're dealing with a local
> variable, what matters is not the load from that variable but rather the
> earlier loads which determined the value that had been stored there.
> Which brings us back to the case of the
>
> dep ; rfi
>
> dependency relation, where the accesses in the middle are plain and
> non-racy. Should the LKMM be changed to allow this?

It would be nice, give or take the potential side effects. ;-)
As in it would be nice, but might not be worthwhile.

> There are other differences to consider. For example:
>
> r = READ_ONCE(x);
> smp_wmb();
> WRITE_ONCE(y, 1);
>
> If the write to r were treated as a marked store, the smp_wmb would
> order it (and consequently the READ_ONCE) before the WRITE_ONCE.
> However we don't want to do this when r is a local variable. Indeed, a
> plain store wouldn't be ordered this way because the compiler might
> optimize the store away entirely, leaving the smp_wmb nothing to act on.

Agreed, having smp_wmb() order things due to a write to a local variable
would not be what we want.

> So overall the situation is rather puzzling. Treating local variables
> as registers is probably the best answer.

That is sounding quite appealing at the moment.

Thanx, Paul