Re: [RFC PATCH] introduce sys_membarrier(): process-wide memory barrier

Steven Rostedt
2010-01-07T21:35:45Z

On Thu, 2010-01-07 at 12:58 -0800, Paul E. McKenney wrote:

> I believe that I am worried about a different scenario. I do not believe
> that the scenario you lay out above can actually happen. The pair of
> schedules on CPU 2 have to act as a full memory barrier, otherwise,
> it would not be safe to resume a task on some other CPU.

I'm not so sure about that. The update of ->curr happens inside a
spinlock, which is a rmb() ... wmb() pair. Must be, because a spin_lock
must be an rmb otherwise the loads could move outside the lock, and the
spin_unlock must be a wmb() otherwise what was written could move
outside the lock.

> If the pair
> of schedules act as a full memory barrier, then the code in
> synchronize_rcu() that looks at the RCU read-side state would see that
> CPU 2 is in an RCU read-side critical section.
>
> The scenario that I am (perhaps wrongly) concerned about is enabled by
> the fact that URCU's rcu_read_lock() has a load, some checks, and a store.
> It has compiler constraints, but no hardware memory barriers. This
> means that CPUs (even x86) can execute an rcu_dereference() before the
> rcu_read_lock()'s store has executed.
>
> Hacking your example above, keeping mind that x86 can reorder subsequent
> loads to precede prior stores:
>
>
> CPU 1 CPU 2
> ----------- -------------
>
> <user space> <kernel space, switching to task>
>
> ->curr updated
>
> <long code path, maybe mb?>
>
> <user space>
>
> rcu_read_lock(); [load only]
>
> obj = list->next
>
> list_del(obj)
>
> sys_membarrier();
> < kernel space >

Well, if we just grab the task_rq(task)->lock here, then we should be
OK? We would guarantee that curr is either the task we want or not.

>
> if (task_rq(task)->curr != task)
> < but load to obj reordered before store to ->curr >
>
> < user space >
>
> < misses that CPU 2 is in rcu section >
>
> [CPU 2's ->curr update now visible]
>
> [CPU 2's rcu_read_lock() store now visible]
>
> free(obj);
>
> use_object(obj); <=== crash!
>
>
>
> If the "long code path" happens to include a full memory barrier, or if it
> happens to be long enough to overflow CPU 2's store buffer, then the
> above scenario cannot happen. Until such time as someone applies some
> unforeseen optimization to the context-switch path.
>
> And, yes, the context-switch path has to have a full memory barrier
> somewhere, but that somewhere could just as easily come before the
> update of ->curr.

Hmm, since ->curr is updated before sched_mm() I'm thinking it would
have to be after the update of curr.

>
> The same scenario applies when using ->cpu_vm_mask instead of ->curr.
>
> Now, I could easily believe that the current context-switch code has
> sufficient atomic operations, memory barriers, and instructions to
> prevent this scenario from occurring, but it is feeling a lot like an
> accident waiting to happen. Hence my strident complaints. ;-)

I'm totally with you on this. I really want a good understanding of what
can go wrong, and show that we have the necessary infrastructure to
prevent it.

-- Steve