Re: [PATCH -tip] introduce sys_membarrier(): process-wide memory barrier (v9)

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

* Nick Piggin (npiggin@suse.de) wrote:
> On Tue, Mar 16, 2010 at 08:36:35AM +0100, Ingo Molnar wrote:
> > 
> > * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> > 
> > > * Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> > > > * Linus Torvalds (torvalds@linux-foundation.org) wrote:
> > > > > >  - SA_RUNNING: a way to signal only running threads - as a way for user-space 
> > > > > >    based concurrency control mechanisms to deschedule running threads (or, like
> > > > > >    in your case, to implement barrier / garbage collection schemes).
> > > > > 
> > > > > Hmm. This sounds less fundamentally broken, but at the same time also 
> > > > > _way_ more invasive in the signal handling layer. It's already one of our 
> > > > > more "exciting" layers out there.
> > > > > 
> > > > 
> > > > Hrm, thinking about it a bit further, the only way I see we could provide a
> > > > usable SA_RUNNING flag would be to add hooks to the scheduler. These hooks would
> > > > somehow have to call user-space code (!) when scheduling in/out a thread. Yes,
> > > > this sounds utterly broken (since these hooks would have to be preemptable).
> > > > 
> > > > The idea is this: if we look, for instance, at the kernel preemptable RCU
> > > > implementations, they consist of two parts: one is iteration on all CPUs to
> > > > consider all active CPUs, and the other is a modification of the scheduler to
> > > > note all preempted tasks that were in a preemptable RCU C.S..
> > > > 
> > > > Just for the memory barrier we consider for sys_membarrier(), I had to ensure
> > > > that the scheduler issues memory barriers to order accesses to user-space memory
> > > > and mm_cpumask modifications. In reality, what we are doing is to ensure that
> > > > the operation required on the running thread is done by the scheduler too when
> > > > scheduling in/out the task.
> > > > 
> > > > As soon as we have signal handlers which perform more than a simple memory 
> > > > barrier (e.g. something that has side-effects outside of the processor), I 
> > > > doubt it would ever make sense to only run the handler on running threads 
> > > > unless we have hooks in the scheduler too.
> > > 
> > > Unless this question is answered, Ingo's SA_RUNNING signal proposal, as 
> > > appealing as it may look at a first glance, falls into the "fundamentally 
> > > broken" category. [...]
> > 
> > How is it different from your syscall? I.e. which lines of code make the 
> > difference? We could certainly apply the (trivial) barrier change to 
> > context_switch().
> 
> I think it is just easy for userspace to misuse or think it does
> something that it doesn't (because of races).
> 

Yep, this is exactly my point.

> If a context switch includes a barrier, then it is easy to know that
> either the task of interest will execute the barrier, or it will have
> context switched.
> 
> What more complex operation could be done in the signal handler that
> isn't broken by races? Programs that use realtime scheduling policies,
> and maybe some statistical or heuristic operations... Any cool use that
> would make anybody other than librcu bother using it?
> 

Yes, there seems to be no point in providing a nice flexible interface through
signals if the only race-less use we can find is to issue memory barriers
(which would be race-less because we add the proper barriers to the scheduler mm
switch code). And even if we find a userland use for such a signal, I tend to
think that the inherent risk of misuse and races would overweight its benefit.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency Consultant
EfficiOS Inc.
http://www.efficios.com