Re: [RFC PATCH v6 1/5] Thread-local ABI system call: cache CPU number of running thread

[Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>]

----- On Apr 7, 2016, at 4:22 PM, Andi Kleen andi-Vw/NltI1exuRpAAqCnN02g@public.gmane.org wrote:

>> One basic use of cpu id cache is to speed up the sched_getcpu(3)
>> implementation in glibc. This is why I'm proposing it as a stand-alone
> 
> I don't think rseq is needed for faster getcpu.

I agree that rseq is not needed for faster getcpu. This is why I was proposing
to make "cpu_id" feature configurable separately from the rseq feature.
E.g. a kernel configuration that don't want to take the hit of rseq handling
in signal delivery and preemption could just enable the cpu_id feature, and
thus only need to add work in the migration code path, and when returning to
userspace. Also, if a thread only registers the cpu_id feature, the kernel
can skip the rseq code quickly in signal delivery and preemption too.

> 
> User space has to be able handle stale return values anyways, as it
> has no way to lock itself to a cpu while it is using the return value.
> So it can be only a hint.
> 
> The original version of getcpu just had a jiffies based cache. The CPU
> value was valid up to a jiffie (the next time jiffie changes), and then it
> gets looked up again.
> 
> Processes are unlikely to switch CPUs more often than a jiffie, so it's
> good enough as a hint.

One example use-case where this would hurt: we use the CPU id heavily when
tracing to a ring buffer in user-space. Having one event written into the
wrong buffer once in a while is not a big deal, but tracing a whole burst
of events within a jiffy (e.g. 4ms at 250Hz) to the wrong cpu buffer
whenever the thread migrates is really an unwanted side-effect latency-wise.

> 
> This doesn't need any new kernel interfaces at all because jiffies is already
> exported to the vdso.

My understanding is that although your assumptions about availability of
those features in vdso are true for x86 32/64, but do not currently apply
to ARM32.

ARM32 is my main target architecture for the CPU id cache work. x86 32/64
simply also happen to benefit from that work too (see my benchmark numbers
in changelog of patch 1/5).

> It just needs a new entry point into the vdso that handles the jiffie
> check.

This would likely require to extend the ARM vdso page to expose the jiffies
counter to user-space, and update user-space libraries to use this counter
in sched_getcpu. But it would still be slower than the cpu_id cache I propose,
due to the required function call to sched_getcpu, unless you want to open-code
the jiffies check within all applications as an ABI. It would also be bad for
fast bursts of cpu id use (e.g. per-cpu ring buffers).

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com