Re: [Xenomai-core] [RFC][PATCH 4/4] Recursive FIFO ticket xnlock

[Jan Kiszka <jan.kiszka@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 2853 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > Gilles Chanteperdrix wrote:
>  > > Jan Kiszka wrote:
>  > >  > The root of all this: When Nick Piggin posted his first suggestion for
>  > >  > ticket spinlocks on LKML, I immediately liked the idea. For details
>  > >  > check LWN [1], in a nutshell: This algorithm enforces strict FIFO order
>  > >  > for the admission to contended spinlocks, thus it improves the
>  > >  > determinism on SMP systems with more than 2 CPUs.
>  > >  > 
>  > >  > Meanwhile, ticket spinlocks are mainline (2.6.25). But that version has
>  > >  > to drawbacks for us: it doesn't support nesting like xnlock does, and it
>  > >  > is x86-only so far.
>  > >  > 
>  > >  > So I designed a version for Xenomai which is both nestable and
>  > >  > arch-independent. It is certainly not as optimal as mainline's version,
>  > >  > but our code path stresses the locking code differently anyway.
>  > >  > 
>  > >  > This thing here /seems/ to work, but I'm lacking CPUs at home to test.
>  > >  > You can't truly stress ticket locks with only a single dual-core :-/.
>  > >  > QEMU runs into a live-lock with -smp 2, this patch applied and two
>  > >  > moderate latency loops, but that might be an artifact of its
>  > >  > single-threaded VCPU scheduling. And kvm currently locks up under SMP
>  > >  > even without any change, but kvm and SMP is a story of its own. There is
>  > >  > hope: 16-way is waiting at work... :)
>  > > 
>  > > Xenomai uses a Big Kernel Lock, an approach known to not scale very
>  > > well. So, if we want to scale correctly on machines with many cpus, we
>  > > should change our locking strategy first.
>  > 
>  > Per-cpu IPC objects, per-cpu nklock - I'm all with you! But that's stuff
>  > for a massive restructuring we could schedule for Xenomai 3.
> 
> This does not look that easy. What I meant is rather that Xenomai is
> probably run mostly on machines with not so many cpus.

You do not have to be an oracle to predict that the number of cores will
grow quickly in the foreseeable future on almost any arch, and that
users will like to _use_  more of them for RT. Running Xenomai on big
boxes is already no big deal today, you just have to keep the RT (and
borderline) load confined to a few of them. (Statically!) spreading the
load over more cores is then just the next logical step.

For sure, what I'm proposing, strict user-manageable cpu-locality of
objects and thus of lock mechanisms, is not simple to achieve - but it
would scale like hell and it would keep the costs mostly at UP level, on
huge SMP boxes, NUMA included. But even in that case, you will need
spinlocks from time to time, and you will want them to remain as
predictable as possible. So this patch is also a bit of an investment in
the future. :)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]