jitter test scalability problem - my theory

[Mike Galbraith <mgalbraith@suse.de>]

Greetings,

2.6.33.15-rt31 on a 64 core DL980, running a jitter test proggy that
fires up an executive (a model thereof actually) on 56 isolated cores,
which in turn fires up 3 or 4 workers in descending priority.  All tasks
are pinned. Box booted isolcpus=8-63.  Jitter test runs on these, 0-7
handle everything else, including interrupts.

Test proggy is looking to achieve a max jitter of +-30us, and that goal
is met.. until I increase the load to > 32 isolated cores.

Profiling, my kernel overhead is mostly (surprise) locks, but it seems
my scalability issue may be coming from...

static void 
inc_rt_prio_smp(struct rt_rq *rt_rq, int prio, int prev_prio)
{
...
                if (rq->online)
                        cpupri_set(&rq->rd->cpupri, rq->cpu, prio);
      
void cpupri_set(struct cpupri *cp, int cpu, int newpri)
{
... 
        if (likely(newpri != CPUPRI_INVALID)) {
                struct cpupri_vec *vec = &cp->pri_to_cpu[newpri];

                raw_spin_lock_irqsave(&vec->lock, flags);  <== here

So it seems push/pull logic _may_ be my scalability problem.  Even
though there's nothing that can be pushed/pulled, it's hammering a few
locks from many cores, so cores perturb each other enough despite
isolation, to fail once enough cores are active.

Does that look like a reasonable explanation for my jitter increase?

I'm going to hack up a test for my wild theory, but since I don't have a
lot of experience hunting jitter sources, looking for a few lousy usecs,
it couldn't hurt to ask whether I'm barking up the wrong tree or not :)


(in previous run,  _raw_spin_lock_irqsave() was > 60% of kernel
overhead, this is just the last run I did)

# dso: [kernel.kallsyms]
# Events: 214K cycles
#
# Overhead                               Symbol
# ........  ...................................
#
    29.53%  [k] _raw_spin_lock_irqsave
            |
            |--61.57%-- cpupri_set
            |          |
            |          |--81.55%-- dequeue_rt_stack
            |          |          dequeue_task_rt
            |          |          dequeue_task
            |          |          |
            |          |          |--100.00%-- deactivate_task
            |          |          |          __schedule
            |          |          |          schedule
            |          |          |          |
            |          |          |          |--69.94%-- run_ksoftirqd
            |          |          |          |          kthread
            |          |          |          |          kernel_thread_helper
            |          |          |          |
            |          |          |          |--15.23%-- sys_semtimedop
            |          |          |          |          sys_semop
            |          |          |          |          system_call_fastpath
            |          |          |          |          |
            |          |          |          |          |--4.11%-- 0x7f54c025ae37
            |          |          |          |          |          __semop