Re: [PATCH 0/5] Generic smp_call_function(), improvements, and smp_call_function_single()

[Jens Axboe <jens.axboe@oracle.com>]

On Thu, Mar 27 2008, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > which is pretty much identical to io-cpu-affinity, except it uses 
> > kernel threads for completion.
> > 
> > The reason why I dropped the kthread approach is that it was slower. 
> > Time from signal to run was about 33% faster with IPI than with 
> > wake_up_process(). Doing benchmark runs, and the IPI approach won 
> > hands down in cache misses as well.
> 
> with irq threads we'll have all irq context run in kthread context 
> again. Could you show me how you measured the performance of the kthread 
> approach versus the raw-IPI approach?

There were 3 different indicators that the irq thread approach was
slower:

- Time from signal to actual run of the trigger was ~2usec vs ~3usec for
  IPI vs kthread. That was a microbenchmark.
- Cache misses were higher with the kthread approach.
- Actual performance in non-micro benchmarks was lower with the kthread
  approach.

I'll defer to Alan for the actual numbers, most of this was done in
private mails back and forth doing performance analysis. The initial
testing was done with the IPI hack, then we moved to the kthread
approach. Later the two were pitted against each other and the kthread
part was definitely slower. It ended up using more system time than the
IPI approach. So the kthread approach was than abandoned and all testing
has been on the smp_call_function_single() branch since then.

I very much wanted the kthread approach to work, since it's easier to
work with. It's not for lack of will or trying... I'll be happy to
supply you otherwise identical patches for this, the only difference
being kthread of IPI completions if you want to play with this.

> we can do a million kthread context switches per CPU per second, so 
> kthread context-switch cost cannot be a true performance limit, unless 
> you micro-benchmarked this.

At which point you wont be doing much else, so a cs microbenchmark is
not really that interesting.

-- 
Jens Axboe