Re: PREEMPT_RT and I-PIPE: the numbers, part 4

[Ingo Molnar <mingo@elte.hu>]

* Kristian Benoit <kbenoit@opersys.com> wrote:

[...]
> "plain" run:
> 
> Measurements   |   Vanilla   |  preempt_rt    |   ipipe
> ---------------+-------------+----------------+-------------
> fork           |      97us   |   91us (-6%)   |  101us (+4%)
> mmap           |     776us   |  629us (-19%)  |  794us (+2%)

some of you have wondered how it's possible that the PREEMPT_RT kernel 
is _faster_ than the vanilla kernel in these two metrics.

I've done some more profiling, and one reason is kmap_atomic(). As i 
pointed out in an earlier mail, in your tests you not only had HIGHMEM64 
enabled, but also HIGHPTE, which is a heavy kmap_atomic() user. [and 
which is an option meant for systems with 8GB or more RAM, not the 
typical embedded target.]

kmap_atomic() is a pretty preemption-unfriendly per-CPU construct, which 
under PREEMPT_RT had to be changed and was mapped into kmap(). The 
performance advantage comes from the caching built into kmap() and not 
having to do per-page invlpg calls. (which can be pretty slow, 
expecially on highmem64) The 'mapping kmap_atomic into kmap' technique 
is perfectly fine under PREEMPT_RT because all kernel code is 
preemptible, but it's not really possible in the vanilla kernel due to 
the fundamental non-preemptability of interrupts, the preempt-off-ness 
of the mmu_gather mechanism, the atomicity of the ->page_table_lock 
spinlock, etc.

so this is a case of 'fully preemptible beats non-preemptible due to 
flexibility', but it should be more of an exception than the rule, 
because generally the fully preemptible kernel tries to be 1:1 identical 
to the vanilla kernel. But it's an interesting phenomenon from a 
conceptual angle nevertheless.

	Ingo