Re: [RFC][PATCH] Improving directed yield scalability for PLE handler

[Avi Kivity <avi@redhat.com>]

On 09/11/2012 09:27 PM, Andrew Theurer wrote:
> 
> So, having both is probably not a good idea.  However, I feel like
> there's more work to be done.  With no over-commit (10 VMs), total
> throughput is 23427 +/- 2.76%.  A 2x over-commit will no doubt have some
> overhead, but a reduction to ~4500 is still terrible.  By contrast,
> 8-way VMs with 2x over-commit have a total throughput roughly 10% less
> than 8-way VMs with no overcommit (20 vs 10 8-way VMs on 80 cpu-thread
> host).  We still have what appears to be scalability problems, but now
> it's not so much in runqueue locks for yield_to(), but now
> get_pid_task():
> 
> perf on host:
> 
> 32.10% 320131 qemu-system-x86 [kernel.kallsyms] [k] get_pid_task
> 11.60% 115686 qemu-system-x86 [kernel.kallsyms] [k] _raw_spin_lock
> 10.28% 102522 qemu-system-x86 [kernel.kallsyms] [k] yield_to
>  9.17%  91507 qemu-system-x86 [kvm]             [k] kvm_vcpu_on_spin
>  7.74%  77257 qemu-system-x86 [kvm]             [k] kvm_vcpu_yield_to
>  3.56%  35476 qemu-system-x86 [kernel.kallsyms] [k] __srcu_read_lock
>  3.00%  29951 qemu-system-x86 [kvm]             [k] __vcpu_run
>  2.93%  29268 qemu-system-x86 [kvm_intel]       [k] vmx_vcpu_run
>  2.88%  28783 qemu-system-x86 [kvm]             [k] vcpu_enter_guest
>  2.59%  25827 qemu-system-x86 [kernel.kallsyms] [k] __schedule
>  1.40%  13976 qemu-system-x86 [kernel.kallsyms] [k] _raw_spin_lock_irq
>  1.28%  12823 qemu-system-x86 [kernel.kallsyms] [k] resched_task
>  1.14%  11376 qemu-system-x86 [kvm_intel]       [k] vmcs_writel
>  0.85%   8502 qemu-system-x86 [kernel.kallsyms] [k] pick_next_task_fair
>  0.53%   5315 qemu-system-x86 [kernel.kallsyms] [k] native_write_msr_safe
>  0.46%   4553 qemu-system-x86 [kernel.kallsyms] [k] native_load_tr_desc
> 
> get_pid_task() uses some rcu fucntions, wondering how scalable this
> is....  I tend to think of rcu as -not- having issues like this... is
> there a rcu stat/tracing tool which would help identify potential
> problems?

It's not, it's the atomics + cache line bouncing.  We're basically
guaranteed to bounce here.

Here we're finally paying for the ioctl() based interface.  A syscall
based interface would have a 1:1 correspondence between vcpus and tasks,
so these games would be unnecessary.

-- 
error compiling committee.c: too many arguments to function