Re: KVM lock contention on 48 core AMD machine

[Avi Kivity <avi@redhat.com>]

On 03/21/2011 06:54 PM, Eric Dumazet wrote:
> Le lundi 21 mars 2011 à 22:01 +0545, Ben Nagy a écrit :
> >  >>  On Mon, Mar 21, 2011 at 7:38 PM, Avi Kivity<avi@redhat.com>   wrote:
> >  >>  >   In the future, please post the binary perf.dat.
> >  >>
> >  >>  Hi Avi,
> >  >>
> >  >>  How do I do that?
> >  >
> >  >  'make nconfig' and go to the kernel hacking section.
> >
> >  Imprecise question sorry, I meant how do I get the perf.dat not how do
> >  I disable the debugging.
> >
> >  On the non-debug kernel: Linux eax 2.6.38-7-server #36-Ubuntu SMP Fri
> >  Mar 18 23:36:13 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> >
> >             150512.00 67.8% __ticket_spin_lock
> >  [kernel.kallsyms]
> >              11126.00  5.0% memcmp_pages
> >  [kernel.kallsyms]
> >               9563.00  4.3% native_safe_halt
> >  [kernel.kallsyms]
> >               8965.00  4.0% svm_vcpu_run
> >  /lib/modules/2.6.38-7-server/kernel/arch/x86/kvm/kvm-amd.ko
> >               6489.00  2.9% tg_load_down
> >  [kernel.kallsyms]
> >               4676.00  2.1% kvm_get_cs_db_l_bits
> >  /lib/modules/2.6.38-7-server/kernel/arch/x86/kvm/kvm.ko
> >               1931.00  0.9% load_balance_fair
> >  [kernel.kallsyms]
> >               1917.00  0.9% ktime_get
> >  [kernel.kallsyms]
> >               1624.00  0.7% walk_tg_tree.clone.129
> >  [kernel.kallsyms]
> >               1542.00  0.7% find_busiest_group
> >  [kernel.kallsyms]
> >               1326.00  0.6% find_next_bit
> >  [kernel.kallsyms]
> >                673.00  0.3% lock_hrtimer_base.clone.25
> >  [kernel.kallsyms]
> >                624.00  0.3% copy_user_generic_string    [kernel.kallsyms]
> >
> >  top now says:
> >
> >  top - 00:11:35 up 22 min,  4 users,  load average: 0.11, 6.15, 7.78
> >  Tasks: 491 total,   3 running, 488 sleeping,   0 stopped,   0 zombie
> >  Cpu(s):  0.9%us, 15.4%sy,  0.0%ni, 83.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> >  Mem:  99068660k total, 70831760k used, 28236900k free,    10036k buffers
> >  Swap:  2438140k total,  2173652k used,   264488k free,  3396144k cached
> >
> >  With average 'resting cpu' per idle guest 8%, 96 guests running.
> >
> >  Is this as good as I am going to get? It seems like I can't really
> >  debug that lock contention without blowing stuff up because of the
> >  load of the lock debugging...
> >
> >  Don't know if I mentioned this before, but those guests are each
> >  pinned to a cpu (cpu guestnum%48) with taskset.
> >
> >  Cheers,
>
> It seems you hit idr_lock contention (used in kernel/posix-timers.c)
>

Any ideas on how to fix it?  We could pre-allocate IDs and batch them in 
per-cpu caches, but it seems like a lot of work.

-- 
error compiling committee.c: too many arguments to function