From mboxrd@z Thu Jan  1 00:00:00 1970
From: Joerg Roedel <joro@8bytes.org>
Subject: Re: KVM lock contention on 48 core AMD machine
Date: Fri, 18 Mar 2011 13:30:31 +0100
Message-ID: <20110318123031.GB6066@8bytes.org>
References: <AANLkTikZZ4MXZjx_+6u-vKVSgxULQo2BkQKRO6K_vzFy@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: kvm@vger.kernel.org
To: Ben Nagy <ben@iagu.net>
Return-path: <kvm-owner@vger.kernel.org>
Received: from 8bytes.org ([88.198.83.132]:52830 "EHLO 8bytes.org"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1752800Ab1CRMjf (ORCPT <rfc822;kvm@vger.kernel.org>);
	Fri, 18 Mar 2011 08:39:35 -0400
Content-Disposition: inline
In-Reply-To: <AANLkTikZZ4MXZjx_+6u-vKVSgxULQo2BkQKRO6K_vzFy@mail.gmail.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

Hi Ben,

On Fri, Mar 18, 2011 at 07:02:40PM +0700, Ben Nagy wrote:
> Here's some output from perf top while the system is locky:
>            263832.00 46.3% delay_tsc
> [kernel.kallsyms]
>            231491.00 40.7% __ticket_spin_trylock
> [kernel.kallsyms]
>             14609.00  2.6% native_read_tsc
> [kernel.kallsyms]
>              9414.00  1.7% do_raw_spin_lock
> [kernel.kallsyms]
>              8041.00  1.4% local_clock
> [kernel.kallsyms]
>              6081.00  1.1% native_safe_halt
> [kernel.kallsyms]
>              3901.00  0.7% __lock_acquire.clone.18
> [kernel.kallsyms]
>              3665.00  0.6% do_raw_spin_unlock
> [kernel.kallsyms]
>              3042.00  0.5% __delay
> [kernel.kallsyms]
>              2484.00  0.4% lock_contended
> [kernel.kallsyms]
>              2484.00  0.4% sched_clock_cpu
> [kernel.kallsyms]
>              1906.00  0.3% sched_clock_local
> [kernel.kallsyms]
>              1419.00  0.2% lock_acquire
> [kernel.kallsyms]
>              1332.00  0.2% lock_release
> [kernel.kallsyms]
>               987.00  0.2% tg_load_down
> [kernel.kallsyms]
>               895.00  0.2% _raw_spin_lock_irqsave
> [kernel.kallsyms]
>               686.00  0.1% find_busiest_group
> [kernel.kallsyms]

Can you try to run

# perf record -a -g

for a while when your VMs are up and unresponsive? This will monitor the
whole system and collects stack-traces. When you have done so
please run

# perf report > locks.txt

and upload the locks.txt file somewhere. The result might give us some
glue where the high lock-contention comes from.

Regards,

	Joerg