From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Glauber Subject: Re: RCU stall with high number of KVM vcpus Date: Tue, 14 Nov 2017 08:52:49 +0100 Message-ID: <20171114075249.GB16731@hc> References: <20171113131000.GA10546@hc> <2832f775-3cbe-d984-fe4f-e018642b6f1d@arm.com> <20171113173552.GA13282@hc> <7dda7be2-f392-8056-d4d3-372bb867729a@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: kvm@vger.kernel.org, Paolo Bonzini , Radim =?utf-8?B?S3LEjW3DocWZ?= , Christoffer Dall , linux-arm-kernel@lists.infradead.org To: Marc Zyngier Return-path: Received: from mail-dm3nam03on0045.outbound.protection.outlook.com ([104.47.41.45]:42304 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751872AbdKNHxE (ORCPT ); Tue, 14 Nov 2017 02:53:04 -0500 Content-Disposition: inline In-Reply-To: <7dda7be2-f392-8056-d4d3-372bb867729a@arm.com> Sender: kvm-owner@vger.kernel.org List-ID: On Mon, Nov 13, 2017 at 06:11:19PM +0000, Marc Zyngier wrote: > On 13/11/17 17:35, Jan Glauber wrote: [...] > >>> numbers don't look good, see waittime-max: > >>> > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg > >>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >>> > >>> &(&kvm->mmu_lock)->rlock: 99346764 99406604 0.14 1321260806.59 710654434972.0 7148.97 154228320 225122857 0.13 917688890.60 3705916481.39 16.46 > >>> ------------------------ > >>> &(&kvm->mmu_lock)->rlock 99365598 [] kvm_handle_guest_abort+0x4c0/0x950 > >>> &(&kvm->mmu_lock)->rlock 25164 [] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 > >>> &(&kvm->mmu_lock)->rlock 14934 [] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 > >>> &(&kvm->mmu_lock)->rlock 908 [] __cond_resched_lock+0x68/0xb8 > >>> ------------------------ > >>> &(&kvm->mmu_lock)->rlock 3 [] stage2_flush_vm+0x60/0xd8 > >>> &(&kvm->mmu_lock)->rlock 99186296 [] kvm_handle_guest_abort+0x4c0/0x950 > >>> &(&kvm->mmu_lock)->rlock 179238 [] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8 > >>> &(&kvm->mmu_lock)->rlock 19181 [] kvm_mmu_notifier_invalidate_range_end+0x24/0x68 > >>> > >>> ............................................................................................................................................................................................................................. > >> [slots of stuff] > >> > >> Well, the mmu_lock is clearly contended. Is the box in a state where you > >> are swapping? There seem to be as many faults as contentions, which is a > >> bit surprising... > > > > I don't think it is swapping but need to double check. > > It is the number of aborts that is staggering. And each one of them > leads to the mmu_lock being contended. So something seems to be taking > its sweet time holding the damned lock. Can you elaborate on the aborts, I'm not familiar with KVM but from a first look I thought kvm_handle_guest_abort() is in the normal path when a vcpu is stopped. Is that wrong? --Jan