From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D696225D7 for ; Wed, 17 Jan 2024 16:51:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705510266; cv=none; b=G0HcOeTqo0P383X+IbAowlytjfDNJBsFk7N4V/H4vU4++QrW6z+BK5AeAlALRqP1JSKRG4xwqPMqLnIAa5ZxRKtLBXLgB2bt70abY32lRNLYH5WR0t/LzP2KhdPCEu7IoThqnNmnrCxd95Cp8NuxB0DvYqu9B87kf9dD9iIUWEY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705510266; c=relaxed/simple; bh=faT2uRrQt3V7tKi/jCpPpD6mGOoqk18135qnLaMNQ6w=; h=Date:DKIM-Signature:X-Report-Abuse:From:To:Cc:Subject:Message-ID: References:MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:X-Migadu-Flow; b=BCpQc7jnhw52gpz0VMclEX3bIjnAfW/fN6+qwMUFnbyR08WoADN4AdNMJm5n7Mu0GkFWX/ZYQ1mbsoPpTLG5GkBF8cVJ6uhjfc3oyji2jWXU5TExk2Cw2Hi9bIEgSYbl8a1eqdLZGkye4f06ks9BjYe9dQ/oV9qsCKuBxiVCRvs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=BspoeQJt; arc=none smtp.client-ip=91.218.175.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="BspoeQJt" Date: Wed, 17 Jan 2024 17:50:28 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1705510261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zsMBTqAbxvDSq34Y8PBcrq1QpEjPS4gVtR8AVs1aPo0=; b=BspoeQJtQ3UPfpuJmKzVnJGZaqu6V2IgvIXmri1tYRZK2y0xcxPjnsCRf7Xs6Hj33dsr19 MNRPJ4vUXh4baAxJUXOshQkM6a9UfonXktAjkK426LQgS0DFmZbAQ1G6W9LQlRCO/cssRo KcxndU/YA7T217PRGN6/swdvxkMLfmA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: "sundongxu (A)" Cc: Marc Zyngier , yuzenghui@huawei.com, james.morse@arm.com, suzuki.poulose@arm.com, will@kernel.org, catalin.marinas@arm.com, wanghaibin.wang@huawei.com, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, jiangkunkun@huawei.com Subject: Re: [bug report] GICv4.1: VM performance degradation due to not trapping vCPU WFI Message-ID: References: <86v87t8ras.wl-maz@kernel.org> <89fe1503-6d62-90ca-5edb-e11c74846a00@huawei.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <89fe1503-6d62-90ca-5edb-e11c74846a00@huawei.com> X-Migadu-Flow: FLOW_OUT On Wed, Jan 17, 2024 at 10:20:32PM +0800, sundongxu (A) wrote: > On 2024/1/16 19:13, Marc Zyngier wrote: > > On Tue, 16 Jan 2024 03:26:08 +0000, "sundongxu (A)" wrote: > >> We found a problem about GICv4/4.1, for example: > >> We use QEMU to start a VM (4 vCPUs and 8G memory), VM disk was > >> configured with virtio, and the network is configured with vhost-net, > >> the CPU affinity of the vCPU and emulator is as follows, in VM xml: > >> > >> > >> > >> > >> > >> > >> > > Effectively, we apply the same principle to vSGIs as to vLPIs, and it > > was found that this heuristic was pretty beneficial to vLPIs. I'm a > > bit surprised that vSGIs are so different in their usage pattern. > > IMO, the point is hypervisor not trapping vCPU WFI, rather than > vSGI/vLPI usage pattern. Sure, that's what's affecting your use case, but the logic in the kernel came about because improving virtual interrupt injection has been found to be generally useful. > > > > Does it help if you move your "emulatorpin" to some other physical > > CPUs? > > Yes，it does, in kernel 5.10 or 6.5rc1. Won't your VM have a poor experience in this configuration regardless of WFx traps? The value of vCPU pinning is to *isolate* the vCPU threads from noise/overheads of the host and scheduler latencies. Seems to me that VMM overhead threads are being forced to take time away from the guest. Nevertheless, disabling WFI traps isn't going to work well for overcommitted scenarios. The thought of tacking on more hacks in KVM has be a bit uneasy, perhaps instead we can give userspace an interface to explicitly enable/disable WFx traps and let it pick a suitable policy. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B21B9C47DA2 for ; Wed, 17 Jan 2024 16:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YDT/X1GxSSyZEFTOP8poqvqvjsu1GYSaCA51BXV1eik=; b=jXWosR/hplNdue Vi+R/nmcy/Zs974NsjvRd439pLwvZMWXxake2gwzGrBc+Xy5XJoZswVz5OdGw7kM9T1eJ4QQChnSA u2JrHaEun2eYJ0HvHAq3k4hJw+jWuTVj9w0AhBlJ/Jj9cJS2nx/Nt+9vloBJa7VmTbaA+CQMrIGUv jNgrF5b2fGGzSKI2K5YQ00DcJE7ZD+E5BJWhd7HvOpNoTNs+L/xjKPiXTHkvDBIw37cw9bo5WtWhM rLcq3rd6WlIyU/aAtpQa1DdbxXSewQwAqxYy2ROeTTf16JovosrhzJxmK/FZzzrs2ZcQWkhVrJ7HY L18A5XmwJQ/NmMHWaA7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rQ98N-0008Yo-38; Wed, 17 Jan 2024 16:51:23 +0000 Received: from out-181.mta0.migadu.com ([2001:41d0:1004:224b::b5]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rQ98B-0008Sv-1J for linux-arm-kernel@lists.infradead.org; Wed, 17 Jan 2024 16:51:22 +0000 Date: Wed, 17 Jan 2024 17:50:28 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1705510261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zsMBTqAbxvDSq34Y8PBcrq1QpEjPS4gVtR8AVs1aPo0=; b=BspoeQJtQ3UPfpuJmKzVnJGZaqu6V2IgvIXmri1tYRZK2y0xcxPjnsCRf7Xs6Hj33dsr19 MNRPJ4vUXh4baAxJUXOshQkM6a9UfonXktAjkK426LQgS0DFmZbAQ1G6W9LQlRCO/cssRo KcxndU/YA7T217PRGN6/swdvxkMLfmA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: "sundongxu (A)" Cc: Marc Zyngier , yuzenghui@huawei.com, james.morse@arm.com, suzuki.poulose@arm.com, will@kernel.org, catalin.marinas@arm.com, wanghaibin.wang@huawei.com, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, jiangkunkun@huawei.com Subject: Re: [bug report] GICv4.1: VM performance degradation due to not trapping vCPU WFI Message-ID: References: <86v87t8ras.wl-maz@kernel.org> <89fe1503-6d62-90ca-5edb-e11c74846a00@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <89fe1503-6d62-90ca-5edb-e11c74846a00@huawei.com> X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240117_085118_906216_1FD87BAB X-CRM114-Status: GOOD ( 15.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBKYW4gMTcsIDIwMjQgYXQgMTA6MjA6MzJQTSArMDgwMCwgc3VuZG9uZ3h1IChBKSB3 cm90ZToKPiBPbiAyMDI0LzEvMTYgMTk6MTMsIE1hcmMgWnluZ2llciB3cm90ZToKPiA+IE9uIFR1 ZSwgMTYgSmFuIDIwMjQgMDM6MjY6MDggKzAwMDAsICJzdW5kb25neHUgKEEpIiA8c3VuZG9uZ3h1 M0BodWF3ZWkuY29tPiB3cm90ZToKPiA+PiBXZSBmb3VuZCBhIHByb2JsZW0gYWJvdXQgR0lDdjQv NC4xLCBmb3IgZXhhbXBsZToKPiA+PiBXZSB1c2UgUUVNVSB0byBzdGFydCBhIFZNICg0IHZDUFVz IGFuZCA4RyBtZW1vcnkpLCBWTSBkaXNrIHdhcwo+ID4+IGNvbmZpZ3VyZWQgd2l0aCB2aXJ0aW8s IGFuZCB0aGUgbmV0d29yayBpcyBjb25maWd1cmVkIHdpdGggdmhvc3QtbmV0LAo+ID4+IHRoZSBD UFUgYWZmaW5pdHkgb2YgdGhlIHZDUFUgYW5kIGVtdWxhdG9yIGlzIGFzIGZvbGxvd3MsIGluIFZN IHhtbDoKCjxzbmlwPgoKPiA+PiAgIDxjcHV0dW5lPgo+ID4+ICAgICA8dmNwdXBpbiB2Y3B1PScw JyBjcHVzZXQ9JzQnLz4KPiA+PiAgICAgPHZjcHVwaW4gdmNwdT0nMScgY3B1c2V0PSc1Jy8+Cj4g Pj4gICAgIDx2Y3B1cGluIHZjcHU9JzInIGNwdXNldD0nNicvPgo+ID4+ICAgICA8dmNwdXBpbiB2 Y3B1PSczJyBjcHVzZXQ9JzcnLz4KPiA+PiAgICAgPGVtdWxhdG9ycGluIGNwdXNldD0nNCw1LDYs NycvPgo+ID4+ICAgPC9jcHV0dW5lPgoKPC9zbmlwPgoKPiA+IEVmZmVjdGl2ZWx5LCB3ZSBhcHBs eSB0aGUgc2FtZSBwcmluY2lwbGUgdG8gdlNHSXMgYXMgdG8gdkxQSXMsIGFuZCBpdAo+ID4gd2Fz IGZvdW5kIHRoYXQgdGhpcyBoZXVyaXN0aWMgd2FzIHByZXR0eSBiZW5lZmljaWFsIHRvIHZMUElz LiBJJ20gYQo+ID4gYml0IHN1cnByaXNlZCB0aGF0IHZTR0lzIGFyZSBzbyBkaWZmZXJlbnQgaW4g dGhlaXIgdXNhZ2UgcGF0dGVybi4KPiAKPiBJTU8sIHRoZSBwb2ludCBpcyBoeXBlcnZpc29yIG5v dCB0cmFwcGluZyB2Q1BVIFdGSSwgcmF0aGVyIHRoYW4KPiB2U0dJL3ZMUEkgdXNhZ2UgcGF0dGVy bi4KClN1cmUsIHRoYXQncyB3aGF0J3MgYWZmZWN0aW5nIHlvdXIgdXNlIGNhc2UsIGJ1dCB0aGUg bG9naWMgaW4gdGhlIGtlcm5lbApjYW1lIGFib3V0IGJlY2F1c2UgaW1wcm92aW5nIHZpcnR1YWwg aW50ZXJydXB0IGluamVjdGlvbiBoYXMgYmVlbiBmb3VuZAp0byBiZSBnZW5lcmFsbHkgdXNlZnVs LgoKPiA+IAo+ID4gRG9lcyBpdCBoZWxwIGlmIHlvdSBtb3ZlIHlvdXIgImVtdWxhdG9ycGluIiB0 byBzb21lIG90aGVyIHBoeXNpY2FsCj4gPiBDUFVzPwo+IAo+IFllc++8jGl0IGRvZXMsIGluIGtl cm5lbCA1LjEwIG9yIDYuNXJjMS4KCldvbid0IHlvdXIgVk0gaGF2ZSBhIHBvb3IgZXhwZXJpZW5j ZSBpbiB0aGlzIGNvbmZpZ3VyYXRpb24gcmVnYXJkbGVzcyBvZiBXRngKdHJhcHM/IFRoZSB2YWx1 ZSBvZiB2Q1BVIHBpbm5pbmcgaXMgdG8gKmlzb2xhdGUqIHRoZSB2Q1BVIHRocmVhZHMgZnJvbQpu b2lzZS9vdmVyaGVhZHMgb2YgdGhlIGhvc3QgYW5kIHNjaGVkdWxlciBsYXRlbmNpZXMuIFNlZW1z IHRvIG1lIHRoYXQKVk1NIG92ZXJoZWFkIHRocmVhZHMgYXJlIGJlaW5nIGZvcmNlZCB0byB0YWtl IHRpbWUgYXdheSBmcm9tIHRoZSBndWVzdC4KCk5ldmVydGhlbGVzcywgZGlzYWJsaW5nIFdGSSB0 cmFwcyBpc24ndCBnb2luZyB0byB3b3JrIHdlbGwgZm9yCm92ZXJjb21taXR0ZWQgc2NlbmFyaW9z LiBUaGUgdGhvdWdodCBvZiB0YWNraW5nIG9uIG1vcmUgaGFja3MgaW4gS1ZNIGhhcyBiZQphIGJp dCB1bmVhc3ksIHBlcmhhcHMgaW5zdGVhZCB3ZSBjYW4gZ2l2ZSB1c2Vyc3BhY2UgYW4gaW50ZXJm YWNlIHRvIGV4cGxpY2l0bHkKZW5hYmxlL2Rpc2FibGUgV0Z4IHRyYXBzIGFuZCBsZXQgaXQgcGlj ayBhIHN1aXRhYmxlIHBvbGljeS4KCi0tIApUaGFua3MsCk9saXZlcgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5n IGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5p bmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=