From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B8E5C4706C for ; Tue, 16 Jan 2024 11:14:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XWoyBnzmJJDySXzb01vchn0zkqQ8M9E1fGNXTxqeZBs=; b=bYhBsxLM+wgPhz TNlv+fvhpIdf7lj6AFeg3VFVqnNvl5eHqhvHOlEHNlCHsHATKy/PHhHd6SzCG6mLbceMFRvBZaHZI mxucu1ACQs/vY9F6dxrWHKmwqH6JM1NLYS2w8EtqZ+kcs25GjCM4ASMutFa9mEg2Wv3NB4wBpUYhl DmYXiKPbJVfc2oJLVEOIzEIop8RZJq7pqBD+9qhEU+8Dwzok+mrAtvDx0XTvlu+GWQYYbx/Hu8jh3 tKOfWFL7lnAjJcPBzwvK2dd2A8EzZDB4lWjuByqVaMPRr5RwwXtWNTbeAOCyLq5+lTIA8hUWSeWQj IfYsOMH1Gun6qNV1gpFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rPhNq-00Bnj1-22; Tue, 16 Jan 2024 11:13:30 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rPhNm-00Bnd8-2I for linux-arm-kernel@lists.infradead.org; Tue, 16 Jan 2024 11:13:28 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id AB926CE171F; Tue, 16 Jan 2024 11:13:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E0E8EC433F1; Tue, 16 Jan 2024 11:13:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1705403598; bh=S+0SZtA/75E2v3VkKaIyy4uPLM3LshStUgu21aLTubc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=iDmOVxuYwMchFyoYdVADTNZz9TMlhzIa/aKxHixxz0KQ41cIc3Y+CMlgbWUx83ddD QoXselPmeixw44LsSSnOmrPS3qexnnY1/yFH8E7RR1Aw4kayf7uE1aNLP2piWMUWYT QInccIBD99HZ3hTVX4RUuPjzGMOwgU9aB8Ct5rd9amKF/e1m4+LMkatl1xxoB0gzF0 WDDceGzuBCmWj4bM8tTkSaebP0wjHUVI5QJY0Lc8/6ed7KRKmRpfoGlJ7NVVABvCBn 7nga2kTsKzerF2P51PCnof/yJec1QCwRTbQSRZbQ4FoLBl0oTI6baBXqBHard97zcH uNQslch4RLRlQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rPhNc-00C5Q7-AA; Tue, 16 Jan 2024 11:13:16 +0000 Date: Tue, 16 Jan 2024 11:13:15 +0000 Message-ID: <86v87t8ras.wl-maz@kernel.org> From: Marc Zyngier To: "sundongxu (A)" Cc: , , , , , , , , , Subject: Re: [bug report] GICv4.1: VM performance degradation due to not trapping vCPU WFI In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: sundongxu3@huawei.com, oliver.upton@linux.dev, yuzenghui@huawei.com, james.morse@arm.com, suzuki.poulose@arm.com, will@kernel.org, catalin.marinas@arm.com, wanghaibin.wang@huawei.com, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240116_031327_129261_6A8F0312 X-CRM114-Status: GOOD ( 26.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 16 Jan 2024 03:26:08 +0000, "sundongxu (A)" wrote: > > Hi Guys, > > We found a problem about GICv4/4.1, for example: > We use QEMU to start a VM (4 vCPUs and 8G memory), VM disk was > configured with virtio, and the network is configured with vhost-net, > the CPU affinity of the vCPU and emulator is as follows, in VM xml: > > > > > > > > > > Running Mysql in the VM, and sysbench (Mysql benchmark) on the host, > the performance index is tps, the higher the better. > If the host only enabled GICv3, the tps will be around 1400. > If the host enabled GICv4.1, other configurations remain unchanged, the > tps will be around 40. > > We found that when the host enabled GICv4.1, because vSGI is directly > injected to VM, and most time vCPU exclusively occupy the pCPU, vCPU > will not trap when executing the WFI instruction. Then from the host > view, the CPU usage of vCPU0~vCPU3 is almost 100%. When running mysql > service in VM, the vhost-net and qemu processes also need to obtain > enough CPU time, but unfortunately these processes cannot get that much > time (for example, only GICv3 enabled, the cpu usage of vhost-net is > about 43%, but with GICv4.1 enabled, it becomes 0~2%). During the test, > it was found that vhost-net sleeps and wakes up very frequently. When > vhost-net wakes up, it often cannot obtain CPU in time (because of > wake-up preemption check). After waking up, vhost-net will usually run > for a short period of time before going to sleep again. Can you elaborate on this preemption check issue? > > If the host enabled GICv4.1, and force vCPU to trap when executing WFI, > the tps will be around 1400. > > On the other hand, when vCPU executes WFI instruction without trapping, > the vcpu wake-up delay will be significantly improved. For example, the > result of running cyclictest in VM: > WFI trap 6us > WFI no trap 2us > > Currently, I add a KVM module parameter to control whether the vCPU > traps (by set or clear HCR_TWI) when executing the WFI instruction with > host enabled GICv4/4.1, and by default, vCPU traps are set. > > Or, it there a better way? As you foudn out, KVM has an adaptive way of dealing with HCR_TWI, turning it off when the vcpu is alone in the run queue. Which means it doesn't compete with any other thread. How comes the other threads don't register as being runnable? Effectively, we apply the same principle to vSGIs as to vLPIs, and it was found that this heuristic was pretty beneficial to vLPIs. I'm a bit surprised that vSGIs are so different in their usage pattern. Does it help if you move your "emulatorpin" to some other physical CPUs? Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel