From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01A2FC4332F for ; Fri, 18 Nov 2022 00:21:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nifo40Ao/NfKLORhebUYqyE0kIIh8Kj4WrkaxWDjoos=; b=hdkCQP1BtDu8FV XLEAnTkT8w8sYx/1BsduNkXz/6vH7vexTV85Mxv/iEYgY0Orc5uowZpAXCUtVOO1Bx5YJl8XyBoq1 ArhFn5L49St4Bpthb5HMlr5HhCwrtczW6IiOptHj0ozZKmiL1CSRXpSJldxfzNRlfis1W/h9j7H3+ 89lpYgvQAjHxbGTV99ZnPUWvWL44eaWr1B2YEujBbHucEmxkcJ72Cr+KMVenWrQ+ToLKpoldOd+sX 2+72yFviHGTOeta1l5xXFbq1HJODWAoVm+YBVUhYQTeep5FyAvzURCHfjcqS1Uoe/aZDIl89ou+El qIspUHn9mcTbEO3xa0aA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ovp7Y-000slO-6g; Fri, 18 Nov 2022 00:20:40 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ovp7T-000sk0-Nw for linux-arm-kernel@lists.infradead.org; Fri, 18 Nov 2022 00:20:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 36264B810DF; Fri, 18 Nov 2022 00:20:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E0DD1C433C1; Fri, 18 Nov 2022 00:20:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668730832; bh=UPKFz2DsiakqWhGmlH6ehaMqM4wm6M3vd2T7GaE1SQw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=imroKEniBTTIqxq1/mcUCDjw0gXXBCz/5lchVAcVG5jEO3ZeNbN2jlyHR7W2fLaPq WsKOtqPfrLtoxMNvxsBVu0rUhlnsnlrAGbyaILTeLAuRa0I2QJqysQbak1S8I2EkS+ SLMy4MKsd/r2wmkrT9XiSrRjf96AaDtD6zD0OlQCdaWnSP9LViEUcpuBmVWxMJHdmy m7rESH7yae05V8s5QOYoMy5IiOIcMESXCWdzXbHOeof6TwOv715rmmPk/YXN2d59Pv +QE4ZIxIJnjN9ifVukmDw7c/VfTTija0hA6eDgt92nbUmt/LKBVFBdHMi++HjOhLGX wELms2dbIY27w== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1ovp7O-006roN-TM; Fri, 18 Nov 2022 00:20:31 +0000 Date: Fri, 18 Nov 2022 00:20:30 +0000 Message-ID: <86r0y1nmep.wl-maz@kernel.org> From: Marc Zyngier To: Usama Arif Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux-foundation.org, linux@armlinux.org.uk, yezengruan@huawei.com, catalin.marinas@arm.com, will@kernel.org, steven.price@arm.com, mark.rutland@arm.com, bagasdotme@gmail.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com Subject: Re: [External] Re: [v2 0/6] KVM: arm64: implement vcpu_is_preempted check In-Reply-To: <180b91af-a2aa-2cfd-eb7f-b2825c4e3dbe@bytedance.com> References: <20221104062105.4119003-1-usama.arif@bytedance.com> <87k048f3cm.wl-maz@kernel.org> <180b91af-a2aa-2cfd-eb7f-b2825c4e3dbe@bytedance.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: usama.arif@bytedance.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux-foundation.org, linux@armlinux.org.uk, yezengruan@huawei.com, catalin.marinas@arm.com, will@kernel.org, steven.price@arm.com, mark.rutland@arm.com, bagasdotme@gmail.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221117_162036_075997_35D4F374 X-CRM114-Status: GOOD ( 34.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 07 Nov 2022 12:00:44 +0000, Usama Arif wrote: > > > > On 06/11/2022 16:35, Marc Zyngier wrote: > > On Fri, 04 Nov 2022 06:20:59 +0000, > > Usama Arif wrote: > >> > >> This patchset adds support for vcpu_is_preempted in arm64, which > >> allows the guest to check if a vcpu was scheduled out, which is > >> useful to know incase it was holding a lock. vcpu_is_preempted can > >> be used to improve performance in locking (see owner_on_cpu usage in > >> mutex_spin_on_owner, mutex_can_spin_on_owner, rtmutex_spin_on_owner > >> and osq_lock) and scheduling (see available_idle_cpu which is used > >> in several places in kernel/sched/fair.c for e.g. in wake_affine to > >> determine which CPU can run soonest): > > > > [...] > > > >> pvcy shows a smaller overall improvement (50%) compared to > >> vcpu_is_preempted (277%). Host side flamegraph analysis shows that > >> ~60% of the host time when using pvcy is spent in kvm_handle_wfx, > >> compared with ~1.5% when using vcpu_is_preempted, hence > >> vcpu_is_preempted shows a larger improvement. > > > > And have you worked out *why* we spend so much time handling WFE? > > > > M. > > Its from the following change in pvcy patchset: > > diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c > index e778eefcf214..915644816a85 100644 > --- a/arch/arm64/kvm/handle_exit.c > +++ b/arch/arm64/kvm/handle_exit.c > @@ -118,7 +118,12 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu) > } > > if (esr & ESR_ELx_WFx_ISS_WFE) { > - kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu)); > + int state; > + while ((state = kvm_pvcy_check_state(vcpu)) == 0) > + schedule(); > + > + if (state == -1) > + kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu)); > } else { > if (esr & ESR_ELx_WFx_ISS_WFxT) > vcpu_set_flag(vcpu, IN_WFIT); > > > If my understanding is correct of the pvcy changes, whenever pvcy > returns an unchanged vcpu state, we would schedule to another > vcpu. And its the constant scheduling where the time is spent. I guess > the affects are much higher when the lock contention is very > high. This can be seem from the pvcy host side flamegraph as well with > (~67% of the time spent in the schedule() call in kvm_handle_wfx), For > reference, I have put the graph at: > https://uarif1.github.io/pvlock/perf_host_pvcy_nmi.svg The real issue here is that we don't try to pick the right vcpu to run, and strictly rely on schedule() to eventually pick something that can run. An interesting to do would be to try and fit the directed yield mechanism there. It would be a lot more interesting than the one-off vcpu_is_preempted hack, as it gives us a low-level primitive on which to construct things (pvcy is effectively a mwait-like primitive). M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel