From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5F3D1C16; Thu, 25 Jan 2024 11:02:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706180524; cv=none; b=QZ77GNqM3EDgLBqh/mA53uH87f5QOAl20LuIoQtk+eSU+iBBo3u4CPBZOWyrjQUrqFKoKq8+qhEv3iw0pvbhCSJ1NCxOcz9DjLAFeh3Zddewf64g5fY5HtjJFVfQBLvMkrrdIycn6PBCdBgJvmNFFATjyME7+3xZvjGa0KjFQBc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706180524; c=relaxed/simple; bh=QoLtJhax5FRP1/wP5osvaDsb01kHkav7FL5tHy/UekQ=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=ed7OL78+krY58aSDD6uRIOMmEZYOvGMhuPdeV59kDrDW5A1ULkP4RUy62qL4tLx9yTj/0JX57LZW2f/A9TCbW5S2uSM7Mzc5O+AKuBTXbIMZehJfE6ibGbKSSSrck40K1Z0S3CpCnb2KfItim6/+cj7RL/oC+a7hwMFkhqLGNKg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=enn/eet3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="enn/eet3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FDADC433C7; Thu, 25 Jan 2024 11:02:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1706180524; bh=QoLtJhax5FRP1/wP5osvaDsb01kHkav7FL5tHy/UekQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=enn/eet3O116NLrhiq7jlG6ANuEaBW6zFAvQdy07perA1dTDRc9THMguITl/NOlPN AXDsffIfgCgsZByTce3FxALTs9aWIAaCry/LETWFiBKwesDBwOMFRqh3MGB3oLsToV xpD3286z1wfNpGlRs8HNIfg+mD5M79xcMgz0Ph2V1zUqajdCKfRIdhX9SraeypljnY LuL5xMSpQRCiXgQ3eGfcaVMiMwRG2ZF5Pk3DOWq2dE+EY2I4AnIs5YUSQG68oGWe6g 4rde6c5igjr7+Dz2+vkUHM+yqF1H54V9Cs2cvvKs+iJ07kbhNmV9g/kbqzYuX97y8A XhOeNMydS7j8Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rSxUf-00Eh8C-TE; Thu, 25 Jan 2024 11:02:02 +0000 Date: Thu, 25 Jan 2024 11:02:01 +0000 Message-ID: <86zfwt7k2e.wl-maz@kernel.org> From: Marc Zyngier To: Oliver Upton Cc: kvmarm@lists.linux.dev, kvm@vger.kernel.org, James Morse , Suzuki K Poulose , Zenghui Yu , Raghavendra Rao Ananta , Jing Zhang Subject: Re: [PATCH 00/15] KVM: arm64: Improvements to GICv3 LPI injection In-Reply-To: <20240124204909.105952-1-oliver.upton@linux.dev> References: <20240124204909.105952-1-oliver.upton@linux.dev> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: oliver.upton@linux.dev, kvmarm@lists.linux.dev, kvm@vger.kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, rananta@google.com, jingzhangos@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Hi Oliver, On Wed, 24 Jan 2024 20:48:54 +0000, Oliver Upton wrote: > > The unfortunate reality is there are increasingly large systems that are > shipping today without support for GICv4 vLPI injection. Serialization > in KVM's LPI routing/injection code has been a significant bottleneck > for VMs on these machines when under a high load of LPIs (e.g. a > multi-queue NIC). > > Even though the long-term solution is quite clearly **direct > injection**, we really ought to do something about the LPI scaling > issues within KVM. > > This series aims to improve the performance of LPI routing/injection in > KVM by moving readers of LPI configuration data away from the > lpi_list_lock in favor or using RCU. > > Patches 1-5 change out the representation of LPIs in KVM from a > linked-list to an xarray. While not strictly necessary for making the > locking improvements, this seems to be an opportune time to switch to a > data structure that can actually be indexed. > > Patches 6-10 transition vgic_get_lpi() and vgic_put_lpi() away from > taking the lpi_list_lock in favor of using RCU for protection. Note that > this requires some rework to the way references are taken on LPIs and > how reclaim works to be RCU safe. > > Lastly, patches 11-15 rework the LRU policy on the LPI translation cache > to not require moving elements in the linked-list and take advantage of > this to make it an rculist readable outside of the lpi_list_lock. I quite like the overall direction. I've left a few comments here and there, and will probably get back to it after I try to run some tests on a big-ish box. > All of this was tested on top of v6.8-rc1. Apologies if any of the > changelogs are a bit too light, I'm happy to rework those further in > subsequent revisions. > > I would've liked to have benchmark data showing the improvement on top > of upstream with this series, but I'm currently having issues with our > internal infrastructure and upstream kernels. However, this series has > been found to have a near 2x performance improvement to redis-memtier [*] > benchmarks on our kernel tree. It'd be really good to have upstream-based numbers, with details of the actual setup (device assignment? virtio?) so that we can compare things and even track regressions in the future. Thanks, M. -- Without deviation from the norm, progress is not possible.