From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 147C33B2D39; Sat, 19 Sep 2026 09:22:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789809775; cv=none; b=RxSSElwBg4F1WCiKzZRuAzrFcqqFgow59MvS+YZOrQqhzAg/M+GALlx88uzLDlo0jUjTAx/4LSNxXwzUBGGvTvJaDDCX+jkf5Ep3buT5PSlIBRpmw72ms9de71R71ED5bjg80vC44+8/FxfcVpj7TH1XosEwoVadnQf6uwxkmuA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789809775; c=relaxed/simple; bh=s0Q9Jr7Ajsxct8ZN+S1ktDBZtevCFOzsYwmtNormGOg=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=jbJ43ZA5/j1nfeG6YCWB+Ax7Ha/Gc6LCnYn4s2+Bw6v4ksO1PFRoTfOBQDzmtJXHEllfvlTRAlGlYXAEBTx8Qm1eam4LVnmqF5N7Wja4enszZw7sDQesRQK3x+yPeP3NodSbantGNAZsZsik5VV+tK3l8aSF1ZqyHzIb+AeD49g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=moAckOx/; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="moAckOx/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F9801F000FF; Sat, 19 Sep 2026 09:22:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789809773; bh=p2++DSH7qzxPkUFh8j6R7Yh1w31wukoOyoqI9yjfdzM=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=moAckOx/BPVEJ1RzH3Y4pPsnwzsAWrGRbxm3lk29vHVa/fqmDMYCpF8+laeQ8Fiqm vJhorci6+huPu6TpBGzoelksDEEX/IQiY0dRJ0mKPS8vlQ/rAh0mISFmMwPMe8fkf1 VoCJfmw5k5d/ljYIKnyFd/KA8OvSz+JnXAT/Mfq4+z/EkVjnzbktETwTRcTKJDyUeL qmOr6mQ+wNCd2qyZnDppyIokInddhCdc52KGjbvXGaA4q4msaibojUzdBfB+kvBbj6 eqmb2BzIbv7iOmAHMreAvi62AE1MhsBlYOLjxCeddTeWJ5Igubyss5hwZ4xz0RXamk 5cGYpvHhLpGjg== Received: from sofa.misterjones.org ([185.219.108.64] helo=lobster-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1x7rHX-0000000AvKU-1Eeq; Sat, 19 Sep 2026 09:22:51 +0000 Date: Sat, 19 Sep 2026 10:25:57 +0100 Message-ID: <87h5jl4moq.wl-maz@kernel.org> From: Marc Zyngier To: Sascha Bischoff Cc: "linux-arm-kernel@lists.infradead.org" , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , nd , "oliver.upton@linux.dev" , Joey Gouly , Suzuki Poulose , "yuzenghui@huawei.com" , "peter.maydell@linaro.org" , "lpieralisi@kernel.org" , Timothy Hayes , "fuad.tabba@linux.dev" Subject: Re: [PATCH v6 09/49] KVM: arm64: gic-v5: Create and manage VM and VPE tables In-Reply-To: <20260904113404.4051341-10-sascha.bischoff@arm.com> References: <20260904113404.4051341-1-sascha.bischoff@arm.com> <20260904113404.4051341-10-sascha.bischoff@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: Sascha.Bischoff@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, nd@arm.com, oliver.upton@linux.dev, Joey.Gouly@arm.com, Suzuki.Poulose@arm.com, yuzenghui@huawei.com, peter.maydell@linaro.org, lpieralisi@kernel.org, Timothy.Hayes@arm.com, fuad.tabba@linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Fri, 04 Sep 2026 12:38:51 +0100, Sascha Bischoff wrote: > > GICv5 uses a set of in-memory tables to track and manage VM state. > These must be allocated by the hypervisor and provided to the IRS. > > The VMT (Virtual Machine Table) is a linear or two-level table > comprising VMT Entries (VMTEs). Each VMTE describes the state for a > single VM. This state includes things such as the SPI and LPI IST > configuration (coming in a future commit), an implementation-defined > VM Descriptor, and a VPE Table (VPET). > > The VPET contains one entry per possible VPE ID belonging to a VM. It > is used to mark a VPE as valid and provide the address of an > implementation-defined VPE Descriptor (VPED), which the hardware uses > to track and manage VPE state. > > Allocate each VM's VPEDs as a single dense array indexed by vcpu_idx, > while the VPET remains indexed by the userspace-provided vcpu_id. This > keeps VPED storage proportional to the number of vCPUs even when their > IDs are sparse. > > The VMT and VPET are shared with the IRS. On systems with a > non-coherent IRS, cache maintenance operates at cache-line > granularity, while multiple entries can occupy the same cache line. > Use a common lock for CPU accesses to these tables and IRS command > processing so that writing back one entry cannot overwrite an IRS > update to a neighbouring entry. > > The implementation-defined VMD and VPED storage is also visible to the > IRS. Round these allocations up to whole cache lines to prevent cache > maintenance from corrupting unrelated slab objects. Initialise the > storage before publishing its addresses to the IRS. > > This commit adds support for allocating the VMT and its descriptor > backing state, and for managing VMTEs. The VMTEs can be initialised or > released for reuse. VM IDs are allocated with an IDA, while an XArray > tracks the host-side allocations associated with populated VMTEs. > > Signed-off-by: Sascha Bischoff > --- > arch/arm64/kvm/Makefile | 2 +- > arch/arm64/kvm/vgic/vgic-init.c | 2 + > arch/arm64/kvm/vgic/vgic-v5-tables.c | 681 +++++++++++++++++++++++++++ > arch/arm64/kvm/vgic/vgic-v5-tables.h | 100 ++++ > arch/arm64/kvm/vgic/vgic-v5.c | 19 + > drivers/irqchip/irq-gic-v5-irs.c | 12 +- > include/kvm/arm_vgic.h | 4 + > include/linux/irqchip/arm-gic-v5.h | 14 +- > 8 files changed, 824 insertions(+), 10 deletions(-) > create mode 100644 arch/arm64/kvm/vgic/vgic-v5-tables.c > create mode 100644 arch/arm64/kvm/vgic/vgic-v5-tables.h > [...] The lkp bot is reporting some interesting issues in 202609190650.liMAzwjF-lkp@intel.com: > + /* Allocate and assign the VM Descriptor, if required. */ > + if (vmt_info->vmd_size != 0) { > + vmd_alloc_size = round_up(vmt_info->vmd_size, > + dma_get_cache_alignment()); > + vmd = kzalloc(vmd_alloc_size, GFP_KERNEL); > + if (!vmd) { > + ret = -ENOMEM; > + goto out_fail; > + } > + > + /* Stash the VA so we can free it later */ > + vmi->vmd_base = vmd; The VMD is dynamically allocated. > + > + tmp = FIELD_PREP(GICV5_VMTEL2E_VMD_ADDR, > + virt_to_phys(vmd) >> GICV5_VMTEL2E_VMD_ADDR_SHIFT); > + vmte_val0 = tmp; > + } > + > + /* > + * Allocate and assign the VPE Table. > + * > + * First of all, iterate over all vcpus to find the highest VPE ID we > + * require - we need to ensure that we have enough storage for all > + * vcpu_id values that userspace has picked and not just the total > + * number of vcpus. This gives us the number of VPEs required for the > + * VM. > + * > + * Round up the number of VPEs to a whole power of two as we cannot > + * describe non-powers-of-two in the VMTE field as it conveys the number > + * of ID bits used and not the number of vPEs. IRS_IDR1.IAFFID_BITS is > + * encoded as N - 1, so expose at least one VPE ID bit even for a > + * single-vCPU VM to keep the views consistent. > + */ > + kvm_for_each_vcpu(i, vcpu, kvm) { > + u16 vpe_id = vgic_v5_vpe_id(vcpu); > + > + if (vpe_id > max_vpe_id) > + max_vpe_id = vpe_id; > + } > + > + nr_cpus = max(2UL, roundup_pow_of_two(max_vpe_id + 1)); > + vmi->vpe_id_bits = fls(nr_cpus) - 1; > + > + vpet_alloc_size = round_up((size_t)nr_cpus * sizeof(*vpet), > + dma_get_cache_alignment()); > + vpet = kzalloc(vpet_alloc_size, GFP_KERNEL); > + if (!vpet) { > + ret = -ENOMEM; > + goto out_fail; > + } > + > + /* Stash the VA so we can free it later */ > + vmi->vpet_base = vpet; and so is the VPET. [...] > diff --git a/arch/arm64/kvm/vgic/vgic-v5-tables.h b/arch/arm64/kvm/vgic/vgic-v5-tables.h > new file mode 100644 > index 0000000000000..962be0c7cd3f6 > --- /dev/null > +++ b/arch/arm64/kvm/vgic/vgic-v5-tables.h > @@ -0,0 +1,100 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +/* > + * Copyright (C) 2025, 2026 Arm Ltd. > + */ > + > +#ifndef __KVM_ARM_VGICV5_TABLES_H__ > +#define __KVM_ARM_VGICV5_TABLES_H__ > + > +#include > +#include > +#include > + > +/* Level 1 Virtual Machine Table Entry */ > +typedef __le64 vmtl1_entry; > + > +/* Level 2 Virtual Machine Table Entry */ > +struct vmtl2_entry { > + __le64 val[4]; > +}; > + > +/* Virtual PE Table Entry */ > +typedef __le64 vpe_entry; > + > +struct vgic_v5_vm_info { > + void __iomem *vmd_base; > + vpe_entry __iomem *vpet_base; But pointers to allocated memory cannot be __iomem, as this qualifies MMIO registers. I'll queue the change below on top. M. diff --git a/arch/arm64/kvm/vgic/vgic-v5-tables.h b/arch/arm64/kvm/vgic/vgic-v5-tables.h index b2ffbe68c05b4..89096926563f7 100644 --- a/arch/arm64/kvm/vgic/vgic-v5-tables.h +++ b/arch/arm64/kvm/vgic/vgic-v5-tables.h @@ -23,8 +23,8 @@ struct vmtl2_entry { typedef __le64 vpe_entry; struct vgic_v5_vm_info { - void __iomem *vmd_base; - vpe_entry __iomem *vpet_base; + void *vmd_base; + vpe_entry *vpet_base; void *vped_base; u8 vpe_id_bits; -- Jazz isn't dead. It just smells funny.