From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E2043D6CDA for ; Fri, 12 Jun 2026 16:24:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781281459; cv=none; b=Tg0Drt2lxIIr1lViErayPQw5VcpMJMPW/fy8CMLQ//CKn6y4tfVtHd0Vrf88ASOHaB7faDcotd16/W8RSXUsXiyfT2R5qubk4XsF/oVGIsTxRTTYlXDprmJTsJXqQNr4nqP5OgLCyZqhd+IKpMw8DY9QsK9nsETPfuSrC0SnT6s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781281459; c=relaxed/simple; bh=QRDRXTCX50amxKD5gLh7D6OBaXCfaBfzgoB4N+6bbzg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KxP/+EuwSPe6iFf47y8Fhh440xIyAMrDhGVd/UK+yRKX1gSGhumZFsqdMyu+y2DYPIJ4M0+XJnJO+zj8cr7oUVu1nkS92900xZWFXVvqy77a6/WO+0hl2dJm8PNylMLnneMJExAtpT9BJqkOECi0AY+Z76k/+PiNTavPJ3C9zz4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NoBrRehd; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NoBrRehd" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-490cdae130cso6509215e9.0 for ; Fri, 12 Jun 2026 09:24:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781281456; x=1781886256; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mJahK7oX1jpIBSr/85+UyDd//PxGmbpfN024eZe1Ebc=; b=NoBrRehdDwJVQdKa0C37zfYA0hO1eLZxOeIoJgnQZPq8ucrodATeQo4X4YpaNNQ0Md osOLlTZeLA1VzYGwGsJ1wpA7PP5QDgn+Nrsv8nw9J1odCsDJegjQPp3SMJ08I2uXX1F6 QYz4/D/vh4dg9d3gkQGcPd/glSiOyVYmDIE8MKfJskbXNXTJei7HetJpnJCe+uAYMUkF pHlPu51NPntaMF7PAHlWoxozVaeoxn9wFg5js6aWzEvv/Yb0bmsgxnd0iab5CZjazxI6 UzVV+9zw/Zs35zUxaIgihk+CTgZEjHE08QS1KiRmzBNfSsgPUFrpL9Gx8qCf3XFxTkPi dlWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781281456; x=1781886256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=mJahK7oX1jpIBSr/85+UyDd//PxGmbpfN024eZe1Ebc=; b=Sd88P6kZ5FDbT51RlMkVTkGbKOjj/2E7zftagfu08VTa420EUer0M6hACQCkjceTse ALKL0JITLZ2iqz7YepVrHVkrrZuaK7j3r6RaNKu3RDn7edDmYTuozVszWw5QLJzg9gR8 etUWhX021LHZ/pHtqJMa/rA/DCyfvMDwbCZ3mRjuKhstVANnhe+s+RU787uL8SNZGGES mlW4um7yrDO8iDIr5tQHfAKjZrqU5l7qT5o/NTVJP0TvPvHYjqGDAcy7Va2SLay3flpM VwyHJ35WtD+thAFRp8tF6ngAaByfrqQTgSCM22qxoyB5kaVXDFA2IeQOv/6+jDdxd4ah Mg+w== X-Forwarded-Encrypted: i=1; AFNElJ8kYy5M9CS4U9CDC2PdfbRGuQfDv6HdQue6+MJckRZX/9cq1Y0uzbyAvsmKJv4aHVCdl/vTfdSL38I=@vger.kernel.org X-Gm-Message-State: AOJu0Yz5fc5CsBCK4AB79OCyZOruLWb+uZGugb5xN3misLGFt2FesEEu VosRSQr6liZA4rstQMNfOlfmJlVzIPutThLz1rSa+lk+KPKuQO7JzQxS X-Gm-Gg: Acq92OFMNOW8Y3s6FeJYDJNyqISZaqTKu1B73txTc970wGyUFE5Rete7vA5QP1dKsMj 5DitH3WyZqafXjt7Uaf+bVuULBlj4gd6TD3yOO3aAGdJ3PPmVQ2b/h0OL/Bkk7F83PWG5nFh57o PNdG9TR2s7YeNV2gYJbpAQHR0Ss/4yUUYimh8+D5qwhKv5qvM+qHqUGDtBVCPDIBnxknxpVz1ns V3wx+vl2AgARKx4rELKtl9P1DKRQhwc4EHY+3/0ekzu1cMzyyjS0IRLWZxK623ANLezjmagBjHY CgiibUNCAFGjUHcWKszzjker50NM/QVsvwM5PwilkUsEqcq4RA+BdC19zTvb7h+KifQ2+FEVZBc MoKVKPb7/2pUkqQrzGqU9fEd4YAB3+mIePzZ0WazTDwISWPNWt4yUBot7TKageSlFX6zmJB3+Nv D3gCiq3shWKCOV9JSsmh1k8UIGqfcn76mPOrWwPHBh1jYS7q+g/r98xQhr/Q3+GQ== X-Received: by 2002:a05:600c:4745:b0:490:bd66:e523 with SMTP id 5b1f17b1804b1-492200c04a9mr1802785e9.20.1781281455606; Fri, 12 Jun 2026 09:24:15 -0700 (PDT) Received: from f4d4888f22f2.ant.amazon.com.com ([15.248.2.31]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490ea95c51dsm57620935e9.1.2026.06.12.09.24.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 12 Jun 2026 09:24:15 -0700 (PDT) From: Jack Thomson To: maz@kernel.org, oupton@kernel.org, pbonzini@redhat.com Cc: joey.gouly@arm.com, seiden@linux.ibm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, corbet@lwn.net, vladimir.murzin@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, isaku.yamahata@intel.com, Jack Thomson Subject: [PATCH v5 2/5] KVM: arm64: Add pre_fault_memory implementation Date: Fri, 12 Jun 2026 17:23:50 +0100 Message-ID: <20260612162354.73378-3-jackabt.amazon@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260612162354.73378-1-jackabt.amazon@gmail.com> References: <20260612162354.73378-1-jackabt.amazon@gmail.com> Precedence: bulk X-Mailing-List: linux-doc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Jack Thomson Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data abort and routing it through the existing stage-2 fault handlers. Treat the requested GPA as an IPA in the userspace-owned VM's memslot space and always target the canonical stage-2, even if the vCPU last ran with a nested/shadow MMU selected. If the vCPU last ran in a nested context, switch to the canonical stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU refcount state stay consistent. Leave the switch in place for the ioctl; vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load() reselects the correct MMU from vCPU state. Check existing mappings with a shared page-table walk under the MMU read lock, and use the resulting walk level when constructing the synthetic fault. Report poisoned pages through the ioctl return path with -EHWPOISON instead of also queueing SIGBUS, and use the installed mapping size to advance the prefault range. Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain unsupported: pKVM filters the capability, and the ioctl returns -EOPNOTSUPP if invoked anyway. Signed-off-by: Jack Thomson --- Documentation/virt/kvm/api.rst | 18 +++- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+), 4 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 52bbbb553ce1..657e05656fa6 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error @@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A memslot update raced with the ioctl before any page was + processed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero). ENOENT The specified `gpa` is outside defined memslots. EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN) @@ -6494,7 +6497,14 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed. + +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's +memslot address space. If the vCPU most recently ran a nested guest, KVM +still targets the VM's canonical stage-2, and does not interpret `gpa` as +a nested guest IPA or target the nested/shadow stage-2 selected by the +vCPU's last run state. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl @@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 449154f9a485..6b89262e8ba7 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -24,6 +24,7 @@ menuconfig KVM select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT + select KVM_GENERIC_PRE_FAULT_MEMORY select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_REL diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9453321ef8c6..dcb92bee13af 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_COUNTER_OFFSET: case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS: case KVM_CAP_ARM_SEA_TO_USER: + case KVM_CAP_PRE_FAULT_MEMORY: r = 1; break; case KVM_CAP_SET_GUEST_DEBUG2: diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c720f07cb82e..4bf048bbcf8b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc { struct kvm_s2_trans *nested; struct kvm_memory_slot *memslot; unsigned long hva; + unsigned long *page_size; + bool prefault; }; static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) @@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, &s2vi->map_writable, &s2vi->page); if (unlikely(is_error_noslot_pfn(s2vi->pfn))) { if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) { + /* + * When prefaulting, report the poison via -EHWPOISON + * only; don't also queue a SIGBUS as the run path + * does for the faulting vCPU thread. + */ + if (s2fd->prefault) + return -EHWPOISON; kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize)); return 0; } @@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd, kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable); kvm_fault_unlock(kvm); + if (s2fd->page_size && !ret) + *s2fd->page_size = mapping_size; + /* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled); } + +/* + * Prefaulting always targets the canonical stage-2. If the vCPU last ran + * in a nested context, swap in the canonical MMU via the vCPU put/load + * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount + * state stay consistent. + * + * The swap is deliberately not undone: nothing runs in between the + * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the + * generic prefault loop, and the vcpu_put() at ioctl exit discards + * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next + * vcpu_load() re-derives the correct MMU from the vCPU's context. If the + * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu() + * keeps the canonical MMU in place for the reload. Leaving the swap in + * place also bounds the cost to at most one put/load pair per ioctl, + * rather than two pairs per prefaulted page. + */ +static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu) +{ + if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu) + return; + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); +} + +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, + struct kvm_pre_fault_memory *range) +{ + struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault; + struct kvm_vcpu_fault_info fault_backup = *fault_info; + s8 walk_level = KVM_PGTABLE_LAST_LEVEL; + unsigned long page_size = PAGE_SIZE; + struct kvm_memory_slot *memslot; + phys_addr_t gpa = range->gpa; + struct kvm_pgtable *pgt; + phys_addr_t end; + kvm_pte_t pte; + hva_t hva; + gfn_t gfn; + long ret; + + if (vcpu_is_protected(vcpu)) + return -EOPNOTSUPP; + + /* + * Interpret range->gpa in the userspace-owned VM's IPA space, not in + * any nested guest IPA space that may have been active on the vCPU's + * last run. Always target the canonical stage-2. + */ + kvm_pre_fault_load_canonical_mmu(vcpu); + + if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) { + ret = -ENOENT; + goto out; + } + + gfn = gpa_to_gfn(gpa); + memslot = gfn_to_memslot(vcpu->kvm, gfn); + if (!memslot) { + ret = -ENOENT; + goto out; + } + + /* + * A racing memslot deletion or move installs an invalid slot before + * zapping stage-2. Ask userspace to retry once the update settles. + */ + if (memslot->flags & KVM_MEMSLOT_INVALID) { + ret = -EAGAIN; + goto out; + } + + /* + * pKVM stage-2 mappings aren't directly walkable from the host; let + * the fault path handle both new and existing mappings. + */ + if (!is_protected_kvm_enabled()) { + pgt = vcpu->arch.hw_mmu->pgt; + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) { + ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level, + KVM_PGTABLE_WALK_SHARED); + } + if (ret) + goto out; + + if (kvm_pte_valid(pte)) { + page_size = kvm_granule_size(walk_level); + if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF)) + handle_access_fault(vcpu, gpa); + goto out_success; + } + } + + /* + * Synthesize a read translation fault for the canonical IPA, at the + * level where the stage-2 walk currently ends (the last level under + * pKVM, where stage-2 isn't walkable from the host). + */ + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | + ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level); + fault_info->hpfar_el2 = HPFAR_EL2_NS | + FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12); + + struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested = NULL, + .memslot = memslot, + .page_size = &page_size, + .prefault = true, + }; + + /* + * As in the run path, -EAGAIN from the abort handlers is treated as + * progress: either a parallel fault installed the mapping, or a racing + * invalidation is in flight and the next access will refault. + */ + if (kvm_slot_has_gmem(memslot)) { + ret = gmem_abort(&s2fd); + } else { + hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL); + if (kvm_is_error_hva(hva)) { + ret = -EFAULT; + goto out; + } + + s2fd.hva = hva; + ret = user_mem_abort(&s2fd); + } + + if (ret < 0) + goto out; + +out_success: + end = ALIGN_DOWN(gpa, page_size) + page_size; + ret = min_t(u64, range->size, end - gpa); +out: + /* + * Restore the synthetic fault state so a subsequent KVM_RUN does not + * observe it. kvm_handle_mmio_return() runs before guest entry can + * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR + * in place would corrupt the completion of a pending MMIO exit. + */ + *fault_info = fault_backup; + return ret; +} -- 2.43.0