From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D328B499F0C; Wed, 9 Sep 2026 14:25:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788963902; cv=none; b=faV0mdlAKUWcG6keFuHC6pQat9Mmc8LeqfzNnm1ziz5y7B0yS2znNN66dSBrqz9JbO8IQBWi06EcMkC2wp1DWd9gcbx3h3JVM4RIF3IUyhUaFqXVHEiYe1oexa1XSxgA4qRa9x9ZJgxlqwRKb+KcfGoE5NcYWlKQc82queDVgyU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788963902; c=relaxed/simple; bh=ULpncWDJMgomUF68aAY/PeSV4wBrO68j1eZMZMsP9ms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DOs3gxkmwXQkjAQHpRxtNq0g+rp4B8qt/rweVD5Zs5MPwSSRjBZOoZKauxo4M2fxk3HqrBLa03CPu39QADDf/nW4fIxhk7uKlSSIVG7wizvIG80JrE91EcUWGe+SyRopUHIpOdhQsHW83pL8aFzO1/N/VBoUBZsozsjBJ0y6EIw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b=k139WFNa; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b="k139WFNa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37E9D1F00A3A; Wed, 9 Sep 2026 14:25:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1788963900; bh=ltU/gqfcoFvcP7yHRVS2ZylT5segreVKUm5AdAvSOeY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=k139WFNaVZv0qiigoK6ESfLSiUIndXLxpvUt2aInhHhh1I/fJGZG21Go++XtU1RLr I8QG5qHkXt/LJb8OvTohDUbepDPxpjqDV2+6GHO/fp5JMeEg0OfiSqMVnIOn2SKh0V IAnYbH6eE5Rdds5wJKOE1v8rzq3xdWBz+tPBQzME= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Sean Christopherson , Phil Rosenthal Subject: [PATCH 6.18 239/583] KVM: x86/mmu: Consume the locked rmap value in the lockless rmap walk Date: Wed, 9 Sep 2026 15:38:44 +0200 Message-ID: <20260909134246.429442043@linuxfoundation.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260909134237.773280130@linuxfoundation.org> References: <20260909134237.773280130@linuxfoundation.org> User-Agent: quilt/0.69 X-stable: review X-Patchwork-Hint: ignore Precedence: bulk X-Mailing-List: patches@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit 6.18-stable review patch. If anyone has any objections, please let me know. ------------------ From: Phil Rosenthal commit e428f9779a43737d830111238816f1928b07aefb upstream. __kvm_rmap_lock() deliberately elides the rmap lock when it observes an empty rmap. In that case kvm_rmap_lock_readonly() also re-enables preemption and returns zero, so the caller holds neither the rmap lock nor a preemption reference. The elision documents the invariant it relies on: * Elide the lock if the rmap is empty, as lockless walkers (read-only * mode) don't need to (and can't) walk an empty rmap, nor can they add * entries to the rmap. I.e. the only paths that process empty rmaps * do so while holding mmu_lock for write, and are mutually exclusive. kvm_rmap_age_gfn_range() ignores the returned value and unconditionally enters for_each_rmap_spte_lockless(). The iterator started with rmap_get_first(), which re-reads rmap_head->val rather than using the value returned by the lock. If a writer populates the rmap between the lock's read and the iterator's re-read, the aging path walks the newly installed rmap without holding its lock. For a KVM_RMAP_MANY rmap this leaves the walker following a pte_list_desc chain that it never locked. A writer holding mmu_lock for write may free that chain (e.g. kvm_zap_all_rmap_sptes() on the recycle path, or any rmap zap) via kmem_cache_free() while the walk is in progress, giving a slab use-after-free. Nothing serialises the two: the aging path runs without mmu_lock when CONFIG_KVM_MMU_LOCKLESS_AGING=y, and the rmap lock that would otherwise exclude the writer was elided. Because the empty path re-enables preemption, the interval between the two reads can span an arbitrary scheduling delay. Fix the class of bug by having the lockless walk consume the value returned by the lock instead of re-reading the rmap. Split rmap_get_first() into __rmap_get_first(), which starts an iterator from an already-read rmap value, and make for_each_rmap_spte_lockless() take that value and call __rmap_get_first() directly. kvm_rmap_age_gfn_range() passes the value returned by kvm_rmap_lock_readonly(): when the lock was elided the value is zero, __rmap_get_first() returns NULL, and the walk is skipped. No lockless walker re-reads the rmap, so the lock-elision invariant cannot be violated, and no lock()-without-paired-unlock() path is added to the aging code. Fixes: af3b6a9eba48 ("KVM: x86/mmu: Walk rmaps (shadow MMU) without holding mmu_lock when aging gfns") Suggested-by: Sean Christopherson Cc: stable@vger.kernel.org Signed-off-by: Phil Rosenthal Link: https://patch.msgid.link/20260720-rmap-age-elided-submit-v2-1-668973030d47@phil.gs Signed-off-by: Sean Christopherson Signed-off-by: Greg Kroah-Hartman --- arch/x86/kvm/mmu/mmu.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1218,18 +1218,9 @@ struct rmap_iterator { int pos; /* index of the sptep */ }; -/* - * Iteration must be started by this function. This should also be used after - * removing/dropping sptes from the rmap link because in such cases the - * information in the iterator may not be valid. - * - * Returns sptep if found, NULL otherwise. - */ -static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, - struct rmap_iterator *iter) +static u64 *__rmap_get_first(unsigned long rmap_val, + struct rmap_iterator *iter) { - unsigned long rmap_val = kvm_rmap_get(rmap_head); - if (!rmap_val) return NULL; @@ -1244,6 +1235,19 @@ static u64 *rmap_get_first(struct kvm_rm } /* + * Iteration must be started by this function. This should also be used after + * removing/dropping sptes from the rmap link because in such cases the + * information in the iterator may not be valid. + * + * Returns sptep if found, NULL otherwise. + */ +static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, + struct rmap_iterator *iter) +{ + return __rmap_get_first(kvm_rmap_get(rmap_head), iter); +} + +/* * Must be used with a valid iterator: e.g. after rmap_get_first(). * * Returns sptep if found, NULL otherwise. @@ -1277,8 +1281,9 @@ static u64 *rmap_get_next(struct rmap_it __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ if (!WARN_ON_ONCE(!is_shadow_present_pte(*(_sptep_)))) \ -#define for_each_rmap_spte_lockless(_rmap_head_, _iter_, _sptep_, _spte_) \ - __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ +#define for_each_rmap_spte_lockless(_rmap_val_, _iter_, _sptep_, _spte_) \ + for (_sptep_ = __rmap_get_first(_rmap_val_, _iter_); \ + _sptep_; _sptep_ = rmap_get_next(_iter_)) \ if (is_shadow_present_pte(_spte_ = mmu_spte_get_lockless(sptep))) static void drop_spte(struct kvm *kvm, u64 *sptep) @@ -1716,7 +1721,7 @@ static bool kvm_rmap_age_gfn_range(struc rmap_head = gfn_to_rmap(gfn, level, range->slot); rmap_val = kvm_rmap_lock_readonly(rmap_head); - for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { + for_each_rmap_spte_lockless(rmap_val, &iter, sptep, spte) { if (!is_accessed_spte(spte)) continue;