From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FC98CCFA13 for ; Thu, 30 Apr 2026 11:16:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BOtF7KKTWEC524hrl0lXwosAKe0svdMvHPFsDgnjuNI=; b=A4mmT9FWuv5Zwm6t0kjWkbO08l 0JCznnV2ik/Ao8b5tzHdw0ob5TKCiqin9p2PBqOcOv6YbZDbnMYuNAZ4NTY47W6zJCEmugCzeIXbz 9g3lpsOwp7Fi1rwWoi3muzHJUx30UugH9ILxfsc0jLCc7+m59ln88WMvsZyjDPjig10pAdzXgksm5 bT6rifJ93RJK1RuUp0G62KEStEzE9us9ftZWIj87ls3V6cEIS0bBY9xDBszQ9Pe4aChXIjbZ6NlTV HDjmwdSaJup3UY7lse94Z04TwqItIaBF0HDYIYp6yiVvJVJgdq8xaV7Wd237km5DyD44QEVqg7PPR 0dXZNtHw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1wIPNf-00000005LnP-0aXM; Thu, 30 Apr 2026 11:16:31 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1wIPNR-00000005LaE-1Net for linux-arm-kernel@bombadil.infradead.org; Thu, 30 Apr 2026 11:16:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=BOtF7KKTWEC524hrl0lXwosAKe0svdMvHPFsDgnjuNI=; b=b75KHsOMSuM80GJvPGRuxlxDfS zx33hpKkjwxn2RiH/T3wwn42wSZ0qFqZvy0Nq6rTMtyqQMgKnw2Jv7syy5fIJWZtOZtBaLtuNSFdP eeyNowLnq1EhxshRQBEcL8bk3HpxD504SdorRoX3iP3kZ+hNS7pLqrJB1uOc6RxKQL98/c6Cd8MNN mRKz83IQ5wCjU8UHE9nPgJeBy1D14mQzersL6NJlDiDCfHRY2GcaKJQvgMijOrCxwD1yPoWYoyaU1 2HsBkOz19Uf+K5sG13//1X6v7rZEoodaG5dYP3D+YnvMLF31DpmKWzLaC9KW17gOWLywU3Ny6GDv/ TMduRsfQ==; Received: from foss.arm.com ([217.140.110.172]) by desiato.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1wIPNL-00000007Drb-2vqC for linux-arm-kernel@lists.infradead.org; Thu, 30 Apr 2026 11:16:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5AA86356B; Thu, 30 Apr 2026 04:16:05 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5F1B03F763; Thu, 30 Apr 2026 04:16:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547770; bh=fPffxhpzXsn8izUh0F0BUdbzOcL0eKYDh++M+0U1pJY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F2FribgvepEhj4KmtyowEbRPZ1k2UjwoGhsaeKYWQ/KdYmhB4cIuuC7AG87K6o8Kr NwV01a4Jq8sUYRkmJ/N9azU5WKc96Ax+s+LKNDutqQIndBM0DZSPzZ/CizOJBU6yq8 qFSgzNc/18I0HhkgGm5tOqP81PSmbV2v//O2qyYc= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 11/12] KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine Date: Thu, 30 Apr 2026 12:14:15 +0100 Message-ID: <20260430111424.3479613-13-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260430_121612_104309_D86AC089 X-CRM114-Status: GOOD ( 17.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement arm64 version of kvm_arch_dirty_ring_clear() making use of FEAT_HACDBS. It works by transversing the dirty-ring and converting its entries into HDBSS entries based on the slot offset. The resulting HDBSS array is then fed to the HACDBS mechanism that walks the pagetable marking writable-dirty pages as writable-clean. Only successfully cleaned entries are set as invalid on the dirty-ring, so in case of error, falling back to generic software cleaning will take care of any remaining entry in the dirty-ring. Signed-off-by: Leonardo Bras --- arch/arm64/include/asm/kvm_dirty_bit.h | 13 +++++ arch/arm64/kvm/dirty_bit.c | 66 ++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/asm/kvm_dirty_bit.h index 3d749f979c67..d76c109937d8 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -26,29 +26,42 @@ DECLARE_PER_CPU(struct hacdbs, hacdbs_pcp); void __init kvm_hacdbs_init(void); void kvm_hacdbs_cpu_up(void); void kvm_hacdbs_cpu_down(void); int __kvm_arch_dirty_log_clear(struct kvm *kvm, struct kvm_memory_slot *memslot, struct kvm_clear_dirty_log *log, unsigned long *bitmap, bool *flush); +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring, + int *nr_entries_reset); + static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm) { return this_cpu_read(hacdbs_pcp.status) == HACDBS_IDLE && (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM); } static inline int kvm_arch_dirty_log_clear(struct kvm *kvm, struct kvm_memory_slot *memslot, struct kvm_clear_dirty_log *log, unsigned long *bitmap, bool *flush) { if (!kvm_arch_dirty_clear_enabled(kvm)) return -EPERM; return __kvm_arch_dirty_log_clear(kvm, memslot, log, bitmap, flush); } +static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm, + struct kvm_dirty_ring *ring, + int *nr_entries_reset) +{ + if (!kvm_arch_dirty_clear_enabled(kvm)) + return -EPERM; + + return __kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset); +} + #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 0b7dcb8467c0..b99808a36469 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -256,20 +256,86 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm, ret = -EAGAIN; } write_unlock(&kvm->mmu_lock); kfree(hw_entries); return ret; } +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ring, + int *nr_entries_reset) +{ + u64 *hw_entries; + u64 slot_offset = 0; + u64 ttwl; + int i, ret; + u32 slot = -1; + + if (signal_pending(current)) + return -EINTR; + + ttwl = hdbss_get_ttwl(kvm->arch.mmu.split_page_chunk_size); + + hw_entries = kmalloc(max(ring->size * sizeof(u64), PAGE_SIZE), GFP_KERNEL); + if (!hw_entries) + return -ENOMEM; + + for (i = 0; i < ring->size; i++) { + struct kvm_dirty_gfn *entry; + gfn_t gfn; + + entry = &ring->dirty_gfns[(ring->reset_index + i) & + (ring->size - 1)]; + + if (!kvm_dirty_gfn_harvested(entry)) + break; + + if (entry->slot != slot) { + struct kvm_memory_slot *memslot; + + memslot = kvm_dirty_ring_get_memslot(kvm, entry->slot); + slot = entry->slot; + slot_offset = memslot->base_gfn; + } + + gfn = slot_offset + entry->offset; + + hw_entries[i] = (gfn_to_gpa(gfn) & HDBSS_ENTRY_IPA) | + ttwl | HDBSS_ENTRY_VALID; + } + + ret = dirty_bit_clear(kvm, hw_entries, i); + + /* Set as invalid all successfully cleaned entries */ + for (int j = 0; j < ret; j++) { + struct kvm_dirty_gfn *entry; + + entry = &ring->dirty_gfns[(ring->reset_index + j) & + (ring->size - 1)]; + + kvm_dirty_gfn_set_invalid(entry); + } + + /* In case of error, try software cleaning from the faulting entry */ + ring->reset_index += ret; + *nr_entries_reset += ret; + + kfree(hw_entries); + + if (ret < i) + return -EFAULT; + + return ret; +} + static irqreturn_t hacdbsirq_handler(int irq, void *pcpu) { u64 cons = read_sysreg_s(SYS_HACDBSCONS_EL2); unsigned long err = FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons); switch (err) { case HACDBSCONS_EL2_ERR_REASON_NOF: this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); break; case HACDBSCONS_EL2_ERR_REASON_IPAHACF: -- 2.54.0