* [PATCH 6.18.y 1/6] arm64: tlb: Allow XZR argument to TLBI ops
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 2/6] arm64: tlb: Optimize ARM64_WORKAROUND_REPEAT_TLBI Catalin Marinas
` (4 subsequent siblings)
5 siblings, 0 replies; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
From: Mark Rutland <mark.rutland@arm.com>
commit bfd9c931d19aa59fb8371d557774fa169b15db9a upstream.
The TLBI instruction accepts XZR as a register argument, and for TLBI
operations with a register argument, there is no functional difference
between using XZR or another GPR which contains zeroes. Operations
without a register argument are encoded as if XZR were used.
Allow the __TLBI_1() macro to use XZR when a register argument is all
zeroes.
Today this only results in a trivial code saving in
__do_compat_cache_op()'s workaround for Neoverse-N1 erratum #1542419. In
subsequent patches this pattern will be used more generally.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 18a5dc0c9a54..0ddb344f83b4 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -38,12 +38,12 @@
: : )
#define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE \
- "tlbi " #op ", %0\n" \
+ "tlbi " #op ", %x0\n" \
ALTERNATIVE("nop\n nop", \
- "dsb ish\n tlbi " #op ", %0", \
+ "dsb ish\n tlbi " #op ", %x0", \
ARM64_WORKAROUND_REPEAT_TLBI, \
CONFIG_ARM64_WORKAROUND_REPEAT_TLBI) \
- : : "r" (arg))
+ : : "rZ" (arg))
#define __TLBI_N(op, arg, n, ...) __TLBI_##n(op, arg)
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6.18.y 2/6] arm64: tlb: Optimize ARM64_WORKAROUND_REPEAT_TLBI
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 1/6] arm64: tlb: Allow XZR argument to TLBI ops Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 3/6] arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance Catalin Marinas
` (3 subsequent siblings)
5 siblings, 0 replies; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
From: Mark Rutland <mark.rutland@arm.com>
commit a8f78680ee6bf795086384e8aea159a52814f827 upstream.
The ARM64_WORKAROUND_REPEAT_TLBI workaround is used to mitigate several
errata where broadcast TLBI;DSB sequences don't provide all the
architecturally required synchronization. The workaround performs more
work than necessary, and can have significant overhead. This patch
optimizes the workaround, as explained below.
The workaround was originally added for Qualcomm Falkor erratum 1009 in
commit:
d9ff80f83ecb ("arm64: Work around Falkor erratum 1009")
As noted in the message for that commit, the workaround is applied even
in cases where it is not strictly necessary.
The workaround was later reused without changes for:
* Arm Cortex-A76 erratum #1286807
SDEN v33: https://developer.arm.com/documentation/SDEN-885749/33-0/
* Arm Cortex-A55 erratum #2441007
SDEN v16: https://developer.arm.com/documentation/SDEN-859338/1600/
* Arm Cortex-A510 erratum #2441009
SDEN v19: https://developer.arm.com/documentation/SDEN-1873351/1900/
The important details to note are as follows:
1. All relevant errata only affect the ordering and/or completion of
memory accesses which have been translated by an invalidated TLB
entry. The actual invalidation of TLB entries is unaffected.
2. The existing workaround is applied to both broadcast and local TLB
invalidation, whereas for all relevant errata it is only necessary to
apply a workaround for broadcast invalidation.
3. The existing workaround replaces every TLBI with a TLBI;DSB;TLBI
sequence, whereas for all relevant errata it is only necessary to
execute a single additional TLBI;DSB sequence after any number of
TLBIs are completed by a DSB.
For example, for a sequence of batched TLBIs:
TLBI <op1>[, <arg1>]
TLBI <op2>[, <arg2>]
TLBI <op3>[, <arg3>]
DSB ISH
... the existing workaround will expand this to:
TLBI <op1>[, <arg1>]
DSB ISH // additional
TLBI <op1>[, <arg1>] // additional
TLBI <op2>[, <arg2>]
DSB ISH // additional
TLBI <op2>[, <arg2>] // additional
TLBI <op3>[, <arg3>]
DSB ISH // additional
TLBI <op3>[, <arg3>] // additional
DSB ISH
... whereas it is sufficient to have:
TLBI <op1>[, <arg1>]
TLBI <op2>[, <arg2>]
TLBI <op3>[, <arg3>]
DSB ISH
TLBI <opX>[, <argX>] // additional
DSB ISH // additional
Using a single additional TBLI and DSB at the end of the sequence can
have significantly lower overhead as each DSB which completes a TLBI
must synchronize with other PEs in the system, with potential
performance effects both locally and system-wide.
4. The existing workaround repeats each specific TLBI operation, whereas
for all relevant errata it is sufficient for the additional TLBI to
use *any* operation which will be broadcast, regardless of which
translation regime or stage of translation the operation applies to.
For example, for a single TLBI:
TLBI ALLE2IS
DSB ISH
... the existing workaround will expand this to:
TLBI ALLE2IS
DSB ISH
TLBI ALLE2IS // additional
DSB ISH // additional
... whereas it is sufficient to have:
TLBI ALLE2IS
DSB ISH
TLBI VALE1IS, XZR // additional
DSB ISH // additional
As the additional TLBI doesn't have to match a specific earlier TLBI,
the additional TLBI can be implemented in separate code, with no
memory of the earlier TLBIs. The additional TLBI can also use a
cheaper TLBI operation.
5. The existing workaround is applied to both Stage-1 and Stage-2 TLB
invalidation, whereas for all relevant errata it is only necessary to
apply a workaround for Stage-1 invalidation.
Architecturally, TLBI operations which invalidate only Stage-2
information (e.g. IPAS2E1IS) are not required to invalidate TLB
entries which combine information from Stage-1 and Stage-2
translation table entries, and consequently may not complete memory
accesses translated by those combined entries. In these cases,
completion of memory accesses is only guaranteed after subsequent
invalidation of Stage-1 information (e.g. VMALLE1IS).
Taking the above points into account, this patch reworks the workaround
logic to reduce overhead:
* New __tlbi_sync_s1ish() and __tlbi_sync_s1ish_hyp() functions are
added and used in place of any dsb(ish) which is used to complete
broadcast Stage-1 TLB maintenance. When the
ARM64_WORKAROUND_REPEAT_TLBI workaround is enabled, these helpers will
execute an additional TLBI;DSB sequence.
For consistency, it might make sense to add __tlbi_sync_*() helpers
for local and stage 2 maintenance. For now I've left those with
open-coded dsb() to keep the diff small.
* The duplication of TLBIs in __TLBI_0() and __TLBI_1() is removed. This
is no longer needed as the necessary synchronization will happen in
__tlbi_sync_s1ish() or __tlbi_sync_s1ish_hyp().
* The additional TLBI operation is chosen to have minimal impact:
- __tlbi_sync_s1ish() uses "TLBI VALE1IS, XZR". This is only used at
EL1 or at EL2 with {E2H,TGE}=={1,1}, where it will target an unused
entry for the reserved ASID in the kernel's own translation regime,
and have no adverse affect.
- __tlbi_sync_s1ish_hyp() uses "TLBI VALE2IS, XZR". This is only used
in hyp code, where it will target an unused entry in the hyp code's
TTBR0 mapping, and should have no adverse effect.
* As __TLBI_0() and __TLBI_1() no longer replace each TLBI with a
TLBI;DSB;TLBI sequence, batching TLBIs is worthwhile, and there's no
need for arch_tlbbatch_should_defer() to consider
ARM64_WORKAROUND_REPEAT_TLBI.
When building defconfig with GCC 15.1.0, compared to v6.19-rc1, this
patch saves ~1KiB of text, makes the vmlinux ~42KiB smaller, and makes
the resulting Image 64KiB smaller:
| [mark@lakrids:~/src/linux]% size vmlinux-*
| text data bss dec hex filename
| 21179831 19660919 708216 41548966 279fca6 vmlinux-after
| 21181075 19660903 708216 41550194 27a0172 vmlinux-before
| [mark@lakrids:~/src/linux]% ls -l vmlinux-*
| -rwxr-xr-x 1 mark mark 157771472 Feb 4 12:05 vmlinux-after
| -rwxr-xr-x 1 mark mark 157815432 Feb 4 12:05 vmlinux-before
| [mark@lakrids:~/src/linux]% ls -l Image-*
| -rw-r--r-- 1 mark mark 41007616 Feb 4 12:05 Image-after
| -rw-r--r-- 1 mark mark 41073152 Feb 4 12:05 Image-before
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 59 ++++++++++++++++++-------------
arch/arm64/kernel/sys_compat.c | 2 +-
arch/arm64/kvm/hyp/nvhe/mm.c | 2 +-
arch/arm64/kvm/hyp/nvhe/tlb.c | 8 ++---
arch/arm64/kvm/hyp/pgtable.c | 2 +-
arch/arm64/kvm/hyp/vhe/tlb.c | 10 +++---
6 files changed, 47 insertions(+), 36 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 0ddb344f83b4..c87d13bee37d 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -31,18 +31,10 @@
*/
#define __TLBI_0(op, arg) asm (ARM64_ASM_PREAMBLE \
"tlbi " #op "\n" \
- ALTERNATIVE("nop\n nop", \
- "dsb ish\n tlbi " #op, \
- ARM64_WORKAROUND_REPEAT_TLBI, \
- CONFIG_ARM64_WORKAROUND_REPEAT_TLBI) \
: : )
#define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE \
"tlbi " #op ", %x0\n" \
- ALTERNATIVE("nop\n nop", \
- "dsb ish\n tlbi " #op ", %x0", \
- ARM64_WORKAROUND_REPEAT_TLBI, \
- CONFIG_ARM64_WORKAROUND_REPEAT_TLBI) \
: : "rZ" (arg))
#define __TLBI_N(op, arg, n, ...) __TLBI_##n(op, arg)
@@ -181,6 +173,34 @@ static inline unsigned long get_trans_granule(void)
(__pages >> (5 * (scale) + 1)) - 1; \
})
+#define __repeat_tlbi_sync(op, arg...) \
+do { \
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_REPEAT_TLBI)) \
+ break; \
+ __tlbi(op, ##arg); \
+ dsb(ish); \
+} while (0)
+
+/*
+ * Complete broadcast TLB maintenance issued by the host which invalidates
+ * stage 1 information in the host's own translation regime.
+ */
+static inline void __tlbi_sync_s1ish(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale1is, 0);
+}
+
+/*
+ * Complete broadcast TLB maintenance issued by hyp code which invalidates
+ * stage 1 translation information in any translation regime.
+ */
+static inline void __tlbi_sync_s1ish_hyp(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale2is, 0);
+}
+
/*
* TLB Invalidation
* ================
@@ -266,7 +286,7 @@ static inline void flush_tlb_all(void)
{
dsb(ishst);
__tlbi(vmalle1is);
- dsb(ish);
+ __tlbi_sync_s1ish();
isb();
}
@@ -278,7 +298,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm)
asid = __TLBI_VADDR(0, ASID(mm));
__tlbi(aside1is, asid);
__tlbi_user(aside1is, asid);
- dsb(ish);
+ __tlbi_sync_s1ish();
mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
}
@@ -305,20 +325,11 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
unsigned long uaddr)
{
flush_tlb_page_nosync(vma, uaddr);
- dsb(ish);
+ __tlbi_sync_s1ish();
}
static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
{
- /*
- * TLB flush deferral is not required on systems which are affected by
- * ARM64_WORKAROUND_REPEAT_TLBI, as __tlbi()/__tlbi_user() implementation
- * will have two consecutive TLBI instructions with a dsb(ish) in between
- * defeating the purpose (i.e save overall 'dsb ish' cost).
- */
- if (alternative_has_cap_unlikely(ARM64_WORKAROUND_REPEAT_TLBI))
- return false;
-
return true;
}
@@ -334,7 +345,7 @@ static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- dsb(ish);
+ __tlbi_sync_s1ish();
}
/*
@@ -469,7 +480,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
{
__flush_tlb_range_nosync(vma->vm_mm, start, end, stride,
last_level, tlb_level);
- dsb(ish);
+ __tlbi_sync_s1ish();
}
static inline void flush_tlb_range(struct vm_area_struct *vma,
@@ -501,7 +512,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
dsb(ishst);
__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
- dsb(ish);
+ __tlbi_sync_s1ish();
isb();
}
@@ -515,7 +526,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
dsb(ishst);
__tlbi(vaae1is, addr);
- dsb(ish);
+ __tlbi_sync_s1ish();
isb();
}
diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c
index 4a609e9b65de..b9d4998c97ef 100644
--- a/arch/arm64/kernel/sys_compat.c
+++ b/arch/arm64/kernel/sys_compat.c
@@ -37,7 +37,7 @@ __do_compat_cache_op(unsigned long start, unsigned long end)
* We pick the reserved-ASID to minimise the impact.
*/
__tlbi(aside1is, __TLBI_VADDR(0, 0));
- dsb(ish);
+ __tlbi_sync_s1ish();
}
ret = caches_clean_inval_user_pou(start, start + chunk);
diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c
index ae8391baebc3..218976287d3f 100644
--- a/arch/arm64/kvm/hyp/nvhe/mm.c
+++ b/arch/arm64/kvm/hyp/nvhe/mm.c
@@ -271,7 +271,7 @@ static void fixmap_clear_slot(struct hyp_fixmap_slot *slot)
*/
dsb(ishst);
__tlbi_level(vale2is, __TLBI_VADDR(addr, 0), level);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
}
diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c
index 48da9ca9763f..3dc1ce0d27fe 100644
--- a/arch/arm64/kvm/hyp/nvhe/tlb.c
+++ b/arch/arm64/kvm/hyp/nvhe/tlb.c
@@ -169,7 +169,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
*/
dsb(ish);
__tlbi(vmalle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -226,7 +226,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
dsb(ish);
__tlbi(vmalle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -240,7 +240,7 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
enter_vmid_context(mmu, &cxt, false);
__tlbi(vmalls12e1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -266,5 +266,5 @@ void __kvm_flush_vm_context(void)
/* Same remark as in enter_vmid_context() */
dsb(ish);
__tlbi(alle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
}
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index c351b4abd5db..cbf8cd2e1673 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -483,7 +483,7 @@ static int hyp_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
*unmapped += granule;
}
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
mm_ops->put_page(ctx->ptep);
diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c
index ec2569818629..35855dadfb1b 100644
--- a/arch/arm64/kvm/hyp/vhe/tlb.c
+++ b/arch/arm64/kvm/hyp/vhe/tlb.c
@@ -115,7 +115,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
*/
dsb(ish);
__tlbi(vmalle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -176,7 +176,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
dsb(ish);
__tlbi(vmalle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -192,7 +192,7 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
enter_vmid_context(mmu, &cxt);
__tlbi(vmalls12e1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
exit_vmid_context(&cxt);
@@ -217,7 +217,7 @@ void __kvm_flush_vm_context(void)
{
dsb(ishst);
__tlbi(alle1is);
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
}
/*
@@ -358,7 +358,7 @@ int __kvm_tlbi_s1e2(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding)
default:
ret = -EINVAL;
}
- dsb(ish);
+ __tlbi_sync_s1ish_hyp();
isb();
if (mmu)
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6.18.y 3/6] arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 1/6] arm64: tlb: Allow XZR argument to TLBI ops Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 2/6] arm64: tlb: Optimize ARM64_WORKAROUND_REPEAT_TLBI Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-23 9:01 ` Patch "arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance" has been added to the 6.18-stable tree gregkh
2026-04-21 10:00 ` [PATCH 6.18.y 4/6] arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish() Catalin Marinas
` (2 subsequent siblings)
5 siblings, 1 reply; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
commit 6bfbf574a39139da11af9fdf6e8d56fe1989cd3e upstream.
Add __tlbi_sync_s1ish_kernel() similar to __tlbi_sync_s1ish() and use it
for kernel TLB maintenance. Also use this function in flush_tlb_all()
which is only used in relation to kernel mappings. Subsequent patches
can differentiate between workarounds that apply to user only or both
user and kernel.
A subsequent patch will add mm_struct to __tlbi_sync_s1ish(). Since
arch_tlbbatch_flush() is not specific to an mm, add a corresponding
__tlbi_sync_s1ish_batch() helper.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index c87d13bee37d..387bd86af702 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -191,6 +191,18 @@ static inline void __tlbi_sync_s1ish(void)
__repeat_tlbi_sync(vale1is, 0);
}
+static inline void __tlbi_sync_s1ish_batch(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale1is, 0);
+}
+
+static inline void __tlbi_sync_s1ish_kernel(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale1is, 0);
+}
+
/*
* Complete broadcast TLB maintenance issued by hyp code which invalidates
* stage 1 translation information in any translation regime.
@@ -286,7 +298,7 @@ static inline void flush_tlb_all(void)
{
dsb(ishst);
__tlbi(vmalle1is);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
@@ -345,7 +357,7 @@ static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_batch();
}
/*
@@ -512,7 +524,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
dsb(ishst);
__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
@@ -526,7 +538,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
dsb(ishst);
__tlbi(vaae1is, addr);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
^ permalink raw reply related [flat|nested] 10+ messages in thread* Patch "arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance" has been added to the 6.18-stable tree
2026-04-21 10:00 ` [PATCH 6.18.y 3/6] arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance Catalin Marinas
@ 2026-04-23 9:01 ` gregkh
0 siblings, 0 replies; 10+ messages in thread
From: gregkh @ 2026-04-23 9:01 UTC (permalink / raw)
To: catalin.marinas, gregkh, linux-arm-kernel, mark.rutland, will
Cc: stable-commits
This is a note to let you know that I've just added the patch titled
arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance
to the 6.18-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
arm64-tlb-introduce-__tlbi_sync_s1ish_-kernel-batch-for-tlb-maintenance.patch
and it can be found in the queue-6.18 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@vger.kernel.org> know about it.
From stable+bounces-240111-greg=kroah.com@vger.kernel.org Tue Apr 21 12:07:14 2026
From: Catalin Marinas <catalin.marinas@arm.com>
Date: Tue, 21 Apr 2026 11:00:14 +0100
Subject: arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance
To: stable@vger.kernel.org
Cc: Will Deacon <will@kernel.org>, linux-arm-kernel@lists.infradead.org
Message-ID: <20260421100018.335793-4-catalin.marinas@arm.com>
From: Catalin Marinas <catalin.marinas@arm.com>
commit 6bfbf574a39139da11af9fdf6e8d56fe1989cd3e upstream.
Add __tlbi_sync_s1ish_kernel() similar to __tlbi_sync_s1ish() and use it
for kernel TLB maintenance. Also use this function in flush_tlb_all()
which is only used in relation to kernel mappings. Subsequent patches
can differentiate between workarounds that apply to user only or both
user and kernel.
A subsequent patch will add mm_struct to __tlbi_sync_s1ish(). Since
arch_tlbbatch_flush() is not specific to an mm, add a corresponding
__tlbi_sync_s1ish_batch() helper.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/arm64/include/asm/tlbflush.h | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -191,6 +191,18 @@ static inline void __tlbi_sync_s1ish(voi
__repeat_tlbi_sync(vale1is, 0);
}
+static inline void __tlbi_sync_s1ish_batch(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale1is, 0);
+}
+
+static inline void __tlbi_sync_s1ish_kernel(void)
+{
+ dsb(ish);
+ __repeat_tlbi_sync(vale1is, 0);
+}
+
/*
* Complete broadcast TLB maintenance issued by hyp code which invalidates
* stage 1 translation information in any translation regime.
@@ -286,7 +298,7 @@ static inline void flush_tlb_all(void)
{
dsb(ishst);
__tlbi(vmalle1is);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
@@ -345,7 +357,7 @@ static inline bool arch_tlbbatch_should_
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_batch();
}
/*
@@ -512,7 +524,7 @@ static inline void flush_tlb_kernel_rang
dsb(ishst);
__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
@@ -526,7 +538,7 @@ static inline void __flush_tlb_kernel_pg
dsb(ishst);
__tlbi(vaae1is, addr);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish_kernel();
isb();
}
Patches currently in stable-queue which might be from catalin.marinas@arm.com are
queue-6.18/arm64-tlb-allow-xzr-argument-to-tlbi-ops.patch
queue-6.18/arm64-tlb-introduce-__tlbi_sync_s1ish_-kernel-batch-for-tlb-maintenance.patch
queue-6.18/arm64-tlb-optimize-arm64_workaround_repeat_tlbi.patch
queue-6.18/arm64-tlb-pass-the-corresponding-mm-to-__tlbi_sync_s1ish.patch
queue-6.18/arm64-cputype-add-c1-pro-definitions.patch
queue-6.18/arm64-errata-work-around-early-cme-dvmsync-acknowledgement.patch
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 6.18.y 4/6] arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish()
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
` (2 preceding siblings ...)
2026-04-21 10:00 ` [PATCH 6.18.y 3/6] arm64: tlb: Introduce __tlbi_sync_s1ish_{kernel,batch}() for TLB maintenance Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-23 9:01 ` Patch "arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish()" has been added to the 6.18-stable tree gregkh
2026-04-21 10:00 ` [PATCH 6.18.y 5/6] arm64: cputype: Add C1-Pro definitions Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 6/6] arm64: errata: Work around early CME DVMSync acknowledgement Catalin Marinas
5 siblings, 1 reply; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
commit d9fb08ba946a6190c371dcd9f9e465d0d52c5021 upstream.
The mm structure will be used for workarounds that need limiting to
specific tasks.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 8 ++++----
arch/arm64/kernel/sys_compat.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 387bd86af702..ba36e91aefb8 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -185,7 +185,7 @@ do { \
* Complete broadcast TLB maintenance issued by the host which invalidates
* stage 1 information in the host's own translation regime.
*/
-static inline void __tlbi_sync_s1ish(void)
+static inline void __tlbi_sync_s1ish(struct mm_struct *mm)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
@@ -310,7 +310,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm)
asid = __TLBI_VADDR(0, ASID(mm));
__tlbi(aside1is, asid);
__tlbi_user(aside1is, asid);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(mm);
mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
}
@@ -337,7 +337,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
unsigned long uaddr)
{
flush_tlb_page_nosync(vma, uaddr);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(vma->vm_mm);
}
static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
@@ -492,7 +492,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
{
__flush_tlb_range_nosync(vma->vm_mm, start, end, stride,
last_level, tlb_level);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(vma->vm_mm);
}
static inline void flush_tlb_range(struct vm_area_struct *vma,
diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c
index b9d4998c97ef..03fde2677d5b 100644
--- a/arch/arm64/kernel/sys_compat.c
+++ b/arch/arm64/kernel/sys_compat.c
@@ -37,7 +37,7 @@ __do_compat_cache_op(unsigned long start, unsigned long end)
* We pick the reserved-ASID to minimise the impact.
*/
__tlbi(aside1is, __TLBI_VADDR(0, 0));
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(current->mm);
}
ret = caches_clean_inval_user_pou(start, start + chunk);
^ permalink raw reply related [flat|nested] 10+ messages in thread* Patch "arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish()" has been added to the 6.18-stable tree
2026-04-21 10:00 ` [PATCH 6.18.y 4/6] arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish() Catalin Marinas
@ 2026-04-23 9:01 ` gregkh
0 siblings, 0 replies; 10+ messages in thread
From: gregkh @ 2026-04-23 9:01 UTC (permalink / raw)
To: catalin.marinas, gregkh, linux-arm-kernel, mark.rutland, will
Cc: stable-commits
This is a note to let you know that I've just added the patch titled
arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish()
to the 6.18-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
arm64-tlb-pass-the-corresponding-mm-to-__tlbi_sync_s1ish.patch
and it can be found in the queue-6.18 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@vger.kernel.org> know about it.
From stable+bounces-240112-greg=kroah.com@vger.kernel.org Tue Apr 21 12:02:34 2026
From: Catalin Marinas <catalin.marinas@arm.com>
Date: Tue, 21 Apr 2026 11:00:15 +0100
Subject: arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish()
To: stable@vger.kernel.org
Cc: Will Deacon <will@kernel.org>, linux-arm-kernel@lists.infradead.org
Message-ID: <20260421100018.335793-5-catalin.marinas@arm.com>
From: Catalin Marinas <catalin.marinas@arm.com>
commit d9fb08ba946a6190c371dcd9f9e465d0d52c5021 upstream.
The mm structure will be used for workarounds that need limiting to
specific tasks.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/arm64/include/asm/tlbflush.h | 8 ++++----
arch/arm64/kernel/sys_compat.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -185,7 +185,7 @@ do { \
* Complete broadcast TLB maintenance issued by the host which invalidates
* stage 1 information in the host's own translation regime.
*/
-static inline void __tlbi_sync_s1ish(void)
+static inline void __tlbi_sync_s1ish(struct mm_struct *mm)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
@@ -310,7 +310,7 @@ static inline void flush_tlb_mm(struct m
asid = __TLBI_VADDR(0, ASID(mm));
__tlbi(aside1is, asid);
__tlbi_user(aside1is, asid);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(mm);
mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
}
@@ -337,7 +337,7 @@ static inline void flush_tlb_page(struct
unsigned long uaddr)
{
flush_tlb_page_nosync(vma, uaddr);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(vma->vm_mm);
}
static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
@@ -492,7 +492,7 @@ static inline void __flush_tlb_range(str
{
__flush_tlb_range_nosync(vma->vm_mm, start, end, stride,
last_level, tlb_level);
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(vma->vm_mm);
}
static inline void flush_tlb_range(struct vm_area_struct *vma,
--- a/arch/arm64/kernel/sys_compat.c
+++ b/arch/arm64/kernel/sys_compat.c
@@ -37,7 +37,7 @@ __do_compat_cache_op(unsigned long start
* We pick the reserved-ASID to minimise the impact.
*/
__tlbi(aside1is, __TLBI_VADDR(0, 0));
- __tlbi_sync_s1ish();
+ __tlbi_sync_s1ish(current->mm);
}
ret = caches_clean_inval_user_pou(start, start + chunk);
Patches currently in stable-queue which might be from catalin.marinas@arm.com are
queue-6.18/arm64-tlb-allow-xzr-argument-to-tlbi-ops.patch
queue-6.18/arm64-tlb-introduce-__tlbi_sync_s1ish_-kernel-batch-for-tlb-maintenance.patch
queue-6.18/arm64-tlb-optimize-arm64_workaround_repeat_tlbi.patch
queue-6.18/arm64-tlb-pass-the-corresponding-mm-to-__tlbi_sync_s1ish.patch
queue-6.18/arm64-cputype-add-c1-pro-definitions.patch
queue-6.18/arm64-errata-work-around-early-cme-dvmsync-acknowledgement.patch
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 6.18.y 5/6] arm64: cputype: Add C1-Pro definitions
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
` (3 preceding siblings ...)
2026-04-21 10:00 ` [PATCH 6.18.y 4/6] arm64: tlb: Pass the corresponding mm to __tlbi_sync_s1ish() Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-21 10:00 ` [PATCH 6.18.y 6/6] arm64: errata: Work around early CME DVMSync acknowledgement Catalin Marinas
5 siblings, 0 replies; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
commit 2c99561016c591f4c3d5ad7d22a61b8726e79735 upstream.
Add cputype definitions for C1-Pro. These will be used for errata
detection in subsequent patches.
These values can be found in "Table A-303: MIDR_EL1 bit descriptions" in
issue 07 of the C1-Pro TRM:
https://documentation-service.arm.com/static/6930126730f8f55a656570af
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: James Morse <james.morse@arm.com>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/cputype.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index 9b00b75acbf2..18f98fb7ee78 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -98,6 +98,7 @@
#define ARM_CPU_PART_CORTEX_A725 0xD87
#define ARM_CPU_PART_CORTEX_A720AE 0xD89
#define ARM_CPU_PART_NEOVERSE_N3 0xD8E
+#define ARM_CPU_PART_C1_PRO 0xD8B
#define APM_CPU_PART_XGENE 0x000
#define APM_CPU_VAR_POTENZA 0x00
@@ -189,6 +190,7 @@
#define MIDR_CORTEX_A725 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A725)
#define MIDR_CORTEX_A720AE MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A720AE)
#define MIDR_NEOVERSE_N3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N3)
+#define MIDR_C1_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_C1_PRO)
#define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX)
#define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX)
#define MIDR_THUNDERX_83XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_83XX)
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6.18.y 6/6] arm64: errata: Work around early CME DVMSync acknowledgement
2026-04-21 10:00 [PATCH 6.18.y 0/6] arm64: Stable backport of the C1-Pro erratum 4193714 workaround Catalin Marinas
` (4 preceding siblings ...)
2026-04-21 10:00 ` [PATCH 6.18.y 5/6] arm64: cputype: Add C1-Pro definitions Catalin Marinas
@ 2026-04-21 10:00 ` Catalin Marinas
2026-04-23 9:01 ` Patch "arm64: errata: Work around early CME DVMSync acknowledgement" has been added to the 6.18-stable tree gregkh
5 siblings, 1 reply; 10+ messages in thread
From: Catalin Marinas @ 2026-04-21 10:00 UTC (permalink / raw)
To: stable; +Cc: Will Deacon, linux-arm-kernel
commit 0baba94a9779c13c857f6efc55807e6a45b1d4e4 upstream.
C1-Pro acknowledges DVMSync messages before completing the SME/CME
memory accesses. Work around this by issuing an IPI to the affected CPUs
if they are running in EL0 with SME enabled.
Note that we avoid the local DSB in the IPI handler as the kernel runs
with SCTLR_EL1.IESB=1. This is sufficient to complete SME memory
accesses at EL0 on taking an exception to EL1. On the return to user
path, no barrier is necessary either. See the comment in
sme_set_active() and the more detailed explanation in the link below.
To avoid a potential IPI flood from malicious applications (e.g.
madvise(MADV_PAGEOUT) in a tight loop), track where a process is active
via mm_cpumask() and only interrupt those CPUs.
Link: https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Mark Brown <broonie@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
Documentation/arch/arm64/silicon-errata.rst | 2 +
arch/arm64/Kconfig | 12 ++++
arch/arm64/include/asm/cpucaps.h | 2 +
arch/arm64/include/asm/fpsimd.h | 21 ++++++
arch/arm64/include/asm/tlbbatch.h | 10 ++-
arch/arm64/include/asm/tlbflush.h | 72 ++++++++++++++++++-
arch/arm64/kernel/cpu_errata.c | 30 ++++++++
arch/arm64/kernel/entry-common.c | 3 +
arch/arm64/kernel/fpsimd.c | 79 +++++++++++++++++++++
arch/arm64/kernel/process.c | 36 ++++++++++
arch/arm64/tools/cpucaps | 1 +
11 files changed, 264 insertions(+), 4 deletions(-)
diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index a7ec57060f64..93cdf1693715 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -202,6 +202,8 @@ stable kernels.
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Neoverse-V3AE | #3312417 | ARM64_ERRATUM_3194386 |
+----------------+-----------------+-----------------+-----------------------------+
+| ARM | C1-Pro | #4193714 | ARM64_ERRATUM_4193714 |
++----------------+-----------------+-----------------+-----------------------------+
| ARM | MMU-500 | #841119,826419 | ARM_SMMU_MMU_500_CPRE_ERRATA|
| | | #562869,1047329 | |
+----------------+-----------------+-----------------+-----------------------------+
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6663ffd23f25..840a945cb4ac 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1154,6 +1154,18 @@ config ARM64_ERRATUM_3194386
If unsure, say Y.
+config ARM64_ERRATUM_4193714
+ bool "C1-Pro: 4193714: SME DVMSync early acknowledgement"
+ depends on ARM64_SME
+ default y
+ help
+ Enable workaround for C1-Pro acknowledging the DVMSync before
+ the SME memory accesses are complete. This will cause TLB
+ maintenance for processes using SME to also issue an IPI to
+ the affected CPUs.
+
+ If unsure, say Y.
+
config CAVIUM_ERRATUM_22375
bool "Cavium erratum 22375, 24313"
default y
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 9d769291a306..121210b7ffd0 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -66,6 +66,8 @@ cpucap_is_possible(const unsigned int cap)
return IS_ENABLED(CONFIG_ARM64_WORKAROUND_REPEAT_TLBI);
case ARM64_WORKAROUND_SPECULATIVE_SSBS:
return IS_ENABLED(CONFIG_ARM64_ERRATUM_3194386);
+ case ARM64_WORKAROUND_4193714:
+ return IS_ENABLED(CONFIG_ARM64_ERRATUM_4193714);
case ARM64_MPAM:
/*
* KVM MPAM support doesn't rely on the host kernel supporting MPAM.
diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index b8cf0ea43cc0..0fa8d1d5722e 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -428,6 +428,24 @@ static inline size_t sme_state_size(struct task_struct const *task)
return __sme_state_size(task_get_sme_vl(task));
}
+void sme_enable_dvmsync(void);
+void sme_set_active(void);
+void sme_clear_active(void);
+
+static inline void sme_enter_from_user_mode(void)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714) &&
+ test_thread_flag(TIF_SME))
+ sme_clear_active();
+}
+
+static inline void sme_exit_to_user_mode(void)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714) &&
+ test_thread_flag(TIF_SME))
+ sme_set_active();
+}
+
#else
static inline void sme_user_disable(void) { BUILD_BUG(); }
@@ -456,6 +474,9 @@ static inline size_t sme_state_size(struct task_struct const *task)
return 0;
}
+static inline void sme_enter_from_user_mode(void) { }
+static inline void sme_exit_to_user_mode(void) { }
+
#endif /* ! CONFIG_ARM64_SME */
/* For use by EFI runtime services calls only */
diff --git a/arch/arm64/include/asm/tlbbatch.h b/arch/arm64/include/asm/tlbbatch.h
index fedb0b87b8db..6297631532e5 100644
--- a/arch/arm64/include/asm/tlbbatch.h
+++ b/arch/arm64/include/asm/tlbbatch.h
@@ -2,11 +2,17 @@
#ifndef _ARCH_ARM64_TLBBATCH_H
#define _ARCH_ARM64_TLBBATCH_H
+#include <linux/cpumask.h>
+
struct arch_tlbflush_unmap_batch {
+#ifdef CONFIG_ARM64_ERRATUM_4193714
/*
- * For arm64, HW can do tlb shootdown, so we don't
- * need to record cpumask for sending IPI
+ * Track CPUs that need SME DVMSync on completion of this batch.
+ * Otherwise, the arm64 HW can do tlb shootdown, so we don't need to
+ * record cpumask for sending IPI
*/
+ cpumask_var_t cpumask;
+#endif
};
#endif /* _ARCH_ARM64_TLBBATCH_H */
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index ba36e91aefb8..f53ab3ba0c48 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -80,6 +80,71 @@ static inline unsigned long get_trans_granule(void)
}
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+void sme_do_dvmsync(const struct cpumask *mask);
+
+static inline void sme_dvmsync(struct mm_struct *mm)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ sme_do_dvmsync(mm_cpumask(mm));
+}
+
+static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
+ struct mm_struct *mm)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ /*
+ * Order the mm_cpumask() read after the hardware DVMSync.
+ */
+ dsb(ish);
+ if (cpumask_empty(mm_cpumask(mm)))
+ return;
+
+ /*
+ * Allocate the batch cpumask on first use. Fall back to an immediate
+ * IPI for this mm in case of failure.
+ */
+ if (!cpumask_available(batch->cpumask) &&
+ !zalloc_cpumask_var(&batch->cpumask, GFP_ATOMIC)) {
+ sme_do_dvmsync(mm_cpumask(mm));
+ return;
+ }
+
+ cpumask_or(batch->cpumask, batch->cpumask, mm_cpumask(mm));
+}
+
+static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ if (!cpumask_available(batch->cpumask))
+ return;
+
+ sme_do_dvmsync(batch->cpumask);
+ cpumask_clear(batch->cpumask);
+}
+
+#else
+
+static inline void sme_dvmsync(struct mm_struct *mm)
+{
+}
+static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
+ struct mm_struct *mm)
+{
+}
+static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
+{
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
/*
* Level-based TLBI operations.
*
@@ -189,12 +254,14 @@ static inline void __tlbi_sync_s1ish(struct mm_struct *mm)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
+ sme_dvmsync(mm);
}
-static inline void __tlbi_sync_s1ish_batch(void)
+static inline void __tlbi_sync_s1ish_batch(struct arch_tlbflush_unmap_batch *batch)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
+ sme_dvmsync_batch(batch);
}
static inline void __tlbi_sync_s1ish_kernel(void)
@@ -357,7 +424,7 @@ static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- __tlbi_sync_s1ish_batch();
+ __tlbi_sync_s1ish_batch(batch);
}
/*
@@ -546,6 +613,7 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
struct mm_struct *mm, unsigned long start, unsigned long end)
{
__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
+ sme_dvmsync_add_pending(batch, mm);
}
#endif
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 8cb3b575a031..6c8c4301d9c6 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -11,6 +11,7 @@
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/cpufeature.h>
+#include <asm/fpsimd.h>
#include <asm/kvm_asm.h>
#include <asm/smp_plat.h>
@@ -551,6 +552,23 @@ static const struct midr_range erratum_spec_ssbs_list[] = {
};
#endif
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+static bool has_sme_dvmsync_erratum(const struct arm64_cpu_capabilities *entry,
+ int scope)
+{
+ if (!id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1)))
+ return false;
+
+ return is_affected_midr_range(entry, scope);
+}
+
+static void cpu_enable_sme_dvmsync(const struct arm64_cpu_capabilities *__unused)
+{
+ if (this_cpu_has_cap(ARM64_WORKAROUND_4193714))
+ sme_enable_dvmsync();
+}
+#endif
+
#ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38
static const struct midr_range erratum_ac03_cpu_38_list[] = {
MIDR_ALL_VERSIONS(MIDR_AMPERE1),
@@ -870,6 +888,18 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
ERRATA_MIDR_RANGE_LIST(erratum_spec_ssbs_list),
},
#endif
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+ {
+ .desc = "C1-Pro SME DVMSync early acknowledgement",
+ .capability = ARM64_WORKAROUND_4193714,
+ .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM,
+ .matches = has_sme_dvmsync_erratum,
+ .cpu_enable = cpu_enable_sme_dvmsync,
+ /* C1-Pro r0p0 - r1p2 (the latter only when REVIDR_EL1[0]==0) */
+ .midr_range = MIDR_RANGE(MIDR_C1_PRO, 0, 0, 1, 2),
+ MIDR_FIXED(MIDR_CPU_VAR_REV(1, 2), BIT(0)),
+ },
+#endif
#ifdef CONFIG_ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
{
.desc = "ARM errata 2966298, 3117295",
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index a9c81715ce59..5b97dfcf796d 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -21,6 +21,7 @@
#include <asm/daifflags.h>
#include <asm/esr.h>
#include <asm/exception.h>
+#include <asm/fpsimd.h>
#include <asm/irq_regs.h>
#include <asm/kprobes.h>
#include <asm/mmu.h>
@@ -84,6 +85,7 @@ static __always_inline void __enter_from_user_mode(struct pt_regs *regs)
{
enter_from_user_mode(regs);
mte_disable_tco_entry(current);
+ sme_enter_from_user_mode();
}
static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
@@ -102,6 +104,7 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
local_irq_disable();
exit_to_user_mode_prepare(regs);
local_daif_mask();
+ sme_exit_to_user_mode();
mte_check_tfsr_exit();
exit_to_user_mode();
}
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index e3f8f51748bc..ca18214ce2ab 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -15,6 +15,7 @@
#include <linux/compiler.h>
#include <linux/cpu.h>
#include <linux/cpu_pm.h>
+#include <linux/cpumask.h>
#include <linux/ctype.h>
#include <linux/kernel.h>
#include <linux/linkage.h>
@@ -28,6 +29,7 @@
#include <linux/sched/task_stack.h>
#include <linux/signal.h>
#include <linux/slab.h>
+#include <linux/smp.h>
#include <linux/stddef.h>
#include <linux/sysctl.h>
#include <linux/swab.h>
@@ -1384,6 +1386,83 @@ void do_sve_acc(unsigned long esr, struct pt_regs *regs)
put_cpu_fpsimd_context();
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+/*
+ * SME/CME erratum handling.
+ */
+static cpumask_t sme_dvmsync_cpus;
+
+/*
+ * These helpers are only called from non-preemptible contexts, so
+ * smp_processor_id() is safe here.
+ */
+void sme_set_active(void)
+{
+ unsigned int cpu = smp_processor_id();
+
+ if (!cpumask_test_cpu(cpu, &sme_dvmsync_cpus))
+ return;
+
+ cpumask_set_cpu(cpu, mm_cpumask(current->mm));
+
+ /*
+ * A subsequent (post ERET) SME access may use a stale address
+ * translation. On C1-Pro, a TLBI+DSB on a different CPU will wait for
+ * the completion of cpumask_set_cpu() above as it appears in program
+ * order before the SME access. The post-TLBI+DSB read of mm_cpumask()
+ * will lead to the IPI being issued.
+ *
+ * https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3
+ */
+}
+
+void sme_clear_active(void)
+{
+ unsigned int cpu = smp_processor_id();
+
+ if (!cpumask_test_cpu(cpu, &sme_dvmsync_cpus))
+ return;
+
+ /*
+ * With SCTLR_EL1.IESB enabled, the SME memory transactions are
+ * completed on entering EL1.
+ */
+ cpumask_clear_cpu(cpu, mm_cpumask(current->mm));
+}
+
+static void sme_dvmsync_ipi(void *unused)
+{
+ /*
+ * With SCTLR_EL1.IESB on, taking an exception is sufficient to ensure
+ * the completion of the SME memory accesses, so no need for an
+ * explicit DSB.
+ */
+}
+
+void sme_do_dvmsync(const struct cpumask *mask)
+{
+ /*
+ * This is called from the TLB maintenance functions after the DSB ISH
+ * to send the hardware DVMSync message. If this CPU sees the mask as
+ * empty, the remote CPU executing sme_set_active() would have seen
+ * the DVMSync and no IPI required.
+ */
+ if (cpumask_empty(mask))
+ return;
+
+ preempt_disable();
+ smp_call_function_many(mask, sme_dvmsync_ipi, NULL, true);
+ preempt_enable();
+}
+
+void sme_enable_dvmsync(void)
+{
+ cpumask_set_cpu(smp_processor_id(), &sme_dvmsync_cpus);
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
/*
* Trapped SME access
*
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 489554931231..4c328b7c79ba 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -26,6 +26,7 @@
#include <linux/reboot.h>
#include <linux/interrupt.h>
#include <linux/init.h>
+#include <linux/cpumask.h>
#include <linux/cpu.h>
#include <linux/elfcore.h>
#include <linux/pm.h>
@@ -339,8 +340,41 @@ void flush_thread(void)
flush_gcs();
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+static void arch_dup_tlbbatch_mask(struct task_struct *dst)
+{
+ /*
+ * Clear the inherited cpumask with memset() to cover both cases where
+ * cpumask_var_t is a pointer or an array. It will be allocated lazily
+ * in sme_dvmsync_add_pending() if CPUMASK_OFFSTACK=y.
+ */
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ memset(&dst->tlb_ubc.arch.cpumask, 0,
+ sizeof(dst->tlb_ubc.arch.cpumask));
+}
+
+static void arch_release_tlbbatch_mask(struct task_struct *tsk)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ free_cpumask_var(tsk->tlb_ubc.arch.cpumask);
+}
+
+#else
+
+static void arch_dup_tlbbatch_mask(struct task_struct *dst)
+{
+}
+
+static void arch_release_tlbbatch_mask(struct task_struct *tsk)
+{
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
void arch_release_task_struct(struct task_struct *tsk)
{
+ arch_release_tlbbatch_mask(tsk);
fpsimd_release_task(tsk);
}
@@ -356,6 +390,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
*dst = *src;
+ arch_dup_tlbbatch_mask(dst);
+
/*
* Drop stale reference to src's sve_state and convert dst to
* non-streaming FPSIMD mode.
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 1b32c1232d28..16d123088ddd 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -101,6 +101,7 @@ WORKAROUND_2077057
WORKAROUND_2457168
WORKAROUND_2645198
WORKAROUND_2658417
+WORKAROUND_4193714
WORKAROUND_AMPERE_AC03_CPU_38
WORKAROUND_AMPERE_AC04_CPU_23
WORKAROUND_TRBE_OVERWRITE_FILL_MODE
^ permalink raw reply related [flat|nested] 10+ messages in thread* Patch "arm64: errata: Work around early CME DVMSync acknowledgement" has been added to the 6.18-stable tree
2026-04-21 10:00 ` [PATCH 6.18.y 6/6] arm64: errata: Work around early CME DVMSync acknowledgement Catalin Marinas
@ 2026-04-23 9:01 ` gregkh
0 siblings, 0 replies; 10+ messages in thread
From: gregkh @ 2026-04-23 9:01 UTC (permalink / raw)
To: broonie, catalin.marinas, gregkh, james.morse, linux-arm-kernel,
mark.rutland, will
Cc: stable-commits
This is a note to let you know that I've just added the patch titled
arm64: errata: Work around early CME DVMSync acknowledgement
to the 6.18-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
arm64-errata-work-around-early-cme-dvmsync-acknowledgement.patch
and it can be found in the queue-6.18 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@vger.kernel.org> know about it.
From stable+bounces-240114-greg=kroah.com@vger.kernel.org Tue Apr 21 12:07:30 2026
From: Catalin Marinas <catalin.marinas@arm.com>
Date: Tue, 21 Apr 2026 11:00:17 +0100
Subject: arm64: errata: Work around early CME DVMSync acknowledgement
To: stable@vger.kernel.org
Cc: Will Deacon <will@kernel.org>, linux-arm-kernel@lists.infradead.org
Message-ID: <20260421100018.335793-7-catalin.marinas@arm.com>
From: Catalin Marinas <catalin.marinas@arm.com>
commit 0baba94a9779c13c857f6efc55807e6a45b1d4e4 upstream.
C1-Pro acknowledges DVMSync messages before completing the SME/CME
memory accesses. Work around this by issuing an IPI to the affected CPUs
if they are running in EL0 with SME enabled.
Note that we avoid the local DSB in the IPI handler as the kernel runs
with SCTLR_EL1.IESB=1. This is sufficient to complete SME memory
accesses at EL0 on taking an exception to EL1. On the return to user
path, no barrier is necessary either. See the comment in
sme_set_active() and the more detailed explanation in the link below.
To avoid a potential IPI flood from malicious applications (e.g.
madvise(MADV_PAGEOUT) in a tight loop), track where a process is active
via mm_cpumask() and only interrupt those CPUs.
Link: https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Mark Brown <broonie@kernel.org>
Reviewed-by: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
Documentation/arch/arm64/silicon-errata.rst | 2
arch/arm64/Kconfig | 12 ++++
arch/arm64/include/asm/cpucaps.h | 2
arch/arm64/include/asm/fpsimd.h | 21 +++++++
arch/arm64/include/asm/tlbbatch.h | 10 ++-
arch/arm64/include/asm/tlbflush.h | 72 ++++++++++++++++++++++++-
arch/arm64/kernel/cpu_errata.c | 30 ++++++++++
arch/arm64/kernel/entry-common.c | 3 +
arch/arm64/kernel/fpsimd.c | 79 ++++++++++++++++++++++++++++
arch/arm64/kernel/process.c | 36 ++++++++++++
arch/arm64/tools/cpucaps | 1
11 files changed, 264 insertions(+), 4 deletions(-)
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -202,6 +202,8 @@ stable kernels.
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Neoverse-V3AE | #3312417 | ARM64_ERRATUM_3194386 |
+----------------+-----------------+-----------------+-----------------------------+
+| ARM | C1-Pro | #4193714 | ARM64_ERRATUM_4193714 |
++----------------+-----------------+-----------------+-----------------------------+
| ARM | MMU-500 | #841119,826419 | ARM_SMMU_MMU_500_CPRE_ERRATA|
| | | #562869,1047329 | |
+----------------+-----------------+-----------------+-----------------------------+
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1154,6 +1154,18 @@ config ARM64_ERRATUM_3194386
If unsure, say Y.
+config ARM64_ERRATUM_4193714
+ bool "C1-Pro: 4193714: SME DVMSync early acknowledgement"
+ depends on ARM64_SME
+ default y
+ help
+ Enable workaround for C1-Pro acknowledging the DVMSync before
+ the SME memory accesses are complete. This will cause TLB
+ maintenance for processes using SME to also issue an IPI to
+ the affected CPUs.
+
+ If unsure, say Y.
+
config CAVIUM_ERRATUM_22375
bool "Cavium erratum 22375, 24313"
default y
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -66,6 +66,8 @@ cpucap_is_possible(const unsigned int ca
return IS_ENABLED(CONFIG_ARM64_WORKAROUND_REPEAT_TLBI);
case ARM64_WORKAROUND_SPECULATIVE_SSBS:
return IS_ENABLED(CONFIG_ARM64_ERRATUM_3194386);
+ case ARM64_WORKAROUND_4193714:
+ return IS_ENABLED(CONFIG_ARM64_ERRATUM_4193714);
case ARM64_MPAM:
/*
* KVM MPAM support doesn't rely on the host kernel supporting MPAM.
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -428,6 +428,24 @@ static inline size_t sme_state_size(stru
return __sme_state_size(task_get_sme_vl(task));
}
+void sme_enable_dvmsync(void);
+void sme_set_active(void);
+void sme_clear_active(void);
+
+static inline void sme_enter_from_user_mode(void)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714) &&
+ test_thread_flag(TIF_SME))
+ sme_clear_active();
+}
+
+static inline void sme_exit_to_user_mode(void)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714) &&
+ test_thread_flag(TIF_SME))
+ sme_set_active();
+}
+
#else
static inline void sme_user_disable(void) { BUILD_BUG(); }
@@ -456,6 +474,9 @@ static inline size_t sme_state_size(stru
return 0;
}
+static inline void sme_enter_from_user_mode(void) { }
+static inline void sme_exit_to_user_mode(void) { }
+
#endif /* ! CONFIG_ARM64_SME */
/* For use by EFI runtime services calls only */
--- a/arch/arm64/include/asm/tlbbatch.h
+++ b/arch/arm64/include/asm/tlbbatch.h
@@ -2,11 +2,17 @@
#ifndef _ARCH_ARM64_TLBBATCH_H
#define _ARCH_ARM64_TLBBATCH_H
+#include <linux/cpumask.h>
+
struct arch_tlbflush_unmap_batch {
+#ifdef CONFIG_ARM64_ERRATUM_4193714
/*
- * For arm64, HW can do tlb shootdown, so we don't
- * need to record cpumask for sending IPI
+ * Track CPUs that need SME DVMSync on completion of this batch.
+ * Otherwise, the arm64 HW can do tlb shootdown, so we don't need to
+ * record cpumask for sending IPI
*/
+ cpumask_var_t cpumask;
+#endif
};
#endif /* _ARCH_ARM64_TLBBATCH_H */
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -80,6 +80,71 @@ static inline unsigned long get_trans_gr
}
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+void sme_do_dvmsync(const struct cpumask *mask);
+
+static inline void sme_dvmsync(struct mm_struct *mm)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ sme_do_dvmsync(mm_cpumask(mm));
+}
+
+static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
+ struct mm_struct *mm)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ /*
+ * Order the mm_cpumask() read after the hardware DVMSync.
+ */
+ dsb(ish);
+ if (cpumask_empty(mm_cpumask(mm)))
+ return;
+
+ /*
+ * Allocate the batch cpumask on first use. Fall back to an immediate
+ * IPI for this mm in case of failure.
+ */
+ if (!cpumask_available(batch->cpumask) &&
+ !zalloc_cpumask_var(&batch->cpumask, GFP_ATOMIC)) {
+ sme_do_dvmsync(mm_cpumask(mm));
+ return;
+ }
+
+ cpumask_or(batch->cpumask, batch->cpumask, mm_cpumask(mm));
+}
+
+static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
+{
+ if (!alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ return;
+
+ if (!cpumask_available(batch->cpumask))
+ return;
+
+ sme_do_dvmsync(batch->cpumask);
+ cpumask_clear(batch->cpumask);
+}
+
+#else
+
+static inline void sme_dvmsync(struct mm_struct *mm)
+{
+}
+static inline void sme_dvmsync_add_pending(struct arch_tlbflush_unmap_batch *batch,
+ struct mm_struct *mm)
+{
+}
+static inline void sme_dvmsync_batch(struct arch_tlbflush_unmap_batch *batch)
+{
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
/*
* Level-based TLBI operations.
*
@@ -189,12 +254,14 @@ static inline void __tlbi_sync_s1ish(str
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
+ sme_dvmsync(mm);
}
-static inline void __tlbi_sync_s1ish_batch(void)
+static inline void __tlbi_sync_s1ish_batch(struct arch_tlbflush_unmap_batch *batch)
{
dsb(ish);
__repeat_tlbi_sync(vale1is, 0);
+ sme_dvmsync_batch(batch);
}
static inline void __tlbi_sync_s1ish_kernel(void)
@@ -357,7 +424,7 @@ static inline bool arch_tlbbatch_should_
*/
static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
{
- __tlbi_sync_s1ish_batch();
+ __tlbi_sync_s1ish_batch(batch);
}
/*
@@ -546,6 +613,7 @@ static inline void arch_tlbbatch_add_pen
struct mm_struct *mm, unsigned long start, unsigned long end)
{
__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
+ sme_dvmsync_add_pending(batch, mm);
}
#endif
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -11,6 +11,7 @@
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/cpufeature.h>
+#include <asm/fpsimd.h>
#include <asm/kvm_asm.h>
#include <asm/smp_plat.h>
@@ -551,6 +552,23 @@ static const struct midr_range erratum_s
};
#endif
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+static bool has_sme_dvmsync_erratum(const struct arm64_cpu_capabilities *entry,
+ int scope)
+{
+ if (!id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1)))
+ return false;
+
+ return is_affected_midr_range(entry, scope);
+}
+
+static void cpu_enable_sme_dvmsync(const struct arm64_cpu_capabilities *__unused)
+{
+ if (this_cpu_has_cap(ARM64_WORKAROUND_4193714))
+ sme_enable_dvmsync();
+}
+#endif
+
#ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38
static const struct midr_range erratum_ac03_cpu_38_list[] = {
MIDR_ALL_VERSIONS(MIDR_AMPERE1),
@@ -870,6 +888,18 @@ const struct arm64_cpu_capabilities arm6
ERRATA_MIDR_RANGE_LIST(erratum_spec_ssbs_list),
},
#endif
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+ {
+ .desc = "C1-Pro SME DVMSync early acknowledgement",
+ .capability = ARM64_WORKAROUND_4193714,
+ .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM,
+ .matches = has_sme_dvmsync_erratum,
+ .cpu_enable = cpu_enable_sme_dvmsync,
+ /* C1-Pro r0p0 - r1p2 (the latter only when REVIDR_EL1[0]==0) */
+ .midr_range = MIDR_RANGE(MIDR_C1_PRO, 0, 0, 1, 2),
+ MIDR_FIXED(MIDR_CPU_VAR_REV(1, 2), BIT(0)),
+ },
+#endif
#ifdef CONFIG_ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
{
.desc = "ARM errata 2966298, 3117295",
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -21,6 +21,7 @@
#include <asm/daifflags.h>
#include <asm/esr.h>
#include <asm/exception.h>
+#include <asm/fpsimd.h>
#include <asm/irq_regs.h>
#include <asm/kprobes.h>
#include <asm/mmu.h>
@@ -84,6 +85,7 @@ static __always_inline void __enter_from
{
enter_from_user_mode(regs);
mte_disable_tco_entry(current);
+ sme_enter_from_user_mode();
}
static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
@@ -102,6 +104,7 @@ static __always_inline void arm64_exit_t
local_irq_disable();
exit_to_user_mode_prepare(regs);
local_daif_mask();
+ sme_exit_to_user_mode();
mte_check_tfsr_exit();
exit_to_user_mode();
}
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -15,6 +15,7 @@
#include <linux/compiler.h>
#include <linux/cpu.h>
#include <linux/cpu_pm.h>
+#include <linux/cpumask.h>
#include <linux/ctype.h>
#include <linux/kernel.h>
#include <linux/linkage.h>
@@ -28,6 +29,7 @@
#include <linux/sched/task_stack.h>
#include <linux/signal.h>
#include <linux/slab.h>
+#include <linux/smp.h>
#include <linux/stddef.h>
#include <linux/sysctl.h>
#include <linux/swab.h>
@@ -1384,6 +1386,83 @@ void do_sve_acc(unsigned long esr, struc
put_cpu_fpsimd_context();
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+/*
+ * SME/CME erratum handling.
+ */
+static cpumask_t sme_dvmsync_cpus;
+
+/*
+ * These helpers are only called from non-preemptible contexts, so
+ * smp_processor_id() is safe here.
+ */
+void sme_set_active(void)
+{
+ unsigned int cpu = smp_processor_id();
+
+ if (!cpumask_test_cpu(cpu, &sme_dvmsync_cpus))
+ return;
+
+ cpumask_set_cpu(cpu, mm_cpumask(current->mm));
+
+ /*
+ * A subsequent (post ERET) SME access may use a stale address
+ * translation. On C1-Pro, a TLBI+DSB on a different CPU will wait for
+ * the completion of cpumask_set_cpu() above as it appears in program
+ * order before the SME access. The post-TLBI+DSB read of mm_cpumask()
+ * will lead to the IPI being issued.
+ *
+ * https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3
+ */
+}
+
+void sme_clear_active(void)
+{
+ unsigned int cpu = smp_processor_id();
+
+ if (!cpumask_test_cpu(cpu, &sme_dvmsync_cpus))
+ return;
+
+ /*
+ * With SCTLR_EL1.IESB enabled, the SME memory transactions are
+ * completed on entering EL1.
+ */
+ cpumask_clear_cpu(cpu, mm_cpumask(current->mm));
+}
+
+static void sme_dvmsync_ipi(void *unused)
+{
+ /*
+ * With SCTLR_EL1.IESB on, taking an exception is sufficient to ensure
+ * the completion of the SME memory accesses, so no need for an
+ * explicit DSB.
+ */
+}
+
+void sme_do_dvmsync(const struct cpumask *mask)
+{
+ /*
+ * This is called from the TLB maintenance functions after the DSB ISH
+ * to send the hardware DVMSync message. If this CPU sees the mask as
+ * empty, the remote CPU executing sme_set_active() would have seen
+ * the DVMSync and no IPI required.
+ */
+ if (cpumask_empty(mask))
+ return;
+
+ preempt_disable();
+ smp_call_function_many(mask, sme_dvmsync_ipi, NULL, true);
+ preempt_enable();
+}
+
+void sme_enable_dvmsync(void)
+{
+ cpumask_set_cpu(smp_processor_id(), &sme_dvmsync_cpus);
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
/*
* Trapped SME access
*
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -26,6 +26,7 @@
#include <linux/reboot.h>
#include <linux/interrupt.h>
#include <linux/init.h>
+#include <linux/cpumask.h>
#include <linux/cpu.h>
#include <linux/elfcore.h>
#include <linux/pm.h>
@@ -339,8 +340,41 @@ void flush_thread(void)
flush_gcs();
}
+#ifdef CONFIG_ARM64_ERRATUM_4193714
+
+static void arch_dup_tlbbatch_mask(struct task_struct *dst)
+{
+ /*
+ * Clear the inherited cpumask with memset() to cover both cases where
+ * cpumask_var_t is a pointer or an array. It will be allocated lazily
+ * in sme_dvmsync_add_pending() if CPUMASK_OFFSTACK=y.
+ */
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ memset(&dst->tlb_ubc.arch.cpumask, 0,
+ sizeof(dst->tlb_ubc.arch.cpumask));
+}
+
+static void arch_release_tlbbatch_mask(struct task_struct *tsk)
+{
+ if (alternative_has_cap_unlikely(ARM64_WORKAROUND_4193714))
+ free_cpumask_var(tsk->tlb_ubc.arch.cpumask);
+}
+
+#else
+
+static void arch_dup_tlbbatch_mask(struct task_struct *dst)
+{
+}
+
+static void arch_release_tlbbatch_mask(struct task_struct *tsk)
+{
+}
+
+#endif /* CONFIG_ARM64_ERRATUM_4193714 */
+
void arch_release_task_struct(struct task_struct *tsk)
{
+ arch_release_tlbbatch_mask(tsk);
fpsimd_release_task(tsk);
}
@@ -356,6 +390,8 @@ int arch_dup_task_struct(struct task_str
*dst = *src;
+ arch_dup_tlbbatch_mask(dst);
+
/*
* Drop stale reference to src's sve_state and convert dst to
* non-streaming FPSIMD mode.
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -101,6 +101,7 @@ WORKAROUND_2077057
WORKAROUND_2457168
WORKAROUND_2645198
WORKAROUND_2658417
+WORKAROUND_4193714
WORKAROUND_AMPERE_AC03_CPU_38
WORKAROUND_AMPERE_AC04_CPU_23
WORKAROUND_TRBE_OVERWRITE_FILL_MODE
Patches currently in stable-queue which might be from catalin.marinas@arm.com are
queue-6.18/arm64-tlb-allow-xzr-argument-to-tlbi-ops.patch
queue-6.18/arm64-tlb-introduce-__tlbi_sync_s1ish_-kernel-batch-for-tlb-maintenance.patch
queue-6.18/arm64-tlb-optimize-arm64_workaround_repeat_tlbi.patch
queue-6.18/arm64-tlb-pass-the-corresponding-mm-to-__tlbi_sync_s1ish.patch
queue-6.18/arm64-cputype-add-c1-pro-definitions.patch
queue-6.18/arm64-errata-work-around-early-cme-dvmsync-acknowledgement.patch
^ permalink raw reply [flat|nested] 10+ messages in thread