[PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue

[PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue

[Peng Fan (OSS)]

From: Peng Fan <peng.fan@nxp.com>

According to NXP errata document IMX8_1N94W[1], the i.MX8QuadMax SoC
suffers from a serious cache coherency issue (ERR050104). The upper
bits, above bit 35, of the ARADDR and ACADDR buses within the Arm A53
subsystem have been incorrectly connected. This causes some TLBI and IC
maintenance operations exchanged between the A53 and A72 core clusters
to be corrupted.

The workaround requires:

  - Downgrading targeted TLBI operations to broadcast-all variants.
    Instead of patching the low-level __TLBI_1 macro (which interferes
    with the REPEAT_TLBI workaround and causes excessive over-
    invalidation), redirect high-level TLB flush functions
    (flush_tlb_mm, __do_flush_tlb_range, flush_tlb_kernel_range,
    __flush_tlb_kernel_pgtable) to use VMALLE1IS via static key checks.

  - Upgrading IC IVAU to IC IALLUIS for both kernel (via ALTERNATIVE in
    invalidate_icache_by_line) and EL0 userspace (via trap-and-upgrade
    in user_cache_maint_handler with SCTLR_EL1.UCI=0).

  - Disabling KVM since correct TLB maintenance cannot be guaranteed
    for guests without the proper devicetree.

  - No need to touch SMMU Broadcast TLB Maintenance (BTM) since i.MX8QM
    does not support broadcast TLB.

SoC detection uses devicetree compatible string "fsl,imx8qm" since the
boot CPU MIDR_EL1 (0x410fd034) and AIDR_EL1 (0) are not unique to this
SoC.

[1] https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf

[ Reworked per review feedback from Will Deacon and Mark Rutland:
  - Move TLBI workaround from __TLBI_1 macro to high-level flush
    functions to avoid REPEAT_TLBI interaction issues
  - Add kernel IC IVAU upgrade via ALTERNATIVE in assembler.h
  - Add cpucap_is_possible() entry for compile-time elimination]

Co-developed-by: Ivan T. Ivanov <iivanov@suse.de>
Signed-off-by: Ivan T. Ivanov <iivanov@suse.de>
Link: https://lore.kernel.org/all/20230420112952.28340-1-iivanov@suse.de/
Signed-off-by: Peng Fan <peng.fan@nxp.com>
---
This picks up the work originally done by Ivan T. Ivanov in 2023 [1],
reworked to address review feedback from Will Deacon and Mark Rutland,
and rebased onto linux-next (next-20260723).

Changes from v2 [1]:
  - Moved TLBI workaround from __TLBI_1 macro to high-level flush
    functions (flush_tlb_mm, __do_flush_tlb_range, flush_tlb_kernel_range,
    __flush_tlb_kernel_pgtable) to avoid REPEAT_TLBI interaction issues,
    per Will Deacon suggestion to use static keys at the higher level.
  - Added kernel IC IVAU -> IC IALLUIS upgrade via ALTERNATIVE in
    assembler.h, per Mark Rutland review noting that only EL0 traps
    were handled in v2.
  - Added cpucap_is_possible() entry for compile-time elimination when
    CONFIG_NXP_IMX8QM_ERRATUM_ERR050104 is not set.
  - Dropped SMMU BTM changes since iMX8QM does not support broadcast TLB.

[1] https://lore.kernel.org/all/20230420112952.28340-1-iivanov@suse.de/
[2] https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf
---
Changes in v2:
- Skip the workaround for local (TLBF_NOBROADCAST) TLB flushes; they
  don't cross the core's external interface (Mark Rutland).
- Preserve IC IVAU fault reporting: run "ic ivau" for the local fault
  check, then "ic ialluis" only on success (Mark Rutland).
- Link to v1: https://patch.msgid.link/20260729-imx8qm-cache-coherency-v1-1-c9864e280437@nxp.com
---
 Documentation/arch/arm64/silicon-errata.rst |  2 +
 arch/arm64/Kconfig                          | 16 ++++++++
 arch/arm64/include/asm/assembler.h          |  5 +++
 arch/arm64/include/asm/cpucaps.h            |  2 +
 arch/arm64/include/asm/tlbflush.h           | 62 ++++++++++++++++++-----------
 arch/arm64/kernel/cpu_errata.c              | 20 ++++++++++
 arch/arm64/kernel/traps.c                   |  2 +
 arch/arm64/kvm/arm.c                        |  5 +++
 arch/arm64/tools/cpucaps                    |  1 +
 9 files changed, 91 insertions(+), 24 deletions(-)

diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index 868bd9eed9a6..a90fd2ec532b 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -321,6 +321,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | Freescale/NXP  | LS2080A/LS1043A | A-008585        | FSL_ERRATUM_A008585         |
 +----------------+-----------------+-----------------+-----------------------------+
+| Freescale/NXP  | i.MX 8QuadMax   | ERR050104       | NXP_IMX8QM_ERRATUM_ERR050104|
++----------------+-----------------+-----------------+-----------------------------+
 +----------------+-----------------+-----------------+-----------------------------+
 | Hisilicon      | Hip0{5,6,7}     | #161010101      | HISILICON_ERRATUM_161010101 |
 +----------------+-----------------+-----------------+-----------------------------+
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index faccdf847a67..eea6774baf0d 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1454,6 +1454,22 @@ config ROCKCHIP_ERRATUM_3588001
 
 	  If unsure, say Y.
 
+config NXP_IMX8QM_ERRATUM_ERR050104
+	bool "NXP iMX8QM ERR050104: broken cache/TLB invalidation broadcast"
+	default y
+	help
+	  On iMX8QM, address bits above bit 35 in the A53 subsystem ARADDR and
+	  ACADDR buses are incorrectly connected. This corrupts targeted TLBI
+	  and IC broadcasts exchanged between the A53 and A72 core clusters.
+
+	  Work around this by redirecting targeted TLBI operations to
+	  broadcast-all variants (VMALLE1IS) in the high-level TLB flush
+	  functions, upgrading IC IVAU to IC IALLUIS for both kernel and
+	  user-space, and disabling KVM since correct TLB maintenance
+	  cannot be guaranteed for guests.
+
+	  If unsure, say Y.
+
 config SOCIONEXT_SYNQUACER_PREITS
 	bool "Socionext Synquacer: Workaround for GICv3 pre-ITS"
 	default y
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index effae53e9739..b35c9308e32b 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -455,6 +455,10 @@ alternative_else_nop_endif
  * 	Corrupts:	tmp1, tmp2
  */
 	.macro invalidate_icache_by_line start, end, tmp1, tmp2, fixup
+alternative_if ARM64_WORKAROUND_NXP_ERR050104
+	ic	ialluis
+	b	.Licache_done\@
+alternative_else_nop_endif
 	icache_line_size \tmp1, \tmp2
 	sub	\tmp2, \tmp1, #1
 	bic	\tmp2, \start, \tmp2
@@ -463,6 +467,7 @@ alternative_else_nop_endif
 	add	\tmp2, \tmp2, \tmp1
 	cmp	\tmp2, \end
 	b.lo	.Licache_op\@
+.Licache_done\@:
 	dsb	ish
 	isb
 
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 76350b38f0d7..0c3bfbe99aad 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -66,6 +66,8 @@ cpucap_is_possible(const unsigned int cap)
 		return IS_ENABLED(CONFIG_ARM64_ERRATUM_3194386);
 	case ARM64_WORKAROUND_4193714:
 		return IS_ENABLED(CONFIG_ARM64_ERRATUM_4193714);
+	case ARM64_WORKAROUND_NXP_ERR050104:
+		return IS_ENABLED(CONFIG_NXP_IMX8QM_ERRATUM_ERR050104);
 	case ARM64_MPAM:
 		/*
 		 * KVM MPAM support doesn't rely on the host kernel supporting MPAM.
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 14a78ac0f800..a05520d3a013 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -378,9 +378,13 @@ static inline void flush_tlb_mm(struct mm_struct *mm)
 	unsigned long asid;
 
 	dsb(ishst);
-	asid = __TLBI_VADDR(0, ASID(mm));
-	__tlbi(aside1is, asid);
-	__tlbi_user(aside1is, asid);
+	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104)) {
+		__tlbi(vmalle1is);
+	} else {
+		asid = __TLBI_VADDR(0, ASID(mm));
+		__tlbi(aside1is, asid);
+		__tlbi_user(aside1is, asid);
+	}
 	__tlbi_sync_s1ish(mm);
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
 }
@@ -580,23 +584,28 @@ static __always_inline void __do_flush_tlb_range(struct vm_area_struct *vma,
 
 	asid = ASID(mm);
 
-	switch (flags & (TLBF_NOWALKCACHE | TLBF_NOBROADCAST)) {
-	case TLBF_NONE:
-		__flush_s1_tlb_range_op(vae1is, start, pages, stride,
-					asid, tlb_level);
-		break;
-	case TLBF_NOWALKCACHE:
-		__flush_s1_tlb_range_op(vale1is, start, pages, stride,
-					asid, tlb_level);
-		break;
-	case TLBF_NOBROADCAST:
-		/* Combination unused */
-		BUG();
-		break;
-	case TLBF_NOWALKCACHE | TLBF_NOBROADCAST:
-		__flush_s1_tlb_range_op(vale1, start, pages, stride,
-					asid, tlb_level);
-		break;
+	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104) &&
+	    !(flags & TLBF_NOBROADCAST)) {
+		__tlbi(vmalle1is);
+	} else {
+		switch (flags & (TLBF_NOWALKCACHE | TLBF_NOBROADCAST)) {
+		case TLBF_NONE:
+			__flush_s1_tlb_range_op(vae1is, start, pages, stride,
+						asid, tlb_level);
+			break;
+		case TLBF_NOWALKCACHE:
+			__flush_s1_tlb_range_op(vale1is, start, pages, stride,
+						asid, tlb_level);
+			break;
+		case TLBF_NOBROADCAST:
+			/* Combination unused */
+			BUG();
+			break;
+		case TLBF_NOWALKCACHE | TLBF_NOBROADCAST:
+			__flush_s1_tlb_range_op(vale1, start, pages, stride,
+						asid, tlb_level);
+			break;
+		}
 	}
 
 	if (!(flags & TLBF_NONOTIFY))
@@ -657,7 +666,8 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
 	end = round_up(end, stride);
 	pages = (end - start) >> PAGE_SHIFT;
 
-	if (__flush_tlb_range_limit_excess(pages, stride)) {
+	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104) ||
+	    __flush_tlb_range_limit_excess(pages, stride)) {
 		flush_tlb_all();
 		return;
 	}
@@ -675,10 +685,14 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
  */
 static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
 {
-	unsigned long addr = __TLBI_VADDR(kaddr, 0);
-
 	dsb(ishst);
-	__tlbi(vaae1is, addr);
+	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104)) {
+		__tlbi(vmalle1is);
+	} else {
+		unsigned long addr = __TLBI_VADDR(kaddr, 0);
+
+		__tlbi(vaae1is, addr);
+	}
 	__tlbi_sync_s1ish_kernel();
 	isb();
 }
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 5db8f0619e4b..1866eeec27ca 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -6,6 +6,7 @@
  */
 
 #include <linux/arm-smccc.h>
+#include <linux/of.h>
 #include <linux/types.h>
 #include <linux/cpu.h>
 #include <asm/cpu.h>
@@ -200,6 +201,16 @@ cpu_enable_cache_maint_trap(const struct arm64_cpu_capabilities *__unused)
 	sysreg_clear_set(sctlr_el1, SCTLR_EL1_UCI, 0);
 }
 
+#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
+static bool
+is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	WARN_ON(preemptible());
+
+	return of_machine_is_compatible("fsl,imx8qm");
+}
+#endif
+
 #define CAP_MIDR_RANGE(model, v_min, r_min, v_max, r_max)	\
 	.matches = is_affected_midr_range,			\
 	.midr_range = MIDR_RANGE(model, v_min, r_min, v_max, r_max)
@@ -1030,6 +1041,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
 		.matches = has_broken_gic_v3_seis,
 	},
+#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
+	{
+		.desc = "NXP erratum ERR050104",
+		.capability = ARM64_WORKAROUND_NXP_ERR050104,
+		.type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE,
+		.matches = is_imx8qm_soc,
+		.cpu_enable = cpu_enable_cache_maint_trap,
+	},
+#endif
 	{
 	}
 };
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..d6391ab20db3 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -586,6 +586,8 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
 		break;
 	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
 		__user_cache_maint("ic ivau", address, ret);
+		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret)
+			asm volatile("ic ialluis");
 		break;
 	default:
 		force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 9a6c72a18672..734ccd7ec056 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -3032,6 +3032,11 @@ static __init int kvm_arm_init(void)
 
 	in_hyp_mode = is_kernel_in_hyp_mode();
 
+	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
+		kvm_info("KVM not supported on NXP iMX8QM (ERR050104)\n");
+		return -ENODEV;
+	}
+
 	if (cpus_have_final_cap(ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) ||
 	    cpus_have_final_cap(ARM64_WORKAROUND_1508412))
 		kvm_info("Guests without required CPU erratum workarounds can deadlock system!\n" \
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index fbf27e887e71..fb3fe9736cc6 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -123,6 +123,7 @@ WORKAROUND_DEVICE_LOAD_ACQUIRE
 WORKAROUND_DISABLE_CNP
 WORKAROUND_GICv3_BROKEN_SEIS
 WORKAROUND_NVIDIA_OLYMPUS_1027
+WORKAROUND_NXP_ERR050104
 WORKAROUND_PMUV3_IMPDEF_TRAPS
 WORKAROUND_QCOM_FALKOR_E1003
 WORKAROUND_QCOM_ORYON_CNTVOFF

---
base-commit: 9eebf259d5352b87080d67758f483583d9e763d7
change-id: 20260729-imx8qm-cache-coherency-3129d5da3de1

Best regards,
--  
Peng Fan <peng.fan@nxp.com>

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing Context Synchronization Event (CSE) after writing to `SCTLR_EL1`.
--

--- Patch [1]: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue ---
commit 174c9667365c79e4a6011c183eeb84b1ff94a880
Author: Peng Fan <peng.fan@nxp.com>

arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue

This commit adds a workaround for the NXP i.MX8QuadMax SoC cache coherency
issue (ERR050104). It upgrades targeted TLBI and IC IVAU operations to
broadcast-all variants via static keys and alternatives, and disables KVM
support for the affected hardware.

Link: https://lore.kernel.org/all/20230420112952.28340-1-iivanov@suse.de/

> diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
> index 5db8f0619e4bc..1866eeec27cab 100644
> --- a/arch/arm64/kernel/cpu_errata.c
> +++ b/arch/arm64/kernel/cpu_errata.c

[ ... ]

> @@ -200,6 +201,16 @@ cpu_enable_cache_maint_trap(const struct arm64_cpu_capabilities *__unused)
>  	sysreg_clear_set(sctlr_el1, SCTLR_EL1_UCI, 0);

[Severity: High]
This is a pre-existing issue, but does this write to SCTLR_EL1 require an
immediate isb() barrier before returning?

Writing to a control-plane system register like SCTLR_EL1 in
cpu_enable_cache_maint_trap() without an immediate Context Synchronization
Event (CSE) violates architectural invariants. Returning without an isb()
leaves the CPU pipeline in an undefined state, which could lead to spurious
exceptions or boot-time hangs on affected platforms.

>  }

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260730-imx8qm-cache-coherency-v2-1-5651aeb1af16@nxp.com?part=1

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 cache coherency issue

[Marc Zyngier]

On Thu, 30 Jul 2026 14:53:02 +0100,
"Peng Fan (OSS)" <peng.fan@oss.nxp.com> wrote:
> 
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 9a6c72a18672..734ccd7ec056 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -3032,6 +3032,11 @@ static __init int kvm_arm_init(void)
>  
>  	in_hyp_mode = is_kernel_in_hyp_mode();
>  
> +	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> +		kvm_info("KVM not supported on NXP iMX8QM (ERR050104)\n");
> +		return -ENODEV;
> +	}
> +
>  	if (cpus_have_final_cap(ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) ||
>  	    cpus_have_final_cap(ARM64_WORKAROUND_1508412))
>  		kvm_info("Guests without required CPU erratum workarounds can deadlock system!\n" \

I really dislike this sort of hacks in the KVM code.

Instead, please find a way to set 'kvm_mode' to KVM_MODE_NONE from
your erratum handler, and KVM will gracefully disable itself, as if
that was passed on the command line.

It shouldn't be too hard to do.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

[PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

According to NXP errata document[1] i.MX8QuadMax SoC suffers from
serious cache coherence issue. It was also mentioned in initial
support[2] for imx8qm mek machine.

I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
for this issue with the hope to reduce effect of this fix on unaffected
platforms.

Unfortunately I was unable to find a way to identify SoC ID using
registers. Boot CPU MIDR_EL1 is equal to 0x410fd034, AIDR_EL1 is
equal to 0. So I fallback to using devicetree compatible strings for this.
And because we can not guarantee that VMs on top of KVM will get
correct devicetree KVM is disabled.

Also make sure that SMMU "Broadcast TLB Maintenance" is not used even
SMMU device build in this SoC supports it.

I know this fix is a suboptimal solution for affected machines, but I
haven't been able to come up with a less intrusive fix.  And I hope once
TLB caches are invalidated any immediate attempt to invalidate them again
will be close to NOP operation (flush_tlb_kernel_range())

I have run few simple benchmarks and perf tests on affected and unaffected
machines and I was not able see any obvious issues. iMX8QM "performance"
was nearly doubled with 2 A72 bringed online.

Following is excerpt from NXP IMX8_1N94W "Mask Set Errata" document
Rev. 5, 3/2023. Just in case it gets lost somehow.

--
"ERR050104: Arm/A53: Cache coherency issue"

Description

Some maintenance operations exchanged between the A53 and A72
core clusters, involving some Translation Look-aside Buffer
Invalidate (TLBI) and Instruction Cache (IC) instructions can
be corrupted. The upper bits, above bit-35, of ARADDR and ACADDR
buses within in Arm A53 sub-system have been incorrectly connected.
Therefore ARADDR and ACADDR address bits above bit-35 should not
be used.

Workaround

The following software instructions are required to be downgraded
to TLBI VMALLE1IS:  TLBI ASIDE1, TLBI ASIDE1IS, TLBI VAAE1,
TLBI VAAE1IS, TLBI VAALE1, TLBI VAALE1IS, TLBI VAE1, TLBI VAE1IS,
TLBI VALE1, TLBI VALE1IS

The following software instructions are required to be downgraded
to TLBI VMALLS12E1IS: TLBI IPAS2E1IS, TLBI IPAS2LE1IS

The following software instructions are required to be downgraded
to TLBI ALLE2IS: TLBI VAE2IS, TLBI VALE2IS.

The following software instructions are required to be downgraded
to TLBI ALLE3IS: TLBI VAE3IS, TLBI VALE3IS.

The following software instructions are required to be downgraded
to TLBI VMALLE1IS when the Force Broadcast (FB) bit [9] of the
Hypervisor Configuration Register (HCR_EL2) is set:
TLBI ASIDE1, TLBI VAAE1, TLBI VAALE1, TLBI VAE1, TLBI VALE1

The following software instruction is required to be downgraded
to IC IALLUIS: IC IVAU, Xt

Specifically for the IC IVAU, Xt downgrade, setting SCTLR_EL1.UCI
to 0 will disable EL0 access to this instruction. Any attempt to
execute from EL0 will generate an EL1 trap, where the downgrade to
IC ALLUIS can be implemented.
--

[1] https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf
[2] commit 307fd14d4b14c ("arm64: dts: imx: add imx8qm mek support")
[3] https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/arch/arm64/include/asm/tlbflush.h#L19

Signed-off-by: Ivan T. Ivanov <iivanov@suse.de>
---

Changes since v1:
https://lore.kernel.org/linux-arm-kernel/20230412125506.21634-1-iivanov@suse.de/

 - Disable KVM on affected SoC add a note in commit message why this is done.
 - Updated Kconfig help message and made option enabled by default, like
   the rest of the workarounds.
 - Removed unnecessary DSB ISH + ISB instructions from the ALTERNATIVES.
 - Reworked handling of user-space IC IVAU.
 - Adjusted position of workaround definition in cpucaps file
 - Make sure that "Broadcast TLB Maintenance" is not used even if SMMU device
   build in this SoC supports it. And note it in commit message.

 Documentation/arm64/silicon-errata.rst |  2 ++
 arch/arm64/Kconfig                     | 12 ++++++++++++
 arch/arm64/include/asm/tlbflush.h      |  6 +++++-
 arch/arm64/kernel/cpu_errata.c         | 18 ++++++++++++++++++
 arch/arm64/kernel/traps.c              |  5 +++++
 arch/arm64/kvm/arm.c                   |  5 +++++
 arch/arm64/tools/cpucaps               |  1 +
 drivers/iommu/arm/arm-smmu/arm-smmu.c  |  4 ++++
 drivers/iommu/arm/arm-smmu/arm-smmu.h  |  1 +
 9 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst
index ec5f889d7681..fce231797184 100644
--- a/Documentation/arm64/silicon-errata.rst
+++ b/Documentation/arm64/silicon-errata.rst
@@ -175,6 +175,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | Freescale/NXP  | LS2080A/LS1043A | A-008585        | FSL_ERRATUM_A008585         |
 +----------------+-----------------+-----------------+-----------------------------+
+| Freescale/NXP  | i.MX 8QuadMax   | ERR050104       | NXP_IMX8QM_ERRATUM_ERR050104|
++----------------+-----------------+-----------------+-----------------------------+
 +----------------+-----------------+-----------------+-----------------------------+
 | Hisilicon      | Hip0{5,6,7}     | #161010101      | HISILICON_ERRATUM_161010101 |
 +----------------+-----------------+-----------------+-----------------------------+
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1023e896d46b..a5fe6ffb8150 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1159,6 +1159,18 @@ config SOCIONEXT_SYNQUACER_PREITS
 
 	  If unsure, say Y.
 
+config NXP_IMX8QM_ERRATUM_ERR050104
+	bool "NXP iMX8QM ERR050104: broken cache/tlb invalidation"
+	default y
+	help
+	  On iMX8QM, addresses above bit 35 are not broadcast correctly for
+	  TLBI or IC operations, making TLBI and IC unreliable.
+
+	  Work around this erratum by using TLBI *ALL*IS and IC IALLUIS
+	  operations. EL0 use of IC IVAU is trapped and upgraded to IC IALLUIS.
+
+	  If unsure, say Y.
+
 endmenu # "ARM errata workarounds via the alternatives framework"
 
 choice
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 412a3b9a3c25..e3bad2298ea5 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -37,7 +37,11 @@
 			    : : )
 
 #define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE			       \
-			       "tlbi " #op ", %0\n"			       \
+		   ALTERNATIVE("tlbi " #op ", %0\n",			       \
+			       "tlbi vmalle1is\n",			       \
+			       ARM64_WORKAROUND_NXP_ERR050104)		       \
+			    : : "r" (arg));				       \
+			  asm (ARM64_ASM_PREAMBLE			       \
 		   ALTERNATIVE("nop\n			nop",		       \
 			       "dsb ish\n		tlbi " #op ", %0",     \
 			       ARM64_WORKAROUND_REPEAT_TLBI,		       \
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 307faa2b4395..b0647b64dbb8 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -8,6 +8,7 @@
 #include <linux/arm-smccc.h>
 #include <linux/types.h>
 #include <linux/cpu.h>
+#include <linux/of.h>
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpufeature.h>
@@ -55,6 +56,14 @@ is_kryo_midr(const struct arm64_cpu_capabilities *entry, int scope)
 	return model == entry->midr_range.model;
 }
 
+static bool __maybe_unused
+is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	WARN_ON(preemptible());
+
+	return of_machine_is_compatible("fsl,imx8qm");
+}
+
 static bool
 has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry,
 			  int scope)
@@ -729,6 +738,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)),
 		.cpu_enable = cpu_clear_bf16_from_user_emulation,
 	},
+#endif
+#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
+	{
+		.desc = "NXP erratum ERR050104",
+		.capability = ARM64_WORKAROUND_NXP_ERR050104,
+		.type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE,
+		.matches = is_imx8qm_soc,
+		.cpu_enable = cpu_enable_cache_maint_trap,
+	},
 #endif
 	{
 	}
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 4a79ba100799..265b6334291b 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
 		__user_cache_maint("dc civac", address, ret);
 		break;
 	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
+		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
+			asm volatile("ic ialluis");
+			ret = 0;
+			break;
+		}
 		__user_cache_maint("ic ivau", address, ret);
 		break;
 	default:
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 4b2e16e696a8..5066332302d2 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -2239,6 +2239,11 @@ static __init int kvm_arm_init(void)
 		return -ENODEV;
 	}
 
+	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
+		kvm_info("KVM is not supported on this system\n");
+		return -ENODEV;
+	}
+
 	err = kvm_sys_reg_table_init();
 	if (err) {
 		kvm_info("Error initializing system register tables");
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 37b1340e9646..c1de9235f922 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -87,6 +87,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM
 WORKAROUND_CLEAN_CACHE
 WORKAROUND_DEVICE_LOAD_ACQUIRE
 WORKAROUND_NVIDIA_CARMEL_CNP
+WORKAROUND_NXP_ERR050104
 WORKAROUND_QCOM_FALKOR_E1003
 WORKAROUND_REPEAT_TLBI
 WORKAROUND_SPECULATIVE_AT
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
index 2ff7a72cf377..29bbb16bae6e 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
@@ -1680,6 +1680,10 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
 	/* ID0 */
 	id = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_ID0);
 
+	/* Do not use Broadcast TLB Maintenance on affected platforms */
+	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104))
+		id &= ~ARM_SMMU_ID0_BTM;
+
 	/* Restrict available stages based on module parameter */
 	if (force_stage == 1)
 		id &= ~(ARM_SMMU_ID0_S2TS | ARM_SMMU_ID0_NTS);
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
index 703fd5817ec1..1589acfdbed9 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
@@ -52,6 +52,7 @@
 #define ARM_SMMU_ID0_PTFS_NO_AARCH32S	BIT(24)
 #define ARM_SMMU_ID0_NUMIRPT		GENMASK(23, 16)
 #define ARM_SMMU_ID0_CTTW		BIT(14)
+#define ARM_SMMU_ID0_BTM		BIT(13)
 #define ARM_SMMU_ID0_NUMSIDB		GENMASK(12, 9)
 #define ARM_SMMU_ID0_EXIDS		BIT(8)
 #define ARM_SMMU_ID0_NUMSMRG		GENMASK(7, 0)
-- 
2.35.3


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

Hi Catalin, Will,

Is there something more that I could for this patch or
it could be merged as is?

Regards,
Ivan


On 04-20 14:29, Ivan T. Ivanov wrote:
> Message-Id: <20230420112952.28340-1-iivanov@suse.de>
> 
> According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> serious cache coherence issue. It was also mentioned in initial
> support[2] for imx8qm mek machine.
> 
> I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
> for this issue with the hope to reduce effect of this fix on unaffected
> platforms.
> 
> Unfortunately I was unable to find a way to identify SoC ID using
> registers. Boot CPU MIDR_EL1 is equal to 0x410fd034, AIDR_EL1 is
> equal to 0. So I fallback to using devicetree compatible strings for this.
> And because we can not guarantee that VMs on top of KVM will get
> correct devicetree KVM is disabled.
> 
> Also make sure that SMMU "Broadcast TLB Maintenance" is not used even
> SMMU device build in this SoC supports it.
> 
> I know this fix is a suboptimal solution for affected machines, but I
> haven't been able to come up with a less intrusive fix.  And I hope once
> TLB caches are invalidated any immediate attempt to invalidate them again
> will be close to NOP operation (flush_tlb_kernel_range())
> 
> I have run few simple benchmarks and perf tests on affected and unaffected
> machines and I was not able see any obvious issues. iMX8QM "performance"
> was nearly doubled with 2 A72 bringed online.
> 
> Following is excerpt from NXP IMX8_1N94W "Mask Set Errata" document
> Rev. 5, 3/2023. Just in case it gets lost somehow.
> 
> --
> "ERR050104: Arm/A53: Cache coherency issue"
> 
> Description
> 
> Some maintenance operations exchanged between the A53 and A72
> core clusters, involving some Translation Look-aside Buffer
> Invalidate (TLBI) and Instruction Cache (IC) instructions can
> be corrupted. The upper bits, above bit-35, of ARADDR and ACADDR
> buses within in Arm A53 sub-system have been incorrectly connected.
> Therefore ARADDR and ACADDR address bits above bit-35 should not
> be used.
> 
> Workaround
> 
> The following software instructions are required to be downgraded
> to TLBI VMALLE1IS:  TLBI ASIDE1, TLBI ASIDE1IS, TLBI VAAE1,
> TLBI VAAE1IS, TLBI VAALE1, TLBI VAALE1IS, TLBI VAE1, TLBI VAE1IS,
> TLBI VALE1, TLBI VALE1IS
> 
> The following software instructions are required to be downgraded
> to TLBI VMALLS12E1IS: TLBI IPAS2E1IS, TLBI IPAS2LE1IS
> 
> The following software instructions are required to be downgraded
> to TLBI ALLE2IS: TLBI VAE2IS, TLBI VALE2IS.
> 
> The following software instructions are required to be downgraded
> to TLBI ALLE3IS: TLBI VAE3IS, TLBI VALE3IS.
> 
> The following software instructions are required to be downgraded
> to TLBI VMALLE1IS when the Force Broadcast (FB) bit [9] of the
> Hypervisor Configuration Register (HCR_EL2) is set:
> TLBI ASIDE1, TLBI VAAE1, TLBI VAALE1, TLBI VAE1, TLBI VALE1
> 
> The following software instruction is required to be downgraded
> to IC IALLUIS: IC IVAU, Xt
> 
> Specifically for the IC IVAU, Xt downgrade, setting SCTLR_EL1.UCI
> to 0 will disable EL0 access to this instruction. Any attempt to
> execute from EL0 will generate an EL1 trap, where the downgrade to
> IC ALLUIS can be implemented.
> --
> 
> [1] https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf
> [2] commit 307fd14d4b14c ("arm64: dts: imx: add imx8qm mek support")
> [3] https://github.com/nxp-imx/linux-imx/blob/lf-6.1.y/arch/arm64/include/asm/tlbflush.h#L19
> 
> Signed-off-by: Ivan T. Ivanov <iivanov@suse.de>
> ---
> 
> Changes since v1:
> https://lore.kernel.org/linux-arm-kernel/20230412125506.21634-1-iivanov@suse.de/
> 
>  - Disable KVM on affected SoC add a note in commit message why this is done.
>  - Updated Kconfig help message and made option enabled by default, like
>    the rest of the workarounds.
>  - Removed unnecessary DSB ISH + ISB instructions from the ALTERNATIVES.
>  - Reworked handling of user-space IC IVAU.
>  - Adjusted position of workaround definition in cpucaps file
>  - Make sure that "Broadcast TLB Maintenance" is not used even if SMMU device
>    build in this SoC supports it. And note it in commit message.
> 
>  Documentation/arm64/silicon-errata.rst |  2 ++
>  arch/arm64/Kconfig                     | 12 ++++++++++++
>  arch/arm64/include/asm/tlbflush.h      |  6 +++++-
>  arch/arm64/kernel/cpu_errata.c         | 18 ++++++++++++++++++
>  arch/arm64/kernel/traps.c              |  5 +++++
>  arch/arm64/kvm/arm.c                   |  5 +++++
>  arch/arm64/tools/cpucaps               |  1 +
>  drivers/iommu/arm/arm-smmu/arm-smmu.c  |  4 ++++
>  drivers/iommu/arm/arm-smmu/arm-smmu.h  |  1 +
>  9 files changed, 53 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst
> index ec5f889d7681..fce231797184 100644
> --- a/Documentation/arm64/silicon-errata.rst
> +++ b/Documentation/arm64/silicon-errata.rst
> @@ -175,6 +175,8 @@ stable kernels.
>  +----------------+-----------------+-----------------+-----------------------------+
>  | Freescale/NXP  | LS2080A/LS1043A | A-008585        | FSL_ERRATUM_A008585         |
>  +----------------+-----------------+-----------------+-----------------------------+
> +| Freescale/NXP  | i.MX 8QuadMax   | ERR050104       | NXP_IMX8QM_ERRATUM_ERR050104|
> ++----------------+-----------------+-----------------+-----------------------------+
>  +----------------+-----------------+-----------------+-----------------------------+
>  | Hisilicon      | Hip0{5,6,7}     | #161010101      | HISILICON_ERRATUM_161010101 |
>  +----------------+-----------------+-----------------+-----------------------------+
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 1023e896d46b..a5fe6ffb8150 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1159,6 +1159,18 @@ config SOCIONEXT_SYNQUACER_PREITS
>  
>  	  If unsure, say Y.
>  
> +config NXP_IMX8QM_ERRATUM_ERR050104
> +	bool "NXP iMX8QM ERR050104: broken cache/tlb invalidation"
> +	default y
> +	help
> +	  On iMX8QM, addresses above bit 35 are not broadcast correctly for
> +	  TLBI or IC operations, making TLBI and IC unreliable.
> +
> +	  Work around this erratum by using TLBI *ALL*IS and IC IALLUIS
> +	  operations. EL0 use of IC IVAU is trapped and upgraded to IC IALLUIS.
> +
> +	  If unsure, say Y.
> +
>  endmenu # "ARM errata workarounds via the alternatives framework"
>  
>  choice
> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> index 412a3b9a3c25..e3bad2298ea5 100644
> --- a/arch/arm64/include/asm/tlbflush.h
> +++ b/arch/arm64/include/asm/tlbflush.h
> @@ -37,7 +37,11 @@
>  			    : : )
>  
>  #define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE			       \
> -			       "tlbi " #op ", %0\n"			       \
> +		   ALTERNATIVE("tlbi " #op ", %0\n",			       \
> +			       "tlbi vmalle1is\n",			       \
> +			       ARM64_WORKAROUND_NXP_ERR050104)		       \
> +			    : : "r" (arg));				       \
> +			  asm (ARM64_ASM_PREAMBLE			       \
>  		   ALTERNATIVE("nop\n			nop",		       \
>  			       "dsb ish\n		tlbi " #op ", %0",     \
>  			       ARM64_WORKAROUND_REPEAT_TLBI,		       \
> diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
> index 307faa2b4395..b0647b64dbb8 100644
> --- a/arch/arm64/kernel/cpu_errata.c
> +++ b/arch/arm64/kernel/cpu_errata.c
> @@ -8,6 +8,7 @@
>  #include <linux/arm-smccc.h>
>  #include <linux/types.h>
>  #include <linux/cpu.h>
> +#include <linux/of.h>
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/cpufeature.h>
> @@ -55,6 +56,14 @@ is_kryo_midr(const struct arm64_cpu_capabilities *entry, int scope)
>  	return model == entry->midr_range.model;
>  }
>  
> +static bool __maybe_unused
> +is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	WARN_ON(preemptible());
> +
> +	return of_machine_is_compatible("fsl,imx8qm");
> +}
> +
>  static bool
>  has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry,
>  			  int scope)
> @@ -729,6 +738,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
>  		MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)),
>  		.cpu_enable = cpu_clear_bf16_from_user_emulation,
>  	},
> +#endif
> +#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
> +	{
> +		.desc = "NXP erratum ERR050104",
> +		.capability = ARM64_WORKAROUND_NXP_ERR050104,
> +		.type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE,
> +		.matches = is_imx8qm_soc,
> +		.cpu_enable = cpu_enable_cache_maint_trap,
> +	},
>  #endif
>  	{
>  	}
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 4a79ba100799..265b6334291b 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
>  		__user_cache_maint("dc civac", address, ret);
>  		break;
>  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> +			asm volatile("ic ialluis");
> +			ret = 0;
> +			break;
> +		}
>  		__user_cache_maint("ic ivau", address, ret);
>  		break;
>  	default:
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 4b2e16e696a8..5066332302d2 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -2239,6 +2239,11 @@ static __init int kvm_arm_init(void)
>  		return -ENODEV;
>  	}
>  
> +	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> +		kvm_info("KVM is not supported on this system\n");
> +		return -ENODEV;
> +	}
> +
>  	err = kvm_sys_reg_table_init();
>  	if (err) {
>  		kvm_info("Error initializing system register tables");
> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index 37b1340e9646..c1de9235f922 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -87,6 +87,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM
>  WORKAROUND_CLEAN_CACHE
>  WORKAROUND_DEVICE_LOAD_ACQUIRE
>  WORKAROUND_NVIDIA_CARMEL_CNP
> +WORKAROUND_NXP_ERR050104
>  WORKAROUND_QCOM_FALKOR_E1003
>  WORKAROUND_REPEAT_TLBI
>  WORKAROUND_SPECULATIVE_AT
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
> index 2ff7a72cf377..29bbb16bae6e 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
> @@ -1680,6 +1680,10 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
>  	/* ID0 */
>  	id = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_ID0);
>  
> +	/* Do not use Broadcast TLB Maintenance on affected platforms */
> +	if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104))
> +		id &= ~ARM_SMMU_ID0_BTM;
> +
>  	/* Restrict available stages based on module parameter */
>  	if (force_stage == 1)
>  		id &= ~(ARM_SMMU_ID0_S2TS | ARM_SMMU_ID0_NTS);
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> index 703fd5817ec1..1589acfdbed9 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> @@ -52,6 +52,7 @@
>  #define ARM_SMMU_ID0_PTFS_NO_AARCH32S	BIT(24)
>  #define ARM_SMMU_ID0_NUMIRPT		GENMASK(23, 16)
>  #define ARM_SMMU_ID0_CTTW		BIT(14)
> +#define ARM_SMMU_ID0_BTM		BIT(13)
>  #define ARM_SMMU_ID0_NUMSIDB		GENMASK(12, 9)
>  #define ARM_SMMU_ID0_EXIDS		BIT(8)
>  #define ARM_SMMU_ID0_NUMSMRG		GENMASK(7, 0)
> -- 
> 2.35.3
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Will Deacon]

Hi Ivan,

Sorry for the delay in getting to this.

On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> serious cache coherence issue. It was also mentioned in initial
> support[2] for imx8qm mek machine.
> 
> I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
> for this issue with the hope to reduce effect of this fix on unaffected
> platforms.
> 
> Unfortunately I was unable to find a way to identify SoC ID using
> registers. Boot CPU MIDR_EL1 is equal to 0x410fd034, AIDR_EL1 is
> equal to 0. So I fallback to using devicetree compatible strings for this.
> And because we can not guarantee that VMs on top of KVM will get
> correct devicetree KVM is disabled.
> 
> Also make sure that SMMU "Broadcast TLB Maintenance" is not used even
> SMMU device build in this SoC supports it.
> 
> I know this fix is a suboptimal solution for affected machines, but I
> haven't been able to come up with a less intrusive fix.  And I hope once
> TLB caches are invalidated any immediate attempt to invalidate them again
> will be close to NOP operation (flush_tlb_kernel_range())

[...]

> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 1023e896d46b..a5fe6ffb8150 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1159,6 +1159,18 @@ config SOCIONEXT_SYNQUACER_PREITS
>  
>  	  If unsure, say Y.
>  
> +config NXP_IMX8QM_ERRATUM_ERR050104
> +	bool "NXP iMX8QM ERR050104: broken cache/tlb invalidation"
> +	default y
> +	help
> +	  On iMX8QM, addresses above bit 35 are not broadcast correctly for
> +	  TLBI or IC operations, making TLBI and IC unreliable.
> +
> +	  Work around this erratum by using TLBI *ALL*IS and IC IALLUIS
> +	  operations. EL0 use of IC IVAU is trapped and upgraded to IC IALLUIS.
> +
> +	  If unsure, say Y.
> +
>  endmenu # "ARM errata workarounds via the alternatives framework"
>  
>  choice
> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> index 412a3b9a3c25..e3bad2298ea5 100644
> --- a/arch/arm64/include/asm/tlbflush.h
> +++ b/arch/arm64/include/asm/tlbflush.h
> @@ -37,7 +37,11 @@
>  			    : : )
>  
>  #define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE			       \
> -			       "tlbi " #op ", %0\n"			       \
> +		   ALTERNATIVE("tlbi " #op ", %0\n",			       \
> +			       "tlbi vmalle1is\n",			       \
> +			       ARM64_WORKAROUND_NXP_ERR050104)		       \
> +			    : : "r" (arg));				       \
> +			  asm (ARM64_ASM_PREAMBLE			       \
>  		   ALTERNATIVE("nop\n			nop",		       \
>  			       "dsb ish\n		tlbi " #op ", %0",     \
>  			       ARM64_WORKAROUND_REPEAT_TLBI,		       \

I can see two problems with this:

  1. It doesn't interact properly with the ARM64_WORKAROUND_REPEAT_TLBI
     workaround

  2. You're nuking the entire TLB in the low-level helper, so things like
     flush_tlb_range() are going to over-invalidate excessively

Granted, you may not care about (1) for your SoC, but I would prefer not
to introduce artificial constraints on the workaround. I think both of
the issues above stem from plumbing this in too low in the stack.

Can you instead use a static key to redirect the higher-level control flow
to flush_tlb_mm() or flush_tlb_all() for user and kernel respectively? I'm
assuming the ASID _is_ carried on the interconnect?

> diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
> index 307faa2b4395..b0647b64dbb8 100644
> --- a/arch/arm64/kernel/cpu_errata.c
> +++ b/arch/arm64/kernel/cpu_errata.c
> @@ -8,6 +8,7 @@
>  #include <linux/arm-smccc.h>
>  #include <linux/types.h>
>  #include <linux/cpu.h>
> +#include <linux/of.h>
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/cpufeature.h>
> @@ -55,6 +56,14 @@ is_kryo_midr(const struct arm64_cpu_capabilities *entry, int scope)
>  	return model == entry->midr_range.model;
>  }
>  
> +static bool __maybe_unused
> +is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	WARN_ON(preemptible());
> +
> +	return of_machine_is_compatible("fsl,imx8qm");
> +}
> +
>  static bool
>  has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry,
>  			  int scope)
> @@ -729,6 +738,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
>  		MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)),
>  		.cpu_enable = cpu_clear_bf16_from_user_emulation,
>  	},
> +#endif
> +#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
> +	{
> +		.desc = "NXP erratum ERR050104",
> +		.capability = ARM64_WORKAROUND_NXP_ERR050104,
> +		.type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE,
> +		.matches = is_imx8qm_soc,
> +		.cpu_enable = cpu_enable_cache_maint_trap,
> +	},
>  #endif
>  	{
>  	}
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 4a79ba100799..265b6334291b 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
>  		__user_cache_maint("dc civac", address, ret);
>  		break;
>  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> +			asm volatile("ic ialluis");

Hmm, one oddity here is that you can pass a faulting address and not see
the fault. It looks like that's already IMP DEF, so it's probably ok, but
might be worth a comment.

> +			ret = 0;

'ret' is initialised to 0 already?

Finally, how come you don't need to upgrade I-cache invalidation by-VA
in the kernel? It looks like you're only handling operations trapped
from EL0.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

On 06-02 11:34, Will Deacon wrote:
> 
> Hi Ivan,
> 
> Sorry for the delay in getting to this.
> 
> On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> > According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> > serious cache coherence issue. It was also mentioned in initial
> > support[2] for imx8qm mek machine.
> > 
> > I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
> > for this issue with the hope to reduce effect of this fix on unaffected
> > platforms.
> > 
> > Unfortunately I was unable to find a way to identify SoC ID using
> > registers. Boot CPU MIDR_EL1 is equal to 0x410fd034, AIDR_EL1 is
> > equal to 0. So I fallback to using devicetree compatible strings for this.
> > And because we can not guarantee that VMs on top of KVM will get
> > correct devicetree KVM is disabled.
> > 
> > Also make sure that SMMU "Broadcast TLB Maintenance" is not used even
> > SMMU device build in this SoC supports it.
> > 
> > I know this fix is a suboptimal solution for affected machines, but I
> > haven't been able to come up with a less intrusive fix.  And I hope once
> > TLB caches are invalidated any immediate attempt to invalidate them again
> > will be close to NOP operation (flush_tlb_kernel_range())
> 
> [...]
> 
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index 1023e896d46b..a5fe6ffb8150 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -1159,6 +1159,18 @@ config SOCIONEXT_SYNQUACER_PREITS
> >  
> >  	  If unsure, say Y.
> >  
> > +config NXP_IMX8QM_ERRATUM_ERR050104
> > +	bool "NXP iMX8QM ERR050104: broken cache/tlb invalidation"
> > +	default y
> > +	help
> > +	  On iMX8QM, addresses above bit 35 are not broadcast correctly for
> > +	  TLBI or IC operations, making TLBI and IC unreliable.
> > +
> > +	  Work around this erratum by using TLBI *ALL*IS and IC IALLUIS
> > +	  operations. EL0 use of IC IVAU is trapped and upgraded to IC IALLUIS.
> > +
> > +	  If unsure, say Y.
> > +
> >  endmenu # "ARM errata workarounds via the alternatives framework"
> >  
> >  choice
> > diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> > index 412a3b9a3c25..e3bad2298ea5 100644
> > --- a/arch/arm64/include/asm/tlbflush.h
> > +++ b/arch/arm64/include/asm/tlbflush.h
> > @@ -37,7 +37,11 @@
> >  			    : : )
> >  
> >  #define __TLBI_1(op, arg) asm (ARM64_ASM_PREAMBLE			       \
> > -			       "tlbi " #op ", %0\n"			       \
> > +		   ALTERNATIVE("tlbi " #op ", %0\n",			       \
> > +			       "tlbi vmalle1is\n",			       \
> > +			       ARM64_WORKAROUND_NXP_ERR050104)		       \
> > +			    : : "r" (arg));				       \
> > +			  asm (ARM64_ASM_PREAMBLE			       \
> >  		   ALTERNATIVE("nop\n			nop",		       \
> >  			       "dsb ish\n		tlbi " #op ", %0",     \
> >  			       ARM64_WORKAROUND_REPEAT_TLBI,		       \
> 
> I can see two problems with this:
> 
>   1. It doesn't interact properly with the ARM64_WORKAROUND_REPEAT_TLBI
>      workaround

From what I see, they can not happen at the same time on the same SoC,
so this should not be an issue, but I could be missing something.

> 
>   2. You're nuking the entire TLB in the low-level helper, so things like
>      flush_tlb_range() are going to over-invalidate excessively

Right, this is noted in the commit message.

> 
> Granted, you may not care about (1) for your SoC, but I would prefer not
> to introduce artificial constraints on the workaround. I think both of
> the issues above stem from plumbing this in too low in the stack.
> 
> Can you instead use a static key to redirect the higher-level control flow
> to flush_tlb_mm() or flush_tlb_all() for user and kernel respectively?

This was the other option considered, but it was looking more intrusive.

> I'm
> assuming the ASID _is_ carried on the interconnect?

Looking at history of in tlbflush in the vendor tree [4] I think ASID
is not properly wired either [5]. But I am failing to see why this
should matter.

[4] https://github.com/nxp-imx/linux-imx/commits/lf-6.1.y/arch/arm64/include/asm/tlbflush.h
[5] https://github.com/nxp-imx/linux-imx/commit/db6c157fff2a39c378e41e63f78d6fba18578758

> 
> > diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
> > index 307faa2b4395..b0647b64dbb8 100644
> > --- a/arch/arm64/kernel/cpu_errata.c
> > +++ b/arch/arm64/kernel/cpu_errata.c
> > @@ -8,6 +8,7 @@
> >  #include <linux/arm-smccc.h>
> >  #include <linux/types.h>
> >  #include <linux/cpu.h>
> > +#include <linux/of.h>
> >  #include <asm/cpu.h>
> >  #include <asm/cputype.h>
> >  #include <asm/cpufeature.h>
> > @@ -55,6 +56,14 @@ is_kryo_midr(const struct arm64_cpu_capabilities *entry, int scope)
> >  	return model == entry->midr_range.model;
> >  }
> >  
> > +static bool __maybe_unused
> > +is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope)
> > +{
> > +	WARN_ON(preemptible());
> > +
> > +	return of_machine_is_compatible("fsl,imx8qm");
> > +}
> > +
> >  static bool
> >  has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry,
> >  			  int scope)
> > @@ -729,6 +738,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
> >  		MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)),
> >  		.cpu_enable = cpu_clear_bf16_from_user_emulation,
> >  	},
> > +#endif
> > +#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104
> > +	{
> > +		.desc = "NXP erratum ERR050104",
> > +		.capability = ARM64_WORKAROUND_NXP_ERR050104,
> > +		.type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE,
> > +		.matches = is_imx8qm_soc,
> > +		.cpu_enable = cpu_enable_cache_maint_trap,
> > +	},
> >  #endif
> >  	{
> >  	}
> > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > index 4a79ba100799..265b6334291b 100644
> > --- a/arch/arm64/kernel/traps.c
> > +++ b/arch/arm64/kernel/traps.c
> > @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
> >  		__user_cache_maint("dc civac", address, ret);
> >  		break;
> >  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> > +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> > +			asm volatile("ic ialluis");
> 
> Hmm, one oddity here is that you can pass a faulting address and not see
> the fault. It looks like that's already IMP DEF, so it's probably ok, but
> might be worth a comment.

I am not sure what should be expected behavior, but I could
add comment, sure.

> 
> > +			ret = 0;
> 
> 'ret' is initialised to 0 already?

Oh, yes.

> 
> Finally, how come you don't need to upgrade I-cache invalidation by-VA
> in the kernel? It looks like you're only handling operations trapped
> from EL0.

Hm, I was thinking that __tlbi() is taking care for this or you mean 
something else, like locations in assembler.h?

Regards,
Ivan


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Mark Rutland]

On Thu, Jun 08, 2023 at 04:39:29PM +0300, Ivan T. Ivanov wrote:
> On 06-02 11:34, Will Deacon wrote:
> > On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> > > According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> > > serious cache coherence issue. It was also mentioned in initial
> > > support[2] for imx8qm mek machine.
> > > 
> > > I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
> > > for this issue with the hope to reduce effect of this fix on unaffected
> > > platforms.

> > > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > > index 4a79ba100799..265b6334291b 100644
> > > --- a/arch/arm64/kernel/traps.c
> > > +++ b/arch/arm64/kernel/traps.c
> > > @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
> > >  		__user_cache_maint("dc civac", address, ret);
> > >  		break;
> > >  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> > > +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> > > +			asm volatile("ic ialluis");
> > 
> > Hmm, one oddity here is that you can pass a faulting address and not see
> > the fault. It looks like that's already IMP DEF, so it's probably ok, but
> > might be worth a comment.
> 
> I am not sure what should be expected behavior, but I could
> add comment, sure.

Another option is to make this:

	 case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
	 	__user_cache_maint("ic ivau", address, ret)
		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret)
			asm volatile("ic ialluis");
		break;

Which'll ensure that if the regular IC IVAU faults we'll handle that, and if
not we'll do the IC IALLUIS.

I think that looks a bit cleaner, too.

> > Finally, how come you don't need to upgrade I-cache invalidation by-VA
> > in the kernel? It looks like you're only handling operations trapped
> > from EL0.
> 
> Hm, I was thinking that __tlbi() is taking care for this or you mean 
> something else, like locations in assembler.h?

The __tlbi macro handles only TLBI instructions.

The trap handler above *only* handles IC instructions trapped from userspace;
we have IC IVAU instructions elsewhere in the kernel (e.g.
arch/arm64/mm/cache.S).

Thanks,
Mark.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

On 06-08 15:16, Mark Rutland wrote:
> 
> On Thu, Jun 08, 2023 at 04:39:29PM +0300, Ivan T. Ivanov wrote:
> > On 06-02 11:34, Will Deacon wrote:
> > > On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> > > > According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> > > > serious cache coherence issue. It was also mentioned in initial
> > > > support[2] for imx8qm mek machine.
> > > > 
> > > > I chose to use an ALTERNATIVE() framework, instead downstream solution[3],
> > > > for this issue with the hope to reduce effect of this fix on unaffected
> > > > platforms.
> 
> > > > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > > > index 4a79ba100799..265b6334291b 100644
> > > > --- a/arch/arm64/kernel/traps.c
> > > > +++ b/arch/arm64/kernel/traps.c
> > > > @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
> > > >  		__user_cache_maint("dc civac", address, ret);
> > > >  		break;
> > > >  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> > > > +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> > > > +			asm volatile("ic ialluis");
> > > 
> > > Hmm, one oddity here is that you can pass a faulting address and not see
> > > the fault. It looks like that's already IMP DEF, so it's probably ok, but
> > > might be worth a comment.
> > 
> > I am not sure what should be expected behavior, but I could
> > add comment, sure.
> 
> Another option is to make this:
> 
> 	 case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
> 	 	__user_cache_maint("ic ivau", address, ret)
> 		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret)
> 			asm volatile("ic ialluis");
> 		break;
> 
> Which'll ensure that if the regular IC IVAU faults we'll handle that, and if
> not we'll do the IC IALLUIS.
> 
> I think that looks a bit cleaner, too.

What I am afraid is that "ic ivau address" will do cache invalidation of a 
random address, because of wrong address wiring. So the end result will not
be what should be expected.

> 
> > > Finally, how come you don't need to upgrade I-cache invalidation by-VA
> > > in the kernel? It looks like you're only handling operations trapped
> > > from EL0.
> > 
> > Hm, I was thinking that __tlbi() is taking care for this or you mean 
> > something else, like locations in assembler.h?
> 
> The __tlbi macro handles only TLBI instructions.
> 
> The trap handler above *only* handles IC instructions trapped from userspace;

Yep, I get this.

> we have IC IVAU instructions elsewhere in the kernel (e.g.
> arch/arm64/mm/cache.S).

But I have missed this one :-)

I think that this is working because these are used only 
in operations witch work up to the Point of Unification,
thus not messing up with caches of the rest of PE.

Regards,
Ivan


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Mark Rutland]

On Thu, Jun 08, 2023 at 06:05:54PM +0300, Ivan T. Ivanov wrote:
> On 06-08 15:16, Mark Rutland wrote:
> > On Thu, Jun 08, 2023 at 04:39:29PM +0300, Ivan T. Ivanov wrote:
> > > On 06-02 11:34, Will Deacon wrote:
> > > > On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:

> > > > > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > > > > index 4a79ba100799..265b6334291b 100644
> > > > > --- a/arch/arm64/kernel/traps.c
> > > > > +++ b/arch/arm64/kernel/traps.c
> > > > > @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
> > > > >  		__user_cache_maint("dc civac", address, ret);
> > > > >  		break;
> > > > >  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> > > > > +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> > > > > +			asm volatile("ic ialluis");
> > > > 
> > > > Hmm, one oddity here is that you can pass a faulting address and not see
> > > > the fault. It looks like that's already IMP DEF, so it's probably ok, but
> > > > might be worth a comment.
> > > 
> > > I am not sure what should be expected behavior, but I could
> > > add comment, sure.
> > 
> > Another option is to make this:
> > 
> > 	 case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
> > 	 	__user_cache_maint("ic ivau", address, ret)
> > 		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret)
> > 			asm volatile("ic ialluis");
> > 		break;
> > 
> > Which'll ensure that if the regular IC IVAU faults we'll handle that, and if
> > not we'll do the IC IALLUIS.
> > 
> > I think that looks a bit cleaner, too.
> 
> What I am afraid is that "ic ivau address" will do cache invalidation of a 
> random address, because of wrong address wiring. So the end result will not
> be what should be expected.

I don't follow:

 - The faulting logic is entirely local (and has to happen before the
   broadcast), so the faulting logic shouldn't be affected by the erratum and
   will use the correct address.

 - We're going to do an IC IALLUIS anyway, so we're going to do invalidation of
   *every* address. So randomly invalidating a random address at the same time
   isn't going to cause a functiona problem because we're going to invaldiate
   it anyway.

> > > > Finally, how come you don't need to upgrade I-cache invalidation by-VA
> > > > in the kernel? It looks like you're only handling operations trapped
> > > > from EL0.
> > > 
> > > Hm, I was thinking that __tlbi() is taking care for this or you mean 
> > > something else, like locations in assembler.h?
> > 
> > The __tlbi macro handles only TLBI instructions.
> > 
> > The trap handler above *only* handles IC instructions trapped from userspace;
> 
> Yep, I get this.
> 
> > we have IC IVAU instructions elsewhere in the kernel (e.g.
> > arch/arm64/mm/cache.S).
> 
> But I have missed this one :-)

Looking again, those all use the common invalidate_icache_by_line macro in
arch/arm64/include/asm/assembler.h

Lukily that seems to be all:

| [mark@lakrids:~/src/linux]% git grep -iw ivau -- arch/arm64 
| arch/arm64/include/asm/assembler.h:     ic      ivau, \tmp2                     // invalidate I line PoU
| arch/arm64/kernel/traps.c:      case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
| arch/arm64/kernel/traps.c:              __user_cache_maint("ic ivau", address, ret);

> I think that this is working because these are used only in operations witch
> work up to the Point of Unification, thus not messing up with caches of the
> rest of PE.

I think you have a misunderstanding of the architecture, because that doesn't
make sense. All IC IVAU operations operate to the Point of Unification. That's
what the 'U' in 'IVAU' stands for. The Point of Unification is the point in the
memory system where instruction fetches and data fetches see the same thing.
 
If the VA passed to IC IVAU isn't correctly broadcast, it's broken regardless
of where that IC IVAU is executed from, because it will leave stale
instructions in the I-caches of other PEs.

Thanks,
Mark.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

On 06-08 16:32, Mark Rutland wrote:
> On Thu, Jun 08, 2023 at 06:05:54PM +0300, Ivan T. Ivanov wrote:
> > On 06-08 15:16, Mark Rutland wrote:
> > > On Thu, Jun 08, 2023 at 04:39:29PM +0300, Ivan T. Ivanov wrote:
> > > > On 06-02 11:34, Will Deacon wrote:
> > > > > On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> 
> > > > > > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > > > > > index 4a79ba100799..265b6334291b 100644
> > > > > > --- a/arch/arm64/kernel/traps.c
> > > > > > +++ b/arch/arm64/kernel/traps.c
> > > > > > @@ -556,6 +556,11 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs)
> > > > > >  		__user_cache_maint("dc civac", address, ret);
> > > > > >  		break;
> > > > > >  	case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:	/* IC IVAU */
> > > > > > +		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104)) {
> > > > > > +			asm volatile("ic ialluis");
> > > > > 
> > > > > Hmm, one oddity here is that you can pass a faulting address and not see
> > > > > the fault. It looks like that's already IMP DEF, so it's probably ok, but
> > > > > might be worth a comment.
> > > > 
> > > > I am not sure what should be expected behavior, but I could
> > > > add comment, sure.
> > > 
> > > Another option is to make this:
> > > 
> > > 	 case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
> > > 	 	__user_cache_maint("ic ivau", address, ret)
> > > 		if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret)
> > > 			asm volatile("ic ialluis");
> > > 		break;
> > > 
> > > Which'll ensure that if the regular IC IVAU faults we'll handle that, and if
> > > not we'll do the IC IALLUIS.
> > > 
> > > I think that looks a bit cleaner, too.
> > 
> > What I am afraid is that "ic ivau address" will do cache invalidation of a 
> > random address, because of wrong address wiring. So the end result will not
> > be what should be expected.
> 
> I don't follow:
> 
>  - The faulting logic is entirely local (and has to happen before the
>    broadcast), so the faulting logic shouldn't be affected by the erratum and
>    will use the correct address.

Perhaps is all me really not understanding PoU, so please 
bear with me :-). This SoC hasi two CPU clusters.

4x Cortex A53
• 32KB-I and 32KB-D cache
• 1MB Shared L2 Cache

2x Cortex A72
• 48KB-I and 32KB-D cache
• 1MB Shared L2 Cache

If it is all local then how far this broadcast will go? Up to the
L2 cache? But then errata says that: ".. maintenance operations 
exchanged between the A53 and A72 core clusters.. " are affected.
So in case of IC IAVU there is no interaction between clusters?

> 
>  - We're going to do an IC IALLUIS anyway, so we're going to do invalidation of
>    *every* address. So randomly invalidating a random address at the same time
>    isn't going to cause a functiona problem because we're going to invaldiate
>    it anyway.

Yes, but my point was that this random address might generate a translation
error where there shouldn't be one.

> 
> > > > > Finally, how come you don't need to upgrade I-cache invalidation by-VA
> > > > > in the kernel? It looks like you're only handling operations trapped
> > > > > from EL0.
> > > > 
> > > > Hm, I was thinking that __tlbi() is taking care for this or you mean 
> > > > something else, like locations in assembler.h?
> > > 
> > > The __tlbi macro handles only TLBI instructions.
> > > 
> > > The trap handler above *only* handles IC instructions trapped from userspace;
> > 
> > Yep, I get this.
> > 
> > > we have IC IVAU instructions elsewhere in the kernel (e.g.
> > > arch/arm64/mm/cache.S).
> > 
> > But I have missed this one :-)
> 
> Looking again, those all use the common invalidate_icache_by_line macro in
> arch/arm64/include/asm/assembler.h
> 
> Lukily that seems to be all:
> 
> | [mark@lakrids:~/src/linux]% git grep -iw ivau -- arch/arm64 
> | arch/arm64/include/asm/assembler.h:     ic      ivau, \tmp2                     // invalidate I line PoU
> | arch/arm64/kernel/traps.c:      case ESR_ELx_SYS64_ISS_CRM_IC_IVAU:     /* IC IVAU */
> | arch/arm64/kernel/traps.c:              __user_cache_maint("ic ivau", address, ret);
> 
> > I think that this is working because these are used only in operations witch
> > work up to the Point of Unification, thus not messing up with caches of the
> > rest of PE.
> 
> I think you have a misunderstanding of the architecture, because that doesn't
> make sense. All IC IVAU operations operate to the Point of Unification. That's
> what the 'U' in 'IVAU' stands for. The Point of Unification is the point in the
> memory system where instruction fetches and data fetches see the same thing.
>  
> If the VA passed to IC IVAU isn't correctly broadcast, it's broken regardless
> of where that IC IVAU is executed from, because it will leave stale
> instructions in the I-caches of other PEs.

I am totally confused :-) If this is true, and I am not saying it is not,
then we can't expect that IC IVAU will generate correct translation might
fault, no?

Regards,
Ivan


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Will Deacon]

On Mon, Jun 12, 2023 at 01:22:37PM +0300, Ivan T. Ivanov wrote:
> On 06-12 10:33, Mark Rutland wrote:
> > I'm saying that the *faulting logic* is local, not the broadcast.
> > 
> > IIUC the erratum is due to the wiring between clusters losing some bits. The
> > *recipient* of a broadcast DVM message (which is what TLBI or IC instructions
> > will generate) will receive a bogus address (and other context) to use for the
> > invalidate.
> > 
> > The CPU which executes the `IC IVAU <Xt>` instruction will check the address in
> > `<Xt>` using its local MMU to determine whether the access should fault
> > *before* sending the broadcast DVM message, and the recipients will not perform
> > any MMU check (since e.g. they could be running a different process with a
> > different VA space anyway).
> > 
> > The MMU check is local to the CPU, and doesn't depend on any broadcast; that
> > should be unaffected by the erratum. If that is affected then the erratum
> > description is wrong and we have a bigger set of problems.
> > 
> 
> Thanks, I think I get it. Now, what will be preferred way to fix IC
> TLBI instructions for this errata? Using static_key for TLBI and
> alternatives for IC instruction or something else?

As I mentioned in a previous comment, I think this should use a static_key
to drive the high-level behaviour instead of patching the low-level code
with alternatives.

> Keep trapping userspace IC instruction seems ok to me. Perhaps
> alternative for IC in invalidate_icache_by_line macro? About
> TLBI, it will be really invasive if static_key is used, I think.
> Yes, there is over invalidation, but still overall performance of
> SoC is almost doubled because of enabled 2 CA72 :-)
> 
> Another point is, should I keep hunk in arm-smmu.c, because
> cpus_have_final_cap() is not generally available for drivers?

What do you mean here? We obviously can't break the build, but I think
the easiest thing to do is add a clause to `arm_smmu_sva_supported()`
to return false if this erratum is present.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

On 06-26 13:15, Will Deacon wrote:
> > 
> > Another point is, should I keep hunk in arm-smmu.c, because
> > cpus_have_final_cap() is not generally available for drivers?
> 
> What do you mean here? We obviously can't break the build,

Build breaks on x86_64 for example. I will do what Robin suggest
in the other email.

Regards,
Ivan

> but I think
> the easiest thing to do is add a clause to `arm_smmu_sva_supported()`
> to return false if this erratum is present.
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Francesco Dolcini]

Hello Ivan,

On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
> According to NXP errata document[1] i.MX8QuadMax SoC suffers from
> serious cache coherence issue. It was also mentioned in initial
> support[2] for imx8qm mek machine.

Ivan, from what I can see this patch was never merged and there was not any
follow-up on this. What's the status? Are you planning to work on it?

Frank, it would be nice to have this issue sorted out in mainline, any
chance that NXP can help move this forward?

Francesco

Re: [PATCH v2] arm64: errata: Add NXP iMX8QM workaround for A53 Cache coherency issue

[Ivan T. Ivanov]

Hi,

> On 10 Dec 2025, at 9:45, Francesco Dolcini <francesco@dolcini.it> wrote:
> 
> Hello Ivan,
> 
> On Thu, Apr 20, 2023 at 02:29:52PM +0300, Ivan T. Ivanov wrote:
>> According to NXP errata document[1] i.MX8QuadMax SoC suffers from
>> serious cache coherence issue. It was also mentioned in initial
>> support[2] for imx8qm mek machine.
> 
> Ivan, from what I can see this patch was never merged and there was not any
> follow-up on this. What's the status? Are you planning to work on it?

Yep, it was put in low priority list and it was forgotten successfully.

I don't have plans to work on this in foreseeable future.

Regards,
Ivan