[PATCH v2 3/3] arm64: use generic MMIO accessors

[PATCH v2 3/3] arm64: use generic MMIO accessors

[Qingfang Deng]

Use the generic raw write accessors in place of the inline assembly
stores. Include asm-generic/mmio.h after the arm64 ordering hooks so
these accessors are also available to the write-combining helpers
before asm-generic/io.h is included.

Retain the raw read overrides when either the Cortex-A57 erratum 832075
or NVIDIA Olympus erratum 1027 workaround is configured. This preserves
the alternative-patched load-acquire instructions and pre-load barriers.
Use generic raw reads when neither workaround is enabled.

Assisted-by: Codex:gpt-6-astra
Signed-off-by: Qingfang Deng <qingfang.deng@linux.dev>
---
v2: new patch
---
 arch/arm64/include/asm/io.h | 37 ++++++-------------------------------
 1 file changed, 6 insertions(+), 31 deletions(-)

diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 49a7002661a9..2fdac25bc31b 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -19,37 +19,9 @@
 #include <asm/cpufeature.h>
 #include <asm/rsi.h>
 
-/*
- * Generic IO read/write.  These perform native-endian accesses.
- */
-#define __raw_writeb __raw_writeb
-static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr)
-{
-	volatile u8 __iomem *ptr = addr;
-	asm volatile("strb %w0, %1" : : "rZ" (val), "Qo" (*ptr));
-}
-
-#define __raw_writew __raw_writew
-static __always_inline void __raw_writew(u16 val, volatile void __iomem *addr)
-{
-	volatile u16 __iomem *ptr = addr;
-	asm volatile("strh %w0, %1" : : "rZ" (val), "Qo" (*ptr));
-}
-
-#define __raw_writel __raw_writel
-static __always_inline void __raw_writel(u32 val, volatile void __iomem *addr)
-{
-	volatile u32 __iomem *ptr = addr;
-	asm volatile("str %w0, %1" : : "rZ" (val), "Qo" (*ptr));
-}
-
-#define __raw_writeq __raw_writeq
-static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr)
-{
-	volatile u64 __iomem *ptr = addr;
-	asm volatile("str %x0, %1" : : "rZ" (val), "Qo" (*ptr));
-}
-
+#if defined(CONFIG_ARM64_ERRATUM_832075) || \
+	defined(CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM)
+/* Keep the raw read overrides for the device-load erratum alternatives. */
 #define __raw_readb __raw_readb
 static __always_inline u8 __raw_readb(const volatile void __iomem *addr)
 {
@@ -102,6 +74,7 @@ static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
 		     : "=r" (val) : "r" (addr));
 	return val;
 }
+#endif
 
 /* IO barriers */
 #define __io_ar(v)							\
@@ -125,6 +98,8 @@ static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
 #define __io_br(v)
 #define __io_aw(v)
 
+#include <asm-generic/mmio.h>
+
 /* arm64-specific, don't use in portable drivers */
 #define __iormb(v)		__io_ar(v)
 #define __iowmb()		__io_bw()
-- 
2.43.0

Re: [PATCH v2 3/3] arm64: use generic MMIO accessors

[Arnd Bergmann]

On Sun, Sep 20, 2026, at 03:43, Qingfang Deng wrote:
> Use the generic raw write accessors in place of the inline assembly
> stores. Include asm-generic/mmio.h after the arm64 ordering hooks so
> these accessors are also available to the write-combining helpers
> before asm-generic/io.h is included.
>
> Retain the raw read overrides when either the Cortex-A57 erratum 832075
> or NVIDIA Olympus erratum 1027 workaround is configured. This preserves
> the alternative-patched load-acquire instructions and pre-load barriers.
> Use generic raw reads when neither workaround is enabled.
>
> Assisted-by: Codex:gpt-6-astra
> Signed-off-by: Qingfang Deng <qingfang.deng@linux.dev>

Sorry, this is no good, we need the compiler to issue the exact
instructions from the inline asm here, both in order to guarantee
these can be virtualized, and to prevent it from splitting up
larger register accesses into byte load/store on unaligned pointers.

     Arnd

Re: [PATCH v2 3/3] arm64: use generic MMIO accessors

[Qingfang Deng]

Hi,

On 2026/9/20 16:08, Arnd Bergmann wrote:
> On Sun, Sep 20, 2026, at 03:43, Qingfang Deng wrote:
>> Use the generic raw write accessors in place of the inline assembly
>> stores. Include asm-generic/mmio.h after the arm64 ordering hooks so
>> these accessors are also available to the write-combining helpers
>> before asm-generic/io.h is included.
>>
>> Retain the raw read overrides when either the Cortex-A57 erratum 832075
>> or NVIDIA Olympus erratum 1027 workaround is configured. This preserves
>> the alternative-patched load-acquire instructions and pre-load barriers.
>> Use generic raw reads when neither workaround is enabled.
>>
>> Assisted-by: Codex:gpt-6-astra
>> Signed-off-by: Qingfang Deng <qingfang.deng@linux.dev>
> Sorry, this is no good, we need the compiler to issue the exact
> instructions from the inline asm here, both in order to guarantee
> these can be virtualized, and to prevent it from splitting up
> larger register accesses into byte load/store on unaligned pointers.

You're right about virtualization: an AArch64 compiler can generate 
writeback addressing loads/stores, which KVM's MMIO handler does not 
support.

Regarding unaligned accesses, the splitting can only happen when the 
address is known to be unaligned at compile-time. Do these accessors 
need to support unaligned MMIO? If so, I could add 
OPTIMIZER_HIDE_VAR(addr) to the generic accessors.

Best regards,
Qingfang

Re: [PATCH v2 3/3] arm64: use generic MMIO accessors

[Arnd Bergmann]

On Sun, Sep 20, 2026, at 11:24, Qingfang Deng wrote:
>
> Regarding unaligned accesses, the splitting can only happen when the 
> address is known to be unaligned at compile-time. Do these accessors 
> need to support unaligned MMIO? If so, I could add 
> OPTIMIZER_HIDE_VAR(addr) to the generic accessors.

I think all architectures that have had this problem just started
using the inline asm variant, and I'd probably try to encourage
that more.

What might be helpful here would be to split out the __raw_* helpers
into a separate header that is explicitly included on architectures
without their own definitions, and an updated comment that explains
that architectures are encouraged to implement these as inline asm
instead of using the volatile load/store.

     Arnd

Re: [PATCH v2 3/3] arm64: use generic MMIO accessors

[Arnd Bergmann]

On Sun, Sep 20, 2026, at 11:30, Arnd Bergmann wrote:
> On Sun, Sep 20, 2026, at 11:24, Qingfang Deng wrote:
>>
>> Regarding unaligned accesses, the splitting can only happen when the 
>> address is known to be unaligned at compile-time. Do these accessors 
>> need to support unaligned MMIO? If so, I could add 
>> OPTIMIZER_HIDE_VAR(addr) to the generic accessors.
>
> I think all architectures that have had this problem just started
> using the inline asm variant, and I'd probably try to encourage
> that more.

Also, I have only seen this problem in combination with driver bugs
that define their MMIO registers as a __packed data structure
instead of the usual 'void __iomem*' with constant offset or a
normal structure without packing.

     Arnd