Re: [PATCH v4 1/2] arm64: errata: Workaround NVIDIA Olympus device store/load ordering

[Shanker Donthineni <sdonthineni@nvidia.com>]

Hi Will,

On 6/30/2026 9:21 AM, Will Deacon wrote:

> External email: Use caution opening links or attachments
>
>
> On Thu, Jun 25, 2026 at 01:24:24PM -0500, Shanker Donthineni wrote:
>> On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
>> observed by a peripheral before an older, non-overlapping Device-nGnR*
>> store to the same peripheral. This breaks the program-order guarantee
>> that software expects for Device-nGnR* accesses and can leave a
>> peripheral in an incorrect state, as a load is observed before an
>> earlier store takes effect.
>>
>> The erratum can occur only when all of the following apply:
>>
>>    - A PE executes a Device-nGnR* store followed by a younger
>>      Device-nGnR* load.
>>    - The store is not a store-release.
>>    - The accesses target the same peripheral and do not overlap in bytes.
>>    - There is at most one intervening Device-nGnR* store in program
>>      order, and there are no intervening Device-nGnR* loads.
>>    - There is no DSB, and no DMB that orders loads, between the store and
>>      the load.
> Does that mean that a DMB LD between the store and the load would
> solve the problem?

I appreciate your suggestion to leave __raw_writeX() unchanged and apply
the workaround on the read side. It results in a much better
approach, write-combining performance is preserved and avoids dgh()
promotion to dmb.

The hardware team has confirmed that DMB OSH between the store and load
prevents T410-OLY-1027. They are still validating whether DMB LD is
sufficient, so I do not want to rely on DMB LD until that confirmation is
available.

> It would be interesting to see how your benchmarks motivating patch 2
> look if you leave __raw_writeX as-is and instead add a barrier in
> __raw_readX before the load instruction.

I profiled memcpy_fromio() after implementing your suggested read-side
workaround using DMB OSH:
   - Patch 1 leaves __raw_writeX() unchanged and inserts DMB OSH before
     each load in __raw_readX().
   - Patch 2 provides an arm64 memcpy_fromio() implementation that applies
     one DMB OSH before the block copy and then uses direct Device loads.

With patch 2 applying the barrier once per block, the results show no
noticeable performance regression when the workaround is active. The
micro-benchmark uses a write-combined MMIO buffer and is pinned to one
PE. The loop count is adjusted so each row performs approximately 10,000
64-bit MMIO loads in total. The table reports the per-call latency of
memcpy_fromio() in nanoseconds, with CPU cycles measured using the PMU
cycle counter shown in parentheses.

+-------+--------------------+----------------------+------------------------+
|  size | WAR off ns (cyc)   | OSH/load P1 ns (cyc) | OSH/block P1+P2 ns(cyc)|
+-------+--------------------+----------------------+------------------------+
|    8B |       830.4 (2735) |         835.0 (2750) |           835.1 (2750) |
|   16B |      1660.1 (5468) |        1669.6 (5499) |          1664.8 (5484) |
|   32B |     3319.7 (10934) |       3339.1 (10998) |         3324.1 (10953) |
|   64B |     6638.6 (21866) |       6677.5 (21994) |         6642.3 (21880) |
|  128B |    13275.8 (43729) |      13355.3 (43989) |        13279.0 (43747) |
|  256B |    26549.7 (87480) |      26714.5 (87993) |        26552.5 (87475) |
+-------+--------------------+----------------------+------------------------+

Micro-bench test:
        local_irq_save(flags);
        off = 0U;
        c0 = wc_pmu_read();
        t0 = ktime_get();
        for (i = 0UL; i < 10000; i++) {
                memcpy_fromio(dst, map + off, n * sizeof(u64));
                off ^= buf_size;
        }
        t1 = ktime_get();
        c1 = wc_pmu_read();
        local_irq_restore(flags);

Patch 2:
void memcpy_fromio(void *dst, const volatile void __iomem *src, size_t count)
{
...
     asm volatile(ALTERNATIVE("nop", "dmb osh",
                  ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
              : : : "memory");

     while (count &&
            !IS_ALIGNED((__force unsigned long)src, sizeof(u64))) {
         u8 val;
         asm volatile(ALTERNATIVE("ldrb %w0, [%1]",
                      "ldarb %w0, [%1]",
                      ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
                  : "=r" (val) : "r" (src));
         *(u8 *)dst = val;
         src++;
         dst++;
         count--;
     }
     while (count >= sizeof(u64)) {
         u64 val;
         asm volatile(ALTERNATIVE("ldr %0, [%1]",
                      "ldar %0, [%1]",
                      ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
                  : "=r" (val) : "r" (src));
         *(u64 *)dst = val;
         src += sizeof(u64);
         dst += sizeof(u64);
         count -= sizeof(u64);
     }
     while (count) {
         u8 val;
         asm volatile(ALTERNATIVE("ldrb %w0, [%1]",
                      "ldarb %w0, [%1]",
                      ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
                  : "=r" (val) : "r" (src));
         *(u8 *)dst = val;
         src++;
         dst++;
         count--;
     }
}

I am also discussing with the hardware team to understand any broader
implications of using a read-side DMB instead of store-release writes,
and to evaluate the correctness and performance differences between
DMB OSH and DMB LD. If we proceed with the load-side workaround, I will
drop patch 2 and keep the implementation limited to the raw read
helpers. I will post v5 after receiving their feedback.

-Shanker