[RFC PATCH 0/1] target/arm: Fix FEAT_HADFS when page tables are in MMIO (cxl)

[RFC PATCH 0/1] target/arm: Fix FEAT_HADFS when page tables are in MMIO (cxl)

[Jonathan Cameron via]

https://lore.kernel.org/qemu-devel/20240215150133.2088-1-Jonathan.Cameron@huawei.com/
gives more details of the reason the CXL emulation uses MMIO space
(short answer is it has very fine grained address routing).

Obviously this one is lower priority for me to upstream as the ARM support in
general is only in my CXL staging tree (I'll get back to that soonish).

The equivalent tests with CXL memory and forcing applications to run from it
on ARM64 TCG ran into an additional issue.  We need to be able to do
atomic compare and swap to update the access and dirty flags as part of
a page walk if we have the ARM 8.1 FEAT_HADFS (note you can disable this in
the kernel then this problem isn't hit but that's not a viable long term
solution though it did help me diagnose the problem)

As the patch says I'm far from confident on this fix. Whilst it 'works'
for little endian at least I'm hoping someone more familiar with the locking
requirements etc can let me know if the BQL is the right option here and
people can give a general opinion on whether such a hack is the right
solution.

Jonathan Cameron (1):
  arm/ptw: Handle atomic updates of page tables entries in MMIO during
    PTW.

 target/arm/ptw.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

-- 
2.39.2

[RFC PATCH 1/1] arm/ptw: Handle atomic updates of page tables entries in MMIO during PTW.

[Jonathan Cameron via]

I'm far from confident this handling here is correct. Hence
RFC.  In particular not sure on what locks I should hold for this
to be even moderately safe.

The function already appears to be inconsistent in what it returns
as the CONFIG_ATOMIC64 block returns the endian converted 'eventual'
value of the cmpxchg whereas the TCG_OVERSIZED_GUEST case returns
the previous value.

Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 target/arm/ptw.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/target/arm/ptw.c b/target/arm/ptw.c
index 5eb3577bcd..37ddb4a4db 100644
--- a/target/arm/ptw.c
+++ b/target/arm/ptw.c
@@ -711,8 +711,38 @@ static uint64_t arm_casq_ptw(CPUARMState *env, uint64_t old_val,
     void *host = ptw->out_host;
 
     if (unlikely(!host)) {
-        fi->type = ARMFault_UnsuppAtomicUpdate;
-        return 0;
+        /* Can I do a load and store via the physical address */
+
+        bool locked = bql_locked();
+        if (!locked) {
+            bql_lock();
+        }
+        /* Page table in MMIO Memory Region */
+        if (ptw->out_be) {
+            old_val = cpu_to_be64(old_val);
+            new_val = cpu_to_be64(new_val);
+            cpu_physical_memory_read(ptw->out_phys, &cur_val, 8);
+            if (cur_val == old_val) {
+                cpu_physical_memory_write(ptw->out_phys, &new_val, 8);
+                cur_val = be64_to_cpu(new_val);
+            } else {
+                cur_val = be64_to_cpu(cur_val);
+            }
+        } else {
+            old_val = cpu_to_le64(old_val);
+            new_val = cpu_to_le64(new_val);
+            cpu_physical_memory_read(ptw->out_phys, &cur_val, 8);
+            if (cur_val == old_val) {
+                cpu_physical_memory_write(ptw->out_phys, &new_val, 8);
+                cur_val = le64_to_cpu(new_val);
+            } else {
+                cur_val = le64_to_cpu(cur_val);
+            }
+        }
+        if (!locked) {
+            bql_unlock();
+        }
+        return cur_val;
     }
 
     /*
-- 
2.39.2

Re: [RFC PATCH 1/1] arm/ptw: Handle atomic updates of page tables entries in MMIO during PTW.

[Peter Maydell]

On Thu, 15 Feb 2024 at 15:18, Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> I'm far from confident this handling here is correct. Hence
> RFC.  In particular not sure on what locks I should hold for this
> to be even moderately safe.

> The function already appears to be inconsistent in what it returns
> as the CONFIG_ATOMIC64 block returns the endian converted 'eventual'
> value of the cmpxchg whereas the TCG_OVERSIZED_GUEST case returns
> the previous value.
>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  target/arm/ptw.c | 34 ++++++++++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)
>
> diff --git a/target/arm/ptw.c b/target/arm/ptw.c
> index 5eb3577bcd..37ddb4a4db 100644
> --- a/target/arm/ptw.c
> +++ b/target/arm/ptw.c
> @@ -711,8 +711,38 @@ static uint64_t arm_casq_ptw(CPUARMState *env, uint64_t old_val,
>      void *host = ptw->out_host;
>
>      if (unlikely(!host)) {
> -        fi->type = ARMFault_UnsuppAtomicUpdate;
> -        return 0;
> +        /* Can I do a load and store via the physical address */
> +
> +        bool locked = bql_locked();
> +        if (!locked) {
> +            bql_lock();
> +        }
> +        /* Page table in MMIO Memory Region */
> +        if (ptw->out_be) {
> +            old_val = cpu_to_be64(old_val);
> +            new_val = cpu_to_be64(new_val);
> +            cpu_physical_memory_read(ptw->out_phys, &cur_val, 8);

This is definitely wrong because it takes no account of
address spaces. You need to also account for ptw->out_space.
Compare what arm_ldq_ptw() does.

> +            if (cur_val == old_val) {
> +                cpu_physical_memory_write(ptw->out_phys, &new_val, 8);
> +                cur_val = be64_to_cpu(new_val);
> +            } else {
> +                cur_val = be64_to_cpu(cur_val);
> +            }
> +        } else {
> +            old_val = cpu_to_le64(old_val);
> +            new_val = cpu_to_le64(new_val);
> +            cpu_physical_memory_read(ptw->out_phys, &cur_val, 8);
> +            if (cur_val == old_val) {
> +                cpu_physical_memory_write(ptw->out_phys, &new_val, 8);
> +                cur_val = le64_to_cpu(new_val);
> +            } else {
> +                cur_val = le64_to_cpu(cur_val);
> +            }
> +        }
> +        if (!locked) {
> +            bql_unlock();
> +        }
> +        return cur_val;
>      }


thanks
-- PMM